Yes, you are being paranoid.
Stop lugging it in and out. The temperatures are fine for your tomato, especially in a protected area. But, the continuous shifting of the soil (as you lug it in and out) is probably playing havoc with the roots.
The fertilizer in the soil may not last 6 months as tomatoes are heavy feeders. When it starts yellowing a wee bit, fertilize but as its in a container, mix the fertilizer at 1/2 strength with a well balanced fertilizer formulated for tomato plants.
If you are growing in a container and using commercially produced soil, you needed worry about all those letters and numbers behind the name - the bagged soil is clean.