Kayleigh, fun addition! I kept some inside over winter time for a few yrs and agree, they kept blooming whenever I was able to put them where the sun hit them for at least a few minutes. Your resolve is impressive, I can hardly remove Coleus blooms inside even though I know the little brown bits are going to fall on the floor. Seems so significant inside during winter, any little bloom. If you firmly believe your plant would be more healthy if yours are removed, it's the right thing to do. When plants sense our confident vibes, I think they do better. There's no biological reason to remove them though, if you would prefer to see them, not true annuals at all.
I no longer have Impatiens, they finally caught downy mildew in 2014.
Wax Begonias are another worth mentioning here. They'll bloom steadily all winter with some direct sun. (And yes, drop brown, shriveled old flowers on the floor too.) I haven't brought any in this year, but now that I think of this, I wouldn't be surprised if I found myself outside with the hand trowel, grabbing a few, then trying to find a spot to cram them into an existing pot. I left them all in the ground since they usually survive winter here, but are boring out there (for a few months, after a few cold nights put them in dormancy.)