Absolutely don't give up, just don't take it as a great failure if your plant doesn't look great all of the time. If you want to give it a little fertilizer, that's fine. I'm sure I have before at various times over the years, but only occasionally/sporadically while outside, maybe 3 times in a summer, maybe not at all for a few years. AFAIK, I've never killed any plant by not fertilizing it, but there's a long list of plants no longer alive that seemed to croak from getting too much.
