It's a lot of work and I had always said I hate gardening, but I discovered a creative outlet at this house I'm in now. Late in life; started while in my mid-40's. This house and lot just challenged me and we were new to Alberta, fresh from beautiful BC and missing that beauty. And I've learned it gets me outside in the fresh air, doing something close to exercise... which is something I would not get otherwise. Those are all positive things. It is expensive and when we were younger we could not have afforded it. As you mentioned, there are always setbacks and those can be expensive. But I'm a practical girl and careful with my money. I (and my husband) have decided the results are worth it. And I don't have any other expensive hobbies. 😝
I would encourage you to keep at it. Some years are better than others. Get help. Do your research on line. Make some gardening friends. It's a healthy endeavour from many points of view (exercise, frame of mind, fresh air, positive out look on life, gives you something to look forward to each new spring, friends with things in common, keeping you away from less healthy things...). Keep it up!