I typically always wear gloves mostly because I am in an out of the house a lot so gardening is sparse. I dont know if anyone else has thought of this, but when I am doing light work just with dirt in a bed or moving plants around, and seeding, I wear latex/plastic gloves. This way i can keep my hands clean if I need to, but I can also feel the dirt and squishyness of the plants as I work. Then I wear the thicker gardening gloves for using the shovel or pulling up thorny plants to protect my hands against potential injury. So I think it is a win-win with the latex gloves and I can always go without if I dont have any plans that day!
