I occasionally get into the dirt without gloves, but mostly if I'm seriously going to get anything done, the gloves must go on! It's a mental thing with me, putting on the gloves means 'time to get to work'. I love those nitrile gloves, too. I get the mesh part wet in the hot weather and it helps to keep me a little cooler.
Another glove phenomenon: in our school garden, the kids really get into it with the gloves on. (we MUST have them wear gloves, since if someone gets injured, bitten or has an allergic attack we risk loss of the whole garden project). But, it's truly magic what putting on a pair of gloves does for a child who has never had their hands in the dirt before. Even the girly-girls get right in there and dig, pick, scout for bugs and generally are so much more engaged. Gloves bridge a gap for the kids, I think.