Yup, I use them, since some of my succulents have poisonous sap or prickly needles Or even some leaves may have particulates that can easily irritate membranes of your eyes. Sometimes one can unknowingly rub ones eyes, wearing gloves will deter you a bit from doing so. I actually wear those safety glasses to protect my eyes too, at times on a windy day, cannot help some dirt flying around.
Though it really feels nice to feel the soil, but I would rather be safe than sorry.
When it gets too warm outside, but I do not want to wear long sleeves, I wear those white driving arm covers, it also comes in printed colors, easily allows air on my short sleeves, relieves me of heat and some sun protection too while doing yard work. And I would also be wearing a straw hat by then
Our tolerances to the environment are changing too through the years, so got to protect our skin, hands etc.