I found this little guy growing in the garden yesterday and would like to find out what it is. The leaves have a slight coarse texture.

Could it be a dogwood seedling?

When I looked at the first image my brain also said "Dogwood". Hope it is.

Dogwood was also the first thing that came to mind when looking at it. I wasn't sure if dogwood leaves, shrub or otherwise had a coarse texture to them. I was also unsure of the purple stem. I too would hope it's a dogwood.

It may be a Cornus sericea

I thought of some others, like weigela or diervilla, but I keep coming back to dogwood. But I am not entirely convinced. It would have to be a species with very short leaf stems. That would rule out Cornus sericea. I'm not familiar enough with C. florida or C. kousa to know if they are good candidates.