It's a lot of fun!
I too grow cacti in containers but the majority of my plants are in the ground. I find it far easier than container gardening because the plant can be left to its own devices and do what comes 'naturally'. I never have to 'repot' it if it gets too big. I never have to change the soil. I do not have to struggle lifting heavy pots to 'relocate it' to a sunnier or shadier spot. I may have to put some shade tent over it but that's easy.
Just water when needed, fertilize if you desire and treat for the appropriate pests if you have to. Let 'Mother Nature' do the rest and enjoy.

It is also a joy because it invites butterflies, bees, hummingbirds, etc. into the yard and I've made many a friend chatting with neighbors who have just stopped to admire and/or ask questions as they pass and see me at work (usually I'm pulling weeds

).
I realize a lot of people don't think they have the room in their yards to spare or may only have the tiniest of outdoor spaces but that shouldn't stop you from gardening outside. You do not need a 'huge' space or a big yard to have a lovely cacti garden. Just follow a few simple rules:
1.) Grow what will grow and be happy where you plant it.
2.) Pick the right plant for the right spot.
3.) Get creative and have fun.
Here is a tiny little spot along the walkway of my house. It measures only 16 inches deep but is rather long.
I had no idea what to do with this space. Obviously my beloved Agaves, pole cacti and p.pears that would become huge wouldn't fit here. So, I saw a bunch of tiny cacti in a bargain bin at a local Big Box store that I fell in love with. The little round globular ones that I call 'bubbles' and created my 'bubble garden' out of this space. Each little cacti in here was probably between $2.50- $8.00 max. Goes to show you that you can also have a lot of plant purchasing power with very little money as well.
Two and a half years later, this is what I've wound up with. I get a variety of blooms throughout the year from these Echinopsis, Parodia, Gymnocalycium, Chamaelobivia, Mammillarias and others similar.
Another reason I like gardening 'in ground' is there are no restrictions when it comes to the size of the plant you can grow or its potential if you have the room.
This was taken 6-19-2016. Notice the size of the pole cacti and the Eve's Needles.

This picture was taken 3-25-18. Notice the size of the pole cacti and the Eve's Needles!

And they're still growing! (And you can see the changes and additions I've made too.

)