I'm betting hornworm too but I've had many occasions where the caterpillar, not a hornworm, is brown, the same color as the soil & only comes out at night & eats. During the day, they seem to hide on the soil surface maybe making their way into dips & hollows & are the very devil to find as they are so well camouflaged. I don't know what kind they are but they do the kind of damage in your photos.
