I would be inquisitive enough to call that real estate agent back and explain what happened. If nothing else, the agent needs to know that he/she made a mistake, giving you access to the yard. If the agent confirms that indeed the owner lives across the street, I would ask her/him to please call the owner and explain that the mistake was his/hers.
