Sounds like you've had what is known as "June drop". Here's a link to a good article about fruit drop. Second page after the table talks about it best.
http://irrec.ifas.ufl.edu/flci...
Can be related to high temperatures alone, big temp fluctuations (which you did have in May, didn't you?) not enough water, or the tree just setting more fruit than it can support.
Doesn't sound like there's much you can do about it except make sure the tree gets enough water, and if you have a big fruit crop coming, that can be a big water demand.
