While Irises are not delicate, the size of the rhizome often will affect the length of time it will take to bloom. It could be years before a small rhizome will bloom. A tall bearded (TB) iris rhizome that is the size of a pea or a dwarf would not make me happy! Will it grow? Yes, but I won't be able to enjoy the bloom for a long time. However, it has happened that I have received rather small rhizomes and had them bloom the following year.
The geographic location where the iris was grown often has an impact on the size of the rhizome. The west coast iris rhizomes are usually larger, but that does not mean that the iris is healthier or better in any way. The East coast and central USA will not grow the big rhizomes, but you should not shy away from buying them. They will still produce the same beautiful blooms.
Remember that the genetics of some TB irises will not allow the rhizomes to be large. Examples of this are the irises by the hybridizer Barry Blyth. The rhizomes never grew bigger than a Standard Dwarf Bearded (SDB) iris rhizome, but they bloomed the following spring.
On the other hand, most of the rhizomes for 'Bev,' by Gerald Richardson, are huge.
Irises generally are very robust plants that will grow and thrive in all types of conditions no matter the size of the rhizome you start with!