Yes, you're correct: You have an iris garden, which contains many irises.
The names of public gardens usually don't use the plural for any type of flower. If you find it confusing, try comparing it to nouns with undisputed plural forms, such as "rose" and "roses." You'll never hear about a municipal roses garden or a memorial roses garden. The singular form is almost always used in those cases.
