Sorry that I won't be of much help here, but I do sympathize. (I'm getting to the point where I am looking to lower garden maintenance chores, and I'm taking a hard look at my irises and daylilies, many of which are in pots.)
If you do decide to downsize, maybe you should keep your few favorites in mixed perennial beds (might be harder for the borers to find? we don't have borers here, thankfully), or else in pots. (Pots are NOT care-free by any means, but it may be less of a weeding chore than a bed. The downside, of course, is that even with a large 5 gal or so pot, you are only going to be getting a few stalks of anything, and you'll have to divide and re-pot every other year or so.)
