Being a collector is different from being an accumulator. Though, many gardeners start off as accumulators and at first take any plant that they are offered. Eventually, however, most of us catch on to the demands of our site and learn to say, "No thanks, I don't have enough sun to grow this or that."
Or, "I don't have any space for it."
Or as our resolve stiffens, "I don't want any more black-eyed Susans."
We become very picky, or more specialized, about what we collect or accept as pass-along plants.
At different times in my gardening career I have specialized in collecting various things. I remember trying to collect every hellebore that I could find, and then it was dwarf asters for a while, followed by sedums and hydrangeas. Now that the deer have adopted my garden as their very own salad bar, my rule for collecting plants is very specific. If deer dislike it, then I love it and can't get enough of it.
Thus I have come full circle and after years of scorning offers of bee balm, I am gladly accepting more. The deer have demolished every stem of my sedum, but just look at my wonderful collection of irises! And in May my peonies were breathtaking.
Necessity is indeed the mother of invention, and circumstances shape our preferences and thus our plant collections.