I like the concept of paid users conserving the number of free (or group-paid) slots for other users who can afford it. However, here are some other things that need to be thought about:
Easy: Any group requiring approval for new members needs an indicator that an incoming member is paid, so they know approval won't cost anything.
More difficult: What happens if someone stops paying? There are multiple complications with this. You'd think for a free group, if there are enough free slots available, they can stay, otherwise remove them automatically. But what if the newly non-paying member is an owner? What if they're the group's only owner? Maybe that situation is rare enough that it'd be OK for groups.io support to clean up any resulting mess if the non-payment was accidental. For premium groups, there may need to be an option for this: Start paying for the user (if they're over the limit), or kick them out?
I don't think there needs to be a free trial period as long as groups like beta and GMF exist and allow unlimited free members. If someone wants to join a group that will require a fee but isn't sure they like groups.io, they can be directed to one or more of these to get the hang of what groups are like. Alternatively, you could allow a one-time refund if a user is dissatisfied within 5 days.
You may have someone wanting a "family plan" where several household members could be paid from one account, but that wouldn't have to be available right away.
Once again, just because I come up with issues doesn't mean I don't think it's a good idea. :-)