Yes, rejecting and handling rejection are tough.
No-one really wants to do that, so we go to meetings that are a waste of time, and kvetch about it in the next meeting.
Of course how about asking up-front — for the invitee— “What do you really need? Maybe I have it.” and for the organiser “I think I need something like this”.
“Like that’s going to happen” ;)
It seems your idea is safe for the time being.