Don't know if this is the case already, but it should be that you can only have one invite active at a time, so when a random invite comes in, you just move the window out of the way and continue about your business, and meanwhile the inviter is unable to bother anyone else
