Deleting the invite at the point it's accepted is easy to do. TheI agree with that reason.
In particular it would be confusing (to me as a moderator) if a particularly prompt member accepted the invitation and it disappeared before I'd noticed it appear in the list. Even ignoring that case, I prefer having this nice list of invitations at hand rather than have to dredge them out of the Activity log.
Why do people want to remove accepted Invitations from this list? The list can be sorted by Status or invitation date, isn't that sufficiently tidy?
But maybe I go ahead and delete it then. Then, the only invites thatOr, if you don't, the action could be "Cancel/Remove Invitation" to cover both cases.