It's the combination of the notification going to the person who did the action plus the delay. They compound each other.
I don't find it to be irritating, but you can avoid both issues by choosing Web/app notification instead of email.
Of course that has the downside that you (generally) don't have a record of the events that happened while you were sleeping or otherwise occupied. I say "generally" because your OS or device may have a means to keep and review a log of recent web/app notifications.
