moderated Delayed email delivery #outage


 

All,

Last night at 9:31pm Pacific Time, the database that stores email messages in the process of being delivered corrupted itself. I don't know how. Because of that, we stopped processing/delivering new email messages. We were able to continue retrying existing messages. This problem was not picked up on our monitoring systems; we were still delivering email, just not new ones. I fixed the problem and restarted email delivery at 7:50am Pacific Time this morning. We're now working through that immense backlog.

I have already added additional monitoring that will catch this in the future, and will be looking at other ways to ensure this doesn't happen again.

Thanks, Mark

Join main@beta.groups.io to automatically receive all group messages.