moderated Delayed email delivery #outage



Last night at 9:31pm Pacific Time, the database that stores email messages in the process of being delivered corrupted itself. I don't know how. Because of that, we stopped processing/delivering new email messages. We were able to continue retrying existing messages. This problem was not picked up on our monitoring systems; we were still delivering email, just not new ones. I fixed the problem and restarted email delivery at 7:50am Pacific Time this morning. We're now working through that immense backlog.

I have already added additional monitoring that will catch this in the future, and will be looking at other ways to ensure this doesn't happen again.

Thanks, Mark

Join to automatically receive all group messages.