Last night at 9:31pm Pacific Time, the database that stores email messages in the process of being delivered corrupted itself. I don't know how. Because of that, we stopped processing/delivering new email messages. We were able to continue retrying existing messages. This problem was not picked up on our monitoring systems; we were still delivering email, just not new ones. I fixed the problem and restarted email delivery at 7:50am Pacific Time this morning. We're now working through that immense backlog.
I have already added additional monitoring that will catch this in the future, and will be looking at other ways to ensure this doesn't happen again.
