1. main@beta.groups.io

moderated Re: Delayed email delivery #outage

J_Catlady
 

This appears to have affected things besides sent messages. Our six invites last night were marked as "sending" instead of "sent" until this morning, causing me to think something else was the cause. I sent a message about that before seeing this, and it's all fixed now and the invites have gone through. (The bug report I sent is still relevant and is about the display order of sent invites.)


On Fri, Mar 20, 2020 at 8:42 AM Mark Fletcher <markf@corp.groups.io> wrote:

All,

Last night at 9:31pm Pacific Time, the database that stores email messages in the process of being delivered corrupted itself. I don't know how. Because of that, we stopped processing/delivering new email messages. We were able to continue retrying existing messages. This problem was not picked up on our monitoring systems; we were still delivering email, just not new ones. I fixed the problem and restarted email delivery at 7:50am Pacific Time this morning. We're now working through that immense backlog.

I have already added additional monitoring that will catch this in the future, and will be looking at other ways to ensure this doesn't happen again.

Thanks, Mark


--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu
Join main@beta.groups.io to automatically receive all group messages.