Topics

Delayed email delivery #outage

 

All,

Last night at 9:31pm Pacific Time, the database that stores email messages in the process of being delivered corrupted itself. I don't know how. Because of that, we stopped processing/delivering new email messages. We were able to continue retrying existing messages. This problem was not picked up on our monitoring systems; we were still delivering email, just not new ones. I fixed the problem and restarted email delivery at 7:50am Pacific Time this morning. We're now working through that immense backlog.

I have already added additional monitoring that will catch this in the future, and will be looking at other ways to ensure this doesn't happen again.

Thanks, Mark

 

This appears to have affected things besides sent messages. Our six invites last night were marked as "sending" instead of "sent" until this morning, causing me to think something else was the cause. I sent a message about that before seeing this, and it's all fixed now and the invites have gone through. (The bug report I sent is still relevant and is about the display order of sent invites.)


On Fri, Mar 20, 2020 at 8:42 AM Mark Fletcher <markf@corp.groups.io> wrote:

All,

Last night at 9:31pm Pacific Time, the database that stores email messages in the process of being delivered corrupted itself. I don't know how. Because of that, we stopped processing/delivering new email messages. We were able to continue retrying existing messages. This problem was not picked up on our monitoring systems; we were still delivering email, just not new ones. I fixed the problem and restarted email delivery at 7:50am Pacific Time this morning. We're now working through that immense backlog.

I have already added additional monitoring that will catch this in the future, and will be looking at other ways to ensure this doesn't happen again.

Thanks, Mark


--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu

 

p.s. Mark: Actually I think the display order problem was probably also caused by the same glitch, so you can ignore/delete my bug report about it. Thanks.


On Fri, Mar 20, 2020 at 8:52 AM J_Olivia Catlady <j.olivia.catlady@...> wrote:
This appears to have affected things besides sent messages. Our six invites last night were marked as "sending" instead of "sent" until this morning, causing me to think something else was the cause. I sent a message about that before seeing this, and it's all fixed now and the invites have gone through. (The bug report I sent is still relevant and is about the display order of sent invites.)

On Fri, Mar 20, 2020 at 8:42 AM Mark Fletcher <markf@corp.groups.io> wrote:

All,

Last night at 9:31pm Pacific Time, the database that stores email messages in the process of being delivered corrupted itself. I don't know how. Because of that, we stopped processing/delivering new email messages. We were able to continue retrying existing messages. This problem was not picked up on our monitoring systems; we were still delivering email, just not new ones. I fixed the problem and restarted email delivery at 7:50am Pacific Time this morning. We're now working through that immense backlog.

I have already added additional monitoring that will catch this in the future, and will be looking at other ways to ensure this doesn't happen again.

Thanks, Mark


--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu

 

Mark,I think there might still be a delay. I resent several confirmation emails a few minutes ago and the people are showing no email deliveries in their history.


On Fri, Mar 20, 2020 at 9:05 AM J_Catlady via Groups.Io <j.olivia.catlady=gmail.com@groups.io> wrote:
This appears to have affected things besides sent messages. Our six invites last night were marked as "sending" instead of "sent" until this morning, causing me to think something else was the cause. I sent a message about that before seeing this, and it's all fixed now and the invites have gone through. (The bug report I sent is still relevant and is about the display order of sent invites.)

On Fri, Mar 20, 2020 at 8:42 AM Mark Fletcher <markf@corp.groups.io> wrote:

All,

Last night at 9:31pm Pacific Time, the database that stores email messages in the process of being delivered corrupted itself. I don't know how. Because of that, we stopped processing/delivering new email messages. We were able to continue retrying existing messages. This problem was not picked up on our monitoring systems; we were still delivering email, just not new ones. I fixed the problem and restarted email delivery at 7:50am Pacific Time this morning. We're now working through that immense backlog.

I have already added additional monitoring that will catch this in the future, and will be looking at other ways to ensure this doesn't happen again.

Thanks, Mark


--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu

Andy Wedge
 

On Fri, Mar 20, 2020 at 04:06 PM, J_Catlady wrote:
people are showing no email deliveries in their history.
I had this for a new member I had added. It got updated as messages were delivered. They came out of the normal sequence but they all got there in the end.  I'm sure it'll all be OK once the back end system has caught up.

Andy

 

Ok, I didn't realize it was still working on catching up. Our pending subscription notices and confirmation notices are not going out at all.


On Fri, Mar 20, 2020 at 9:11 AM Andy Wedge <andy_wedge@...> wrote:
On Fri, Mar 20, 2020 at 04:06 PM, J_Catlady wrote:
people are showing no email deliveries in their history.
I had this for a new member I had added. It got updated as messages were delivered. They came out of the normal sequence but they all got there in the end.  I'm sure it'll all be OK once the back end system has caught up.

Andy


--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu

Andy Wedge
 

On Fri, Mar 20, 2020 at 04:23 PM, J_Catlady wrote:
I didn't realize it was still working on catching up.
I'm still getting emails from online posts that I can see were done hours ago so it's obviously still catching up.  Patience is a virtue :-)

Hopefully Mark will confirm when he thinks all is fully OK and we can review then.

Andy

 

FWIW we are now receiving some "subscription approval needed" emails multiple times.


On Fri, Mar 20, 2020 at 9:23 AM J_Catlady via Groups.Io <j.olivia.catlady=gmail.com@groups.io> wrote:
Ok, I didn't realize it was still working on catching up. Our pending subscription notices and confirmation notices are not going out at all.

On Fri, Mar 20, 2020 at 9:11 AM Andy Wedge <andy_wedge@...> wrote:
On Fri, Mar 20, 2020 at 04:06 PM, J_Catlady wrote:
people are showing no email deliveries in their history.
I had this for a new member I had added. It got updated as messages were delivered. They came out of the normal sequence but they all got there in the end.  I'm sure it'll all be OK once the back end system has caught up.

Andy


--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu

Duane
 

On Fri, Mar 20, 2020 at 11:23 AM, J_Catlady wrote:
I didn't realize it was still working on catching up.
Mark's message said, "We're now working through that immense backlog."  Some folks on GMF are reporting that things are starting to come through.  Maybe by tomorrow? ;>)

Duane

 

Patience is one thing I tend to sorely lack.;) Thanks.


On Fri, Mar 20, 2020 at 9:27 AM Andy Wedge <andy_wedge@...> wrote:
On Fri, Mar 20, 2020 at 04:23 PM, J_Catlady wrote:
I didn't realize it was still working on catching up.
I'm still getting emails from online posts that I can see were done hours ago so it's obviously still catching up.  Patience is a virtue :-)

Hopefully Mark will confirm when he thinks all is fully OK and we can review then.

Andy


--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu

 

Hi All,

We're through the backlog now and delivery should be more or less back to normal with perhaps some continuing delays for cox.net and comcast customers because we have to strictly limit how many messages we send to those services.

Thanks,
Mark

On Fri, Mar 20, 2020 at 9:33 AM J_Catlady <j.olivia.catlady@...> wrote:
Patience is one thing I tend to sorely lack.;) Thanks.

On Fri, Mar 20, 2020 at 9:27 AM Andy Wedge <andy_wedge@...> wrote:
On Fri, Mar 20, 2020 at 04:23 PM, J_Catlady wrote:
I didn't realize it was still working on catching up.
I'm still getting emails from online posts that I can see were done hours ago so it's obviously still catching up.  Patience is a virtue :-)

Hopefully Mark will confirm when he thinks all is fully OK and we can review then.

Andy


--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu

 

Mark,

Just letting you know that something is still awry (which may or may not be related to the glitch). About 10 minutes ago, I ran a test with a test member subscribing via email to a restricted group. The member received and succesfully replied to the confirmation email, but the email delivery history still shows "no email deliveries." That would normally cause me to resend it (not knowing they actually got it). Also, they did not receive the pending member email at all. This is a gmail address. 

Hopefully this will all iron itself out eventually. I can send you details offlist if you want them, and/or just wait and hope for the best.

Thanks.



On Fri, Mar 20, 2020 at 10:12 AM Mark Fletcher <markf@corp.groups.io> wrote:
Hi All,

We're through the backlog now and delivery should be more or less back to normal with perhaps some continuing delays for cox.net and comcast customers because we have to strictly limit how many messages we send to those services.

Thanks,
Mark

On Fri, Mar 20, 2020 at 9:33 AM J_Catlady <j.olivia.catlady@...> wrote:
Patience is one thing I tend to sorely lack.;) Thanks.

On Fri, Mar 20, 2020 at 9:27 AM Andy Wedge <andy_wedge@...> wrote:
On Fri, Mar 20, 2020 at 04:23 PM, J_Catlady wrote:
I didn't realize it was still working on catching up.
I'm still getting emails from online posts that I can see were done hours ago so it's obviously still catching up.  Patience is a virtue :-)

Hopefully Mark will confirm when he thinks all is fully OK and we can review then.

Andy


--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu

Mark
 

Yep, I just sent a message to one of my groups via the web interface, and it immediately appeared in the online message list ... but the e-mailed version of the message still hasn't arrived.

Mark

 

It was taking up to three hours as of last night.


On Mar 21, 2020, at 10:48 AM, Mark via Groups.Io <pitamakan@...> wrote:

Yep, I just sent a message to one of my groups via the web interface, and it immediately appeared in the online message list ... but the e-mailed version of the message still hasn't arrived.

Mark

--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu