Mark,

We're glad that your daughter recovered from whatever it was.

In Ohio (and some other states), many businesses are closed. You can't
get a haircut. Some stores have special early hours only for seniors,
which I thought was nice. Our business might have to close for a while
but some among us can work from home, at least for now.

A friend in Montreal said that they are working with the doors all
opened and have a disinfectant team working to keep everything
sanitized.

In some places it's crazy. It will get worse before it gets better, so
all we can do is try to stay as healthy as possible so we can weather
this storm.


Donald


----------------------------------------------------
Some ham radio groups you may be interested in:
https://groups.io/g/ICOM
https://groups.io/g/Ham-Antennas
https://groups.io/g/HamRadioHelp

Hi All,

We're through the backlog now and delivery should be more or less back to normal with perhaps some continuing delays for cox.net and comcast customers because we have to strictly limit how many messages we send to those services.

Thanks,

Mark

On Fri, Mar 20, 2020 at 9:33 AM J_Catlady <j.olivia.catlady@...> wrote:

Patience is one thing I tend to sorely lack.;) Thanks.

On Fri, Mar 20, 2020 at 9:27 AM Andy Wedge <andy_wedge@...> wrote:

On Fri, Mar 20, 2020 at 04:23 PM, J_Catlady wrote:

I didn't realize it was still working on catching up.

I'm still getting emails from online posts that I can see were done hours ago so it's obviously still catching up.  Patience is a virtue :-)

Hopefully Mark will confirm when he thinks all is fully OK and we can review then.

Andy

--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu

Patience is one thing I tend to sorely lack.;) Thanks.

On Fri, Mar 20, 2020 at 9:27 AM Andy Wedge <andy_wedge@...> wrote:

On Fri, Mar 20, 2020 at 04:23 PM, J_Catlady wrote:

I didn't realize it was still working on catching up.

I'm still getting emails from online posts that I can see were done hours ago so it's obviously still catching up.  Patience is a virtue :-)

Hopefully Mark will confirm when he thinks all is fully OK and we can review then.

Andy

--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu

On Fri, Mar 20, 2020 at 04:23 PM, J_Catlady wrote:

I didn't realize it was still working on catching up.

I'm still getting emails from online posts that I can see were done hours ago so it's obviously still catching up.  Patience is a virtue :-)

Hopefully Mark will confirm when he thinks all is fully OK and we can review then.

Andy

Ok, I didn't realize it was still working on catching up. Our pending subscription notices and confirmation notices are not going out at all.

On Fri, Mar 20, 2020 at 9:11 AM Andy Wedge <andy_wedge@...> wrote:

On Fri, Mar 20, 2020 at 04:06 PM, J_Catlady wrote:

people are showing no email deliveries in their history.

I had this for a new member I had added. It got updated as messages were delivered. They came out of the normal sequence but they all got there in the end.  I'm sure it'll all be OK once the back end system has caught up.

Andy

--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu

We ran into a problem last night trying to resend some invites, where we were sending a new invite to several people who had been sent invites in the past. We sent new invites to avoid the (previously reported) issue wherein you can't change obsolete custom text when resending. At first the invites did not go through at all, and were logged on the "sent invite" page as "sending" rather than sent. That was still the case this morning, so we resent at that point. When we resent them this morning, they went through. However, and here's the bug: when clicking on the "invited" column to see exactly which ones were (successfully) sent this morning, only three of the six showed up in the correct date order, i.e., at the top, for this morning. The other three, although also sent at the same time, showed up way down in the list, as if ordered by their originally invited dates.

This is a minor problem in the big scheme of things but made it very hard to sort out who successfully received the new invites today and who did not.

Bug: it seems that the date sent for (some? which ones I'm not sure of) invites that were previously sent in the past is not updated when resent.

--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu

On Fri, Mar 20, 2020 at 04:06 PM, J_Catlady wrote:

people are showing no email deliveries in their history.

I had this for a new member I had added. It got updated as messages were delivered. They came out of the normal sequence but they all got there in the end.  I'm sure it'll all be OK once the back end system has caught up.

Andy

This appears to have affected things besides sent messages. Our six invites last night were marked as "sending" instead of "sent" until this morning, causing me to think something else was the cause. I sent a message about that before seeing this, and it's all fixed now and the invites have gone through. (The bug report I sent is still relevant and is about the display order of sent invites.)

On Fri, Mar 20, 2020 at 8:42 AM Mark Fletcher <markf@corp.groups.io> wrote:

All,

Last night at 9:31pm Pacific Time, the database that stores email messages in the process of being delivered corrupted itself. I don't know how. Because of that, we stopped processing/delivering new email messages. We were able to continue retrying existing messages. This problem was not picked up on our monitoring systems; we were still delivering email, just not new ones. I fixed the problem and restarted email delivery at 7:50am Pacific Time this morning. We're now working through that immense backlog.

I have already added additional monitoring that will catch this in the future, and will be looking at other ways to ensure this doesn't happen again.

Thanks, Mark

--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu

This appears to have affected things besides sent messages. Our six invites last night were marked as "sending" instead of "sent" until this morning, causing me to think something else was the cause. I sent a message about that before seeing this, and it's all fixed now and the invites have gone through. (The bug report I sent is still relevant and is about the display order of sent invites.)

On Fri, Mar 20, 2020 at 8:42 AM Mark Fletcher <markf@corp.groups.io> wrote:

All,

Last night at 9:31pm Pacific Time, the database that stores email messages in the process of being delivered corrupted itself. I don't know how. Because of that, we stopped processing/delivering new email messages. We were able to continue retrying existing messages. This problem was not picked up on our monitoring systems; we were still delivering email, just not new ones. I fixed the problem and restarted email delivery at 7:50am Pacific Time this morning. We're now working through that immense backlog.

I have already added additional monitoring that will catch this in the future, and will be looking at other ways to ensure this doesn't happen again.

Thanks, Mark

All,

Last night at 9:31pm Pacific Time, the database that stores email messages in the process of being delivered corrupted itself. I don't know how. Because of that, we stopped processing/delivering new email messages. We were able to continue retrying existing messages. This problem was not picked up on our monitoring systems; we were still delivering email, just not new ones. I fixed the problem and restarted email delivery at 7:50am Pacific Time this morning. We're now working through that immense backlog.

I have already added additional monitoring that will catch this in the future, and will be looking at other ways to ensure this doesn't happen again.

Thanks, Mark