Hi All,

Here's a summary of what happened during the email outage on Monday and the steps I'm taking to address it.

What Happened

I noticed around 9am that the program that sends emails, called karld, had crashed and restarted several times during the night. I had not been paged. Checking the mail queue, it appeared that there were a large number of emails that had not been sent. I began to investigate.

karld was running out of memory and restarting. It would only be able to send a few emails before each restart. I initially believed it was because there were about 600k messages to gmail.com addresses waiting to be sent (of a total of 900k messages in the queue), and we simply didn't have enough memory to support that. At the time I thought it was an issue with delivering email to Gmail that was the main problem.

As a short term measure, I greatly increased the swap on the main email sending machine, to try and clear the message backlog. I also decreased the number of outbound email connections. This started to clear the backlog, but tripped another issue; the machine was now running out of socket resources, which had never happened before. I increased the kernel socket resource limits and that allowed the backlog to continue to drain. But I noticed that email to gmail.com was not being sent in any significant numbers, so the backlog was not draining quickly. Restarting the karld process would cause some gmail.com messages to be sent, so I began restarting the process every few minutes, while I tried to puzzle out what was happening. I discovered two inter-related bugs.

I discovered that a change I made last week introduced a bug where we were including a large amount of extra, unneeded data with every recipient for each message. This greatly increased the memory usage of the system. This did not cause a problem until sometime early Monday morning, when enough messages were in the unsent queue that karld started taking up too much memory.

But that didn't explain the initial reason that messages were not being sent, which I had initially thought was caused by an intermittent Gmail delivery issue. In reality, it was caused by a second bug, also caused by the change I made last week. We have to track how many concurrent connections we have to each email service, because many services restrict the number of connections we can have to them. This second bug was causing an accounting issue where we would lose track of how many open connections we had with each email service. In some cases, we thought we had open connections, when we actually had none. Specifically, this was triggered for Gmail, and it meant that we weren't sending many messages to them. Which caused the large number of unsent messages, which in turn caused the out of memory issue.

I fixed these bugs, and the email queue was back to normal by around 2pm.

Changes Already Made

• I fixed the two bugs at the root of the downtime.
• I re-enabled monitoring of karld, which I had disabled last week for an unrelated issue.
• During the downtime, I discovered that status.groups.io was not working. I fixed the DNS issue causing that and that website is back up.
• I fixed several inaccurate metrics on the karld internal dashboard, which will help diagnose issues in the future.

Changes Still To Be Made

• I am going to accelerate the addition of a second karld instance. Emails from beta are already being sent from the new instance.
• I am going to make it easier to add a status banner to the website. Right now it requires a software release, which is an unnecessary speed bump when I'm diagnosing a problem.
• I am going to automate posting of issues to the status.groups.io page. Right now, it has to be done by hand, which means it doesn't get done.

Please let me know if you have any questions.

Thanks, Mark

Thank you Mark for the update. 

You are correct. The system has been working great and has not missed a beat since Tuesday.

No problem with the missing messages. I will deal with it on my end. 

Again thank you for being "on-it".

kl

I can confirm the same issue, regardless of extension, albeit in my case the old vs new distinction is not as clear, i.e. I can search for the keyword that exists in two TXT files but only one is returned and in a particular case it finds a newer one, but (and it may be a clue), the newer one that is found was updated at some point in time.

It almost sounds to me like a search index needs rebuilding or something.

Cheers,
Christos

On Wed, Sep 23, 2020 at 05:35 PM, Mark Fletcher wrote:

On Wed, Sep 23, 2020 at 8:59 AM Andy Wedge <andy_wedge@...> wrote:

it seems that setting View Table permissions to 'Moderators and the Table Owner' does not stop a member viewing the table if they have the URL that contains the table id.

This has been fixed.

Thanks Mark,

If you try to access a restricted table by the URL it now just displays a blank screen. An error message of some description would be nice :-)

Regards
Andy

 

Hi Mark,

it seems that setting View Table permissions to 'Moderators and the Table Owner' does not stop a member viewing the table if they have the URL that contains the table id.

Regards,
Andy

On Mon, Sep 21, 2020 at 07:22 PM, Bruce Bowman wrote:

Peter -- Well, now I'm just being dumb.

Or, like me, you're dealing with email overload.

Anyway, I rechecked with this moderator and she no longer sees the "create/edit" button on the hashtags page. So although the mystery isn't solved I'm going to assume the problem is.

Thank you all for helping me with this!

Pete

I have done sufficient testing with PDF files and .txt files to be certain that text indexing that happened to old files is no longer happening with new files. 

Discussed in GMF at https://groups.io/g/GroupManagersForum/topic/76990562

I am the owner of a group used for a condo association. The group was started on 03/19/19 so is grandfathered in to have more features than newly created groups.  We have stored files in folders in the file section, including folder for "Board Minutes" with sub folders for each year.  Board minutes are stored as "Print to PDF" files and are searchable.
 
I open a PDF file that was uploaded last year and find an uncommon word in it.
I search for the word using the PDF reader and it is found.
I search for the word using the groups.io Files/Search and the file shows up in the results

I upload a new file (created from Print to PDF from MS Word) with the word "abracadabra" in it.
I search for the word abracadabra using the groups.io Files/Search and the file does not show up in the results.

Mark

I have a number of emails going back to last Wednesday that still have not emailed out to members. I have provided support with the message ID numbers. Should we do a new support ticket?

Thank you

Thanks Mark. I know the ICS stuff has been a game a Whack-A-Mole for you.
--
Kenny Paul, Technical Program Manager for ONAP
The Linux Foundation
Pacific Time Zone

Thanks Bruce.  good call, but I've already tried that.  The requirements list is coincidently one that was renamed, but the generation of new subscriptions has not resolved the problem.  Also there are numerous lists that have not changed since the original port to Gio over 2 years ago are also impacted by the same behavior.
--
Kenny Paul, Technical Program Manager for ONAP
The Linux Foundation
Pacific Time Zone

Hi Mark,

If one group's settings is FollowingOnly and they click on the Follow This Topic link at the footer of an emailed message from the group, as expected they are taken to the Follow this Yes/No page (which btw still uses "thread" instead of "topic").

If one clicks on No there however, it does the same thing as Yes does, adds it to your FollowedTopics list and lands you in the FollowedTopics page.

Cheers,
Christos

On Mon, Sep 21, 2020 at 07:21 PM, Kenny Paul wrote:

Gsuite account, non-gsuite account, old account, new account, I just cannot get stuff to work in any consistent fashion.

If you change the name of a group/subgroup that also affects the feed address. You will have to click the Subscribe to Calendar button again to get a new GIO feed and re-sub that to the Google calendar.

Does that help?

I am not aware of any way to go in the other direction (i.e.: subscribe a Google calendar to groups.io). Others more knowledgeable than I am may have ideas.

Regards,
Bruce

On Mon, Sep 21, 2020 at 07:10 PM, I wrote:

Discrepancies:  

• My Owner account is not actually moderated. The message went into moderation because it contained a new hashtag.
• I did not add the hashtag before approving the message. The act of approving the message added the hashtag and generated both log entries.

Peter -- Well, now I'm just being dumb.

Apparently, my Owner account was actually moderated in my test group. I must have left it that way from some previous test I was doing. So it actually did go into moderation for that reason, and the log entry was correct.

What actually happens [under normal circumstances] is that incoming emails with new hashtags, when sent by Moderators/Owners, override the group hashtag setting -- going directly into the message base and generating those two log entries -- without any further intervention by anybody.

If you haven't given this person the permission to create new hashtags this certainly does seem like a way to circumvent that.

As I said before, I'd prefer that Moderator/Owner posts be treated no differently from Member posts in this regard.

Duh.

Bruce