Date   

moderated delay in approvals of pending messages #bug

 

Mark,

I'm seeing similar delay behavior in message approvals (via emal) today to what I saw when there was the email delay problem. For example, I approved a message via email and it had not gone through within several minutes. I eventually had to go to the site to approve it, and by that time other messages in the thread had already gone through that made it somewhat obsolete. I probably would not have bothered to report this except that I know there was the other delay problem prior to this, and perhaps they are related.
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


moderated Re: Downtime on Monday, September 21, 2020 #postmortem

RCardona
 

Andy,

Reread the 21 Sept message. . .  status.groups.io is listed under the section, "Changes Still to be Made."  

Give the man some time to implement it.

Rob


On Sat, Sep 26, 2020 at 05:18 PM, Andy wrote:
What is "status.groups.io"?
When I go there, it says (among other things):
    Sep 20, 2020
    No incidents reported.

Seriously?
I guess you're saying things get reported only when you manually enter them, which tends to be never.  Got it.

Andy


moderated Re: Activity log navlinks hidden #fixed #bug

Alan Epstein
 

Thank you, Chris.


moderated Re: Downtime on Monday, September 21, 2020 #postmortem

Andy
 

What is "status.groups.io"?

When I go there, it says (among other things):

    Sep 20, 2020
    No incidents reported.

Seriously?

I guess you're saying things get reported only when you manually enter them, which tends to be never.  Got it.

Andy


moderated Re: Activity log navlinks hidden #fixed #bug

Chris Jones
 

On Sat, Sep 26, 2020 at 02:20 PM, Alan Epstein wrote:
With members, mine gives a long rolling list, whereas my colleague sees a shorter list with the nav control at the bottom. Not sure why we should be seeing differently unless it has to do with him being owner of a paid account.
Probably because you have different Items per Page settings under Preferences in your Account settings. These settings are account - wide and not per group.

Chris


moderated Re: Activity log navlinks hidden #fixed #bug

Alan Epstein
 

Thanks, Mark! That's a big help.

I think this bug may inhabit a few other places, such as the Groups list and Members list. With members, mine gives a long rolling list, whereas my colleague sees a shorter list with the nav control at the bottom. Not sure why we should be seeing differently unless it has to do with him being owner of a paid account.

_Alan


moderated Site updates #changelog

 

Changes to the site this week:

  • BUGFIX: For people with infinite scroll set, pagination was missing on the subgroup direct add page.
  • INTERNAL: More work in transitioning to quicktemplate.
  • CHANGE: Notification messages no longer obey group moderation settings, only hashtag moderation settings. This makes them consistent with #cal-invite notices and integration messages.
  • CHANGE: Pending notification messages no longer are tagged as Integration messages and are now tagged with a Notification badge.
  • CHANGE: Notification messages now are sent with an envelope sender of noreply@groups.io, to be more consistent with other automated messages.
  • CHANGE: Various message activity log messages were re-worded for notifications.
  • BUGFIX: Could not delete a specific wiki page revision.
  • BUGFIX: In the activity log, we were not shortening some messages that we should have been.
  • BUGFIX: Sorting in the Files search results page now works the same as the Files section.
  • BUGFIX: Clicking No on the Follow Topic page would still cause you to follow the topic. Also, cleaned up the page and fixed two references to thread.
  • BUGFIX: Newly uploaded files were not being indexed in the search cluster.
  • BUGFIX: In the Files section, even if you didn't have permission to upload a file, you could still bring up the File Upload dialog box (but the upload would fail).
  • BUGFIX: If a user knew the URL for a database table, they could view it even if they didn't have permission to do so.
  • BUGFIX: Our new DNS server was not answering CNAME queries with appropriate A records. This prevented status.groups.io from resolving.
  • BUGFIX: There were some semicolon escaping issues when we generated ICS files. This would cause Google Calendar to not parse the files correctly.

Take care everyone.

Mark


moderated Re: (revised) file upload notification requires pending message approvel for uploaders on mod #bug

Glenn Glazer
 

On 9/25/2020 15:31, Mark Fletcher wrote:
On Fri, Sep 25, 2020 at 3:27 PM Glenn Glazer <glenn.glazer@...> wrote:
On 9/25/2020 14:43, Mark Fletcher wrote:

- CHANGE: Notification messages no longer obey group moderation settings, only hashtag moderation settings. This makes them consistent with #cal-invite notices and integration messages.

This is hopefully something that I wanted, but to be sure, if I leave #file-notice unmoderated, then the upload notice generated by the "Notify Members" box on the upload will be unmoderated regardless of who uploads it. Is that correct?

Yes.


Mark 

Awesome, you're the best.

Best,

Glenn

--
PG&E Delenda Est

Virus-free. www.avast.com


moderated Re: (revised) file upload notification requires pending message approvel for uploaders on mod #bug

 

On Fri, Sep 25, 2020 at 3:27 PM Glenn Glazer <glenn.glazer@...> wrote:
On 9/25/2020 14:43, Mark Fletcher wrote:

- CHANGE: Notification messages no longer obey group moderation settings, only hashtag moderation settings. This makes them consistent with #cal-invite notices and integration messages.

This is hopefully something that I wanted, but to be sure, if I leave #file-notice unmoderated, then the upload notice generated by the "Notify Members" box on the upload will be unmoderated regardless of who uploads it. Is that correct?

Yes.


Mark 


moderated Re: (revised) file upload notification requires pending message approvel for uploaders on mod #bug

Glenn Glazer
 

On 9/25/2020 14:43, Mark Fletcher wrote:

- CHANGE: Notification messages no longer obey group moderation settings, only hashtag moderation settings. This makes them consistent with #cal-invite notices and integration messages.

This is hopefully something that I wanted, but to be sure, if I leave #file-notice unmoderated, then the upload notice generated by the "Notify Members" box on the upload will be unmoderated regardless of who uploads it. Is that correct?

Best,

Glenn

--
PG&E Delenda Est

Virus-free. www.avast.com


moderated Re: (revised) file upload notification requires pending message approvel for uploaders on mod #bug

 

Looks like a home run. Thanks!
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


moderated Re: Activity log navlinks hidden #fixed #bug

 

On Fri, Sep 25, 2020 at 10:20 AM Alan Epstein <ade26g@...> wrote:
Thanks, Mark.

Here's a scrunched down screenshot. The red oval shows where I would expect the nav control to get to the 400 other members. Another admin on this account can see them, but I can't. And it's consistent in other groups. I've tried it on Mac, PC, Firefox and Chrome.

This has been fixed.

Thanks,
Mark 


moderated Re: (revised) file upload notification requires pending message approvel for uploaders on mod #bug

 

Hello,

I've made the following changes:

- CHANGE: Notification messages no longer obey group moderation settings, only hashtag moderation settings. This makes them consistent with #cal-invite notices and integration messages.
- CHANGE: Pending notification messages no longer are tagged as `Integration` messages and are now tagged with a `Notification` badge.
- CHANGE: Notification messages now are sent with an envelope sender of noreply@groups.io, to be more consistent with other automated messages.
- CHANGE: Various message activity log messages were re-worded for notifications.

Please let me know if you have any questions.

Thanks,
Mark


moderated Login example in API docs missing quotes around credentials string #misc

Sarah Tappon
 

The example currently looks like this:
$ curl "https://groups.io/api/v1/login" \
-c "cookies.curl" \
-d email=test@...&password=qwerty

I've found this always gives an invalid password error (despite no special characters in my password). It works if you add quotes around the email&password string. Would be good to edit the docs to reflect this.

Thanks!


moderated Re: Pending message stuck after email address change #bug

 

On Fri, Sep 18, 2020 at 12:26 AM Andy Wedge <andy_wedge@...> wrote:

Hmm. There is a further problem now in that trying to delete the message doesn't work.

I'm not sure what the problem was. I was able to delete the message without error. Please let me know if you see this again.

Thanks,
Mark 


moderated Re: Wiki - can't delete revisions #bug

 

On Sat, Sep 19, 2020 at 9:19 AM Duane <txpigeon@...> wrote:
I noticed this morning that I'm unable to delete revisions from the Wiki pages.  After I click on Delete This Revision, it takes me to the Wiki home page.  When I go back to the page I was trying to delete the revision from, it shows the original number of revisions in the upper right corner.  Click on that and see all of them.

This should be fixed now.

Thanks,
Mark 


moderated Re: Activity log navlinks hidden #fixed #bug

 

On Sun, Sep 20, 2020 at 4:21 AM Alan Epstein <ade26g@...> wrote:
I am also finding a lack of navigation controls for the Direct Invite screen. I see only the 1st 20 records and no way to get to the rest. True on all groups, all platforms, all browsers.

Are you referring to the Sent Invitations screen? I just checked that and was able to page through the results. If you're still seeing the problem, please send the URL and I'll investigate.

Thanks,
Mark 


moderated Re: FILES-Search, UPLOADED criteria sort is using incorrect date field for either sorting or displaying the results #bug

 

On Mon, Sep 21, 2020 at 3:55 PM Christos G. Psarras <christos@...> wrote:


The Upload sort order must be using some other date field for the sorting, because it doesn't sort them correctly, in either ASC or DESC order; or it actually does sort them correctly, but it displays a different (DateUpdated?) field date which results in the list showing like it's not sorted correctly, something to that effect, see attached.

It only happens when the search box is being used, it sorts them correctly if it's not.

Sorting on the Files search results page should be fixed now.

Thanks,
Mark 


moderated Downtime on Monday, September 21, 2020 #postmortem

 

Hi All,

Here's a summary of what happened during the email outage on Monday and the steps I'm taking to address it.

What Happened

I noticed around 9am that the program that sends emails, called karld, had crashed and restarted several times during the night. I had not been paged. Checking the mail queue, it appeared that there were a large number of emails that had not been sent. I began to investigate.

karld was running out of memory and restarting. It would only be able to send a few emails before each restart. I initially believed it was because there were about 600k messages to gmail.com addresses waiting to be sent (of a total of 900k messages in the queue), and we simply didn't have enough memory to support that. At the time I thought it was an issue with delivering email to Gmail that was the main problem.

As a short term measure, I greatly increased the swap on the main email sending machine, to try and clear the message backlog. I also decreased the number of outbound email connections. This started to clear the backlog, but tripped another issue; the machine was now running out of socket resources, which had never happened before. I increased the kernel socket resource limits and that allowed the backlog to continue to drain. But I noticed that email to gmail.com was not being sent in any significant numbers, so the backlog was not draining quickly. Restarting the karld process would cause some gmail.com messages to be sent, so I began restarting the process every few minutes, while I tried to puzzle out what was happening. I discovered two inter-related bugs.

I discovered that a change I made last week introduced a bug where we were including a large amount of extra, unneeded data with every recipient for each message. This greatly increased the memory usage of the system. This did not cause a problem until sometime early Monday morning, when enough messages were in the unsent queue that karld started taking up too much memory.

But that didn't explain the initial reason that messages were not being sent, which I had initially thought was caused by an intermittent Gmail delivery issue. In reality, it was caused by a second bug, also caused by the change I made last week. We have to track how many concurrent connections we have to each email service, because many services restrict the number of connections we can have to them. This second bug was causing an accounting issue where we would lose track of how many open connections we had with each email service. In some cases, we thought we had open connections, when we actually had none. Specifically, this was triggered for Gmail, and it meant that we weren't sending many messages to them. Which caused the large number of unsent messages, which in turn caused the out of memory issue.

I fixed these bugs, and the email queue was back to normal by around 2pm.

Changes Already Made

  • I fixed the two bugs at the root of the downtime.
  • I re-enabled monitoring of karld, which I had disabled last week for an unrelated issue.
  • During the downtime, I discovered that status.groups.io was not working. I fixed the DNS issue causing that and that website is back up.
  • I fixed several inaccurate metrics on the karld internal dashboard, which will help diagnose issues in the future.

Changes Still To Be Made

  • I am going to accelerate the addition of a second karld instance. Emails from beta are already being sent from the new instance.
  • I am going to make it easier to add a status banner to the website. Right now it requires a software release, which is an unnecessary speed bump when I'm diagnosing a problem.
  • I am going to automate posting of issues to the status.groups.io page. Right now, it has to be done by hand, which means it doesn't get done.

Please let me know if you have any questions.

Thanks, Mark


moderated Re: Email delivery issues #update

KL
 

Thank you Mark for the update. 

You are correct. The system has been working great and has not missed a beat since Tuesday.

No problem with the missing messages. I will deal with it on my end. 

Again thank you for being "on-it".

kl