Date   

moderated Site updates #changelog

 

Changes to the site this week:

  • BUGFIX: For people with infinite scroll set, pagination was missing on the subgroup direct add page.
  • INTERNAL: More work in transitioning to quicktemplate.
  • CHANGE: Notification messages no longer obey group moderation settings, only hashtag moderation settings. This makes them consistent with #cal-invite notices and integration messages.
  • CHANGE: Pending notification messages no longer are tagged as Integration messages and are now tagged with a Notification badge.
  • CHANGE: Notification messages now are sent with an envelope sender of noreply@groups.io, to be more consistent with other automated messages.
  • CHANGE: Various message activity log messages were re-worded for notifications.
  • BUGFIX: Could not delete a specific wiki page revision.
  • BUGFIX: In the activity log, we were not shortening some messages that we should have been.
  • BUGFIX: Sorting in the Files search results page now works the same as the Files section.
  • BUGFIX: Clicking No on the Follow Topic page would still cause you to follow the topic. Also, cleaned up the page and fixed two references to thread.
  • BUGFIX: Newly uploaded files were not being indexed in the search cluster.
  • BUGFIX: In the Files section, even if you didn't have permission to upload a file, you could still bring up the File Upload dialog box (but the upload would fail).
  • BUGFIX: If a user knew the URL for a database table, they could view it even if they didn't have permission to do so.
  • BUGFIX: Our new DNS server was not answering CNAME queries with appropriate A records. This prevented status.groups.io from resolving.
  • BUGFIX: There were some semicolon escaping issues when we generated ICS files. This would cause Google Calendar to not parse the files correctly.

Take care everyone.

Mark


moderated Re: (revised) file upload notification requires pending message approvel for uploaders on mod #bug

Glenn Glazer
 

On 9/25/2020 15:31, Mark Fletcher wrote:
On Fri, Sep 25, 2020 at 3:27 PM Glenn Glazer <glenn.glazer@...> wrote:
On 9/25/2020 14:43, Mark Fletcher wrote:

- CHANGE: Notification messages no longer obey group moderation settings, only hashtag moderation settings. This makes them consistent with #cal-invite notices and integration messages.

This is hopefully something that I wanted, but to be sure, if I leave #file-notice unmoderated, then the upload notice generated by the "Notify Members" box on the upload will be unmoderated regardless of who uploads it. Is that correct?

Yes.


Mark 

Awesome, you're the best.

Best,

Glenn

--
PG&E Delenda Est

Virus-free. www.avast.com


moderated Re: (revised) file upload notification requires pending message approvel for uploaders on mod #bug

 

On Fri, Sep 25, 2020 at 3:27 PM Glenn Glazer <glenn.glazer@...> wrote:
On 9/25/2020 14:43, Mark Fletcher wrote:

- CHANGE: Notification messages no longer obey group moderation settings, only hashtag moderation settings. This makes them consistent with #cal-invite notices and integration messages.

This is hopefully something that I wanted, but to be sure, if I leave #file-notice unmoderated, then the upload notice generated by the "Notify Members" box on the upload will be unmoderated regardless of who uploads it. Is that correct?

Yes.


Mark 


moderated Re: (revised) file upload notification requires pending message approvel for uploaders on mod #bug

Glenn Glazer
 

On 9/25/2020 14:43, Mark Fletcher wrote:

- CHANGE: Notification messages no longer obey group moderation settings, only hashtag moderation settings. This makes them consistent with #cal-invite notices and integration messages.

This is hopefully something that I wanted, but to be sure, if I leave #file-notice unmoderated, then the upload notice generated by the "Notify Members" box on the upload will be unmoderated regardless of who uploads it. Is that correct?

Best,

Glenn

--
PG&E Delenda Est

Virus-free. www.avast.com


moderated Re: (revised) file upload notification requires pending message approvel for uploaders on mod #bug

 

Looks like a home run. Thanks!
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu


moderated Re: Activity log navlinks hidden #fixed #bug

 

On Fri, Sep 25, 2020 at 10:20 AM Alan Epstein <ade26g@...> wrote:
Thanks, Mark.

Here's a scrunched down screenshot. The red oval shows where I would expect the nav control to get to the 400 other members. Another admin on this account can see them, but I can't. And it's consistent in other groups. I've tried it on Mac, PC, Firefox and Chrome.

This has been fixed.

Thanks,
Mark 


moderated Re: (revised) file upload notification requires pending message approvel for uploaders on mod #bug

 

Hello,

I've made the following changes:

- CHANGE: Notification messages no longer obey group moderation settings, only hashtag moderation settings. This makes them consistent with #cal-invite notices and integration messages.
- CHANGE: Pending notification messages no longer are tagged as `Integration` messages and are now tagged with a `Notification` badge.
- CHANGE: Notification messages now are sent with an envelope sender of noreply@groups.io, to be more consistent with other automated messages.
- CHANGE: Various message activity log messages were re-worded for notifications.

Please let me know if you have any questions.

Thanks,
Mark


moderated Login example in API docs missing quotes around credentials string #misc

Sarah Tappon
 

The example currently looks like this:
$ curl "https://groups.io/api/v1/login" \
-c "cookies.curl" \
-d email=test@...&password=qwerty

I've found this always gives an invalid password error (despite no special characters in my password). It works if you add quotes around the email&password string. Would be good to edit the docs to reflect this.

Thanks!


moderated Re: Pending message stuck after email address change #bug

 

On Fri, Sep 18, 2020 at 12:26 AM Andy Wedge <andy_wedge@...> wrote:

Hmm. There is a further problem now in that trying to delete the message doesn't work.

I'm not sure what the problem was. I was able to delete the message without error. Please let me know if you see this again.

Thanks,
Mark 


moderated Re: Wiki - can't delete revisions #bug

 

On Sat, Sep 19, 2020 at 9:19 AM Duane <txpigeon@...> wrote:
I noticed this morning that I'm unable to delete revisions from the Wiki pages.  After I click on Delete This Revision, it takes me to the Wiki home page.  When I go back to the page I was trying to delete the revision from, it shows the original number of revisions in the upper right corner.  Click on that and see all of them.

This should be fixed now.

Thanks,
Mark 


moderated Re: Activity log navlinks hidden #fixed #bug

 

On Sun, Sep 20, 2020 at 4:21 AM Alan Epstein <ade26g@...> wrote:
I am also finding a lack of navigation controls for the Direct Invite screen. I see only the 1st 20 records and no way to get to the rest. True on all groups, all platforms, all browsers.

Are you referring to the Sent Invitations screen? I just checked that and was able to page through the results. If you're still seeing the problem, please send the URL and I'll investigate.

Thanks,
Mark 


moderated Re: FILES-Search, UPLOADED criteria sort is using incorrect date field for either sorting or displaying the results #bug

 

On Mon, Sep 21, 2020 at 3:55 PM Christos G. Psarras <christos@...> wrote:


The Upload sort order must be using some other date field for the sorting, because it doesn't sort them correctly, in either ASC or DESC order; or it actually does sort them correctly, but it displays a different (DateUpdated?) field date which results in the list showing like it's not sorted correctly, something to that effect, see attached.

It only happens when the search box is being used, it sorts them correctly if it's not.

Sorting on the Files search results page should be fixed now.

Thanks,
Mark 


moderated Downtime on Monday, September 21, 2020 #postmortem

 

Hi All,

Here's a summary of what happened during the email outage on Monday and the steps I'm taking to address it.

What Happened

I noticed around 9am that the program that sends emails, called karld, had crashed and restarted several times during the night. I had not been paged. Checking the mail queue, it appeared that there were a large number of emails that had not been sent. I began to investigate.

karld was running out of memory and restarting. It would only be able to send a few emails before each restart. I initially believed it was because there were about 600k messages to gmail.com addresses waiting to be sent (of a total of 900k messages in the queue), and we simply didn't have enough memory to support that. At the time I thought it was an issue with delivering email to Gmail that was the main problem.

As a short term measure, I greatly increased the swap on the main email sending machine, to try and clear the message backlog. I also decreased the number of outbound email connections. This started to clear the backlog, but tripped another issue; the machine was now running out of socket resources, which had never happened before. I increased the kernel socket resource limits and that allowed the backlog to continue to drain. But I noticed that email to gmail.com was not being sent in any significant numbers, so the backlog was not draining quickly. Restarting the karld process would cause some gmail.com messages to be sent, so I began restarting the process every few minutes, while I tried to puzzle out what was happening. I discovered two inter-related bugs.

I discovered that a change I made last week introduced a bug where we were including a large amount of extra, unneeded data with every recipient for each message. This greatly increased the memory usage of the system. This did not cause a problem until sometime early Monday morning, when enough messages were in the unsent queue that karld started taking up too much memory.

But that didn't explain the initial reason that messages were not being sent, which I had initially thought was caused by an intermittent Gmail delivery issue. In reality, it was caused by a second bug, also caused by the change I made last week. We have to track how many concurrent connections we have to each email service, because many services restrict the number of connections we can have to them. This second bug was causing an accounting issue where we would lose track of how many open connections we had with each email service. In some cases, we thought we had open connections, when we actually had none. Specifically, this was triggered for Gmail, and it meant that we weren't sending many messages to them. Which caused the large number of unsent messages, which in turn caused the out of memory issue.

I fixed these bugs, and the email queue was back to normal by around 2pm.

Changes Already Made

  • I fixed the two bugs at the root of the downtime.
  • I re-enabled monitoring of karld, which I had disabled last week for an unrelated issue.
  • During the downtime, I discovered that status.groups.io was not working. I fixed the DNS issue causing that and that website is back up.
  • I fixed several inaccurate metrics on the karld internal dashboard, which will help diagnose issues in the future.

Changes Still To Be Made

  • I am going to accelerate the addition of a second karld instance. Emails from beta are already being sent from the new instance.
  • I am going to make it easier to add a status banner to the website. Right now it requires a software release, which is an unnecessary speed bump when I'm diagnosing a problem.
  • I am going to automate posting of issues to the status.groups.io page. Right now, it has to be done by hand, which means it doesn't get done.

Please let me know if you have any questions.

Thanks, Mark


moderated Re: Email delivery issues #update

KL
 

Thank you Mark for the update. 

You are correct. The system has been working great and has not missed a beat since Tuesday.

No problem with the missing messages. I will deal with it on my end. 

Again thank you for being "on-it".

kl


moderated Re: Email delivery issues #update

 

On Tue, Sep 22, 2020 at 8:24 AM KL <kal564@...> wrote:

I have a number of emails going back to last Wednesday that still have not emailed out to members. I have provided support with the message ID numbers. Should we do a new support ticket?

The bug affecting your group (and I believe only your group) has been fixed. It was related to the outage on Monday (#postmortem forthcoming). Emails should be flowing again, although unfortunately there's no easy way to resend the ones that didn't go out.

Thanks,
Mark 


moderated Re: FollowTopic Yes or No page doesn't obey No #bug

 

On Mon, Sep 21, 2020 at 8:06 PM Christos G. Psarras <christos@...> wrote:

If one group's settings is FollowingOnly and they click on the Follow This Topic link at the footer of an emailed message from the group, as expected they are taken to the Follow this Yes/No page (which btw still uses "thread" instead of "topic").

If one clicks on No there however, it does the same thing as Yes does, adds it to your FollowedTopics list and lands you in the FollowedTopics page.


This should be fixed now.

Thanks,
Mark 


moderated Re: Files recently uploaded are not found when you do a word search from the Files area. #bug

 

Hello,

On Wed, Sep 23, 2020 at 7:36 AM Jud Eson <eson.jud@...> wrote:
I have done sufficient testing with PDF files and .txt files to be certain that text indexing that happened to old files is no longer happening with new files. 

The bug has been fixed and I'm in the process of re-indexing the files section. This will probably take several hours, but should be done by tomorrow.

Thanks,
Mark 


moderated Re: Files recently uploaded are not found when you do a word search from the Files area. #bug

Christos G. Psarras
 

I can confirm the same issue, regardless of extension, albeit in my case the old vs new distinction is not as clear, i.e. I can search for the keyword that exists in two TXT files but only one is returned and in a particular case it finds a newer one, but (and it may be a clue), the newer one that is found was updated at some point in time.

It almost sounds to me like a search index needs rebuilding or something.

Cheers,
Christos


On 2020-09-23 08:32, Jud Eson via groups.io wrote:
I have done sufficient testing with PDF files and .txt files to be certain that text indexing that happened to old files is no longer happening with new files. 

Discussed in GMF at https://groups.io/g/GroupManagersForum/topic/76990562


moderated Re: Members still able to view database tables when permissions are Mods & Table Owner #bug

Andy Wedge
 

On Wed, Sep 23, 2020 at 05:35 PM, Mark Fletcher wrote:
On Wed, Sep 23, 2020 at 8:59 AM Andy Wedge <andy_wedge@...> wrote:
it seems that setting View Table permissions to 'Moderators and the Table Owner' does not stop a member viewing the table if they have the URL that contains the table id.
This has been fixed.
Thanks Mark,

If you try to access a restricted table by the URL it now just displays a blank screen. An error message of some description would be nice :-)

Regards
Andy

 


moderated Re: Members still able to view database tables when permissions are Mods & Table Owner #bug

 

On Wed, Sep 23, 2020 at 8:59 AM Andy Wedge <andy_wedge@...> wrote:

it seems that setting View Table permissions to 'Moderators and the Table Owner' does not stop a member viewing the table if they have the URL that contains the table id.

This has been fixed.

Thanks,
Mark