Date   

moderated Re: FILES-Search, UPLOADED criteria sort is using incorrect date field for either sorting or displaying the results #bug

 

On Mon, Sep 21, 2020 at 3:55 PM Christos G. Psarras <christos@...> wrote:


The Upload sort order must be using some other date field for the sorting, because it doesn't sort them correctly, in either ASC or DESC order; or it actually does sort them correctly, but it displays a different (DateUpdated?) field date which results in the list showing like it's not sorted correctly, something to that effect, see attached.

It only happens when the search box is being used, it sorts them correctly if it's not.

Sorting on the Files search results page should be fixed now.

Thanks,
Mark 


moderated Downtime on Monday, September 21, 2020 #postmortem

 

Hi All,

Here's a summary of what happened during the email outage on Monday and the steps I'm taking to address it.

What Happened

I noticed around 9am that the program that sends emails, called karld, had crashed and restarted several times during the night. I had not been paged. Checking the mail queue, it appeared that there were a large number of emails that had not been sent. I began to investigate.

karld was running out of memory and restarting. It would only be able to send a few emails before each restart. I initially believed it was because there were about 600k messages to gmail.com addresses waiting to be sent (of a total of 900k messages in the queue), and we simply didn't have enough memory to support that. At the time I thought it was an issue with delivering email to Gmail that was the main problem.

As a short term measure, I greatly increased the swap on the main email sending machine, to try and clear the message backlog. I also decreased the number of outbound email connections. This started to clear the backlog, but tripped another issue; the machine was now running out of socket resources, which had never happened before. I increased the kernel socket resource limits and that allowed the backlog to continue to drain. But I noticed that email to gmail.com was not being sent in any significant numbers, so the backlog was not draining quickly. Restarting the karld process would cause some gmail.com messages to be sent, so I began restarting the process every few minutes, while I tried to puzzle out what was happening. I discovered two inter-related bugs.

I discovered that a change I made last week introduced a bug where we were including a large amount of extra, unneeded data with every recipient for each message. This greatly increased the memory usage of the system. This did not cause a problem until sometime early Monday morning, when enough messages were in the unsent queue that karld started taking up too much memory.

But that didn't explain the initial reason that messages were not being sent, which I had initially thought was caused by an intermittent Gmail delivery issue. In reality, it was caused by a second bug, also caused by the change I made last week. We have to track how many concurrent connections we have to each email service, because many services restrict the number of connections we can have to them. This second bug was causing an accounting issue where we would lose track of how many open connections we had with each email service. In some cases, we thought we had open connections, when we actually had none. Specifically, this was triggered for Gmail, and it meant that we weren't sending many messages to them. Which caused the large number of unsent messages, which in turn caused the out of memory issue.

I fixed these bugs, and the email queue was back to normal by around 2pm.

Changes Already Made

  • I fixed the two bugs at the root of the downtime.
  • I re-enabled monitoring of karld, which I had disabled last week for an unrelated issue.
  • During the downtime, I discovered that status.groups.io was not working. I fixed the DNS issue causing that and that website is back up.
  • I fixed several inaccurate metrics on the karld internal dashboard, which will help diagnose issues in the future.

Changes Still To Be Made

  • I am going to accelerate the addition of a second karld instance. Emails from beta are already being sent from the new instance.
  • I am going to make it easier to add a status banner to the website. Right now it requires a software release, which is an unnecessary speed bump when I'm diagnosing a problem.
  • I am going to automate posting of issues to the status.groups.io page. Right now, it has to be done by hand, which means it doesn't get done.

Please let me know if you have any questions.

Thanks, Mark


moderated Re: Email delivery issues #update

KL
 

Thank you Mark for the update. 

You are correct. The system has been working great and has not missed a beat since Tuesday.

No problem with the missing messages. I will deal with it on my end. 

Again thank you for being "on-it".

kl


moderated Re: Email delivery issues #update

 

On Tue, Sep 22, 2020 at 8:24 AM KL <kal564@...> wrote:

I have a number of emails going back to last Wednesday that still have not emailed out to members. I have provided support with the message ID numbers. Should we do a new support ticket?

The bug affecting your group (and I believe only your group) has been fixed. It was related to the outage on Monday (#postmortem forthcoming). Emails should be flowing again, although unfortunately there's no easy way to resend the ones that didn't go out.

Thanks,
Mark 


moderated Re: FollowTopic Yes or No page doesn't obey No #bug

 

On Mon, Sep 21, 2020 at 8:06 PM Christos G. Psarras <christos@...> wrote:

If one group's settings is FollowingOnly and they click on the Follow This Topic link at the footer of an emailed message from the group, as expected they are taken to the Follow this Yes/No page (which btw still uses "thread" instead of "topic").

If one clicks on No there however, it does the same thing as Yes does, adds it to your FollowedTopics list and lands you in the FollowedTopics page.


This should be fixed now.

Thanks,
Mark 


moderated Re: Files recently uploaded are not found when you do a word search from the Files area. #bug

 

Hello,

On Wed, Sep 23, 2020 at 7:36 AM Jud Eson <eson.jud@...> wrote:
I have done sufficient testing with PDF files and .txt files to be certain that text indexing that happened to old files is no longer happening with new files. 

The bug has been fixed and I'm in the process of re-indexing the files section. This will probably take several hours, but should be done by tomorrow.

Thanks,
Mark 


moderated Re: Files recently uploaded are not found when you do a word search from the Files area. #bug

Christos G. Psarras
 

I can confirm the same issue, regardless of extension, albeit in my case the old vs new distinction is not as clear, i.e. I can search for the keyword that exists in two TXT files but only one is returned and in a particular case it finds a newer one, but (and it may be a clue), the newer one that is found was updated at some point in time.

It almost sounds to me like a search index needs rebuilding or something.

Cheers,
Christos


On 2020-09-23 08:32, Jud Eson via groups.io wrote:
I have done sufficient testing with PDF files and .txt files to be certain that text indexing that happened to old files is no longer happening with new files. 

Discussed in GMF at https://groups.io/g/GroupManagersForum/topic/76990562


moderated Re: Members still able to view database tables when permissions are Mods & Table Owner #bug

Andy Wedge
 

On Wed, Sep 23, 2020 at 05:35 PM, Mark Fletcher wrote:
On Wed, Sep 23, 2020 at 8:59 AM Andy Wedge <andy_wedge@...> wrote:
it seems that setting View Table permissions to 'Moderators and the Table Owner' does not stop a member viewing the table if they have the URL that contains the table id.
This has been fixed.
Thanks Mark,

If you try to access a restricted table by the URL it now just displays a blank screen. An error message of some description would be nice :-)

Regards
Andy

 


moderated Re: Members still able to view database tables when permissions are Mods & Table Owner #bug

 

On Wed, Sep 23, 2020 at 8:59 AM Andy Wedge <andy_wedge@...> wrote:

it seems that setting View Table permissions to 'Moderators and the Table Owner' does not stop a member viewing the table if they have the URL that contains the table id.

This has been fixed.

Thanks,
Mark 


moderated Members still able to view database tables when permissions are Mods & Table Owner #bug

Andy Wedge
 

Hi Mark,

it seems that setting View Table permissions to 'Moderators and the Table Owner' does not stop a member viewing the table if they have the URL that contains the table id.

Regards,
Andy


moderated Re: Moderator created new hashtag without permission #bug

Peter Cook
 

On Mon, Sep 21, 2020 at 07:22 PM, Bruce Bowman wrote:
Peter -- Well, now I'm just being dumb.
Or, like me, you're dealing with email overload.

Anyway, I rechecked with this moderator and she no longer sees the "create/edit" button on the hashtags page. So although the mystery isn't solved I'm going to assume the problem is.

Thank you all for helping me with this!

Pete


moderated Files recently uploaded are not found when you do a word search from the Files area. #bug

Jud Eson
 

I have done sufficient testing with PDF files and .txt files to be certain that text indexing that happened to old files is no longer happening with new files. 

Discussed in GMF at https://groups.io/g/GroupManagersForum/topic/76990562

I am the owner of a group used for a condo association. The group was started on 03/19/19 so is grandfathered in to have more features than newly created groups.  We have stored files in folders in the file section, including folder for "Board Minutes" with sub folders for each year.  Board minutes are stored as "Print to PDF" files and are searchable.
 
I open a PDF file that was uploaded last year and find an uncommon word in it.
I search for the word using the PDF reader and it is found.
I search for the word using the groups.io Files/Search and the file shows up in the results

I upload a new file (created from Print to PDF from MS Word) with the word "abracadabra" in it.
I search for the word abracadabra using the groups.io Files/Search and the file does not show up in the results.


moderated Re: Email delivery issues #update

KL
 

Mark

I have a number of emails going back to last Wednesday that still have not emailed out to members. I have provided support with the message ID numbers. Should we do a new support ticket?

Thank you


moderated Re: Ongoing Calendar Saga #bug

Kenny Paul
 

Thanks Mark. I know the ICS stuff has been a game a Whack-A-Mole for you.
--
Kenny Paul, Technical Program Manager for ONAP
The Linux Foundation
Pacific Time Zone


moderated Re: Ongoing Calendar Saga #bug

Kenny Paul
 

Thanks Bruce.  good call, but I've already tried that.  The requirements list is coincidently one that was renamed, but the generation of new subscriptions has not resolved the problem.  Also there are numerous lists that have not changed since the original port to Gio over 2 years ago are also impacted by the same behavior.
--
Kenny Paul, Technical Program Manager for ONAP
The Linux Foundation
Pacific Time Zone


moderated Re: Ongoing Calendar Saga #bug

 

Hi Kenny,

On Mon, Sep 21, 2020 at 4:21 PM Kenny Paul <kpaul@...> wrote:

I've about reached my breaking point related to Gio cal vs. G-cal and the unhealthy frienemy relationship they have had over the past couple of years.   You can subscribe to the a couple of our Gio sub-group cals from and G-cal and they the work just fine and others just will not do squat. Gsuite account, non-gsuite account, old account, new account, I just cannot get stuff to work in any consistent fashion.  The community is in disarray at this point given the fact that so many folks do not use Gio natively and I simply have run out of ideas.

I am sorry about the bugs you're experiencing. The ICS file spec is not my favorite. I have fixed the escaping issue that was causing GCal to not parse that file. It now parses it correctly. You will notice that it doesn't load 2 of the events into the Google calendar. I'm still researching that; it's because the two events have the same UID, but I don't yet know how that happened, if it's normal, and what to do about it.

Thanks,
Mark 


moderated FollowTopic Yes or No page doesn't obey No #bug

Christos G. Psarras
 

Hi Mark,

If one group's settings is FollowingOnly and they click on the Follow This Topic link at the footer of an emailed message from the group, as expected they are taken to the Follow this Yes/No page (which btw still uses "thread" instead of "topic").

If one clicks on No there however, it does the same thing as Yes does, adds it to your FollowedTopics list and lands you in the FollowedTopics page.

Cheers,
Christos


moderated Re: Ongoing Calendar Saga #bug

Bruce Bowman
 

On Mon, Sep 21, 2020 at 07:21 PM, Kenny Paul wrote:
Gsuite account, non-gsuite account, old account, new account, I just cannot get stuff to work in any consistent fashion.
If you change the name of a group/subgroup that also affects the feed address. You will have to click the Subscribe to Calendar button again to get a new GIO feed and re-sub that to the Google calendar.

Does that help?

I am not aware of any way to go in the other direction (i.e.: subscribe a Google calendar to groups.io). Others more knowledgeable than I am may have ideas.

Regards,
Bruce


moderated Re: Moderator created new hashtag without permission #bug

Bruce Bowman
 

On Mon, Sep 21, 2020 at 07:10 PM, I wrote:
Discrepancies:  
  • My Owner account is not actually moderated. The message went into moderation because it contained a new hashtag.
  • I did not add the hashtag before approving the message. The act of approving the message added the hashtag and generated both log entries.
Peter -- Well, now I'm just being dumb.

Apparently, my Owner account was actually moderated in my test group. I must have left it that way from some previous test I was doing. So it actually did go into moderation for that reason, and the log entry was correct.

What actually happens [under normal circumstances] is that incoming emails with new hashtags, when sent by Moderators/Owners, override the group hashtag setting -- going directly into the message base and generating those two log entries -- without any further intervention by anybody.

If you haven't given this person the permission to create new hashtags this certainly does seem like a way to circumvent that.

As I said before, I'd prefer that Moderator/Owner posts be treated no differently from Member posts in this regard.

Duh.

Bruce


moderated Ongoing Calendar Saga #bug

Kenny Paul
 

I've about reached my breaking point related to Gio cal vs. G-cal and the unhealthy frienemy relationship they have had over the past couple of years.   You can subscribe to the a couple of our Gio sub-group cals from and G-cal and they the work just fine and others just will not do squat. Gsuite account, non-gsuite account, old account, new account, I just cannot get stuff to work in any consistent fashion.  The community is in disarray at this point given the fact that so many folks do not use Gio natively and I simply have run out of ideas.

Random list where cal subscription works:  onap-arc@...
Random list where cal subscription doesn't work: 
onap-requirements-sub@... 

Ideas, help, xanax or tequila all welcome.

--
Kenny Paul, Technical Program Manager for ONAP
The Linux Foundation
Pacific Time Zone