Date   

moderated Reduced email delivery overnight #outage

 

Outage Summary

Email delivery was dramatically slowed overnight due to a crashing bug in the karl process. I fixed the bug around 6:25am pacific time, and it took about 25 minutes for all queued email to be delivered.

Duration

From approximately around midnight pacific time until 6:25am pacific time.

Cause

A group was transferred from Yahoo yesterday with a lot of users from a bogus/typo domain, yahooo.com. That domain has an MX record that is blank. We were not ignoring blank MX records. What that meant is that for email to those users, we'd connect to the localhost SMTP port to send this email, essentially DOSing ourselves. I was alerted to this behavior on Friday evening via our normal alerts. I wrote a fix and pushed it to the site. Because of how we retry sending messages, the code introduced with the fix was not run until after I went to bed. I did not consider the fix to be risky, but the fix contained a divide by zero, causing a crash. The karl process then continued to crash until I woke up and saw the problem. I was not alerted to the crashes.

Areas Of Improvement
  • I was not alerted to the crashes because our alerts are emailed to Pagerduty using karl. Alerts should be sent directly to Pagerduty using their API instead of via email.

Thanks, Mark


moderated Site updates #changelog

 

Changes to the site this week:

  • SYSADMIN: Added more database indexes to speed up operations.
  • SYSADMIN: Worked on better database monitoring tools.
  • CHANGE: In the Month calendar view, show all times as hh:mmt (if on the hour, it used to skip the :00 part)
  • SYSADMIN: Moved the redis process from the main database machine to the backup.
  • INTERNAL: Skip blank MX records when sending email.
  • SYSADMIN: Upgraded the main database machine.
  • CHANGE: Revamped the download archives function. Fixed a couple of bugs and it now downloads a compressed zip file.
  • SYSADMIN: Separate web serving and email delivery onto different machines.
  • CHANGE: Don't force a digest at 10pm if a digest has been delivered in the last 6 hours.
  • SYSADMIN: Test existing database backups.
  • BUGFIX: Post to sub email footers had the wrong domain for enterprise groups.
  • CHANGE: In posttosub messages, include the group subject tag in the email subject.
  • BUGFIX: When a moderator wanted to post to a subgroup they were not a member of, but had appropriate subgroup perms, it would crash.
  • BUGFIX: For groups that do not resize photos in email, we did not resize them at all in digests (when we should be resizing them to a max of 400x400).
  • NEW: We're now tracking confirmation email delivery attempts and bounces.
  • CHANGE: Added Timezone selector to add event page, so you can explicitly select the timezone instead of your current timezone being silently used.
  • CHANGE: For +owner messages, do same header processing we do with normal group messages (remove a bunch of headers, rename 'Received:' to 'X-Received:'). This is to hopefully increase deliverability of those messages to owners.
  • BUGFIX: In files, if a folder had a # in the name, it screwed up paging through the files list.
  • CHANGE: For messages with more than one rcpt tos, process all of them instead of stopping at the first error.

Have a good weekend everybody.

Mark


moderated Re: Daily Summary Email

 

Maria,

I like this whole thing. People who subscribe to digest currently use it (at least in many cases) just to get an idea of what's happening, and then they reply on the site. So this would work for them.
--
J

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


moderated Re: Daily Summary Email

Maria
 

On Thu, Feb 2, 2017 at 01:02 pm, J_Catlady wrote:
Absolutely! My understanding was that this was going to be a new subscription option. They would choose *either* the summary *or* the digest. Is that a misunderstanding? 

Yes, that's what i meant. Another option not an additional email on top of digests. So you pick: special notice, digest, or daily summary, or individual posts.

I think it might be nice if it started with a summary like:

Your group has N new topics

Your group has N new replies

Then the index of topics  - and I love the idea of including the number icon to indicate how many posts exist in that topic and the graphic for the hashtag assigned to the topic. And would be cool if the hashtag were clickable to bring people to the page with other posts with that hashtag.

Then graphic links to : "Post a New Topic" "Vist Your Group" "Go to Your Account Settings" (just ideas...)

If it's a daily summary - i think it's OK to NOT have the actual posts in there but rather just have it all link one by one via the index to the individual posts/topics on the web interface.

Again - the issue with truncation in the gmail app is so annoying because those who get emails there think it's the service's fault when an email is truncated ( with no link to load rest). I guess it's hard for people to imagine that Gmail's app could be so silly as to do that...So on that account I'd vote to avoid anything that results in clipping.

Also - another reason to not include the whole posts in a daily summary of activity email is because of attachments. On groups where there is an image or 10 on each post, that summary email could be loaded with images and each IOS mail client kinda handles those differently - some well and some not so good.

I think a summary of activity that is easy to skim gives you an idea of what's going on and the opportunity to engage via web. It also helps people not "forget" about the group - something you don't get when you are on special notice only.

Re: snippet: It's tricky... I think the first sentence with an ellipsis to indicate there is more, could be useful to give a feeling that you are skimming, but you'd probably have to show the most recent post in that topic for the snippet, otherwise people will see same snippet each day for as long as that topics is active.

Maria


moderated Re: Daily Summary Email

 

On Thu, Feb 2, 2017 at 11:58 am, Frances wrote:
If someone is overwhelmed with email (even by digests), won’t another email be annoying? If someone is getting 10 or more digests a day, that one summary email may be long.

Absolutely! My understanding was that this was going to be a new subscription option. They would choose *either* the summary *or* the digest. Is that a misunderstanding? 
--
J

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


moderated Re: Daily Summary Email

 

Mark,

I like Maria's idea of a daily summary email. What should it contain?
I like Frances' example (#12836), which looks a lot like the Topics View absent message snippets.
https://groups.io/g/beta/message/12836

If a snippet were included that would open again Brian's question about whether it should be from the most recent message rather than the topic starter. Or the oldest message covered by this summary.

Should it just contain a list of subjects of messages that were sent
over the previous 24 hours?
That might be best, as being the least likely to get snipped by a receiving mail interface.

Also, when should it be sent? I'm thinking perhaps morning time.
"Morning time" where? The subscriber's time zone? That would probably work for most people. Or have it be a subscription parameter.

And of course, is this summary subject to the advanced subscription options? I'd say yes it should be, for the same reasons the Digest is.

--
Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum


moderated Re: Daily Summary Email

Steph Mathews <smathews@...>
 

I think that it should contain links to all of the topics from that day and the replies within that topic.  Steph


On 2/2/2017 1:43 PM, Mark Fletcher wrote:
Hi All,

I like Maria's idea of a daily summary email. What should it contain? If it was just like a digest, it'd be way too long. Should it just contain a list of subjects of messages that were sent over the previous 24 hours?

Also, when should it be sent? I'm thinking perhaps morning time.

Thanks,
Mark


moderated Re: Daily Summary Email

Frances
 

Hi again,

If someone is overwhelmed with email (even by digests), won’t another email be annoying? If someone is getting 10 or more digests a day, that one summary email may be long.

For less active groups, it might be a useful tool though and encourage participation.

Frances


moderated Re: Digests tweak

 

Hi All,

For now, I've changed it so that a digest will not be forced at 10pm pacific time if a digest has been sent out within the previous 6 hours. We'll see how that goes; I can always tweak it.

Thanks,
Mark

On Wed, Feb 1, 2017 at 6:33 PM, Shal Farley <shals2nd@...> wrote:
Mark,

> I am proposing changing the algorithm that forces a digest. My
> proposal is that at 10pm pacific, if there are pending digest
> messages *AND* the person has not received a digest in the last 12
> hours, to generate a digest.

This suggests to me another way of looking at Digests. Maybe instead of a time of day, you can trigger them by a maximum age for the oldest message in the digest when the max count/size hasn't been reached.

The extra bit of complexity might be offset by smoothing out the delivery burden (on the average - you'd still see the occasional rogue wave).

Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum





moderated Re: Daily Summary Email

Frances
 

Hi everyone,

Would it be possible to contain the number of messages for each subject?

It might be interesting but ?not necessary? for the tags to be shown:

But things that do show should be clickable in the email.

Morning would be useful.

Frances

On Feb 2 17, at 2:43 PM, Mark Fletcher <markf@corp.groups.io> wrote:

Hi All,

I like Maria's idea of a daily summary email. What should it contain? If it was just like a digest, it'd be way too long. Should it just contain a list of subjects of messages that were sent over the previous 24 hours?

Also, when should it be sent? I'm thinking perhaps morning time.

Thanks,
Mark


moderated Daily Summary Email

 

Hi All,

I like Maria's idea of a daily summary email. What should it contain? If it was just like a digest, it'd be way too long. Should it just contain a list of subjects of messages that were sent over the previous 24 hours?

Also, when should it be sent? I'm thinking perhaps morning time.

Thanks,
Mark


moderated Re: Digests tweak

 

That was a typo. She received 12 digests, not 11.

Sent from my iPhone

On Feb 2, 2017, at 11:35 AM, J_Catlady <j.olivia.catlady@...> wrote:

In the group in question, the traffic has always been several hundred messages per day. The person who complained is a very experienced group member and practically (although I believe, unofficially) runs the group in question. Yesterday she literally received 11 digests from the IBD group. This is almost causing her to go no-email. 

Those sound like good suggestions.

J

Sent from my iPhone

On Feb 2, 2017, at 10:38 AM, HR Tech via Groups.Io <m.conway11@...> wrote:

In busy groups what I am seeing is: the increase in digests has helped increase activity. So that's a good thing.

But yes, perhaps there are those who are wondering why they are getting 2x the amount of digests, but still may prefer the fact that finally they work well - lets remember that not only were they clipped - they were truncated with no link in gmail app PLUS the reply to links didn't work properly in the expanded version - so big issues.

But I did receive one feedback from a person who felt the frequency of digests was too much (even though it's exactly the way it was on Y! now) - this was a newbie so the prior user "training" didn't apply, and I suspect that even the 1 digest every 25 messages would have felt like too much for that person as they still would have gotten 2 or even 3 in a 24 hour period from 2 of our busy groups. I'm actively requesting feedback so I may learn more.

My suggestion at the moment, is to have an additional email delivery option: "daily summary of activity". This could be every 24 hours/ daily. I'd suggest on that one (since it will be clipped/truncated) to make the index titles link to the actual web version of the posts. I'd also suggest that the reply links link to the web interface so people can use that to reply (because the reply links don't work well after the digest gets truncated). And this could be an elective option for those who don't want multiple digests but don't want special notices only. 

Also - i think once the issue of that 1AM digest is resolved that will help too.

Maria


--
J

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


--
J

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


moderated Re: Digests tweak

 

In the group in question, the traffic has always been several hundred messages per day. The person who complained is a very experienced group member and practically (although I believe, unofficially) runs the group in question. Yesterday she literally received 11 digests from the IBD group. This is almost causing her to go no-email. 

Those sound like good suggestions.

J

Sent from my iPhone

On Feb 2, 2017, at 10:38 AM, HR Tech via Groups.Io <m.conway11@...> wrote:

In busy groups what I am seeing is: the increase in digests has helped increase activity. So that's a good thing.

But yes, perhaps there are those who are wondering why they are getting 2x the amount of digests, but still may prefer the fact that finally they work well - lets remember that not only were they clipped - they were truncated with no link in gmail app PLUS the reply to links didn't work properly in the expanded version - so big issues.

But I did receive one feedback from a person who felt the frequency of digests was too much (even though it's exactly the way it was on Y! now) - this was a newbie so the prior user "training" didn't apply, and I suspect that even the 1 digest every 25 messages would have felt like too much for that person as they still would have gotten 2 or even 3 in a 24 hour period from 2 of our busy groups. I'm actively requesting feedback so I may learn more.

My suggestion at the moment, is to have an additional email delivery option: "daily summary of activity". This could be every 24 hours/ daily. I'd suggest on that one (since it will be clipped/truncated) to make the index titles link to the actual web version of the posts. I'd also suggest that the reply links link to the web interface so people can use that to reply (because the reply links don't work well after the digest gets truncated). And this could be an elective option for those who don't want multiple digests but don't want special notices only. 

Also - i think once the issue of that 1AM digest is resolved that will help too.

Maria


--
J

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


moderated Re: Digests tweak

Frances
 

Hi

Perhaps more info to subscribers about alternative ways to subscribe?

There is a link in the footer to Change Your Subscription. If you get every post as a separate email, we get follow / unfollow.

Not sure about digest. Perhaps links or help there would be good? Or clearer?

Frances


moderated Re: Digests tweak

Maria
 

In busy groups what I am seeing is: the increase in digests has helped increase activity. So that's a good thing.

But yes, perhaps there are those who are wondering why they are getting 2x the amount of digests, but still may prefer the fact that finally they work well - lets remember that not only were they clipped - they were truncated with no link in gmail app PLUS the reply to links didn't work properly in the expanded version - so big issues.

But I did receive one feedback from a person who felt the frequency of digests was too much (even though it's exactly the way it was on Y! now) - this was a newbie so the prior user "training" didn't apply, and I suspect that even the 1 digest every 25 messages would have felt like too much for that person as they still would have gotten 2 or even 3 in a 24 hour period from 2 of our busy groups. I'm actively requesting feedback so I may learn more.

My suggestion at the moment, is to have an additional email delivery option: "daily summary of activity". This could be every 24 hours/ daily. I'd suggest on that one (since it will be clipped/truncated) to make the index titles link to the actual web version of the posts. I'd also suggest that the reply links link to the web interface so people can use that to reply (because the reply links don't work well after the digest gets truncated). And this could be an elective option for those who don't want multiple digests but don't want special notices only. 

Also - i think once the issue of that 1AM digest is resolved that will help too.

Maria


moderated Re: Digests tweak

 

Just got an offlist message from a member of my group who's also in another group, that one with a ginormous amount of daily traffic. She is being driven crazy, says she's now getting about a dozen digests a day. Suggests that the number of posts in a digest could be a member option. She had no problem with her email trimming long messages.

It does seem optimal, if at all feasible to implement, to let the number of posts be a member option. 
--
J

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


moderated Re: Event: Site maintenance, Thursday, 2 February 2017 #downtime #cal-invite

 

I love the banner, BTW, "down for upgrades."

"Are you down for upgrades?" "Yeah I'm down" LOL

--
J

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


moderated Re: Event: Site maintenance, Thursday, 2 February 2017 #downtime #cal-invite

Joseph Hudson <jhud7789@...>
 

Oh okay thanks.

On Feb 1, 2017, at 6:13 PM, Shal Farley <shals2nd@gmail.com> wrote:

Joeseph,

Sugar, is that you trying to agree with me? I've never heard of that
terminology before.
Yes, it is a cry of agreement. Mostly British, I think.
https://en.wiktionary.org/wiki/hear,_hear

--
Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum



moderated Re: Digests tweak

 

Mark,

I am proposing changing the algorithm that forces a digest. My
proposal is that at 10pm pacific, if there are pending digest
messages *AND* the person has not received a digest in the last 12
hours, to generate a digest.
This suggests to me another way of looking at Digests. Maybe instead of a time of day, you can trigger them by a maximum age for the oldest message in the digest when the max count/size hasn't been reached.

The extra bit of complexity might be offset by smoothing out the delivery burden (on the average - you'd still see the occasional rogue wave).

Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum


moderated Re: Event: Site maintenance, Thursday, 2 February 2017 #downtime #cal-invite

 

Joeseph,

Sugar, is that you trying to agree with me? I've never heard of that
terminology before.
Yes, it is a cry of agreement. Mostly British, I think.
https://en.wiktionary.org/wiki/hear,_hear

--
Shal
https://groups.io/g/Group_Help
https://groups.io/g/GroupManagersForum

19261 - 19280 of 31945