Very impressive. Just imagine, if you will, yahoo creating a report like this for every outage.

https://www.youtube.com/watch?v=NzlG28B-R8Y

--
J

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu

Outage Summary

Email delivery was dramatically slowed overnight due to a crashing bug in the karl process. I fixed the bug around 6:25am pacific time, and it took about 25 minutes for all queued email to be delivered.

Duration

From approximately around midnight pacific time until 6:25am pacific time.

Cause

A group was transferred from Yahoo yesterday with a lot of users from a bogus/typo domain, yahooo.com. That domain has an MX record that is blank. We were not ignoring blank MX records. What that meant is that for email to those users, we'd connect to the localhost SMTP port to send this email, essentially DOSing ourselves. I was alerted to this behavior on Friday evening via our normal alerts. I wrote a fix and pushed it to the site. Because of how we retry sending messages, the code introduced with the fix was not run until after I went to bed. I did not consider the fix to be risky, but the fix contained a divide by zero, causing a crash. The karl process then continued to crash until I woke up and saw the problem. I was not alerted to the crashes.

Areas Of Improvement

• I was not alerted to the crashes because our alerts are emailed to Pagerduty using karl. Alerts should be sent directly to Pagerduty using their API instead of via email.

Thanks, Mark

Changes to the site this week:

• SYSADMIN: Added more database indexes to speed up operations.
• SYSADMIN: Worked on better database monitoring tools.
• CHANGE: In the Month calendar view, show all times as hh:mmt (if on the hour, it used to skip the :00 part)
• SYSADMIN: Moved the redis process from the main database machine to the backup.
• INTERNAL: Skip blank MX records when sending email.
• SYSADMIN: Upgraded the main database machine.
• CHANGE: Revamped the download archives function. Fixed a couple of bugs and it now downloads a compressed zip file.
• SYSADMIN: Separate web serving and email delivery onto different machines.
• CHANGE: Don't force a digest at 10pm if a digest has been delivered in the last 6 hours.
• SYSADMIN: Test existing database backups.
• BUGFIX: Post to sub email footers had the wrong domain for enterprise groups.
• CHANGE: In posttosub messages, include the group subject tag in the email subject.
• BUGFIX: When a moderator wanted to post to a subgroup they were not a member of, but had appropriate subgroup perms, it would crash.
• BUGFIX: For groups that do not resize photos in email, we did not resize them at all in digests (when we should be resizing them to a max of 400x400).
• NEW: We're now tracking confirmation email delivery attempts and bounces.
• CHANGE: Added Timezone selector to add event page, so you can explicitly select the timezone instead of your current timezone being silently used.
• CHANGE: For +owner messages, do same header processing we do with normal group messages (remove a bunch of headers, rename 'Received:' to 'X-Received:'). This is to hopefully increase deliverability of those messages to owners.
• BUGFIX: In files, if a folder had a # in the name, it screwed up paging through the files list.
• CHANGE: For messages with more than one rcpt tos, process all of them instead of stopping at the first error.

Have a good weekend everybody.

Mark

Maria,

I like this whole thing. People who subscribe to digest currently use it (at least in many cases) just to get an idea of what's happening, and then they reply on the site. So this would work for them.
--
J

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu

Absolutely! My understanding was that this was going to be a new subscription option. They would choose *either* the summary *or* the digest. Is that a misunderstanding? 

Yes, that's what i meant. Another option not an additional email on top of digests. So you pick: special notice, digest, or daily summary, or individual posts.

I think it might be nice if it started with a summary like:

Your group has N new topics

Your group has N new replies

Then the index of topics  - and I love the idea of including the number icon to indicate how many posts exist in that topic and the graphic for the hashtag assigned to the topic. And would be cool if the hashtag were clickable to bring people to the page with other posts with that hashtag.

Then graphic links to : "Post a New Topic" "Vist Your Group" "Go to Your Account Settings" (just ideas...)

If it's a daily summary - i think it's OK to NOT have the actual posts in there but rather just have it all link one by one via the index to the individual posts/topics on the web interface.

Again - the issue with truncation in the gmail app is so annoying because those who get emails there think it's the service's fault when an email is truncated ( with no link to load rest). I guess it's hard for people to imagine that Gmail's app could be so silly as to do that...So on that account I'd vote to avoid anything that results in clipping.

Also - another reason to not include the whole posts in a daily summary of activity email is because of attachments. On groups where there is an image or 10 on each post, that summary email could be loaded with images and each IOS mail client kinda handles those differently - some well and some not so good.

I think a summary of activity that is easy to skim gives you an idea of what's going on and the opportunity to engage via web. It also helps people not "forget" about the group - something you don't get when you are on special notice only.

Re: snippet: It's tricky... I think the first sentence with an ellipsis to indicate there is more, could be useful to give a feeling that you are skimming, but you'd probably have to show the most recent post in that topic for the snippet, otherwise people will see same snippet each day for as long as that topics is active.

Maria

If someone is overwhelmed with email (even by digests), won’t another email be annoying? If someone is getting 10 or more digests a day, that one summary email may be long.

Absolutely! My understanding was that this was going to be a new subscription option. They would choose *either* the summary *or* the digest. Is that a misunderstanding? 
--
J

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu

I like Maria's idea of a daily summary email. What should it contain?

Should it just contain a list of subjects of messages that were sent
over the previous 24 hours?

Also, when should it be sent? I'm thinking perhaps morning time.

I think that it should contain links to all of the topics from that day and the replies within that topic.  Steph

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu

In busy groups what I am seeing is: the increase in digests has helped increase activity. So that's a good thing.

But yes, perhaps there are those who are wondering why they are getting 2x the amount of digests, but still may prefer the fact that finally they work well - lets remember that not only were they clipped - they were truncated with no link in gmail app PLUS the reply to links didn't work properly in the expanded version - so big issues.

But I did receive one feedback from a person who felt the frequency of digests was too much (even though it's exactly the way it was on Y! now) - this was a newbie so the prior user "training" didn't apply, and I suspect that even the 1 digest every 25 messages would have felt like too much for that person as they still would have gotten 2 or even 3 in a 24 hour period from 2 of our busy groups. I'm actively requesting feedback so I may learn more.

My suggestion at the moment, is to have an additional email delivery option: "daily summary of activity". This could be every 24 hours/ daily. I'd suggest on that one (since it will be clipped/truncated) to make the index titles link to the actual web version of the posts. I'd also suggest that the reply links link to the web interface so people can use that to reply (because the reply links don't work well after the digest gets truncated). And this could be an elective option for those who don't want multiple digests but don't want special notices only. 

Also - i think once the issue of that 1AM digest is resolved that will help too.

Maria

Just got an offlist message from a member of my group who's also in another group, that one with a ginormous amount of daily traffic. She is being driven crazy, says she's now getting about a dozen digests a day. Suggests that the number of posts in a digest could be a member option. She had no problem with her email trimming long messages.

It does seem optimal, if at all feasible to implement, to let the number of posts be a member option. 
--
J

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu

I love the banner, BTW, "down for upgrades."

"Are you down for upgrades?" "Yeah I'm down" LOL

--
J

Messages are the sole opinion of the author. Especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu