moderated Event: Unplanned downtime #outage - Monday, 2 July 2018 #cal-invite #outage Calendar <main@...>

Unplanned downtime #outage

Monday, 2 July 2018
9:00pm to 9:05pm
(GMT-07:00) America/Los Angeles


An update to the site went badly, and I had to revert the changes. The site was down for about 5 minutes.

Gritty details: I changed a low level library that deals with converting data to and from JSON. This mistakenly caused many integers to be output as named strings in JSON, which caused problems for the code that was expecting to deal with integers. This caused some errors to show on the website and some emails to be sent without footers. I caught it within about 10 seconds of going live.


Hi All,

As I discovered early this morning, the site rollback didn't restart some cron jobs. This meant that event reminders and some digests were delayed, and summaries did not go out for 4 hours. My apologies; rollbacks don't happen much at all (I cannot remember the last time I had to do one), so it was some dusty code. It's been fixed for the future.

Thanks, Mark


On Tue, Jul 3, 2018 at 05:54 am, Mark Fletcher wrote:
so it was some dusty code. It's been fixed for the future.
And I'd bet that by the time this new code is needed, it will have become dusty! ;>)