moderated Event: Website outage #outage - Tuesday, 21 August 2018 #outage #cal-invite Calendar <main@...>

Website outage #outage

Tuesday, 21 August 2018
6:50am to 7:50am
(GMT-07:00) America/Los Angeles


At approximately 6:50am, the website started to become unresponsive. Internally, the web servers were rebooting because they were running out of database connections. I originally thought that this was a database issue, but after much investigation, I was able to track down the problem. A fix I pushed to the site yesterday afternoon affecting Chat was now causing database connections to leak. I have temporarily disabled the Chat function until I can fix the bugfix. Email delivery was not affected during this time.

I will post a post mortem either later today or tomorrow, after I've reinstated Chat and had some time to consider what, if any, changes to make based on this outage.