moderated Ongoing issue #outage


 

Hi All,

The site is having issues. Something is up with either the Linode internal network or Amazon S3. I'm working to figure it out.

Thanks,
Mark


Glenn Glazer
 

On 12/15/2021 07:58, Mark Fletcher wrote:

Hi All,

The site is having issues. Something is up with either the Linode internal network or Amazon S3. I'm working to figure it out.

Thanks,
Mark


Since we were also dealing with this at work, AWS had a "connectivity outage" in US-WEST-1, US-WEST-2 and US-GOV-WEST-1. Our first message from their bot was at 1558 UTC, seems to have started resolving approximately 15 minutes ago.

Best,

Glenn

--
#calcare
PG&E Delenda Est


 

On Wed, Dec 15, 2021 at 9:31 AM Glenn Glazer <glenn.glazer@...> wrote:

Since we were also dealing with this at work, AWS had a "connectivity outage" in US-WEST-1, US-WEST-2 and US-GOV-WEST-1. Our first message from their bot was at 1558 UTC, seems to have started resolving approximately 15 minutes ago.

Good to know. Our issues started at around 7:19am pacific time, so a bit before AWS's issues were reported to you (we only use S3 from AWS). I did see a lot of connection issues to S3, but I don't know yet if that was because of an S3 issue or because of an internal Linode network issue. I do know that I never got a page about the problems, because Pagerduty (the service we use to send me pages) had problems at the same time we did (which makes me think it was a much wider issue with AWS and not Linode). Fortunately I was on the system when it started to happen, so I saw it immediately.

I'm still working through all the outage data and doing a postmortem, which I'll post here probably tomorrow. We've been back and normal since 7:57am pacific time. 

Thanks,
Mark


Glenn Glazer
 

On 12/15/2021 09:54, Mark Fletcher wrote:
On Wed, Dec 15, 2021 at 9:31 AM Glenn Glazer <glenn.glazer@...> wrote:

Since we were also dealing with this at work, AWS had a "connectivity outage" in US-WEST-1, US-WEST-2 and US-GOV-WEST-1. Our first message from their bot was at 1558 UTC, seems to have started resolving approximately 15 minutes ago.

Good to know. Our issues started at around 7:19am pacific time, so a bit before AWS's issues were reported to you (we only use S3 from AWS). I did see a lot of connection issues to S3, but I don't know yet if that was because of an S3 issue or because of an internal Linode network issue. I do know that I never got a page about the problems, because Pagerduty (the service we use to send me pages) had problems at the same time we did (which makes me think it was a much wider issue with AWS and not Linode). Fortunately I was on the system when it started to happen, so I saw it immediately.

I'm still working through all the outage data and doing a postmortem, which I'll post here probably tomorrow. We've been back and normal since 7:57am pacific time. 

Thanks,
Mark

You're welcome. By way of commiseration, we had the same problems with PagerDuty. XD

From https://support.pagerduty.com/docs/service-regions:

Hosting Locations
For customer using our US service region, PagerDuty could be running from any of the following data centers at any point in time:

AWS US West (Northern California)
AWS US West (Oregon)
AWS US East (Ohio)

For customers using our EU service region, PagerDuty could be running from either of the below at any point in time:

AWS EU Central (Frankfurt)
AWS EU West (Ireland)

So I think they are part of the AWS impact. Linode, obviously not.

Best,

Glenn

--
#calcare
PG&E Delenda Est