moderated Happy New Year!


 

Hi All,

I am back from vacation. I hope you all had a good end of 2017! It will take me a couple of days to work through support emails. Also, there's a backlog of group transfer requests that I need to work through.

I am speculating here, but there may have to be some downtime in the next week or so. There's been a major security flaw discovered in all Intel CPUs from the past decade, and all operating systems will need to be patched to fix it. Linode, our hosting provider, has not yet said when or if our instances will have to be rebooted or migrated to install the security fixes, but I'm guessing that it will need to happen. As soon as I know more, I'll post. Nerdy details on the flaw are here: https://www.theregister.co.uk/2018/01/02/intel_cpu_design_flaw/

Thanks,
Mark


 

Mark,

Welcome back! Happy New Year to all!

--
J

 

Messages are the sole opinion of the author, especially the fishy ones.

I wish I could shut up, but I can't, and I won't. - Desmond Tutu


Ginny T.
 

Happy New Year to you, too Mark (and everyone else!). Welcome back and I hope that you had a good and much-deserved rest and break.
Thanks again for everything,
Ginny
--
Ginny T.  gttemari21@...
========
TemariKai.com


 

Hi All,

As I mentioned, I thought there would have to be some downtime soon to patch a couple of security flaws. Perhaps you've seen the coverage of the Meltdown and Spectre flaws in the news. Anyways, I still don't have a timeframe yet as to when it will happen. If I had to guess, it'll be some time this weekend, but I'm still waiting on Linode to give us answers. There is also a possibility that there might have to be two downtime periods. I just don't know yet. Hopefully, the downtime will be limited to at most an hour. I will post more when I know.

I am still working through the backlog of group transfers, but should be caught up tomorrow. Support email should also be caught up after tomorrow. And then I will get to the beta@ messages.

Thanks, Mark


Douglas Swearingen
 

Thank You Mark for everything you do to make groups.io the success it is.

Doug


On Thursday, January 4, 2018 10:16 PM, Mark Fletcher <markf@corp.groups.io> wrote:


Hi All,
As I mentioned, I thought there would have to be some downtime soon to patch a couple of security flaws. Perhaps you've seen the coverage of the Meltdown and Spectre flaws in the news. Anyways, I still don't have a timeframe yet as to when it will happen. If I had to guess, it'll be some time this weekend, but I'm still waiting on Linode to give us answers. There is also a possibility that there might have to be two downtime periods. I just don't know yet. Hopefully, the downtime will be limited to at most an hour. I will post more when I know.
I am still working through the backlog of group transfers, but should be caught up tomorrow. Support email should also be caught up after tomorrow. And then I will get to the beta@ messages.
Thanks, Mark



Marv Waschke
 

Are you expecting post-patch performance issues? 30% hits keep being mentioned, but I suspect that will depend a lot on circumstances— some processes hit hard, others barely touched. Best, Marv Waschke

 

From: main@beta.groups.io [mailto:main@beta.groups.io] On Behalf Of Mark Fletcher
Sent: Thursday, January 04, 2018 9:17 PM
To: main@beta.groups.io
Subject: Re: [beta] Happy New Year!

 

Hi All,

As I mentioned, I thought there would have to be some downtime soon to patch a couple of security flaws. Perhaps you've seen the coverage of the Meltdown and Spectre flaws in the news. Anyways, I still don't have a timeframe yet as to when it will happen. If I had to guess, it'll be some time this weekend, but I'm still waiting on Linode to give us answers. There is also a possibility that there might have to be two downtime periods. I just don't know yet. Hopefully, the downtime will be limited to at most an hour. I will post more when I know.

I am still working through the backlog of group transfers, but should be caught up tomorrow. Support email should also be caught up after tomorrow. And then I will get to the beta@ messages.

Thanks, Mark


 

On Fri, Jan 5, 2018 at 10:13 AM, Marv Waschke <marv@...> wrote:

Are you expecting post-patch performance issues? 30% hits keep being mentioned, but I suspect that will depend a lot on circumstances— some processes hit hard, others barely touched. Best, Marv Waschke

 


I've seen various numbers, and it apparently depends on the workload. I'm not worried about most of the instances that run the service. But I am concerned about the main Postgres instance and the Redis instance. Unfortunately, it's not something I can really test ahead of time. And it's not like I can not patch the instances. So, fingers crossed, and also hopefully I'll be able to do the upgrades over the weekend, when there's less traffic, in case there's a problem.

Thanks,
Mark


Marv Waschke
 

Your architecture of separated and simple db instances and more processing in the application layer may have been prescient— that may limit the situations where the context switches and pre-execution  munge performance for PostGres. Not sure of implications for Redis. I see that AWS says the performance hits have not been bad. I’ll quit prattling. Best, Marv Waschke


 

Hi All,

Linode is now saying to expect reboots of all machines in the next couple of days to address the Meltdown bug. Unfortunately, I will only have 24 hours notice for these reboots. Also, there are two machines that are critical for Groups.io (all the other ones have backups/can be rebooted without affecting the site); if those two machines are not rebooted at the same time, then there will have to be two separate downtimes. I will be given 2 hour windows for the reboots. The reboots themselves should only take a few minutes.

Also, to address the Spectre class of bugs, there will have to be another round of reboots at some point in the future.

More when I know.

Thanks, Mark