This is a different #changelog from normal. Most everything outlined below represents internal changes to the site. This likely won't make sense to most people, but I'm writing this all down so that I can remember it in the future. The one new thing that will affect people are the new FBL subscriptions, outlined below.
I switched main database machines late last week because of a (minor) corruption issue. The new main database machine was running on the same type of hardware as the old machine, and I had tested it for several weeks beforehand. However, starting at the beginning of this week (we're busiest on Mondays), it was clear that for some reason the new database machine was not up to handling the load of the site. The site was crashing regularly, and I was getting paged in the middle of the night. It took me a few days to rule out other issues and come to the conclusion that it was a kernel issue. The new database machine was running an older Linux kernel, which normally shouldn't affect performance, but I believe in this case was. As part of my debugging process, I optimized many database access patterns, which will be great going forward. I have changed the Linux kernel on all database machines, and, combined with the other optimizations I did, I believe that the performance issues have been addressed. Hopefully.
At the same time, email to Comcast, Cox and GoDaddy was being delayed. I was finally able to establish contact with both Comcast and Cox and got the throttling lifted. As part of that, Comcast asked that I subscribe to their FBL (Feed Back Loop) service. An FBL is how we get notified when someone marks an email of ours as spam. So we are now receiving spam reports from several more email providers, including Comcast, Cox, Fastmail, Mail.ru, and others. The other recommendation that Comcast had was that we reuse SMTP connections when sending email to them, something we were not doing. So I spent the time to implement that. As of now, email to Comcast and Cox is flowing again. Email to GoDaddy/Secureserver is still being throttled and I have not had any luck contacting them as of yet.
Take care everyone.