We prepare a lot for I’m a Scientist. We check our lists, we cross off tasks, we write new lists of things to check. The one thing we can’t really do is test how the site will operate under stress. It really helps to have a couple of hundred students trying to access the live chats for that. This morning you obliged and we found something was lacking.
A piece of technical wizardry called MemcacheD wasn’t running. MemcacheD basically stores the most used parts of the database in memory and it means less work for the database. Without it the database gets overloaded. This morning with three live chats all at the same time it got overloaded and eventually the server stopped working.
The good news is we got MemcacheD started again just before we needed to reboot the server. Since then the site has been used very heavily again but without slowing down the server at all. Yay!
So, I want to say Sorry to the students, teachers and scientists for disrupting the live chats. If any affected teachers would like to rebook please email us as normal. Good luck with the rest of the event.