Earlier today, between 2-3 AM UTC time, one of our main web servers started to behave strangely. Most of its backend processes (PHP) were hanging, and the remaining could not cope with handling the incoming traffic. Technical details aside, this took Wikidot (and hosted wikis) down and the service was not available for about an hour.
For the last half the year, Wikidot.com was up for more than 99.90% of time, with exceptional february-uptime 99.994% uptime and February and 99.90% in March. Our monitoring and early-alert system prevent most incidents even before they happen. So a situation like this one is really rare.
I am personally sorry about the incident, especially that performance and availability are our priorities.
The only excuse we have is that it happened at early Sunday morning / night, which is when we do not work (usually) and need to rely on automated monitoring and notification system. We will certainly analyze this case and draw conclusions to prevent similar incidents in the future.
Michał Frąckowiak, CEO
Image by BY-YOUR-⌘
Thanks for the update — was wondering what had happened ;-)
~ Leiger - Wikidot Community Admin - Volunteer
Wikidot: Official Documentation | Wikidot Discord server | NEW: Wikiroo, backup tool (in development)
Thank you for posting this so quickly, Michał, and for dealing with it promptly at odd hours.
This week, IE has been hanging a lot when I try to access custom domains. It times out at
I regularly need to clear my cache in order to get back in. For me, it's an annoyance. For many of our users, it's a roadblock.
Any recent changes that would cause this?
Thanks, I will look at this later. After quick testing I can only say that my Windows XP with IE 8.0 does not have any problems with custom domains and authentication.
Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalf.me
I'm running the same.
I'm still getting this in IE8. Nothing loads, and it times out at
I think my IP address changes pretty regularly — would that have something to do with it?
A lot of people have a dynamic IP address, as ISP's usually charge extra for the "privilege" of having a non-dynamic (static) IP. So I doubt that is the problem :S
~ Leiger - Wikidot Community Admin - Volunteer
Wikidot: Official Documentation | Wikidot Discord server | NEW: Wikiroo, backup tool (in development)
Thanks for the explanation anyway. This morning we got a few alerts from our monitoring service, and we were wondering what happened. It's nice to have such a transparent service provider. :-)
I discovered the reason why I did not get notifications about server problems. I rely on notifications sent by email to a special email account that instantly pushes every new email to my iPhone. But the phone for some reason could not log-in into the cellular network and was working only with my WiFi.
Now the thing I learned: Push notifications do not work with WiFi only when the phone is in "sleep" mode, because WiFi connection is closed to save battery. Without 3G/Edge connection, the phone does not have an open internet connection and cannot receive messages. When I touched the phone to check the time, I instantly got about 50 emails.
To prevent this, the simplest solution is to go back to SMS notifications as the second delivery channel.
Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalf.me
Switch to an Android smartphone perhaps? :-)
Or just go Settings » Mail » Fetch New Data » Fetch every 15 minutes when “Push” is unavailable
Yes, this would work, but:
1. then I would not get notifications instantly, but in bulk with a long delay.
2. I do not want to automatically be notified about any other emails from other mailboxes — I read them in ~ 1h intervals, that is enough, and does not disturb my workflow.
What I really like about the current setup is that when I start receiving emails every 10 seconds, it is a clear indications something is wrong and intervention is required. It easily wakes me up at night too.
Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalf.me
Reminds me of my life some 10 years ago…
Hire someone who wakes up for you, instead. :D
Well, we are still a small team, and we share various responsibilities ;-)
Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalf.me