I am very sad to report that one of our servers has been hit by a power failure that resulted in filesystem inconsistency. Unfortunately this resulted in the global unavailability of Wikidot. But before I explain any further, I must say that:
No data is lost
except for file uploads from the last half hour before the crash.
While recovering the failed server, we have reconfigured the rest of our servers to bring back as much functionality as possible.
The services that do not work yet include:
- activities: no new activities are created when pages are saved or edited; activities cannot be browsed FIXED!
- thumbnails: some need to be re-generated; until then some can be missing
- uploaded files: it might take longer than usual to request uploaded files until our cache servers "warm up"
- translations might not work on all sites FIXED!
- site statistics are not available right now, and some data from today might be missing
-
- update: site statistics are available for 2012 now, stats for yesterday and today are being generated
-
Unfortunately some time passed before we could bring Wikidot up after the failure. Meanwhile we have reported our Facebook profile and Twitter page.
I am very sorry about the issue — believe me, we are working as hard as we can to bring remaining functionality back and to minimize the negative impact of the failure.
I will be posting updates about our status.
Best regards and thanks for your patience,
Michal
Nice to know that you guys are on top of it. By the way, the following update page, whose link is given in the "Please try again" page, is not updated at all. The last update listed there is from 2009:
http://www.wikidot.com/current-problems
Thanks for taking care of this issue, and good luck with everything!
Eduardo R. Ribeiro
http://www.etnolinguistica.org
Perhaps this page should be updated to include links to the Twitter and Facebook accounts? That seems to be where things are announced when problems occur.
~ Leiger - Wikidot Community Admin - Volunteer
Wikidot: Official Documentation | Wikidot Discord server | NEW: Wikiroo, backup tool (in development)
Great idea, Shane. I love the Facebook and Twitter updates, so that I can share with/retweet to my users anytime there's such an issue. Also, it would be nice if the "Please try again" page were available from minute one (would it be the case of hosting it somewhere else, just in case?).
The usefulness of Twitter and Facebook in this situation shows the value of outside services to keep our communities informed when something goes wrong. I know problems such as these are bound to happen; what can and should be avoided is leaving the public in the dark. If we could have "emergency" landing pages for each site, with a customizable message, that would be perfect.
Eduardo R. Ribeiro
http://www.etnolinguistica.org
Michal, what the heck is going on with my site?
That was my test account so I meant to post with this one. What is going on with my site, all my downloads are gone!!!!!!!!!!! WTF!! Do you uderstand how critical this is to happen today of all days! My sites core feature is the photoshop downloads, now what the hell am I supposed to do???? Almost every single form data .zip upload will not upload or downlload!!! I know other sites aren't that much of a priority like my site is. I should be getting almost 200,000 hits today, but I'm not now because this happened! Now my site rank as well as it's popularity is going to go down because of this. Do you know how many people view my site everyday as well as bringing a sh** load of unique visits to wikidot. I'm busy every day now because of my site, now what is going on??
CEO of Icon Deposit
Take a look at me via Twitter, Dribbble, and Google +
We are checking it now, me will provide more information asap.
Bartłomiej Bąkowski @ Wikidot Inc.
';.;' TeRq (Write PM)
This needs to be fixed today or now. Plus this had to happen when I release something HUGE on my site!!
CEO of Icon Deposit
Take a look at me via Twitter, Dribbble, and Google +
It's fixed.
Bartłomiej Bąkowski @ Wikidot Inc.
';.;' TeRq (Write PM)
Hey Matt,
Looking at your Icon Deposit site, about 50% of the images are loading for me. If I click on the "Files" link on your home page, there are about two pages worth of files, so I suspect all of them are there.
On this page, the download button image does not load, but I was able to guess at its location - clicking there allows me to download the PSD file. So it seems that the data form files are still available.
I understand that this is causing problems for you, but just thought I'd let you know that I was able to access those files … and therefore, all of your work and the work of your contributors is not permanently lost.
~ Leiger - Wikidot Community Admin - Volunteer
Wikidot: Official Documentation | Wikidot Discord server | NEW: Wikiroo, backup tool (in development)
Shane, Thank you man so much! You just lifted a lot of weight off my shoulders bud. So when this is all fixed, everything will be back the way it was yesterday?
CEO of Icon Deposit
Take a look at me via Twitter, Dribbble, and Google +
Why on earth Wikidot does not use fault tolerant architecture? Should we move some critical files outside wikidot?
All the files are securely hosted on Amazon S3 and we only cache them locally. All the "missing" files are stored fine in our S3 bucket. We are trying to find out why the hell they are not being downloaded and served by our servers.
Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalf.me
Michal, go to my site, if you downloaded something from my site before or viewed it before, you know what it should look like, I need those files back asap, none of the .zip form data will download. It's very critical especially for today when I released a new CSS3 UI Kit. I'm supposed to be getting a lot of traffic today, but when they go to the site, they are very disappointed.
CEO of Icon Deposit
Take a look at me via Twitter, Dribbble, and Google +
U said "No data is lost except for file uploads from the last half hour before the crush." In fact all my data uploaded one - two - three - and so on weeks ago are lost . They are very critical as contain some important scripts. Thinking to move from Wikidot..
Given that the problem only started a few hours ago and that Wikidot is currently working on resolving the problem, it may pay to wait a while - your files are still available (as Michal stated, everything is backed-up onto Amazon S3 servers).
The files are still there, it's just that you're temporarily unable to access them & I'm sure that'll be sorted out soon.
~ Leiger - Wikidot Community Admin - Volunteer
Wikidot: Official Documentation | Wikidot Discord server | NEW: Wikiroo, backup tool (in development)
Please check again — ALL files should be back. If they are not, we still have backups, but I can bet they will not be necessary.
I know how frustrating such situations are. For all of us. But things DO happen — no matter how redundant the infrastructure is there is always place for errors. Definitely there are still things we can work on.
People, check your websites and give us feedback if things work as expected. We will highly appreciate it!
Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalf.me
You are the mann! … /genius ;) <3
CEO of Icon Deposit
Take a look at me via Twitter, Dribbble, and Google +
The problem seems to be solved. Everything is back. What was that? As i got is you relay files stored on Amazon Cloud? Should we expect something similar in the future? Need to know as I have to make decision.
The last problem was with improperly cleared memory cache (Memcached) — our file caching servers got outdated information which files should be proxied on our disk, but when not finding them they were throwing errors (we had to start the file cache from zero, copying everything from S3).
Now it seems Wikidot is 100% up and running. I owe you huge apologies — I hope you will accept it!
Michał Frąckowiak @ Wikidot Inc.
Visit my blog at michalf.me
No problem, you got it up and running quicker than I anticipated.
Michal, my site statistics are saying they do not exist. Do you know what happened with that?
CEO of Icon Deposit
Take a look at me via Twitter, Dribbble, and Google +
Matt, they don't do it deliberately to spite you! And yours is not the only site that a) was affected and b) is critical to the people that run or use it. I had the Scottish Government on the phone several times today being a bit unhappy that their sites and the databases we deliver through them were down. In the UK we have a saying that stuff like this is Sod's Law; in other words it always happens just when you really really need Wikidot to work well. With me it's usually when I am doing a presentation about how good Wikidot is!
But this happens with any platform, hardware and software from time to time. The important thing is whether Wikidot is responsive when it comes to fixing the problems. And I think they are.
Rob Elliott - Strathpeffer, Scotland - Wikidot first line support & community admin team.
I know that, I'm not blaming them for that one, it was just a coincidence.
CEO of Icon Deposit
Take a look at me via Twitter, Dribbble, and Google +