As most of you know, one of cool Wikidot features is traffic analysis. For each site (actually only paid sites till now) we collect available traffic information and every few hours we analyze it using popular analytics software AWStats. The result — number of page views, visits, traffic sources, browser info, crawler stats and tons of other metrics — is then available from within ManageSite panel.
For some of you (especially managing high-traffic sites) it became clear that something had been wrong with stats for the last few weeks. And by "something" I mean large delays in analyzing logs, missing traffic and temporarily unavailable result pages. We are really sorry about these problems and we have been working hard (often till late night hours) to fix them.
The reason for these problems was the faulty (non-scaling) design of our log storage and log processing architecture. Two bottlenecks have been causing a lot of problems recently: central database used to store log files from several web servers and shared filesystem for analyzed data. These bottlenecks became even more apparent after moving servers to AWS.
Yesterday we moved to a completely redesigned (shared-nothing) architecture. We created it from scratch. All data and logs are now stored in a scalable storage (Amazon S3) and are being processed by independent workers.
A few days ago we needed over 6 hours to calculate web traffic for all eligible sites. Now it takes 30 minutes. 12-fold improvement. And we can make it even faster by increasing the number of workers. The instant benefit is obvious: more up-to-date and reliable results. Finally!
Last but not least: as the post title says, we are now adding web analysis to all free educational sites. We believe it is a nice addition to this increasingly popular plan!