New ListPages Implementation

by Gabrys on 29 May 2010 10:08

flickr:4643329494

Yesterday we replaced ListPages module with its new implementation. It works exactly the same for users, but is much more elegant for a programmer.

The module is now divided into two layers. One layer is selecting pages (named SelectPages API) based on criteria and the second layer is displaying the pages using given format. SelectPages API is reusable so we will use it at least for generating RSS feeds (so that they can have the same parameters passed by URL as ListPages has).

The reason ListPages was not divided that way from the beginning was that it was developed rapidly and improved over months — by adding new stuff, so it eventually grew up to over 1400 lines. RSS feed generation is an older feature of Wikidot that was never meant to support ListPages-like options.

Over time the ListPages module has emerged as a standard way of doing things at Wikidot so we wanted to make more things behave like ListPages, and to support the same parameters.

Testing the new module

To make sure we didn't break anything, we tested over 1000 real pages containing ListPages module from Wikidot sites. We tested whether both old and new ListPages module implementation return the same HTML. Doing this we fixed bugs in the new module and discovered a few never reported bugs in the old one including:

  • links to RSS feeds were sometimes generated wrongly (categories had random minuses in front of their names)
  • passing tags=" " as parameter to ListPages worked the same as tags="-" and should just be ignored
  • %%total%% symbol in the module body ignored the limit parameter, and so, reported the wrong number of pages.

All those bugs are fixed in the new ListPages. Good thing was all bugs in the new module were not caused by bad SelectPages API. This means the API was designed well and testing it separately in advance revealed all bugs.

The future of ListPages module

We want to make the order parameter validated, so that if you ask to order by something that is not supported, you are given an error message. We decided not to do it yet, because we found many modules with invalid order parameters and the change would just break them. We need to figure out what to do in order to let users know something is wrong with their modules while not breaking too many pages.

If you still wonder why we moved selecting pages to another level of abstraction, there's one more reason. In future we want to replace this code. Now it queries the database for pages matching given criteria and in future we want a separate Lucene index just for this purpose. This would allow us to make queries based on data form fields and other page properties without killing the database server.

This separation means that even if the ListPages index breaks for some reason, you can still view static Wikidot pages. Having particular features (especially those that are CPU-, memory- or disk-intensive) separate from others raises the stability of Wikidot. We have already separated the search service and user-uploaded files from the rest of Wikidot. Internally — we moved all background tasks out of application servers, to make them only do their job, nothing more.


The image comes from Flickr

Comments: 28

Add a New Comment