Doing risky things can be a lot of fun. A solid dose of adrenaline, quick decisions, testing limits of your abilities and equipment.

The long story short, yesterday after work we went for little sight-seeing with my car. The day before I left all my recovery equipment at home (even including my air compressor), so we just wanted to explore the area of the Tążyna river and find remains of an old village.

One false move and we ended with the Jeep stuck in some swampy mud in the middle of nowhere. Right. And all we had was one folding shovel. No ropes, no farm jack, no winch. Even mobile coverage was not available most of the time.

After a few minutes of digging it was clear we cannot do it by ourselves. All our off-road friends we called were out of town.

Piotr and Łukasz went to find help in the nearest village. Meantime I was clearing the ground with the shovel and was all dirty in mud. After 2 hours they came with a guy on a tractor. We thought we were saved. At last! But after a few minutes the tractor got stuck too. And we were a feet deeper in the mud. Total failure.

So the guy went for a bigger (4x4) tractor. And this time, it did work. We still had to find someone with an air compressor because we deflated tires for better traction. We were not sure whether we would get home the same day but we did.

Overall it was fun, although there were moments that laughing was the last thing we thought about. We did not even take many photos.

Every fail leaves us with a new experience (like: "watch for swamps" or "back up your work regularly in case your computer dies") which will affect future decisions. Sure it is much better to avoid failures, but if they happen, learn from them.

There is an ancient Chinese proverb: "Always take recovery equipment when you're going off-road with your Jeep." And: "The better off-roading car you have, the longer it takes to get help where you are stuck". :-D

But put fun aside, at Wikidot we try to learn from all problems, failures and issues we face. We have an efficient monitoring system, database replication and server fail-over and many other solutions mostly as a result of a failure we had (or could have had) and we wanted to protect against.

As a part of our wisdom-building effort I wrote down a few points that might help when dealing with failures:

1. Know what you are doing and how

The first and the most important rule. Make sure you know what you are planning to do and what result you expect. When doing complex database operations or programming, having the actual plan is a good thing. Sometimes you might find out there is a simpler way. Or that the result is not worth the effort and possible risk. Know your goal!

2. Make sure you have skills, knowledge and equipment

Running a marathon when you are a novice runner is not going to work. Joining a car race without a proper car will not bring you victory either.

In our story, the Jeep was not really prepared for crossing such deep and swampy mud (although it runs 31" mud tires and has 3" lift) and our first mistake was not recognizing this.

3. Think again "what can go wrong" and "what if it fails"

Imagine what can fail. Could your query delete user data from the database? When mountaineering or hiking, is there a chance you cannot make it to the shelter before night? Can your car get stuck in the mud when you try to cross a swamp?

4. Have backup plans. The more the better.

Plan ahead if you can. Finding a solution after the failure happens can work, but it generates a lot of stress.

Make database backups. Do server administration in a way that allows for a quick roll-back. When in mountains, make sure you have a phone to call for help or can spend a night up there.

We failed a bit yesterday. Plan B failed, because we did not have recovery tools. Winch or a line with hi-lift would have saved us within minutes, and a shovel alone was not enough. Plan C failed because we could not get our off-road friends to help us. Plan D almost failed because the first tractor almost got stuck in the swampy mud. But it worked in the end.

The planning does not spoil the fun

Planning at work is considered a good thing. But putting work aside - many people claim that planning kills the fun and spontaneity. No way. It is often simple troubles that are annoying and time-consuming enough to kill the fun. Actually solving the simple problems ahead of time can move you to even more exciting limits.

Personally I believe the experience that comes from failures is one of the most valuable.