Chesley Sullenberger, the pilot who landed an Airbus A320 on the Hudson explained in an interview that “Everything we know in aviation, every rule in the rule book, every procedure we have, we know because someone somewhere died…. We have purchased at great cost, lessons literally bought with blood that we have to preserve as institutional knowledge and pass on to succeeding generations. We cannot have the moral failure of forgetting these lessons and then having to relearn them.”
How would our practice of software testing improve and our discipline be enhanced if mistakes we made resulted in people dying? For some of us they can and do result in people dying. In many industries like medical devices and air traffic control mistakes can result in lives being lost. For the vast majority of us though a bug missed won’t result in catastrophic results.
Does that mean we shouldn’t do everything we can to learn lessons after things have gone wrong?
We talk about retrospectives in agile software development but they don’t get anywhere near to the thoroughness of the retrospectives implemented in the airline industry. There are principals as stake in the airline industry that are critical to the ability to learn. Principals that focus on not just learning, but putting those lessons into practice.
One of those principals is the ability to be honest. When was the last time you held your hand up and said “Sorry I made that mistake”? For many we’re part of a culture where we feel like our reputation, and our credibility, will be irrevocably damaged if we admit to our mistakes. Yet, the first part of fixing anything comes down to diagnosis. If we’re not going to admit and diagnose our mistakes because we fear damage to our reputations then we’re not even going to start to improve.
I write this wondering if I should provide an example. Write about a significant mistake I’ve made early on in my career. I’m reluctant to write about this and make this public because I fear that it’ll impact future work and damage my reputation as a tester. Yet if I, and the rest of us, are conditioned not to talk about our failures then we’ll never learn. So here goes!
The example I’m thinking of was a key product demo. Having completed a few product demos already I knew that once you have a demo working you don’t touch the system. You don’t fiddle with it. You don’t try to improve things. You don’t modify even the smallest bit of data. If it’s working and ready to go you leave it. Then, when you’re ready, you just run the demo.
So this was a lesson I already knew. Just that on this occasion I decided not put it into practice. Time to learn this lesson all over again the hard way.
We only had one test/dev system and that was being used for this product demo too. I’d set the demo up for a senior executive to run and left him to it. This was the same senior executive that was really putting the pressure on to deliver this project. So I went back to running more tests on the system. Found a small issue and asked a developer to modify some test data. What we both thought was an insignificant field in a database. I’ve no idea why but we lost a couple of database tables. This took the whole system out!
I’ve never seen an executive exit a meeting room so quickly. Exit a meeting room with a mission to, shall we say, ‘talk with me’. The system had gone down right at a key point in the demo with a client.
I should have known better. I should have left that system well alone. Under pressure to meet deadlines I continued working on the system. Our one and only test system that was being used for this demo.
Let’s just say that the following day the language was more aggressive and colourful than I felt comfortable with. To the extent that I felt very uncomfortable discussing this issue in any depth or even talking with others about what had happened. The net effect is that I may have remembered my lesson about running demos but I learnt nothing from the technical aspects of this failure. Nobody else would have learnt from it either.
This is the key point. If people are EVER discouraged from admitting to, and analysing mistakes, then we’re all losing out. We’ll never share those mistakes and others will never learn from them. They’ll never learn from them because they get hidden.
This is the complete opposite of the airline industry. Mistakes are analysed, detailed and publicised. Get that! They are publicised. This way everyone can learn from them. Only when everyone learns from them are we in a position to make sure they don’t happen again.
Interestingly part of this culture to engender openness is covered by the rule of law. Evidence that pilots give can not be used in court against themselves. This ethos of encouraging disclosure without incriminating the individual removes an important barrier to the process of learning from mistakes. It’s a culture that encourages people to speak up.
This presents the question, “do you feel like you’re working in a culture that encourages you to speak up?”
After this demo do you think I was happy to speak up again? Was I happy to get involved and work on pulling product demos together again? You can bet I wasn’t. Do you think I was going to be happy raising issues on this project and helping present solutions? Was I keen to talk about and analyse mistakes anymore? You can bet I wasn’t. And if we’re not highlighting where things are going wrong how can we put them right?
It’s not just learning the lesson that’s important though. It’s putting lessons learnt into practice that is critical. We learn from our mistakes but the moment the pressure is on we can be tempted, even encouraged, to bypass a lesson and revert back to old habits. Reverting to old habits can sometime seem like the shortest route to meeting some arbitrary deadline. A short route that very often doesn’t turn out to be the quickest route.
And that for me was the real mistake I made. When the pressure of a deadline was on I decided I’d take the short cut. I decided to ignore a lesson I’d already learnt and revert back to an old habit. Well I’ve learnt that lesson again and more this time round.
If you were to admit to some of your past failures and make public the lessons you’ve learnt what they be?
