Saturday, 25 January 2014

Making decisions by the seat of IT’s pants?

In his brilliant book ‘The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty’ Dr Sam Savage shows us why we need to connect the ‘seat of the intellect to the seat of the pants’.

Savage’s message is that making decisions under uncertainty requires probabilistic methods rather than averages in everything from movie portfolios to the height of flood water*. A simpler message is that we can fly intuitively by the seat of our pants for some types of decisions but that failing to engage the brain for others could mean our pants end up around our ankles.

Does your IT organisation make decisions by the seat of the pants or the seat of the intellect? Does the word ‘pants’ make you smile? Be honest now.

I suspect that decisions and behaviours in IT organisations are distorted by intuition and judgement more than we’d like to admit. We all fall prey to these cognitive biases, however rational we like to think we are. Don’t worry, its human. So is our denial. However when the effects of cognitive bias aren’t understood, recognised and mitigated the stage is set for some questionable decision-making. There are great examples of how this plot unfolds in ‘Why plans fail – Cognitive Bias and Decision-Making’ by Jim Benson. IT is a regular cast member.

If you’re still not convinced, read ‘Thinking Fast and Slow’ by Daniel Kahneman and his earlier work with Amos Tversky. The former won a Nobel Prize for his work in behavioural economics which describes how our intuition, the ‘gut feel’, makes irrational decisions in the presence of uncertainty. Combine the ‘halo effect’ with the ‘illusion of knowledge’, throw in some ‘cognitive dissonance’ and we can also see why IT decision-makers are held up to be (and may even believe themselves) immune to these effects.

How do we know when decisions have been based on good evidence, bad evidence or faulty judgement and how do we know if they were successful? If IT is investing in a vendor’s technology, letting an outsourcing contract, reorganising, shedding headcount or making service improvements then its making bets on some outcome. Perhaps the outcome itself is a vague articulation of some flawed reasoning (but that’s another story). Our natural over-confidence and biases make success seem inevitable in our heads when there’s actually some probability of a negative outcome. If no-one is asking “Where’s the evidence?” then alarm bells should be ringing.

Making IT decisions using selective evidence or using biased sources such as vendors and suppliers can be a problem too, whether unintentional or deliberate. Decisions can also be distorted by a culture of fear and blame where evidence is suppressed or sanitised to duck punishment for ‘bad’ decisions whilst at the same time losing the ability to experiment, learn and celebrate ‘good’ ones.

There are also dangers lurking in the things which might be considered self-evident in some IT departments as Pfeffer & Sutton describe in their book ‘Hard Facts, Dangerous Half-Truths, and Total Nonsense: Profiting from Evidence-based Management’. Doing things which are believed to have worked elsewhere (but proof is scarce) or following deeply held (but untested) ideologies are just two examples. Does that remind you of a new technology hype, a ‘best practice’ framework, a new boss keen to make an impact? If unchallenged or unsupported by evidence these decisions could either seep into the collective sub-consciousness or breed resentment and even sabotage.

Evidence-based Management might sound like a new-fangled fad but it isn’t. Not where decisions really matter, in health for example. Take Stacey Barr’s recent post about why we should trust scientific evidence in spite of personal bias or even the media’s pursuit of a story. Ben Goldacre has a done a lot to expose this kind of ‘Bad Science’ in the media and yet the IT media and their marketing paymasters have been jumping, unskeptical, from one bandwagon to another for decades.

Making good IT decisions needs evidence and evidence needs measurement. This is measurement in the broadest sense:

‘A quantitatively expressed reduction of uncertainty based on one or more observations’

according to Doug Hubbard’s ‘How to Measure Anything: Finding the Value of Intangibles in Business’. Hubbard, like Savage, is an exponent of probabilistic methods and creator of Applied Information Economics for measuring the value of information itself and reducing uncertainty in the variables most likely to have an economic impact.

Maybe IT has something learn from other parts of the business about how to use Monte Carlo models and other statistical tools (from Lean and SixSigma for example) to focus on the best improvement returns rather than relying on perception and intuition.

As shareholders, tax payers and employers, shouldn’t we expect IT leaders to make decisions using more robust methods and then prove that they’ve created, not destroyed, value? I’d like to think so.

*Savage’s dad Leonard Jimmie Savage was a giant of Bayesian decision theory in the 1950s so you might expect him to know a thing or two about probability.