Wednesday, 4 September 2013

Our Net Promoter Score is 74. So what?

A big thank you to everyone who completed the IT Performance Company Customer Survey. The results are in and here’s our Net Promoter Score compared with a clutch of IT Services companies:

IT Performance Company Customer Survey August 2013 of 31 respondents
compared with Temkin Group Q1 2012 Tech Vendor NPS Benchmark Survey

We can say, hand on heart, that our lovely survey respondents gave us a better Net Promoter Score (NPS) than 800 US IT professionals gave IBM IT Services last year. What does that tell us? Is this a valid comparison? Is NPS useful for internal IT providers?

The Net Promoter Score is undoubtedly fashionable at the moment and the market awareness driven by Satmetrix and Bain & Company seems to have endowed it with some credibility. According to Google Trends, interest is still on the increase.

Google Trends Web Search Interest and Forecast for 'Net Promoter Score'

The Net Promoter Score has its critics too. Wikipedia covers some of the arguments and Customer Champions have a nicely balanced article. It's right for measures to be held up for scrutiny, especially when attempting to use NPS to predict financial performance or benchmark disjoint groups (as with our example above). Equally we shouldn’t obsess about data quality perfection because the presence of types of error doesn’t invalidate the measure as long we interpret it in context coupled with cause-effect improvement action.

NPS is derived from the response to the so-called ‘Ultimate Question’ :  “How likely are you (on a scale of 0-10) to recommend our company/product/service to your friends and colleagues?”. Respondents who score you a 9 or 10 are ‘Promoters’, those giving you 0-6 are ‘Detractors’ and the 7-8 are ‘Passives’. The % of Promoters minus the % Detractors yields the NPS score on a scale of -100 to +100.

The recommendation question itself is powerful because it goes beyond satisfaction and loyalty to seek a measure of active advocacy. Customers are being asked if they would ‘sell’ on your behalf and in doing so make a prediction about the outcome and put their own reputation on the line. In essence they are expressing their confidence that you will do a good job and not embarrass them.

Its very, very hard to get an NPS of 100 because even those who score you an 8 don’t get counted and the scale is very heavily weighted towards the Detractor buckets. The aggregate NPS score throws away the Likert Scale resolution so it's important to look at the distribution of scores, ours for example:

Distribution of Responses to 'Likely to recommend' question
in IT Performance Company Customer Survey August 2013 of 31 respondents.

Would we have shared this chart if there had been any Detractors? The point is that having both an aggregate comparator score and a differentiating scale yields useful insight from a single recommendation question. The insight is richer still if a second qualifying question is asked about what could have been done better eg. “What would have made you score us higher?”.

I’d expect all enlightened internal or external IT providers running ‘IT as a business’ to have some strategic customer objectives. These may be coated in slippery management-speak which needs to be scraped off to expose an observable performance result. See StacyBarr’s post on weasel words.

If there’s an objective in the customer perspective of the IT strategy map like: “Customers trust us enough to recommend us” then the Net Promoter Score will be a strong candidate for an outcome measure. The responses to the open-ended qualifying question should give clues to the  results which should drive improvements in things that customers value such as “Call-back commitments are met”.

Monopoly IT providers might be tempted to dismiss the relevance of the ‘Ultimate Question’ when their consumers don’t have an alternative, or external benchmarks aren’t valid (and anyway, their strategy is cost reduction). This is to reject the power of the recommendation question for hearing the ‘Voice of the Customer’, the foundation of Lean IT Thinking for reducing waste. Experiments should instead be used to analyse (by correlation) whether the Net Promoter Score is a strong indicator of some dimension of the IT service to prioritise improvements by customer value and answer questions like: “If we change X, what happens to NPS?”.

By virtue of simplicity and low cost, the two complementary NPS questions lend themselves to transactional interactions such as exit polls from service desks. Care is needed though to avoid types of sampling bias and distortion of the respondent's ‘stated preference’.

You don’t need a large sample to begin to see signals and to paraphrase Doug Hubbard: “Small measurements reduce uncertainty a lot”.  An IT provider might choose to use stratified NPS sampling to compare service perception gaps at different delivery points, internal or outsourced. NPS could be used for longitudinal tracking of how IT is improving its primary value streams. More in-depth, periodic surveys like variants of the Gallup CE11 will still be needed to get a fuller understanding of IT customer engagement.

As a small, growing professional services company, our very survival depends on reputation, trust and enduring relationships. If we have earned our customer’s willingness to promote us then we’re proud, honoured and probably doing a good job. So yes, the Net Promoter Score is an outcome measure in the Customer Perspective of our very own Balanced Scorecard.

A more recent Temkin 2013 Tech Vendor survey can be purchased here in which VMware came top (47) and CSC came bottom (-12).