Blog Layout


Quantifying impact: The art & science of measuring the value of design

This article is based on a recent presentation that Cole Armstrong and Suzi Goodwin from NeuroSpot gave to the CX Collective, a global community of customer advocates run out of Aotearoa New Zealand. We greatly appreciate the input from Sarah Clearwater and Baruk Jacob from the CX Collective, and from the participants on the day.

A key reason for tackling this topic was because we know the work that goes on in organisations across the country, work delivered by people passionate about the customer experience, work that provides real value to organisations. But work which is at times hard to justify, seemingly impossible to quantify and difficult to see the tangible evidence of positive shifts in behaviour. Work that at times can be viewed as a cost centre rather than a value driver. 


CX as a function is not always well understood at all levels of a company – McKinsey reports show that companies with an embedded design practice are growing at nearly twice the rate of their peers, yet only a third of CEO's and their direct reports could state confidently how design delivers on business metrics. We wanted to share ways that you could actively measure not only the impact you have delivered, but the potential of testing different assumptions and interventions from the beginning of a project, giving you opportunities to set every project up for success, and highlighting the value you provide.

You need to quantify your impact

In the fast-paced world where organisations are constantly moving onto the next thing, we rarely get a chance to look back and see how what we delivered made a difference. Retrospectives are great, but not always well timed or they can be rushed as a box ticking exercise as we all try to focus on our current challenge. If quantifying your impact takes extra effort, why bother?


  • We want to make a positive difference. If we don't measure our intervention, we can't say for sure that it worked. Maybe all that resource resulted in no change, or potentially, having made things worse?


  • Quantifying impact helps mitigate risk. Every new innovation or experience brings a measure of risk, and it takes up a lot of stakeholder headspace. How can you alleviate this?


  • Look at the cost/benefit. Experimentation helps you see if the expected gain justifies the resources required.


  • Build your professional standing. Building trust with your stakeholders, showing that you deliver well executed interventions with measurable outcomes can help with internal support for your next challenge.

Use experimentation to articulate value

Hand drawn diagram of a hypothetical test, showing users going into the test enviroment, being directed to either Option A or B, and then the results.

Experimentation goes beyond A|B tests, and you're probably familiar with the basics from high school science - you have your hypothesis, a control and treatment group, there's some analysis and an outcome. That's pretty much how it works in design too. 


You'll need the following:


Objective: a clear sense of what you want to achieve - the behaviour or response that you expect to see. E.g. greater satisfaction with our insurance claims process.


Metric: something observable, reliable and related to your objective. Preferably actual behaviour, and ideally already collected - e.g. people tagging onto public transport, sales records. Or maybe you need to send someone out with a clipboard to count people. Other metric types can be used as well - claimed behaviour (however this can introduce human bias) or perceptions (e.g. via a survey).

Users: you need a group of people to test on. How many depends on a number of factors but for some projects 100-200 per group could be enough.


Intervention: this is the thing you actually want to test, the change you are wanting to make to a process, product or experience.


Randomisation: your users need to be randomly assigned to your control and treatment groups. Randomisation means you can be confident you're not inadvertently introducing bias (e.g. males vs females). While straightforward in the digital space, in other settings this could involve splitting a call centre in two, sending two different letters or changing instore displays from one hour or day to the next.


What will the decision drive?: consider all the potential outcomes of the experiment. What decisions would it drive if your outcome metric shot up, or down - or stayed the same?


Having these clear in your head will help you both articulate the experiment and its intent to stakeholders, as well as provide an instruction manual of sorts to the team implementing it.

Experimentation adds value earlier in the design process

One of the key takeout’s from our presentation is this: don't wait till you're delivering the work to experiment. There is often far greater value in incorporating experimentation towards the start of the design process. Why not test:


  • Stakeholder assumptions. Experimentation at this stage helps ensure you focus on the issues that matter, and allows you to engage stakeholders into the process, getting buy in for down the track. 



  • Alternative concepts. How do you objectively know the intervention your team is looking at is better than the one tossed aside? Or, how can you test your intervention at a small scale, to estimate the impact it could have when rolled out? Experimenting at this stage gives you the opportunity to test some of the crazier, counter-intuitive ideas that could be a huge competitive advantage for your organisation.

This is not to say there is no merit in experimentation at the end of a project - it can help ensure you've delivered a project of value and should absolutely be used. You'll want to be able to share those results with stakeholders, and add them to your CV...

Finally, your experiments need to be ethical

Ensuring you have some ethical boundaries in your experiments means you're avoiding brand damage and a loss of trust to your organisation, and hopefully not keeping yourself awake at night with a moral dilemma. Some questions to keep in mind:


  • Is the objective ethical? This should be universal to all projects.


  • Does it infringe on your user's welfare? The objective might be worthwhile, but it shouldn't leave people worse off.

 

  • Is there anything more than minimal risk? Holding a conversation for example, carries a risk - what's the risk from your experiment?


And at the end of the day, is it ethical to offer a new product or service that makes people worse off?

This is just a summary of what we covered, the video of our presentation and the resources discussed in the session are all available from the CX Collective Learning Library. But we're here to help too, so if you have any questions about what we've discussed or have a challenge you need a sounding board on - get in touch. You can also check out some of our other work here.

Remember, there are no bad ideas - just unproven ones.

By Cole Armstrong 15 Mar, 2024
How do we create persuasive touchpoints that make a difference? By considering how simple ways of reframing our messages, using insights from psychology and behavioural science, can create greater motivation to act.
By Cole Armstrong 19 Jul, 2023
If I asked you to think back about an event, maybe a holiday or your last plane trip, your last dinner out, or a shopping experience, what would you remember? If I asked you to describe the experience, chances are you’d feel pretty confident about your memory, or at least some of the key elements. It turns out though, that confidence you’re feeling - it doesn’t relate to the accuracy of your memory. Faulty memories You’re not losing your mind, it’s just that your mind is playing tricks... sort of. We’ve spent quite a bit of time using eye tracking technology through our client projects. It enables us to see a participant’s behaviour – what they actually see and engage with - and the journeys people take through a physical environment, like a mall or retail setting. One project saw participants navigating a store with eye tracking glasses, getting items off a shopping list. As soon as they’d completed their journey, we asked which way they’d walked. Participants confidently recounted their route, and yet despite having literally just finished their journey, consistently missed out details. In another project we asked focus group participants about an image we’d shown them 20 minutes earlier. This elicited quite a spirited conversation about skin colour and how the illustrator’s choice of using a dark skin colour for all of the characters pointed to the racism of the illustrator and client. The thing was though, the characters weren’t dark skinned. Not one of them. And yet all of the participants convinced themselves this was the case. We’re certainly not the first to have encountered this phenomenon. There’s quite an active scene looking into issues with eyewitness testimony, and under which conditions our memory maybe unduly swayed or prone to errors. As you can imagine, the consequences of this could be huge. How can we stop getting it wrong? We’re not saying that our memories are always wrong – clearly that’s not the case! But there’s a rhyme and reason behind how our memory operates – both for good and bad. Our brains are BUSY. It’s like a hamster wheel going full on 24/7. Even when sleeping our brain is taking stock of the day, filing away moments into short and long-term memory. In order to look after us, our brains have to prioritise its resources, and it essentially takes shortcuts wherever possible, driving the same way to work each day, ordering the same coffee and so on. Imagine the fatigue we would face if we had to make every decision and action consciously, rather than letting our brains run the show. Which moments matter? So when our brain – a lazy but efficient workaholic – is sorting through events and the happenings of our day, it throws out the mundane, peripheral information it deems unimportant. It instead focuses on creating a highlight reel, and takes the moment of the events and experiences that were the most emotionally intensive, and the final moment. The concept is known as The Peak-End rule, and comes from Daniel Kahneman and his colleague Donald Redelmeier. Their 1996 study, which I am very pleased to not have been a participant in, involved 154 colonoscopy patients rating their level of discomfort at 60-second intervals throughout the procedure, as well as being asked to retrospectively describe how uncomfortable the procedure was. The level of discomfort during the procedure had no correlation to the discomfort they reported retrospectively. As an aside, they followed-up this study with yet more colonoscopy patients, who were split into two groups. One group had the standard procedure and experience of the camera being somewhat painfully removed, and the other had an amended experience that lasted three minutes longer, but which the camera removal was more uncomfortable than painful. The second group – with a longer procedure but less discomfort in the final moments – rated the procedure as less painful than the first group and were more likely to return for subsequent procedures. What was relevant was the peak level of discomfort experienced, as well as the level of discomfort in the final, end moments of the procedure. So what does this tell us? Firstly, that our memory is more fallible than we’d like to realise, more often made up of a series of stitched together moments and thoughts that can be revised and reinterpreted after the fact. Here’s an example - one of the best flights I’ve had was on Air New Zealand to Sydney – the first time we’d flown with my then-infant daughter (you can imagine our nerves!). At the end of the flight we apologised to the man next to us who’d (somehow) been working the whole 3.5 hours. We were suitably self-conscious at the amount of screaming he’d been subjected to but were greatly surprised at how nice he was – telling us that she’d been great, and how his (now teenage) children had subjected him through worse. Then some of the other passengers near us congratulated us on surviving a flight with an infant and how good a flyer she’d been – alongside the cabin staff who were making faces at our daughter to get her to laugh. It's honestly one of the best flights I remember – but clearly it wasn’t that pleasant at the time! The Peak-End rule in action – our actual and remembered experiences diverting wildly. How to apply these learnings to your work Don’t rely on people providing an accurate testimony of their experience. It’s more important to look at what a customer does vs what they say they do. When reviewing a process, journey or customer experience, focus on the moments that matter – the peak emotional moment and the final moment. This provides direction, stopping you from spreading your resources too thin and helping concentrate efforts on the moments most likely to have an impact. Be creative when designing experiences. Because our remembered experience is more important than our actual experience, you have a unique opportunity where you can creatively leave customers with an experience perhaps better than what they had… If you know there’s a frustration or issue during a process, while working on a fix for that, make sure your final moment knocks their socks off.
08 Nov, 2022
JCDecaux is one of the largest Out-of-Home businesses worldwide; in New Zealand it specialises in high quality Large Format and Airport touchpoints. JCDecaux is committed to delivering research-led validation to its partners regarding Out-of-Home effectiveness and looks for partners who can deliver neuro or behavioural methodologies that can deliver on this objective.
Show More
Share by: