Image Testing / Chartbeat
A tool for Running multivariate Image Tests
Problem
Editors want to maximize traffic and learn which types of images lead to higher click through.
Solution
A tool that allows home page editors to run multivariate tests of images, in addition to image headline pairs.
Result
Decision to invest in project based on 1-week design sprint which ultimately led to the release of a new revenue generating product to market.
Role
Design Direction, Lead Sprint Designer
Background
When a publisher adds a new article to their homepage, one of the key decisions that gets made is which image and headline to include. Given that articles may have taken days or even months to produce, it’s important the headline and image that are chosen are compelling, leading to the maximum number of possible audience members for that article.
Previously, most editors charged with writing home page headlines and choosing images had to rely on their instincts. Very few tools existed that had been built to serve the publishing use case, and those that did had a reputation for poor usability and narrow adoption.
Chartbeat had previously launched a headline testing tool that had been successful both financially and in terms of user adoption. While users were happy with the tool and seeing positive results in terms of click-through-rate lift, we began to get numerous requests for an image testing tool, including from some of our highest revenue customers.
This led to us deciding to run a Design Sprint in late 2018. We chose to run a Design Sprint because we knew there were many crucial questions that needed to be answered quickly regarding the scope and design. A Design Sprint allowed us to get key stakeholders in a room so there could be broad alignment in terms of what we learned, so we could make a decision on how to proceed shortly after the sprint.
User Research
Early in the design sprint we interviewed internal SMEs as well as external customers. The key takeaways were:
Editors work in extremely fast paced environments. We need to make choosing image variants as efficient as possible if we hoped editors would adopt it as part of their workflow.
Users wanted to be able to test just headlines, just images, or headlines and images at the same time. Building on top of the existing headline testing workflow was likely the best solution since it was familiar, and would simplify the task of running an image test.
Photos, compared to headlines, are costly and time-consuming to produce. Some publishers even have limits to how many images they can use per month based on their contracts with their image providers.
Users wanted to be able to make decisions based on data, not just gut.
Prototyping
Mapping the Problem & How Might We
One of the early activities in a design sprint is to create a map that includes the actors, steps, and outcomes. This sounds easy but considering the rest of the sprint is based on you map, it’s important that the team is aligned not only on which users you want to prioritize, but the goal and key steps to get there. Luckily, we had done some pre-sprint work to ensure that we were more or less on the same page. Even still, this took some time to get right.
From there we did a How Might We exercise, which had each of us envision opportunity areas and vote on which we thought were most worth pursuing.'
Sketching
Next, we split up and individually sketched out ideas for the flow or parts of the flow. This is one of the steps I enjoy most about Design Sprints. I find that having a cross-functional team contributing to design ideas gives you a richer set of options to choose from, and in my experience, the ideas that get acted on often come from non-designers.
Once everyone had sketched out an option, we used dot voting to highlight our favorite parts.
Sketching the user flow
Another of my favorite parts of a design sprint is the collaborative research note taking and synthesis. Having everyone listen in to the interviews and take notes ensures everyone is on the same page regarding what we heard, and it helped us find themes in the feedback more easily. One person on the sprint team was responsible for conducting the interview, while the others took notes. The notes in red documented critical feedback, while the notes in green documented things people liked about the prototype. We then bucketed them based on what they were about (e.g. image sources, usability, etc.)
Outcome
Based in part on the learnings, Chartbeat decided to fund the building of Image Testing shortly after the sprint.
The project was a large technical undertaking but we onboarded our first customers in 2020. Some of the highest trafficked news sites in the world are currently using it on their home pages.