Storytelling through Data: Using Research to Guide Product Development

Developers today are experiencing problems posed by what economists call productive efficiency: how do we create and do more, faster and smarter, with dwindling time, resources, and input? At times, developers are asked to achieve the impossible, and this is why our team at Acquia is always striving to create the most innovative products to ensure they can deliver at the highest level. 

We are always interested in understanding how the user experience of Acquia’s products can be improved. Recently, our UX team embarked on an in-depth research project to understand the complex and varying needs of our users so we can make the best decisions about our product offerings. We analyzed the Acquia Cloud hosting user experience and compared it with several alternative cloud hosting platforms, a project referred to as a competitive analysis. 

Our 26 user interviews emphasized both the many responsibilities of cloud-hosting users and their need for efficient products and robust developer tools. Through this research, we have learned the importance of seamless integration with version control systems and continuous integration/continuous delivery tools and for creating solutions to facilitate, or, ideally, to automate the time-consuming parts of their workflow. With this insight, Acquia has focused its efforts on delivering tools that solve these challenges, helping developers to create digital experiences with unprecedented efficiency and design capability. 

A Study in Two Parts

Acquia’s lead researcher of the study, Adam Young, Ph.D., described the importance of the research methodology, noting, “For any research project, you will want both qualitative and quantitative data to weigh in your solutions.” For the quantitative part of the study, he designed a survey that was sent via multiple channels in pursuit of the opinions of Drupalists with varying perspectives. This survey reached respondents on Twitter, Reddit and Drupal Slack as well as Acquia’s blog for developers, dev.acquia.com/blog. We asked the respondents to rate their experiences with different local development tools on various dimensions such as: 

  • How often they use various development tools
  • How easy they were to learn
  • How easy they were to use

And whether they were perceived as:

  • Modern or outdated
  • Premium or basic
  • Empowering or constraining
  • Straightforward or confusing

The research team also gathered satisfaction scores such as NPS (Net Promoter Score) and CSAT (Customer Satisfaction) to further quantify user responses. The survey’s results allowed us to complement the qualitative in-depth interviews with data that provided reliability (i.e., confidence) and generalizability (i.e., wide applicability) to the study. 

What, Why and How: The Importance of Both Qualitative and Quantitative Data

When reflecting on the importance of using quantitative and qualitative data, Dr. Young details how this research project was approached with different methodologies in mind. The quantitative survey was conducted to provide data related to the dimensions of user experience defined above, while the interviews were designed to gain insight to their context and surroundings. Quantitative data provides answers about how much, and, indirectly, what, whereas qualitative data answers questions about how and why, so it is important to use both methods to gain the most accurate and useful answers to the problems at hand.

“Both are necessary for a well-rounded research project to make sure you don’t have any blind spots. If you are doing only qualitative research, you might be overgeneralizing from the experiences of a few people and mistakenly ascribing characteristics to people they don’t truly apply to,” said Young. “If you are doing only quantitative research, you are getting answers that may lack necessary context, and often your methods are constraining the solution space; if a person’s opinion can’t be accurately expressed with a checkbox, you might be missing their perspective entirely. Both types of data are valuable in making decisions in a well-informed way.”

Storytelling through Data

One of the questions asked in our survey was “What are the different actions you perform in your workflow and how frequently are they used?” We provided a rating scale from 1 to 5, in which a rating of 1 represented “I never work with (x),” and 5 represented, “I am always doing (x).” This enabled us to establish relative importance based upon frequency of use across the evaluated tools; for example, using version control was rated 4.6 out of 5, indicating its significant role in respondents’ workflows.

Bonnie Bohan, a customer experience designer and fellow member of Acquia’s UX team, described how once the data was collected, she and Young worked together to process and analyze the data. Visual design tools were essential in this process, enabling the team to present the data in a seamless way. 

They worked together through the entire competitive analysis process: “While Adam was dedicated to writing the script and conducting the interviews, I would observe and actively take notes, which helped us collect and analyze the data more quickly. By focusing on patterns and transcribing direct user quotes, while there was a lot of information to process, we were able to achieve a quick turnaround in terms of creating a presentation deck for internal stakeholders.”

Bohan describes how she used journey mapping to represent the user journeys the interviews revealed. Subjects walked through Acquia’s website to show how they would discover and evaluate Acquia’s developer products. She then created journey maps to describe how our process looks in comparison to our competitors, grounding the results with a visual component to the data. This helped to identify where users were experiencing obstacles, which the development team could then address. She also populated a feature matrix to compare the different capabilities across each product. This further enriched the data’s presentation, enhancing stakeholders’ ability to comprehend and act on the study’s results. At times, she said, they may have six paragraphs about a subject, yet using these techniques, they were able to turn those into a clear, but comprehensive, set of conclusions to present across the organization. 

The Findings

Productive research always raises more questions than it answers. While this project provided a better understanding of the problems our users face, some key unknowns related to the developers themselves still remain. Given the importance of understanding the goals and motivations of our users, and to clarify points of frustration, the team conducted another series of interviews to elicit further insight. Those questions included what their day-to-day working life was like, what they enjoy most in their work as a developer, and what usual — or unanticipated  — challenges they have faced. With this data in hand, we were able to better establish who our users are, clarifying key user persona details.

Here are some of our key findings: 

Developers express great enjoyment when they are actually building the product and when they are provided the time and latitude to work out a problem-solving process and mindset. Anything that takes them out of that mindset, whether it may be the myriad daily interruptions that can occur or a slowdown of the workflow, a product must be intuitive and fast enough for them to do what they need quickly.

We learned that developers do not mind being confronted with challenging problems, so long as they have the right tools, i.e., they are enabled rather than constrained. 

Finally, the primary motivations of developers who use our products are the ability to build at an accelerated rate, stay in the flow of work and not expend a moment’s time trying to hunt down things unnecessarily.

Conclusion 

Our research studies have been instrumental in our ability to truly listen and understand the needs of our users, and we are using this data to drive our decisions across the organization. With that said, our research is not restricted purely to the developer community. Our research team is working hard each quarter to develop surveys and interviews so we are staying in tune with the wishes, aims and goals of users. 

Most recently, we sent a similarly designed survey to participants inquiring what they believe to be the most interesting initiatives Acquia could be working on. Similarly, we asked about which Drupal initiatives subjects were most interested in, and the top three answers were: New admin UI and JavaScript modernization; configuration management 2.0; and layout. 

We have taken the most highly ranked features users described as most important, and our product development team has set their priorities based on this data. They are now actively pursuing development of these products, including Acquia Developer Studio, which is scheduled to be released in the next year. 

The Acquia team is committed to figuring out what users want and directly addressing this feedback with new products. We want our users to know we truly value their input, and as a Drupal contributor, we consider it Acquia’s unifying purpose to keep pushing boundaries, to be at the forefront of innovation and to develop solutions that the Drupal community perceives as most important to advancing their work. So thank you for raising your voice, and we hope you will continue to do so. 

Senior Editorial and Web Operations Manager, Digital Marketing

Adrianna Shukla

Former Senior Manager of Editorial and Web Operations Acquia