Role
My role during this project was a User Researcher II working within the Grammarly Business self-serve team whose main focus was on driving retention and expansion of Grammarly Business within professional teams. The self-serve team was the arm that focused on non-managed accounts, which meant the team sizes of the accounts were normally under 50 seats.
Problem
Grammarly was once one of the few players in the AI writing space before the introduction of ChatGPT and other AI tools. After the rise of AI tools/interfaces, there was a need for Grammarly to adapt its existing interface to include an AI assistant. While AI assistants are often fun to use and have a novelty about them for individual use, there were many unknowns when considering use in a business environment. We were not only unsure how users would interact with the system and what they might use it for, but we were also unsure if people would be comfortable using it for work-related tasks in the first place. Because company work often relies on understanding context, and that context comes from existing potentially confidential documents, would users want to use an assistant that can read through that context to deliver whatever is asked of it, and what would that experience look like? These were the problems this research into early concept testing for Grammarly’s AI writing assistant took on.
Challenges
The main challenge in this research was aligning stakeholders in the same direction. Because AI assistants were such a ‘hot’ topic around the industry there were many ideas on how we should build out our own and what were the most important things to focus on. Addressing this required staying in constant contact with stakeholders throughout the project and ensuring we were answering questions that would be impactful for the feature’s development while also understanding how the feature would add value for our users.
Research Goal
The main goals of this research were to refine initial designs into integrating an AI assistant within the Grammarly product, and to understand what existing mental models people had for using and interacting with an AI assistant by answering the following research questions:
- How do people want an AI assistant to help them with their writing while working?
- How do people interact with an AI writing assistant?
- What are the motivators and roadblocks to adopting an AI writing assistant for professional use?
- What design/usability elements do people look for with an AI writing assistant?
My Process
- Learn: The AI assistant space was not only new for Grammarly, but also for me. In order to bring about the best outcome in this research, I thought it was best I first learn more about the space. This included meeting with internal Grammarly engineers who could explain how the AI assistant could work, as well as exploring what the experience looked like at potential competitors. By developing my technical knowledge of AI assistants, I would make sure the research I did answered questions that were actionable.
- Communicate: After immersing myself in all things AI, I moved on to a meeting with my stakeholders to learn more about their vision of what an AI writing assistant looks like at Grammarly and what we are trying to accomplish by introducing it as a feature. It was important that I made sure stakeholders saw this as a feature to add value to our users which would then translate into higher retention and new bookings by making it a valuable experience rather than a trendy technology of the moment that is only capitalizing on the buzzword AI assistant. Through these meetings with stakeholders in product, marketing, design, brand, and engineering, I was able to gather the questions that would be most helpful in shaping the AI assistant experience at Grammarly and how potential insights would be used for each team.
- Prioritize: With the AI assistant space being so new, there were many questions my stakeholders were interested in, and while each question could have potential value down the developmental road, it was important to answer the most important ones first. To do this, I looked at what was gatekeeping our teams’ design and product decisions and what insights would allow them to move forward. Because there was a company goal to move fast and not fall behind in this space, I would also be partnering with a designer to concept-test potential AI interface designs. This meant the questions I answered had to focus on getting feedback on design elements in addition to understanding the root behaviors behind using an AI assistant.
- Plan: Once I had organized the research questions I began to plan how to best answer them with my design partner. We worked through several potential testing methods but settled on doing a series of iterative concept tests. This methodology would allow us to receive direct feedback on specific design elements and usability, as well as explore participants’ thoughts and attitudes towards AI assistants during discussion.
- Research: This research used a series of concept tests of early-stage designs for an AI assistant interface within Grammarly’s existing interface. These tests would happen in three rounds over the course of a month with updates being made to the design concepts being tested in between each round. To facilitate quick updates and recommendations, I set up participant sessions with my designer to ensure they were getting insights as soon as possible. At the end of each round, we would debrief what we saw and then make changes for the next round. Once all rounds were completed, I would compile all of the learnings and changes made into a slide deck to share with the remaining stakeholders.
- Analyze: Due to the fast nature of the concept tests; the results would have to be analyzed in real-time with my design partner to ensure they had the time they needed to make updates to their designs before the next round of testing. To do this, I made sure to connect with my design partner at the end of each session, as well as at the end of a test round, to ensure we were aligned on what we saw and how to best implement changes into the next round. Once finished with the concept tests themselves I moved on to compiling the insights gained in each round into a singular slide deck which highlighted both the key insights learned as well as the changes made throughout the testing phase.
- Share: Once I had compiled all of the insights from each concept testing round into a singular slide deck, I set up a meeting with my remaining stakeholders to go over what we learned and what recommendations I had for the next steps as we continued with the development of our AI assistant. Following the presentation, I also participated in several team meetings where I further dived into the results to better align with each team’s specific needs and connect insights to the specifics of what they were working on. This ensured that all of my stakeholders were able to connect the value of the research to their roadmap and development cycle.
Methodology
This research was qualitative in focus and used a series of concept tests of early-stage designs in a 2-1 interview setting where myself and my design partner observed how a participant interacted with the design in simulated tasks they might do at work (as these designs are still not released I am unable to show them for confidentiality reasons). During the tasks I also led participants in a discussion around their experience with AI assistants and how they want them to integrate with their workflow. The concept tests were broken into three rounds which included participants from each of the below groups:
| Group | User Type | AI Familiarity | Count |
| 1 | Grammarly Free/Premium | Familiar | n=6 (2/round) |
| 2 | Grammarly Free/Premium | Not Familiar | n=6 (2/round) |
| 3 | Non-Grammarly user | Familiar | n=6 (2/round) |
| 4 | Non-Grammarly user | Not Familiar | n=6 (2/round) |
These groups were chosen after consolidating my planning sessions with stakeholders to understand who our key audiences might be. It was important to understand how this new feature would be received by both our current Grammarly users as well as non-Grammarly users. Additionally, it was critical to understand how experience with AI might impact a user’s perception of our AI assistant feature. With that in mind, the users were split into groups first based on whether they were/were not a Grammarly user, and then again based on their self-reported familiarity with AI using a Likert scale; those that rated themselves as “familiar” or “very familiar” being categorizes in the familiar category.
Key Findings:
This research highlighted several key improvement areas with our AI assistant to ensure a successful launch of the beta experience. Below is a summarized list of the insights learned through the three rounds of concept testing (Notes: The below summary is not from the slide deck but includes the main takeaways that were included).
Outcome
This research led to two main outcomes. The first was the usability refinement of our AI interface design, which allowed users to much more easily interact with and use the AI assistant; the second was an in-depth understanding of where users and potential users ran into problems with using an AI assistant. By highlighting these problems early on, we were able to steer our product development in a direction that not only made a usable AI assistant but an intuitive one, which was one of the greatest issues highlighted in this research. This led to several further studies which targeted the onboarding into the feature, which had a significant impact on the adoption of the tool once it was released to users in experiments.