Crunchyroll Shuffle Redesign Research

This project is an example of how I applied the Crunchyroll User Research process to inform the redesign of a legacy content discovery feature for the updated Crunchyroll 2.0 website.

1. Context

​Product Details

• Shuffle is only available on the top right navigation of Crunchyroll's legacy webpages and is not available for the updated experience
• Shuffle plays a random anime series the account has not watched previously​
  
• Existing Insights: Shuffle was created very early in Crunchyroll's history, so there were few updated insights, other than its low usage rate

Company Context

• Crunchyroll was redesigning the website from the "legacy" experience to the new "2.0" experience
• The company's main goals were subscriber retention and acquisition
• A lack of content discovery was hypothesized to be a driver of cancellations (i.e., users join to watch a seasonal premiere and cancel after the series is over because of an inability to find new series to watch, despite being interested in anime)

Project Context

• Product Management requested User Research to inform the redesign of Shuffle for the 2.0 experience
• The project referenced the existing legacy Shuffle feature, but did not include new designs or prototypes
• The Product Manager was interested in improving not only the feature's interaction and layout, but also its core functionality
• Timeline: The project had a 3-week turn-around time and was scheduled in parallel with other ongoing projects
• Stakeholders: The project was conducted for Product Management and Design, but may have benefitted Content Operations
• Tools Available: Usertesting.com, Qualtrics, Qualaroo

Research Questions:

• How much value does Shuffle provide users by showing a random anime series?
• Is a random series the preferred method of discovery for users? If not, what is?
• How do users react to a poor Shuffle result (i.e., a series they are uninterested in)?
• How do users react to high-quality Shuffle results (i.e., series that interest them)?
  • What features can support this successful discovery?
• How do users expect Shuffle to work?

2. Study Design

Methodology
Qualitative Semi-Structured Moderated Remote User Interviews via Usertesting.com were selected after considering the project context, described below.

• Research Questions: Qualitative interviews are valuable for generating hypotheses about and observing the experience of using the feature.
• Project Timeline: 3 weeks - Using Usertesting.com greatly reduces the turnaround time, typically 1 week for data collection, and manpower, which was limited due to concurrent projects, for research operations. The project also required 1 week for stakeholder review of the test plan due to time differences between California and Moldova.

Participants: 8 participants were recruited initially and 2 were added after the initial interviews to saturate the results (i.e., until participants started repeating insights)

• Participant demographics: Age, Country, and Gender were controlled for to be representative of Crunchyroll's users.
• Familiarity: 1/2 of the participants had used the feature within the past year and 1/2 had not. All participants were Crunchyroll users. This sampling was used to collect the perspectives of both user groups.

3. Data Collection

• Stakeholders were provided links and session information for all interview sessions and were encouraged to attend. I have found that stakeholders digest insights more deeply when they attend the initial data collection.
• Participants were screened, recruited, and interviewed on the Usertesting.com platform.
• 2 participants who were not eligible despite passing the screener were quickly (0-2 days) replaced using the platform.
• Semi-Structured Interviews enabled me to ensure I answered the core research questions while probing relevant areas that were not considered during the planning phase (e.g., what were users' experiences of Shuffle on other platforms? How do they differ from Crunchyroll?).

4. Data Analysis and Insights

• Notes were taken and video clips were created after interviews using Usertesting's video clipping and transcription features.
• Participant responses were coded into a spreadsheet and analyzed for patterns.
• Some selected insights:
  • Hardcore, engaged users preferred random discovery via Shuffle because they were searching for "the needle in the haystack", whereas casual users all preferred a curated recommendation based on their watch history because they "just need something to watch" from content discovery features.
  • All users would try to use Shuffle again in response to poor recommendations, up to a varying tolerance, but were unlikely to use the feature ever again if it failed to produce a successful result. Participants supported this finding with this behavior in response to failed shuffles on other streaming platforms.

5. Reporting

The insights were delivered via report slide deck (with relevant video clips) and readout presentation. Although some qualitative studies spawn quantitative follow-up studies to quantify the patterns seen in the research, the results of this project aligned with the stakeholders' understanding of the feature and its users, so the project did not require immediate additional research until new concepts for content discovery were generated.