This report is a summary of the research and design outcomes throughout a 10 week usability testing project conducted with Torin Blankensmith, Rashmi Srinivas, and Hailey Yu, exploring the following design question for Spotify Mobile:
“Does the current content organization and navigation scheme of Spotify Mobile allow the users to access new music easily and efficiently?”
Through a User-Centered Design approach, we developed a full usability study test kit based off of results from an initial heuristic evaluation. We conducted 7 usability study sessions with 7 participants, who had used Spotify Mobile and listen to music 2-5 hours on mobile devices per day. Each usability testing session contained a pre-test Q&A session, 6 tasks with 6 corresponding post-task surveys, and a post-test Q&A session. The 6 tasks were designed to cover social, music discovery, and playlist features on the premium version of Spotify Mobile App on an iOS device.
We compiled all the qualitative and quantitative data into a single Google Docs, categorized by tasks, and grouped the data into 4 themes through an affinity diagram approach:
1) Unclear Terminology,
2) Confusing Navigation,
3) Struggles to Find Personalized New Music
4) The Gap between Expectations and Reality.
Below are the two major findings that derived from the 4 themes as well as our redesign recommendations accordingly:
Users can’t easily access their friends’ playlists and share content as the social feature in Spotify.
Personalized new music can be difficult and overwhelming as a discovery feature in Spotify.
Implementing these redesign recommendations will help Spotify generate a more user-friendly mobile App on IOS devices, and accomplish Spotify’s mission of “[helping] people listen to whatever music they want, whenever they want, wherever they want.”
To determine what areas of Spotify mobile our team should focus on, we created a heuristic evaluation. the heuristic evaluation was broken up into larger categories including Navigation menu overall, Search, Browse, Radio, Your Music, Customization, and Error messages. These categories contained evaluation questions such as “are the user’s past keywords saved? Do the icons relate to the labels in the sub navigation?” and so on. Our entire Heuristic Evaluation can be found in the Appendix.
Each team member explored the app and answered the evaluation questions with a “yes” or “no” as applicable. After compiling all three evaluation forms, we found that the areas that were least effective in terms of functionality and clarity included finding music through the Browse page, consistency between terminologies, and using the hamburger icon to navigate and explore the app.
With the information from the Heuristic Evaluations, we then brainstormed a list of possible research questions which would reflect how we design our usability test later on. An example of our list of questions is below:
These questions started off as too specific and task-related, however we focused on what all of the questions had in common. An overarching theme throughout these questions centered around the user’s ability to find new music through various methods including Browse, Radio, and social features. Going back to Spotify’s mission to make music as a whole accessible to their users - we decided to focus our usability study on tasks that show us how users find, share, and music and playlists, and how users customize their content. With this area of focus, we were able to evaluate where the Spotify mobile app could be improved to make these core processes easier for their users.
For the study, we recruited a total of seven participants. Four participants are female, three are male, and all participants are college students at the University of Washington between the ages of 18-25 years old. All of the participants chosen met our pre-screening requirements and listen to Spotify on their mobile devices an average of 2-4 hours per day. Six participants use an iPhone and one participant uses an Android device.
Our target audience for Spotify Mobile are college-aged students who are familiar with the app, but do not necessarily know about every feature Spotify mobile has to offer. Due in part to our screening questionnaire and environment, all of our study participants matched the criteria to be the target audience.
Before starting the task walkthroughs, we asked each participant to answer a few questions to get an idea of their experience with Spotify mobile. Questions asked included how the participant would rate the ease of use of Spotify mobile on a Likert scale of 1-10, 10 being the most difficult; what is the participant’s favorite features on Spotify; what are the other music streaming services the participant uses, if any; and in what situations does the participant use Spotify mobile.
The pre-test questionnaire took an average of 4 minutes to complete per participant.
Since we wanted to measure the ease and efficiency of Spotify’s mobile features, we broke up the usability test into six tasks - each testing out a different action that a user might do to discover new music through Spotify recommendations, radio, and social circles/friends. Some tasks contained subtasks that further tested out the same feature, or otherwise was an action that would naturally follow in the task order. Follow up questions were asked depending on the actions of the participant during the task.
For each task we started by providing the participant with an applicable scenario to provide context and observe the participant’s natural response. From our screening questionnaire, we were able to determine that most users use Spotify mobile when they are walking to/from a destination, working out, or at a social event without access to their laptops or tablets.
Scenario: You are taking a bus home and you want to listen to something new that's not in your playlist.
Task: Find a recommended playlist by Spotify that you are interested in and play a song in the playlist
Scenario: You really enjoy listening to a specific artist and you want to listen to a similar artist. Please go back to the Navigation page and…
Task: Discover a similar artist and play a songFollow up question: What content do you expect to be under each of these subheadings? (“suggested for you based on…” and “because you listened to…”)
Sub Task: Now follow the suggested artist
Follow up question: What type of content would you expect to be notified about
Scenario: You have a favorite artist that you listen to when you go on a run. This time, you want to listen to a radio station with songs from that artist. Please start at the Navigation page and
Task: Start a radio station on an artist that you have recently played
Scenario: You want to rearrange the order of the songs in one of your playlists for a party at your house tonight. Please start at the Navigation page, open a playlist and
Task: Change the order of the first and second song.
Sub Task 1: You really like the first song in the playlist, go ahead and save itSub Task 2: Navigate to where the location of the saved song.
Scenario: One of your friends has had good music suggestions and you want to take your friendship to the next level on Spotify
Task: Starting from the Navigation page, follow a friend on Spotify.
Follow up question: Now that you are following a friend, what type of content would you expect to be notified about?
Sub Task: Play a public playlist from your friend’s Spotify music collection
Scenario: You are having a party tonight and want to create a new playlist.
Task: Please start at the navigation page and create a new playlist and include a song of your choice.
Sub Task: Your friend is co-hosting this party with you and you would like to share this playlist with him/her.
Outside of sub-tasks and follow up questions, each task was followed by a general post-task questionnaire. Questions included whether or not the participant completes the task, the time it took to complete the task, and on average if it took the participant more, less, or average time to complete than they expected.
Each post-task questionnaire took an average of 1 minute to complete.
The purpose of the post-test questionnaire was to debrief the entire test with the participant and give the participant time to talk through any particular task or challenge. The questions included how the participant rates the ease of use of Spotify mobile on a Likert scale of 1-10, 10 being the most difficult; what tasks the participant enjoyed/did not like, if any; what the participant likes/dislikes about Spotify’s mobile app; and if there were any new features discovered through the usability test.
On average, the post-test questionnaire took 5 minutes to complete for each participant.
Our usability tests were held at the Allen Research Commons at the University of Washington. The Allen Research Commons is an open space on the bottom floor of a library containing multiple tables, booths, and group work areas. This environment was chosen for usability testing because it is an area where students often work in groups, therefore the noise level is comfortable for people to talk at a normal volume. We wanted our participants to be able to talk freely and openly during testing without worrying about being too loud in a library setting.
For each usability test, we had one moderator, one quantitative notetaker, and one qualitative notetaker.
Prior to running the usability tests, we had set up a Google Form containing the pre-test questionnaire, 6 post-task questionnaires, and the post-test questionnaire. The quantitative note-taker filled out one Google Form for each participant.
We also used a usability tool called Lookback on the testing device. Lookback recorded a screen-captured video of the user going through the screens for each task. After uploading the video for each participant, Lookback automatically marked timestamps for when the user switched screens. This allowed us to collect quantitative data on the time it took the participants to complete each task.
Qualitative notes were collected via the think-aloud protocol conducted on each participant. The qualitative notetaker listen and typed up everything the participant said out loud and also watched the testing device marking down the interactions.
We compiled our qualitative and quantitative raw data into a single Google Doc, attached in Appendix, which is organized by tasks. The quantitative data includes the timing for task completion, variability for task timing, likert scale in pre-test and post-test questionnaires, and task completion evaluation in post-task questionnaires.
We started our data analysis with reading through the compiled data file. Dividing the file into 3 portions, each team member read through the data and jotted down information discovered onto sticky notes. We ended up with putting all the sticky notes randomly on the wall, and grouping them into different themes: unclear terminology, confusing navigation, the differences between participants’ expectation and reality, struggles to find personalized new music and struggles to find recommended artists.
Based off the themes we summarized through affinity diagraming, we went through the compiled raw data again, attaching evidences corresponding to each theme, both qualitatively and quantitatively.
Result Summary
The Spotify mobile App contains terminologies that are unclear in accordance with the outcomes of those actions.
The information flow under navigation menu makes some features hidden and hard to find.
Expected Notifications from Friends
Expected Notifications from Artists
Expected to Find Recommended Music on Browse or Home Page
Random Titles for Personalized New Music
After asking the participants what they expect to happen after following a friend or an artist, 100% of participants expected to be notified about an artist’s new releases. 3 out of 7 participants expected to notified when an artist is playing a concert near by. 2 participants expected to be notified when an artist’s song or album is trending, or hits the top charts.
The participants then found that there were no notifications about any of the above unless a post is created or sent directly to another user. While 2 out of 7 participants did mention they prefer not to receive many notifications, most participants expressed interest in knowing what their friends were listening to within the Spotify app versus juts through the Facebook integration.
When asked to follow a friend on Spotify, 3 out of 7 participants started off by searching for a specific friend in the search bar. Once 2 out of 7 participants were prompted to try another way to follow a friend, it took an average of 4 clicks to access a friend’s profile from the navigation menu. Most of the participants went through each item in the navigation menu before reaching their personal profile to access their friends list.
When asked to first share a playlist, then make the playlist open to friends to add to, our participants were confused with the terminology presented in the edit menu. Spotify has two actions: Share Playlist and Make Playlist Collaborative. Share is meant to have multiple functions including sharing the playlist on social media as well as opening up the playlist for a friend to add to it once it becomes collaborative. Making a playlist collaborative allows the user to then share the playlist with a friend to add to it. These two actions caused confusion for the participants and 6 out of 7 struggled with adding a friend to a playlist. In a follow up question the participant that did not struggle had performed this action before, while another said that they couldn’t “figure out how to add someone to the playlist”.
Another major finding that we revealed is that finding personalized new music can be difficult and overwhelming, which contradicts Spotify’s mission, “...the right music is always at your fingertips.” By personalized new music, we are referring to recommended music suggested by Spotify based on users’ personal music preferences, which could be new releases from artists that users followed, playlist containing songs in the same genre with the ones in users’ library and etc.
Hidden Location of Discover
Spotify has a specific section called Browse, as shown in Figure 1, as a main option under the navigation menu. However, the quickly accessible music on the Browse page are not unique music recommendations for different users; most of the contents are either general new released music or music in trend. In fact, Spotify does have a section called Discover for personalized new music for different users; unfortunately, it has been placed in a spot, which seems not to be designed for quick access, as shown in Figure 2, in the current version of Spotify.
We also asked our participants to play a recommended playlist by Spotify, hoping to see where they would expect to find recommended music. As a result, 3 out of 7 participants reached the “Just for You” section on top of the Home page, which owns the same new music from Browse page in a slightly different layout F. In addition, 2 out of 7 participants took minutes to explore tabs on the navigation menu, searching for the keyword “recommended/recommendation” on top of the hidden discover menu.
Vague Titles for Music Recommendation Categories
We also uncovered that some titles for recommended music categories in Discover need more clarifications on the differences of the contents under those titles. For example, Spotify labeled recommended music based off artists that users listened with “Because You Listened to...” and “Suggested for You Based on…,” as shown in Figure 3, and we asked our participants for their opinions on the differences between the contents under each of the two titles. We found that 6 out of 7 participants didn’t give us a confident explanation, with 4 out those 6 guessing that wording could be the only difference.
First, we recommend changing the navigation style from the hamburger menu to a tab bar. The navigation style dominates the Information Flow/Organization in Spotify Mobile, so it is important that it is easy to use and takes up less space than necessary. We found that users revisited the hamburger menu and selected the same page multiple times because the menu was always hidden. Users could not remember what was in the navigation menu so they overused it. A Tab bar would show the user that all of the content that they want is right there on the screen.
Second, we suggest making the “Discover” content the first page seen when searching for music instead of the current Browse page. “Discover” is more relevant content to the user versus the current Browse page and it reduces number of taps t takes to get to relevant content.
Lastly, we would want to create a new page just for a “Social News Feed” where users can see information about people they are following without relying on notifications. A Social News Feed also provides a new way of finding music - an action that is very important to Spotify and Spotify’s users.
