Contact Us

Use the form on the right to contact us.

You can edit the text in this area, and change where the contact form on the right submits to, by entering edit mode using the modes on the bottom right. 


Chicago, Illinois

(815) 575-0265

I have an ambition to research, discover and share knowledge in human-computer interaction and user-centered design.

NetHuluImg2.jpg

Netflix vs. Hulu - User Test

The following is a user testing project that was conducted for my Human-Computer Interaction master's program. 

STUDY OVERVIEW

METHODS

RESEARCH OBJECTIVE

A within-subjects summative usability test was performed on two of the most popular streaming video platforms: Hulu and Netflix. The test was created to see which of the two interfaces provided a better user experience and to see which interface allowed users to find and save movies in a way that was easy and efficient. 

MY CONTRIBUTIONS

  • A/B testing 
  • Moderation guide writing
  • Interview transcripts
  • Findings coding and summary
  • Recommendations
  • Report writing and editing

MAJOR TAKEAWAYS

Overall, the success rate for completion was very high. Only Hulu experienced instances where users failed to complete tasks -- specifically, when it came to users adding a movie to their personalized movie list, and then subsequently finding and viewing their personalized movie list. The time-on-task data illustrated there were significant challenges when it came to saving a movie to a personalized list on both Hulu and Netflix. Ease-of-use ratings showed fluctuations by task, but in overall ease-of-use, users rated Netflix more favorably.

STUDY Breakdown

USABILITY TEST PLAN

Objective

The focus of this summative study is to compare the usability of two competing streaming video platforms, Netflix and Hulu, and evaluate which platform has the most effective and satisfying interface. Specifically, we would like to answer the following questions:

  • Can users efficiently find the movie they’re looking for?
  • Can users save their movies to a list to refer to later?
  • Can users remove movies from their list?
  • Are users satisfied with their overall experience?

Stimuli

Two different streaming video interfaces will be compared: Netflix and Hulu.

*Note to moderator: ensure that the order of the sites is randomized for the test users.

Tasks

Participants will be asked to perform a few tasks using the two interfaces. The tasks will cover the following cases:

  • Finding a desired movie to watch
  • Adding a movie to their personalized movie list
  • Finding and viewing their personalized movie list
  • Removing a saved movie from their personalized movie list
  • Measures: Quantitative and Qualitative 

Measures: Quantitative and Qualitative

Quantitative measures (Primary):

1.    Time on task: Total amount of time spent on task will be recorded
2.    Ease-of-use ratings for each task: At the end of each task, the participants will be asked to rate their ability to use the interface from 1 (very difficult) to 10 (very easy)

Qualitative measures (Secondary): 

1.    Usability issues observed
2.    Participant comments: Spontaneous comments from the participant will be recorded and clarified
3.    Follow up questions will be asked concerning the participant’s ease-of-use ratings 

Participants

15 participants will be recruited for this study. 

Participants’ profile

  • Mix of ages 
  • Mix of genders 
  • Moderate familiarity with streaming video interfaces or platforms

Click here to view the moderator guide

 

Quantitative FINDINGS

Click here for the full quantitative and qualitative findings as well as recommendations

Task 1: Can users efficiently find the movie they’re looking for?

Compare success rate

  • We conducted Fisher’s Exact test to compare the Task 1 completion rate with Netflix (100%) to the Task 1 completion rate with Hulu (100%) and found no statistically significant difference at alpha level .05.

Compare time on task

  • We conducted a paired t-test to compare the time on Task 1 between Netflix and Hulu. We found no significant difference (t(14) = 0.43, p > .05) between Netflix (M = 46.1 s, SD = 54.0 s) and Hulu (M = 34.5 s, SD = 31.7 s).

Compare ease of use ratings (1 very difficult, 10 very easy)

  • We conducted a paired t-test to compare the ease of use rating for Task 1 between Netflix and Hulu. We found no significant difference in the participant’s ease of use rating (t(14) = 0.59, p > .05) between Netflix (M = 8.80, SD = 2.34) and Hulu (M = 9.20, SD = 1.42). All participants successfully completed Task 1, with a mean time of completion at 43.1 seconds for Netflix and 34.5 seconds for Hulu. Users also gave similarly high ratings for their ease of use experience for both sites. The two main reasons for the high ease of use ratings are the search bar locations and how the search results are presented.

TASK 2: Can users save their movies to a list to refer to later?

Compare success rate

  • We conducted Fisher’s Exact test to compare the Task 2 completion rate with Netflix (100%) to the Task 2 completion rate with Hulu (60%). The completion rates were significantly different (p < .05), showing that more participants successfully completed Task 2 when using Netflix than when using Hulu.

Compare Time on Task

  • We conducted a paired t-test to compare the time on Task 2 between Netflix and Hulu. We found no significant difference (t(8) = .57, p > .05) between Netflix (M = 59.33 s, SD = 56.93 s) and Hulu (M = 72.67 s, SD = 37.33 s).

Compare ease of use ratings (1 very difficult, 10 very easy)

  • We conducted a paired t-test to compare the ease of use rating for Task 2 between Netflix and Hulu. We found a significant difference (t(14) = 0.03, p < .05), showing that participants gave a higher ease of use rating when using Netflix (M = 6.93, SD = 2.55) than when using Hulu (M = 4.13, SD = 2.90).

Task 3: Can users find and view their personalized movie list?

Compare success rate

  • We conducted Fisher’s Exact test to compare the Task 3 completion rate with Netflix (100%) to the Task 3 completion rate with Hulu (93.33%) and found no statistically significant difference at alpha level .05.

Compare Time on Task

  • We conducted a paired t-test to compare the time on Task 3 between Netflix and Hulu. We found no significant difference (t(13) = . 0.23, p > .05) between Netflix (M = 31.86 s, SD = 31.89 s) and Hulu (M = 51.50 s, SD= 55.17 s).

Compare ease of use ratings (1 very difficult, 10 very easy)

  • We conducted a paired t-test to compare the ease of use rating for Task 3 between Netflix and Hulu. We found no significant difference in the participant’s ease of use rating (t(14) = 0.25, p > .05) between Netflix (M = 8.07, SD = 1.75) and Hulu (M = 7.20, SD = 2.51).

Task 4: Remove a saved movie from their personalized movie list? 

Compare success rate

  • We conducted Fisher’s Exact test to compare the Task 4 completion rate with Netflix (100%) to the Task 4 completion rate with Hulu (100%) and found no statistically significant difference at alpha level .05.

Compare Time on Task

  • We conducted a paired t-test to compare the time on Task 4 between Netflix and Hulu. We found no significant difference (t(14) = .0.31, p > .05) between Netflix (M = 21.13 s, SD = 30.24 s) and Hulu (M = 12.93 s, SD = 13.37 s).

Compare ease of use ratings (1 very difficult, 10 very easy)

  • We conducted a paired t-test to compare the ease of use rating for Task 4 between Netflix and Hulu. We found a significant difference (t(14) = 0.03, p < .05), showing that participants gave a lower ease of use rating when using Netflix (M = 8.80, SD = 1.78) than when using Hulu (M = 9.93, SD = 0.26).