The base material for the user study is taken from the Open Source movie Tears of Steel by the Blender Foundation. Three clips, where each consists of 12 seconds of content without scene cut, were extracted from the movie. For each of these clips, 4 versions were created: one re-encoded with default quality, one where the right 25% of pixels are blurred, one where the right 50% of pixels are blurred, and one that is completely blurred. The videos are contained in this tarball.
Before the test starts, you should explain the test to your candidates. To show an example of the distortions, use this tarball.
Situations where parts of images are blurred can occur when a panorama video is encoded in several qualities to save bandwidth for web delivery, but a consumer is turning their head unexpectedly to look at a part of the panorama that is currently transmitted in low quality. This scenario has not been conclusively studied so far.
The method for collecting user data must be as follows:
- We use a single stimulus method (with dual stimulus, a user would have to watch (12 choose 2) = 66 videos clips, with single stimulus only 12).
- The user has to rate every clip on a 5-point Likert scale to collect categorical data. This should happen by rating how strongly the user agrees with the following statement:
"The quality of the video was satisfactory."
(1) strongly disagree, (2) disagree, (3) undecided, (4) agree, (5) strongly agree
The user should not see the numbers.
You may translate the statement and ratings to other languages. - The user should answer the question quickly, immediately after watching the a clip. A user may take pauses between clips.
- It is extremely important that every user watches all the videos, and that the order is completely randomized for every user.
Make sure that you can associate the scores with the right video! - You should make sure that users are able to understand the statement and the task they are given, and assert that they are able to discern the different video qualities (don't show the clips to them with strong backlighting, do they have appropriately corrected eyesight while watching etc etc).
- The clips are encoded in 1920x800 resolution. The user should watch them on a screen where the blur matters.
Finally, store the results in a file that can be imported into a common collection. It should be a CSV file containing lines as follows:
<id>,<candidate number>,<filename of the clip>,<rating>
where id is a unique number for each participant in the course. It is the same for all candidates tested by this participant. Choose an id randomly.