Design Project 7: User Testing
Studio: 5/29 (Mon), slides by 5pm
Final Write-up: 11:59pm on 6/1 (Thu)
6% toward your total grade
What do we do?
Now let's show your working prototype to representative users and get their feedback. Find at least 3 users representative of the target population, let them test your prototype, and learn how you might improve your interface. Note that participants should not be students taking this class. We suggest that your testing process follow the three steps outlined below:
Planning: Written protocol
You should prepare a written protocol -- a planning document -- before you run sessions. It is important to plan and rehearse in advance so that your participants receive consistent experimental treatment, which in turn leads to valid and reliable data. Your protocol document should include:
- Instructions for preparation and setting up the testing environment: Anyone in your team should be able to set up and run the experiment just by referring to this document. Example information includes any URL to open, web browser to use, example data to load, etc.
- Introduction and informed consent: Make sure you read this introduction exactly as written in a study session. For informed consent, we don't require formal paperwork, but you should explain what information you're collecting (e.g., photos or voice recording) and get participants' consent.
- Tutorial or training (if needed): If your tasks could benefit from participants' familiarity with the interface, we advise adding a tutorial or a training session. It's a good idea to use a simple, concrete example task to walk through the interface.
- Task list & instruction: What are the tasks participants should perform, and what instruction should they be given? This is another part that should be delivered exactly as written to all participants.
- Recording strategy: How will you record your observations and findings? We advise against video recording, but instead recommend sketches, photos, voice recordings, screen captures, screen recordings, and written notes as you see fit. Also discuss what role members of your team will play in each session: facilitator, observer, and optionally tech person in charge of monitoring if the system is working fine and data is collected properly.
- Questionnaires, Interview questions: After the session, it's a good idea to get qualitative comments about the task experience from participants. In doing so, we suggest preparing a list of questions to ask in advance. For interviews, consider semi-structured format: follow the initial plan you came up with, but ask additional questions based on participants' responses.
- Debrief prompt: After the session, thank your participants and give them a short debrief about the experience and the project.
Testing: Session observations
Before running actual sessions, do a couple dry rehearsals within your team or with your friends. Carefully follow the protocol you have written up, and debug it. As you make observations, focus on usability issues and whether participants find the interface actually useful.
Reflection: Usability lessons
What did you learn about the usability of your interface? Revisit your interface design, tasks, personas, POV, and user needs, and review if any of them needs revision or additional work.
- Written protocol
- Session observations
- Participants: Who are they? How did you recruit them? Why are they representative target users? Add brief demographic, context information about each participant that is relevant to your interface.
- Use at least one photo or sketch for each participant and provide a summary description of each session (e.g., What was unique about this participant? Main takeaway from this participant?).
- Usability lessons
- List at least 10 usability problems you discovered. Organize them by high-level task or theme, not by each participant or time. But mention which participant ran into the problem by referring to them as P1, P2, ... (e.g., search results did not show the price information (P1, P3)). For each problem, indicate how critical the problem is: high, medium, and low. Finally, show how you plan to address each of the problems.
- High-level reflections: What did you overall learn from the user testing experience? What would you do differently for better results and insights?
- Studio Reflections: Summarize the feedback from the studio session, and mention how you addressed it or will address it later in the process.
- Plan for iteration: You'll have an additional week to finalize your overall design process. Discuss in your team how to use this time most effectively, and make a few concrete goals. Justify your goals.
- Written protocol (30%)
- Preparation and setting information is detailed enough?
- Introduction is concise, to the point, and readable?
- Tutorial or training is well-designed? (if exists)
- Task list and instruction is clearly written and readable?
- Recording strategy is carefully planned, in a way that can capture useful insights?
- Questionnaires and interview questions are carefully thought out to get at deep insights and thought process of participants?
- A debrief prompt is well-written in a polite and informative manner?
- Session observations (20%)
- Participants are representative?
- Participant information is descriptive and contextual?
- Visual aids (photo or sketch) are added?
- Summary of each session is provided?
- Usability lessons (20%)
- 10+ usability issues submitted?
- Are the usability issues described concretely and clearly?
- Organized by task and theme?
- Level of criticality included?
- High-level reflections are thoughtful and insightful?
- Plan for iteration (10%)
- Is the plan for improvement reasonable and concrete?
- Are the goals well-justified?
- Studio Reflections (10%)
- Is feedback well summarized?
- Is feedback addressed, or is the plan for addressing feedback concrete?
- Studio Presentation (10%)
- Preparation and organization?
- Articulation and clear delivery?
- Effective use of visual aids?
- Time management?
: In studio, your team will present your main findings for 5 minutes, with 5 minutes for Q&A and feedback. You need to prepare a Google Slides presentation, by adding your slides to the link your TA will send you. The slides should be sent to your TA by 5pm on Monday
. Every team member
needs to participate in the presentation. Write down the questions and feedback you receive during your presentation, and reflect on them in your report.
: One report per team. Your report should be submitted as a zip file. The main report should be written in Markdown
(please use the .md
extension). We're going to publish your reports on the course website. Submit your team's report on KLMS