Design Project

What do we do?

This is a team-based, semester-long design project, in which you'll create an original crowdsourcing and social computing application on your own. You'll work in teams of 3-4 people. More information about each milestone will be added to this page. Here is the timeline and grading weights for each milestone:

• Team Formation (week 2): 0%
• Idea (week 3): 10%
• Story (week 5): 10%
• Pitch (week 7): 15%
• Low-fi Prototype (week 8): 10%
• Mid-fi Prototype (week 10): 10%
• High-fi Prototype (week 13): 10%
• Final Presentation (week 15): 15%
• Final Paper & Video (week 16): 20%

Please note that all team members will receive the same grade for all the milestones.

Why do this?

Now that you've mastered the arts and skills of crowdsourcing and social computing, it's important that you apply what you learned to a problem you deeply care about. It's a great way to learn further, and potentially make impact.

Does my project idea qualify?

Projects are open-ended, so please be creative. But here are some requirements:

• Your project should involve a VOLUNTARY OR INTRINSICALLY MOTIVATED CROWD, which means you cannot use paid crowds (e.g., MTurk, Upwork).
• Your project should have some user interface. I'm expecting most projects to be a novel crowdsourcing and social computing system that can be tested by others (e.g., a prototype website or a mobile app). Data mining or analysis focused projects are okay, but there needs to be some visualization or analytics dashboard component that is accessible by others to understand your results.
• Your prototype doesn't need to be a fully implemented platform, menaing that it needs to focus on the core crowdsourcing / social computing concept you're introducing, rather than to build all the database backend, login, etc.
• Connecting to your own research is encouraged, but please make sure to talk to course staff early about how to make it happen.

Examples

Here's the project gallery from the 2016 class.

How do we submit?

We'll create an assignment in KLMS for each milestone.

Late Policy

For each milestone deadline, all members of your team will lose 10% for each late day. Submissions will be accepted until three days after the deadline. After then you'll get 0 on that assignment. Please note that late submissions for midterm and final presentations are not allowed.

Milestone 0: Team Formation

Due: 11:59pm on 9/8 (Fri)

What do I do?

You'll need to find teammates to work on an exciting project this semester. Each team should have 4 people by default. In exceptional cases we'll accept 3-person teams.

How do I find teammates?

Here are three methods you can use:

• You can use Piazza to post advertisements for finding teammates.
• We'll give 5 minutes at the end of two classes for in-person team formation.
• Use office hours to talk to us about project ideas so that we can make connections, and you can also run into other classmates.

Please try to have diversity in your own team crowd, in terms of skillset, interest, and background.

How do I submit?

Please fill out the design project sign-up form.

Milestone 1: Problem Identification and Ideas

What do we do?

In a team, you'll identify a problem that you'd like to tackle with your project, and brainstorm approaches to solving the problem.

Your report

In your report, please answer the following questions:

1. What is the problem your team is trying to solve? (one sentence)
2. How do you know this problem exists? Why is this problem important? Include both (1) external references (e.g., academic papers, news articles, or published surveys), as well as (2) internal investigation (e.g., results from making observations, personal experiences, or interviews with target requesters or workers). (maximum one paragraph)
3. Why use crowdsourcing or social computing for the problem? Why not use machines or individual experts? (maximum one paragraph)
4. For the identified problem, discuss with your teammates what specific challenges exist. State these challenges as "How might we..." (HMW) questions. HMW questions serve as a bridge between the identified problem and solution ideas. They are short questions that should be broad enough to allow for open ideas yet narrow enough to set meaningful boundaries. Report at least 10 HMW questions for your team's problem, and pick top 3 HMW questions. This method card from Stanford d.school should be a useful resource.
5. Solution Ideas: Solution ideas are your attempts at solving the HMW questions. In your team, discuss how you might address these questions with crowdsourcing or social computing. Report at least 10 solution ideas per each of the top 3 HMW questions, and pick top 3 solution ideas overall. Note that these 3 ideas should be distinct from one another, covering a wide solution space. The best ideas don't necessarily have to represent each HMW; multiple ideas can belong to a single HMW.

Grading

• Problem (10%)
  • Clearly presented in a sentence?
  • Unique?
  • Non-trivial?
• Problem Background (20%)
  • Clearly presented in a paragraph?
  • Convincing evidence or references presented?
  • Importance of the problem highlighted?
• Motivation (20%)
  • Clearly presented in a paragraph?
  • Suitable for crowdsourcing or social computing?
  • Convincing reasons for using crowdsourcing or social computing presented?
• HMW (20%)
  • 10+ HMWs submitted?
  • Scope not too broad or narrow?
  • Distinct, broad, and creative?
• Solution Ideas (30%)
  • 10+ solution ideas submitted for each HMW?
  • Top 3 ideas are noted?
  • Distinct, broad, and creative?

How do we submit?

One report per team. Your report should be submitted as a zip file. The main report should be written in Markdown (please use the .md extension). We're going to publish your reports on the course website. Submit your team's report on KLMS.

Milestone 2: Tasks and Storyboards

What do we do?

Your team will now further develop the ideas that you came up with in Milestone #1. You'll create two types of artifacts that will capture your design process. Please follow the order presented below.

• Tasks: Based on the three solution ideas your team came up with, brainstorm three major tasks your solution wants to support. Tasks represent what the user (including but not limited to crowd workers, requesters, or community members) wants to achieve, rather than specific system features or UI elements. For each task, list three requirements or design goals that a solution should support. Some tips:
  • Make sure the three tasks are distinct from one another and cover different levels of complexity.
  • Be specific, and include details and context. "The user wants to learn math." is way too broad. Think about a particular user with their likely demographic, constraints, and motivation: "Seung Hyun, a 20-year old college student, wants to learn Chinese in his spare time. While he doesn't have time to take classes or get a private tutor, he wants to use his commute time to learn useful words and phrases."
• Storyboards: Create 3 storyboard sketches, one for each of the three tasks you identified. A storyboard is a set of comic-strip-like drawings that visually walks through a concrete scenario and a task that your user experiences. Show the challenge the user encounters, what environment the user is surrounded by, what motivates the user to use the solution idea you're suggesting, and how your solution idea actually addresses the challenge. The storyboard shouldn't be about specific system features or UI elements. You don't even need to show the detailed screen layout of your UI at this point. Focus on the concept and context, rather than pretty UIs or sophisticated system features.
  • Make sure to hand-draw your storyboards, with a thick pen so that sketches are visible when digitally scanned and excessive details are not included. Using a pencil or a thin ball-point pen is not allowed.
  • Here are some good examples (Verbivore and Let's Read), except for the fact that some of them are not drawn with a thick pen.
  • Resources: Amal Dar Aziz's guide to storyboarding is a highly recommended resource. Also, watch the first 6 minutes of this video by Scott Klemmer, which gives a nice introduction to storyboarding.

Your report

Your report should include the following:

• Tasks: Description of three tasks, each about a paragraph long. For each task, include three requirements or design goals, and explain why they matter.
• Storyboards: Three storyboards, hand-drawn.

Grading

• Tasks (40%)
  • 3 tasks submitted?
  • Are the tasks distinct from each other?
  • Do the tasks contain detailed context?
  • 3 requirements of goals submitted per task?
  • Strong rationale provided??
• Storyboards (60%)
  • 3 storyboards submitted?
  • Flow easy to follow and understand?
  • Easy to read (thick pen used)?
  • Not solution-driven but user- and scenario-driven?

How do we submit?

One report per team. Your report should be submitted as a zip file.

• Tasks: In Markdown (please use the .md extension)
• Sketches: Scan in png or jpg. Images need to be in a directory called images.

Submit your team's report on KLMS.

Milestone 3: Pitch

What do I do?

Now that you've identified an interesting problem, a set of concrete tasks you want to support, and a set of possible solutions, it's time to turn these into a convincing pitch!

You'll have 10 minutes to do the following:

• Motivate the problem.
• Present the identified tasks and explain why they matter.
• State what your proposed solution is, and why it'll be able to solve the problem and support the tasks well.
• Share your plan: (1) In your team, who'll be responsible for what? (2) How will you find the crowd or users to use and test your system?

After the pitch, you'll have three minutes for Q and A.

Note: We'll enforce a strict 10-minute time limit by cutting off the presentation. Please plan and rehearse.

Grading

• Organization (10%): Overall structure and flow of the presentation.
• Problem (20%): Well defined? Is it a real problem? What's the evidence?
• Solution (30%): Novel? Feasible? Quality control / aggregation / motivation... thought out?
• Plan (10%): Who does what and deployment plan.
• Articulation (10%): Delivery and clarity of the presentation.
• Visual aids (10%): Design and readability of the slides, use of effective visual aids and examples.
• Overall (10%): How engaging was the overall talk?

Your report

You'll present in class and submit your slides after the class.

How do I submit?

Your team's slides should be submitted as a PDF file, via KLMS.

Milestone 4: Low-Fi Prototype

What do we do?

Let's start giving your design idea a digital life by creating a low-fidelity digital prototype. Rather than use HTML and Javascript, you'll first use a prototyping tool to implement the tasks and scenarios. Specifically, we recommend Marvel, InVision, or proto.io. You can also choose any prototyping tool of your choice.

Functionlity does not need to be fully implemented. Please use hard-coded but realistic data and fake results, so that you can focus on the essential: the user's task, scenario, and UI components to support them rather than the underlying implementation. You need to build a prototype that supports end-to-end scenarios. Your prototype needs to support at least three distinct tasks. This does not mean you need to build three separate prototypes, but rather this means you need to build one complete prototype that is flexible enough to support the three tasks.

After building your digital prototype, find at least three participants to test it. Make sure they are not friends who know about your project already, or classmates. Preferably, try to find participants who are close to your target user group.

Your report

Your report should include:

• Problem Statement: Re-state the problem you're tackling in your project. Based on what you've learned so far, you can revise your problem statement.
• Tasks: List three core tasks you've decided to support in your prototype. Again, you can reuse or improve what you had earlier.
• Prototype:
  • Prototyping tool: Which tool did you choose? Why? What worked well with the tool? What didn't?
  • Design choices: What did you intentionally choose NOT to implement (e.g., fake or hard-coded data, manual algorithm, etc.), and why?
  • Representative screenshots: Include a few most important screenshots that showcase the uniqueness of your application.
  • Instructions: Your prototype needs to be accessible and executable by anyone with the link. Please include instructions for running your prototype in detail, if needed.
• Observations: List at least 10 usability problems you discovered. Organize them by high-level task or theme, not by each participant or time. But mention which participant ran into the problem by referring to them as P1, P2, ... (e.g., search results did not show the price information (P1, P3)). For each problem, indicate how critical the problem is: high, medium, and low. Finally, show how you plan to address each of the problems in the later stage of your design process.

Grading

• Problem Statement (5%)
  • Problem statement is clearly written?
  • Lists a problem not a solution?
  Tasks (15%)
  • Tasks align with the problem statement?
  • Three or more tasks?
  • Are the tasks distinct from each other?
  • Are the tasks described concretely and clearly?
  • User-level description not functionality description?
• Prototype (50%)
  • Tool description and justfication are clear?
  • Design choices are justified?
  • Design choices mention what's not selected by the team?
  • Screenshots are added?
  • Screenshots capture representative moments of the prototype?
  • The prototype captures the three tasks?
  • The prototype is complete in that it supports an end-to-end scenario?
  • The prototype is accessible and executable?
  • Instructions for running the prototype are clear?
• Observations (30%)
  • 10+ submitted?
  • Are the usability issues described concretely and clearly?
  • Organized by task and theme?
  • Level of criticality included?
  • Is the plan for improvements reasonable and sound?

How do I submit?

One report per team. Your report should be submitted as a zip file. The main report should be written in Markdown (please use the .md extension). We're going to publish your reports on the course website. Submit using KLMS.

Milestone 5: Mid-Fi Prototype

What do I do?

Finally, the tasks and scenarios you want to support need to be available in an interactive prototype.

Functionlity does not need to be fully implemented. Please use hard-coded but realistic data, Wizard-of-Oz, and fake results, so that you can focus on the essential: the UI and the crowd workflow rather than the underlying implementation.

Your report

Your report should include:

• Instruction: Include any information for someone who doesn't know about the project to be able to try the prototype and offer feedback (e.g., what tasks to try, what's working and not working now).
• Prototype: It should be either (1) a URL to your prototype, or (2) files (e.g., HTML, CSS, JavaScript) that can be run locally.

Grading

As long as you submit the report by the deadline, your team will get full credit. This milestone is meant to be a simple checkpoint.

How do I submit?

One report per team. Your report should be submitted as a zip file. The instructions should be written in Markdown (please use the .md extension). Submit using KLMS.

Milestone 6: High-Fi Prototype

What do I do?

Now's the time for a funlly functional and interactive prototype that is ready to be tested by your target users. You need to build a prototype that supports end-to-end scenarios captured in your earlier prototypes. Your prototype needs to support at least three distinct tasks. This does not mean you need to build three separate prototypes, but rather this means you need to build one complete prototype that is flexible enough to support the three tasks. You may choose to reuse or revise the tasks and the UI you created in earlier stages.

Your report

Your report should include:

• Project Summary (maximum three sentences): (1) the problem you're addressing, (2) what your solution is, (3) what unique approach you're taking in your solution (how it's different from other similar solutions). This summary will be used as a blurb on the project gallery.
• Instruction: Give a quick tour of the interface, and also show off some of the highlights of the interface. Note that this should not cover all features you have; focus on the most exciting and important parts. Use screenshots and callouts.
• URL of your prototype: A live version of the prototype for evalation. Note: the URL must work at least until your assignment is graded. Course staff will run your prototype to do a heuristic evaluation for grading. If the link doesn't work, your team will be penalized. If there are specific requirements (e.g., browser or device settings), include them as well.
• URL of your Git repository: Make sure to add a README file that briefly describes the code, e.g., main JavaScript file, or where main feature implementations are, etc. Several lines are enough.
• Libraries and frameworks: List any external dependencies you used for your implementation (e.g., Bootstrap, Semantic UI, etc.).
• Individual Reflections: Each member should write this part on their own, reflecting on their own experience. Merge all members' mini-reports in the final report. Answer the following questions:
  • Which part the UI did you directly contribute to?
  • What were some of the difficulties you faced?
  • List one useful skill you learned while working on the prototype.

Grading

• Project Summary (10%)
  • Clear description of the problem?
  • Clear description of the solution?
  • Clear description of the approach?
• Instruction (20%)
  • Screenshots are added?
  • Is the instruction clear and easy-to-follow?
  • Screenshots capture representative features of the prototype?
• Prototype (40%)
  • The prototype is complete in that it supports an end-to-end scenario?
  • Heuristic evaluation results: how usable is it?
• Implementation Notes (15%)
  • Prototype URL is accessible and works properly?
  • Repository URL is accessible and contains README?
  • Libraries and frameworks used are listed?
• Individual Reflections (15%) -- graded individually
  • Individual contribution clearly specified?
  • Difficulty discussion has enough depth and insight?
  • Non-trivial implementation skill added?

How do I submit?

One report per team. Your report should be submitted as a zip file. The instructions should be written in Markdown (please use the .md extension). Submit using KLMS.

Milestone 7: Final Presentation

What do I do?

Now that you have an awesome prototype that is (hopefully) getting a lot of excitement from your crowd, it's time to tell us about what you built and what you learned from having real users use the system.

You'll have 12 minutes to do the following:

• Motivate the problem.
• List the main tasks you're trying to support.
• What is your solution? How is it different from existing systems? Why is it better?
• Walk through your interface. You can use video or screenshots, or even do a live demo. But be careful with live demos, because they require a lot of practice and include many moving parts.
• Show us how your deployment went. How many users did you attract? What are some collected results suggesting that your application serves the task well? What did you learn from talking to users?
• Limitations, implications, and possible improvements.

After the presentation, you'll have five minutes for Q and A.

Note: We'll enforce a strict 12-minute time limit by cutting off the presentation. Please plan and rehearse.

Grading

Here's how your pitches will be graded.

• Organization (10%): Overall structure and flow of the presentation.
• Problem (10%): Well defined? Is it a real problem? What's the evidence?
• Solution (20%): Does the application deliver its initial promises? Is it novel? Feasible? Are important design factors such as quality control, social interaction, aggregation, motivation, etc. well thought out?
• Results (20%): What did you learn from the deployment? What are the qualitative and quantitative results from users?
• Discussion (10%): Depth in discussion of limitations, implications, and possible improvements.
• Articulation (10%): Delivery and clarity of the presentation.
• Visual aids (10%): Design and readability of the slides, use of effective visual aids and examples.
• Overall (10%): How engaging was the overall talk?

Your report

You'll present in class and submit your slides after the class.

How do I submit?

Your team's slides should be submitted as a PDF file, via KLMS.

Milestone 8: Final Report

What do I do?

Now that you have deployed your system for a couple weeks, seen your crowd use the system, and collected useful data, it's time to write it all up! You've already read and written responses to 20+ crowdsourcing and social computing papers, so let's write one on your project. Also, you'll make an engaging video.

Paper

Find a model paper that your team likes or is inspired by, and use it as a template. Here's a suggested outline:

• Abstract: One paragraph summary of the project. This is the most important part of the paper. After reading it, a reader should understand the problem you're tacking, the approach you're taking, and the results of your evaluation.
• Introduction: What is the state of the world? What's the burning problem? What's your solution? What's the summary of results?
• Background and Related Work: What backgrond knowledge do you want your readers to have to better understand your motivation? What existing techniques or ideas are you inspired by? How does your approach differ from them?
• System: What are the major tasks you're supporting, and how does your system support them? Report on the unique aspect of the interface and the crowdsourcing / social computing design. Include screenshots of important interactions.
• Evaluation: How did your deployment go? Report the number of users, results, analysis, etc. Use visual aids to effectively communicate the results. NOTE: Please do not include results generated by your own team!!
• Discussion: What lessons did you learn? Specifically, discuss what you learned about designing a system for a VOLUNTARY OR INTRINSICALLY MOTIVATED CROWD. Also discuss limitations, design implications, and possible improvements.
• Conclusion: Not necessary.
• References: List of references.

Video

Video is a great way to demonstrate the awesomeness of your application. Record a 2-minute video that captures the user context and the killer features of your UI. Be creative in how you plan, structure, and record the video! Avoid using slides and try to capture realistic context, and don't hesitate to "act". Do not show the UI from the beginning. You need to show parts of your final prototype to demonstrate how the user might perform the task using your system. Rather than describe all the features you implemented, focus on the flow of the task.

• Some examples from CS374: Intro to HCI class.
• Guidelines for making a video (adopted from CHI 2017 Video Showcase)
  • Encoded as MP4 using the H.264 codec. No exceptions!
  • All spoken dialog must be closed captioned (subtitled) to improve accessibility.
  • Resolution of at least 1280px x 720px.
  • The 16:9 aspect ratio is recommended.

Grading

Here's how your report will be graded.

Part 1: Paper (12% toward your project grade)

• Idea (20%): Motivation, problem identification, and approach are clearly and convincingly presented.
• Background / Related Work (10%): Covers relevant background information and research.
• Execution (20%): The system delivers its initial promises. It's technically strong and robust, and involves crowdsourcing in a non-trivial manner.
• Evidence (20%): The evaluation is complete, and shows if and how the system worked as expected. Results are presented in an academically sound manner.
• Discussion (20%): Discussion touches upon important design dimensions and shows reflective thinking.
• Presentation (10%): It is overall well-written. Images, tables, and charts effectively communicating the message.

Part 2: Video (8% toward your project grade)

• 2 minutes or under?
• User needs and contexts well captured?
• Is the overall organization and flow of the story smooth?
• Quality and variety of shots and rhythm?
• UI introduced convincingly but not as dry list of features?
• Engaging and creative?

Your report

• Paper: A maximum 4-page paper in the CHI Proceedings Format, including references.
• Video: A video clip, maximum 2 minutes long. You'll upload your video file on YouTube and also upload the MP4 file to KLMS. Please add a URL of your video to your final paper.

How do I submit?

Your team's report should be submitted as a PDF file, via KLMS.

Extra Credit #1: Cool Video

Up to +10% on the final report

We'll reward a couple teams with most engaging videos. We'll announce the winner(s) and explain why they deserve the "Cool Video" title.

Extra Credit #2: Best Crowdsourcer

Up to +10% on the final report

Teams that successfully engage active users deserve some credit! We'll reward 1-2 teams that most successfully showed the power of the crowd. This does not just mean the most number of users, but includes the overall participation, quality of data, and your analysis of the data. We'll announce the winner(s) and explain why they deserve the "Best Crowdsourcer" title.