Crowd-Powered Systems

– Spring 2017 :: ECE 69500 :: Purdue University

Schedule

Week 1: 1/10
Foundations, terminology
NO READINGS
Warm-up #0: self-introduction in HTML, due 1/15 (Sun)
1/12
"
Human Computation: A Survey and Taxonomy of a Growing FieldHuman Computation: A Survey and Taxonomy of a Growing Field
Week 2: 1/17
Crowd-powered systems
Soylent: a word processor with a crowd insideSoylent: a word processor with a crowd inside
Warm-up #1: post a HIT via AMT web site, due 1/22 (Sun)
1/19
Image understanding
Platemate: crowdsourcing nutritional analysis from food photographs.Platemate: crowdsourcing nutritional analysis from food photographs.
Week 3: 1/24
Interface design: forms
Extensive Guide to Web Form UsabilityExtensive Guide to Web Form Usability
Developing an Online FormDeveloping an Online Form
Instead of a normal response, please send a screenshot of an example from the web, and analyze it in the context of the respective reading. This can be in the body of the email or in an attachment, at your option.
1/26
Interface design: instructions
Instructions: how to write guides for busy, grouchy peopleInstructions: how to write guides for busy, grouchy people
Post a screenshot of a task interface—from AMT or any other online “task”—and analyze it how well it follows the article's guidelines.
Week 4: 1/31
Open problems
The Future of Crowd WorkThe Future of Crowd Work
Warm-up #2: design and implement task UI, due 2/1 (Wed)
Project: declare groups, due 2/3 (Fri)
2/2
Hierarchical task decomposition
Collaboratively crowdsourcing workflows with turkomaticCollaboratively crowdsourcing workflows with turkomatic
Week 5: 2/7
"
Crowdforge: Crowdsourcing complex workCrowdforge: Crowdsourcing complex work
2/9
Computer vision: overview
Crowdsourcing in Computer Vision - Chapter 2Crowdsourcing in Computer Vision - Chapter 2
Week 6: 2/14
Computer vision: overview
Crowdsourcing in Computer Vision - Chapters 3-4Crowdsourcing in Computer Vision - Chapters 3-4
Warm-up #3: implement server backend + add UI tracking, due 2/14 (Tue)
Project: proposal, due 2/15 (Wed)
2/16
Computer vision: image segmentation
Pull the Plug? Predicting If Computers or Humans Should Segment ImagesPull the Plug? Predicting If Computers or Humans Should Segment Images
Week 7: 2/21
Computer vision: city streets
Combining crowdsourcing and google street view to identify street-level accessibility problemsCombining crowdsourcing and google street view to identify street-level accessibility problems
2/23
Image understanding in real-time
VizWiz: Nearly Realtime Answers to Visual QuestionsVizWiz: Nearly Realtime Answers to Visual Questions
Week 8: 2/28
Security
Thirteen principles to ensure enterprise system securityThirteen principles to ensure enterprise system security
Principles of SecurityPrinciples of Security
Security GoalsSecurity Goals
Instead of a normal response, for each principle mentioned in the first two articles, think of one example not mentioned in the articles. Try to find examples others might not think of. We are reading less formal sources to ensure that everyone gets the high level principles as clearly as possible. There is some overlap between the first two.
Project: introduction, due 3/5 (Sun)
3/2
Security
OWASP Top 10: The Ten Most Critical Web Application Security RisksOWASP Top 10: The Ten Most Critical Web Application Security Risks
Week 9: 3/7
Security quiz
No reading
Project: v0.1, due 3/12 (Sun)
3/9
Real-time
Crowds in two seconds: enabling realtime crowd-powered interfacesCrowds in two seconds: enabling realtime crowd-powered interfaces
Week 10: 3/23
Real-time: development
No reading
Project: email check-in, due 3/26 (Sun)
Week 11: 3/28
Programming crowds: SQL (+ AMT APIs)
CrowdDB: answering queries with crowdsourcingCrowdDB: answering queries with crowdsourcing
This paper includes an explanation of AMT platform concepts you need for warm-up #4.
3/30
Programming crowds: development
No reading
Week 12: 4/4
Programming crowds: iterative vs. parallel
Exploring Iterative and Parallel Human Computation ProcessesExploring Iterative and Parallel Human Computation Processes
Warm-up #4: complete crowd-powered system, due 4/5 (Wed)
Project: v0.5, due 4/7 (Fri)
4/6
Machine learning: hybrid classifiers
Flock: Hybrid crowd-machine learning classifiersFlock: Hybrid crowd-machine learning classifiers
Week 13: 4/11
Quality
Financial incentives and the performance of crowdsFinancial incentives and the performance of crowds
Project: email check-in, due 4/12 (Wed)
4/13
Quality
Quality Management on Amazon Mechanical TurkQuality Management on Amazon Mechanical Turk
relatively light
Week 14: 4/18
Beyond AMT: human computation games
Designing Games With a PurposeDesigning Games With a Purpose
relatively light
Project: v1.0, due 4/21 (Fri)
Project: evaluation data, due 4/23 (Sun)
4/20
Beyond AMT: macro-tasks
Expert Crowdsourcing with Flash TeamsExpert Crowdsourcing with Flash Teams
Week 15: 4/25
Beyond AMT: citizen science games
The challenge of designing scientific discovery gamesThe challenge of designing scientific discovery games
This is about Fold.it, a game for predicting protein structures. This project was a big deal. We are reading a minor paper about its design because the Nature article—while far more cited—assumes some knowledge of biology, and has less to offer for our context.
Project: video, due 4/26 (Wed)
Project: report, due 4/29 (Sat)
4/27
Projects
NO READINGS
Assignments and readings may be changed without notice up to 1 week before the due date.

Readings

How to read papers for this course

Many people find it easier to read papers if they have a purpose in mind.  As you read each paper, you might find it helpful to focus on a few questions:

  1. What was the contribution type? (examples)
  2. What do the authors claim as their key contributions?
  3. What strategies, methods, and technologies were used?
  4. What generalizable knowledge does the work contribute?  What research questions does it address?  How will this benefit other researchers?
  5. Do you find the conclusions convincing?  Are the results well-supported by data obtained with sound methods?
  6. What aspects of the work you find strongest? … and weakest?
  7. What would be a natural next step for the work?

Note: These questions are included only to help guide your reading. They won't apply to all of the readings.  Your responses may or may not include your answers to any of these questions.

How to write your response

A good response will clearly express a well-founded opinion about the work.  The requires understanding the paper and thinking about what you liked/disliked about it.  Your response will show that you read and understood the paper, without directly summarizing it.  This might take some practice.

As a starting point, look at the discussion think through your answers to questions #5, #6, and #7 from above.  Next, look at the discussion so far (if any).  Do you agree? … or disagree?  Think of 2-3 insights, including 1-2 points about the paper as a whole, and 1-2 points about specific details (i.e., methods, analysis, etc.).  Try to avoid duplicating existing comments.

Updated 1/12/2017:

Email reading responses to the instructor by 10am before class by email with a subject line in the following format: reading YYYYMMDD Title [ece695cps] For example, the subject for the first reading (due Jan 17), will be as follows: reading 20170110 Soylent: a word processor with a crowd inside [ece695cps] Try to follow this exactly. That will help me keep them straight in my email box. Note there are no colons or quotation marks other than whatever might be in the paper title itself. "reading" is lowercase.

There is no hard requirement on length, but 2-3 paragraphs with 100-200 words (total) is probably a good target.  2 points will be reserved for especially insightful comments.  I expect that most will receive 1 point.

Your lowest 3 reading responses will be dropped. If you have a conference or other super-busy week, you're welcome to use your drops as skips. If you need to do that, please at least read the abstract and conclusion, and skim over the rest before you come to class, since we will be discussing these. Also, send me a response in the format above with just a note saying you are skipping that one. That way, I'll know it wasn't just lost.

You do not need to do a response for any paper of which the instructor is an author.

Company profiles

Find a company related to human computation or crowd labor. Give a 5-10 minute presentation in class about it. Send me the company's name, URL(s), and a short (2-3 sentences) summary by email by the night before your presentation. I will post your summary to the course web site.

Sign up here

Companies focused on crowdsourcing and/or human computation:

For more, search CrunchBase for terms such as “crowdsourcing”, “freelancing”, “task management”, “collaboration”, and “outsourcing”. Then read the description, look at the company's web site, or search around to determine if their operations actually the bill.

Major products driven by crowdsourcing and/or human computation:

* Big companies tend to keep internal processes and architectures private, so this list is undoubtedly incomplete.

Patents held by big companies: