Crowd-Powered Systems

Spring 2017 :: ECE 69500 :: Purdue University


Limits of AI

What Artificial Intelligence Can and Can’t Do Right Now (11/2016) by Andrew Ng - Dr. Ng is co-author on the paper that introduced latent dirichlet allocation (JMLR 2003), as well as one of the most highly cited papers about crowdsourcing (EMNLP 2008)

The humans working behind the AI curtain (1/9/2017) by Mary Gray & Siddharth Suri

The Brain vs Deep Learning Part I: Computational Complexity (7/2015) by Tim Dettmers

Teaching Computers To See By Seeing Like A Computer (2013)

ML + HComp

Crowdsourcing Feature Discovery via Adaptively Chosen Comparisons paper (11/2015)

Flock: Hybrid crowd-machine learning classifiers paper (2015) by Justin Cheng & Michael S Bernstein

Flock: Hybrid crowd-machine learning classifiers paper (2015) by Justin Cheng & Michael S Bernstein

Crowdsourcing in Computer Vision paper (2014) by Kovashka, Russakovsky, Fei-Fei, Grauman


AAAI 2017 conference (2/2017)

AAAI 2017 Workshop on Human-Machine Collaborative Learning (2/2017) - AI-centric

AAAI 2017 Workshop on Crowdsourcing, Deep Learning, and Artificial Intelligence Agents (2/2017) - AI-centric

NIPS 2016 Workshop on Crowdsourcing and Machine Learning workshop (12/2016) - ML-centric

Collective intelligence conference (6/2017) - HCI-centric, 2nd-tier

Mixture of AI, HCI, IR, …

WWW 2017 – Crowdsourcing track conference (4/2017) - mixture of AI, HCI, and other perspectives, 2nd-tier

HCOMP 2017 (AAAI Conference on Human Computation and Crowdsourcing) (10/2017) - mixture of AI and HCI perspectives, 2nd-tier but consistently strong and competitive

HCOMP 2016 Workshop on Human Computation for Image and Video Analysis (11/2016)


CHI 2017 conference (5/2017) - HCI-centric, top-tier, always has multiple sections with high-quality crowd work

UIST 2017 conference (10/2017) - HCI-centric, top-tier, always has high-quality crowd work

CSCW 2017 conference (2/2017) - HCI-centric, top-tier, always has high-quality crowd work

Mechanical Turk

Mechanical Turk: Requester home

Mechanical Turk: Worker home

Requester Best Practices Guide document (6/2011) from Amazon

Writing instructions

Printed instructions: can research make a difference? literature review from book by Prof. Patricia Wright - also freely available via

Instructions and procedures textbook chapter (2009) from Technical Communication by Profs. John Lannon & Laura J. Gurak

Instructions: Tell them how to do it from online technical writing textbook by Dr. David McMurrey

Instructions: how to write guides for busy, grouchy people blog post (7/16/2011) by Prof. Dennis G. Jerz

The elements of good instructions from technical writing course by Prof. Gian Pagnucci

The small print: writing user interface instructions (2/2010) by Connie Malamed

Guidelines for Writing Clear Instructions and Messages for Voters and Poll Workers (5/2009) from NIST (National Institute of Standards and Technology)

Form design

Forms that Work: Designing Web Forms for Usability book (2008) by Caroline Jarrett and Gerry Gaffney - also freely available via

Web Form Design: Filling in the Blanks book (2009) by Luke Wroblewski - not free

Creating Usable Online Forms blog post (4/1/2008) from by Heidi A. Uliasz

Opinion: Placeholders in form fields are harmful blog post (5/11/2014) from Nieldsen Norman Group by Katie Sherwin

Opinion: Progressive disclosure: valid or sneaky? blog post (5/31/2014) from Forms that work (web site about book) by Gianpiero Di Gianvittorio

Optimizing online forms blog post (6/16/2014) from designmodo by Paula Borowska

Extensive guide to web form usability blog post (11/8/2011) from Smashing Magazine

The Form Usability Treasure Trove: 42+ Top Resources blog post (7/7/2014) from by Phil Sharp

23 forms and data entry usability guidelines guidelines (2009) from by Dr. David Travis


Online courses about cybersecurity

Video explanations of the OWASP Top 10

Links for technologies (e.g., Python, Flask, JavaScript, etc.) are on the Tools page.