You will work in groups of 2-3 to design, implement, and evaluate your own crowd-powered system. It should be something within the realm of “human computation” and/or “crowdsourcing”.
Milestones
See the course schedule.
Topic
The course readings and discussions should hopefully give you plenty of open research problems and interesting application ideas. In addition, the following may give you some inspiration:
- project ideas from Crowdsourcing & Human Computation course taught by Prof. Chris Callison-Burch at Penn.
- Follow the Crowd blog
- program from the recent CHI, CSCW, UIST, and HCOMP conferences
Human subjects research
If you are doing something that you might want to publish as “research”, you must obtain approval from the Purdue IRB. This typically takes a few weeks. You can submit your application early, even before you are finished implementing the system. Approval is not needed for a purely educational exercise. See me and check the Purdue IRB web site—especially their Determination of Human Subjects Research Worksheet—for more information about this.
Budget
Each student will have a budget of at least $50 to spend on paid crowd labor. You can pool your allocation with your group. If you need more, let me know and I can most likely accomodate.
Milestones
| 2/3 | declare groups | Send your group declaration by email with the subject line "project milestone 1 group declarations [ece695cps]". |
| 2/15 | proposal | Send your group declaration by email with the subject line "project milestone 2 proposal [ece695cps]". |
| 3/5 | introduction | Create an HTML file with a 600- to 1200-word description of your project plan. If your project is research-oriented, the description should include (a) title, (b) introduction, (c) ≈5-10 references, (d) research question, (e) what you might build as far as you know, (f) how you will test, and (g) how you will recruit participants. For system design projects, it should include (a) title, (b) introduction, (c) ≈5-10 references, (d) description of the target users and user goals this would serve, (e) how you will study the target users, (f) what you might build as far as you know, (g) how you will evaluate its success, and (h) how you will recruit participants. One of the group members should deposit the ZIP file under /d/g/695/$USER on the server and send me email with the SHA1 has (like in warmup #3). Use the subject line “project milestone 3 introduction [ece695cps]”. |
| 3/12 | v0.1 | Create an interface prototype or a detailed mock-up or wireframe and an updated description of your design. The only expectation is that you show evidence that you've started and made some progress beyond the proposal last week. Send your proposal by email with the subject line "project milestone 4 prototype v0.1 [ece695cps]". If appropriate, you can upload the files to the server and email just the path and SHA1 checksum, like before. |
| 3/26 | email check-in | Send an email status report with the subject line "project milestone 5 prototype v0.1 [ece695cps]". |
| 4/7 | v0.5 | At this point, you should have something working and running on a server. Make an appointment to show me your progress and discuss your path forward. In addition, write up the first half of your final report (int). You should introduce your project and why it is needed. Feel free to write as though it is all done now, and just use the <s> tag to cross out anything that is not currently correct. Include at least 15 references to related work in the manner described below. If appropriate, you can upload the files to the server and email just the path and SHA1 checksum, like before. Use the subject line “project milestone 6 code and data [ece695cps]”. |
| 4/12 | email check-in | Send an email status report with the subject line "project milestone 7 prototype v0.1 [ece695cps]". |
| 4/21 | v1.0 | Development and evaluation are now complete. Deposit your code, instruction for running it, evaluation data, and a summary of your accomplishments on the server like before, and send an email with the SHA1 checksum with the subject line “project milestone 8 code and data [ece695cps]”. At this point, the only thing left is to create your project page and video. |
| 4/23 | evaluation data | Send by email with subject line “project milestone 9 evaluation data [ece695cps]” |
| 4/26 | video | Produce a 2- to 5-minute video illustrating your project, following the guidelines below. Submit via the server and email the SHA1 checksum with subject line “project milestone 10 video [ece695cps]” |
| 4/29 | report | Write a report in the form of a web page (≈1500-3000 words) following the guidelines below. Upload to your project web directory on the server and email the SHA1 checksum of just the index.html file with subject line “project milestone 10 video [ece695cps]”. |
Web server
I hope to have a dedicated web server available for all to use in mid- to late-February. That server will provide more flexibility than most other options on campus.
Your evaluation
Evaluation should be performed with the intended worker pool. For example, those who are targeting AMT should run their system on AMT. However, you don't need a large number. In general, projects should be at the proof-of-concept level of quality. As such, your evaluation just needs to be enough to show that your system works on some reasonable inputs.
Project page (final product)
For your final report, you will create a project page (≈1500-3000 words), including a video (≈3-5 minutes) demonstrating and explaining your system.
Content. You should include all of the same information that would be found in an academic paper
- Problem you set out to solve
- Existing solutions and other related work
- Your design (including the process that led you to it)
- Your implementation
- Your evaluation method
- Your results
- Conclusions
- Acknowledgements (including outside help or code not mentioned elsewhere)
Writing. Write in the style and tone of an extended blog post. Start with a summary that explains at a high level what you did, in a way that flows smoothly into the rest of your article. Although this will include all of the same information that would be in an academic paper, it should be much easier to read. Your top priority should be to clearly understood and pique your readers' interest.
Formatting.
Everything should be neatly formatted. Templates are okay if they are
clean and low-key (and acknowledged). Use appropriate spacing between
elements. Embed your video using the <video> tag.
- Resize the window to the smallest size that works for your application. A maximum of 1024 pixels wide is recommended, and narrower is even better.
- Hide/remove any extraneous information from the page. The developer panel in your browser can help with this.
- If you will include the entire browser window, be sure to hide browser buttons for plugins you have installed, etc. In Chrome, just drag the edge of the address bar to the right to hide the buttons.
- If you will not include the entire browser window, then give your
screenshot a bit of frame. Consider something like
<img src="…" alt="…" style="padding:4px; border: solid 1px lightgray">. - If your screenshot will not include the entire page, then try to size the window so that the bottom edge cuts off half a line of text, to make it clear there is more that isn't shown.
References. Include at least 15 references to related work as hyperlinks within the text. For academic articles, use a DOI link (e.g., “http://dx.doi.org/10.1145/368481.368507”) and include the full citation inside the title attribute using a consistent reference format. For non-academic references, try to choose permalinks or main page URLs that are less likely to go away, and include the title and author (if applicable) in the title attribute.
The <a href="http://dx.doi.org/10.1145/368481.368507" title="Donald E. Knuth. 1959. RUNCIBLE—algebraic translation on a limited computer. Commun. ACM 2, 11 (November 1959), 18-21.">Runcible</a> compiler supported algebraic translation on a limited computer.
Video.
Explain your project and give a demo of your system. A narrated screencast is fine. Aim for about 3-5 minutes.
- Length: 2-5 minutes
- Format: MP4/H.264
- Neat, smooth, and understandable
- Legible when viewed at 1024x768 (our classroom projector)
- Include titles, as appropriate, but don't go overboard.
- Before you begin, write out your narration and arrange/size windows. Tip: Setting windows to 1024x768 will ensure that everything is legible.
- Animation is not required and should only be used if it aids communication.
- Avoid gratuitous animation (e.g., flashy transitions).
- System description is not required, but feel free to mention any notable aspects or give a brief outline, especially if it aids communication.
- Do not describe mundane implementation details (e.g., “built using 8,756 lines of Python with the Flask and CrowdLib modules…”).
- Submit via the server and email the SHA1 checksum with subject line “project milestone 9 video [ece695cps]”
Visibility. Project pages and videos will be posted to the course web page unless any team members request that they be omitted or anonymized. I hope these project pages will be something you can refer to as you move along in your endeavors, as well as a resource for future students. However, that is entirely optional.
Grading
Projects will be graded based on the quality of the design (including the process that led you to it), the novelty and boldness (challenge) of the idea, your success at achieving what you set out to do, the quality of your final product (web page and video demo), and the overall effort.