David Martin
ABSTRACT
Crowdsourcing is a key current topic in CSCW. We build upon findings of a few qualitative studies of crowdworkers. We conducted an ethnomethodological analysis of publicly available content on Turker Nation, a general forum for Amazon Mechanical Turk (AMT) users. Using forum data we provide novel depth and detail on how the Turker Nation members operate as economic actors, working out which Requesters and jobs are worthwhile to them. We show some of the key ways Turker Nation functions as a community and also look further into Turker-Requester relationships from the Turker perspective -- considering practical, emotional and moral aspects. Finally, following Star and Strauss [25] we analyse Turking as a form of invisible work. We do this to illustrate practical and ethical issues relating to working with Turkers and AMT, and to promote design directions to support Turkers and their relationships with Requesters.
- Adda, G., Sagot, B., Fort, K., Mariani, J. Crowdsourcing for Language Resource Development. 5th Language and Technology Conference. (2011).Google Scholar
- Antin, J. and Shaw, A. Social desirability bias and selfreports of motivation: A study of amazon mechanical turk in the us and india. In Proc. of CHI'12, ACM press (2012), 2925--2934. Google ScholarDigital Library
- Bederson, B. B. and Quinn, A. J. Web workers unite! addressing challenges of online laborers. In Proc. CHI EA'11. ACM (2011), 97--106. Google ScholarDigital Library
- Felstiner, A. Working the crowd: employment and labor law in the crowdsourcing industry. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=15 93853 (2011).Google Scholar
- Felstiner, A. The weakness of crowds. Limn, Crowds and Clouds (2), 2013. http://limn.it/the-weakness-ofcrowds/.Google Scholar
- Fort, K., Adda, G. and Bretonnel Cohen, K. Amazon Mechanical Turk: Gold Mine or Coal Mine? Computational Linguisitcs, 37, 2 (2011). Google ScholarDigital Library
- Garfinkel, H., Studies in ethnomethodology. Englewood Cliffs, N.J.: Prentice-Hall, (1967)Google Scholar
- Howe, J. Crowdsourcing: A definition. (2006). http://crowdsourcing.typepad.com/cs/2006/06/crowdso urcing_a.htmlGoogle Scholar
- Hughes, J., Randall, D., Shapiro, D. Faltering from Ethnography to Design. In Proc of CSCW'92. ACM (1992), 115--122 Google ScholarDigital Library
- Ipeirotis, P. Analyzing the Amazon Mechanical Turk marketplace. CeDER Working Papers No. CeDER-1004. (2010) http://hdl.handle.net/2451/29801Google Scholar
- Ipeirotis, P. Demographics of Mechanical Turk. CeDER Working Papers No. CeDER-10-01. (2010) http://hdl.handle.net/2451/29585Google Scholar
- Irani, L. and Silberman, M. S. Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk. In Proc. CHI 2013, ACM Press (2013). Google ScholarDigital Library
- Kaufmann, N., Schulze, T., and Veit, D. More than fun and money. Worker Motivation in Crowdsourcing -- A Study on Mechanical Turk. In Proc. Of AMCIS (2011).Google Scholar
- Kittur,A., Chi, E. H., and Suh, B. Crowdsourcing user studies with Mechanical Turk. In Proc. of the CHI'08. ACM, (2008), 453--456 Google ScholarDigital Library
- Kittur, A., Smus, B. and Kraut, R. E. CrowdForge: Crowdsourcing Complex Work. Technical Report Carnegie Mellon University-HCII-11--100, http://crowdresearch.org/blog/wpcontent/.../rowdForge IST 011.pdfGoogle Scholar
- Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M. and Horton, J. The future of crowd work. In Proc. of CSCW'13, ACM press, 1301--1318. Google ScholarDigital Library
- Kochhar, S., Mazzochi, S., Paritosh, P. The Anatomy of a Large-Scale Human Computation Engine. H OMP'10. A M press (2010). 10--17 Google ScholarDigital Library
- Little, G., Chilton, L. B., Goldman, M. and Miller, R. C. Turkit: tools for iterative tasks on mechanical turk. In KDD-HCOMP'09, 2009. Google ScholarDigital Library
- O'Neill, J. & Martin, D. Text hat In Action. In Proc. Group'03, ACM Press (2003), 40--49. Google ScholarDigital Library
- Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., and Tomlinson, B. Who are the crowdworkers?: shifting demographics in mechanical turk. In Proc. of CHI EA'10, ACM (2010), 2863--2872. Google ScholarDigital Library
- Sacks, H. Lectures on conversation. (2 vols). Gail Jefferson (Ed.). Oxford: Basil Blackwell, (1992).Google Scholar
- Silberman, M. S. What's fair? Rational action and its residuals in an electronic market. Unpublished manuscript, (2010). http://www.scribd.com/doc/86592724/Whats-FairGoogle Scholar
- Silberman, M. S., Irani, L., and Ross, J. Ethics and Tactics of Professional Crowdwork. ACM XRDS, 17, 2 (2010) Google ScholarDigital Library
- Silberman, M., Ross, J., Irani, L. and Tomlinson, B. Sellers' problems in human computation markets. In Proc. of HCOMP'10. ACM Press (2010), 18--21. Google ScholarDigital Library
- Star, S.L. and Strauss, A.L. Layers of silence, arenas of voice: The ecology of visible and invisible work'. CSCW Journal, 8: 9--30. (1998) Google ScholarDigital Library
- Suchman, L. Plans and situated action: the problem of human-machine communication. Cambridge: Cambridge University Press (1987) Google ScholarDigital Library
- Watson, R., Analysing Practical and Professional Texts; A Naturalistic Approach. Farnham, Surrey: Ashgate, (2009).Google Scholar
- Worstall, T. On The New York Times Stupidity over Amazon's Mechanical Turk. Forbes. http://www.forbes.com/sites/timworstall/2013/03/19/on-the-new-york-times-stupidity-over-amazonsmechanical-turk/ The columns on the last page should be of approximately equal length. Remove these two lines from your final version.Google Scholar
Index Terms
- Being a turker
Recommendations
A Community Rather Than A Union: Understanding Self-Organization Phenomenon on MTurk and How It Impacts Turkers and Requesters
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing SystemsThis paper aims to understand the self-organization phenomenon among the workers of Amazon Mechanical Turk (MTurk), a well-known crowdsourcing platform. Specifically, we explored 1) why MTurk workers self-organize into online communities (Turker ...
Turker Tales: Integrating Tangential Play into Crowd Work
DIS '19: Proceedings of the 2019 on Designing Interactive Systems ConferenceWhile past work has admirably supported crowd workers in improving their work performance, we argue that there is also value in designing for enjoyment untied from work outcomes--- what we call "tangential play.'' To this end, we present Turker Tales, a ...
Efficient Crowd Exploration of Large Networks: The Case of Causal Attribution
Accurately and efficiently crowdsourcing complex, open-ended tasks can be difficult, as crowd participants tend to favor short, repetitive "microtasks". We study the crowdsourcing of large networks where the crowd provides the network topology via ...
Comments