Amazon Mechanical Turk

The Amazon Mechanical Turk (MTurk) is a crowdsourcing Internet marketplace that enables computer programmers (known as Requesters) to co-ordinate the use of human intelligence to perform tasks that computers are unable to do yet. It is one of the suites of Amazon Web Services. The Requesters are able to post tasks known as HITs (Human Intelligence Tasks), such as choosing the best among several photographs of a store-front, writing product descriptions, or identifying performers on music CDs. Workers (called Providers in Mechanical Turk's Terms of Service) can then browse among existing tasks and complete them for a monetary payment set by the Requester. To place HITs, the requesting programs use an open Application Programming Interface, or the more limited MTurk Requester site ^[1]. Requestors are restricted to US-based entities. ^[2]

Requesters can ask that Workers fulfill Qualifications before engaging a task, and they can set up a test in order to verify the Qualification. They can also accept or reject the result sent by the Worker, which reflects on the Worker's reputation. Currently, workers can have an address anywhere in the world. Payments for completing tasks can be redeemed on Amazon.com via gift certificate or be later transferred to a Worker's U.S. bank account. Requesters, which are typically corporations, pay 10 percent over the price of successfully completed HITs to Amazon.^[3]

Name

The name Mechanical Turk comes from "The Turk," a chess-playing automaton of the 18th century, which was made by Wolfgang von Kempelen. It toured Europe, beating the likes of Napoleon Bonaparte and Benjamin Franklin. It was later revealed that this "machine" was not an automaton at all, but was in fact a chess master hidden in a special compartment controlling its operations. Likewise, the Mechanical Turk web service allows humans to help the machines of today perform tasks they aren't suited for.

History, HIT types, and user demographics

The service was initially invented for Amazon's in-house use by Peter Cohen, to find duplicates among its web pages describing products.^[3]

MTurk was launched publicly on November 2, 2005 and, as of July 2011, is still in beta. Following its launch, the Mechanical Turk user base grew quickly, in part the result of the Slashdot effect. At that time, there were a huge number of "Human Intelligence Tasks" (HITs) in the system. In early- to mid-November 2005, there were tens of thousands of HITs, all of them uploaded to the system by Amazon itself for some of its internal tasks that required human intelligence. Most of these were related to music CD items. Web traffic grew to a massive amount near the beginning of December.

However, the number of HITs in the system soon decreased, and by December 20, there were less than 100 groups of HITs on the average page load. In January, new types of HITs were set up, such as top three lists ranking for the (now defunct) Amazon Unspun site, and third-party HITs began to appear as well. By April 2006, there were only the occasional batch of 25 HIT groups being offered, and the service had slowed to a crawl.

As of January 2007, there were new HITs being offered of podcast transcribing, rating and image tagging. The transcription HITs, mostly offered by CastingWords,^[4] are still posted regularly as of July 2011. Other common HIT types ask Turkers to write or rewrite sentences, paragraphs, or whole articles. These have rewards ranging from one cent to about $10. HITs that reward people for linking to or commenting on a blog, or friending a person on Facebook are also often encountered, as are surveys.

In March 2007, there were reportedly more than 100,000 workers in over 100 countries.^[3] In 2011, techlist published an interactive map pinpointing the locations of 50,000 of their MTurk workers around the world.^[5]

According to a survey conducted through one MTurk HIT, Turkers are primarily located in the United States^[6] with demographics generally similar to the overall Internet population in the US.^[7]

Missing persons searches

In 2007, the service began to be used to search for prominent missing individuals. It was first suggested during the search for James Kim, but his body was found before any technical progress was made. That summer, computer scientist Jim Gray disappeared on his yacht and Amazon's Werner Vogels, a personal friend, made arrangements for DigitalGlobe, which provides satellite data for Google Maps and Google Earth, to put recent photography of the Farallon Islands on the Mechanical Turk. A front-page story on Digg attracted 12,000 searchers who worked with imaging professionals on the same data. The search was unsuccessful.^[8] In September 2007, a similar arrangement was repeated in the search for aviator Steve Fossett. Satellite data was divided into 85 squared meter sections, and Mechanical Turk users were asked to flag images with "foreign objects" that might be a crash site or other evidence that should be examined more closely. ^[9] The search was also unsuccessful; the crash site was eventually found by hikers about a year later.^[10]

Third-party programming

Programmers have developed various browser extensions and scripts designed to simplify the process of completing HITs. According to the Amazon Web Services Blog, however, Amazon appears to disapprove^[11] of the ones that automate the process 100% and take out the human element. Accounts using so-called automated bots have been banned.

API

Amazon makes available an API to give users another access point to the MTurk system. The MTurk API^[12] lets a programmer submit HITs to MTurk, retrieve completed work, and approve/deny that work. Web sites and Web services can use the API to integrate MTurk work into other web applications, providing users with alternatives to the interface Amazon has built for these functions.

Related systems

Main article: Crowdsourcing

MTurk is comparable in some respects to the now discontinued Google Answers service. However, the Mechanical Turk is a more general marketplace that can potentially help distribute any kind of work tasks all over the World. The Collaborative Human Interpreter (CHI) by Philipp Lenssen also suggested using distributed human intelligence to help computer programs perform tasks that computers cannot do well. MTurk could be used as the execution engine for the CHI.

Criticism

Because HITs are typically simple, repetitive tasks and users are paid often only a few cents to complete them, some have criticized Mechanical Turk as a "virtual sweatshop".^[13] At the same time, workers set their own hours and are not under any obligation to accept any work they do not wish to do. Because workers are paid as contractors rather than employees, requesters do not have to file forms for, nor to pay, payroll taxes, and they avoid laws regarding minimum wage, overtime, and workers compensation. Workers, though, must report their income as self-employment income. In addition, some requesters have taken advantage of workers by having them do the tasks, then rejecting their submission in order to avoid paying. However, at least some workers on Mechanical Turk are people who are middle class and do the work for fun. Quality of the entries can vary from usable to useless.^[14]

Amazon.com does not monitor the service and refers all complaints to the poster of the HIT.

See also

• Crowdsourcing
• turKit
• Ubiquitous Human Computing

References

1. ^ http://requester.mturk.com
2. ^ https://requester.mturk.com/mturk/help?helpPage=help_popUp_whyUnitedStates&isPopUp=true
3. ^ ^{a b c} Artificial Intelligence, With Help From the Humans, The New York Times, 25 March 2007
4. ^ http://turkers.castingwords.com
5. ^ Tamir, Dahn. "50000 Worldwide Mechanical Turk Workers". techlist. http://techlist.com/mturk/global-mturk-worker-map.php. Retrieved 8 November 2011. 
6. ^ Panos Ipeirotis (March 19, 2008). "Mechanical Turk: The Demographics". New York University. http://behind-the-enemy-lines.blogspot.com/2008/03/mechanical-turk-demographics.html. Retrieved 2009-07-30. 
7. ^ Panos Ipeirotis (March 16, 2009). "Turker Demographics vs Internet Demographics". New York University. http://behind-the-enemy-lines.blogspot.com/2009/03/turker-demographics-vs-internet.html. Retrieved 2009-07-30. 
8. ^ Steve Silberman (July 24, 2007). "Inside the High-Tech Search for a Silicon Valley Legend". Wired magazine. http://www.wired.com/techbiz/people/magazine/15-08/ff_jimgray?currentPage=5. Retrieved 2007-09-16. 
9. ^ http://www.avweb.com/avwebflash/news/SteveFossettSearch_AmazonMechanicalTurk_PleaseHelp_196097-1.html
10. ^ Jim Christie (October 1, 2008). "Hikers find Steve Fossett's ID, belongings". Reuters. http://www.reuters.com/article/peopleNews/idUSTRE4907G820081001. Retrieved 2008-11-27. 
11. ^ http://aws.typepad.com/aws/2005/12/amazon_mechanic.html
12. ^ http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=28
13. ^ Harris, Mark (2008-12-21). "Email from America". London: Sunday Times. http://technology.timesonline.co.uk/tol/news/tech_and_web/personal_tech/test_bench/article5370360.ece. 
14. ^ Mieszkowski, Katharine (2006-07-24). "I make $1.45 a week and I love it". Salon.com. http://www.salon.com/tech/feature/2006/07/24/turks/. Retrieved 2008-06-15. ^{[dead link]}

External links

• Official website
• Wired Magazine story about "Crowdsourcing," June 2006
• Business Week article on Mechanical Turk
• New York Times article on Mechanical Turk
• Salon.com article on Mechanical Turk
• Technology Review article on Machanical Turk
• Requester Best Practices Guide