Sofia Eleni Spatharioti, & Seth Cooper. (2017). On Variety, Complexity, and Engagement in Crowdsourced Disaster Response Tasks. In eds Aurélie Montarnal Matthieu Lauras Chihab Hanachi F. B. Tina Comes (Ed.), Proceedings of the 14th International Conference on Information Systems for Crisis Response And Management (pp. 489–498). Albi, France: Iscram.
Abstract: Crowdsourcing is used to enlist workers as a resource for a variety of applications, including disaster response. However, simple tasks such as image labeling often feel monotonous and lead to worker disengagement. This provides a challenge for designing successful crowdsourcing systems. Existing research in the design of work indicates that task variety is a key factor in worker motivation. Therefore, we asked Amazon Mechanical Turk workers to complete a series of disaster response related subtasks, consisting of either image labeling or locating photographed areas on a map. We varied the frequency at which workers encountered the dierent subtask types, and found that switching subtask type at dierent frequencies impacted measures of worker engagement. This indicates that a certain amount of variety in subtasks may engage crowdsourcing workers better than uniform subtask types.
|
Sofia Eleni Spatharioti, Rebecca Govoni, Jennifer S. Carrera, Sara Wylie, & Seth Cooper. (2017). A Required Work Payment Scheme for Crowdsourced Disaster Response: Worker Performance and Motivations. In eds Aurélie Montarnal Matthieu Lauras Chihab Hanachi F. B. Tina Comes (Ed.), Proceedings of the 14th International Conference on Iformation Systems for Crisis Response And Management (pp. 475–488). Albi, France: Iscram.
Abstract: Crowdsourcing is an increasingly popular approach for processing data in response to disasters. While volunteer crowdsourcing may suÿce for high-profile disasters, paid crowdsourcing may be necessary to recruit workers for less prominent events. Thus, understanding the impact of payment schemes on worker behavior and motivation may improve outcomes. In this work, we presented workers recruited from Amazon Mechanical Turk with a disaster response task in which they could provide a variable number of image ratings. We paid workers a fixed amount to provide a minimum number of image ratings, allowing them to voluntarily provide more if desired; this allowed us to examine the impact of dierent amounts of required work. We found that requiring no ratings resulted in workers voluntary completing more work, and being more likely to indicate motivation related to interest on a post survey, than when small numbers of ratings were required. This is consistent with the motivational crowding-out eect, even in paid crowdsourcing. We additionally found that providing feedback on progress positively impacted the amount of work done.
|
Sofia Eleni Spatharioti, Sara Wylie, & Seth Cooper. (2018). Does Flight Path Context Matter? Impact on Worker Performance in Crowdsourced Aerial Imagery Analysis. In Kees Boersma, & Brian Tomaszeski (Eds.), ISCRAM 2018 Conference Proceedings – 15th International Conference on Information Systems for Crisis Response and Management (pp. 621–628). Rochester, NY (USA): Rochester Institute of Technology.
Abstract: Natural disasters result in billions of dollars in damages annually and communities left struggling with the difficult task of response and recovery. To this end, small private aircraft and drones have been deployed to gather images along flight paths over the affected areas, for analyzing aerial photography through crowdsourcing. However, due to the volume of raw data, the context and order of these images is often lost when reaching workers. In this work, we explored the effect of contextualizing a labeling task on Amazon Mechanical Turk, by serving workers images in the order they were collected on the flight and showing them the location of the current image on a map. We did not find a negative impact from the loss of contextual information, and found map context had a negative impact on worker performance. This may indicate that ordering images based on other criteria may be more effective.
|
Sofia Eleni Spatharioti, Sara Wylie, & Seth Cooper. (2018). Identifying and Assessing Points of Interest through Crowdsourced Image Analysis. In Kees Boersma, & Brian Tomaszeski (Eds.), ISCRAM 2018 Conference Proceedings – 15th International Conference on Information Systems for Crisis Response and Management (pp. 1123–1125). Rochester, NY (USA): Rochester Institute of Technology.
Abstract: During a natural disaster, major damages to critical structures such as bridges or power lines can severely disrupt community functions for long periods of time, making the task of swiftly identifying this type of damage vital for response and recovery. However, survey flight paths are often designed with a main focus of complete and quick coverage of affected areas through aerial photography, which is then assigned to volunteers to aid in damage report and labeling. We designed a crowdsourcing interface that focuses on locating points of interest and assessing damage using images from survey flights. We tested our design using a disaster and a non-disaster application by recruiting volunteers on Amazon Mechanical Turk. We found that the type of structure may cause difficulties for crowd workers in providing accurate assessments and that designing flights to also target structures may provide higher quality imagery for this type of task.
|
