Crowdsourced Aesthetics: A Reflection on Running Online Cartographic User Studies

Authors: P. William Limpisathian*, University of Oregon, Cary Anderson, The Pennsylvania State University
Topics: Cartography, Geographic Information Science and Systems, Quantitative Methods
Keywords: online, user study, methodology, visualization, cartography, MTurk
Session Type: Paper
Day: 4/11/2018
Start / End Time: 8:00 AM / 9:40 AM
Room: Napoleon C3, Sheraton 3rd Floor
Presentation File: No File Uploaded


There continues to be a surge in experiments and user studies conducted on online crowdsourcing and micro-task platforms within the domain of cartography and geovisualization. At the surface, online user studies, through services such as Qualtrics together with Amazon Mechanical Turk (MTurk), seemingly offer researchers dynamic ways to test visual data while enabling flexible access to a large and diverse participant pool at a fraction of the cost associated with traditional in-person experiment methods. Previous research in other domains has highlighted significant cost reduction potentials without detrimental effects on data quality. Yet the process involved in conducting these online, crowdsourced studies has thus far remained inexact and often exploratory. In this paper, We aim to discuss in detail the tasks of structuring the study for online testing; recruiting, managing and paying study participants; and processing and verifying the raw results. Additionally, we will highlight some of the ways geographers and others researchers in related disciplines structure their experiments on these platforms. Lastly, we will reflect on our own prior research that utilized the MTurk platform and discuss the potential benefits, challenges, and pitfalls that cartographers and researchers may face when conducting online visualization user studies on an online micro-task platform.

Abstract Information

This abstract is already part of a session. View the session here.

To access contact information login