Workshops and Datathon

Volunteer Science was built to be an online laboratory, offering facilities for researchers to create online experiments and for subjects to participate in these experiments. In this session, we provide an overview of our vision for the online laboratory and the battery of experiments we replicated to validate our approach. We then walk participants through the process of starting a research team and creating, testing, and running experiments with volunteers and workers on Mechanical Turk.

For decades, physical behavioral labs have been the primary method for conducting controlled experiments of human behavior. However, the Internet now enables software-based "virtual labs" using online participants, extending the scope of experimental methods and allowing for studies of increasing complexity, size, and scope. Open-source, software-based experiments also allow for faster iteration and improved reproducibility of experimental research.

We will discuss the design and features of an open-source platform for virtual lab experiments aimed at making it easier to build and conduct large-scale, real-time experiments for studying collective behavior. We will review several different studies conducted successfully so far, including a complex collective intelligence task and an experiment on cooperation over a month of real time. We will show how the platform uses Meteor, a real-time web framework, to allow for simple programming of simultaneous interaction, live monitoring of participants, and digital one-way mirrors for the experimenter.

Although our platform addresses many of the logistical challenges in studying groups online, we will also discuss various additional issues to consider in designing such experiments. We highlight how open-source software speeds iteration and facilitates reproducibility by allowing researchers to share not only data, but entire experiment protocols. We will conclude with a brainstorming session on future directions for virtual lab research and the types of novel behavioral questions that it enables (computational) social scientists to answer.

From recommending what to buy, which movies to watch, to selecting the news to read, people to follow and jobs to apply for, online systems have become an important part of our daily lives. A natural question to ask is how these socio-technical systems impact our behavior. However, because of the intricate interplay between the outputs of these systems and people's actions, identifying their impact on people's behavior is non-trivial.

Fortunately, there is a rich body of work on causal inference that we can build on. In the first part of the tutorial, I will show the value of counterfactual reasoning for studying socio-technical systems, by demonstrating how predictive modeling based on correlations can be counterproductive. Then, we will discuss different approaches to causal inference, including randomized experiments, natural experiments such as instrumental variables and regression discontinuities, and observational methods such as stratification and matching. Throughout, we will try to make connections with graphical models, machine learning and past work in the social sciences.

The second half will be more hands-on. We will work through a practical example of estimating the causal impact of a recommender system, starting from simple to more complex methods. The goal of the practical exercise will be to appreciate the pitfalls in different approaches to causal reasoning and take away best practices for doing causal inference with messy, real-world data.

Those attending this workshop should have R, the R library dplyr, and Git installed on their laptops and should clone this repository: https://github.com/amit-sharma/causal-inference-tutorial/

Web-based visualization is gaining popularity thanks to its wide availability and ease of use. In this workshop you will discover the basics of D3.js, a Javascript library that helps create powerful visualizations on the web. D3 lets you combine data from multiple sources and gives you full control on the final look of your data. We will go through the steps needed to set up a basic development environment, create a simple visualization and make it accessible to other people online. All you need to follow along are your laptop with your favorite browser and text editor installed.