The “Assembly Line” for the Information Age Human-Computer Cooperation for Large-Scale Product Classification Jianfu Chen Computer Science Department, Stony Brook University

Machines Transform Human History

People have always been seeking the optimal way of integrating machine and human labor.

20 th Century Ford Assembly Line Integrates Machine and Human Labor Efficiently

21 st Century – Information Age “Mass Production” of Information

We want to find the optimal ways to integrate machine and human intelligence. NOT all products could be produced fully automatically by machines – assembly line integrated machine & human labor NOT all information can be produced fully automatically by computers – We want to find optimal ways to integrate machine and human intelligence What’s the “Assembly Line” for the Information Age?

A Case Study: Large scale product classification Kindle Fire HD 8.9" 4G LTE Wireless 8.9" HD Display, Dolby Audio, Dual- Band Dual-Antenna Wi-Fi, 4G LTE, 32GB or 64GB Goal: optimally integrate computer and human effort Achieve a lower unit cost for product classification More precisely, optimize the accuracy-cost tradeoff

An “Assembly Line” for Human Computer Cooperation 3Com V.35 cable V.35 cable ( DTE ) - DB-50 (M) - M/34 (V.35) (M) - 10 ft A list of K candidate classes System Accuracy Machine Accuracy Human Accuracy X Cost is Human labor cost, i.e., the salary paid to workers, which is proportional to the working time spent. =

A quick glance at Accuracy-Cost Relation

There is an optimal cost that gives the highest accuracy.

Towards a more realistic analysis of accuracy-cost relationship With the above “assembly line” model, human accuracy and working time are influenced by a set of factors – K – Task difficulty – Expertise I am familiar with office supplies, but not familiar with nuts and bolts. – Cognitive characteristics Careful, smart, quick Independent of the task

Use a probabilistic graphical model to capture the cognitive process of human classification A probabilistic graphical model shows how the above different factors interact with each other, and influence the accuracy and cost. Specifically, we use Bayesian Network, which characterizes the causal relationships of different factors.

Use a Bayesian Network to predict accuracy and cost

Not only visually intuitive, but also formal

Inference and learning

usage of the model Predict the accuracy-cost tradeoff – Given certain budget, what’s the highest accuracy we can achieve? – To achieve certain accuracy, what’s the lowest expected cost? How to charge customers? Optimally assign the workers to the tasks

Related Works time and motion study – Scientific management (Taylorism) Crowdsourcing – Amazon Mechanical Turk – learning worker expertise and accuracy Item Response Theory – Psychometrics IQ test, GRE, GMAT

Conclusion In information age, we need a new “assembly line” to integrate human and machine intelligence. We try to model human accuracy and working time by considering the interactions of a set of relevant factors, using a probabilistic graphical model. We use the model to predict the accuracy-cost tradeoff, decide how to charge customers, and optimally assign tasks to human workers.

Thank you!