1. Presentation at Nowcasting Symposium, Design/Media Arts Department, UCLA, Los Angeles, October 16-17, 2009 (original title: Cultural Analytics annual report 6/2008-9/2009) Dr. Lev Manovich Director, Software Studies Initiative, Calit2 + UCSD Professor, Visual Arts Department manovich.lev@gmail.com You can capture this lecture using any media and share it. manovich.lev@gmail.com Follow our research: softwarestudies.com Cultural Visualization techniques

2. cultural analytics research @Software Studies Initiative at Calit2/UCSD - key people: Dr. Jeremy Douglass | Postdoctoral Researcher Tara Zepel | PhD student, Art History Sunsern Cheamanunkul | PhD student, Computer Science So Yamaoka | PhD student, Computer Science William Huber | PhD student, Art History for the expanded list of participants, see softwarestudies.com

3. Our research is made possible by the support from: Center for Research in Computing and the Arts (CRCA) California Institute for Telecommunication and Information (Calit2) NEH Office of Digital Humanities National Energy Research Scientific Computing Center Singapore Ministry of Education Bergen University (Norway) UCHRI

4. Software Studies Initiative Collaborators: Yuri Tsivian, Department of Art History, University of Chicago: cinemetrics.lv | film analysis Adele Eisenstein: Digital Formalism project (Department for Theatre, Film and Media Studies (TFM), Vienna University; the Austrian Film Museum; Interactive Media Systems Group, Vienna University of Technology) | film analysis Netherlands Institute for Sound and Vision | television and motion graphics analysis San Diego Museum of Contemporary Art | mapping art for an exhibition Isabel Galhano Rodrigues, University of Porto, Portugal | gesture analysis David Kirsh, Cognitive Science, UCSD | dance video analysis Ph.D. students, Art History, UCSD | art history and visual culture Jim Hollan, Cognitive Science, UCSD | visualization | cultural analytics software Falko Kuester, Structural Engineering, UCSD + Calit2 | visual analytics | cultural analytics software Yoav Freund, Computer Science and Engineering, UCSD | machine learning and machine vision | cultural analytics software Kay OHalloran, Multimodal Analysis Lab, National University of Singapore | Mapping Asian Cultures project Giorgos Cheliotis: Communication and New Media, National University of Singapore | Mapping Asian Cultures project Matthew Fuller: Goldsmiths College, University of London | software studies, game studies

5. cultural analytics = quantitative analysis and visualization of cultural data

6. goals of cultural analytics: - being able to better represent the complexity, diversity, variability, and uniqueness of cultural processes and artifacts - develop techniques to describe the dimensions of cultural artifacts and cultural processes which until now received little or no attention (such as gradual temporal change) - create much more inclusive cultural histories and analysis - ideally taking into account all available cultural objects created in particular cultural area and time period (art history without names) - democratize cultural research by creating open-source tools for cultural analysis and visualization - create interfaces for exploration of cultural data which operate across multiple scales - from details of structure of a particular individual cultural artifact/processes to massive cultural data sets/flows

7. cultural analytics - typical steps: -1) description (i.e, culture into data): a) manual: annotation, tagging b) automatic: software analysis of media; capturing user activity our focus: easy-to-use techniques for automatic description of visual and interactive media 2) optional: statistical data analysis 3) data visualization (reduction, summarization) and data mapping (expansion, outlining, layering) our focus: new visualization + mapping techniques appropriate for interactive exploration of large sets of visual objects 4) interpretation (humanities), or explanation (science), or correlation (social science)

8. visualization of cultural data - visualization types: 1) visualization without doing additional annotation / automatic analysis a) display all objects in a set together organized by exiting metadata (for instance, dates, artist names, etc.) b) sample and re-order (for instance: montage, slice) 2) visualization after doing additional data analysis/annotation c) visualization of newly generated metadata (graph) d) display objects organized by metadata (image graph) examples: d1) 2D sorted view d2) 2D image graph - using single feature for each dimension d3) 2D image graph - using combination of features (PCA, etc.)

9. examples of manual annotations and results of automatic analysis of visual data as tab delimited files (Excel spredsheets)

10. data exploration using currently available software used in science and business - limited because does not show images or video or 3D models or other media demo: Mondrian software

11. our techniques: analytical browsing - interfaces which combine media browsing and graphs to enable visual exploration of cultural data

12. The aim pursued with visual exploration is to give an overview of the data and to allow users to interactively browse through different portions of the data. In this scenario users have no or only vague hypotheses about the data; their aim is to find some. In this sense, visual exploration can be understood as an undirected search for relevant information within the data. To support users in the search process, a high degree of interactivity must be a key feature of visual exploration techniques. Christian Tominski, Event-Based Visualization for User-Centered Visual Analysis, PhD Thesis, Institute for Computer Science, Department of Computer Science and Electrical Engineering, University of Rostock, 2006. Exploration denotes an undirected search for interesting features in a data set. Kreuseler, M., Nocke, T., and Schumann, H. A History Mechanism for Visual Data Mining. In Proceedings of the IEEE Symposium on information Visualization (infovis'04) - Volume 00 (October 10 - 12, 2004). INFOVIS. IEEE Computer Society, Washington, DC, 49-56. 2004. source: www.infovis-wiki.net.

13. Cultural Analytics software running on HIPerSpace (May 2009)

16. our techniques: image graphs - graphs which show the actual media objects (as opposed to only points)

20. a video of cultural analytics software tool running on HIPerSpace (available on Youtube)

21. bringing the same techniques to the desktop: demo: mapping development of Piet Mondrian career with our imageJ tool currently in development: interactive desktop software for image set exploration (Flash/Flex/Flare)

22. recent examples of our image graphs: a complete set of all Time covers (4553 images, 1923-2008) - www.flickr.com/photos/culturevis/ Google logos arranged by type of design modification - www.flickr.com/photos/culturevis/www.flickr.com/photos/culturevis/

23. our techniques: data mapping - highlighting the structure a cultural artifact while displaying this artifact (all or sampled) traditional visualization - reduction, summarization, abstraction data mapping (or supervisualization ) - augmentation, expansion

24. text visualizations www.flickr.com/photos/culturevis/ www.flickr.com/photos/culturevis/ SIGGRAPH 09 media species poster (www.flickr.com/photos/culturevis/

25. our techniques: sampling and re-ordering (re-mapping) demo - representing a shot structure of a feature film using imageJ

26. our techniques: visualization of user activity using automatic software analysis of gameplay video game play timelines (available on softwarestudies.com) SIGGRAPH 09 games poster

27. in progress: analysis and visualization of a set of 48,000 comic book/manga/graphic novels titles = 1,2 million pages = appr. 10 million panels data processing @ NERSC (National Energy Research Scientific Computing Center) using some of the fastest supercomputers in the world all our open-source tools are available from Google code depository