1. “Introduction to Media Computation” A New Core Area B Course Mark Guzdial, 18 July 2002

2. Today’s story What’s computation good for –Why you should care –Examples A course in media computation –Yes, still teaching programming How the course will be organized Your feedback!!!

3. What’s computation good for Computer science is the study of recipes Computer scientists study… –How the recipes are written (algorithms, software engineering) –The units used in the recipes (data structures, databases) –What can recipes be written for (systems, intelligent systems, theory) –How well the recipes work (human-computer interfaces)

4. Specialized Recipes Some people specialize in crepes or barbeque Computer scientists can also specialize on special kinds of recipes –Recipes that create pictures, sounds, movies, animations (graphics, computer music) Still others look at emergent properties of computer “recipes” –What happens when lots of recipes talk to one another (networking, non-linear systems)

5. KEY: The COMPUTER does the recipe! Make it as hard, tedious, complex as you want! Crank through a million genomes? No problem! Find one person in a 30,000 campus? Yawn! Process a million dots on the screen or a bazillion sound samples…MEDIA COMPUTATION!

6. Why should you need to study “recipes”? To understand better the recipe-way of thinking –It’s influencing everything, from computational science to bioinformatics –Eventually, it’s going to become part of everyone’s notion of a liberal education To work with and manage computer scientists AND…to communicate! –Writers, marketers, producers communicate through computation

7. Computation for Communication All media are going digital Digital media are manipulated with software You are limited in your communication by what your software allows –What if you want to say something that Microsoft or Adobe or Apple doesn’t let you say?

8. Programming is a communications skill If you want to say something that your tools don’t allow, program it yourself If you want to understand what your tools can or cannot do, you should understand what the programs are doing If you care about preparing media for the Web, for marketing, for print, for broadcast… then it’s worth your while to understand how the media are manipulated. PROGRAMMING KNOWLEDGE = FREEDOM

9. “But PhotoShop is great!” Okay, but so are ProAudio Tools, ImageMagick and the GIMP, and Java and Visual Basic You can learn lots of tools OR You can learn a key set of tools, and a general set of principles and programming skills AND learn new tools easier

10. It’s still computer science For example, all the “under the hood” standard data structures are there still: –Sounds are arrays of samples –Pictures are matrices (two-dimensional arrays) of pixels (x, y, color in red+green+blue) –Movies are arrays of pictures!

11. Example: Opening a Picture fp=pickafile() p=picture(fp) show(p)

12. Example: A “Photoshop- like” filter def decreaseRed(pic): for x in pixels(pic): setRed(x, 0.95 * getRed(x)) Before After (twice)

13. Example: Background Subtraction Let’s say that you want to build a picture of your daughter on the moon. Take a picture of her against the wall, then just the wall.

14. Example Solution For each pixel (dot) in the picture –Get the pixel in the picture –Get the pixel in the frame (without the kid) –Are the colors pretty darn close? Must be where the kid ISN’T, so grab the picture of the moon

16. Example code #Picture with person, background, and newbackground def swapbg(pic1, bg, newbg): for x in range(1,pic1.getWidth()): for y in range(1,pic1.getHeight()): p1px = getPixel(pic1,x,y) bgpx = getPixel(bg,x,y) if (distance(getColor(p1px),getColor(bgpx)) < 15.0): setColor(p1px,getColor(getPixel(newbg,x,y))) return pic1

17. What if we used a different threshold?

18. Why doesn’t it look better? Can you figure out where the light source was? The colors on the shirt and the colors of the wall were awfully similar

19. New Approach: Chromakey! It’s what weather people do Pose in front of a blue screen Swap all “blue” for the background

21. Example Solution def chromakey2(source,bg): for p in pixels(source): if (getRed(p)+getGreen(p) < getBlue(p)): setColor(p,getColor(getPixel(bg,x(p),y(p)))) return source

22. Another way of saying the same thing def chromakey(source,bg): # source should have something in front of blue, bg is the new background for x in range(1,source.getWidth()): for y in range(1,source.getHeight()): p = getPixel(source,x,y) # My definition of blue: If the redness + greenness < blueness if (getRed(p) + getGreen(p) < getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y))) return source

23. Can I do this by masking in Photoshop? Of course! –How do you think Photoshop does it? But you can do it better, differently, faster, and for more kinds of pictures if you know how it works

24. Example: Segmenting and Splicing Sounds Recorded myself reading the first part of the preamble Then used a sound editor to find the sample number where each word ended

25. Let’s keep only the first part #Segmenting # Here's what I got from exploring with the editor # Word Endpoint # We 15730 # the 17407 # People 26726 # of 32131 # the 33413 # United 40052 # States 55510

26. def segmentMain(): # Grab the sound JUST up to "We the people of the United States" fs="/Users/guzdial/mediasources/preamble.wav" s=sound(fs) nfs = "/Users/guzdial/mediasources/sec1silence.wav" #An empty sound ns=sound(nfs) # This is where we'll build the new sound nsi=1 # New sound index, starting from 1 for si in range(1,55510): # Where the samples are in the sound setSampleAt(ns,nsi, getSampleAt(s,si)) nsi = nsi + 1 play(ns) writeSoundTo(ns,"preamble-start.wav")

27. Now, let’s make me say something I didn’t say “We the UNITED People of the UNITED States”

28. # Splicing # Using the preamble piece, making "We the united people" def spliceMain(): fs="/Users/guzdial/mediasources/preamble-start.wav" s=sound(fs) ns=sound(fs) # This is where we'll build the new sound nsi=17408 # New Sound Index starts at just after "We the" in the new sound for si in range(33414,40052): # Where the word "United" is in the sound setSampleAt(ns,nsi, getSampleAt(s,si)) nsi = nsi + 1 for si in range(17408, 26726): # Where the word "People" is in the sound etSampleAt(ns,nsi, getSampleAt(s,si)) nsi = nsi + 1 for index in range(1,1000): #Stick some empty space after that setSampleAt(ns,nsi,0) nsi = nsi + 1 play(ns) #Let's here and return the result return ns

29. Last Example: Titling # # Try titling # def testTitle(): f = "/Users/guzdial/medias ources/barbara.jpg" newpic = title(f,"Copyright 2001 Mark Guzdial") writePictureTo(newpic,"/ Users/guzdial/mediaso urces/t-barbara.jpg") def title(fname,string): p = picture(fname) drawText(p,30,30,string) show(p) return p

30. What if you have 100’s of files? # # Try titling a whole directory # def testTitling(): import os.path os.path.walk("/Users/guzdial/mediasources/kid-in-bg- seq/",titleEach,"") def titleEach(arg,dir,files): for f in files: newpic = title(dir+f,"Copyright 2001 Mark Guzdial") writePictureTo(newpic,dir+"t-"+f)

31. Other Fun Examples “Morphing” one image into another Animations Digital video special effects, e.g., adding an animated figure to a video Creating a synthesizer Creating hip-hop forward-reverse record sounds

32. What is this language? Jython Python is an easy-to-use language that’s aimed at Web and database programming (http://www.python.org)http://www.python.org –It is something you’ll see in want ads –Neat features: Indentation defines blocks Jython (http://www.jython.or) is Python written in Javahttp://www.jython.or –Anything one can do in Java, one can do in Jython

33. How the class will be set up Lecture and optional recitation –3 credit hours Several kinds of assignments: –Pre-quizzes (worksheet like a quiz) given just before a quiz (3) –Homework, collaborative (4-6 per semester) –Labettes (weekly, on-own activities) –Projects: NON-collaborative homework (4 per semester)

34. Not all programming assignments For example, a lot of reading and modifying code at first There will be a good bit of math involved, but high-school level—and very different from CS1321 –For example, computing a sine wave at the right frequency to generate a given sound

35. Rough Syllabus Weeks 1-2: Getting started with programming –How to debug Weeks 3-10: Lots of experience with little/no new CS –Sounds, pictures, movies –Manipulating files –Media transformations, e.g., text (numbers) to graphics Weeks 11-15: Can more CS make this easier? –Objects, functional decomposition, recursion, a little Java –Why do some programs take so long?

36. JES: Jython Environment for Students

37. Planning Right now: Developing the materials, technology, course notes Fall 2002: Planning assignments, First offering: Spring 2003 for 100 students Summer 2003: Train TAs, set up procedures for scaling Full-scale: Fall 2003, two sections of 200/250/300 each

38. For more information… http://coweb.cc.gatech.edu/mediaCo mp-planhttp://coweb.cc.gatech.edu/mediaCo mp-plan guzdial@cc.gatech.edu

39. Feedback? Will this interest students in your major? –If not, how could it be made a better fit? What do I absolutely have to do to make this work?