1. ACM SIGCSE 2003: Multimedia Construction Projects Mark Guzdial College of Computing Georgia Institute of Technology guzdial@cc.gatech.edu http://www.cc.gatech.edu/~mark.guzdial

2. Plan 7-7:15: Introduction  What’s on your CD  Why media computation 7:15-7:45: Picture manipulations in Python 7:45-8:15: Sound manipulations in Python 8:15-8:30: Break 8:30-8:45: How to do this in Java and Squeak 8:45-9:45: You Play 9:45-10: Wrap-up

3. What’s on your CD Materials from our Introduction to Media Computation course  Jython Environment for Students (JES)  Book  Course slides  MediaTools: Squeak-based media exploration tools Material for this workshop  Java and Squeak API slides  Java source code  Squeak computer music essays

4. Computer science is more important than Calculus In 1961, Alan Perlis argued that computer science is more important in a liberal education than calculus  Explicitly, he argued that all students should learn to program. Calculus is about rates, and that’s important to many. Computer science is about process, which is important to everyone

5. How close are we to being able to teach everyone CS? Not very  CS1 is one of the most despised courses for non-majors At many departments, CS retention rates are lower than the rest of campus  At Georgia Tech: 65% for 1995 cohort, vs. 73% for Engineeering Drop-out rates near 50% at many institutions Female enrollment in CS has been dropping nationally

6. Why? Several recent studies and books claim that CS instruction tends to dissuade anyone but white males  “Tedious,” “taught without application relevance,” “boring,” “lacking creativity,” “asocial”

7. The potential of computing How can we realize the potential impact of computing if future professionals in other disciplines despise computer science? To realize that potential, we need practitioners in other disciplines to understand computing.  At least some of those practitioners need to understand it as a creative, exciting venture—as we do!

8. The best uses for computing technologies will come from others Thomas Edison vs. D.W. Griffith If we want computing technologies to become useful, they have to get out of our hands. It can’t be just through applications  That presumes that the current technologists know how everyone else wants to do things and should do things  Suggestion: D.W. Griffith knew things that Edison didn’t.

9. An attempt at relevant, creative computing: Introduction to Media Computation A course for non-CS and non-Engineering majors  International Affairs, Literature, Public Policy, Architecture, Management, Biology, etc. 120 students this semester, planning 400-600 in the Fall  2/3 female in this semester’s CS1315 Focus: Learning programming within the context of media manipulation and creation

10. Motivating the Computing As professionals, these students will often the use the computer as a communications medium. All media are going digital, and digital media are manipulated with software. Knowing how to program, then, is a communications skill.

11. Python as the programming language Huge and contentious issue Use in commercial contexts legitimatizes the choice  Industrial Light & Magic, Google, Nextel, etc. Minimal syntax Looks like other programming languages  Potential for knowledge transfer for students  “Executable pseudocode” for this workshop Actually using Jython (http://www.jython.org) for Java class librarieshttp://www.jython.org

12. JES - Jython Environment for Students

13. Simple Python in JES >>> print 34 + 56 90 >>> print 34.1/46.5 0.7333333333333334 >>> print 22 * 33 726 >>> print 14 - 15 >>> print "Hello" Hello >>> print "Hello" + "Mark" HelloMark

14. Command Area Editing Up/down arrows walk through command history You can edit the line at the bottom  Just put the cursor at the end of the line before hitting Return/Enter.

15. Basic Picture Functions makePicture(filename) creates and returns a picture object, from the JPEG file at the filename show(picture) displays a picture in a window We’ll learn functions for manipulating pictures later, like getColor, setColor, and repaint

16. Basic Sound Functions makeSound(filename) creates and returns a sound object, from the WAV file at the filename play(sound) makes the sound play (but doesn’t wait until it’s done) blockingPlay(sound) waits for the sound to finish We’ll learn more later like getSample and setSample

17. Demonstrating simple JES >>> print pickAFile() /Users/guzdial/mediasources/barbara.jpg >>> print makePicture(pickAFile()) Picture, filename /Users/guzdial/mediasources/barbara.jpg height 294 width 222 >>> print pickAFile() /Users/guzdial/mediasources/hello.wav >>> print makeSound(pickAFile()) Sound of length 54757 >>> print play(makeSound(pickAFile())) None >>> myfilename = pickAFile() >>> print myfilename /Users/guzdial/mediasources/barbara.jpg >>> mypicture = makePicture(myfilename) >>> print mypicture Picture, filename /Users/guzdial/mediasources/barbara.jpg height 294 width 222 >>> show(mypicture)

18. Writing a recipe: Making our own functions To make a function, use the command def Then, the name of the function, and the names of the input values between parentheses (“(input1)”) End the line with a colon (“:”) The body of the recipe is indented (Hint: Use two spaces) Your function does NOT exist for JES until you load it

19. Functions for picking and playing/showing files def pickAndShow(): myfile = pickAFile() mypict = makePicture(myfile) show(mypict) def pickAndPlay(): myfile = pickAFile() mysound = makeSound(myfile) play(mysound)

20. Functions that take input def playNamed(myfile): mysound = makeSound(myfile) play(mysound) def showNamed(myfile): mypict = makePicture(myfile) show(mypict)

21. CS1315: Introduction to Media Computation Picture encoding and manipulation

22. Pictures and Pixels A picture is a matrix of pixels  A picture has two dimensions: Width and Height Pixels are picture elements  Each pixel object knows its color using RGB encoding  It also knows where it is in its picture

23. RGB In RGB, each color has three component colors:  Amount of redness  Amount of greenness  Amount of blueness Each does appear as a separate dot on most devices, but our eye blends them. In most computer-based models of RGB, a single byte (8 bits) is used for each  So a complete RGB color is 24 bits, 8 bits of each

24. Encoding RGB Each component color (red, green, and blue) is encoded as a single byte Colors go from (0,0,0) to (255,255,255)  If all three components are the same, the color is in greyscale (50,50,50) at (2,2)  (0,0,0) (at position (1,2) in example) is black  (255,255,255) is white

25. Manipulating pixels >>> pixel=getPixel(picture,1,1) >>> print pixel Pixel, color=color r=168 g=131 b=105 >>> pixels=getPixels(picture) >>> print pixels[0] Pixel, color=color r=168 g=131 b=105 getPixel(picture,x,y) gets a single pixel. getPixels(picture) gets all of them in an array. (Square brackets is a standard array reference notation—which we’ll generally not use.)

26. What can we do with a pixel? getRed, getGreen, and getBlue are functions that take a pixel as input and return a value between 0 and 255 setRed, setGreen, and setBlue are functions that take a pixel as input and a value between 0 and 255

27. We can also get, set, and make Colors getColor takes a pixel as input and returns a Color object with the color at that pixel setColor takes a pixel as input and a Color, then sets the pixel to that color makeColor takes red, green, and blue values (in that order) between 0 and 255, and returns a Color object pickAColor lets you use a color chooser and returns the chosen color We also have functions that can makeLighter and makeDarker an input color

28. We can change pixels directly… >>> file="/Users/guzdial/mediasources/barbara.jpg" >>> pict=makePicture(file) >>> show(pict) >>> setColor(getPixel(pict,10,100),yellow) >>> setColor(getPixel(pict,11,100),yellow) >>> setColor(getPixel(pict,12,100),yellow) >>> setColor(getPixel(pict,13,100),yellow) >>> repaint(pict) But that’s really dull and boring… That’s the subject of the next lecture

29. If you make something you like… writePictureTo(picture,”filename”) Writes the picture out as a JPEG  Be sure to end your filename as “.jpg”! If you don’t specify a full path, will be saved in the same directory as JES.

30. How do you find out what RGB values you have? And where? Use the MediaTools!

31. Use a loop! Our first picture recipe def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5) Used like this: >>> file="/Users/guzdial/mediasources/barbara.jpg" >>> picture=makePicture(file) >>> show(picture) >>> decreaseRed(picture) >>> repaint(picture)

32. getPixels returns a sequence of pixels Each pixel knows its color and its original picture Change the pixel, you change the picture So the loop below assigns the index variable p to each pixel in the picture picture, one at a time. def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5)

33. Let’s walk that through slowly… def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5) Here we get a picture object in as input and call it picture picture

34. Now, get the pixels def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5) We get all the pixels from the picture, then make p be the name of each one one at a time picture Pixel, color r=168 g=131 b=105 Pixel, color r=160 g=131 b=105 Pixel, color r=168 g=132 b=106 … p getPixels()

35. Get the red value from pixel def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5) We get the red value of pixel p and name it value picture Pixel, color r=168 g=131 b=105 Pixel, color r=168 g=132 b=106 … p value = 168 Pixel, color r=160 g=131 b=105

36. Now change the pixel def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5) Set the red value of pixel p to 0.5 (50%) of value picture Pixel, color r=84 g=131 b=105 Pixel, color r=168 g=132 b=106 … p value = 168 Pixel, color r=160 g=131 b=105

37. Then move on to the next pixel def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5) Move on to the next pixel and name it p picture Pixel, color r=84 g=131 b=105 Pixel, color r=168 g=132 b=106 … p value = 168 Pixel, color r=160 g=131 b=105

38. Get its red value def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5) Change value to the new red value at the new p picture Pixel, color r=84 g=131 b=105 Pixel, color r=168 g=131 b=105 Pixel, color r=168 g=132 b=106 … value = 160 p Get its red value def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5) Change value to the new red value at the new p picture Pixel, color r=84 g=131 b=105 Pixel, color r=168 g=132 b=106 … p Pixel, color r=160 g=131 b=105

39. And change this red value def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5) Change the red value at pixel p to 50% of value picture Pixel, color r=84 g=131 b=105 Pixel, color r=80 g=131 b=105 Pixel, color r=168 g=132 b=106 … value = 160 p

40. And eventually, we do all pixels We go from this…to this!

41. Increasing Red def increaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*1.2) What happened here?!? Remember that the limit for redness is 255. If you go beyond 255, all kinds of weird things can happen: Wrap around

42. Clearing Blue def clearBlue(picture): for p in getPixels(picture): setBlue(p,0)

43. Combining into a sunset function How do we turn this beach scene into a sunset? What happens at sunset?  At first, I tried increasing the red, but that made things like red specks in the sand REALLY prominent. That can’t be how it really works  New Theory: As the sun sets, less blue and green is visible, which makes things look more red.

44. A Sunset-generation Function def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.7) value=getGreen(p) setGreen(p,value*0.7)

45. Creating a negative Let’s think it through  R,G,B go from 0 to 255  Let’s say Red is 10. That’s very light red. What’s the opposite? LOTS of Red!  The negative of that would be 245: 255-10 So, for each pixel, if we negate each color component in creating a new color, we negate the whole picture.

46. Recipe for creating a negative def negative(picture): for px in getPixels(picture): red=getRed(px) green=getGreen(px) blue=getBlue(px) negColor=makeColor( 255-red, 255-green, 255-blue) setColor(px,negColor)

47. Converting to greyscale We know that if red=green=blue, we get grey  But what value do we set all three to? What we need is a value representing the darkness of the color, the luminance There are lots of ways of getting it, but one way that works reasonably well is dirt simple—simply take the average:

48. Converting to greyscale def greyScale(picture): for p in getPixels(picture): intensity = (getRed(p)+getGreen(p)+getBlue(p))/3 setColor(p,makeColor(intensity,intensity,intensity))

49. But that’s not really the best greyscale In reality, we don’t perceive red, green, and blue as equal in their amount of luminance: How bright (or non-bright) something is.  We tend to see blue as “darker” and red as “brighter”  Even if, physically, the same amount of light is coming off of each Photoshop’s greyscale is very nice: Very similar to the way that our eye sees it  B&W TV’s are also pretty good

50. Building a better greyscale We’ll weight red, green, and blue based on how light we perceive them to be, based on laboratory experiments. def greyScaleNew(picture): for px in getPixels(picture): newRed = getRed(px) * 0.299 newGreen = getGreen(px) * 0.587 newBlue = getBlue(px) * 0.114 luminance = newRed+newGreen+newBlue setColor(px,makeColor(luminance,luminance,luminance))

51. Comparing the two greyscales: Average on left, weighted on right

52. Let’s try making Barbara a redhead! We could just try increasing the redness, but as we’ve seen, that has problems.  Overriding some red spots  And that’s more than just her hair If only we could increase the redness only of the brown areas of Barb’s head…

53. Making Barb a redhead def turnRed(): brown = makeColor(57,16,8) file = r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\barbara.jpg" picture=makePicture(file) for px in getPixels(picture): color = getColor(px) if distance(color,brown)<50.0: redness=getRed(px)*1.5 setRed(px,redness) show(picture) return(picture) Original:

54. Tuning our color replacement If you want to get more of Barb’s hair, just increasing the threshold doesn’t work  Wood behind becomes within the threshold value How could we do it better?  Lower our threshold, but then miss some of the hair  Work only within a range…

55. Introducing the function range Range returns a sequence between its first two inputs, possibly using a third input as the increment >>> print range(1,4) [1, 2, 3] >>> print range(-1,3) [-1, 0, 1, 2] >>> print range(1,10,2) [1, 3, 5, 7, 9]

56. Replacing colors in a range def turnRedInRange(): brown = makeColor(57,16,8) file=r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\barbara.jpg" picture=makePicture(file) for x in range(70,168): for y in range(56,190): px=getPixel(picture,x,y) color = getColor(px) if distance(color,brown)<50.0: redness=getRed(px)*1.5 setRed(px,redness) show(picture) return(picture) Get the range using MediaTools

57. Mirroring Imagine a mirror horizontally across the picture, or vertically What would we see? How do generate that digitally?  We simply copy the colors of pixels from one place to another

58. Mirroring a picture Slicing a picture down the middle and sticking a mirror on the slice Do it by using a loop to measure a difference  The index variable is actually measuring distance from the mirrorpoint Then reference to either side of the mirror point using the difference

59. Recipe for mirroring def mirrorVertical(source): mirrorpoint = int(getWidth(source)/2) for y in range(1,getHeight(source)): for x in range(1,mirrorpoint): p = getPixel(source, x+mirrorpoint,y) p2 = getPixel(source, mirrorpoint-x,y) c = getColor(p2) setColor(p,c)

60. Can we do it with a horizontal mirror? def mirrorHorizontal(source): mirrorpoint = int(getHeight(source)/2) for y in range(1,mirrorpoint): for x in range(1,getWidth(source)): p = getPixel(source,x,y+mirrorpoint) p2 = getPixel(source,x,mirrorpoint-y) setColor(p,getColor(p2))

61. Of course!

62. What if we wanted to copy bottom to top? Very simple: Swap the p and p2 in the bottom line  Copy from p to p2, instead of from p2 to p def mirrorHorizontal(source): mirrorpoint = int(getHeight(source)/2) for y in range(1,mirrorpoint): for x in range(1,getWidth(source)): p = getPixel(source,x,y+mirrorpoint) p2 = getPixel(source,x,mirrorpoint-y) setColor(p2,getColor(p))

63. Messing with Santa some more

64. Copying pixels In general, what we want to do is to keep track of a sourceX and sourceY, and a targetX and targetY.  We increment (add to them) in pairs sourceX and targetX get incremented together sourceY and targetY get incremented together  The tricky parts are: Setting values inside the body of loops Incrementing at the bottom of loops

65. Copying Barb to a canvas def copyBarb(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying targetX = 1 for sourceX in range(1,getWidth(barb)): targetY = 1 for sourceY in range(1,getHeight(barb)): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetX,targetY), color) targetY = targetY + 1 targetX = targetX + 1 show(barb) show(canvas) return canvas

66. Transformation = Small changes in copying Making relatively small changes in this basic copying program can make a variety of transformations.  Change the targetX and targetY, and you copy wherever you want  Cropping: Change the sourceX and sourceY range, and you copy only part of the program.  Rotating: Swap targetX and targetY, and you end up copying sideways  Scaling: Change the increment on sourceX and sourceY, and you either grow or shrink the image.

67. Copying into the middle of the canvas def copyBarbMidway(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying targetX = 100 for sourceX in range(1,getWidth(barb)): targetY = 100 for sourceY in range(1,getHeight(barb)): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetX,targetY), color) targetY = targetY + 1 targetX = targetX + 1 show(barb) show(canvas) return canvas

68. Rotating the copy def copyBarbSideways(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying targetX = 1 for sourceX in range(1,getWidth(barb)): targetY = 1 for sourceY in range(1,getHeight(barb)): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetY,targetX), color) targetY = targetY + 1 targetX = targetX + 1 show(barb) show(canvas) return canvas

69. Cropping: Just the face def copyBarbsFace(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying targetX = 100 for sourceX in range(45,200): targetY = 100 for sourceY in range(25,200): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetX,targetY), color) targetY = targetY + 1 targetX = targetX + 1 show(barb) show(canvas) return canvas

70. Scaling the picture down def copyBarbsFaceSmaller(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)/2)): sourceY = 25 for targetY in range(100,100+((200-25)/2)): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 show(barb) show(canvas) return canvas

71. Scaling the picture up def copyBarbsFaceLarger(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)*2)): sourceY = 25 for targetY in range(100,100+((200-25)*2)): color = getColor(getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 0.5 sourceX = sourceX + 0.5 show(barb) show(canvas) return canvas

72. Background subtraction Let’s say that you have a picture of someone, and a picture of the same place (same background) without the someone there, could you subtract out the background and leave the picture of the person? Maybe even change the background? Let’s take that as our problem!

73. Person (Katie) and Background

74. Background Subtraction Code def swapbg(person, bg, newbg): for x in range(1,getWidth(person)): for y in range(1,getHeight(person)): personPixel = getPixel(person,x,y) bgpx = getPixel(bg,x,y) personColor= getColor(personPixel) bgColor = getColor(bgpx) if distance(personColor,bgColor) < 10: bgcolor = getColor(getPixel(newbg,x,y)) setColor(personPixel, bgcolor)

75. Putting Katie in a Jungle

76. But why isn’t it alot better? We’ve got places where we got pixels swapped that we didn’t want to swap  See Katie’s shirt stripes We’ve got places where we want pixels swapped, but didn’t get them swapped  See where Katie made a shadow

77. Another way: Chromakey Have a background of a known color  Some color that won’t be on the person you want to mask out  Pure green or pure blue is most often used  I used my son’s blue bedsheet This is how the weather people seem to be in front of a map— they’re actually in front of a blue sheet.

78. Chromakey recipe def chromakey(source,bg): # source should have something in front of blue, bg is the new background for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of blue: If the redness + greenness < blueness if (getRed(p) + getGreen(p) < getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y)))

79. Can also do this with getPixels() def chromakey2(source,bg): # source should have something in front of blue, bg is the new background for p in getPixels(source): # My definition of blue: If the redness + greenness < blueness if (getRed(p) + getGreen(p) < getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,getX(p),getY(p))))

80. Example results

81. Just trying the obvious thing for Red def chromakey2(source,bg): # source should have something in front of red, bg is the new background for p in getPixels(source): if getRed(p) > (getGreen(p) + getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,getX(p),getY(p))))

82. Doesn’t always work as you expect

83. Let’s try that with green def chromakeyGreen(source,bg): # source should have something in front of green, bg is the new background for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of green: If the greenness > redness + blueness if getGreen(p) > getBlue(p) + getRed(p): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y)))

84. The same definition of green doesn’t work

85. Tweaking Chromakey def chromakeyGreen(source,bg): # source should have something in front of green, bg is the new background for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of green: If the greenness > redness and blueness if getGreen(p) > getBlue(p) and getGreen(p) > getRed(p): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y)))

86. That looks better

87. New functions for drawing on pictures addText(pict,x,y,string) puts the string starting at position (x,y) in the picture addLine(picture,x1,y1,x2,y2) draws a line from position (x1,y1) to (x2,y2) addRect(pict,x1,y1,w,h) draws a black rectangle (unfilled) with the upper left hand corner of (x1,y1) and a width of w and height of h addRectFilled(pict,x1,y1,w,h,color) draws a rectangle filled with the color of your choice with the upper left hand corner of (x1,y1) and a width of w and height of h

88. Making it easier to deal with the media folder >>> setMediaFolder() New media folder: /Users/guzdial/mediasources/ >>> print getMediaPath("barbara.jpg") /Users/guzdial/mediasources/barbara.jpg >>> print getMediaPath("sec1silence.wav") /Users/guzdial/mediasources/sec1silence.wav

89. Example picture def littlepicture(): canvas=makePicture(getMediaPath("640x480.jpg")) addText(canvas,10,50,"This is not a picture") addLine(canvas,10,20,300,50) addRectFilled(canvas,0,200,300,500,yellow) addRect(canvas,10,210,290,490) return canvas

90. CS1315: Introduction to Media Computation Sound Encoding and Manipulation

91. How sound works: Acoustics, the physics of sound Sounds are waves of air pressure  Sound comes in cycles  The frequency of a wave is the number of cycles per second (cps), or Hertz (Complex sounds have more than one frequency in them.)  The amplitude is the maximum height of the wave

92. Volume and pitch: Psychoacoustics, the psychology of sound Our perception of volume is related (logarithmically) to changes in amplitude  If the amplitude doubles, it’s about a 3 decibel (dB) change Our perception of pitch is related (logarithmically) to changes in frequency  Higher frequencies are perceived as higher pitches  We can hear between 5 Hz and 20,000 Hz (20 kHz)  A above middle C is 440 Hz

93. “Logarithmically?” It’s strange, but our hearing works on ratios not differences, e.g., for pitch.  We hear the difference between 200 Hz and 400 Hz, as the same as 500 Hz and 1000 Hz  Similarly, 200 Hz to 600 Hz, and 1000 Hz to 3000 Hz Intensity (volume) is measured as watts per meter squared  A change from 0.1W/m 2 to 0.01 W/m 2, sounds the same to us as 0.001W/m 2 to 0.0001W/m 2

94. Decibel is a logarithmic measure A decibel is a ratio between two intensities: 10 * log 10 (I 1 /I 2 )  As an absolute measure, it’s in comparison to threshold of audibility  0 dB can’t be heard.  Normal speech is 60 dB.  A shout is about 80 dB

95. Digitizing Sound: How do we get that into numbers? Remember in calculus, estimating the curve by creating rectangles? We can do the same to estimate the sound curve  Analog-to-digital conversion (ADC) will give us the amplitude at an instant as a number: a sample  How many samples do we need?

96. Nyquist Theorem We need twice as many samples as the maximum frequency in order to represent (and recreate, later) the original sound. The number of samples recorded per second is the sampling rate  If we capture 8000 samples per second, the highest frequency we can capture is 4000 Hz That’s how phones work  If we capture more than 44,000 samples per second, we capture everything that we can hear (max 22,000 Hz) CD quality is 44,100 samples per second

97. Digitizing sound in the computer Each sample is stored as a number (two bytes) What’s the range of available combinations?  16 bits, 2 16 = 65,536  But we want both positive and negative values To indicate compressions and rarefactions.  What if we use one bit to indicate positive (0) or negative (1)?  That leaves us with 15 bits  15 bits, 2 15 = 32,768  One of those combinations will stand for zero We’ll use a “positive” one, so that’s one less pattern for positives Each sample can be between -32,768 and 32,767

98. Working with sounds We’ll use pickAFile and makeSound as we have before.  But now we want.wav files We’ll use getSamples to get all the sample objects out of a sound We can also get the value at any index with getSampleValueAt Sounds also know their length (getLength) and their sampling rate (getSamplingRate) Can save sounds with writeSoundTo(sound,”file.wav”)

99. Demonstrating Working with Sound in JES >>> filename=pickAFile() >>> print filename /Users/guzdial/mediasources/preamble.wav >>> sound=makeSound(filename) >>> print sound Sound of length 421109 >>> samples=getSamples(sound) >>> print samples Samples, length 421109 >>> print getSampleValueAt(sound,1) 36 >>> print getSampleValueAt(sound,2) 29

100. Demonstrating working with samples >>> print getLength(sound) 220568 >>> print getSamplingRate(sound) 22050.0 >>> print getSampleValueAt(sound,220568) 68 >>> print getSampleValueAt(sound,220570) I wasn't able to do what you wanted. The error java.lang.ArrayIndexOutOfBoundsException has occured Please check line 0 of >>> print getSampleValueAt(sound,1) 36 >>> setSampleValueAt(sound,1,12) >>> print getSampleValueAt(sound,1) 12

101. Working with Samples We can get sample objects out of a sound with getSamples(sound) or getSampleObjectAt(sound,index) A sample object remembers its sound, so if you change the sample object, the sound gets changed. Sample objects understand getSample(sample) and setSample(sample,value)

102. Example: Manipulating Samples >>> soundfile=pickAFile() >>> sound=makeSound(soundfile) >>> sample=getSampleObjectAt(sound,1) >>> print sample Sample at 1 value at 59 >>> print sound Sound of length 387573 >>> print getSound(sample) Sound of length 387573 >>> print getSample(sample) 59 >>> setSample(sample,29) >>> print getSample(sample) 29

103. Recipe to Increase the Volume def increaseVolume(sound): for sample in getSamples(sound): value = getSample(sample) setSample(sample,value * 2) Using it: >>> f="/Users/guzdial/mediasources/gettysburg10.wav" >>> s=makeSound(f) >>> increaseVolume(s) >>> play(s) >>> writeSoundTo(s,"/Users/guzdial/mediasources/louder-g10.wav")

104. Decreasing the volume def decreaseVolume(sound): for sample in getSamples(sound): value = getSample(sample) setSample(sample,value * 0.5) This works just like increaseVolume, but we’re lowering each sample by 50% instead of doubling it.

105. Maximizing volume How do we get maximal volume? It’s a three-step process:  First, figure out the loudest sound (largest sample).  Next, figure out a multiplier needed to make that sound fill the available space. We want to solve for x where x * loudest = 32767 So, x = 32767/loudest  Finally, multiply the multiplier times every sample

106. Maxing (normalizing) the sound def normalize(sound): largest = 0 for s in getSamples(sound): largest = max(largest,getSample(s) ) multiplier = 32767.0 / largest print "Largest sample value in original sound was", largest print "Multiplier is", multiplier for s in getSamples(sound): louder = multiplier * getSample(s) setSample(s,louder)

107. Could also do this with IF def normalize(sound): largest = 0 for s in getSamples(sound): if getSample(s) > largest: largest = getSample(s) multiplier = 32767.0 / largest print "Largest sample value in original sound was", largest print "Multiplier is", multiplier for s in getSamples(sound): louder = multiplier * getSample(s) setSample(s,louder)

108. Increasing volume by sample index def increaseVolumeByRange(sound): for sampleIndex in range(1,getLength(sound)+1): value = getSampleValueAt(sound,sampleIndex) setSampleValueAt(sound,sampleIndex,value * 2) This really is the same as: def increaseVolume(sound): for sample in getSamples(sound): value = getSample(sample) setSample(sample,value * 2)

109. Recipe to play a sound backwards def backwards(filename): source = makeSound(filename) target = makeSound(filename) sourceIndex = getLength(source) for targetIndex in range(1,getLength(target)+1): sourceValue = getSampleValueAt(source,sourceIndex) setSampleValueAt(target,targetIndex,sourceValue) sourceIndex = sourceIndex - 1 return target Note use of return for returning the processed sound

110. Recipe for halving the frequency of a sound def half(filename): source = makeSound(filename) target = makeSound(filename) sourceIndex = 1 for targetIndex in range(1, getLength( target)+1): setSampleValueAt( target, targetIndex, getSampleValueAt( source, int(sourceIndex))) sourceIndex = sourceIndex + 0.5 play(target) return target This is how a sampling synthesizer works! Here are the pieces that do it

111. Compare these two def half(filename): source = makeSound(filename) target = makeSound(filename) sourceIndex = 1 for targetIndex in range(1, getLength( target)+1): setSampleValueAt( target, targetIndex, getSampleValueAt( source, int(sourceIndex))) sourceIndex = sourceIndex + 0.5 play(target) return target def copyBarbsFaceLarger(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)*2)): sourceY = 25 for targetY in range(100,100+((200-25)*2)): color = getColor( getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 0.5 sourceX = sourceX + 0.5 show(barb) show(canvas) return canvas

112. Both of them are sampling Both of them have three parts:  A start where objects are set up  A loop where samples or pixels are copied from one place to another To decrease the frequency or the size, we take each sample/pixel twice In both cases, we do that by incrementing the index by 0.5 and taking the integer of the index  Finishing up and returning the result

113. Recipe to double the frequency of a sound def double(filename): source = makeSound(filename) target = makeSound(filename) targetIndex = 1 for sourceIndex in range(1, getLength(source)+1, 2): setSampleValueAt( target, targetIndex, getSampleValueAt( source, sourceIndex)) targetIndex = targetIndex + 1 #Clear out the rest of the target sound -- it's only half full! for secondHalf in range( getLength( target)/2, getLength( target)): setSampleValueAt(target,targetIndex,0) targetIndex = targetIndex + 1 play(target) return target Here’s the critical piece: We skip every other sample in the source!

114. What happens if we don’t “clear out” the end? Try it! def double(filename): source = makeSound(filename) target = makeSound(filename) targetIndex = 1 for sourceIndex in range(1, getLength(source)+1, 2): setSampleValueAt( target, targetIndex, getSampleValueAt( source, sourceIndex)) targetIndex = targetIndex + 1 play(target) return target

115. Splicing Sounds (page 81-85) Splicing gets its name from literally cutting and pasting pieces of magnetic tape together Doing it digitally is easy, but not short  We find where the end points of words are  We copy the samples into the right places to make the words come out as we want them  (We can also change the volume of the words as we move them, to increase or decrease emphasis and make it sound more natural.)

116. Finding the word end-points Using MediaTools and play before/after cursor, can figure out the index numbers where each word ends

117. Now, it’s all about copying We have to keep track of the source and target indices. targetIndex = Where-the-incoming-sound-should-start for sourceIndex in range(startingPoint,endingPoint) setSampleValueAt( target, targetIndex, getSampleValueAt( source, sourceIndex)) targetIndex = targetIndex + 1

118. def splicePreamble(): file = "/Users/guzdial/mediasources/preamble10.wav" source = makeSound(file) target = makeSound(file) # This will be the newly spliced sound targetIndex=17408 # targetIndex starts at just after "We the" in the new sound for sourceIndex in range( 33414, 40052): # Where the word "United" is in the sound setSampleValueAt(target, targetIndex, getSampleValueAt( source, sourceIndex)) targetIndex = targetIndex + 1 for sourceIndex in range(17408, 26726): # Where the word "People" is in the sound setSampleValueAt(target, targetIndex, getSampleValueAt( source, sourceIndex)) targetIndex = targetIndex + 1 for index in range(1,1000):#Stick some quiet space after that setSampleValueAt(target, targetIndex,0) targetIndex = targetIndex + 1 play(target) #Let's hear and return the result return target The Whole Splice

119. What’s going on here? First, set up a source and target. Next, we copy “United” (samples 33414 to 40052) after “We the” (sample 17408)  That means that we end up at 17408+(40052-33414) = 17408+6638=24046  Where does “People” start? Next, we copy “People” (17408 to 26726) immediately afterward.  Do we have to copy “of” to?  Or is there a pause in there that we can make use of? Finally, we insert a little (1/441-th of a second) of space – 0’s

120. What if we didn’t do that second copy? Or the pause? def spliceSimpler(): file = "C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\preamble10.wav" source = makeSound(file) target = makeSound(file) # This will be the newly spliced sound targetIndex=17408 # targetIndex starts at just after "We the" in the new sound for sourceIndex in range( 33414, 40052): # Where the word "United" is in the sound setSampleValueAt( target, targetIndex, getSampleValueAt( source, sourceIndex)) targetIndex = targetIndex + 1 play(target) #Let's hear and return the result return target

121. Can we generalize shifting a sound into other frequencies? This way does NOT work: def shift(filename,factor): source = makeSound(filename) target = makeSound(filename) sourceIndex = 1 for targetIndex in range(1, getLength( target)+1): setSampleValueAt( target, targetIndex, getSampleValueAt( source, int(sourceIndex))) sourceIndex = sourceIndex + factor play(target) return target

122. Watching it not work It’ll work for shifting down, but not shifting up. Why? >>> hello=pickAFile() >>> print hello /Users/guzdial/mediasources/hello.wav >>> lowerhello=shift(hello,0.75) >>> higherhello=shift(hello,1.5) I wasn't able to do what you wanted. The error java.lang.ArrayIndexOutOfBoundsException has occured Please check line 7 of /Users/guzdial/shift-broken.py

123. We need to prevent going past the end of the sound def shift(filename,factor): source = makeSound(filename) target = makeSound(filename) sourceIndex = 1 for targetIndex in range(1, getLength( target)+1): setSampleValueAt( target, targetIndex, getSampleValueAt( source, int(sourceIndex))) sourceIndex = sourceIndex + factor if sourceIndex > getLength(source): sourceIndex = 1 play(target) return target

124. Now we have the basics of a sampling synthesizer For a desired frequency f we want a sampling interval like this: Useful exercise: Build a shift function that takes a frequency as input.

125. Modules: Moving towards movies and animations The OS module offers a number of powerful capabilities for dealing with files, e.g., renaming files, finding out when a file was last modified, and so on. We start accessing the OS module by typing:  import os The function that knows about directories is listdir(), used as os.listdir()  listdir takes a path to a directory as input.

126. Using os.listdir >>> import os >>> print getMediaPath("barbara.jpg") C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\barbara.jpg >>> print getMediaPath("pics") Note: There is no file at C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics >>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics") ['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg', 'students7.jpg', 'students8.jpg']

127. Writing a program to title pictures We’ll input a directory We’ll use os.listdir() to get each filename in the directory  We’ll open the file as a picture.  We’ll title it.  We’ll save it out as “titled-” and the filename.

128. A Working Titling Program import os def titleDirectory(dir): for file in os.listdir(dir): print "Processing:",dir+"//"+file picture = makePicture(dir+"//"+file) addText(picture,10,10,"This is from CS1315 Spring 2003") writePictureTo(picture,dir+"//"+"titled-"+file)

129. Showing it work >>> titleDirectory("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics") Processing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students1.jpg Processing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students2.jpg Processing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students5.jpg Processing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students6.jpg Processing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students7.jpg Processing: C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics//students8.jpg >>> print os.listdir("C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\pics") ['students1.jpg', 'students2.jpg', 'students5.jpg', 'students6.jpg', 'students7.jpg', 'students8.jpg', 'titled- students1.jpg', 'titled-students2.jpg', 'titled-students5.jpg', 'titled-students6.jpg', 'titled-students7.jpg', 'titled-students8.jpg']

130. Comparing before and after

131. What if you want to make sure you’ve got JPEG files? import os def titleDirectory(dir): for file in os.listdir(dir): print "Processing:",dir+"//"+file if file.endswith(".jpg"): picture = makePicture(dir+"//"+file) addText(picture,10,10,"This is from CS1315 Spring 2003") writePictureTo(picture,dir+"//"+"titled-"+file)

132. Video Processing Burst a movie into a series of JPEG frames  MediaTools or QuickTime Pro will do this for you. Process each frame using os.listdir

133. Lightening every frame of a movie def lightenMovie(folder): import os for file in os.listdir(folder): picture=makePicture(folder+file) for px in getPixels(picture): color=getColor(px) makeLighter(color) setColor(px,color) writePictureTo(picture,folder+“L"+file)

134. Creating animations Simply draw on JPEG frames And save each out as a consecutively numbered picture

135. Simplest Animation def AnimationSimple(): frames = 50 rx = 0 ry = 0 rw = 50 rh = 50 for f in range(1,frames+1): pic=makePicture(getMediaPath("640x480.jpg")) addRectFilled(pic,rx,ry,rw,rh,red) if(f < 10): writePictureTo(pic,'test0%d.jpg'%(f)) else: writePictureTo(pic,'test0%d.jpg'%(f)) rx = 5 + rx ry = 5 + ry