1. Web Audio API Let your WebGL program dance

2. Announcement Final project proposal due in one week. Choice of demo date? Plan for the 12/23 class. 2

3. Final Projects Individual or Team of 2. Voting during the demo. Do NOT use THREE.js Themes for this semester: Interactive Art in WebGL –Animating objects with changing colors, positions, shapes, …etc. –Interacts with mouse input or audio input (music). 3

4. Possible Interactions Input: –Mouse, keyboard –Audio –Video or camera? Output – changing the following –Size, number, position, color, …etc. –Viewpoint –Lighting 4

5. For Further Reading Web Audio API: –Good introduction at: http://www.html5rocks.com/en/tutorials/webaudio/intro/ http://www.html5rocks.com/en/tutorials/webaudio/intro/ –More information on the frequency analyzer: http://srchea.com/experimenting-with-web-audio-api- three-js-webgl http://srchea.com/experimenting-with-web-audio-api- three-js-webgl Watch out for deprecated functions: –e.g., createJavaScriptNode() replaced by createScriptProcessor() 5

6. WEB AUDIO 6

7. Loading the Music: Option #1 Option #1: element in HTML The following code produces an audio element with playback controls. 7 Rotate X Rotate Y Rotate Z Pause Depth Test

8. A Simple Player // Experimenting with HTML5 audio var context = new AudioContext(); var audio = document.getElementById('myAudio'); var audioSrc = context.createMediaElementSource(audio); audioSrc.connect(context.destination); audio.play();

9. Adding Frequency Analyzer var context = new AudioContext(); var audio = document.getElementById('myAudio'); var audioSrc = context.createMediaElementSource(audio); var sourceJs = context.createScriptProcessor(2048); analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.6; analyser.fftSize = 512; // Connect the MediaElementSource with the analyser audioSrc.connect(analyser); analyser.connect(sourceJs); sourceJs.buffer = audioSrc.buffer; sourceJs.connect(context.destination); audioSrc.connect(context.destination);

10. sourceJs.onaudioprocess = function(e) { // frequencyBinCount: how many values from the analyser frequencyData = new Uint8Array( analyser.frequencyBinCount); analyser.getByteFrequencyData( frequencyData ); };

11. Visualization function render() {... // update data in frequencyData analyser.getByteFrequencyData(frequencyData); // render frame based on values in frequencyData gl.uniform1f( volumeLoc, frequencyData[160] / 255 ); gl.drawArrays( gl.TRIANGLES, 0, numVertices ); requestAnimFrame( render ); }

12. Loading the Music: Option #2 Option #2: XMLHttpRequest() in JS 12 loadSound("your_song.mp3"); function loadSound(url) { var request = new XMLHttpRequest(); request.open('GET', url, true); request.responseType = 'arraybuffer'; // When loaded decode the data request.onload = function() { context.decodeAudioData( request.response,...... } request.send(); }

13. Lab Time! Download cube3_music.zip Change frequencyData[160] to other entry. –How does it respond to the music? Uncomment console.log(frequencyData) –What do you see in the console? How about visualizing the whole array of frequencyData[0..255]? 13