Let your WebGL program dance Web Audio API Let your WebGL program dance

Announcement Final project proposal due in one week. Demo on Jan. 16, 2017.

Final Projects Individual or Team of 2. Voting during the demo. Do NOT use THREE.js Themes for this semester: Interactive Art in WebGL Animating objects with changing colors, positions, shapes, …etc. Interacts with mouse input, audio input (music), or gyro input.

Win Me

Possible Interactions Input: Mouse, keyboard, gyroscope Audio Video or camera? Output – changing the following Size, number, position, color, texture, …etc. Viewpoint Lighting

For Further Reading Web Audio API: Watch out for deprecated functions: Good introduction at: http://www.html5rocks.com/en/tutorials/webaudio/intro/ More information on the frequency analyzer: http://srchea.com/experimenting-with-web-audio-api-three-js-webgl Watch out for deprecated functions: e.g., createJavaScriptNode() replaced by createScriptProcessor()

WEB AUDIO

Loading the Music: Option #1 Option #1: <audio> element in HTML The following code produces an audio element with playback controls. <button id="xButton">Rotate X</button> <button id="yButton">Rotate Y</button> <button id="zButton">Rotate Z</button> <br/> <button id="pButton" ...>Pause</button> <button id="dButton" ...>Depth Test</button> <audio id="myAudio" src="Sleep Away.mp3" controls></audio>

A Simple Player // Experimenting with HTML5 audio var context = new AudioContext(); var audio = document.getElementById('myAudio'); var audioSrc = context.createMediaElementSource(audio); audioSrc.connect(context.destination); audio.play(); Source: https://www.html5rocks.com/en/tutorials/webaudio/intro/

Adding Frequency Analyzer var context = new AudioContext(); var audio = document.getElementById('myAudio'); var audioSrc = context.createMediaElementSource(audio); var sourceJs = context.createScriptProcessor(2048); analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.6; analyser.fftSize = 512; // Connect the MediaElementSource with the analyser audioSrc.connect(analyser); analyser.connect(sourceJs); sourceJs.buffer = audioSrc.buffer; sourceJs.connect(context.destination); audioSrc.connect(context.destination);

sourceJs.onaudioprocess = function(e) { // frequencyBinCount: how many values from the analyser frequencyData = new Uint8Array( analyser.frequencyBinCount); analyser.getByteFrequencyData( frequencyData ); }; Source: https://www.html5rocks.com/en/tutorials/webaudio/intro/

Visualization function render() { ... // update data in frequencyData analyser.getByteFrequencyData(frequencyData); // render frame based on values in frequencyData gl.uniform1f( volumeLoc, frequencyData[160] / 255 ); gl.drawArrays( gl.TRIANGLES, 0, numVertices ); requestAnimFrame( render ); }

Loading the Music: Option #2 Option #2: XMLHttpRequest() in JS loadSound("your_song.mp3"); function loadSound(url) { var request = new XMLHttpRequest(); request.open('GET', url, true); request.responseType = 'arraybuffer'; // When loaded decode the data request.onload = function() { context.decodeAudioData( request.response, ... ... } request.send();

Lab Time! Download cube3_music.zip Change frequencyData[160] to other entry. How does it respond to the music? Uncomment console.log(frequencyData) What do you see in the console? How about visualizing the whole array of frequencyData[0..255]?

Attack of the Clones Image source: https://en.wikipedia.org/wiki/File:Imperialmarch.jpg

Clones of the Cube

5x5x5 Array of Cubes (JS code) function render() { ... gl.uniformMatrix4fv( viewingLoc, 0, flatten(viewing) ); gl.uniformMatrix4fv( projectionLoc, 0, flatten(projection) ); gl.uniformMatrix4fv( lightMatrixLoc,0, flatten(moonRotationMatrix) ); for (i=-2; i<=2; i++) { for (j=-2; j<=2; j++) { for (k=-2; k<=2; k++) { var cloned = mult(modeling, mult( translate(0.2*i, 0.2*j, 0.2*k), scale(0.12, 0.12, 0.12))); gl.uniformMatrix4fv( modelingLoc, 0, flatten(cloned) ); gl.drawArrays( gl.TRIANGLES, 0, numVertices ); }

MV.Js bug Two versions of scale() in MV.js function Scale( x, y, z ) function Scale( s, u ) The solution is rename scale(s, u).

Lab Time! Uncomment the line that sets “var cloned” and see what happens. Create more cube! How about 10x10x10? 100x100x100? Can your PC handle them? Can you display 5x5x5 cows using the OBJViewer example of last week? How about using different colors for the cubes? Passing different color attributes to the shaders? How about changing the colors directly within the shaders?