1. Let your WebGL program dance
Web Audio API
Let your WebGL program dance

2. Announcement Final project proposal due in one week.
Demo on Jan. 16, 2017.

3. Final Projects Individual or Team of 2. Voting during the demo.
Do NOT use THREE.js
Themes for this semester: Interactive Art in WebGL
Animating objects with changing colors, positions, shapes, …etc.
Interacts with mouse input, audio input (music), or gyro input.

4. Win Me

5. Possible Interactions
Input:
Mouse, keyboard, gyroscope
Audio
Video or camera?
Output – changing the following
Size, number, position, color, texture, …etc.
Viewpoint
Lighting

6. For Further Reading Web Audio API: Watch out for deprecated functions:
Good introduction at:
More information on the frequency analyzer:
Watch out for deprecated functions:
e.g., createJavaScriptNode() replaced by createScriptProcessor()

7. WEB AUDIO

8. Loading the Music: Option #1
Option #1: <audio> element in HTML
The following code produces an audio element with playback controls.
<button id="xButton">Rotate X</button>
<button id="yButton">Rotate Y</button>
<button id="zButton">Rotate Z</button>
<br/>
<button id="pButton" ...>Pause</button>
<button id="dButton" ...>Depth Test</button>
<audio id="myAudio" src="Sleep Away.mp3" controls></audio>

9. A Simple Player
// Experimenting with HTML5 audio var context = new AudioContext(); var audio = document.getElementById('myAudio'); var audioSrc = context.createMediaElementSource(audio); audioSrc.connect(context.destination); audio.play();
Source:

10. Adding Frequency Analyzer
var context = new AudioContext(); var audio = document.getElementById('myAudio'); var audioSrc = context.createMediaElementSource(audio); var sourceJs = context.createScriptProcessor(2048); analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.6; analyser.fftSize = 512; // Connect the MediaElementSource with the analyser audioSrc.connect(analyser); analyser.connect(sourceJs); sourceJs.buffer = audioSrc.buffer; sourceJs.connect(context.destination); audioSrc.connect(context.destination);

11. sourceJs.onaudioprocess = function(e) { // frequencyBinCount: how many values from the analyser frequencyData = new Uint8Array( analyser.frequencyBinCount); analyser.getByteFrequencyData( frequencyData ); };
Source:

12. Visualization
function render() { ... // update data in frequencyData analyser.getByteFrequencyData(frequencyData); // render frame based on values in frequencyData gl.uniform1f( volumeLoc, frequencyData[160] / 255 ); gl.drawArrays( gl.TRIANGLES, 0, numVertices ); requestAnimFrame( render ); }

13. Loading the Music: Option #2
Option #2: XMLHttpRequest() in JS
loadSound("your_song.mp3");
function loadSound(url) {
var request = new XMLHttpRequest();
request.open('GET', url, true);
request.responseType = 'arraybuffer';
// When loaded decode the data
request.onload = function() {
context.decodeAudioData( request.response, ...
... }
request.send();

14. Lab Time! Download cube3_music.zip
Change frequencyData[160] to other entry.
How does it respond to the music?
Uncomment console.log(frequencyData)
What do you see in the console?
How about visualizing the whole array of frequencyData[0..255]?

15. Attack of the Clones
Image source:

16. Clones of the Cube

17. 5x5x5 Array of Cubes (JS code)
function render() { ... gl.uniformMatrix4fv( viewingLoc, 0, flatten(viewing) ); gl.uniformMatrix4fv( projectionLoc, 0, flatten(projection) ); gl.uniformMatrix4fv( lightMatrixLoc,0, flatten(moonRotationMatrix) ); for (i=-2; i<=2; i++) { for (j=-2; j<=2; j++) { for (k=-2; k<=2; k++) { var cloned = mult(modeling, mult( translate(0.2*i, 0.2*j, 0.2*k), scale(0.12, 0.12, 0.12))); gl.uniformMatrix4fv( modelingLoc, 0, flatten(cloned) ); gl.drawArrays( gl.TRIANGLES, 0, numVertices ); }

18. MV.Js bug Two versions of scale() in MV.js
function Scale( x, y, z )
function Scale( s, u )
The solution is rename scale(s, u).

19. Lab Time!
Uncomment the line that sets “var cloned” and see what happens.
Create more cube! How about 10x10x10? 100x100x100? Can your PC handle them?
Can you display 5x5x5 cows using the OBJViewer example of last week?
How about using different colors for the cubes?
Passing different color attributes to the shaders?
How about changing the colors directly within the shaders?