Web Audio API Let your WebGL program dance

Announcement Final project proposal due in one week. Choice of demo date? Plan for the 12/23 class. 2

Final Projects Individual or Team of 2. Voting during the demo. Do NOT use THREE.js Themes for this semester: Interactive Art in WebGL –Animating objects with changing colors, positions, shapes, …etc. –Interacts with mouse input or audio input (music). 3

Possible Interactions Input: –Mouse, keyboard –Audio –Video or camera? Output – changing the following –Size, number, position, color, …etc. –Viewpoint –Lighting 4

For Further Reading Web Audio API: –Good introduction at: –More information on the frequency analyzer: three-js-webgl three-js-webgl Watch out for deprecated functions: –e.g., createJavaScriptNode() replaced by createScriptProcessor() 5

WEB AUDIO 6

Loading the Music: Option #1 Option #1: element in HTML The following code produces an audio element with playback controls. 7 Rotate X Rotate Y Rotate Z Pause Depth Test

A Simple Player // Experimenting with HTML5 audio var context = new AudioContext(); var audio = document.getElementById('myAudio'); var audioSrc = context.createMediaElementSource(audio); audioSrc.connect(context.destination); audio.play();

Adding Frequency Analyzer var context = new AudioContext(); var audio = document.getElementById('myAudio'); var audioSrc = context.createMediaElementSource(audio); var sourceJs = context.createScriptProcessor(2048); analyser = context.createAnalyser(); analyser.smoothingTimeConstant = 0.6; analyser.fftSize = 512; // Connect the MediaElementSource with the analyser audioSrc.connect(analyser); analyser.connect(sourceJs); sourceJs.buffer = audioSrc.buffer; sourceJs.connect(context.destination); audioSrc.connect(context.destination);

sourceJs.onaudioprocess = function(e) { // frequencyBinCount: how many values from the analyser frequencyData = new Uint8Array( analyser.frequencyBinCount); analyser.getByteFrequencyData( frequencyData ); };

Visualization function render() {... // update data in frequencyData analyser.getByteFrequencyData(frequencyData); // render frame based on values in frequencyData gl.uniform1f( volumeLoc, frequencyData[160] / 255 ); gl.drawArrays( gl.TRIANGLES, 0, numVertices ); requestAnimFrame( render ); }

Loading the Music: Option #2 Option #2: XMLHttpRequest() in JS 12 loadSound("your_song.mp3"); function loadSound(url) { var request = new XMLHttpRequest(); request.open('GET', url, true); request.responseType = 'arraybuffer'; // When loaded decode the data request.onload = function() { context.decodeAudioData( request.response, } request.send(); }

Lab Time! Download cube3_music.zip Change frequencyData[160] to other entry. –How does it respond to the music? Uncomment console.log(frequencyData) –What do you see in the console? How about visualizing the whole array of frequencyData[0..255]? 13