1. Web Applications and Multimedia
Martin Kruliš
by Martin Kruliš (v1.2)

2. Images
Images and CSS3
Wise application of CCS3 can produce unexpected effects, including animations
HTMLImageElement DOM API
var img = new Image();
img.src = "image.png";
elem.appendChild(img);
Important properties and events
complete – is image loaded and ready
naturalWidth, naturalHeight – real proportions
onload – event fired when loading completes
Example 1
by Martin Kruliš (v1.2)

3. Create temporary URL, which is valid only within the page
Images
Creating Images from Blob Data
var xhr = new XMLHttpRequest();
xhr.open("GET", "picture.jpg", true);
xhr.responseType = "arraybuffer";
xhr.onload = function(e) {
var abView = new Uint8Array(this.response);
var blob = new Blob( [ abView ], { type: "image/jpeg" } );
var urlCreator = window.URL || window.webkitURL;
var imageUrl = urlCreator.createObjectURL(blob);
var img = new Image();
img.src = imageUrl;
parentElem.appendChild(img); };
xhr.send();
Another (yet less elegant) option is to encode image data in base64 and create image source URL as "data:image/jpeg;base64,followedbybase64encodeddata…".
Create temporary URL, which is valid only within the page
by Martin Kruliš (v1.2)

4. Animation Frames Creating Animations
Script can request a callback for next animation frame by window.requestAnimationFrame(anim)
Invocation returns request ID (similar to timers)
window.cancelAnimationFrame(id) aborts request
The callback itself can update the page just before the window is re-painted
It gets time argument, which gets current time in milliseconds from window.performance.now()
The callback is invoked only once, but it can register new animation frame request
Example 2
by Martin Kruliš (v1.2)

5. The same video in different formats
HTML <video> Element
Creates embedded screen for video playback
<video controls>
<source src="myvid.webm" type="video/webm">
<source src="myvid.ogg" type="video/ogg">
<source src="myvid.mp4" type="video/mp4">
Browser does not support HTML5 video.
</video>
The controls attribute makes controls visible
If the controls are hidden, the element may be controlled by Javascript
The same video in different formats
The video element has also src attribute, which can be used instead of source sub-elements, if the video has only one source.
The MPEG4 using H.264 encoding is currently the most supported format across web browsers.
HTML5 vs. Flash: It should be noted, that video playback is one of the last few domains, where Adobe Flash technology still prevails. However, the <video> element is currently quite stable and capable of replacing Flash in its function.
by Martin Kruliš (v1.2)

6. Video HTML <video> Element DOM API
Important Methods and Properties
load(), readyState
play(), pause(), paused, ended
duration, currentTime, seekable
volume, muted
Events
loadstart, loadeddata, canplay, canplaythrough
pause, play, playing, progress, stalled
seeking, seeked
volumechange
by Martin Kruliš (v1.2)

7. SVG Scalable Vector Graphics
Quite complex vector graphics and animations
HTML5 allows direct embedding of SVG code
<svg>
<circle id="circle1" ... >
</svg>
The SVG data are visible through DOM API
document.getElementById("circle1")…
Example 3
by Martin Kruliš (v1.2)

8. Canvas The canvas Element HTML5 element for JS drawing
<canvas id="mycanvas" width="640" height="480">
Your browser does not support HTML5 canvas.
</canvas>
Canvas can be sized by CSS, but width and height define the resolution of the viewport
Canvas is cleared when resized
Drawing is performed through context
var context = mycanvas.getContext("2d");
Various types of context (for drawing, 3D graphics, …)
Animations use timers or animation frame events
Currently, only 2D context and webGL context (will be described later) are implemented in the browsers.
by Martin Kruliš (v1.2)

9. Canvas 2D Context Features Drawing rectangles Drawing paths
strokeRect(), fillRect(), clearRect()
Drawing paths
beginPath(), closePath()
moveTo(), lineTo()
arc(), quadraticCurveTo(), bezierCurveTo()
stroke(), fill()
Setting drawing styles
strokeStyle, fillStyle, globalAlpha
lineWidth, lineCap, lineJoin
by Martin Kruliš (v1.2)

10. Canvas 2D Context Features Color gradients Patterns Text rendering
createLinearGradient(), createRadialGradient()
gradient.addColorStop()
Can be used in strokeStyle, fillStyle
Patterns
var img = new Image();
img.src = "fill-pattern.png";
ctx.strokeStyle = ctx.createPattern(img,"repeat");
Text rendering
fillText(), strokeText(), font
by Martin Kruliš (v1.2)

11. Canvas 2D Context Features Transformations Compositing Clipping paths
Global transformation matrix for all drawings
translate(), rotate(), scale()
transform(), setTransform()
Compositing
How is new drawing combined with existing drawings
globalCompositeOperation
Clipping paths
Path terminated with clip(), instead of closePath()
All subsequent drawing is clipped accordingly
by Martin Kruliš (v1.2)

12. Canvas 2D Context Features Context state stack
save(), restore()
Retrieving/drawing image data
ImageData object – raw RGB data in binary form
createImageData()
getImageData(), putImageData()
Saving canvas as image
canvas.toDataURL()
Drawing images/video on canvas
drawImage(imgElem, …)
The canvas.toDataURL() function generates URL with base64 encoded image data (default format is png).
Note: drawImage(elem) renders HTML graphical element such as image or video to the canvas. It can be used to either integrate it into existing scene, or for post-processing (as it can be accessed by getImageData()).
Example 4
by Martin Kruliš (v1.2)

13. 2D Graphics General Libraries Specialized PixiJS Paper.js Fabric.js
two.js
Easel.js
Graphics2D.js …
Specialized
Graphs (Chart.js, plotly.js, D3.js, Google charts, …)
Diagrams (JointJS, GOJs, mxGraph, Flowchart, …)
by Martin Kruliš (v1.2)

14. PixiJS PixiJS Modern, promising 2D graphics library
Fast, hardware accelerated
Open source
Features
Basic shapes, sprites, text
Textures, blending
Transformations, grouping
Video
Animations, skeletons, meshes
Example
by Martin Kruliš (v1.2)

15. DELETE 2D Graphic Libraries
KineticJS
The programmer does not draw, but create objects that are automatically rendered
Similar to DOM elements
Supports geometric shapes, images, …
Modifying properties causes automatic repainting
Complex object management
Layers, object groups, …
Events
Extending event model to graphic objects
Drag and drop support
KineticJS is just an example – one library of many. Check for a more broader list.
by Martin Kruliš (v1.2)

16. WebGL Web API for OpenGL A way to render 3D graphic in web browser
Based on OpenGL ES 3.0 specification
canvas element is used as viewport
Using ‘webgl’ context, which provides bindings to OpenGL functions
var canvas = document.getElementById('glcanvas');
var gl = canvas.getContext('webgl');
gl.viewport(0, 0, canvas.width, canvas.height);
gl.clearColor(0.0, 0.0, 0.0, 1.0);
...
by Martin Kruliš (v1.2)

17. WebGL WebGL Context Data buffers (vertices, colors, …)
var buf = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, buf);
var v = new Float32Array([ 1.0, -1.0, ... ]);
gl.bufferData(gl.ARRAY_BUFFER, v, gl.STATIC_DRAW);
Vertex and fragment shaders
May be stored along with HTML page in <script> elem.
var shader = gl.createShader(gl.FRAGMENT_SHADER);
gl.shaderSource(shader, shaderSourceCode);
gl.compileShader(shader);
...
Example 5
by Martin Kruliš (v1.2)

18. 3D Libraries Three.js Lightweight 3D graphic library
Primarily uses webgl, but can fallback to SW rendering
Object based approach
3D entities are JS objects
Camera, lights, scene, …
Hides complexities of 3D math
Additional features
Object materials – different surface effects
Import/export 3D data …
Three.js is one of the most popular 3D rendering libraries. More libraries can be found here
Example 6
by Martin Kruliš (v1.2)

19. Audio HTML <audio> Element Web Audio API
Similar to <video> element
Controlled from user panel or from Javascript
Web Audio API
Playback from media files, oscillators, and binary data buffers
Adding audio filters and effects
Related to WebRTC media streams
Recording from microphone
Sending audio to RTC channel
by Martin Kruliš (v1.2)

20. Web Audio API Audio Context A context object must be created first
var audioCtx = new (window.AudioContext || window.webkitAudioContext)(); if (!audioCtx) ... // no audio
Within the context a pipeline of audio nodes has to be assembled
Inputs
Effects
Destination
by Martin Kruliš (v1.2)

21. Web Audio API Audio Nodes Audio Sources Audio Destinations
OscillatorNode
AudioBuffer, AudioBufferSourceNode
MediaElementAudioSourceNode – e.g., <audio>
MediaStreamAudioSourceNode – WebRTC media stream
Audio Destinations
AudioDestinationNode
audioCtx.destination – default destination (speakers)
MediaStreamAudioDestinationNode – WebRTC channel
by Martin Kruliš (v1.2)

22. Web Audio API Audio Nodes Effects and Filters Other Nodes DelayNode
GainNode …
Other Nodes
AnalysisNode – analysis and visualization
ChannelSplitterNode, ChannelMergerNode
AudioListener – position the sound in space
PannerNode – signal behavior in space
Example 7
by Martin Kruliš (v1.2)

23. Web Audio API Audio Buffer Playback Example
var audioCtx = new (window.AudioContext || window.webkitAudioContext)();
var channels = 2, sampleRate = 44100;
var nextTime = 0;
function enqueueAudioData(dataArrayBuffer) {
var data = new Int16Array(dataArrayBuffer);
var buf = audioCtx.createBuffer(channels, data.length / channels, sampleRate);
// ... Copy samples from data to buf ...
if (nextTime == 0)
nextTime = audioCtx.currentTime ;
var source = audioCtx.createBufferSource();
source.buffer = buf;
source.connect(audioCtx.destination);
source.start(nextTime);
nextTime += buf.duration; }
Puts data from ArrayBuffer to playback queue
Small constant to prevent data jittering
BufferSource wraps the buffer for playback
nextTime is used to mark, when the playback starts
by Martin Kruliš (v1.2)

24. Discussion
by Martin Kruliš (v1.2)