1. Lecture 5 Multimedia Software Tools Sandra Donohue

2. Video Antialiasing movie In common usage 'Video' usually refers to the combination of visual and audio signals - e.g. "lets rent a video". In strict usage ‘Video’ is the visual part alone and does not include the sound track. Video Tools (i.e. Premiere) allow us to edit both the video track and the sound track This lecture we will only look at the visual side of video

3. Analog There are two basic types of video: analog and digital With either production for the web or TV previously you would have been exposed mostly to analog however digital video has taken over. Most things in nature are analog: real images and sounds are based on light intensity and sound pressure values, which are continuous functions in space and time For broadcasting we must convert images and sounds to electrical signals

4. Analog That is done by appropriate use of sensors, also called transducers. Sensors for converting images and sounds to electronic signals have traditionally been analog devices, with analog outputs. The world of television and sound recording was based on these devices.

5. Raster Scanning Video cameras convert an image in front of the camera into an electrical signal An electrical signal has only one value at any instant in time-it is one-dimensional, but an image is two- dimensional Conversion of the two-dimensional image into a one- dimensional electrical signal is accomplished by scanning that image in an orderly pattern called a raster

6. Raster Scanning A sensing point is moved rapidly over the image-fast enough to capture the complete image before it moves too much. As the sensing point moves, the electrical output changes in response to the brightness or color of the image point beneath the sensing point. The varying electrical signal from the sensor then represents the image as a series of values spread out in time-this is called a video signal

7. Raster Scanning In Out

8. Raster Scanning Scanning of the image begins at the upper left corner and progresses horizontally across the image, making a scanning line across the image. At the same time, the scanning point is being moved down at a much slower rate. When the right side of the image is reached, the scanning point snaps back to the left side of the image Because of the slow vertical motion of the scanning point, it is now a little below the starting point of the first line. It then scans across again on the next line, snaps back, and continues until the full height of the image has been scanned by a series of lines.

9. Raster Scanning During each line scanned, the electrical output from the scanning sensor represents the light intensity of the image at each position of the scanning point During the snap-back time (known as the horizontal blanking interval) it is customary to turn off the sensor so a zero-output (or blanking level) signal is sent out The signal from a complete scan of the image is a sequence of line signals, separated by horizontal blanking intervals. This set of scanning lines is called a frame

10. Aspect Ratio A parameter of scanning is the aspect ratio It is the ratio of the length of a scanning line horizontally on the image, to the distance covered vertically on the image by all the scanning lines Aspect ratio can also be thought of as the width-to- height ratio of a frame. In present-day TV aspect ratio is standardised at 4:3. Other imaging systems, such as movies*, use different aspect ratios ranging as high as 2:1 for some systems.

11. Synchronization The electrical signal from a scanning process is used to modulate the brightness of the beam in a cathode-ray tube (CRT) that is being scanned exactly the same way as the sensor reproducing the original image This is what happens in a TV or a video monitor The electrical signals sent to the monitor contain some additional information to ensure that the monitor scanning will be in synchronism with the sensor's scanning. This information is called "sync" signal, and it may be included within the video signal itself during the blanking intervals, or it may be sent on a separate cable (or cables)

12. Resolution Resolution is the ability of a television system to reproduce fine detail in the scene. It is expressed separately for horizontal and vertical directions. i.e. 640 x 480

13. Horizontal Resolution As the scanning point moves across one line, the electrical signal output from the sensor changes continuously in response to the light level of the part of the image that the sensor sees. One measure of scanning performance is the Horizontal resolution of the pickup system, which depends on the size of the scanning sensitive point. A smaller sensitive point will give higher resolution.

14. Horizontal Resolution The output is zero in the black area, and the output begins to rise as the sensor moves partially onto the white area. Full output (100%) is reached when the sensor is completely on the white area.

15. Vertical Resolution The vertical resolution of a video system depends on the number of scanning lines used in one frame. The more lines there are, the higher is the vertical resolution Broadcast television systems uses between 500 and 600 per frame In both cases the resolution is very poor compared to standard video monitor (1024x768)

16. Color TV Systems: Composite Colour cameras producing three output signals - red, green, and blue = RGB. Most uses of video involve more than a single camera connected to a single monitor. We may wish to combine the outputs of several cameras together in different ways, and almost always we will want to have more than one viewing monitor.

17. Color TV Systems: Composite In RGB systems, all parts of the system are interconnected with three parallel video cables, one for each of the colour channels. Due to the complexities of distributing three signals in exact synchronism and relationship, most colour TV systems do not handle RGB (except within cameras), the camera signals are encoded into a composite format which may be distributed on a single cable, used throughout TV studios, for video recording, and for broadcasting.

18. Color TV Systems: Composite Different composite formats are used around the world:  NTSC (US and Japan) ~30 frames per second (29.97 fps) 640 Horizontal Pixels 480 Vertical Pixels 100 levels of brightness (~7 bits 128) 70 levels of hue (~6 bits: 64) 100 levels of saturation  PAL (Europe, Australia and Pacific) 25 frames per second 704 Horizontal Pixels 512 Vertical Pixels 100 levels of brightness (~7 bits 128) 100 levels of hue 100 levels of saturation  SECAM (Europe)  As a result an NTSC signal supports approximately 20 bits of colour and PAL signal supports ~21 bits.

19. Color TV Systems: Composite When we record a television/video signal to a computer this is no longer true. The computer's capture card reconstructs the original RGB format in order to allow efficient compression

20. Luminance/Chrominance All composite formats make use of the luminance/ chrominance principle for their basic structure:- any colour signal may be broken into two parts  luminance, a monochrome video signal that controls only the brightness (or luminance) of the image  chrominance, the colouring information for the image. However, because a tri-stimulus colour system requires three independent signals for complete representation of all colours, the chrominance signal is actually two signals (X&Y), called colour differences:

21. Luminance/Chrominance X=R+B Y=G-B and we also send the luminance (L) which is = R+G+B so if want green: G=L-X -> G = (R+G+B) - (R+B) and then we can substitute so now Y=L-X-B rearranged to B=L-X-Y -> (R+G+B) - (R+B) - (G-B) = R + G + B - R - B - G + B = B Lastly R = L-B-G! With a little algebra you can extract all the colours from the signals

22. Video The principle of video is quite simple and has been known for several thousand years - flick cards! Which give the impression of a moving object Our brains cannot see very rapid changes. The fluorescent lights above us are actually blinking on and off 50 times per second (the incandescent are not!) and yet we see a steady stream of light.

23. Video Video is based on the principle that if you show a person a set of images in a rapid manner, that viewer will not see the individual images, but rather sees a constant image. This is called ‘Persistence of Vision’

24. Video If we slow the rate of change down progressively we can see the individual frames appear and the motion becomes jerky. The same principle is used for all tape and disk videos, movies, TV and computer video clips. See here (movie) (movie)

25. Video If we go too slow it looks jerky and awkward, conversely if we go too fast it looks fake and unreal. For those of you who have experimented with 3D and rendered some animations to video files the package did all the ‘in-between frames for you (movie).(movie) This is a single image in Lightwave 5.5 where the viewpoint has been animated by LW from one point to another.

26. Video Each one of these images is called a frame The rate at which we display them (usually expressed frames per second - fps) is called the ‘frame rate’ The frame rate is obviously important to how we perceive a piece of video!

27. Frame Rates for Motion For motion video, many frames must be scanned each second to produce the effect of smooth motion In standard broadcast video systems, normal frame rates are 25 or 30 frames per second, depending on the country These frame rates are not high enough to prevent a video display from having a flicker For the human eye not to perceive flicker in a bright image, the refresh rate of the image must be higher than 50 per second

28. Frame Rates for Motion To speed up the frame rate to that range while preserving horizontal resolution would require speeding up of all the scanning, both horizontal and vertical, therefore increasing the bandwidth required to send the signal. Because this is a problem for systems that send through the air - all television systems use interlacing whereas other systems (computers etc) do not!

29. Frame Rates for Motion Interlace in a television system means that more than one vertical scan is used to reproduce a complete frame Broadcast television uses 2:1 interlace-2 vertical scans (fields) make up a complete frame. With 2:1 interlace, one vertical scan displays all the odd lines of a frame, and then a second vertical scan puts in all the even lines.

30. Frame Rates for Motion Old 1930's movies were filmed at approximately 13 frames per second because of limitations of the equipment of that day. These old movies appear jerky because the individual frames (images) are visible for too long. To achieve non-jerky or smooth motion frame rates need to be above 24 frames per second

31. Frame Rates for Motion Even at 24 frames per second there are visual anomalies: one of the obvious ones we have all seen is the backward spinning wheels or propellers on cars/planes in the movies. This does not happen in real life and is an anomaly of the video capture scanning process.

32. Frame Rates for Motion Why the Backwards spinning wheel occurs: - Imagine yourself as the camera or recording device you are only shown the 'world' 30 times in a second 50/s 20/s 10/s 4/s What you see is a sequence of 4 images of a wheel spinning in a CLOCKWISE direction at various rates

33. Motion Blur This occurs when the object moves during the time which the "shutter" is open In fact it is possible that a fast moving object is off to the left when the first frame is taken and off to the right when the next frame is taken. In short it may not appear on the film at all

34. Motion Blur One solution is instead of getting a perfect image of the state of the world at each /50th (or whatever) cameras can actually record a frame capturing the view over the period of time between frame 1 and 2 i.e. a longer exposure to the view. But this produces an effect called 'motion blur' Motion blur represents a loss of image accuracy in order to achieve believable motion

35. Motion Blur What the camera records is the 'blur' of the fast moving object (or the camera can be on a fast moving object!) At no point does the viewer see an accurate image of the object, but does get the sense of 'motion' How do we create this effect in a rendered environment?

36. Motion Blur In a rendered (i.e. non real) environment this is probably one of the easiest effects to implement. Simply re-render a scene multiple times, incrementing the position and/or orientation of an object in the scene The object will appear blurred, suggesting motion. This effect can be incorporated in the frames of an animation sequence to improve its realism, especially when simulating high-speed motion

37. Motion Blur The apparent speed of the object can be increased by dimming its blurred path by reducing the brightness of the copies

38. Motion Blur A 'jerky' video sequence can be the result of 2 major problems, both relating to the viewer seeing each individual frame too sharply: Too slow a frame rate (i.e. the viewer sees each frame for too long) No motion blur (the viewer sees each frames image too sharply and loses the sense of motion)

39. Digitising Video Digital recording of a video signal requires large amounts of disc space. Just a few quick calculations will tell you that 640 x 480 x 24bit x 30fps x10s = 289,440,000 bytes that's 285MB for 10 seconds Bandwidth requirements on the CPU's bus of 30MB/s means most computers cannot sustain such a high bandwidth. To allow 'desktop video' processing like Premiere there has to be a compromise: compression -> loss of quality

40. Digital Video Compression Video sequence contains an enormous amount of information to capture and store this we need to use very good compression techniques These techniques must also allow for fast decompression they need to decompress and play at least at 30 frames per second  Fast Compression (to capture 30 fps)  Highly effective Compression (large file size reduction needed)  Fast De-Compression (to decompress and display at 30 fps)  The process of compression must be handled in hardware, software is just too slow

41. Digital Video Compression A special hardware device (known as a CODEC) takes the video signal and compresses it as it records The compression technique is sometimes called the CODEC (i.e. compression- /decompression) Compression techniques vary significantly in terms of:  Compression (i.e. how much they reduce the file size)  Result Quality (i.e. how much quality they give up in compression)  Speed (i.e. the rate at which they can compress/decompress) An ideal system would achieve high compression at high speeds (30fps) in a lossless mode

42. Digital Video Compression Lossless compression schemes preserve the original data, ensuring that the image is the same before and after compression Most lossless compression schemes use some from Run Length Encoding (RLE)

43. Digital Video Compression In general 'lossless' compression techniques don't achieve very high compression rates because most video is 'photograph' oriented and Run Length Encoding isn't effective on photos Lossy compression schemes on the other hand are willing to change the original data in subtle ways to achieve better compression but as a result they often lose image quality Most lossy compressors suffer 'additive' loss

44. Digital Video Compression When dealing with video there are several types of compression: Spatial Compression compresses the data in each frame of the video clip Temporal Compression compresses the data by comparing frames over time Each method has side effects (some overlap): Spatial Temporal Contouring Blockiness Blurring Blockiness Streaking

45. Digital Video Compression Numerous video formats exist (too many to list), the main ones are:  DVI: 160:1 (lossy) best compression (owned by Intel) 160:1  MPEG: 40:1 (lossy) endorsed broadcast standard  JPEG - a stream of JPEG images  QuickTime - a whole lot of formats which can use multiple compression techniques

46. Capturing & Editing Motion Video allows us to capture motion by recording 1 / 30th transitions in time onto single images or frames. These frames can be edited just like any other still image. Video editing tools allow us to edit the individual frames, by editing sections or on a frame by frame basis if necessary

47. Capturing & Editing Motion Video Editing enables us to achieve many effects  Overlaying/merging 1 video onto another  Cutting/trimming/joining video sequences  Adding transitions between joins  Adding/Overlaying Items (text, etc.)  Adjusting Visual factors (brightness, colours etc)

48. Example Video Sequence The following is an example of a heavily edited video sequence from Premiere's sample directory: Watch the video and try to count the number of pieces of video editing sample.avi

49. Example Video Sequence This sequence contains 10 significant video only edits These are mainly - Joins and Transitions between clips and overlays Software tools such as Premiere will allow us to implement any of these effects Video packages also enable us to  attach audio tracks to our visual sequences  edit the sound tracks much as we do the visual tracks  output our video in a range of formats (Premiere allows QT and AVI formats)

50. Overlaying Video Sequences Overlaying 2 video signals is a very powerful tool which is widely used in the film and television post-production industry. Famous examples of this technique include: Terminator Jurassic Park

51. Blue screen techniques Using a tool like Premiere we can implement similar overlays although nowhere near the standard of facilities such as anti-aliasing. Steps in doing a Premiere overlay involve:  Obtain the two segments  Specify the transparent elements  Generate the overlay This is a simple example here (movie)here The same methods are used to develop complex commercial overlays.

52. Cutting and Transitions The cutting room used in the film and television industries is no longer required Machines can undertake these tasks more effectively by dealing with 'virtual' film in a machine (allows for mistakes, multiple versions etc.) Taking a sequence with a join in the middle, insert a suitable transition over the join:  Clip 1 Clip 2 Clip 1Clip 2  Joined with a transition By adding a transition of any (large range) type we generate a smooth move between the 2 sequencestransition

53. Advanced Video Effects Digital Composition is the digitally manipulated integration of at least two source images to produce a new image Integration is the heart of compositing: The new image must appear real-completely and seamlessly integrated ­ as if it were actually photographed by a single camera at one instant, or the illusion of compositing has failed. Digital compositing is the art of the invisible effect

54. Basic Compositing Theory The colour of each pixel in a digital image is defined by numbers. These numbers con be manipulated mathematically to modify or to combine images. Most mathematical operations for compositing break down to Add, Subtract, Multiply, Divide, or combinations thereof

55. Basic Compositing Theory One of the most basic and simple operations is Add. This simply adds the value of a pixel from the first image to the value of the corresponding pixel in the second image. In shorthand:' Output = A Image + B Image, or O = A+B An Add operation is similar to double-exposing film, with the same drawbacks

56. Basic Compositing Theory Almost all compositing operations are based on values between 0 and 1, with 0 equal to black or the minimum value, and 1 equal to white or the maximum value. The bit depth of an image doesn't change the equations, because bit depth simply defines the number of divisions between black and white. Eight bits defines 256 divisions, 10 bits defines 1024, and 16 bits divides the 0-to-1 range into 65, 536 discrete values

57. Rendering The process of applying one or more operations to can image is called rendering. The result of rendering is the new image. The more complex the operations, the longer the rendering takes Some high-end compositing systems use hardware acceleration to reduce rendering time. Others, like After Effects and Digital Fusion, rely on multiple processors to share the load

58. Channels A channel is a subset of image values from a single component of the image. A colour image usually has at least three channels: R, G, B, for red, green, and blue colour component information For compositing, images often use additional channels to control the transparency of the colour image, indicate the depth or texture of objects, or provide other information to be used by a filter or process Four or five total channels are common, but up to 24 are available in some compositing software.

59. Mattes A matte is an image in which its pixels' values are used to control the transparency of a layer in a composite

60. Mattes WalkM276.jpg is a greyscale image

61. Mattes Whereas Walkr276.jpg is a colour image

62. Mattes If you use WalkM276 to control the transparency of Walkr276, the black areas around the robot will be completely transparent, the white area of the robot's body will be completely opaque, and the shaded grey areas of the robot's shadows will be partially transparent

63. Mattes In other words, multiply Image A times the matte, multiply Image B times the inverse of the matte, then add them to get the output image. For example, if you composite the robot image and the matte using the Over operation, with a background image such as a chessboard, the background will show through the transparent areas with an effect like this

64. Mattes

65. An alpha channel is commonly used to composite a foreground layer over a background layer, with the alpha channel controlling how much and where the background shows through. For example, open WALKRGBA.TGA in Photoshop, or any other program that displays alpha channel information. Then open a channel display and select the alpha channel. You should see the alpha channel information.

66. Key Key is a synonym for matte but is more generally used in reference to chroma keying for bluescreen work. Keying is the process of separating a foreground object from its existing background, creating a matte that then can be used to composite the foreground over a different background. Chroma keying creates a matte by subtracting (making transparent) a particular colour or range of colours. Common subsets of chroma keying include bluescreen and greenscreen

67. Key An actor filmed against a pure, saturated blue (bluescreen) background is a good candidate for chroma keying

68. Key Luma keying creates a matte by subtracting a particular brightness value or range of values Difference keying creates a difference matte by subtracting the values of one image from those of another

69. Matte types A fixed or static matte is set up at the first frame and does not change throughout the image sequence. A travelling matte moves but does not necessarily change shape. A travelling matte is often used on a rigid object that does not rotate or change profile during the sequence- for example, a microphone or other rig element An articulate matte is one that is animated to change shape over the course of an image sequence. Generally, a matte is articulated in order to closely follow the contours of the subject being extracted

70. Articulate Matte An extreme form of articulate matte is the rotoscope (or roto-) matte, which is animated frame-by­frame to extremely tight tolerances. Roto-matte work is generally reserved only for shots that have no clean plate, when the camera was not locked down or motion controlled, and when both the background and foreground are too complex for other matte extraction methods. In most cases, roto-matte work is the last resort simply because of the time and expense to tweak the matte controls on every frame. A roto-matte for a single actor may have dozens of control points; multiply that by 1,440 frames per minute of film (1,800 for video) and you con see the huge amount of work involved

71. Complementary Matte A complementary matte is the inverse of a primary matte, having precisely opposite greyscale values for each pixel. Complementary mattes are useful for controlling effects that do not have an invert option.

72. G-matte A garbage matte, or G-matte, is a quick-and-dirty way of blocking off part of an image with a hand-drawn polygon or bitmap shape. The portion of the image blocked off is usually rigging or other garbage, hence the name. The matte may be as simple as a triangle; the simpler and faster to draw, the better. Garbage mattes save time for the compositor by reducing the area that requires more painstaking work, and they may save rendering time as well

73. Edge Matte An edge matte contains only the boundary or outline of the subject. This is a useful tool for creating either inner or outer garbage mattes that conform closely to irregular contours. Edge mattes are also handy for masking the effects of a process to the edges of a foreground element. In addition to compositing, edge mattes can be applied to a single image as the basis of energy-discharge effects such as lightning, St. Elmo's fire, or energy-weapon hits

74. Mask A mask is a special matte that restricts or modifies the area affected by a process, rather than separating part of an image from its background A mask can have sharp or feathered edges and may be any shape, limited only by your software's masking tools. The most common use of masks is to limit keying effects to the immediate area of the keyed subject, excluding un­wanted effects in the rest of the image. Masks can be animated by hand or tracked to moving patterns in an image sequence

75. Filters A filter is an algorithm used to sample and modify an image pixel by pixel. There are several types of filters – spatial filters, spatial convolution filters, sharpening filters, gaussian filter, Mitchell filter and sinc filter

76. Geometric Transformations Whereas a filter changes an image by modifying each pixel's colour values, a geometric transformation, or transform, changes an image by moving the pixels around without changing their colour values. You might think of a transform as rearranging a mosaic; all the tiles are still present, but in different places. The most basic transform is the flip, an inversion of the image along the horizontal or X-axis. The flip is especially useful for reflection maps and other tricks to make a composited image appear correctly in shiny surfaces

77. Other Effects A dissolve is one of the oldest, most basic, and still popular effects. Although there are a plethora of variations on the basic dissolve, at core it is simply a gradual replacement of one image with another A fade is related to a dissolve, except that the target image is usually solid white or (more commonly) solid black. Many film scripts actually end with the phrase, FADE TO BLACK. Even if none of the scene transitions are done with fades, you will generally need to do a fade to black under the end credits

78. Other Effects One of the common challenges in digital compositing is matching CGI elements to live footage. CGI tends to be too clean, with every surface in perfect focus. In reality, haze, dust, and other atmospheric influences tend to wash out, or desaturate, the colours of distant objects. This atmosphere is an unconscious depth cue for the audience, and if your composite does not re-create this effect, the shot will look subtly unreal Look at this sample!!