1. 1 DATA AND FILE FORMATS Data and file format standardization is crucial for sharing of data among multiple applications and for exchanging information between applications. Personal - computer(PC) industry has generated many different standards. Text based file and data formats have been replaced by multifunction formats which can handle graphics, audio, video and color images. A few of the commonly used data and file formats are: 1.Rich Text Format ( RTF)

2. 2 Early text editors could not carry through formatting information when transmitting files. This limited data interchange because when text was moved from one application to other, all the formatting information was lost and had to be re- entered. The RTF format extended this range of information. RTF is capable to handle binary files, audio files and video files to a certain extent. 2.Tagged Image File Format ( TIFF) Tagged Image File Format has been around for a long time. In this format, tags are used to keep all the attribute information in a standard manner.

3. 3 TIFF file provides tags that store information about resolution of the image, fonts, format color, compression scheme, date and time of capture, decompression, etc. A search through file is quick since these can be found easily. In case you have to extend the file, it is done through pointers and links. And by creating extra blocks. This approach represents industry standard to represent raster image data generated by scanners, frame grabbers and paint/photo retouching applications. It can represent color images in different types representations. This standard can handle color images very well.

4. 4 3.Resource Interchange File Format (RIFF) RIFF is not really a new file format. Rather, it provides a framework for multimedia file format for Microsoft windows based applications. It can be used to convert a custom file format into a RIFF format and transmit the file. For example a MIDI file format is converted to RIFF by adding the RIFF structure around it. Information is in blocks - called chunks. Like TIFF, RIFF is a tagged file format and uses tags to store information in the header, about the file. RIFF can handle MIDI, DIB, PAL, AVI files. MIDI - is musical instrument digital interface form.

5. 5 DIB: Device Independent Bit-Map file It can synchronize audio and vidio in movies. Other commonly used format are: 4.Joint Photographic Expert Group (JPEG) - DIB Format for Motion Images Microsoft has extended the DIB file format for both JPEG still and motion images. This can be used with RIFF and AVI file format. 5.Motion Pictures Expert Group (MPEG) Format MPEG1, MPEG2 are existing standards 6.Audio-Video interlaced file format ( AVI) 7.TWAIN Format ( for multimedia applications).

6. 6 Multi-Media Input/Output Technologies Multimedia can mean different things. It can be an encyclopedia on a CD-ROM or a hypermedia message composed by a user consisting of text, images, full motion video. Hypermedia links allow tracking of a subject matter through a variety of topics. It takes specialized equipment to capture and store multimedia objects. Keyboard has been traditional input device for entering data into computer system. It has changed from simple numeric device to alphanumeric and multifunction device over the years.

7. 7 With the advent of GUIs, pointing devices, such as mouse or a pen, have become essential for selecting or moving graphical objects. Window based GUI applications require a mouse or pen for selecting various objects, push buttons, data entry boxes, so on. In addition to traditional alphanumeric data entry, multimedia technology requires a variety of other types data inputs including voice or audio, full motion video, still photos and images. These inputs require special devices such as digital pens, audio equipment, video cameras, and image scanners. In case of text, there was no measure of quality. Text was stored in ASCII/EBCIDIC formats. Now with

8. 8 higher quality multi-font printers, the text quality is measured in terms of print matrix resolution, text color, font types, etc. The text capturing device does not determine the end quality of the text. Multimedia objects such as images, audio and video depend on input device and storage for quality. However, the capture device determine the outer bound of the quality. The display device, at best, can match the resolution of capturing device. Digital verses Analog Inputs Another important distinction with multimedia objects is the need to convert data from analog to digital form.

9. 9 For example a scanner scans an object into scan lines and pixels and then converts analog amplitude of each pixel into a digital measure. 0 or 1 for black and white. 0 - 255 for gray scale pixel points. HSI or RGB for color objects. TV signal ( NTSC) is also analog. It needs to be converted into digital form for use in computer system. The process for converting analog to digital and digital to analog are called as coding and decoding. Hardware devices and software programs for this implementation are called as codecs.

10. 10 Codecs usually include compressions and decompression algorithms. Different codecs are required for each type of multimedia inputs. Display and Encoding Technologies Since multimedia systems include a variety of object types, a number of different technologies are required for compression, decompression and display of multimedia objects. Almost all multimedia objects are based on a graphical user interface (GUI). Most graphical user interface are based on VGA ( 640x480 pixels) or SVGA (800 x 600 pixels)

11. 11 or even XVGA(1280 x 1024 pixels). Some imaging applications may require 150 - 200 pixels per inch or better resolution. Voice mail system store analog sound and are usually based on adaptive differential PCM technology. Codecs are required for converting analog sound to digital formats such as WAVE or AVI. Video cameras provide input in analog formats such as NTSC ( national television system committee ) standard, PAL ( phase alteration line standard) or SECAM ( France). Input from either source must be encoded to digital format and decoded for transfer back for analog play back. Encoded compressed digital signals are based on JPEG or

12. 12 MPEG standards. Other formats include AVI and RIFF. Resolution and Bandwidth Issues Each object type has some resolution. Images are measured in pixels per inch. Higher the resolution, better the object quality. For document imaging systems, screen resolution of 100 pixels per inch ( ppi) are required. The quality of 200 ppi is very good. Laser printers and office copiers can provide a quality of 300 - 600 ppi. Published ( professional quality books) have resolution of 1200-1800 ppi.

13. 13 Sound quality is measured in terms of sampling rate and number of bits used for representing magnitude of the sample. A higher sampling rate allows capturing of higher frequency details. Higher number of bits allow capturing of amplitude changes more accurately. Both factors contribute to the tone quality. A sampling rate of 4 kHz at 8 bits is considered as minimal acceptable for voice grade sound. A sampling rate of 8kHz at 16 bits is required for music quality.

14. 14 For CD-quality stereo sound, the sampling rate of 44.1kHz at 16 bits is required. Multi-channel stereophonic sound requires even higher resolution. The VCR quality is considered a minimum for video display which is defined as 300 lines visible on the screen. The minimum acceptable resolution is 320 x 240 pixels. HDTV quality is 1280 x 1024 pixel range Another measure of video quality is number of bits being used for color definition. A 16 bit palette is common. Higher color resolution is required for HDTV quality.

15. 15 This will be 24 bit colors ( full color). A third measure is the number of frames per seconds. TV operates at 60 FPS page 192. Multimedia Object Quality and Transmission Bandwidth

16. 16 Multimedia Input and Output Devices Electric Pen When an electric pen is used to write or draw, the digitizer encodes the x and y coordinates of the pen, and the pen status, which includes whether the pen is touching the digitizer surface ( usually the screen) or not, pen pressure, pen angle, rotation, etc. Most electric pen contain a micro-switch at the tip that behaves like left button on the mouse. Some pens are capable of measuring accurate pen pressure while others can measure the proximity. Pen computing requires generating x-y coordinates at least

17. 17 120 times per second with 200 dpi resolution. The minimum sampling rate generates sufficient data to track pen movement. Most pen digitizers produce an accuracy of 0.005 to 0.02 inch resolution. Resolution is defined as the number of points digitizer is able to digitize in one inch. Video and Image Display Systems TV is video technology. Live pictures bring reality in our environment. They educate us. Introduction of video game took younger generation by a storm.

18. 18 Virtual reality will be the next major advance in game technology, in military technology, and in training environment. VGA, SVGA, XVGA, 8514A, are some of the existing video technologies. Display Performance Issues There are three main factors that affect the performance: 1. Network bandwidth: the play back becomes choppy and incoherent if the bandwidth is insufficient to support minimum data rate. There are JPEG and MPEG standards to define this parameter. 2. Decompression or Decoding: once again, while in the

19. 19 case of poor decompression, performance causes irritation delays. In case of full motion video, poor decompression causes same effect as poor network bandwidth. 3. Performance of Display Technology: if the technology is not appropriate, the device may not display full motion or graphics properly. Tables show various video standards.

20. 20

21. 21

22. 22 Typically, 14 inches monitors have active display area of 9.875” x 7.125” and diagonal of 12.25”. If we assume a resolution of 1024 x 768, the dot pitch can be defined as ( distance between two pixels): This means that to display a resolution of 1024 x 768 very clearly, we need a 0.24 mm dot pitch monitor. Most 14-inch monitors come in 0.28 - 0.30 mm dp. When these monitors display 1024 x 768 pixels, the accuracy and crispness are compromised.

23. 23 Now let us repeat the same calculations for a 17-inch monitor which has 12.9” x 9.76” size with a diagonal of 16.125 inchesThe dp for 1024 x 768 pixels will be This means to display a resolution of 1024 x 768 on a 17- inch monitor, you would require a dp of 0.32 or better. Most of the 17-inch monitors have a dot pitch of 0.28- 0.30. This is sufficient even to display a resolution of 1280 x 1024.

24. 24 Smaller dot pitch gives a perception of clearer picture. Horizontal Refresh Rate: is a measure of the rate at which scan lines are painted. It is measured in kHz and a standard VGA monitor has a horizontal refresh rate of 31.5 kHz Vertical Refresh Rate: is closely tied to horizontal refresh rate. It the rate at which whole screen is painted ( counting all scan lines) and return to the top of the screen.

25. 25 It is measured in Hertz. Typically it is 50-72 Hz. Human eye is sensitive to lower vertical refresh rates and is more likely to perceive flicker at lower rate. It can be annoying and tiring to eyes. Print Output Technologies Laser print technology has continued to evolve and print quality at 600 dpi is starting to make this technology useful for high speed process. Offset printer resolution is around 1200 - 1800 dpi. 600 dpi is sufficient for common print applications. Digital Voice and Audio Multimedia technology is multidimensional and audio is

26. 26 Comparison of Print Technologies

27. 27 one of the dimensions that adds voice, music and sound capabilities. Until 1990, PC applications were visuals. Game applications added voice/music dimension. Today, some applications utilize sound boards whereby audio inputs may be through keyboards, microphones, etc. Digital Audio When voice or music is captured by a microphone, it generates electrical signal. The signal consists of fundamental sine wave with certain frequency and amplitude. The fundamental sine wave is accompanied by harmonics.

28. 28 Adding the fundamental to harmonics, forms a composite sinusoidal signal that represents the original sound. Analog sinusoidal waveforms are converted to digital format by feeding the analog signal to A/D converter (ADC) where the analog signal goes through the sampling process. Sampling Process: The analog signal is sampled over time at regular intervals to obtain amplitudes of the signal at sampling time. The regular interval at which sampling occurs is called the sampling rate

29. 29 T = Time interval between two samples

30. 30 The sample amplitude obtained at sampling instants is represented by an 8-bit value (one byte) or 16-bit (two bytes) value. Higher values can also be used for higher resolution systems ( high fidelity sound). A composite signal of 11.025 kHz sampled 4 times every cycle will yield 44.1 kHz sampling rate. If you sample at higher rate, you need to store more samples. For CD quality music at 44.1 kHz rate at 16-bit resolution, a one minute recording will require 44.1 x 1000 x 16 x 60 / 8 = 5.292 Mbytes.

31. 31 Audio objects generate a large volume of data. This poses two problems. First, it requires a large volume on disk space to store, and Second, it takes longer to transmit this data. To solve these problems, the data is compressed. Compression helps shrinks the volume of data and less disk space is required. It also helps to reduce network time. Audio industry uses 5.0125kHz, 22.05kHz and 44.1kHz as standard sampling frequencies. These frequencies are supported by most of the sound cards.

32. 32 Digital Camera Digital cameras are being used increasingly for multimedia applications due to internet advantage they provide in applications where very high resolution is not required. The advantages are: Digital images can be viewed immediately for proofing Digital images can be printed immediately and any number of times for duplications Digital images can be integrated with word- processor documents Can be embedded in emails or faxed by computers.

33. 33 Can be enhanced/altered for effective presentations Can be archived - minimizing the risk of loss or damage to the image. Can take images of 3-D objects and store as 3-D images Digital cameras are portable and can be used in environment where film cameras can not be used. Possible uses: for fingerprint analysis, for drivers’ licenses, insurance companies, bank-customers’ signatures, security installations, etc.

34. 34 Full Motion Video Although video image processing is not very common for full motion video, there is no reason why it will not be done in future. Video capture circuit cards capture from NTSC/PAL/SECAM signals from video cameras or VCRs or even s-video inputs ( RS 170 inputs). Video capture boards can handle audio signals as well, they can convert analog signals to digital (ADC) and digital to analog forms(DAC) Normally, a video capture board is used to capture real- time video, and the digitized raw data is then compressed in real-time.

35. 35 The compressed data is subsequently moved to CPU over ISA or local bus. The CPU then builds the AVI file format for the compressed data and stores the file. During the playback, the file is read in blocks by the CPU, and the data is decompressed as blocks of audio and video. The data can be decompressed in either software or hardware and sent to VGA card for display. To understand performance issues, let us take an example: calculate the bandwidth required to display a real-time video at 640 x 480 resolution at 30 Hz frame rate in true 24-bit color.

36. 36 Bus Bandwidth The bandwidth required for display of full motion video is- resolution x frames per sec x pixels per bit for color. 640 x 480 x 30 x 24 = 27.648 Mbytes/sec This bandwidth is required to display real-time video in true color 24-bit per pixel mode. If you want to reduce from full color mode to 256 colors only ( 8-bit color), the bandwidth requirement will reduce to 9.216 Mbytes/sec The ISA bus operates at 8 MHz and has a bandwidth of 2 Mbytes/sec, which is not sufficient for above application. This gives you following options:

37. 37 Display a video window of 300x200 at 30 frames per seconds with 256 colors. 300x200x30 x 8 = 1.98 Mbytes/sec Display the video window at full VGA resolution of 640 x 480 at 6 frames per second with 256 colors. 640 x 480 x 6 x 8 = 1.84 Mbytes/sec It is clear that ISA bus is a big bottleneck. However, the bus bandwidth problem can be solved by using other bus architectures, such as local bus, VESA, VL bus or PCI bus. Both VL and PCI buses have bandwidth in excess of 100Mbits per second. In theory, local bus operates at CPU speed.

38. 38 To achieve good performance, every link in the chain for capture and playback must be examined carefully to ensure that the required bandwidth can be carried by that link, be it network, video server, compression or decompression hardware or even display system. Animation: It is an illusion of movement created by sequentially playing still image frames at a rate of 15-20 frames per second ( close to full-motion video range). Animation film contains a series of frames with incremental movement of objects in each frame. It may not be necessary to move objects to create the illusion of movement. Color and background can be changed from

39. 39 frame to frame so that there is perception of moving object. MEMORY SYSTEMS Memory systems for computers have been changing to meet the needs of high resolution graphic displays. The demand on memory systems will even be greater with the increased use of multimedia applications. Memory Types Different types of memories are used for different purposes due to retention factors, performance parameters, and cost trade-offs.

40. 40 Memory types that may be used in multimedia systems include the following: 1.ROM (read only memory): is read only. Instructions and/or data is burned into the memory permanently, and the contents are non-volatile. ROM is used for firmware. That is: for operating systems, software programs that have to reside permanently inside computer. 2. PROM (Programmable ROM): is semiconductor memory that contains an array of fuses. These fuses are blown according to the word to be programmed. To program, a specialized PROM programmer (PROM burner) is used. This burner blows the fuse.

41. 41 The contents of this PROM are non-volatile. The access to PROM is random. Typical data-path is 16-bits wide. 3.RAM (Random Access Memory): is also semiconductor type of memory that allows random access to its contents.That is, the word can be accessed by directly addressing it. It is organized in an array form so that it can be read and written efficiently. All words are addressable. There are several types of RAMs: SRAM (static RAM): It is semiconductor memory consisting of transistors which can remember the information.

42. 42 These transistors do not require periodic charging to maintain the information. It is read/write type. The organization is array type to facilitate read and write operations. SRAM access speed ranges from a few nanoseconds to 30 nanoseconds. SRAM is volatile and loses the information when power is switched off. 4.DRAM (Dynamic RAM): it is semiconductor memory where information is stored in a capacitor. The term dynamic is used because capacitors require periodic charging to maintain the information. This process is known as periodic refreshing.

43. 43 Capacitors are used as memory cells and can achieve high cell density. The trade-off to high density is periodic refresh. DRAM is mainly used as main memory of the computer. The access speed ranges 50-80 ns. It is volatile and the information is lost if no power or no refresh. 5.VRAM ( Video RAM): It is like DRAM. The only difference is that it is dual-ported. The CPU port ( processor port) is standard port, similar to DRAM, containing data path and address path. In addition, there is a video port. The video port contains a buffer to hold a complete row of data.

44. 44 This buffer is organized in such a way that it can hold a complete data for a horizontal line. Each horizontal line represents one row of the screen data. The advantage of the VRAM is that the whole horizontal line of video screen information is loaded into the buffer in one scoop. Buffer’s output is then converted from parallel to serial and output as a video stream. With dual porting screen, updates can be done in almost half the time. With VRAM, the port to the CPU is available 90% of the time to do the updates.