ENREVIIST ENREVI ENhanced REality for the VIdeo October 4 Concertation

ENREVIIST ENREVI :The partners Symah Vision(.fr) as project co-ordinator and bringing expertise in tracking and video systems. Peak(.au) bringing expertise in 3D real time synthesis. Crystal Vision(.uk) in advanced keying know-how. Fraunhofer Institut (.de) for advance tracking research.

ENREVIIST The development objectives: Improving real time 3D camera tracking in an undefined environment Improve keying techniques in particular in undefined environment such as outside the studio. Provide real time rendering software for 3D objects with broadcast quality, in particular in respect to anti-aliasing techniques, on affordable hardware platforms. Provide system integration and ergonomics in concordance to the professional standards.

ENREVIIST Technologies involved: 2D and 3D graphics design. Computer vision. Robotics and sensors. Video professional system integration.

ENREVIIST Objectives reformulation: Toward scene description authoring Tracking/Keying : Scene description ==> MPEG4 authoring. Low cost 3D rendering addresses MPEG4 player graphic capabilities ENREVI system layer: how to store transport and retrieve specific metadata: camera position, camera calibration, camera movement, object position in camera FOV.

ENREVIIST Summary of project Status System specifications and interfaces (SV) Tracking options and final specification (IGD) Status on low cost rendering options (Peak+SV) Detailed specifications on Keyer (Crystal) Achievements, problems and prospects

ENREVIIST System specifications and interfaces (SV)

ENREVIIST Camera (Real Scene) Tracking Module Control Module Keyer Renderer Display of Enhanced scene Video Data Network Storage Transport Virtual Objects Design Inside project Outside Project ENREVI General System Architecture

ENREVIIST Communications library : The development status (2) Communication with PEAK software (Rendering): Shared MEMORY Peak rendering Tracking independent interface Daemon for X Tracking Tracking independent interface SV communication library Daemon for ENREVI Tracking

ENREVIIST Communication library : The development status (3) Communication with IGD software (Tracking): IGD Tracking Algorithm IGD tracking API SV tracking application SV communication library Remote control +data

ENREVIIST Communication library : The development status (4) Communication with Crystal Vision software (CK): Cristal Vision dedicated Hardware Crystal Vision TCP/IP or RS485 Communication protocol SV Chroma Key control GUI SV communication library (Optionnal) Remote control +data

ENREVIIST We tried : –Native libraries : XVT DSC++, Twin, YAAF, Zinc, Microsoft MFC, WxWindows, Open UI, CPAT and XVT DSC++ but the conclusion is : no native library is simultaneously platform independent, easy to use and complete. –Interpreted languages : Two serious candidates are available : TCL/TK and JAVA. The two complete the requirements. Graphics Library : the choice

ENREVIIST Specifications for communication Library Bi-directional data transmission. Real time transmission. Synchronisation of data by time stamping Real time losses compensation. Multiple application on the same machine. User defined data

ENREVIIST Standardisation activities Focused on MPEG4 during 6 first months French delegation registration WG11 meetings participation Outside Enrevi: –Development of an Authoring Tool for scene description using BIFS syntax –Development of a Production Tool for « on line » scene updates

ENREVIIST Tracking options and final specifications (IGD)

ENREVIIST General Steps Calibration of Stereo System Calibration of Hybrid System Tracking –Detection/tracking of 2D-coordinates of scene features –Calculation of 3D-coordinates of scene features –Calculation of camera parameter for each frame

ENREVIIST Tracking II Tracking is optimization of camera parameter for each frame Prediction can be done in several ways: –polynomial extrapolation –motion models (e.g.kalman filter) –external sensors (inertial, magnetic) Outlier detection by using M-estimators

ENREVIIST D Reconstruction I From matrix F the camera projection matrices for each image can be computed : P and P, w.r.t the chosen projective basis. Reconstruction: E matrix is obtained from F matrix E is decomposed in a rotation (R) and translation (T) matrix P i is established.

ENREVIIST System Design Stereo Calibration C1 C2 Feat 1Feat 2 n frames Calib ext./ int. Comm Modul

ENREVIIST System Design Reconstruction C1 C2 n frames Calib ext./ int. Comm Modul 3D-Recon- struction 3D-Coor- dinates Feat 1 Feat 2

ENREVIIST System Design Tracking C1 C2 n frames Optimi- zation Comm Modul 3D-Coor- dinates Calib ext./ int. Feat

ENREVIIST Next Steps Finalising and testing of implementation of calibration method (stereo, hybrid) Work on self calibration Improving image processing Implementing and testing reconstruction methods First tests of tracking in natural scenes

ENREVIIST Status on low cost rendering options (Peak+SV)

ENREVIIST Low cost HW for renderer The renderer HW main requirements: –Low cost (inferior to 20 kE). –Compatible with Linux. –Powerful graphics accelerator. –Real time mix 3D/video. The candidates are : –SGI Linux workstations : no video device is planned –integrated video/3D cards like MATROX SG2000 and the wildcat –Home made solution based on nVidia GeForce II. SV explores the home made solution in order to: –have a back up solution. –Have rackable HW –be independent from a integrator (like SGI or Intergraph)

ENREVIIST HW for renderer : the home made solution (1) The video environment Video Out Keyer Channel 422 out 3D Renderer Tracking Data ( Video Texture In ) Background video

ENREVIIST HW for renderer : the home made solution (2) The principle: Video and 3D acceleration are processed by separated devices. 2 D1 boards provide video output in D format. One card is dedicated to the 422 video and the other one to the alpha channel. These cards load their video content in real time from the mother boards RAM by DMA transfers. A nVidia GeForce II accelerates the 3D rendering. The result buffer is transferred to the mother board.

ENREVIIST HW for renderer : the home made solution (3) The architecture System Memory Textures 1.0 GB/s Pentium® III processor Core 2.0 GB/s L2 Cache Intel i840 AGPset 1.0 GB/s Nvidia Quadro Graphic Board 1.0 GB/s AGP 4x 4.47 Gb/s Local Memory Resident Textures Graphic Board DVS Alpha Out DVS Video Out PCI 132 MB/s Personnal Computer Keyer Live Video In Broadcast Video Out

ENREVIIST HW for renderer : the home made solution (4) Conclusion : This architecture is sufficient for simple 3D rendering. But for advanced 3D products like Everest the lost of 50% of the graphics processing power may be unacceptable.

ENREVIIST Detailed specifications on Keyer (Crystal)

ENREVIIST ENREVI Keyer Functionality Block Diagrams (Mark 2 - June 2000) Top Level SDI Input SDI Output OPEN Architecture CPU and Communications Linear Key Generator Functionality of ENREVI chroma keyer (Mark 2 prototype)

ENREVIIST GPI Input RS422 To Control Panel RS422 From Control Panel RS422 Receiver Hitachi H8 Microcontroller RS422 Transmitter 16K Bits Non-volatile RAM 32K Bytes SRAM 1M Byte EPROM Watchdog and Reset Generator To/From Coefficient Stores Program Port For Programmable Logic GPI Input Processing CPU and Communications Block Diagram of ENREVI Keyer Functionality of ENREVI chroma keyer (Mark 2 prototype)

ENREVIIST Mai n Out Foreground In Unipolar to Bipolar Data Conversion Wipe Generator Rectangular Mask Generator Matte Generator Select Foreground Main Key Processing Engine Background Multiplier Select, Add and Multiplex Main Output Preview Out Select and Multiplex Preview Output Coefficients Store From CPU Background In External Key In Suppressed Foregroun d In Chroma Key In Unipolar to Bipolar Data Conversion External Key Processing Non Additive Mix Strip and delay Non- active Picture Information Combine Active and Non-active Picture Information Mix Generator Fade Generator Self Key Generator Black Generator Chroma Key Processing Foreground Multiplier Linear Key Generator Block Diagram of ENREVI Keyer

ENREVIIST Detailled Project status WP1 Management WP2 System architecture WP3 Tracking WP4 Keying WP5 Rendering WP6 System and Integration

ENREVIIST Hiring of engineers in direct relation with project accounts to 5 people Equipment related to project was purchased by partners. Web site of project has been completed and permanently updated. Symah Vision entered into the national initiative for standardisation (AFNOR). Contact with main national active bodies have been initiated and SV personnel attented several meetings Review documents after month 6 have been posted to private section of web site. Activity is generally in line with plan. WP1 Project management

ENREVIIST WP1 (Project management) Initial payment was transferred to consortium partners Consortium agreement is being circulated PISTE project has been contacted to study concertation potential Concertation subgroup on AR and VR imaging has been formed under the leadership of Fabio LAVAGETTO from University of Genova. Dissemination strategy:. Technical paper should be ready by october to be submitted to adequate conferences.

ENREVIIST WP2 (System studies, specifications) First version of system specifications being revised ans updated. Preliminary experiments on new hardware plateforms have been performed. Linux investigations are completed. Evaluation of various Digital Video I/O cards for industry standard PCs in under way. Concept of how to antialias 3D objects in OpenGL has been studied. Documentation of interface tracker-renderer finalized. Documentation of external control for renderer finalised. Documentation of Keyer external control finalised.

ENREVIIST WP3 (Tracking development) Selection and ordering of components for the tracking subsystem Continuing investigation of hybrid tracking systems; selecting of promising approach and starting of development of detailed mathematical formulation Starting of work on extending of existing marker-based tracking algorithm to achieve more stability, robustness and speed. The work that will be performed includes: Improvement of robustness of marker detection Making algorithm platform independent Improving speed by parallizing The marker-based algorithm will be as a reference algorithm for verifying the new non-marker based tracking algorithm and the used algorithms for stereo and hybrid calibration. Continuing investigation of stable, reliable and practicable approaches for calibration of stereo- systems (e.g. self calibration, calibration from planar patterns) Continuing of investigation of techniques and algorithms for feature tracking (points, lines, line segments) Some High frequency tracking research contribution performed by SV.

ENREVIIST WP4 (Keying development) Mathematical model for the chroma keyer completed in January Programmable logic coding for 1 st version started in February 1 st Prototype was built in April Chroma keying demonstrated on new hardware. All basic functions working. Select colour, acceptance angle, chrominance suppression. Y suppression, suppression angle. Purchase Altera FIR filter software to design better interpolation filters. Start using new generation Altera design software (Quartus) despite it being unfinished. Better release promised in September. Update hardware design in June to include new Altera logic family. PCB layout for new design.Demonstrate new Altera design using 2 prototypes (1/2 processing on each) Transfer new Altera design to 20K logic ready for new hardware is completed.

ENREVIIST WP5 (3D real time rendering) WP5 (3D real time rendering) Peak and Symah have initiated discussion on various tracks to generate broadcast quality genlocked video signal from general purpose platforms. Implementation of a first step for realtime antialiasing in OpenGL on a PC. First system studies for porting renderer to Linux. First tests with video cards from Matrox and Pinnacle Symah Vision has obtained some results in preliminary architecture of low cost 3D real time renderer. Peak described how they plug their renderer to external trackers : a deamon external to the application takes the data coming via the tracking providers protocol (ex : SV communication library) and send them into a memory shared by the renderer. The market for low cost renderers has been shacked by discontinuation of SGI NT solutions and change of Intergraph/Wildcat strategy. SV is proposing to try a home made solution where video and 3D acceleration are processed by separated devices. The corresponding architecture has been reviewed at consortium meeting in Cambridge.

ENREVIIST WP6 (System Integration) Final demo scenario has been agreed: A real time insertion of 2D and 3D objects will be positioned on a mock-up of a stadium. The size would be approx 10 square meters. This solution has been preferred to an outdoor demo to allow possibility to implement the demo for IBC 2001.

ENREVIIST Issues Consortium Agreement late Cost statements dry run late Report on standardisation activity late No visibility on low cost renderer from the trade SV proposes to test home made solution with no global budget impact.

ENREVIIST Meetings Attended Concertation meeting in Brussels on 24 and 25 Jan 2000 Kickoff meeting on 26 and 27 Jan 200 in Insbrück National meeting of MPEG initiative in Nice on 7 Feb 2000 Meeting between Crystal and BBC on 14 Feb for preliminary specifications. Many meetings between Crystal, Peak and SV in Las Vegas during NAB in april, IGD was unfortunately unable to attend – 10 th to 13 th April European MPEG meeting in Noordwijkerhout on 22 nd and 23 rd March MPEG application meeting in Montreux from 23 rd to 25 th May Concertation meeting in Brussels 24 th and 25 th May Meeting SV/Peak in Innsbrück on 28 th June Meeting SV/IGD in Darmstad on 3 rd July Consortium meeting in Cambridge on 5 th and 6 th July 2000, all partners attented.