THE EYESWEB PLATFORM - GDE The EyesWeb XMI multimodal platform GDE 5 March 2015

THE EYW GDE THE EYESWEB PLATFORM - GDE

THE EYESWEB GDE (II) THE EYESWEB PLATFORM - GDE

THE MENU BAR THE EYESWEB PLATFORM - GDE

THE MESSAGE CONSOLE The Message Console is part of the GDE. It is represented by a frame where messages, errors and warning notifications are displayed. I It logs internal state information about data and blocks of a patch at design time and runtime. I It logs debug information that are needed to be displayed for tracing the flow of the elaborating data value. THE EYESWEB PLATFORM - GDE

THE MESSAGE CONSOLE (II) THE EYESWEB PLATFORM - GDE

THE WORKSPACE In the workspace, drag the blocks in order to link themtogether and create a patch. THE EYESWEB PLATFORM - GDE

THE WORKSPACE (II) - PATCHES AND BLOCKS THE EYESWEB PLATFORM - GDE

THE WORKSPACE (III) - PINS THE EYESWEB PLATFORM - GDE Input pins represent the data arriving at the block from another block, in order for the block to process it. Output pins represent data being sent from the block to some other block, after processing. Parameter pins represent information or data arriving at the block, which affects the processing work carried out by the block.

EYESWEB DATATYPES THE EYESWEB PLATFORM - GDE Each input, output, or parameter pins are associated to a datatype. Datatypes define the set of values that can be associated to a pin and the operations that can be performed on such values. Conversion rules are used to establish compatibilities among different datatypes. A generic datatype is also provided by EyesWeb. For each datatype, EyesWeb usually provides blocks for generating it, serialize it (i.e. load/save to file), process it.

COMMON DATATYPES AND BLOCKS THE EYESWEB PLATFORM - GDE Common data types may include: integers, floating-point numbers, alphanumeric strings, dates, times, video streams, audio streams

LINKING BLOCKS THE EYESWEB PLATFORM - GDE Blocks can be linked to one another by piping the output pin of one block to the input or parameter pin of another block. Block A can be linked to block B if and only if an output datatype of block A is compatible with an input datatype of block B. An input pin requiring a generic datatype can be connected to whatever output pin whose datatype provides the interfaces (operations) required by the receiving block. When positioning the mouse on a pin, the EyesWeb status bar shows the associated datatype.

LIBRARIES THE EYESWEB PLATFORM - GDE The EyesWeb environment includes libraries of blocks, which can be used through a drag-and-drop procedure to create new patches. Blocks functionalities roughly include: o Input and output; o Mathematical operations; o Machine Learning o Signal operations; o Data processing functions (image, audio, 3D, depthmaps...); o Management / flow control functions;

LIBRARIES (II) THE EYESWEB PLATFORM - GDE Kernel blocks and datatypes are statically linked against the kernel. Blocks and datatypes are contained in separate libraries, included in catalogs: o User catalogs: Base, System, and most third parties modules are contained in catalogs which are registered through the Eyw RegisterModule tool (catalog stored in xml files) o Catalog factories: Steinberg, FreeFrame, Legacy use a custom (not xml-based) mechanism to store the catalog of modules (e.g., legacy uses Windows registry, FreeFrame enumerates files in some given folders, etc)

LIBRARIES (III) - THE LIBRARY VIEW THE EYESWEB PLATFORM - GDE Blocks are grouped in a collection of libraries. Top level libraries:

THE SEARCH VIEW THE EYESWEB PLATFORM - GDE A search window option is available to help the user find a particular block.

THE PROPERTIES VIEW THE EYESWEB PLATFORM - GDE

THE PARAM SHEET THE EYESWEB PLATFORM - GDE

THE DESCR(IPTION) SHEET THE EYESWEB PLATFORM - GDE

THE EYESWEB PLATFORM - KERNEL The EyesWeb XMI multimodal platform The EyesWeb Kernel 5 March 2015

THE EYW KERNEL THE EYESWEB PLATFORM - KERNEL

THE EYESWEB KERNEL (II) It is the core of the EyesWeb platform. It is implemented as a Dynamic Link Library, in order to be reusable in different applications. Responsibilities: o Register the available modules in the system; o Manage patches; o Load/Save patches (as Xml files; however other formats are supported too); o Run patches (engine/scheduler); o Provide network interface for remote GUIs (EyesWeb o Mobile). THE EYESWEB PLATFORM - KERNEL

THE EYESWEB KERNEL (III) - STATE MACHINE DIAGRAM THE EYESWEB PLATFORM - KERNEL

THE EYESWEB KERNEL (IV) - STATE MACHINE DIAGRAM THE EYESWEB PLATFORM - KERNEL

EYESWEB STATES Design time: the user is editing the patch, or the patch is being loaded/saved. Runtime: the patch is being executed: o Init: first initialization phase, blocks allocate resources or initialize devices; o Check: second phase of initialization, topological sort, loops check, input/output check; o Start: the patch is ready for the execution, notify all blocks that patch is going to start, start devices data capture; o Stop: the patch execution is interrupted (usually on user request). Output streams and devices are closed o Done: block releases resources then the application goes back to design time. THE EYESWEB PLATFORM - KERNEL

EYESWEB SCHEDULER - RUNNING Works during runtime (or on Initialization/Check). Its activation can be: o Time Based: Periodic, on absolute time. o Event Based: new input event, user defined event. o On Request: e.g., interruptions for data from external devices, triggered from parameters.. o Heterogeneous: any combination of the above options. Triggers blocks execution, the order of execution is given by the patch topology THE EYESWEB PLATFORM - KERNEL

TOPOLOGICAL SORT IN EYESWEB Blocks are sorted with a topological algorithm. Loops of links are forbidden, the following exceptions are allowed: o links connected to a parameter pins. o links connected to input pins that were declared as not necessary for execution. Loops are managed through specifically designed blocks. The solution is not unique. THE EYESWEB PLATFORM - KERNEL

TOPOLOGICAL SORT IN EYESWEB (II) - EXAMPLE THE EYESWEB PLATFORM - KERNEL

TOPOLOGICAL SORT (III): PATCHES AND LOOPS THE EYESWEB PLATFORM - KERNEL This block has two inputs: o the first input is required for execution o the second item is not required for execution I In this case the graph is cyclic: links B -> C, C -> B are involved in the loop. Since the first input pin of block B is required, the topological sort fails. The execution of the patch is interrupted.

TOPOLOGICAL SORT (IV): PATCHES AND LOOPS THE EYESWEB PLATFORM - KERNEL As before, this patch contains a loop: links B -> C, C -> B are compose the loop. Since the second input pin of block B is not required, link C -> B is ignored by the topological sort. The sorting algorithm succeeds. In this case link B -> C, that is part of a loop, connects to a parameter. The B -> C link is ignored by the topological sort. The sorting algorithm succeeds.

TOPOLOGICAL SORT (V): PATCHES AND PARALLELISM THE EYESWEB PLATFORM - KERNEL Two possible ordered sequences exist o ABCDE or o ACBDE In such cases parallelism can be exploited. A [B | C] DE, where B and C are executed in parallel on multicore / multiprocessor computers.