1. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy RTAI: a Linux extension to hard real time applications Dipartimento di Ingegneria Aerospaziale, Politecnico di Milano, via La Masa 34, 20156, Milano, Italy Paolo Mantegazza (mantegazza@aero.polimi.it)

2. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy Presentation overview Operational view of the hard real time definition. Operational view of the hard real time definition. Appropriate perspective of why custom off the shelf (COTS) solutions are usable for high demanding local/distributed real time systems. Appropriate perspective of why custom off the shelf (COTS) solutions are usable for high demanding local/distributed real time systems. Advantages in adopting a free open source distributed real time operating system. Advantages in adopting a free open source distributed real time operating system. The RTAI solution: The RTAI solution: overview of the core system (RTHAL/ADEOS) and services; overview of the core system (RTHAL/ADEOS) and services; description of the implementation and features of RTAI middleware layer (net_rpc); description of the implementation and features of RTAI middleware layer (net_rpc); presentation of RTAI-Lab: a comprehensive middleware tool to develop and monitor local/distributed RTAI applications. presentation of RTAI-Lab: a comprehensive middleware tool to develop and monitor local/distributed RTAI applications. DIAPM and non-DIAPM RTAI applications. DIAPM and non-DIAPM RTAI applications. Concluding remarks. Concluding remarks.

3. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy Soft and hard real time Computer Science literature definitions: Computer Science literature definitions: a soft real time (SRT) system is characterized by its ability to execute a task according to a desired time schedule on the average (average determinism) a soft real time (SRT) system is characterized by its ability to execute a task according to a desired time schedule on the average (average determinism) video display an hard real time (HRT) system embodies guaranteed timing, cannot miss deadlines and must have bounded latencies (strict determinism) an hard real time (HRT) system embodies guaranteed timing, cannot miss deadlines and must have bounded latencies (strict determinism) plant controlled by a digital controller Strict determinism is possible only with dedicated CPU, i.e. DSP and DSP like microcontrollers, having a guaranteed interrupt latency in the range of a single/few execution cycles. Strict determinism is possible only with dedicated CPU, i.e. DSP and DSP like microcontrollers, having a guaranteed interrupt latency in the range of a single/few execution cycles.

4. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy Hard real time using COTS hardware GPCPUs have non deterministic GPCPUs have non deterministic latencies and jitter due to: latencies and jitter due to: virtual memory and MMU; virtual memory and MMU; multi level caches; multi level caches; bus arbitration from intelligent I/O subsystems; bus arbitration from intelligent I/O subsystems; high depth piped execution and speculation. high depth piped execution and speculation. GPCPUs can meet high requirements at unprecedented cost/performance figures (mega/giga FLOPS capabilities). GPCPUs can meet high requirements at unprecedented cost/performance figures (mega/giga FLOPS capabilities). Even higher performances using low cost Multi Processors and networked GPCPUs. Even higher performances using low cost Multi Processors and networked GPCPUs. Is it viable the solution of adopting COTS hardware, like general purpose CPUs (GPCPUs), for HRT systems?

5. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy High precision profiling machine example MIMO controller at 10 KHz with some form of adaptivity, force control and active vibration suppression loop forever acquire data (sensors/commands) compute control action output data (actuators/logs) wait for the next sample (latency computation) Worst case latencies in the range of few  s if the controller is run standalone on a GPCPU. Worst case latencies in the range of few  s if the controller is run standalone on a GPCPU. Typically the controller has to work cooperatively in a prioritised fully preemptable multitasking execution environment, e.g. for data visualisation, monitoring, logging and communication. Typically the controller has to work cooperatively in a prioritised fully preemptable multitasking execution environment, e.g. for data visualisation, monitoring, logging and communication. Such a mix can lead to non deterministic latencies (jitter) up to 30  s (30% of the sampling period). Such a mix can lead to non deterministic latencies (jitter) up to 30  s (30% of the sampling period). A system having such a jitter can be still considered HRT? A system having such a jitter can be still considered HRT?

6. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy Hard real time specifications The finishing and tolerances of our machined items verify our top expectations, around the clock, seven days a week. The finishing and tolerances of our machined items verify our top expectations, around the clock, seven days a week. A comparison against any equivalent dedicated DSP implementation will show no difference in results. A comparison against any equivalent dedicated DSP implementation will show no difference in results. Our real time specifications must be related to the actual system bandwidth and only indirectly to the sampling rate. Our real time specifications must be related to the actual system bandwidth and only indirectly to the sampling rate. Our 10 KHz controller will likely be controlling a 1 KHz bandwidth system for which a worst case 3% timing uncertainty will produce measurements and command errors that are guaranteed to be filtered, down to an acceptable noise level, by the inherent system low pass attenuation. Our 10 KHz controller will likely be controlling a 1 KHz bandwidth system for which a worst case 3% timing uncertainty will produce measurements and command errors that are guaranteed to be filtered, down to an acceptable noise level, by the inherent system low pass attenuation. No clear cut between hard and soft real time but simply a continuous transition from low to high bandwidths. No clear cut between hard and soft real time but simply a continuous transition from low to high bandwidths.

7. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy Hard real time using FOSS Real time operating systems (RTOS) running on GPCPUs will afford equivalent performances because these are dominated by the architectural features of GPCPUs. Real time operating systems (RTOS) running on GPCPUs will afford equivalent performances because these are dominated by the architectural features of GPCPUs. At this point FOSS RTOSes can have an edge, with the added advantage of full code sources availability. At this point FOSS RTOSes can have an edge, with the added advantage of full code sources availability. Is it possible to meet the required HRT specifications within an existing truly FOSS operating system such as Linux? Is it possible to meet the required HRT specifications within an existing truly FOSS operating system such as Linux? Linux is a non preemptable UNIX like OS, with superb performances as a general purpose computational/desktop/server OS, but with limited soft and hard real time capability. However, it offers a very good base to which HRT can be added. Linux is a non preemptable UNIX like OS, with superb performances as a general purpose computational/desktop/server OS, but with limited soft and hard real time capability. However, it offers a very good base to which HRT can be added. This goal is achieved easily setting a strict separation of hard and non hard real time processes, while affording efficient communication and synchronization services to allow an effective interaction between the two environments. RTAI follows such a concept. This goal is achieved easily setting a strict separation of hard and non hard real time processes, while affording efficient communication and synchronization services to allow an effective interaction between the two environments. RTAI follows such a concept.

8. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy RTAI core system: RTHAL/ADEOS A Real Time Hardware Abstraction Layer (RTHAL) is installed that gathers all the pointers to the time critical kernel internal data and functions into a single structure so that they can be dynamically substituted by RTAI when HRT is required. Linux is changed to use what pointed in RTHAL for its operation. The Adaptive Domain Environment for operating Systems (ADEOS) nanokernel is installed that implements a pipeline scheme into which every domain (OS) has an entry with a predefined priority. RTAI is the highest priority domain which always processes interrupts before the Linux domain, thus serving any hard real time activity either before or fully preempting anything that is not hard real time. RTAI is integrated into Linux through a kernel patch and a series of add on programs (loadable modules) expanding the Linux kernel to hard real time. The patch can be RTHAL or ADEOS based.

9. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy RTAI schedulers Fully preemptable. Fully preemptable. Can be scheduled from within interrupt handlers so that Linux has no chance of delaying any RTAI hard real time activity. Can be scheduled from within interrupt handlers so that Linux has no chance of delaying any RTAI hard real time activity. Allow to symmetrically work inter-intra both user and kernel space by using the same set of APIs everywhere. Allow to symmetrically work inter-intra both user and kernel space by using the same set of APIs everywhere. Optimised for kernel uniprocessor (UP), kernel symmetric multiprocessors (SMP), kernel multi uniprocessor (MUP), integrated UP/MP kernel/user space processing (LXRT). Optimised for kernel uniprocessor (UP), kernel symmetric multiprocessors (SMP), kernel multi uniprocessor (MUP), integrated UP/MP kernel/user space processing (LXRT). Possibility of choosing between a base periodic timing, with a fixed assigned time resolution tick, and an arbitrary timing, allowing scheduling a task at the resolution of the available clock by firing a oneshot timer at the time instant imposed by the highest priority task waiting on the timed list. Possibility of choosing between a base periodic timing, with a fixed assigned time resolution tick, and an arbitrary timing, allowing scheduling a task at the resolution of the available clock by firing a oneshot timer at the time instant imposed by the highest priority task waiting on the timed list. Different scheduling policies: FIFO, Round Robin, Early Deadline First. Different scheduling policies: FIFO, Round Robin, Early Deadline First.

10. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy User space interrupts RTAI services Mailboxes Binary/resource semaphore Bitssynchronization FIFOcommunications Tasklets and timers Distributed services (net_rpc) Direct intertask messages User space interrupts Basic task management

11. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy Distributed RTAI: net_rpc Small (20 KBs) and effective real time middleware layer to allow native usage of any of the RTAI APIs on distributed local/remote control nodes. Small (20 KBs) and effective real time middleware layer to allow native usage of any of the RTAI APIs on distributed local/remote control nodes. The integration is provided by just adding a node/port identifier in front of any RTAI function call. Thus any application can be run on a single machine or on many networked machines without changing a single line of source code. The integration is provided by just adding a node/port identifier in front of any RTAI function call. Thus any application can be run on a single machine or on many networked machines without changing a single line of source code. Lean implementation of the RPC idea. Lean implementation of the RPC idea. The scheme uses a stub (buddy, proxy, agent) task to execute the remote APIs. The scheme uses a stub (buddy, proxy, agent) task to execute the remote APIs. Stubs and ports are statically connected to guarantee maximum determinism. Stubs and ports are statically connected to guarantee maximum determinism. All services are executed in a strict synchronous request_service/return_result mode, asynchronous usage is also possible. All services are executed in a strict synchronous request_service/return_result mode, asynchronous usage is also possible. Port requests/releases are not real time operation. Port requests/releases are not real time operation.

12. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy Distributed RTAI: net_rpc (cont.) Ports are owned by a task and a task can own multiple ports having different identifiers. Ports are owned by a task and a task can own multiple ports having different identifiers. Capability of dynamically expanding the services at run time. Capability of dynamically expanding the services at run time. Actual available support of low level net service through UDP, with dedicated connections and switches. Actual available support of low level net service through UDP, with dedicated connections and switches. Sharing Linux networking possible but forgetting real time performances. Sharing Linux networking possible but forgetting real time performances. Ethernet UDP protocol is non strictly deterministic but using a dedicated high speed hardware guarantees acceptable performances for low to medium bandwidth, say up to 2 KHz, distributed controllers. Ethernet UDP protocol is non strictly deterministic but using a dedicated high speed hardware guarantees acceptable performances for low to medium bandwidth, say up to 2 KHz, distributed controllers. Higher bandwidth at an increased cost (GigaEthernet or SCI adapter) or using the free open source project RTNet. Higher bandwidth at an increased cost (GigaEthernet or SCI adapter) or using the free open source project RTNet. Security? Security?

13. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy RTAI-Lab Provides a common structured framework to design, build, run and monitor locally/remotely any suite of RTAI-based single/multitasking real time applications, either specifically coded in a high level procedural language (C/C++), or automatically generated by proprietary packages (MATLAB/Simulink/Real-Time-Workshop), and/or fully open source projects such as SCILAB/Scicos/CodeGen. Provides a common structured framework to design, build, run and monitor locally/remotely any suite of RTAI-based single/multitasking real time applications, either specifically coded in a high level procedural language (C/C++), or automatically generated by proprietary packages (MATLAB/Simulink/Real-Time-Workshop), and/or fully open source projects such as SCILAB/Scicos/CodeGen. Allows two separate systems, the host and the target(s), to communicate for changing/tuning parameters on the fly and/or monitoring/logging signals. Allows two separate systems, the host and the target(s), to communicate for changing/tuning parameters on the fly and/or monitoring/logging signals. The host is the machine where RTAI-Lab is executing in soft real time; the target(s) is/are the machine(s) where the local/distributed generated code runs in hard real time. The host is the machine where RTAI-Lab is executing in soft real time; the target(s) is/are the machine(s) where the local/distributed generated code runs in hard real time.

14. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy Matlab case Simulink model (model.mdl) Real-Time Workshop build process 1) Target Language Compiler - rtai.tlc (model.c, model.h) 2) Template Makefile - rtai.tmf (model.mk) Real time executable(model) Making process make, gcc RTAI-Lab Graphical User Interface

15. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy DIAPM control applications (1) Tracking control of a flexible manipulator Structural flexibility of the links. Structural flexibility of the links. Integrated active damping of structural vibrations. Integrated active damping of structural vibrations. Few sensors (angular potentiometers at the joints and strain gauges along the links) and actuators (direct drive brushless motors) to limit weight and costs. Few sensors (angular potentiometers at the joints and strain gauges along the links) and actuators (direct drive brushless motors) to limit weight and costs.

16. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy DIAPM control applications (2) Fatigue test system High flexibility in programming arbitrarily complex coordinated multi-load histories. High flexibility in programming arbitrarily complex coordinated multi-load histories. Load signals generation, test results logging, display and recording Load signals generation, test results logging, display and recording Optimisation of test conditions and controller parameters Optimisation of test conditions and controller parameters

17. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy DIAPM control applications (3) MPWM control of the vibrations of a large space structure Modular beam-like (19 m) plastic tubes-made (75 Kg) truss suspended from the ceiling by means of six soft springs. Modular beam-like (19 m) plastic tubes-made (75 Kg) truss suspended from the ceiling by means of six soft springs. Equipped with six pairs of on/off air jet thrusters operating within the horizontal plane, and six colocated capacitive accelerometers. Equipped with six pairs of on/off air jet thrusters operating within the horizontal plane, and six colocated capacitive accelerometers. Multi PWM deadbeat predictive control law. Multi PWM deadbeat predictive control law.

18. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy DIAPM control applications (4) Active control of combustion instabilities on a Rijke tube

19. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy DIAPM control applications (5) Reduction of vibrations transmitted to helicopter airframes using active gearbox struts Agusta A109 mockup Agusta A109 mockup Active gearbox struts provided with surface bonded piezoelectric patches Active gearbox struts provided with surface bonded piezoelectric patches Feedforward multi-input multi-output FXLMS control algorithm. Feedforward multi-input multi-output FXLMS control algorithm.

20. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy Non-DIAPM control applications Marble and granite automatic cutter machine Marble and granite automatic cutter machine Ball bearing rings shape control Ball bearing rings shape control Laser cutting machine Laser cutting machine Robot control Robot control Universal measuring machines Universal measuring machines Electric power conversion Electric power conversion Electroerosion machines Electroerosion machines Plastic deformation machines Plastic deformation machines

21. RTAI: a Linux extension to hard real time applications Politecnico di Milano - Italy Concluding remarks Since its very beginning RTAI has grown in an impressive way in robustness, services and programming tools. Since its very beginning RTAI has grown in an impressive way in robustness, services and programming tools. RTAI is a FOSS development effort with a team of developers worldwide. RTAI is a FOSS development effort with a team of developers worldwide. RTAI numbers many academic and industrial applications and its usage is more and more expanding. RTAI numbers many academic and industrial applications and its usage is more and more expanding. RTAI might still lack some of the bells and whistles of commercial RTOSes, but it does not fear any performance challenge. RTAI might still lack some of the bells and whistles of commercial RTOSes, but it does not fear any performance challenge. Bugs are always there: with FOSS, if nobody cares, you are free to loose some sleeping and fix bugs yourselves (feeling proud to contribute your fixes back to help the community). Bugs are always there: with FOSS, if nobody cares, you are free to loose some sleeping and fix bugs yourselves (feeling proud to contribute your fixes back to help the community).