Semantic Web Services for Smart Devices based on Mobile Agents Vagan Terziyan Industrial Ontologies Group University of Jyväskylä

2 of 40 ContentContent Resources in Semantic Web and Beyond Global Understanding Environment Resource Adaptation Remote Diagnostics of Resources Resource Maintenance and Networking Mobile Service Components (Agents)

3 of 40 Semantic Web in Networked Business Environment Semantic Web technology provides standards for metadata and ontology development such as semantic annotations (Resource Description Framework) and knowledge representation (Web Ontology Language). It facilitates interoperability of heterogeneous components, authoring reusable data and intelligent, automated processing of data. Semantic Web is an enabling technology for the future Networked Business Environment “In a networked business environment Metso will be a business hub controlling the flow of information in the network of installed Metso devices and solutions, and Metso’s customers and partners.” Networked Business Environment requires new advanced ways of data and knowledge management Industrial Maintenance domain is a good application case for the concept of the Networked Business Environment Networked Maintenance Environment Networked Maintenance Environment will bring all benefits of the knowledge management, delivering value-added services and integration of businesses

MAIN RESEARCH OBJECTIVE to combine the emerging Semantic Web, Web Services, Peer-to-Peer, Machine Learning and Agent technologies for the development of a global and smart maintenance management environment, to provide Web- based support for the predictive maintenance of industrial devices by utilizing heterogeneous and interoperable Web resources, services and human experts

5 of 40 resources Classes of resources in maintenance systems: Devices - increasingly complex machines, equipment, etc., that require costs-demanding support Processing Units (Services) – embedded, local and remote systems, for automated intelligent monitoring, diagnostics and control over devices Humans (Experts) – qualified users of the system, operators, maintenance experts, a limited resource that should be reused Industrial Resources

6 of 40 Smart Maintenance Environment “Devices with on-line data” “Experts” Maintenance On-line learning “Services”exchange data Maintenance data exchange

7 of 40 Self-maintenanceSelf-maintenance Do not expect that someone cares about you, take care yourself even if you are just an industrial device ! proactiveYou should be proactive enough to “realize” that you exist and want to be in a “good shape”; sensitiveYou should be sensitive enough to “feel” your own state and condition; smartYou should be smart enough to “understand” that you need some maintenance.

8 of 40 Resource Agents 2. “Yeah, your condition is not good. You need urgent help” 1. “I feel bad, temperature 40, pain in stomach, … Who can advise what to do ? “ 3. “Hey, I have some pills for you” Resource agents Resource agents are intelligent “supplements” of various resources. They represent these resources in Semantic Web-enabled environment and interoperate, realizing resource’s (pro-)active behavior

9 of 40 Research Challenges Resource Adaptation and Interoperability (Semantic Web )  Unify data representation for heterogeneous environment  Provide basis for communication Resource Proactivity and Mobility (Agent Technology)  Design of framework for delivering self-maintained resources to industrial systems Resource Interaction (Peer-to-Peer, Web Services technologies)  Design of goal-driven co-operating resources  Resource-to-Resource communication models in distributed environment (in the context of industrial maintenance)  Design of communication infrastructure

10 of 40 GUN Concept Global Understanding eNvironment

First Slice of Gun Architecture RESOURCE ADAPTATION

12 of 40 Research and Development: Resource State/Condition Description Framework (RSCDF) based on Semantic Web and extension of RDF (Resource Description Framework) RSCDF adapters (wrappers) for devices, services and experts: - browsable devices - application-expert interface - RSCDF -enabled services GoalsGoals Define Semantic Web-based framework for unification of maintenance data and interoperability in maintenance system “Device”“Expert”“Service” RSCDF

13 of 40 Generic Semantic Adapter A generic resource-access mechanism (semantic adapter) for devices, diagnostic services and humans An environment for remote access and resource browsing via semantic-based communication interface

14 of 40 Generic Semantic Adapter The integration requires development of the Generic Resource Adapter, which will provide basic tools for adaptation of the resource to Semantic Environment. It should have open modular architecture, extendable for support of variety low- and high-level protocols of the resources and semantic translation modules specific for every resource (e.g. human, device, database). Generic Resource Adapter must be configurable for individual resource. Configuration includes setting up of communication specific parameters, choosing messaging mechanism, establishing messaging rules for the resource and providing a semantic description of the resource interface. GUN- resource Communication-specific connector of a resource Resource-specific messaging Semantic “wrapping” of resource actions; translation of external messages into resource-native formats Connectivity Layer Semantic Layer GUN environment Generic Adapter configuration Messaging Layer

15 of 40 Semantic adapter for Devices API Semantic environment If to consider field devices as data sources, then information to be annotated is data from sensors, control parameters and other data that presents relevant state of the device for the maintenance process. Special piece of device-specific software (Semantic Adapter) is used for translation of raw diagnostic data into standardized maintenance data based on shared ontology. Shared ontology Adapter Semantic message Device-specific calls

16 of 40 Semantic adapters for Services Semantic environment The purpose of Service Semantic Adapter is to make service component semantic web enabled, allowing communication with service on semantic level regardless of the incompatibility on protocol levels, both low-level (data communication protocol) and high-level (messaging rules, message syntax, data encoding, etc.). Shared ontology Adapter Semantic message Service- specific calls

17 of 40 Semantic Adapters for Human-experts Human in the system is an initiator and coordinator of the resource maintenance process. The significant challenge is development of effective and handy tools for human interaction with Semantic Web-based environment. Human will interact with the environment via special communication and semantic adapter. User interface Human GUN- resource Action translated into semantic message Semantic message that will be visualized Shared ontology

Second Slice of Gun Architecture REMOTE DIAGNOSTICS

19 of 40 Adding agents to resources Making resource proactive Enabling communication with resource Implementation of agent-communication scenarios service learning remote diagnostics GoalsGoals Development of agent-based resource management framework and enabling meaningful resource interaction ”Adapter”“Expert”“Service” “Device” Resource Agent Smart Maintenance Environment Environment “Device”“Expert”“Service” Remote diagnostics Service learning and remote diagnostics Expert~Service Expert ~ Service

20 of 40 Device – Expert : interactions “Expert” “Device” Querying diagnostic results Labelled data Watching and querying diagnostic data Labelled data History data  Accepts semantic description of device state and can respond with classification label (semantic description of diagnosis)  Can make semantic query to request device-state data (also labeled history data), get response from Device and provide own label for observed device state Expert :

21 of 40 Device – Service, learning “Service” “Device” Querying data for learning Diagnostic model Learning sample Labelled data History data Learning process: creation of the Diagnostic Model

22 of 40 Device – Service, servicing “Device” Querying diagnostic results Labelled data “Service” Diagnostic model History data Labelled data

23 of 40 System structure “Expert” “Service” Labelled data Diagnostic model Querying diagnostic results Labelled data Watching and querying diagnostic data Labelled data History data “Device” Querying data for learning Learning sample and Querying diagnostic results Simple remote diagnostic model with semantic-based communication, expert and diagnostic service with learning capabilities.

Third Slice of Gun Architecture MAINTENANCE NETWORKING

25 of 40 GoalsGoals P2P agent-communication system Resource Discovery Maintenance Data & Knowledge Integration Certification and credibility assessment of services Resource Goal/Behaviour Description Framework Semantic modelling of a resource proactive behaviour Exchanging & integrating models of resource (maintenance) behaviour Development of networked maintenance environment GB

26 of 40 NetworkingNetworking

27 of 40 P2P networking - highly scalable - fault-tolerable - supports dynamic changes of network structure - supports dynamic changes of network structure - does not need administration - does not need administration Why to interact? 1. Resource summarizes “opinions” from multiple services; 2. Services “learns” from multiple teachers; 3. One service for multiple similar clients; 4. Resources exchange lists of services; 5. Services exchange lists of clients. - network of hubs

28 of 40 Notice boards Service 1 Service 2 Service 3 Client 1 Client 2 Client 3 Component advertisement solution Allows search for new partners Source of new entry points into P2P network Allows automated search based on semantic profiles

29 of 40 Discovery: sample scenario Number of queried peers is restricted due to: superhub based structure; query forwarding mechanism based on analysis of semantic profile; Resource Service Matched service Wrong service Response Query propagation

30 of 40 Learning and test sample. Querying diagnostic results. Devices: multiple services “Service” “Device” Labelled data Learning sample Test sample “Service” Diagnostic model w1w1w1w1 w2w2w2w2 w3w3w3w3 w4w4w4w4 w5w5w5w5 Evaluation and Result integration mechanism … Labelled data Device will support service composition in form of ensembles using own models of service quality estimation. Service composition is made with goal of increasing diagnostic performance.

31 of 40 Services: multiple devices “Service” Diagnostic model …“Device” Labelled data “Device” “Device” …“Device” “Device” “Device” 1 n Device-specific diagnostic model Device Class-specific diagnostic model Service builds classification model; many techniques are possible, e.g.:  own model for each device  one model from several devices of same type (provide device experience exchange)

32 of 40 Results of Networking Decentralized environment that integrates many devices, many services, many human experts and supports : Establishment of new peer-to-peer links through NoticeBoards, advertisement mechanism Semantic based discovery of necessary network components Service Interaction ”One service – many devices” Interaction ”One device – many services” Exchange of contact lists between neigbor peers

33 of 40 Device-to-Device “opinion” exchange Device Device 1 Device 2 Service 1 Service 2 trust = 100 trust = ? ? 4 8 Device will be able to derive service quality estimates basing on analysis of ”opinions” of other devices and trust to them. Service quality evaluations

34 of 40 Service-to- Service “model” exchange and integration Diagnostic models exchange Diagnostic models integration entails creation of a more complex model extension or a service with new diagnostic model

35 of 40 CertificationCertification Certifying party Device Service 1 Service 2 Service Own evaluations trust Sure, there are security threats as in any open environment. Security is to be ensured using existing solutions for Internet environment. Existence of certification authorities is required in the network. Certificates gained by services and trust to the certificate issuer are factors that influence optimal service selection. The quality of service is evaluated by users as well. Sure, there are security threats as in any open environment. Security is to be ensured using existing solutions for Internet environment. Existence of certification authorities is required in the network. Certificates gained by services and trust to the certificate issuer are factors that influence optimal service selection. The quality of service is evaluated by users as well.

36 of 40 Maintenance “executive” services Device Service Control Support for maintenance services that can influence on device state and perform maintenance actions upon it (automated control system, maintenance personnel). They complete the minimal working set of maintenance system components. data diagnosis control

37 of 40 “Expert” Network “Device” “Service” Diagnostic model “Expert” User interface Labelled data Labelled data Learning diagnostic results and querying sample Labelled data Querying diagnostic results Maintenance Networking Environment “Device” Network “Service” Network ”Adapter” “RSCDF Alarm Service” Sensor data RSCDF data ”Adapter” Sensor data RSCDF data “Embedded Alarm Service” Local (Embedded) Platform ”Adapter” RSCDF data RSCDF data Remote Expert Platform ”Adapter” Learning process RSCDF data Remote Service Platform History data Resource Agent

38 of 40 Service Platform Environment where service components perform: Condition monitoring Maintenance activities Maintenance Platform Environment to run Maintenance Services, contains a set of expert-agents both in maintenance and diagnostics. Agents are “service components” Internal and External Service Platforms

39 of 40 Mobility of Service Components Service Agents Host Agent Embedded Platform Based on the online diagnostics, a service agent, selected for the specific emergency situation, moves to the embedded platform to help the host agent to manage it and to carry out the predictive maintenance activities Maintenance Service

40 of 40 Conclusion: Summary of Concepts and Requirements Service Interaction ”One device – many services” P2P environment that integrates many devices, many services, many human experts and supports: Discovery of necessary network components using their profiles Interaction ”One service – many devices” Adaptation of resources (devices, services, experts) to the Environment Support for services that are able to learrn Unification of maintenance data Proactive and Mobile Resources Mobile Resource Agent GB