EdgeX Face to Face Working Agenda and Deck

EdgeX Face to Face Working Agenda and Deck
Fuji Release Seoul Korea, April 29 – May 2, 2019

WG Chairperson Meeting Day (chair invite only)
Architecture and Design Sessions for Specific Topics

Architect’s Day Topics (April 29th)
9am - Core and General (Conn) High Availability Application of 12 factor principles Service orchestration For non-HA scenarios Instrumentation Database persistence Decoupling from specific providers Long v short term storage (e.g. core-metadata v. core-data) Migration when models change General Topics Refactoring needs Store & Forward 11am – Application Services v2 (Johanson & Goodell) Next functions Rules Engine integration Command support Vault integration Binary data support Retiring Export Services How to support all the cloud systems Language bindings (C, C++, Node, Python) 1pm - Security (White) HW secure storage (TPM, TEE, …) How to insure the right services are running? Service-to-service comms 3pm – Swagger vs. RAML (Foster) 4pm – Device Service SDK Items not yet implemented from the req doc

EdgeX Training & Business Issues (open)
Introductory to Deep Dive Tutorials (April 30) Business and Other Non-Technical Topics (April 30)

Training and Business Topics Day (April 30th) Open to all
9am - EdgeX Introductory Tutorial (Hall) Basics - What and why of EdgeX Getting EdgeX Get it running Quick exploration of services & their purpose 10:45am – Getting started as an EdgeX developer (Conn) Getting EdgeX setup in GoLand Running EdgeX in development/debug environments Setting up and running blackbox tests 1pm – Building Application Services using the SDK (Johanson & Goodell) Getting the SDK Using the SDK to create an application service How/where to extend the SDK and the services Roadmap and the future of the application services 2:45pm – Building Device Services using the SDK (Tony/Toby/Cloud) Getting the Go SDK Using the SDK to create a simple DS High level overview of the functions Where/how to extend the SDK or DS 9am – EdgeX v1 (Steele) Marketing approach Referenceable clients 11am – Commerce Project (Corrion) 4pm – Getting to Certification (Thompson) Certification process Who will do the testing Implementation plan

EdgeX TSC 2-day F2F (open)
General Meetings ( May 1-2)

Day 1 May 1

Fuji F2F Meeting Agenda Day 1 – Planning & Scoping Day
9am Welcome and intro by Host and Chairman 10am Architecture issues tee-up Review and explanation of upcoming items of discussion for day 2 11:30am Edinburgh Wrap Up Planning 1pm Project Cadence Check I Release Naming 1:30pm Fuji Planning – what’s in/out Scope definition Work Group Kanban Creation 4pm Future Release Roadmaps - Geneva and beyond Long range scoping and roadmap review Day 2 – Architecture issues day 9am Developer Advocate Perspective 10am Architecture discussion and decisions 3pm Action item Review 3:30pm Lessons Learned and Chandler Meeting Changes

Introductions Host Welcome (Moonki) Logistics and schedules
Attendee introductions Chairman Welcome (Steele)

Architecture Issues Tee Up
10am Day 1

Architecture Issues Just teeing up the conversation today
Get familiar and discuss with fellow attendees before tomorrow Divided into 2 categories Tech Debt - things we need to fix or address architecturally from the existing code base Things we want to add (new features, functions, technology, etc.) Goal: resolve and target a release if necessary Priority to Fuji items Pick the release for other items

Tech Debt Issues More unit & integration tests
Identify the real short comings Release manager – we have a process, now what? Mongo & Redis – what should be our reference implementation database(s) Traceability – we have a correlation id. What’s next? Rules Engine replacement (the last of the Java) Is there a small size/simple rules engine replacement? Until one is found/devised we don’t exactly know how to deal with it in application services (TBD). In the meantime, we continue to use the Java/Drools implemention and we need application service example to call existing rules engine Environment support Windows development & 0MQ (any change??) ARM 32 support (any change??) Metadata needs to implement registration/callback mechanism for any metadata changes (ex: call me anytime the device changes) Value descriptors creation should be the responsibility of core not device services Events should be timestamped in nanoseconds

Tech Debt Issues part 2 Device Service improvements
Clean up of device profile Adding automatic discovery and provisioning (or at least a place in the SDK for DS creators to use to do automatic discovery and provisioning using blacklist/whitelist configuration). First – need requirements/design work to help define Need device service/device deregistration Cache of readings: to avoid hitting the device in some cases and to avoid sending data to core under some requests Gain an appreciation of EdgeX performance and size characteristics What are the resource requirements for EdgeX Answering the question: will it fit on my system? What is the speed of an EdgeX transaction? I.e. how long does it take for a sensor reading to be ingested, perform analytic logic, and trigger actuation on a device (and be exported to the north side). How many things can be attached to EdgeX at one time? How much data can be ingested and how often can it be sampled? These questions must be answered on real platforms (both Intel and Arm) – not just on a virtual environment. The results must also include information about the use case, data type, circumstances of execution The performance tests/harness/tool description must be made available for others to be able to replicate and/or extend How do we deal with the rate of code change going forward? Edinburgh is V1.0. We must beware of non-backward compatible changes We must find a cadence that allows consumers to keep up – example, Redis developing core services with Redis that replace those of the Mongo implementation. Go plugins – don’t appear to be on the Go roadmap We are checking with the Go community and groups like Hashicorp that is using Plugins – what is their vision and options thought Because of plugins issue, we don’t really know how to effectively bring on other implementations (ex: ObjectBox) without many issues We will look to move from RAML to Swagger (aka OpenAPI)

New features, tools, processes, etc. Issues
Need to handle the “forward” part of store and forward “Disconnected or situational” Store and Forward - need to support (via application services) use case where the application services are not able to connect to and pass data to the desired endpoint (where EdgeX has lost connectivity to the north) In this requirement, there are scenarios where all or a subset of unsent data is filtered and then sent when reconnected “Batch mode or deliberate” Store and Forward – need to support (via application services) use where the application services is always connected to the endpoint, but the use case requires not sending events all the time, but batching the events up periodically and sending them in bulk to the north Certification support Which services first, full certification vs self assessment, who pays, how managed Documentation versioning Semantic versioning Kubernetes (K8s and K3s) support How do we facilitate under scale of solutions? Naming of Snaps and other deployment artifacts going forward SMA’s responsibilities (and non-responsibilities) & design Versus registry/config, other services or outside systems LF Edge What are the projects and what is the potential impact

New features, tools, processes, etc. part 2
Application services and cloud providers Change in philosophy -> provide example code for the big 3 (Amazon, Azure, Google) We also need to support MQTTS/HTTPS Will require application services (and SDK) to use be able to use Vault for certs, tokens, etc. Schedule service will be used by app functions/service for triggering things like store/forward needs. We need the ability to incorporate scheduling as module/library or as REST call client May need additional scheduling options (CRON job support, etc.) We need to consider automatic/dynamic device

Deployment Artifact discussion – time permitting
Today We produce many services with a reference implementation deployment artifact for each (offering two different types: Docker and Snaps) Tomorrow Might we want to create deployment artifacts that contain multiple services? All of core services running in a single container/snap – simplifying deployment under circumstances/use cases Might we want to make/build some type of combined service? All core services in a single executable – for example – in order to improve performance, lower footprint, etc.

Edinburgh Planning Issue Resolution & Wrap up Planning
11:30am Day 1

Edinburgh Dates Freeze Date – May 28 Release Date – June 20

Strike Team Reports CBOR/binary support - Toby on point with Trevor and Janko to assist with core, supporting and export services changes Redis implementations of DB services - Andre & Trevor Initial performance (and load) testing infrastructure and results - Andy LF support for TIG stack Certification Program - Randy and Jim SDK Unit testing (to the point short of any major refactoring) - Steve SIGL Static Analysis - James (with LF support) Kong Upgrade - Tingyu Device services (Edinburgh public availability of Modbus, MQTT, BACNet, BLE, Virtual Device, SNMP) - Steve Dell to provide SNMP DS Security storing service secrets in Vault and protecting the master token - Jim and Tingyu with Intel resource assistance Documentation clean up - Michael with WG leads to nominate areas of need first and complete doc updates by deadline UI updates IoTech UI - Keith VMWare UI - Jim with Huaqiao UIs not able to be updated will be archived to the Delhi release

Issues, Roadblocks & Discussion
Critical Edinburgh items

EdgeX Release Cadence Semi-Annual Pulse Check
1pm Day 1

Cadence Check April & Oct remain target release months
Fuji – Oct 2019 Geneva – April 2020 Hanoi release – Oct 2020 “I” release – April 2020 Chairmen’s Honor proclamation of the new name by James Gregg, Lenny Goodell, & Mike Johanson F2F planning around time of completion of each release Chandler – Oct/Nov 2019 ??? – April 2020 (venue to be selected – nominations??) Conferences At least 2 x large marketing/promotional events (Hannover Messe, IoT SWC) At least 1 x developer focused event Internet of Things World – May

Chairman and TSC Elections
Current TSC terms run through 6/2019 The Contributors and Maintainers of each Working Group shall elect a Working Group Chair for their Working Group. Each Working Group Chair will serve as a voting member of the TSC. All Contributors and Maintainers in the Project shall elect three TSC at-large representatives to serve as voting members of the TSC. The voting members of the TSC will elect a TSC Chair annually TSC Chair is a voting member of the Governing Board, and will liaise between the Governing Board and technical leadership of the EdgeX Foundry Project. The TSC Chair must be representative of a member company

Definitions & Process Contributors: anyone in the technical community that contributes code, documentation or other technical artifacts to the EdgeX Foundry Project codebase. Process Typically managed over . Working Groups manage their own elections. LF can support if needed. Once the TSC has been put in place, the new TSC will take nominations for Chair. If there is more than one nomination, there will be a vote. One week for a nomination period, and one week for voting (if needed).

Fuji Planning Scope Discussions
1:30pm Day 1

Scope Only 3-4 month cycle Major themes Other efforts
Support High Availability/Better Distribution Improved Security First Certifications Application Services over Export Services Other efforts ???

Fuji – General (cross area)
In Move to Go 2.0 Upgrade to Mongo 4.0 If Mongo is still a reference implementation Apply “12 Factor Apps” to all services Services can live on any platform Services can have multiple instances Watchers/callbacks for config or data changes Swagger (in addition to RAML) API documentation Sponsor some hackathons Out ARM 32 support Windows development support 0MQ issue

Fuji – Core (and Supporting)
Out Consul 1.4 Blacklist/whitelist of devices Store and Forward ability When north side connectivity is lost Design/document vs implementation? High Availability Application of 12 factor principles Service orchestration For non-HA scenarios Instrumentation Database persistence Reference Implementation Database Selection Decoupling from specific providers Long v short term storage (e.g. core-metadata v. core-data) Migration when models change Refactoring needs Command refactoring (device profile v command service) Data Filter between DS and Core Data Add SMS to notifications Logging service rework Intel has developed a Logging/Telemetry as a Service – Replace/Augmenting service Support for alternate logging format XML & CSV in addition to JSON Command improvements Min/max limits Security authorization option(s)

Fuji – Application WG In Out Rules Engine Replacement
Rules engine to trigger notifications Next iteration of Application Services and App Functions SDK App Service to send data to Rules Engine Message envelop changes Added functions Filters: for value ranges, ??? Compression Signing Additional endpoints (HTTPS/MQTTS) More dynamic topic configuration / addressing Export commands Alternate message bus support Other functions (transforms, enriching, encryption, …???) Blockchain/digital ledger attribution Additional language SDKs

Fuji – DS & SDK In Out Support for mesh network protocols
Improvements to the SDK Cache of readings Device dynamic discovery Intel Camera device integration Camera Discovery Components (SAF Bus) Intel RFID device services Support for mesh network protocols Support device hierarchy Tooling for SDK (CLI, JetBrains, or Eclipse plugins, etc.) Tooling for Device Profile creation Downsampling – DS throttle back on readings if nothing is changing Additional language SDKs

Fuji – System Management
In Out Implementation of SMA as lightweight registry provider (when Consul not there) Know what services are available when registry not there Start/stop/restart all done by the executor to include stop/restart of SMA Executor tracks completion and returns results Executor for metrics collection Setting configuration SMA Translation layer (stretch) Pick one protocol to start (LWM2M) Storing metrics collected locally Callbacks (alert on changes to config/metric) – SMA translations to other protocols Redfish OMADM Actuation based on metric change “rules engine” for control plane data Consider use of QoS and blockchain to prioritize resource usage by certain services. SMA to store configuration – lightweight configuration store In replacement of Consul or whatever provider we have for reg/config). Discussed at recent sys mgmt. WG meeting. Bigger question is do we still want Consul.

Fuji - Security In Out Generation of PKI
Distribution of per service Vault secrets HW secure storage abstraction layer How to protect the Vault Master Key Secure Service to Service requirements and design Ensuring the services running are those expected (and authorized) Renew/refresh threat assessment Out Implementation of service-to-service communications Securely providing service updates How to securely provision new devices/sensor Device identification and authentication/authorization Hyperledger/blockchain/digital ledger integration Protect data at rest In DB like Mongo In log files Privacy concerns (HIP-A, GDRP, …)

Fuji – Test/QA/Documentation
In Improved unit tests across services (improvements should be driven by specific WG dev teams) Improve blackbox test structure including reorganization of the tests and better test case documentation New test framework (e.g. Robot or Cucumber) to support additional types of functional/blackbox and system integration tests e.g. Device Service or system level latency tests Blackbox tests for new Application Services microservices Blackbox test for Device Services - Virtual, Modbus, MQTT, BACNet, OPC UA. Initially develop common set of tests that can be used against all Device Services. Automated performance testing API Load testing (measure response time) and metrics (CPU, memory) collection for all EdgeX microservices (this work was started during the Edinburgh iteration) EdgeX microservice startup times Test coverage analysis e.g. using tools such as Codecov.io Out Automated Performance Testing Automated system level latency and throughout testing (e.g. device read to export or device read to analytics to device actuation) - STRETCH GOAL The ability to create summary reports/dashboards of key EdgeX performance indicators with alerts if thresholds have been exceeded Blackbox and performance test runs against other container technologies supported (e.g. snaps Automated performance testing - baseline performance of service binaries no container Configuration testing Existing testing uses a single static configuration Need to identify and add additional testing configurations to automated blackbox testing Static code analysis Using tools such as SonarQube or Coverity to identify badly written code, memory leaks and security vulnerabilities – STRETCH GOAL Documentation Replacement of RAML API documentation with Swagger – STRETCH GOAL Tracing During testing, configuration Candidate tools/technology based on OpenTracing standard: Zipkin, Jaeger

Fuji Planning – DevOps In Out
Static code analysis tool identified and integrated into the EdgeX Jenkins Pipeline Code and artifact signing with semantic versioning Fix Documentation – edgex-go Build Performance Optimizations Pipelines for EdgeX Foundry base build images Basebuild images managed locally within Nexus Leverage PyPi Proxy for local pip dependencies ARM builds – optimization leveraging different high CPU build nodes / OS (ARM team) Alternate deployment/orchestration Beyond Docker/Snaps Kubernetes Kata Containers … SonarQube – SonarCloud is already in play in the LF Decision: wait to see what codecov.io offers Suggestion to rename all of the Jenkins “arm” jobs so as to differentiate 32bit / 64bit architectures Full Pipeline transformation for EdgeX services arm64 or aarch64 rather than just generically "ARM", since really we don't support arm 32-bit, and typically folks will assume you mean arm 32-bit if you just say "ARM“ related to my point above about saying arm64 instead of arm, we should really rename the jenkins jobs to be "arm64" instead of "arm", since they're not arm 32-bit and it would be confusing if or once we start adding arm 32-bit builders Use Pypi proxy in nexus

Fuji – Vertical Solutions
In Out Additional project groups Oil/Gas Smart Factory Gap analysis by each project group What is needed by the vertical that is not provided by EdgeX today? Proposed EdgeX roadmap additions High level architectural needs/changes High level designs Technological suggestions/input Propose additions??? Propose additions???

Fuji - Certification Self Assessment process
In Out Self Assessment process Device Service Certification Application, Core Certification

Backlog Review What else do we need to consider? New additions/opens
Edinburgh Roadmap review (what did we not get done) New additions/opens

Future Roadmapping Beyond Fuji
4pm Day 1

Future Roadmaps What moves to Geneva Do we have plans for Hanoi
What’s the major theme(s) for spring release Do we have plans for Hanoi What are some big rocks we want to set out as markers for the nex few releases

Day 2 May 2

Admin and Schedule Review
Action items, announcements from Day 1 Agenda for Day 2 9am Developer Advocate Perspective 10am Architecture discussion and decisions 3pm Action item Review 3:30pm Lessons Learned and Chandler Meeting Changes

Developer Advocate Report Current Project Community and how to shape it
9am Day 2

State of the Development Community
What’s the current community of developers look like? Stats and information on contributors, reviewers, etc. What’s the current community of users look like? What are the common questions What are the common needs

Areas of Improvement How do we onboard new developers better?
How do we provide better help to the community of developers? How do we provide better help to the community of users?

Architecture Issues 10am Day 2

Tech Debt Issues More unit & integration tests
Identify the real short comings Release manager – we have a process, now what? Mongo & Redis – what should be our reference implementation database(s) Traceability – we have a correlation id. What’s next? Rules Engine replacement (the last of the Java) Is there a small size/simple rules engine replacement? Until one is found/devised we don’t exactly know how to deal with it in application services (TBD). In the meantime, we continue to use the Java/Drools implemention and we need application service example to call existing rules engine Environment support Windows development & 0MQ (any change??) ARM 32 support (any change??) Metadata needs to implement registration/callback mechanism for any metadata changes (ex: call me anytime the device changes) Value descriptors creation should be the responsibility of core not device services Events should be timestamped in nanoseconds

Tech Debt Issues part 2 Device Service improvements
Clean up of device profile Adding automatic discovery and provisioning (or at least a place in the SDK for DS creators to use to do automatic discovery and provisioning using blacklist/whitelist configuration). First – need requirements/design work to help define Need device service/device deregistration Cache of readings: to avoid hitting the device in some cases and to avoid sending data to core under some requests Gain an appreciation of EdgeX performance and size characteristics What are the resource requirements for EdgeX Answering the question: will it fit on my system? What is the speed of an EdgeX transaction? I.e. how long does it take for a sensor reading to be ingested, perform analytic logic, and trigger actuation on a device (and be exported to the north side). How many things can be attached to EdgeX at one time? How much data can be ingested and how often can it be sampled? These questions must be answered on real platforms (both Intel and Arm) – not just on a virtual environment. The results must also include information about the use case, data type, circumstances of execution The performance tests/harness/tool description must be made available for others to be able to replicate and/or extend How do we deal with the rate of code change going forward? Edinburgh is V1.0. We must beware of non-backward compatible changes We must find a cadence that allows consumers to keep up – example, Redis developing core services with Redis that replace those of the Mongo implementation. Go plugins – don’t appear to be on the Go roadmap We are checking with the Go community and groups like Hashicorp that is using Plugins – what is their vision and options thought Because of plugins issue, we don’t really know how to effectively bring on other implementations (ex: ObjectBox) without many issues We will look to move from RAML to Swagger (aka OpenAPI)

New features, tools, processes, etc. Issues
Need to handle the “forward” part of store and forward “Disconnected or situational” Store and Forward - need to support (via application services) use case where the application services are not able to connect to and pass data to the desired endpoint (where EdgeX has lost connectivity to the north) In this requirement, there are scenarios where all or a subset of unsent data is filtered and then sent when reconnected “Batch mode or deliberate” Store and Forward – need to support (via application services) use where the application services is always connected to the endpoint, but the use case requires not sending events all the time, but batching the events up periodically and sending them in bulk to the north Certification support Which services first, full certification vs self assessment, who pays, how managed Documentation versioning Semantic versioning Kubernetes (K8s and K3s) support How do we facilitate under scale of solutions? Naming of Snaps and other deployment artifacts going forward SMA’s responsibilities (and non-responsibilities) & design Versus registry/config, other services or outside systems LF Edge What are the projects and what is the potential impact

New features, tools, processes, etc. part 2
Application services and cloud providers Change in philosophy -> provide example code for the big 3 (Amazon, Azure, Google) We also need to support MQTTS/HTTPS Will require application services (and SDK) to use be able to use Vault for certs, tokens, etc. Schedule service will be used by app functions/service for triggering things like store/forward needs. We need the ability to incorporate scheduling as module/library or as REST call client May need additional scheduling options (CRON job support, etc.) We need to consider automatic/dynamic device

Deployment Artifact discussion – time permitting
Today We produce many services with a reference implementation deployment artifact for each (offering two different types: Docker and Snaps) Tomorrow Might we want to create deployment artifacts that contain multiple services? All of core services running in a single container/snap – simplifying deployment under circumstances/use cases Might we want to make/build some type of combined service? All core services in a single executable – for example – in order to improve performance, lower footprint, etc.

Action Item Review 3pm Day 2

Chandler Meeting 3:30pm Day 2

Next Meetings Planning
For Chandler Lessons learned What worked well and what did not? Architect’s day Training/Business day Spring 2020 meeting Venue Host

Goodbyes 4pm

EdgeX Face to Face Working Agenda and Deck

Similar presentations

Presentation on theme: "EdgeX Face to Face Working Agenda and Deck"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

EdgeX Face to Face Working Agenda and Deck

Similar presentations

Presentation on theme: "EdgeX Face to Face Working Agenda and Deck"— Presentation transcript:

Similar presentations

About project

Feedback