1. Data Centric Continuous Architectures
“Data is a precious thing and will last longer than the systems themselves.”
Tim Berners-Lee

2. Data Centric Continuous Architectures
Introduction
Context
The Continuous Architecture Principles
Business Trends
Emergence of Chief Data Officers
Data Science and Analytics
Technology Trends
Micro services – impact on data
Event Sourcing
NoSQL databases
Semantic approaches and linked data
Practical Experience
Challenge 1: Giving meaning to your interfaces and data lineage
Challenge 2: Microservices and sharing of data
Wrap up and Q&A
© Pierre Pureur & Murat Erder

3. Context – A Few Thoughts
What is architecture?
Dancing House by Frank Gehry and Vlado Milunic in Prague
The basics tenets of good architecture have not changed.
What has evolved is the technology landscape
Increased ability to work at larger scale and in a distributed manner
Desire for quicker delivery timelines
What has not changed is the organizational hurdles that are the same as in Greek Antiquity
© Pierre Pureur & Murat Erder

4. Context - Brief History
Euler - Seven Bridges of Königsberg
1736
Hierarchical
Databases
1960s
Ingres
System R
1974
SQL
Standard
1986
PL/SQL
1992
RDF
2004
Hadoop
2006
SPARQL
2008
Apache Flink
2014
Data
© Pierre Pureur & Murat Erder

5. Software Architectures Can Evolve Continuously If We….
1. Architect products, not just solutions for projects 
2. Focus on Quality Attributes, not on functional requirements 
3. Delay design decisions until they are absolutely necessary 
4. Architect for change- leverage ”the power of small” 
5. Architect for build, test and deploy 
6. Model the organization after the design of the system 
What About Data?
© Pierre Pureur & Murat Erder

6. Data Centric Continuous Architectures
Introduction
Context
The Continuous Architecture Principles
Business Trends
Emergence of Chief Data Officers
Data Science and Analytics
Technology Trends
Micro services – impact on data
Event Sourcing
NoSQL databases
Semantic approaches and linked data
Practical Experience
Challenge 1: Giving meaning to your interfaces and data lineage
Challenge 2: Microservices and sharing of data
Wrap up and Q&A
© Pierre Pureur & Murat Erder

7. © Pierre Pureur & Murat Erder
Data – What is it?
© Pierre Pureur & Murat Erder

8. Emergence of Chief Data Officers
CIOs started as the custodian of ‘Information’. Somewhere along the way this was forgotten and now we have ‘Chief Data Officers’
Regulators started putting pressure on institutions…
Companies that deal with a lot of data turned the data technology landscape upside down….
Gartner News Release (January, 2016)
Gartner Estimates That 90 Percent of Large Organizations Will Have a Chief Data Officer by 2019
Gartner News Release (November, 2016)
Main business objectives of the CDO are to
increase customer intimacy (62 percent)
competitive advantage (60 percent) and efficiency (54 percent).
Respondents said that the main organization-wide responsibilities are oversight of analytics initiatives (69 percent) and data governance (68 percent). Next come responsibilities for defining the analytics strategy for the organization and ensuring information reliability and value (64 percent each).
© Pierre Pureur & Murat Erder

9. Data Science and Analytics
Data Science is a "concept to unify statistics, data analysis and their related methods" in order to "understand and analyze actual phenomena" with data.  It employs techniques and theories drawn from many fields including mathematics, statistics information science, and computer science.
Data Science, also known as data-driven science, is an interdisciplinary field about scientific methods, processes, and systems to extract knowledge or insights from data in various forms, either structured or unstructured, similar to data mining.
© Pierre Pureur & Murat Erder

10. Data Science and Analytics
Data Science is evolving: As companies need to better understand what their customers and prospects want, data science approaches are being used to analyze customers’ behavior as close to real time as possible.
This means that the type of raw data being collected as part of our interactions with our customers and prospects as well as the corresponding data products may continuously evolve depending on the results of the continuous data science process
Leveraging Data Science Techniques for Real Time Analytics creates a need for continuous data analysis and design
© Pierre Pureur & Murat Erder

11. Data Centric Continuous Architectures
Introduction
Context
The Continuous Architecture Principles
Business Trends
Emergence of Chief Data Officers
Data Science and Analytics
Technology Trends
Micro services – impact on data
Event Sourcing
NoSQL databases
Semantic approaches and linked data
Practical Experience
Challenge 1: Giving meaning to your interfaces and data lineage
Challenge 2: Microservices and sharing of data
Wrap up and Q&A
© Pierre Pureur & Murat Erder

12. What About Microservices?
Basic Definition: A microservice is a collection of one or more operations within a single business domain that performs a singularly focused task and provides, manipulates, or stores data for an application. Examples could include employee profile retrieval, document storage and quote rating
Microservice architectures can be used to break up a large monolithic application into smaller, loosely coupled parts. This enables independent development/testing and scaling, and therefore agility.
There are more moving parts in a microservice architecture, which creates complexity, and potentially increases latency
Microservices promote “share nothing” architectures – and cause us to think differently about data
© Pierre Pureur & Murat Erder

13. Microservices – Impact on Data
As a result, microservices need to handle functionality normally provided by a central database management system in order to share data between them
Microservice Architectures follow a “share nothing” model. They do not share data stores between microservices
© Pierre Pureur & Murat Erder

14. Event Sourcing – A Different Way Of Thinking About Data
Event Sourcing is “an architectural pattern which warrants that your entities do not track their internal state by means of direct serialization or O/R mapping, but by means of reading and committing events to an event store” (Wikipedia)
One way of thinking about Event Sourcing is to use a checking account analogy. The transactions against a checking account are permanently stored. This ensures that the balance of the account can be (re)created when necessary – at any point of time in the present or the past. This also ensures that adjustments can be applied retroactively
Order Item
Customer
Order
Shipping Address
Billing Address
Current State
Events
Customer Onboarded
Address Changed
Order Created
Order Item Added
Order Item Removed
Order Item Added
Shipping Address Provided
© Pierre Pureur & Murat Erder

15. Data Modeling is Dead! Long Live Event Modeling
Event Storming is an example of new modeling techniques which are replacing Data Modeling. The brainchild of Alberto Brandolini, it's a synthesis of facilitated group learning practices from Gamestorming and the principles of domain-driven design (DDD).
© Pierre Pureur & Murat Erder

16. Document Graph Key Value Wide Column
NoSQL Databases – Four Data Store Types
Document
Graph
Key Value
Wide Column
NoSQL Databases tend to be specialized by usage types, and work well with Microservices
© Pierre Pureur & Murat Erder

17. © Pierre Pureur & Murat Erder
NoSQL Databases
Document
Typical applications: IoT data capture, single integrated (360 degree) view of subject, Insurance Policy Administration, Legal Contracts
Strengths: Flexible data model, scalability, availability
Weaknesses: No standard query syntax, analytic processing, time series analysis, transactional processing
Characteristics:
Document object model maps directly to objects in modern object oriented languages.
Rich query functionality
Key Value
Graph
Typical applications: Storage of session data, chat room enablement
Strengths: Flexible data model, availability and partition tolerance
Weaknesses: Data consistency, granularity, analytic processing
Characteristics:
Data model hidden in the system - only the primary key can be queried.
Single means of accessing data: by primary key. Very limited query functionality. Potential additional development costs and application-level requirements to provide more than the most basic query features
Typical applications: Social networking, recommendations, n-degree relationships
Strengths: Graph algorithms e.g. shortest path, connected-ness
Weaknesses: Has to traverse the entire graph to achieve a definitive answer. Not easy to cluster. Analytic Processing
Characteristics:
Best suited for social networking applications
Can be used by any application where you need to understand what people are doing, buying or enjoying to provide recommendations
For graph-style, rich or complex, interconnected data.
Wide Column
Typical applications: Catalog searches, Time series data
Strengths: Consistency and partition tolerance
Weaknesses: No ACID compliance; analytic processing
Characteristics: Provides more granular access to data than the key value model, but less flexibility than the document data model.
© Pierre Pureur & Murat Erder

18. NoSQL Databases – let’s not forget the Hadoop stack
© Pierre Pureur & Murat Erder
Image Source: Edureka!

19. Semantic Approaches – Linked Data
In computing, linked data (often capitalized as Linked Data) is a method of publishing structured data so that it can be interlinked and become more useful through semantic queries. It builds upon standard Web technologies such as HTTP, RDF and URIs, but rather than using them to serve web pages for human readers, it extends them to share information in a way that can be read automatically by computers. (Wikipedia)
Has four principles (based on Tim Berners Lee note 2006):
Use URIs (Universal Resource Identifiers) as names for things
Use HTTP URIs so that people can look up those names
When someone looks up a URI, provide useful information, using the standards (RDF*. SPARQL)
Include links to other URIs so that they can discover more things.
London
Murat
Subject
Pinar
Object UK
Wife
predicate
Lives in
City of
Triple
© Pierre Pureur & Murat Erder

20. Data Centric Continuous Architectures
Introduction
Context
The Continuous Architecture Principles
Business Trends
Emergence of Chief Data Officers
Data Science and Analytics
Technology Trends
Micro services – impact on data
Event Sourcing
NoSQL databases
Semantic approaches and linked data
Practical Experience
Challenge 1: Giving meaning to your interfaces and data lineage
Challenge 2: Microservices and sharing of data
Wrap up and Q&A
© Pierre Pureur & Murat Erder

21. Challenge 1: Giving meaning to your interfaces and data lineage3
Run time applications can refer to required data (e.g. data sources) and feedback data (e.g. lineage) to the set of mastering services. 1
Mastering is done on each data element through a service that can implement appropriate workflow and security 2
Mastering
Services
Design time, relevant data can be pulled together via a common ‘shopfront’ 4
The data is available for analytical purposes.
Analytics Lake
© Pierre Pureur & Murat Erder

22. Challenge 2: Dealing with Data in a Microservice Architecture
How would the team have implemented the data structures in a traditional “SOA” Approach?
They would minimize the number of data stores - Reuse and share existing data stores rather creating new ones; organize data by layers; and leverage an SQL Database
They would rely on DBMS software to keep data consistent (e.g. referential integrity) where data is distributed
They would use Data Modeling to eliminate data redundancies
© Pierre Pureur & Murat Erder

23. Challenge 2: Dealing with Data in a Microservice Architecture
How would the team implement the data structures with microservices?
They would decompose Integration Services and Business Services into microservices that perform atomic tasks
Instead of storing the data in a few relational DBMSs, they would store that data in several non-SQL DB’s, selecting the best technology for each data type
They would use Event Modeling instead of Data Modeling techniques and use event queues to communicate asynchronously between microservices
© Pierre Pureur & Murat Erder

24. Challenge 2: Microservice Event Exchange Patterns
Traditional Web Services e.g. REST via HTTP
Event Trigger
File change
Scheduled Activity
User Action
event
MS1
MS2
MS3
Target 1
MS6
Target 2
Queuing e.g. Apache Kafka or RabbitMQ
Queue Topics
MS1
New topic subscribers and publishers can be added seamlessly
Microservices do not need to know about their dependencies as service flow is not hard coded
Microservices can still be auto scaled individually
Support for event driven design and event playback for state reconstruction from event logs
event
Event Collector
event
MS2
File change
Scheduled Activity
User Action
event
MS3
Event Log
MS6
Target 1, 2, etc…
© Pierre Pureur & Murat Erder

25. © Pierre Pureur & Murat Erder
Challenge 2: Dealing with Data in a Microservice Architecture – Lessons Learned
Many “Micro Service Architectures” are not fully based on Microservices. Can an architecture truly be made of 100% non-reusable components?
Microservices increase agility – but increase complexity and eventually cost of ownership. The effort to fully refactor a monolith into microservices is hard and may not be cost justified
Fully distributing data across Microservices (“share nothing”) tends to increase complexity. It may make sense to compromise by having a few related microservices share the same data
Event Modeling feels unusual to most traditional architects, designers and developers. The learning curve could be steep
© Pierre Pureur & Murat Erder

26. Data Centric Continuous Architectures
Introduction
Context
The Continuous Architecture Principles
Business Trends
Emergence of Chief Data Officers
Data Science and Analytics
Technology Trends
Micro services – impact on data
Event Sourcing
NoSQL databases
Semantic approaches and linked data
Practical Experience
Challenge 1: Giving meaning to your interfaces and data lineage
Challenge 2: Microservices and sharing of data
Wrap up and Q&A
© Pierre Pureur & Murat Erder

27. What should our Data Principle Be?
7. Data Driven? Data First? Don’t Forget the Data? 
© Pierre Pureur & Murat Erder

28. © Pierre Pureur & Murat Erder
Questions?
© Pierre Pureur & Murat Erder

29. © Pierre Pureur & Murat Erder
Book: store.elsevier.com/
Blog:
Murat Erder: @muraterder
Pierre Pureur: @pgp60
© Pierre Pureur & Murat Erder