1. Extending the Dynamics 365 World With Microsoft Azure
By Nelson Johnson
Solution Architect
BroadPoint Technologies
Note: This presentation was developed in October 2017, updated February 18, 2018.
The services in the cloud are constantly changing.
This presentation is available on slideshare:

2. Target Audience Interest in Cloud Architecture
IT Managers
Developers
System Architect
Azure curious
On-prem CRM and considering moving to Dynamics 365 (D365)
Direct integration with SQL
Dealing with D365 Limitations

3. Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Choices in Azure Architecture Share our Best Practices

4. Who is Nelson Johnson?
Started programming 1977…in high school…on a mainframe
Became full time software consultant in 1982
BS in MIS and Decision Science from George Mason University
Worked at DARPA creating web apps in 1996
Working with CRM since version 3
Solution Architect for BroadPoint Technologies, LLC

5. Why would you want to extend D365 to Azure?
Inconsistent workload demands
Consolidate business logic
Scheduled jobs are not easily managed within D365
SQL stored procedures are not an option
Limitations of FetchXML, Rollup fields
Offload processing
Plugins, workflows, and API in D365 have a timeout limit of 2 minutes
Chain of plugins or workflows are getting too long
Need to create or update a large number of records
Waiting for external services to respond
SQL timeout
There are a lot of good reasons for using Azure, such as reliability. Cost should not be the only consideration.
Consider the learning curve to maintain apps in house vs in the cloud, and the cost of keeping updates in lock-step with the platform.
Inconsistent workload demands – Web Portal with usage spikes or scheduled business process that does not need to run all the time
Good for offload batch processing – Asynchronous process to create 20k records, such as monthly billing
Use it to consolidate business logic – such as an integration with Microsoft Graph services, machine learning, or handling complex calculations like linear equations for product mix.

6. Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Choices in Azure Architecture Share our Best Practices

7. Tools for extending to Azure
Compute
Azure Functions
Azure Web Jobs
Azure Service Fabric
Azure Worker Role
Worker Roles (see Best Practices)
Service Plan
Choose your CPUs, Memory
Integration
Endpoint Config in D365
Service Bus / REST API
Azure Logic Apps
Storage
SQL Server Replication
Azure Cosmos DB
Azure Tables
When you build extensions to D365, you have many options available to run your code in Azure. These are listed in the Compute column.
We will discuss the first 5 of these because they provide a diverse set of features that work for many different applications.
But the code does not run itself. Somehow the program you have hosted in Azure must determine when to run and which records in CRM they apply to.
This is where we introduce the integration options because they control which programs will run in response to user activity in CRM.
The most common integration works through defining an endpoint in D365 that connects to a Service Bus.
An optional component to consider when extending your D365 to Azure involves using one of several storage options.
Lets start with the Integration part first.

8. What is Service Bus? A first-in-first-out message queue
Separates events from actions
Enforces decoupled architecture
Increases options for reusuable business logic
Lock & complete
High availability
Scalable
Handles failures
The Service Bus is the most common way to communicate events in D365 out to your Azure programs.
Service Bus is a first-in-first-out message queue. It is like a database pipe – messages go in one side and come out the other.
When you create a Service Bus Namespace in Azure, it gives you a connection string to connect to it.
Your custom code would be the message receiver, and that code must retrieve each message (in effect lock it) run a process based on what is in the message, then mark it as complete.
Why would we use it?
D365 is super simple to send messages to it, it can queue up 1M messages, and it handles failures for us without writing any logic.
Source:

9. How does D365 integrate with Azure?
In general, D365 is easily configured to send messages through a Service Bus to trigger your programs in Azure to start running.
When an event happens in D365 that we want to trigger some Azure code to process, we have a workflow that creates a record in our custom Service Bus Entity
It is configured to push messages into separate queues or topics depending on what the event was.
The event message sits in the Service Bus queue until the Service Bus Manager picks it up and calls your Line of Business (LoB) service.
LoB Service connects to CRM and performs work
Errors are logged for an operator to fix and restart the process

10. Service Bus vs Webhooks (D365 CE 9)
Service bus is always up – Web hooks only work when the service is working
SB has low latency
SB can ingest 1000’s of messages per second
1000 concurrent connections
Publish/subscribe supports a decoupled architecture
Do you need to use the Service Bus? No.
You could use Webhooks if you are using D365 CD 9 or later.
Here are some reasons you should use the Service Bus:
Consider the scenario when you need to update a back-end service. By using a service bus, you can stop and start the back end service without disrupting the other business processes.

11. How do you connect an entity to the Service bus?
Developer Sidebar
How do you connect an entity to the Service bus?
Plugin Registration Tool – no code needed!
Endpoint
Step
Create a custom entity
Register the endpoint
Register the step (CRM event message)

12. Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Choices in Azure Architecture Share our Best Practices
Service Fabric, WebJobs, Functions, Logic Apps

13. How low will you go? How much control do you want?
You might need to give up some control to get scalability, quick deployment, security updates
Can you reassign your team to higher value tasks?
Source:

14. Azure Compute Light – Azure Functions, Logic Apps
Mid-size – Azure Webjobs
Heavy – Service Fabric
Which service is right for you?
There is a tradeoff between what is quick and easy to code and the need for robust enterprise features.
I would categorize the services this way:

15. Azure Functions Consumption plan – ideal for intermittent services
Pros
Cons
Deployment
Consumption plan – ideal for intermittent services
Dynamic Scaling
More triggers than WebJobs
Integration with more services than WebJobs
Complements Azure Logic Apps
Control – what is that?
Up to 10 minute Timeout unless you pay for a service plan
Local Dev and Debug
Version Control
Deploy directly from VS
Continuous Integration/deployment
“Server-less” just means you don’t own the server.
Languages: JavaScript, C#, and F#, and scripting options such as Python, PHP, Bash, Batch, and PowerShell.
Triggers on Microsoft Graph Events (Excel Table events, One Drive file events, Outlook), Azure Cosmos DB, Blob storage, Events, HTTP REST or Webhook, Service Bus, and Timer.
Integration with: Logic Apps, Storage tables, SQL tables, Twilio, SendGrid, Push Notification Hubs, and Auth Tokens.
Complements Azure Logic Apps
Run this with service bus topics and subscriptions

16. Logic Apps Pros Cons Deployment Can schedule jobs
Can query and update D365 data (workflows with loops!)
Can interface with Service Bus
Can connect to over 176 other services
Configuration through JSON – no code
Price based on # of executions
“Server-less”
Weekly Updates
No code – anything more than simple tasks requires a Function
UI to develop is inside Azure – not easy to figure out
Documentation is sparse
Programing skill level
Limited triggers:
Min trigger schedule
15 seconds
60 seconds
Version control built in to web UI
Visual Studio – Publish
ARM template deployment
Another “Serverless” option
Technology that Flow is built on
Example: Poll the database every minute for 10 days + execute 500 workflows = $15/10 days
Glue between applications
Triggers: recurring schedule, on demand, HTTP, API
News features released in frequent updates

17. Azure App Service WebJobs
Pros
Cons
Deployment
PaaS – Managed VM, High availability
Multiple apps on single service plan
Easier (than SF) for developer to start developing apps
Can run continuously or triggered
Scales with App Service plan – setup manually
You pay for it while it is deployed
Timeout limit
No remote desktop
Local Dev and Debug
Version Control
Deploy directly from VS
Continuous Integration/deployment
Easier for a developer to get up to speed
PCI, SOC, ISO compliant
Triggers: Blobs, Service Bus, Queue, Timer, continuous, custom
No remote desktop

18. Azure Service Fabric Best for Microservices
Pros
Cons
Deployment
Best for Microservices
Control over how workloads are run
Load-balancing, scaling, failover, replication, stateful
VM Scale Sets in clusters
Self-hostable
Fault Analysis Service
Learning curve
Code changes will be necessary to forklift code. Re-architect is better approach.
Expensive initially because a production system uses 5 VM’s
- Value increases as you add more services
No remote debugging
Local Dev and Debug
Deploy directly from VS
PowerShell and CLI deployment
Deploy many services at one time
Big learning curve.
Better choice if you plan to migrate legacy applications away from an enterprise data center
Stateful microservice removes the need for queues and caches. Stateful services are high availability and low latency.
Auto-scaling built in to clusters.
With the Fault Analysis Service, you can induce meaningful faults and run complete test scenarios against your applications

19. Wrap up - Which one is best?
Azure Functions
Just getting started and want to experiment
Specialized integration
Extending Azure Logic Apps
Azure Logic Apps
Glue for mashups
Azure WebJobs
Already have a website and want to piggy-back on that service plan
Credit card processing
Azure Service Fabric
Re-architecting a legacy environment
Require high level of control and resilience

20. Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Azure Architecture Share our Best Practices
Microservice

21. Design Considerations
Must read: Azure Architecture Center (
Application Architecture Guide
Reference Architectures
Cloud Design Patterns (32 of them)
Best Practices
Performance Antipatterns
Design Checklists
Reference architectures – Network DMZ, Hybrid, Identity Management, Managed web apps
Performance Antipatterns – Busy Database, Chatty I/O, Extraneous Fetching (too much data),

22. Microservice Architecture Design Patterns
Increase the velocity of application releases by decomposing the application into small autonomous services that can be deployed independently
D365
Shows some example cloud architecture design patterns. There are over 30 to choose from.
The Sidecar pattern is for features like logging, and security that may be part of your framework to provide orthogonal services.
Anti-corruption layer deals with exposing views of data from the legacy system in a manner consistent with the new system and allows the legacy system to function without modification.
The Strangler is used to direct data requests to specific pieces of functionality that is being incrementally moved form the legacy system to a microservice architecture.
Ambassador patterns act as proxy to services to allow control over routing, resiliency, security features, and to avoid any host-related access restrictions.
Source:

23. BASE - Design Patterns for Eventual Consistency
Use Event Sourcing and CQRS Pattern
Data Consistency
Decouple recording the transaction from updating
Reconciliation between services is easier
Retry Pattern
Best for transient faults like timeouts
Service Bus provides this (to a degree)
Idempotent methods
Same result if called multiple times
Your on-premesis systems typically are using a SQL technology that guarantees data consistency, (also called ACID: Atomicity, Consistency, Isolation, Durability)
Dynamics 365 lives in the BASE world. Basically Available, Soft state, Eventual consistency means that transactions are distributed and they are eventually updated in the database(s).
This means that D365 may not be 100% accurate all the time and you must design processes that will catch inconsistencies and trigger a process to fix it.
Event Sourcing is used to keep object state history so that it can be re-evaluated and corrected
Command Query Responsibility Separation is used to de-couple the recording of the transaction from an actual update
Use Materialized Views design pattern to present the result of the transactions
Retry Pattern depends on what the fault is – cant help you if your code is broken.

24. Potential for Inconsistent Data
Two transactions are created to update the same account
They both get the starting balance
They both are adding a different amount to the balance
Last one “wins”
Example of what could go wrong if you create a system with ACID in mind, but you are really using a BASE system:
You start with a known account balance
But you end with an incorrect balance
What happened? What was the balance before the transactions?
If this is milliseconds apart, it would be very hard to tell what is going on.
You could use logging, but then someone has to monitor it, and how do you fix it?

25. Event Sourcing and CQRS to the rescue!
Two transactions are created to update the same account
They both append their update to a journal
They both send a message to another process to update the balance
The balance is calculated twice, but it is accurate
Command Query Responsibility Separation
By keeping the entire history of changes, we no longer really need the Balance (State) of the account
We can completely recalculate the balance, or re-play a series of events (think debugging).
We can also see the balance at any point in time.
Eventually all accesses to an item will return the last updated value.
By pushing updates into a transaction log, process does not have to wait for CRUD operations
The cost: storing more data, overall system is more complex because it is many smaller parts, overhead to complete the task is more

26. Plan for Failure! Compensating transaction Pattern
Un-do in reverse order
Create Exception entity to manage resolution
Provide a view of exceptions in the dashboard
Provide error with steps to resolve it
For each transaction, keep track of steps taken
If the process is not completed, have a process that will walk back through and undo what was done
You are effectively creating what you might have done If you have used transactions in SQL Server
Source:

27. Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Azure Architecture Share our Best Practices

28. Cloud Service Lessons Learned
“Classic” service – not part of ARM deployments
It was an easy starting point
No timeouts or code size limits
You should employ Async Tasks
Doesn’t run under CSP license
Runs all the time like a VM
No options to use KeyVault
No options for dynamic configuration of Web.config/App.config
ARM – Azure Resource Manager allows you to build scripts to deploy services
None of these reasons should stop you from using it, unless you are selling D365 licenses under a CSP model
Microsoft has not been putting any effort into this area – seems like it is going away
Our customers only get about 5% of CPU during peak periods, but pay all month for it

29. Best Practices Test Location and latency
Check which datacenter your CRM instances is located
Server Name First Two Letters
Region BL
East US BN
North Central US BY
West US
Location, location, location – Minimize the latency because it can really cost you

30. Best Practices Time Zone Service Plan Azure Resource Manager
Integration account in customers timezone
Work with localized datetime fields using UTC
Service Plan
Stress test with the smallest one and work up
Azure Resource Manager
Group by Resource (compute, service plan, storage, service bus)
Tags: Owner, Environment, LoB Function(s), Tier, Project
Test Driven Development
Minimize “chatter” using API Transactions, ExecuteMultiple
Monitor your usage
My profile was for East coast time, but customer was in Houston and they saw my changes as having the wrong time.
Keeping track of your services using ARM is really important to containing costs
- Make a governance policy on who owns what using tags!
TDD costs a bit more up front, but really pays for itself after sprint 1

31. Best Practices
Do not include the Service Endpoints in the D365 Solution file that gets promoted to production
Have your subscribing service validate that the message is coming from the right domain
SB message has the sending domain – compare it to the D365 connection
Use one service bus “namespace” per D365 instance
Don’t Choose free tier of Service bus for production
Configure to use many queues (or topics, depending on subscription)
Use a non-interactive account in D365
Reduce license cost
If you include the Service Endpoints in the solution file that you promote from dev to production, you will be connecting the production instance back to the development Azure environment
The message on the service bus has the domain of the system that created the message. Compare it to the connection string domain to make sure they are the same.
You can differentiate which messages belong to which programs by using queues and topics – don’t use multiple service buses
In D365, check the “non-interactive” box on the user account you use for integration.

32. Logic App Best Practices
Azure Logic Apps
Use webhook for trigger on the “business logic”
Modular design
Gives you multiple options to consume the logic later
Avoid Excessive polling
Cost is $ /action
Ex: every 10 seconds= 259,200 actions=$6.48
Avoid IoT and Database conversions
Best used for stochastic loads

33. Use EntityCollection() for Transactions
Developer Sidebar
Use EntityCollection() for Transactions
EntityCollection()
Relationship()
EntityCollection Entities.Add()
<entity>.RelatedEntities.Add() (early bound)
Create(<entity>)
These are the classes you need to combine to into creating a transaction
The benefit is to reduce the amount of chatter with the API, and it is an all-or-nothing, which saves you from having to clean up partial transactions

34. Transaction code sample
Developer Sidebar
Transaction code sample
Code
Description
Guid duesHeaderId = Guid.NewGuid();
Create a new ID so that we can create a reference in each detail record
EntityCollection duesHeaderDetailCollection = new EntityCollection();
Create a collection that will hold all the child records
Relationship duesHeaderDetailRelationship = new Relationship("bpt_bpt_duesheader_bpt_duesdetail");
Define the relationship that links the entities together
duesHeader.bpt_DuesHeaderId = duesHeaderId;
Assign the PK of the header record
duesHeader.bpt_MemberId = Contact.ToEntityReference();
Assign some values to the header - this is the member
Xrm.bpt_duesdetail duesDetail = new Xrm.bpt_duesdetail();
Create an instance of the child entity record
duesDetail.bpt_DuesBillId = new CrmEntityReference( Xrm.bpt_DuesHeader.EntityLogicalName, duesHeaderId);
Populate the entityreference from the child back to the parent record
duesHeaderDetailCollection.Entities.Add(duesDetail);
Add the child record to the parent
duesHeader.RelatedEntities.Add(duesHeaderDetailRelationship, duesHeaderDetailCollection);
Complete the relationship by adding the child collection to the parent entity
service.Create(duesHeader);
Call the service to create both records at the same time
Some key elements are color coded to make it easier to follow where they are initialized and used.
This sample is a membership dues Header / Detail
A key point – the duesHeaderId is manually assigned in order to build the link between the parent and child record
Typically, you would have more than one detail (child) record.

35. Resources:
Platform feature comparison: web-site-cloud-service-vm
Azure Architecture:
Azure Functions Tools for Visual Studio: functions/functions-develop-vs
Write and test functions locally:
Azure Functions forum: US/home?forum=AzureFunctions&filter=alltypes&sort=firstpostdesc
Continuous Deployment: continuous-deployment
Subscription Governance: manager/resource-manager-subscription-governance

36. Resources (cont):
Design Patterns for Microservices: patterns-for-microservices/
WebJobs Extensible Triggers: and-binders-with-azure-webjobs-sdk alpha1/
Be careful of the Logic App Consumption Prizing model:

37. Bright ideas for better performance
Q&A

38. Bright ideas for better performance
Thank You!
Nelson Johnson