Presentation is loading. Please wait.

Presentation is loading. Please wait.

Extending the Dynamics 365 World With Microsoft Azure

Similar presentations


Presentation on theme: "Extending the Dynamics 365 World With Microsoft Azure"— Presentation transcript:

1 Extending the Dynamics 365 World With Microsoft Azure
By Nelson Johnson Solution Architect BroadPoint Technologies Note: This presentation was developed in October 2017, updated February 18, 2018. The services in the cloud are constantly changing. This presentation is available on slideshare:

2 Target Audience Interest in Cloud Architecture
IT Managers Developers System Architect Azure curious On-prem CRM and considering moving to Dynamics 365 (D365) Direct integration with SQL Dealing with D365 Limitations

3 Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Choices in Azure Architecture Share our Best Practices

4 Who is Nelson Johnson? Started programming 1977…in high school…on a mainframe Became full time software consultant in 1982 BS in MIS and Decision Science from George Mason University Worked at DARPA creating web apps in 1996 Working with CRM since version 3 Solution Architect for BroadPoint Technologies, LLC

5 Why would you want to extend D365 to Azure?
Inconsistent workload demands Consolidate business logic Scheduled jobs are not easily managed within D365 SQL stored procedures are not an option Limitations of FetchXML, Rollup fields Offload processing Plugins, workflows, and API in D365 have a timeout limit of 2 minutes Chain of plugins or workflows are getting too long Need to create or update a large number of records Waiting for external services to respond SQL timeout There are a lot of good reasons for using Azure, such as reliability. Cost should not be the only consideration. Consider the learning curve to maintain apps in house vs in the cloud, and the cost of keeping updates in lock-step with the platform. Inconsistent workload demands – Web Portal with usage spikes or scheduled business process that does not need to run all the time Good for offload batch processing – Asynchronous process to create 20k records, such as monthly billing Use it to consolidate business logic – such as an integration with Microsoft Graph services, machine learning, or handling complex calculations like linear equations for product mix.

6 Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Choices in Azure Architecture Share our Best Practices

7 Tools for extending to Azure
Compute Azure Functions Azure Web Jobs Azure Service Fabric Azure Worker Role Worker Roles (see Best Practices) Service Plan Choose your CPUs, Memory Integration Endpoint Config in D365 Service Bus / REST API Azure Logic Apps Storage SQL Server Replication Azure Cosmos DB Azure Tables When you build extensions to D365, you have many options available to run your code in Azure. These are listed in the Compute column. We will discuss the first 5 of these because they provide a diverse set of features that work for many different applications. But the code does not run itself. Somehow the program you have hosted in Azure must determine when to run and which records in CRM they apply to. This is where we introduce the integration options because they control which programs will run in response to user activity in CRM. The most common integration works through defining an endpoint in D365 that connects to a Service Bus. An optional component to consider when extending your D365 to Azure involves using one of several storage options. Lets start with the Integration part first.

8 What is Service Bus? A first-in-first-out message queue
Separates events from actions Enforces decoupled architecture Increases options for reusuable business logic Lock & complete High availability Scalable Handles failures The Service Bus is the most common way to communicate events in D365 out to your Azure programs. Service Bus is a first-in-first-out message queue. It is like a database pipe – messages go in one side and come out the other. When you create a Service Bus Namespace in Azure, it gives you a connection string to connect to it. Your custom code would be the message receiver, and that code must retrieve each message (in effect lock it) run a process based on what is in the message, then mark it as complete. Why would we use it? D365 is super simple to send messages to it, it can queue up 1M messages, and it handles failures for us without writing any logic. Source:

9 How does D365 integrate with Azure?
In general, D365 is easily configured to send messages through a Service Bus to trigger your programs in Azure to start running. When an event happens in D365 that we want to trigger some Azure code to process, we have a workflow that creates a record in our custom Service Bus Entity It is configured to push messages into separate queues or topics depending on what the event was. The event message sits in the Service Bus queue until the Service Bus Manager picks it up and calls your Line of Business (LoB) service. LoB Service connects to CRM and performs work Errors are logged for an operator to fix and restart the process

10 Service Bus vs Webhooks (D365 CE 9)
Service bus is always up – Web hooks only work when the service is working SB has low latency SB can ingest 1000’s of messages per second 1000 concurrent connections Publish/subscribe supports a decoupled architecture Do you need to use the Service Bus? No. You could use Webhooks if you are using D365 CD 9 or later. Here are some reasons you should use the Service Bus: Consider the scenario when you need to update a back-end service. By using a service bus, you can stop and start the back end service without disrupting the other business processes.

11 How do you connect an entity to the Service bus?
Developer Sidebar How do you connect an entity to the Service bus? Plugin Registration Tool – no code needed! Endpoint Step Create a custom entity Register the endpoint Register the step (CRM event message)

12 Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Choices in Azure Architecture Share our Best Practices Service Fabric, WebJobs, Functions, Logic Apps

13 How low will you go? How much control do you want?
You might need to give up some control to get scalability, quick deployment, security updates Can you reassign your team to higher value tasks? Source:

14 Azure Compute Light – Azure Functions, Logic Apps
Mid-size – Azure Webjobs Heavy – Service Fabric Which service is right for you? There is a tradeoff between what is quick and easy to code and the need for robust enterprise features. I would categorize the services this way:

15 Azure Functions Consumption plan – ideal for intermittent services
Pros Cons Deployment Consumption plan – ideal for intermittent services Dynamic Scaling More triggers than WebJobs Integration with more services than WebJobs Complements Azure Logic Apps Control – what is that? Up to 10 minute Timeout unless you pay for a service plan Local Dev and Debug Version Control Deploy directly from VS Continuous Integration/deployment “Server-less” just means you don’t own the server. Languages: JavaScript, C#, and F#, and scripting options such as Python, PHP, Bash, Batch, and PowerShell. Triggers on Microsoft Graph Events (Excel Table events, One Drive file events, Outlook), Azure Cosmos DB, Blob storage, Events, HTTP REST or Webhook, Service Bus, and Timer. Integration with: Logic Apps, Storage tables, SQL tables, Twilio, SendGrid, Push Notification Hubs, and Auth Tokens. Complements Azure Logic Apps Run this with service bus topics and subscriptions

16 Logic Apps Pros Cons Deployment Can schedule jobs
Can query and update D365 data (workflows with loops!) Can interface with Service Bus Can connect to over 176 other services Configuration through JSON – no code Price based on # of executions “Server-less” Weekly Updates No code – anything more than simple tasks requires a Function UI to develop is inside Azure – not easy to figure out Documentation is sparse Programing skill level Limited triggers: Min trigger schedule 15 seconds 60 seconds Version control built in to web UI Visual Studio – Publish ARM template deployment Another “Serverless” option Technology that Flow is built on Example: Poll the database every minute for 10 days + execute 500 workflows = $15/10 days Glue between applications Triggers: recurring schedule, on demand, HTTP, API News features released in frequent updates

17 Azure App Service WebJobs
Pros Cons Deployment PaaS – Managed VM, High availability Multiple apps on single service plan Easier (than SF) for developer to start developing apps Can run continuously or triggered Scales with App Service plan – setup manually You pay for it while it is deployed Timeout limit No remote desktop Local Dev and Debug Version Control Deploy directly from VS Continuous Integration/deployment Easier for a developer to get up to speed PCI, SOC, ISO compliant Triggers: Blobs, Service Bus, Queue, Timer, continuous, custom No remote desktop

18 Azure Service Fabric Best for Microservices
Pros Cons Deployment Best for Microservices Control over how workloads are run Load-balancing, scaling, failover, replication, stateful VM Scale Sets in clusters Self-hostable Fault Analysis Service Learning curve Code changes will be necessary to forklift code. Re-architect is better approach. Expensive initially because a production system uses 5 VM’s - Value increases as you add more services No remote debugging Local Dev and Debug Deploy directly from VS PowerShell and CLI deployment Deploy many services at one time Big learning curve. Better choice if you plan to migrate legacy applications away from an enterprise data center Stateful microservice removes the need for queues and caches. Stateful services are high availability and low latency. Auto-scaling built in to clusters. With the Fault Analysis Service, you can induce meaningful faults and run complete test scenarios against your applications

19 Wrap up - Which one is best?
Azure Functions Just getting started and want to experiment Specialized integration Extending Azure Logic Apps Azure Logic Apps Glue for mashups Azure WebJobs Already have a website and want to piggy-back on that service plan Credit card processing Azure Service Fabric Re-architecting a legacy environment Require high level of control and resilience

20 Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Azure Architecture Share our Best Practices Microservice

21 Design Considerations
Must read: Azure Architecture Center ( Application Architecture Guide Reference Architectures Cloud Design Patterns (32 of them) Best Practices Performance Antipatterns Design Checklists Reference architectures – Network DMZ, Hybrid, Identity Management, Managed web apps Performance Antipatterns – Busy Database, Chatty I/O, Extraneous Fetching (too much data),

22 Microservice Architecture Design Patterns
Increase the velocity of application releases by decomposing the application into small autonomous services that can be deployed independently D365 Shows some example cloud architecture design patterns. There are over 30 to choose from. The Sidecar pattern is for features like logging, and security that may be part of your framework to provide orthogonal services. Anti-corruption layer deals with exposing views of data from the legacy system in a manner consistent with the new system and allows the legacy system to function without modification. The Strangler is used to direct data requests to specific pieces of functionality that is being incrementally moved form the legacy system to a microservice architecture. Ambassador patterns act as proxy to services to allow control over routing, resiliency, security features, and to avoid any host-related access restrictions. Source:

23 BASE - Design Patterns for Eventual Consistency
Use Event Sourcing and CQRS Pattern Data Consistency Decouple recording the transaction from updating Reconciliation between services is easier Retry Pattern Best for transient faults like timeouts Service Bus provides this (to a degree) Idempotent methods Same result if called multiple times Your on-premesis systems typically are using a SQL technology that guarantees data consistency, (also called ACID: Atomicity, Consistency, Isolation, Durability) Dynamics 365 lives in the BASE world. Basically Available, Soft state, Eventual consistency means that transactions are distributed and they are eventually updated in the database(s). This means that D365 may not be 100% accurate all the time and you must design processes that will catch inconsistencies and trigger a process to fix it. Event Sourcing is used to keep object state history so that it can be re-evaluated and corrected Command Query Responsibility Separation is used to de-couple the recording of the transaction from an actual update Use Materialized Views design pattern to present the result of the transactions Retry Pattern depends on what the fault is – cant help you if your code is broken.

24 Potential for Inconsistent Data
Two transactions are created to update the same account They both get the starting balance They both are adding a different amount to the balance Last one “wins” Example of what could go wrong if you create a system with ACID in mind, but you are really using a BASE system: You start with a known account balance But you end with an incorrect balance What happened? What was the balance before the transactions? If this is milliseconds apart, it would be very hard to tell what is going on. You could use logging, but then someone has to monitor it, and how do you fix it?

25 Event Sourcing and CQRS to the rescue!
Two transactions are created to update the same account They both append their update to a journal They both send a message to another process to update the balance The balance is calculated twice, but it is accurate Command Query Responsibility Separation By keeping the entire history of changes, we no longer really need the Balance (State) of the account We can completely recalculate the balance, or re-play a series of events (think debugging). We can also see the balance at any point in time. Eventually all accesses to an item will return the last updated value. By pushing updates into a transaction log, process does not have to wait for CRUD operations The cost: storing more data, overall system is more complex because it is many smaller parts, overhead to complete the task is more

26 Plan for Failure! Compensating transaction Pattern
Un-do in reverse order Create Exception entity to manage resolution Provide a view of exceptions in the dashboard Provide error with steps to resolve it For each transaction, keep track of steps taken If the process is not completed, have a process that will walk back through and undo what was done You are effectively creating what you might have done If you have used transactions in SQL Server Source:

27 Introduction Reasons to extend D365 to Azure High level concepts Services that fit your needs Azure Architecture Share our Best Practices

28 Cloud Service Lessons Learned
“Classic” service – not part of ARM deployments It was an easy starting point No timeouts or code size limits You should employ Async Tasks Doesn’t run under CSP license Runs all the time like a VM No options to use KeyVault No options for dynamic configuration of Web.config/App.config ARM – Azure Resource Manager allows you to build scripts to deploy services None of these reasons should stop you from using it, unless you are selling D365 licenses under a CSP model Microsoft has not been putting any effort into this area – seems like it is going away Our customers only get about 5% of CPU during peak periods, but pay all month for it

29 Best Practices Test Location and latency
Check which datacenter your CRM instances is located Server Name First Two Letters Region BL East US BN North Central US BY West US Location, location, location – Minimize the latency because it can really cost you

30 Best Practices Time Zone Service Plan Azure Resource Manager
Integration account in customers timezone Work with localized datetime fields using UTC Service Plan Stress test with the smallest one and work up Azure Resource Manager Group by Resource (compute, service plan, storage, service bus) Tags: Owner, Environment, LoB Function(s), Tier, Project Test Driven Development Minimize “chatter” using API Transactions, ExecuteMultiple Monitor your usage My profile was for East coast time, but customer was in Houston and they saw my changes as having the wrong time. Keeping track of your services using ARM is really important to containing costs - Make a governance policy on who owns what using tags! TDD costs a bit more up front, but really pays for itself after sprint 1

31 Best Practices Do not include the Service Endpoints in the D365 Solution file that gets promoted to production Have your subscribing service validate that the message is coming from the right domain SB message has the sending domain – compare it to the D365 connection Use one service bus “namespace” per D365 instance Don’t Choose free tier of Service bus for production Configure to use many queues (or topics, depending on subscription) Use a non-interactive account in D365 Reduce license cost If you include the Service Endpoints in the solution file that you promote from dev to production, you will be connecting the production instance back to the development Azure environment The message on the service bus has the domain of the system that created the message. Compare it to the connection string domain to make sure they are the same. You can differentiate which messages belong to which programs by using queues and topics – don’t use multiple service buses In D365, check the “non-interactive” box on the user account you use for integration.

32 Logic App Best Practices
Azure Logic Apps Use webhook for trigger on the “business logic” Modular design Gives you multiple options to consume the logic later Avoid Excessive polling Cost is $ /action Ex: every 10 seconds= 259,200 actions=$6.48 Avoid IoT and Database conversions Best used for stochastic loads

33 Use EntityCollection() for Transactions
Developer Sidebar Use EntityCollection() for Transactions EntityCollection() Relationship() EntityCollection Entities.Add() <entity>.RelatedEntities.Add() (early bound) Create(<entity>) These are the classes you need to combine to into creating a transaction The benefit is to reduce the amount of chatter with the API, and it is an all-or-nothing, which saves you from having to clean up partial transactions

34 Transaction code sample
Developer Sidebar Transaction code sample Code Description Guid duesHeaderId = Guid.NewGuid(); Create a new ID so that we can create a reference in each detail record EntityCollection duesHeaderDetailCollection = new EntityCollection(); Create a collection that will hold all the child records Relationship duesHeaderDetailRelationship = new Relationship("bpt_bpt_duesheader_bpt_duesdetail"); Define the relationship that links the entities together duesHeader.bpt_DuesHeaderId = duesHeaderId; Assign the PK of the header record duesHeader.bpt_MemberId = Contact.ToEntityReference(); Assign some values to the header - this is the member Xrm.bpt_duesdetail duesDetail = new Xrm.bpt_duesdetail(); Create an instance of the child entity record duesDetail.bpt_DuesBillId = new CrmEntityReference( Xrm.bpt_DuesHeader.EntityLogicalName, duesHeaderId); Populate the entityreference from the child back to the parent record duesHeaderDetailCollection.Entities.Add(duesDetail); Add the child record to the parent duesHeader.RelatedEntities.Add(duesHeaderDetailRelationship, duesHeaderDetailCollection); Complete the relationship by adding the child collection to the parent entity service.Create(duesHeader); Call the service to create both records at the same time Some key elements are color coded to make it easier to follow where they are initialized and used. This sample is a membership dues Header / Detail A key point – the duesHeaderId is manually assigned in order to build the link between the parent and child record Typically, you would have more than one detail (child) record.

35 Resources: Platform feature comparison: web-site-cloud-service-vm Azure Architecture: Azure Functions Tools for Visual Studio: functions/functions-develop-vs Write and test functions locally: Azure Functions forum: US/home?forum=AzureFunctions&filter=alltypes&sort=firstpostdesc Continuous Deployment: continuous-deployment Subscription Governance: manager/resource-manager-subscription-governance

36 Resources (cont): Design Patterns for Microservices: patterns-for-microservices/ WebJobs Extensible Triggers: and-binders-with-azure-webjobs-sdk alpha1/ Be careful of the Logic App Consumption Prizing model:

37 Bright ideas for better performance
Q&A

38 Bright ideas for better performance
Thank You! Nelson Johnson


Download ppt "Extending the Dynamics 365 World With Microsoft Azure"

Similar presentations


Ads by Google