2. Zero-Downtime/Blue-Green Sitecore Deployments

3. Agenda What is DevOps Provisioning and Deployment
Branching Strategy - GitFlow
MS deployments slots
Blue/Green deployments

4. What I do???

5. What is a Blue/Green Deployment?
Two Identical environments
Blue
Green
One environment is live
One environment is not live
Load balancer used to direct traffic to the ‘live’ environment

6. Why Do We Care? Deployment technique that reduces downtime
Rollback to prior version if needed

7. What is the Challenge? Sitecore is a CMS with a shared data layer
Data isn’t owned solely by development team
Sitecore owns the schema
Content authors own ”content”
End-users indirectly generate content
Complex inter-relationship between xDB and SQL and search indices

8. Cattle vs Pets Analogy

9. DevOps
DevOps is the combination of cultural philosophies, practices, and tools that increases an organization's ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes.

10. DevOps – Conversation

11. DevOps Life Cycle & Habits

12. DevOps Practices & 7Cs

13. Continuous Confusion

14. Environment Provisioning

15. isango! – First step towards DevOps

16. CI & CD Process and Tools
Plan
CODE
Build
Code Quality
Release
QA Verification
Performance Monitoring
BlazeMeter
ReSharper

17. Standard Deployment Process

18. Gitflow - Smart Branching
A Gitflow is a recipe or recommendation for how to use Git to accomplish work in a consistent and productive manner. Gitflow encourage users to leverage Git effectively and consistently. Git offers a lot of flexibility in how users manage changes. Given Git's focus on flexibility, there is no standardized process on how to interact with Git. When working with a team on a Git managed project, it’s important to make sure the team is all in agreement on how the flow of changes will be applied. To ensure the team is on the same page, an agreed upon Git workflow should be developed or selected.

19. Branch Workflow Finishing Feature branch Starting feature Branch
Start a release
Finishing release
Starting hotfix
Finishing hotfix

20. Traditional Deployment

21. Traditional Deployment

22. B/G Deployment- Concept

23. B/G Deployment

24. Deployment Slots

25. What Sticks and What Doesn’t
SWAP Operation: Let’s understand the sequence of actions that happens when a staging slot is swapped to production.
The App settings and connection strings that are marked as “Slot” are read from the production slot and applied to the site in the staging slot. That causes the site’s worker process to be recycled for those changes to take effect and become visible as process environment variables.
Then the site in the staging slot gets warmed up. To warm up the site an HTTP request is made to the root directory of the site to every VM instance (each web worker) where site is supposed to run.
After warm up has completed the host names for the sites in production and staging slots get swapped.
To avoid any downtime and to keep swap operation’s time period minimum, we do not want Sitecore Application to restart as Sitecore has a high initialization time. To achieve this, we do not change any App settings or connection strings during this operation. And we make the Staging Slot with its settings to be a Production Slot. And here we use Blue/Green Deployments.
As all the traffic goes to production slot and the Worker Process is Swapped a client might lose the connection if we keep the session state within the process. So to avoid this we store the session state out of the worker process, to be specific in Redis. This way a user’s session is not affected by the swap operation.

26. Release flow in B/G deployment

27. Sitecore B/G Deployment
Blue/Green Deployment: Above we mentioned about B/G deployment. B/G deployment is a technique that reduces downtime and risk by running two identical production environments called Blue and Green. As you prepare a new version of your software, deployment and testing takes place in the Slots (Staging Slot) that is not live: in this example, Green. Once you have deployed and fully tested the software in Green, you swap the Slots so all incoming requests now go to Green instead of Blue. Green is now live, and Blue is idle. In addition, blue-green deployment reduces risk: if something unexpected happens with your new version on Green, you can immediately Swap again to the last version.
Web Database to be used should not be the same for the slots. Else you will end up with New Templates and data on Live Sites which can cause issues. So we need B/G in web database so that production site remains unaffected from data side as well.
Now for a website like Isango search is very important. So to avoid any update of new data or removal of old data or fields in SolrCloud and Coveo indexes we need B/G in indexes as well. To achieve this, SolrCloud web indexes are kept separate for the two slots and Coveo farms are also kept separate for the two slots.
Now we have separate Web Database, SolrCloud and Coveo indexes for our Staging Slot. This makes sure that our deployment to the production environment does not affect Live sites in any aspect.

28. Release, Branches and Environments

29. CD Auto Scale and Zero Downtime High-Level Arch.

30. Sitecore B/G Deployment in isango!
Content Management
Web root
Modules
Updated packages
TDS Deployment
Publishing Service
Content Management
Web root
Modules
Updated packages
TDS Deployment
Publishing Service
Content Delivery
Staging Slot
( idle state)
Production Slot
Master DB
Content Freeze
Blue Green Selection ( Kudu + PowerShell)
Master DB
Deploy & Substitute variables web packages
Content Delivery
Deploy TDS packages
Production Slot
Octopus Deploy is the deployment automation server designed to make it easy to orchestrate releases and deploy applications, whether on-premises or in the cloud. As an Octopus user, we define the process for deploying our software. We specify the environments and the targets where we will deploy your software. We manage our deployments to Slots and Blue Green Deployments using Octopus Deploy. Let’s discuss the process below:
Post a message in Microsoft Teams so that all the members of the team know that a deployment is started.
A PowerShell Script which uses KUDU API to get the Connection String from the Web App Production Slot (Live Site) than it checks for two words in the connection String ‘blue’ and ‘green’. If it finds ‘blue’ then it sets the value of a variable (Environment.Color) in Octopus to be ‘green’. If it finds ‘green’ then it sets the value of a variable (Environment.Color) in Octopus to be ‘blue’. If neither ‘blue’ nor ‘green’ value is found, then the script fails and the deployment stops.
This way we make sure that if the running environment is blue then deployments go to green and vice-versa.
with Blue Green Details. This tells us about the color of the environment which will be used in the deployment and which environment color is on Live Site currently. Just for the information so we do not have to check configuration files manually in the web app to figure out the color of the environment. This is important to know the color of the environments so that one can check the correct Indexes and database if required in future.
Now we check Staging Slot of Content Delivery and if it is running than we stop it (Production slot gets the 100% traffic always so it does not affect the live sites). We also stop our Web App of Content Management (Content Management runs without slots and this can be brought down as it is internal and everyone is aware about the content freeze).
Now we Deploy Web Package to Staging Slot of Content Delivery and Production Slot of Content Management. This web Package contains all the code as defined earlier.
Now we Deploy Environment Package to Staging Slot of Content Delivery and Production Slot of Content Management. And during this deployment we substitute values of the variables in the configuration files.
This package contains multiple configuration patch files which includes SolrCloud configuration, Coveo Configuration, Connection String, Web configuration, ReleaseTag for CDN Files and so on.
In SolrCloud Configuration Patch file we use variables for core and its Alias names. For web Indexes these Variables are concatenated with Environment.Color variable whose value is set in step 2. So the SolrCloud Web index will be switched to Blue or Green accordingly and Live Web Indexes will not be affected by the Deployment
In Coveo Configuration Patch file we use variable for Farm Name. This Variable is concatenated with Environment.Color variable whose value is set in step 2. As the Coveo Farm is changed, Indexes of Coveo are also changed and will be switched to Blue or Green accordingly and Live Indexes will not be affected by the Deployment.
In the Deployed Connection String file, we use variable for Web Database Name and this Variable is concatenated with Environment.Color variable. Then we use a PowerShell script which uses kudu API also to connect to App Service to do the transformation of original Connection String with the Deployed one and change the web Database to Blue or Green accordingly and Live Web Database will not be affected by the Deployment.
We also use a Release Tag whose value is a Date-Time String which is used in the URLs of static files (JS and CSS) for the versioning purpose.
We use AWS Cloud Front as our CDN. CDN has one origin which is our Production Slot of Content Delivery so that it can cache static files like css, js and media from sitecore. If we add one more origin (Staging Slot) for the same files than during deployment all the new Static files will be cached at the CDN and the live site will start using them. This can create problems on Live sites. To avoid this situation, we introduce Function with CloudFront. This function is executed before CloudFront forwards a request to the origin. In this function we check for two headers in the request ‘Referer’ and ‘Origin’.
If referrer value contains a Defined word (which is usually the URL of site in staging slot), in our case (*staging.isango.com*) then it uses a Secondary Origin ID of CloudFront which has a different origin (Staging Slot). Else it returns production Origin ID of CloudFront
If this function cannot find any referrer and cannot decide the Origin ID to be used on the basis of the referrer than it checks for the origin Header value, if origin value contains a Defined word (which is usually the URL of site in staging slot), in our case (*staging.isango.com*) then it uses a Secondary Origin ID which has a different origin (Staging Slot). Else it returns production Origin ID of CloudFront
Note: At this stage our Staging Slot of Content Delivery and Production Slot of Content Management are connected with the same Web database, SolrCloud and Coveo Indexes and if we make any Item related changes in the CM or publish anything or Index anything then it will not affect the Production Slot of Content Delivery as it is isolated in the context of site, code, database and Indexes. Now if any business user wants to make any changes in the Live Sites then they cannot until the SWAP operation is performed.
Now we deploy Sitecore Package Deployer. The Sitecore Package Deployer uses a Sitecore Job to automatically deploy update packages that have been copied into a folder on the Sitecore server. More information about it:
Now we deploy Toolcare Publishing Service. Which we use to publish site in the end of the deployment pipeline Remotely.
Now we Start our Staging Slot of Content Delivery and Production Slot of Content Management
Once CM is up we deploy TDS update packages and using Sitecore Package Deployer we process these packages.
Once the packages are processed successfully Deployment Pipeline Initiate a Full Site Smart Publish using Toolcare Publishing Service and Publishing Runner on remote machine.
This steps takes time if deployments are infrequent as the data differences between master and web-green or web-blue (whichever is used on the slot during the particular deployment) will be large.
Once it is done, Testing is started on staging Slot. And if everything seems fine then SWAP operation is performed.
We just need content freeze during the deployments but live sites are unaffected.
Testing in staging Slot
Staging Slot
Swap

31. Questions?

32. If you’d like to talk more about Blue/Green deployments or have any question, feel free give me a holler.
Prashant Kumar
Head Of Technology
isango! (Hotelbeds)
@ThePrashantyagi

33. Feedback