1. Provisioning Best Practices for MongoDB Mrinal Sarkar Solutions Architect

2. What We’ll Cover Provisioning approaches MongoDB Ops considerations
Hardware considerations
Case studies
Recommended Best Practices
How to get started

3. Audience
Operations teams responsible for MongoDB provisioning, sizing projects
At the end of this session, you should have a general idea of:
Best Practices, Recommendations* for MongoDB provisioning
What factors affect system performance
How to leverage MongoDB tools & Consulting services for sizing & provisioning projects
*This discussion is aimed as a general Framework/guidebook NOT a prescriptive formula as each project is different & unique in scope.

4. Ever Heard these?
“Our systems have not kept pace with data growth. Last month we experienced a 4 hour downtime…”
“My report is so slow today. It was fine when we ran it on Test cluster last week, but now that its on prod, its terrible…”
“I’m seeing a lot of Cache usage even though my DB is not so busy. What’s going on?”

5. Provisioning Methods Theoretical Approach Empirical Approach
Based on past experience/data/metrics
Gross estimates driven
Unreliable & routinely prone to errors
Quite common (unfortunately)
Empirical Approach
Based on Projected Specs & Metrics
Sample Test Estimates / POC Driven
Detailed, backed by data. Essentially a Scientific Method
In this discussion, we will be reviewing the Empirical Approach

6. Empirical Approach Steps:
Start with Business Requirements
Translate Reqs to Specs & SLAs
Design Test Schema
Configure Test Hardware
Conduct POC to Validate Design & Hardware.

7. Empirical Approach is Scientific Approach
Scientific Method
Empirical Approach
Observe & Gather Facts
Start with Business Requirements
Formulate Hypotheses
Translate Reqs to SLAs
Develop Test Plans
Design Test Schema
Configure Test Hardware
Conduct Experiments
Conduct POC
Validate Hypotheses (& repeat)
Validate Design & Hardware.

8. Typical Examples: Business Reqs SLAs Schema Test Hardware POC
eg: Globally available, Real time App
SLAs
eg: Query response in <50ms, Failover in <2s
Schema
eg: 3 collections, 7 indexes on fields x,y,z, Data size n GB
Test Hardware
eg: N Cpu, m Cores, x RAM, y GB Disk, z Gbps nw
POC
eg: Sample Tests under different Loads and iterations

9. MongoDB Operational considerations

10. Operational Considerations – Key Areas
In our observation, the most frequent cause of Performance issues is:
Poor Application Design:
Sub-optimal Design/Model & config-issues, missing Indexes, etc.
RAM, Storage and Network are really secondary factors

11. Operational Considerations - Performance
In provisioning projects for MongoDB workloads and performance SLAs a vital consideration is:
Working Set
Bottlenecks happen when workloads arrive at a given point more quickly than system can handle.

12. Operational Considerations – Working Set
What is a “Working Set”?
The Working Set is the portion of your data that clients access most often
i.e.
frequently accessed collections and related Indexes.
Note: Working set need not be ALL your data set.
Indexes are vital part of Working Set in MongoDB

13. Explain plan output (no index)…
winningPlan" : “{
...
”stage" : ”COLLSCAN",
"nReturned" : 2,
"executionTimeMillis" : 270,
"totalKeysExamined" : 25820,
"totalDocsExamined" : 25820,
"executionStages" : { }
COLLSCAN
Full collection scan
25820 docs scanned
No Indexes!

14. Explain plan output (Index)
Create Index:
db.zips.createIndex({state:1, city:1, pop:1})
IXSCAN
Same # of docs
Much fewer Docs & Keys examined
Much Faster than before
"winningPlan" : {
"stage" : "IXSCAN",
...
"executionSuccess" : true,
"nReturned" : 25820,
"executionTimeMillis" : 40,
"totalKeysExamined" : 450,
"totalDocsExamined" : 100, }

15. MongoDB Memory Optimization

16. Memory – size it for Working Set
Ask this Question:
What is the size of your most frequently queried data and their indexes?
Note:
Working Set should Ideally reside in RAM
At the very least, Indexes must fit in RAM (need not be entire data set)
Good place to start – db.collection.stats(), explain(), visual Profiler, etc

17. Why should Working Set be in RAM?
FACT: In terms of speed, RAM beats disk 1000x.
Hence, to extract maximum performance from your system the Working-set should stay in RAM.
Ideally, at least all the Indexes in Working Set must fit in RAM
Bottlenecks happen when workloads arrive at a given point more quickly than system can handle.

18. Ensure indexes fit in RAM
Good
Okay
Risky!
Data not in the Working Set must be retrieved from the disk
When the Working Set exceeds RAM, you have the options of scaling out(sharding) or scaling up(add more memory).

19. MongoDB Storage Engines

20. MongoDB Storage Engines
WiredTiger - default SE v3.2 (introduced in v3.0)
Document-level concurrency
Disk Compression
Multi threaded
Better performance on many workloads
In-Memory - new in predictable latencies, volatile
MMAPv1 - original SE - high volume, in-place updates
Key Takeaway - WiredTiger is the preferred SE for most MongoDB deployments.

21. 7x Performance, 50% Less Storage
How: WiredTiger Storage Engine
Same data model, same query language, same ops
Write performance gains driven by document-level concurrency control
Storage savings driven by native compression
MongoDB 3.0
MongoDB 2.6
Performance

22. WiredTiger Cache & Data Blocks
Total RAM
Default 60% of RAM – 1GB
WiredTiger Cache
(uncompressed)
File System Cache
(compressed)
Non-mapped
storage.wiredTiger.engineConfig.cacheSizeGB

23. WiredTiger YCSB Benchmarking

24. WiredTiger YCSB Benchmarking

25. Other Storage Considerations
7,200 rpm SATA
~ IOPS
15,000 rpm SAS
~ IOPS
Amazon SSD EBS
~ 4000 PIOPS / Volume
~ 48,000 PIOPS / Instance
Intel X25-E series SSD
~ 5, IOPS
Fusion IO
~ 135, IOPS
Violin Memory 6000
~ 1,000,000 IOPS
Key Takeaway - Higher IOPS generally preferable. Use WiredTiger to maximize hardware.

26. CPU & Network Considerations

27. Factors affecting CPU usage
Index Sorts vs In-memory Sorts
Multithreading support (WiredTiger)
Parallelization (Sharding)
Aggregations
NUMA, Encryption, Compression etc
Key Takeaway – Use optimal Indexes & WiredTiger SE. Use multi-core, hi-speed CPUs whenever possible.

28. Factors affecting Network Usage
Latency
WriteConcern
ReadConcern
ReadPreference
Throughput
Update/Write Patterns
Reads/Queries
Sharding
Chunk size and migrations
Rebalancing
Key Takeaway – Localize & Parallelize Data access.
Choose High Bandwidth, Low Latency network.

29. Great, so how do I Track & Monitor Performance issues?

30. How to Diagnose Bottlenecks?
Command-line Tools
Ops Manager / Cloud Manager
Always available, no extra setup required generally
Intuitive Graphical Interface. Automated Provisioning, Monitoring (and Backup)
Manual & Time consuming
Simple One time setup
Reactive
Proactive

31. The Best Way to Setup and Manage your MongoDB Infrastructure
MongoDB Ops Manager
The Best Way to Setup and Manage your MongoDB Infrastructure
Up to 95% Reduction in Operational Overhead
Single-click provisioning, scaling & upgrades, admin tasks Monitoring, with charts, dashboards and alerts on 100+ metrics Visual query performance diagnostics, index suggestions, and automated index rollouts

32. Key Takeaway: Leverage Ops Manager for Provisioning, Monitoring and Diagnosis.

33. Key Provisioning Questions
What is you Data access pattern – Read/Write/Mixed?
What is your Working Set size?
Does the Working Set fit in RAM?
Are there any missing Indexes?
What is you initial data volume and growth rate?
What is the peak load you need to provision for?
How Geographically dispersed is your architecture?

34. Case Studies

35. Case Study #1: A Large Storage Vendor
Provisioning: Ops Manager Diagnosis

36. Case Study #1: A Large Storage Vendor
Problem statement:
Rapid Data Growth 2 TB -> 10 TB in 6 months
Replication bottlenecks, Stale secondary
Query Performance severely impacted

37. Case Study #1: A Large Storage Vendor
Initial setup:
3 Node RS, 2 DC
Bare metal, Quad core x86
256 GB RAM
7.2K SATA disks
10Gb/s
v3.0

38. Case Study #1: Diagnosis
Ops Manager Metrics & Graphs showed
Severe Network choking
on one member
Oplog maxed out
High Cache usage
Ops Manager Profiler revealed
Hotspots (3 Main collections in TB range) – Working set 300 GB
Unoptimized Queries, Collection scans, missing Indexes

39. Case Study #1: Recommendations
Identify and remove faulty NIC
Improved data replication
Increase Oplog size to 100GB
Oplog GB/hour stuck around 60GB
To cover Initial sync duration
Overcomes stale secondary issue
Continued..

40. Case Study #1: Recommendations
Increase RAM to 384 GB per node
More space to WT Cache
Working set now in Memory
Previous memory allocation (256GB was < Working Set of 300 GB)
Create 2 new Indexes & Leverage Prefix-compression
Improve Query Performance
Recommended by Ops Manager Profiler
Continued..

41. Case Study #1: Recommendations (contd.)
For Future Growth & Scale…
Shard the 3 Large collections
Start with 2 Shards in 2 DC
Use a compound shard key
Engage PS for Performance Eval & Tuning
For Future Growth & readiness
Key Takeaway – Ops Manager – Key to Diagnosis, Performance & Scale.

42. Case Study #2: A Large Online Retailer
Provisioning from a POC driven Approach

43. Case Study #2: A Large Online Retailer
Problem statement:
Migrate prod-catalog from SQL Server to MongoDB
2 main DCs running active/active
400* requests/sec
*(Fact: Includes spikes, On Cyber Monday they peaked at 214 reqs/sec )

44. Case Study #2: POC A POC yielded the following numbers:
4 million product SKUs, avg JSON doc-size 30KB
Need to service requests for:
a specific product (ie by product_id)
Products in a specific category (i.e. “Disks” or “Hard Drives”)

45. Case Study #2: The Math Partition by category
products in multiple categories duplicated
The average product appears in 2 categories, so we actually need to store 8M SKU documents, not 4M
8M docs * 30KB/doc = 240GB of data
270 GB with indexes = Working Set

46. Case Study #2: Recommendation
A single Replica Set
with enough RAM per node (384GB RAM/server)
4 node Replica Set
2 nodes in each DC, 1 in 2nd and 1 arbiter in a 3rd DC
Deploy using read preference ‘nearest’
Key Takeaway – conduct POC, Start with a simple Replica set design (& then shard if necessary)

47. Provisioning method best suited for your Use case depends on your Reqs.
There are some Best Practices though…

48. Recommended Best Practices – Data Model
Understand the Data
Workload type - Read/Write/Mixed?
Size of Data on Disk (Avg doc size, num docs)
Size of Indexes on Disk
Data Protection & Availability needs
Expected growth
WiredTiger SE enables Efficient Data model

49. Recommended Best Practices – Working Set
Understand Query Patterns
Understand Performance Reqs
Ensure well designed Indexes
Working Set resident in Memory is the key

50. Recommended Best Practices – Conduct POC
POC Forces you to:
Optimize schema / index design
Understand query patterns
Get a handle on Working Set size
POC / Load Test under multiple scenarios & durations
Note: In certain cases, benchmarking using frameworks like YCSB is quite helpful in metrics/sizing.

51. Recommended Best Practices – Ops Manager
Once you deploy initially, it is common for requirements to change.
Hence, Monitor and collect metrics regularly
Leverage Ops Manager Automation & Monitoring
Leverage Query Profiler

52. Pre-Production & Production
MongoDB Enablement
Consulting, training, and professional services throughout your project lifecycle
Design &
Development
Ops Rapid Start
Pre-Production & Production
Production Readiness
Health Check
Performance Evaluation & Tuning
Expansion
MongoDB Private Cloud Accelerator
We offer professional service for all teams throughout your development lifecycle, from design to production.

53. MongoDB Consulting Services.
In summary
MongoDB Provisioning Best Practices:
Start with Business Requirements
Translate Reqs to Specs & SLAs
Design Test Schema
Configure Test Hardware and Ops Manager
Conduct POC to Validate Design & Hardware.
Leverage Ops Manager
&
MongoDB Consulting Services.