1. HyScale: Hybrid Scaling of Dockerized Microservices Architectures
Before diving into the scaling aspect of things you might be wondering what Mas are and where they fit in the full stack and why we want to use them
What/where do they fit in or why are we interested in dockerized microservices architectures?
M.A.Sc. Thesis presented by Jonathon Wong
Department of Electrical and Computer Engineering
University of Toronto

2. Cloud Architecture
Application level (e.g. Netflix, includes login, browsing/search, streaming, etc)
Virtualization (i.e. house application) level two types but we deal with Docker containers, introduce at a high level (compare to VM)
Docker/container/microservice interchangeably
Resource management level: placement and resource allocation, scaling happens (the layer of interest)
Physical hardware/resources level: also enables virtualization
What are these mysterious microservices architectures?

3. Microservices Architecture
Catalog Service
Product Service
Libraries
Java 7
Python 2.7
Javascript
Libraries
Java 8
Python 2.7
Javascript
Customer Service
Cart Service
Loyalty Service
Order Service
Traditionally applications followed the monolithic architecture
Break into many small decoupled microservices and compare
A microservices architecture is a method of developing software applications as a suite of independently deployable, small, modular services, in which each service runs a unique process and communicates through a well-defined, lightweight mechanism to serve a business goal.
Login Service
Account Service

4. Benefits of Microservices Architectures
Independent redeployments without compromising application integrity
Only need to change one or more distinct services instead of redeploying entire application
Decoupled nature copes with failures
One service failing does not affect other services
Promotes and facilitates scalability
An overloaded service can easily be replicated to distribute load

5. Monetizing Microservices Architectures
Hosting your own microservices architecture (MA) is expensive
Customers/tenants typically pay cloud data centres to host their MA instances
Modern data centres categorize various classes of tenants
Each class associated with a service-level agreement (SLA) dictating various quality of service (QoS) requirements
QoS metrics usually revolve around availability of microservices
Violation of SLAs incur a penalty on the provider
Yeah I want to host my own! …
Now every business is doing it!
Problem hasn’t gone away, just shifted to the cloud data centre to deal with D:

6. Scalability Issues Data centres reaching physical limits in terms of:
Hardware resources
Operating costs
Space/Real estate
Data centre resources are also often underutilized by tenants
Data centres tend to overprovision resources to tenants to prevent SLA violations
Resources allocated to a tenant could be given to another instead!
Even worse, sometimes it’s not used at all!
Better ways of utilizing data centre resources must be identified

7. Cloud Goals Reduce overall operating costs
Increased physical resource utilization μ1 μ2 μ3
Want to achieve lower operating costs, usually through higher resource efficiencies
Resource management layer comes into play

8. Cloud Goals Reduce overall operating costs
Increased physical resource utilization
Reduce SLA violations
Add resources/scale services that are about to violate the performance SLA
Achieve high availability
Replication
Fault tolerance
Even though we can share resources, don’t want to impact performance (SLAs)
Still want to maintain high availability

9. Scaling Approaches Horizontal vs. Vertical Scaling
Granularity/Sizing
Placement
Reactive vs. Predictive Scaling
Timing
2 characteristics
HvsV: coarse vs fine grained (placement is implicit)
PvsR: do we perform action for now/future

10. Horizontal and Vertical Scaling
Traditional scaling techniques use either horizontal or vertical scaling on its own
Hybrid scaling leverages both horizontal and vertical scaling techniques simultaneously to achieve both high resource utilization and high availability
Vertical Scaling
Service
Memory
CPU
Disk
High load detected
Horizontal Scaling
Service
High load detected
Traditionally, scaling techniques applied to both VMs and containers either follow horizontal scaling or vertical scaling on its own.
Horizontal is good for fault tolerance and availability
Vertical is good for resource allocation granularity

11. Reactive and Predictive Scaling
Reactive scaling is popular in industry
Predictive scaling much harder to accomplish in practice
Trends are highly non-linear
Difficult to account for external factors
Reactive is simple, look at current/past values and accommodate (usually too late if spiky)
Good when unexpected fluctuations occur (usually overprovisions with a % buffer)
Predictive assumes trends exist (e.g. getting off work to play video games/annual sales)
External factors such as fads, political events
Examples are using ARIMA models or machine learning

12. Scaling Difficulties Multidimensional problem
There exists an infinite number of different resource configurations at a given point in time
How many machines (horizontal scaling)?
How many replicas per microservice (horizontal scaling)?
How many resources per microservice (vertical scaling)?
CPU
Memory
I/O
Intractable to iterate over every possible configuration!
Dynamic bin packing (alluded to in previous slide)
How do we tackle this?

13. Contributions
Develop novel predictive and reactive hybrid algorithms for Docker containers
Resource allocation granularity of vertical scaling
High availability and fault tolerance of horizontal scaling
Quick response to sudden fluctuations and flash crowds of reactive scaling
Pre-emptive resource allocation for workload trends of predictive scaling
Explore both predictive and reactive sides

14. HyScale Architecture Autoscaler Monitor Load Balancers
Main components:
Docker containers/ Microservices
Load Balancers and Clients
Node Manager (NM)
Monitor and Autoscaler NM NM NM
Due to lack of hybrid scaling architectures for containers, and to implement and benchmark/evaluate our scaling techniques
Microservices + replicas run on nodes
Clients speak to the server-side load balancers to identify which container instance to connect to.
Node managers issue docker commands to bring up/down containers and resize
Monitor is central arbiter which has view of all nodes and their usage, queries autoscaling module for scaling decisions and issues to NMs
Autoscaler can have any scaling algorithm implemented onto it
Load Balancers

15. Reactive Scaling: HyScaleres Algorithm
𝑀𝑖𝑠𝑠𝑖𝑛 𝑔 𝑚,𝑟𝑒𝑠 = 𝑟 𝑢𝑠𝑎𝑔 𝑒 𝑟 − 𝑟 𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑒 𝑑 𝑟 ∗𝑇𝑎𝑟𝑔𝑒 𝑡 𝑚 𝑇𝑎𝑟𝑔𝑒 𝑡 𝑚 𝑅𝑒𝑐𝑙𝑎𝑖𝑚𝑎𝑏𝑙 𝑒 𝑟,𝑟𝑒𝑠 =𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑒 𝑑 𝑟 − 𝑟 𝑢𝑠𝑎𝑔 𝑒 𝑟 𝑇𝑎𝑟𝑔𝑒 𝑡 𝑚 ∗0.9 𝑅𝑒𝑞𝑢𝑖𝑟𝑒 𝑑 𝑟,𝑟𝑒𝑠 = 𝑟 𝑢𝑠𝑎𝑔 𝑒 𝑟 𝑇𝑎𝑟𝑔𝑒 𝑡 𝑚 ∗0.9 −𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑒 𝑑 𝑟
The first approach I will show…
Usage is past values seen
- Reclaimable will tell us how much to scale down by
+ Acquired will tell us how much to scale up by

16. Evaluation: Microbenchmarks
“Microservices” running small CPU and memory workloads
Workload is triggered upon receiving request from client
Vary microservice resource type heterogeneity
CPU only
Memory only
Mixture of both
Emulate client request load to microservices
Constant
Wave
Mem
Mem + CPU
CPU

17. Compared our algorithms against Google’s popular Kubernetes horizontal scaling technique
Lower response times for CPU (negligible failures)

18. Results: Resource Utilization
Constant CPU Experiment
Wave CPU Experiment
Average CPU Usage (%)
RTECPU (ms/%)
Kubernetes
37.38
90.88
HyScaleCPU
34.21
86.17
HyScaleCPU+Mem
35.41
67.41
Average CPU Usage (%)
RTECPU (ms/%)
Kubernetes
50.34
60.47
HyScaleCPU
44.72
59.43
HyScaleCPU+Mem
45.94
44.47
Average CPU utilization per node
Explain RTE being a relative metric comparing response times overhead generated to resources expended
HyScale generates less response time overhead per % of resource used

19. Predictive Approach: Machine Learning
ANNs (artificial neural networks) are usually trained on a static dataset
Models old data trends well
Known for its generalization capabilities
Can result in poor predictions when encountering new data
Requires dataset to encapsulate all anticipated data
Online learning attempts to “learn” on the spot
Allows model to incorporate new data trends
Double-edged sword
Overwrites old learned trends
Next I want to introduce our predictive approach using machine learning
Dataset coverage is good
Online learning is especially good when you have lots of different jobs/workloads coming in dynamically (don’t have to do offline profiling)

20. Online Learning Technique
Experience buffer to encapsulate new and old trend information
Old data samples are filtered by Pearson’s product-moment coefficient
New data samples are pushed into a FIFO queue
Experience buffer is randomly sampled to create a training batch
Online training batch created for the ANN to train on in real-time

21. Results: Parameter Space Exploration
Parameter space search
Learning Rate and Update Rate are about how fast model learns off new online training batches
Similarity threshold and prior sample percentage are characteristics of the old data
Numbers aren’t that important here, trends are important
Sweet spot indicates where you don’t forget too much old, but learn enough new
Update rate number of iterations before each training step

22. Conclusions
Developed autoscaling architecture for benchmarking of various autoscaling algorithms
Supports hybrid scaling of Docker containers
Developed novel reactive hybrid algorithms (HyScale)
Leverages availability of horizontal scaling and granularity of vertical scaling
Developed novel predictive algorithm using ANNs and online learning
Model learns new trends while retaining old trends

23. Future Work
Implement predictive approach onto HyScale architecture for benchmarking
Combine reactive and predictive approaches into a single algorithm
Incorporate network and disk I/O into algorithms
CRIU? Weighted?
Support stateful microservices

24. Thank You!
Questions?

25. Containers vs. Virtual Machines

26. Reactive Scaling Example: Kubernetes
Horizontally scales containers according to the following equations:
𝑢𝑡𝑖𝑙𝑖𝑧𝑎𝑡𝑖𝑜 𝑛 𝑟 = 𝑢𝑠𝑎𝑔 𝑒 𝑟 𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑒 𝑑 𝑟
𝑁𝑢𝑚𝑅𝑒𝑝𝑙𝑖𝑐𝑎 𝑠 𝑚 = 𝑟 𝑢𝑡𝑖𝑙𝑖𝑧𝑎𝑡𝑖𝑜 𝑛 𝑟 𝑇𝑎𝑟𝑔𝑒 𝑡 𝑚
To avoid thrashing, the following condition must also be met:
𝑟 𝑢𝑠𝑎𝑔 𝑒 𝑟 𝑁𝑢𝑚𝑅𝑒𝑝𝑙𝑖𝑐𝑎 𝑠 𝑚 ∗𝑇𝑎𝑟𝑔𝑒 𝑡 𝑚 −1 >0.1

27. Reactive Scaling Example: ElasticDocker
Vertically scales Docker containers
Memory scaled up by 256MB and down by 128MB (threshold- based)
vCPU cores added or removed based on CPU utilization thresholds (90% up / 70% down)

28. Reactive Scaling Example: Self-Adaptive
While response time SLA violated, increase resources above utilization threshold for that service, create new VM instance if necessary
Threshold based removal of resources, VM removed if no more resources on it

29. Predictive Scaling Examples
Qazi Ullah uses ARNN and compares to ARIMA in the BitBrains fastStorage workload trace
Binbin Song uses LSTMs for predicting workloads in the Google cloud trace

30. Predictive-Reactive Scaling Examples
Multi-tiered (VM horizontal) probability distribution for predictive and threshold based for reactive
DoCloud (container horizontal) ARMA model for predictive and threshold based for reactive

31. CPU Container Experimental Setup*
A baseline microservice calculates 20,000 prime numbers per client request
Response times are measured
To create contention of CPU resources, the microservice is run alongside another third party container
Progrium stress consumes CPU resources by performing sqrt(rand()) calculations μ1 μ2
This simulates CPU load on the system from the request/response framework that is inherent in a microservices architecture.

32. Docker CPU Vertical Scaling*
Docker CPU shares:
Defines a container’s proportion of CPU cycles on a single machine μ1 μ2
1024
CPU shares
1024
CPU shares
3072
CPU shares
By tuning the CPU shares allocated to each microservice, we effectively control their relative weights.
For example, if two microservice containers run on a single machine with CPU shares of 1024 and 3072, the containers will have 1/4 and 3/4 of the access time to the CPU, respectively
4096
CPU shares
2048
CPU shares

33. Horizontal vs. Vertical Scaling: Experimental Scenarios: μ1 μ2
1024
400% CPU
5 scenarios to see the effects of each
For example, in the vertical scaling emulation, we allocated 1024 CPU shares to both the microservice and the progrium stress container, splitting CPU access time equally between the two. Assuming that nodes have 4 CPU cores, both are provisioned 2 CPU cores worth of access time. An equivalent horizontally-scaled resource allocation with 3 microservices running over 3 machines allocates 1024 and 5120 CPU shares to the microservice and the progrium stress container, respectively. This results in 1/6 of the CPU access time of each node for each microservice, again totaling to 2 CPU cores worth of access time (i.e., 1/6 of the 12 CPU cores). This equivalence was reproduced with 2 microservices and 2 nodes, and 4 microservices and 4 nodes.
Horizontal vs. Vertical Scaling: Experimental Scenarios
1. No CPU contention

34. Horizontal vs. Vertical Scaling: Experimental Scenarios: : μ1 μ2
2048
1024
200% CPU
For example, in the vertical scaling emulation, we allocated 1024 CPU shares to both the microservice and the progrium stress container, splitting CPU access time equally between the two. Assuming that nodes have 4 CPU cores, both are provisioned 2 CPU cores worth of access time. An equivalent horizontally-scaled resource allocation with 3 microservices running over 3 machines allocates 1024 and 5120 CPU shares to the microservice and the progrium stress container, respectively. This results in 1/6 of the CPU access time of each node for each microservice, again totaling to 2 CPU cores worth of access time (i.e., 1/6 of the 12 CPU cores). This equivalence was reproduced with 2 microservices and 2 nodes, and 4 microservices and 4 nodes.
Horizontal vs. Vertical Scaling: Experimental Scenarios
2. CPU contention

35. Horizontal vs. Vertical Scaling: Experimental Scenarios: : μ1 μ2 μ1 μ2
4096
4096
100% CPU +
100% CPU =
200% CPU
For example, in the vertical scaling emulation, we allocated 1024 CPU shares to both the microservice and the progrium stress container, splitting CPU access time equally between the two. Assuming that nodes have 4 CPU cores, both are provisioned 2 CPU cores worth of access time. An equivalent horizontally-scaled resource allocation with 3 microservices running over 3 machines allocates 1024 and 5120 CPU shares to the microservice and the progrium stress container, respectively. This results in 1/6 of the CPU access time of each node for each microservice, again totaling to 2 CPU cores worth of access time (i.e., 1/6 of the 12 CPU cores). This equivalence was reproduced with 2 microservices and 2 nodes, and 4 microservices and 4 nodes.
Horizontal vs. Vertical Scaling: Experimental Scenarios
3. Equivalent horizontal scaling with 2 replicas

36. Horizontal vs. Vertical Scaling: Experimental Scenarios: : : μ1 μ2 μ1 μ2 μ1 μ2
6144
6144
6144
67% CPU +
67% CPU +
67% CPU =
200% CPU
For example, in the vertical scaling emulation, we allocated 1024 CPU shares to both the microservice and the progrium stress container, splitting CPU access time equally between the two. Assuming that nodes have 4 CPU cores, both are provisioned 2 CPU cores worth of access time. An equivalent horizontally-scaled resource allocation with 3 microservices running over 3 machines allocates 1024 and 5120 CPU shares to the microservice and the progrium stress container, respectively. This results in 1/6 of the CPU access time of each node for each microservice, again totaling to 2 CPU cores worth of access time (i.e., 1/6 of the 12 CPU cores). This equivalence was reproduced with 2 microservices and 2 nodes, and 4 microservices and 4 nodes.
Horizontal vs. Vertical Scaling: Experimental Scenarios
4. Equivalent horizontal scaling with 3 replicas

37. Horizontal vs. Vertical Scaling: Experimental Scenarios: : : : μ1 μ2 μ1 μ2 μ1 μ2 μ1 μ2
8192
8192
8192
8192
50% CPU +
50% CPU +
50% CPU +
50% CPU =
200% CPU
For example, in the vertical scaling emulation, we allocated 1024 CPU shares to both the microservice and the progrium stress container, splitting CPU access time equally between the two. Assuming that nodes have 4 CPU cores, both are provisioned 2 CPU cores worth of access time. An equivalent horizontally-scaled resource allocation with 3 microservices running over 3 machines allocates 1024 and 5120 CPU shares to the microservice and the progrium stress container, respectively. This results in 1/6 of the CPU access time of each node for each microservice, again totaling to 2 CPU cores worth of access time (i.e., 1/6 of the 12 CPU cores). This equivalence was reproduced with 2 microservices and 2 nodes, and 4 microservices and 4 nodes.
Horizontal vs. Vertical Scaling: Experimental Scenarios
5. Equivalent horizontal scaling with 4 replicas

38. Horizontal vs. Vertical Scaling Analysis
Contention over CPU resources introduces 17% overhead
Replicated instances decrease overall CPU performance
Overhead within applications (in our case, the JVM) replicated several times affect response times
Preference for vertical scaling. Vertical scaling provided fastest request processing times when compared to the equivalently horizontally scaled instances.
Although Docker containers, themselves, have negligible overhead, contention over shared CPU resources introduces significant overhead
Contention would be further exacerbated by the presence of more co-located containers.
Vertical Scaling
Horizontal Scaling

39. Dockerized Microservices
Autoscaler
Packages all libraries and dependencies within an isolated container
Each Docker container hosts a microservice
Microservices receive and perform task requests
Each task consumes computing resources (CPU, memory, IO)
Microservices periodically heartbeat the Node Manager (NM)
Ensures liveness
Provides resource usage information
Monitor NM
Load Balancers
The heart of our architecture lies the actual Docker containers running the microservices hosted on each of the nodes.
Docker containers, as we all know, package all libraries and dependencies within an isolated virtual container. In our architecture, each of these Docker container hosts a microservice that receive and perform task requests. These requests can come from clients as well as other services running on the same node as well as other nodes in the network.
In our implementation, the microservices have been fitted with component that periodically heartbeats its Node Manager, sending information about the resource usage about the microservice to the Node Manager. (Point to the interaction on the diagram)
The heartbeats are also used by the Node Manager to ensure liveness and availability of each container.

40. Node Manager Node Manager
Ensures containers are running via heartbeats
Restarts service if crash detected
Heartbeats the Monitor
Ensures liveness
Forwards resource utilization of all containers running on node to Monitor
Performs resource update commands received by Monitor
Reallocates resources from one container to another
Uses docker-java client library to interface with Docker
Node managers, as briefly mentioned on the previous slide, receive resource info from the containers as well as ensure that containers are running via the heartbeats. If a container fails, the service is then restarted.
The node manager gathers all of the resource utilization information for all of its monitored microservices and forwards this to the central Monitor in the form of heartbeats. These heartbeats are also used by the monitor to ensure liveness for each of the nodes.
Node manager uses Docker java client library to interface with Docker. After the monitor makes a resource reconfiguration and scaling decision, it will forward this to the Node managers who execute it using Docker java client interface, which is just an open source Docker interface written in Java.

41. Monitor and Autoscaler
Ensures nodes are running via heartbeats
Receives resource usage information from all Node Managers (NM)
Has a centralized view of resource utilization across all machines
Performs resource adjustments
Provides the autoscaler module cluster state information (e.g., monitored node usage)
Receives resource adjustment decisions from the autoscaler
Autoscaler
The monitor is the main central arbiter of the system, tasked with gathering all the resource usage information from all the Node managers and making resource adjustment and scaling decisions. The resource adjustment decisions performed by the monitor are hybrid in nature; utilizing both vertical scaling and horizontal scaling.
The monitor periodically invokes the cost function to identify the resource configuration with the lowest cost and adjusts/scales resources to reach the optimal cost effective resource configuration.
The monitor also has a role in keeping the nodes available. The heartbeats sent by the Node Manager are used to keep track of liveness of the nodes by the Monitor.
Monitor NM

42. Clients and Load Balancers
The load balancers have knowledge of all microservices and their respective IP addresses
Perform load balancing for replicas of microservices
Clients contact load balancer to discover service location
Load balancer responds with IP address
Afterwards, clients directly send requests to the IP address
The load balancer has knowledge of all microservices, their replicas that exist because of horizontal scaling, as well as their respective IP addresses. It uses this information to load balance the incoming client requests among replicas of microservices
Clients first contact the load balancer to discover the service location. The load balancer performs its balancing technique and responds with an IP address of one of the replicas. Afterwards, the clients will send all of its requests directly to the IP address.

43. HyScaleCPU Algorithm: CPU Shares
𝑡𝑜𝑡𝑎𝑙𝐶𝑃𝑈𝑆ℎ𝑎𝑟𝑒 𝑠 𝑛 =𝑛𝑢𝑚𝑅𝑒𝑝𝑙𝑖𝑐𝑎 𝑠 𝑛 ∗1024 𝐶𝑃𝑈 𝑆ℎ𝑎𝑟𝑒𝑠 𝑛𝑒𝑤𝑅𝑒𝑞𝑢𝑒𝑠𝑡𝑒𝑑𝐶𝑃𝑈 𝑠 𝑟 =𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑒 𝑑 𝑟 −𝑅𝑒𝑐𝑙𝑎𝑖𝑚𝑎𝑏𝑙𝑒𝐶𝑃𝑈 𝑠 𝑟 +𝐴𝑐𝑞𝑢𝑖𝑟𝑒𝑑𝐶𝑃𝑈 𝑠 𝑟 𝑛𝑒𝑤𝐶𝑃𝑈𝑆ℎ𝑎𝑟𝑒 𝑠 𝑟 = 𝑛𝑒𝑤𝑅𝑒𝑞𝑢𝑒𝑠𝑡𝑒𝑑𝐶𝑃𝑈 𝑠 𝑟 𝑛𝑢𝑚𝐶𝑃𝑈 𝑠 𝑛 ∗𝑡𝑜𝑡𝑎𝑙𝐶𝑃𝑈𝑆ℎ𝑎𝑟𝑒 𝑠 𝑛

44. HyScaleCPU+Mem Algorithm
𝑀𝑖𝑠𝑠𝑖𝑛𝑔𝑀𝑒 𝑚 𝑚 = 𝑟 𝑢𝑠𝑎𝑔 𝑒 𝑟 − 𝑟 𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑒 𝑑 𝑟 ∗𝑇𝑎𝑟𝑔𝑒 𝑡 𝑚 𝑇𝑎𝑟𝑔𝑒 𝑡 𝑚 𝑅𝑒𝑐𝑙𝑎𝑖𝑚𝑎𝑏𝑙𝑒𝑀𝑒 𝑚 𝑟 =𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑒 𝑑 𝑟 − 𝑟 𝑢𝑠𝑎𝑔 𝑒 𝑟 𝑇𝑎𝑟𝑔𝑒 𝑡 𝑚 ∗0.9 𝑅𝑒𝑞𝑢𝑖𝑟𝑒𝑑𝐶𝑃𝑈𝑀𝑒 𝑚 𝑟 = 𝑟 𝑢𝑠𝑎𝑔 𝑒 𝑟 𝑇𝑎𝑟𝑔𝑒 𝑡 𝑚 ∗0.9 −𝑟𝑒𝑞𝑢𝑒𝑠𝑡𝑒 𝑑 𝑟 𝐴𝑐𝑞𝑢𝑖𝑟𝑒𝑑𝑀𝑒 𝑚 𝑟 =𝑚𝑖𝑛 𝑅𝑒𝑞𝑢𝑖𝑟𝑒𝑑𝑀𝑒 𝑚 𝑟 , 𝐴𝑣𝑎𝑖𝑙𝑎𝑏𝑙𝑒𝑀𝑒 𝑚 𝑟

45. Evaluation “Microservices” running miniature CPU and memory benchmarks
Benchmark is triggered during an incoming request from client
Vary SLA heterogeneity for each microservice
Vary microservice resource type heterogeneity
Memory only
CPU only
Mixture of both
Emulate client request load to microservices
Constant
Wave
CPU
Mem
Mem
Mem + CPU
CPU
Mem + CPU

46. User-perceived metrics
Explain response time
Lower response times for CPU (negligible failures)
Kubernetes lower failed requests due to replication adding more memory implicitly
Connection failures make request return quickly lowering average response times

47. SLA metrics
Explain num SLA violations, violation time, and total violation cost
Lower violation time and and costs for hyscale algos
Wavemix failed requests return too quickly and reduce the number of violations

48. HyScaleCPU is better due to lots of docker update commands being issued

49. Results: Resource Utilization for ConstantCPU and WaveCPU Experiments
CPU Usage (%)
Average
Min
Max
RTECPU (ms/%)
Kubernetes
50.34
21.56
83.75
60.47
HyScaleCPU
44.72
40.89
53.18
59.43
HyScaleCPU+Mem
45.94
35.96
54.95
44.47
Explain RTE being a relative metric comparing response times overhead generated to resources expended
HyScale generates less response time overhead per % of resource used
CPU Usage (%)
Average
Min
Max
RTECPU (ms/%)
Kubernetes
37.38
8.98
67.13
90.88
HyScaleCPU
34.21
26.10
39.90
86.17
HyScaleCPU+Mem
35.41
32.63
41.19
67.41

50. Results: Resource Utilization for ConstantMix
CPU Usage (%)
Mem Usage (%)
Average
Min
Max
Kubernetes
38.86
24.08
62.21
42.54
27.78
50.79
HyScaleCPU
36.25
33.42
42.56
41.00
28.21
48.61
HyScaleCPU+Mem
37.48
34.72
46.81
41.03
28.17
48.10
RTECPU (ms/%)
RTEMem (ms/%)
Kubernetes
49.28
45.02
HyScaleCPU
61.99
54.80
HyScaleCPU+Mem
42.26
38.61

51. Results: Resource Utilization for the WaveMix Experiment
CPU Usage (%)
Mem Usage (%)
Average
Min
Max
Kubernetes
16.30
9.72
21.58
42.08
31.66
47.82
HyScaleCPU
19.98
17.03
23.11
41.04
27.68
48.70
HyScaleCPU+Mem
24.22
22.13
28.74
42.16
29.35
50.36
RTECPU (ms/%)
RTEMem (ms/%)
Kubernetes
96.38
37.33
HyScaleCPU
73.02
35.55
HyScaleCPU+Mem
62.59
35.96
RTEmem is skewed due to low response times generated by failed requests
Note that most real spikey world workloads are represented by wave mix

52. Evaluation: BitBrains Workload
GWA-T-12 BitBrains Rnd workload trace containing 500 VMs
Add RTE table

53. Results: Resource Utilization for BitBrains Workload
CPU Usage (%)
Mem Usage (%)
Average
Min
Max
Kubernetes
10.57
5.66
33.19
21.01
19.41
22.63
HyScaleCPU
12.52
4.90
67.48
20.75
19.43
22.09
HyScaleCPU+Mem
8.08
6.05
15.40
21.30
19.50
23.19
RTECPU (ms/%)
RTEMem (ms/%)
Kubernetes
68.21
34.32
HyScaleCPU
113.34
68.39
HyScaleCPU+Mem
55.82
21.17

54. ANN Models xt-n Basic ANN LSTM LN-LSTM Multi-layer LN-LSTM
Convolutional LN-LSTM
xt+1 xt
Blackbox: any ANN could be used for the online learning technique (so we tried a few)
Explain basic input and output (black box) of models

55. Unrolled RNN Layer

56. LSTM Block

57. Custom Convolutional LN-LSTM
Convolutional filters reduce frequency variance in input
Lstms model input in time
Custom Convolutional LN-LSTM

58. Prediction Errors of Pre-trained ANNs
ANN Model
RMSE (%)
Standard ANN
4.1264
LSTM
4.0696
LN-LSTM
4.0118
Multi-layer LN-LSTM
4.0163
Conv + LN-LSTMs
3.9302
36 sample history
SMA = 8.99%

59. Results: Parameter Space Exploration
3.7694/ = %
3.7694/ = %
Almost 9% gap closure to perfect predictions or
More than 4% gap closure to perfect predictions
Diminishing returns