Telemetry Aware Scheduling (TAS) makes telemetry data available to scheduling and descheduling decisions in Kubernetes.
Read Documentation
Overview
Telemetry Aware Scheduling (TAS) makes telemetry data available to scheduling and descheduling decisions in Kubernetes. Through a user defined policy, TAS enables rule based decisions on pod placement powered by up to date platform metrics. Policies can be applied on a workload by workload basis - allowing the right indicators to be used to place the right pod.
For example - a pod that requires certain cache characteristics can be schedule on output from Intel® RDT metrics. Likewise a combination of RDT, RAS and other platform metrics can be used to provide a signal for the overall health of a node and be used to proactively ensure workload resiliency.
Design and Architecture
Telemetry Aware Scheduling is made up of two components deployed in a single pod on a Kubernetes Cluster.
Telemetry Aware Scheduler Extender
Telemetry Aware Scheduler Extender is contacted by the generic Kubernetes Scheduler every time it needs to make a scheduling decision. The extender checks if there is a telemetry policy associated with the workload. If so, it inspects the strategies associated with the policy and returns opinions on pod placement to the generic scheduler. The scheduler extender has two strategies it acts on - scheduleonmetric and dontschedule. This is implemented and configured as a Kubernetes Scheduler Extender.
Telemetry Policy Controller
The Telemetry Policy Controller consumes TAS Policies - a Custom Resource. The controller parses this policy for deschedule, scheduleonmetric and dontschedule strategies and places them in a cache to make them locally available to all TAS components. It consumes new Telemetry Policies as they are created, removes them when deleted, and updates them as they are changed. The policy controller also monitors the current state of policies to see if they are violated. For example if it notes that a deschedule policy is violated it labels the node as a violator allowing pods relating to that policy to be descheduled.
Installation and Usage
Strategies
There are three strategies that TAS acts on.
1 scheduleonmetric has only one rule. It is consumed by the Telemetry Aware Scheduling Extender and prioritizes nodes based on a comparator and an up-to-date metric value.
- example: scheduleonmetric when cache_hit_ratio is GreaterThan
2 dontschedule strategy has multiple rules, each with a metric name and operator and a target. A pod with this policy will never be scheduled on a node breaking any one of these rules.
- example: dontschedule if gpu_usage is GreaterThan 10
3 deschedule is consumed by the Telemetry Policy Controller. If a pod with this policy is running on a node that violates it can be descheduled with the kubernetes descheduler.
- example: deschedule if network_bandwidth_percent_free is LessThan 10
The policy definition section below describes how to actually create these strategies in a kubernetes cluster.
Quick set up
TAS relies on metrics from the custom metrics pipeline. If this pipeline isn't set up, and node level metrics aren't exposed through it, TAS will have no metrics on which to make decisions.
Deploy TAS
Telemetry Aware Scheduling uses go modules. It requires Go 1.13+ with modules enabled in order to build. TAS has been tested with Kubernetes 1.14+. TAS was tested on Intel® Server Board S2600WF-Based Systems (Wolf Pass). A yaml file for TAS is contained in the deploy folder along with its service and RBAC roles and permissions.
Note: If run without the unsafe flag (described in the table below) a secret called extender-secret will need to be created with the cert and key for the TLS endpoint. TAS will not deploy if there is no secret available with the given deployment file.
A secret can be created with:
kubectl create secret tls extender-secret --cert /etc/kubernetes/<PATH_TO_CERT> --key /etc/kubernetes/<PATH_TO_KEY> In order to build and deploy run:
make build && make image && kubectl apply -f deploy/
After this is run TAS should be operable in the cluster and should be visible after running kubectl get pods
Demo Videos
Power Driven Scheduling and Scaling with CPU Telemetry
Events
KubeCon + CloudNativeCon Europe Virtual
Visit our booth HERE! (requires to be logged in the tool)
Aug 17, 2020 to Aug 20, 2020