Metric API Server with k8s

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

It wont work due some certificate issue, for a local cluster i can tell the deployment to ignore using TLS

kubectl edit deploy metrics-server -n kube-system

And add this line to the Args in container spec:

- --kubelet-insecure-tls

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"k8s-app":"metrics-server"},"name":"metrics-server","namespace":"kube-system"},"spec":{"selector":{"matchLabels":{"k8s-app":"metrics-server"}},"strategy":{"rollingUpdate":{"maxUnavailable":0}},"template":{"metadata":{"labels":{"k8s-app":"metrics-server"}},"spec":{"containers":[{"args":["--cert-dir=/tmp","--secure-port=4443","--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname","--kubelet-use-node-status-port","--metric-resolution=15s"],"image":"registry.k8s.io/metrics-server/metrics-server:v0.6.4","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/livez","port":"https","scheme":"HTTPS"},"periodSeconds":10},"name":"metrics-server","ports":[{"containerPort":4443,"name":"https","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/readyz","port":"https","scheme":"HTTPS"},"initialDelaySeconds":20,"periodSeconds":10},"resources":{"requests":{"cpu":"100m","memory":"200Mi"}},"securityContext":{"allowPrivilegeEscalation":false,"readOnlyRootFilesystem":true,"runAsNonRoot":true,"runAsUser":1000},"volumeMounts":[{"mountPath":"/tmp","name":"tmp-dir"}]}],"nodeSelector":{"kubernetes.io/os":"linux"},"priorityClassName":"system-cluster-critical","serviceAccountName":"metrics-server","volumes":[{"emptyDir":{},"name":"tmp-dir"}]}}}}
  creationTimestamp: "2023-09-12T09:44:33Z"
  generation: 2
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
  resourceVersion: "555634"
  uid: b175795b-aaf9-48f6-9cce-a6eec20d30fe
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls
        image: registry.k8s.io/metrics-server/metrics-server:v0.6.4
...

Create the HorizontalPodAutoscaler

Create a deployment with the it’s service, i will set a CPU limit for this pod, basically it is a web server and i will hammer it with siege utility from another machin client

This is the content of the hpa_examle.yml manifest file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nignxwebsite
spec:
  selector:
    matchLabels:
      appname: bizwebsite
  template:
    metadata:
      labels:
        appname: bizwebsite
    spec: 
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        resources:
#          limits:
#            cpu: 500m
          requests:
            cpu: 50m
---
apiVersion: v1
kind: Service
metadata:
  name: php-apache
  labels:
    run: php-apache
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30007
  selector:
    appname: bizwebsite

Install siege util: sudo apt install siege

siege http://192.168.1.13:30007

touk@k8smaster:~$ k top pods
NAME                            CPU(cores)   MEMORY(bytes)
nignxwebsite-6d4bd7c9b4-smx77   0m           6Mi

The output of the k top pods command shows that the nignxwebsite pod is currently using 0m CPU, which means it's not being utilized at all. This is a good opportunity to test the HorizontalPodAutoscaler by increasing the load on the server and observing if the number of replicas increases accordingly.

After collecting the metrics, i may take some secondes:

touk@k8smaster:~$ k top pods
NAME                            CPU(cores)   MEMORY(bytes)
nignxwebsite-6d4bd7c9b4-smx77   380m         6Mi

And it exceeds the limit i have set for it

Let’s Configure Autoscaler resource

From the command line utility we can use kbectl to create an autoscale

kubectl autoscale deployment nginxwebsite --cpu-percent=50 --min=1 --max=10

Or using a manifest file:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nignxwebsite-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nignxwebsite # Targetting the specifi deployment we want
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50
touk@k8smaster:~$ k get all
NAME                                READY   STATUS    RESTARTS   AGE
pod/nignxwebsite-8594759588-xkjr2   1/1     Running   0          4m6s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        45h
service/php-apache   NodePort    10.107.246.64   <none>        80:30007/TCP   32m

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nignxwebsite   1/1     1            1           32m

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/nignxwebsite-6d4bd7c9b4   0         0         0       32m
replicaset.apps/nignxwebsite-8594759588   1         1         1       4m6s

NAME                                                   REFERENCE                 TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/nignxwebsite-hpa   Deployment/nignxwebsite   0%/50%    1         3         1          62s

After using the siege utility to hammer the website

touk@k8snode3:~$ siege http://192.168.1.13:30007
** SIEGE 4.0.4
** Preparing 25 concurrent users for battle.
The server is now under siege...
touk@k8smaster:~$ k get all
NAME                                READY   STATUS              RESTARTS   AGE
pod/nignxwebsite-8594759588-ft44q   0/1     ContainerCreating   0          2s
pod/nignxwebsite-8594759588-x2wwd   0/1     ContainerCreating   0          2s
pod/nignxwebsite-8594759588-xkjr2   1/1     Running             0          4m22s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        45h
service/php-apache   NodePort    10.107.246.64   <none>        80:30007/TCP   32m

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nignxwebsite   1/3     3            1           32m

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/nignxwebsite-6d4bd7c9b4   0         0         0       32m
replicaset.apps/nignxwebsite-8594759588   3         3         1       4m22s

NAME                                                   REFERENCE                 TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/nignxwebsite-hpa   Deployment/nignxwebsite   275%/50%   1         3         1          78s

It started creating 3 pods, so the deployment tells the Replicaset spins up extra pods

touk@k8smaster:~$ k top pods
NAME                            CPU(cores)   MEMORY(bytes)
nignxwebsite-8594759588-xkjr2   407m         7Mi
touk@k8smaster:~$ k top pods
NAME                            CPU(cores)   MEMORY(bytes)
nignxwebsite-8594759588-ft44q   149m         3Mi
nignxwebsite-8594759588-x2wwd   156m         6Mi
nignxwebsite-8594759588-xkjr2   171m         8Mi

As you can see the load is

PreviousTaint and TolerationNextCRICTL Setup - Cluster troubleshoot