TiDB monitor and alerts

reference

• https://docs.pingcap.com/zh/tidb/stable/tidb-monitoring-framework
• https://docs.pingcap.com/zh/tidb-in-kubernetes/stable/monitor-a-tidb-cluster

installation

1. prepare the tidb-monitor.yaml

   • apiVersion: pingcap.com/v1alpha1
     kind: TidbMonitor
     metadata:
       name: basic
     spec:
       clusters:
         - name: basic
       prometheus:
         baseImage: m.daocloud.io/docker.io/prom/prometheus
         version: v2.27.1
       grafana:
         baseImage: m.daocloud.io/docker.io/grafana/grafana
         version: 7.5.11
         usernameSecret:
           name: basic-grafana-credentials
           key: username
         passwordSecret:
           name: basic-grafana-credentials
           key: password
         ingress:
           hosts:
             - tidb-monitor.dev.geekcity.tech
           annotations:
             kubernetes.io/ingress.class: nginx
             cert-manager.io/issuer: self-signed-issuer
           tls:
             - hosts:
                 - tidb-monitor.dev.geekcity.tech
               secretName: tidb-monitor.dev.geekcity.tech-tls
       initializer:
         baseImage: m.daocloud.io/docker.io/pingcap/tidb-monitor-initializer
         version: v8.5.0
       reloader:
         baseImage: m.daocloud.io/docker.io/pingcap/tidb-monitor-reloader
         version: v1.0.1
       prometheusReloader:
         baseImage: m.daocloud.io/quay.io/prometheus-operator/prometheus-config-reloader
         version: v0.49.0
       imagePullPolicy: IfNotPresent
     
     

2. apply TiDB monitor components
   • create a secret named basic-grafana-credentials to store the credentials of the grafana admin user

     • kubectl -n tidb-cluster create secret generic basic-grafana-credentials \
         --from-literal=username=admin \
         --from-literal=password=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 16)
       

   • apply resources

     • kubectl -n tidb-cluster apply -f tidb-monitor.yaml
       

check the metrics of the TiDB cluster

1. extract credentials

   • kubectl -n tidb-cluster get secret basic-grafana-credentials -o jsonpath="{.data.username}" | base64 -d && echo
     kubectl -n tidb-cluster get secret basic-grafana-credentials -o jsonpath="{.data.password}" | base64 -d && echo
     

2. visit with web browser
   • url: https://tidb-monitor.dev.geekcity.tech:32443

uninstallation

• kubectl -n tidb-cluster delete -f tidb-monitor.yaml