CentOS7 上 Kubernetes 1.20.15 集群安装 Prometheus 和 Grafana，监控所有 POD 应用，动态添加规则

以下是在 CentOS7 系统上部署 Kubernetes 1.20.15 集群并安装 Prometheus 和 Grafana 的步骤：

安装 Kubernetes 集群

首先需要安装 Kubernetes 集群。可以选择使用 kubeadm 工具来安装和管理集群。具体步骤如下：

1.1 安装 Docker

$ sudo yum install -y docker
$ sudo systemctl enable docker
$ sudo systemctl start docker

1.2 安装 kubeadm、kubelet 和 kubectl

$ cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

$ sudo yum install -y kubelet kubeadm kubectl
$ sudo systemctl enable kubelet
$ sudo systemctl start kubelet

1.3 初始化 Kubernetes 集群

$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16

1.4 安装网络插件

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

安装 Prometheus 和 Grafana

2.1 创建 namespace 和 configmap

$ kubectl create namespace monitoring
$ kubectl create configmap prometheus-config --from-file=prometheus.yml -n monitoring
$ kubectl create configmap grafana-config --from-file=grafana.ini -n monitoring

2.2 部署 Prometheus

$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.48/example/prometheus-operator-crd/alertmanager.crd.yaml
$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.48/example/prometheus-operator-crd/prometheus.crd.yaml
$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.48/example/prometheus-operator-crd/prometheusrule.crd.yaml
$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.48/example/prometheus-operator-crd/servicemonitor.crd.yaml
$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.48/example/prometheus-operator-crd/podmonitor.crd.yaml
$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.48/bundle.yaml
$ kubectl apply -f prometheus.yaml

2.3 部署 Grafana

$ kubectl apply -f grafana.yaml

监控所有 POD 应用

在应用部署时，需要注入 Prometheus 的 sidecar 容器，并在应用的 yaml 文件中添加 Prometheus 的相关配置。具体步骤如下：

3.1 创建 serviceaccount

$ kubectl create serviceaccount prometheus

3.2 创建 clusterrole 和 clusterrolebinding

$ kubectl create clusterrole prometheus --verb=get,list,watch --resource=pods
$ kubectl create clusterrolebinding prometheus --clusterrole=prometheus --serviceaccount=default:prometheus

3.3 部署应用在部署应用时，需要注入 Prometheus 的 sidecar 容器，并在应用的 yaml 文件中添加 Prometheus 的相关配置。具体示例如下：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/path: '/metrics'
        prometheus.io/port: '8080'
    spec:
      serviceAccountName: prometheus
      containers:
        - name: my-app
          image: my-app:v1
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
      # 注入 Prometheus 的 sidecar 容器
      - name: prometheus-sidecar
        image: prom/prometheus:v2.28.1
        args:
          - '--web.listen-address=:8080'
          - '--log.level=debug'
          - '--config.file=/etc/prometheus/prometheus.yml'
          - '--storage.tsdb.path=/prometheus'
        ports:
          - name: prometheus
            containerPort: 8080
        volumeMounts:
          - name: prometheus-config
            mountPath: /etc/prometheus/prometheus.yml
            subPath: prometheus.yml
      volumes:
        # 挂载 Prometheus 配置
        - name: prometheus-config
          configMap:
            name: prometheus-config

动态添加规则

可以通过修改 Prometheus 的配置文件来动态添加规则。具体步骤如下：

4.1 修改 Prometheus 配置

$ kubectl edit configmap prometheus-config -n monitoring

4.2 添加规则在 configmap 的 prometheus.yml 文件中添加如下内容：

rule_files:
  - /etc/prometheus/rules/*.rules

scrape_configs:
  - job_name: 'my-app'
    static_configs:
    - targets: ['my-app:8080']
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_container_port_number]
      separator: ;
      regex: "8080"
      replacement: "$1"
      action: keep
  - job_name: 'kubernetes-services'
    kubernetes_sd_configs:
    - role: service
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
      separator: ;
      regex: "true"
      replacement: "$1"
      action: keep
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
      separator: ;
      regex: "(https?)"
      target_label: __scheme__
      replacement: "$1"
      action: replace
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
      separator: ;
      regex: "(.+)"
      target_label: __metrics_path__
      replacement: "$1"
      action: replace
    - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
      separator: ;
      regex: "([^:]+)(?::\d+)?;?(\d+) ?"
      target_label: __address__
      replacement: "$1:$2"
      action: replace
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)    - source_labels: [__meta_kubernetes_namespace]
      separator: ;
      regex: "(.+)"
      target_label: kubernetes_namespace
      replacement: "$1"
      action: replace
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: "(.+)"
      target_label: kubernetes_name
      replacement: "$1"
      action: replace

4.3 重启 Prometheus

$ kubectl delete pod -l app=prometheus -n monitoring

以上是在 CentOS7 系统上部署 Kubernetes 1.20.15 集群并安装 Prometheus 和 Grafana，并监控所有 POD 应用，并且可以动态添加规则的步骤。