CentOS7系统kubernetes 12015 集群内安装Prometheus和Grafana监控所有的POD应用动态添加规则。使用yaml安装给出步骤。

以下是在CentOS7系统上部署Kubernetes 1.20.15集群并安装Prometheus和Grafana的步骤：

安装Kubernetes集群

首先需要安装Kubernetes集群。可以选择使用kubeadm工具来安装和管理集群。具体步骤如下：

1.1 安装Docker

$ sudo yum install -y docker
$ sudo systemctl enable docker
$ sudo systemctl start docker

1.2 安装kubeadm、kubelet和kubectl

$ cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

$ sudo yum install -y kubelet kubeadm kubectl
$ sudo systemctl enable kubelet
$ sudo systemctl start kubelet

1.3 初始化Kubernetes集群

$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16

1.4 安装网络插件

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

安装Prometheus和Grafana

2.1 创建namespace和configmap

$ kubectl create namespace monitoring
$ kubectl create configmap prometheus-config --from-file=prometheus.yml -n monitoring
$ kubectl create configmap grafana-config --from-file=grafana.ini -n monitoring

2.2 部署Prometheus

$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.48/example/prometheus-operator-crd/alertmanager.crd.yaml
$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.48/example/prometheus-operator-crd/prometheus.crd.yaml
$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.48/example/prometheus-operator-crd/prometheusrule.crd.yaml
$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.48/example/prometheus-operator-crd/servicemonitor.crd.yaml
$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.48/example/prometheus-operator-crd/podmonitor.crd.yaml
$ kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/release-0.48/bundle.yaml
$ kubectl apply -f prometheus.yaml

2.3 部署Grafana

$ kubectl apply -f grafana.yaml

监控所有POD应用

在应用部署时，需要注入Prometheus的sidecar容器，并在应用的yaml文件中添加Prometheus的相关配置。具体步骤如下：

3.1 创建serviceaccount

$ kubectl create serviceaccount prometheus

3.2 创建clusterrole和clusterrolebinding

$ kubectl create clusterrole prometheus --verb=get,list,watch --resource=pods
$ kubectl create clusterrolebinding prometheus --clusterrole=prometheus --serviceaccount=default:prometheus

3.3 部署应用在部署应用时，需要注入Prometheus的sidecar容器，并在应用的yaml文件中添加Prometheus的相关配置。具体示例如下：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "8080"
    spec:
      serviceAccountName: prometheus
      containers:
        - name: my-app
          image: my-app:v1
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
      # 注入Prometheus的sidecar容器
      - name: prometheus-sidecar
        image: prom/prometheus:v2.28.1
        args:
          - "--web.listen-address=:8080"
          - "--log.level=debug"
          - "--config.file=/etc/prometheus/prometheus.yml"
          - "--storage.tsdb.path=/prometheus"
        ports:
          - name: prometheus
            containerPort: 8080
        volumeMounts:
          - name: prometheus-config
            mountPath: /etc/prometheus/prometheus.yml
            subPath: prometheus.yml
      volumes:
        # 挂载Prometheus配置
        - name: prometheus-config
          configMap:
            name: prometheus-config

动态添加规则

可以通过修改Prometheus的配置文件来动态添加规则。具体步骤如下：

4.1 修改Prometheus配置

$ kubectl edit configmap prometheus-config -n monitoring

4.2 添加规则在configmap的prometheus.yml文件中添加如下内容：

rule_files:
  - /etc/prometheus/rules/*.rules

scrape_configs:
  - job_name: 'my-app'
    static_configs:
    - targets: ['my-app:8080']
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_container_port_number]
      separator: ;
      regex: "8080"
      replacement: "$1"
      action: keep
  - job_name: 'kubernetes-services'
    kubernetes_sd_configs:
    - role: service
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
      separator: ;
      regex: "true"
      replacement: "$1"
      action: keep
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
      separator: ;
      regex: "(https?)"
      target_label: __scheme__
      replacement: "$1"
      action: replace
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
      separator: ;
      regex: "(.+)"
      target_label: __metrics_path__
      replacement: "$1"
      action: replace
    - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
      separator: ;
      regex: "([^:]+)(?::\\d+)?;?(\\d+)?"
      target_label: __address__
      replacement: "$1:$2"
      action: replace
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      separator: ;
      regex: "(.+)"
      target_label: kubernetes_namespace
      replacement: "$1"
      action: replace
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: "(.+)"
      target_label: kubernetes_name
      replacement: "$1"
      action: replace

4.3 重启Prometheus

$ kubectl delete pod -l app=prometheus -n monitoring

以上是在CentOS7系统上部署Kubernetes 1.20.15集群并安装Prometheus和Grafana，并监控所有的POD应用，并且可以动态添加规则的步骤。