Kubernetes 1.20.15 上使用 YAML 部署 ELK 采集 Pod 日志 - 高可用方案

以下是在 CentOS7 系统上部署 ELK 在 Kubernetes 1.20.15 的步骤：

1. 在 Kubernetes 集群中创建一个新的命名空间：

kubectl create namespace elk

2. 创建一个配置文件，例如 elk-config.yaml，用于存储 ELK 的配置信息：

apiVersion: v1
kind: ConfigMap
metadata:
  name: elk-config
  namespace: elk
data:
  elasticsearch.yml: |
    cluster.name: 'my-elasticsearch-cluster'
    node.name: 'node-1'
    network.host: '0.0.0.0'
    discovery.seed_hosts: ['elasticsearch-discovery']
    cluster.initial_master_nodes: ['node-1']
    xpack.security.enabled: false
  kibana.yml: |
    server.name: 'kibana'
    server.host: '0.0.0.0'
    elasticsearch.hosts: ['http://elasticsearch:9200']
    xpack.security.enabled: false
  logstash.yml: |
    http.host: '0.0.0.0'
    xpack.monitoring.enabled: false

3. 创建一个 StatefulSet 来部署 Elasticsearch：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: elk
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.15.0
        ports:
        - containerPort: 9200
        - containerPort: 9300
        env:
        - name: 'cluster.name'
          value: 'my-elasticsearch-cluster'
        - name: 'node.name'
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: 'discovery.seed_hosts'
          value: 'elasticsearch-discovery'
        - name: 'cluster.initial_master_nodes'
          value: 'elasticsearch-0,elasticsearch-1,elasticsearch-2'
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /usr/share/elasticsearch/data
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ['sh', '-c', 'chown -R 1000:1000 /usr/share/elasticsearch/data']
        securityContext:
          privileged: true
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
  - metadata:
      name: elasticsearch-data
      annotations:
        volume.beta.kubernetes.io/storage-class: 'standard'
    spec:
      accessModes: [ 'ReadWriteOnce' ]
      storageClassName: standard
      resources:
        requests:
          storage: 10Gi

4. 创建一个 Headless Service 用于 Elasticsearch 集群的发现：

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-discovery
  namespace: elk
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
  - name: discovery
    port: 9300
    targetPort: 9300

5. 创建一个 ConfigMap 来存储 Logstash 的配置文件：

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-config
  namespace: elk
data:
  logstash.conf: |
    input {
      tcp {
        port => 5000
      }
    }
    filter {
      grok {
        match => { 'message' => '%{COMBINEDAPACHELOG}' }
      }
      date {
        match => [ 'timestamp' , 'dd/MMM/yyyy:HH:mm:ss Z' ]
      }
    }
    output {
      elasticsearch {
        hosts => ['http://elasticsearch:9200']
      }
    }

6. 创建一个 Deployment 来部署 Logstash：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash
  namespace: elk
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logstash
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
      - name: logstash
        image: docker.elastic.co/logstash/logstash:7.15.0
        ports:
        - containerPort: 5000
        volumeMounts:
        - name: logstash-config
          mountPath: /usr/share/logstash/pipeline/logstash.conf
          subPath: logstash.conf
      volumes:
      - name: logstash-config
        configMap:
          name: logstash-config

7. 创建一个 Deployment 来部署 Kibana：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: elk
spec:
  replicas: 2
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.15.0
        ports:
        - containerPort: 5601
        volumeMounts:
        - name: kibana-config
          mountPath: /usr/share/kibana/config/kibana.yml
          subPath: kibana.yml
      volumes:
      - name: kibana-config
        configMap:
          name: elk-config
          items:
          - key: kibana.yml
            path: kibana.yml

8. 创建一个 NodePort Service 来暴露 Kibana：

apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: elk
spec:
  type: NodePort
  ports:
  - name: http
    port: 5601
    targetPort: 5601
  selector:
    app: kibana

这样就完成了在 CentOS7 系统上部署 ELK 在 Kubernetes 1.20.15 的步骤。

高可用方案部署

为了实现高可用方案，需要将 Elasticsearch 的 replicas 参数改为 3 或以上，将 Kibana 的 replicas 参数改为 2 或以上。

• 将 Elasticsearch 的 replicas: 3 修改为所需的副本数。
• 将 Kibana 的 replicas: 2 修改为所需的副本数。

通过以上步骤，您可以成功在 Kubernetes 1.20.15 上部署 ELK，并实现高可用性，以确保您的日志收集和分析服务的稳定性。