Docker 部署 ELK + Fluentd 日志分析平台

  • A+
所属分类：Docker 云原生

详细部署步骤

 创建项目目录

mkdir elk-fluentd && cd elk-fluentd
mkdir -p fluentd/conf logstash/pipeline

编写 docker-compose.yml

version: '3.8'

services:
  # Elasticsearch: 存储和索引日志
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms2g -Xmx2g  # 根据内存调整
      - xpack.security.enabled=false # 测试环境禁用安全认证
    ports:
      - "9200:9200"
    volumes:
      - es_data:/usr/share/elasticsearch/data
    networks:
      - elk
    ulimits:
      memlock:
        soft: -1
        hard: -1

  # Kibana: 日志可视化
  kibana:
    image: docker.elastic.co/kibana/kibana:8.12.0
    container_name: kibana
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    networks:
      - elk

  # Fluentd: 日志收集器
  fluentd:
    image: fluent/fluentd:v1.16-1
    container_name: fluentd
    volumes:
      - ./fluentd/conf:/fluentd/etc
      - /var/lib/docker/containers:/var/lib/docker/containers:ro  # 读取Docker日志
    ports:
      - "24224:24224"
    depends_on:
      - elasticsearch
    networks:
      - elk

  # Logstash（可选）: 复杂日志处理
  logstash:
    image: docker.elastic.co/logstash/logstash:8.12.0
    container_name: logstash
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    ports:
      - "5044:5044"  # Beats输入端口
    depends_on:
      - elasticsearch
    networks:
      - elk

volumes:
  es_data:
    driver: local

networks:
  elk:
    driver: bridge

配置 Fluentd

创建 fluentd/conf/fluent.conf

<source>
  @type tail
  path /var/lib/docker/containers/*/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag docker.*
  <parse>
    @type json
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

<match docker.**>
  @type elasticsearch
  host elasticsearch
  port 9200
  logstash_format true
  index_name fluentd-${tag}
  <buffer>
    @type file
    path /var/log/fluentd-buffer
    timekey 1d
    timekey_wait 10m
  </buffer>
</match>

配置 Logstash

创建 logstash/pipeline/logstash.conf
input {
  beats {
    port => 5044
  }
}

filter {
  # 示例：解析Nginx日志
  if [type] == "nginx" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

启动服务

docker compose up -d

验证部署

检查容器状态

docker compose ps
Docker 部署 ELK + Fluentd 日志分析平台

检查 Elasticsearch 健康状态

curl -XGET "http://localhost:9200/_cluster/health?pretty"
Docker 部署 ELK + Fluentd 日志分析平台

访问 Kibana

打开浏览器访问 http://localhost:5601

  1. 点击 Explore on my own。
  2. 进入 Management > Stack Management > Index Pattern。
  3. 创建索引模式（如 fluentd-*）。

发送测试日志

echo '{"message": "Hello, ELK+Fluentd!"}' | nc localhost 24224
Docker 部署 ELK + Fluentd 日志分析平台

在 Kibana 中搜索 Hello, ELK+Fluentd 确认日志到达

xxx