收集哪些日志
K8S 系统的组件日志
K8S Cluster 里面部署的应用程序日志
日志方案 Filebeat+ELK
Filebeat(日志采集工具)+Logstach(数据处理引擎)+Elasticserch(数据存储, 全文检索, 分布式搜索引擎)+Kibana(展示数据, 绘图, 搜索)
3 容器中的日志怎么收集
收集方案: Pod 中附加专用日志收集的容器
优点: 低耦合
缺点: 每个 Pod 启动一个日志收集代理, 增加资源消耗和运维维护成本
4 部署 DaemonSet 采取 k8s 组件日志 / var/log/messages
4.1 ELK 安装在 harbor 节点
安装, 配置 ELK
1) 安装 openjdk
yum -y install java-1.8.0-openjdk
2) 安装 Logstash
配置网络源
- /etc/yum.repos.d/logstash.repo
- [logstash-7.x]
- name=Elastic repository for 7.x packages
- baseurl= https://artifacts.elastic.co/packages/7.x/yum
- gpgcheck=1
- gpgkey= https://artifacts.elastic.co/GPG-KEY-elasticsearch
- enabled=1
- autorefresh=1
- type=rpm-md
3) 安装 ELK 组件
yum -y install logstash Elasticsearch kibana
4) 配置组件
修改 kibana 配置
- vi /etc/kibana/kibana.YAML
- server.port: 5601
- server.host: "0.0.0.0"
- Elasticsearch.hosts: ["http://localhost:9200/"]
修改 logstash 配置
- vi /etc/logstash/conf.d/logstash-to-es.conf
- input {
- beats {
- port => 5044
- }
- }
- filter {
- }
- output {
- Elasticsearch {
- hosts => ["http://127.0.0.1:9200/"]
- index => "k8s-log-%{ YYYY.MM.dd}" #索引格式
- }
- stdout {
- codec => rubydebug
- }
- }
启动服务
- systemctl start kibana #启动 kibana
- systemctl start Elasticsearch #启动 Elasticsearch
- /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-to-es.conf &
- 192.168.1.143:5601 #访问 kibana
5) 创建 pod(master 节点执行) 收集组件日志
- kubectl apply -f k8s-logs.YAML
- apiVersion: v1
- kind: ConfigMap
- metadata:
- name: k8s-logs-filebeat-config
- namespace: kube-system
- data:
- filebeat.YAML: |-
- filebeat.prospectors:
- type: log
- paths:
- /messages
- fields:
- App: k8s
- type: module
- fields_under_root: true
- output.logstash:
- hosts: ['192.168.1.143:5044']
- apiVersion: apps/v1
- kind: DaemonSet
- metadata:
- name: k8s-logs
- namespace: kube-system
- spec:
- selector:
- matchLabels:
- project: k8s
- App: filebeat
- template:
- metadata:
- labels:
- project: k8s
- App: filebeat
- spec:
- containers:
- name: filebeat
- image: docker.elastic.co/beats/filebeat:6.4.2
- args: [
- "-c", "/etc/filebeat.yml",
- "-e",
- ]
- resources:
- requests:
- CPU: 100m
- memory: 100Mi
- limits:
- CPU: 500m
- memory: 500Mi
- securityContext:
- runAsUser: 0
- volumeMounts:
- name: filebeat-config
- mountPath: /etc/filebeat.YAML
subPath: filebeat.YAML
- name: k8s-logs
- mountPath: /messages
- volumes:
- name: k8s-logs
- hostPath:
- path: /var/log/messages
- type: File
- name: filebeat-config
- configMap:
6)kibana 添加索引
kibana 界面 ->Index Patterns-》k8s-log-*->@timestamp->Create index pattern->Discover
7) 查看日志
6 收集 nginx 日志
1) 创建名称空间
cat namespace.YAML
- apiVersion: v1
- kind: Namespace
- metadata:
- name: test
2) 创建 pod 的 configmap(保存 filebeat 的配置文件)
- kubectl apply -f filebeat-nginx-configmap.YAML
- cat filebeat-nginx-configmap.YAML
- apiVersion: v1
- kind: ConfigMap
- metadata:
- name: filebeat-nginx-config
- namespace: test
- data:
- filebeat.YAML: |-
- filebeat.prospectors:
- type: log
- paths:
- /usr/local/nginx/logs/access.log
- fields: #添加额外字段, 表示字段来源和类型, 日志收集时做配置
- App: www
- type: nginx-access
- fields_under_root: true #将收集的日志放在 kibana 界面顶级
- type: log
- paths:
- /usr/local/nginx/logs/error.log
- fields:
- App: www
- type: nginx-error
- fields_under_root: true
- output.logstash: #收集的日志数据输出到 logstash 中
- hosts: ['192.168.1.143:5044']
3) 创建 secret 用于保存 harbor 用户的密码
kubectl create secret docker-registry harborsecret123 --docker-server=192.168.1.143 --docker-username='zhanghai' --docker-password='[email protected]' -n test
4) 修改 logstash 配置
- /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-to-es.conf &
- cat logstash-to-es.conf
- input {
- beats {
- port => 5044
- }
- }
- filter {
- }
- output {
- if [App] == "www" {
- if [type] == "nginx-access" {
- Elasticsearch {
- hosts => ["http://127.0.0.1:9200/"]
- index => "nginx-access-%{ YYYY.MM.dd}"
- }
- }
- else if [type] == "nginx-error" {
- Elasticsearch {
- hosts => ["http://127.0.0.1:9200/"]
- index => "nginx-error-%{ YYYY.MM.dd}"
- }
- }
- else if [type] == "tomcat-catalina" {
- Elasticsearch {
- hosts => ["http://127.0.0.1:9200/"]
- index => "tomcat-catalina-%{ YYYY.MM.dd}"
- }
- }
- else if [App] == "k8s" {
- if [type] == "module" {
- Elasticsearch {
- hosts => ["http://127.0.0.1:9200/"]
- index => "k8s-log-%{ YYYY.MM.dd}"
- }
- }
- }
- }
- }
5) 创建 nginx 的 pod
- kubectl apply -f nginx-deployment.YAML
- cat nginx-deployment.YAML
- apiVersion: apps/v1beta1
- kind: Deployment
- metadata:
- name: PHP-demo
- namespace: test
- spec:
- replicas: 3
- selector:
- matchLabels:
- project: www
- App: PHP-demo
- template:
- metadata:
- labels:
- project: www
- App: PHP-demo
- spec:
- imagePullSecrets:
- name: harborsecret123
- containers:
- name: nginx
- image: 192.168.1.143/project/PHP-demo:latest
- imagePullPolicy: Always
- ports:
- containerPort: 80
- name: web
- protocol: TCP
- resources:
- requests:
- CPU: 0.5
- memory: 256Mi
- limits:
- CPU: 1
- memory: 1Gi
- resources:
- requests:
- CPU: 0.5
- memory: 256Mi
- limits:
- CPU: 1
- memory: 1Gi
- livenessProbe:
- httpGet:
path: /status.PHP
- port: 80
- initialDelaySeconds: 6
- timeoutSeconds: 20
- volumeMounts:
- name: nginx-logs
- mountPath: /usr/local/nginx/logs
- name: filebeat
- image: docker.elastic.co/beats/filebeat:6.4.2
- args: [
- "-c", "/etc/filebeat.yml",
- "-e",
- ]
- resources:
- limits:
- memory: 500Mi
- requests:
- CPU: 100m
- memory: 100Mi
- securityContext:
- runAsUser: 0
- volumeMounts:
- name: filebeat-config
- mountPath: /etc/filebeat.YAML
subPath: filebeat.YAML
- name: nginx-logs
- mountPath: /usr/local/nginx/logs
- volumes:
- name: nginx-logs
- emptyDir: {
- }
- name: filebeat-config
- configMap:
- name: filebeat-nginx-config
5) 配置 kibana 前台
kibana 界面 ->Kibana Index Patterns-》nginx-access-*->@timestamp->Create index pattern->Discover
收集 tomcat 日志
1) 创建 pod 的 configmap
- cat filebeat-tomcat-configmap.YAML
- apiVersion: v1
- kind: ConfigMap
- metadata:
- name: filebeat-config
- namespace: test
- data:
- filebeat.YAML: |-
- filebeat.prospectors:
- type: log
- paths:
- /usr/local/tomcat/logs/catalina.*
- fields:
- App: www
- type: tomcat-catalina
- fields_under_root: true
- multiline: #多行匹配, 已 [开头为记录一行日志
- pattern: '^['
- negate: true
- match: after
- output.logstash:
- hosts: ['192.168.1.143:5044']
- kubectl apply -f filebeat-tomcat-configmap.YAML
2) 创建 tomcat 的 pod
- kubectl apply -f tomcat-deployment.YAML
- cat tomcat-deployment.YAML
- apiVersion: apps/v1beta1
- kind: Deployment
- metadata:
- name: tomcat-java-demo
- namespace: test
- spec:
- replicas: 3
- selector:
- matchLabels:
- project: www
- App: java-demo
- template:
- metadata:
- labels:
- project: www
- App: java-demo
- spec:
- imagePullSecrets:
- name: harborsecret123
- containers:
- name: tomcat
- image: 192.168.1.143/project/tomcat-java-demo:latest
- imagePullPolicy: Always
- ports:
- containerPort: 8080
- name: Web
- protocol: TCP
- resources:
- requests:
- CPU: 0.5
- memory: 1Gi
- limits:
- CPU: 1
- memory: 2Gi
- livenessProbe:
- httpGet:
- path: /
- port: 8080
- initialDelaySeconds: 60
- timeoutSeconds: 20
- readinessProbe:
- httpGet:
- path: /
- port: 8080
- initialDelaySeconds: 60
- timeoutSeconds: 20
- volumeMounts:
- name: tomcat-logs
- mountPath: /usr/local/tomcat/logs
- name: filebeat
- image: docker.elastic.co/beats/filebeat:6.4.2
- args: [
- "-c", "/etc/filebeat.yml",
- "-e",
- ]
- resources:
- limits:
- memory: 500Mi
- requests:
- CPU: 100m
- memory: 100Mi
- securityContext:
- runAsUser: 0
- volumeMounts:
- name: filebeat-config
- mountPath: /etc/filebeat.YAML
subPath: filebeat.YAML
- name: tomcat-logs
- mountPath: /usr/local/tomcat/logs
- volumes:
- name: tomcat-logs
- emptyDir: {
- }
- name: filebeat-config
- configMap:
- name: filebeat-config
3) 配置 kibana 前台
kibana 界面 ->Kibana Index Patterns-tomcat-catalina-*->@timestamp->Create index pattern->Discover
来源: http://www.bubuko.com/infodetail-3492523.html