一、概述
Elasticsearch 是个开源分布式搜索引擎,它的特点有:分布式,零配置,自动发现,索引自动分片,索引副本机制,restful 风格接口,多数据源,自动搜索负载等。
Logstash 是一个开源的用于收集, 分析和存储日志的工具。
Kibana 也是一个开源和免费的工具,Kibana 可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 web 界面,可以汇总、分析和搜索重要数据日志。
Beats 是 elasticsearch 公司开源的一款采集系统监控数据的代理 agent,是在被监控服务器上以客户端形式运行的数据收集器的统称,可以直接把数据发送给 Elasticsearch 或者通过 Logstash 发送给 Elasticsearch,然后进行后续的数据分析活动。Beats 由如下组成:
1.Packetbeat:是一个网络数据包分析器,用于监控、收集网络流量信息,
Packetbeat 嗅探服务器之间的流量,解析应用层协议,并关联到消息的处理, 其支 持 ICMP (v4 and v6)、DNS、HTTP、Mysql、PostgreSQL、Redis、
MongoDB、Memcache 等协议;
2. Filebeat:用于监控、收集服务器日志文件,其已取代 logstash forwarder;
3. Metricbeat:可定期获取外部系统的监控指标信息,其可以监控、收集
Apache、HAProxy、MongoDB、MySQL、Nginx、PostgreSQL、
Redis、System、Zookeeper 等服务;
4. Winlogbeat:用于监控、收集 Windows 系统的日志信息;
5. Create your own Beat:自定义 beat ,如果上面的指标不能满足需求,elasticsarch 鼓励开发者 使用 go 语言,扩展实现自定义的 beats,只需要按照模板,实现监控的输入,日志,输出等即可。
Beats 将搜集到的数据发送到 Logstash,经 Logstash 解析、过滤后,将其发送到 Elasticsearch 存储,并由 Kibana 呈现给用户。
Beats 作为日志搜集器没有 Logstash 作为日志搜集器消耗资源,解决了 Logstash 在各服务器节点上占用系统资源高的问题。
Elastic Stack 官方下载地址: 。
二、开源实时日志分析系统 Elastic Stack 5.0 部署:
A. 安装依赖包 JDK
①关闭防火墙
- #cat / etc / selinux / config | grep - v "#"SELINUX = disabled##关闭selinux SELINUXTYPE = targeted#systemctl stop firewalld##关闭防火墙
②下载安装 JDK
- #wget http: //dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
- #yum install jre##这里我只安装的是jre环境,一样可用#java - version openjdk version "1.8.0_111"OpenJDK Runtime Environment(build 1.8.0_111 - b15) OpenJDK 64 - Bit Server VM(build 25.111 - b15, mixed mode
B. 安装 Elasticsearch
①下载安装 elasticsearch
- #rpm--import https: //artifacts.elastic.co/GPG-KEY-elasticsearch
- #cat > /etc/yum.repos.d / elasticsearch.repo << EOF[elasticsearch - 5.x] name = Elasticsearch repository
- for 5.x packages baseurl = https: //artifacts.elastic.co/packages/5.x/yum
- gpgcheck = 1 gpgkey = https: //artifacts.elastic.co/GPG-KEY-elasticsearch
- enabled = 1 autorefresh = 1 type = rpm - md EOF#yum install elasticsearch#systemctl start elasticsearch#systemctl status elasticsearch# / usr / share / elasticsearch / bin / elasticsearch - V##查看elasticsearch版本Version: 5.0.1,
- Build: 080bb47 / 2016 - 11 - 11T22: 08 : 49.812Z,
- JVM: 1.8.0_111
注: 安装后出现如下问题,致使 elasticsearch 无法启动
- #systemctl status elasticsearch OpenJDK 64 - Bit Server VM warning: If the number of processors is expected to increase from one,
- then you should conf...CThreads = N OpenJDK 64 - Bit Server VM warning: INFO: os: :commit_memory(0x0000000085330000, 2060255232, 0) failed;
- error = 'Cannot a ...' (errno = 12)#There is insufficient memory
- for the Java Runtime Environment to
- continue.#Native memory allocation(mmap) failed to map 2060255232 bytes
- for committing reserved memory.#An error report file with more information is saved as: # / tmp / hs_err_pid11084.log
解决:
配置 elasticsearch 下的 jvm.options:
- #vi / etc / elasticsearch / jvm.options - Xms4g##启用如下两项 - Xmx4g## - Xms2g##关闭如下两项## - Xmx2g
②ElasticSearch 默认的对外服务的 HTTP 端口是 9200,节点间交互的 TCP 端口是 9300。
- #ss - tlnp | grep - E '9200|9300'
③测试服务
- #curl - X GET http: //localhost:9200
- {
- "name": "XVY0Ovb",
- "cluster_name": "elasticsearch",
- "cluster_uuid": "tR_H9avzT6Kf4hXWTIfWyA",
- "version": {
- "number": "5.0.1",
- "build_hash": "080bb47",
- "build_date": "2016-11-11T22:08:49.812Z",
- "build_snapshot": false,
- "lucene_version": "6.2.1"
- },
- "tagline": "You Know, for Search"
- }
也可以使用如下命令测试
- #curl - i - XGET 'localhost:9200/'HTTP / 1.1 200 OK content - type: application / json;
- charset = UTF - 8 content - length: 327 {
- "name": "XVY0Ovb",
- "cluster_name": "elasticsearch",
- "cluster_uuid": "tR_H9avzT6Kf4hXWTIfWyA",
- "version": {
- "number": "5.0.1",
- "build_hash": "080bb47",
- "build_date": "2016-11-11T22:08:49.812Z",
- "build_snapshot": false,
- "lucene_version": "6.2.1"
- },
- "tagline": "You Know, for Search"
- }
C. 安装 Logstash
- #rpm--import https: //artifacts.elastic.co/GPG-KEY-elasticsearch
- #cat > /etc/yum.repos.d / logstash.repo << EOF[logstash - 5.x] name = Elastic repository
- for 5.x packages baseurl = https: //artifacts.elastic.co/packages/5.x/yum
- gpgcheck = 1 gpgkey = https: //artifacts.elastic.co/GPG-KEY-elasticsearch
- enabled = 1 autorefresh = 1 type = rpm - md EOF#yum install logstash#systemctl start logstash#systemctl status logstash# / usr / share / logstash / bin / logstash - V##查看logstash版本logstash 5.0.1
D. 安装 Kibana
①安装 Kibana
- #rpm--import https: //artifacts.elastic.co/GPG-KEY-elasticsearch
- #cat > /etc/yum.repos.d / kibana.repo << EOF[kibana - 5.x] name = Kibana repository
- for 5.x packages baseurl = https: //artifacts.elastic.co/packages/5.x/yum
- gpgcheck = 1 gpgkey = https: //artifacts.elastic.co/GPG-KEY-elasticsearch
- enabled = 1 autorefresh = 1 type = rpm - md EOF#yum install kibana#systemctl start kibana#systemctl status kibana# / usr / share / kibana / bin / kibana - V##查看kibana版本5.0.1
②只需更改如下配置
- #cat / etc / kibana / kibana.yml | grep - v "#"server.host: "192.168.147.128"
③检测服务
- #ss - tlnp | grep 5601##Kibana默认进程名:node,端口5601
浏览器输入
E、Beats 安装部署
a. 安装部署 Filebeat
①安装 Filebeat
- #curl - L - O https: //artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.0.1-x86_64.rpm
- #rpm - ivh filebeat - 5.0.1 - x86_64.rpm#systemctl start filebeat#systemctl status filebeat#filebeat.sh - version##查看版本filebeat version 5.0.1(amd64),
- libbeat 5.0.1
②配置 Filebeat
- #cp / etc / filebeat#cp filebeat.yml filebeat.yml.bak#vi / etc / filebeat / filebeat.yml##配置filebeat# === ==========Filebeat prospectors === ============filebeat.prospectors: -input_type: log paths: -/var/log
- /*.log
- #==================== Outputs =====================
- #------------- Elasticsearch output ---------------
- output.elasticsearch:
- # Array of hosts to connect to.
- hosts: ["localhost:9200"]
- #---------------- Logstash output -----------------
- output.logstash:
- # The Logstash hosts
- hosts: ["localhost:5043"] ##只需配置该处,其他默认即可
- # filebeat.sh -configtest -e ##验证配置文件*/
③配置 Logstash
- #cat / etc / logstash / conf.d / logstash.conf input {
- beats {
- port = >"5043"
- }
- }
- filter {
- grok {
- match = >{
- "message" = >"%{COMBINEDAPACHELOG}"
- }
- }
- geoip {
- source = >"clientip"
- }
- }
- output {
- elasticsearch {
- hosts = >["localhost:9200"]
- }
- }#systemctl restart logstash##重启logstash# / usr / share / logstash / bin / logstash - f / etc / logstash / conf.d / logstash.conf--config.test_and_exit##验证配置文件Sending Logstash logs to /
- var / log / logstash which is now configured via log4j2.properties Configuration OK#ss - tlnp | grep - E '5043|9600'
④配置 kibana
浏览器输入 http://localhost:5601,配置 filebeat 的索引(只需输入 filebeat-* 即可)。
在第一个框里输入 filebeat-* 后稍等片刻,kibana 会自动识别,OK 后下面的按钮会由灰色变为可操控的按钮 "Create", 如上图所示。点击该按钮后,最后就会呈现如下图所示:
我们再会过头新建 logstash 的索引,浏览器输入 ,点击左边栏的 "Management"===> 然后点击 "index Patterns"===>
===> 然后点击 "Add New"===>
===> 点击 "Crete" 按钮创建 logstash 索引,创建完成后即会展现如下图所示:
b. 安装部署 Packetbeat
①安装配置 Packetbeat
- #yum install libpcap#curl - L - O https: //artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-5.0.1-x86_64.rpm
- #rpm - ivh packetbeat - 5.0.1 - x86_64.rpm#cat / etc / packetbeat / packetbeat.yml === =================Network device === ================packetbeat.interfaces.device: any##捕获所有消息发送或接收的网络接口 === =====================Flows === =====================packetbeat.flows: timeout: 30s period: 10s === ===============Transaction protocols === ===========##如下是packetbeat默认支持的主要协议及端口packetbeat.protocols.icmp: enabled: true packetbeat.protocols.amqp: ports: [5672] packetbeat.protocols.cassandra: ports: [9042] packetbeat.protocols.dns: ports: [53] include_authorities: true include_additionals: true packetbeat.protocols.http: ports: [80, 8080, 8000, 5000, 8002] packetbeat.protocols.memcache: ports: [11211] packetbeat.protocols.mysql: ports: [3306] packetbeat.protocols.pgsql: ports: [5432] packetbeat.protocols.redis: ports: [6379] packetbeat.protocols.thrift: ports: [9090] packetbeat.protocols.mongodb: ports: [27017] packetbeat.protocols.nfs: ports: [2049] === ======================General === ===============================================Outputs === ======================-------------------Elasticsearch output------------------output.elasticsearch: hosts: ["localhost:9200"]---------------------Logstash output---------------------output.logstash: hosts: ["localhost:5043"]##只需配置该处,其他默认即可 === ==========================Logging === ==================
②验证配置并启动 packetbeat
- #packetbeat.sh - version##查看packetbeat版本packetbeat version 5.0.1(amd64),
- libbeat 5.0.1#packetbeat.sh - configtest - e##测试配置文件......Config OK#systemctl start packetbeat#systemctl status packetbeat
③配置 Kibana(新建 packetbeat 索引)
在 下新建索引页面输入 "packetbeat-*",之后 kibana 会自动更新,在 "Time-field name" 下面的三个选项中选择 "@timestamp",最后点击 "Create" 创建即可。
创建完成后,kibana 显示如下:
c. 安装部署 Metricbeat
①安装配置 metricbeat
- #curl - L - O https: //artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-5.0.1-x86_64.rpm
- #rpm - ivh metricbeat - 5.0.1 - x86_64.rpm#cat / etc / metricbeat / metricbeat.yml === ==============Modules configuration === ==============metricbeat.modules: ----------------------System Module----------------------module: system metricsets: -cpu - load - filesystem - fsstat - memory - network - process enabled: true period: 10s processes: ['.*'] === ======================General === =============================================Outputs === ====================-------------------Elasticsearch output----------------output.elasticsearch: hosts: ["localhost:9200"]---------------------Logstash output-------------------output.logstash: hosts: ["localhost:5043"]##只需配置该项,其他默认即可 === ====================Logging === ======================
②验证配置并启动 metricbeat
- #metricbeat.sh - version##查看版本信息metricbeat version 5.0.1(amd64),
- libbeat 5.0.1#metricbeat.sh - configtest - e##验证配置文件#systemctl start metricbeat#systemctl status metricbeat
③配置 kibana(新建 metricbeat 索引)
在 http://localhost:5601 下新建索引页面输入 "metricbeat-*",之后 kibana 会自动更新,在 "Time-field name" 下面的选项中选择 "@timestamp",最后点击 "Create" 创建即可。
最后呈现如下图所示:
注 1:
关于 ELK Stack 的一些查询语句:
①查询 filebeat
- #curl - XGET 'http://localhost:9200/filebeat-*/_search?pretty'
②查询 packetbeat
- #curl - XGET 'http://localhost:9200/packetbeat-*/_search?pretty'
③查询 metricbeat
- #curl - XGET 'http://localhost:9200/metricbeat-*/_search?pretty'
④查询集群健康度
- #curl 'localhost:9200/_cat/health?v'
⑤查看节点列表
- #curl 'localhost:9200/_cat/nodes?v'ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 127.0.0.1 37 93 3 0.05 0.07 0.41 mdi * XVY0Ovb
⑥列出所有索引
- #curl 'localhost:9200/_cat/indices?v'health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open filebeat - 2016.11.28 Mn4MzxdTRaCj9iseutcmqA 5 1 2 0 12kb 12kb yellow open filebeat - 2016.11.29 iMrr710mT42mApxdV62k - A 5 1 159 0 65.9kb 65.9kb yellow open packetbeat - 2016.11.29 wkTcIwD6RgiiCFwlWBIILA 5 1 5652 0 1.6mb 1.6mb yellow open customer NvxXLgHoREefJLRhot13Ug 5 1 0 0 800b 800b yellow open packetbeat - 2016.11.28 Beoe07S7QB - dntNV4nxJNQ 5 1 2446 0 676.4kb 676.4kb yellow open test M7WbkYq2QNmeJ9NOyMfMZA 5 1 0 0 800b 800b yellow open logstash - 2016.11.28 pcb_84ChSBe9A7VRd - SQNw 5 1 161 0 123.2kb 123.2kb yellow open metricbeat - 2016.11.29 AmVeT1xCQGCnxlAFXUxhYw 5 1 94459 0 37.6mb 37.6mb yellow open logstash - 2016.11.29 6PCKMYKCSVmPfdg - Sx2ARA 5 1 85772 0 20.3mb 20.3mb yellow open.kibana QYTg0I5KS - yc3d7GSey3Zw 1 1 5 0 102kb 102kb
注 2:
如果搭建期间有什么不清楚或不了解的,建议去看官方文档,文档地址如下:
来源: http://www.tuicool.com/articles/VbmMFnu