配置环境
准备三台主机，将三台主机的信息分别写入/etc/hosts文件中
192.168.100.115 server.example.com server
192.168.100.116 agent1.example.com agent1
192.168.100.117 grafana.example.com grafana
[root@server ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.115 server.example.com server
192.168.100.116 agent1.example.com agent1
192.168.100.117 grafana.example.com grafana三台主机都需要有以下操作
安装工具
yum -y install tar lrzsz chrony
时钟同步
systemctl restart chronyd
systemctl enable chronyd
hwclock -w
prometheus
一、简介
Prometheus的主要特性有:
- 多维度数据模型
- 灵活的查询语言
- 不依赖分布式存储，单个服务器节点是自主的
- 以HTTP方式，通过pull模型拉去时间序列数据
- 也可以通过中间网关支持push模型
- 通过服务发现或者静态配置, 来发现目标服务对象
- 支持多种多样的图表和界面展示
pormetheus原理架构图
二、安装prometheus
1、server端安装
将prometheus包文件拖进来
[root@server ~]# rz -E
rz waiting to receive.
[root@server ~]# ls
anaconda-ks.cfg -e -i.bak prometheus-2.54.0.linux-amd64.tar.gz//prometheus这个包文件是二进制文件，解压就能用
[root@server ~]# tar -zxvf prometheus-2.54.0.linux-amd64.tar.gz -C /usr/local///文件名字太长，改一下文件名
[root@server ~]# cd /usr/local/
[root@server local]# ls
bin etc games include lib lib64 libexec prometheus-2.54.0.linux-amd64 sbin share src
[root@server local]# mv prometheus-2.54.0.linux-amd64 prometheus
[root@server local]# ls
bin etc games include lib lib64 libexec prometheus sbin share src2、启动prometheus
直接使用默认配置文件启动, 建议加&后台符号，否则当前命令窗口会被前台占用
[root@server local]# /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml" &//查看9090端口是否启动
[root@server local]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 4096 *:9090 *:*
LISTEN 0 128 [::]:22 [::]:*
3、访问prometheusdeweb界面
通过浏览器访问 http://服务器IP:9090 就可以访问到prometheus的主界面
查看监控主机的信息
通过 http://服务器IP:9090/metrics 可以查看到监控的数据
说明: 这里的metrics你可以类比成zabbix里的监控项
在web主界面可以通过关键字查询metrics, 并显示图形
比如process_cpu_seconds_total
点Graph是查看图形化
三、监控远程linux主机
1、安装node_exporter组件
[root@agent1 ~]# rz -E
rz waiting to receive.
[root@agent1 ~]# ls
anaconda-ks.cfg -e -i.bak node_exporter-1.8.2.linux-amd64.tar.gz
[root@agent1 ~]# tar -zxvf node_exporter-1.8.2.linux-amd64.tar.gz -C /usr/local///文件名太长，更改文件名
[root@agent1 ~]# cd /usr/local/
[root@agent1 local]# ls
bin etc games include lib lib64 libexec node_exporter-1.8.2.linux-amd64 sbin share src
[root@agent1 local]# mv node_exporter-1.8.2.linux-amd64/ node_exporter/
[root@agent1 local]# ls
bin etc games include lib lib64 libexec node_exporter sbin share src2、启动node_exporter, 并验证端口
[root@agent1 local]# nohup node_exporter/node_exporter &
[1] 1124
[root@agent1 local]# nohup: ignoring input and appending output to 'nohup.out'//9100端口启动成功
[root@agent1 local]# ss -anlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 4096 *:9100 *:*
3、修改prometheus.yml配置文件并重启服务
[root@server ~]# vim /usr/local/prometheus/prometheus.yml # metrics_path defaults to '/metrics'# scheme defaults to 'http'.static_configs:- targets: ["localhost:9090"] //在文件最后写入以下三行- job_name: "agent1" static_configs:- targets: ["192.168.100.116:9100"]
~
重启prometheus服务
[root@server ~]# /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml" &//查看进程和端口
[root@server ~]# ps -ef | grep prometheus
root 952 671 0 15:57 pts/0 00:00:00 /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
root 959 671 0 15:58 pts/0 00:00:00 grep --color=auto prometheus
[root@server ~]# ss -anlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 4096 *:9090 *:*
LISTEN 0 128 [::]:22 [::]:*
网页刷新一下
四、监控自己
1、server端安装node_exporter
将包文件拖进来
[root@server ~]# rz -E
rz waiting to receive.
[root@server ~]# ls
anaconda-ks.cfg data -e -i.bak node_exporter-1.8.2.linux-amd64.tar.gz prometheus-2.54.0.linux-amd64.tar.gz
[root@server ~]# tar -zxvf node_exporter-1.8.2.linux-amd64.tar.gz -C /usr/local/
//文件名太长了改下名字
[root@server ~]# cd /usr/local/
[root@server local]# ls
bin data etc games include lib lib64 libexec exporter-1.8.2.linux-amd64 nohup.out prometheus sbin share src
[root@server local]# mv node_exporter-1.8.2.linux-amd64/ node_exporter/
[root@server local]# ls
bin data etc games include lib lib64 libexec node_exporter nohup.out prometheus sbin share src2、启动node_exporter
[root@server local]# nohup node_exporter/node_exporter &
[2] 716
[root@server local]# nohup: ignoring input and appending output to 'nohup.out'[root@server local]# ss -anlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 4096 *:9100 *:*
LISTEN 0 4096 *:9090 *:*
LISTEN 0 128 [::]:22 [::]:*
//9100端口启动成功
3、修改prometheus.yml文件并重启服务
[root@server ~]# vim /usr/local/prometheus/prometheus.yml
//在文件最后写入- job_name: "server"static_configs:- targets: ["192.168.100.115:9100"]
//kill掉这个进程，然后再启动
[root@server ~]# ps -ef | grep prometheus
root 694 669 0 15:30 pts/0 00:00:00 /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
root 802 669 0 15:48 pts/0 00:00:00 grep --color=auto prometheus
[root@server ~]# kill -9 694
[root@server ~]# ps -ef | grep prometheus
root 804 669 0 15:48 pts/0 00:00:00 grep --color=auto prometheus
[1]- Killed /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml"[root@server ~]# /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml" &刷新以下web界面，这就有了
五、监控远程mysql
这里我是将mariadb部署在agent1主机上
1、安装mysqld_exporter与mariadb
先将包文件拖进来
[root@agent1 ~]# rz -E
rz waiting to receive.
[root@agent1 ~]# ls
anaconda-ks.cfg -e -i.bak mysqld_exporter-0.15.1.linux-amd64.tar.gz node_exporter-1.8.2.linux-amd64.tar.gz nohup.out//解压包文件
[root@agent1 ~]# tar -zxvf mysqld_exporter-0.15.1.linux-amd64.tar.gz -C /usr/local/
[root@agent1 ~]# cd /usr/local/
[root@agent1 local]# ls
bin etc games include lib lib64 libexec mysqld_exporter-0.15.1.linux-amd64 node_exporter nohup.out sbin share src
[root@agent1 local]# mv mysqld_exporter-0.15.1.linux-amd64/ mysqld_exporter/
[root@agent1 local]# ls
bin etc games include lib lib64 libexec mysqld_exporter node_exporter nohup.out sbin share src安装mariadb
[root@agent1 local]# yum -y install mariadb-server mariadb
[root@agent1 local]# systemctl restart mariadb.service
[root@agent1 local]# systemctl enable mariadb
Created symlink /etc/systemd/system/mysql.service → /usr/lib/systemd/system/mariadb.service.
Created symlink /etc/systemd/system/mysqld.service → /usr/lib/systemd/system/mariadb.service.
Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → /usr/lib/systemd/system/mariadb.service.
[root@agent1 local]# 2、授权
[root@agent1 local]# mysql
MariaDB [(none)]> grant all ON *.* to 'mysql_monitor'@'localhost' identified by 'redhat';
Query OK, 0 rows affected (0.001 sec)MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.000 sec)
MariaDB [(none)]> exit
Bye3、 创建连接mariadb配置文件
[root@agent1 ~]# vim /usr/local/mysqld_exporter/.my.cnf//在文件内写入以下内容
[client]
user=mysql_monitor
password=redhat4、 启动mysqld_exporter
[root@agent1 ~]# nohup /usr/local/mysqld_exporter/mysqld_exporter --config.mycnf=/usr/local/mysqld_exporter/.my.cnf &
[2] 2766
[root@agent1 ~]# nohup: ignoring input and appending output to 'nohup.out'[2]+ Exit 1 nohup /usr/local/mysqld_exporter/mysqld_exporter --config.mycnf=/usr/local/mysqld_exporter/.my.cnf
[root@agent1 ~]# ss -anlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 4096 *:9100 *:*
LISTEN 0 80 *:3306 *:*
5、更改prometheus配置文件并重启服务
[root@server ~]# vim /usr/local/prometheus/prometheus.yml
//在文件最后添加信息- job_name: "agent1-mariadb"static_configs:- targets: ["192.168.100.116:9100"]//重新启动服务
[root@server ~]# ps -ef | grep prometheus
root 806 669 0 15:49 pts/0 00:00:00 /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
root 895 669 0 16:19 pts/0 00:00:00 grep --color=auto prometheus
[root@server ~]# kill -9 806
[root@server ~]# /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml" &刷新web界面，这就有了
grafana
一、安装grafana
1、在grafana端安装grafana
[root@grafana ~]# rz -E
rz waiting to receive.
[root@grafana ~]# ls
anaconda-ks.cfg -e grafana-6.4.2-1.x86_64.rpm -i.bak
[root@grafana ~]# rpm -ivh grafana-6.4.2-1.x86_64.rpm
warning: grafana-6.4.2-1.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 24098cb6: NOKEY
error: Failed dependencies:fontconfig is needed by grafana-6.4.2-1.x86_64freetype is needed by grafana-6.4.2-1.x86_64urw-fonts is needed by grafana-6.4.2-1.x86_64//这里报错是正常的，需要安装依赖包
[root@grafana ~]# yum -y install fontconfig freetype urw-fonts[root@grafana ~]# rpm -ivh grafana-6.4.2-1.x86_64.rpm
warning: grafana-6.4.2-1.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 24098cb6: NOKEY
Verifying... ################################# [100%]
Preparing... ################################# [100%]
Updating / installing...1:grafana-6.4.2-1 ################################# [100%]
### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemdsudo /bin/systemctl daemon-reloadsudo /bin/systemctl enable grafana-server.service
### You can start grafana-server by executingsudo /bin/systemctl start grafana-server.service
POSTTRANS: Running script2、启动grafana
[root@grafana ~]# systemctl restart grafana-server
[root@grafana ~]# systemctl enable grafana-server
Created symlink /etc/systemd/system/multi-user.target.wants/grafana-server.service → /usr/lib/systemd/system/grafana-server.service.
[root@grafana ~]# ss -anlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 4096 *:3000 *:*
LISTEN 0 128 [::]:22 [::]:*
//3000端口启动，就表名grafana启动成功二、浏览器访问并添加数据
默认登陆用户和密码均为admin
添加完成
三、grafana实现自定义监控cpu负载
这就有了
grafana+onealert报警
prometheus报警需要使用alertmanager这个组件，而且报警规则需要手动编写(对运维来说不友好)。
所以我这里选用grafana+onealert报警。
一、 在onealert里添加grafana应用
这里我适使用：http://www.onealert.com
作为告警平台
下滑找到
配置通知策略
在grafana增加通知通道
测试结果
