russel053/ 六月 29, 2018/ Linux

以下方式可以採用LXC方式 拆分 Prometheus /Grafana/以及各node_exproter

新增使用者:

useradd --no-create-home --shell /bin/false prometheus
useradd --no-create-home --shell /bin/false node_exporter

建立資料夾:

mkdir /etc/prometheus
mkdir /var/lib/prometheus

給予群組:

chown prometheus:prometheus /etc/prometheus
chown prometheus:prometheus /var/lib/prometheus

下載Prometheus:

wget https://github.com/prometheus/prometheus/releases/download/v2.3.1/prometheus-2.3.1.linux-amd64.tar.gz

最新版:https://prometheus.io/download/

解壓縮之後複製:

tar zxvf prometheus-XXX.tar.gz

cp prometheus-XXX/prometheus /usr/local/bin/
cp prometheus-XXX/promtool /usr/local/bin/
cp -r prometheus-XXX/consoles /etc/prometheus
cp -r prometheus-XXX/console_libraries /etc/prometheus

給予群組:

chown prometheus:prometheus /usr/local/bin/prometheus
chown prometheus:prometheus /usr/local/bin/promtool
chown -R prometheus:prometheus /etc/prometheus/consoles
chown -R prometheus:prometheus /etc/prometheus/console_libraries

建立設定檔:

nano /etc/prometheus/prometheus.yml
global:
  scrape_interval:     50s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 30s # Evaluate rules every 15 seconds. The default is every 1 minute.

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093
rule_files:
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node'
    scrape_interval: 50s
    scrape_timeout: 20s
    static_configs:
      - targets: ['10.0.1.108:9100']
      - targets: ['10.0.1.110:9100']
      - targets: ['10.0.1.111:9100']
      - targets: ['10.0.1.112:9100']
      - targets: ['10.0.1.113:9100']
      - targets: ['10.0.1.114:9100']
      - targets: ['10.0.1.115:9100']
      - targets: ['10.0.1.119:9100']
  - job_name: 'Proxmox'
    metrics_path: /pve
    static_configs:
      - targets: ['10.0.1.110:9221']
      - targets: ['10.0.1.111:9221']
      - targets: ['10.0.1.112:9221']
      - targets: ['10.0.1.113:9221']
      - targets: ['10.0.1.114:9221']
      - targets: ['10.0.1.115:9221']
  - job_name: 'netdata'
    metrics_path: /api/v1/allmetrics
    params:
      format: [ prometheus ]
    static_configs:
      - targets: ['10.0.1.60:19999']
      - targets: ['10.0.1.108:19999']
      - targets: ['10.0.1.110:19999']
      - targets: ['10.0.1.111:19999']
      - targets: ['10.0.1.112:19999']
      - targets: ['10.0.1.113:19999']
      - targets: ['10.0.1.114:19999']
      - targets: ['10.0.1.115:19999']
      - targets: ['10.0.1.119:19999']
  - job_name: 'Nginx'
    static_configs:
      - targets: ['10.0.1.60:9145']

建立服務:

nano /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries
選配:--web.listen-address=:9091 \
    --storage.tsdb.retention.time=60d \
    --storage.tsdb.retention=60d \

[Install]
WantedBy=multi-user.target

重新載入並開啟服務:

systemctl daemon-reload
systemctl start prometheus
systemctl status prometheus
systemctl enable prometheus

下載node_exproter:

wget https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz

最新:https://prometheus.io/download/
or
echo "deb http://ftp.debian.org/debian stretch-backports main" >> /etc/apt/sources.list
apt update
apt -t stretch-backports install prometheus-node-exporter -y;
or
apt install prometheus-node-exporter 即可快速安裝

解壓縮之後複製到指定位置:

tar zxvf node_exporter-0.17.0-rc.0.linux-amd64.tar.gz
cp node_exporter-0.17.0-rc.0.linux-amd64/node_exporter /opt/node_exporter
chown node_exporter:node_exporter /opt/node_exporter/

建立服務:

nano /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/opt/node_exporter/node_exporter

[Install]
WantedBy=multi-user.target

重新載入並開啟服務:

systemctl daemon-reload
systemctl start node_exporter
systemctl status node_exporter
systemctl enable node_exporter

Node_exporter 額外分支 text_collector(須預先安裝ipmitool):

apt-get install python-sponge
crontab -e

* * * * * /opt/node_exporter/text_collector/nvme_temp.bsh |/opt/node_exporter/text_collector/nvme_temp.prom
* * * * * /opt/node_exporter/text_collector/mellanox_hca_temp.bsh | /opt/node_exporter/text_collector/mellanox_hca_temp.prom
* * * * * /usr/bin/ipmitool sensor | /opt/node_exporter/text_collector/ipmitool.bsh > /opt/node_exporter/text_collector/ipmitool.prom

nano /etc/default/prometheus-node-exporter
ARGS="--collector.textfile.directory=/opt/node_exporter/text_collector"

bsh/prom File: Download


建立 pve exporter:

apt install virtualenv
virtualenv /opt/prometheus-pve-exporter
/opt/prometheus-pve-exporter/bin/pip install prometheus-pve-exporter

mkdir -p /etc/prometheus
nano /etc/prometheus/pve.yml
default:
    user: [email protected]
    password: admin
    verify_ssl: false

cat < /etc/systemd/system/prometheus-pve-exporter.service
[Unit]
Description=Prometheus exporter for Proxmox VE
Documentation=https://github.com/znerol/prometheus-pve-exporter
[Service]
Restart=always
User=root
ExecStart=/opt/prometheus-pve-exporter/bin/pve_exporter /etc/prometheus/pve.yml
[Install]
WantedBy=multi-user.target
EOF

新增監控使用者與服務:

pveum useradd [email protected] -comment "prometheus plugin user"
pveum passwd [email protected]
pveum groupadd monitors
pveum aclmod / -group monitors -role PVEAuditor
pveum usermod [email protected] -group monitors
pveum aclmod / -user [email protected] -role PVEAuditor

systemctl daemon-reload
systemctl start prometheus-pve-exporter
apt install prometheus-node-exporter
systemctl enable prometheus-node-exporter
systemctl start prometheus-node-exporter

snmp_exporter: 安裝:

virtualenv /opt/prometheus-snmp-exporter
/opt/prometheus-snmp-exporter/bin/pip install snmp_exporter

指定ver.:

virtualenv --python=python3 /opt/prometheus-pve-exporter
virtualenv --python=python /opt/prometheus-snmp-exporter

建立服務:

cat <<EOF> /etc/systemd/system/prometheus-snmp-exporter.service

[Unit]

Description=Prometheus exporter for SNMP
Documentation=https://github.com/prometheus/snmp_exporter
[Service]

Restart=always
User=root

ExecStart=/opt/prometheus-snmp-exporter/bin/snmp_exporter 
[Install]
WantedBy=multi-user.target

EOF

重新載入並開啟服務:

systemctl daemon-reload
systemctl start prometheus-snmp-exporter
systemctl status prometheus-snmp-exporter
systemctl enable prometheus-snmp-exporter

Prometheus metric library for Nginx:

下載來源:

git clone http://www.github.com/knyar/nginx-lua-prometheus

到 Web Server 內的 Nginx 新增:

nano /etc/nginx/nginx.conf
區段http{ 加入

lua_shared_dict prometheus_metrics 10M;
lua_package_path "/path/to/nginx-lua-prometheus/?.lua";
init_by_lua '
  prometheus = require("prometheus").init("prometheus_metrics")
  metric_requests = prometheus:counter(
    "nginx_http_requests_total", "Number of HTTP requests", {"host", "status"})
  metric_latency = prometheus:histogram(
    "nginx_http_request_duration_seconds", "HTTP request latency", {"host"})
  metric_connections = prometheus:gauge(
    "nginx_http_connections", "Number of HTTP connections", {"state"})
';
log_by_lua '
  metric_requests:inc(1, {ngx.var.server_name, ngx.var.status})
  metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.server_name})
';

新增 Site:

server {
  listen 9145;
  allow 192.168.0.0/16;
  deny all;
  location /metrics {
    content_by_lua '
      metric_connections:set(ngx.var.connections_reading, {"reading"})
      metric_connections:set(ngx.var.connections_waiting, {"waiting"})
      metric_connections:set(ngx.var.connections_writing, {"writing"})
      prometheus:collect()
    ';
  }
}

WMI_exporter: Download

msiexec /i c:\users\user\downloads\wmi_exporter-0.5.0-amd64.msi ENABLED_COLLECTORS=cpu,cs,logical_disk,net,os,system,textfile TEXTFILE_DIR="C:\wmi_exporter"

安裝完成後:

enable remote powershell:  Enable-PSRemoting -Force
add optional feature openssh server

安裝 Grafana:
新增來源:

nano /etc/apt/sources.list

deb https://packagecloud.io/grafana/stable/debian/ stretch main

增加key:

curl https://packagecloud.io/gpg.key | sudo apt-key add -

安裝:

sudo apt-get update
sudo apt-get install grafana

啟用服務:

service grafana-server start
update-rc.d grafana-server defaults


or

systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server
systemctl enable grafana-server.service

安裝CloudFlare 插件:

grafana-cli plugins install cloudflare-app

篩選:

node_disk_io_now{device!~"md.*|dm.*|loop.*|rbd.*|zd.*"}

Docker 使用方式: Docker:prometheus and exporter pull

Share this Post