We will be creating our project on AWS Console.
We don't need many resources for it just an - EC2 Instance (t2 micro free tier eligible) is enough
1- Launch an EC2 instance
2- Connect the EC2 Instance through ssh in your System for a better working environment or you can just directly connect to it.
3- Open the official Grafana download webpage,
Select the Edition as "OSS" (open sources) and select the Operating System - in our case, we are using Linux (Ubuntu)
Link - https://grafana.com/grafana/download
4- After selecting the OS, we will be able the see the relevant commands to install Garafana in our system
5- Copy, Paste, and Run all the 3 commands one by one into the system
6- After successful installation, it shows commands to start the Grafana in the system, just copy and run the command to start Grafana
7- Run sudo systemctl status grafana-server command to see check if Grafana is started or not
Here we can see Grafana is "Active"
8- Now in order to gain access to Grafana we need to expose port - 3000 because Grafana runs on port 3000.
Go to EC2 instance Security Group - Inbound rules - Add rule
9- Copy the Instance Public IP address and add port 3000 and open in a new tab, we can see the Grafana login page, and the default "username" and "password" is "admin"
10- we can add a new username and password or just skip and move to the Grafana homepage
11- Next, we Install Prometheus in our system
Open the official Prometheus download page.
Link - https://prometheus.io/download/
12- As we scroll down a little we can see the tar file for various OS, select the OS and copy the "Link address"
13- paste the link address in the system with the "wget" command before it and ls (list) we can see the tar file downloaded
14- Next, we will uncompress the tar file with the command tar -xvf file_name , and do ls (list) we can see the file is uncompressed, now you can remove the unnecessary tar file if you want
15- Move into the Prometheus file and do ls -lrt (list) we can multiple files:
prometheus: This is the Prometheus server binary executable. It is the main component responsible for collecting, storing, and querying metrics.
promtool: This is a command-line tool that provides various utilities for working with Prometheus, such as metric validation and rule testing.
prometheus.yml (or prometheus.yaml): This is the configuration file for Prometheus. It contains the settings and rules that define how Prometheus collects and processes metrics. You can customize this file to specify target endpoints, scrape intervals, alerting rules, and more.
16- We need to create a service file for Prometheus in order to use systemctl to start, stop, or restart them. This is optional, but it is recommended because it will ensure that the services are started automatically when your system boots up.
sudo vim /etc/systemd/system/prometheus.service
17- This is the service file configs for Prometheus.
The Important part is the "Execstart" where we have put the correct path of prometheus file and --config.file= path/to/prometheus.yml and how to get the correct path of both prometheus and prometheus.yml file.
First, both are in the same folder
Second just do "pwd" and copy the path for both and paste it and don't forget to add config syntax before prometheus.yml file path
[Unit]
Description=Prometheus server
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
ExecStart=/home/ubuntu/prometheus-2.45.0.linux-amd64/prometheus --config.file=/home/ubuntu/prometheus-2.45.0.linux-amd64/prometheus.yml
[Install]
WantedBy=multi-user.target
18- After saving the service file we will run a few commands to start Prometheus.
sudo systemctl daemon-reload- The sudo systemctl daemon-reload command is used to reload systemd configuration files without restarting the system.
When you make changes to a systemd unit file, such as a service file, the changes will not take effect immediately. Instead, you need to reload the systemd daemon to apply the updated configuration.
sudo systemctl start prometheussudo systemctl enable prometheus
19- After starting prometheus we can run "systemctl status" command to check if prometheus is active or not, here we can see prometheus is showing active
20- Now we look at prometheus.yml file to check in which port prometheus runs, well its shows target localhost port 9090
21- So, again we add port-9090 in our EC2 instance by going to the security group - inbound rule
22- Now, copy the Instance public IP address and add port 9090, and open a new tab, we will be able to see the Prometheus page.
But the work does not end here because -
Prometheus itself is a monitoring and alerting system that is designed to collect, store, and query metrics from various targets. However, Prometheus does not have built-in capabilities to directly collect metrics from the system or applications. This is where exporters come into play.
23- Node Exporter is one such exporter specifically designed for Prometheus. It is responsible for collecting system-level metrics from the host machine on which it is installed. It exposes these metrics in a format that Prometheus can understand and scrape.
Our next step is to install a node exporter in our system, we will get a node exporter on the same Prometheus download page, just scroll down you will find it.
- We will repeat the same process of installing node exporter as we did prometheus its same, just copy the link address
24- paste the link with "wget" commands and run
and if we do ls we can see the node exporter tar file.
25- Next, as we know we need to uncompress the tar file, and after uncompressing the file, delete the tar file if you want
26- Move to the node exporter folder by cd and do ls to see files it contains
here we have the actual node_exporter binary file and do "pwd" to get the path of this folder we will need it later on
27- Now, we will service files just like Prometheus in the same location where service files are kept in /etc/systemd
sudo vim /etc/systemd/service/node_exporter.service , we need to create a file named node_exporter.service
28- Contents of the node exporter service file will be as same as the Prometheus service file just copy-paste the syntax, the only important syntax we need to edit is the "ExecStart" Here we will put the correct node exporter path location, and we get the path by going to node exporter folder and do "pwd" and copy paste the paste in service file and forget to add "node_exporter" binary file name with the path
29- Next we run the 3 commands that we need to run after saving a service file just like we did for Prometheus.
systemctl daemon-reload, start and enable
30- Do systemctl status for node exporter to check if it is active
31- Now we again another port for node exporter which runs on 9100 by default, go to EC2 instance security group - inbound rules, and add port
32- In order to get metric data from node-exporter we need to add another target in the prometheus.yml file
33- Edit the prometheus.yml file using Vim or any editor and add the target
As prometheus.yml already existed with default syntax we only need to add one line target syntax just copy and paste the target syntax and add port 9100 and save the file
34- After we make changes to the YAML file we need to restart Prometheus in order to register the change made in the prometheus.yml file
35- Open the Prometheus webpage and run the Prom QL query - "up" and execute to check if it shows both targets for Prometheus and node exporter which it does showing
36- We don't just need data but we need monitoring and visualisation for this we Open the Grafana homepage.
Here, select the data source - Prometheus
37- In the data source add server URL - just copy paste the prometheus IP address and in place of IP edit to - "localhost" because in our yml file we added target as "localhost"
38- Next scroll down and select - "save and test" and see if it successfully gets the Prometheus server and that click on "Explore view"
40- This is the Prometheus web page let's run a query to see how data is represented in Prometheus
Here we ran a query to check CPU Utilization, the query is in Prom QL which I got from Google by searching -
sum by(mode) (irate(node_cpu_seconds_total{mode!="idle"}[5m])) > 0
"Select the option from Table to Graph" - which option can be seen just below the Query
40- Now let's run the same query on Grafana - In order to add code we need to select the option - see Image and on the right-hand side below "Query Inspector" we see an option for "Builder - Code" - here just select code and add the code the run the query and if time series graph is not shown then select option visualization, you get the graph
And add this visual in the dashboard - select apply
41- Now you might be wondering why are running code instead of selecting the data
Hereafter selecting Builder instead of Code and selecting Type in data we see numbers of data which is really intimidating to see which is why adding code is sometimes easier to get data
42- Next we add code to see "Free Space" and select our visualization as "Gauge" and "Standard option" as - Data - bytes(SI). It's important to select the standard option because the query will give data in bytes which is hard to understand, so we select the standard option as bytes which represents data in MB
43- Next, we add threshold-
Adding a threshold in Grafana for free space allows you to set up visual indicators or alerts when the free space of a particular resource, such as a disk or storage volume, falls below a specified threshold
We add the threshold in bytes - just convert MB to bytes and add it here - you set the threshold limit according to yourself
Blue shows free space having the same or above 1024 MB
Orange shows free space having the same or above 200 MB
Green is the base
And add this visual in the dashboard - select apply
44- Next we add Uptime visualization by running a query, we get this query in builder mode and select the query by searching for uptime.
Here we select visualization as Stat and select a standard option - Time - seconds
because the data will give you data in seconds in order to convert it into hours we select the standard option of seconds and add Apply
46- Here we can our Dashboard is created, now we can add multiple datas by selecting the option "Add" and as many data as per our requirement.
47- But creating dashboards one after another is very time-consuming, let me show you an easy way to get an Awesome Dashboard with all relevant data
Now, select Dashboard, we can save or discard our previously built dashboard.
Here select - New - Import
48- Open a new tab and search for Grafana dashboard and you will the for Node Exporter Full (if you didn't get then search for node exporter full in the search bar)
link - https://grafana.com/grafana/dashboards/
49- Select the Node Exporter Full and scroll down, we see an ID - copy the ID number
50- Go back to the Grafana page - last we selected import remember - now on this import page - paste the ID no. and select - Load
51- Now we do not need to do anything just select data sources as Prometheus - which is above option "import" and after selecting Prometheus last click "import"
52- Here we get an awesome Dashboard to work with which give us all the relevant data
If we scroll down we will get more data information options with visualization
53- We can edit visualization from "time series to gauge" or to any other visualization,
I prefer Gauge visualization because its easier to read- So select the visualization you need to edit and right click and select edit and change visualization and panel name as per your requirement
That's it for Grafana and Prometheus monitoring and visualization, At first it feels intimidating but as you start to get hands-on it gets interesting and fun to learn
Thank you for reading the blog I hope you found the content somehow interesting and again thank you for your time!!!!