This article is an in-depth write-up on Grafana – An open-source tool for running analytics & monitoring our systems online. It contains answers to all our questions about it such as What is it? Why use it? Can I deploy it on-prem? Are there any companies using it already? How popular is it?

Also, I’ll share a bit of my industry experience with the tool.

So, without any further ado.
Let’s get started.

For a full list of articles on the software engineering category here you go

Affiliate Disclaimer: A few of the resources stated in this article contain affiliate links. That means if you find these resources helpful, worthy of spending your money on & you buy a subscription, a course or a book; I get a small cut without you paying anything extra.

I recommend these resources to you because I think the content they offer is pretty good & these will assist you big time in upskilling yourself enabling you to soar in your career.


1. What is Grafana & What Is It Used For?

Grafana is an open source solution for running data analytics, pulling up metrics that make sense of the massive amount of data & to monitor our apps with the help of cool customizable dashboards.

Grafana connects with every possible data source, commonly referred to as databases such as Graphite, Prometheus, Influx DB, ElasticSearch, MySQL, PostgreSQL etc.

Grafana being an open source solution also enables us to write plugins from scratch for integration with several different data sources.

The tool helps us study, analyse & monitor data over a period of time, technically called time series analytics.

It helps us track the user behaviour, application behaviour, frequency of errors popping up in production or a pre-prod environment, type of errors popping up & the contextual scenarios by providing relative data.

A big upside of the project is it can be deployed on-prem by organizations which do not want their data to be streamed over to a vendor cloud for security & other reasons.

Over time this framework has gained a lot of popularity in the industry & is deployed by big guns such as PayPal, eBay, Intel & many more. I’ll talk about the industry use cases up ahead in the article.

Besides the core open source solution there are other two services offered by the Grafana team for businesses known as the Grafana Cloud & the Enterprise. What are they? More on that up ahead in the article.

Before that, let’s dig a little deeper into the functionality & the architectural flow of the tool with having an understanding of what dashboards are? & How do they work?


Zero to Software/Application Architect learning track is a series of four courses that I am writing with an aim to educate you, step by step, on the domain of software architecture & distributed system design. The learning track takes you right from having no knowledge in it to making you a pro in designing large scale distributed systems like YouTube, Netflix, Google Stadia & so on. Check it out.


2. What Is a Grafana Dashboard?

Here is a snapshot of a Grafana dashboard, monitoring stuff.

8bitmen.com Grafana Dashboard

The dashboards pull data from the plugged-in data sources such as Graphite, Prometheus, Influx DB, ElasticSearch, MySQL, PostgreSQL etc.  These are a few of many data sources which Grafana supports by default.

The dashboards contain a gamut of visualization options such as geo maps, heat maps, histograms, all the variety of charts & graphs which a business typically requires to study data.

A dashboard contains several different individual panels on the grid. Each panel has different functionalities.

2.1 My Industry Experience with Grafana

In my former organization where I worked as a backend dev, we used Grafana for monitoring purposes, well naturally.

First on the pre-production environment to study the errors popping up & the server uptime.

The app instances were deployed as Docker containers managed by a docker swarm. There were times when the instances were down or a critical issue caused the system to crash. All of these scenarios were tracked on the Grafana dashboard. Which made our lives a lot easier.

The data was pulled from Prometheus which was plugged-in to the Grafana dashboard as a data source. Queries were fired from the dashboard with different expressions such as min, avg etc.

And Prometheus pulled data from cAdvisor. Have a look at the architectural flow.

8bitmen.com Grafana Prometheus cAdvisor Architecture dashboard monitoring

After the initial testing & monitoring was complete.

The tool was used to monitor the stuff happening when the app was live, in the production environment. Several pre-meditated checks were put in place & alarms were configured when they occurred. This helped us massively, in gaining an in-depth understanding of the system’s behaviour.

We also used past data, which we could track on the dashboard by filtering down by time range for planning out the future operations.

We also used Kibana for monitoring but that was primarily for tracking & studying logs.

To educate yourself on software architecture from the right resources, to master the art of designing large scale distributed systems that would scale to millions of users, to understand what tech companies are really looking for in a candidate during their system design interviews. Read my blog post on master system design for your interviews or web startup.


3. What Are the Features Offered by Grafana?

Let’s go through the features offered by the open-source analytics dashboard framework.

This open-source framework takes care of all the analytics of our app. We can easily query, visualize, set up alerts, understand the data with the help of metrics.

The dashboard is pretty equipped, & continually evolving, to make sense of complex data. From displaying graphs to heatmaps, histograms, Geo maps. The tool has a plethora of visualization options to understand data as per our business requirements.

Alerts are set up & triggered like trip wires whenever an anticipated scenario occurs. These happenings can be notified on Slack or whatever communication platform the monitoring team uses.

Grafana has native support for approx. a dozen databases. And with many more, facilitated by respective plugins.

Either host it on-prem or any cloud platform of your choice.

It has built-in support for Graphite & expressions like add, filter, avg, min, max functions etc. to custom fetch data. What is Graphite? I’ll come to that.

It also has a built-in Influx DB, Prometheus, ElasticSearch, CloudWatch support. I’ll talk about all up ahead.

Become a Pro In DevOps Starting Right From The Basics 🙂

KodeKloud is a platform that trains you right from the basics – on the fundamentals of DevOps. You can go through the courses on technologies such as Kubernetes, Docker, Ansible & many more & can also take the certification preparation courses on the same. The platform offers interactive hands-on training in real environments right in your browser. They have already trained over 180,000 students on DevOps. Check it out.

Educative.io’s – DevOps For Developers Learning Track gets you up to speed on everything you need to know to master Docker and Kubernetes. By the time you’re done, you’ll be able to incorporate Docker and Kubernetes into your work projects with confidence. Educative.io helps software developers level up on in-demand technologies & prepare for their interviews via interactive text-based courses with embedded coding environments. They have over 300,000 learners on their platform.


4. What is Grafana Cloud?

Grafana Cloud is a cloud-native, highly available, fast fully managed open SaaS Software as a Service metrics platform. Pretty helpful for those who do not want to take the load of hosting the solution on-prem & want to stay worry free of managing the entire deployment infrastructure.

It runs on Kubernetes clusters. The backend is Prometheus & Graphite compatible. So, we can either chose Grafana cloud instance or both.  For more information on navigate here.


5. What is Grafana Enterprise?

The Enterprise service comes with all the Grafana Cloud features plus premium plugins, data sources, premium support from the core team. We get response SLAs, trainings & a lot more. For more information visit


6. What Are Some of the Real-World Industry Use Cases of Grafana?

Grafana dashboards are deployed all over the industry be it Gaming, IoT, FinTech or E-Comm.

StackOverflow used the tool to enable their developers & site reliability teams create tailored dashboards to visualize data & optimize their server performance.

Digital Ocean uses Grafana to share visualization data between their teams & have in place a common visual data sharing platform.

These are some of the instances, why we love this tool. For further reading on the industry use cases here you go

By now I am pretty sure you have an idea what Grafana is & why use it?

Now let’s find out what are Graphite & Prometheus?


7. What Is Prometheus Grafana?

Prometheus is an open source data monitoring tool. The combination of Prometheus & Grafana is the de-facto tool combination in the industry for deploying a data visualization setup. Grafana dashboard is used for visualizing the data whereas the backend is powered by Prometheus.

Though Prometheus too has data visualization features & stuff. But still, Grafana is preferred for visualizing data. Queries are fired from the dashboard & the data is fetched from Prometheus.
It acts as a perfect open source data model for storing time series data.


8. What Is Graphite Grafana?

Graphite, again, is a monitoring tool. It facilitates storage & visualization of time series data. Ideally, Graphite is used as a data source for Grafana dashboard in a data monitoring setup.

Grafana has a pretty advanced Graphite query editor which enables us to interact with the data with the help of expressions & functions.

Should you wish to delve into the details. How Grafana, Graphite & Prometheus are different in terms of features they offer? This is a nice read.


9. Grafana Vs Kibana?

As I stated earlier. In my previous project, Kibana was primarily used for analyzing & monitoring logs. Kibana is the K in the ELK stack. The whole intention of writing Kibana by the ElasticSearch team was to have an effective tool to monitor logs. Just click around & track the context of exceptions occurring in prod instead of running Linux commands in the console to find them. It was a pain, believe me.

Well, on the other hand, Grafana is written as a generic monitoring solution for running monitoring & analytics on pretty much anything. Well, this is a very bird’s eye view of the difference between both the tools.

Read this, if you wish to get a deeper insight.

Recommended Read: Master System Design For Your Interviews Or Your Web Startup


ReadBest resources to learn software architecture & system design
I’ve put together a list of resources (online courses + books) that I believe are super helpful in building a solid foundation in software architecture & designing large scale distributed systems like Facebook, YouTube, Gmail, Uber & so on.  


Subscribe to the newsletter to stay notified of the new posts.

Your subscription could not be saved. Please try again.
Please check your inbox to confirm the subscription.

Subscribe to my newsletter

Get updates on the new content published on the blog by joining my newsletter


If you liked the article, share it with your folks. You can follow 8bitmen on social media to stay notified of the new content published – 
Twitter     
Facebook
LinkedIn 

I am Shivang, here is my LinkedIn profile in case you want to say Hello!


More on the Blog

Web Application Architecture & Software Architecture 101 Course

Instagram Architecture – How Does It Store & Search Billions of Images

YouTube Database – How Does It Store So Many Videos Without Running Out Of Storage Space?

Data Analytics in E-Sports – Future Prospects – Jobs – Everything You Should Know

What is Data Ingestion? How to Pick the Right Data Ingestion Tool?