{"id":3500,"date":"2021-05-27T15:03:48","date_gmt":"2021-05-27T15:03:48","guid":{"rendered":"https:\/\/contentlab.com\/how-to-build-a-scalable-prometheus-architecture\/"},"modified":"2021-11-11T04:25:18","modified_gmt":"2021-11-11T04:25:18","slug":"how-to-build-a-scalable-prometheus-architecture","status":"publish","type":"post","link":"https:\/\/contentlab.com\/how-to-build-a-scalable-prometheus-architecture\/","title":{"rendered":"How to Build a Scalable Prometheus Architecture"},"content":{"rendered":"
\n
\n

When building distributed, scalable cloud-native apps containing dozens or even hundreds of microservices, you need reliable monitoring and alerting. If you\u2019re monitoring cloud-native apps in 2021, there\u2019s a good chance you\u2019ve chosen Prometheus<\/a>.<\/p>\n

Prometheus is an excellent choice for monitoring containerized microservices and the infrastructure that runs them \u2014 often Kubernetes.<\/p>\n

However, Prometheus has an Achilles\u2019 heel: it doesn\u2019t scale well by default. It runs on a single machine and periodically connects to an endpoint on each of the containers, servers, and VMs it is monitoring to scrape metrics data from. Large organizations running multiple instances of each of hundreds of microservices quickly exceed the scraping capabilities of a single Prometheus server.<\/p>\n

Prometheus stores the metrics it scrapes on a local disk in a time-series database. This is a problem because the metrics generated by growing cloud-native applications can quickly fill the disk of a Prometheus server.<\/p>\n

Without a way to scale Prometheus, you\u2019ll be forced to reduce the granularity of the metrics you\u2019re collecting or start downsampling your data. Neither of these are good choices because they make the metrics you\u2019re collecting less useful.<\/p>\n

Scalable cloud applications need monitoring that scales with them to ensure you\u2019re able to collect and store all of the monitoring data you need as your application grows. For a deeper dive into the factors driving the need to scale Prometheus, check out our article on Monitoring Microservices the Right Way<\/a>.<\/p>\n

Prometheus Scaling Options<\/h2>\n

Fortunately, there are several ways to build a scalable Prometheus back-end architecture that keeps up with the growth of the application it monitors. By \u201cback end\u201d we mean a central location to store and query data gathered by the Prometheus instances monitoring your application.<\/p>\n

Deciding how many Prometheus instances to run \u2014 and where to run them \u2014 is a separate scaling question that depends heavily on your application architecture. A single Prometheus server can easily pull metrics from a thousand other servers<\/a>.<\/p>\n

So, you might run one Prometheus instance per datacenter if you\u2019re monitoring physical servers, one per region if you\u2019re monitoring cloud virtual machines (VMs), or one per Kubernetes cluster if you\u2019re monitoring containers.<\/p>\n

To build a scalable Prometheus back end, you have two primary choices: manual scaling using Prometheus federation and shardable solutions backed by long-term storage (LTS) that scale automatically to meet demand. You can get a general intro to the platform with our Prometheus tutorial<\/a>.<\/p>\n

Prometheus at Scale with Federation<\/h3>\n

The most straightforward way to scale Prometheus is by using federation<\/a>. Simply put, federation is the ability of one Prometheus server to scrape time-series data from another <\/em>Prometheus server. Prometheus offers two types of federation: hierarchical and cross-service.<\/p>\n

Hierarchical federation is a tree structure, where each Prometheus server gathers and stores data from several Prometheus servers beneath it in the tree. We\u2019ll go into it a little deeper.<\/p>\n

Hierarchical Federation<\/h3>\n
\"\"
Hierarchical Federation Structure<\/sup><\/figcaption><\/figure>\n

Cross-service federation is similar but simpler. Instead of pulling data from multiple Prometheus servers into a single higher-level server, cross-service federation lets a Prometheus server pull selected time-series metrics from sibling Prometheus servers. Each sibling server will contain a copy of the data it pulls from the others, so can query all of your data from any of the Prometheus servers.<\/p>\n

Cross-Service Federation<\/h3>\n
\"Cross-Service
Cross-Service Federation Example<\/sup><\/figcaption><\/figure>\n

To a certain extent, both types of federations help address the distributed query issue. Instead of querying multiple Prometheus servers to get the information we need, we should query a single, federated Prometheus server.<\/p>\n

This is especially useful for sharding Prometheus. As mentioned, a single Prometheus instance can scrape data from a thousand services. But when this isn\u2019t enough, you can set up several Prometheus servers with each of them scraping metrics from a subset of your services. You would then set up a single Prometheus server that scrapes data from each of the shards and aggregates it in one place.<\/p>\n

It\u2019s also useful for rolling up metrics when you have multiple instances of the same microservice being monitored by several Prometheus servers, but you\u2019d like to query a single Prometheus server to understand how that microservice is performing.<\/p>\n

You can set this up quickly and easily using a recording rule that creates a reduced-cardinality time series for the parent Prometheus instance to scrape.<\/p>\n

Drawbacks to the Cross-Service Federation Model<\/h3>\n

However, there are a couple of tradeoffs.<\/p>\n

The first is local storage space. Since each Prometheus server stores its data on a local disk, pulling data from other Prometheus servers means we\u2019ll have more data about what\u2019s happening now<\/em>, but we\u2019ll end up with a shorter retention period.<\/p>\n

You can scale up by attaching large disks to your Prometheus servers. But, as your application grows, you\u2019ll inevitably hit a wall\u2026<\/p>\n

You can\u2019t keep all the historical metrics on a single server.<\/p>\n

Next, managing a growing fleet of Prometheus servers quickly becomes a job unto itself. You want your monitoring infrastructure to be fast and reliable. With that in mind, you\u2019ll be on the hook for a few things, such as:<\/p>\n

    \n
  1. Setting up and maintaining replicas to ensure high availability,<\/li>\n
  2. Rolling out upgrades across all of your prometheus servers, and<\/li>\n
  3. Scrambling to troubleshoot when something goes wrong.<\/li>\n<\/ol>\n

    Neither of these tradeoffs mean that federation is always the wrong choice for scaling Prometheus. That being said, it\u2019s important to be aware of what you\u2019re getting yourself into.<\/p>\n

    A Prometheus Federation Example<\/h2>\n

    A common solution is to use a hierarchical federation where each level in the hierarchy only scrapes aggregate <\/em>time-series data from the servers beneath it. For example, imagine you\u2019re using Prometheus to monitor a multi-cluster Kubernetes<\/a> application. Inside each cluster, a Prometheus server scrapes metrics from each pod running in the cluster.<\/p>\n

    Above these cluster-level servers, you have a single Prometheus server that scrapes aggregate metrics from each cluster-level server.<\/p>\n

    At the cluster level, you\u2019d likely append Kubernetes<\/a> metadata to each Prometheus sample \u2014 such as pod labels or node names. Those labels show you what service the pod belongs to. Node names record the physical node on which the pod and its containers are running.<\/p>\n

    Adding this metadata to each sample increases your Prometheus labels\u2019 cardinality, which in turn increases the number of time-series Prometheus needs to store.<\/p>\n

    You\u2019ll appreciate this tradeoff if, for example, you need to determine if an anomaly you\u2019ve noticed in your app\u2019s metrics only occurs on a single node.<\/p>\n

    Additional Aggregation for Prometheus<\/h3>\n

    You can\u2019t afford to ship all this data to the higher-level Prometheus server, however. You\u2019ll have to create an additional time-series that aggregates the data using a recording rule<\/a>:<\/p>\n

    If you\u2019re trying to aggregate a metric named open_websocket_connections<\/code>, you can create a recording rule that sums the number of open WebSocket connections across all pods in a service. Choose a common prefix like \u2018federate:<\/code>\u2018 when naming aggregate time-series so the federated parent server can scrape them easily.<\/p>\n

    Then, when configuring the parent Prometheus server<\/a>, specify \u2018{__name__=~\"federate:.*\"}<\/code>\u2018 as a match parameter to ensure that the parent server scrapes aggregate data from all its children but leaves everything else behind.<\/p>\n

    Once set up, you can query the parent server about metrics from containerized services running in all your Kubernetes clusters. That is, albeit at reduced fidelity than querying each cluster-level Prometheus server individually.<\/p>\n

    Although federation works and is easy enough to understand, it\u2019s tedious to set up and maintain. As your application grows, you\u2019ll have to add additional layers to your federation\u2019s hierarchy. Consequently, you\u2019ll end up dropping more labels so you can create new aggregate time-series as you move up the hierarchy.<\/p>\n

    Is there a better approach that doesn\u2019t involve so many tradeoffs? Ideally, we\u2019d send all our Prometheus data to a central datastore, keep it there indefinitely, and query it at will.<\/p>\n

    Remote Storage to the Rescue<\/h2>\n

    Fortunately, Prometheus can write to<\/a> and read from<\/a> remote storage. Prometheus has many remote storage integrations<\/a>, including some names you\u2019ll recognize, like Google Bigtable. Some datastores behind these integrations offer virtually limitless capacity, overcoming limitations imposed by Prometheus\u2019 reliance on local storage.<\/p>\n

    \"Prometheus
    Prometheus Storage<\/sub><\/a><\/figcaption><\/figure>\n

    Note that if you set up all your Prometheus servers to write to a remote datastore, it isn\u2019t necessarily<\/em> a scalable Prometheus architecture.<\/p>\n

    Prometheus isn\u2019t designed to let one server remotely read data written by another <\/em>server. However, you can save as much historical data as you\u2019d like and query it at will. Also, there\u2019s a limit to how much query traffic a single Prometheus server can handle.<\/p>\n

    This need not be a deal-breaker. If you\u2019re comfortable with this limitation, adding a scalable cloud database as remote storage might be the last hurdle to a scalable Prometheus backend.<\/p>\n

    You can even use your database\u2019s native query capabilities instead of running PromQL queries<\/a>, but you\u2019ll lose some excellent features.<\/p>\n

    For instance, there\u2019s easy integration with Grafana<\/a> for visualizations. Note that if you intend to use InfluxDB<\/a>, OpenTSDB<\/a>, or TimescaleDB<\/a> for long-term storage, they have their own<\/em> scale challenges.<\/p>\n

    \"Basic
    Basic Cortex Architecture<\/a> for Metrics<\/sup><\/figcaption><\/figure>\n

    Fortunately, a few remote storage integrations offer robust long-term storage, easy deployment and scaling, and <\/em>a fast, scalable PromQL-compatible query interface.<\/p>\n

    These integrations let you stream metrics from all your Prometheus servers directly into an S3-backed time-series database.<\/p>\n

    Cortex vs. Thanos<\/h3>\n

    Two of the most popular integrations follow this pattern: Cortex and Thanos.<\/p>\n

    Cortex and Thanos are similar in that they offer:<\/p>\n