Vector

Vector is a high-performance observability data pipeline designed to collect, transform, and route all of your logs and metrics. Built in Rust, it provides a unified tool that replaces the need to patch multiple agents and forwarders together. Whether deployed as an agent on edge nodes or as a centralized aggregator, Vector allows platform engineers to take control of their observability data.

For more details on its architecture, you can check the Vector repository or its official website.

The Problem with Observability Data

Modern infrastructure produces an overwhelming amount of telemetry data. Managing this flow typically requires deploying different agents for logs and metrics, routing them through heavy aggregators like Logstash or Fluentd, and struggling with complex, vendor-specific configurations. This often leads to vendor lock-in, where data is shaped and stored according to what is most convenient for the vendor rather than the engineering team.

Vector addresses this by acting as a single, vendor-neutral tool for processing observability data. It unifies logs and metrics collection and provides programmable transforms to enrich, filter, and normalize data in transit.

Core Concepts of Vector Topologies

Vector topologies are defined using a configuration file that tells it which components to run and how they should interact. A pipeline consists of three main component types:

1. Sources: These components collect or receive data from observability data sources into Vector.
2. Transforms: These manipulate or change the observability data as it passes through your topology.
3. Sinks: These components send data onwards from Vector to external services or destinations.

Because it is built in Rust, Vector handles demanding workloads efficiently, delivering high performance with a low memory footprint. It can function as an end-to-end platform—deploying as a daemon, sidecar, or aggregator.

Getting Started with Vector

You can install Vector using various methods, including package managers, Docker, or an installation script. To install via the script, run the following command:

Once installed, you can define your first pipeline. Let's create a configuration file called vector.yaml that reads from standard input and writes to standard output.

In this configuration, sources.in uses the stdin source, and sinks.out uses the console sink to print unencoded text. The inputs option explicitly routes the output of in to out.

To test the pipeline, you can pipe an event directly into Vector:

This basic workflow illustrates how events enter Vector, pass between components, and exit to their destination. From here, you can extend the configuration to read from complex sources like Syslog or Kafka, apply programmable transforms to parse or sample the data, and route it to multiple sinks simultaneously. For a full list of supported components, consult the Vector quickstart guide.