Log forwarder

A simple log forwarder written in Go. This was purely an exercise for fun and to practise concurrency. It should not be used in any production environment as it is not in a production ready state. Use a battle tested solution instead e.g. Filebeat, Grafana Alloy etc.

Preview of logs being forwarded to Grafana Loki in real-time.

Features

• Watches log files for changes and forwards new entries to target location.
• Configurable; users can define which log files to watch, set forwarder type, and other metadata like environment and service names.
• Keeps track of last line read; note this is only in-memory for now, work is needed for persistence which will be a future addition.
• Ability to batch logs before sending them downstream. Controllable via forwarder implementation for now (i.e. constant in code), but future additions will allow it to be set via the config file.
• Ability to support different centralised logging platforms (provided they have an API). Only Loki implementation has been introduced at the moment, however the forwarder has been designed in a manner that enables other platforms to be added easily.
• Test environment for development; Grafana, Loki and fake log generator run in local docker containers which allows for testing the log forwarder end to end while developing.

Future changes

• Persist 'last line read' rather than storing in memory.
• Harden log extract logic - it should take into account multi line logs e.g. stack traces
• Enable configurable batch size - controllable via config file (under forwarder).
• Enable configurable rate limit - controllable via config file (under forwarder).
• Limit the number of log entries read at once to control how much data is held in memory at once e.g. if forwarder is turned on and an existing log contains 50k lines then we probably don't want to read all new lines in one go
• Add retry mechanism with exponential back off to log forwarder. It should be reusable across different forwarder types.

Data flow diagram

Basic design idea for data flow

Usage

First thing you'll want to do is create a config file. This is required because it gives the program instructions on what to watch, and where to forward the results.

You can use the example as a reference. The miminal amount of config needed is; at least one log file with a path, and a Loki forwarder with an API url e.g.

environment: development # Optional metadata

forwarder: 
  name: loki 
  api_url: "http://grafana-loki:3100/loki/api/v1/push"

logs:
  - service_name: app 1 # Optional metadata
    path: /var/log/app/log1.log

By default the forwarder will look for a config.yaml in the same directory. There is an option to send the config path as an argument --config ./mydir/config.yaml.

Loki is the only fowarder currently supported and it must be in lowercase format like so loki.

Developing and testing locally

Run docker compose from the development directory to start grafana, fake log generator and log forwarder.

Access grafana UI: http://localhost:3000/

You can use docker compose watch which will rebuild the log-fowarder on change.

Helpful commands

To rebuild log forwarder on change run watch not up

docker compose watch

Accessing container's shell

docker compose exec <service_name> sh

Container logs

docker compose logs -f <service_name>

Useful resources

• Grafana docker: https://grafana.com/docs/grafana/latest/setup-grafana/installation/docker/
• Loki docker: https://grafana.com/docs/loki/latest/setup/install/docker/
• Grafana data source configuration: https://grafana.com/docs/grafana/latest/administration/provisioning/#data-sources
• Loki API: https://grafana.com/docs/loki/latest/reference/loki-http-api/
• Execution traces: https://go.dev/blog/execution-traces-2024