flux-task-lint

Static analysis for InfluxDB Flux task scripts — catches scheduling, query-bounding, aggregation and cardinality mistakes before the task is deployed.

The problem

An InfluxDB task is accepted if it parses and its option task record is well formed. Almost nothing else is checked. A task can be scheduled every hour while reading a fifteen-minute window, and InfluxDB will run it happily, report every execution as a success, and quietly drop three quarters of the source data on the floor. There is no error, no failed run, no alert — just a rollup bucket whose numbers are wrong in a way nobody notices until somebody compares them against the raw data months later.

The same shape of failure recurs across the whole family of task bugs:

an offset longer than the every interval, so runs overlap or skip;
no offset at all, so points that arrive a second late are never aggregated;
range(start: 0), which scans the bucket's entire history on every run and works fine right up until the bucket gets big;
a rollup that writes back into the bucket it reads from, so each run aggregates its own previous output;
a group() that keeps a per-request tag, so the rollup bucket inherits the cardinality it was created to collapse.

Every one of these is visible in the source. flux-task-lint reads the task file, builds a model of the schedule and the pipeline, and reports them with a rule ID, a source excerpt and a concrete suggestion — in a pre-commit hook, in CI, or straight from the terminal.

Demo

The same run as text:

$ flux-task-lint late_arrivals.flux feedback_loop.flux
late_arrivals.flux
  error  TASK004  offset 90m exceeds the 1h schedule interval; runs will overlap or skip windows
  6 |     offset: 90m,
    |     ^^^^^^
    = hint: use an offset well below 1h, for example 5m
    = background: flux-task-lint --explain TASK004

  error  QRY004  `range` window of 15m is narrower than the 1h task interval; 45m of every run is never read
  10 |     |> range(start: -15m)
     |        ^^^^^
     = hint: widen the window to at least 1h

  info  AGG002  `aggregateWindow` without `createEmpty`: empty windows become null points by default
  12 |     |> aggregateWindow(every: 7m, fn: mean)
     |        ^^^^^^^^^^^^^^^
     = hint: add `createEmpty: false` to drop empty windows, or `createEmpty: true` to keep the cadence

  warning  AGG001  aggregateWindow every 7m does not divide the 15m query window evenly; the final window covers only 1m
  12 |     |> aggregateWindow(every: 7m, fn: mean)
     |                               ^^
     = hint: choose a window that divides 15m exactly
     = background: flux-task-lint --explain AGG001

feedback_loop.flux
  error  TASK003  `every` and `cron` are mutually exclusive; keep only one
  6 |     cron: "0 * * * *",
    |     ^^^^
    = hint: delete whichever field no longer describes the intended schedule

  warning  QRY003  `range` window of 1d is 24x the 1h task interval; each run re-reads data the previous run already processed
  11 |     |> range(start: -24h)
     |        ^^^^^
     = hint: narrow the window towards 1h, or raise settings.range_multiple if the overlap is deliberate
     = background: flux-task-lint --explain QRY003

  warning  CARD001  rollup groups by 'request_id', which is normally unbounded; the target bucket inherits the source cardinality
  13 |     |> group(columns: ["host", "request_id"])
     |                                ^^^^^^^^^^^^
     = hint: group by the dimensions dashboards slice on, or drop this tag before aggregating
     = background: flux-task-lint --explain CARD001

  error  AGG004  rollup reads and writes bucket 'telemetry'; each run will aggregate its own previous output
  15 |     |> to(bucket: "telemetry")
     |        ^^
     = hint: write to a dedicated rollup bucket, e.g. telemetry_1h
     = background: flux-task-lint --explain AGG004

8 problems (4 errors, 3 warnings, 1 info) in 2 files

Both example files are committed under examples/broken/, so the run above is reproducible from a fresh clone.

Install

There is nothing to install. The linter is pure standard library on Python 3.11 or newer.

git clone https://github.com/taskautomation-org/flux-task-lint.git
cd flux-task-lint
./flux-task-lint examples/

The flux-task-lint script at the repo root is a launcher that puts the bundled src/ on sys.path for you, so a fresh clone runs with no setup at all. To lint tasks anywhere on disk, call it by absolute path:

/path/to/flux-task-lint/flux-task-lint tasks/

…or install it into a virtualenv, which also gives you the flux-task-lint entry point:

python -m pip install -e .
flux-task-lint tasks/

For development (tests and linting):

python -m pip install -r requirements-dev.txt
python -m pytest

Usage

Lint a directory

Directories are searched recursively for *.flux. Explicit file paths are linted whatever their extension, so a one-off script with an unusual name still works.

./flux-task-lint tasks/
./flux-task-lint tasks/hourly_rollup.flux tasks/daily_rollup.flux
cat task.flux | ./flux-task-lint -

Exit codes and gating

Code	Meaning
`0`	Nothing at or above the `--max-severity` threshold.
`1`	Findings at or above the threshold.
`2`	The tool could not run: bad usage, unreadable path, broken config.

--max-severity defaults to error, so warnings and info notes are reported without failing the run. Tighten it once a codebase is clean:

./flux-task-lint tasks/ --max-severity warning   # warnings fail too
./flux-task-lint tasks/ --max-severity off       # report only, never fail

Distinguishing exit code 1 from 2 matters in CI: a 1 means the linter worked and found something, a 2 means the step itself is broken and the absence of findings proves nothing.

Understanding a rule

--explain prints a rule's full documentation — the rationale, a flagged example, an accepted example, and any settings that tune it:

$ flux-task-lint --explain TASK004
TASK004  (error)  [Scheduling]

`offset` is greater than or equal to `every`.

The offset delays each run relative to its schedule boundary. Once the
offset reaches the interval, run N is still waiting to start when run
N+1 becomes due. In practice the task either overlaps itself
continuously or skips windows, and the rollup output develops holes
that are very hard to attribute back to the schedule.

Keep the offset to a small fraction of the interval -- long enough to
cover the write latency of the slowest measurement in the source
bucket, and no longer.

Flagged:

    option task = {name: "hourly rollup", every: 1h, offset: 90m}

Accepted:

    option task = {name: "hourly rollup", every: 1h, offset: 5m}

Further reading: Per-measurement offset tuning for late IoT data
https://taskautomation.org/automated-task-scheduling-orchestration/cron-interval-scheduling-logic/per-measurement-offset-tuning-for-late-iot-data/

--list-rules prints every rule with its default severity.

Output formats

./flux-task-lint tasks/                      # pretty (default)
./flux-task-lint tasks/ --format json        # machine-readable report
./flux-task-lint tasks/ --format github      # ::error / ::warning annotations
./flux-task-lint tasks/ --format sarif --output flux.sarif

json carries the full diagnostic including the hint and any structured details a rule computed:

{
  "rule": "QRY004",
  "severity": "error",
  "message": "`range` window of 15m is narrower than the 1h task interval; 45m of every run is never read",
  "path": "tasks/late_arrivals.flux",
  "line": 10,
  "column": 8,
  "endColumn": 13,
  "hint": "widen the window to at least 1h",
  "details": { "window": "15m", "interval": "1h" }
}

sarif emits SARIF 2.1.0 with one rule descriptor per rule that fired, ready for github/codeql-action/upload-sarif and the GitHub code-scanning UI.

Colour is enabled when stdout is a TTY. NO_COLOR and --no-color disable it; FORCE_COLOR forces it on for capture.

GitHub Action

The repository ships a composite action, so a workflow does not need to install anything:

name: Lint InfluxDB tasks

on: [push, pull_request]

permissions:
  contents: read

jobs:
  flux-task-lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: taskautomation-org/flux-task-lint@v1
        with:
          paths: tasks/
          format: github
          max-severity: error

Findings appear inline on the diff. To feed code scanning instead, write SARIF and upload it:

      - uses: taskautomation-org/flux-task-lint@v1
        with:
          paths: tasks/
          format: sarif
          output: flux-task-lint.sarif
          max-severity: "off"     # let code scanning own the gating
      - uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: flux-task-lint.sarif

Input	Default	Description
`paths`	`.`	Files or directories to lint, whitespace separated.
`format`	`github`	`pretty`, `json`, `github` or `sarif`.
`max-severity`	`error`	Lowest severity that fails the step, or `off`.
`config`	(discovered)	Explicit config file path.
`output`	(stdout)	Write the report to a file instead.
`working-directory`	`.`	Directory to run from.
`python-version`	`3.12`	Python used to run the linter.

Outputs: exit-code and report.

Rules

Twenty rules across six families. Full documentation — rationale, a flagged example and an accepted example for each — lives in the rule reference, and --explain RULE prints the same content in the terminal.

Scheduling

Rule	Severity	Checks
`TASK001`	error	The file declares no `option task` record.
`TASK002`	error	The task declares neither `every` nor `cron`.
`TASK003`	error	The task declares both `every` and `cron`.
`TASK004`	error	`offset` is greater than or equal to `every`.
`TASK005`	warning	An aggregating task declares no `offset`, so late data is lost.
`TASK006`	error	The `cron` expression is not valid in InfluxDB's dialect.
`TASK007`	warning	The task is scheduled more often than once a minute.

Query bounding

Rule	Severity	Checks
`QRY001`	error	`from()` with no `range()` — InfluxDB rejects unbounded reads.
`QRY002`	error	`range(start: 0)` or an absolute timestamp — full-history scan.
`QRY003`	warning	The `range` window is much wider than the task interval.
`QRY004`	error	The `range` window is narrower than the interval, leaving gaps.
`QRY005`	warning	No `_measurement` filter, so the query scans the whole bucket.

Aggregation

Rule	Severity	Checks
`AGG001`	warning	`aggregateWindow(every:)` does not tile the query window evenly.
`AGG002`	info	`aggregateWindow()` leaves `createEmpty` implicit.
`AGG003`	warning	The aggregate window is finer than the source resolution.
`AGG004`	error	A rollup writes back into the bucket it reads from.

Write targets and cardinality

Rule	Severity	Checks
`OUT001`	warning	A rollup never calls `to()`, so the result is discarded.
`OUT002`	error	`to()` targets a bucket the same pipeline reads from.
`CARD001`	warning	A rollup groups by a tag that is effectively unbounded.

Syntax

Rule	Severity	Checks
`SYN001`	error	The file could not be parsed as Flux.

Configuration

Drop a .flux-task-lint.toml beside your tasks, or anywhere above them — the linter walks up from the first path it is given. A [tool.flux-task-lint] table in pyproject.toml works too. --config FILE overrides discovery and --no-config skips it entirely.

# Per-rule severity. Any rule can be silenced with "off", promoted to "error",
# or demoted to "warning" / "info".
[rules]
AGG002 = "off"
QRY005 = "error"
TASK007 = "warning"

[settings]
# QRY003 fires when the range window exceeds this multiple of the interval.
range_multiple = 4.0

# TASK007's floor for the scheduling interval.
min_schedule = "1m"

# TASK005 only asks for an offset on tasks that run at least this often.
require_offset_below = "24h"

# Tags CARD001 treats as unbounded, replacing the built-in list.
high_cardinality_tags = ["request_id", "session_id", "trace_id"]

# What each rollup bucket stores, so AGG003 can detect upsampling.
[settings.bucket_resolution]
telemetry_1h = "1h"
telemetry_1d = "24h"

# Per-path exemptions. Omit `rules` to exempt the path from everything.
[[ignore]]
path = "tasks/vendor/*"
rules = ["QRY005", "AGG002"]

[[ignore]]
path = "tasks/legacy/*"

Key	Type	Default	Used by
`rules.<ID>`	severity or bool	rule default	all
`settings.range_multiple`	number	`4.0`	`QRY003`
`settings.min_schedule`	duration	`"1m"`	`TASK007`
`settings.require_offset_below`	duration	`"24h"`	`TASK005`
`settings.high_cardinality_tags`	list of strings	20 built-in tags	`CARD001`
`settings.bucket_resolution`	table of durations	empty	`AGG003`
`ignore[].path`	glob	—	all
`ignore[].rules`	list of rule IDs	all rules	all

Inline suppression

When a finding is genuinely intentional, say so at the line it applies to rather than turning the rule off globally:

from(bucket: "telemetry")
    // A 24h overlap tolerates gateways that batch up to a day of readings.
    // flux-task-lint: disable-next-line QRY003
    |> range(start: -24h)

Three directives are recognised, all accepting a comma- or space-separated list of rule IDs. Omitting the IDs suppresses every rule at that location.

Directive	Scope
`// flux-task-lint: disable-next-line QRY003`	The following line.
`// flux-task-lint: disable-line QRY003`	The line the comment is on.
`// flux-task-lint: disable-file AGG002`	The whole file.

Suppressed findings are counted in the summary line, so a file that silences everything still says so.

How it works

The linter is a short pipeline, one module per stage:

lexer -> parser -> model -> rules -> formatters

lexer.py tokenizes Flux: identifiers, keywords, strings with escapes, durations (1h30m, -15m, 1mo), date/time literals, regular expressions (disambiguated from division by looking at the preceding token), operators and comments. Comments are kept, because suppression directives live in them.

parser.py is a recursive-descent parser producing the small syntax tree in ast.py: option statements, assignments, pipelines, calls with named arguments, record literals (including {r with ...} extension), arrays, arrow functions, and generic binary/unary/conditional nodes for everything else.

This is a pragmatic parser, not a Flux implementation. It covers the constructs task scripts are made of and parses the rest loosely enough to keep the surrounding structure intact. It does not type-check, does not resolve imports, and does not evaluate anything. When it cannot parse a file it says so as SYN001 rather than skipping the file silently — a valid Flux construct that trips it is a bug worth reporting.

model.py turns the tree into the semantic model rules actually use. It interprets the option task record (parsing durations and validating the cron expression), collects pipelines, and — importantly — flattens them: a chain assembled across several assignments looks identical to a rule as one written in a single expression.

source = from(bucket: "telemetry") |> range(start: -1h)

source
    |> aggregateWindow(every: 1h, fn: mean, createEmpty: false)
    |> to(bucket: "telemetry_1h")

A rule asking "does this pipeline write back to its own source bucket?" sees all four stages, not two disconnected fragments. Pipelines that are a strict prefix of a longer chain are dropped so nothing is reported twice.

rules/ holds the checks, one module per family. Each is a generator function registered with a @rule decorator that carries the ID, default severity, rationale and examples. That registry is the single source of truth: --explain, --list-rules and docs/rules.md are all generated from it, and a test fails if the committed documentation drifts.

duration.py and cron.py do the arithmetic. Durations are split into calendar months and fixed nanoseconds, so 1mo is only collapsed into an approximate figure when a comparison demands it — and the message then says "about 30x" rather than pretending to precision it does not have. The cron validator implements the robfig/cron dialect InfluxDB uses, including the optional seconds field and the @ macros, and estimates the shortest interval between firings so a cron task can be compared against a duration.

formatters/ renders the report. The pretty formatter is the only one that knows about colour; JSON, GitHub annotations and SARIF are pure data.

Contributing

See CONTRIBUTING.md. New rules are welcome; each needs a rationale, a flagged and an accepted example, and both a positive and a negative test.

License

MIT — see LICENSE.

Navigation Menu

Repository files navigation

flux-task-lint

The problem

Demo

Install

Usage

Lint a directory

Exit codes and gating

Understanding a rule

Output formats

GitHub Action

Rules

Scheduling

Query bounding

Aggregation

Write targets and cardinality

Syntax

Configuration

Inline suppression

How it works

Further reading

Contributing

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
.github/workflows		.github/workflows
docs		docs
examples		examples
src/flux_task_lint		src/flux_task_lint
tests		tests
.flux-task-lint.toml		.flux-task-lint.toml
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
action.yml		action.yml
flux-task-lint		flux-task-lint
pyproject.toml		pyproject.toml
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation

flux-task-lint

The problem

Demo

Install

Usage

Lint a directory

Exit codes and gating

Understanding a rule

Output formats

GitHub Action

Rules

Scheduling

Query bounding

Aggregation

Write targets and cardinality

Syntax

Configuration

Inline suppression

How it works

Further reading

Contributing

License

About

Topics

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages