Ghent analysis/master claude docs

.claude/rules/01-pipeline.md

@@ -0,0 +1,44 @@
+# Columnflow Pipeline
+
+## Standard task order
+
+```
+GetDatasetLFNs → CalibrateEvents → SelectEvents → ReduceEvents
+→ MergeReducedEvents → ProduceColumns → CreateHistograms
+→ MergeHistograms → PlotVariables1D / CreateDatacards
+```
+
+Law resolves upstream dependencies automatically. Running a downstream task triggers all missing upstream tasks.
+
+## Five TAF types
+
+| TAF | Class | Task | CLI flag | Count |
+|---|---|---|---|---|
+| Calibrator | `Calibrator` | `CalibrateEvents` | `--calibrators` | 0..N |
+| Selector | `Selector` | `SelectEvents` | `--selector` | exactly 1 |
+| Reducer | `Reducer` | `ReduceEvents` | `--reducer` | exactly 1 |
+| Producer | `Producer` | `ProduceColumns` | `--producers` | 0..N |
+| HistProducer | `HistProducer` | `CreateHistograms` | `--hist-producer` | exactly 1 |
+
+## What each task produces
+
+- **CalibrateEvents** → Parquet with additional/corrected columns
+- **SelectEvents** → Parquet with event/object masks + `stats.json` (event counts, MC weight sums)
+- **ReduceEvents** → Parquet with selected events only; columns not in `cfg.x.keep_columns["cf.ReduceEvents"]` are permanently dropped
+- **ProduceColumns** → Parquet with new columns alongside the reduced events
+- **CreateHistograms** → pickle with `Hist` histograms per dataset/shift/category
+
+## law.cfg: module registration
+
+Every new Python file containing a TAF must be added to `law.cfg`:
+
+```ini
+calibration_modules:   myanalysis.calibration.{default,jets}
+selection_modules:     myanalysis.selection.{default,objects}
+production_modules:    myanalysis.production.{default,weights}
+categorization_modules: myanalysis.categorization.categories
+hist_production_modules: myanalysis.histogramming.default
+inference_modules:     myanalysis.inference.default
+```
+
+No spaces after commas inside `{}` brace expansions.

.claude/rules/02-invariants.md

@@ -0,0 +1,105 @@
+# Columnflow Coding Invariants
+
+## Column writes — always use set_ak_column
+
+```python
+# CORRECT — reassign events
+from columnflow.columnar_util import set_ak_column
+events = set_ak_column(events, "ht", ak.sum(events.Jet.pt, axis=1), value_type=np.float32)
+
+# WRONG — direct field mutation does not work
+events["ht"] = ...   # wrong
+events.ht    = ...   # wrong
+```
+
+## uses / produces — declare every column
+
+- Every column read from `events` must be in `uses`.
+- Every column written via `set_ak_column` must be in `produces`.
+- Include sub-TAF objects in both sets to propagate their column declarations:
+
+```python
+@producer(uses={sub_producer, "extra"}, produces={sub_producer, "my_col"})
+def parent(self, events, **kwargs):
+    events = self[sub_producer](events, **kwargs)   # reassign!
+```
+
+## Imports — use maybe_import for heavy packages
+
+```python
+# At module level — deferred import (CORRECT)
+from columnflow.util import maybe_import
+ak = maybe_import("awkward")
+np = maybe_import("numpy")
+
+# coffea — inside function body only, never at module level
+def my_func(self, events, **kwargs):
+    import coffea.nanoevents.methods.vector
+```
+
+## Selectors — return signature and event mask
+
+```python
+def my_selector(self, events, stats, **kwargs) -> tuple[ak.Array, SelectionResult]:
+    ...
+    return events, SelectionResult(steps={"jet": mask})
+
+# Exposed selector must set results.event:
+from operator import and_
+from functools import reduce
+results.event = reduce(and_, results.steps.values())
+```
+
+## Monte Carlo guard
+
+```python
+if self.dataset_inst.is_mc:
+    events = self[mc_weight](events, **kwargs)
+```
+
+## Vectorized operations — no Python loops over events
+
+```python
+# WRONG
+for event in events: ...
+
+# CORRECT
+n_jet = ak.sum(events.Jet.pt > 25, axis=1)   # axis=1 = per-event
+total = ak.sum(events.mc_weight)              # axis=0 = global scalar
+```
+
+## keep_columns — required for pre-Reduce columns
+
+Columns produced in Calibrators/Selectors that are needed downstream must be listed:
+
+```python
+from columnflow.util import DotDict
+from columnflow.columnar_util import ColumnCollection
+
+cfg.x.keep_columns = DotDict.wrap({
+    "cf.ReduceEvents": {
+        "Jet.{pt,eta,phi,mass,btagDeepFlavB}",
+        "Electron.{pt,eta,phi,mass,charge}",
+        "MET.{pt,phi}",
+        "event", "run", "luminosityBlock",
+        ColumnCollection.ALL_FROM_SELECTOR,
+    },
+})
+```
+
+## TAF lifecycle hooks — when to use each
+
+```python
+@my_taf.init      # dynamic uses/produces/shifts; no task access
+@my_taf.post_init # first hook with task access (for late registration)
+@my_taf.requires  # add law task requirements (e.g. BundleExternalFiles)
+@my_taf.setup     # load external resources (files, scale factors) onto self
+@my_taf.teardown  # free memory
+```
+
+## Calling sub-TAFs
+
+```python
+events = self[other_producer](events, **kwargs)
+events, sub_result = self[sub_selector](events, **kwargs)
+```

.claude/rules/calibration.md

@@ -0,0 +1,56 @@
+---
+paths:
+  - "**/calibration/**"
+---
+
+# Calibrator Rules
+
+## Decorator pattern
+
+```python
+from columnflow.calibration import Calibrator, calibrator
+from columnflow.columnar_util import set_ak_column
+from columnflow.util import maybe_import
+ak = maybe_import("awkward")
+
+@calibrator(
+    uses={"Jet.{pt,eta,phi,mass,rawFactor,area}", "fixedGridRhoFastjetAll"},
+    produces={"Jet.{pt,eta,phi,mass}"},   # nominal; add varied columns in init
+)
+def default(self: Calibrator, events: ak.Array, **kwargs) -> ak.Array:
+    corrected_pt = events.Jet.pt * self.jec_factor   # loaded in setup()
+    events = set_ak_column(events, "Jet.pt", corrected_pt)
+    return events
+```
+
+## Hooks for external files and shifts
+
+```python
+@default.requires
+def default_requires(self, task, reqs):
+    from columnflow.tasks.external import BundleExternalFiles
+    reqs["ext"] = BundleExternalFiles.req(task)
+
+@default.setup
+def default_setup(self, task, reqs, inputs, reader_targets):
+    bundle = inputs["ext"]["collection"][0]
+    # parse JEC file from bundle["jec"] and store correction objects on self
+    self.jec_factor = 1.02  # placeholder
+
+@default.init
+def default_init(self):
+    from columnflow.config_util import get_shifts_from_sources
+    # declare which systematic shifts this calibrator is responsible for
+    self.shifts |= set(get_shifts_from_sources(self.config_inst, "jec"))
+    # add varied output columns for each shift
+    for shift_inst in self.shifts:
+        self.produces.add(f"Jet.pt_{shift_inst.name}")
+```
+
+## Key differences from Producers
+
+- Calibrators run on **raw NanoAOD events** (before selection/reduction).
+- They typically **overwrite** kinematic columns (e.g. `Jet.pt`) rather than creating new named columns.
+- Separate task branches run for each registered systematic shift.
+- Varied columns (e.g. `Jet.pt_jec_up`) must be declared in `produces` dynamically in the `init` hook.
+- Column aliases in `cfg` map `"Jet.pt"` → `"Jet.pt_jec_up"` when the `jec_up` shift is active.

.claude/rules/categorization.md

@@ -0,0 +1,42 @@
+---
+paths:
+  - "**/categorization/**"
+---
+
+# Categorizer Rules
+
+## Decorator and return signature
+
+```python
+from columnflow.categorization import Categorizer, categorizer
+from columnflow.util import maybe_import
+ak = maybe_import("awkward")
+
+@categorizer(uses={"event"})
+def cat_incl(self: Categorizer, events: ak.Array, **kwargs) -> tuple[ak.Array, ak.Array]:
+    # Returns (events, boolean_mask_1d) — True = event belongs to this category
+    return events, ak.ones_like(events.event, dtype=bool)
+
+@categorizer(uses={"n_jet", "n_bjet"})
+def cat_sr(self: Categorizer, events: ak.Array, **kwargs) -> tuple[ak.Array, ak.Array]:
+    return events, (events.n_jet >= 4) & (events.n_bjet >= 2)
+```
+
+## Registering in config
+
+```python
+from columnflow.config_util import add_category
+
+add_category(cfg, name="incl",  id=1, selection="cat_incl",  label="Inclusive")
+add_category(cfg, name="sr",    id=2, selection="cat_sr",    label="Signal Region")
+add_category(cfg, name="cr_1b", id=3, selection="cat_cr_1b", label="CR (1b)")
+```
+
+The `selection` argument is the **name of the Categorizer function** (a string), not a boolean expression.
+
+## Key rules
+
+- Always include `"event"` in `uses` for inclusive categorizers (it's always available).
+- Categorizers run inside `CreateHistograms` after `ProduceColumns`; they read columns produced by Producers.
+- Categories are mutually exclusive by convention; combine them via the `--categories` CLI flag to run several in parallel.
+- Register the file in `law.cfg` under `categorization_modules`.

.claude/rules/config.md

@@ -0,0 +1,100 @@
+---
+paths:
+  - "**/config/**"
+---
+
+# Config Object Rules
+
+## Hierarchy: Analysis → Config → Campaign → Dataset
+
+```python
+import order as od
+
+analysis = od.Analysis(name="my_analysis", id=1)
+cpn = od.Campaign(name="run2_2018", id=4, ecm=13, aux={"tier": "NanoAOD", "year": 2018})
+cfg = analysis.add_config(cpn, name="run2_2018", id=4)
+```
+
+## Adding order objects to Config
+
+```python
+# Process (must be added before datasets that reference it)
+cfg.add_process(procs.tt)
+
+# Dataset (fetched from Campaign by name)
+cfg.add_dataset(cpn.get_dataset("tt_dl_powheg"))
+
+# Shift
+cfg.add_shift(name="nominal", id=0)
+cfg.add_shift(name="mu_up", id=1, type=od.Shift.SHAPE)
+
+# Variable
+cfg.add_variable(
+    name="jet1_pt",
+    expression="Jet.pt[:,0]",   # awkward expression evaluated in HistProducer
+    null_value=EMPTY_FLOAT,
+    binning=(40, 0.0, 400.0),
+    unit="GeV",
+    x_title=r"Leading jet $p_T$",
+)
+
+# Category
+from columnflow.config_util import add_category
+add_category(cfg, name="incl", id=1, selection="cat_incl", label="Inclusive")
+```
+
+## Required auxiliaries (cfg.x.*)
+
+```python
+from scinum import Number
+cfg.x.luminosity = Number(59740, {"lumi_13TeV_2018": 0.025j})
+
+from columnflow.util import DotDict
+from columnflow.columnar_util import ColumnCollection
+
+cfg.x.keep_columns = DotDict.wrap({
+    "cf.ReduceEvents": {
+        "Jet.{pt,eta,phi,mass,btagDeepFlavB}",
+        "Electron.{pt,eta,phi,mass,charge}",
+        "Muon.{pt,eta,phi,mass,charge}",
+        "MET.{pt,phi}",
+        "event", "run", "luminosityBlock",
+        ColumnCollection.ALL_FROM_SELECTOR,
+    },
+})
+
+# Default CLI argument values (override on command line)
+cfg.x.default_calibrator = "default"
+cfg.x.default_selector   = "default"
+cfg.x.default_producer   = "default"
+cfg.x.default_variables  = ("n_jet", "jet1_pt")
+```
+
+## Shift aliases (weight-based uncertainties)
+
+```python
+from columnflow.config_util import add_shift_aliases
+
+# Maps "muon_weight" → "muon_weight_up" when mu_up shift is active
+add_shift_aliases(cfg, "mu", {
+    "muon_weight": "muon_weight_{direction}",
+})
+```
+
+## External files
+
+```python
+cfg.x.external_files = DotDict.wrap({
+    "muon_sf": "/path/to/muon_sf.json",
+    "btag_sf": ("/path/to/btag_sf.json", "v2"),   # with version
+})
+```
+
+## Variable expression formats
+
+```python
+"Jet.pt[:,0]"    # leading jet pT (EMPTY_FLOAT if no jets — set null_value)
+"Jet.pt"         # all jets flattened (1D histogram across all events and jets)
+"n_jet"          # scalar column (from ProduceColumns)
+"MET.pt"         # scalar per-event field
+```