Fix duplicate OTel metric streams after re-init and fork

[dstandish]

The OTel metrics pipeline can end up with two writers on one cumulative series — one climbing,
one frozen — in two independent ways. This fixes both.

A second get_otel_logger() call in the same process leaves the first pipeline running.
Every MeterProvider owns a PeriodicExportingMetricReader whose constructor starts a daemon
export thread, and shutdown_on_exit=False means nothing reaps one that gets replaced. Its
instruments stop being recorded to, so it republishes frozen totals under a different
start_time_unix_nano for the life of the process. The scheduler reaches this on every start:
BaseExecutor.__init__ and SchedulerJobRunner._execute each call stats.initialize(), with a
stats.incr("schedulerjob_start") in between that materialises the first pipeline. Fixed by
building the pipeline once per process rather than shutting down and replacing it — no second
pipeline to reap, and no cumulative reset partway through startup.

A forked child re-exports the pipeline it inherited. fork() copies the parent's provider,
and the SDK registers register_at_fork(after_in_child=...) for every
PeriodicExportingMetricReader, so the child restarts the exporter thread behind it. Nothing in
the child records to that pipeline, so it republishes the totals the parent held at the instant
of the fork, once per export interval, for as long as the child lives. Airflow forks constantly
and the long-lived children make it permanent: LocalExecutor pool workers, the OpenLineage
dag-state-change ProcessPoolExecutor, and the scheduler's log and health-check servers all fork
from a scheduler whose pipeline is already live.

Measured in the child, at a 200 ms export interval over a ~1 s child lifetime — exports of a
metric only the parent ever recorded:

	child that never re-inits	child that re-inits
before	4	4
after	0	0

Setting the reader's shutdown event alone is not enough: the ticker publishes one last collection
on its way out. Dropping the collect callback keeps that final pass from carrying the parent's
totals with it. A child that emits metrics of its own still builds its own pipeline, which the
test covers.

A provider the SDK built for the deployment reaches the child the same way. A MeterProvider
from OTEL_CONFIG_FILE, or from an opentelemetry-instrument agent, defaults to
shutdown_on_exit=True and registers its own atexit shutdown (_internal/__init__.py:503). A fork
hands that registration to the child, which then dumps everything the parent accumulated on its way
out. Its readers are left running — on that path they are the only pipeline the child has, and
declarative configuration is the sole source of SDK construction, so the child cannot rebuild what
it would lose — but the inherited copy of the exit hook is dropped.

Only the provider's own _metric_readers are ever stopped, never the class-level
_all_metric_readers that every provider in the process shares, so an agent's pipeline keeps
working in forked children. Both properties have tests.

The first commit is Jed Cunningham's.

related: #71800 — that PR fixes the first defect above by shutting down the replaced provider
instead. The two approaches are alternatives for that half; the fork half is independent of it,
and I measured that #71800 alone leaves 4 stale exports in a child that never re-inits and 1 in a
child that does (the final collect on the way out of shutdown()).

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

[dstandish]

Superseded by #71855, which carries the same two fixes — building the metrics pipeline once per
process, and stopping a forked child from exporting the pipeline it inherited — in the form their
author wrote them, plus the handling for an SDK-built provider's inherited atexit shutdown.

Closing this one. Nothing here is lost: the fork coverage and the newsfragment have been carried
over to #71855.

---

Drafted-by: Claude Code (Opus 5) (no human review before posting)