Stop forked children re-exporting the parent's OTel metrics

[dstandish]

The OTel metrics pipeline can end up with two writers on one cumulative series — one climbing,
one frozen — in two independent ways. This fixes both.

A second get_otel_logger() call in the same process leaves the first pipeline running. Every
MeterProvider owns a PeriodicExportingMetricReader whose constructor starts a daemon export
thread, and shutdown_on_exit=False means nothing reaps one that gets replaced. Its instruments
stop being recorded to, so it republishes frozen totals under a different start_time_unix_nano
for the life of the process. The scheduler reaches this on every start: BaseExecutor.__init__ and
SchedulerJobRunner._execute each call stats.initialize(), with a
stats.incr("schedulerjob_start") in between that materialises the first pipeline. Fixed by
building the pipeline once per process, so there is no second one to reap and no cumulative reset
partway through startup.

A forked child re-exports the pipeline it inherited. fork() copies the parent's provider, and
the SDK registers register_at_fork(after_in_child=...) for every PeriodicExportingMetricReader,
so the child restarts the export thread behind it. Nothing in the child records to that pipeline,
so it republishes the totals the parent held at the instant of the fork, once per export interval,
for as long as the child lives. Airflow forks constantly and the long-lived children make it
permanent: LocalExecutor pool workers, the OpenLineage dag-state-change ProcessPoolExecutor, and
the scheduler's log and health-check servers all fork from a scheduler whose pipeline is already
live. Stopping the reader is not enough on its own — the revived ticker does one final collect on
its way out — so the collect callback is dropped too.

A MeterProvider the SDK built for the deployment, from OTEL_CONFIG_FILE or an
opentelemetry-instrument agent, reaches the child the same way, carrying the atexit shutdown that
shutdown_on_exit=True registered for it. Its readers are left running, since on that path they
are the only pipeline the child has, but the inherited copy of that hook is dropped so the child
cannot dump the parent's state on the way out. Only a provider's own _metric_readers are ever
stopped, never the class-level _all_metric_readers every provider shares, so an agent's pipeline
keeps working in forked children.

related: #71800 — that PR fixes the first defect above by shutting down the replaced provider
instead of never building a second one. The two are alternatives for that half; shutting down and
replacing force-flushes the old pipeline and restarts the series under a new
start_time_unix_nano, which building once avoids.

supersedes: #71804, which carried an earlier version of this work.

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5), for this description and the verification runs

Generated-by: Claude Code (Opus 5) following the guidelines