refactor: de-vendor the certificate transfer and tracing interface handling

[tonyandrewmeyer]

This PR replaces the two charm libraries vendored under tracing/ops_tracing/vendor/charms/ with small requirer-side dataclasses driven through ops.Relation.load / ops.Relation.save, removing pydantic (and its pydantic-core, annotated-types, typing-inspection transitive deps) from ops_tracing's dependency tree.

The vendored copies had exactly one consumer (ops_tracing itself) and were already a fork, so this also drops the upstream API surface ops_tracing never exercises: the provider-side classes, the TracingEndpointRequirer / CertificateTransferRequires wrappers, the charmhub publish metadata, the redundant relation-direction validation, the charm_tracing_config convenience wrapper, and the pydantic-v1 compatibility branches.

As well as dropping 4 dependencies, including ones that are not pure Python, we drop hundreds of lines of unused code, significantly simplifying the production side code of ops-tracing, while gaining some additional tests.

• tracing/ops_tracing/_tracing_models.py: TracingProviderAppData, TracingRequirerAppData, and CertificateTransferProviderAppData as plain @dataclasses. Relation.load recurses into the nested Receiver / ProtocolType dataclasses and the Set[str] of PEMs; the lib-local validation helpers are gone.
• tracing/ops_tracing/_api.py: Tracing observes the tracing and certificate-transfer Juju events directly. Reading the provider databag goes through relation.load; writing the requested receivers goes through relation.save. The two old wrapper classes and their inlined databag helpers are gone.
• tracing/test/: add test_tracing_v2_conformance.py and test_certificate_transfer_v1_conformance.py pinning requirer-side behaviour against the verbatim clauses in the upstream interface docs, plus test_upstream_schemas.py (opt-in via tox -e upstream-schemas) which fetches the canonical/charmlibs schemas at HEAD and asserts our dataclasses haven't drifted from them.

There is a small enhancement to the Relation.load functionality to support nested dataclasses more cleanly. This is technically breaking, as anyone using nested dataclasses without a custom load would have got plain dicts beyond the top level, but this seems like a bug fix to me, and exercising a more complicated example ourselves makes it so that we notice when the feature is not convenient enough for other users.

[james-garner-canonical]

Driveby comment -- there's a larger discussion to be had here as to how we want to handle this long term. With the tracing charm library migrating to charmlibs this cycle, we could in principle depend on it a as a regular Python dependency instead of vendoring. On the other hand, if our needs really do diverge, then maybe we'd prefer to switch to vendoring our own tracing logic in a more first-class way (with this PR being one step on that road).

[tonyandrewmeyer]

Driveby comment -- there's a larger discussion to be had here as to how we want to handle this long term.

Yes, that's one of the reasons this is in draft. I'll address it before moving it out.

With the tracing charm library migrating to charmlibs this cycle, we could in principle depend on it a as a regular Python dependency instead of vendoring.

We obviously should do that when it happens, if we're still vendoring. But that's not the point of the issue. The point is to get rid of large dependencies that are not really needed, and slim down the code to only what is actually required.

[james-garner-canonical]

Driveby comment -- there's a larger discussion to be had here as to how we want to handle this long term.

Yes, that's one of the reasons this is in draft. I'll address it before moving it out.

:)

With the tracing charm library migrating to charmlibs this cycle, we could in principle depend on it a as a regular Python dependency instead of vendoring.

We obviously should do that when it happens, if we're still vendoring.

It's not entirely clear to me that we should, given the desire to keep Ops as dependency free as possible, and given the trickiness introduced by the mutual dependency between Ops and the tracing library. If we can feasibly maintain our own slim subset of the functionality the library provides instead, I think that's worth considering.

But that's not the point of the issue. The point is to get rid of large dependencies that are not really needed, and slim down the code to only what is actually required.

The more our vendored copy drifts (e.g. dropping Pydantic), the more involved migrating to the charmlibs version becomes. If we definitely intend to use the charmlibs version, which we have roadmap time allocated to migrating this cycle, then IMO we shouldn't be aiming to merge this PR -- and I figured it was worth raising that point while it's still in draft.

But I fully support the idea of exploring how slim we can make this, that's definitely an important point in the dependency vs own-our-own discussion.

[tonyandrewmeyer]

The more our vendored copy drifts (e.g. dropping Pydantic),

What's currently here is not a vendored copy that drops Pydantic, it's a library built for just ops-tracing. It's not in a vendor folder, and it doesn't have anything in it that isn't used by ops-tracing. It might be too similar to the old version and need reworking - it's a draft and I haven't looked very closely at the code yet, just given instructions to an agent.

If we definitely intend to use the charmlibs version,

My understanding was that we definitely intend to not use the regular library, wherever it is hosted. I thought when we'd discussed this previously everyone was in favour of doing that, just not of allocating roadmap time to it.

which we have roadmap time allocated to migrating this cycle

My understanding was we are doing that because it's the library charms should use for workload tracing, and we want to accelerate getting people away from Charmhub hosted libraries, rather than anything to do with our own use. Did I miss something?

But I fully support the idea of exploring how slim we can make this, that's definitely an important point in the dependency vs own-our-own discussion.

If this is more of an open question than I understood, I don't mind that. The direction here will be more-or-less the same either way. I am aware of the love for keeping Pydantic in charms and wouldn't be surprised if, even though my understanding is we already agreed to it, this gets rejected.

[james-garner-canonical]

I'm definitely in favour of the 'maintain our own slim, Pydantic-free tracing stuff' direction, so that's music to my ears. I must have forgotten the previous discussion where we agreed to that, and gotten confused by the PR description (I haven't looked at the code).