Document all hand-written source in codegen/opentelemetry-codegen-json

.changelog/5704.fixed

@@ -0,0 +1 @@
+`opentelemetry-exporter-zipkin-json`: warn and fall back to the default instead of raising `ValueError` when `OTEL_EXPORTER_ZIPKIN_TIMEOUT` holds a non-integer value; the same fix is applied to `opentelemetry-exporter-zipkin-proto-http`.

.pylintrc

@@ -36,10 +36,6 @@ persistent=yes
 # Specify a configuration file.
 #rcfile=
 
-# When enabled, pylint would attempt to guess common misconfiguration and emit
-# user-friendly hints instead of false-positive error messages.
-suggestion-mode=yes
-
 # Allow loading of arbitrary C extensions. Extensions are imported into the
 # active Python interpreter and may run arbitrary code.
 unsafe-load-any-extension=no

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/__init__.py

@@ -0,0 +1,14 @@
+# Copyright The OpenTelemetry Authors
+# SPDX-License-Identifier: Apache-2.0
+
+"""Build time code generators for OpenTelemetry Python.
+
+Nothing under ``opentelemetry.codegen`` runs while an application is emitting telemetry. These
+modules are invoked by ``protoc`` during development and read protocol buffer descriptors, then
+hand the Python source they build back to ``protoc``, which is the program that writes the files.
+Those files are then committed to the repository and shipped as ordinary modules.
+
+``opentelemetry.codegen.json`` is the only generator that currently lives here. It produces the
+generated Python modules of the ``opentelemetry-proto-json`` distribution, under its ``src``
+directory. The packaging files and the tests of that distribution are hand-written.
+"""

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/__init__.py

@@ -0,0 +1,27 @@
+# Copyright The OpenTelemetry Authors
+# SPDX-License-Identifier: Apache-2.0
+
+"""Protocol buffer plugin that generates OTLP/JSON dataclasses.
+
+``protoc`` discovers this generator as the plugin named ``otlp_json``, through the
+``protoc-gen-otlp_json`` console script declared in the distribution metadata of
+``opentelemetry-codegen-json``. Passing ``--otlp_json_out=<directory>`` to ``protoc`` makes it run
+``opentelemetry.codegen.json.plugin.main``, which hands the request to
+``opentelemetry.codegen.json.generator``.
+
+For each ``.proto`` file named in ``file_to_generate`` on the request, and not for the descriptors
+that are only dependencies, the generator emits one Python module holding a ``slots=True``
+dataclass per protobuf message and an ``enum.IntEnum`` per protobuf enum. Every generated message
+class carries ``to_dict`` and ``from_dict`` methods that implement the Proto3 JSON mapping, with
+the OTLP deviation that ``trace_id``, ``span_id`` and ``parent_span_id`` are hex encoded rather
+than base64 encoded. Alongside those modules it copies
+``opentelemetry.codegen.json.runtime.json_codec`` into the output tree as ``_json_codec``. The
+generated code imports that copy, which belongs to the generated distribution, so no part of
+``opentelemetry-codegen-json`` is imported at runtime.
+
+The modules here are hand-written. Everything emitted into the ``opentelemetry-proto-json``
+distribution is generated and must never be edited by hand, though only the per-message modules
+say so in an ``AUTO-GENERATED`` header. The copied ``_json_codec.py``, the generated
+``version/__init__.py`` and the empty ``__init__.py`` files carry no header and are regenerated
+just the same.
+"""

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/generator.py

@@ -3,6 +3,31 @@
 
 # pylint: disable=no-member,invalid-name,too-many-lines
 
+"""Turns protobuf descriptors into OTLP/JSON dataclass modules.
+
+This module is hand-written and runs at build time. Everything it produces is a string of Python
+source destined for the ``opentelemetry-proto-json`` distribution, where it is committed and must
+not be edited by hand. Only the per-message modules say so in an ``AUTO-GENERATED`` header; the
+copied ``_json_codec.py``, the generated ``version/__init__.py`` and the empty ``__init__.py``
+files carry no header and are overwritten by a regeneration just the same. Changing the shape of
+the generated code means changing the code here and regenerating, which
+``scripts/proto_codegen_json.sh`` does.
+
+``OtlpJsonGenerator`` holds the whole translation. ``generate_plugin_response`` is what
+``opentelemetry.codegen.json.plugin`` calls, and it turns the generator's output into the
+``CodeGeneratorResponse`` that ``protoc`` reads back. ``generate_code`` is the inner form it is
+built from, returning a plain mapping from output path to source text.
+
+The output layout mirrors the input layout. A ``package_transform`` callable rewrites each
+``.proto`` path, by default replacing the substring ``opentelemetry/proto/`` with
+``opentelemetry/proto_json/``, and the longest directory prefix shared by every transformed path
+becomes the common root. Two things are written at that common root rather than per proto file:
+``_json_codec.py``, copied verbatim from
+``opentelemetry.codegen.json.runtime.json_codec``, and ``version/__init__.py``, holding
+``__version__``. An empty ``__init__.py`` is then added to every ancestor directory of a
+generated file, apart from the ``opentelemetry`` namespace directory itself.
+"""
+
 from __future__ import annotations
 
 import logging
@@ -34,6 +59,24 @@ class OtlpJsonGenerator:
     """
     Generates Python dataclasses and JSON serialization/deserialization code
     from protobuf descriptors.
+
+    Construction indexes every message and enum in every descriptor of ``request``, including the
+    descriptors that are only dependencies and are not themselves being generated, because a field
+    in a generated file may name a type defined in one of those. Generation itself happens in
+    ``generate_all``.
+
+    An instance produces source text only, and nothing here touches the filesystem except to read
+    the codec module. ``protoc`` is what writes the files, from the response that
+    ``opentelemetry.codegen.json.plugin`` hands back to it.
+
+    Args:
+        request: The ``CodeGeneratorRequest`` that ``protoc`` wrote to the plugin's standard input.
+            Its ``proto_file`` field carries every descriptor, and its ``file_to_generate`` field
+            names the subset to emit modules for.
+        package_transform: Rewrites a ``.proto`` path into the path of the Python file to generate
+            from it. The ``.proto`` suffix is replaced with ``.py`` afterwards, so this callable
+            only has to deal with the directory part.
+        version: The value written as ``__version__`` into the generated ``version/__init__.py``.
     """
 
     def __init__(
@@ -43,12 +86,7 @@ def __init__(
         version: str,
     ) -> None:
         """
-        Initialize the generator.
-
-        Args:
-            request: Protobuf compiler plugin request
-            package_transform: A callable that transforms the proto file path.
-            version: Version string for the generated code.
+        Initialize the generator. The parameters are documented on the class.
         """
         self._request = request
         self._package_transform = package_transform
@@ -616,9 +654,25 @@ def _get_serialization_expr(self, field_desc: descriptor.FieldDescriptorProto, v
         """
         Get the Python expression to serialize a value of a given type for JSON output.
 
+        The expression is a fragment of generated source, so ``var_name`` is spliced into it as
+        written rather than evaluated here. For a repeated field the caller passes the loop
+        variable of a lambda, and for a singular field it passes an attribute access on ``self``.
+
+        A field named ``trace_id``, ``span_id`` or ``parent_span_id`` is hex encoded in preference
+        to the base64 encoding that Proto3 JSON gives bytes, which is the OTLP deviation. The type
+        is consulted first: a message-typed or enum-typed field is handled by its type whatever it
+        is called. Among the remaining types the name alone decides, so a field with one of those
+        three names is hex encoded even when it is declared as something other than ``bytes``.
+
         Args:
             field_desc: Field descriptor for the value being serialized
             var_name: Variable name representing the value to serialize
+
+        Returns:
+            A Python expression that evaluates to the JSON form of the value. For a type that
+            needs no conversion, such as a string or a 32-bit integer, this is ``var_name``
+            unchanged. On the repeated branch the caller compares the result against ``var_name``
+            to decide whether the list needs a mapping call at all.
         """
         codec = self._get_codec_module_path()
         if field_desc.type == descriptor.FieldDescriptorProto.TYPE_MESSAGE:
@@ -858,8 +912,15 @@ def _load_codec_source() -> str:
     """
     Load the source code for the codec module from its source file.
 
+    The text is copied into the generated package unchanged, as ``_json_codec.py``, so that the
+    generated code carries its own runtime helpers and does not import this distribution.
+
     Returns:
         Source code as a string
+
+    Raises:
+        RuntimeError: If ``runtime/json_codec.py`` cannot be read. Generation cannot continue,
+            because every generated module imports the codec module by name.
     """
     codec_src_path = Path(__file__).parent / "runtime" / "json_codec.py"
     try:
@@ -908,12 +969,22 @@ def generate_code(
     """
     Main entry point for code generation.
 
+    ``generate_plugin_response`` is the only caller. This form exists because a mapping from path
+    to source is easier to read and to assert against than a protobuf response message.
+
     Args:
         request: Protobuf compiler plugin request
-        package_transform: Package transformation string or callable
+        package_transform: Rewrites a ``.proto`` path into the path of the Python file to generate
+            from it. The default substitutes ``opentelemetry/proto_json/`` for
+            ``opentelemetry/proto/`` wherever that substring occurs, and leaves a path that does
+            not contain it alone.
 
     Returns:
-        Dictionary mapping output file paths to generated code
+        Dictionary mapping output file paths to generated code. Besides one entry per file in
+        ``request.file_to_generate``, it holds ``_json_codec.py`` and ``version/__init__.py`` at
+        the common root, and an ``__init__.py`` for every ancestor directory of a generated file
+        apart from the ``opentelemetry`` namespace directory. It is empty when
+        ``file_to_generate`` is.
     """
     generator = OtlpJsonGenerator(request, package_transform, version="0.0.0")
     return generator.generate_all()
@@ -926,12 +997,20 @@ def generate_plugin_response(
     """
     Generate plugin response with all generated files.
 
+    Besides the files, the response declares which ``protoc`` features this plugin understands.
+    Without ``FEATURE_PROTO3_OPTIONAL`` declared, ``protoc`` refuses to pass a descriptor
+    containing an ``optional`` proto3 field, and the edition range states which editions of the
+    language the plugin accepts.
+
     Args:
         request: Protobuf compiler plugin request
-        package_transform: Package transformation string
+        package_transform: Rewrites a ``.proto`` path into the path of the Python file to generate
+            from it. The default substitutes ``opentelemetry/proto_json/`` for
+            ``opentelemetry/proto/`` wherever that substring occurs, and leaves a path that does
+            not contain it alone.
 
     Returns:
-        Plugin response with generated files
+        Plugin response with generated files, ready for ``protoc`` to write to disk.
     """
     response = plugin.CodeGeneratorResponse()
 

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/plugin.py

@@ -3,6 +3,27 @@
 
 # pylint: disable=no-member
 
+"""The ``protoc`` plugin entry point for the OTLP/JSON generator.
+
+``protoc`` invokes a plugin as a separate process, with an empty argument vector. It writes a
+serialized ``CodeGeneratorRequest`` to the standard input of that process and reads a serialized
+``CodeGeneratorResponse`` back from its standard output, so a plugin must keep both streams free
+of anything else. Diagnostics therefore go to standard error.
+
+Nothing configures logging on the path ``protoc`` takes. The ``logging.basicConfig`` call below
+runs only when this module is executed as ``__main__``, and ``protoc`` reaches ``main`` through
+the console script instead, which imports the module under its own name. A record of ``WARNING``
+or above therefore arrives at standard error through ``logging.lastResort``, carrying its message
+alone and none of the formatting set up below, and a record under ``WARNING`` is dropped.
+
+``protoc`` finds this module through the ``protoc-gen-otlp_json`` console script, which the
+``opentelemetry-codegen-json`` distribution declares. The name after ``protoc-gen-`` is the plugin
+name, so ``--otlp_json_out=<directory>`` on a ``protoc`` command line is what runs ``main``.
+
+The actual generation lives in ``opentelemetry.codegen.json.generator``. This module only handles
+the process protocol around it.
+"""
+
 import logging
 import sys
 from collections.abc import Iterator
@@ -18,7 +39,25 @@
 @contextmanager
 def code_generation() -> Iterator[tuple[plugin.CodeGeneratorRequest, plugin.CodeGeneratorResponse],]:
     """
-    Context manager for handling the code generation process.
+    Read the ``protoc`` request from standard input and write the response to standard output.
+
+    Unless the version branch below is taken, on entry this reads every byte of standard input,
+    parses it as a ``CodeGeneratorRequest``, and yields that request together with an empty
+    ``CodeGeneratorResponse`` for the body of the ``with`` statement to fill in. On exit it
+    serializes the response and writes it to standard output.
+
+    Two cases write no response to standard output at all. A first command line argument of
+    ``-V`` or ``--version`` prints the version and calls ``sys.exit`` before anything is read from
+    standard input, so the body never runs and the call does not block on a stream that may never
+    close. That branch answers a version query typed directly at the console script;
+    ``protoc`` never reaches it, because ``protoc`` runs a plugin with an empty argument vector.
+    An exception raised inside the body propagates instead, and because the serialization is
+    placed after the ``yield`` rather than in a ``finally`` block, it is skipped, so ``protoc``
+    reports the failure instead of accepting a partial result.
+
+    Returns:
+        A context manager yielding the parsed ``CodeGeneratorRequest`` and the empty
+        ``CodeGeneratorResponse`` that the caller is expected to populate.
     """
     if len(sys.argv) > 1 and sys.argv[1] in ("-V", "--version"):
         print("opentelemetry-codegen-json 0.0.0")
@@ -39,7 +78,18 @@ def code_generation() -> Iterator[tuple[plugin.CodeGeneratorRequest, plugin.Code
 
 def main() -> None:
     """
-    Main entry point for the protoc plugin.
+    Run one ``protoc`` plugin invocation.
+
+    This is the callable behind the ``protoc-gen-otlp_json`` console script. It passes the request
+    read from standard input to
+    ``opentelemetry.codegen.json.generator.generate_plugin_response``, then merges the feature
+    flags of that result into the response that ``code_generation`` writes out and copies every
+    generated file across one at a time.
+
+    Both steps are needed because ``code_generation`` owns the response object that gets
+    serialized. The flags are merged with ``|=`` rather than assigned, so a flag the response
+    already carries survives. The files are copied one at a time because a repeated protobuf
+    message field cannot be rebound to another message, only added to and merged into.
     """
     with code_generation() as (request, response):
         generated_response = generate_plugin_response(request)

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/runtime/__init__.py

@@ -0,0 +1,15 @@
+# Copyright The OpenTelemetry Authors
+# SPDX-License-Identifier: Apache-2.0
+
+"""Source modules that the generator copies verbatim into its output.
+
+The generator does not import a module under ``opentelemetry.codegen.json.runtime`` in order to
+call it. It reads the text of the module off disk and writes that text into the generated package,
+so that the generated code never requires ``opentelemetry-codegen-json`` to be installed.
+
+``opentelemetry.codegen.json.runtime.json_codec`` is the only such module. It lands in the output
+tree as ``_json_codec``, next to the generated message modules.
+
+Changing a module here changes every package generated afterwards, so the copies already committed
+under ``opentelemetry-proto-json`` stay as they are until the next regeneration.
+"""