Add opt-in X.509 cross-consistency checks (EKU chaining, RSA-PSS key params, critical BasicConstraints)

[Frauschi]

Background

Follow-up to the algorithm-confusion hardening in wolfSSL#10131 (SigOidMatchesKeyOid,
which makes a certificate's signature-algorithm OID agree with the issuer key's
OID). An audit of the rest of wolfSSL's X.509 parsing/validation looked for the
same class of issue — logically related fields/extensions that should agree
or constrain each other but are not cross-checked — and surfaced three further
gaps. This PR drafts fixes for them.

Each fix is gated behind a new, default-off macro, so existing builds are
byte-for-byte unchanged. The full testsuite.test passes with the macros off,
and also passes with B1+B2 on.

Changes

Files: wolfcrypt/src/asn.c (+106), wolfssl/wolfcrypt/asn.h (+21).

WOLFSSL_CHECK_EKU_CHAIN (B1)

The issuing CA's extendedKeyUsage did not constrain the certs it issued — only
the leaf's EKU was checked, at TLS time. The CA's effective EKU is now propagated
onto its Signer in FillSigner, and during chain verification a certificate
that asserts an EKU not permitted by the issuing CA is rejected
(EXTKEYUSAGE_E). An absent CA EKU or anyExtendedKeyUsage imposes no
restriction; the restriction propagates transitively down the path.

WOLFSSL_CHECK_RSAPSS_KEY_PARAMS (B2)

An id-RSASSA-PSS issuer key's restricting parameters (RFC 4055) were parsed and
then discarded — the long-standing TODO: store parameters so that usage can be checked in DecodeRsaPublicKey. The SPKI PSS parameters are now captured on
the DecodedCert/Signer, and a signature made by that key is required to use
the mandated hash/MGF/salt (ASN_SIG_OID_E on mismatch).

WOLFSSL_REQUIRE_CRITICAL_BASIC_CONSTRAINTS (B7)

RFC 5280 §4.2.1.9 requires CA certificates to mark basicConstraints critical;
wolfSSL stored the critical bit but never required it. An intermediate CA
whose basicConstraints is not critical is now rejected (ASN_CRIT_EXT_E).
Self-signed trust anchors are exempt, since their own extensions are not
processed during RFC 5280 §6.1 path validation.

Validation

• Default build (all macros off): full testsuite.test passes — zero
  behavioral change for existing users.
• B1: proof-of-concept — an OCSPSigning-only CA issuing a serverAuth
  leaf now fails verification (-247); positive/negative EKU subset controls
  behave correctly; testsuite.test passes with B1 on.
• B2: compiles and testsuite.test passes (normal RSA-PSS certs use an
  rsaEncryption SPKI, so the check is skipped). A runtime PoC is not feasible
  with the bundled certgen, which cannot emit a parameter-constrained
  id-RSASSA-PSS SPKI; the gap and fix are unambiguous from source.
• B7: proof-of-concept — an intermediate CA with non-critical
  basicConstraints now fails verification (-160), self-signed roots still
  load.

Notes / discussion points

• These macros tighten RFC conformance in ways that reject some
  non-conforming-but-common certificates, which is why they are opt-in. In
  particular, B7 is strict: with it enabled, wolfSSL's own intentionally
  lenient (non-critical-BC) test intermediates are rejected, so the bundled
  testsuite is expected to surface rejections when B7 is on.
• If preferred, B7 could surface via the verify callback (a soft signal) rather
  than a hard parse rejection — happy to adjust.
• Lower-severity hardening items from the audit (e.g. byte-comparing non-PSS
  inner/outer AlgorithmIdentifier parameters, serial-number positivity/length)
  were left out of this PR and can be added if wanted.

https://claude.ai/code/session_01NSH5QDCbE9n1hKYQUYbKQA

---

Generated by Claude Code