From b7ffb3ad496535909215e43b0bb8dfea446aa0a5 Mon Sep 17 00:00:00 2001 From: ryankert01 Date: Sun, 2 Aug 2026 20:03:13 +0800 Subject: [PATCH 01/48] feat(ws2): add TP-aware logprob contract and dispatch metadata Implements PR 1 of issue #241: a typed contract for vocab-parallel selected-token logprob, mirroring the WS2 attention contract pattern. - rl_engine/kernels/logprob_contract.py: LogprobContract, ShardingSpec (per-rank vocab shard bounds, padded-vs-real vocab, TP/CP rank metadata, owner_rank resolution), MaskSpec (active-token mask, ignore_index), ReductionSpec (fp32 (max, sumexp) merge in fixed global vocab-shard index order, all-gather transport, CP declared a non-merge axis), and LogprobBackendCapability. - KernelRegistry.get_logprob_op(contract): contract-aware dispatch that only selects backends with a declared capability; incompatible or undeclared candidates are rejected with explicit reasons and never used as a silent fallback. Existing WS1 batch-invariant logp backends are declared truthfully as single-shard references, so strict WS2 requests fail loudly until the deterministic vocab-parallel TP reference (PR 3) lands. Legacy get_op() behavior is unchanged. - Design doc, runtime-dispatch and operator doc updates, and CPU-safe contract/dispatch tests covering the Qwen3-8B TP=2 BF16 target and the TP=1/2/4 sweep shapes. Tolerance values remain owned by #108. --- .github/workflows/ci.yml | 3 + docs/design/runtime-dispatch.md | 7 + docs/design/ws2-tp-logprob-contract.md | 196 ++++++++++ docs/operators/batch-invariant-logp.md | 13 + rl_engine/kernels/logprob_contract.py | 501 +++++++++++++++++++++++++ rl_engine/kernels/registry.py | 192 ++++++++++ tests/test_logprob_contract.py | 402 ++++++++++++++++++++ 7 files changed, 1314 insertions(+) create mode 100644 docs/design/ws2-tp-logprob-contract.md create mode 100644 rl_engine/kernels/logprob_contract.py create mode 100644 tests/test_logprob_contract.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 92cd0433..28cdb58d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -76,6 +76,9 @@ jobs: run: | python -m pytest tests/test_kv_cache_attention.py -v -k "not large and not gpu" + - name: Run WS2 Logprob Contract Tests (CPU-safe) + run: python -m pytest tests/test_logprob_contract.py -v + docs: runs-on: ubuntu-latest steps: diff --git a/docs/design/runtime-dispatch.md b/docs/design/runtime-dispatch.md index bedf3475..84f29ec3 100644 --- a/docs/design/runtime-dispatch.md +++ b/docs/design/runtime-dispatch.md @@ -11,6 +11,13 @@ logical type, and the registry selects the first available backend for the curre 4. Cache successfully constructed operator instances. 5. Skip backends that already failed in the current process. +WS2 TP-aware logprob uses the stricter `KernelRegistry.get_logprob_op(contract)` path. In +addition to platform priority, this path requires a backend capability descriptor and checks +the requested role, dtype, TP/CP layout, padded-vs-real vocab masking, inactive-token +support, vocab-domain LSE export, and deterministic TP merge semantics. Incompatible +candidates produce explicit rejection reasons and are never used as an undeclared fallback. +See [WS2 TP-aware logprob contract](ws2-tp-logprob-contract.md). + ## LogP Priority | Platform | Priority | diff --git a/docs/design/ws2-tp-logprob-contract.md b/docs/design/ws2-tp-logprob-contract.md new file mode 100644 index 00000000..a392393b --- /dev/null +++ b/docs/design/ws2-tp-logprob-contract.md @@ -0,0 +1,196 @@ +# WS2 TP-Aware Logprob Contract + +Status: PR1 contract and dispatch metadata + +Tracking and shared contracts: + +- [#241: TP-aware deterministic logprob](https://github.com/RL-Align/RL-Kernel/issues/241) +- [#83: WS2 roadmap](https://github.com/RL-Align/RL-Kernel/issues/83) +- [#108: WS1 numerical contract](https://github.com/RL-Align/RL-Kernel/issues/108) +- [#111: WS2 cross-config alignment](https://github.com/RL-Align/RL-Kernel/issues/111) +- [#116: WS2 tolerance and drift-report format](https://github.com/RL-Align/RL-Kernel/issues/116) +- [Cross-config logprob drift contract](ws2_cross_config_logprob_drift_contract.md) + +## Scope + +This contract describes the logical inputs and deterministic reduction semantics for +selected-token log-probability under vocab-parallel tensor parallelism (TP): + +```text +selected_logp[t] = logits[t, target[t]] - logsumexp_vocab(logits[t, :]) +``` + +Under vocab-parallel TP each rank holds one vocabulary shard, so the vocabulary-wide +`logsumexp` requires a cross-rank reduction. This contract lets runtime dispatch reject a +backend whose numerical semantics do not match the requested layout. + +This PR1 layer does not shard tensors, launch a collective, merge `(max, sumexp)` partial +states, or implement a kernel. The single-GPU harness registration, the deterministic +vocab-parallel TP reference, and the cross-config integration belong to later PRs in #241. + +Context parallelism (CP) is a declared non-merge axis. CP partitions tokens, never the +vocabulary, so the logprob reduction spans TP vocab shards only. CP rank metadata is carried +for provenance and must never widen the merge. + +## Contract Objects + +`rl_engine.kernels.logprob_contract` defines: + +- `LogprobContract`: role, logits dtype, mask, sharding, reduction, and LSE export; +- `ShardingSpec`: per-rank vocab-shard bounds, padded-vs-real vocabulary, TP/CP rank + metadata, and target-token ownership; +- `MaskSpec`: active-token mask and ignore index; +- `ReductionSpec`: fixed `(max, sumexp)` merge semantics; +- `LogprobBackendCapability`: the layouts and semantics a backend explicitly supports. + +Construction performs validation immediately. A structurally valid contract means that the +request is complete and internally consistent; it does not mean that an installed backend can +materialize it. + +`ShardingSpec.vocab_shard_bounds` lists every TP rank's half-open `[start, end)` vocab range +indexed by TP rank. The full table is required on every rank: it defines target ownership +and the fixed merge order without any collective, and it makes an incomplete or overlapping +partition a loud construction-time error instead of a silent runtime divergence. +`ShardingSpec.owner_rank(token_id)` resolves the unique owning rank for a real-vocab token +and rejects everything else. + +`padded_vocab_size` is the shard-covered (weight) vocabulary; `real_vocab_size` is the +tokenizer vocabulary. Padding columns occupy `[real_vocab_size, padded_vocab_size)` and must +be excluded from the logsumexp by any conforming implementation. The two sizes are equal +when the vocabulary is unpadded. + +Inactive tokens (prompt, padding, masked-out response positions) are excluded from every +drift aggregate and are exempt from the exactly-one-owner target gather; their targets may +legally hold `ignore_index`. `ignore_index` must not collide with the real vocabulary. + +## Qwen3-8B TP=2 BF16 Example + +```python +from rl_engine.kernels.logprob_contract import ( + LogprobContract, + MaskSpec, + ReductionSpec, + ShardingSpec, +) + +sharding = ShardingSpec( + tp_rank=0, + tp_world_size=2, + vocab_shard_bounds=((0, 76032), (76032, 152064)), + real_vocab_size=151936, + padded_vocab_size=152064, + cp_rank=0, + cp_world_size=2, +) + +contract = LogprobContract( + role="train", + dtype="bf16", + mask=MaskSpec( + num_tokens=8, + active_mask=(False, False, True, True, True, True, True, False), + ignore_index=-100, + ), + sharding=sharding, + reduction=ReductionSpec(), +) +``` + +Each rank owns one contiguous vocab shard; the 128 padding columns at the end of rank 1's +shard are outside the real vocabulary and never contribute to the logsumexp. The two leading +prompt tokens and the trailing padding token are inactive. + +## Reduction Semantics + +The only PR1 reduction contract is: + +```text +partial state: (local_max, local_sumexp), fp32 +merge: max_sumexp +merge_axis: tp_vocab +order: global_vocab_shard_index +transport: all_gather +downcast_at: final_write +engine: in_op_reference +``` + +Every rank computes `m_l = max(local_logits)` and `s_l = sum(exp(local_logits - m_l))` in +fp32, the partials travel by all-gather (collectives are transport only, never a numerical +reduction), and every rank merges in fixed global vocab-shard index order: + +```text +M = max_l(m_l) +S = sum_l(s_l * exp(m_l - M)) +LSE = M + log(S) +selected_logp = target_logit - LSE +``` + +The selected target logit comes from a masked single-owner gather: exactly one rank holds +each active token's target column. Downcast happens only at the final write. Because the +merge order is fixed by shard index, TP=2 is bitwise-equal to TP=1 by construction; averaging +per-rank logsumexp values or letting a collective reduce numerically is not conformant. + +The acceptable LSE and selected-token drift thresholds remain owned by #108, and drift +reports follow the #116 format. This contract does not introduce another tolerance table. +The selected-token metric remains the cross-config convention: + +```text +dlogp = training-side recomputed logp - rollout-side old logp +``` + +computed over active response tokens only. + +## Contract-Aware Dispatch + +Legacy callers continue to use `KernelRegistry.get_op()`. WS2 callers use: + +```python +result = kernel_registry.get_logprob_op(contract) +op = result.op +provenance = result.provenance +``` + +Dispatch considers only backends with a `LogprobBackendCapability`. It checks role, dtype, +TP/CP degree, padded-vs-real vocab masking, inactive-token support, vocab-domain LSE export, +and deterministic TP merge. An undeclared or incompatible backend is skipped with an +explicit rejection reason; there is no silent fallback. + +`requested_backend` accepts a case-insensitive policy keyword (`auto` | `production` | +`reference` | `deterministic`; default `auto`) or an exact, case-sensitive stable backend +id. Strictness comes from the contract's capability checks, not from the policy string. A +backend id may never shadow a policy keyword; capability construction rejects that. The +provenance `fallback` flag reports only capability or load rejections of otherwise-eligible +candidates — skips caused purely by the caller's own policy filter are not fallbacks. + +WS2 dispatch resolves from its own candidate list, seeded from but decoupled from the legacy +`batch_invariant_logp` priority list: registering a TP-vocab backend for WS2 dispatch does +not change what legacy `get_op("batch_invariant_logp")` returns to WS1 callers. + +The current WS1 batch-invariant logp implementations are single-shard (TP=1) references: +they accept full-vocabulary logits with ignore-index masking but carry no vocab-shard +metadata, no padded-vs-real vocab distinction, and no public vocab-domain LSE export. Strict +WS2 requests therefore fail clearly today. The later deterministic vocab-parallel reference +becomes selectable by registering a capability that truthfully declares those features; no +controller branch or silent fallback is required. + +Successful dispatch provenance records: + +- requested and actual backend ids; +- platform and fallback status; +- prior candidate rejection reasons; +- the complete requested contract, including shard bounds, padded and real vocab sizes, + merge semantics, and the explicit `cp_is_merge_axis: false` declaration; +- the selected backend capability descriptor. + +## Validation + +Contract and dispatch behavior are covered by: + +```bash +python -m pytest tests/test_logprob_contract.py -q +``` + +The tests include Qwen3-8B TP=2 BF16 construction with padded vocab, the TP=1/2/4 sweep +shapes, incomplete/overlapping shard-bound rejection, owner-rank resolution, active-mask and +ignore-index validation, fp32-accumulation and merge-semantics enforcement, undeclared +backend rejection, no incompatible fallback, and JSON-compatible provenance. diff --git a/docs/operators/batch-invariant-logp.md b/docs/operators/batch-invariant-logp.md index fbc0e9f1..b0671c40 100644 --- a/docs/operators/batch-invariant-logp.md +++ b/docs/operators/batch-invariant-logp.md @@ -54,6 +54,19 @@ CUDA priority list when the extension exposes `_C.batch_invariant_logp_sm90` (built with `KERNEL_ALIGN_FORCE_SM90=1`) on an SM90 device. On any other build or device, dispatch is unchanged (Triton -> PyTorch). +### WS2 TP-aware dispatch + +WS2 distributed callers use a separate contract-aware entry point, +`kernel_registry.get_logprob_op(contract)`. It validates explicit vocab-shard ownership, +padded-vs-real vocab metadata, active-token masking, and fixed `(max, sumexp)` merge +semantics before selecting a backend. Legacy `get_op("batch_invariant_logp")` behavior +remains unchanged. + +The backends above are single-shard (TP=1) references and do not yet export vocab-domain +LSE or carry vocab-shard metadata, so they are declared incompatible with strict WS2 +requests instead of being selected as a silent fallback. See +[WS2 TP-aware logprob contract](../design/ws2-tp-logprob-contract.md). + ## Benchmarks `benchmarks/benchmark_batch_invariant_logp.py` compares Native, Triton, and the diff --git a/rl_engine/kernels/logprob_contract.py b/rl_engine/kernels/logprob_contract.py new file mode 100644 index 00000000..8f2667e1 --- /dev/null +++ b/rl_engine/kernels/logprob_contract.py @@ -0,0 +1,501 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Typed WS2 contract for TP-aware selected-token log-probability. + +The objects in this module describe a vocab-parallel logprob invocation: + +``selected_logp[t] = logits[t, target[t]] - logsumexp_vocab(logits[t, :])`` + +Under vocab-parallel tensor parallelism the vocabulary-wide ``logsumexp`` +requires cross-rank reduction. This module only *describes* that invocation +(shard ownership, merge semantics, mask/ignore-index metadata); it does not +shard tensors, launch collectives, or implement the ``(max, sumexp)`` merge. +Keeping description and materialization separate lets dispatch reject an +incompatible backend before any numerically different path is launched. + +Context parallelism is a declared non-merge axis: CP partitions tokens, never +the vocabulary, so the logprob reduction spans TP vocab shards only. CP rank +metadata is carried for provenance and must never widen the merge. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from enum import Enum +from typing import Any, TypeVar + +_EnumT = TypeVar("_EnumT", bound=Enum) + +# Dispatch policy keywords accepted by KernelRegistry.get_logprob_op; a stable +# backend id must never shadow one of these, or it becomes unselectable by id. +RESERVED_DISPATCH_POLICIES = frozenset({"auto", "production", "reference", "deterministic"}) + + +class LogprobContractError(ValueError): + """Raised when logprob metadata does not describe a valid invocation.""" + + +class LogprobRole(str, Enum): + TRAIN = "train" + INFER = "infer" + + +class LogprobDType(str, Enum): + BF16 = "bf16" + FP16 = "fp16" + FP32 = "fp32" + + +class LogprobMerge(str, Enum): + """Merge primitive for per-shard partial states. + + Every rank contributes ``(local_max, local_sumexp)`` computed in the + accumulation dtype; the merged result is + ``M = max(m_l)``, ``S = sum(s_l * exp(m_l - M))``, ``LSE = M + log(S)``. + """ + + MAX_SUMEXP = "max_sumexp" + + +class MergeAxis(str, Enum): + """The only reduction axis of this contract; CP is a non-merge axis.""" + + TP_VOCAB = "tp_vocab" + + +class ReductionOrder(str, Enum): + GLOBAL_VOCAB_SHARD_INDEX = "global_vocab_shard_index" + + +class ReductionTransport(str, Enum): + """Collectives move partial states only; they never reduce numerically.""" + + ALL_GATHER = "all_gather" + + +class DowncastPoint(str, Enum): + FINAL_WRITE = "final_write" + + +class ReductionEngine(str, Enum): + IN_OP_REFERENCE = "in_op_reference" + + +def _enum_value(enum_type: type[_EnumT], value: Any, field: str) -> _EnumT: + try: + return enum_type(value) + except (TypeError, ValueError) as exc: + allowed = ", ".join(item.value for item in enum_type) + raise LogprobContractError(f"{field} must be one of: {allowed}; got {value!r}") from exc + + +def _positive_int(value: Any, field: str) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + raise LogprobContractError(f"{field} must be a positive integer; got {value!r}") + return value + + +def _non_negative_int(value: Any, field: str) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise LogprobContractError(f"{field} must be a non-negative integer; got {value!r}") + return value + + +def _plain_int(value: Any, field: str) -> int: + if isinstance(value, bool) or not isinstance(value, int): + raise LogprobContractError(f"{field} must be an integer; got {value!r}") + return value + + +@dataclass(frozen=True) +class ShardingSpec: + """Logical vocab-parallel TP ownership for one logprob invocation. + + ``vocab_shard_bounds`` lists every TP rank's half-open ``[start, end)`` + vocab range indexed by TP rank. The full table is required on every rank: + it defines target-token ownership and the fixed global-shard-index merge + order without any collective, and makes an incomplete partition a loud + construction-time error instead of a silent runtime divergence. + + ``padded_vocab_size`` is the shard-covered (weight) vocabulary; + ``real_vocab_size`` is the tokenizer vocabulary. Padding columns occupy + ``[real_vocab_size, padded_vocab_size)`` and must be excluded from the + logsumexp by any conforming implementation. + """ + + tp_rank: int + tp_world_size: int + vocab_shard_bounds: tuple[tuple[int, int], ...] + real_vocab_size: int + padded_vocab_size: int + cp_rank: int = 0 + cp_world_size: int = 1 + + def __post_init__(self) -> None: + tp_world_size = _positive_int(self.tp_world_size, "tp_world_size") + tp_rank = _non_negative_int(self.tp_rank, "tp_rank") + if tp_rank >= tp_world_size: + raise LogprobContractError( + f"tp_rank={tp_rank} must be smaller than tp_world_size={tp_world_size}" + ) + cp_world_size = _positive_int(self.cp_world_size, "cp_world_size") + cp_rank = _non_negative_int(self.cp_rank, "cp_rank") + if cp_rank >= cp_world_size: + raise LogprobContractError( + f"cp_rank={cp_rank} must be smaller than cp_world_size={cp_world_size}" + ) + + real_vocab_size = _positive_int(self.real_vocab_size, "real_vocab_size") + padded_vocab_size = _positive_int(self.padded_vocab_size, "padded_vocab_size") + if padded_vocab_size < real_vocab_size: + raise LogprobContractError( + f"padded_vocab_size={padded_vocab_size} must not be smaller than " + f"real_vocab_size={real_vocab_size}" + ) + + try: + bounds = tuple((pair[0], pair[1]) for pair in self.vocab_shard_bounds) + except (TypeError, IndexError) as exc: + raise LogprobContractError( + "vocab_shard_bounds must be an iterable of (start, end) integer pairs" + ) from exc + if len(bounds) != tp_world_size: + raise LogprobContractError( + "vocab_shard_bounds must declare exactly one (start, end) pair per TP rank; " + f"got {len(bounds)} pairs for tp_world_size={tp_world_size}" + ) + expected_start = 0 + for rank, (start, end) in enumerate(bounds): + start = _plain_int(start, f"vocab_shard_bounds[{rank}][0]") + end = _plain_int(end, f"vocab_shard_bounds[{rank}][1]") + if end <= start: + raise LogprobContractError( + f"vocab_shard_bounds[{rank}] must satisfy end > start; got [{start}, {end})" + ) + if start != expected_start: + raise LogprobContractError( + "vocab_shard_bounds must form a contiguous [0, padded_vocab_size) " + f"partition in TP-rank order; rank {rank} starts at {start}, " + f"expected {expected_start}" + ) + expected_start = end + if expected_start != padded_vocab_size: + raise LogprobContractError( + "vocab_shard_bounds must cover padded_vocab_size exactly; " + f"covered {expected_start}, declared {padded_vocab_size}" + ) + object.__setattr__(self, "vocab_shard_bounds", bounds) + + @property + def local_vocab_start(self) -> int: + return self.vocab_shard_bounds[self.tp_rank][0] + + @property + def local_vocab_end(self) -> int: + return self.vocab_shard_bounds[self.tp_rank][1] + + @property + def local_vocab_size(self) -> int: + start, end = self.vocab_shard_bounds[self.tp_rank] + return end - start + + def owner_rank(self, token_id: int) -> int: + """Return the unique TP rank owning ``token_id``; error outside real vocab.""" + + token_id = _plain_int(token_id, "token_id") + if token_id < 0 or token_id >= self.real_vocab_size: + raise LogprobContractError( + f"token_id={token_id} is outside the real vocabulary " + f"[0, {self.real_vocab_size}); mask it as inactive instead" + ) + for rank, (start, end) in enumerate(self.vocab_shard_bounds): + if start <= token_id < end: + return rank + raise LogprobContractError( + f"token_id={token_id} is not covered by any declared vocab shard" + ) + + +@dataclass(frozen=True) +class MaskSpec: + """Active-token ownership for one logprob invocation. + + Inactive tokens are excluded from every drift aggregate and are exempt + from the exactly-one-owner target gather; their targets may legally hold + ``ignore_index``. + """ + + num_tokens: int + active_mask: tuple[bool, ...] + ignore_index: int = -100 + _active_token_count: int = field(init=False, repr=False, compare=False) + + def __post_init__(self) -> None: + num_tokens = _positive_int(self.num_tokens, "num_tokens") + _plain_int(self.ignore_index, "ignore_index") + try: + active_mask = tuple(self.active_mask) + except TypeError as exc: + raise LogprobContractError("active_mask must be an iterable of booleans") from exc + for index, value in enumerate(active_mask): + if not isinstance(value, bool): + raise LogprobContractError(f"active_mask[{index}] must be a bool; got {value!r}") + if len(active_mask) != num_tokens: + raise LogprobContractError( + "active_mask must contain exactly one entry per token; " + f"got {len(active_mask)} entries for num_tokens={num_tokens}" + ) + object.__setattr__(self, "active_mask", active_mask) + object.__setattr__(self, "_active_token_count", sum(active_mask)) + + @property + def active_token_count(self) -> int: + return self._active_token_count + + +@dataclass(frozen=True) +class ReductionSpec: + """Deterministic TP-vocab ``(max, sumexp)`` merge semantics.""" + + merge: LogprobMerge = LogprobMerge.MAX_SUMEXP + merge_axis: MergeAxis = MergeAxis.TP_VOCAB + acc_dtype: LogprobDType = LogprobDType.FP32 + order: ReductionOrder = ReductionOrder.GLOBAL_VOCAB_SHARD_INDEX + transport: ReductionTransport = ReductionTransport.ALL_GATHER + downcast_at: DowncastPoint = DowncastPoint.FINAL_WRITE + engine: ReductionEngine = ReductionEngine.IN_OP_REFERENCE + + def __post_init__(self) -> None: + object.__setattr__(self, "merge", _enum_value(LogprobMerge, self.merge, "merge")) + object.__setattr__( + self, "merge_axis", _enum_value(MergeAxis, self.merge_axis, "merge_axis") + ) + object.__setattr__( + self, "acc_dtype", _enum_value(LogprobDType, self.acc_dtype, "acc_dtype") + ) + object.__setattr__(self, "order", _enum_value(ReductionOrder, self.order, "order")) + object.__setattr__( + self, "transport", _enum_value(ReductionTransport, self.transport, "transport") + ) + object.__setattr__( + self, "downcast_at", _enum_value(DowncastPoint, self.downcast_at, "downcast_at") + ) + object.__setattr__(self, "engine", _enum_value(ReductionEngine, self.engine, "engine")) + if self.acc_dtype is not LogprobDType.FP32: + raise LogprobContractError( + f"TP logprob accumulation must be fp32; got {self.acc_dtype.value}" + ) + + +@dataclass(frozen=True) +class LogprobContract: + """Complete semantic request consumed by contract-aware dispatch.""" + + role: LogprobRole + dtype: LogprobDType + mask: MaskSpec + sharding: ShardingSpec + reduction: ReductionSpec + export_lse: bool = True + + def __post_init__(self) -> None: + object.__setattr__(self, "role", _enum_value(LogprobRole, self.role, "role")) + object.__setattr__(self, "dtype", _enum_value(LogprobDType, self.dtype, "dtype")) + if not isinstance(self.mask, MaskSpec): + raise LogprobContractError("mask must be a MaskSpec") + if not isinstance(self.sharding, ShardingSpec): + raise LogprobContractError("sharding must be a ShardingSpec") + if not isinstance(self.reduction, ReductionSpec): + raise LogprobContractError("reduction must be a ReductionSpec") + if not isinstance(self.export_lse, bool) or not self.export_lse: + raise LogprobContractError( + "export_lse must be True for the WS2 vocab-domain LSE drift contract" + ) + if 0 <= self.mask.ignore_index < self.sharding.real_vocab_size: + raise LogprobContractError( + f"ignore_index={self.mask.ignore_index} must not collide with the real " + f"vocabulary [0, {self.sharding.real_vocab_size})" + ) + + def to_dict(self) -> dict[str, Any]: + """Return stable, JSON-compatible requested-contract provenance.""" + + sharding = { + "tp_rank": self.sharding.tp_rank, + "tp_world_size": self.sharding.tp_world_size, + "cp_rank": self.sharding.cp_rank, + "cp_world_size": self.sharding.cp_world_size, + "vocab_shard_bounds": [list(pair) for pair in self.sharding.vocab_shard_bounds], + "real_vocab_size": self.sharding.real_vocab_size, + "padded_vocab_size": self.sharding.padded_vocab_size, + "local_vocab_start": self.sharding.local_vocab_start, + "local_vocab_end": self.sharding.local_vocab_end, + } + reduction = { + "merge": self.reduction.merge.value, + "merge_axis": self.reduction.merge_axis.value, + "acc_dtype": self.reduction.acc_dtype.value, + "order": self.reduction.order.value, + "transport": self.reduction.transport.value, + "downcast_at": self.reduction.downcast_at.value, + "engine": self.reduction.engine.value, + "cp_is_merge_axis": False, + } + mask = { + "num_tokens": self.mask.num_tokens, + "active_token_count": self.mask.active_token_count, + "active_mask": list(self.mask.active_mask), + "ignore_index": self.mask.ignore_index, + } + return { + "semantic_operator": "selected_token_logprob", + "role": self.role.value, + "dtype": self.dtype.value, + "export_lse": self.export_lse, + "lse_domain": "vocab", + "mask": mask, + "sharding": sharding, + "reduction": reduction, + } + + +@dataclass(frozen=True) +class LogprobBackendCapability: + """Capabilities a concrete backend declares to contract-aware dispatch.""" + + backend_id: str + roles: frozenset[LogprobRole] + dtypes: frozenset[LogprobDType] + tp_world_sizes: tuple[int, ...] | None = None + cp_world_sizes: tuple[int, ...] | None = None + supports_vocab_padding: bool = False + supports_inactive_tokens: bool = False + exports_vocab_lse: bool = False + deterministic_tp_merge: bool = False + implementation_kind: str = "production" + + def __post_init__(self) -> None: + if not isinstance(self.backend_id, str) or not self.backend_id.strip(): + raise LogprobContractError("backend_id must be a non-empty string") + if self.backend_id.strip().lower() in RESERVED_DISPATCH_POLICIES: + raise LogprobContractError( + f"backend_id={self.backend_id!r} shadows a reserved dispatch policy keyword" + ) + roles = frozenset(_enum_value(LogprobRole, value, "roles") for value in self.roles) + dtypes = frozenset(_enum_value(LogprobDType, value, "dtypes") for value in self.dtypes) + if not roles or not dtypes: + raise LogprobContractError("backend roles and dtypes must not be empty") + tp_world_sizes = self._validated_world_sizes(self.tp_world_sizes, "tp_world_sizes") + cp_world_sizes = self._validated_world_sizes(self.cp_world_sizes, "cp_world_sizes") + for flag_name in ( + "supports_vocab_padding", + "supports_inactive_tokens", + "exports_vocab_lse", + "deterministic_tp_merge", + ): + if not isinstance(getattr(self, flag_name), bool): + raise LogprobContractError(f"{flag_name} must be a bool") + if self.implementation_kind not in {"production", "reference", "deterministic"}: + raise LogprobContractError( + "implementation_kind must be production, reference, or deterministic" + ) + object.__setattr__(self, "roles", roles) + object.__setattr__(self, "dtypes", dtypes) + object.__setattr__(self, "tp_world_sizes", tp_world_sizes) + object.__setattr__(self, "cp_world_sizes", cp_world_sizes) + + @staticmethod + def _validated_world_sizes( + values: tuple[int, ...] | None, field: str + ) -> tuple[int, ...] | None: + if values is None: + return None + try: + sizes = tuple(values) + except TypeError as exc: + raise LogprobContractError(f"{field} must be an iterable of integers") from exc + if not sizes: + raise LogprobContractError(f"{field} must not be empty; use None for unrestricted") + for value in sizes: + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + raise LogprobContractError(f"{field} must contain positive values; got {value!r}") + if len(set(sizes)) != len(sizes): + raise LogprobContractError(f"{field} must not contain duplicates") + return sizes + + def incompatibilities(self, contract: LogprobContract) -> tuple[str, ...]: + """Explain every reason this backend cannot materialize ``contract``.""" + + reasons: list[str] = [] + if contract.role not in self.roles: + reasons.append(f"role={contract.role.value} is unsupported") + if contract.dtype not in self.dtypes: + reasons.append(f"dtype={contract.dtype.value} is unsupported") + tp_size = contract.sharding.tp_world_size + cp_size = contract.sharding.cp_world_size + if self.tp_world_sizes is not None and tp_size not in self.tp_world_sizes: + reasons.append(f"TP={tp_size} is unsupported") + if self.cp_world_sizes is not None and cp_size not in self.cp_world_sizes: + reasons.append(f"CP={cp_size} is unsupported") + if ( + contract.sharding.padded_vocab_size != contract.sharding.real_vocab_size + and not self.supports_vocab_padding + ): + reasons.append("padded-vs-real vocab masking is unsupported") + if ( + contract.mask.active_token_count != contract.mask.num_tokens + and not self.supports_inactive_tokens + ): + reasons.append("inactive-token (ignore_index) masking is unsupported") + if contract.export_lse and not self.exports_vocab_lse: + reasons.append("vocab-domain LSE export is unsupported") + if tp_size > 1 and not self.deterministic_tp_merge: + reasons.append("deterministic TP (max, sumexp) merge is unsupported") + return tuple(reasons) + + def supports(self, contract: LogprobContract) -> bool: + return not self.incompatibilities(contract) + + def to_dict(self) -> dict[str, Any]: + return { + "backend_id": self.backend_id, + "roles": sorted(role.value for role in self.roles), + "dtypes": sorted(dtype.value for dtype in self.dtypes), + "tp_world_sizes": list(self.tp_world_sizes) if self.tp_world_sizes else None, + "cp_world_sizes": list(self.cp_world_sizes) if self.cp_world_sizes else None, + "supports_vocab_padding": self.supports_vocab_padding, + "supports_inactive_tokens": self.supports_inactive_tokens, + "exports_vocab_lse": self.exports_vocab_lse, + "deterministic_tp_merge": self.deterministic_tp_merge, + "implementation_kind": self.implementation_kind, + } + + +@dataclass(frozen=True) +class LogprobDispatchResult: + """A concrete backend plus the actual provenance bound to the request.""" + + op: Any + capability: LogprobBackendCapability + provenance: dict[str, Any] + + +__all__ = [ + "DowncastPoint", + "LogprobBackendCapability", + "LogprobContract", + "LogprobContractError", + "LogprobDispatchResult", + "LogprobDType", + "LogprobMerge", + "LogprobRole", + "MaskSpec", + "MergeAxis", + "RESERVED_DISPATCH_POLICIES", + "ReductionEngine", + "ReductionOrder", + "ReductionSpec", + "ReductionTransport", + "ShardingSpec", +] diff --git a/rl_engine/kernels/registry.py b/rl_engine/kernels/registry.py index fb2feb6f..35ae951b 100644 --- a/rl_engine/kernels/registry.py +++ b/rl_engine/kernels/registry.py @@ -8,6 +8,14 @@ import torch +from rl_engine.kernels.logprob_contract import ( + LogprobBackendCapability, + LogprobContract, + LogprobContractError, + LogprobDispatchResult, + LogprobDType, + LogprobRole, +) from rl_engine.platforms.device import device_ctx from rl_engine.utils.logger import logger @@ -164,6 +172,51 @@ def __init__(self): self._instance_cache: Dict[str, Any] = {} self._failed_backends: Set[str] = set() + # These descriptors report what the existing WS1 batch-invariant logp + # implementations actually support: single-shard (TP=1) logits with + # ignore-index masking, no vocab-shard metadata, no padded-vs-real + # vocab distinction, and no public vocab-domain LSE export. A strict + # WS2 request is rejected with explicit reasons until the deterministic + # vocab-parallel TP reference backend lands (issue #241 PR 3) instead + # of silently selecting an incompatible fallback. + common_logprob_roles = frozenset({LogprobRole.TRAIN, LogprobRole.INFER}) + common_logprob_dtypes = frozenset({LogprobDType.BF16, LogprobDType.FP16, LogprobDType.FP32}) + self._logprob_capabilities = { + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP: LogprobBackendCapability( + backend_id="pytorch-batch-invariant-logp-ws1", + roles=common_logprob_roles, + dtypes=common_logprob_dtypes, + tp_world_sizes=(1,), + supports_vocab_padding=False, + supports_inactive_tokens=True, + exports_vocab_lse=False, + deterministic_tp_merge=False, + implementation_kind="reference", + ), + OpBackend.TRITON_BATCH_INVARIANT_LOGP: LogprobBackendCapability( + backend_id="triton-batch-invariant-logp-ws1", + roles=common_logprob_roles, + dtypes=common_logprob_dtypes, + tp_world_sizes=(1,), + supports_vocab_padding=False, + supports_inactive_tokens=True, + exports_vocab_lse=False, + deterministic_tp_merge=False, + implementation_kind="deterministic", + ), + OpBackend.CUDA_BATCH_INVARIANT_LOGP_SM90: LogprobBackendCapability( + backend_id="cuda-batch-invariant-logp-sm90-ws1", + roles=common_logprob_roles, + dtypes=frozenset({LogprobDType.BF16, LogprobDType.FP32}), + tp_world_sizes=(1,), + supports_vocab_padding=False, + supports_inactive_tokens=True, + exports_vocab_lse=False, + deterministic_tp_merge=False, + implementation_kind="deterministic", + ), + } + self._priority_map = { "cuda": { "logp": [ @@ -280,6 +333,17 @@ def __init__(self): self._adjust_priority_for_hardware() self._adjust_priority_from_env() + # WS2 contract-aware dispatch owns its candidate list. It is seeded + # from the legacy batch_invariant_logp priority (after hardware/env + # adjustments) but deliberately decoupled afterwards: registering a + # TP-vocab backend for WS2 dispatch must not change what legacy + # get_op("batch_invariant_logp") returns to WS1 callers, and vice + # versa. + self._logprob_candidates: Dict[str, list] = { + platform: list(ops.get("batch_invariant_logp", [])) + for platform, ops in self._priority_map.items() + } + def _adjust_priority_from_env(self): rocm_attn_backend = os.getenv("RL_KERNEL_ROCM_ATTN_BACKEND", "").strip().lower() if rocm_attn_backend in {"flash_attn", "flash-attn", "flash_attention"}: @@ -389,6 +453,134 @@ def get_op(self, op_type: str, device: torch.device | str | None = None) -> Any: raise RuntimeError(f"No functional backend found for {op_type} on {platform}") + def get_logprob_op( + self, + contract: LogprobContract, + *, + requested_backend: str = "auto", + ) -> LogprobDispatchResult: + """Resolve only a backend that explicitly supports the WS2 logprob contract. + + This entry point is intentionally separate from legacy ``get_op`` so + existing callers retain their current behavior while WS2 callers cannot + silently fall back to a backend with different distributed semantics. + + ``requested_backend`` is either a case-insensitive policy keyword + (``auto`` | ``production`` | ``reference`` | ``deterministic``) or an + exact, case-sensitive stable backend id. Strictness comes from the + contract's capability checks, not from this policy string, so the + default is ``auto``. + """ + + if not isinstance(contract, LogprobContract): + raise LogprobContractError("contract must be a LogprobContract") + if not isinstance(requested_backend, str) or not requested_backend.strip(): + raise LogprobContractError("requested_backend must be a non-empty string") + requested_backend = requested_backend.strip() + + platform = self._platform() + candidates = self._logprob_candidates.get(platform, []) + rejected: list[str] = [] + # provenance["fallback"] reports only capability/load rejections of + # otherwise-eligible candidates; skips caused purely by the caller's + # own requested_backend policy filter are not fallbacks. + capability_rejections = 0 + + for backend in candidates: + capability = self._logprob_capabilities.get(backend) + if capability is None: + rejected.append(f"{backend.name}: no LogprobBackendCapability declared") + capability_rejections += 1 + continue + capability_incompat = list(capability.incompatibilities(contract)) + policy_mismatch = self._logprob_policy_mismatch(requested_backend, capability) + reasons = capability_incompat + ([policy_mismatch] if policy_mismatch else []) + if reasons: + rejected.append(f"{backend.name}: " + "; ".join(reasons)) + if capability_incompat: + capability_rejections += 1 + continue + + op = self._get_or_create_backend(backend) + if op is None: + rejected.append(f"{backend.name}: backend could not be loaded or instantiated") + capability_rejections += 1 + continue + + provenance = { + "requested_backend": requested_backend, + "actual_backend": capability.backend_id, + "backend_enum": backend.name, + "platform": platform, + "fallback": capability_rejections > 0, + "prior_rejections": list(rejected), + "contract": contract.to_dict(), + "capability": capability.to_dict(), + } + return LogprobDispatchResult( + op=op, + capability=capability, + provenance=provenance, + ) + + details = " | ".join(rejected) if rejected else "no candidates registered" + requested = contract.to_dict() + raise RuntimeError( + "No logprob backend supports the requested WS2 contract on " + f"{platform}: role={requested['role']}, dtype={requested['dtype']}, " + f"TP={contract.sharding.tp_world_size}, CP={contract.sharding.cp_world_size}, " + f"padded_vocab={contract.sharding.padded_vocab_size}, " + f"real_vocab={contract.sharding.real_vocab_size}. Rejections: {details}" + ) + + @staticmethod + def _logprob_policy_mismatch( + requested_backend: str, + capability: LogprobBackendCapability, + ) -> str | None: + policy = requested_backend.lower() + if policy == "auto": + return None + if policy in {"production", "reference", "deterministic"}: + if capability.implementation_kind == policy: + return None + return ( + f"implementation_kind={capability.implementation_kind} does not satisfy " + f"requested_backend={policy}" + ) + if capability.backend_id == requested_backend: + return None + return ( + f"backend_id={capability.backend_id} does not match " + f"requested_backend={requested_backend}" + ) + + def _platform(self) -> str: + if device_ctx.is_rocm: + return "rocm" + if device_ctx.device_type == "cuda": + return "cuda" + return "cpu" + + def _get_or_create_backend(self, backend: OpBackend) -> Any | None: + if backend.name in self._instance_cache: + return self._instance_cache[backend.name] + if backend.name in self._failed_backends: + return None + + op_class = self._load_backend(backend) + if op_class is None: + self._failed_backends.add(backend.name) + return None + try: + op = op_class() + except Exception as exc: + logger.error(f"Failed to instantiate {backend.name}: {exc}") + self._failed_backends.add(backend.name) + return None + self._instance_cache[backend.name] = op + return op + def _platform_for_device(self, device: torch.device | str | None) -> str: if device is None: if device_ctx.is_rocm: diff --git a/tests/test_logprob_contract.py b/tests/test_logprob_contract.py new file mode 100644 index 00000000..d18083f4 --- /dev/null +++ b/tests/test_logprob_contract.py @@ -0,0 +1,402 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""WS2 TP-aware logprob contract and contract-aware dispatch tests (issue #241).""" + +from __future__ import annotations + +import json +from dataclasses import replace + +import pytest + +from rl_engine.kernels.logprob_contract import ( + LogprobBackendCapability, + LogprobContract, + LogprobContractError, + LogprobDType, + LogprobRole, + MaskSpec, + ReductionSpec, + ShardingSpec, +) +from rl_engine.kernels.registry import KernelRegistry, OpBackend + +QWEN3_REAL_VOCAB = 151936 +QWEN3_PADDED_VOCAB = 152064 + + +def _even_bounds(padded_vocab: int, tp_world_size: int) -> tuple[tuple[int, int], ...]: + shard = padded_vocab // tp_world_size + return tuple((rank * shard, (rank + 1) * shard) for rank in range(tp_world_size)) + + +def _sharding( + *, + tp_rank: int = 0, + tp_world_size: int = 2, + cp_rank: int = 0, + cp_world_size: int = 2, + real_vocab_size: int = QWEN3_REAL_VOCAB, + padded_vocab_size: int = QWEN3_PADDED_VOCAB, + vocab_shard_bounds: tuple[tuple[int, int], ...] | None = None, +) -> ShardingSpec: + return ShardingSpec( + tp_rank=tp_rank, + tp_world_size=tp_world_size, + vocab_shard_bounds=( + vocab_shard_bounds + if vocab_shard_bounds is not None + else _even_bounds(padded_vocab_size, tp_world_size) + ), + real_vocab_size=real_vocab_size, + padded_vocab_size=padded_vocab_size, + cp_rank=cp_rank, + cp_world_size=cp_world_size, + ) + + +def _mask( + *, + num_tokens: int = 8, + active_mask: tuple[bool, ...] | None = None, + ignore_index: int = -100, +) -> MaskSpec: + return MaskSpec( + num_tokens=num_tokens, + active_mask=( + active_mask + if active_mask is not None + else (False, False, True, True, True, True, True, False) + ), + ignore_index=ignore_index, + ) + + +def _contract( + *, + role: str = "train", + dtype: str = "bf16", + mask: MaskSpec | None = None, + sharding: ShardingSpec | None = None, + reduction: ReductionSpec | None = None, +) -> LogprobContract: + return LogprobContract( + role=role, + dtype=dtype, + mask=mask if mask is not None else _mask(), + sharding=sharding if sharding is not None else _sharding(), + reduction=reduction if reduction is not None else ReductionSpec(), + ) + + +def _declared_tp_backend() -> LogprobBackendCapability: + return LogprobBackendCapability( + backend_id="test-deterministic-tp-logprob", + roles=frozenset({LogprobRole.TRAIN, LogprobRole.INFER}), + dtypes=frozenset({LogprobDType.BF16}), + tp_world_sizes=(1, 2, 4), + cp_world_sizes=None, + supports_vocab_padding=True, + supports_inactive_tokens=True, + exports_vocab_lse=True, + deterministic_tp_merge=True, + implementation_kind="deterministic", + ) + + +def test_qwen3_tp2_bf16_contract_is_representable_and_serializable(): + contract = _contract() + + assert contract.sharding.tp_world_size == 2 + assert contract.sharding.cp_world_size == 2 + assert contract.sharding.local_vocab_start == 0 + assert contract.sharding.local_vocab_end == QWEN3_PADDED_VOCAB // 2 + assert contract.sharding.local_vocab_size == QWEN3_PADDED_VOCAB // 2 + assert contract.mask.active_token_count == 5 + assert contract.reduction.acc_dtype is LogprobDType.FP32 + assert contract.to_dict()["reduction"] == { + "merge": "max_sumexp", + "merge_axis": "tp_vocab", + "acc_dtype": "fp32", + "order": "global_vocab_shard_index", + "transport": "all_gather", + "downcast_at": "final_write", + "engine": "in_op_reference", + "cp_is_merge_axis": False, + } + json.dumps(contract.to_dict()) + + +@pytest.mark.parametrize("tp_world_size", [1, 2, 4]) +def test_pr4_sweep_tp_degrees_are_representable(tp_world_size): + sharding = _sharding(tp_world_size=tp_world_size, cp_world_size=1) + + assert len(sharding.vocab_shard_bounds) == tp_world_size + assert sharding.vocab_shard_bounds[-1][1] == QWEN3_PADDED_VOCAB + assert sharding.owner_rank(QWEN3_REAL_VOCAB - 1) == tp_world_size - 1 + + +@pytest.mark.parametrize( + ("field", "value", "message"), + [ + ("tp_rank", 2, "tp_rank=2"), + ("cp_rank", 2, "cp_rank=2"), + ("real_vocab_size", 0, "positive integer"), + ("padded_vocab_size", QWEN3_REAL_VOCAB - 1, "must not be smaller"), + ], +) +def test_invalid_rank_and_vocab_metadata_fail_loudly(field, value, message): + values = { + "tp_rank": 0, + "tp_world_size": 2, + "cp_rank": 0, + "cp_world_size": 2, + "real_vocab_size": QWEN3_REAL_VOCAB, + "padded_vocab_size": QWEN3_PADDED_VOCAB, + "vocab_shard_bounds": _even_bounds(QWEN3_PADDED_VOCAB, 2), + } + values[field] = value + + with pytest.raises(LogprobContractError, match=message): + ShardingSpec(**values) + + +@pytest.mark.parametrize( + ("bounds", "message"), + [ + ((), "one \\(start, end\\) pair per TP rank"), + (((0, 76032),), "one \\(start, end\\) pair per TP rank"), + (((0, 76032), (76032, 76032)), "end > start"), + (((0, 76000), (76032, 152064)), "contiguous"), + (((0, 76064), (76032, 152064)), "contiguous"), + (((0, 76032), (76032, 152000)), "cover padded_vocab_size exactly"), + ], +) +def test_incomplete_or_overlapping_vocab_shard_bounds_fail_loudly(bounds, message): + with pytest.raises(LogprobContractError, match=message): + _sharding(vocab_shard_bounds=bounds) + + +def test_owner_rank_is_unique_and_rejects_out_of_real_vocab_targets(): + sharding = _sharding() + + assert sharding.owner_rank(0) == 0 + assert sharding.owner_rank(QWEN3_PADDED_VOCAB // 2 - 1) == 0 + assert sharding.owner_rank(QWEN3_PADDED_VOCAB // 2) == 1 + assert sharding.owner_rank(QWEN3_REAL_VOCAB - 1) == 1 + + with pytest.raises(LogprobContractError, match="outside the real vocabulary"): + sharding.owner_rank(-1) + with pytest.raises(LogprobContractError, match="outside the real vocabulary"): + sharding.owner_rank(QWEN3_REAL_VOCAB) + + +def test_active_token_mask_metadata_is_validated(): + with pytest.raises(LogprobContractError, match="one entry per token"): + _mask(num_tokens=4) + + with pytest.raises(LogprobContractError, match="must be a bool"): + MaskSpec(num_tokens=2, active_mask=(True, 1)) + + all_inactive = _mask(num_tokens=3, active_mask=(False, False, False)) + assert all_inactive.active_token_count == 0 + + +def test_reduction_requires_fp32_accumulation_and_known_semantics(): + with pytest.raises(LogprobContractError, match="must be fp32"): + ReductionSpec(acc_dtype="bf16") + + with pytest.raises(LogprobContractError, match="merge must be one of"): + ReductionSpec(merge="lse_average") + + with pytest.raises(LogprobContractError, match="transport must be one of"): + ReductionSpec(transport="all_reduce") + + +def test_contract_component_types_and_lse_export_are_enforced(): + with pytest.raises(LogprobContractError, match="mask must be a MaskSpec"): + LogprobContract( + role="train", + dtype="bf16", + mask=None, + sharding=_sharding(), + reduction=ReductionSpec(), + ) + + with pytest.raises(LogprobContractError, match="export_lse must be True"): + replace(_contract(), export_lse=False) + + +def test_ignore_index_must_not_collide_with_the_real_vocabulary(): + with pytest.raises(LogprobContractError, match="must not collide"): + _contract(mask=_mask(ignore_index=5)) + + padding_column = QWEN3_REAL_VOCAB + 1 + contract = _contract(mask=_mask(ignore_index=padding_column)) + assert contract.mask.ignore_index == padding_column + + +def test_current_ws1_backend_rejects_strict_tp_contract_without_fallback(): + registry = KernelRegistry() + + with pytest.raises(RuntimeError) as exc_info: + registry.get_logprob_op(_contract()) + + message = str(exc_info.value) + assert "TP=2 is unsupported" in message + assert "vocab-domain LSE export is unsupported" in message + assert "deterministic TP (max, sumexp) merge is unsupported" in message + assert "padded-vs-real vocab masking is unsupported" in message + + +def test_current_ws1_backend_rejects_padded_vocab_even_at_tp1(): + registry = KernelRegistry() + contract = _contract(sharding=_sharding(tp_world_size=1, cp_world_size=1)) + + with pytest.raises(RuntimeError) as exc_info: + registry.get_logprob_op(contract) + + message = str(exc_info.value) + assert "TP=1 is unsupported" not in message + assert "padded-vs-real vocab masking is unsupported" in message + + +def test_undeclared_backend_capability_is_never_selected(): + registry = KernelRegistry() + platform = registry._platform() + registry._logprob_candidates[platform] = [OpBackend.PYTORCH_NATIVE] + + with pytest.raises(RuntimeError, match="no LogprobBackendCapability declared"): + registry.get_logprob_op(_contract()) + + +def test_declared_compatible_backend_resolves_and_records_provenance(): + registry = KernelRegistry() + platform = registry._platform() + registry._logprob_candidates[platform] = [OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] + registry._logprob_capabilities[OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] = _declared_tp_backend() + + result = registry.get_logprob_op(_contract(), requested_backend="deterministic") + + assert result.op is not None + assert result.capability.backend_id == "test-deterministic-tp-logprob" + assert result.provenance["requested_backend"] == "deterministic" + assert result.provenance["actual_backend"] == "test-deterministic-tp-logprob" + assert result.provenance["fallback"] is False + assert result.provenance["contract"]["sharding"]["tp_world_size"] == 2 + assert result.provenance["contract"]["sharding"]["real_vocab_size"] == QWEN3_REAL_VOCAB + assert result.provenance["contract"]["reduction"]["cp_is_merge_axis"] is False + json.dumps(result.provenance) + + +def test_requested_stable_backend_id_is_enforced(): + registry = KernelRegistry() + platform = registry._platform() + registry._logprob_candidates[platform] = [OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] + registry._logprob_capabilities[OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] = _declared_tp_backend() + + with pytest.raises(RuntimeError, match="does not match requested_backend=another-backend"): + registry.get_logprob_op(_contract(), requested_backend="another-backend") + + result = registry.get_logprob_op(_contract(), requested_backend="test-deterministic-tp-logprob") + assert result.provenance["actual_backend"] == "test-deterministic-tp-logprob" + + +def test_cp_is_a_non_merge_axis_and_cp_agnostic_backends_accept_any_cp_degree(): + capability = _declared_tp_backend() + cp2_contract = _contract(sharding=_sharding(cp_world_size=2, cp_rank=1)) + + assert capability.incompatibilities(cp2_contract) == () + + cp_restricted = replace(capability, cp_world_sizes=(1,)) + assert cp_restricted.incompatibilities(cp2_contract) == ("CP=2 is unsupported",) + + +def test_inactive_tokens_require_declared_backend_support(): + capability = replace(_declared_tp_backend(), supports_inactive_tokens=False) + contract = _contract() + + assert "inactive-token (ignore_index) masking is unsupported" in ( + capability.incompatibilities(contract) + ) + + fully_active = _contract(mask=_mask(num_tokens=3, active_mask=(True, True, True))) + assert capability.incompatibilities(fully_active) == () + + +def test_backend_id_must_not_shadow_a_reserved_policy_keyword(): + with pytest.raises(LogprobContractError, match="reserved dispatch policy keyword"): + replace(_declared_tp_backend(), backend_id="Deterministic") + + +def test_default_auto_policy_resolves_any_compatible_implementation_kind(): + registry = KernelRegistry() + platform = registry._platform() + registry._logprob_candidates[platform] = [OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] + registry._logprob_capabilities[OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] = replace( + _declared_tp_backend(), implementation_kind="reference" + ) + + result = registry.get_logprob_op(_contract()) + + assert result.provenance["requested_backend"] == "auto" + assert result.capability.implementation_kind == "reference" + + +def test_policy_keywords_are_case_insensitive_but_backend_ids_are_exact(): + registry = KernelRegistry() + platform = registry._platform() + registry._logprob_candidates[platform] = [OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] + registry._logprob_capabilities[OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] = _declared_tp_backend() + + result = registry.get_logprob_op(_contract(), requested_backend="DETERMINISTIC") + assert result.capability.backend_id == "test-deterministic-tp-logprob" + + with pytest.raises(RuntimeError, match="does not match requested_backend"): + registry.get_logprob_op(_contract(), requested_backend="Test-Deterministic-TP-Logprob") + + +def test_policy_only_skips_are_not_reported_as_fallback(): + registry = KernelRegistry() + platform = registry._platform() + registry._logprob_candidates[platform] = [ + OpBackend.TRITON_BATCH_INVARIANT_LOGP, + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, + ] + registry._logprob_capabilities[OpBackend.TRITON_BATCH_INVARIANT_LOGP] = replace( + _declared_tp_backend(), backend_id="other-compatible-backend" + ) + registry._logprob_capabilities[OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] = _declared_tp_backend() + + result = registry.get_logprob_op(_contract(), requested_backend="test-deterministic-tp-logprob") + + assert result.provenance["fallback"] is False + assert len(result.provenance["prior_rejections"]) == 1 + + +def test_capability_rejections_are_reported_as_fallback(): + registry = KernelRegistry() + platform = registry._platform() + registry._logprob_candidates[platform] = [ + OpBackend.TRITON_BATCH_INVARIANT_LOGP, + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, + ] + registry._logprob_capabilities[OpBackend.TRITON_BATCH_INVARIANT_LOGP] = replace( + _declared_tp_backend(), backend_id="tp1-only-backend", tp_world_sizes=(1,) + ) + registry._logprob_capabilities[OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] = _declared_tp_backend() + + result = registry.get_logprob_op(_contract()) + + assert result.provenance["fallback"] is True + assert "TP=2 is unsupported" in result.provenance["prior_rejections"][0] + + +def test_ws2_candidate_list_is_decoupled_from_the_legacy_priority_map(): + registry = KernelRegistry() + platform = registry._platform() + registry._logprob_candidates[platform].insert(0, OpBackend.PYTORCH_NATIVE) + + legacy = registry._priority_map[platform]["batch_invariant_logp"] + assert OpBackend.PYTORCH_NATIVE not in legacy From cdc11ba17621ef52e94f90fec8a477d0b9990075 Mon Sep 17 00:00:00 2001 From: ryankert01 Date: Sun, 2 Aug 2026 22:50:48 +0800 Subject: [PATCH 02/48] fix(ws2): address CodeRabbit review on logprob contract PR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - docs: correct the TP-invariance claim — fixed merge order gives determinism per TP degree; cross-degree bitwise equality additionally requires a TP-degree-independent local tile decomposition (PR 3 obligation), otherwise #108 tolerances apply - contract: store backend_id stripped so id-based dispatch matches; summarize the active mask in to_dict() provenance instead of copying every per-token boolean; sort __all__ per RUF022 - registry: add public register_logprob_backend() seam for PR 3 and tests; delegate _platform() to _platform_for_device(None); reuse _get_or_create_backend() in get_op so WS2 and legacy dispatch share one cache/blacklist code path - tests: use the registration seam instead of poking private state, pin _even_bounds' last bound for non-divisible vocabularies, assert candidate-list decoupling in both directions, cover registration replace semantics and backend_id normalization --- docs/design/ws2-tp-logprob-contract.md | 16 +++-- rl_engine/kernels/logprob_contract.py | 8 ++- rl_engine/kernels/registry.py | 54 ++++++++++------ tests/test_logprob_contract.py | 88 +++++++++++++++++++------- 4 files changed, 116 insertions(+), 50 deletions(-) diff --git a/docs/design/ws2-tp-logprob-contract.md b/docs/design/ws2-tp-logprob-contract.md index a392393b..3b1cf2fd 100644 --- a/docs/design/ws2-tp-logprob-contract.md +++ b/docs/design/ws2-tp-logprob-contract.md @@ -127,8 +127,15 @@ selected_logp = target_logit - LSE The selected target logit comes from a masked single-owner gather: exactly one rank holds each active token's target column. Downcast happens only at the final write. Because the -merge order is fixed by shard index, TP=2 is bitwise-equal to TP=1 by construction; averaging -per-rank logsumexp values or letting a collective reduce numerically is not conformant. +merge order is fixed by shard index, the result is deterministic and reproducible at every +TP degree by construction. Cross-degree bitwise equality (TP=2 equal to TP=1, the #241 PR 3 +acceptance target) requires one further condition: the local per-shard reduction must use a +TP-degree-independent tile decomposition, so that the same partial sums are formed in the +same order regardless of how the vocabulary is sharded. Providing that decomposition is an +obligation of the deterministic reference implementation; a backend without it is still +deterministic per degree, and its cross-degree drift is judged against the #108 tolerance +table instead. Averaging per-rank logsumexp values or letting a collective reduce +numerically is not conformant in either case. The acceptable LSE and selected-token drift thresholds remain owned by #108, and drift reports follow the #116 format. This contract does not introduce another tolerance table. @@ -170,8 +177,9 @@ The current WS1 batch-invariant logp implementations are single-shard (TP=1) ref they accept full-vocabulary logits with ignore-index masking but carry no vocab-shard metadata, no padded-vs-real vocab distinction, and no public vocab-domain LSE export. Strict WS2 requests therefore fail clearly today. The later deterministic vocab-parallel reference -becomes selectable by registering a capability that truthfully declares those features; no -controller branch or silent fallback is required. +becomes selectable through `KernelRegistry.register_logprob_backend(backend, capability)` +by declaring a capability that truthfully describes those features; no controller branch or +silent fallback is required. Successful dispatch provenance records: diff --git a/rl_engine/kernels/logprob_contract.py b/rl_engine/kernels/logprob_contract.py index 8f2667e1..815d232f 100644 --- a/rl_engine/kernels/logprob_contract.py +++ b/rl_engine/kernels/logprob_contract.py @@ -342,10 +342,11 @@ def to_dict(self) -> dict[str, Any]: "engine": self.reduction.engine.value, "cp_is_merge_axis": False, } + # The per-token mask is deliberately summarized: provenance exists for + # logging/serialization and the raw mask would dominate its size. mask = { "num_tokens": self.mask.num_tokens, "active_token_count": self.mask.active_token_count, - "active_mask": list(self.mask.active_mask), "ignore_index": self.mask.ignore_index, } return { @@ -382,6 +383,7 @@ def __post_init__(self) -> None: raise LogprobContractError( f"backend_id={self.backend_id!r} shadows a reserved dispatch policy keyword" ) + object.__setattr__(self, "backend_id", self.backend_id.strip()) roles = frozenset(_enum_value(LogprobRole, value, "roles") for value in self.roles) dtypes = frozenset(_enum_value(LogprobDType, value, "dtypes") for value in self.dtypes) if not roles or not dtypes: @@ -482,17 +484,17 @@ class LogprobDispatchResult: __all__ = [ + "RESERVED_DISPATCH_POLICIES", "DowncastPoint", "LogprobBackendCapability", "LogprobContract", "LogprobContractError", - "LogprobDispatchResult", "LogprobDType", + "LogprobDispatchResult", "LogprobMerge", "LogprobRole", "MaskSpec", "MergeAxis", - "RESERVED_DISPATCH_POLICIES", "ReductionEngine", "ReductionOrder", "ReductionSpec", diff --git a/rl_engine/kernels/registry.py b/rl_engine/kernels/registry.py index 35ae951b..d9b741bf 100644 --- a/rl_engine/kernels/registry.py +++ b/rl_engine/kernels/registry.py @@ -433,25 +433,41 @@ def get_op(self, op_type: str, device: torch.device | str | None = None) -> Any: candidates = self._priority_map.get(platform, {}).get(op_type, [OpBackend.PYTORCH_NATIVE]) for backend in candidates: - if backend.name in self._instance_cache: - return self._instance_cache[backend.name] + op_instance = self._get_or_create_backend(backend) + if op_instance is not None: + return op_instance - if backend.name in self._failed_backends: - continue + raise RuntimeError(f"No functional backend found for {op_type} on {platform}") - op_class = self._load_backend(backend) - if op_class: - try: - op_instance = op_class() - self._instance_cache[backend.name] = op_instance - return op_instance - except Exception as e: - logger.error(f"Failed to instantiate {backend.name}: {e}") - self._failed_backends.add(backend.name) - else: - self._failed_backends.add(backend.name) + def register_logprob_backend( + self, + backend: OpBackend, + capability: LogprobBackendCapability, + *, + platform: Optional[str] = None, + prepend: bool = False, + ) -> None: + """Register (or replace) a backend for WS2 contract-aware logprob dispatch. + + This is the supported seam for making a new backend selectable by + ``get_logprob_op`` (e.g. the deterministic vocab-parallel TP reference + from issue #241 PR 3) without touching the legacy ``get_op`` priority + lists. Registering the same backend again replaces its capability + without duplicating the candidate entry. + """ - raise RuntimeError(f"No functional backend found for {op_type} on {platform}") + if not isinstance(backend, OpBackend): + raise LogprobContractError("backend must be an OpBackend") + if not isinstance(capability, LogprobBackendCapability): + raise LogprobContractError("capability must be a LogprobBackendCapability") + resolved_platform = platform if platform is not None else self._platform() + candidates = self._logprob_candidates.setdefault(resolved_platform, []) + self._logprob_capabilities[backend] = capability + if backend not in candidates: + if prepend: + candidates.insert(0, backend) + else: + candidates.append(backend) def get_logprob_op( self, @@ -556,11 +572,7 @@ def _logprob_policy_mismatch( ) def _platform(self) -> str: - if device_ctx.is_rocm: - return "rocm" - if device_ctx.device_type == "cuda": - return "cuda" - return "cpu" + return self._platform_for_device(None) def _get_or_create_backend(self, backend: OpBackend) -> Any | None: if backend.name in self._instance_cache: diff --git a/tests/test_logprob_contract.py b/tests/test_logprob_contract.py index d18083f4..8a7a1ec5 100644 --- a/tests/test_logprob_contract.py +++ b/tests/test_logprob_contract.py @@ -28,7 +28,10 @@ def _even_bounds(padded_vocab: int, tp_world_size: int) -> tuple[tuple[int, int], ...]: shard = padded_vocab // tp_world_size - return tuple((rank * shard, (rank + 1) * shard) for rank in range(tp_world_size)) + return tuple( + (rank * shard, padded_vocab if rank == tp_world_size - 1 else (rank + 1) * shard) + for rank in range(tp_world_size) + ) def _sharding( @@ -274,8 +277,10 @@ def test_undeclared_backend_capability_is_never_selected(): def test_declared_compatible_backend_resolves_and_records_provenance(): registry = KernelRegistry() platform = registry._platform() - registry._logprob_candidates[platform] = [OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] - registry._logprob_capabilities[OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] = _declared_tp_backend() + registry._logprob_candidates[platform] = [] + registry.register_logprob_backend( + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, _declared_tp_backend(), platform=platform + ) result = registry.get_logprob_op(_contract(), requested_backend="deterministic") @@ -293,8 +298,10 @@ def test_declared_compatible_backend_resolves_and_records_provenance(): def test_requested_stable_backend_id_is_enforced(): registry = KernelRegistry() platform = registry._platform() - registry._logprob_candidates[platform] = [OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] - registry._logprob_capabilities[OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] = _declared_tp_backend() + registry._logprob_candidates[platform] = [] + registry.register_logprob_backend( + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, _declared_tp_backend(), platform=platform + ) with pytest.raises(RuntimeError, match="does not match requested_backend=another-backend"): registry.get_logprob_op(_contract(), requested_backend="another-backend") @@ -333,9 +340,11 @@ def test_backend_id_must_not_shadow_a_reserved_policy_keyword(): def test_default_auto_policy_resolves_any_compatible_implementation_kind(): registry = KernelRegistry() platform = registry._platform() - registry._logprob_candidates[platform] = [OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] - registry._logprob_capabilities[OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] = replace( - _declared_tp_backend(), implementation_kind="reference" + registry._logprob_candidates[platform] = [] + registry.register_logprob_backend( + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, + replace(_declared_tp_backend(), implementation_kind="reference"), + platform=platform, ) result = registry.get_logprob_op(_contract()) @@ -347,8 +356,10 @@ def test_default_auto_policy_resolves_any_compatible_implementation_kind(): def test_policy_keywords_are_case_insensitive_but_backend_ids_are_exact(): registry = KernelRegistry() platform = registry._platform() - registry._logprob_candidates[platform] = [OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] - registry._logprob_capabilities[OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] = _declared_tp_backend() + registry._logprob_candidates[platform] = [] + registry.register_logprob_backend( + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, _declared_tp_backend(), platform=platform + ) result = registry.get_logprob_op(_contract(), requested_backend="DETERMINISTIC") assert result.capability.backend_id == "test-deterministic-tp-logprob" @@ -360,14 +371,15 @@ def test_policy_keywords_are_case_insensitive_but_backend_ids_are_exact(): def test_policy_only_skips_are_not_reported_as_fallback(): registry = KernelRegistry() platform = registry._platform() - registry._logprob_candidates[platform] = [ + registry._logprob_candidates[platform] = [] + registry.register_logprob_backend( OpBackend.TRITON_BATCH_INVARIANT_LOGP, - OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, - ] - registry._logprob_capabilities[OpBackend.TRITON_BATCH_INVARIANT_LOGP] = replace( - _declared_tp_backend(), backend_id="other-compatible-backend" + replace(_declared_tp_backend(), backend_id="other-compatible-backend"), + platform=platform, + ) + registry.register_logprob_backend( + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, _declared_tp_backend(), platform=platform ) - registry._logprob_capabilities[OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] = _declared_tp_backend() result = registry.get_logprob_op(_contract(), requested_backend="test-deterministic-tp-logprob") @@ -378,14 +390,15 @@ def test_policy_only_skips_are_not_reported_as_fallback(): def test_capability_rejections_are_reported_as_fallback(): registry = KernelRegistry() platform = registry._platform() - registry._logprob_candidates[platform] = [ + registry._logprob_candidates[platform] = [] + registry.register_logprob_backend( OpBackend.TRITON_BATCH_INVARIANT_LOGP, - OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, - ] - registry._logprob_capabilities[OpBackend.TRITON_BATCH_INVARIANT_LOGP] = replace( - _declared_tp_backend(), backend_id="tp1-only-backend", tp_world_sizes=(1,) + replace(_declared_tp_backend(), backend_id="tp1-only-backend", tp_world_sizes=(1,)), + platform=platform, + ) + registry.register_logprob_backend( + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, _declared_tp_backend(), platform=platform ) - registry._logprob_capabilities[OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] = _declared_tp_backend() result = registry.get_logprob_op(_contract()) @@ -400,3 +413,34 @@ def test_ws2_candidate_list_is_decoupled_from_the_legacy_priority_map(): legacy = registry._priority_map[platform]["batch_invariant_logp"] assert OpBackend.PYTORCH_NATIVE not in legacy + + legacy.insert(0, OpBackend.PYTORCH_GEMM) + assert OpBackend.PYTORCH_GEMM not in registry._logprob_candidates[platform] + + +def test_register_logprob_backend_is_the_public_registration_seam(): + registry = KernelRegistry() + platform = registry._platform() + registry._logprob_candidates[platform] = [] + capability = _declared_tp_backend() + + registry.register_logprob_backend( + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, capability, platform=platform + ) + registry.register_logprob_backend( + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, + replace(capability, backend_id="replacement-backend"), + platform=platform, + ) + + assert registry._logprob_candidates[platform] == [OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] + result = registry.get_logprob_op(_contract()) + assert result.capability.backend_id == "replacement-backend" + + with pytest.raises(LogprobContractError, match="capability must be"): + registry.register_logprob_backend(OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, None) + + +def test_backend_id_whitespace_is_normalized_for_dispatch(): + capability = replace(_declared_tp_backend(), backend_id=" padded-id ") + assert capability.backend_id == "padded-id" From 6455715eb09cfd4189f53a3aceb0a0df342a4dd8 Mon Sep 17 00:00:00 2001 From: ryankert01 Date: Sun, 2 Aug 2026 23:06:28 +0800 Subject: [PATCH 03/48] fix(ws2): address second CodeRabbit round on logprob contract - docs: state that cross-TP bitwise equality needs a global tile-level merge structure independent of TP partitioning (per-shard tiles alone leave different grouping at shard boundaries), and that padded columns are masked to -inf before the local (max, sumexp) partials - registry: scope logprob capabilities per platform so the same backend enum can declare different support on cuda/rocm/cpu; validate the platform argument of register_logprob_backend against known platforms - contract: derive IMPLEMENTATION_KINDS from RESERVED_DISPATCH_POLICIES and use it for the kind check; wrap non-iterable roles/dtypes in LogprobContractError for consistent error handling - tests: cover per-platform capability scoping, unknown-platform rejection, and non-iterable roles/dtypes --- docs/design/ws2-tp-logprob-contract.md | 24 +++++++++------ rl_engine/kernels/logprob_contract.py | 15 ++++++--- rl_engine/kernels/registry.py | 22 ++++++++++++-- tests/test_logprob_contract.py | 42 ++++++++++++++++++++++++++ 4 files changed, 87 insertions(+), 16 deletions(-) diff --git a/docs/design/ws2-tp-logprob-contract.md b/docs/design/ws2-tp-logprob-contract.md index 3b1cf2fd..28685bb4 100644 --- a/docs/design/ws2-tp-logprob-contract.md +++ b/docs/design/ws2-tp-logprob-contract.md @@ -114,9 +114,11 @@ downcast_at: final_write engine: in_op_reference ``` -Every rank computes `m_l = max(local_logits)` and `s_l = sum(exp(local_logits - m_l))` in -fp32, the partials travel by all-gather (collectives are transport only, never a numerical -reduction), and every rank merges in fixed global vocab-shard index order: +Every rank first masks every local column whose global id lies in +`[real_vocab_size, padded_vocab_size)` to `-inf`, so padding never contributes to the +logsumexp, then computes `m_l = max(local_logits)` and `s_l = sum(exp(local_logits - m_l))` +in fp32. The partials travel by all-gather (collectives are transport only, never a +numerical reduction), and every rank merges in fixed global vocab-shard index order: ```text M = max_l(m_l) @@ -129,12 +131,16 @@ The selected target logit comes from a masked single-owner gather: exactly one r each active token's target column. Downcast happens only at the final write. Because the merge order is fixed by shard index, the result is deterministic and reproducible at every TP degree by construction. Cross-degree bitwise equality (TP=2 equal to TP=1, the #241 PR 3 -acceptance target) requires one further condition: the local per-shard reduction must use a -TP-degree-independent tile decomposition, so that the same partial sums are formed in the -same order regardless of how the vocabulary is sharded. Providing that decomposition is an -obligation of the deterministic reference implementation; a backend without it is still -deterministic per degree, and its cross-degree drift is judged against the #108 tolerance -table instead. Averaging per-rank logsumexp values or letting a collective reduce +acceptance target) requires one further condition: the entire reduction must follow a +global tile-level structure that is independent of TP partitioning — a fixed tile +decomposition of the vocabulary plus a fixed merge order and rescaling tree over those +tiles, identical at every TP degree, so that the TP degree only selects which rank computes +which tiles and never changes the floating-point grouping. A TP-degree-independent +decomposition inside each shard is not sufficient on its own, because shard boundaries +would still group the combines differently across degrees. Providing that global structure +is an obligation of the deterministic reference implementation; a backend without it is +still deterministic per degree, and its cross-degree drift is judged against the #108 +tolerance table instead. Averaging per-rank logsumexp values or letting a collective reduce numerically is not conformant in either case. The acceptable LSE and selected-token drift thresholds remain owned by #108, and drift diff --git a/rl_engine/kernels/logprob_contract.py b/rl_engine/kernels/logprob_contract.py index 815d232f..2cb22a25 100644 --- a/rl_engine/kernels/logprob_contract.py +++ b/rl_engine/kernels/logprob_contract.py @@ -30,6 +30,9 @@ # Dispatch policy keywords accepted by KernelRegistry.get_logprob_op; a stable # backend id must never shadow one of these, or it becomes unselectable by id. RESERVED_DISPATCH_POLICIES = frozenset({"auto", "production", "reference", "deterministic"}) +# Policies a backend can declare as its implementation kind; "auto" is a +# selection strategy, not an implementation kind. +IMPLEMENTATION_KINDS = RESERVED_DISPATCH_POLICIES - {"auto"} class LogprobContractError(ValueError): @@ -384,8 +387,11 @@ def __post_init__(self) -> None: f"backend_id={self.backend_id!r} shadows a reserved dispatch policy keyword" ) object.__setattr__(self, "backend_id", self.backend_id.strip()) - roles = frozenset(_enum_value(LogprobRole, value, "roles") for value in self.roles) - dtypes = frozenset(_enum_value(LogprobDType, value, "dtypes") for value in self.dtypes) + try: + roles = frozenset(_enum_value(LogprobRole, value, "roles") for value in self.roles) + dtypes = frozenset(_enum_value(LogprobDType, value, "dtypes") for value in self.dtypes) + except TypeError as exc: + raise LogprobContractError("roles and dtypes must be iterables of enum values") from exc if not roles or not dtypes: raise LogprobContractError("backend roles and dtypes must not be empty") tp_world_sizes = self._validated_world_sizes(self.tp_world_sizes, "tp_world_sizes") @@ -398,9 +404,9 @@ def __post_init__(self) -> None: ): if not isinstance(getattr(self, flag_name), bool): raise LogprobContractError(f"{flag_name} must be a bool") - if self.implementation_kind not in {"production", "reference", "deterministic"}: + if self.implementation_kind not in IMPLEMENTATION_KINDS: raise LogprobContractError( - "implementation_kind must be production, reference, or deterministic" + f"implementation_kind must be one of: {', '.join(sorted(IMPLEMENTATION_KINDS))}" ) object.__setattr__(self, "roles", roles) object.__setattr__(self, "dtypes", dtypes) @@ -484,6 +490,7 @@ class LogprobDispatchResult: __all__ = [ + "IMPLEMENTATION_KINDS", "RESERVED_DISPATCH_POLICIES", "DowncastPoint", "LogprobBackendCapability", diff --git a/rl_engine/kernels/registry.py b/rl_engine/kernels/registry.py index d9b741bf..0f4f2875 100644 --- a/rl_engine/kernels/registry.py +++ b/rl_engine/kernels/registry.py @@ -181,7 +181,7 @@ def __init__(self): # of silently selecting an incompatible fallback. common_logprob_roles = frozenset({LogprobRole.TRAIN, LogprobRole.INFER}) common_logprob_dtypes = frozenset({LogprobDType.BF16, LogprobDType.FP16, LogprobDType.FP32}) - self._logprob_capabilities = { + base_logprob_capabilities = { OpBackend.PYTORCH_BATCH_INVARIANT_LOGP: LogprobBackendCapability( backend_id="pytorch-batch-invariant-logp-ws1", roles=common_logprob_roles, @@ -343,6 +343,16 @@ def __init__(self): platform: list(ops.get("batch_invariant_logp", [])) for platform, ops in self._priority_map.items() } + # Capabilities are scoped per platform: the same backend enum may + # truthfully declare different support on cuda vs rocm vs cpu. + self._logprob_capabilities: Dict[str, Dict[OpBackend, LogprobBackendCapability]] = { + platform: { + backend: base_logprob_capabilities[backend] + for backend in candidates + if backend in base_logprob_capabilities + } + for platform, candidates in self._logprob_candidates.items() + } def _adjust_priority_from_env(self): rocm_attn_backend = os.getenv("RL_KERNEL_ROCM_ATTN_BACKEND", "").strip().lower() @@ -461,8 +471,13 @@ def register_logprob_backend( if not isinstance(capability, LogprobBackendCapability): raise LogprobContractError("capability must be a LogprobBackendCapability") resolved_platform = platform if platform is not None else self._platform() + if resolved_platform not in self._priority_map: + raise LogprobContractError( + f"unsupported platform {resolved_platform!r}; expected one of " + f"{sorted(self._priority_map)}" + ) candidates = self._logprob_candidates.setdefault(resolved_platform, []) - self._logprob_capabilities[backend] = capability + self._logprob_capabilities.setdefault(resolved_platform, {})[backend] = capability if backend not in candidates: if prepend: candidates.insert(0, backend) @@ -502,8 +517,9 @@ def get_logprob_op( # own requested_backend policy filter are not fallbacks. capability_rejections = 0 + platform_capabilities = self._logprob_capabilities.get(platform, {}) for backend in candidates: - capability = self._logprob_capabilities.get(backend) + capability = platform_capabilities.get(backend) if capability is None: rejected.append(f"{backend.name}: no LogprobBackendCapability declared") capability_rejections += 1 diff --git a/tests/test_logprob_contract.py b/tests/test_logprob_contract.py index 8a7a1ec5..670225f0 100644 --- a/tests/test_logprob_contract.py +++ b/tests/test_logprob_contract.py @@ -444,3 +444,45 @@ def test_register_logprob_backend_is_the_public_registration_seam(): def test_backend_id_whitespace_is_normalized_for_dispatch(): capability = replace(_declared_tp_backend(), backend_id=" padded-id ") assert capability.backend_id == "padded-id" + + +def test_capabilities_are_scoped_per_platform(): + registry = KernelRegistry() + platform = registry._platform() + other = "rocm" if platform != "rocm" else "cpu" + registry._logprob_candidates[platform] = [] + registry.register_logprob_backend( + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, _declared_tp_backend(), platform=platform + ) + registry.register_logprob_backend( + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, + replace(_declared_tp_backend(), backend_id="other-platform-backend"), + platform=other, + ) + + result = registry.get_logprob_op(_contract()) + + assert result.capability.backend_id == "test-deterministic-tp-logprob" + assert ( + registry._logprob_capabilities[other][OpBackend.PYTORCH_BATCH_INVARIANT_LOGP].backend_id + == "other-platform-backend" + ) + + +def test_register_logprob_backend_rejects_unknown_platform(): + registry = KernelRegistry() + + with pytest.raises(LogprobContractError, match="unsupported platform"): + registry.register_logprob_backend( + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, + _declared_tp_backend(), + platform="cuda-typo", + ) + + +def test_non_iterable_roles_and_dtypes_raise_contract_errors(): + with pytest.raises(LogprobContractError, match="roles and dtypes must be iterables"): + replace(_declared_tp_backend(), roles=None) + + with pytest.raises(LogprobContractError, match="roles and dtypes must be iterables"): + replace(_declared_tp_backend(), dtypes=42) From 3b4eaef3dd7ccb3645653e79166b03d0492355d1 Mon Sep 17 00:00:00 2001 From: ryankert01 Date: Sun, 2 Aug 2026 23:48:14 +0800 Subject: [PATCH 04/48] feat(ws2): make determinism scope and invocation surface part of the typed contract Address external review: the cross-TP bitwise guarantee lived only in prose, so a fixed-topology-deterministic backend could pass dispatch as fully conformant. - DeterminismScope (fixed_topology | cross_tp_bitwise): requested via ReductionSpec (default cross_tp_bitwise, the #241 PR 3 target), declared per backend via determinism_scopes, enforced by dispatch; replaces the deterministic_tp_merge bool - MaskMode (explicit_active_mask | ignore_index) replaces supports_inactive_tokens: the contract permits inactive targets that do not hold ignore_index, so ignore-index-only backends are rejected for contracts with inactive tokens - LogprobOutputSpec pins the output surface: fp32 selected logprob and fp32 vocab LSE, replicated across the TP group - implementation_kind is now a tier (reference | production); determinism is no longer conflated with it, and requesting "deterministic" as a policy raises a loud error pointing at determinism_scope - fallback provenance: policy evaluation now precedes capability checks, so a candidate excluded by the caller's own policy never counts as a fallback even when it also lacks capabilities - docs: define the (-inf, 0) identity partial for padding-only or all--inf shards; document that requested_backend="auto" is not distributed-safe and specify the preflight fingerprint agreement - LogprobContract.cross_rank_fingerprint(): rank-independent identity for that preflight; provenance now records active_mask_sha256 so masks with equal active counts remain distinguishable --- docs/design/ws2-tp-logprob-contract.md | 68 +++++++--- rl_engine/kernels/logprob_contract.py | 175 ++++++++++++++++++++++--- rl_engine/kernels/registry.py | 42 +++--- tests/test_logprob_contract.py | 116 ++++++++++++++-- 4 files changed, 343 insertions(+), 58 deletions(-) diff --git a/docs/design/ws2-tp-logprob-contract.md b/docs/design/ws2-tp-logprob-contract.md index 28685bb4..cf9179b8 100644 --- a/docs/design/ws2-tp-logprob-contract.md +++ b/docs/design/ws2-tp-logprob-contract.md @@ -36,12 +36,17 @@ for provenance and must never widen the merge. `rl_engine.kernels.logprob_contract` defines: -- `LogprobContract`: role, logits dtype, mask, sharding, reduction, and LSE export; +- `LogprobContract`: role, logits dtype, mask, sharding, reduction, output surface, and + LSE export, plus a rank-independent `cross_rank_fingerprint()`; - `ShardingSpec`: per-rank vocab-shard bounds, padded-vs-real vocabulary, TP/CP rank metadata, and target-token ownership; - `MaskSpec`: active-token mask and ignore index; -- `ReductionSpec`: fixed `(max, sumexp)` merge semantics; -- `LogprobBackendCapability`: the layouts and semantics a backend explicitly supports. +- `ReductionSpec`: fixed `(max, sumexp)` merge semantics and the requested determinism + scope; +- `LogprobOutputSpec`: the output surface — fp32 selected logprob and fp32 vocab-domain + LSE, replicated across the TP group; +- `LogprobBackendCapability`: the layouts and semantics a backend explicitly supports, + including its mask modes and determinism scopes. Construction performs validation immediately. A structurally valid contract means that the request is complete and internally consistent; it does not mean that an installed backend can @@ -140,8 +145,21 @@ decomposition inside each shard is not sufficient on its own, because shard boun would still group the combines differently across degrees. Providing that global structure is an obligation of the deterministic reference implementation; a backend without it is still deterministic per degree, and its cross-degree drift is judged against the #108 -tolerance table instead. Averaging per-rank logsumexp values or letting a collective reduce -numerically is not conformant in either case. +tolerance table instead. The contract expresses this distinction as +`ReductionSpec.determinism_scope`: `cross_tp_bitwise` (the #241 target and the default) +versus `fixed_topology`. A backend declares the scopes it honors in +`LogprobBackendCapability.determinism_scopes`, and dispatch rejects a backend that cannot +honor the requested scope — prose obligations are not enough; the guarantee is part of the +typed contract. + +A shard may lie entirely inside the padded region, and a row's local columns may all be +`-inf` after masking. The identity partial for these cases is defined as +`(m_l, s_l) = (-inf, 0)`: a partial with `s_l = 0` contributes nothing to the merge +regardless of its `m_l`, and implementations must use this identity directly rather than +evaluating `exp(-inf - (-inf))`, which would poison the merge with NaN. + +Averaging per-rank logsumexp values or letting a collective reduce numerically is never +conformant, at either determinism scope. The acceptable LSE and selected-token drift thresholds remain owned by #108, and drift reports follow the #116 format. This contract does not introduce another tolerance table. @@ -164,16 +182,20 @@ provenance = result.provenance ``` Dispatch considers only backends with a `LogprobBackendCapability`. It checks role, dtype, -TP/CP degree, padded-vs-real vocab masking, inactive-token support, vocab-domain LSE export, -and deterministic TP merge. An undeclared or incompatible backend is skipped with an -explicit rejection reason; there is no silent fallback. +TP/CP degree, padded-vs-real vocab masking, explicit active-mask support, vocab-domain LSE +export, and the requested determinism scope. An undeclared or incompatible backend is +skipped with an explicit rejection reason; there is no silent fallback. `requested_backend` accepts a case-insensitive policy keyword (`auto` | `production` | -`reference` | `deterministic`; default `auto`) or an exact, case-sensitive stable backend -id. Strictness comes from the contract's capability checks, not from the policy string. A -backend id may never shadow a policy keyword; capability construction rejects that. The -provenance `fallback` flag reports only capability or load rejections of otherwise-eligible -candidates — skips caused purely by the caller's own policy filter are not fallbacks. +`reference`; default `auto`) or an exact, case-sensitive stable backend id. The keywords +select an implementation tier; determinism is not a tier — it is requested through +`ReductionSpec.determinism_scope`, so `requested_backend="deterministic"` raises a loud +error instead of silently matching nothing. Strictness comes from the contract's +capability checks, not from the policy string. A backend id may never shadow a reserved +keyword; capability construction rejects that. The provenance `fallback` flag reports only +capability or load rejections of policy-eligible candidates — a candidate excluded by the +caller's own policy never counts as a fallback, even if it would also have failed +capability checks. WS2 dispatch resolves from its own candidate list, seeded from but decoupled from the legacy `batch_invariant_logp` priority list: registering a TP-vocab backend for WS2 dispatch does @@ -192,10 +214,26 @@ Successful dispatch provenance records: - requested and actual backend ids; - platform and fallback status; - prior candidate rejection reasons; -- the complete requested contract, including shard bounds, padded and real vocab sizes, - merge semantics, and the explicit `cp_is_merge_axis: false` declaration; +- the complete dispatch-relevant contract, including shard bounds, padded and real vocab + sizes, merge and output semantics, the explicit `cp_is_merge_axis: false` declaration, + and the active-mask digest (`active_mask_sha256`) — the mask's identity without its + per-token payload; - the selected backend capability descriptor. +### Distributed dispatch safety + +`get_logprob_op` resolves locally on each rank, so `requested_backend="auto"` is not +distributed-safe on its own: a load failure on one rank can resolve a different backend +than its peers, which for a collective-bearing implementation means divergent numerical +schedules or a deadlock. For `tp_world_size > 1` a caller must either request an exact +backend id or run a preflight agreement before any collective: all-gather the resolved +backend id together with `LogprobContract.cross_rank_fingerprint()` — a rank-independent +hash covering the shard-bounds table, vocab sizes, reduction/output semantics, and the +active-mask digest, excluding rank-local fields — and abort on any mismatch. Implementing +this preflight is an obligation of the #241 PR 3/PR 4 work; the backend invocation +protocol (how the contract and mask reach the implementation) is likewise defined there, +against this contract. + ## Validation Contract and dispatch behavior are covered by: diff --git a/rl_engine/kernels/logprob_contract.py b/rl_engine/kernels/logprob_contract.py index 2cb22a25..6941c78e 100644 --- a/rl_engine/kernels/logprob_contract.py +++ b/rl_engine/kernels/logprob_contract.py @@ -21,6 +21,8 @@ from __future__ import annotations +import hashlib +import json from dataclasses import dataclass, field from enum import Enum from typing import Any, TypeVar @@ -29,10 +31,15 @@ # Dispatch policy keywords accepted by KernelRegistry.get_logprob_op; a stable # backend id must never shadow one of these, or it becomes unselectable by id. +# "deterministic" stays reserved even though it is no longer a policy: +# determinism is expressed through DeterminismScope, and requesting it as a +# policy is a loud error rather than a silent id mismatch. RESERVED_DISPATCH_POLICIES = frozenset({"auto", "production", "reference", "deterministic"}) -# Policies a backend can declare as its implementation kind; "auto" is a -# selection strategy, not an implementation kind. -IMPLEMENTATION_KINDS = RESERVED_DISPATCH_POLICIES - {"auto"} +# Implementation tiers a backend can declare. Determinism is deliberately a +# separate axis (DeterminismScope): a backend can be a deterministic reference, +# a deterministic production implementation, or a non-deterministic production +# implementation. +IMPLEMENTATION_KINDS = frozenset({"production", "reference"}) class LogprobContractError(ValueError): @@ -85,6 +92,40 @@ class ReductionEngine(str, Enum): IN_OP_REFERENCE = "in_op_reference" +class DeterminismScope(str, Enum): + """Strength of the reduction's determinism guarantee. + + ``fixed_topology``: bitwise-reproducible for one fixed TP degree; results + at different TP degrees are compared against the #108 tolerance table. + + ``cross_tp_bitwise``: additionally bitwise-equal across TP degrees. This + requires the entire reduction to follow a global tile-level structure that + is independent of TP partitioning (see the design doc); fixed shard-order + merging alone is not sufficient. + """ + + FIXED_TOPOLOGY = "fixed_topology" + CROSS_TP_BITWISE = "cross_tp_bitwise" + + +class MaskMode(str, Enum): + """How a backend consumes inactive-token information. + + ``explicit_active_mask``: the backend honors an arbitrary active-token + mask. ``ignore_index``: the backend only recognizes inactive tokens whose + target id equals ``ignore_index``. The contract permits inactive targets + that do NOT hold ``ignore_index``, so an ignore-index-only backend cannot + serve a contract with inactive tokens. + """ + + EXPLICIT_ACTIVE_MASK = "explicit_active_mask" + IGNORE_INDEX = "ignore_index" + + +class TPPlacement(str, Enum): + REPLICATED = "replicated" + + def _enum_value(enum_type: type[_EnumT], value: Any, field: str) -> _EnumT: try: return enum_type(value) @@ -233,6 +274,7 @@ class MaskSpec: active_mask: tuple[bool, ...] ignore_index: int = -100 _active_token_count: int = field(init=False, repr=False, compare=False) + _active_mask_sha256: str = field(init=False, repr=False, compare=False) def __post_init__(self) -> None: num_tokens = _positive_int(self.num_tokens, "num_tokens") @@ -251,11 +293,19 @@ def __post_init__(self) -> None: ) object.__setattr__(self, "active_mask", active_mask) object.__setattr__(self, "_active_token_count", sum(active_mask)) + object.__setattr__( + self, "_active_mask_sha256", hashlib.sha256(bytes(active_mask)).hexdigest() + ) @property def active_token_count(self) -> int: return self._active_token_count + @property + def active_mask_sha256(self) -> str: + """Compact mask identity for provenance and cross-rank agreement.""" + return self._active_mask_sha256 + @dataclass(frozen=True) class ReductionSpec: @@ -268,8 +318,14 @@ class ReductionSpec: transport: ReductionTransport = ReductionTransport.ALL_GATHER downcast_at: DowncastPoint = DowncastPoint.FINAL_WRITE engine: ReductionEngine = ReductionEngine.IN_OP_REFERENCE + determinism_scope: DeterminismScope = DeterminismScope.CROSS_TP_BITWISE def __post_init__(self) -> None: + object.__setattr__( + self, + "determinism_scope", + _enum_value(DeterminismScope, self.determinism_scope, "determinism_scope"), + ) object.__setattr__(self, "merge", _enum_value(LogprobMerge, self.merge, "merge")) object.__setattr__( self, "merge_axis", _enum_value(MergeAxis, self.merge_axis, "merge_axis") @@ -291,6 +347,39 @@ def __post_init__(self) -> None: ) +@dataclass(frozen=True) +class LogprobOutputSpec: + """Output surface every conforming backend must produce. + + Selected logprob and vocab-domain LSE are fp32 and replicated across the + TP group; the ``downcast_at: final_write`` rule applies to any consumer + downcast after these outputs, never inside the reduction. + """ + + selected_logp_dtype: LogprobDType = LogprobDType.FP32 + lse_dtype: LogprobDType = LogprobDType.FP32 + tp_placement: TPPlacement = TPPlacement.REPLICATED + + def __post_init__(self) -> None: + object.__setattr__( + self, + "selected_logp_dtype", + _enum_value(LogprobDType, self.selected_logp_dtype, "selected_logp_dtype"), + ) + object.__setattr__( + self, "lse_dtype", _enum_value(LogprobDType, self.lse_dtype, "lse_dtype") + ) + object.__setattr__( + self, "tp_placement", _enum_value(TPPlacement, self.tp_placement, "tp_placement") + ) + if self.selected_logp_dtype is not LogprobDType.FP32: + raise LogprobContractError( + f"selected logprob output must be fp32; got {self.selected_logp_dtype.value}" + ) + if self.lse_dtype is not LogprobDType.FP32: + raise LogprobContractError(f"vocab LSE output must be fp32; got {self.lse_dtype.value}") + + @dataclass(frozen=True) class LogprobContract: """Complete semantic request consumed by contract-aware dispatch.""" @@ -300,6 +389,7 @@ class LogprobContract: mask: MaskSpec sharding: ShardingSpec reduction: ReductionSpec + output: LogprobOutputSpec = field(default_factory=LogprobOutputSpec) export_lse: bool = True def __post_init__(self) -> None: @@ -311,6 +401,8 @@ def __post_init__(self) -> None: raise LogprobContractError("sharding must be a ShardingSpec") if not isinstance(self.reduction, ReductionSpec): raise LogprobContractError("reduction must be a ReductionSpec") + if not isinstance(self.output, LogprobOutputSpec): + raise LogprobContractError("output must be a LogprobOutputSpec") if not isinstance(self.export_lse, bool) or not self.export_lse: raise LogprobContractError( "export_lse must be True for the WS2 vocab-domain LSE drift contract" @@ -343,15 +435,24 @@ def to_dict(self) -> dict[str, Any]: "transport": self.reduction.transport.value, "downcast_at": self.reduction.downcast_at.value, "engine": self.reduction.engine.value, + "determinism_scope": self.reduction.determinism_scope.value, "cp_is_merge_axis": False, } # The per-token mask is deliberately summarized: provenance exists for - # logging/serialization and the raw mask would dominate its size. + # logging/serialization and the raw mask would dominate its size. The + # digest keeps the mask *identity* observable, so two masks with the + # same active count still produce distinguishable provenance. mask = { "num_tokens": self.mask.num_tokens, "active_token_count": self.mask.active_token_count, + "active_mask_sha256": self.mask.active_mask_sha256, "ignore_index": self.mask.ignore_index, } + output = { + "selected_logp_dtype": self.output.selected_logp_dtype.value, + "lse_dtype": self.output.lse_dtype.value, + "tp_placement": self.output.tp_placement.value, + } return { "semantic_operator": "selected_token_logprob", "role": self.role.value, @@ -361,7 +462,28 @@ def to_dict(self) -> dict[str, Any]: "mask": mask, "sharding": sharding, "reduction": reduction, + "output": output, + } + + def cross_rank_fingerprint(self) -> str: + """Rank-independent identity for preflight agreement across ranks. + + Excludes ``tp_rank``/``cp_rank`` (and their derived local bounds) so + every rank of one logical invocation computes the same value. + All-gathering this fingerprint together with the resolved backend id + and aborting on mismatch is the documented preflight for distributed + dispatch; ``requested_backend="auto"`` is not distributed-safe + without it. + """ + + payload = self.to_dict() + payload["sharding"] = { + key: value + for key, value in payload["sharding"].items() + if key not in {"tp_rank", "cp_rank", "local_vocab_start", "local_vocab_end"} } + encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")) + return hashlib.sha256(encoded.encode("utf-8")).hexdigest() @dataclass(frozen=True) @@ -374,9 +496,9 @@ class LogprobBackendCapability: tp_world_sizes: tuple[int, ...] | None = None cp_world_sizes: tuple[int, ...] | None = None supports_vocab_padding: bool = False - supports_inactive_tokens: bool = False + mask_modes: frozenset[MaskMode] = frozenset() exports_vocab_lse: bool = False - deterministic_tp_merge: bool = False + determinism_scopes: frozenset[DeterminismScope] = frozenset() implementation_kind: str = "production" def __post_init__(self) -> None: @@ -396,12 +518,19 @@ def __post_init__(self) -> None: raise LogprobContractError("backend roles and dtypes must not be empty") tp_world_sizes = self._validated_world_sizes(self.tp_world_sizes, "tp_world_sizes") cp_world_sizes = self._validated_world_sizes(self.cp_world_sizes, "cp_world_sizes") - for flag_name in ( - "supports_vocab_padding", - "supports_inactive_tokens", - "exports_vocab_lse", - "deterministic_tp_merge", - ): + try: + mask_modes = frozenset( + _enum_value(MaskMode, value, "mask_modes") for value in self.mask_modes + ) + determinism_scopes = frozenset( + _enum_value(DeterminismScope, value, "determinism_scopes") + for value in self.determinism_scopes + ) + except TypeError as exc: + raise LogprobContractError( + "mask_modes and determinism_scopes must be iterables of enum values" + ) from exc + for flag_name in ("supports_vocab_padding", "exports_vocab_lse"): if not isinstance(getattr(self, flag_name), bool): raise LogprobContractError(f"{flag_name} must be a bool") if self.implementation_kind not in IMPLEMENTATION_KINDS: @@ -412,6 +541,8 @@ def __post_init__(self) -> None: object.__setattr__(self, "dtypes", dtypes) object.__setattr__(self, "tp_world_sizes", tp_world_sizes) object.__setattr__(self, "cp_world_sizes", cp_world_sizes) + object.__setattr__(self, "mask_modes", mask_modes) + object.__setattr__(self, "determinism_scopes", determinism_scopes) @staticmethod def _validated_world_sizes( @@ -453,13 +584,17 @@ def incompatibilities(self, contract: LogprobContract) -> tuple[str, ...]: reasons.append("padded-vs-real vocab masking is unsupported") if ( contract.mask.active_token_count != contract.mask.num_tokens - and not self.supports_inactive_tokens + and MaskMode.EXPLICIT_ACTIVE_MASK not in self.mask_modes ): - reasons.append("inactive-token (ignore_index) masking is unsupported") + # The contract does not require inactive targets to hold + # ignore_index, so ignore-index-only masking is insufficient. + reasons.append("explicit active-token masking is unsupported") if contract.export_lse and not self.exports_vocab_lse: reasons.append("vocab-domain LSE export is unsupported") - if tp_size > 1 and not self.deterministic_tp_merge: - reasons.append("deterministic TP (max, sumexp) merge is unsupported") + if contract.reduction.determinism_scope not in self.determinism_scopes: + reasons.append( + f"determinism_scope={contract.reduction.determinism_scope.value} is unsupported" + ) return tuple(reasons) def supports(self, contract: LogprobContract) -> bool: @@ -473,9 +608,9 @@ def to_dict(self) -> dict[str, Any]: "tp_world_sizes": list(self.tp_world_sizes) if self.tp_world_sizes else None, "cp_world_sizes": list(self.cp_world_sizes) if self.cp_world_sizes else None, "supports_vocab_padding": self.supports_vocab_padding, - "supports_inactive_tokens": self.supports_inactive_tokens, + "mask_modes": sorted(mode.value for mode in self.mask_modes), "exports_vocab_lse": self.exports_vocab_lse, - "deterministic_tp_merge": self.deterministic_tp_merge, + "determinism_scopes": sorted(scope.value for scope in self.determinism_scopes), "implementation_kind": self.implementation_kind, } @@ -492,6 +627,7 @@ class LogprobDispatchResult: __all__ = [ "IMPLEMENTATION_KINDS", "RESERVED_DISPATCH_POLICIES", + "DeterminismScope", "DowncastPoint", "LogprobBackendCapability", "LogprobContract", @@ -499,7 +635,9 @@ class LogprobDispatchResult: "LogprobDType", "LogprobDispatchResult", "LogprobMerge", + "LogprobOutputSpec", "LogprobRole", + "MaskMode", "MaskSpec", "MergeAxis", "ReductionEngine", @@ -507,4 +645,5 @@ class LogprobDispatchResult: "ReductionSpec", "ReductionTransport", "ShardingSpec", + "TPPlacement", ] diff --git a/rl_engine/kernels/registry.py b/rl_engine/kernels/registry.py index 0f4f2875..b17acf3e 100644 --- a/rl_engine/kernels/registry.py +++ b/rl_engine/kernels/registry.py @@ -9,12 +9,15 @@ import torch from rl_engine.kernels.logprob_contract import ( + IMPLEMENTATION_KINDS, + DeterminismScope, LogprobBackendCapability, LogprobContract, LogprobContractError, LogprobDispatchResult, LogprobDType, LogprobRole, + MaskMode, ) from rl_engine.platforms.device import device_ctx from rl_engine.utils.logger import logger @@ -188,9 +191,9 @@ def __init__(self): dtypes=common_logprob_dtypes, tp_world_sizes=(1,), supports_vocab_padding=False, - supports_inactive_tokens=True, + mask_modes=frozenset({MaskMode.IGNORE_INDEX}), exports_vocab_lse=False, - deterministic_tp_merge=False, + determinism_scopes=frozenset({DeterminismScope.FIXED_TOPOLOGY}), implementation_kind="reference", ), OpBackend.TRITON_BATCH_INVARIANT_LOGP: LogprobBackendCapability( @@ -199,10 +202,10 @@ def __init__(self): dtypes=common_logprob_dtypes, tp_world_sizes=(1,), supports_vocab_padding=False, - supports_inactive_tokens=True, + mask_modes=frozenset({MaskMode.IGNORE_INDEX}), exports_vocab_lse=False, - deterministic_tp_merge=False, - implementation_kind="deterministic", + determinism_scopes=frozenset({DeterminismScope.FIXED_TOPOLOGY}), + implementation_kind="production", ), OpBackend.CUDA_BATCH_INVARIANT_LOGP_SM90: LogprobBackendCapability( backend_id="cuda-batch-invariant-logp-sm90-ws1", @@ -210,10 +213,10 @@ def __init__(self): dtypes=frozenset({LogprobDType.BF16, LogprobDType.FP32}), tp_world_sizes=(1,), supports_vocab_padding=False, - supports_inactive_tokens=True, + mask_modes=frozenset({MaskMode.IGNORE_INDEX}), exports_vocab_lse=False, - deterministic_tp_merge=False, - implementation_kind="deterministic", + determinism_scopes=frozenset({DeterminismScope.FIXED_TOPOLOGY}), + implementation_kind="production", ), } @@ -508,6 +511,12 @@ def get_logprob_op( if not isinstance(requested_backend, str) or not requested_backend.strip(): raise LogprobContractError("requested_backend must be a non-empty string") requested_backend = requested_backend.strip() + if requested_backend.lower() == "deterministic": + raise LogprobContractError( + 'requested_backend="deterministic" is not a dispatch policy; request ' + "determinism through ReductionSpec.determinism_scope and match it against " + "backend determinism_scopes instead" + ) platform = self._platform() candidates = self._logprob_candidates.get(platform, []) @@ -524,13 +533,16 @@ def get_logprob_op( rejected.append(f"{backend.name}: no LogprobBackendCapability declared") capability_rejections += 1 continue - capability_incompat = list(capability.incompatibilities(contract)) policy_mismatch = self._logprob_policy_mismatch(requested_backend, capability) - reasons = capability_incompat + ([policy_mismatch] if policy_mismatch else []) - if reasons: - rejected.append(f"{backend.name}: " + "; ".join(reasons)) - if capability_incompat: - capability_rejections += 1 + if policy_mismatch is not None: + # Excluded by the caller's own policy: never a fallback, even + # if the candidate would also have failed capability checks. + rejected.append(f"{backend.name}: {policy_mismatch}") + continue + capability_incompat = list(capability.incompatibilities(contract)) + if capability_incompat: + rejected.append(f"{backend.name}: " + "; ".join(capability_incompat)) + capability_rejections += 1 continue op = self._get_or_create_backend(backend) @@ -573,7 +585,7 @@ def _logprob_policy_mismatch( policy = requested_backend.lower() if policy == "auto": return None - if policy in {"production", "reference", "deterministic"}: + if policy in IMPLEMENTATION_KINDS: if capability.implementation_kind == policy: return None return ( diff --git a/tests/test_logprob_contract.py b/tests/test_logprob_contract.py index 670225f0..40cea4e2 100644 --- a/tests/test_logprob_contract.py +++ b/tests/test_logprob_contract.py @@ -11,14 +11,18 @@ import pytest from rl_engine.kernels.logprob_contract import ( + DeterminismScope, LogprobBackendCapability, LogprobContract, LogprobContractError, LogprobDType, + LogprobOutputSpec, LogprobRole, + MaskMode, MaskSpec, ReductionSpec, ShardingSpec, + TPPlacement, ) from rl_engine.kernels.registry import KernelRegistry, OpBackend @@ -101,10 +105,12 @@ def _declared_tp_backend() -> LogprobBackendCapability: tp_world_sizes=(1, 2, 4), cp_world_sizes=None, supports_vocab_padding=True, - supports_inactive_tokens=True, + mask_modes=frozenset({MaskMode.EXPLICIT_ACTIVE_MASK, MaskMode.IGNORE_INDEX}), exports_vocab_lse=True, - deterministic_tp_merge=True, - implementation_kind="deterministic", + determinism_scopes=frozenset( + {DeterminismScope.CROSS_TP_BITWISE, DeterminismScope.FIXED_TOPOLOGY} + ), + implementation_kind="reference", ) @@ -126,6 +132,7 @@ def test_qwen3_tp2_bf16_contract_is_representable_and_serializable(): "transport": "all_gather", "downcast_at": "final_write", "engine": "in_op_reference", + "determinism_scope": "cross_tp_bitwise", "cp_is_merge_axis": False, } json.dumps(contract.to_dict()) @@ -249,7 +256,7 @@ def test_current_ws1_backend_rejects_strict_tp_contract_without_fallback(): message = str(exc_info.value) assert "TP=2 is unsupported" in message assert "vocab-domain LSE export is unsupported" in message - assert "deterministic TP (max, sumexp) merge is unsupported" in message + assert "determinism_scope=cross_tp_bitwise is unsupported" in message assert "padded-vs-real vocab masking is unsupported" in message @@ -282,11 +289,11 @@ def test_declared_compatible_backend_resolves_and_records_provenance(): OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, _declared_tp_backend(), platform=platform ) - result = registry.get_logprob_op(_contract(), requested_backend="deterministic") + result = registry.get_logprob_op(_contract(), requested_backend="reference") assert result.op is not None assert result.capability.backend_id == "test-deterministic-tp-logprob" - assert result.provenance["requested_backend"] == "deterministic" + assert result.provenance["requested_backend"] == "reference" assert result.provenance["actual_backend"] == "test-deterministic-tp-logprob" assert result.provenance["fallback"] is False assert result.provenance["contract"]["sharding"]["tp_world_size"] == 2 @@ -320,11 +327,11 @@ def test_cp_is_a_non_merge_axis_and_cp_agnostic_backends_accept_any_cp_degree(): assert cp_restricted.incompatibilities(cp2_contract) == ("CP=2 is unsupported",) -def test_inactive_tokens_require_declared_backend_support(): - capability = replace(_declared_tp_backend(), supports_inactive_tokens=False) +def test_inactive_tokens_require_explicit_active_mask_support(): + capability = replace(_declared_tp_backend(), mask_modes=frozenset({MaskMode.IGNORE_INDEX})) contract = _contract() - assert "inactive-token (ignore_index) masking is unsupported" in ( + assert "explicit active-token masking is unsupported" in ( capability.incompatibilities(contract) ) @@ -361,7 +368,7 @@ def test_policy_keywords_are_case_insensitive_but_backend_ids_are_exact(): OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, _declared_tp_backend(), platform=platform ) - result = registry.get_logprob_op(_contract(), requested_backend="DETERMINISTIC") + result = registry.get_logprob_op(_contract(), requested_backend="REFERENCE") assert result.capability.backend_id == "test-deterministic-tp-logprob" with pytest.raises(RuntimeError, match="does not match requested_backend"): @@ -486,3 +493,92 @@ def test_non_iterable_roles_and_dtypes_raise_contract_errors(): with pytest.raises(LogprobContractError, match="roles and dtypes must be iterables"): replace(_declared_tp_backend(), dtypes=42) + + +def test_requested_deterministic_policy_is_a_loud_error(): + registry = KernelRegistry() + + with pytest.raises(LogprobContractError, match="determinism_scope"): + registry.get_logprob_op(_contract(), requested_backend="deterministic") + + +def test_determinism_scope_is_part_of_the_typed_contract(): + fixed_only = replace( + _declared_tp_backend(), + determinism_scopes=frozenset({DeterminismScope.FIXED_TOPOLOGY}), + ) + + assert "determinism_scope=cross_tp_bitwise is unsupported" in ( + fixed_only.incompatibilities(_contract()) + ) + + relaxed = _contract(reduction=ReductionSpec(determinism_scope="fixed_topology")) + assert fixed_only.incompatibilities(relaxed) == () + + +def test_policy_filtered_candidates_never_count_toward_fallback(): + registry = KernelRegistry() + platform = registry._platform() + registry._logprob_candidates[platform] = [] + registry.register_logprob_backend( + OpBackend.TRITON_BATCH_INVARIANT_LOGP, + replace(_declared_tp_backend(), backend_id="tp1-only-backend", tp_world_sizes=(1,)), + platform=platform, + ) + registry.register_logprob_backend( + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, _declared_tp_backend(), platform=platform + ) + + result = registry.get_logprob_op(_contract(), requested_backend="test-deterministic-tp-logprob") + + assert result.provenance["fallback"] is False + assert len(result.provenance["prior_rejections"]) == 1 + + +def test_output_spec_is_pinned_to_fp32_replicated(): + with pytest.raises(LogprobContractError, match="must be fp32"): + LogprobOutputSpec(selected_logp_dtype="bf16") + with pytest.raises(LogprobContractError, match="must be fp32"): + LogprobOutputSpec(lse_dtype="bf16") + + assert LogprobOutputSpec().tp_placement is TPPlacement.REPLICATED + assert _contract().to_dict()["output"] == { + "selected_logp_dtype": "fp32", + "lse_dtype": "fp32", + "tp_placement": "replicated", + } + + +def test_cross_rank_fingerprint_is_rank_independent_and_content_sensitive(): + rank0 = _contract(sharding=_sharding(tp_rank=0)) + rank1 = _contract(sharding=_sharding(tp_rank=1, cp_rank=1)) + + assert rank0.cross_rank_fingerprint() == rank1.cross_rank_fingerprint() + + different_mask = _contract( + mask=_mask(active_mask=(True, True, True, True, True, True, True, False)) + ) + assert rank0.cross_rank_fingerprint() != different_mask.cross_rank_fingerprint() + + +def test_provenance_records_the_active_mask_digest(): + provenance_mask = _contract().to_dict()["mask"] + + assert provenance_mask["active_mask_sha256"] == _mask().active_mask_sha256 + assert len(provenance_mask["active_mask_sha256"]) == 64 + + same_count_different_mask = _mask( + active_mask=(True, True, True, True, True, False, False, False) + ) + assert same_count_different_mask.active_token_count == _mask().active_token_count + assert same_count_different_mask.active_mask_sha256 != _mask().active_mask_sha256 + + +def test_padding_only_shard_is_constructible_for_the_identity_partial(): + sharding = _sharding( + vocab_shard_bounds=((0, QWEN3_REAL_VOCAB), (QWEN3_REAL_VOCAB, QWEN3_PADDED_VOCAB)), + ) + + assert sharding.local_vocab_start == 0 + assert sharding.vocab_shard_bounds[1] == (QWEN3_REAL_VOCAB, QWEN3_PADDED_VOCAB) + assert sharding.owner_rank(QWEN3_REAL_VOCAB - 1) == 0 From e6dbeefaba51f23176ddd72e70f257b81f070e1a Mon Sep 17 00:00:00 2001 From: ryankert01 Date: Mon, 3 Aug 2026 06:09:14 +0800 Subject: [PATCH 05/48] docs(ws2): drop standalone design doc per review Fold the normative reduction semantics (padded-column masking, fp32 (max, sumexp) merge formulas, the (-inf, 0) identity partial, and the cross-TP tile-structure requirement) into the ReductionSpec and DeterminismScope docstrings, and repoint the runtime-dispatch and batch-invariant-logp doc references at the module. The contract summary moves to the PR description. --- docs/design/runtime-dispatch.md | 3 +- docs/design/ws2-tp-logprob-contract.md | 248 ------------------------- docs/operators/batch-invariant-logp.md | 4 +- rl_engine/kernels/logprob_contract.py | 34 +++- 4 files changed, 35 insertions(+), 254 deletions(-) delete mode 100644 docs/design/ws2-tp-logprob-contract.md diff --git a/docs/design/runtime-dispatch.md b/docs/design/runtime-dispatch.md index 84f29ec3..23c1586a 100644 --- a/docs/design/runtime-dispatch.md +++ b/docs/design/runtime-dispatch.md @@ -16,7 +16,8 @@ addition to platform priority, this path requires a backend capability descripto the requested role, dtype, TP/CP layout, padded-vs-real vocab masking, inactive-token support, vocab-domain LSE export, and deterministic TP merge semantics. Incompatible candidates produce explicit rejection reasons and are never used as an undeclared fallback. -See [WS2 TP-aware logprob contract](ws2-tp-logprob-contract.md). +The contract objects and their normative reduction semantics are documented in +`rl_engine.kernels.logprob_contract`. ## LogP Priority diff --git a/docs/design/ws2-tp-logprob-contract.md b/docs/design/ws2-tp-logprob-contract.md deleted file mode 100644 index cf9179b8..00000000 --- a/docs/design/ws2-tp-logprob-contract.md +++ /dev/null @@ -1,248 +0,0 @@ -# WS2 TP-Aware Logprob Contract - -Status: PR1 contract and dispatch metadata - -Tracking and shared contracts: - -- [#241: TP-aware deterministic logprob](https://github.com/RL-Align/RL-Kernel/issues/241) -- [#83: WS2 roadmap](https://github.com/RL-Align/RL-Kernel/issues/83) -- [#108: WS1 numerical contract](https://github.com/RL-Align/RL-Kernel/issues/108) -- [#111: WS2 cross-config alignment](https://github.com/RL-Align/RL-Kernel/issues/111) -- [#116: WS2 tolerance and drift-report format](https://github.com/RL-Align/RL-Kernel/issues/116) -- [Cross-config logprob drift contract](ws2_cross_config_logprob_drift_contract.md) - -## Scope - -This contract describes the logical inputs and deterministic reduction semantics for -selected-token log-probability under vocab-parallel tensor parallelism (TP): - -```text -selected_logp[t] = logits[t, target[t]] - logsumexp_vocab(logits[t, :]) -``` - -Under vocab-parallel TP each rank holds one vocabulary shard, so the vocabulary-wide -`logsumexp` requires a cross-rank reduction. This contract lets runtime dispatch reject a -backend whose numerical semantics do not match the requested layout. - -This PR1 layer does not shard tensors, launch a collective, merge `(max, sumexp)` partial -states, or implement a kernel. The single-GPU harness registration, the deterministic -vocab-parallel TP reference, and the cross-config integration belong to later PRs in #241. - -Context parallelism (CP) is a declared non-merge axis. CP partitions tokens, never the -vocabulary, so the logprob reduction spans TP vocab shards only. CP rank metadata is carried -for provenance and must never widen the merge. - -## Contract Objects - -`rl_engine.kernels.logprob_contract` defines: - -- `LogprobContract`: role, logits dtype, mask, sharding, reduction, output surface, and - LSE export, plus a rank-independent `cross_rank_fingerprint()`; -- `ShardingSpec`: per-rank vocab-shard bounds, padded-vs-real vocabulary, TP/CP rank - metadata, and target-token ownership; -- `MaskSpec`: active-token mask and ignore index; -- `ReductionSpec`: fixed `(max, sumexp)` merge semantics and the requested determinism - scope; -- `LogprobOutputSpec`: the output surface — fp32 selected logprob and fp32 vocab-domain - LSE, replicated across the TP group; -- `LogprobBackendCapability`: the layouts and semantics a backend explicitly supports, - including its mask modes and determinism scopes. - -Construction performs validation immediately. A structurally valid contract means that the -request is complete and internally consistent; it does not mean that an installed backend can -materialize it. - -`ShardingSpec.vocab_shard_bounds` lists every TP rank's half-open `[start, end)` vocab range -indexed by TP rank. The full table is required on every rank: it defines target ownership -and the fixed merge order without any collective, and it makes an incomplete or overlapping -partition a loud construction-time error instead of a silent runtime divergence. -`ShardingSpec.owner_rank(token_id)` resolves the unique owning rank for a real-vocab token -and rejects everything else. - -`padded_vocab_size` is the shard-covered (weight) vocabulary; `real_vocab_size` is the -tokenizer vocabulary. Padding columns occupy `[real_vocab_size, padded_vocab_size)` and must -be excluded from the logsumexp by any conforming implementation. The two sizes are equal -when the vocabulary is unpadded. - -Inactive tokens (prompt, padding, masked-out response positions) are excluded from every -drift aggregate and are exempt from the exactly-one-owner target gather; their targets may -legally hold `ignore_index`. `ignore_index` must not collide with the real vocabulary. - -## Qwen3-8B TP=2 BF16 Example - -```python -from rl_engine.kernels.logprob_contract import ( - LogprobContract, - MaskSpec, - ReductionSpec, - ShardingSpec, -) - -sharding = ShardingSpec( - tp_rank=0, - tp_world_size=2, - vocab_shard_bounds=((0, 76032), (76032, 152064)), - real_vocab_size=151936, - padded_vocab_size=152064, - cp_rank=0, - cp_world_size=2, -) - -contract = LogprobContract( - role="train", - dtype="bf16", - mask=MaskSpec( - num_tokens=8, - active_mask=(False, False, True, True, True, True, True, False), - ignore_index=-100, - ), - sharding=sharding, - reduction=ReductionSpec(), -) -``` - -Each rank owns one contiguous vocab shard; the 128 padding columns at the end of rank 1's -shard are outside the real vocabulary and never contribute to the logsumexp. The two leading -prompt tokens and the trailing padding token are inactive. - -## Reduction Semantics - -The only PR1 reduction contract is: - -```text -partial state: (local_max, local_sumexp), fp32 -merge: max_sumexp -merge_axis: tp_vocab -order: global_vocab_shard_index -transport: all_gather -downcast_at: final_write -engine: in_op_reference -``` - -Every rank first masks every local column whose global id lies in -`[real_vocab_size, padded_vocab_size)` to `-inf`, so padding never contributes to the -logsumexp, then computes `m_l = max(local_logits)` and `s_l = sum(exp(local_logits - m_l))` -in fp32. The partials travel by all-gather (collectives are transport only, never a -numerical reduction), and every rank merges in fixed global vocab-shard index order: - -```text -M = max_l(m_l) -S = sum_l(s_l * exp(m_l - M)) -LSE = M + log(S) -selected_logp = target_logit - LSE -``` - -The selected target logit comes from a masked single-owner gather: exactly one rank holds -each active token's target column. Downcast happens only at the final write. Because the -merge order is fixed by shard index, the result is deterministic and reproducible at every -TP degree by construction. Cross-degree bitwise equality (TP=2 equal to TP=1, the #241 PR 3 -acceptance target) requires one further condition: the entire reduction must follow a -global tile-level structure that is independent of TP partitioning — a fixed tile -decomposition of the vocabulary plus a fixed merge order and rescaling tree over those -tiles, identical at every TP degree, so that the TP degree only selects which rank computes -which tiles and never changes the floating-point grouping. A TP-degree-independent -decomposition inside each shard is not sufficient on its own, because shard boundaries -would still group the combines differently across degrees. Providing that global structure -is an obligation of the deterministic reference implementation; a backend without it is -still deterministic per degree, and its cross-degree drift is judged against the #108 -tolerance table instead. The contract expresses this distinction as -`ReductionSpec.determinism_scope`: `cross_tp_bitwise` (the #241 target and the default) -versus `fixed_topology`. A backend declares the scopes it honors in -`LogprobBackendCapability.determinism_scopes`, and dispatch rejects a backend that cannot -honor the requested scope — prose obligations are not enough; the guarantee is part of the -typed contract. - -A shard may lie entirely inside the padded region, and a row's local columns may all be -`-inf` after masking. The identity partial for these cases is defined as -`(m_l, s_l) = (-inf, 0)`: a partial with `s_l = 0` contributes nothing to the merge -regardless of its `m_l`, and implementations must use this identity directly rather than -evaluating `exp(-inf - (-inf))`, which would poison the merge with NaN. - -Averaging per-rank logsumexp values or letting a collective reduce numerically is never -conformant, at either determinism scope. - -The acceptable LSE and selected-token drift thresholds remain owned by #108, and drift -reports follow the #116 format. This contract does not introduce another tolerance table. -The selected-token metric remains the cross-config convention: - -```text -dlogp = training-side recomputed logp - rollout-side old logp -``` - -computed over active response tokens only. - -## Contract-Aware Dispatch - -Legacy callers continue to use `KernelRegistry.get_op()`. WS2 callers use: - -```python -result = kernel_registry.get_logprob_op(contract) -op = result.op -provenance = result.provenance -``` - -Dispatch considers only backends with a `LogprobBackendCapability`. It checks role, dtype, -TP/CP degree, padded-vs-real vocab masking, explicit active-mask support, vocab-domain LSE -export, and the requested determinism scope. An undeclared or incompatible backend is -skipped with an explicit rejection reason; there is no silent fallback. - -`requested_backend` accepts a case-insensitive policy keyword (`auto` | `production` | -`reference`; default `auto`) or an exact, case-sensitive stable backend id. The keywords -select an implementation tier; determinism is not a tier — it is requested through -`ReductionSpec.determinism_scope`, so `requested_backend="deterministic"` raises a loud -error instead of silently matching nothing. Strictness comes from the contract's -capability checks, not from the policy string. A backend id may never shadow a reserved -keyword; capability construction rejects that. The provenance `fallback` flag reports only -capability or load rejections of policy-eligible candidates — a candidate excluded by the -caller's own policy never counts as a fallback, even if it would also have failed -capability checks. - -WS2 dispatch resolves from its own candidate list, seeded from but decoupled from the legacy -`batch_invariant_logp` priority list: registering a TP-vocab backend for WS2 dispatch does -not change what legacy `get_op("batch_invariant_logp")` returns to WS1 callers. - -The current WS1 batch-invariant logp implementations are single-shard (TP=1) references: -they accept full-vocabulary logits with ignore-index masking but carry no vocab-shard -metadata, no padded-vs-real vocab distinction, and no public vocab-domain LSE export. Strict -WS2 requests therefore fail clearly today. The later deterministic vocab-parallel reference -becomes selectable through `KernelRegistry.register_logprob_backend(backend, capability)` -by declaring a capability that truthfully describes those features; no controller branch or -silent fallback is required. - -Successful dispatch provenance records: - -- requested and actual backend ids; -- platform and fallback status; -- prior candidate rejection reasons; -- the complete dispatch-relevant contract, including shard bounds, padded and real vocab - sizes, merge and output semantics, the explicit `cp_is_merge_axis: false` declaration, - and the active-mask digest (`active_mask_sha256`) — the mask's identity without its - per-token payload; -- the selected backend capability descriptor. - -### Distributed dispatch safety - -`get_logprob_op` resolves locally on each rank, so `requested_backend="auto"` is not -distributed-safe on its own: a load failure on one rank can resolve a different backend -than its peers, which for a collective-bearing implementation means divergent numerical -schedules or a deadlock. For `tp_world_size > 1` a caller must either request an exact -backend id or run a preflight agreement before any collective: all-gather the resolved -backend id together with `LogprobContract.cross_rank_fingerprint()` — a rank-independent -hash covering the shard-bounds table, vocab sizes, reduction/output semantics, and the -active-mask digest, excluding rank-local fields — and abort on any mismatch. Implementing -this preflight is an obligation of the #241 PR 3/PR 4 work; the backend invocation -protocol (how the contract and mask reach the implementation) is likewise defined there, -against this contract. - -## Validation - -Contract and dispatch behavior are covered by: - -```bash -python -m pytest tests/test_logprob_contract.py -q -``` - -The tests include Qwen3-8B TP=2 BF16 construction with padded vocab, the TP=1/2/4 sweep -shapes, incomplete/overlapping shard-bound rejection, owner-rank resolution, active-mask and -ignore-index validation, fp32-accumulation and merge-semantics enforcement, undeclared -backend rejection, no incompatible fallback, and JSON-compatible provenance. diff --git a/docs/operators/batch-invariant-logp.md b/docs/operators/batch-invariant-logp.md index b0671c40..4bca1f8c 100644 --- a/docs/operators/batch-invariant-logp.md +++ b/docs/operators/batch-invariant-logp.md @@ -64,8 +64,8 @@ remains unchanged. The backends above are single-shard (TP=1) references and do not yet export vocab-domain LSE or carry vocab-shard metadata, so they are declared incompatible with strict WS2 -requests instead of being selected as a silent fallback. See -[WS2 TP-aware logprob contract](../design/ws2-tp-logprob-contract.md). +requests instead of being selected as a silent fallback. The contract objects are +documented in `rl_engine.kernels.logprob_contract`. ## Benchmarks diff --git a/rl_engine/kernels/logprob_contract.py b/rl_engine/kernels/logprob_contract.py index 6941c78e..9f6d72ba 100644 --- a/rl_engine/kernels/logprob_contract.py +++ b/rl_engine/kernels/logprob_contract.py @@ -100,8 +100,12 @@ class DeterminismScope(str, Enum): ``cross_tp_bitwise``: additionally bitwise-equal across TP degrees. This requires the entire reduction to follow a global tile-level structure that - is independent of TP partitioning (see the design doc); fixed shard-order - merging alone is not sufficient. + is independent of TP partitioning: a fixed tile decomposition of the + vocabulary plus a fixed merge order and rescaling tree over those tiles, + identical at every TP degree, so the TP degree only selects which rank + computes which tiles and never changes the floating-point grouping. + Fixed shard-order merging alone is not sufficient, because shard + boundaries would still group the combines differently across degrees. """ FIXED_TOPOLOGY = "fixed_topology" @@ -309,7 +313,31 @@ def active_mask_sha256(self) -> str: @dataclass(frozen=True) class ReductionSpec: - """Deterministic TP-vocab ``(max, sumexp)`` merge semantics.""" + """Deterministic TP-vocab ``(max, sumexp)`` merge semantics. + + Every rank first masks local columns whose global id lies in + ``[real_vocab_size, padded_vocab_size)`` to ``-inf`` (padding never + contributes to the logsumexp), then computes ``m_l = max(local_logits)`` + and ``s_l = sum(exp(local_logits - m_l))`` in fp32. Partials travel by + all-gather -- collectives are transport only, never a numerical + reduction -- and every rank merges in fixed global vocab-shard index + order:: + + M = max_l(m_l) + S = sum_l(s_l * exp(m_l - M)) + LSE = M + log(S) + selected_logp = target_logit - LSE + + The selected target logit comes from a masked single-owner gather; + downcast happens only at the final write. The identity partial for a + padding-only shard, or a row whose local columns are all ``-inf`` after + masking, is ``(m_l, s_l) = (-inf, 0)``: a partial with ``s_l = 0`` + contributes nothing to the merge regardless of its ``m_l``, and + implementations must use this identity directly rather than evaluate + ``exp(-inf - (-inf))``, which would poison the merge with NaN. Averaging + per-rank logsumexp values, or letting a collective reduce numerically, is + never conformant at either determinism scope. + """ merge: LogprobMerge = LogprobMerge.MAX_SUMEXP merge_axis: MergeAxis = MergeAxis.TP_VOCAB From 878ba88a44e5f9ac24d1a38b4536a60c02721602 Mon Sep 17 00:00:00 2001 From: ryankert01 Date: Mon, 3 Aug 2026 06:21:30 +0800 Subject: [PATCH 06/48] style(ws2): align comment density with sibling kernel modules Shrink class docstrings toward the attention-contract one-liner style and cut design-rationale comments; the normative reduction semantics stay in the ReductionSpec and DeterminismScope docstrings. --- rl_engine/kernels/logprob_contract.py | 65 +++++++++------------------ rl_engine/kernels/registry.py | 19 +++----- 2 files changed, 26 insertions(+), 58 deletions(-) diff --git a/rl_engine/kernels/logprob_contract.py b/rl_engine/kernels/logprob_contract.py index 9f6d72ba..b0bcd7ab 100644 --- a/rl_engine/kernels/logprob_contract.py +++ b/rl_engine/kernels/logprob_contract.py @@ -29,16 +29,10 @@ _EnumT = TypeVar("_EnumT", bound=Enum) -# Dispatch policy keywords accepted by KernelRegistry.get_logprob_op; a stable -# backend id must never shadow one of these, or it becomes unselectable by id. -# "deterministic" stays reserved even though it is no longer a policy: -# determinism is expressed through DeterminismScope, and requesting it as a -# policy is a loud error rather than a silent id mismatch. +# Policy keywords accepted by KernelRegistry.get_logprob_op; a backend id must +# never shadow one of these, or it becomes unselectable by id. RESERVED_DISPATCH_POLICIES = frozenset({"auto", "production", "reference", "deterministic"}) -# Implementation tiers a backend can declare. Determinism is deliberately a -# separate axis (DeterminismScope): a backend can be a deterministic reference, -# a deterministic production implementation, or a non-deterministic production -# implementation. +# Backend tiers; determinism is a separate axis (DeterminismScope). IMPLEMENTATION_KINDS = frozenset({"production", "reference"}) @@ -58,12 +52,7 @@ class LogprobDType(str, Enum): class LogprobMerge(str, Enum): - """Merge primitive for per-shard partial states. - - Every rank contributes ``(local_max, local_sumexp)`` computed in the - accumulation dtype; the merged result is - ``M = max(m_l)``, ``S = sum(s_l * exp(m_l - M))``, ``LSE = M + log(S)``. - """ + """Merge primitive for per-shard ``(local_max, local_sumexp)`` partials.""" MAX_SUMEXP = "max_sumexp" @@ -115,11 +104,9 @@ class DeterminismScope(str, Enum): class MaskMode(str, Enum): """How a backend consumes inactive-token information. - ``explicit_active_mask``: the backend honors an arbitrary active-token - mask. ``ignore_index``: the backend only recognizes inactive tokens whose - target id equals ``ignore_index``. The contract permits inactive targets - that do NOT hold ``ignore_index``, so an ignore-index-only backend cannot - serve a contract with inactive tokens. + The contract permits inactive targets that do not hold ``ignore_index``, + so an ``ignore_index``-only backend cannot serve a contract with inactive + tokens. """ EXPLICIT_ACTIVE_MASK = "explicit_active_mask" @@ -161,15 +148,11 @@ class ShardingSpec: """Logical vocab-parallel TP ownership for one logprob invocation. ``vocab_shard_bounds`` lists every TP rank's half-open ``[start, end)`` - vocab range indexed by TP rank. The full table is required on every rank: - it defines target-token ownership and the fixed global-shard-index merge - order without any collective, and makes an incomplete partition a loud - construction-time error instead of a silent runtime divergence. - - ``padded_vocab_size`` is the shard-covered (weight) vocabulary; - ``real_vocab_size`` is the tokenizer vocabulary. Padding columns occupy - ``[real_vocab_size, padded_vocab_size)`` and must be excluded from the - logsumexp by any conforming implementation. + vocab range, indexed by rank; the full table is required on every rank and + must form a contiguous ``[0, padded_vocab_size)`` partition. + ``padded_vocab_size`` is the shard-covered (weight) vocabulary, + ``real_vocab_size`` the tokenizer vocabulary; padding columns occupy + ``[real_vocab_size, padded_vocab_size)``. """ tp_rank: int @@ -267,11 +250,10 @@ def owner_rank(self, token_id: int) -> int: @dataclass(frozen=True) class MaskSpec: - """Active-token ownership for one logprob invocation. + """Active-token mask and ignore index for one logprob invocation. - Inactive tokens are excluded from every drift aggregate and are exempt - from the exactly-one-owner target gather; their targets may legally hold - ``ignore_index``. + Inactive tokens are excluded from drift aggregates and from the + single-owner target gather; their targets may legally hold ``ignore_index``. """ num_tokens: int @@ -377,12 +359,8 @@ def __post_init__(self) -> None: @dataclass(frozen=True) class LogprobOutputSpec: - """Output surface every conforming backend must produce. - - Selected logprob and vocab-domain LSE are fp32 and replicated across the - TP group; the ``downcast_at: final_write`` rule applies to any consumer - downcast after these outputs, never inside the reduction. - """ + """Output surface every conforming backend must produce: fp32 selected + logprob and fp32 vocab-domain LSE, replicated across the TP group.""" selected_logp_dtype: LogprobDType = LogprobDType.FP32 lse_dtype: LogprobDType = LogprobDType.FP32 @@ -466,10 +444,8 @@ def to_dict(self) -> dict[str, Any]: "determinism_scope": self.reduction.determinism_scope.value, "cp_is_merge_axis": False, } - # The per-token mask is deliberately summarized: provenance exists for - # logging/serialization and the raw mask would dominate its size. The - # digest keeps the mask *identity* observable, so two masks with the - # same active count still produce distinguishable provenance. + # The digest stands in for the raw per-token mask, which would + # dominate the provenance size. mask = { "num_tokens": self.mask.num_tokens, "active_token_count": self.mask.active_token_count, @@ -614,8 +590,7 @@ def incompatibilities(self, contract: LogprobContract) -> tuple[str, ...]: contract.mask.active_token_count != contract.mask.num_tokens and MaskMode.EXPLICIT_ACTIVE_MASK not in self.mask_modes ): - # The contract does not require inactive targets to hold - # ignore_index, so ignore-index-only masking is insufficient. + # Inactive targets need not hold ignore_index (see MaskMode). reasons.append("explicit active-token masking is unsupported") if contract.export_lse and not self.exports_vocab_lse: reasons.append("vocab-domain LSE export is unsupported") diff --git a/rl_engine/kernels/registry.py b/rl_engine/kernels/registry.py index b17acf3e..7093b320 100644 --- a/rl_engine/kernels/registry.py +++ b/rl_engine/kernels/registry.py @@ -175,13 +175,9 @@ def __init__(self): self._instance_cache: Dict[str, Any] = {} self._failed_backends: Set[str] = set() - # These descriptors report what the existing WS1 batch-invariant logp - # implementations actually support: single-shard (TP=1) logits with - # ignore-index masking, no vocab-shard metadata, no padded-vs-real - # vocab distinction, and no public vocab-domain LSE export. A strict - # WS2 request is rejected with explicit reasons until the deterministic - # vocab-parallel TP reference backend lands (issue #241 PR 3) instead - # of silently selecting an incompatible fallback. + # Truthful descriptors for the existing WS1 batch-invariant logp + # implementations: single-shard (TP=1), ignore-index masking only, no + # vocab-shard metadata, no vocab-domain LSE export. common_logprob_roles = frozenset({LogprobRole.TRAIN, LogprobRole.INFER}) common_logprob_dtypes = frozenset({LogprobDType.BF16, LogprobDType.FP16, LogprobDType.FP32}) base_logprob_capabilities = { @@ -336,12 +332,9 @@ def __init__(self): self._adjust_priority_for_hardware() self._adjust_priority_from_env() - # WS2 contract-aware dispatch owns its candidate list. It is seeded - # from the legacy batch_invariant_logp priority (after hardware/env - # adjustments) but deliberately decoupled afterwards: registering a - # TP-vocab backend for WS2 dispatch must not change what legacy - # get_op("batch_invariant_logp") returns to WS1 callers, and vice - # versa. + # WS2 dispatch owns its candidate list, seeded from the legacy + # batch_invariant_logp priority but decoupled afterwards: neither + # path's registrations may affect the other. self._logprob_candidates: Dict[str, list] = { platform: list(ops.get("batch_invariant_logp", [])) for platform, ops in self._priority_map.items() From 934bc5be978b2291b374766228edf2c9ccc67079 Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Tue, 4 Aug 2026 17:04:05 +0800 Subject: [PATCH 07/48] feat: add single-gpu logprob comparison harness --- docs/design/ws2-logprob-single-gpu-harness.md | 94 +++++ .../ops/cuda/loss/batch_invariant_logp.py | 45 +++ .../ops/pytorch/loss/batch_invariant_logp.py | 28 +- .../ops/triton/loss/batch_invariant_logp.py | 88 ++++- rl_engine/testing/__init__.py | 20 + rl_engine/testing/logprob_comparison.py | 361 ++++++++++++++++++ scripts/compare_logprob.py | 96 +++++ tests/test_logprob_comparison.py | 214 +++++++++++ 8 files changed, 925 insertions(+), 21 deletions(-) create mode 100644 docs/design/ws2-logprob-single-gpu-harness.md create mode 100644 rl_engine/testing/logprob_comparison.py create mode 100644 scripts/compare_logprob.py create mode 100644 tests/test_logprob_comparison.py diff --git a/docs/design/ws2-logprob-single-gpu-harness.md b/docs/design/ws2-logprob-single-gpu-harness.md new file mode 100644 index 00000000..7440d465 --- /dev/null +++ b/docs/design/ws2-logprob-single-gpu-harness.md @@ -0,0 +1,94 @@ +# WS2 Single-GPU Logprob Comparison Harness + +This harness is the TP=1 registration and regression guard for issue #241. It compares +selected-token logprob implementations before any distributed communication is introduced. + +## Contract + +For each logical token row, every backend returns direct FP32 values: + +```text +LSE = logsumexp(logits[..., vocab]) +logp = selected_logit - LSE +``` + +The harness uses the merged WS1 batch-invariant PyTorch implementation as its reference. +Reference logp is obtained through the unchanged production call, while reference LSE is +obtained through the diagnostic entry point. The TP=1 PyTorch candidate follows the same +core computation and must be bitwise equal. This is a regression guard, not new +distributed mathematics. + +LSE drift is reported over every logical token row. Selected-token dlogp drift is reported +only over active response/action tokens. Both reports contain max, mean, p95, p99, and the +number of compared values. + +## Exact Backend Selection + +Supported backend names are: + +- `pytorch` +- `triton` +- `cuda-sm90` + +The comparison path does not use registry fallback. An explicitly requested backend must +run exactly or raise `LogprobBackendUnavailable`. In particular, `cuda-sm90` requires a +compiled SM90 extension, Hopper hardware, BF16/FP32 logits, and a compatible vocab row +stride. The production operator may retain its normal fallback behavior outside the +harness. + +Each backend exposes a diagnostic-only `forward_with_lse` method. Existing production +calls remain unchanged: + +```text +op(logits, target_ids) -> logp +op.forward_with_lse(logits, target_ids) -> (logp, lse) +``` + +## Usage + +CPU TP=1 regression guard: + +```bash +python scripts/compare_logprob.py \ + --candidate pytorch \ + --device cpu \ + --dtype fp32 \ + --batch 2 \ + --seq 16 \ + --vocab 257 +``` + +GPU comparison: + +```bash +python scripts/compare_logprob.py \ + --candidate triton \ + --candidate cuda-sm90 \ + --device cuda \ + --dtype bf16 \ + --batch 2 \ + --seq 16 \ + --vocab 151936 +``` + +The command prints a structured JSON report containing input dtype/shape, active-token +count, TP world size, communication mode, requested and actual backends, direct-LSE +provenance, bitwise logp status, and LSE/dlogp drift statistics. + +## Scope Boundary + +This harness is intentionally single-GPU and records `tp_world=1` and +`communication=none`. It does not implement vocab-shard metadata, all-gather transport, +fixed-order cross-rank LSE merging, CP reconstruction, or distributed artifacts. Those +belong to the later PR3 and PR4 work in issue #241. + +## Tests + +```bash +python -m pytest tests/test_logprob_comparison.py -q +``` + +The focused tests cover bitwise TP=1 regression, direct LSE identity, active-token-only +percentiles, zero active tokens, invalid ignore-index usage, structured serialization, +generic operator-harness registration, exact GPU backend diagnostics, and fail-closed +backend provenance. diff --git a/rl_engine/kernels/ops/cuda/loss/batch_invariant_logp.py b/rl_engine/kernels/ops/cuda/loss/batch_invariant_logp.py index a68781d9..aa62b6ef 100644 --- a/rl_engine/kernels/ops/cuda/loss/batch_invariant_logp.py +++ b/rl_engine/kernels/ops/cuda/loss/batch_invariant_logp.py @@ -161,3 +161,48 @@ def apply( ) return _BatchInvariantLogpSM90Function.apply(logits, target_ids, ignore_index) + + def forward_with_lse( + self, + logits: torch.Tensor, + target_ids: torch.Tensor, + ignore_index: int = -100, + *, + validate: bool = True, + ) -> tuple[torch.Tensor, torch.Tensor]: + """Run the exact SM90 path and return its direct FP32 logprob/LSE outputs. + + Unlike the production ``apply`` method, this diagnostic entry point never + falls back to Triton or PyTorch, so comparison provenance stays truthful. + """ + if logits.dim() < 2: + raise ValueError( + f"logits must be at least 2-D ([*lead, V]), got shape {tuple(logits.shape)}" + ) + if logits.shape[:-1] != target_ids.shape: + raise ValueError( + f"logits leading shape {tuple(logits.shape[:-1])} must match " + f"target_ids shape {tuple(target_ids.shape)}" + ) + if not _sm90_supported(logits): + raise RuntimeError( + "exact cuda-sm90 logprob diagnostics require Hopper, CUDA BF16/FP32 logits, " + "and a 16-byte-aligned vocab row stride; fallback is disabled" + ) + if validate: + vocab_size = logits.size(-1) + valid_targets = target_ids.reshape(-1) + valid_targets = valid_targets[valid_targets != ignore_index] + if valid_targets.numel() and ( + (valid_targets < 0).any() or (valid_targets >= vocab_size).any() + ): + bad = valid_targets[(valid_targets < 0) | (valid_targets >= vocab_size)] + raise ValueError( + f"target_ids contains values outside [0, {vocab_size}): {bad.tolist()}" + ) + + lead_shape = logits.shape[:-1] + logits_2d = logits.reshape(-1, logits.size(-1)).contiguous() + target_1d = target_ids.reshape(-1).to(device=logits.device, dtype=torch.int64).contiguous() + logp, lse = _C.batch_invariant_logp_sm90(logits_2d, target_1d, int(ignore_index)) + return logp.reshape(lead_shape), lse.reshape(lead_shape) diff --git a/rl_engine/kernels/ops/pytorch/loss/batch_invariant_logp.py b/rl_engine/kernels/ops/pytorch/loss/batch_invariant_logp.py index 4ac8bd37..80e08b5f 100644 --- a/rl_engine/kernels/ops/pytorch/loss/batch_invariant_logp.py +++ b/rl_engine/kernels/ops/pytorch/loss/batch_invariant_logp.py @@ -45,24 +45,42 @@ def apply( logits_2d = logits.reshape(-1, vocab_size).float() target_1d = target_ids.reshape(-1).to(logits.device, dtype=torch.long) - selected_logp = self._row_wise_selected_logprob( + selected_logp, _ = self._row_wise_selected_logprob_with_lse( logits_2d, target_1d, ignore_index=ignore_index, validate=validate ) return selected_logp.reshape(lead_shape) + def forward_with_lse( + self, + logits: torch.Tensor, + target_ids: torch.Tensor, + ignore_index: int = -100, + *, + validate: bool = True, + ) -> tuple[torch.Tensor, torch.Tensor]: + """Return selected logprob and the FP32 vocab-domain LSE for diagnostics.""" + self._validate_shapes(logits, target_ids) + lead_shape = logits.shape[:-1] + logits_2d = logits.reshape(-1, logits.size(-1)).float() + target_1d = target_ids.reshape(-1).to(logits.device, dtype=torch.long) + logp, lse = self._row_wise_selected_logprob_with_lse( + logits_2d, target_1d, ignore_index=ignore_index, validate=validate + ) + return logp.reshape(lead_shape), lse.reshape(lead_shape) + # ---------------------------------------------------------------------- # # Core Computation # ---------------------------------------------------------------------- # @staticmethod - def _row_wise_selected_logprob( + def _row_wise_selected_logprob_with_lse( logits_2d: torch.Tensor, target_1d: torch.Tensor, *, ignore_index: int, validate: bool = True, - ) -> torch.Tensor: - """Per-row selected logprob with locked reduction order. + ) -> tuple[torch.Tensor, torch.Tensor]: + """Per-row selected logprob and LSE with locked reduction order. The three reduction steps (max, sum-exp, gather) operate on each row independently. PyTorch's ``max(dim=-1)`` and ``sum(dim=-1)`` iterate @@ -104,7 +122,7 @@ def _row_wise_selected_logprob( selected_logp = selected_logp.where(valid_mask, torch.zeros_like(selected_logp)) - return selected_logp + return selected_logp, log_sum_exp # ---------------------------------------------------------------------- # # Helper diff --git a/rl_engine/kernels/ops/triton/loss/batch_invariant_logp.py b/rl_engine/kernels/ops/triton/loss/batch_invariant_logp.py index 66b99757..804341f3 100644 --- a/rl_engine/kernels/ops/triton/loss/batch_invariant_logp.py +++ b/rl_engine/kernels/ops/triton/loss/batch_invariant_logp.py @@ -10,6 +10,27 @@ _BLOCK_V: int = 1024 +def _launch_batch_invariant_logp( + logits_2d: torch.Tensor, target_1d: torch.Tensor, ignore_index: int +) -> tuple[torch.Tensor, torch.Tensor]: + num_tokens = logits_2d.shape[0] + vocab_size = logits_2d.shape[1] + output = torch.empty(num_tokens, device=logits_2d.device, dtype=torch.float32) + lse = torch.empty(num_tokens, device=logits_2d.device, dtype=torch.float32) + _batch_invariant_logp_kernel[(num_tokens,)]( + logits_2d, + target_1d, + output, + lse, + num_tokens, + vocab_size, + logits_2d.stride(0), + ignore_index=ignore_index, + BLOCK_V=_BLOCK_V, + ) + return output, lse + + @triton.jit def _batch_invariant_logp_kernel( logits_ptr, # logits [N, V] @@ -126,22 +147,7 @@ def forward(ctx, logits, target_ids, ignore_index): logits_2d = logits.reshape(-1, vocab_size).contiguous() target_1d = target_ids.reshape(-1).to(device=logits.device, dtype=torch.int64).contiguous() - num_tokens = logits_2d.shape[0] - output = torch.empty(num_tokens, device=logits.device, dtype=torch.float32) - lse = torch.empty(num_tokens, device=logits.device, dtype=torch.float32) - - grid = (num_tokens,) - _batch_invariant_logp_kernel[grid]( - logits_2d, - target_1d, - output, - lse, - num_tokens, - vocab_size, - logits_2d.stride(0), - ignore_index=ignore_index, - BLOCK_V=_BLOCK_V, - ) + output, lse = _launch_batch_invariant_logp(logits_2d, target_1d, ignore_index) ctx.save_for_backward(logits_2d, target_1d, lse) ctx.ignore_index = ignore_index @@ -237,3 +243,53 @@ def apply( ) return _BatchInvariantLogpFunction.apply(logits, target_ids, ignore_index) + + def forward_with_lse( + self, + logits: torch.Tensor, + target_ids: torch.Tensor, + ignore_index: int = -100, + *, + validate: bool = True, + ) -> tuple[torch.Tensor, torch.Tensor]: + """Return direct FP32 logprob/LSE outputs without an autograd wrapper.""" + self._validate_inputs(logits, target_ids, ignore_index=ignore_index, validate=validate) + lead_shape = logits.shape[:-1] + logits_2d = logits.reshape(-1, logits.size(-1)).contiguous() + target_1d = target_ids.reshape(-1).to(device=logits.device, dtype=torch.int64).contiguous() + logp, lse = _launch_batch_invariant_logp(logits_2d, target_1d, ignore_index) + return logp.reshape(lead_shape), lse.reshape(lead_shape) + + @staticmethod + def _validate_inputs( + logits: torch.Tensor, + target_ids: torch.Tensor, + *, + ignore_index: int, + validate: bool, + ) -> None: + if logits.device.type not in ("cuda", "xpu", "hip"): + raise RuntimeError( + "TritonBatchInvariantLogpOp requires a GPU tensor " + f"(CUDA / ROCm / XPU), got device '{logits.device}'." + ) + if logits.dim() < 2: + raise ValueError( + f"logits must be at least 2-D ([*lead, V]), got shape {tuple(logits.shape)}" + ) + if logits.shape[:-1] != target_ids.shape: + raise ValueError( + f"logits leading shape {tuple(logits.shape[:-1])} must match " + f"target_ids shape {tuple(target_ids.shape)}" + ) + if validate: + vocab_size = logits.size(-1) + valid_targets = target_ids.reshape(-1) + valid_targets = valid_targets[valid_targets != ignore_index] + if valid_targets.numel() and ( + (valid_targets < 0).any() or (valid_targets >= vocab_size).any() + ): + bad = valid_targets[(valid_targets < 0) | (valid_targets >= vocab_size)] + raise ValueError( + f"target_ids contains values outside [0, {vocab_size}): {bad.tolist()}" + ) diff --git a/rl_engine/testing/__init__.py b/rl_engine/testing/__init__.py index 42be8c1b..cc11e625 100644 --- a/rl_engine/testing/__init__.py +++ b/rl_engine/testing/__init__.py @@ -3,6 +3,17 @@ """Testing helpers for RL-shaped kernel validation.""" +from .logprob_comparison import ( + DriftStats, + LogprobBackendUnavailable, + LogprobCandidate, + LogprobComparisonInputs, + LogprobComparisonReport, + LogprobPathDrift, + LogprobPathResult, + compare_single_gpu_logprob, + make_logprob_candidate, +) from .reference_ops import ( active_token_count, compute_policy_ratio, @@ -15,10 +26,19 @@ from .rl_batch import SyntheticRLKernelBatch, make_synthetic_rl_kernel_batch __all__ = [ + "DriftStats", + "LogprobBackendUnavailable", + "LogprobCandidate", + "LogprobComparisonInputs", + "LogprobComparisonReport", + "LogprobPathDrift", + "LogprobPathResult", "SyntheticRLKernelBatch", "active_token_count", + "compare_single_gpu_logprob", "compute_policy_ratio", "compute_reference_kl", + "make_logprob_candidate", "make_synthetic_rl_kernel_batch", "masked_mean", "masked_sum", diff --git a/rl_engine/testing/logprob_comparison.py b/rl_engine/testing/logprob_comparison.py new file mode 100644 index 00000000..601dbced --- /dev/null +++ b/rl_engine/testing/logprob_comparison.py @@ -0,0 +1,361 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Single-GPU WS2 selected-logprob cross-implementation harness.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any, Callable, Sequence + +import torch + + +class LogprobBackendUnavailable(RuntimeError): + """Raised when an explicitly requested comparison backend cannot run exactly.""" + + +@dataclass(frozen=True) +class LogprobComparisonInputs: + """Logical TP=1 inputs shared by every comparison path.""" + + logits: torch.Tensor + target_ids: torch.Tensor + active_token_mask: torch.Tensor | None = None + ignore_index: int = -100 + + +@dataclass(frozen=True) +class LogprobPathResult: + """Direct selected-logprob and vocab-LSE outputs from one backend.""" + + name: str + logp: torch.Tensor + lse: torch.Tensor + provenance: dict[str, Any] + + +@dataclass(frozen=True) +class LogprobCandidate: + """One exact backend materialization used by the harness.""" + + name: str + requested_backend: str + actual_backend: str + fn: Callable[[torch.Tensor, torch.Tensor, int], tuple[torch.Tensor, torch.Tensor]] + provenance: dict[str, Any] = field(default_factory=dict) + + +@dataclass(frozen=True) +class DriftStats: + """Absolute drift statistics over a declared comparison population.""" + + max_abs: float + mean_abs: float + p95_abs: float + p99_abs: float + active_count: int + + def to_dict(self) -> dict[str, Any]: + return { + "max_abs": self.max_abs, + "mean_abs": self.mean_abs, + "p95_abs": self.p95_abs, + "p99_abs": self.p99_abs, + "active_count": self.active_count, + } + + +@dataclass(frozen=True) +class LogprobPathDrift: + """Candidate-vs-reference LSE and active-token dlogp drift.""" + + candidate_name: str + lse: DriftStats + dlogp: DriftStats + bitwise_logp: bool + provenance: dict[str, Any] + + def to_dict(self) -> dict[str, Any]: + return { + "candidate_name": self.candidate_name, + "lse": self.lse.to_dict(), + "dlogp": self.dlogp.to_dict(), + "bitwise_logp": self.bitwise_logp, + "provenance": self.provenance, + } + + +@dataclass(frozen=True) +class LogprobComparisonReport: + """Structured single-GPU report consumed by later WS2 integration.""" + + reference_name: str + drifts: tuple[LogprobPathDrift, ...] + input_provenance: dict[str, Any] + + def to_dict(self) -> dict[str, Any]: + return { + "reference_name": self.reference_name, + "drifts": [drift.to_dict() for drift in self.drifts], + "input_provenance": self.input_provenance, + } + + +def make_logprob_candidate(backend: str) -> LogprobCandidate: + """Materialize an exact built-in backend without registry fallback.""" + + normalized = backend.strip().lower().replace("_", "-") + if normalized in {"pytorch", "native"}: + from rl_engine.kernels.ops.pytorch.loss.batch_invariant_logp import ( + NativeBatchInvariantLogpOp, + ) + + op = NativeBatchInvariantLogpOp() + actual = "pytorch" + elif normalized == "triton": + try: + from rl_engine.kernels.ops.triton.loss.batch_invariant_logp import ( + TritonBatchInvariantLogpOp, + ) + + op = TritonBatchInvariantLogpOp() + except Exception as exc: + raise LogprobBackendUnavailable(f"triton backend is unavailable: {exc}") from exc + actual = "triton" + elif normalized in {"cuda-sm90", "sm90"}: + try: + from rl_engine.kernels.ops.cuda.loss.batch_invariant_logp import ( + BatchInvariantLogpSM90Op, + ) + + op = BatchInvariantLogpSM90Op() + except Exception as exc: + raise LogprobBackendUnavailable(f"cuda-sm90 backend is unavailable: {exc}") from exc + actual = "cuda-sm90" + else: + raise ValueError( + f"unsupported logprob comparison backend {backend!r}; " + "expected pytorch, triton, or cuda-sm90" + ) + + diagnostic = getattr(op, "forward_with_lse", None) + if not callable(diagnostic): + raise LogprobBackendUnavailable( + f"backend {normalized!r} does not expose the required direct LSE diagnostic" + ) + + def run( + logits: torch.Tensor, target_ids: torch.Tensor, ignore_index: int + ) -> tuple[torch.Tensor, torch.Tensor]: + try: + return diagnostic(logits, target_ids, ignore_index=ignore_index, validate=True) + except (RuntimeError, NotImplementedError, OSError) as exc: + raise LogprobBackendUnavailable( + f"exact backend {normalized!r} cannot execute this input: {exc}" + ) from exc + + return LogprobCandidate( + name=f"{actual}-batch-invariant-logp", + requested_backend=actual, + actual_backend=actual, + fn=run, + provenance={ + "requested_alias": normalized, + "implementation": f"{type(op).__module__}.{type(op).__qualname__}", + }, + ) + + +def compare_single_gpu_logprob( + inputs: LogprobComparisonInputs, + *, + candidates: Sequence[str | LogprobCandidate] = ("pytorch",), +) -> LogprobComparisonReport: + """Compare exact TP=1 implementations against the WS1 deterministic path.""" + + active_mask, effective_targets = _validate_inputs(inputs) + reference = _run_ws1_reference(inputs.logits, effective_targets, inputs.ignore_index) + + drifts = tuple( + _compare_path( + _run_candidate( + ( + candidate + if isinstance(candidate, LogprobCandidate) + else make_logprob_candidate(candidate) + ), + inputs.logits, + effective_targets, + inputs.ignore_index, + ), + reference, + active_mask, + ) + for candidate in candidates + ) + return LogprobComparisonReport( + reference_name=reference.name, + drifts=drifts, + input_provenance={ + "device": str(inputs.logits.device), + "input_dtype": str(inputs.logits.dtype), + "output_dtype": str(reference.logp.dtype), + "shape": list(inputs.logits.shape), + "ignore_index": inputs.ignore_index, + "active_token_count": int(active_mask.sum().item()), + "tp_world": 1, + "communication": "none", + }, + ) + + +def _run_ws1_reference( + logits: torch.Tensor, + target_ids: torch.Tensor, + ignore_index: int, +) -> LogprobPathResult: + """Run the existing deterministic logp path and its direct-LSE diagnostic.""" + from rl_engine.kernels.ops.pytorch.loss.batch_invariant_logp import ( + NativeBatchInvariantLogpOp, + ) + + op = NativeBatchInvariantLogpOp() + logp = op(logits, target_ids, ignore_index=ignore_index, validate=True) + _, lse = op.forward_with_lse( + logits, target_ids, ignore_index=ignore_index, validate=True + ) + return LogprobPathResult( + name="pytorch-batch-invariant-logp", + logp=logp.detach(), + lse=lse.detach(), + provenance={ + "requested_backend": "pytorch", + "actual_backend": "pytorch", + "tp_world": 1, + "communication": "none", + "logp_source": "production", + "lse_source": "direct", + }, + ) + + +def _run_candidate( + candidate: LogprobCandidate, + logits: torch.Tensor, + target_ids: torch.Tensor, + ignore_index: int, +) -> LogprobPathResult: + if candidate.requested_backend != candidate.actual_backend: + raise LogprobBackendUnavailable( + f"requested backend {candidate.requested_backend!r} materialized as " + f"{candidate.actual_backend!r}; silent fallback is forbidden" + ) + logp, lse = candidate.fn(logits, target_ids, ignore_index) + expected_shape = logits.shape[:-1] + for name, value in (("logp", logp), ("lse", lse)): + if not isinstance(value, torch.Tensor): + raise TypeError(f"candidate {candidate.name!r} {name} must be a tensor") + if value.shape != expected_shape: + raise ValueError( + f"candidate {candidate.name!r} {name} shape {tuple(value.shape)} " + f"does not match {tuple(expected_shape)}" + ) + if value.dtype != torch.float32: + raise ValueError(f"candidate {candidate.name!r} {name} must be FP32") + return LogprobPathResult( + name=candidate.name, + logp=logp.detach(), + lse=lse.detach(), + provenance={ + "requested_backend": candidate.requested_backend, + "actual_backend": candidate.actual_backend, + "tp_world": 1, + "communication": "none", + "lse_source": "direct", + **candidate.provenance, + }, + ) + + +def _compare_path( + candidate: LogprobPathResult, + reference: LogprobPathResult, + active_mask: torch.Tensor, +) -> LogprobPathDrift: + return LogprobPathDrift( + candidate_name=candidate.name, + lse=_drift_stats(candidate.lse, reference.lse), + dlogp=_drift_stats(candidate.logp, reference.logp, mask=active_mask), + bitwise_logp=torch.equal(candidate.logp, reference.logp), + provenance=candidate.provenance, + ) + + +def _drift_stats( + candidate: torch.Tensor, + reference: torch.Tensor, + *, + mask: torch.Tensor | None = None, +) -> DriftStats: + if candidate.shape != reference.shape: + raise ValueError( + f"candidate shape {tuple(candidate.shape)} must match reference shape " + f"{tuple(reference.shape)}" + ) + diff = (candidate.float() - reference.float()).abs() + values = diff.reshape(-1) if mask is None else diff[mask.to(device=diff.device)] + count = int(values.numel()) + if count == 0: + return DriftStats(0.0, 0.0, 0.0, 0.0, 0) + return DriftStats( + max_abs=float(values.max().item()), + mean_abs=float(values.mean().item()), + p95_abs=float(torch.quantile(values, 0.95).item()), + p99_abs=float(torch.quantile(values, 0.99).item()), + active_count=count, + ) + + +def _validate_inputs( + inputs: LogprobComparisonInputs, +) -> tuple[torch.Tensor, torch.Tensor]: + if inputs.logits.dim() < 2: + raise ValueError("logits must be at least 2-D [*lead, vocab]") + if inputs.logits.shape[:-1] != inputs.target_ids.shape: + raise ValueError("target_ids shape must match logits leading shape") + if not inputs.logits.is_floating_point(): + raise ValueError("logits must be floating point") + + if inputs.active_token_mask is None: + active = inputs.target_ids != inputs.ignore_index + else: + if inputs.active_token_mask.shape != inputs.target_ids.shape: + raise ValueError("active_token_mask shape must match target_ids") + if inputs.active_token_mask.dtype != torch.bool: + raise ValueError("active_token_mask must be bool") + active = inputs.active_token_mask.to(device=inputs.target_ids.device) + if bool(((inputs.target_ids == inputs.ignore_index) & active).any().item()): + raise ValueError("active target_ids cannot equal ignore_index") + + effective = inputs.target_ids.to(device=inputs.logits.device, dtype=torch.long).clone() + active = active.to(device=inputs.logits.device, dtype=torch.bool) + effective.masked_fill_(~active, inputs.ignore_index) + valid = effective[active] + vocab_size = inputs.logits.size(-1) + if valid.numel() and ((valid < 0).any() or (valid >= vocab_size).any()): + raise ValueError(f"active target_ids must be in [0, {vocab_size})") + return active, effective + + +__all__ = [ + "DriftStats", + "LogprobBackendUnavailable", + "LogprobCandidate", + "LogprobComparisonInputs", + "LogprobComparisonReport", + "LogprobPathDrift", + "LogprobPathResult", + "compare_single_gpu_logprob", + "make_logprob_candidate", +] diff --git a/scripts/compare_logprob.py b/scripts/compare_logprob.py new file mode 100644 index 00000000..a5feb570 --- /dev/null +++ b/scripts/compare_logprob.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +from __future__ import annotations + +import argparse +import json +import pathlib +import sys + +import torch + +REPO_ROOT = pathlib.Path(__file__).resolve().parents[1] +if str(REPO_ROOT) not in sys.path: + sys.path.insert(0, str(REPO_ROOT)) + +from rl_engine.testing import ( # noqa: E402 + LogprobComparisonInputs, + compare_single_gpu_logprob, +) + + +def _dtype(name: str) -> torch.dtype: + return { + "fp32": torch.float32, + "bf16": torch.bfloat16, + "fp16": torch.float16, + }[name] + + +def _device(name: str) -> torch.device: + if name == "auto": + return torch.device("cuda" if torch.cuda.is_available() else "cpu") + return torch.device(name) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Run the WS2 TP=1 selected-logprob/LSE comparison harness." + ) + parser.add_argument( + "--candidate", + action="append", + choices=("pytorch", "triton", "cuda-sm90"), + help="Exact backend to compare. Repeat for multiple backends; defaults to pytorch.", + ) + parser.add_argument("--device", default="auto") + parser.add_argument("--dtype", choices=("fp32", "bf16", "fp16"), default="fp32") + parser.add_argument("--batch", type=int, default=2) + parser.add_argument("--seq", type=int, default=16) + parser.add_argument("--vocab", type=int, default=257) + parser.add_argument("--prompt-tokens", type=int, default=8) + parser.add_argument("--seed", type=int, default=123) + return parser.parse_args() + + +def main() -> None: + args = parse_args() + device = _device(args.device) + if args.batch < 1 or args.seq < 1 or args.vocab < 1: + raise ValueError("batch, seq, and vocab must be positive") + if not 0 <= args.prompt_tokens <= args.seq: + raise ValueError("prompt-tokens must be in [0, seq]") + + generator = torch.Generator(device=device).manual_seed(args.seed) + logits = torch.randn( + args.batch, + args.seq, + args.vocab, + generator=generator, + device=device, + dtype=_dtype(args.dtype), + ) + target_ids = torch.randint( + 0, + args.vocab, + (args.batch, args.seq), + generator=generator, + device=device, + ) + active_mask = torch.ones((args.batch, args.seq), device=device, dtype=torch.bool) + active_mask[:, : args.prompt_tokens] = False + report = compare_single_gpu_logprob( + LogprobComparisonInputs( + logits=logits, + target_ids=target_ids, + active_token_mask=active_mask, + ), + candidates=tuple(args.candidate or ("pytorch",)), + ) + print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/tests/test_logprob_comparison.py b/tests/test_logprob_comparison.py new file mode 100644 index 00000000..f5051f5d --- /dev/null +++ b/tests/test_logprob_comparison.py @@ -0,0 +1,214 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +import argparse + +import pytest +import torch + +from rl_engine.kernels.gtest import run_operator_suite +from rl_engine.kernels.gtest.operator_specs import make_candidate, make_operator_case +from rl_engine.kernels.ops.pytorch.loss.batch_invariant_logp import ( + NativeBatchInvariantLogpOp, +) +from rl_engine.testing.logprob_comparison import ( + LogprobBackendUnavailable, + LogprobCandidate, + LogprobComparisonInputs, + compare_single_gpu_logprob, + make_logprob_candidate, +) +from scripts.compare_logprob import _device + + +def _inputs() -> LogprobComparisonInputs: + generator = torch.Generator().manual_seed(17) + logits = torch.randn(2, 4, 257, generator=generator, dtype=torch.float32) + target_ids = torch.tensor([[3, 5, 7, 11], [13, 17, 19, 23]]) + active = torch.tensor([[False, False, True, True], [False, True, True, True]]) + return LogprobComparisonInputs(logits, target_ids, active_token_mask=active) + + +def test_single_gpu_pytorch_path_is_bitwise_regression_guard(): + report = compare_single_gpu_logprob(_inputs(), candidates=("pytorch",)) + + assert report.reference_name == "pytorch-batch-invariant-logp" + assert len(report.drifts) == 1 + drift = report.drifts[0] + assert drift.bitwise_logp + assert drift.lse.max_abs == 0.0 + assert drift.dlogp.max_abs == 0.0 + assert drift.dlogp.active_count == 5 + assert drift.provenance["requested_backend"] == "pytorch" + assert drift.provenance["actual_backend"] == "pytorch" + assert drift.provenance["lse_source"] == "direct" + assert report.input_provenance["tp_world"] == 1 + assert report.input_provenance["communication"] == "none" + + +def test_report_serializes_lse_and_active_token_percentiles(): + inputs = _inputs() + reference = make_logprob_candidate("pytorch") + + def shifted(logits, target_ids, ignore_index): + logp, lse = reference.fn(logits, target_ids, ignore_index) + logp = logp.clone() + logp[0, 0] += 100.0 # inactive and therefore excluded from dlogp + logp[0, 2] += 1.0 + lse = lse + torch.arange(lse.numel(), dtype=lse.dtype).reshape_as(lse) * 0.1 + return logp, lse + + candidate = LogprobCandidate( + name="shifted", + requested_backend="shifted", + actual_backend="shifted", + fn=shifted, + ) + report = compare_single_gpu_logprob(inputs, candidates=(candidate,)) + payload = report.to_dict() + drift = payload["drifts"][0] + + assert drift["dlogp"]["active_count"] == 5 + assert drift["dlogp"]["max_abs"] == pytest.approx(1.0) + assert drift["dlogp"]["p95_abs"] == pytest.approx(0.8) + assert drift["dlogp"]["p99_abs"] == pytest.approx(0.96) + assert drift["lse"]["active_count"] == 8 + assert drift["lse"]["p99_abs"] == pytest.approx(0.693, abs=1e-5) + + +def test_all_inactive_tokens_produce_zero_dlogp_statistics(): + inputs = _inputs() + inputs = LogprobComparisonInputs( + inputs.logits, + inputs.target_ids, + active_token_mask=torch.zeros_like(inputs.target_ids, dtype=torch.bool), + ) + drift = compare_single_gpu_logprob(inputs).drifts[0] + + assert drift.dlogp.active_count == 0 + assert drift.dlogp.max_abs == 0.0 + assert drift.dlogp.p95_abs == 0.0 + assert drift.lse.active_count == inputs.target_ids.numel() + + +def test_explicit_backend_mismatch_fails_closed(): + native = make_logprob_candidate("pytorch") + disguised = LogprobCandidate( + name="fallback", + requested_backend="cuda-sm90", + actual_backend="pytorch", + fn=native.fn, + ) + + with pytest.raises(LogprobBackendUnavailable, match="silent fallback is forbidden"): + compare_single_gpu_logprob(_inputs(), candidates=(disguised,)) + + +def test_active_ignore_index_is_rejected(): + inputs = _inputs() + targets = inputs.target_ids.clone() + targets[0, 2] = -100 + + with pytest.raises(ValueError, match="active target_ids cannot equal ignore_index"): + compare_single_gpu_logprob( + LogprobComparisonInputs( + inputs.logits, + targets, + active_token_mask=inputs.active_token_mask, + ) + ) + + +def test_native_diagnostic_lse_satisfies_selected_logit_identity(): + inputs = _inputs() + candidate = make_logprob_candidate("pytorch") + effective = inputs.target_ids.masked_fill(~inputs.active_token_mask, -100) + logp, lse = candidate.fn(inputs.logits, effective, -100) + production_logp = NativeBatchInvariantLogpOp()( + inputs.logits, effective, ignore_index=-100, validate=True + ) + safe_targets = effective.masked_fill(~inputs.active_token_mask, 0) + selected = torch.gather(inputs.logits, -1, safe_targets.unsqueeze(-1)).squeeze(-1) + + assert torch.equal(logp, production_logp) + assert torch.equal(logp[inputs.active_token_mask], (selected - lse)[inputs.active_token_mask]) + + +def test_unsupported_backend_name_is_rejected(): + with pytest.raises(ValueError, match="unsupported logprob comparison backend"): + make_logprob_candidate("unknown") + + +def test_cli_auto_device_resolves_without_constructing_auto(monkeypatch): + monkeypatch.setattr(torch.cuda, "is_available", lambda: False) + + assert _device("auto") == torch.device("cpu") + + +def test_operator_comparison_specs_register_batch_invariant_logp(): + args = argparse.Namespace( + op="batch_invariant_logp", + candidate="pytorch", + arch_key=None, + batch=2, + seq=4, + vocab=17, + seed=7, + input_mode="random", + constant_value=0.5, + token_value=3, + normalized_dim=128, + k_dim=16, + n_dim=32, + theta=1.0e6, + eps=1.0e-6, + ) + + case = make_operator_case(args, torch.float32, torch.device("cpu")) + candidate = make_candidate(args) + report = run_operator_suite( + "batch_invariant_logp", candidates=[candidate], cases=[case] + ) + + assert report.passed + assert report.candidates[0].cases[0].op_class == "logprob" + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required") +def test_triton_diagnostic_path_reports_direct_lse(): + try: + candidate = make_logprob_candidate("triton") + except LogprobBackendUnavailable as exc: + pytest.skip(str(exc)) + logits = torch.randn(4, 1024, device="cuda", dtype=torch.bfloat16) + targets = torch.tensor([0, 17, 511, 1023], device="cuda") + try: + report = compare_single_gpu_logprob( + LogprobComparisonInputs(logits, targets), candidates=(candidate,) + ) + except LogprobBackendUnavailable as exc: + if isinstance(exc.__cause__, PermissionError): + pytest.skip(str(exc)) + raise + + assert report.drifts[0].provenance["actual_backend"] == "triton" + assert report.drifts[0].provenance["lse_source"] == "direct" + + +@pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability()[0] != 9, + reason="Hopper CUDA device required", +) +def test_sm90_diagnostic_path_reports_direct_lse_without_fallback(): + try: + candidate = make_logprob_candidate("cuda-sm90") + except LogprobBackendUnavailable as exc: + pytest.skip(str(exc)) + logits = torch.randn(4, 1024, device="cuda", dtype=torch.bfloat16) + targets = torch.tensor([0, 17, 511, 1023], device="cuda") + report = compare_single_gpu_logprob( + LogprobComparisonInputs(logits, targets), candidates=(candidate,) + ) + + assert report.drifts[0].provenance["actual_backend"] == "cuda-sm90" + assert report.drifts[0].provenance["lse_source"] == "direct" From b69426d28bcbdc002a33baed9522e18bde3311e9 Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Tue, 4 Aug 2026 20:22:12 +0800 Subject: [PATCH 08/48] fix: keep logprob CLI stdout machine readable --- docs/design/ws2-logprob-single-gpu-harness.md | 7 +++-- scripts/compare_logprob.py | 10 +++++++ tests/test_logprob_comparison.py | 30 ++++++++++++++++++- 3 files changed, 43 insertions(+), 4 deletions(-) diff --git a/docs/design/ws2-logprob-single-gpu-harness.md b/docs/design/ws2-logprob-single-gpu-harness.md index 7440d465..c5b869b0 100644 --- a/docs/design/ws2-logprob-single-gpu-harness.md +++ b/docs/design/ws2-logprob-single-gpu-harness.md @@ -71,9 +71,10 @@ python scripts/compare_logprob.py \ --vocab 151936 ``` -The command prints a structured JSON report containing input dtype/shape, active-token -count, TP world size, communication mode, requested and actual backends, direct-LSE -provenance, bitwise logp status, and LSE/dlogp drift statistics. +The command prints a structured JSON report to stdout containing input dtype/shape, +active-token count, TP world size, communication mode, requested and actual backends, +direct-LSE provenance, bitwise logp status, and LSE/dlogp drift statistics. Backend +diagnostic logs are routed to stderr so redirected stdout remains valid JSON. ## Scope Boundary diff --git a/scripts/compare_logprob.py b/scripts/compare_logprob.py index a5feb570..6b42d4ea 100644 --- a/scripts/compare_logprob.py +++ b/scripts/compare_logprob.py @@ -6,6 +6,7 @@ import argparse import json +import logging import pathlib import sys @@ -19,6 +20,7 @@ LogprobComparisonInputs, compare_single_gpu_logprob, ) +from rl_engine.utils.logger import logger # noqa: E402 def _dtype(name: str) -> torch.dtype: @@ -35,6 +37,13 @@ def _device(name: str) -> torch.device: return torch.device(name) +def _route_rl_kernel_logs_to_stderr() -> None: + """Keep stdout machine-readable while preserving backend diagnostics.""" + for handler in logger.handlers: + if isinstance(handler, logging.StreamHandler): + handler.setStream(sys.stderr) + + def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser( description="Run the WS2 TP=1 selected-logprob/LSE comparison harness." @@ -56,6 +65,7 @@ def parse_args() -> argparse.Namespace: def main() -> None: + _route_rl_kernel_logs_to_stderr() args = parse_args() device = _device(args.device) if args.batch < 1 or args.seq < 1 or args.vocab < 1: diff --git a/tests/test_logprob_comparison.py b/tests/test_logprob_comparison.py index f5051f5d..67c1d422 100644 --- a/tests/test_logprob_comparison.py +++ b/tests/test_logprob_comparison.py @@ -2,6 +2,10 @@ # Copyright (c) 2026 RL-Kernel Contributors import argparse +import io +import json +import logging +import sys import pytest import torch @@ -18,7 +22,8 @@ compare_single_gpu_logprob, make_logprob_candidate, ) -from scripts.compare_logprob import _device +from rl_engine.utils.logger import logger +from scripts.compare_logprob import _device, _route_rl_kernel_logs_to_stderr def _inputs() -> LogprobComparisonInputs: @@ -145,6 +150,29 @@ def test_cli_auto_device_resolves_without_constructing_auto(monkeypatch): assert _device("auto") == torch.device("cpu") +def test_cli_routes_rl_kernel_logs_to_stderr_for_machine_readable_stdout(monkeypatch): + stdout = io.StringIO() + stderr = io.StringIO() + original_streams = [ + (handler, handler.stream) + for handler in logger.handlers + if isinstance(handler, logging.StreamHandler) + ] + monkeypatch.setattr(sys, "stdout", stdout) + monkeypatch.setattr(sys, "stderr", stderr) + + try: + _route_rl_kernel_logs_to_stderr() + logger.info("test backend diagnostic") + print(json.dumps({"ok": True})) + finally: + for handler, stream in original_streams: + handler.setStream(stream) + + assert json.loads(stdout.getvalue()) == {"ok": True} + assert "test backend diagnostic" in stderr.getvalue() + + def test_operator_comparison_specs_register_batch_invariant_logp(): args = argparse.Namespace( op="batch_invariant_logp", From 0efcfe12883482247d7544af50fbb1c0adf2a049 Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Tue, 4 Aug 2026 20:39:54 +0800 Subject: [PATCH 09/48] refactor: simplify logprob comparison harness --- rl_engine/testing/__init__.py | 6 - rl_engine/testing/logprob_comparison.py | 177 +++++++----------------- 2 files changed, 53 insertions(+), 130 deletions(-) diff --git a/rl_engine/testing/__init__.py b/rl_engine/testing/__init__.py index cc11e625..51759abd 100644 --- a/rl_engine/testing/__init__.py +++ b/rl_engine/testing/__init__.py @@ -4,13 +4,10 @@ """Testing helpers for RL-shaped kernel validation.""" from .logprob_comparison import ( - DriftStats, LogprobBackendUnavailable, LogprobCandidate, LogprobComparisonInputs, LogprobComparisonReport, - LogprobPathDrift, - LogprobPathResult, compare_single_gpu_logprob, make_logprob_candidate, ) @@ -26,13 +23,10 @@ from .rl_batch import SyntheticRLKernelBatch, make_synthetic_rl_kernel_batch __all__ = [ - "DriftStats", "LogprobBackendUnavailable", "LogprobCandidate", "LogprobComparisonInputs", "LogprobComparisonReport", - "LogprobPathDrift", - "LogprobPathResult", "SyntheticRLKernelBatch", "active_token_count", "compare_single_gpu_logprob", diff --git a/rl_engine/testing/logprob_comparison.py b/rl_engine/testing/logprob_comparison.py index 601dbced..ad2ef76c 100644 --- a/rl_engine/testing/logprob_comparison.py +++ b/rl_engine/testing/logprob_comparison.py @@ -1,44 +1,31 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 RL-Kernel Contributors -"""Single-GPU WS2 selected-logprob cross-implementation harness.""" +"""Single-GPU selected-logprob comparison.""" from __future__ import annotations -from dataclasses import dataclass, field -from typing import Any, Callable, Sequence +from collections.abc import Callable, Sequence +from dataclasses import asdict, dataclass, field +from typing import Any import torch class LogprobBackendUnavailable(RuntimeError): - """Raised when an explicitly requested comparison backend cannot run exactly.""" + pass @dataclass(frozen=True) class LogprobComparisonInputs: - """Logical TP=1 inputs shared by every comparison path.""" - logits: torch.Tensor target_ids: torch.Tensor active_token_mask: torch.Tensor | None = None ignore_index: int = -100 -@dataclass(frozen=True) -class LogprobPathResult: - """Direct selected-logprob and vocab-LSE outputs from one backend.""" - - name: str - logp: torch.Tensor - lse: torch.Tensor - provenance: dict[str, Any] - - @dataclass(frozen=True) class LogprobCandidate: - """One exact backend materialization used by the harness.""" - name: str requested_backend: str actual_backend: str @@ -47,64 +34,34 @@ class LogprobCandidate: @dataclass(frozen=True) -class DriftStats: - """Absolute drift statistics over a declared comparison population.""" - +class _DriftStats: max_abs: float mean_abs: float p95_abs: float p99_abs: float active_count: int - def to_dict(self) -> dict[str, Any]: - return { - "max_abs": self.max_abs, - "mean_abs": self.mean_abs, - "p95_abs": self.p95_abs, - "p99_abs": self.p99_abs, - "active_count": self.active_count, - } - @dataclass(frozen=True) -class LogprobPathDrift: - """Candidate-vs-reference LSE and active-token dlogp drift.""" - +class _LogprobPathDrift: candidate_name: str - lse: DriftStats - dlogp: DriftStats + lse: _DriftStats + dlogp: _DriftStats bitwise_logp: bool provenance: dict[str, Any] - def to_dict(self) -> dict[str, Any]: - return { - "candidate_name": self.candidate_name, - "lse": self.lse.to_dict(), - "dlogp": self.dlogp.to_dict(), - "bitwise_logp": self.bitwise_logp, - "provenance": self.provenance, - } - @dataclass(frozen=True) class LogprobComparisonReport: - """Structured single-GPU report consumed by later WS2 integration.""" - reference_name: str - drifts: tuple[LogprobPathDrift, ...] + drifts: tuple[_LogprobPathDrift, ...] input_provenance: dict[str, Any] def to_dict(self) -> dict[str, Any]: - return { - "reference_name": self.reference_name, - "drifts": [drift.to_dict() for drift in self.drifts], - "input_provenance": self.input_provenance, - } + return asdict(self) def make_logprob_candidate(backend: str) -> LogprobCandidate: - """Materialize an exact built-in backend without registry fallback.""" - normalized = backend.strip().lower().replace("_", "-") if normalized in {"pytorch", "native"}: from rl_engine.kernels.ops.pytorch.loss.batch_invariant_logp import ( @@ -172,35 +129,38 @@ def compare_single_gpu_logprob( *, candidates: Sequence[str | LogprobCandidate] = ("pytorch",), ) -> LogprobComparisonReport: - """Compare exact TP=1 implementations against the WS1 deterministic path.""" - active_mask, effective_targets = _validate_inputs(inputs) - reference = _run_ws1_reference(inputs.logits, effective_targets, inputs.ignore_index) - - drifts = tuple( - _compare_path( - _run_candidate( - ( - candidate - if isinstance(candidate, LogprobCandidate) - else make_logprob_candidate(candidate) - ), - inputs.logits, - effective_targets, - inputs.ignore_index, - ), - reference, - active_mask, - ) - for candidate in candidates + reference_logp, reference_lse = _run_ws1_reference( + inputs.logits, effective_targets, inputs.ignore_index ) + + drifts = [] + for candidate in candidates: + if isinstance(candidate, str): + candidate = make_logprob_candidate(candidate) + logp, lse = _run_candidate( + candidate, + inputs.logits, + effective_targets, + inputs.ignore_index, + ) + drifts.append( + _LogprobPathDrift( + candidate_name=candidate.name, + lse=_drift_stats(lse, reference_lse), + dlogp=_drift_stats(logp, reference_logp, mask=active_mask), + bitwise_logp=torch.equal(logp, reference_logp), + provenance=_candidate_provenance(candidate), + ) + ) + return LogprobComparisonReport( - reference_name=reference.name, - drifts=drifts, + reference_name="pytorch-batch-invariant-logp", + drifts=tuple(drifts), input_provenance={ "device": str(inputs.logits.device), "input_dtype": str(inputs.logits.dtype), - "output_dtype": str(reference.logp.dtype), + "output_dtype": str(reference_logp.dtype), "shape": list(inputs.logits.shape), "ignore_index": inputs.ignore_index, "active_token_count": int(active_mask.sum().item()), @@ -214,8 +174,7 @@ def _run_ws1_reference( logits: torch.Tensor, target_ids: torch.Tensor, ignore_index: int, -) -> LogprobPathResult: - """Run the existing deterministic logp path and its direct-LSE diagnostic.""" +) -> tuple[torch.Tensor, torch.Tensor]: from rl_engine.kernels.ops.pytorch.loss.batch_invariant_logp import ( NativeBatchInvariantLogpOp, ) @@ -225,19 +184,7 @@ def _run_ws1_reference( _, lse = op.forward_with_lse( logits, target_ids, ignore_index=ignore_index, validate=True ) - return LogprobPathResult( - name="pytorch-batch-invariant-logp", - logp=logp.detach(), - lse=lse.detach(), - provenance={ - "requested_backend": "pytorch", - "actual_backend": "pytorch", - "tp_world": 1, - "communication": "none", - "logp_source": "production", - "lse_source": "direct", - }, - ) + return logp.detach(), lse.detach() def _run_candidate( @@ -245,7 +192,7 @@ def _run_candidate( logits: torch.Tensor, target_ids: torch.Tensor, ignore_index: int, -) -> LogprobPathResult: +) -> tuple[torch.Tensor, torch.Tensor]: if candidate.requested_backend != candidate.actual_backend: raise LogprobBackendUnavailable( f"requested backend {candidate.requested_backend!r} materialized as " @@ -263,33 +210,18 @@ def _run_candidate( ) if value.dtype != torch.float32: raise ValueError(f"candidate {candidate.name!r} {name} must be FP32") - return LogprobPathResult( - name=candidate.name, - logp=logp.detach(), - lse=lse.detach(), - provenance={ - "requested_backend": candidate.requested_backend, - "actual_backend": candidate.actual_backend, - "tp_world": 1, - "communication": "none", - "lse_source": "direct", - **candidate.provenance, - }, - ) + return logp.detach(), lse.detach() -def _compare_path( - candidate: LogprobPathResult, - reference: LogprobPathResult, - active_mask: torch.Tensor, -) -> LogprobPathDrift: - return LogprobPathDrift( - candidate_name=candidate.name, - lse=_drift_stats(candidate.lse, reference.lse), - dlogp=_drift_stats(candidate.logp, reference.logp, mask=active_mask), - bitwise_logp=torch.equal(candidate.logp, reference.logp), - provenance=candidate.provenance, - ) +def _candidate_provenance(candidate: LogprobCandidate) -> dict[str, Any]: + return { + "requested_backend": candidate.requested_backend, + "actual_backend": candidate.actual_backend, + "tp_world": 1, + "communication": "none", + "lse_source": "direct", + **candidate.provenance, + } def _drift_stats( @@ -297,7 +229,7 @@ def _drift_stats( reference: torch.Tensor, *, mask: torch.Tensor | None = None, -) -> DriftStats: +) -> _DriftStats: if candidate.shape != reference.shape: raise ValueError( f"candidate shape {tuple(candidate.shape)} must match reference shape " @@ -307,8 +239,8 @@ def _drift_stats( values = diff.reshape(-1) if mask is None else diff[mask.to(device=diff.device)] count = int(values.numel()) if count == 0: - return DriftStats(0.0, 0.0, 0.0, 0.0, 0) - return DriftStats( + return _DriftStats(0.0, 0.0, 0.0, 0.0, 0) + return _DriftStats( max_abs=float(values.max().item()), mean_abs=float(values.mean().item()), p95_abs=float(torch.quantile(values, 0.95).item()), @@ -349,13 +281,10 @@ def _validate_inputs( __all__ = [ - "DriftStats", "LogprobBackendUnavailable", "LogprobCandidate", "LogprobComparisonInputs", "LogprobComparisonReport", - "LogprobPathDrift", - "LogprobPathResult", "compare_single_gpu_logprob", "make_logprob_candidate", ] From 115d86c7a1ac9054ee2990ddb82a61869c4d8449 Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Tue, 4 Aug 2026 21:30:27 +0800 Subject: [PATCH 10/48] docs: document SM90 logprob validation --- docs/design/ws2-logprob-sm90-validation.md | 134 +++++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100644 docs/design/ws2-logprob-sm90-validation.md diff --git a/docs/design/ws2-logprob-sm90-validation.md b/docs/design/ws2-logprob-sm90-validation.md new file mode 100644 index 00000000..e94cb52d --- /dev/null +++ b/docs/design/ws2-logprob-sm90-validation.md @@ -0,0 +1,134 @@ +# WS2 Logprob PR2 SM90 Validation + +This document records the Hopper SM90 validation procedure for the PR2 single-GPU +logprob comparison harness from issue #241. It is a validation note for maintainers; +the cloud setup wrapper used during development is intentionally kept outside the +repository. + +## Prerequisites + +The validation host must provide: + +- Python 3.10 or newer; +- CUDA-enabled PyTorch; +- an NVIDIA Hopper GPU with compute capability 9.0, such as H100, H800, or H200; +- `nvidia-smi` and `nvcc`; +- a CUDA development environment capable of compiling the RL-Kernel extension. + +The CUDA version reported by `nvcc` must match `torch.version.cuda`. A runtime-only +image is insufficient because it normally does not include the CUDA compiler. + +## Build + +Activate an environment containing the repository dependencies and a CUDA-enabled +PyTorch installation, then build the editable extension with SM90 enabled: + +```bash +export FORCE_CUDA=1 +export KERNEL_ALIGN_FORCE_SM90=1 +export TORCH_CUDA_ARCH_LIST="9.0+PTX" +export MAX_JOBS=2 + +python -m pip install --no-build-isolation --no-deps -e . +``` + +Verify the extension and SM90 symbol after the build. Import PyTorch first so its +runtime libraries are available to the extension loader: + +```bash +python - <<'PY' +import torch +from rl_engine import _C + +print("torch:", torch.__version__) +print("torch CUDA:", torch.version.cuda) +print("extension:", _C.__file__) +print("SM90 symbol:", hasattr(_C, "batch_invariant_logp_sm90")) +PY +``` + +The final line must report `SM90 symbol: True`. + +## Validation commands + +Run the focused PR2 tests and the complete batch-invariant logprob suite: + +```bash +python -m pytest \ + tests/test_logprob_comparison.py \ + tests/test_operator_inputs.py \ + tests/test_op_checks.py -q + +python -m pytest tests/test_batch_invariant_logp.py -q +``` + +Run the two explicit SM90 comparisons: + +```bash +python scripts/compare_logprob.py \ + --candidate cuda-sm90 \ + --device cuda \ + --dtype bf16 \ + --batch 2 \ + --seq 8 \ + --vocab 1024 \ + --prompt-tokens 3 \ + --seed 7 + +python scripts/compare_logprob.py \ + --candidate cuda-sm90 \ + --device cuda \ + --dtype bf16 \ + --batch 2 \ + --seq 16 \ + --vocab 151936 \ + --prompt-tokens 8 \ + --seed 241 +``` + +The comparison command writes the JSON report to stdout. Diagnostic log messages are +written to stderr so stdout can be redirected directly to a `.json` file. + +## Expected report + +The report must identify the requested and actual backend as `cuda-sm90`, use the +`BatchInvariantLogpSM90Op` implementation, and record: + +```text +tp_world=1 +communication=none +lse_source=direct +``` + +LSE drift is measured over all logical token rows. Selected-logprob drift is measured +only over active response/action tokens. Each drift section includes maximum, mean, +p95, p99, and active-count values. + +## Validation result + +The procedure was validated on: + +```text +GPU: NVIDIA H800 PCIe +Compute capability: 9.0 +Python: 3.11.15 +PyTorch: 2.11.0+cu128 +CUDA toolkit / nvcc: 12.8 +Triton: 3.6.0 +``` + +Results: + +```text +PR2 focused tests: 41 passed +Complete batch-invariant logprob suite: 67 passed +``` + +Observed BF16 SM90 drift against the PyTorch reference: + +| Shape | LSE max abs | dlogp max abs | +| --- | ---: | ---: | +| `[2, 8, 1024]` | `4.76837158203125e-07` | `4.76837158203125e-07` | +| `[2, 16, 151936]` | `9.5367431640625e-07` | `9.5367431640625e-07` | + +Both runs used TP=1, no communication, and the explicit SM90 backend without fallback. From c028b5b71b4d29c3539dd059d36a65008e7bd40e Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Tue, 4 Aug 2026 21:50:52 +0800 Subject: [PATCH 11/48] fix: address logprob harness lint and provenance --- rl_engine/testing/logprob_comparison.py | 10 +++----- scripts/compare_logprob.py | 5 +--- tests/test_logprob_comparison.py | 33 ++++++++++++++++++++----- 3 files changed, 31 insertions(+), 17 deletions(-) diff --git a/rl_engine/testing/logprob_comparison.py b/rl_engine/testing/logprob_comparison.py index ad2ef76c..0be7fba4 100644 --- a/rl_engine/testing/logprob_comparison.py +++ b/rl_engine/testing/logprob_comparison.py @@ -175,15 +175,11 @@ def _run_ws1_reference( target_ids: torch.Tensor, ignore_index: int, ) -> tuple[torch.Tensor, torch.Tensor]: - from rl_engine.kernels.ops.pytorch.loss.batch_invariant_logp import ( - NativeBatchInvariantLogpOp, - ) + from rl_engine.kernels.ops.pytorch.loss.batch_invariant_logp import NativeBatchInvariantLogpOp op = NativeBatchInvariantLogpOp() logp = op(logits, target_ids, ignore_index=ignore_index, validate=True) - _, lse = op.forward_with_lse( - logits, target_ids, ignore_index=ignore_index, validate=True - ) + _, lse = op.forward_with_lse(logits, target_ids, ignore_index=ignore_index, validate=True) return logp.detach(), lse.detach() @@ -215,12 +211,12 @@ def _run_candidate( def _candidate_provenance(candidate: LogprobCandidate) -> dict[str, Any]: return { + **candidate.provenance, "requested_backend": candidate.requested_backend, "actual_backend": candidate.actual_backend, "tp_world": 1, "communication": "none", "lse_source": "direct", - **candidate.provenance, } diff --git a/scripts/compare_logprob.py b/scripts/compare_logprob.py index 6b42d4ea..b4736db7 100644 --- a/scripts/compare_logprob.py +++ b/scripts/compare_logprob.py @@ -16,10 +16,7 @@ if str(REPO_ROOT) not in sys.path: sys.path.insert(0, str(REPO_ROOT)) -from rl_engine.testing import ( # noqa: E402 - LogprobComparisonInputs, - compare_single_gpu_logprob, -) +from rl_engine.testing import LogprobComparisonInputs, compare_single_gpu_logprob # noqa: E402 from rl_engine.utils.logger import logger # noqa: E402 diff --git a/tests/test_logprob_comparison.py b/tests/test_logprob_comparison.py index 67c1d422..d4fece0a 100644 --- a/tests/test_logprob_comparison.py +++ b/tests/test_logprob_comparison.py @@ -12,9 +12,7 @@ from rl_engine.kernels.gtest import run_operator_suite from rl_engine.kernels.gtest.operator_specs import make_candidate, make_operator_case -from rl_engine.kernels.ops.pytorch.loss.batch_invariant_logp import ( - NativeBatchInvariantLogpOp, -) +from rl_engine.kernels.ops.pytorch.loss.batch_invariant_logp import NativeBatchInvariantLogpOp from rl_engine.testing.logprob_comparison import ( LogprobBackendUnavailable, LogprobCandidate, @@ -81,6 +79,31 @@ def shifted(logits, target_ids, ignore_index): assert drift["lse"]["p99_abs"] == pytest.approx(0.693, abs=1e-5) +def test_canonical_provenance_cannot_be_overridden(): + native = make_logprob_candidate("pytorch") + candidate = LogprobCandidate( + name="custom", + requested_backend="pytorch", + actual_backend="pytorch", + fn=native.fn, + provenance={ + "actual_backend": "fallback", + "tp_world": 8, + "communication": "all-gather", + "lse_source": "reconstructed", + "implementation": "custom", + }, + ) + + provenance = compare_single_gpu_logprob(_inputs(), candidates=(candidate,)).drifts[0].provenance + + assert provenance["actual_backend"] == "pytorch" + assert provenance["tp_world"] == 1 + assert provenance["communication"] == "none" + assert provenance["lse_source"] == "direct" + assert provenance["implementation"] == "custom" + + def test_all_inactive_tokens_produce_zero_dlogp_statistics(): inputs = _inputs() inputs = LogprobComparisonInputs( @@ -194,9 +217,7 @@ def test_operator_comparison_specs_register_batch_invariant_logp(): case = make_operator_case(args, torch.float32, torch.device("cpu")) candidate = make_candidate(args) - report = run_operator_suite( - "batch_invariant_logp", candidates=[candidate], cases=[case] - ) + report = run_operator_suite("batch_invariant_logp", candidates=[candidate], cases=[case]) assert report.passed assert report.candidates[0].cases[0].op_class == "logprob" From 7ba09b523b031b5e21de172a1d4d0766fc254e2f Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Tue, 4 Aug 2026 22:03:37 +0800 Subject: [PATCH 12/48] fix: type heterogeneous logprob backends --- rl_engine/testing/logprob_comparison.py | 1 + 1 file changed, 1 insertion(+) diff --git a/rl_engine/testing/logprob_comparison.py b/rl_engine/testing/logprob_comparison.py index 0be7fba4..52ca82d7 100644 --- a/rl_engine/testing/logprob_comparison.py +++ b/rl_engine/testing/logprob_comparison.py @@ -63,6 +63,7 @@ def to_dict(self) -> dict[str, Any]: def make_logprob_candidate(backend: str) -> LogprobCandidate: normalized = backend.strip().lower().replace("_", "-") + op: Any if normalized in {"pytorch", "native"}: from rl_engine.kernels.ops.pytorch.loss.batch_invariant_logp import ( NativeBatchInvariantLogpOp, From a63bea2dc0269cba82fca44e36cf3941de5aae33 Mon Sep 17 00:00:00 2001 From: KJLdefeated Date: Wed, 5 Aug 2026 16:05:57 +0800 Subject: [PATCH 13/48] init vocab parallel logp --- .github/workflows/ci.yml | 3 + docs/operators/batch-invariant-logp.md | 36 +- .../ops/pytorch/loss/vocab_parallel_logp.py | 408 +++++++++++++++++ rl_engine/kernels/registry.py | 27 ++ tests/test_logprob_contract.py | 32 ++ tests/test_operator_inputs.py | 26 ++ tests/test_vocab_parallel_logp.py | 426 ++++++++++++++++++ 7 files changed, 948 insertions(+), 10 deletions(-) create mode 100644 rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py create mode 100644 tests/test_vocab_parallel_logp.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 28cdb58d..2d9ba91e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,6 +79,9 @@ jobs: - name: Run WS2 Logprob Contract Tests (CPU-safe) run: python -m pytest tests/test_logprob_contract.py -v + - name: Run WS2 Vocab-Parallel Logprob Tests (CPU-safe) + run: python -m pytest tests/test_vocab_parallel_logp.py -v + docs: runs-on: ubuntu-latest steps: diff --git a/docs/operators/batch-invariant-logp.md b/docs/operators/batch-invariant-logp.md index 4bca1f8c..5068c33f 100644 --- a/docs/operators/batch-invariant-logp.md +++ b/docs/operators/batch-invariant-logp.md @@ -54,18 +54,31 @@ CUDA priority list when the extension exposes `_C.batch_invariant_logp_sm90` (built with `KERNEL_ALIGN_FORCE_SM90=1`) on an SM90 device. On any other build or device, dispatch is unchanged (Triton -> PyTorch). -### WS2 TP-aware dispatch +## Tensor Parallel -WS2 distributed callers use a separate contract-aware entry point, -`kernel_registry.get_logprob_op(contract)`. It validates explicit vocab-shard ownership, -padded-vs-real vocab metadata, active-token masking, and fixed `(max, sumexp)` merge -semantics before selecting a backend. Legacy `get_op("batch_invariant_logp")` behavior -remains unchanged. +`VocabParallelLogprobOp` +(`rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py`) +**TP=1, TP=2, and TP=4 produce bit-identical results.** -The backends above are single-shard (TP=1) references and do not yet export vocab-domain -LSE or carry vocab-shard metadata, so they are declared incompatible with strict WS2 -requests instead of being selected as a silent fallback. The contract objects are -documented in `rl_engine.kernels.logprob_contract`. +1. Split the padded vocabulary into `num_vocab_tiles` fixed tiles. +2. Each rank computes fp32 `(max, sumexp)` for the tiles it owns. Every tile + is reduced as the same contiguous `[n, tile]` shape, on any rank. +3. All tile partials are shared with `all_gather`. The collective only moves + bytes; it never does math, so it cannot round anything. +4. Every rank merges all tiles in the same fixed order, over the same + `[n, num_vocab_tiles]` shape. `LSE = M + log(sum(s_t * exp(m_t - M)))`. +5. The target logit is copied from the rank that owns it (never summed). +6. `logp = target_logit - LSE`. Inactive rows become `0.0`. + +Usage goes through the contract-aware entry point: + +```python +from rl_engine.kernels.registry import kernel_registry + +result = kernel_registry.get_logprob_op(contract) # LogprobContract from +op = result.op # rl_engine.kernels.logprob_contract +logp, lse = op(local_logits, target_ids, contract=contract, tp_group=tp_group) +``` ## Benchmarks @@ -237,3 +250,6 @@ WSL/Linux with CUDA. - `rl_engine/kernels/registry.py` - `tests/test_batch_invariant_logp.py` - `benchmarks/benchmark_batch_invariant_logp.py` +- `rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py` +- `rl_engine/kernels/logprob_contract.py` +- `tests/test_vocab_parallel_logp.py` diff --git a/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py b/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py new file mode 100644 index 00000000..06eded1c --- /dev/null +++ b/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py @@ -0,0 +1,408 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Deterministic vocab-parallel TP selected-token logprob reference (issue #241 PR3). + +Implements the WS2 contract in ``rl_engine.kernels.logprob_contract`` with a +TP-independent vocab tile decomposition: the padded vocabulary is split into +``num_vocab_tiles`` fixed tiles, every tile's fp32 ``(max, sumexp)`` partial is +computed from a contiguous ``[n, tile]`` tensor, all tile partials travel by +all-gather (transport only), and every rank merges them in global tile-index +order over a fixed ``[n, num_vocab_tiles]`` shape. The TP degree only decides +which rank computes which tiles and never changes any floating-point grouping, +so outputs and gradients are bitwise-identical across TP degrees +(``DeterminismScope.CROSS_TP_BITWISE``) as long as ``num_vocab_tiles`` is held +fixed. A fixed per-shard merge order alone cannot provide this property: +shard boundaries would regroup the combines differently at each degree. + +Consequences of the tile structure: + +- ``num_vocab_tiles`` is part of the numerical identity. It must be pinned + across ranks (enforced by the preflight) and across the TP degrees being + compared; it is never derived from the shard layout. +- Every shard boundary must be tile-aligned; misalignment fails loudly. +- At TP=1 the result matches the WS1 ``NativeBatchInvariantLogpOp`` only + within the #108 logprob tolerance, not bitwise — the WS1 op reduces the + whole ``[n, V]`` row at once, which groups the sums differently. + +Preconditions: logits over the real vocabulary must be finite. A row whose +real-vocab logits are all ``-inf`` has no finite logsumexp; with +``validate=True`` such a row fails loudly if it is active. + +The selected logprob is zero-filled at inactive rows (``MaskSpec.active_mask`` +is the sole authority; with validation enabled an active row can never legally +hold ``ignore_index``). The vocab-domain LSE is returned for every row and is +differentiable everywhere, including inactive rows. +""" + +from __future__ import annotations + +from typing import Any + +import torch + +from rl_engine.kernels.logprob_contract import LogprobContract, LogprobContractError, LogprobDType + +BACKEND_ID = "pytorch-vocab-parallel-logp-ws2" +DEFAULT_NUM_VOCAB_TILES = 64 + +_TORCH_TO_CONTRACT_DTYPE = { + torch.bfloat16: LogprobDType.BF16, + torch.float16: LogprobDType.FP16, + torch.float32: LogprobDType.FP32, +} + + +def _require_distributed_initialized(): + import torch.distributed as dist + + if not dist.is_available(): + raise LogprobContractError("vocab-parallel logprob requires torch.distributed.") + if not dist.is_initialized(): + raise LogprobContractError( + "vocab-parallel logprob requires an initialized process group when " + "the contract declares tp_world_size > 1." + ) + return dist + + +def _tile_size(contract: LogprobContract, num_vocab_tiles: int) -> int: + if isinstance(num_vocab_tiles, bool) or not isinstance(num_vocab_tiles, int): + raise LogprobContractError( + f"num_vocab_tiles must be a positive integer; got {num_vocab_tiles!r}" + ) + if num_vocab_tiles <= 0: + raise LogprobContractError( + f"num_vocab_tiles must be a positive integer; got {num_vocab_tiles}" + ) + padded = contract.sharding.padded_vocab_size + if padded % num_vocab_tiles != 0: + raise LogprobContractError( + f"num_vocab_tiles={num_vocab_tiles} must divide " f"padded_vocab_size={padded} exactly" + ) + tile = padded // num_vocab_tiles + for rank, (start, end) in enumerate(contract.sharding.vocab_shard_bounds): + if start % tile != 0 or end % tile != 0: + raise LogprobContractError( + f"vocab_shard_bounds[{rank}]=[{start}, {end}) is not aligned to the " + f"vocab tile size {tile} (num_vocab_tiles={num_vocab_tiles}); " + "cross-TP bitwise determinism requires tile-aligned shard bounds" + ) + return tile + + +def _validate_invocation( + local_logits: torch.Tensor, + target_ids: torch.Tensor, + contract: LogprobContract, + tp_group: Any, +) -> None: + if not isinstance(contract, LogprobContract): + raise LogprobContractError("contract must be a LogprobContract") + if local_logits.dim() != 2: + raise LogprobContractError( + f"local_logits must be 2-D [num_tokens, local_vocab]; got {local_logits.dim()}-D" + ) + if target_ids.dim() != 1 or target_ids.shape[0] != local_logits.shape[0]: + raise LogprobContractError( + f"target_ids must be 1-D with one entry per token; got shape " + f"{tuple(target_ids.shape)} for {local_logits.shape[0]} tokens" + ) + sharding = contract.sharding + if local_logits.shape[1] != sharding.local_vocab_size: + raise LogprobContractError( + f"local_logits has {local_logits.shape[1]} vocab columns but the contract " + f"declares local shard [{sharding.local_vocab_start}, " + f"{sharding.local_vocab_end}) of size {sharding.local_vocab_size}" + ) + if local_logits.shape[0] != contract.mask.num_tokens: + raise LogprobContractError( + f"local_logits has {local_logits.shape[0]} tokens but MaskSpec declares " + f"num_tokens={contract.mask.num_tokens}" + ) + declared = _TORCH_TO_CONTRACT_DTYPE.get(local_logits.dtype) + if declared is not contract.dtype: + raise LogprobContractError( + f"local_logits dtype {local_logits.dtype} does not match the contract " + f"dtype {contract.dtype.value}" + ) + if sharding.tp_world_size > 1: + dist = _require_distributed_initialized() + group_rank = dist.get_rank(group=tp_group) + group_world = dist.get_world_size(group=tp_group) + if group_world != sharding.tp_world_size: + raise LogprobContractError( + f"tp_group world size {group_world} does not match the contract " + f"tp_world_size={sharding.tp_world_size}; pass the TP subgroup, " + "not the global group" + ) + if group_rank != sharding.tp_rank: + raise LogprobContractError( + f"tp_group rank {group_rank} does not match the contract " + f"tp_rank={sharding.tp_rank}" + ) + + +def _validate_active_targets( + target_1d: torch.Tensor, active_mask: torch.Tensor, real_vocab_size: int +) -> None: + bad = active_mask & ((target_1d < 0) | (target_1d >= real_vocab_size)) + if bool(bad.any().item()): + bad_values = target_1d[bad] + raise LogprobContractError( + "active target_ids must lie in the real vocabulary " + f"[0, {real_vocab_size}); got values in " + f"[{int(bad_values.min().item())}, {int(bad_values.max().item())}] " + "on active rows" + ) + + +def _preflight_cross_rank_agreement( + contract: LogprobContract, tp_group: Any, num_vocab_tiles: int +) -> None: + """All-gather (fingerprint, backend id, tile count) and abort on mismatch.""" + + dist = _require_distributed_initialized() + payload = (contract.cross_rank_fingerprint(), BACKEND_ID, int(num_vocab_tiles)) + world = dist.get_world_size(group=tp_group) + gathered: list[Any] = [None] * world + dist.all_gather_object(gathered, payload, group=tp_group) + mismatched = [(rank, other) for rank, other in enumerate(gathered) if other != payload] + if mismatched: + rank, other = mismatched[0] + raise LogprobContractError( + "cross-rank preflight failed: rank " + f"{contract.sharding.tp_rank} has {payload} but rank {rank} has {other}; " + "all TP ranks must agree on the contract fingerprint, backend id, and " + "num_vocab_tiles before any collective" + ) + + +def _local_tile_stats(z_masked: torch.Tensor, tile: int) -> tuple[torch.Tensor, torch.Tensor]: + """fp32 per-tile ``(max, sumexp)`` partials for this rank's shard. + + Each tile is reduced as a contiguous ``[n, tile]`` tensor so the reduction + shape and layout are identical no matter which rank computes the tile or + what the local shard size is. An all-``-inf`` (padding-only) tile yields + the identity partial ``(-inf, 0)`` without evaluating ``exp(-inf - (-inf))``. + """ + + n, local_vocab = z_masked.shape + m_parts: list[torch.Tensor] = [] + s_parts: list[torch.Tensor] = [] + for tile_index in range(local_vocab // tile): + block = z_masked[:, tile_index * tile : (tile_index + 1) * tile].contiguous() + m_t = block.max(dim=-1).values + finite = m_t > float("-inf") + m_safe = torch.where(finite, m_t, torch.zeros_like(m_t)) + s_t = (block - m_safe.unsqueeze(-1)).exp().sum(dim=-1) + s_t = torch.where(finite, s_t, torch.zeros_like(s_t)) + m_parts.append(m_t) + s_parts.append(s_t) + return torch.stack(m_parts, dim=1), torch.stack(s_parts, dim=1) + + +def _gather_tile_stats( + local_m: torch.Tensor, + local_s: torch.Tensor, + contract: LogprobContract, + tp_group: Any, + tile: int, +) -> tuple[torch.Tensor, torch.Tensor]: + """Assemble all ``num_vocab_tiles`` partials in global tile order.""" + + sharding = contract.sharding + tile_counts = [(end - start) // tile for start, end in sharding.vocab_shard_bounds] + if sharding.tp_world_size == 1: + return local_m.contiguous(), local_s.contiguous() + + dist = _require_distributed_initialized() + n = local_m.shape[0] + max_tiles = max(tile_counts) + packed = local_m.new_zeros((n, max_tiles, 2)) + packed[:, : local_m.shape[1], 0] = local_m + packed[:, : local_s.shape[1], 1] = local_s + packed = packed.contiguous() + gathered = [torch.empty_like(packed) for _ in range(sharding.tp_world_size)] + dist.all_gather(gathered, packed, group=tp_group) + + m_parts = [gathered[rank][:, : tile_counts[rank], 0] for rank in range(len(tile_counts))] + s_parts = [gathered[rank][:, : tile_counts[rank], 1] for rank in range(len(tile_counts))] + return torch.cat(m_parts, dim=1).contiguous(), torch.cat(s_parts, dim=1).contiguous() + + +def _gather_target_logit( + z_masked: torch.Tensor, + safe_target: torch.Tensor, + contract: LogprobContract, + tp_group: Any, +) -> torch.Tensor: + """Exact selected-target logit via a select-by-owner copy.""" + + sharding = contract.sharding + n = z_masked.shape[0] + start = sharding.local_vocab_start + local_vocab = sharding.local_vocab_size + local_idx = (safe_target - start).clamp(0, max(local_vocab - 1, 0)) + owns = (safe_target >= start) & (safe_target < sharding.local_vocab_end) + rows = torch.arange(n, device=z_masked.device) + local_contrib = torch.where( + owns, z_masked[rows, local_idx], torch.zeros_like(safe_target, dtype=z_masked.dtype) + ).contiguous() + + if sharding.tp_world_size == 1: + stacked = local_contrib.unsqueeze(0) + else: + dist = _require_distributed_initialized() + gathered = [torch.empty_like(local_contrib) for _ in range(sharding.tp_world_size)] + dist.all_gather(gathered, local_contrib, group=tp_group) + stacked = torch.stack(gathered, dim=0) + + starts = torch.tensor( + [bound_start for bound_start, _ in sharding.vocab_shard_bounds], + device=safe_target.device, + dtype=torch.long, + ) + owner = torch.bucketize(safe_target, starts, right=True) - 1 + return stacked[owner, rows] + + +def _merge_tile_partials(m_all: torch.Tensor, s_all: torch.Tensor) -> torch.Tensor: + """Fixed-order (max, sumexp) merge over [n, num_vocab_tiles].""" + + M = m_all.max(dim=1).values + finite = M > float("-inf") + M_safe = torch.where(finite, M, torch.zeros_like(M)) + terms = s_all * (m_all - M_safe.unsqueeze(1)).exp() + S = terms.sum(dim=1) + return M + S.log() + + +class _VocabParallelLogprobFunction(torch.autograd.Function): + @staticmethod + def forward(ctx, local_logits, target_1d, active_mask, contract, tp_group, tile): + z_masked = local_logits.float() + sharding = contract.sharding + global_ids = torch.arange( + sharding.local_vocab_start, sharding.local_vocab_end, device=z_masked.device + ) + padding_cols = global_ids >= sharding.real_vocab_size + if bool(padding_cols.any()): + z_masked = z_masked.masked_fill(padding_cols.unsqueeze(0), float("-inf")) + + safe_target = torch.where(active_mask, target_1d, torch.zeros_like(target_1d)) + + local_m, local_s = _local_tile_stats(z_masked, tile) + m_all, s_all = _gather_tile_stats(local_m, local_s, contract, tp_group, tile) + target_logit = _gather_target_logit(z_masked, safe_target, contract, tp_group) + lse = _merge_tile_partials(m_all, s_all) + + selected_logp = torch.where(active_mask, target_logit - lse, torch.zeros_like(lse)) + + ctx.save_for_backward(z_masked, lse, safe_target, active_mask, padding_cols) + ctx.local_vocab_start = sharding.local_vocab_start + ctx.local_vocab_size = sharding.local_vocab_size + ctx.input_dtype = local_logits.dtype + ctx.set_materialize_grads(False) + return selected_logp, lse + + @staticmethod + def backward(ctx, grad_logp, grad_lse): + if not ctx.needs_input_grad[0] or (grad_logp is None and grad_lse is None): + return None, None, None, None, None, None + + z_masked, lse, safe_target, active_mask, padding_cols = ctx.saved_tensors + n, local_vocab = z_masked.shape + finite_row = torch.isfinite(lse) + lse_safe = torch.where(finite_row, lse, torch.zeros_like(lse)) + p = (z_masked - lse_safe.unsqueeze(1)).exp() + p = torch.where(finite_row.unsqueeze(1), p, torch.zeros_like(p)) + + grad = torch.zeros_like(z_masked) + if grad_logp is not None: + local_idx = (safe_target - ctx.local_vocab_start).clamp(0, max(local_vocab - 1, 0)) + owns = (safe_target >= ctx.local_vocab_start) & ( + safe_target < ctx.local_vocab_start + local_vocab + ) + onehot = torch.zeros_like(z_masked) + hit = owns & active_mask + rows = torch.arange(n, device=z_masked.device)[hit] + onehot[rows, local_idx[hit]] = 1.0 + g_logp = torch.where(active_mask, grad_logp, torch.zeros_like(grad_logp)) + grad = grad + g_logp.unsqueeze(1) * (onehot - p) + if grad_lse is not None: + grad = grad + grad_lse.unsqueeze(1) * p + if bool(padding_cols.any()): + grad = grad.masked_fill(padding_cols.unsqueeze(0), 0.0) + return grad.to(ctx.input_dtype), None, None, None, None, None + + +class VocabParallelLogprobOp: + """Deterministic vocab-parallel selected-token logprob (WS2 reference).""" + + op_class = "logprob" + is_batch_invariant = True + + def __init__(self) -> None: + pass + + def __call__( + self, + local_logits: torch.Tensor, + target_ids: torch.Tensor, + *, + contract: LogprobContract, + tp_group: Any = None, + num_vocab_tiles: int = DEFAULT_NUM_VOCAB_TILES, + validate: bool = True, + ) -> tuple[torch.Tensor, torch.Tensor]: + return self.apply( + local_logits, + target_ids, + contract=contract, + tp_group=tp_group, + num_vocab_tiles=num_vocab_tiles, + validate=validate, + ) + + def apply( + self, + local_logits: torch.Tensor, + target_ids: torch.Tensor, + *, + contract: LogprobContract, + tp_group: Any = None, + num_vocab_tiles: int = DEFAULT_NUM_VOCAB_TILES, + validate: bool = True, + ) -> tuple[torch.Tensor, torch.Tensor]: + if not isinstance(contract, LogprobContract): + raise LogprobContractError("contract must be a LogprobContract") + tile = _tile_size(contract, num_vocab_tiles) + _validate_invocation(local_logits, target_ids, contract, tp_group) + + target_1d = target_ids.reshape(-1).to(device=local_logits.device, dtype=torch.long) + active_mask = torch.tensor( + contract.mask.active_mask, dtype=torch.bool, device=local_logits.device + ) + if validate: + _validate_active_targets(target_1d, active_mask, contract.sharding.real_vocab_size) + if contract.sharding.tp_world_size > 1: + _preflight_cross_rank_agreement(contract, tp_group, num_vocab_tiles) + + selected_logp, lse = _VocabParallelLogprobFunction.apply( + local_logits, target_1d, active_mask, contract, tp_group, tile + ) + + if validate and bool((~torch.isfinite(lse) & active_mask).any().item()): + raise LogprobContractError( + "non-finite logsumexp on an active row: logits over the real " + "vocabulary must be finite for every active token" + ) + return selected_logp, lse + + +__all__ = [ + "BACKEND_ID", + "DEFAULT_NUM_VOCAB_TILES", + "VocabParallelLogprobOp", +] diff --git a/rl_engine/kernels/registry.py b/rl_engine/kernels/registry.py index 7093b320..213aa7a2 100644 --- a/rl_engine/kernels/registry.py +++ b/rl_engine/kernels/registry.py @@ -85,6 +85,10 @@ class OpBackend(Enum, metaclass=_KernelEnumMeta): CUDA_BATCH_INVARIANT_LOGP_SM90 = ( "rl_engine.kernels.ops.cuda.loss.batch_invariant_logp.BatchInvariantLogpSM90Op" ) + # Deterministic vocab-parallel TP logprob reference (WS2 #241 PR3) + PYTORCH_VOCAB_PARALLEL_LOGP = ( + "rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp.VocabParallelLogprobOp" + ) # RMSNorm(pre-norm / QK-Norm) - pure Pytorch reference(ws1 ground-truth) PYTORCH_NATIVE_RMS_NORM = "rl_engine.kernels.ops.pytorch.norm.rms_norm.NativeRMSNormOp" @@ -350,6 +354,29 @@ def __init__(self): for platform, candidates in self._logprob_candidates.items() } + # deterministic vocab-parallel TP logprob reference. + ws2_tp_logprob_capability = LogprobBackendCapability( + backend_id="pytorch-vocab-parallel-logp-ws2", + roles=common_logprob_roles, + dtypes=common_logprob_dtypes, + tp_world_sizes=None, + cp_world_sizes=None, + supports_vocab_padding=True, + mask_modes=frozenset({MaskMode.EXPLICIT_ACTIVE_MASK, MaskMode.IGNORE_INDEX}), + exports_vocab_lse=True, + determinism_scopes=frozenset( + {DeterminismScope.CROSS_TP_BITWISE, DeterminismScope.FIXED_TOPOLOGY} + ), + implementation_kind="reference", + ) + for ws2_platform in self._priority_map: + self.register_logprob_backend( + OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP, + ws2_tp_logprob_capability, + platform=ws2_platform, + prepend=True, + ) + def _adjust_priority_from_env(self): rocm_attn_backend = os.getenv("RL_KERNEL_ROCM_ATTN_BACKEND", "").strip().lower() if rocm_attn_backend in {"flash_attn", "flash-attn", "flash_attention"}: diff --git a/tests/test_logprob_contract.py b/tests/test_logprob_contract.py index 40cea4e2..a961f312 100644 --- a/tests/test_logprob_contract.py +++ b/tests/test_logprob_contract.py @@ -247,8 +247,20 @@ def test_ignore_index_must_not_collide_with_the_real_vocabulary(): assert contract.mask.ignore_index == padding_column +def _restrict_to_ws1_candidates(registry: KernelRegistry) -> None: + """Drop the #241 PR3 vocab-parallel reference so only WS1 backends remain.""" + + platform = registry._platform() + registry._logprob_candidates[platform] = [ + backend + for backend in registry._logprob_candidates[platform] + if backend is not OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP + ] + + def test_current_ws1_backend_rejects_strict_tp_contract_without_fallback(): registry = KernelRegistry() + _restrict_to_ws1_candidates(registry) with pytest.raises(RuntimeError) as exc_info: registry.get_logprob_op(_contract()) @@ -262,6 +274,7 @@ def test_current_ws1_backend_rejects_strict_tp_contract_without_fallback(): def test_current_ws1_backend_rejects_padded_vocab_even_at_tp1(): registry = KernelRegistry() + _restrict_to_ws1_candidates(registry) contract = _contract(sharding=_sharding(tp_world_size=1, cp_world_size=1)) with pytest.raises(RuntimeError) as exc_info: @@ -272,6 +285,25 @@ def test_current_ws1_backend_rejects_padded_vocab_even_at_tp1(): assert "padded-vs-real vocab masking is unsupported" in message +def test_ws1_rejections_recorded_when_vocab_parallel_reference_resolves(): + """The WS1 backends still reject strict contracts; they are skipped with + recorded reasons while dispatch resolves the #241 PR3 reference.""" + + registry = KernelRegistry() + platform = registry._platform() + # Order the WS1 backends ahead of the reference so their rejections are + # exercised on the way to a successful resolution. + candidates = registry._logprob_candidates[platform] + candidates.remove(OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP) + candidates.append(OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP) + + result = registry.get_logprob_op(_contract()) + assert result.capability.backend_id == "pytorch-vocab-parallel-logp-ws2" + assert result.provenance["fallback"] is True + rejections = " | ".join(result.provenance["prior_rejections"]) + assert "vocab-domain LSE export is unsupported" in rejections + + def test_undeclared_backend_capability_is_never_selected(): registry = KernelRegistry() platform = registry._platform() diff --git a/tests/test_operator_inputs.py b/tests/test_operator_inputs.py index 4f742734..2080f6fd 100644 --- a/tests/test_operator_inputs.py +++ b/tests/test_operator_inputs.py @@ -40,6 +40,7 @@ def _args(**overrides): "logp", "linear_logp", "batch_invariant_logp", + "vocab_parallel_logp", "rope", "silu", "swiglu", @@ -72,6 +73,31 @@ def test_constant_batch_invariant_logp_inputs_match_operator_contract(): assert torch.equal(inputs["target_ids"], torch.full((1, 2), 3, dtype=torch.long)) +def test_constant_vocab_parallel_logp_inputs_match_operator_contract(): + args = _args(input_mode="constant", constant_value=0.5, token_value=3) + inputs = make_operator_inputs("vocab_parallel_logp", args, torch.float32, torch.device("cpu")) + + # vocab=17 rounds up to padded=20 with 4 tiles; tokens flatten to batch*seq. + assert torch.equal(inputs["local_logits"], torch.full((2, 20), 0.5)) + assert torch.equal(inputs["target_ids"], torch.full((2,), 3, dtype=torch.long)) + assert inputs["contract"].sharding.real_vocab_size == 17 + assert inputs["contract"].sharding.padded_vocab_size == 20 + assert inputs["num_vocab_tiles"] == 4 + assert operator_shape_name("vocab_parallel_logp", args) == "2x20" + + +def test_vocab_parallel_logp_inputs_run_through_the_operator(): + from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import VocabParallelLogprobOp + + args = _args(input_mode="random", seed=7) + inputs = make_operator_inputs("vocab_parallel_logp", args, torch.float32, torch.device("cpu")) + + logp, lse = VocabParallelLogprobOp()(**inputs) + assert logp.shape == inputs["target_ids"].shape + assert logp.dtype == torch.float32 and lse.dtype == torch.float32 + assert torch.isfinite(logp).all() and torch.isfinite(lse).all() + + def test_random_logp_inputs_are_seeded(): args = _args(input_mode="random", seed=7) first = make_operator_inputs("logp", args, torch.float32, torch.device("cpu")) diff --git a/tests/test_vocab_parallel_logp.py b/tests/test_vocab_parallel_logp.py new file mode 100644 index 00000000..510aec70 --- /dev/null +++ b/tests/test_vocab_parallel_logp.py @@ -0,0 +1,426 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Deterministic vocab-parallel TP logprob reference tests (issue #241 PR3). + +Bit-level determinism assertions compare raw bit patterns via +``tensor.view(torch.int32)`` rather than ``torch.equal``: value equality +treats ``-0.0 == 0.0`` as equal and ``NaN != NaN`` as different, neither of +which is what a bitwise claim means. +""" + +from __future__ import annotations + +import queue +import tempfile +import traceback +from pathlib import Path + +import pytest +import torch +import torch.multiprocessing as mp + +from rl_engine.kernels.gtest.tolerance import load_contract +from rl_engine.kernels.logprob_contract import ( + DeterminismScope, + LogprobContract, + LogprobContractError, + MaskSpec, + ReductionSpec, + ShardingSpec, +) +from rl_engine.kernels.ops.pytorch.loss.batch_invariant_logp import NativeBatchInvariantLogpOp +from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import ( + BACKEND_ID, + VocabParallelLogprobOp, +) +from rl_engine.kernels.registry import KernelRegistry, OpBackend + +REAL_VOCAB = 27 +PADDED_VOCAB = 32 +NUM_TILES = 8 +NUM_TOKENS = 6 +ACTIVE = (True, True, True, True, True, False) + + +def _even_bounds(padded: int, world: int) -> tuple[tuple[int, int], ...]: + shard = padded // world + return tuple( + (rank * shard, padded if rank == world - 1 else (rank + 1) * shard) for rank in range(world) + ) + + +def _contract( + *, + tp_rank: int = 0, + tp_world_size: int = 1, + bounds: tuple[tuple[int, int], ...] | None = None, + real_vocab: int = REAL_VOCAB, + padded_vocab: int = PADDED_VOCAB, + num_tokens: int = NUM_TOKENS, + active: tuple[bool, ...] = ACTIVE, + dtype: str = "fp32", +) -> LogprobContract: + return LogprobContract( + role="train", + dtype=dtype, + mask=MaskSpec(num_tokens=num_tokens, active_mask=active), + sharding=ShardingSpec( + tp_rank=tp_rank, + tp_world_size=tp_world_size, + vocab_shard_bounds=( + bounds if bounds is not None else _even_bounds(padded_vocab, tp_world_size) + ), + real_vocab_size=real_vocab, + padded_vocab_size=padded_vocab, + ), + reduction=ReductionSpec(), + ) + + +def _inputs(dtype=torch.float32, seed: int = 2026): + torch.manual_seed(seed) + logits = torch.randn(NUM_TOKENS, PADDED_VOCAB, dtype=torch.float32).to(dtype) + targets = torch.tensor([1, 5, REAL_VOCAB - 1, 0, 13, -100]) + return logits, targets + + +def _bits(tensor: torch.Tensor) -> torch.Tensor: + view_dtype = {torch.float32: torch.int32, torch.bfloat16: torch.int16}[tensor.dtype] + return tensor.contiguous().view(view_dtype) + + +def _bitwise_equal(a: torch.Tensor, b: torch.Tensor) -> bool: + return a.shape == b.shape and bool((_bits(a) == _bits(b)).all()) + + +def _case_shard_size_mismatch(): + logits, targets = _inputs() + return logits, targets, _contract(tp_rank=0, tp_world_size=2), NUM_TILES, "vocab columns" + + +def _case_mask_length_mismatch(): + logits, targets = _inputs() + contract = _contract(num_tokens=NUM_TOKENS + 1, active=ACTIVE + (True,)) + return logits, targets, contract, NUM_TILES, "num_tokens" + + +def _case_dtype_mismatch(): + logits, targets = _inputs() + return logits, targets, _contract(dtype="bf16"), NUM_TILES, "dtype" + + +def _case_tile_misaligned_bounds(): + # Tile size is 32/8 = 4; a boundary at 6 is misaligned. + logits, targets = _inputs() + contract = _contract(tp_world_size=2, bounds=((0, 6), (6, 32))) + return logits[:, :6], targets, contract, NUM_TILES, "tile" + + +def _case_bad_num_vocab_tiles(): + logits, targets = _inputs() + return logits, targets, _contract(), 7, "num_vocab_tiles" + + +def _case_active_target_out_of_real_vocab(): + logits, targets = _inputs() + bad_targets = targets.clone() + bad_targets[0] = REAL_VOCAB # padding column, active row + return logits, bad_targets, _contract(), NUM_TILES, "real vocabulary" + + +def _case_all_inf_active_row(): + logits, targets = _inputs() + poisoned = logits.clone() + poisoned[0, :] = float("-inf") + return poisoned, targets, _contract(), NUM_TILES, "non-finite" + + +@pytest.mark.parametrize( + "case", + [ + _case_shard_size_mismatch, + _case_mask_length_mismatch, + _case_dtype_mismatch, + _case_tile_misaligned_bounds, + _case_bad_num_vocab_tiles, + _case_active_target_out_of_real_vocab, + _case_all_inf_active_row, + ], + ids=lambda fn: fn.__name__.removeprefix("_case_"), +) +def test_invalid_invocations_fail_loudly(case): + logits, targets, contract, num_tiles, match = case() + with pytest.raises(LogprobContractError, match=match): + VocabParallelLogprobOp()(logits, targets, contract=contract, num_vocab_tiles=num_tiles) + + +class TestSingleRank: + def test_repeated_runs_are_bitwise_identical(self): + contract = _contract() + logits, targets = _inputs() + op = VocabParallelLogprobOp() + logp_a, lse_a = op(logits, targets, contract=contract, num_vocab_tiles=NUM_TILES) + logp_b, lse_b = op(logits, targets, contract=contract, num_vocab_tiles=NUM_TILES) + assert _bitwise_equal(logp_a, logp_b) + assert _bitwise_equal(lse_a, lse_b) + + def test_batch_invariance_same_row_any_context(self): + contract_full = _contract() + logits, targets = _inputs() + op = VocabParallelLogprobOp() + logp_full, lse_full = op(logits, targets, contract=contract_full, num_vocab_tiles=NUM_TILES) + + contract_single = _contract(num_tokens=1, active=(True,)) + logp_one, lse_one = op( + logits[2:3], targets[2:3], contract=contract_single, num_vocab_tiles=NUM_TILES + ) + assert _bitwise_equal(logp_full[2:3], logp_one) + assert _bitwise_equal(lse_full[2:3], lse_one) + + def test_matches_ws1_batch_invariant_logp_within_contract_tolerance(self): + tolerance = load_contract()["accuracy"]["default"]["logprob"]["float32"] + contract = _contract(padded_vocab=REAL_VOCAB + 5) + # Use a real==padded contract so the WS1 op sees identical logits. + contract = _contract(real_vocab=PADDED_VOCAB, padded_vocab=PADDED_VOCAB) + logits, targets = _inputs() + logp, _ = VocabParallelLogprobOp()( + logits, targets, contract=contract, num_vocab_tiles=NUM_TILES + ) + ws1 = NativeBatchInvariantLogpOp().apply(logits, targets) + active = torch.tensor(ACTIVE) + assert torch.allclose( + logp[active], ws1[active], atol=tolerance["atol"], rtol=tolerance["rtol"] + ) + + def test_padding_columns_are_excluded_and_finite(self): + contract = _contract() + logits, targets = _inputs() + boosted = logits.clone() + boosted[:, REAL_VOCAB:] = 1e4 # huge padding logits must not leak into LSE + logp, lse = VocabParallelLogprobOp()( + boosted, targets, contract=contract, num_vocab_tiles=NUM_TILES + ) + ref_lse = torch.logsumexp(boosted[:, :REAL_VOCAB].float(), dim=-1) + assert torch.isfinite(logp).all() and torch.isfinite(lse).all() + assert torch.allclose(lse, ref_lse, atol=1e-5) + + def test_inactive_rows_zero_filled_lse_still_exported(self): + contract = _contract() + logits, targets = _inputs() + logp, lse = VocabParallelLogprobOp()( + logits, targets, contract=contract, num_vocab_tiles=NUM_TILES + ) + assert logp[-1].item() == 0.0 + assert torch.isfinite(lse[-1]) + + +class TestBackward: + def test_grads_match_autograd_oracle(self): + tolerance = load_contract()["accuracy"]["default"]["logprob"]["float32"] + contract = _contract() + logits, targets = _inputs() + x = logits.clone().requires_grad_(True) + logp, lse = VocabParallelLogprobOp()( + x, targets, contract=contract, num_vocab_tiles=NUM_TILES + ) + (logp.sum() + 0.5 * lse.sum()).backward() + + y = logits.clone().requires_grad_(True) + ref_lse = torch.logsumexp(y[:, :REAL_VOCAB].float(), dim=-1) + safe = targets.clamp(0, REAL_VOCAB - 1) + ref_logp = y[torch.arange(NUM_TOKENS), safe].float() - ref_lse + ref_logp = torch.where(torch.tensor(ACTIVE), ref_logp, torch.zeros_like(ref_logp)) + (ref_logp.sum() + 0.5 * ref_lse.sum()).backward() + + assert torch.allclose(x.grad, y.grad, atol=tolerance["atol"], rtol=tolerance["rtol"]) + assert bool((x.grad[:, REAL_VOCAB:] == 0).all()) + + # No grad requested -> outputs detached from autograd entirely. + logp_ng, lse_ng = VocabParallelLogprobOp()( + logits, targets, contract=contract, num_vocab_tiles=NUM_TILES + ) + assert not logp_ng.requires_grad and not lse_ng.requires_grad + + def test_inactive_rows_grad_asymmetry(self): + """The logp term is zeroed on inactive rows; the lse term still flows — + lse is a row property exported (and differentiable) for every row.""" + + contract = _contract() + logits, targets = _inputs() + + x = logits.clone().requires_grad_(True) + _, lse = VocabParallelLogprobOp()(x, targets, contract=contract, num_vocab_tiles=NUM_TILES) + lse.sum().backward() + assert bool((x.grad[-1, :REAL_VOCAB].abs() > 0).any()) + + z = logits.clone().requires_grad_(True) + logp, _ = VocabParallelLogprobOp()(z, targets, contract=contract, num_vocab_tiles=NUM_TILES) + logp.sum().backward() + assert bool((z.grad[-1] == 0).all()) + + +def test_dispatch_resolves_reference_and_leaves_legacy_untouched(): + registry = KernelRegistry() + contract = _contract() + + result = registry.get_logprob_op(contract) + assert result.capability.backend_id == BACKEND_ID + assert result.provenance["fallback"] is False + assert isinstance(result.op, VocabParallelLogprobOp) + assert ( + result.provenance["contract"]["reduction"]["determinism_scope"] + == DeterminismScope.CROSS_TP_BITWISE.value + ) + + by_id = registry.get_logprob_op(contract, requested_backend=BACKEND_ID) + assert by_id.capability.backend_id == BACKEND_ID + by_kind = registry.get_logprob_op(contract, requested_backend="reference") + assert by_kind.capability.backend_id == BACKEND_ID + + for ops in registry._priority_map.values(): + for candidates in ops.values(): + assert OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP not in candidates + + +# --------------------------------------------------------------------------- +# Multi-rank gloo tests (spawn pattern from tests/test_linear_logp.py) +# --------------------------------------------------------------------------- + + +def _gloo_available() -> bool: + return torch.distributed.is_available() and torch.distributed.is_gloo_available() + + +requires_gloo = pytest.mark.skipif( + not _gloo_available(), reason="requires torch.distributed with the gloo backend" +) + +_WORLD_SIZE = 4 +_UNEVEN_BOUNDS = ((0, 4), (4, 16), (16, 24), (24, 32)) # tile-aligned (tile=4) + + +def _tp_worker(rank, world_size, init_method, result_queue, scenario): + import torch.distributed as dist + + torch.set_num_threads(1) + try: + dist.init_process_group( + backend="gloo", init_method=init_method, rank=rank, world_size=world_size + ) + dtype = torch.bfloat16 if scenario == "bf16" else torch.float32 + dtype_name = "bf16" if scenario == "bf16" else "fp32" + bounds = _UNEVEN_BOUNDS if scenario == "uneven" else _even_bounds(PADDED_VOCAB, world_size) + logits, targets = _inputs(dtype=dtype) + start, end = bounds[rank] + + op = VocabParallelLogprobOp() + tiles = 16 if scenario == "preflight" and rank == 0 else NUM_TILES + contract_tp = _contract( + tp_rank=rank, tp_world_size=world_size, bounds=bounds, dtype=dtype_name + ) + + if scenario == "preflight": + try: + op( + logits[:, start:end].contiguous().clone(), + targets, + contract=contract_tp, + tp_group=dist.group.WORLD, + num_vocab_tiles=tiles, + ) + result_queue.put({"ok": False, "rank": rank, "traceback": "no error raised"}) + except LogprobContractError: + result_queue.put({"ok": True, "rank": rank}) + return + + shard = logits[:, start:end].contiguous().clone().requires_grad_(True) + logp_tp, lse_tp = op( + shard, + targets, + contract=contract_tp, + tp_group=dist.group.WORLD, + num_vocab_tiles=NUM_TILES, + ) + (logp_tp.sum() + 0.5 * lse_tp.sum()).backward() + + # In-process TP=1 run of the same op on the full logits: the cross-TP + # bitwise claim is TP=n output == TP=1 output, bit for bit. + full = logits.clone().requires_grad_(True) + contract_tp1 = _contract(dtype=dtype_name) + logp_one, lse_one = op(full, targets, contract=contract_tp1, num_vocab_tiles=NUM_TILES) + (logp_one.sum() + 0.5 * lse_one.sum()).backward() + + result_queue.put( + { + "ok": True, + "rank": rank, + "logp_bits_match": _bitwise_equal(logp_tp, logp_one), + "lse_bits_match": _bitwise_equal(lse_tp, lse_one), + "grad_bits_match": _bitwise_equal(shard.grad, full.grad[:, start:end]), + "logp": logp_tp.detach().float(), + "lse": lse_tp.detach().float(), + } + ) + except Exception: + result_queue.put({"ok": False, "rank": rank, "traceback": traceback.format_exc()}) + raise + finally: + if torch.distributed.is_initialized(): + torch.distributed.destroy_process_group() + + +def _run_gloo_scenario(scenario): + ctx = mp.get_context("spawn") + with tempfile.TemporaryDirectory() as tmpdir: + init_method = (Path(tmpdir) / "gloo_init").as_uri() + result_queue = ctx.Queue() + processes = [ + ctx.Process( + target=_tp_worker, + args=(rank, _WORLD_SIZE, init_method, result_queue, scenario), + ) + for rank in range(_WORLD_SIZE) + ] + results = [] + try: + for process in processes: + process.start() + for _ in range(_WORLD_SIZE): + try: + results.append(result_queue.get(timeout=60)) + except queue.Empty: + for process in processes: + if process.is_alive(): + process.terminate() + pytest.fail("timed out waiting for vocab-parallel gloo workers") + finally: + for process in processes: + process.join(timeout=10) + if process.is_alive(): + process.terminate() + results.sort(key=lambda item: item["rank"]) + for result in results: + assert result["ok"], result.get("traceback") + for process in processes: + assert process.exitcode == 0 + return results + + +@requires_gloo +@pytest.mark.parametrize("scenario", ["even", "uneven", "bf16"]) +def test_tp4_bitwise_identical_to_tp1(scenario): + results = _run_gloo_scenario(scenario) + for result in results: + assert result["logp_bits_match"], f"rank {result['rank']} logp bits differ from TP=1" + assert result["lse_bits_match"], f"rank {result['rank']} lse bits differ from TP=1" + assert result["grad_bits_match"], f"rank {result['rank']} grad bits differ from TP=1" + # Outputs are replicated: every rank must hold identical bits. + for other in results[1:]: + assert _bitwise_equal(results[0]["logp"], other["logp"]) + assert _bitwise_equal(results[0]["lse"], other["lse"]) + + +@requires_gloo +def test_preflight_rejects_mismatched_num_vocab_tiles(): + _run_gloo_scenario("preflight") From 4fcdc30cded0b25498874a966b38ac75fe31cb13 Mon Sep 17 00:00:00 2001 From: KJLdefeated Date: Wed, 5 Aug 2026 16:10:26 +0800 Subject: [PATCH 14/48] init vocab parallel logp --- tests/test_operator_inputs.py | 26 -------------------------- 1 file changed, 26 deletions(-) diff --git a/tests/test_operator_inputs.py b/tests/test_operator_inputs.py index 2080f6fd..4f742734 100644 --- a/tests/test_operator_inputs.py +++ b/tests/test_operator_inputs.py @@ -40,7 +40,6 @@ def _args(**overrides): "logp", "linear_logp", "batch_invariant_logp", - "vocab_parallel_logp", "rope", "silu", "swiglu", @@ -73,31 +72,6 @@ def test_constant_batch_invariant_logp_inputs_match_operator_contract(): assert torch.equal(inputs["target_ids"], torch.full((1, 2), 3, dtype=torch.long)) -def test_constant_vocab_parallel_logp_inputs_match_operator_contract(): - args = _args(input_mode="constant", constant_value=0.5, token_value=3) - inputs = make_operator_inputs("vocab_parallel_logp", args, torch.float32, torch.device("cpu")) - - # vocab=17 rounds up to padded=20 with 4 tiles; tokens flatten to batch*seq. - assert torch.equal(inputs["local_logits"], torch.full((2, 20), 0.5)) - assert torch.equal(inputs["target_ids"], torch.full((2,), 3, dtype=torch.long)) - assert inputs["contract"].sharding.real_vocab_size == 17 - assert inputs["contract"].sharding.padded_vocab_size == 20 - assert inputs["num_vocab_tiles"] == 4 - assert operator_shape_name("vocab_parallel_logp", args) == "2x20" - - -def test_vocab_parallel_logp_inputs_run_through_the_operator(): - from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import VocabParallelLogprobOp - - args = _args(input_mode="random", seed=7) - inputs = make_operator_inputs("vocab_parallel_logp", args, torch.float32, torch.device("cpu")) - - logp, lse = VocabParallelLogprobOp()(**inputs) - assert logp.shape == inputs["target_ids"].shape - assert logp.dtype == torch.float32 and lse.dtype == torch.float32 - assert torch.isfinite(logp).all() and torch.isfinite(lse).all() - - def test_random_logp_inputs_are_seeded(): args = _args(input_mode="random", seed=7) first = make_operator_inputs("logp", args, torch.float32, torch.device("cpu")) From 6ffade40974e1fba5a1c5fbbd7ad9e6d6c9fe9ad Mon Sep 17 00:00:00 2001 From: KJLdefeated Date: Wed, 5 Aug 2026 21:13:04 +0800 Subject: [PATCH 15/48] adding cross tp testing --- tests/test_vocab_parallel_logp.py | 257 +++++++++++++++++++++--------- 1 file changed, 184 insertions(+), 73 deletions(-) diff --git a/tests/test_vocab_parallel_logp.py b/tests/test_vocab_parallel_logp.py index 510aec70..87da2ef0 100644 --- a/tests/test_vocab_parallel_logp.py +++ b/tests/test_vocab_parallel_logp.py @@ -1,13 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 RL-Kernel Contributors -"""Deterministic vocab-parallel TP logprob reference tests (issue #241 PR3). - -Bit-level determinism assertions compare raw bit patterns via -``tensor.view(torch.int32)`` rather than ``torch.equal``: value equality -treats ``-0.0 == 0.0`` as equal and ``NaN != NaN`` as different, neither of -which is what a bitwise claim means. -""" +"""Deterministic vocab-parallel TP logprob reference tests""" from __future__ import annotations @@ -86,7 +80,11 @@ def _inputs(dtype=torch.float32, seed: int = 2026): def _bits(tensor: torch.Tensor) -> torch.Tensor: - view_dtype = {torch.float32: torch.int32, torch.bfloat16: torch.int16}[tensor.dtype] + view_dtype = { + torch.float32: torch.int32, + torch.bfloat16: torch.int16, + torch.float16: torch.int16, + }[tensor.dtype] return tensor.contiguous().view(view_dtype) @@ -283,72 +281,143 @@ def test_dispatch_resolves_reference_and_leaves_legacy_untouched(): assert OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP not in candidates -# --------------------------------------------------------------------------- -# Multi-rank gloo tests (spawn pattern from tests/test_linear_logp.py) -# --------------------------------------------------------------------------- +# Cross-TP bitwise determinism on real ranks (NCCL, one CUDA device per rank) +TP_REAL_VOCAB = 1000 +TP_PADDED_VOCAB = 1024 +TP_NUM_TILES = 32 # tile = 32 columns +TP_TILE = TP_PADDED_VOCAB // TP_NUM_TILES +TP_NUM_TOKENS = 48 +TP_ACTIVE = tuple(index % 7 != 5 for index in range(TP_NUM_TOKENS)) +TP_DTYPES = {"fp32": torch.float32, "bf16": torch.bfloat16} +_SPAWN_TIMEOUT_S = 300 -def _gloo_available() -> bool: - return torch.distributed.is_available() and torch.distributed.is_gloo_available() +def _cuda_device_count() -> int: + return torch.cuda.device_count() if torch.cuda.is_available() else 0 -requires_gloo = pytest.mark.skipif( - not _gloo_available(), reason="requires torch.distributed with the gloo backend" -) +def _requires_gpus(count: int): + return pytest.mark.skipif( + _cuda_device_count() < count, + reason=f"cross-TP determinism needs {count} CUDA devices to place one rank per device", + ) + + +def _tile_counts(world_size: int, uneven: bool) -> list[int]: + """Tiles per rank; bounds are built from whole tiles so they stay tile-aligned.""" + + counts = [TP_NUM_TILES // world_size for _ in range(world_size)] + counts[-1] += TP_NUM_TILES % world_size + if uneven: + for rank in range(world_size - 1): + if counts[rank] > 1: + counts[rank] -= 1 + counts[-1] += 1 + return counts + + +def _tp_bounds(world_size: int, uneven: bool) -> tuple[tuple[int, int], ...]: + bounds, cursor = [], 0 + for count in _tile_counts(world_size, uneven): + bounds.append((cursor, cursor + count * TP_TILE)) + cursor += count * TP_TILE + return tuple(bounds) + + +def _tp_contract(tp_rank: int, tp_world_size: int, bounds, dtype_name: str) -> LogprobContract: + return _contract( + tp_rank=tp_rank, + tp_world_size=tp_world_size, + bounds=bounds, + real_vocab=TP_REAL_VOCAB, + padded_vocab=TP_PADDED_VOCAB, + num_tokens=TP_NUM_TOKENS, + active=TP_ACTIVE, + dtype=dtype_name, + ) -_WORLD_SIZE = 4 -_UNEVEN_BOUNDS = ((0, 4), (4, 16), (16, 24), (24, 32)) # tile-aligned (tile=4) +def _tp_inputs(device, dtype, seed: int = 2026): + """Identical logits and targets on every rank, seeded on CPU.""" -def _tp_worker(rank, world_size, init_method, result_queue, scenario): + gen = torch.Generator(device="cpu").manual_seed(seed) + logits = torch.randn(TP_NUM_TOKENS, TP_PADDED_VOCAB, generator=gen, dtype=torch.float32) + targets = torch.randint(0, TP_REAL_VOCAB, (TP_NUM_TOKENS,), generator=gen) + active = torch.tensor(TP_ACTIVE) + # Inactive rows carry ignore_index; active_mask stays the sole authority. + targets = torch.where(active, targets, torch.full_like(targets, -100)) + return logits.to(device=device, dtype=dtype), targets.to(device) + + +def _nccl_worker(rank, world_size, init_method, result_queue, scenario, uneven, dtype_name): import torch.distributed as dist - torch.set_num_threads(1) try: + torch.cuda.set_device(rank) + device = torch.device("cuda", rank) dist.init_process_group( - backend="gloo", init_method=init_method, rank=rank, world_size=world_size + backend="nccl", init_method=init_method, rank=rank, world_size=world_size ) - dtype = torch.bfloat16 if scenario == "bf16" else torch.float32 - dtype_name = "bf16" if scenario == "bf16" else "fp32" - bounds = _UNEVEN_BOUNDS if scenario == "uneven" else _even_bounds(PADDED_VOCAB, world_size) - logits, targets = _inputs(dtype=dtype) - start, end = bounds[rank] - + dtype = TP_DTYPES[dtype_name] op = VocabParallelLogprobOp() - tiles = 16 if scenario == "preflight" and rank == 0 else NUM_TILES - contract_tp = _contract( - tp_rank=rank, tp_world_size=world_size, bounds=bounds, dtype=dtype_name - ) - - if scenario == "preflight": + bounds = _tp_bounds(world_size, uneven) + logits, targets = _tp_inputs(device, dtype) + tiles = TP_NUM_TILES + + if scenario in {"preflight", "misaligned"}: + if scenario == "preflight": + if rank == 0: + tiles = TP_NUM_TILES * 2 + else: + # Nudge the first boundary off the tile grid, on every rank. + split = bounds[0][1] + TP_TILE // 4 + bounds = ((0, split), (split, bounds[1][1])) + bounds[2:] + + start, end = bounds[rank] try: op( logits[:, start:end].contiguous().clone(), targets, - contract=contract_tp, + contract=_tp_contract(rank, world_size, bounds, dtype_name), tp_group=dist.group.WORLD, num_vocab_tiles=tiles, ) result_queue.put({"ok": False, "rank": rank, "traceback": "no error raised"}) - except LogprobContractError: - result_queue.put({"ok": True, "rank": rank}) + except LogprobContractError as exc: + result_queue.put({"ok": True, "rank": rank, "message": str(exc)}) return + start, end = bounds[rank] shard = logits[:, start:end].contiguous().clone().requires_grad_(True) + tp_contract = _tp_contract(rank, world_size, bounds, dtype_name) logp_tp, lse_tp = op( shard, targets, - contract=contract_tp, + contract=tp_contract, tp_group=dist.group.WORLD, - num_vocab_tiles=NUM_TILES, + num_vocab_tiles=TP_NUM_TILES, ) (logp_tp.sum() + 0.5 * lse_tp.sum()).backward() - # In-process TP=1 run of the same op on the full logits: the cross-TP - # bitwise claim is TP=n output == TP=1 output, bit for bit. + # Same ranks, same inputs, run again: the collectives must not perturb bits. + rerun = logits[:, start:end].contiguous().clone() + logp_re, lse_re = op( + rerun, + targets, + contract=tp_contract, + tp_group=dist.group.WORLD, + num_vocab_tiles=TP_NUM_TILES, + ) + + # In-process TP=1 run on the full logits: the cross-TP claim is that a + # TP=n result equals the TP=1 result, bit for bit. full = logits.clone().requires_grad_(True) - contract_tp1 = _contract(dtype=dtype_name) - logp_one, lse_one = op(full, targets, contract=contract_tp1, num_vocab_tiles=NUM_TILES) + logp_one, lse_one = op( + full, + targets, + contract=_tp_contract(0, 1, ((0, TP_PADDED_VOCAB),), dtype_name), + num_vocab_tiles=TP_NUM_TILES, + ) (logp_one.sum() + 0.5 * lse_one.sum()).backward() result_queue.put( @@ -358,45 +427,50 @@ def _tp_worker(rank, world_size, init_method, result_queue, scenario): "logp_bits_match": _bitwise_equal(logp_tp, logp_one), "lse_bits_match": _bitwise_equal(lse_tp, lse_one), "grad_bits_match": _bitwise_equal(shard.grad, full.grad[:, start:end]), - "logp": logp_tp.detach().float(), - "lse": lse_tp.detach().float(), + "rerun_bits_match": ( + _bitwise_equal(logp_re, logp_tp) and _bitwise_equal(lse_re, lse_tp) + ), + "logp_bit_pattern": _bits(logp_tp.detach().float().cpu()).tolist(), + "lse_bit_pattern": _bits(lse_tp.detach().float().cpu()).tolist(), } ) - except Exception: + except Exception: # pragma: no cover - forwarded to the parent process result_queue.put({"ok": False, "rank": rank, "traceback": traceback.format_exc()}) raise finally: - if torch.distributed.is_initialized(): - torch.distributed.destroy_process_group() + import torch.distributed as dist + if dist.is_initialized(): + dist.destroy_process_group() -def _run_gloo_scenario(scenario): + +def _run_nccl_scenario(world_size, scenario="correctness", uneven=False, dtype_name="fp32"): ctx = mp.get_context("spawn") with tempfile.TemporaryDirectory() as tmpdir: - init_method = (Path(tmpdir) / "gloo_init").as_uri() + init_method = (Path(tmpdir) / "nccl_init").as_uri() result_queue = ctx.Queue() processes = [ ctx.Process( - target=_tp_worker, - args=(rank, _WORLD_SIZE, init_method, result_queue, scenario), + target=_nccl_worker, + args=(rank, world_size, init_method, result_queue, scenario, uneven, dtype_name), ) - for rank in range(_WORLD_SIZE) + for rank in range(world_size) ] results = [] try: for process in processes: process.start() - for _ in range(_WORLD_SIZE): + for _ in range(world_size): try: - results.append(result_queue.get(timeout=60)) + results.append(result_queue.get(timeout=_SPAWN_TIMEOUT_S)) except queue.Empty: for process in processes: if process.is_alive(): process.terminate() - pytest.fail("timed out waiting for vocab-parallel gloo workers") + pytest.fail(f"timed out waiting for NCCL workers (scenario={scenario})") finally: for process in processes: - process.join(timeout=10) + process.join(timeout=30) if process.is_alive(): process.terminate() results.sort(key=lambda item: item["rank"]) @@ -407,20 +481,57 @@ def _run_gloo_scenario(scenario): return results -@requires_gloo -@pytest.mark.parametrize("scenario", ["even", "uneven", "bf16"]) -def test_tp4_bitwise_identical_to_tp1(scenario): - results = _run_gloo_scenario(scenario) - for result in results: - assert result["logp_bits_match"], f"rank {result['rank']} logp bits differ from TP=1" - assert result["lse_bits_match"], f"rank {result['rank']} lse bits differ from TP=1" - assert result["grad_bits_match"], f"rank {result['rank']} grad bits differ from TP=1" - # Outputs are replicated: every rank must hold identical bits. - for other in results[1:]: - assert _bitwise_equal(results[0]["logp"], other["logp"]) - assert _bitwise_equal(results[0]["lse"], other["lse"]) - - -@requires_gloo -def test_preflight_rejects_mismatched_num_vocab_tiles(): - _run_gloo_scenario("preflight") +class TestCrossTPBitwise: + """TP=n output == TP=1 output, bit for bit, on real NCCL ranks.""" + + @_requires_gpus(2) + @pytest.mark.parametrize("dtype_name", ["fp32", "bf16"]) + @pytest.mark.parametrize("uneven", [False, True], ids=["even", "uneven"]) + def test_tp2_bitwise_identical_to_tp1(self, uneven, dtype_name): + self._assert_matches_tp1(_run_nccl_scenario(2, uneven=uneven, dtype_name=dtype_name)) + + @_requires_gpus(4) + @pytest.mark.parametrize("dtype_name", ["fp32", "bf16"]) + @pytest.mark.parametrize("uneven", [False, True], ids=["even", "uneven"]) + def test_tp4_bitwise_identical_to_tp1(self, uneven, dtype_name): + self._assert_matches_tp1(_run_nccl_scenario(4, uneven=uneven, dtype_name=dtype_name)) + + @staticmethod + def _assert_matches_tp1(results): + for result in results: + rank = result["rank"] + assert result["logp_bits_match"], f"rank {rank} logp bits differ from TP=1" + assert result["lse_bits_match"], f"rank {rank} lse bits differ from TP=1" + assert result["grad_bits_match"], f"rank {rank} grad bits differ from TP=1" + assert result["rerun_bits_match"], f"rank {rank} bits changed between identical runs" + # Outputs are replicated: every rank must hold identical bits. + for other in results[1:]: + assert results[0]["logp_bit_pattern"] == other["logp_bit_pattern"] + assert results[0]["lse_bit_pattern"] == other["lse_bit_pattern"] + + @_requires_gpus(2) + def test_tp2_and_tp4_agree_with_each_other(self): + """The claim is over TP degrees, so pin TP=2 against TP=4 directly.""" + + if _cuda_device_count() < 4: + pytest.skip("needs 4 CUDA devices to compare TP=2 against TP=4") + tp2 = _run_nccl_scenario(2) + tp4 = _run_nccl_scenario(4) + assert tp2[0]["logp_bit_pattern"] == tp4[0]["logp_bit_pattern"] + assert tp2[0]["lse_bit_pattern"] == tp4[0]["lse_bit_pattern"] + + +class TestCrossTPGuards: + """A disagreement must abort loudly on every rank, not strand ranks in a collective.""" + + @_requires_gpus(2) + def test_preflight_rejects_mismatched_num_vocab_tiles(self): + results = _run_nccl_scenario(2, scenario="preflight") + for result in results: + assert "cross-rank preflight failed" in result["message"] + + @_requires_gpus(2) + def test_misaligned_shard_bounds_rejected(self): + results = _run_nccl_scenario(2, scenario="misaligned") + for result in results: + assert "not aligned to the vocab tile size" in result["message"] From 4231625fbd3c436affc7cf701d8f8c1459cf8ec4 Mon Sep 17 00:00:00 2001 From: KJLdefeated Date: Thu, 6 Aug 2026 09:31:02 +0800 Subject: [PATCH 16/48] fix comment --- rl_engine/kernels/logprob_contract.py | 3 +++ rl_engine/kernels/registry.py | 5 +++++ 2 files changed, 8 insertions(+) diff --git a/rl_engine/kernels/logprob_contract.py b/rl_engine/kernels/logprob_contract.py index b0bcd7ab..1267740a 100644 --- a/rl_engine/kernels/logprob_contract.py +++ b/rl_engine/kernels/logprob_contract.py @@ -486,6 +486,9 @@ def cross_rank_fingerprint(self) -> str: for key, value in payload["sharding"].items() if key not in {"tp_rank", "cp_rank", "local_vocab_start", "local_vocab_end"} } + # Note: Any future extensions to this payload MUST maintain strict JSON + # serialization determinism across environments to prevent cross-rank + # hashing mismatches. encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")) return hashlib.sha256(encoded.encode("utf-8")).hexdigest() diff --git a/rl_engine/kernels/registry.py b/rl_engine/kernels/registry.py index 7093b320..60f92527 100644 --- a/rl_engine/kernels/registry.py +++ b/rl_engine/kernels/registry.py @@ -510,6 +510,11 @@ def get_logprob_op( "determinism through ReductionSpec.determinism_scope and match it against " "backend determinism_scopes instead" ) + if requested_backend.strip().lower() == "auto" and contract.sharding.tp_world_size > 1: + raise LogprobContractError( + "Unsafe dispatch: requested_backend='auto' is not permitted when tp_world_size > 1 " + "without explicit cross-rank preflighting." + ) platform = self._platform() candidates = self._logprob_candidates.get(platform, []) From 2360a7106de7b6e42a4aee89f0ae4a7ac7709ad0 Mon Sep 17 00:00:00 2001 From: KJLdefeated Date: Fri, 7 Aug 2026 20:31:56 +0800 Subject: [PATCH 17/48] test(ws2): align dispatch tests with the auto+TP>1 unsafe-dispatch guard The guard added per review rejects requested_backend="auto" whenever tp_world_size > 1, so TP-sharded dispatch tests now name an explicit policy and auto-policy tests use TP=1 contracts. Add coverage for the guard itself and document the restriction in get_logprob_op. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_012PyjQEqDJwy9Cos4Sb9QBK --- rl_engine/kernels/registry.py | 7 +++++-- tests/test_logprob_contract.py | 32 ++++++++++++++++++++++++++------ 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/rl_engine/kernels/registry.py b/rl_engine/kernels/registry.py index 60f92527..32cfdc37 100644 --- a/rl_engine/kernels/registry.py +++ b/rl_engine/kernels/registry.py @@ -496,7 +496,10 @@ def get_logprob_op( (``auto`` | ``production`` | ``reference`` | ``deterministic``) or an exact, case-sensitive stable backend id. Strictness comes from the contract's capability checks, not from this policy string, so the - default is ``auto``. + default is ``auto``. With ``tp_world_size > 1``, ``auto`` is rejected: + per-rank auto resolution can diverge across ranks, so distributed + callers must name a policy or backend id and preflight agreement via + ``LogprobContract.cross_rank_fingerprint``. """ if not isinstance(contract, LogprobContract): @@ -510,7 +513,7 @@ def get_logprob_op( "determinism through ReductionSpec.determinism_scope and match it against " "backend determinism_scopes instead" ) - if requested_backend.strip().lower() == "auto" and contract.sharding.tp_world_size > 1: + if requested_backend.lower() == "auto" and contract.sharding.tp_world_size > 1: raise LogprobContractError( "Unsafe dispatch: requested_backend='auto' is not permitted when tp_world_size > 1 " "without explicit cross-rank preflighting." diff --git a/tests/test_logprob_contract.py b/tests/test_logprob_contract.py index 40cea4e2..42dc6dd1 100644 --- a/tests/test_logprob_contract.py +++ b/tests/test_logprob_contract.py @@ -251,7 +251,7 @@ def test_current_ws1_backend_rejects_strict_tp_contract_without_fallback(): registry = KernelRegistry() with pytest.raises(RuntimeError) as exc_info: - registry.get_logprob_op(_contract()) + registry.get_logprob_op(_contract(), requested_backend="reference") message = str(exc_info.value) assert "TP=2 is unsupported" in message @@ -277,8 +277,9 @@ def test_undeclared_backend_capability_is_never_selected(): platform = registry._platform() registry._logprob_candidates[platform] = [OpBackend.PYTORCH_NATIVE] + tp1_contract = _contract(sharding=_sharding(tp_world_size=1, cp_world_size=1)) with pytest.raises(RuntimeError, match="no LogprobBackendCapability declared"): - registry.get_logprob_op(_contract()) + registry.get_logprob_op(tp1_contract) def test_declared_compatible_backend_resolves_and_records_provenance(): @@ -354,7 +355,8 @@ def test_default_auto_policy_resolves_any_compatible_implementation_kind(): platform=platform, ) - result = registry.get_logprob_op(_contract()) + tp1_contract = _contract(sharding=_sharding(tp_world_size=1, cp_world_size=1)) + result = registry.get_logprob_op(tp1_contract) assert result.provenance["requested_backend"] == "auto" assert result.capability.implementation_kind == "reference" @@ -407,7 +409,7 @@ def test_capability_rejections_are_reported_as_fallback(): OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, _declared_tp_backend(), platform=platform ) - result = registry.get_logprob_op(_contract()) + result = registry.get_logprob_op(_contract(), requested_backend="reference") assert result.provenance["fallback"] is True assert "TP=2 is unsupported" in result.provenance["prior_rejections"][0] @@ -441,7 +443,7 @@ def test_register_logprob_backend_is_the_public_registration_seam(): ) assert registry._logprob_candidates[platform] == [OpBackend.PYTORCH_BATCH_INVARIANT_LOGP] - result = registry.get_logprob_op(_contract()) + result = registry.get_logprob_op(_contract(), requested_backend="reference") assert result.capability.backend_id == "replacement-backend" with pytest.raises(LogprobContractError, match="capability must be"): @@ -467,7 +469,7 @@ def test_capabilities_are_scoped_per_platform(): platform=other, ) - result = registry.get_logprob_op(_contract()) + result = registry.get_logprob_op(_contract(), requested_backend="reference") assert result.capability.backend_id == "test-deterministic-tp-logprob" assert ( @@ -502,6 +504,24 @@ def test_requested_deterministic_policy_is_a_loud_error(): registry.get_logprob_op(_contract(), requested_backend="deterministic") +def test_auto_policy_is_rejected_for_tp_sharded_contracts(): + registry = KernelRegistry() + platform = registry._platform() + registry._logprob_candidates[platform] = [] + registry.register_logprob_backend( + OpBackend.PYTORCH_BATCH_INVARIANT_LOGP, _declared_tp_backend(), platform=platform + ) + + with pytest.raises(LogprobContractError, match="Unsafe dispatch"): + registry.get_logprob_op(_contract()) + + with pytest.raises(LogprobContractError, match="Unsafe dispatch"): + registry.get_logprob_op(_contract(), requested_backend=" AUTO ") + + tp1_contract = _contract(sharding=_sharding(tp_world_size=1, cp_world_size=1)) + assert registry.get_logprob_op(tp1_contract).provenance["requested_backend"] == "auto" + + def test_determinism_scope_is_part_of_the_typed_contract(): fixed_only = replace( _declared_tp_backend(), From 4eebb3bd1dd6be2f8c1ef2f2ae54ce80973132ce Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Sat, 8 Aug 2026 17:37:29 +0800 Subject: [PATCH 18/48] refactor: colocate logprob harness tooling and docs --- docs/design/ws2-logprob-single-gpu-harness.md | 95 ------------- docs/design/ws2-logprob-sm90-validation.md | 134 ------------------ docs/operators/batch-invariant-logp.md | 103 +++++++++++++- rl_engine/testing/logprob_comparison.py | 94 ++++++++++++ scripts/compare_logprob.py | 103 -------------- tests/test_logprob_comparison.py | 32 ++++- 6 files changed, 226 insertions(+), 335 deletions(-) delete mode 100644 docs/design/ws2-logprob-single-gpu-harness.md delete mode 100644 docs/design/ws2-logprob-sm90-validation.md delete mode 100644 scripts/compare_logprob.py diff --git a/docs/design/ws2-logprob-single-gpu-harness.md b/docs/design/ws2-logprob-single-gpu-harness.md deleted file mode 100644 index c5b869b0..00000000 --- a/docs/design/ws2-logprob-single-gpu-harness.md +++ /dev/null @@ -1,95 +0,0 @@ -# WS2 Single-GPU Logprob Comparison Harness - -This harness is the TP=1 registration and regression guard for issue #241. It compares -selected-token logprob implementations before any distributed communication is introduced. - -## Contract - -For each logical token row, every backend returns direct FP32 values: - -```text -LSE = logsumexp(logits[..., vocab]) -logp = selected_logit - LSE -``` - -The harness uses the merged WS1 batch-invariant PyTorch implementation as its reference. -Reference logp is obtained through the unchanged production call, while reference LSE is -obtained through the diagnostic entry point. The TP=1 PyTorch candidate follows the same -core computation and must be bitwise equal. This is a regression guard, not new -distributed mathematics. - -LSE drift is reported over every logical token row. Selected-token dlogp drift is reported -only over active response/action tokens. Both reports contain max, mean, p95, p99, and the -number of compared values. - -## Exact Backend Selection - -Supported backend names are: - -- `pytorch` -- `triton` -- `cuda-sm90` - -The comparison path does not use registry fallback. An explicitly requested backend must -run exactly or raise `LogprobBackendUnavailable`. In particular, `cuda-sm90` requires a -compiled SM90 extension, Hopper hardware, BF16/FP32 logits, and a compatible vocab row -stride. The production operator may retain its normal fallback behavior outside the -harness. - -Each backend exposes a diagnostic-only `forward_with_lse` method. Existing production -calls remain unchanged: - -```text -op(logits, target_ids) -> logp -op.forward_with_lse(logits, target_ids) -> (logp, lse) -``` - -## Usage - -CPU TP=1 regression guard: - -```bash -python scripts/compare_logprob.py \ - --candidate pytorch \ - --device cpu \ - --dtype fp32 \ - --batch 2 \ - --seq 16 \ - --vocab 257 -``` - -GPU comparison: - -```bash -python scripts/compare_logprob.py \ - --candidate triton \ - --candidate cuda-sm90 \ - --device cuda \ - --dtype bf16 \ - --batch 2 \ - --seq 16 \ - --vocab 151936 -``` - -The command prints a structured JSON report to stdout containing input dtype/shape, -active-token count, TP world size, communication mode, requested and actual backends, -direct-LSE provenance, bitwise logp status, and LSE/dlogp drift statistics. Backend -diagnostic logs are routed to stderr so redirected stdout remains valid JSON. - -## Scope Boundary - -This harness is intentionally single-GPU and records `tp_world=1` and -`communication=none`. It does not implement vocab-shard metadata, all-gather transport, -fixed-order cross-rank LSE merging, CP reconstruction, or distributed artifacts. Those -belong to the later PR3 and PR4 work in issue #241. - -## Tests - -```bash -python -m pytest tests/test_logprob_comparison.py -q -``` - -The focused tests cover bitwise TP=1 regression, direct LSE identity, active-token-only -percentiles, zero active tokens, invalid ignore-index usage, structured serialization, -generic operator-harness registration, exact GPU backend diagnostics, and fail-closed -backend provenance. diff --git a/docs/design/ws2-logprob-sm90-validation.md b/docs/design/ws2-logprob-sm90-validation.md deleted file mode 100644 index e94cb52d..00000000 --- a/docs/design/ws2-logprob-sm90-validation.md +++ /dev/null @@ -1,134 +0,0 @@ -# WS2 Logprob PR2 SM90 Validation - -This document records the Hopper SM90 validation procedure for the PR2 single-GPU -logprob comparison harness from issue #241. It is a validation note for maintainers; -the cloud setup wrapper used during development is intentionally kept outside the -repository. - -## Prerequisites - -The validation host must provide: - -- Python 3.10 or newer; -- CUDA-enabled PyTorch; -- an NVIDIA Hopper GPU with compute capability 9.0, such as H100, H800, or H200; -- `nvidia-smi` and `nvcc`; -- a CUDA development environment capable of compiling the RL-Kernel extension. - -The CUDA version reported by `nvcc` must match `torch.version.cuda`. A runtime-only -image is insufficient because it normally does not include the CUDA compiler. - -## Build - -Activate an environment containing the repository dependencies and a CUDA-enabled -PyTorch installation, then build the editable extension with SM90 enabled: - -```bash -export FORCE_CUDA=1 -export KERNEL_ALIGN_FORCE_SM90=1 -export TORCH_CUDA_ARCH_LIST="9.0+PTX" -export MAX_JOBS=2 - -python -m pip install --no-build-isolation --no-deps -e . -``` - -Verify the extension and SM90 symbol after the build. Import PyTorch first so its -runtime libraries are available to the extension loader: - -```bash -python - <<'PY' -import torch -from rl_engine import _C - -print("torch:", torch.__version__) -print("torch CUDA:", torch.version.cuda) -print("extension:", _C.__file__) -print("SM90 symbol:", hasattr(_C, "batch_invariant_logp_sm90")) -PY -``` - -The final line must report `SM90 symbol: True`. - -## Validation commands - -Run the focused PR2 tests and the complete batch-invariant logprob suite: - -```bash -python -m pytest \ - tests/test_logprob_comparison.py \ - tests/test_operator_inputs.py \ - tests/test_op_checks.py -q - -python -m pytest tests/test_batch_invariant_logp.py -q -``` - -Run the two explicit SM90 comparisons: - -```bash -python scripts/compare_logprob.py \ - --candidate cuda-sm90 \ - --device cuda \ - --dtype bf16 \ - --batch 2 \ - --seq 8 \ - --vocab 1024 \ - --prompt-tokens 3 \ - --seed 7 - -python scripts/compare_logprob.py \ - --candidate cuda-sm90 \ - --device cuda \ - --dtype bf16 \ - --batch 2 \ - --seq 16 \ - --vocab 151936 \ - --prompt-tokens 8 \ - --seed 241 -``` - -The comparison command writes the JSON report to stdout. Diagnostic log messages are -written to stderr so stdout can be redirected directly to a `.json` file. - -## Expected report - -The report must identify the requested and actual backend as `cuda-sm90`, use the -`BatchInvariantLogpSM90Op` implementation, and record: - -```text -tp_world=1 -communication=none -lse_source=direct -``` - -LSE drift is measured over all logical token rows. Selected-logprob drift is measured -only over active response/action tokens. Each drift section includes maximum, mean, -p95, p99, and active-count values. - -## Validation result - -The procedure was validated on: - -```text -GPU: NVIDIA H800 PCIe -Compute capability: 9.0 -Python: 3.11.15 -PyTorch: 2.11.0+cu128 -CUDA toolkit / nvcc: 12.8 -Triton: 3.6.0 -``` - -Results: - -```text -PR2 focused tests: 41 passed -Complete batch-invariant logprob suite: 67 passed -``` - -Observed BF16 SM90 drift against the PyTorch reference: - -| Shape | LSE max abs | dlogp max abs | -| --- | ---: | ---: | -| `[2, 8, 1024]` | `4.76837158203125e-07` | `4.76837158203125e-07` | -| `[2, 16, 151936]` | `9.5367431640625e-07` | `9.5367431640625e-07` | - -Both runs used TP=1, no communication, and the explicit SM90 backend without fallback. diff --git a/docs/operators/batch-invariant-logp.md b/docs/operators/batch-invariant-logp.md index fbc0e9f1..d8e95616 100644 --- a/docs/operators/batch-invariant-logp.md +++ b/docs/operators/batch-invariant-logp.md @@ -179,6 +179,101 @@ fp16/bf16 backward: checked against fp32 reference with relaxed tolerance CPU-vs-CUDA comparisons use tolerance-based checks; batch-invariance checks within the same backend use exact equality where appropriate. +## TP=1 Comparison Harness + +The single-GPU comparison harness is the TP=1 registration and regression guard +for issue #241. It uses the batch-invariant PyTorch implementation as the +reference and compares exact `pytorch`, `triton`, or `cuda-sm90` backends before +distributed communication is introduced. + +Each backend exposes a diagnostic-only entry point while the production contract +remains unchanged: + +```text +op(logits, target_ids) -> logp +op.forward_with_lse(logits, target_ids) -> (logp, lse) +``` + +The harness reports LSE drift over every logical token row and selected-logprob +drift over active response/action tokens only. Drift summaries contain max, +mean, p95, p99, and the number of compared values. Reports also record requested +and actual backends, implementation, direct-LSE provenance, input shape and +dtype, `tp_world=1`, and `communication=none`. + +Backend selection is exact and does not use registry fallback. In particular, +an explicit `cuda-sm90` comparison fails unless the compiled SM90 extension, +Hopper hardware, input dtype, and vocab row stride satisfy the kernel contract. + +Run the PyTorch TP=1 guard directly from the kernel-specific testing module: + +```bash +python rl_engine/testing/logprob_comparison.py \ + --candidate pytorch \ + --device cpu \ + --dtype fp32 \ + --batch 2 \ + --seq 16 \ + --vocab 257 +``` + +On a GPU, repeat `--candidate` to compare multiple exact backends: + +```bash +python rl_engine/testing/logprob_comparison.py \ + --candidate triton \ + --candidate cuda-sm90 \ + --device cuda \ + --dtype bf16 \ + --batch 2 \ + --seq 16 \ + --vocab 151936 +``` + +The command writes structured JSON to stdout and routes backend diagnostics to +stderr. The harness does not implement vocab sharding, collective communication, +cross-rank LSE merging, or CP reconstruction. + +### SM90 validation + +SM90 validation requires a Hopper GPU, CUDA-enabled PyTorch, and an `nvcc` +toolkit matching `torch.version.cuda`. Build the extension with: + +```bash +export FORCE_CUDA=1 +export KERNEL_ALIGN_FORCE_SM90=1 +export TORCH_CUDA_ARCH_LIST="9.0+PTX" + +python -m pip install --no-build-isolation --no-deps -e . +``` + +Run the focused harness tests, the complete operator suite, and an explicit +SM90 comparison: + +```bash +python -m pytest \ + tests/test_logprob_comparison.py \ + tests/test_operator_inputs.py \ + tests/test_op_checks.py -q + +python -m pytest tests/test_batch_invariant_logp.py -q + +python rl_engine/testing/logprob_comparison.py \ + --candidate cuda-sm90 \ + --device cuda \ + --dtype bf16 \ + --batch 2 \ + --seq 16 \ + --vocab 151936 \ + --prompt-tokens 8 \ + --seed 241 +``` + +The PR2 path was validated on an NVIDIA H800 PCIe with PyTorch 2.11.0+cu128, +CUDA 12.8, and Triton 3.6.0. The focused tests passed 41 cases and the complete +batch-invariant suite passed 67 cases. For BF16 shape `[2, 16, 151936]`, both +LSE and active-token dlogp had maximum absolute drift +`9.5367431640625e-07` against the PyTorch reference, with no backend fallback. + ## Minimal Example ```python @@ -206,11 +301,14 @@ out.sum().backward() python -m pytest tests/test_batch_invariant_logp.py -q -rs ``` -All backends (Native, Triton) are tested in a single file. Coverage includes: +All production backends are tested in a single file. Coverage includes correctness, leading-shape preservation, batch-invariance (bitwise), validation, ignore-index behavior, backward correctness, CUDA smoke cases, registry dispatch, and Triton-specific fp32/fp16/bf16 correctness, large vocab, backward -gradient batch-invariance, and ignored-row zero gradients. +gradient batch-invariance, and ignored-row zero gradients. The focused +`tests/test_logprob_comparison.py` suite covers TP=1 bitwise regression, direct +LSE identity, active-token drift statistics, structured serialization, exact +backend diagnostics, and fail-closed provenance. Triton tests skip when Triton or CUDA is unavailable. On Windows, run via WSL/Linux with CUDA. @@ -223,4 +321,5 @@ WSL/Linux with CUDA. - `csrc/cuda/batch_invariant_logp_kernel_sm90.cu` - `rl_engine/kernels/registry.py` - `tests/test_batch_invariant_logp.py` +- `tests/test_logprob_comparison.py` - `benchmarks/benchmark_batch_invariant_logp.py` diff --git a/rl_engine/testing/logprob_comparison.py b/rl_engine/testing/logprob_comparison.py index 52ca82d7..d207f75b 100644 --- a/rl_engine/testing/logprob_comparison.py +++ b/rl_engine/testing/logprob_comparison.py @@ -5,12 +5,22 @@ from __future__ import annotations +import argparse +import json +import logging +import pathlib +import sys from collections.abc import Callable, Sequence from dataclasses import asdict, dataclass, field from typing import Any import torch +if __package__ in (None, ""): + repo_root = pathlib.Path(__file__).resolve().parents[2] + if str(repo_root) not in sys.path: + sys.path.insert(0, str(repo_root)) + class LogprobBackendUnavailable(RuntimeError): pass @@ -277,6 +287,86 @@ def _validate_inputs( return active, effective +def _dtype(name: str) -> torch.dtype: + return { + "fp32": torch.float32, + "bf16": torch.bfloat16, + "fp16": torch.float16, + }[name] + + +def _device(name: str) -> torch.device: + if name == "auto": + return torch.device("cuda" if torch.cuda.is_available() else "cpu") + return torch.device(name) + + +def _route_rl_kernel_logs_to_stderr() -> None: + from rl_engine.utils.logger import logger + + for handler in logger.handlers: + if isinstance(handler, logging.StreamHandler): + handler.setStream(sys.stderr) + + +def _parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Run the WS2 TP=1 selected-logprob/LSE comparison harness." + ) + parser.add_argument( + "--candidate", + action="append", + choices=("pytorch", "triton", "cuda-sm90"), + help="Exact backend to compare. Repeat for multiple backends; defaults to pytorch.", + ) + parser.add_argument("--device", default="auto") + parser.add_argument("--dtype", choices=("fp32", "bf16", "fp16"), default="fp32") + parser.add_argument("--batch", type=int, default=2) + parser.add_argument("--seq", type=int, default=16) + parser.add_argument("--vocab", type=int, default=257) + parser.add_argument("--prompt-tokens", type=int, default=8) + parser.add_argument("--seed", type=int, default=123) + return parser.parse_args(argv) + + +def main(argv: Sequence[str] | None = None) -> None: + _route_rl_kernel_logs_to_stderr() + args = _parse_args(argv) + device = _device(args.device) + if args.batch < 1 or args.seq < 1 or args.vocab < 1: + raise ValueError("batch, seq, and vocab must be positive") + if not 0 <= args.prompt_tokens <= args.seq: + raise ValueError("prompt-tokens must be in [0, seq]") + + generator = torch.Generator(device=device).manual_seed(args.seed) + logits = torch.randn( + args.batch, + args.seq, + args.vocab, + generator=generator, + device=device, + dtype=_dtype(args.dtype), + ) + target_ids = torch.randint( + 0, + args.vocab, + (args.batch, args.seq), + generator=generator, + device=device, + ) + active_mask = torch.ones((args.batch, args.seq), device=device, dtype=torch.bool) + active_mask[:, : args.prompt_tokens] = False + report = compare_single_gpu_logprob( + LogprobComparisonInputs( + logits=logits, + target_ids=target_ids, + active_token_mask=active_mask, + ), + candidates=tuple(args.candidate or ("pytorch",)), + ) + print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) + + __all__ = [ "LogprobBackendUnavailable", "LogprobCandidate", @@ -285,3 +375,7 @@ def _validate_inputs( "compare_single_gpu_logprob", "make_logprob_candidate", ] + + +if __name__ == "__main__": + main() diff --git a/scripts/compare_logprob.py b/scripts/compare_logprob.py deleted file mode 100644 index b4736db7..00000000 --- a/scripts/compare_logprob.py +++ /dev/null @@ -1,103 +0,0 @@ -#!/usr/bin/env python -# SPDX-License-Identifier: Apache-2.0 -# Copyright (c) 2026 RL-Kernel Contributors - -from __future__ import annotations - -import argparse -import json -import logging -import pathlib -import sys - -import torch - -REPO_ROOT = pathlib.Path(__file__).resolve().parents[1] -if str(REPO_ROOT) not in sys.path: - sys.path.insert(0, str(REPO_ROOT)) - -from rl_engine.testing import LogprobComparisonInputs, compare_single_gpu_logprob # noqa: E402 -from rl_engine.utils.logger import logger # noqa: E402 - - -def _dtype(name: str) -> torch.dtype: - return { - "fp32": torch.float32, - "bf16": torch.bfloat16, - "fp16": torch.float16, - }[name] - - -def _device(name: str) -> torch.device: - if name == "auto": - return torch.device("cuda" if torch.cuda.is_available() else "cpu") - return torch.device(name) - - -def _route_rl_kernel_logs_to_stderr() -> None: - """Keep stdout machine-readable while preserving backend diagnostics.""" - for handler in logger.handlers: - if isinstance(handler, logging.StreamHandler): - handler.setStream(sys.stderr) - - -def parse_args() -> argparse.Namespace: - parser = argparse.ArgumentParser( - description="Run the WS2 TP=1 selected-logprob/LSE comparison harness." - ) - parser.add_argument( - "--candidate", - action="append", - choices=("pytorch", "triton", "cuda-sm90"), - help="Exact backend to compare. Repeat for multiple backends; defaults to pytorch.", - ) - parser.add_argument("--device", default="auto") - parser.add_argument("--dtype", choices=("fp32", "bf16", "fp16"), default="fp32") - parser.add_argument("--batch", type=int, default=2) - parser.add_argument("--seq", type=int, default=16) - parser.add_argument("--vocab", type=int, default=257) - parser.add_argument("--prompt-tokens", type=int, default=8) - parser.add_argument("--seed", type=int, default=123) - return parser.parse_args() - - -def main() -> None: - _route_rl_kernel_logs_to_stderr() - args = parse_args() - device = _device(args.device) - if args.batch < 1 or args.seq < 1 or args.vocab < 1: - raise ValueError("batch, seq, and vocab must be positive") - if not 0 <= args.prompt_tokens <= args.seq: - raise ValueError("prompt-tokens must be in [0, seq]") - - generator = torch.Generator(device=device).manual_seed(args.seed) - logits = torch.randn( - args.batch, - args.seq, - args.vocab, - generator=generator, - device=device, - dtype=_dtype(args.dtype), - ) - target_ids = torch.randint( - 0, - args.vocab, - (args.batch, args.seq), - generator=generator, - device=device, - ) - active_mask = torch.ones((args.batch, args.seq), device=device, dtype=torch.bool) - active_mask[:, : args.prompt_tokens] = False - report = compare_single_gpu_logprob( - LogprobComparisonInputs( - logits=logits, - target_ids=target_ids, - active_token_mask=active_mask, - ), - candidates=tuple(args.candidate or ("pytorch",)), - ) - print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) - - -if __name__ == "__main__": - main() diff --git a/tests/test_logprob_comparison.py b/tests/test_logprob_comparison.py index d4fece0a..98f976f9 100644 --- a/tests/test_logprob_comparison.py +++ b/tests/test_logprob_comparison.py @@ -5,6 +5,7 @@ import io import json import logging +import subprocess import sys import pytest @@ -17,11 +18,12 @@ LogprobBackendUnavailable, LogprobCandidate, LogprobComparisonInputs, + _device, + _route_rl_kernel_logs_to_stderr, compare_single_gpu_logprob, make_logprob_candidate, ) from rl_engine.utils.logger import logger -from scripts.compare_logprob import _device, _route_rl_kernel_logs_to_stderr def _inputs() -> LogprobComparisonInputs: @@ -196,6 +198,34 @@ def test_cli_routes_rl_kernel_logs_to_stderr_for_machine_readable_stdout(monkeyp assert "test backend diagnostic" in stderr.getvalue() +def test_cli_runs_directly_from_testing_module(): + result = subprocess.run( + [ + sys.executable, + "rl_engine/testing/logprob_comparison.py", + "--candidate", + "pytorch", + "--device", + "cpu", + "--batch", + "1", + "--seq", + "2", + "--vocab", + "17", + "--prompt-tokens", + "1", + ], + check=True, + capture_output=True, + text=True, + ) + + payload = json.loads(result.stdout) + assert payload["drifts"][0]["provenance"]["actual_backend"] == "pytorch" + assert payload["input_provenance"]["communication"] == "none" + + def test_operator_comparison_specs_register_batch_invariant_logp(): args = argparse.Namespace( op="batch_invariant_logp", From 19488cc7baf127be2b5ce7820223a51c5a556fe5 Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Sat, 8 Aug 2026 17:51:35 +0800 Subject: [PATCH 19/48] test: resolve logprob CLI path reliably --- tests/test_logprob_comparison.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/test_logprob_comparison.py b/tests/test_logprob_comparison.py index 98f976f9..4fc62a13 100644 --- a/tests/test_logprob_comparison.py +++ b/tests/test_logprob_comparison.py @@ -7,6 +7,7 @@ import logging import subprocess import sys +from pathlib import Path import pytest import torch @@ -199,10 +200,11 @@ def test_cli_routes_rl_kernel_logs_to_stderr_for_machine_readable_stdout(monkeyp def test_cli_runs_directly_from_testing_module(): + script = Path(__file__).resolve().parents[1] / "rl_engine" / "testing" / "logprob_comparison.py" result = subprocess.run( [ sys.executable, - "rl_engine/testing/logprob_comparison.py", + str(script), "--candidate", "pytorch", "--device", From b7d9d895580bc60558bdacadd097baa8e7b676d4 Mon Sep 17 00:00:00 2001 From: KJLdefeated Date: Wed, 5 Aug 2026 16:05:57 +0800 Subject: [PATCH 20/48] init vocab parallel logp --- .github/workflows/ci.yml | 3 + docs/operators/batch-invariant-logp.md | 36 +- .../ops/pytorch/loss/vocab_parallel_logp.py | 408 +++++++++++++++++ rl_engine/kernels/registry.py | 27 ++ tests/test_logprob_contract.py | 32 ++ tests/test_operator_inputs.py | 26 ++ tests/test_vocab_parallel_logp.py | 426 ++++++++++++++++++ 7 files changed, 948 insertions(+), 10 deletions(-) create mode 100644 rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py create mode 100644 tests/test_vocab_parallel_logp.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 28cdb58d..2d9ba91e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,6 +79,9 @@ jobs: - name: Run WS2 Logprob Contract Tests (CPU-safe) run: python -m pytest tests/test_logprob_contract.py -v + - name: Run WS2 Vocab-Parallel Logprob Tests (CPU-safe) + run: python -m pytest tests/test_vocab_parallel_logp.py -v + docs: runs-on: ubuntu-latest steps: diff --git a/docs/operators/batch-invariant-logp.md b/docs/operators/batch-invariant-logp.md index c4d553c5..3c2fdfc8 100644 --- a/docs/operators/batch-invariant-logp.md +++ b/docs/operators/batch-invariant-logp.md @@ -54,18 +54,31 @@ CUDA priority list when the extension exposes `_C.batch_invariant_logp_sm90` (built with `KERNEL_ALIGN_FORCE_SM90=1`) on an SM90 device. On any other build or device, dispatch is unchanged (Triton -> PyTorch). -### WS2 TP-aware dispatch +## Tensor Parallel -WS2 distributed callers use a separate contract-aware entry point, -`kernel_registry.get_logprob_op(contract)`. It validates explicit vocab-shard ownership, -padded-vs-real vocab metadata, active-token masking, and fixed `(max, sumexp)` merge -semantics before selecting a backend. Legacy `get_op("batch_invariant_logp")` behavior -remains unchanged. +`VocabParallelLogprobOp` +(`rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py`) +**TP=1, TP=2, and TP=4 produce bit-identical results.** -The backends above are single-shard (TP=1) references and do not yet export vocab-domain -LSE or carry vocab-shard metadata, so they are declared incompatible with strict WS2 -requests instead of being selected as a silent fallback. The contract objects are -documented in `rl_engine.kernels.logprob_contract`. +1. Split the padded vocabulary into `num_vocab_tiles` fixed tiles. +2. Each rank computes fp32 `(max, sumexp)` for the tiles it owns. Every tile + is reduced as the same contiguous `[n, tile]` shape, on any rank. +3. All tile partials are shared with `all_gather`. The collective only moves + bytes; it never does math, so it cannot round anything. +4. Every rank merges all tiles in the same fixed order, over the same + `[n, num_vocab_tiles]` shape. `LSE = M + log(sum(s_t * exp(m_t - M)))`. +5. The target logit is copied from the rank that owns it (never summed). +6. `logp = target_logit - LSE`. Inactive rows become `0.0`. + +Usage goes through the contract-aware entry point: + +```python +from rl_engine.kernels.registry import kernel_registry + +result = kernel_registry.get_logprob_op(contract) # LogprobContract from +op = result.op # rl_engine.kernels.logprob_contract +logp, lse = op(local_logits, target_ids, contract=contract, tp_group=tp_group) +``` ## Benchmarks @@ -336,3 +349,6 @@ WSL/Linux with CUDA. - `tests/test_batch_invariant_logp.py` - `tests/test_logprob_comparison.py` - `benchmarks/benchmark_batch_invariant_logp.py` +- `rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py` +- `rl_engine/kernels/logprob_contract.py` +- `tests/test_vocab_parallel_logp.py` diff --git a/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py b/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py new file mode 100644 index 00000000..06eded1c --- /dev/null +++ b/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py @@ -0,0 +1,408 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Deterministic vocab-parallel TP selected-token logprob reference (issue #241 PR3). + +Implements the WS2 contract in ``rl_engine.kernels.logprob_contract`` with a +TP-independent vocab tile decomposition: the padded vocabulary is split into +``num_vocab_tiles`` fixed tiles, every tile's fp32 ``(max, sumexp)`` partial is +computed from a contiguous ``[n, tile]`` tensor, all tile partials travel by +all-gather (transport only), and every rank merges them in global tile-index +order over a fixed ``[n, num_vocab_tiles]`` shape. The TP degree only decides +which rank computes which tiles and never changes any floating-point grouping, +so outputs and gradients are bitwise-identical across TP degrees +(``DeterminismScope.CROSS_TP_BITWISE``) as long as ``num_vocab_tiles`` is held +fixed. A fixed per-shard merge order alone cannot provide this property: +shard boundaries would regroup the combines differently at each degree. + +Consequences of the tile structure: + +- ``num_vocab_tiles`` is part of the numerical identity. It must be pinned + across ranks (enforced by the preflight) and across the TP degrees being + compared; it is never derived from the shard layout. +- Every shard boundary must be tile-aligned; misalignment fails loudly. +- At TP=1 the result matches the WS1 ``NativeBatchInvariantLogpOp`` only + within the #108 logprob tolerance, not bitwise — the WS1 op reduces the + whole ``[n, V]`` row at once, which groups the sums differently. + +Preconditions: logits over the real vocabulary must be finite. A row whose +real-vocab logits are all ``-inf`` has no finite logsumexp; with +``validate=True`` such a row fails loudly if it is active. + +The selected logprob is zero-filled at inactive rows (``MaskSpec.active_mask`` +is the sole authority; with validation enabled an active row can never legally +hold ``ignore_index``). The vocab-domain LSE is returned for every row and is +differentiable everywhere, including inactive rows. +""" + +from __future__ import annotations + +from typing import Any + +import torch + +from rl_engine.kernels.logprob_contract import LogprobContract, LogprobContractError, LogprobDType + +BACKEND_ID = "pytorch-vocab-parallel-logp-ws2" +DEFAULT_NUM_VOCAB_TILES = 64 + +_TORCH_TO_CONTRACT_DTYPE = { + torch.bfloat16: LogprobDType.BF16, + torch.float16: LogprobDType.FP16, + torch.float32: LogprobDType.FP32, +} + + +def _require_distributed_initialized(): + import torch.distributed as dist + + if not dist.is_available(): + raise LogprobContractError("vocab-parallel logprob requires torch.distributed.") + if not dist.is_initialized(): + raise LogprobContractError( + "vocab-parallel logprob requires an initialized process group when " + "the contract declares tp_world_size > 1." + ) + return dist + + +def _tile_size(contract: LogprobContract, num_vocab_tiles: int) -> int: + if isinstance(num_vocab_tiles, bool) or not isinstance(num_vocab_tiles, int): + raise LogprobContractError( + f"num_vocab_tiles must be a positive integer; got {num_vocab_tiles!r}" + ) + if num_vocab_tiles <= 0: + raise LogprobContractError( + f"num_vocab_tiles must be a positive integer; got {num_vocab_tiles}" + ) + padded = contract.sharding.padded_vocab_size + if padded % num_vocab_tiles != 0: + raise LogprobContractError( + f"num_vocab_tiles={num_vocab_tiles} must divide " f"padded_vocab_size={padded} exactly" + ) + tile = padded // num_vocab_tiles + for rank, (start, end) in enumerate(contract.sharding.vocab_shard_bounds): + if start % tile != 0 or end % tile != 0: + raise LogprobContractError( + f"vocab_shard_bounds[{rank}]=[{start}, {end}) is not aligned to the " + f"vocab tile size {tile} (num_vocab_tiles={num_vocab_tiles}); " + "cross-TP bitwise determinism requires tile-aligned shard bounds" + ) + return tile + + +def _validate_invocation( + local_logits: torch.Tensor, + target_ids: torch.Tensor, + contract: LogprobContract, + tp_group: Any, +) -> None: + if not isinstance(contract, LogprobContract): + raise LogprobContractError("contract must be a LogprobContract") + if local_logits.dim() != 2: + raise LogprobContractError( + f"local_logits must be 2-D [num_tokens, local_vocab]; got {local_logits.dim()}-D" + ) + if target_ids.dim() != 1 or target_ids.shape[0] != local_logits.shape[0]: + raise LogprobContractError( + f"target_ids must be 1-D with one entry per token; got shape " + f"{tuple(target_ids.shape)} for {local_logits.shape[0]} tokens" + ) + sharding = contract.sharding + if local_logits.shape[1] != sharding.local_vocab_size: + raise LogprobContractError( + f"local_logits has {local_logits.shape[1]} vocab columns but the contract " + f"declares local shard [{sharding.local_vocab_start}, " + f"{sharding.local_vocab_end}) of size {sharding.local_vocab_size}" + ) + if local_logits.shape[0] != contract.mask.num_tokens: + raise LogprobContractError( + f"local_logits has {local_logits.shape[0]} tokens but MaskSpec declares " + f"num_tokens={contract.mask.num_tokens}" + ) + declared = _TORCH_TO_CONTRACT_DTYPE.get(local_logits.dtype) + if declared is not contract.dtype: + raise LogprobContractError( + f"local_logits dtype {local_logits.dtype} does not match the contract " + f"dtype {contract.dtype.value}" + ) + if sharding.tp_world_size > 1: + dist = _require_distributed_initialized() + group_rank = dist.get_rank(group=tp_group) + group_world = dist.get_world_size(group=tp_group) + if group_world != sharding.tp_world_size: + raise LogprobContractError( + f"tp_group world size {group_world} does not match the contract " + f"tp_world_size={sharding.tp_world_size}; pass the TP subgroup, " + "not the global group" + ) + if group_rank != sharding.tp_rank: + raise LogprobContractError( + f"tp_group rank {group_rank} does not match the contract " + f"tp_rank={sharding.tp_rank}" + ) + + +def _validate_active_targets( + target_1d: torch.Tensor, active_mask: torch.Tensor, real_vocab_size: int +) -> None: + bad = active_mask & ((target_1d < 0) | (target_1d >= real_vocab_size)) + if bool(bad.any().item()): + bad_values = target_1d[bad] + raise LogprobContractError( + "active target_ids must lie in the real vocabulary " + f"[0, {real_vocab_size}); got values in " + f"[{int(bad_values.min().item())}, {int(bad_values.max().item())}] " + "on active rows" + ) + + +def _preflight_cross_rank_agreement( + contract: LogprobContract, tp_group: Any, num_vocab_tiles: int +) -> None: + """All-gather (fingerprint, backend id, tile count) and abort on mismatch.""" + + dist = _require_distributed_initialized() + payload = (contract.cross_rank_fingerprint(), BACKEND_ID, int(num_vocab_tiles)) + world = dist.get_world_size(group=tp_group) + gathered: list[Any] = [None] * world + dist.all_gather_object(gathered, payload, group=tp_group) + mismatched = [(rank, other) for rank, other in enumerate(gathered) if other != payload] + if mismatched: + rank, other = mismatched[0] + raise LogprobContractError( + "cross-rank preflight failed: rank " + f"{contract.sharding.tp_rank} has {payload} but rank {rank} has {other}; " + "all TP ranks must agree on the contract fingerprint, backend id, and " + "num_vocab_tiles before any collective" + ) + + +def _local_tile_stats(z_masked: torch.Tensor, tile: int) -> tuple[torch.Tensor, torch.Tensor]: + """fp32 per-tile ``(max, sumexp)`` partials for this rank's shard. + + Each tile is reduced as a contiguous ``[n, tile]`` tensor so the reduction + shape and layout are identical no matter which rank computes the tile or + what the local shard size is. An all-``-inf`` (padding-only) tile yields + the identity partial ``(-inf, 0)`` without evaluating ``exp(-inf - (-inf))``. + """ + + n, local_vocab = z_masked.shape + m_parts: list[torch.Tensor] = [] + s_parts: list[torch.Tensor] = [] + for tile_index in range(local_vocab // tile): + block = z_masked[:, tile_index * tile : (tile_index + 1) * tile].contiguous() + m_t = block.max(dim=-1).values + finite = m_t > float("-inf") + m_safe = torch.where(finite, m_t, torch.zeros_like(m_t)) + s_t = (block - m_safe.unsqueeze(-1)).exp().sum(dim=-1) + s_t = torch.where(finite, s_t, torch.zeros_like(s_t)) + m_parts.append(m_t) + s_parts.append(s_t) + return torch.stack(m_parts, dim=1), torch.stack(s_parts, dim=1) + + +def _gather_tile_stats( + local_m: torch.Tensor, + local_s: torch.Tensor, + contract: LogprobContract, + tp_group: Any, + tile: int, +) -> tuple[torch.Tensor, torch.Tensor]: + """Assemble all ``num_vocab_tiles`` partials in global tile order.""" + + sharding = contract.sharding + tile_counts = [(end - start) // tile for start, end in sharding.vocab_shard_bounds] + if sharding.tp_world_size == 1: + return local_m.contiguous(), local_s.contiguous() + + dist = _require_distributed_initialized() + n = local_m.shape[0] + max_tiles = max(tile_counts) + packed = local_m.new_zeros((n, max_tiles, 2)) + packed[:, : local_m.shape[1], 0] = local_m + packed[:, : local_s.shape[1], 1] = local_s + packed = packed.contiguous() + gathered = [torch.empty_like(packed) for _ in range(sharding.tp_world_size)] + dist.all_gather(gathered, packed, group=tp_group) + + m_parts = [gathered[rank][:, : tile_counts[rank], 0] for rank in range(len(tile_counts))] + s_parts = [gathered[rank][:, : tile_counts[rank], 1] for rank in range(len(tile_counts))] + return torch.cat(m_parts, dim=1).contiguous(), torch.cat(s_parts, dim=1).contiguous() + + +def _gather_target_logit( + z_masked: torch.Tensor, + safe_target: torch.Tensor, + contract: LogprobContract, + tp_group: Any, +) -> torch.Tensor: + """Exact selected-target logit via a select-by-owner copy.""" + + sharding = contract.sharding + n = z_masked.shape[0] + start = sharding.local_vocab_start + local_vocab = sharding.local_vocab_size + local_idx = (safe_target - start).clamp(0, max(local_vocab - 1, 0)) + owns = (safe_target >= start) & (safe_target < sharding.local_vocab_end) + rows = torch.arange(n, device=z_masked.device) + local_contrib = torch.where( + owns, z_masked[rows, local_idx], torch.zeros_like(safe_target, dtype=z_masked.dtype) + ).contiguous() + + if sharding.tp_world_size == 1: + stacked = local_contrib.unsqueeze(0) + else: + dist = _require_distributed_initialized() + gathered = [torch.empty_like(local_contrib) for _ in range(sharding.tp_world_size)] + dist.all_gather(gathered, local_contrib, group=tp_group) + stacked = torch.stack(gathered, dim=0) + + starts = torch.tensor( + [bound_start for bound_start, _ in sharding.vocab_shard_bounds], + device=safe_target.device, + dtype=torch.long, + ) + owner = torch.bucketize(safe_target, starts, right=True) - 1 + return stacked[owner, rows] + + +def _merge_tile_partials(m_all: torch.Tensor, s_all: torch.Tensor) -> torch.Tensor: + """Fixed-order (max, sumexp) merge over [n, num_vocab_tiles].""" + + M = m_all.max(dim=1).values + finite = M > float("-inf") + M_safe = torch.where(finite, M, torch.zeros_like(M)) + terms = s_all * (m_all - M_safe.unsqueeze(1)).exp() + S = terms.sum(dim=1) + return M + S.log() + + +class _VocabParallelLogprobFunction(torch.autograd.Function): + @staticmethod + def forward(ctx, local_logits, target_1d, active_mask, contract, tp_group, tile): + z_masked = local_logits.float() + sharding = contract.sharding + global_ids = torch.arange( + sharding.local_vocab_start, sharding.local_vocab_end, device=z_masked.device + ) + padding_cols = global_ids >= sharding.real_vocab_size + if bool(padding_cols.any()): + z_masked = z_masked.masked_fill(padding_cols.unsqueeze(0), float("-inf")) + + safe_target = torch.where(active_mask, target_1d, torch.zeros_like(target_1d)) + + local_m, local_s = _local_tile_stats(z_masked, tile) + m_all, s_all = _gather_tile_stats(local_m, local_s, contract, tp_group, tile) + target_logit = _gather_target_logit(z_masked, safe_target, contract, tp_group) + lse = _merge_tile_partials(m_all, s_all) + + selected_logp = torch.where(active_mask, target_logit - lse, torch.zeros_like(lse)) + + ctx.save_for_backward(z_masked, lse, safe_target, active_mask, padding_cols) + ctx.local_vocab_start = sharding.local_vocab_start + ctx.local_vocab_size = sharding.local_vocab_size + ctx.input_dtype = local_logits.dtype + ctx.set_materialize_grads(False) + return selected_logp, lse + + @staticmethod + def backward(ctx, grad_logp, grad_lse): + if not ctx.needs_input_grad[0] or (grad_logp is None and grad_lse is None): + return None, None, None, None, None, None + + z_masked, lse, safe_target, active_mask, padding_cols = ctx.saved_tensors + n, local_vocab = z_masked.shape + finite_row = torch.isfinite(lse) + lse_safe = torch.where(finite_row, lse, torch.zeros_like(lse)) + p = (z_masked - lse_safe.unsqueeze(1)).exp() + p = torch.where(finite_row.unsqueeze(1), p, torch.zeros_like(p)) + + grad = torch.zeros_like(z_masked) + if grad_logp is not None: + local_idx = (safe_target - ctx.local_vocab_start).clamp(0, max(local_vocab - 1, 0)) + owns = (safe_target >= ctx.local_vocab_start) & ( + safe_target < ctx.local_vocab_start + local_vocab + ) + onehot = torch.zeros_like(z_masked) + hit = owns & active_mask + rows = torch.arange(n, device=z_masked.device)[hit] + onehot[rows, local_idx[hit]] = 1.0 + g_logp = torch.where(active_mask, grad_logp, torch.zeros_like(grad_logp)) + grad = grad + g_logp.unsqueeze(1) * (onehot - p) + if grad_lse is not None: + grad = grad + grad_lse.unsqueeze(1) * p + if bool(padding_cols.any()): + grad = grad.masked_fill(padding_cols.unsqueeze(0), 0.0) + return grad.to(ctx.input_dtype), None, None, None, None, None + + +class VocabParallelLogprobOp: + """Deterministic vocab-parallel selected-token logprob (WS2 reference).""" + + op_class = "logprob" + is_batch_invariant = True + + def __init__(self) -> None: + pass + + def __call__( + self, + local_logits: torch.Tensor, + target_ids: torch.Tensor, + *, + contract: LogprobContract, + tp_group: Any = None, + num_vocab_tiles: int = DEFAULT_NUM_VOCAB_TILES, + validate: bool = True, + ) -> tuple[torch.Tensor, torch.Tensor]: + return self.apply( + local_logits, + target_ids, + contract=contract, + tp_group=tp_group, + num_vocab_tiles=num_vocab_tiles, + validate=validate, + ) + + def apply( + self, + local_logits: torch.Tensor, + target_ids: torch.Tensor, + *, + contract: LogprobContract, + tp_group: Any = None, + num_vocab_tiles: int = DEFAULT_NUM_VOCAB_TILES, + validate: bool = True, + ) -> tuple[torch.Tensor, torch.Tensor]: + if not isinstance(contract, LogprobContract): + raise LogprobContractError("contract must be a LogprobContract") + tile = _tile_size(contract, num_vocab_tiles) + _validate_invocation(local_logits, target_ids, contract, tp_group) + + target_1d = target_ids.reshape(-1).to(device=local_logits.device, dtype=torch.long) + active_mask = torch.tensor( + contract.mask.active_mask, dtype=torch.bool, device=local_logits.device + ) + if validate: + _validate_active_targets(target_1d, active_mask, contract.sharding.real_vocab_size) + if contract.sharding.tp_world_size > 1: + _preflight_cross_rank_agreement(contract, tp_group, num_vocab_tiles) + + selected_logp, lse = _VocabParallelLogprobFunction.apply( + local_logits, target_1d, active_mask, contract, tp_group, tile + ) + + if validate and bool((~torch.isfinite(lse) & active_mask).any().item()): + raise LogprobContractError( + "non-finite logsumexp on an active row: logits over the real " + "vocabulary must be finite for every active token" + ) + return selected_logp, lse + + +__all__ = [ + "BACKEND_ID", + "DEFAULT_NUM_VOCAB_TILES", + "VocabParallelLogprobOp", +] diff --git a/rl_engine/kernels/registry.py b/rl_engine/kernels/registry.py index 32cfdc37..4b2c5306 100644 --- a/rl_engine/kernels/registry.py +++ b/rl_engine/kernels/registry.py @@ -85,6 +85,10 @@ class OpBackend(Enum, metaclass=_KernelEnumMeta): CUDA_BATCH_INVARIANT_LOGP_SM90 = ( "rl_engine.kernels.ops.cuda.loss.batch_invariant_logp.BatchInvariantLogpSM90Op" ) + # Deterministic vocab-parallel TP logprob reference (WS2 #241 PR3) + PYTORCH_VOCAB_PARALLEL_LOGP = ( + "rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp.VocabParallelLogprobOp" + ) # RMSNorm(pre-norm / QK-Norm) - pure Pytorch reference(ws1 ground-truth) PYTORCH_NATIVE_RMS_NORM = "rl_engine.kernels.ops.pytorch.norm.rms_norm.NativeRMSNormOp" @@ -350,6 +354,29 @@ def __init__(self): for platform, candidates in self._logprob_candidates.items() } + # deterministic vocab-parallel TP logprob reference. + ws2_tp_logprob_capability = LogprobBackendCapability( + backend_id="pytorch-vocab-parallel-logp-ws2", + roles=common_logprob_roles, + dtypes=common_logprob_dtypes, + tp_world_sizes=None, + cp_world_sizes=None, + supports_vocab_padding=True, + mask_modes=frozenset({MaskMode.EXPLICIT_ACTIVE_MASK, MaskMode.IGNORE_INDEX}), + exports_vocab_lse=True, + determinism_scopes=frozenset( + {DeterminismScope.CROSS_TP_BITWISE, DeterminismScope.FIXED_TOPOLOGY} + ), + implementation_kind="reference", + ) + for ws2_platform in self._priority_map: + self.register_logprob_backend( + OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP, + ws2_tp_logprob_capability, + platform=ws2_platform, + prepend=True, + ) + def _adjust_priority_from_env(self): rocm_attn_backend = os.getenv("RL_KERNEL_ROCM_ATTN_BACKEND", "").strip().lower() if rocm_attn_backend in {"flash_attn", "flash-attn", "flash_attention"}: diff --git a/tests/test_logprob_contract.py b/tests/test_logprob_contract.py index 42dc6dd1..307acfdf 100644 --- a/tests/test_logprob_contract.py +++ b/tests/test_logprob_contract.py @@ -247,8 +247,20 @@ def test_ignore_index_must_not_collide_with_the_real_vocabulary(): assert contract.mask.ignore_index == padding_column +def _restrict_to_ws1_candidates(registry: KernelRegistry) -> None: + """Drop the #241 PR3 vocab-parallel reference so only WS1 backends remain.""" + + platform = registry._platform() + registry._logprob_candidates[platform] = [ + backend + for backend in registry._logprob_candidates[platform] + if backend is not OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP + ] + + def test_current_ws1_backend_rejects_strict_tp_contract_without_fallback(): registry = KernelRegistry() + _restrict_to_ws1_candidates(registry) with pytest.raises(RuntimeError) as exc_info: registry.get_logprob_op(_contract(), requested_backend="reference") @@ -262,6 +274,7 @@ def test_current_ws1_backend_rejects_strict_tp_contract_without_fallback(): def test_current_ws1_backend_rejects_padded_vocab_even_at_tp1(): registry = KernelRegistry() + _restrict_to_ws1_candidates(registry) contract = _contract(sharding=_sharding(tp_world_size=1, cp_world_size=1)) with pytest.raises(RuntimeError) as exc_info: @@ -272,6 +285,25 @@ def test_current_ws1_backend_rejects_padded_vocab_even_at_tp1(): assert "padded-vs-real vocab masking is unsupported" in message +def test_ws1_rejections_recorded_when_vocab_parallel_reference_resolves(): + """The WS1 backends still reject strict contracts; they are skipped with + recorded reasons while dispatch resolves the #241 PR3 reference.""" + + registry = KernelRegistry() + platform = registry._platform() + # Order the WS1 backends ahead of the reference so their rejections are + # exercised on the way to a successful resolution. + candidates = registry._logprob_candidates[platform] + candidates.remove(OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP) + candidates.append(OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP) + + result = registry.get_logprob_op(_contract()) + assert result.capability.backend_id == "pytorch-vocab-parallel-logp-ws2" + assert result.provenance["fallback"] is True + rejections = " | ".join(result.provenance["prior_rejections"]) + assert "vocab-domain LSE export is unsupported" in rejections + + def test_undeclared_backend_capability_is_never_selected(): registry = KernelRegistry() platform = registry._platform() diff --git a/tests/test_operator_inputs.py b/tests/test_operator_inputs.py index 4f742734..2080f6fd 100644 --- a/tests/test_operator_inputs.py +++ b/tests/test_operator_inputs.py @@ -40,6 +40,7 @@ def _args(**overrides): "logp", "linear_logp", "batch_invariant_logp", + "vocab_parallel_logp", "rope", "silu", "swiglu", @@ -72,6 +73,31 @@ def test_constant_batch_invariant_logp_inputs_match_operator_contract(): assert torch.equal(inputs["target_ids"], torch.full((1, 2), 3, dtype=torch.long)) +def test_constant_vocab_parallel_logp_inputs_match_operator_contract(): + args = _args(input_mode="constant", constant_value=0.5, token_value=3) + inputs = make_operator_inputs("vocab_parallel_logp", args, torch.float32, torch.device("cpu")) + + # vocab=17 rounds up to padded=20 with 4 tiles; tokens flatten to batch*seq. + assert torch.equal(inputs["local_logits"], torch.full((2, 20), 0.5)) + assert torch.equal(inputs["target_ids"], torch.full((2,), 3, dtype=torch.long)) + assert inputs["contract"].sharding.real_vocab_size == 17 + assert inputs["contract"].sharding.padded_vocab_size == 20 + assert inputs["num_vocab_tiles"] == 4 + assert operator_shape_name("vocab_parallel_logp", args) == "2x20" + + +def test_vocab_parallel_logp_inputs_run_through_the_operator(): + from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import VocabParallelLogprobOp + + args = _args(input_mode="random", seed=7) + inputs = make_operator_inputs("vocab_parallel_logp", args, torch.float32, torch.device("cpu")) + + logp, lse = VocabParallelLogprobOp()(**inputs) + assert logp.shape == inputs["target_ids"].shape + assert logp.dtype == torch.float32 and lse.dtype == torch.float32 + assert torch.isfinite(logp).all() and torch.isfinite(lse).all() + + def test_random_logp_inputs_are_seeded(): args = _args(input_mode="random", seed=7) first = make_operator_inputs("logp", args, torch.float32, torch.device("cpu")) diff --git a/tests/test_vocab_parallel_logp.py b/tests/test_vocab_parallel_logp.py new file mode 100644 index 00000000..510aec70 --- /dev/null +++ b/tests/test_vocab_parallel_logp.py @@ -0,0 +1,426 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Deterministic vocab-parallel TP logprob reference tests (issue #241 PR3). + +Bit-level determinism assertions compare raw bit patterns via +``tensor.view(torch.int32)`` rather than ``torch.equal``: value equality +treats ``-0.0 == 0.0`` as equal and ``NaN != NaN`` as different, neither of +which is what a bitwise claim means. +""" + +from __future__ import annotations + +import queue +import tempfile +import traceback +from pathlib import Path + +import pytest +import torch +import torch.multiprocessing as mp + +from rl_engine.kernels.gtest.tolerance import load_contract +from rl_engine.kernels.logprob_contract import ( + DeterminismScope, + LogprobContract, + LogprobContractError, + MaskSpec, + ReductionSpec, + ShardingSpec, +) +from rl_engine.kernels.ops.pytorch.loss.batch_invariant_logp import NativeBatchInvariantLogpOp +from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import ( + BACKEND_ID, + VocabParallelLogprobOp, +) +from rl_engine.kernels.registry import KernelRegistry, OpBackend + +REAL_VOCAB = 27 +PADDED_VOCAB = 32 +NUM_TILES = 8 +NUM_TOKENS = 6 +ACTIVE = (True, True, True, True, True, False) + + +def _even_bounds(padded: int, world: int) -> tuple[tuple[int, int], ...]: + shard = padded // world + return tuple( + (rank * shard, padded if rank == world - 1 else (rank + 1) * shard) for rank in range(world) + ) + + +def _contract( + *, + tp_rank: int = 0, + tp_world_size: int = 1, + bounds: tuple[tuple[int, int], ...] | None = None, + real_vocab: int = REAL_VOCAB, + padded_vocab: int = PADDED_VOCAB, + num_tokens: int = NUM_TOKENS, + active: tuple[bool, ...] = ACTIVE, + dtype: str = "fp32", +) -> LogprobContract: + return LogprobContract( + role="train", + dtype=dtype, + mask=MaskSpec(num_tokens=num_tokens, active_mask=active), + sharding=ShardingSpec( + tp_rank=tp_rank, + tp_world_size=tp_world_size, + vocab_shard_bounds=( + bounds if bounds is not None else _even_bounds(padded_vocab, tp_world_size) + ), + real_vocab_size=real_vocab, + padded_vocab_size=padded_vocab, + ), + reduction=ReductionSpec(), + ) + + +def _inputs(dtype=torch.float32, seed: int = 2026): + torch.manual_seed(seed) + logits = torch.randn(NUM_TOKENS, PADDED_VOCAB, dtype=torch.float32).to(dtype) + targets = torch.tensor([1, 5, REAL_VOCAB - 1, 0, 13, -100]) + return logits, targets + + +def _bits(tensor: torch.Tensor) -> torch.Tensor: + view_dtype = {torch.float32: torch.int32, torch.bfloat16: torch.int16}[tensor.dtype] + return tensor.contiguous().view(view_dtype) + + +def _bitwise_equal(a: torch.Tensor, b: torch.Tensor) -> bool: + return a.shape == b.shape and bool((_bits(a) == _bits(b)).all()) + + +def _case_shard_size_mismatch(): + logits, targets = _inputs() + return logits, targets, _contract(tp_rank=0, tp_world_size=2), NUM_TILES, "vocab columns" + + +def _case_mask_length_mismatch(): + logits, targets = _inputs() + contract = _contract(num_tokens=NUM_TOKENS + 1, active=ACTIVE + (True,)) + return logits, targets, contract, NUM_TILES, "num_tokens" + + +def _case_dtype_mismatch(): + logits, targets = _inputs() + return logits, targets, _contract(dtype="bf16"), NUM_TILES, "dtype" + + +def _case_tile_misaligned_bounds(): + # Tile size is 32/8 = 4; a boundary at 6 is misaligned. + logits, targets = _inputs() + contract = _contract(tp_world_size=2, bounds=((0, 6), (6, 32))) + return logits[:, :6], targets, contract, NUM_TILES, "tile" + + +def _case_bad_num_vocab_tiles(): + logits, targets = _inputs() + return logits, targets, _contract(), 7, "num_vocab_tiles" + + +def _case_active_target_out_of_real_vocab(): + logits, targets = _inputs() + bad_targets = targets.clone() + bad_targets[0] = REAL_VOCAB # padding column, active row + return logits, bad_targets, _contract(), NUM_TILES, "real vocabulary" + + +def _case_all_inf_active_row(): + logits, targets = _inputs() + poisoned = logits.clone() + poisoned[0, :] = float("-inf") + return poisoned, targets, _contract(), NUM_TILES, "non-finite" + + +@pytest.mark.parametrize( + "case", + [ + _case_shard_size_mismatch, + _case_mask_length_mismatch, + _case_dtype_mismatch, + _case_tile_misaligned_bounds, + _case_bad_num_vocab_tiles, + _case_active_target_out_of_real_vocab, + _case_all_inf_active_row, + ], + ids=lambda fn: fn.__name__.removeprefix("_case_"), +) +def test_invalid_invocations_fail_loudly(case): + logits, targets, contract, num_tiles, match = case() + with pytest.raises(LogprobContractError, match=match): + VocabParallelLogprobOp()(logits, targets, contract=contract, num_vocab_tiles=num_tiles) + + +class TestSingleRank: + def test_repeated_runs_are_bitwise_identical(self): + contract = _contract() + logits, targets = _inputs() + op = VocabParallelLogprobOp() + logp_a, lse_a = op(logits, targets, contract=contract, num_vocab_tiles=NUM_TILES) + logp_b, lse_b = op(logits, targets, contract=contract, num_vocab_tiles=NUM_TILES) + assert _bitwise_equal(logp_a, logp_b) + assert _bitwise_equal(lse_a, lse_b) + + def test_batch_invariance_same_row_any_context(self): + contract_full = _contract() + logits, targets = _inputs() + op = VocabParallelLogprobOp() + logp_full, lse_full = op(logits, targets, contract=contract_full, num_vocab_tiles=NUM_TILES) + + contract_single = _contract(num_tokens=1, active=(True,)) + logp_one, lse_one = op( + logits[2:3], targets[2:3], contract=contract_single, num_vocab_tiles=NUM_TILES + ) + assert _bitwise_equal(logp_full[2:3], logp_one) + assert _bitwise_equal(lse_full[2:3], lse_one) + + def test_matches_ws1_batch_invariant_logp_within_contract_tolerance(self): + tolerance = load_contract()["accuracy"]["default"]["logprob"]["float32"] + contract = _contract(padded_vocab=REAL_VOCAB + 5) + # Use a real==padded contract so the WS1 op sees identical logits. + contract = _contract(real_vocab=PADDED_VOCAB, padded_vocab=PADDED_VOCAB) + logits, targets = _inputs() + logp, _ = VocabParallelLogprobOp()( + logits, targets, contract=contract, num_vocab_tiles=NUM_TILES + ) + ws1 = NativeBatchInvariantLogpOp().apply(logits, targets) + active = torch.tensor(ACTIVE) + assert torch.allclose( + logp[active], ws1[active], atol=tolerance["atol"], rtol=tolerance["rtol"] + ) + + def test_padding_columns_are_excluded_and_finite(self): + contract = _contract() + logits, targets = _inputs() + boosted = logits.clone() + boosted[:, REAL_VOCAB:] = 1e4 # huge padding logits must not leak into LSE + logp, lse = VocabParallelLogprobOp()( + boosted, targets, contract=contract, num_vocab_tiles=NUM_TILES + ) + ref_lse = torch.logsumexp(boosted[:, :REAL_VOCAB].float(), dim=-1) + assert torch.isfinite(logp).all() and torch.isfinite(lse).all() + assert torch.allclose(lse, ref_lse, atol=1e-5) + + def test_inactive_rows_zero_filled_lse_still_exported(self): + contract = _contract() + logits, targets = _inputs() + logp, lse = VocabParallelLogprobOp()( + logits, targets, contract=contract, num_vocab_tiles=NUM_TILES + ) + assert logp[-1].item() == 0.0 + assert torch.isfinite(lse[-1]) + + +class TestBackward: + def test_grads_match_autograd_oracle(self): + tolerance = load_contract()["accuracy"]["default"]["logprob"]["float32"] + contract = _contract() + logits, targets = _inputs() + x = logits.clone().requires_grad_(True) + logp, lse = VocabParallelLogprobOp()( + x, targets, contract=contract, num_vocab_tiles=NUM_TILES + ) + (logp.sum() + 0.5 * lse.sum()).backward() + + y = logits.clone().requires_grad_(True) + ref_lse = torch.logsumexp(y[:, :REAL_VOCAB].float(), dim=-1) + safe = targets.clamp(0, REAL_VOCAB - 1) + ref_logp = y[torch.arange(NUM_TOKENS), safe].float() - ref_lse + ref_logp = torch.where(torch.tensor(ACTIVE), ref_logp, torch.zeros_like(ref_logp)) + (ref_logp.sum() + 0.5 * ref_lse.sum()).backward() + + assert torch.allclose(x.grad, y.grad, atol=tolerance["atol"], rtol=tolerance["rtol"]) + assert bool((x.grad[:, REAL_VOCAB:] == 0).all()) + + # No grad requested -> outputs detached from autograd entirely. + logp_ng, lse_ng = VocabParallelLogprobOp()( + logits, targets, contract=contract, num_vocab_tiles=NUM_TILES + ) + assert not logp_ng.requires_grad and not lse_ng.requires_grad + + def test_inactive_rows_grad_asymmetry(self): + """The logp term is zeroed on inactive rows; the lse term still flows — + lse is a row property exported (and differentiable) for every row.""" + + contract = _contract() + logits, targets = _inputs() + + x = logits.clone().requires_grad_(True) + _, lse = VocabParallelLogprobOp()(x, targets, contract=contract, num_vocab_tiles=NUM_TILES) + lse.sum().backward() + assert bool((x.grad[-1, :REAL_VOCAB].abs() > 0).any()) + + z = logits.clone().requires_grad_(True) + logp, _ = VocabParallelLogprobOp()(z, targets, contract=contract, num_vocab_tiles=NUM_TILES) + logp.sum().backward() + assert bool((z.grad[-1] == 0).all()) + + +def test_dispatch_resolves_reference_and_leaves_legacy_untouched(): + registry = KernelRegistry() + contract = _contract() + + result = registry.get_logprob_op(contract) + assert result.capability.backend_id == BACKEND_ID + assert result.provenance["fallback"] is False + assert isinstance(result.op, VocabParallelLogprobOp) + assert ( + result.provenance["contract"]["reduction"]["determinism_scope"] + == DeterminismScope.CROSS_TP_BITWISE.value + ) + + by_id = registry.get_logprob_op(contract, requested_backend=BACKEND_ID) + assert by_id.capability.backend_id == BACKEND_ID + by_kind = registry.get_logprob_op(contract, requested_backend="reference") + assert by_kind.capability.backend_id == BACKEND_ID + + for ops in registry._priority_map.values(): + for candidates in ops.values(): + assert OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP not in candidates + + +# --------------------------------------------------------------------------- +# Multi-rank gloo tests (spawn pattern from tests/test_linear_logp.py) +# --------------------------------------------------------------------------- + + +def _gloo_available() -> bool: + return torch.distributed.is_available() and torch.distributed.is_gloo_available() + + +requires_gloo = pytest.mark.skipif( + not _gloo_available(), reason="requires torch.distributed with the gloo backend" +) + +_WORLD_SIZE = 4 +_UNEVEN_BOUNDS = ((0, 4), (4, 16), (16, 24), (24, 32)) # tile-aligned (tile=4) + + +def _tp_worker(rank, world_size, init_method, result_queue, scenario): + import torch.distributed as dist + + torch.set_num_threads(1) + try: + dist.init_process_group( + backend="gloo", init_method=init_method, rank=rank, world_size=world_size + ) + dtype = torch.bfloat16 if scenario == "bf16" else torch.float32 + dtype_name = "bf16" if scenario == "bf16" else "fp32" + bounds = _UNEVEN_BOUNDS if scenario == "uneven" else _even_bounds(PADDED_VOCAB, world_size) + logits, targets = _inputs(dtype=dtype) + start, end = bounds[rank] + + op = VocabParallelLogprobOp() + tiles = 16 if scenario == "preflight" and rank == 0 else NUM_TILES + contract_tp = _contract( + tp_rank=rank, tp_world_size=world_size, bounds=bounds, dtype=dtype_name + ) + + if scenario == "preflight": + try: + op( + logits[:, start:end].contiguous().clone(), + targets, + contract=contract_tp, + tp_group=dist.group.WORLD, + num_vocab_tiles=tiles, + ) + result_queue.put({"ok": False, "rank": rank, "traceback": "no error raised"}) + except LogprobContractError: + result_queue.put({"ok": True, "rank": rank}) + return + + shard = logits[:, start:end].contiguous().clone().requires_grad_(True) + logp_tp, lse_tp = op( + shard, + targets, + contract=contract_tp, + tp_group=dist.group.WORLD, + num_vocab_tiles=NUM_TILES, + ) + (logp_tp.sum() + 0.5 * lse_tp.sum()).backward() + + # In-process TP=1 run of the same op on the full logits: the cross-TP + # bitwise claim is TP=n output == TP=1 output, bit for bit. + full = logits.clone().requires_grad_(True) + contract_tp1 = _contract(dtype=dtype_name) + logp_one, lse_one = op(full, targets, contract=contract_tp1, num_vocab_tiles=NUM_TILES) + (logp_one.sum() + 0.5 * lse_one.sum()).backward() + + result_queue.put( + { + "ok": True, + "rank": rank, + "logp_bits_match": _bitwise_equal(logp_tp, logp_one), + "lse_bits_match": _bitwise_equal(lse_tp, lse_one), + "grad_bits_match": _bitwise_equal(shard.grad, full.grad[:, start:end]), + "logp": logp_tp.detach().float(), + "lse": lse_tp.detach().float(), + } + ) + except Exception: + result_queue.put({"ok": False, "rank": rank, "traceback": traceback.format_exc()}) + raise + finally: + if torch.distributed.is_initialized(): + torch.distributed.destroy_process_group() + + +def _run_gloo_scenario(scenario): + ctx = mp.get_context("spawn") + with tempfile.TemporaryDirectory() as tmpdir: + init_method = (Path(tmpdir) / "gloo_init").as_uri() + result_queue = ctx.Queue() + processes = [ + ctx.Process( + target=_tp_worker, + args=(rank, _WORLD_SIZE, init_method, result_queue, scenario), + ) + for rank in range(_WORLD_SIZE) + ] + results = [] + try: + for process in processes: + process.start() + for _ in range(_WORLD_SIZE): + try: + results.append(result_queue.get(timeout=60)) + except queue.Empty: + for process in processes: + if process.is_alive(): + process.terminate() + pytest.fail("timed out waiting for vocab-parallel gloo workers") + finally: + for process in processes: + process.join(timeout=10) + if process.is_alive(): + process.terminate() + results.sort(key=lambda item: item["rank"]) + for result in results: + assert result["ok"], result.get("traceback") + for process in processes: + assert process.exitcode == 0 + return results + + +@requires_gloo +@pytest.mark.parametrize("scenario", ["even", "uneven", "bf16"]) +def test_tp4_bitwise_identical_to_tp1(scenario): + results = _run_gloo_scenario(scenario) + for result in results: + assert result["logp_bits_match"], f"rank {result['rank']} logp bits differ from TP=1" + assert result["lse_bits_match"], f"rank {result['rank']} lse bits differ from TP=1" + assert result["grad_bits_match"], f"rank {result['rank']} grad bits differ from TP=1" + # Outputs are replicated: every rank must hold identical bits. + for other in results[1:]: + assert _bitwise_equal(results[0]["logp"], other["logp"]) + assert _bitwise_equal(results[0]["lse"], other["lse"]) + + +@requires_gloo +def test_preflight_rejects_mismatched_num_vocab_tiles(): + _run_gloo_scenario("preflight") From 3866d3c18a76668d9eff00804cdd262c63b56f72 Mon Sep 17 00:00:00 2001 From: KJLdefeated Date: Wed, 5 Aug 2026 16:10:26 +0800 Subject: [PATCH 21/48] init vocab parallel logp --- tests/test_operator_inputs.py | 26 -------------------------- 1 file changed, 26 deletions(-) diff --git a/tests/test_operator_inputs.py b/tests/test_operator_inputs.py index 2080f6fd..4f742734 100644 --- a/tests/test_operator_inputs.py +++ b/tests/test_operator_inputs.py @@ -40,7 +40,6 @@ def _args(**overrides): "logp", "linear_logp", "batch_invariant_logp", - "vocab_parallel_logp", "rope", "silu", "swiglu", @@ -73,31 +72,6 @@ def test_constant_batch_invariant_logp_inputs_match_operator_contract(): assert torch.equal(inputs["target_ids"], torch.full((1, 2), 3, dtype=torch.long)) -def test_constant_vocab_parallel_logp_inputs_match_operator_contract(): - args = _args(input_mode="constant", constant_value=0.5, token_value=3) - inputs = make_operator_inputs("vocab_parallel_logp", args, torch.float32, torch.device("cpu")) - - # vocab=17 rounds up to padded=20 with 4 tiles; tokens flatten to batch*seq. - assert torch.equal(inputs["local_logits"], torch.full((2, 20), 0.5)) - assert torch.equal(inputs["target_ids"], torch.full((2,), 3, dtype=torch.long)) - assert inputs["contract"].sharding.real_vocab_size == 17 - assert inputs["contract"].sharding.padded_vocab_size == 20 - assert inputs["num_vocab_tiles"] == 4 - assert operator_shape_name("vocab_parallel_logp", args) == "2x20" - - -def test_vocab_parallel_logp_inputs_run_through_the_operator(): - from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import VocabParallelLogprobOp - - args = _args(input_mode="random", seed=7) - inputs = make_operator_inputs("vocab_parallel_logp", args, torch.float32, torch.device("cpu")) - - logp, lse = VocabParallelLogprobOp()(**inputs) - assert logp.shape == inputs["target_ids"].shape - assert logp.dtype == torch.float32 and lse.dtype == torch.float32 - assert torch.isfinite(logp).all() and torch.isfinite(lse).all() - - def test_random_logp_inputs_are_seeded(): args = _args(input_mode="random", seed=7) first = make_operator_inputs("logp", args, torch.float32, torch.device("cpu")) From 65f3c6f05456cb6e2223c9e267ebb0d86705fcbd Mon Sep 17 00:00:00 2001 From: KJLdefeated Date: Wed, 5 Aug 2026 21:13:04 +0800 Subject: [PATCH 22/48] adding cross tp testing --- tests/test_vocab_parallel_logp.py | 257 +++++++++++++++++++++--------- 1 file changed, 184 insertions(+), 73 deletions(-) diff --git a/tests/test_vocab_parallel_logp.py b/tests/test_vocab_parallel_logp.py index 510aec70..87da2ef0 100644 --- a/tests/test_vocab_parallel_logp.py +++ b/tests/test_vocab_parallel_logp.py @@ -1,13 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 RL-Kernel Contributors -"""Deterministic vocab-parallel TP logprob reference tests (issue #241 PR3). - -Bit-level determinism assertions compare raw bit patterns via -``tensor.view(torch.int32)`` rather than ``torch.equal``: value equality -treats ``-0.0 == 0.0`` as equal and ``NaN != NaN`` as different, neither of -which is what a bitwise claim means. -""" +"""Deterministic vocab-parallel TP logprob reference tests""" from __future__ import annotations @@ -86,7 +80,11 @@ def _inputs(dtype=torch.float32, seed: int = 2026): def _bits(tensor: torch.Tensor) -> torch.Tensor: - view_dtype = {torch.float32: torch.int32, torch.bfloat16: torch.int16}[tensor.dtype] + view_dtype = { + torch.float32: torch.int32, + torch.bfloat16: torch.int16, + torch.float16: torch.int16, + }[tensor.dtype] return tensor.contiguous().view(view_dtype) @@ -283,72 +281,143 @@ def test_dispatch_resolves_reference_and_leaves_legacy_untouched(): assert OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP not in candidates -# --------------------------------------------------------------------------- -# Multi-rank gloo tests (spawn pattern from tests/test_linear_logp.py) -# --------------------------------------------------------------------------- +# Cross-TP bitwise determinism on real ranks (NCCL, one CUDA device per rank) +TP_REAL_VOCAB = 1000 +TP_PADDED_VOCAB = 1024 +TP_NUM_TILES = 32 # tile = 32 columns +TP_TILE = TP_PADDED_VOCAB // TP_NUM_TILES +TP_NUM_TOKENS = 48 +TP_ACTIVE = tuple(index % 7 != 5 for index in range(TP_NUM_TOKENS)) +TP_DTYPES = {"fp32": torch.float32, "bf16": torch.bfloat16} +_SPAWN_TIMEOUT_S = 300 -def _gloo_available() -> bool: - return torch.distributed.is_available() and torch.distributed.is_gloo_available() +def _cuda_device_count() -> int: + return torch.cuda.device_count() if torch.cuda.is_available() else 0 -requires_gloo = pytest.mark.skipif( - not _gloo_available(), reason="requires torch.distributed with the gloo backend" -) +def _requires_gpus(count: int): + return pytest.mark.skipif( + _cuda_device_count() < count, + reason=f"cross-TP determinism needs {count} CUDA devices to place one rank per device", + ) + + +def _tile_counts(world_size: int, uneven: bool) -> list[int]: + """Tiles per rank; bounds are built from whole tiles so they stay tile-aligned.""" + + counts = [TP_NUM_TILES // world_size for _ in range(world_size)] + counts[-1] += TP_NUM_TILES % world_size + if uneven: + for rank in range(world_size - 1): + if counts[rank] > 1: + counts[rank] -= 1 + counts[-1] += 1 + return counts + + +def _tp_bounds(world_size: int, uneven: bool) -> tuple[tuple[int, int], ...]: + bounds, cursor = [], 0 + for count in _tile_counts(world_size, uneven): + bounds.append((cursor, cursor + count * TP_TILE)) + cursor += count * TP_TILE + return tuple(bounds) + + +def _tp_contract(tp_rank: int, tp_world_size: int, bounds, dtype_name: str) -> LogprobContract: + return _contract( + tp_rank=tp_rank, + tp_world_size=tp_world_size, + bounds=bounds, + real_vocab=TP_REAL_VOCAB, + padded_vocab=TP_PADDED_VOCAB, + num_tokens=TP_NUM_TOKENS, + active=TP_ACTIVE, + dtype=dtype_name, + ) -_WORLD_SIZE = 4 -_UNEVEN_BOUNDS = ((0, 4), (4, 16), (16, 24), (24, 32)) # tile-aligned (tile=4) +def _tp_inputs(device, dtype, seed: int = 2026): + """Identical logits and targets on every rank, seeded on CPU.""" -def _tp_worker(rank, world_size, init_method, result_queue, scenario): + gen = torch.Generator(device="cpu").manual_seed(seed) + logits = torch.randn(TP_NUM_TOKENS, TP_PADDED_VOCAB, generator=gen, dtype=torch.float32) + targets = torch.randint(0, TP_REAL_VOCAB, (TP_NUM_TOKENS,), generator=gen) + active = torch.tensor(TP_ACTIVE) + # Inactive rows carry ignore_index; active_mask stays the sole authority. + targets = torch.where(active, targets, torch.full_like(targets, -100)) + return logits.to(device=device, dtype=dtype), targets.to(device) + + +def _nccl_worker(rank, world_size, init_method, result_queue, scenario, uneven, dtype_name): import torch.distributed as dist - torch.set_num_threads(1) try: + torch.cuda.set_device(rank) + device = torch.device("cuda", rank) dist.init_process_group( - backend="gloo", init_method=init_method, rank=rank, world_size=world_size + backend="nccl", init_method=init_method, rank=rank, world_size=world_size ) - dtype = torch.bfloat16 if scenario == "bf16" else torch.float32 - dtype_name = "bf16" if scenario == "bf16" else "fp32" - bounds = _UNEVEN_BOUNDS if scenario == "uneven" else _even_bounds(PADDED_VOCAB, world_size) - logits, targets = _inputs(dtype=dtype) - start, end = bounds[rank] - + dtype = TP_DTYPES[dtype_name] op = VocabParallelLogprobOp() - tiles = 16 if scenario == "preflight" and rank == 0 else NUM_TILES - contract_tp = _contract( - tp_rank=rank, tp_world_size=world_size, bounds=bounds, dtype=dtype_name - ) - - if scenario == "preflight": + bounds = _tp_bounds(world_size, uneven) + logits, targets = _tp_inputs(device, dtype) + tiles = TP_NUM_TILES + + if scenario in {"preflight", "misaligned"}: + if scenario == "preflight": + if rank == 0: + tiles = TP_NUM_TILES * 2 + else: + # Nudge the first boundary off the tile grid, on every rank. + split = bounds[0][1] + TP_TILE // 4 + bounds = ((0, split), (split, bounds[1][1])) + bounds[2:] + + start, end = bounds[rank] try: op( logits[:, start:end].contiguous().clone(), targets, - contract=contract_tp, + contract=_tp_contract(rank, world_size, bounds, dtype_name), tp_group=dist.group.WORLD, num_vocab_tiles=tiles, ) result_queue.put({"ok": False, "rank": rank, "traceback": "no error raised"}) - except LogprobContractError: - result_queue.put({"ok": True, "rank": rank}) + except LogprobContractError as exc: + result_queue.put({"ok": True, "rank": rank, "message": str(exc)}) return + start, end = bounds[rank] shard = logits[:, start:end].contiguous().clone().requires_grad_(True) + tp_contract = _tp_contract(rank, world_size, bounds, dtype_name) logp_tp, lse_tp = op( shard, targets, - contract=contract_tp, + contract=tp_contract, tp_group=dist.group.WORLD, - num_vocab_tiles=NUM_TILES, + num_vocab_tiles=TP_NUM_TILES, ) (logp_tp.sum() + 0.5 * lse_tp.sum()).backward() - # In-process TP=1 run of the same op on the full logits: the cross-TP - # bitwise claim is TP=n output == TP=1 output, bit for bit. + # Same ranks, same inputs, run again: the collectives must not perturb bits. + rerun = logits[:, start:end].contiguous().clone() + logp_re, lse_re = op( + rerun, + targets, + contract=tp_contract, + tp_group=dist.group.WORLD, + num_vocab_tiles=TP_NUM_TILES, + ) + + # In-process TP=1 run on the full logits: the cross-TP claim is that a + # TP=n result equals the TP=1 result, bit for bit. full = logits.clone().requires_grad_(True) - contract_tp1 = _contract(dtype=dtype_name) - logp_one, lse_one = op(full, targets, contract=contract_tp1, num_vocab_tiles=NUM_TILES) + logp_one, lse_one = op( + full, + targets, + contract=_tp_contract(0, 1, ((0, TP_PADDED_VOCAB),), dtype_name), + num_vocab_tiles=TP_NUM_TILES, + ) (logp_one.sum() + 0.5 * lse_one.sum()).backward() result_queue.put( @@ -358,45 +427,50 @@ def _tp_worker(rank, world_size, init_method, result_queue, scenario): "logp_bits_match": _bitwise_equal(logp_tp, logp_one), "lse_bits_match": _bitwise_equal(lse_tp, lse_one), "grad_bits_match": _bitwise_equal(shard.grad, full.grad[:, start:end]), - "logp": logp_tp.detach().float(), - "lse": lse_tp.detach().float(), + "rerun_bits_match": ( + _bitwise_equal(logp_re, logp_tp) and _bitwise_equal(lse_re, lse_tp) + ), + "logp_bit_pattern": _bits(logp_tp.detach().float().cpu()).tolist(), + "lse_bit_pattern": _bits(lse_tp.detach().float().cpu()).tolist(), } ) - except Exception: + except Exception: # pragma: no cover - forwarded to the parent process result_queue.put({"ok": False, "rank": rank, "traceback": traceback.format_exc()}) raise finally: - if torch.distributed.is_initialized(): - torch.distributed.destroy_process_group() + import torch.distributed as dist + if dist.is_initialized(): + dist.destroy_process_group() -def _run_gloo_scenario(scenario): + +def _run_nccl_scenario(world_size, scenario="correctness", uneven=False, dtype_name="fp32"): ctx = mp.get_context("spawn") with tempfile.TemporaryDirectory() as tmpdir: - init_method = (Path(tmpdir) / "gloo_init").as_uri() + init_method = (Path(tmpdir) / "nccl_init").as_uri() result_queue = ctx.Queue() processes = [ ctx.Process( - target=_tp_worker, - args=(rank, _WORLD_SIZE, init_method, result_queue, scenario), + target=_nccl_worker, + args=(rank, world_size, init_method, result_queue, scenario, uneven, dtype_name), ) - for rank in range(_WORLD_SIZE) + for rank in range(world_size) ] results = [] try: for process in processes: process.start() - for _ in range(_WORLD_SIZE): + for _ in range(world_size): try: - results.append(result_queue.get(timeout=60)) + results.append(result_queue.get(timeout=_SPAWN_TIMEOUT_S)) except queue.Empty: for process in processes: if process.is_alive(): process.terminate() - pytest.fail("timed out waiting for vocab-parallel gloo workers") + pytest.fail(f"timed out waiting for NCCL workers (scenario={scenario})") finally: for process in processes: - process.join(timeout=10) + process.join(timeout=30) if process.is_alive(): process.terminate() results.sort(key=lambda item: item["rank"]) @@ -407,20 +481,57 @@ def _run_gloo_scenario(scenario): return results -@requires_gloo -@pytest.mark.parametrize("scenario", ["even", "uneven", "bf16"]) -def test_tp4_bitwise_identical_to_tp1(scenario): - results = _run_gloo_scenario(scenario) - for result in results: - assert result["logp_bits_match"], f"rank {result['rank']} logp bits differ from TP=1" - assert result["lse_bits_match"], f"rank {result['rank']} lse bits differ from TP=1" - assert result["grad_bits_match"], f"rank {result['rank']} grad bits differ from TP=1" - # Outputs are replicated: every rank must hold identical bits. - for other in results[1:]: - assert _bitwise_equal(results[0]["logp"], other["logp"]) - assert _bitwise_equal(results[0]["lse"], other["lse"]) - - -@requires_gloo -def test_preflight_rejects_mismatched_num_vocab_tiles(): - _run_gloo_scenario("preflight") +class TestCrossTPBitwise: + """TP=n output == TP=1 output, bit for bit, on real NCCL ranks.""" + + @_requires_gpus(2) + @pytest.mark.parametrize("dtype_name", ["fp32", "bf16"]) + @pytest.mark.parametrize("uneven", [False, True], ids=["even", "uneven"]) + def test_tp2_bitwise_identical_to_tp1(self, uneven, dtype_name): + self._assert_matches_tp1(_run_nccl_scenario(2, uneven=uneven, dtype_name=dtype_name)) + + @_requires_gpus(4) + @pytest.mark.parametrize("dtype_name", ["fp32", "bf16"]) + @pytest.mark.parametrize("uneven", [False, True], ids=["even", "uneven"]) + def test_tp4_bitwise_identical_to_tp1(self, uneven, dtype_name): + self._assert_matches_tp1(_run_nccl_scenario(4, uneven=uneven, dtype_name=dtype_name)) + + @staticmethod + def _assert_matches_tp1(results): + for result in results: + rank = result["rank"] + assert result["logp_bits_match"], f"rank {rank} logp bits differ from TP=1" + assert result["lse_bits_match"], f"rank {rank} lse bits differ from TP=1" + assert result["grad_bits_match"], f"rank {rank} grad bits differ from TP=1" + assert result["rerun_bits_match"], f"rank {rank} bits changed between identical runs" + # Outputs are replicated: every rank must hold identical bits. + for other in results[1:]: + assert results[0]["logp_bit_pattern"] == other["logp_bit_pattern"] + assert results[0]["lse_bit_pattern"] == other["lse_bit_pattern"] + + @_requires_gpus(2) + def test_tp2_and_tp4_agree_with_each_other(self): + """The claim is over TP degrees, so pin TP=2 against TP=4 directly.""" + + if _cuda_device_count() < 4: + pytest.skip("needs 4 CUDA devices to compare TP=2 against TP=4") + tp2 = _run_nccl_scenario(2) + tp4 = _run_nccl_scenario(4) + assert tp2[0]["logp_bit_pattern"] == tp4[0]["logp_bit_pattern"] + assert tp2[0]["lse_bit_pattern"] == tp4[0]["lse_bit_pattern"] + + +class TestCrossTPGuards: + """A disagreement must abort loudly on every rank, not strand ranks in a collective.""" + + @_requires_gpus(2) + def test_preflight_rejects_mismatched_num_vocab_tiles(self): + results = _run_nccl_scenario(2, scenario="preflight") + for result in results: + assert "cross-rank preflight failed" in result["message"] + + @_requires_gpus(2) + def test_misaligned_shard_bounds_rejected(self): + results = _run_nccl_scenario(2, scenario="misaligned") + for result in results: + assert "not aligned to the vocab tile size" in result["message"] From 05d19eb20bac1fc165c38b6094a658af83b07145 Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Sat, 8 Aug 2026 22:07:36 +0800 Subject: [PATCH 23/48] test: align PR3 dispatch with latest PR1 guard --- tests/test_logprob_contract.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_logprob_contract.py b/tests/test_logprob_contract.py index 307acfdf..a74f3b4c 100644 --- a/tests/test_logprob_contract.py +++ b/tests/test_logprob_contract.py @@ -297,7 +297,7 @@ def test_ws1_rejections_recorded_when_vocab_parallel_reference_resolves(): candidates.remove(OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP) candidates.append(OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP) - result = registry.get_logprob_op(_contract()) + result = registry.get_logprob_op(_contract(), requested_backend="reference") assert result.capability.backend_id == "pytorch-vocab-parallel-logp-ws2" assert result.provenance["fallback"] is True rejections = " | ".join(result.provenance["prior_rejections"]) From f36a63d217623c4b396ed336b8f965a19582dd7a Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Tue, 11 Aug 2026 20:18:05 +0800 Subject: [PATCH 24/48] feat(ws2): add distributed logprob drift runner --- docs/operators/batch-invariant-logp.md | 65 ++ rl_engine/testing/__init__.py | 5 + .../testing/distributed_logprob_comparison.py | 807 ++++++++++++++++++ rl_engine/testing/logprob_comparison.py | 54 +- rl_engine/testing/logprob_drift.py | 57 ++ tests/test_distributed_logprob_comparison.py | 217 +++++ 6 files changed, 1165 insertions(+), 40 deletions(-) create mode 100644 rl_engine/testing/distributed_logprob_comparison.py create mode 100644 rl_engine/testing/logprob_drift.py create mode 100644 tests/test_distributed_logprob_comparison.py diff --git a/docs/operators/batch-invariant-logp.md b/docs/operators/batch-invariant-logp.md index 3c2fdfc8..8d1b6487 100644 --- a/docs/operators/batch-invariant-logp.md +++ b/docs/operators/batch-invariant-logp.md @@ -300,6 +300,65 @@ batch-invariant suite passed 67 cases. For BF16 shape `[2, 16, 151936]`, both LSE and active-token dlogp had maximum absolute drift `9.5367431640625e-07` against the PyTorch reference, with no backend fallback. +## Distributed WS2 Drift Report + +The issue #241 PR4 runner materializes one TP/CP topology per `torchrun` +invocation. TP partitions the vocabulary and is the only numerical merge axis; +CP partitions token rows and is recorded in provenance without participating in +the vocab-domain LSE merge. For global rank `r`: + +```text +tp_rank = r % tp_world_size +cp_rank = r // tp_world_size +``` + +Every case generates the same seeded FP32 logical logits, targets, and active +mask. The candidate receives a BF16 token/vocab shard through the explicit +`pytorch-vocab-parallel-logp-ws2` backend, while the independent oracle computes +`torch.logsumexp` over the complete real-vocab FP32 token slice. Distributed +dispatch rejects `auto`, capability fallback, topology mismatches, non-tileable +vocabularies, and incomplete materialization. + +Reports follow the issue #116 fields and contain per-rank and aggregate LSE and +active-token dlogp summaries: max/mean/p95/p99 absolute drift, max relative +drift, worst global token position, target id, target owner rank, #108 tolerance, +and pass/fail. Provenance includes TP/CP topology, dtype, shard bounds, backend +capability, contract fingerprint, reduction spec, merge order, transport, and +the exact launch command. Replicated TP outputs are checked bitwise before one +representative per CP shard is included in aggregate statistics. + +Print the scoped TP=1/2/4 x CP=1/2 launch matrix without starting workers: + +```bash +python rl_engine/testing/distributed_logprob_comparison.py \ + --plan \ + --device cuda \ + --dtype bf16 \ + --output artifacts/ws2-logprob/report.json +``` + +Run one TP=2, CP=2 Qwen3-vocab case on four local GPUs: + +```bash +torchrun --standalone --nproc-per-node=4 \ + rl_engine/testing/distributed_logprob_comparison.py \ + --tp 2 \ + --cp 2 \ + --dtype bf16 \ + --backend pytorch-vocab-parallel-logp-ws2 \ + --real-vocab 151936 \ + --padded-vocab 151936 \ + --num-vocab-tiles 64 \ + --batch 2 \ + --seq 16 \ + --prompt-tokens 8 \ + --output artifacts/ws2-logprob/tp2-cp2.json +``` + +The full matrix requires up to eight ranks for TP=4, CP=2. CPU/Gloo cases are +available for topology and artifact validation; the scoped numerical gate is +BF16 on CUDA/NCCL. + ## Minimal Example ```python @@ -335,6 +394,9 @@ gradient batch-invariance, and ignored-row zero gradients. The focused `tests/test_logprob_comparison.py` suite covers TP=1 bitwise regression, direct LSE identity, active-token drift statistics, structured serialization, exact backend diagnostics, and fail-closed provenance. +`tests/test_distributed_logprob_comparison.py` covers topology planning, TP/CP +rank mapping, token/vocab sharding, explicit backend materialization, #116 JSON +artifacts, and a real four-process TP=2, CP=2 Gloo smoke case. Triton tests skip when Triton or CUDA is unavailable. On Windows, run via WSL/Linux with CUDA. @@ -348,6 +410,9 @@ WSL/Linux with CUDA. - `rl_engine/kernels/registry.py` - `tests/test_batch_invariant_logp.py` - `tests/test_logprob_comparison.py` +- `rl_engine/testing/logprob_drift.py` +- `rl_engine/testing/distributed_logprob_comparison.py` +- `tests/test_distributed_logprob_comparison.py` - `benchmarks/benchmark_batch_invariant_logp.py` - `rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py` - `rl_engine/kernels/logprob_contract.py` diff --git a/rl_engine/testing/__init__.py b/rl_engine/testing/__init__.py index 51759abd..97b585fb 100644 --- a/rl_engine/testing/__init__.py +++ b/rl_engine/testing/__init__.py @@ -10,7 +10,9 @@ LogprobComparisonReport, compare_single_gpu_logprob, make_logprob_candidate, + route_rl_kernel_logs_to_stderr, ) +from .logprob_drift import LogprobDriftStats, summarize_logprob_drift from .reference_ops import ( active_token_count, compute_policy_ratio, @@ -27,6 +29,7 @@ "LogprobCandidate", "LogprobComparisonInputs", "LogprobComparisonReport", + "LogprobDriftStats", "SyntheticRLKernelBatch", "active_token_count", "compare_single_gpu_logprob", @@ -37,5 +40,7 @@ "masked_mean", "masked_sum", "selected_logprobs_reference", + "route_rl_kernel_logs_to_stderr", + "summarize_logprob_drift", "summarize_kernel_drift", ] diff --git a/rl_engine/testing/distributed_logprob_comparison.py b/rl_engine/testing/distributed_logprob_comparison.py new file mode 100644 index 00000000..b5b8e283 --- /dev/null +++ b/rl_engine/testing/distributed_logprob_comparison.py @@ -0,0 +1,807 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Distributed WS2 comparison for the vocab-parallel logprob reference.""" + +from __future__ import annotations + +import argparse +import contextlib +import hashlib +import json +import os +import pathlib +import shlex +import sys +from dataclasses import asdict, dataclass +from typing import Any, Sequence + +import torch + +if __package__ in (None, ""): + repo_root = pathlib.Path(__file__).resolve().parents[2] + if str(repo_root) not in sys.path: + sys.path.insert(0, str(repo_root)) + +_import_output = ( + contextlib.redirect_stdout(sys.stderr) + if __package__ in (None, "") + else contextlib.nullcontext() +) +with _import_output: + from rl_engine.kernels.gtest.tolerance import load_contract as load_tolerance_contract + from rl_engine.kernels.logprob_contract import ( + LogprobContract, + LogprobDType, + LogprobRole, + MaskSpec, + ReductionSpec, + ShardingSpec, + ) + from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import ( + BACKEND_ID, + DEFAULT_NUM_VOCAB_TILES, + ) + from rl_engine.kernels.registry import KernelRegistry + from rl_engine.testing.logprob_comparison import route_rl_kernel_logs_to_stderr + from rl_engine.testing.logprob_drift import LogprobDriftStats, summarize_logprob_drift + +_DTYPES = { + "bf16": torch.bfloat16, + "fp16": torch.float16, + "fp32": torch.float32, +} +_TOLERANCE_DTYPES = { + "bf16": "bfloat16", + "fp16": "float16", + "fp32": "float32", +} + + +@dataclass(frozen=True) +class DistributedLogprobCase: + tp_world_size: int + cp_world_size: int + dtype: str = "bf16" + requested_backend: str = BACKEND_ID + real_vocab_size: int = 151936 + padded_vocab_size: int = 151936 + num_vocab_tiles: int = DEFAULT_NUM_VOCAB_TILES + batch_size: int = 2 + sequence_length: int = 16 + prompt_tokens: int = 8 + seed: int = 123 + ignore_index: int = -100 + + def __post_init__(self) -> None: + for name in ( + "tp_world_size", + "cp_world_size", + "real_vocab_size", + "padded_vocab_size", + ): + value = getattr(self, name) + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + raise ValueError(f"{name} must be a positive integer") + if self.dtype not in _DTYPES: + raise ValueError(f"dtype must be one of {sorted(_DTYPES)}") + if not self.requested_backend or self.requested_backend.lower() == "auto": + raise ValueError("distributed cases require an explicit non-auto backend") + if self.padded_vocab_size < self.real_vocab_size: + raise ValueError("padded_vocab_size must be at least real_vocab_size") + if self.num_vocab_tiles < self.tp_world_size: + raise ValueError("num_vocab_tiles must be at least tp_world_size") + if self.padded_vocab_size % self.num_vocab_tiles != 0: + raise ValueError("num_vocab_tiles must divide padded_vocab_size exactly") + if self.batch_size <= 0 or self.sequence_length <= 0: + raise ValueError("batch_size and sequence_length must be positive") + if not 0 <= self.prompt_tokens <= self.sequence_length: + raise ValueError("prompt_tokens must be in [0, sequence_length]") + + @property + def world_size(self) -> int: + return self.tp_world_size * self.cp_world_size + + @property + def num_tokens(self) -> int: + return self.batch_size * self.sequence_length + + @property + def case_id(self) -> str: + encoded = json.dumps(asdict(self), sort_keys=True, separators=(",", ":")).encode() + return hashlib.sha256(encoded).hexdigest()[:16] + + +@dataclass(frozen=True) +class RankTopology: + global_rank: int + world_size: int + tp_rank: int + tp_world_size: int + cp_rank: int + cp_world_size: int + tp_group_ranks: tuple[int, ...] + + +@dataclass(frozen=True) +class DriftDetail: + stats: LogprobDriftStats + max_rel: float + worst_global_token: int | None + worst_target_id: int | None + worst_owner_rank: int | None + candidate_value: float | None + reference_value: float | None + atol: float + rtol: float + passed: bool + + +@dataclass(frozen=True) +class RankLogprobReport: + global_rank: int + tp_rank: int + tp_world_size: int + cp_rank: int + cp_world_size: int + sp_world_size: int + dp_world_size: int + token_start: int + token_end: int + vocab_start: int + vocab_end: int + device: str + requested_backend: str + actual_backend: str + fallback: bool + contract_fingerprint: str + contract: dict[str, Any] + capability: dict[str, Any] + tp_outputs_bitwise_replicated: bool + lse: DriftDetail + dlogp: DriftDetail + passed: bool + + +@dataclass(frozen=True) +class DistributedLogprobReport: + schema_version: int + case_id: str + case: dict[str, Any] + launch_command: str + environment: dict[str, Any] + ranks: tuple[RankLogprobReport, ...] + aggregate: dict[str, DriftDetail] + passed: bool + + def to_dict(self) -> dict[str, Any]: + return asdict(self) + + +@dataclass(frozen=True) +class _RankPayload: + report: RankLogprobReport + candidate_logp: torch.Tensor + candidate_lse: torch.Tensor + reference_logp: torch.Tensor + reference_lse: torch.Tensor + active_mask: torch.Tensor + target_ids: torch.Tensor + global_positions: torch.Tensor + + +def plan_distributed_logprob_cases( + *, + tp_world_sizes: Sequence[int] = (1, 2, 4), + cp_world_sizes: Sequence[int] = (1, 2), + **overrides: Any, +) -> tuple[DistributedLogprobCase, ...]: + """Build the scoped issue #241 topology product in deterministic order.""" + + return tuple( + DistributedLogprobCase(tp_world_size=tp, cp_world_size=cp, **overrides) + for tp in tp_world_sizes + for cp in cp_world_sizes + ) + + +def rank_topology(case: DistributedLogprobCase, global_rank: int) -> RankTopology: + if not 0 <= global_rank < case.world_size: + raise ValueError(f"global_rank must be in [0, {case.world_size})") + cp_rank, tp_rank = divmod(global_rank, case.tp_world_size) + group_start = cp_rank * case.tp_world_size + return RankTopology( + global_rank=global_rank, + world_size=case.world_size, + tp_rank=tp_rank, + tp_world_size=case.tp_world_size, + cp_rank=cp_rank, + cp_world_size=case.cp_world_size, + tp_group_ranks=tuple(range(group_start, group_start + case.tp_world_size)), + ) + + +def token_shard_bounds(num_tokens: int, cp_world_size: int) -> tuple[tuple[int, int], ...]: + """Partition token rows contiguously, allowing a one-row imbalance.""" + + if num_tokens < cp_world_size: + raise ValueError("num_tokens must be at least cp_world_size") + quotient, remainder = divmod(num_tokens, cp_world_size) + bounds = [] + cursor = 0 + for cp_rank in range(cp_world_size): + count = quotient + int(cp_rank < remainder) + bounds.append((cursor, cursor + count)) + cursor += count + return tuple(bounds) + + +def vocab_shard_bounds(case: DistributedLogprobCase) -> tuple[tuple[int, int], ...]: + """Assign complete global vocab tiles to TP ranks.""" + + tile_size = case.padded_vocab_size // case.num_vocab_tiles + quotient, remainder = divmod(case.num_vocab_tiles, case.tp_world_size) + bounds = [] + cursor_tiles = 0 + for tp_rank in range(case.tp_world_size): + tile_count = quotient + int(tp_rank < remainder) + start = cursor_tiles * tile_size + cursor_tiles += tile_count + bounds.append((start, cursor_tiles * tile_size)) + return tuple(bounds) + + +def format_launch_command( + case: DistributedLogprobCase, + *, + output: str | pathlib.Path, + device: str = "cuda", + dist_backend: str | None = None, +) -> str: + backend = dist_backend or ("nccl" if device == "cuda" else "gloo") + arguments = [ + "torchrun", + "--standalone", + f"--nproc-per-node={case.world_size}", + "rl_engine/testing/distributed_logprob_comparison.py", + "--tp", + str(case.tp_world_size), + "--cp", + str(case.cp_world_size), + "--dtype", + case.dtype, + "--backend", + case.requested_backend, + "--real-vocab", + str(case.real_vocab_size), + "--padded-vocab", + str(case.padded_vocab_size), + "--num-vocab-tiles", + str(case.num_vocab_tiles), + "--batch", + str(case.batch_size), + "--seq", + str(case.sequence_length), + "--prompt-tokens", + str(case.prompt_tokens), + "--seed", + str(case.seed), + "--device", + device, + "--dist-backend", + backend, + "--output", + str(output), + ] + return shlex.join(arguments) + + +def _canonical_inputs( + case: DistributedLogprobCase, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + generator = torch.Generator(device="cpu").manual_seed(case.seed) + logits = torch.randn( + case.num_tokens, + case.padded_vocab_size, + generator=generator, + dtype=torch.float32, + ) + targets = torch.randint( + 0, + case.real_vocab_size, + (case.num_tokens,), + generator=generator, + dtype=torch.long, + ) + active = torch.ones((case.batch_size, case.sequence_length), dtype=torch.bool) + active[:, : case.prompt_tokens] = False + active = active.reshape(-1) + targets = targets.masked_fill(~active, case.ignore_index) + return logits, targets, active + + +def _make_contract( + case: DistributedLogprobCase, + topology: RankTopology, + active_mask: torch.Tensor, +) -> LogprobContract: + return LogprobContract( + role=LogprobRole.TRAIN, + dtype=LogprobDType(case.dtype), + mask=MaskSpec( + num_tokens=int(active_mask.numel()), + active_mask=tuple(bool(value) for value in active_mask.tolist()), + ignore_index=case.ignore_index, + ), + sharding=ShardingSpec( + tp_rank=topology.tp_rank, + tp_world_size=case.tp_world_size, + vocab_shard_bounds=vocab_shard_bounds(case), + real_vocab_size=case.real_vocab_size, + padded_vocab_size=case.padded_vocab_size, + cp_rank=topology.cp_rank, + cp_world_size=case.cp_world_size, + ), + reduction=ReductionSpec(), + ) + + +def _fp32_oracle( + logits: torch.Tensor, + target_ids: torch.Tensor, + active_mask: torch.Tensor, + real_vocab_size: int, +) -> tuple[torch.Tensor, torch.Tensor]: + real_logits = logits[:, :real_vocab_size].float() + lse = torch.logsumexp(real_logits, dim=-1) + safe_targets = target_ids.masked_fill(~active_mask, 0) + selected = real_logits.gather(1, safe_targets.unsqueeze(1)).squeeze(1) + logp = torch.where(active_mask, selected - lse, torch.zeros_like(lse)) + return logp, lse + + +def _resolve_tolerance(dtype: str) -> tuple[float, float]: + entry = load_tolerance_contract()["accuracy"]["default"]["logprob"] + tolerance = entry[_TOLERANCE_DTYPES[dtype]] + return float(tolerance["atol"]), float(tolerance["rtol"]) + + +def _drift_detail( + candidate: torch.Tensor, + reference: torch.Tensor, + *, + target_ids: torch.Tensor, + global_positions: torch.Tensor, + sharding: ShardingSpec, + atol: float, + rtol: float, + mask: torch.Tensor | None = None, +) -> DriftDetail: + stats = summarize_logprob_drift(candidate, reference, mask=mask) + diff = (candidate.float() - reference.float()).abs() + selected = torch.ones_like(diff, dtype=torch.bool) if mask is None else mask.to(diff.device) + if not bool(selected.any().item()): + return DriftDetail(stats, 0.0, None, None, None, None, None, atol, rtol, True) + + selected_diff = diff[selected] + selected_ref = reference.float()[selected] + relative = selected_diff / selected_ref.abs().clamp_min(torch.finfo(torch.float32).tiny) + selected_indices = torch.arange(diff.numel(), device=diff.device)[selected] + worst_selected = int(selected_diff.argmax().item()) + worst_local = int(selected_indices[worst_selected].item()) + target_id = int(target_ids[worst_local].item()) + close = selected_diff <= atol + rtol * selected_ref.abs() + return DriftDetail( + stats=stats, + max_rel=float(relative.max().item()), + worst_global_token=int(global_positions[worst_local].item()), + worst_target_id=target_id, + worst_owner_rank=sharding.owner_rank(target_id) if target_id >= 0 else None, + candidate_value=float(candidate[worst_local].float().item()), + reference_value=float(reference[worst_local].float().item()), + atol=atol, + rtol=rtol, + passed=bool(close.all().item()), + ) + + +def _tp_outputs_replicated( + logp: torch.Tensor, + lse: torch.Tensor, + *, + tp_group: Any, + tp_world_size: int, +) -> bool: + if tp_world_size == 1: + return True + import torch.distributed as dist + + gathered_logp = [torch.empty_like(logp) for _ in range(tp_world_size)] + gathered_lse = [torch.empty_like(lse) for _ in range(tp_world_size)] + dist.all_gather(gathered_logp, logp.contiguous(), group=tp_group) + dist.all_gather(gathered_lse, lse.contiguous(), group=tp_group) + return all(torch.equal(logp, value) for value in gathered_logp) and all( + torch.equal(lse, value) for value in gathered_lse + ) + + +def _execute_rank( + case: DistributedLogprobCase, + topology: RankTopology, + *, + device: torch.device, + tp_group: Any, +) -> _RankPayload: + full_logits, full_targets, full_active = _canonical_inputs(case) + token_start, token_end = token_shard_bounds(case.num_tokens, case.cp_world_size)[ + topology.cp_rank + ] + vocab_start, vocab_end = vocab_shard_bounds(case)[topology.tp_rank] + token_slice = slice(token_start, token_end) + local_active = full_active[token_slice].to(device=device) + local_targets = full_targets[token_slice].to(device=device) + local_fp32 = full_logits[token_slice].to(device=device) + local_logits = local_fp32[:, vocab_start:vocab_end].to(_DTYPES[case.dtype]).contiguous() + positions = torch.arange(token_start, token_end, device=device, dtype=torch.long) + + contract = _make_contract(case, topology, local_active.cpu()) + dispatch = KernelRegistry().get_logprob_op( + contract, + requested_backend=case.requested_backend, + ) + fallback = bool(dispatch.provenance["fallback"]) + if fallback: + raise RuntimeError("distributed logprob dispatch materialized through a fallback") + requested_policy = case.requested_backend.lower() + if requested_policy not in {"reference", "production"} and ( + case.requested_backend != dispatch.capability.backend_id + ): + raise RuntimeError( + f"requested backend {case.requested_backend!r} materialized as " + f"{dispatch.capability.backend_id!r}" + ) + + candidate_logp, candidate_lse = dispatch.op( + local_logits, + local_targets, + contract=contract, + tp_group=tp_group, + num_vocab_tiles=case.num_vocab_tiles, + validate=True, + ) + reference_logp, reference_lse = _fp32_oracle( + local_fp32, + local_targets, + local_active, + case.real_vocab_size, + ) + replicated = _tp_outputs_replicated( + candidate_logp, + candidate_lse, + tp_group=tp_group, + tp_world_size=case.tp_world_size, + ) + atol, rtol = _resolve_tolerance(case.dtype) + lse_drift = _drift_detail( + candidate_lse, + reference_lse, + target_ids=local_targets, + global_positions=positions, + sharding=contract.sharding, + atol=atol, + rtol=rtol, + ) + dlogp_drift = _drift_detail( + candidate_logp, + reference_logp, + target_ids=local_targets, + global_positions=positions, + sharding=contract.sharding, + atol=atol, + rtol=rtol, + mask=local_active, + ) + rank_report = RankLogprobReport( + global_rank=topology.global_rank, + tp_rank=topology.tp_rank, + tp_world_size=topology.tp_world_size, + cp_rank=topology.cp_rank, + cp_world_size=topology.cp_world_size, + sp_world_size=1, + dp_world_size=1, + token_start=token_start, + token_end=token_end, + vocab_start=vocab_start, + vocab_end=vocab_end, + device=str(device), + requested_backend=case.requested_backend, + actual_backend=dispatch.capability.backend_id, + fallback=fallback, + contract_fingerprint=contract.cross_rank_fingerprint(), + contract=contract.to_dict(), + capability=dispatch.capability.to_dict(), + tp_outputs_bitwise_replicated=replicated, + lse=lse_drift, + dlogp=dlogp_drift, + passed=replicated and lse_drift.passed and dlogp_drift.passed, + ) + return _RankPayload( + report=rank_report, + candidate_logp=candidate_logp.detach().cpu(), + candidate_lse=candidate_lse.detach().cpu(), + reference_logp=reference_logp.detach().cpu(), + reference_lse=reference_lse.detach().cpu(), + active_mask=local_active.cpu(), + target_ids=local_targets.cpu(), + global_positions=positions.cpu(), + ) + + +def _aggregate_payloads( + case: DistributedLogprobCase, + payloads: Sequence[_RankPayload], +) -> dict[str, DriftDetail]: + representatives = sorted( + (payload for payload in payloads if payload.report.tp_rank == 0), + key=lambda payload: payload.report.cp_rank, + ) + if len(representatives) != case.cp_world_size: + raise RuntimeError("missing one or more CP representatives in rank reports") + candidate_logp = torch.cat([payload.candidate_logp for payload in representatives]) + candidate_lse = torch.cat([payload.candidate_lse for payload in representatives]) + reference_logp = torch.cat([payload.reference_logp for payload in representatives]) + reference_lse = torch.cat([payload.reference_lse for payload in representatives]) + active_mask = torch.cat([payload.active_mask for payload in representatives]) + target_ids = torch.cat([payload.target_ids for payload in representatives]) + positions = torch.cat([payload.global_positions for payload in representatives]) + sharding = _make_contract( + case, + rank_topology(case, 0), + active_mask, + ).sharding + atol, rtol = _resolve_tolerance(case.dtype) + return { + "lse": _drift_detail( + candidate_lse, + reference_lse, + target_ids=target_ids, + global_positions=positions, + sharding=sharding, + atol=atol, + rtol=rtol, + ), + "dlogp": _drift_detail( + candidate_logp, + reference_logp, + target_ids=target_ids, + global_positions=positions, + sharding=sharding, + atol=atol, + rtol=rtol, + mask=active_mask, + ), + } + + +def _create_tp_group(case: DistributedLogprobCase, topology: RankTopology) -> Any: + if case.world_size == 1: + return None + import torch.distributed as dist + + selected = None + for cp_rank in range(case.cp_world_size): + start = cp_rank * case.tp_world_size + ranks = list(range(start, start + case.tp_world_size)) + group = dist.new_group(ranks=ranks) + if topology.global_rank in ranks: + selected = group + return selected + + +def run_distributed_logprob_case( + case: DistributedLogprobCase, + *, + device_name: str = "cuda", + dist_backend: str | None = None, + output: str | pathlib.Path, +) -> DistributedLogprobReport | None: + """Run one materialized topology; only global rank zero returns the report.""" + + import torch.distributed as dist + + backend = dist_backend or ("nccl" if device_name == "cuda" else "gloo") + rank = int(os.environ.get("RANK", "0")) + world_size = int(os.environ.get("WORLD_SIZE", "1")) + local_rank = int(os.environ.get("LOCAL_RANK", "0")) + if world_size != case.world_size: + raise RuntimeError( + f"WORLD_SIZE={world_size} does not match TP*CP={case.world_size}; " + "launch exactly the topology declared by the case" + ) + initialized_here = False + if world_size > 1 and not dist.is_initialized(): + dist.init_process_group(backend=backend) + initialized_here = True + if dist.is_initialized(): + if dist.get_world_size() != world_size or dist.get_rank() != rank: + raise RuntimeError("initialized process group does not match RANK/WORLD_SIZE") + + if device_name == "cuda": + if not torch.cuda.is_available(): + raise RuntimeError("CUDA was requested but is unavailable") + device = torch.device("cuda", local_rank) + torch.cuda.set_device(device) + elif device_name == "cpu": + device = torch.device("cpu") + else: + raise ValueError("device must be cuda or cpu") + + topology = rank_topology(case, rank) + tp_group = _create_tp_group(case, topology) + try: + payload = _execute_rank(case, topology, device=device, tp_group=tp_group) + if world_size == 1: + payloads = [payload] + else: + gathered: list[Any] = [None] * world_size + dist.all_gather_object(gathered, payload) + payloads = gathered + + report = None + if rank == 0: + aggregate = _aggregate_payloads(case, payloads) + rank_reports = tuple( + payload.report + for payload in sorted(payloads, key=lambda item: item.report.global_rank) + ) + actual_backends = sorted({rank_report.actual_backend for rank_report in rank_reports}) + reduction_specs = { + json.dumps(rank_report.contract["reduction"], sort_keys=True) + for rank_report in rank_reports + } + materialization_consistent = len(actual_backends) == 1 and len(reduction_specs) == 1 + launch_command = format_launch_command( + case, + output=output, + device=device_name, + dist_backend=backend, + ) + report = DistributedLogprobReport( + schema_version=1, + case_id=case.case_id, + case=asdict(case), + launch_command=launch_command, + environment={ + "python": sys.version.split()[0], + "torch": torch.__version__, + "torch_cuda": torch.version.cuda, + "dist_backend": backend, + "world_size": world_size, + "sp_world_size": 1, + "dp_world_size": 1, + "materialization": { + "actual_backends": actual_backends, + "consistent": materialization_consistent, + }, + "communication": { + "logprob_merge_axis": "tp_vocab", + "cp_is_merge_axis": False, + "report_collection": ("all_gather_object" if world_size > 1 else "none"), + }, + }, + ranks=rank_reports, + aggregate=aggregate, + passed=( + materialization_consistent + and all(rank_report.passed for rank_report in rank_reports) + and all(detail.passed for detail in aggregate.values()) + ), + ) + output_path = pathlib.Path(output) + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text( + json.dumps(report.to_dict(), indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + if world_size > 1: + dist.barrier() + return report + finally: + if initialized_here and dist.is_initialized(): + dist.destroy_process_group() + + +def _case_from_args(args: argparse.Namespace) -> DistributedLogprobCase: + return DistributedLogprobCase( + tp_world_size=args.tp, + cp_world_size=args.cp, + dtype=args.dtype, + requested_backend=args.backend, + real_vocab_size=args.real_vocab, + padded_vocab_size=args.padded_vocab, + num_vocab_tiles=args.num_vocab_tiles, + batch_size=args.batch, + sequence_length=args.seq, + prompt_tokens=args.prompt_tokens, + seed=args.seed, + ) + + +def _parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Run the WS2 distributed logprob drift report.") + parser.add_argument("--plan", action="store_true", help="Print the six scoped launch commands.") + parser.add_argument("--tp", type=int, default=1) + parser.add_argument("--cp", type=int, default=1) + parser.add_argument("--dtype", choices=tuple(_DTYPES), default="bf16") + parser.add_argument("--backend", default=BACKEND_ID) + parser.add_argument("--real-vocab", type=int, default=151936) + parser.add_argument("--padded-vocab", type=int, default=151936) + parser.add_argument("--num-vocab-tiles", type=int, default=DEFAULT_NUM_VOCAB_TILES) + parser.add_argument("--batch", type=int, default=2) + parser.add_argument("--seq", type=int, default=16) + parser.add_argument("--prompt-tokens", type=int, default=8) + parser.add_argument("--seed", type=int, default=123) + parser.add_argument("--device", choices=("cuda", "cpu"), default="cuda") + parser.add_argument("--dist-backend", choices=("nccl", "gloo"), default=None) + parser.add_argument("--output", default="artifacts/ws2-logprob/report.json") + return parser.parse_args(argv) + + +def main(argv: Sequence[str] | None = None) -> None: + route_rl_kernel_logs_to_stderr() + args = _parse_args(argv) + if args.plan: + cases = plan_distributed_logprob_cases( + dtype=args.dtype, + requested_backend=args.backend, + real_vocab_size=args.real_vocab, + padded_vocab_size=args.padded_vocab, + num_vocab_tiles=args.num_vocab_tiles, + batch_size=args.batch, + sequence_length=args.seq, + prompt_tokens=args.prompt_tokens, + seed=args.seed, + ) + commands = [ + format_launch_command( + case, + output=pathlib.Path(args.output).parent + / f"tp{case.tp_world_size}-cp{case.cp_world_size}.json", + device=args.device, + dist_backend=args.dist_backend, + ) + for case in cases + ] + print(json.dumps({"commands": commands}, indent=2)) + return + + case = _case_from_args(args) + report = run_distributed_logprob_case( + case, + device_name=args.device, + dist_backend=args.dist_backend, + output=args.output, + ) + if report is not None: + print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) + if not report.passed: + raise SystemExit(1) + + +__all__ = [ + "DistributedLogprobCase", + "DistributedLogprobReport", + "DriftDetail", + "RankLogprobReport", + "RankTopology", + "format_launch_command", + "plan_distributed_logprob_cases", + "rank_topology", + "run_distributed_logprob_case", + "token_shard_bounds", + "vocab_shard_bounds", +] + + +if __name__ == "__main__": + main() diff --git a/rl_engine/testing/logprob_comparison.py b/rl_engine/testing/logprob_comparison.py index d207f75b..1e9d4e63 100644 --- a/rl_engine/testing/logprob_comparison.py +++ b/rl_engine/testing/logprob_comparison.py @@ -20,6 +20,9 @@ repo_root = pathlib.Path(__file__).resolve().parents[2] if str(repo_root) not in sys.path: sys.path.insert(0, str(repo_root)) + from logprob_drift import LogprobDriftStats, summarize_logprob_drift +else: + from .logprob_drift import LogprobDriftStats, summarize_logprob_drift class LogprobBackendUnavailable(RuntimeError): @@ -43,20 +46,11 @@ class LogprobCandidate: provenance: dict[str, Any] = field(default_factory=dict) -@dataclass(frozen=True) -class _DriftStats: - max_abs: float - mean_abs: float - p95_abs: float - p99_abs: float - active_count: int - - @dataclass(frozen=True) class _LogprobPathDrift: candidate_name: str - lse: _DriftStats - dlogp: _DriftStats + lse: LogprobDriftStats + dlogp: LogprobDriftStats bitwise_logp: bool provenance: dict[str, Any] @@ -158,8 +152,8 @@ def compare_single_gpu_logprob( drifts.append( _LogprobPathDrift( candidate_name=candidate.name, - lse=_drift_stats(lse, reference_lse), - dlogp=_drift_stats(logp, reference_logp, mask=active_mask), + lse=summarize_logprob_drift(lse, reference_lse), + dlogp=summarize_logprob_drift(logp, reference_logp, mask=active_mask), bitwise_logp=torch.equal(logp, reference_logp), provenance=_candidate_provenance(candidate), ) @@ -231,31 +225,6 @@ def _candidate_provenance(candidate: LogprobCandidate) -> dict[str, Any]: } -def _drift_stats( - candidate: torch.Tensor, - reference: torch.Tensor, - *, - mask: torch.Tensor | None = None, -) -> _DriftStats: - if candidate.shape != reference.shape: - raise ValueError( - f"candidate shape {tuple(candidate.shape)} must match reference shape " - f"{tuple(reference.shape)}" - ) - diff = (candidate.float() - reference.float()).abs() - values = diff.reshape(-1) if mask is None else diff[mask.to(device=diff.device)] - count = int(values.numel()) - if count == 0: - return _DriftStats(0.0, 0.0, 0.0, 0.0, 0) - return _DriftStats( - max_abs=float(values.max().item()), - mean_abs=float(values.mean().item()), - p95_abs=float(torch.quantile(values, 0.95).item()), - p99_abs=float(torch.quantile(values, 0.99).item()), - active_count=count, - ) - - def _validate_inputs( inputs: LogprobComparisonInputs, ) -> tuple[torch.Tensor, torch.Tensor]: @@ -301,7 +270,7 @@ def _device(name: str) -> torch.device: return torch.device(name) -def _route_rl_kernel_logs_to_stderr() -> None: +def route_rl_kernel_logs_to_stderr() -> None: from rl_engine.utils.logger import logger for handler in logger.handlers: @@ -309,6 +278,10 @@ def _route_rl_kernel_logs_to_stderr() -> None: handler.setStream(sys.stderr) +def _route_rl_kernel_logs_to_stderr() -> None: + route_rl_kernel_logs_to_stderr() + + def _parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: parser = argparse.ArgumentParser( description="Run the WS2 TP=1 selected-logprob/LSE comparison harness." @@ -330,7 +303,7 @@ def _parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: def main(argv: Sequence[str] | None = None) -> None: - _route_rl_kernel_logs_to_stderr() + route_rl_kernel_logs_to_stderr() args = _parse_args(argv) device = _device(args.device) if args.batch < 1 or args.seq < 1 or args.vocab < 1: @@ -374,6 +347,7 @@ def main(argv: Sequence[str] | None = None) -> None: "LogprobComparisonReport", "compare_single_gpu_logprob", "make_logprob_candidate", + "route_rl_kernel_logs_to_stderr", ] diff --git a/rl_engine/testing/logprob_drift.py b/rl_engine/testing/logprob_drift.py new file mode 100644 index 00000000..4996dfbc --- /dev/null +++ b/rl_engine/testing/logprob_drift.py @@ -0,0 +1,57 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Shared selected-logprob drift summaries.""" + +from __future__ import annotations + +from dataclasses import dataclass + +import torch + + +@dataclass(frozen=True) +class LogprobDriftStats: + max_abs: float + mean_abs: float + p95_abs: float + p99_abs: float + active_count: int + + +def summarize_logprob_drift( + candidate: torch.Tensor, + reference: torch.Tensor, + *, + mask: torch.Tensor | None = None, +) -> LogprobDriftStats: + """Summarize absolute drift, optionally over active rows only.""" + + if candidate.shape != reference.shape: + raise ValueError( + f"candidate shape {tuple(candidate.shape)} must match reference shape " + f"{tuple(reference.shape)}" + ) + diff = (candidate.float() - reference.float()).abs() + if mask is None: + values = diff.reshape(-1) + else: + if mask.shape != diff.shape: + raise ValueError("mask shape must match candidate and reference") + if mask.dtype != torch.bool: + raise ValueError("mask must be bool") + values = diff[mask.to(device=diff.device)] + + count = int(values.numel()) + if count == 0: + return LogprobDriftStats(0.0, 0.0, 0.0, 0.0, 0) + return LogprobDriftStats( + max_abs=float(values.max().item()), + mean_abs=float(values.mean().item()), + p95_abs=float(torch.quantile(values, 0.95).item()), + p99_abs=float(torch.quantile(values, 0.99).item()), + active_count=count, + ) + + +__all__ = ["LogprobDriftStats", "summarize_logprob_drift"] diff --git a/tests/test_distributed_logprob_comparison.py b/tests/test_distributed_logprob_comparison.py new file mode 100644 index 00000000..022ccc16 --- /dev/null +++ b/tests/test_distributed_logprob_comparison.py @@ -0,0 +1,217 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +from __future__ import annotations + +import json +import os +import subprocess +import sys +from pathlib import Path + +import pytest +import torch + +from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import BACKEND_ID +from rl_engine.testing.distributed_logprob_comparison import ( + DistributedLogprobCase, + format_launch_command, + plan_distributed_logprob_cases, + rank_topology, + run_distributed_logprob_case, + token_shard_bounds, + vocab_shard_bounds, +) +from rl_engine.testing.logprob_drift import summarize_logprob_drift + + +def _small_case(*, tp: int = 1, cp: int = 1) -> DistributedLogprobCase: + return DistributedLogprobCase( + tp_world_size=tp, + cp_world_size=cp, + real_vocab_size=13, + padded_vocab_size=16, + num_vocab_tiles=8, + batch_size=1, + sequence_length=4, + prompt_tokens=1, + seed=7, + ) + + +def test_planner_builds_the_scoped_topology_product(): + cases = plan_distributed_logprob_cases( + real_vocab_size=13, + padded_vocab_size=16, + num_vocab_tiles=8, + ) + + assert [(case.tp_world_size, case.cp_world_size) for case in cases] == [ + (1, 1), + (1, 2), + (2, 1), + (2, 2), + (4, 1), + (4, 2), + ] + assert [case.world_size for case in cases] == [1, 2, 2, 4, 4, 8] + + +def test_rank_mapping_keeps_cp_out_of_the_tp_merge_axis(): + case = _small_case(tp=2, cp=2) + + assert rank_topology(case, 0).tp_group_ranks == (0, 1) + assert rank_topology(case, 1).tp_group_ranks == (0, 1) + assert rank_topology(case, 2).tp_group_ranks == (2, 3) + assert rank_topology(case, 3).tp_group_ranks == (2, 3) + assert [ + (rank_topology(case, rank).cp_rank, rank_topology(case, rank).tp_rank) for rank in range(4) + ] == [ + (0, 0), + (0, 1), + (1, 0), + (1, 1), + ] + + +def test_token_and_vocab_bounds_cover_each_axis_once(): + case = _small_case(tp=4, cp=2) + + assert token_shard_bounds(case.num_tokens, case.cp_world_size) == ((0, 2), (2, 4)) + assert vocab_shard_bounds(case) == ((0, 4), (4, 8), (8, 12), (12, 16)) + + +def test_case_rejects_implicit_backend_and_non_tileable_vocab(): + with pytest.raises(ValueError, match="explicit non-auto backend"): + DistributedLogprobCase(tp_world_size=2, cp_world_size=1, requested_backend="auto") + with pytest.raises(ValueError, match="must divide"): + DistributedLogprobCase( + tp_world_size=2, + cp_world_size=1, + padded_vocab_size=15, + real_vocab_size=13, + num_vocab_tiles=8, + ) + + +def test_launch_command_records_the_materialized_case(tmp_path): + case = _small_case(tp=2, cp=2) + command = format_launch_command(case, output=tmp_path / "report.json") + + assert "--nproc-per-node=4" in command + assert "--tp 2 --cp 2" in command + assert f"--backend {BACKEND_ID}" in command + assert "--real-vocab 13 --padded-vocab 16" in command + + +def test_shared_pr2_drift_summary_preserves_active_mask_semantics(): + candidate = torch.tensor([100.0, 1.0, 3.0]) + reference = torch.tensor([0.0, 2.0, 1.0]) + mask = torch.tensor([False, True, True]) + + stats = summarize_logprob_drift(candidate, reference, mask=mask) + + assert stats.active_count == 2 + assert stats.max_abs == 2.0 + assert stats.mean_abs == 1.5 + + +def test_tp1_cpu_case_writes_116_compatible_artifact(tmp_path, monkeypatch): + monkeypatch.delenv("RANK", raising=False) + monkeypatch.delenv("LOCAL_RANK", raising=False) + monkeypatch.delenv("WORLD_SIZE", raising=False) + output = tmp_path / "tp1-cp1.json" + + report = run_distributed_logprob_case( + _small_case(), + device_name="cpu", + dist_backend="gloo", + output=output, + ) + + assert report is not None and report.passed + assert report.aggregate["lse"].stats.active_count == 4 + assert report.aggregate["dlogp"].stats.active_count == 3 + assert report.ranks[0].actual_backend == BACKEND_ID + assert report.ranks[0].fallback is False + assert report.ranks[0].tp_outputs_bitwise_replicated + payload = json.loads(output.read_text(encoding="utf-8")) + assert payload["schema_version"] == 1 + assert payload["ranks"][0]["contract"]["reduction"]["cp_is_merge_axis"] is False + assert payload["ranks"][0]["sp_world_size"] == 1 + assert payload["ranks"][0]["dp_world_size"] == 1 + assert payload["environment"]["materialization"]["consistent"] is True + assert payload["aggregate"]["dlogp"]["worst_target_id"] is not None + assert payload["launch_command"].startswith("torchrun --standalone") + + +def test_world_size_mismatch_fails_before_process_group_init(tmp_path, monkeypatch): + monkeypatch.setenv("WORLD_SIZE", "2") + monkeypatch.setenv("RANK", "0") + + with pytest.raises(RuntimeError, match=r"does not match TP\*CP"): + run_distributed_logprob_case( + _small_case(), + device_name="cpu", + dist_backend="gloo", + output=tmp_path / "unused.json", + ) + + +@pytest.mark.skipif(not torch.distributed.is_available(), reason="torch.distributed required") +def test_tp2_cp2_gloo_cli_emits_per_rank_report(tmp_path): + script = ( + Path(__file__).resolve().parents[1] + / "rl_engine" + / "testing" + / "distributed_logprob_comparison.py" + ) + output = tmp_path / "tp2-cp2.json" + environment = os.environ.copy() + environment.setdefault("OMP_NUM_THREADS", "1") + result = subprocess.run( + [ + sys.executable, + "-m", + "torch.distributed.run", + "--standalone", + "--nproc-per-node=4", + str(script), + "--tp", + "2", + "--cp", + "2", + "--device", + "cpu", + "--dist-backend", + "gloo", + "--real-vocab", + "13", + "--padded-vocab", + "16", + "--num-vocab-tiles", + "8", + "--batch", + "1", + "--seq", + "4", + "--prompt-tokens", + "1", + "--output", + str(output), + ], + check=True, + capture_output=True, + text=True, + timeout=120, + env=environment, + ) + + payload = json.loads(output.read_text(encoding="utf-8")) + assert payload["passed"] + assert len(payload["ranks"]) == 4 + assert all(rank["tp_outputs_bitwise_replicated"] for rank in payload["ranks"]) + assert {rank["actual_backend"] for rank in payload["ranks"]} == {BACKEND_ID} + assert {rank["cp_rank"] for rank in payload["ranks"]} == {0, 1} + assert payload["aggregate"]["dlogp"]["stats"]["active_count"] == 3 + assert json.loads(result.stdout)["case"]["tp_world_size"] == 2 From 1d9bac1a258d857485ae0b4bd00fa0e768f126cc Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Tue, 11 Aug 2026 22:24:25 +0800 Subject: [PATCH 25/48] ci(ws2): run logprob comparison tests --- .github/workflows/ci.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2d9ba91e..993ccf52 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -82,6 +82,11 @@ jobs: - name: Run WS2 Vocab-Parallel Logprob Tests (CPU-safe) run: python -m pytest tests/test_vocab_parallel_logp.py -v + - name: Run WS2 Logprob Comparison Tests (CPU-safe) + run: | + python -m pytest tests/test_logprob_comparison.py -v + python -m pytest tests/test_distributed_logprob_comparison.py -v + docs: runs-on: ubuntu-latest steps: From f6b5a07798996ed235aae3c3c360e2bb0fe1256c Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Tue, 11 Aug 2026 23:02:17 +0800 Subject: [PATCH 26/48] fix(ws2): harden distributed drift reporting --- docs/operators/batch-invariant-logp.md | 5 ++- .../testing/distributed_logprob_comparison.py | 15 ++++++--- tests/test_distributed_logprob_comparison.py | 31 +++++++++++++++++++ 3 files changed, 46 insertions(+), 5 deletions(-) diff --git a/docs/operators/batch-invariant-logp.md b/docs/operators/batch-invariant-logp.md index 8d1b6487..b060c10f 100644 --- a/docs/operators/batch-invariant-logp.md +++ b/docs/operators/batch-invariant-logp.md @@ -58,7 +58,10 @@ or device, dispatch is unchanged (Triton -> PyTorch). `VocabParallelLogprobOp` (`rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py`) -**TP=1, TP=2, and TP=4 produce bit-identical results.** +defines a cross-TP bitwise contract for TP=1, TP=2, and TP=4 when +`num_vocab_tiles` is fixed and every vocabulary-shard boundary is tile-aligned. +The complete BF16 CUDA/NCCL validation matrix for this contract is tracked by +issue #241 PR4. 1. Split the padded vocabulary into `num_vocab_tiles` fixed tiles. 2. Each rank computes fp32 `(max, sumexp)` for the tiles it owns. Every tile diff --git a/rl_engine/testing/distributed_logprob_comparison.py b/rl_engine/testing/distributed_logprob_comparison.py index b5b8e283..ad3bb05d 100644 --- a/rl_engine/testing/distributed_logprob_comparison.py +++ b/rl_engine/testing/distributed_logprob_comparison.py @@ -7,6 +7,7 @@ import argparse import contextlib +import datetime import hashlib import json import os @@ -56,6 +57,8 @@ "fp16": "float16", "fp32": "float32", } +_PROCESS_GROUP_TIMEOUT = datetime.timedelta(minutes=5) +_RELATIVE_ERROR_FLOOR = 1.0e-12 @dataclass(frozen=True) @@ -190,6 +193,10 @@ class _RankPayload: global_positions: torch.Tensor +def _strict_report_json(payload: dict[str, Any]) -> str: + return json.dumps(payload, indent=2, sort_keys=True, allow_nan=False) + + def plan_distributed_logprob_cases( *, tp_world_sizes: Sequence[int] = (1, 2, 4), @@ -385,7 +392,7 @@ def _drift_detail( selected_diff = diff[selected] selected_ref = reference.float()[selected] - relative = selected_diff / selected_ref.abs().clamp_min(torch.finfo(torch.float32).tiny) + relative = selected_diff.double() / selected_ref.double().abs().clamp_min(_RELATIVE_ERROR_FLOOR) selected_indices = torch.arange(diff.numel(), device=diff.device)[selected] worst_selected = int(selected_diff.argmax().item()) worst_local = int(selected_indices[worst_selected].item()) @@ -620,7 +627,7 @@ def run_distributed_logprob_case( ) initialized_here = False if world_size > 1 and not dist.is_initialized(): - dist.init_process_group(backend=backend) + dist.init_process_group(backend=backend, timeout=_PROCESS_GROUP_TIMEOUT) initialized_here = True if dist.is_initialized(): if dist.get_world_size() != world_size or dist.get_rank() != rank: @@ -700,7 +707,7 @@ def run_distributed_logprob_case( output_path = pathlib.Path(output) output_path.parent.mkdir(parents=True, exist_ok=True) output_path.write_text( - json.dumps(report.to_dict(), indent=2, sort_keys=True) + "\n", + _strict_report_json(report.to_dict()) + "\n", encoding="utf-8", ) if world_size > 1: @@ -783,7 +790,7 @@ def main(argv: Sequence[str] | None = None) -> None: output=args.output, ) if report is not None: - print(json.dumps(report.to_dict(), indent=2, sort_keys=True)) + print(_strict_report_json(report.to_dict())) if not report.passed: raise SystemExit(1) diff --git a/tests/test_distributed_logprob_comparison.py b/tests/test_distributed_logprob_comparison.py index 022ccc16..a41113e6 100644 --- a/tests/test_distributed_logprob_comparison.py +++ b/tests/test_distributed_logprob_comparison.py @@ -4,6 +4,7 @@ from __future__ import annotations import json +import math import os import subprocess import sys @@ -12,9 +13,12 @@ import pytest import torch +from rl_engine.kernels.logprob_contract import ShardingSpec from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import BACKEND_ID from rl_engine.testing.distributed_logprob_comparison import ( DistributedLogprobCase, + _drift_detail, + _strict_report_json, format_launch_command, plan_distributed_logprob_cases, rank_topology, @@ -116,6 +120,33 @@ def test_shared_pr2_drift_summary_preserves_active_mask_semantics(): assert stats.mean_abs == 1.5 +def test_relative_drift_near_zero_stays_finite(): + sharding = ShardingSpec( + tp_rank=0, + tp_world_size=1, + vocab_shard_bounds=((0, 16),), + real_vocab_size=13, + padded_vocab_size=16, + ) + detail = _drift_detail( + torch.tensor([1.0]), + torch.tensor([0.0]), + target_ids=torch.tensor([1]), + global_positions=torch.tensor([3]), + sharding=sharding, + atol=0.0, + rtol=0.0, + ) + + assert math.isfinite(detail.max_rel) + assert detail.max_rel == pytest.approx(1.0e12) + assert json.loads(_strict_report_json({"max_rel": detail.max_rel}))["max_rel"] == pytest.approx( + 1.0e12 + ) + with pytest.raises(ValueError, match="Out of range float values"): + _strict_report_json({"max_rel": float("nan")}) + + def test_tp1_cpu_case_writes_116_compatible_artifact(tmp_path, monkeypatch): monkeypatch.delenv("RANK", raising=False) monkeypatch.delenv("LOCAL_RANK", raising=False) From 8a4f4bef5029cbc1711273e68c11979594160943 Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Tue, 11 Aug 2026 23:18:33 +0800 Subject: [PATCH 27/48] fix(ws2): clean up process groups on setup failure --- .../testing/distributed_logprob_comparison.py | 41 ++++++------ tests/test_distributed_logprob_comparison.py | 65 +++++++++++++++++++ 2 files changed, 85 insertions(+), 21 deletions(-) diff --git a/rl_engine/testing/distributed_logprob_comparison.py b/rl_engine/testing/distributed_logprob_comparison.py index ad3bb05d..d3884194 100644 --- a/rl_engine/testing/distributed_logprob_comparison.py +++ b/rl_engine/testing/distributed_logprob_comparison.py @@ -625,27 +625,26 @@ def run_distributed_logprob_case( f"WORLD_SIZE={world_size} does not match TP*CP={case.world_size}; " "launch exactly the topology declared by the case" ) - initialized_here = False - if world_size > 1 and not dist.is_initialized(): - dist.init_process_group(backend=backend, timeout=_PROCESS_GROUP_TIMEOUT) - initialized_here = True - if dist.is_initialized(): - if dist.get_world_size() != world_size or dist.get_rank() != rank: - raise RuntimeError("initialized process group does not match RANK/WORLD_SIZE") - - if device_name == "cuda": - if not torch.cuda.is_available(): - raise RuntimeError("CUDA was requested but is unavailable") - device = torch.device("cuda", local_rank) - torch.cuda.set_device(device) - elif device_name == "cpu": - device = torch.device("cpu") - else: - raise ValueError("device must be cuda or cpu") - - topology = rank_topology(case, rank) - tp_group = _create_tp_group(case, topology) + owns_process_group = world_size > 1 and not dist.is_initialized() try: + if device_name == "cuda": + if not torch.cuda.is_available(): + raise RuntimeError("CUDA was requested but is unavailable") + device = torch.device("cuda", local_rank) + torch.cuda.set_device(device) + elif device_name == "cpu": + device = torch.device("cpu") + else: + raise ValueError("device must be cuda or cpu") + + if owns_process_group: + dist.init_process_group(backend=backend, timeout=_PROCESS_GROUP_TIMEOUT) + if dist.is_initialized(): + if dist.get_world_size() != world_size or dist.get_rank() != rank: + raise RuntimeError("initialized process group does not match RANK/WORLD_SIZE") + + topology = rank_topology(case, rank) + tp_group = _create_tp_group(case, topology) payload = _execute_rank(case, topology, device=device, tp_group=tp_group) if world_size == 1: payloads = [payload] @@ -714,7 +713,7 @@ def run_distributed_logprob_case( dist.barrier() return report finally: - if initialized_here and dist.is_initialized(): + if owns_process_group and dist.is_initialized(): dist.destroy_process_group() diff --git a/tests/test_distributed_logprob_comparison.py b/tests/test_distributed_logprob_comparison.py index a41113e6..8309e469 100644 --- a/tests/test_distributed_logprob_comparison.py +++ b/tests/test_distributed_logprob_comparison.py @@ -12,7 +12,9 @@ import pytest import torch +import torch.distributed as dist +import rl_engine.testing.distributed_logprob_comparison as distributed_comparison from rl_engine.kernels.logprob_contract import ShardingSpec from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import BACKEND_ID from rl_engine.testing.distributed_logprob_comparison import ( @@ -189,6 +191,69 @@ def test_world_size_mismatch_fails_before_process_group_init(tmp_path, monkeypat ) +def test_setup_failure_destroys_owned_process_group(tmp_path, monkeypatch): + state = {"initialized": False, "destroyed": False} + + def init_process_group(*, backend, timeout): + state["initialized"] = True + + def destroy_process_group(): + state["destroyed"] = True + state["initialized"] = False + + def fail_group_setup(case, topology): + raise RuntimeError("group setup failed") + + monkeypatch.setenv("WORLD_SIZE", "2") + monkeypatch.setenv("RANK", "0") + monkeypatch.setenv("LOCAL_RANK", "0") + monkeypatch.setattr(dist, "is_initialized", lambda: state["initialized"]) + monkeypatch.setattr(dist, "init_process_group", init_process_group) + monkeypatch.setattr(dist, "get_world_size", lambda: 2) + monkeypatch.setattr(dist, "get_rank", lambda: 0) + monkeypatch.setattr(dist, "destroy_process_group", destroy_process_group) + monkeypatch.setattr(distributed_comparison, "_create_tp_group", fail_group_setup) + + with pytest.raises(RuntimeError, match="group setup failed"): + run_distributed_logprob_case( + _small_case(tp=2), + device_name="cpu", + dist_backend="gloo", + output=tmp_path / "unused.json", + ) + + assert state == {"initialized": False, "destroyed": True} + + +def test_partial_initialization_failure_destroys_owned_process_group(tmp_path, monkeypatch): + state = {"initialized": False, "destroyed": False} + + def fail_initialization(*, backend, timeout): + state["initialized"] = True + raise RuntimeError("initialization failed") + + def destroy_process_group(): + state["destroyed"] = True + state["initialized"] = False + + monkeypatch.setenv("WORLD_SIZE", "2") + monkeypatch.setenv("RANK", "0") + monkeypatch.setenv("LOCAL_RANK", "0") + monkeypatch.setattr(dist, "is_initialized", lambda: state["initialized"]) + monkeypatch.setattr(dist, "init_process_group", fail_initialization) + monkeypatch.setattr(dist, "destroy_process_group", destroy_process_group) + + with pytest.raises(RuntimeError, match="initialization failed"): + run_distributed_logprob_case( + _small_case(tp=2), + device_name="cpu", + dist_backend="gloo", + output=tmp_path / "unused.json", + ) + + assert state == {"initialized": False, "destroyed": True} + + @pytest.mark.skipif(not torch.distributed.is_available(), reason="torch.distributed required") def test_tp2_cp2_gloo_cli_emits_per_rank_report(tmp_path): script = ( From 3627fcde7c805deb10238109eaf5d21e8b935b1c Mon Sep 17 00:00:00 2001 From: KJLdefeated Date: Wed, 12 Aug 2026 11:34:36 +0800 Subject: [PATCH 28/48] ws2 deterministic grpo loss PR5 --- docs/operators/grpo-loss.md | 146 ++- rl_engine/kernels/loss_contract.py | 784 ++++++++++++++++ .../ops/pytorch/loss/distributed_grpo_loss.py | 447 +++++++++ rl_engine/kernels/registry.py | 173 ++++ tests/test_distributed_grpo_loss.py | 866 ++++++++++++++++++ tests/test_grpo_loss_contract.py | 392 ++++++++ 6 files changed, 2802 insertions(+), 6 deletions(-) create mode 100644 rl_engine/kernels/loss_contract.py create mode 100644 rl_engine/kernels/ops/pytorch/loss/distributed_grpo_loss.py create mode 100644 tests/test_distributed_grpo_loss.py create mode 100644 tests/test_grpo_loss_contract.py diff --git a/docs/operators/grpo-loss.md b/docs/operators/grpo-loss.md index 07fcac16..fbfe5751 100644 --- a/docs/operators/grpo-loss.md +++ b/docs/operators/grpo-loss.md @@ -15,6 +15,11 @@ fused ratio/KL kernel (logits → ratio/KL via online softmax), and the group-no logits --[ratio_kl op]--> (ratio, kl) --[group adv + clipped surrogate]--> loss ``` +The backends above consume dense `[B, T, V]` logits and reduce with a plain masked +mean, so they are single-shard only. For vocab-parallel TP, or when the reduction +must be bitwise reproducible across parallel degrees, see [Tensor and Data +Parallel](#tensor-and-data-parallel) below. + ## Entry Point ```python from rl_engine.kernels.registry import kernel_registry @@ -74,6 +79,117 @@ op mirrors this using `NativeRatioKLOp`. Gradients flow into `policy_logits` only (`ref_logits` is frozen; `old_logps` is cached). +## Tensor and Data Parallel + +`DistributedGRPOLossOp` +(`rl_engine/kernels/ops/pytorch/loss/distributed_grpo_loss.py`) +**Every TP × DP degree produces bit-identical loss, per-sequence totals, and +gradients.** It is a reference backend on top of the deterministic +[vocab-parallel logprob](batch-invariant-logp.md#tensor-parallel); the backends +above stay the default single-GPU path. + +The objective is elementwise — ratio, clipping, reference KL and the group-relative +advantage all act per token or per sequence — so the only place the parallel layout +can change the answer is the final sum over tokens. + +1. Selected logprobs come from `VocabParallelLogprobOp`, so ratio and KL inherit + cross-TP bitwise equality for free. TP needs no further handling here: by the + time the objective sees a logprob, the vocabulary has already been reduced. +2. A DP rank owns each of its sequences **whole**. `sequence_shard_bounds` is a + contiguous `[0, num_sequences)` partition in DP-rank order, exactly like + `ShardingSpec.vocab_shard_bounds` for the vocabulary. +3. Two nested reductions, each with a contract-fixed extent: `padded_seq_len` + token slots → one sequence total (entirely local, since the rank owns the + whole sequence); `num_sequences` totals → the scalar numerator. +4. Only per-sequence totals cross a rank boundary. They travel by `all_gather` + and are concatenated in DP-rank order into a `[num_sequences]` vector. The + collective moves bytes and placement is an exact copy, so every degree + performs identical arithmetic on identical inputs. `all_reduce` is excluded on + purpose: its combine order follows the collective's topology, not the declared + sequence order. +5. Advantages are replicated, not merged — rewards are one scalar per sequence, + so every rank normalizes every group over the identical `[num_sequences]` + tensor and keeps its own slice. This is what lets an advantage group straddle + DP ranks with no extra machinery. +6. The normalizer divides by a **global** active-token count, gathered as + integers so it is exact at every degree. + +Step 3 is why the determinism argument is short: because no sequence's token sum +is ever split across ranks, there is no partial-sequence state to merge and no +alignment rule to get wrong. + +```python +sequence_shard_bounds = ((0, 8),) # DP=1 +sequence_shard_bounds = ((0, 4), (4, 8)) # DP=2 +``` + +### Context parallelism is out of scope + +`cp_world_size` must be 1; anything else raises `LossContractError` rather than +silently reducing over a partial batch. CP is an attention-level concern — attention +is the only op with a cross-token dependency — and this operator consumes logits, by +which point CP has already been resolved upstream. Supporting it here would mean +splitting a sequence's token sum across ranks and reducing over the very axis the +logprob contract declares a *non-merge* axis. In a CP job that reduction belongs to +the caller. `cp_rank`/`cp_world_size` are carried for provenance only, mirroring +`ShardingSpec`. + +### Normalizer semantics + +`TokenNormalizer` makes the GRPO normalizer ambiguity explicit. The modes differ by +more than a scale factor once sequence lengths vary, so the choice is part of the +numerical identity and travels in the contract fingerprint. + +| Mode | Denominator | Notes | +| --- | --- | --- | +| `global_active_tokens` (default) | active tokens in the global batch | Matches `NativeGRPOLossOp`'s masked mean at DP=1. Long sequences weigh more. | +| `per_sequence_then_mean` | per-sequence count, then mean over live sequences | Original GRPO form; sequences weigh equally. | +| `fixed_constant` | declared constant | Dr.GRPO form; independent of the mask. | + +Usage goes through the contract-aware entry point: + +```python +from rl_engine.kernels.registry import kernel_registry + +dispatched = kernel_registry.get_loss_op(contract) # GRPOLossContract from +result = dispatched.op.apply( # rl_engine.kernels.loss_contract + policy_local_logits, # [n, local_vocab] differentiable + action_ids, # [n] + old_logps, # [n] + rewards, # [local_num_sequences] + contract=contract, + ref_local_logits=ref_local_logits, # required when beta > 0 + tp_group=tp_group, # vocab-parallel subgroup + dp_group=dp_group, # data-parallel subgroup +) +result.loss.backward() # gradients flow into policy_local_logits only + +loss, policy_loss, kl = result # unpacks like the single-GPU op +``` + +A preflight `all_gather_object` runs on **both** the DP and TP axes before any other +collective. Neither alone suffices: the loss is replicated across TP, so two TP +siblings disagreeing on `beta` would compute different losses for one sharded model, +and the logprob path's own preflight cannot see that because `beta` is not part of +the logprob contract. Other loud failures, with no silent fallback: sequence bounds +that are non-contiguous or leave a gap; a nested logprob contract whose token count +disagrees with the owned sequences; a determinism scope stronger or weaker than the +logprob path's; and population-std advantages over a singleton group. + +### Comparing configurations + +Compare `per_sequence_policy` / `per_sequence_kl`, not the scalar loss. Measured on +this operator's test inputs, regrouping the token sum — a real change of the +summation tree — moves the per-sequence vector in 12 of 12 seeds but the scalar loss +in only 5 of 12: averaging `num_sequences` totals into one fp32 number rounds most +reorderings away. A drift report that compares only the scalar will under-report +reduction differences. `GRPOLossResult` exposes both, plus `advantages`, +`per_sequence_active_tokens`, and a `provenance` dict. + +Bitwise here means *across parallel degrees on one PyTorch build and GPU model*. It +rests on PyTorch's reduction kernels being deterministic for a fixed shape on a fixed +device; cross-version and cross-architecture equality is neither tested nor claimed. + ## Accuracy Reference semantics (`NativeGRPOLossOp`): @@ -92,6 +208,11 @@ loss = masked_mean(policy, completion_mask) + beta * masked_mean(kl, completion_ The Triton op matches the native reference (forward and backward) to `atol=1e-4`. +For `DistributedGRPOLossOp`, the reference-equals-policy identity is exact rather +than approximate: with `ref_logits is policy_logits` and `old_logps == logp_policy` +the ratio is `exp(0) = 1` bitwise, so the result is invariant to the clip epsilon, +and the KL is exactly `0.0`. + ## Performance Notes The cost is dominated by the [`ratio_kl`](ratio-kl.md) stage (the vocab-dimension work); @@ -118,18 +239,31 @@ online — the forward peak is independent of `V`. ## Tests ```bash -python -m pytest tests/test_grpo_loss.py -v +python -m pytest tests/test_grpo_loss.py -v # single-GPU backends +python -m pytest tests/test_grpo_loss_contract.py -v # TP/DP/CP contract, CPU only +python -m pytest tests/test_distributed_grpo_loss.py -v ``` -Covers the native reference (group advantages + loss from logits), Triton forward/backward -vs native, masked-token invariance, an SGD loss step, and registry dispatch. Triton tests -skip without CUDA + Triton. +`test_grpo_loss.py` covers the native reference (group advantages + loss from logits), +Triton forward/backward vs native, masked-token invariance, an SGD loss step, and +registry dispatch. Triton tests skip without CUDA + Triton. + +`test_distributed_grpo_loss.py` covers every `(TP, DP)` combination reachable with +four ranks — `tp2`, `tp4`, `dp2`, `dp4`, `tp2xdp2` — each compared bitwise against a +single-rank GPU baseline, plus the KL=0 identity, run-to-run stability, negative +controls, and the DP- and TP-axis preflight guards. Larger degrees run unchanged on a +bigger node. Multi-rank tests need one GPU per rank and skip otherwise; they are +deliberately small (1000-token vocabulary, 8 sequences of 32 slots) and each worker +caps itself with `torch.cuda.set_per_process_memory_fraction`, so the suite can share +a node with a running training job. ## Implementation Files - `rl_engine/kernels/ops/pytorch/loss/grpo_loss.py` - `rl_engine/kernels/ops/triton/loss/grpo_loss.py` - `rl_engine/kernels/ops/triton/loss/ratio_kl.py`, `rl_engine/kernels/ops/pytorch/loss/ratio_kl.py` -- `rl_engine/kernels/registry.py` -- `tests/test_grpo_loss.py` +- `rl_engine/kernels/ops/pytorch/loss/distributed_grpo_loss.py` +- `rl_engine/kernels/loss_contract.py` +- `rl_engine/kernels/registry.py` (`register_loss_backend`, `get_loss_op`) +- `tests/test_grpo_loss.py`, `tests/test_grpo_loss_contract.py`, `tests/test_distributed_grpo_loss.py` - `benchmarks/benchmark_ratio_kl.py` diff --git a/rl_engine/kernels/loss_contract.py b/rl_engine/kernels/loss_contract.py new file mode 100644 index 00000000..78f2e280 --- /dev/null +++ b/rl_engine/kernels/loss_contract.py @@ -0,0 +1,784 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Typed WS2 contract for deterministic GRPO loss on the TP-aware logprob path. + +The GRPO objective consumes selected-token log-probabilities and reduces them +to a scalar:: + + ratio_t = exp(logp_policy_t - old_logp_t) + surrogate = -min(ratio_t * adv_t, clip(ratio_t) * adv_t) + loss = normalize(sum_t surrogate_t) + beta * normalize(sum_t kl_t) +""" + +from __future__ import annotations + +import hashlib +import json +import math +from dataclasses import dataclass, field +from enum import Enum +from typing import Any, TypeVar + +from rl_engine.kernels.logprob_contract import ( + IMPLEMENTATION_KINDS, + RESERVED_DISPATCH_POLICIES, + DeterminismScope, + DowncastPoint, + LogprobContract, + LogprobDType, +) + +_EnumT = TypeVar("_EnumT", bound=Enum) + + +class LossContractError(ValueError): + """Raised when loss metadata does not describe a valid GRPO invocation.""" + + +class TokenNormalizer(str, Enum): + """Denominator applied to the summed per-token loss terms. + + ``global_active_tokens``: divide by the number of active tokens in the + *global* batch, gathered across every DP rank. Long sequences therefore + contribute proportionally more. This matches the existing single-GPU + ``NativeGRPOLossOp`` masked mean at DP=1. + + ``per_sequence_then_mean``: divide each sequence's sum by that sequence's + own active-token count, then average over sequences that hold at least one + active token. Sequences are weighted equally regardless of length. + + ``fixed_constant``: divide by a declared constant, independent of the mask. + Requires ``LossReductionSpec.fixed_normalizer_constant``. + + The three differ by more than a scale factor once sequence lengths vary, so + the choice is part of the numerical identity and travels in the fingerprint. + """ + + GLOBAL_ACTIVE_TOKENS = "global_active_tokens" + PER_SEQUENCE_THEN_MEAN = "per_sequence_then_mean" + FIXED_CONSTANT = "fixed_constant" + + +class SummationOrder(str, Enum): + """Fixed combine order for per-token partials. + + ``sequence_major_fixed``: within one sequence, tokens combine over the full + ``padded_seq_len`` extent on the single rank that owns the sequence; + sequences then combine in ascending global sequence index. Both extents are + contract-fixed, so the floating-point grouping is identical at every DP + degree. Only *which rank* computes a sequence changes. + """ + + SEQUENCE_MAJOR_FIXED = "sequence_major_fixed" + + +class LossTransport(str, Enum): + """Collectives move partial sums only; they never reduce numerically. + + ``all_reduce`` is excluded on purpose: NCCL's reduction order depends on + world size and topology, so it would silently regroup the per-sequence + combines and break the cross-DP guarantee ``SummationOrder`` provides. + """ + + ALL_GATHER = "all_gather" + + +class KLEstimator(str, Enum): + """Per-token reference-KL estimator. + + ``k3_unbiased``: ``exp(logp_ref - logp_policy) - (logp_ref - logp_policy) - 1``, + the non-negative low-variance estimator used by the existing ratio/KL op. + + ``k1_log_ratio``: ``logp_policy - logp_ref``, the plain log-ratio. + """ + + K3_UNBIASED = "k3_unbiased" + K1_LOG_RATIO = "k1_log_ratio" + + +class ClipMode(str, Enum): + MIN_OF_UNCLIPPED_AND_CLIPPED = "min_of_unclipped_and_clipped" + + +class AdvantageNormalizer(str, Enum): + """Group-relative reward normalization. + + ``mean_std_population``: ``(r - mean) / std`` with the population (biased) + standard deviation, the original GRPO form. + + ``mean_only``: ``r - mean``, the Dr.GRPO form that drops the std divisor to + avoid its length/difficulty bias. + """ + + MEAN_STD_POPULATION = "mean_std_population" + MEAN_ONLY = "mean_only" + + +class VarianceFormula(str, Enum): + """Only the two-pass form is conformant. + + ``E[x^2] - E[x]^2`` cancels catastrophically once rewards share a large + offset, and its error depends on group size, so it cannot support a bitwise + claim. The two-pass form subtracts the mean before squaring. + """ + + TWO_PASS = "two_pass" + + +class GroupReplication(str, Enum): + """How advantage groups are evaluated when they span DP ranks. + + ``replicated_all_gather``: per-sequence rewards are all-gathered and *every* + rank normalizes *every* group identically, then keeps its own slice. + Rewards are one scalar per sequence, so replicating the whole computation is + cheaper than making a partial-statistic merge bitwise-reproducible. + """ + + REPLICATED_ALL_GATHER = "replicated_all_gather" + + +class LossPlacement(str, Enum): + REPLICATED = "replicated" + + +def _enum_value(enum_type: type[_EnumT], value: Any, field: str) -> _EnumT: + try: + return enum_type(value) + except (TypeError, ValueError) as exc: + allowed = ", ".join(item.value for item in enum_type) + raise LossContractError(f"{field} must be one of: {allowed}; got {value!r}") from exc + + +def _positive_int(value: Any, field: str) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + raise LossContractError(f"{field} must be a positive integer; got {value!r}") + return value + + +def _non_negative_int(value: Any, field: str) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise LossContractError(f"{field} must be a non-negative integer; got {value!r}") + return value + + +def _non_negative_float(value: Any, field: str) -> float: + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise LossContractError(f"{field} must be a real number; got {value!r}") + value = float(value) + if not math.isfinite(value) or value < 0.0: + raise LossContractError(f"{field} must be finite and non-negative; got {value!r}") + return value + + +@dataclass(frozen=True) +class ClipSpec: + """Asymmetric PPO-style ratio clipping bounds. + + Separate low/high epsilons cover the "clip-higher" variants; passing the + same value twice recovers the symmetric ``[1-eps, 1+eps]`` form. + """ + + clip_eps_low: float = 0.2 + clip_eps_high: float = 0.2 + mode: ClipMode = ClipMode.MIN_OF_UNCLIPPED_AND_CLIPPED + + def __post_init__(self) -> None: + object.__setattr__(self, "mode", _enum_value(ClipMode, self.mode, "clip.mode")) + low = _non_negative_float(self.clip_eps_low, "clip_eps_low") + high = _non_negative_float(self.clip_eps_high, "clip_eps_high") + if low >= 1.0: + raise LossContractError( + f"clip_eps_low={low} must be smaller than 1.0; the lower clip bound " + "1 - clip_eps_low must stay positive" + ) + object.__setattr__(self, "clip_eps_low", low) + object.__setattr__(self, "clip_eps_high", high) + + @property + def lower_bound(self) -> float: + return 1.0 - self.clip_eps_low + + @property + def upper_bound(self) -> float: + return 1.0 + self.clip_eps_high + + +@dataclass(frozen=True) +class AdvantageSpec: + """Group-relative advantage normalization semantics.""" + + normalizer: AdvantageNormalizer = AdvantageNormalizer.MEAN_STD_POPULATION + variance: VarianceFormula = VarianceFormula.TWO_PASS + std_eps: float = 1e-6 + replication: GroupReplication = GroupReplication.REPLICATED_ALL_GATHER + + def __post_init__(self) -> None: + object.__setattr__( + self, + "normalizer", + _enum_value(AdvantageNormalizer, self.normalizer, "advantage.normalizer"), + ) + object.__setattr__( + self, "variance", _enum_value(VarianceFormula, self.variance, "advantage.variance") + ) + object.__setattr__( + self, + "replication", + _enum_value(GroupReplication, self.replication, "advantage.replication"), + ) + std_eps = _non_negative_float(self.std_eps, "advantage.std_eps") + if std_eps <= 0.0: + raise LossContractError( + f"advantage.std_eps={std_eps} must be strictly positive; it is the floor " + "that keeps a zero-variance group from dividing by zero" + ) + object.__setattr__(self, "std_eps", std_eps) + + +@dataclass(frozen=True) +class ObjectiveSpec: + """The GRPO objective itself: clipping, reference KL, advantage shaping.""" + + clip: ClipSpec = field(default_factory=ClipSpec) + advantage: AdvantageSpec = field(default_factory=AdvantageSpec) + kl_estimator: KLEstimator = KLEstimator.K3_UNBIASED + beta: float = 0.0 + + def __post_init__(self) -> None: + if not isinstance(self.clip, ClipSpec): + raise LossContractError("objective.clip must be a ClipSpec") + if not isinstance(self.advantage, AdvantageSpec): + raise LossContractError("objective.advantage must be an AdvantageSpec") + object.__setattr__( + self, + "kl_estimator", + _enum_value(KLEstimator, self.kl_estimator, "objective.kl_estimator"), + ) + object.__setattr__(self, "beta", _non_negative_float(self.beta, "objective.beta")) + + @property + def uses_reference_model(self) -> bool: + """Whether reference logits are required at all. + + ``beta == 0`` drops the KL term from the loss, so a backend may skip the + reference forward entirely. The KL is still *reported*, so a caller + that wants the diagnostic must supply reference logits regardless. + """ + + return self.beta > 0.0 + + +@dataclass(frozen=True) +class LossReductionSpec: + """Deterministic token/sequence summation and normalizer semantics. + + Per-token loss terms are accumulated in fp32, combined in the order given by + ``summation_order``, moved between ranks by ``transport`` (never reduced by + it), and divided by the denominator selected by ``token_normalizer``. The + scalar is downcast, if at all, only at ``downcast_at``. + + ``determinism_scope`` reuses the logprob scale. ``cross_tp_bitwise`` here + means the scalar loss and its gradient are bitwise-identical across TP and + DP degrees, given a fixed vocab tile count. + """ + + token_normalizer: TokenNormalizer = TokenNormalizer.GLOBAL_ACTIVE_TOKENS + summation_order: SummationOrder = SummationOrder.SEQUENCE_MAJOR_FIXED + acc_dtype: LogprobDType = LogprobDType.FP32 + transport: LossTransport = LossTransport.ALL_GATHER + downcast_at: DowncastPoint = DowncastPoint.FINAL_WRITE + determinism_scope: DeterminismScope = DeterminismScope.CROSS_TP_BITWISE + fixed_normalizer_constant: int | None = None + + def __post_init__(self) -> None: + object.__setattr__( + self, + "token_normalizer", + _enum_value(TokenNormalizer, self.token_normalizer, "token_normalizer"), + ) + object.__setattr__( + self, + "summation_order", + _enum_value(SummationOrder, self.summation_order, "summation_order"), + ) + object.__setattr__( + self, "acc_dtype", _enum_value(LogprobDType, self.acc_dtype, "acc_dtype") + ) + object.__setattr__( + self, "transport", _enum_value(LossTransport, self.transport, "transport") + ) + object.__setattr__( + self, "downcast_at", _enum_value(DowncastPoint, self.downcast_at, "downcast_at") + ) + object.__setattr__( + self, + "determinism_scope", + _enum_value(DeterminismScope, self.determinism_scope, "determinism_scope"), + ) + if self.acc_dtype is not LogprobDType.FP32: + raise LossContractError(f"loss accumulation must be fp32; got {self.acc_dtype.value}") + + needs_constant = self.token_normalizer is TokenNormalizer.FIXED_CONSTANT + if needs_constant: + object.__setattr__( + self, + "fixed_normalizer_constant", + _positive_int(self.fixed_normalizer_constant, "fixed_normalizer_constant"), + ) + elif self.fixed_normalizer_constant is not None: + raise LossContractError( + "fixed_normalizer_constant is only meaningful for " + f"token_normalizer={TokenNormalizer.FIXED_CONSTANT.value}; got " + f"{self.token_normalizer.value}" + ) + + +@dataclass(frozen=True) +class LossOutputSpec: + """Output surface: fp32 scalars replicated across every DP and TP rank.""" + + loss_dtype: LogprobDType = LogprobDType.FP32 + placement: LossPlacement = LossPlacement.REPLICATED + + def __post_init__(self) -> None: + object.__setattr__( + self, "loss_dtype", _enum_value(LogprobDType, self.loss_dtype, "loss_dtype") + ) + object.__setattr__( + self, "placement", _enum_value(LossPlacement, self.placement, "placement") + ) + if self.loss_dtype is not LogprobDType.FP32: + raise LossContractError(f"loss output must be fp32; got {self.loss_dtype.value}") + + +@dataclass(frozen=True) +class LossShardingSpec: + """Which sequences of the global batch this DP rank owns. + + The global batch is ``num_sequences`` sequences of ``padded_seq_len`` token + slots each. ``sequence_shard_bounds`` lists every DP rank's half-open + ``[start, end)`` sequence range, indexed by rank; the full table is required + on every rank and must form a contiguous ``[0, num_sequences)`` partition, + exactly as ``ShardingSpec.vocab_shard_bounds`` does for the vocabulary. + """ + + dp_rank: int + dp_world_size: int + num_sequences: int + padded_seq_len: int + sequence_shard_bounds: tuple[tuple[int, int], ...] + group_boundaries: tuple[int, ...] + cp_rank: int = 0 + cp_world_size: int = 1 + + def __post_init__(self) -> None: + dp_world_size = _positive_int(self.dp_world_size, "dp_world_size") + dp_rank = _non_negative_int(self.dp_rank, "dp_rank") + if dp_rank >= dp_world_size: + raise LossContractError( + f"dp_rank={dp_rank} must be smaller than dp_world_size={dp_world_size}" + ) + cp_world_size = _positive_int(self.cp_world_size, "cp_world_size") + _non_negative_int(self.cp_rank, "cp_rank") + if cp_world_size != 1: + raise LossContractError( + f"cp_world_size={cp_world_size} is unsupported: context parallelism splits a " + "sequence's tokens across ranks, so the loss reduction would span an axis this " + "contract does not model. Reduce across CP outside this operator." + ) + if self.cp_rank != 0: + raise LossContractError(f"cp_rank must be 0 when cp_world_size=1; got {self.cp_rank}") + + _positive_int(self.num_sequences, "num_sequences") + _positive_int(self.padded_seq_len, "padded_seq_len") + object.__setattr__( + self, + "sequence_shard_bounds", + self._validated_bounds(self.sequence_shard_bounds, dp_world_size, self.num_sequences), + ) + object.__setattr__( + self, + "group_boundaries", + self._validated_groups(self.group_boundaries, self.num_sequences), + ) + + @staticmethod + def _validated_bounds( + raw: Any, dp_world_size: int, num_sequences: int + ) -> tuple[tuple[int, int], ...]: + try: + bounds = tuple((pair[0], pair[1]) for pair in raw) + except (TypeError, IndexError, KeyError) as exc: + raise LossContractError( + "sequence_shard_bounds must be an iterable of (start, end) integer pairs" + ) from exc + if len(bounds) != dp_world_size: + raise LossContractError( + "sequence_shard_bounds must declare exactly one (start, end) pair per DP rank; " + f"got {len(bounds)} pairs for dp_world_size={dp_world_size}" + ) + expected_start = 0 + for rank, (start, end) in enumerate(bounds): + for name, value in ( + (f"sequence_shard_bounds[{rank}][0]", start), + (f"sequence_shard_bounds[{rank}][1]", end), + ): + if isinstance(value, bool) or not isinstance(value, int): + raise LossContractError(f"{name} must be an integer; got {value!r}") + if end <= start: + raise LossContractError( + f"sequence_shard_bounds[{rank}] must satisfy end > start; got [{start}, {end})" + ) + if start != expected_start: + raise LossContractError( + "sequence_shard_bounds must form a contiguous [0, num_sequences) partition " + f"in DP-rank order; rank {rank} starts at {start}, expected {expected_start}" + ) + expected_start = end + if expected_start != num_sequences: + raise LossContractError( + "sequence_shard_bounds must cover num_sequences exactly; covered " + f"{expected_start}, declared {num_sequences}" + ) + return bounds + + @staticmethod + def _validated_groups(raw: Any, num_sequences: int) -> tuple[int, ...]: + try: + offsets = tuple(raw) + except TypeError as exc: + raise LossContractError("group_boundaries must be an iterable of integers") from exc + if len(offsets) < 2: + raise LossContractError( + "group_boundaries must hold num_groups + 1 offsets, so at least 2 entries" + ) + for index, value in enumerate(offsets): + if isinstance(value, bool) or not isinstance(value, int): + raise LossContractError(f"group_boundaries[{index}] must be an integer") + if offsets[0] != 0 or offsets[-1] != num_sequences: + raise LossContractError( + "group_boundaries must start at 0 and end at " + f"num_sequences={num_sequences}; got [{offsets[0]}, ..., {offsets[-1]}]" + ) + for index in range(1, len(offsets)): + if offsets[index] <= offsets[index - 1]: + raise LossContractError( + "group_boundaries must be strictly increasing; " + f"offset {index} is {offsets[index]} after {offsets[index - 1]}" + ) + return offsets + + @property + def num_groups(self) -> int: + return len(self.group_boundaries) - 1 + + @property + def group_sizes(self) -> tuple[int, ...]: + return tuple( + self.group_boundaries[i + 1] - self.group_boundaries[i] for i in range(self.num_groups) + ) + + @property + def local_sequence_start(self) -> int: + return self.sequence_shard_bounds[self.dp_rank][0] + + @property + def local_sequence_end(self) -> int: + return self.sequence_shard_bounds[self.dp_rank][1] + + @property + def local_num_sequences(self) -> int: + start, end = self.sequence_shard_bounds[self.dp_rank] + return end - start + + @property + def local_num_token_slots(self) -> int: + """Token slots this rank holds -- the row count of its logprob call.""" + + return self.local_num_sequences * self.padded_seq_len + + +@dataclass(frozen=True) +class GRPOLossContract: + """Complete semantic request for one deterministic GRPO loss invocation.""" + + logprob: LogprobContract + sharding: LossShardingSpec + objective: ObjectiveSpec = field(default_factory=ObjectiveSpec) + reduction: LossReductionSpec = field(default_factory=LossReductionSpec) + output: LossOutputSpec = field(default_factory=LossOutputSpec) + + def __post_init__(self) -> None: + if not isinstance(self.logprob, LogprobContract): + raise LossContractError("logprob must be a LogprobContract") + if not isinstance(self.sharding, LossShardingSpec): + raise LossContractError("sharding must be a LossShardingSpec") + if not isinstance(self.objective, ObjectiveSpec): + raise LossContractError("objective must be an ObjectiveSpec") + if not isinstance(self.reduction, LossReductionSpec): + raise LossContractError("reduction must be a LossReductionSpec") + if not isinstance(self.output, LossOutputSpec): + raise LossContractError("output must be a LossOutputSpec") + + # The nested logprob contract describes this rank's own rows, so its + # token count must match the cells this rank owns. Catching the + # mismatch here turns a silent shape error deep inside the reduction + # into a contract failure at construction. + expected_tokens = self.sharding.local_num_token_slots + if self.logprob.mask.num_tokens != expected_tokens: + raise LossContractError( + f"logprob.mask.num_tokens={self.logprob.mask.num_tokens} must equal the " + f"{expected_tokens} token slots this rank owns " + f"({self.sharding.local_num_sequences} sequences x " + f"{self.sharding.padded_seq_len} slots per sequence)" + ) + if self.logprob.reduction.determinism_scope is not self.reduction.determinism_scope: + raise LossContractError( + "the loss cannot claim a stronger or weaker determinism scope than the " + f"logprob path it consumes; loss={self.reduction.determinism_scope.value}, " + f"logprob={self.logprob.reduction.determinism_scope.value}" + ) + if self.logprob.sharding.cp_world_size != self.sharding.cp_world_size: + raise LossContractError( + f"cp_world_size disagrees between the logprob contract " + f"({self.logprob.sharding.cp_world_size}) and the loss sharding " + f"({self.sharding.cp_world_size})" + ) + if self.objective.advantage.normalizer is AdvantageNormalizer.MEAN_STD_POPULATION: + # A singleton group has zero population variance, so its advantage + # would collapse to 0 and the sequence would contribute nothing. + # Reject it rather than silently training on a dead group. + small = [index for index, size in enumerate(self.sharding.group_sizes) if size < 2] + if small: + raise LossContractError( + f"advantage normalizer {AdvantageNormalizer.MEAN_STD_POPULATION.value} " + f"needs at least 2 sequences per group; groups {small} are smaller" + ) + + @property + def global_token_slots(self) -> int: + return self.sharding.num_sequences * self.sharding.padded_seq_len + + def to_dict(self) -> dict[str, Any]: + """Return stable, JSON-compatible requested-contract provenance.""" + + sharding = { + "dp_rank": self.sharding.dp_rank, + "dp_world_size": self.sharding.dp_world_size, + "cp_rank": self.sharding.cp_rank, + "cp_world_size": self.sharding.cp_world_size, + "num_sequences": self.sharding.num_sequences, + "padded_seq_len": self.sharding.padded_seq_len, + "sequence_shard_bounds": [list(pair) for pair in self.sharding.sequence_shard_bounds], + "group_boundaries": list(self.sharding.group_boundaries), + "local_sequence_start": self.sharding.local_sequence_start, + "local_sequence_end": self.sharding.local_sequence_end, + } + objective = { + "clip_eps_low": self.objective.clip.clip_eps_low, + "clip_eps_high": self.objective.clip.clip_eps_high, + "clip_mode": self.objective.clip.mode.value, + "kl_estimator": self.objective.kl_estimator.value, + "beta": self.objective.beta, + "advantage_normalizer": self.objective.advantage.normalizer.value, + "advantage_variance": self.objective.advantage.variance.value, + "advantage_std_eps": self.objective.advantage.std_eps, + "advantage_replication": self.objective.advantage.replication.value, + } + reduction = { + "token_normalizer": self.reduction.token_normalizer.value, + "summation_order": self.reduction.summation_order.value, + "acc_dtype": self.reduction.acc_dtype.value, + "transport": self.reduction.transport.value, + "downcast_at": self.reduction.downcast_at.value, + "determinism_scope": self.reduction.determinism_scope.value, + "fixed_normalizer_constant": self.reduction.fixed_normalizer_constant, + "cp_is_merge_axis": False, + "dp_is_merge_axis": True, + } + return { + "semantic_operator": "grpo_loss", + "logprob": self.logprob.to_dict(), + "sharding": sharding, + "objective": objective, + "reduction": reduction, + "output": { + "loss_dtype": self.output.loss_dtype.value, + "placement": self.output.placement.value, + }, + } + + def cross_rank_fingerprint(self) -> str: + """Rank-independent identity for preflight agreement across all ranks. + + Drops every rank-local field so all ``dp_world_size x tp_world_size`` + ranks of one logical invocation agree. That includes the nested + logprob contract's mask: each DP rank holds a different slice of + sequences, so its ``num_tokens`` and mask digest legitimately differ + even though the invocation is the same one. The global token geometry + is still pinned, by ``num_sequences``/``padded_seq_len`` and by the full + ``sequence_shard_bounds`` table, which every rank declares identically. + """ + + payload = self.to_dict() + logprob = payload["logprob"] + logprob.pop("mask", None) + logprob["sharding"] = { + key: value + for key, value in logprob["sharding"].items() + if key not in {"tp_rank", "cp_rank", "local_vocab_start", "local_vocab_end"} + } + payload["sharding"] = { + key: value + for key, value in payload["sharding"].items() + if key not in {"dp_rank", "cp_rank", "local_sequence_start", "local_sequence_end"} + } + encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")) + return hashlib.sha256(encoded.encode("utf-8")).hexdigest() + + +@dataclass(frozen=True) +class LossBackendCapability: + """Capabilities a concrete loss backend declares to contract-aware dispatch.""" + + backend_id: str + token_normalizers: frozenset[TokenNormalizer] + kl_estimators: frozenset[KLEstimator] + advantage_normalizers: frozenset[AdvantageNormalizer] + determinism_scopes: frozenset[DeterminismScope] + dp_world_sizes: tuple[int, ...] | None = None + supports_variable_group_sizes: bool = False + supports_asymmetric_clip: bool = False + implementation_kind: str = "production" + + def __post_init__(self) -> None: + if not isinstance(self.backend_id, str) or not self.backend_id.strip(): + raise LossContractError("backend_id must be a non-empty string") + if self.backend_id.strip().lower() in RESERVED_DISPATCH_POLICIES: + raise LossContractError( + f"backend_id={self.backend_id!r} shadows a reserved dispatch policy keyword" + ) + object.__setattr__(self, "backend_id", self.backend_id.strip()) + for name, enum_type in ( + ("token_normalizers", TokenNormalizer), + ("kl_estimators", KLEstimator), + ("advantage_normalizers", AdvantageNormalizer), + ("determinism_scopes", DeterminismScope), + ): + try: + values = frozenset( + _enum_value(enum_type, value, name) for value in getattr(self, name) + ) + except TypeError as exc: + raise LossContractError(f"{name} must be an iterable of enum values") from exc + if not values: + raise LossContractError(f"{name} must not be empty") + object.__setattr__(self, name, values) + object.__setattr__( + self, + "dp_world_sizes", + self._validated_world_sizes(self.dp_world_sizes, "dp_world_sizes"), + ) + for flag_name in ("supports_variable_group_sizes", "supports_asymmetric_clip"): + if not isinstance(getattr(self, flag_name), bool): + raise LossContractError(f"{flag_name} must be a bool") + if self.implementation_kind not in IMPLEMENTATION_KINDS: + raise LossContractError( + f"implementation_kind must be one of: {', '.join(sorted(IMPLEMENTATION_KINDS))}" + ) + + @staticmethod + def _validated_world_sizes( + values: tuple[int, ...] | None, field: str + ) -> tuple[int, ...] | None: + if values is None: + return None + try: + sizes = tuple(values) + except TypeError as exc: + raise LossContractError(f"{field} must be an iterable of integers") from exc + if not sizes: + raise LossContractError(f"{field} must not be empty; use None for unrestricted") + for value in sizes: + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + raise LossContractError(f"{field} must contain positive values; got {value!r}") + if len(set(sizes)) != len(sizes): + raise LossContractError(f"{field} must not contain duplicates") + return sizes + + def incompatibilities(self, contract: GRPOLossContract) -> tuple[str, ...]: + """Explain every reason this backend cannot materialize ``contract``.""" + + reasons: list[str] = [] + reduction = contract.reduction + objective = contract.objective + sharding = contract.sharding + if reduction.token_normalizer not in self.token_normalizers: + reasons.append(f"token_normalizer={reduction.token_normalizer.value} is unsupported") + if objective.kl_estimator not in self.kl_estimators: + reasons.append(f"kl_estimator={objective.kl_estimator.value} is unsupported") + if objective.advantage.normalizer not in self.advantage_normalizers: + reasons.append( + f"advantage normalizer={objective.advantage.normalizer.value} is unsupported" + ) + if reduction.determinism_scope not in self.determinism_scopes: + reasons.append(f"determinism_scope={reduction.determinism_scope.value} is unsupported") + if self.dp_world_sizes is not None and sharding.dp_world_size not in self.dp_world_sizes: + reasons.append(f"DP={sharding.dp_world_size} is unsupported") + if len(set(sharding.group_sizes)) > 1 and not self.supports_variable_group_sizes: + reasons.append("variable advantage group sizes are unsupported") + if ( + objective.clip.clip_eps_low != objective.clip.clip_eps_high + and not self.supports_asymmetric_clip + ): + reasons.append("asymmetric ratio clipping is unsupported") + return tuple(reasons) + + def supports(self, contract: GRPOLossContract) -> bool: + return not self.incompatibilities(contract) + + def to_dict(self) -> dict[str, Any]: + return { + "backend_id": self.backend_id, + "token_normalizers": sorted(item.value for item in self.token_normalizers), + "kl_estimators": sorted(item.value for item in self.kl_estimators), + "advantage_normalizers": sorted(item.value for item in self.advantage_normalizers), + "determinism_scopes": sorted(item.value for item in self.determinism_scopes), + "dp_world_sizes": list(self.dp_world_sizes) if self.dp_world_sizes else None, + "supports_variable_group_sizes": self.supports_variable_group_sizes, + "supports_asymmetric_clip": self.supports_asymmetric_clip, + "implementation_kind": self.implementation_kind, + } + + +@dataclass(frozen=True) +class LossDispatchResult: + """A concrete backend plus the actual provenance bound to the request.""" + + op: Any + capability: LossBackendCapability + provenance: dict[str, Any] + + +__all__ = [ + "AdvantageNormalizer", + "AdvantageSpec", + "ClipMode", + "ClipSpec", + "GRPOLossContract", + "GroupReplication", + "KLEstimator", + "LossBackendCapability", + "LossContractError", + "LossDispatchResult", + "LossOutputSpec", + "LossPlacement", + "LossReductionSpec", + "LossShardingSpec", + "LossTransport", + "ObjectiveSpec", + "SummationOrder", + "TokenNormalizer", + "VarianceFormula", +] diff --git a/rl_engine/kernels/ops/pytorch/loss/distributed_grpo_loss.py b/rl_engine/kernels/ops/pytorch/loss/distributed_grpo_loss.py new file mode 100644 index 00000000..bbec450d --- /dev/null +++ b/rl_engine/kernels/ops/pytorch/loss/distributed_grpo_loss.py @@ -0,0 +1,447 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Deterministic DP-aware GRPO loss.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any, Iterator + +import torch + +from rl_engine.kernels.loss_contract import ( + AdvantageNormalizer, + GRPOLossContract, + KLEstimator, + LossContractError, + TokenNormalizer, +) +from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import ( + DEFAULT_NUM_VOCAB_TILES, + VocabParallelLogprobOp, +) + +BACKEND_ID = "pytorch-distributed-grpo-loss-ws2" + +# Channel layout of the packed per-sequence tensor moved by the single fp32 +# all-gather; counts travel separately as integers so they stay exact. +_CH_POLICY = 0 +_CH_KL = 1 + + +@dataclass(frozen=True) +class GRPOLossResult: + """Scalar loss terms, per-sequence diagnostics, and bound provenance. + + Unpacks as ``loss, policy_loss, kl`` so it can stand in for the tuple the + single-GPU GRPO ops return. + + The ``per_sequence_*`` vectors are the reduction's last mesh-independent + intermediate, detached and replicated on every rank. They are the right + surface for comparing two configurations: the scalar loss averages + ``num_sequences`` totals into 24 mantissa bits and routinely rounds a real + reordering away, whereas the per-sequence vector preserves it. They also + give a drift report somewhere to point when one sequence is responsible. + """ + + loss: torch.Tensor + policy_loss: torch.Tensor + kl: torch.Tensor + advantages: torch.Tensor + per_sequence_policy: torch.Tensor + per_sequence_kl: torch.Tensor + per_sequence_active_tokens: torch.Tensor + provenance: dict[str, Any] = field(default_factory=dict) + + def __iter__(self) -> Iterator[torch.Tensor]: + yield from (self.loss, self.policy_loss, self.kl) + + +def _require_distributed_initialized(): + import torch.distributed as dist + + if not dist.is_available(): + raise LossContractError("distributed GRPO loss requires torch.distributed.") + if not dist.is_initialized(): + raise LossContractError( + "distributed GRPO loss requires an initialized process group when " + "the contract declares dp_world_size > 1." + ) + return dist + + +def _validate_invocation( + policy_local_logits: torch.Tensor, + ref_local_logits: torch.Tensor | None, + action_ids: torch.Tensor, + old_logps: torch.Tensor, + rewards: torch.Tensor, + contract: GRPOLossContract, + dp_group: Any, +) -> None: + sharding = contract.sharding + num_rows = sharding.local_num_token_slots + if policy_local_logits.dim() != 2: + raise LossContractError( + "policy_local_logits must be 2-D [num_tokens, local_vocab]; got " + f"{policy_local_logits.dim()}-D" + ) + if policy_local_logits.shape[0] != num_rows: + raise LossContractError( + f"policy_local_logits has {policy_local_logits.shape[0]} rows but this rank owns " + f"{num_rows} token slots" + ) + if ref_local_logits is not None and ref_local_logits.shape != policy_local_logits.shape: + raise LossContractError( + f"ref_local_logits shape {tuple(ref_local_logits.shape)} must match " + f"policy_local_logits shape {tuple(policy_local_logits.shape)}" + ) + if ref_local_logits is None and contract.objective.uses_reference_model: + raise LossContractError( + f"objective.beta={contract.objective.beta} puts the reference KL in the loss, " + "so ref_local_logits is required" + ) + for name, tensor in (("action_ids", action_ids), ("old_logps", old_logps)): + if tensor.dim() != 1 or tensor.shape[0] != num_rows: + raise LossContractError( + f"{name} must be 1-D with one entry per owned token slot; got shape " + f"{tuple(tensor.shape)} for {num_rows} slots" + ) + if rewards.dim() != 1 or rewards.shape[0] != sharding.local_num_sequences: + raise LossContractError( + "rewards must be 1-D with one entry per sequence this rank owns; got shape " + f"{tuple(rewards.shape)} for {sharding.local_num_sequences} sequences" + ) + + if sharding.dp_world_size > 1: + dist = _require_distributed_initialized() + group_world = dist.get_world_size(group=dp_group) + group_rank = dist.get_rank(group=dp_group) + if group_world != sharding.dp_world_size: + raise LossContractError( + f"dp_group world size {group_world} does not match the contract " + f"dp_world_size={sharding.dp_world_size}; pass the DP subgroup, " + "not the global group" + ) + if group_rank != sharding.dp_rank: + raise LossContractError( + f"dp_group rank {group_rank} does not match the contract dp_rank={sharding.dp_rank}" + ) + + +def _preflight_cross_rank_agreement( + contract: GRPOLossContract, dp_group: Any, tp_group: Any, num_vocab_tiles: int +) -> None: + """All-gather (fingerprint, backend id, vocab tile count) and abort on mismatch. + + Checked over the DP group *and* the TP group. Neither alone is sufficient: + the loss is replicated across TP, so two TP siblings that disagree on, say, + ``beta`` would compute different losses and produce inconsistent gradients + for one sharded model -- and the logprob path's own preflight cannot catch + that, because ``beta`` is not part of the logprob contract. Agreement + within both groups implies agreement across the whole DP x TP grid by + transitivity. + + Runs before any other collective, including the logprob path's own TP + preflight, so a rank that joined the wrong logical invocation fails here + rather than deadlocking a later reduction. + """ + + checks = [ + (axis, group, world_size) + for axis, group, world_size in ( + ("DP", dp_group, contract.sharding.dp_world_size), + ("TP", tp_group, contract.logprob.sharding.tp_world_size), + ) + if world_size > 1 + ] + if not checks: + return + + dist = _require_distributed_initialized() + payload = (contract.cross_rank_fingerprint(), BACKEND_ID, int(num_vocab_tiles)) + for axis, group, _ in checks: + gathered: list[Any] = [None] * dist.get_world_size(group=group) + dist.all_gather_object(gathered, payload, group=group) + mismatched = [(rank, other) for rank, other in enumerate(gathered) if other != payload] + if mismatched: + rank, other = mismatched[0] + raise LossContractError( + f"cross-rank preflight failed on the {axis} axis: this rank has {payload} " + f"but {axis} rank {rank} has {other}; every rank must agree on the contract " + "fingerprint, backend id and num_vocab_tiles before any collective" + ) + + +def _gather_global_rewards( + rewards: torch.Tensor, contract: GRPOLossContract, dp_group: Any +) -> torch.Tensor: + """Assemble the global ``[num_sequences]`` reward vector on every rank. + + ``sequence_shard_bounds`` is a contiguous partition in DP-rank order, so + concatenating the gathered slices in rank order reproduces the global vector + exactly -- no ownership arbitration is needed. + """ + + sharding = contract.sharding + local = rewards.float() + if sharding.dp_world_size == 1: + return local.contiguous() + + dist = _require_distributed_initialized() + max_local = max(end - start for start, end in sharding.sequence_shard_bounds) + padded = local.new_zeros(max_local) + padded[: local.shape[0]] = local + gathered = [torch.empty_like(padded) for _ in range(sharding.dp_world_size)] + dist.all_gather(gathered, padded.contiguous(), group=dp_group) + return torch.cat( + [ + gathered[rank][: end - start] + for rank, (start, end) in enumerate(sharding.sequence_shard_bounds) + ], + dim=0, + ) + + +def _group_advantages(global_rewards: torch.Tensor, contract: GRPOLossContract) -> torch.Tensor: + """Group-relative advantages over the global reward vector. + + Evaluated identically on every rank from an identically shaped input, so no + merge is involved and the result is bitwise-equal mesh-wide. The variance + is two-pass: centring before squaring keeps the result meaningful when the + rewards share a large offset, which ``E[x^2] - E[x]^2`` does not. + """ + + advantage = contract.objective.advantage + boundaries = contract.sharding.group_boundaries + parts: list[torch.Tensor] = [] + for index in range(len(boundaries) - 1): + start, end = boundaries[index], boundaries[index + 1] + group = global_rewards[start:end] + count = float(end - start) + centered = group - group.sum() / count + if advantage.normalizer is AdvantageNormalizer.MEAN_ONLY: + parts.append(centered) + continue + variance = (centered * centered).sum() / count + parts.append(centered / variance.clamp_min(advantage.std_eps**2).sqrt()) + return torch.cat(parts, dim=0) + + +def _sequence_totals(values: torch.Tensor, contract: GRPOLossContract) -> torch.Tensor: + """Reduce this rank's per-token values to one total per owned sequence. + + The reduced extent is ``padded_seq_len``, which the contract fixes, so this + sum is byte-for-byte the same work at every DP degree. + """ + + sharding = contract.sharding + view = values.reshape(sharding.local_num_sequences, sharding.padded_seq_len) + return view.sum(dim=1) + + +def _assemble_global_vector( + local_totals: torch.Tensor, contract: GRPOLossContract, dp_group: Any +) -> torch.Tensor: + """Place every rank's per-sequence totals into the fixed global vector. + + Returns ``[num_sequences, ...]``. The length is a property of the contract, + never of the DP degree, and filling it is pure placement -- no arithmetic + touches the gathered values -- so the downstream reduction sees identical + inputs at every degree. + """ + + sharding = contract.sharding + if sharding.dp_world_size == 1: + return local_totals + + dist = _require_distributed_initialized() + trailing = local_totals.shape[1:] + max_local = max(end - start for start, end in sharding.sequence_shard_bounds) + padded = local_totals.new_zeros((max_local, *trailing)) + padded[: local_totals.shape[0]] = local_totals.detach() + gathered = [torch.empty_like(padded) for _ in range(sharding.dp_world_size)] + dist.all_gather(gathered, padded.contiguous(), group=dp_group) + + # This rank's own slice comes from the live tensor: all_gather severs the + # graph, and the other ranks' slices are constants here anyway. + pieces = [] + for rank, (start, end) in enumerate(sharding.sequence_shard_bounds): + pieces.append(local_totals if rank == sharding.dp_rank else gathered[rank][: end - start]) + return torch.cat(pieces, dim=0) + + +def _normalized( + per_sequence_totals: torch.Tensor, + per_sequence_counts: torch.Tensor, + contract: GRPOLossContract, +) -> torch.Tensor: + """Apply the declared token normalizer to fixed-order sequence totals.""" + + reduction = contract.reduction + normalizer = reduction.token_normalizer + if normalizer is TokenNormalizer.FIXED_CONSTANT: + return per_sequence_totals.sum() / float(reduction.fixed_normalizer_constant) + if normalizer is TokenNormalizer.GLOBAL_ACTIVE_TOKENS: + return per_sequence_totals.sum() / per_sequence_counts.sum().to(per_sequence_totals.dtype) + + # PER_SEQUENCE_THEN_MEAN: sequences with no active token contribute nothing + # and are excluded from the outer denominator rather than counted as zero. + live = per_sequence_counts > 0 + denominators = per_sequence_counts.to(per_sequence_totals.dtype).clamp_min(1.0) + per_sequence_means = torch.where( + live, per_sequence_totals / denominators, torch.zeros_like(per_sequence_totals) + ) + return per_sequence_means.sum() / live.sum().to(per_sequence_totals.dtype) + + +class DistributedGRPOLossOp: + """Deterministic GRPO loss over TP-sharded logits with a DP-invariant reduction. + + The WS2 reference (issue #241 PR5). ``policy_local_logits`` is this rank's + ``[n, local_vocab]`` vocabulary shard, not a dense ``[n, vocab]`` tensor: + tensor parallelism is delegated to the vocab-parallel logprob path, and this + operator owns the sum over tokens and sequences. + """ + + op_class = "grpo_loss" + is_batch_invariant = True + + def __init__(self) -> None: + self._logprob = VocabParallelLogprobOp() + + def __call__(self, *args: Any, **kwargs: Any) -> GRPOLossResult: + return self.apply(*args, **kwargs) + + def apply( + self, + policy_local_logits: torch.Tensor, + action_ids: torch.Tensor, + old_logps: torch.Tensor, + rewards: torch.Tensor, + *, + contract: GRPOLossContract, + ref_local_logits: torch.Tensor | None = None, + tp_group: Any = None, + dp_group: Any = None, + num_vocab_tiles: int = DEFAULT_NUM_VOCAB_TILES, + validate: bool = True, + ) -> GRPOLossResult: + if not isinstance(contract, GRPOLossContract): + raise LossContractError("contract must be a GRPOLossContract") + _validate_invocation( + policy_local_logits, + ref_local_logits, + action_ids, + old_logps, + rewards, + contract, + dp_group, + ) + sharding = contract.sharding + objective = contract.objective + if validate: + _preflight_cross_rank_agreement(contract, dp_group, tp_group, num_vocab_tiles) + + logp_policy, _ = self._logprob.apply( + policy_local_logits, + action_ids, + contract=contract.logprob, + tp_group=tp_group, + num_vocab_tiles=num_vocab_tiles, + validate=validate, + ) + active = torch.tensor( + contract.logprob.mask.active_mask, + dtype=torch.bool, + device=policy_local_logits.device, + ) + + delta = (logp_policy - old_logps.float()).masked_fill(~active, 0.0) + ratio = delta.exp() + + if ref_local_logits is None: + kl_terms = torch.zeros_like(logp_policy) + else: + with torch.no_grad(): + logp_ref, _ = self._logprob.apply( + ref_local_logits, + action_ids, + contract=contract.logprob, + tp_group=tp_group, + num_vocab_tiles=num_vocab_tiles, + validate=False, + ) + diff = (logp_ref - logp_policy).masked_fill(~active, 0.0) + if objective.kl_estimator is KLEstimator.K3_UNBIASED: + kl_terms = diff.exp() - diff - 1.0 + else: + kl_terms = -diff + kl_terms = kl_terms.masked_fill(~active, 0.0) + + global_rewards = _gather_global_rewards(rewards, contract, dp_group) + advantages = _group_advantages(global_rewards, contract) + adv_tokens = ( + advantages[sharding.local_sequence_start : sharding.local_sequence_end] + .reshape(-1, 1) + .expand(sharding.local_num_sequences, sharding.padded_seq_len) + .reshape(-1) + ) + + clip = objective.clip + unclipped = ratio * adv_tokens + clipped = ratio.clamp(clip.lower_bound, clip.upper_bound) * adv_tokens + policy_terms = (-torch.minimum(unclipped, clipped)).masked_fill(~active, 0.0) + + packed = torch.stack( + (_sequence_totals(policy_terms, contract), _sequence_totals(kl_terms, contract)), + dim=1, + ) + # Fixed-extent reductions: [local_seqs, padded_seq_len] -> [local_seqs], + # gathered into [num_sequences] -> scalar, at every DP degree. + totals = _assemble_global_vector(packed, contract, dp_group) + per_sequence_policy = totals[:, _CH_POLICY] + per_sequence_kl = totals[:, _CH_KL] + per_sequence_counts = _assemble_global_vector( + _sequence_totals(active.to(torch.long), contract), contract, dp_group + ) + + total_active = int(per_sequence_counts.sum().item()) + if validate and total_active == 0: + raise LossContractError( + "the global batch holds no active tokens; the loss normalizer would divide by zero" + ) + + policy_loss = _normalized(per_sequence_policy, per_sequence_counts, contract) + kl = _normalized(per_sequence_kl, per_sequence_counts, contract) + loss = policy_loss + objective.beta * kl + + provenance = { + "backend_id": BACKEND_ID, + "implementation_kind": "reference", + "num_vocab_tiles": int(num_vocab_tiles), + "padded_seq_len": int(sharding.padded_seq_len), + "num_sequences": int(sharding.num_sequences), + "global_active_tokens": total_active, + "reference_model_used": ref_local_logits is not None, + "requested_contract": contract.to_dict(), + "cross_rank_fingerprint": contract.cross_rank_fingerprint(), + } + return GRPOLossResult( + loss=loss, + policy_loss=policy_loss, + kl=kl, + advantages=advantages, + per_sequence_policy=per_sequence_policy.detach(), + per_sequence_kl=per_sequence_kl.detach(), + per_sequence_active_tokens=per_sequence_counts.detach(), + provenance=provenance, + ) + + +__all__ = [ + "BACKEND_ID", + "GRPOLossResult", + "DistributedGRPOLossOp", +] diff --git a/rl_engine/kernels/registry.py b/rl_engine/kernels/registry.py index 213aa7a2..8391dbbb 100644 --- a/rl_engine/kernels/registry.py +++ b/rl_engine/kernels/registry.py @@ -19,6 +19,15 @@ LogprobRole, MaskMode, ) +from rl_engine.kernels.loss_contract import ( + AdvantageNormalizer, + GRPOLossContract, + KLEstimator, + LossBackendCapability, + LossContractError, + LossDispatchResult, + TokenNormalizer, +) from rl_engine.platforms.device import device_ctx from rl_engine.utils.logger import logger @@ -89,6 +98,10 @@ class OpBackend(Enum, metaclass=_KernelEnumMeta): PYTORCH_VOCAB_PARALLEL_LOGP = ( "rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp.VocabParallelLogprobOp" ) + # Deterministic GRPO loss on the TP-aware logprob path (WS2 #241 PR5) + PYTORCH_DISTRIBUTED_GRPO_LOSS = ( + "rl_engine.kernels.ops.pytorch.loss.distributed_grpo_loss.DistributedGRPOLossOp" + ) # RMSNorm(pre-norm / QK-Norm) - pure Pytorch reference(ws1 ground-truth) PYTORCH_NATIVE_RMS_NORM = "rl_engine.kernels.ops.pytorch.norm.rms_norm.NativeRMSNormOp" @@ -377,6 +390,34 @@ def __init__(self): prepend=True, ) + # WS2 loss dispatch starts empty: the existing single-GPU GRPO ops + # declare no LossBackendCapability, so they are unreachable from + # get_loss_op and cannot be picked up as a silent fallback for a + # contract that asks for deterministic mesh-wide reduction. + self._loss_candidates: Dict[str, list] = {platform: [] for platform in self._priority_map} + self._loss_capabilities: Dict[str, Dict[OpBackend, LossBackendCapability]] = { + platform: {} for platform in self._priority_map + } + ws2_grpo_loss_capability = LossBackendCapability( + backend_id="pytorch-distributed-grpo-loss-ws2", + token_normalizers=frozenset(TokenNormalizer), + kl_estimators=frozenset(KLEstimator), + advantage_normalizers=frozenset(AdvantageNormalizer), + determinism_scopes=frozenset( + {DeterminismScope.CROSS_TP_BITWISE, DeterminismScope.FIXED_TOPOLOGY} + ), + dp_world_sizes=None, + supports_variable_group_sizes=True, + supports_asymmetric_clip=True, + implementation_kind="reference", + ) + for ws2_platform in self._priority_map: + self.register_loss_backend( + OpBackend.PYTORCH_DISTRIBUTED_GRPO_LOSS, + ws2_grpo_loss_capability, + platform=ws2_platform, + ) + def _adjust_priority_from_env(self): rocm_attn_backend = os.getenv("RL_KERNEL_ROCM_ATTN_BACKEND", "").strip().lower() if rocm_attn_backend in {"flash_attn", "flash-attn", "flash_attention"}: @@ -597,6 +638,138 @@ def get_logprob_op( f"real_vocab={contract.sharding.real_vocab_size}. Rejections: {details}" ) + def register_loss_backend( + self, + backend: OpBackend, + capability: LossBackendCapability, + *, + platform: Optional[str] = None, + prepend: bool = False, + ) -> None: + """Register (or replace) a backend for WS2 contract-aware loss dispatch. + + The seam that makes a loss backend selectable by ``get_loss_op``. It is + deliberately separate from the logprob registry: a backend may serve one + contract and not the other, and the deterministic GRPO loss consumes a + logprob backend rather than being one. + """ + + if not isinstance(backend, OpBackend): + raise LossContractError("backend must be an OpBackend") + if not isinstance(capability, LossBackendCapability): + raise LossContractError("capability must be a LossBackendCapability") + resolved_platform = platform if platform is not None else self._platform() + if resolved_platform not in self._priority_map: + raise LossContractError( + f"unsupported platform {resolved_platform!r}; expected one of " + f"{sorted(self._priority_map)}" + ) + candidates = self._loss_candidates.setdefault(resolved_platform, []) + self._loss_capabilities.setdefault(resolved_platform, {})[backend] = capability + if backend not in candidates: + if prepend: + candidates.insert(0, backend) + else: + candidates.append(backend) + + def get_loss_op( + self, + contract: GRPOLossContract, + *, + requested_backend: str = "auto", + ) -> LossDispatchResult: + """Resolve only a backend that explicitly supports the WS2 loss contract. + + Mirrors ``get_logprob_op``: strictness comes from the contract's + capability checks rather than from the policy string, and a contract no + registered backend can serve fails loudly instead of falling back to an + op with different reduction semantics. + """ + + if not isinstance(contract, GRPOLossContract): + raise LossContractError("contract must be a GRPOLossContract") + if not isinstance(requested_backend, str) or not requested_backend.strip(): + raise LossContractError("requested_backend must be a non-empty string") + requested_backend = requested_backend.strip() + if requested_backend.lower() == "deterministic": + raise LossContractError( + 'requested_backend="deterministic" is not a dispatch policy; request ' + "determinism through LossReductionSpec.determinism_scope and match it " + "against backend determinism_scopes instead" + ) + + platform = self._platform() + candidates = self._loss_candidates.get(platform, []) + rejected: list[str] = [] + capability_rejections = 0 + + platform_capabilities = self._loss_capabilities.get(platform, {}) + for backend in candidates: + capability = platform_capabilities.get(backend) + if capability is None: + rejected.append(f"{backend.name}: no LossBackendCapability declared") + capability_rejections += 1 + continue + policy_mismatch = self._loss_policy_mismatch(requested_backend, capability) + if policy_mismatch is not None: + rejected.append(f"{backend.name}: {policy_mismatch}") + continue + capability_incompat = list(capability.incompatibilities(contract)) + if capability_incompat: + rejected.append(f"{backend.name}: " + "; ".join(capability_incompat)) + capability_rejections += 1 + continue + + op = self._get_or_create_backend(backend) + if op is None: + rejected.append(f"{backend.name}: backend could not be loaded or instantiated") + capability_rejections += 1 + continue + + provenance = { + "requested_backend": requested_backend, + "actual_backend": capability.backend_id, + "backend_enum": backend.name, + "platform": platform, + "fallback": capability_rejections > 0, + "prior_rejections": list(rejected), + "contract": contract.to_dict(), + "capability": capability.to_dict(), + } + return LossDispatchResult(op=op, capability=capability, provenance=provenance) + + details = " | ".join(rejected) if rejected else "no candidates registered" + raise RuntimeError( + "No loss backend supports the requested WS2 GRPO contract on " + f"{platform}: normalizer={contract.reduction.token_normalizer.value}, " + f"kl={contract.objective.kl_estimator.value}, " + f"TP={contract.logprob.sharding.tp_world_size}, " + f"DP={contract.sharding.dp_world_size}, CP={contract.sharding.cp_world_size}. " + f"Rejections: {details}" + ) + + @staticmethod + def _loss_policy_mismatch( + requested_backend: str, + capability: LossBackendCapability, + ) -> str | None: + policy = requested_backend.lower() + if policy == "auto": + return None + if policy in IMPLEMENTATION_KINDS: + if capability.implementation_kind == policy: + return None + return ( + f"implementation_kind={capability.implementation_kind} does not satisfy " + f"requested_backend={policy}" + ) + if capability.backend_id == requested_backend: + return None + return ( + f"backend_id={capability.backend_id} does not match " + f"requested_backend={requested_backend}" + ) + @staticmethod def _logprob_policy_mismatch( requested_backend: str, diff --git a/tests/test_distributed_grpo_loss.py b/tests/test_distributed_grpo_loss.py new file mode 100644 index 00000000..b0a2321e --- /dev/null +++ b/tests/test_distributed_grpo_loss.py @@ -0,0 +1,866 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Deterministic GRPO loss on the TP-aware logprob path. + +The headline claim under test is that the scalar loss and its gradient are +bitwise-identical across every TP x DP degree, given a fixed vocab tile count. +``TestMeshBitwise`` is the file's centre of gravity: it runs the reachable +degrees on real NCCL ranks and compares raw bit patterns against the single-rank +baseline. + +The remaining classes support that claim rather than duplicate it: the single- +rank tests pin the objective's algebraic identities (ratio exactly 1, KL exactly +0), the negative controls prove the bitwise comparisons are not vacuous, and the +guard tests check that a rank which disagrees about the invocation aborts +instead of corrupting the merge. + +Comparisons are made on ``per_sequence_policy``/``per_sequence_kl`` rather than +on the scalar loss alone. That is not a stylistic choice: measured over this +file's own inputs, regrouping the token sum moves the per-sequence vector in 12 +of 12 seeds but the scalar loss in only 5 of 12, because averaging +``NUM_SEQUENCES`` totals into one fp32 number rounds most reorderings away. +Asserting on the scalar alone would let a wrong reduction pass most of the time. + +Context parallelism is out of scope (see ``LossShardingSpec``); the contract +rejects ``cp_world_size > 1`` and ``TestGuards`` covers that. + +Multi-rank tests need one GPU per rank and skip otherwise. They stay small on +purpose -- a 1000-token vocabulary and 8 sequences of 32 slots -- so they can +share a node with a running training job; each worker additionally caps itself +with ``set_per_process_memory_fraction`` so a regression here cannot starve a +co-tenant. +""" + +from __future__ import annotations + +import math +import os +import queue +import tempfile +import traceback +from pathlib import Path + +import pytest +import torch +import torch.multiprocessing as mp + +from rl_engine.kernels.logprob_contract import ( + LogprobContract, + LogprobDType, + LogprobRole, + MaskSpec, + ReductionSpec, + ShardingSpec, +) +from rl_engine.kernels.loss_contract import ( + AdvantageNormalizer, + AdvantageSpec, + ClipSpec, + GRPOLossContract, + KLEstimator, + LossContractError, + LossReductionSpec, + LossShardingSpec, + ObjectiveSpec, + TokenNormalizer, +) +from rl_engine.kernels.ops.pytorch.loss.distributed_grpo_loss import ( + BACKEND_ID, + DistributedGRPOLossOp, +) + +# Global batch geometry, shared by every configuration so the comparisons are +# between meshes rather than between problems. The degrees below are chosen so +# that DP and CP each reach 4 while every shard stays tile-aligned. +NUM_SEQUENCES = 8 +PADDED_SEQ_LEN = 32 +NUM_TOKEN_SLOTS = NUM_SEQUENCES * PADDED_SEQ_LEN +# Deliberately straddles the DP=2 split at 4 and the DP=4 splits at 2/4/6, so the +# replicated-advantage path is exercised by groups that no single rank owns. +GROUP_BOUNDARIES = (0, 3, NUM_SEQUENCES) +REAL_VOCAB = 1000 +PADDED_VOCAB = 1024 +NUM_VOCAB_TILES = 32 +BETA = 0.04 +SEED = 20260811 + +_SPAWN_TIMEOUT_S = 600 +# Fraction of each card the workers may allocate. The test tensors need a few +# megabytes; the cap exists so a bug cannot balloon into a co-tenant job. +_MEMORY_FRACTION = 0.02 + + +def _bits(tensor: torch.Tensor) -> torch.Tensor: + """Raw bit pattern, so -0.0 vs 0.0 and NaN vs NaN compare honestly.""" + + view_dtype = { + torch.float32: torch.int32, + torch.bfloat16: torch.int16, + torch.float16: torch.int16, + }[tensor.dtype] + return tensor.contiguous().view(view_dtype) + + +def _scalar_bits(tensor: torch.Tensor) -> int: + return int(_bits(tensor.detach().float().cpu()).item()) + + +def _vector_bits(tensor: torch.Tensor) -> list[int]: + return _bits(tensor.detach().float().cpu()).tolist() + + +def _reduction_fingerprint(result) -> tuple: + """Everything a change of reduction order can move, at full resolution. + + The per-sequence vectors come first because they are the sensitive part; + the scalars are carried along so a normalizer bug is caught too. + """ + + return ( + _vector_bits(result.per_sequence_policy), + _vector_bits(result.per_sequence_kl), + _scalar_bits(result.loss), + _scalar_bits(result.policy_loss), + _scalar_bits(result.kl), + ) + + +def _cuda_device_count() -> int: + try: + return torch.cuda.device_count() + except Exception: # pragma: no cover - driver-level failures + return 0 + + +def _requires_gpus(count: int): + return pytest.mark.skipif( + _cuda_device_count() < count, + reason=f"needs {count} CUDA devices for a real {count}-rank mesh", + ) + + +# --------------------------------------------------------------------------- # +# Global problem definition +# --------------------------------------------------------------------------- # +def _global_active_mask() -> tuple[bool, ...]: + """Right-padded sequences of strictly decreasing length. + + The lengths are staggered so that no two DP shards hold the same number of + active tokens; equal counts would let a rank-local normalizer accidentally + agree with the global one and pass a test it should fail. + """ + + mask: list[bool] = [] + for seq in range(NUM_SEQUENCES): + real_len = PADDED_SEQ_LEN - seq * 3 + mask.extend(slot < real_len for slot in range(PADDED_SEQ_LEN)) + return tuple(mask) + + +GLOBAL_ACTIVE_MASK = _global_active_mask() + + +def _global_inputs(seed: int = SEED) -> dict[str, torch.Tensor]: + """Deterministic global tensors every rank slices its own view out of. + + ``old_logps`` is centred on ``-log(REAL_VOCAB)``, the scale of a selected + logprob under near-uniform logits, so the importance ratios land around 1 + with real spread. Leaving it centred on 0 would make every ratio ~1e-3 and + every per-token loss term nearly identical, and a sum of near-identical + values is almost invariant to how it is grouped -- which would quietly + drain the power out of every bitwise comparison in this file. + """ + + gen = torch.Generator().manual_seed(seed) + return { + "policy": torch.randn(NUM_TOKEN_SLOTS, PADDED_VOCAB, generator=gen), + "ref": torch.randn(NUM_TOKEN_SLOTS, PADDED_VOCAB, generator=gen), + "action_ids": torch.randint(0, REAL_VOCAB, (NUM_TOKEN_SLOTS,), generator=gen), + "old_logps": torch.randn(NUM_TOKEN_SLOTS, generator=gen) * 0.5 - math.log(REAL_VOCAB), + "rewards": torch.randn(NUM_SEQUENCES, generator=gen), + } + + +def _dp_bounds(dp: int) -> tuple[tuple[int, int], ...]: + """Contiguous sequence partition in DP-rank order.""" + + seqs = NUM_SEQUENCES // dp + return tuple((d * seqs, (d + 1) * seqs) for d in range(dp)) + + +def _owned_rows(bounds: tuple[int, int]) -> list[int]: + """Global token-slot indices this shard owns, in canonical local row order.""" + + start, end = bounds + return [ + seq * PADDED_SEQ_LEN + slot for seq in range(start, end) for slot in range(PADDED_SEQ_LEN) + ] + + +def _build_contract( + *, + tp_rank: int, + tp: int, + dp_rank: int, + dp: int, + objective: ObjectiveSpec | None = None, + reduction: LossReductionSpec | None = None, +) -> GRPOLossContract: + bounds = _dp_bounds(dp) + rows = _owned_rows(bounds[dp_rank]) + shard = PADDED_VOCAB // tp + logprob = LogprobContract( + role=LogprobRole.TRAIN, + dtype=LogprobDType.FP32, + mask=MaskSpec( + num_tokens=len(rows), + active_mask=tuple(GLOBAL_ACTIVE_MASK[row] for row in rows), + ), + sharding=ShardingSpec( + tp_rank=tp_rank, + tp_world_size=tp, + vocab_shard_bounds=tuple((r * shard, (r + 1) * shard) for r in range(tp)), + real_vocab_size=REAL_VOCAB, + padded_vocab_size=PADDED_VOCAB, + ), + reduction=ReductionSpec(), + ) + sharding = LossShardingSpec( + dp_rank=dp_rank, + dp_world_size=dp, + num_sequences=NUM_SEQUENCES, + padded_seq_len=PADDED_SEQ_LEN, + sequence_shard_bounds=bounds, + group_boundaries=GROUP_BOUNDARIES, + ) + return GRPOLossContract( + logprob=logprob, + sharding=sharding, + objective=objective if objective is not None else ObjectiveSpec(beta=BETA), + reduction=reduction if reduction is not None else LossReductionSpec(), + ) + + +def _rank_inputs( + globals_: dict[str, torch.Tensor], + *, + tp_rank: int, + tp: int, + bounds: tuple[int, int], + device: torch.device, +) -> dict[str, torch.Tensor]: + rows = _owned_rows(bounds) + shard = PADDED_VOCAB // tp + cols = slice(tp_rank * shard, (tp_rank + 1) * shard) + start, end = bounds + return { + "policy": globals_["policy"][rows, cols].clone().to(device).requires_grad_(True), + "ref": globals_["ref"][rows, cols].clone().to(device), + "action_ids": globals_["action_ids"][rows].clone().to(device), + "old_logps": globals_["old_logps"][rows].clone().to(device), + "rewards": globals_["rewards"][start:end].clone().to(device), + } + + +def _single_rank_setup( + *, + objective: ObjectiveSpec | None = None, + reduction: LossReductionSpec | None = None, + device: str = "cpu", + globals_: dict[str, torch.Tensor] | None = None, +) -> tuple[GRPOLossContract, dict[str, torch.Tensor]]: + """Contract and tensors for one rank owning the whole batch.""" + + contract = _build_contract( + tp_rank=0, tp=1, dp_rank=0, dp=1, objective=objective, reduction=reduction + ) + tensors = _rank_inputs( + globals_ if globals_ is not None else _global_inputs(), + tp_rank=0, + tp=1, + bounds=(0, NUM_SEQUENCES), + device=torch.device(device), + ) + return contract, tensors + + +def _run_single_rank( + *, + num_vocab_tiles: int = NUM_VOCAB_TILES, + with_reference: bool = True, + **setup, +): + """Baseline invocation: one rank, no collectives.""" + + contract, tensors = _single_rank_setup(**setup) + result = DistributedGRPOLossOp().apply( + tensors["policy"], + tensors["action_ids"], + tensors["old_logps"], + tensors["rewards"], + contract=contract, + ref_local_logits=tensors["ref"] if with_reference else None, + num_vocab_tiles=num_vocab_tiles, + ) + return result, tensors, contract + + +# --------------------------------------------------------------------------- # +# Multi-rank harness +# --------------------------------------------------------------------------- # +def _mesh_worker(rank, world_size, init_method, result_queue, tp, dp, scenario): + """One NCCL rank of a TP x DP mesh. + + Only plain Python values go back on the queue. Tensors sent over a + multiprocessing queue travel by shared memory and are lost when the sender + exits before the parent maps them, which surfaces as an empty queue rather + than an error. + """ + + payload = {"rank": rank} + try: + import torch.distributed as dist + + device_index = rank % torch.cuda.device_count() + torch.cuda.set_device(device_index) + torch.cuda.set_per_process_memory_fraction(_MEMORY_FRACTION, device_index) + device = torch.device("cuda", device_index) + dist.init_process_group( + backend="nccl", init_method=init_method, world_size=world_size, rank=rank + ) + + # Rank layout puts TP fastest: rank = dp_rank * tp + tp_rank. + tp_rank = rank % tp + dp_rank = rank // tp + # new_group is collective, so every rank builds every subgroup in the + # same order even though it only keeps one of each. + tp_group = None + dp_group = None + if tp > 1: + tp_groups = [ + dist.new_group(ranks=list(range(base * tp, (base + 1) * tp))) for base in range(dp) + ] + tp_group = tp_groups[dp_rank] + if dp > 1: + dp_groups = [ + dist.new_group(ranks=list(range(offset, world_size, tp))) for offset in range(tp) + ] + dp_group = dp_groups[tp_rank] + + objective = ObjectiveSpec(beta=BETA) + if scenario == "preflight_dp_mismatch" and dp_rank == 1: + # Perturb a pure fingerprint field. Changing the batch geometry + # instead would change this rank's local shapes, so it would fail + # while constructing its contract -- before the preflight -- and + # strand the other ranks inside the all-gather. + objective = ObjectiveSpec(beta=BETA * 2) + if scenario == "preflight_tp_mismatch" and tp_rank == 1: + # beta is invisible to the logprob contract, so only the loss + # preflight's TP-axis check can catch this one. + objective = ObjectiveSpec(beta=BETA * 2) + + contract = _build_contract( + tp_rank=tp_rank, tp=tp, dp_rank=dp_rank, dp=dp, objective=objective + ) + tensors = _rank_inputs( + _global_inputs(), + tp_rank=tp_rank, + tp=tp, + bounds=contract.sharding.sequence_shard_bounds[dp_rank], + device=device, + ) + op = DistributedGRPOLossOp() + result = op.apply( + tensors["policy"], + tensors["action_ids"], + tensors["old_logps"], + tensors["rewards"], + contract=contract, + ref_local_logits=tensors["ref"], + tp_group=tp_group, + dp_group=dp_group, + num_vocab_tiles=NUM_VOCAB_TILES, + ) + result.loss.backward() + + grad = tensors["policy"].grad + payload.update( + { + "ok": True, + "loss_bits": _scalar_bits(result.loss), + "policy_bits": _scalar_bits(result.policy_loss), + "kl_bits": _scalar_bits(result.kl), + "per_sequence_policy_bits": _vector_bits(result.per_sequence_policy), + "per_sequence_kl_bits": _vector_bits(result.per_sequence_kl), + "per_sequence_counts": result.per_sequence_active_tokens.cpu().tolist(), + "advantage_bits": _vector_bits(result.advantages), + "global_active": result.provenance["global_active_tokens"], + "backend_id": result.provenance["backend_id"], + "grad_nonzero": bool(grad.abs().sum().item() > 0.0), + # Gradient of one fixed global token slot, keyed by vocab shard + # so the parent can reassemble the full row across TP ranks. + "grad_row_bits": _vector_bits(grad[0]) if dp_rank == 0 else None, + "vocab_start": contract.logprob.sharding.local_vocab_start, + } + ) + except BaseException as exc: # the parent re-raises the text + payload.update( + { + "ok": False, + "error": f"{type(exc).__name__}: {exc}", + "tb": traceback.format_exc(), + } + ) + finally: + try: + import torch.distributed as dist + + if dist.is_initialized(): + dist.barrier() + dist.destroy_process_group() + except Exception: # pragma: no cover - teardown best effort + pass + try: + result_queue.put(payload) + except Exception: # pragma: no cover - queue already closed + pass + + +def _run_mesh(tp: int, dp: int, *, scenario: str = "correctness") -> list[dict]: + """Spawn a ``tp * dp`` NCCL mesh and collect one payload per rank.""" + + world_size = tp * dp + os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True") + ctx = mp.get_context("spawn") + result_queue = ctx.Queue() + with tempfile.TemporaryDirectory() as tmp: + init_method = f"file://{Path(tmp) / 'store'}" + spawned = mp.spawn( + _mesh_worker, + args=(world_size, init_method, result_queue, tp, dp, scenario), + nprocs=world_size, + join=False, + ) + payloads: list[dict] = [] + try: + for _ in range(world_size): + payloads.append(result_queue.get(timeout=_SPAWN_TIMEOUT_S)) + except queue.Empty: # pragma: no cover - only on a genuine hang + pytest.fail( + f"TP={tp} DP={dp}: only {len(payloads)}/{world_size} ranks reported " + f"within {_SPAWN_TIMEOUT_S}s" + ) + finally: + spawned.join(timeout=_SPAWN_TIMEOUT_S) + return sorted(payloads, key=lambda item: item["rank"]) + + +def _require_all_ok(payloads: list[dict]) -> None: + failures = [item for item in payloads if not item.get("ok")] + if failures: + head = failures[0] + pytest.fail(f"rank {head['rank']} failed: {head['error']}\n{head['tb']}") + + +def _assemble_grad_row(payloads: list[dict]) -> list[int]: + """Reassemble the fixed token slot's gradient row across TP vocab shards.""" + + contributions = [item for item in payloads if item["grad_row_bits"] is not None] + contributions.sort(key=lambda item: item["vocab_start"]) + row: list[int] = [] + for item in contributions: + row.extend(item["grad_row_bits"]) + return row + + +def _consensus(payloads: list[dict]) -> dict: + """Collapse the mesh's per-rank payloads into the one replicated answer.""" + + _require_all_ok(payloads) + replicated = ( + "loss_bits", + "policy_bits", + "kl_bits", + "per_sequence_policy_bits", + "per_sequence_kl_bits", + "per_sequence_counts", + "global_active", + "advantage_bits", + ) + for key in replicated: + values = {repr(item[key]) for item in payloads} + assert len(values) == 1, f"ranks disagree on {key}: {values}" + assert all(item["grad_nonzero"] for item in payloads), ( + "some rank produced a zero gradient; the all-gather likely severed the " + "graph for that rank's own block" + ) + head = payloads[0] + consensus = {key: head[key] for key in replicated} + consensus["grad_row"] = _assemble_grad_row(payloads) + return consensus + + +# --------------------------------------------------------------------------- # +# Single-rank behaviour +# --------------------------------------------------------------------------- # +class TestSingleRank: + def test_forward_and_backward_run(self): + result, tensors, contract = _run_single_rank() + result.loss.backward() + assert result.loss.dtype is torch.float32 + assert result.loss.shape == () + assert tensors["policy"].grad.abs().sum().item() > 0.0 + assert result.provenance["backend_id"] == BACKEND_ID + assert result.provenance["global_active_tokens"] == sum(GLOBAL_ACTIVE_MASK) + + def test_unpacks_as_the_legacy_triple(self): + result, _, _ = _run_single_rank() + loss, policy_loss, kl = result + assert _scalar_bits(loss) == _scalar_bits(result.loss) + assert _scalar_bits(policy_loss) == _scalar_bits(result.policy_loss) + assert _scalar_bits(kl) == _scalar_bits(result.kl) + + def test_run_to_run_bitwise_stability(self): + first, _, _ = _run_single_rank() + second, _, _ = _run_single_rank() + assert _scalar_bits(first.loss) == _scalar_bits(second.loss) + assert _scalar_bits(first.policy_loss) == _scalar_bits(second.policy_loss) + assert _scalar_bits(first.kl) == _scalar_bits(second.kl) + + def test_advantages_are_group_centred(self): + result, _, _ = _run_single_rank() + advantages = result.advantages + for start, end in zip(GROUP_BOUNDARIES[:-1], GROUP_BOUNDARIES[1:], strict=True): + assert advantages[start:end].sum().item() == pytest.approx(0.0, abs=1e-5) + + def test_inactive_tokens_do_not_affect_the_loss(self): + # Inactive slots carry real logits here, so a backend that forgot to + # mask them would shift the loss rather than merely change a padding. + baseline, _, _ = _run_single_rank() + perturbed_globals = _global_inputs() + inactive = [i for i, flag in enumerate(GLOBAL_ACTIVE_MASK) if not flag] + perturbed_globals["policy"][inactive] += 7.5 + perturbed_globals["old_logps"][inactive] -= 3.25 + perturbed, _, _ = _run_single_rank(globals_=perturbed_globals) + assert _scalar_bits(perturbed.loss) == _scalar_bits(baseline.loss) + + def test_dispatch_resolves_this_backend(self): + from rl_engine.kernels.registry import kernel_registry + + _, _, contract = _run_single_rank() + dispatched = kernel_registry.get_loss_op(contract) + assert dispatched.capability.backend_id == BACKEND_ID + assert isinstance(dispatched.op, DistributedGRPOLossOp) + assert dispatched.provenance["fallback"] is False + + +class TestKLZeroIdentity: + """Acceptance criterion: the reference-equals-policy identity is exact.""" + + def _identity_run(self, **overrides): + # old_logps set to the operator's own selected logprob, so the ratio is + # exp(0) = 1 exactly rather than approximately. + contract, tensors = _single_rank_setup(**overrides) + op = DistributedGRPOLossOp() + with torch.no_grad(): + logp, _ = op._logprob.apply( + tensors["policy"], + tensors["action_ids"], + contract=contract.logprob, + num_vocab_tiles=NUM_VOCAB_TILES, + ) + result = op.apply( + tensors["policy"], + tensors["action_ids"], + logp, + tensors["rewards"], + contract=contract, + ref_local_logits=tensors["policy"].detach(), + num_vocab_tiles=NUM_VOCAB_TILES, + ) + return result, tensors + + def test_kl_is_exactly_zero(self): + result, _ = self._identity_run() + assert _scalar_bits(result.kl) == _scalar_bits(torch.zeros(())) + + def test_loss_reduces_to_the_policy_term(self): + result, _ = self._identity_run() + assert _scalar_bits(result.loss) == _scalar_bits(result.policy_loss) + + def test_ratio_is_exactly_one_so_clipping_cannot_bind(self): + # A ratio that is only approximately 1 would land outside a sufficiently + # tight clip range and change the answer; an exact 1 cannot. + tight, _ = self._identity_run( + objective=ObjectiveSpec(beta=BETA, clip=ClipSpec(clip_eps_low=1e-7, clip_eps_high=1e-7)) + ) + loose, _ = self._identity_run( + objective=ObjectiveSpec(beta=BETA, clip=ClipSpec(clip_eps_low=0.9, clip_eps_high=0.9)) + ) + assert _scalar_bits(tight.loss) == _scalar_bits(loose.loss) + + def test_policy_term_matches_the_masked_advantage_mean(self): + result, _ = self._identity_run() + advantages = result.advantages + mask = torch.tensor(GLOBAL_ACTIVE_MASK) + per_token = advantages.reshape(-1, 1).expand(NUM_SEQUENCES, PADDED_SEQ_LEN).reshape(-1) + expected = -per_token.masked_fill(~mask, 0.0).sum() / mask.sum() + # Not a bitwise comparison: this reference sums the flat [N] vector, + # while the operator sums through its fixed tile structure. + assert result.policy_loss.item() == pytest.approx(expected.item(), abs=1e-6) + + +class TestNormalizers: + def test_global_active_tokens_matches_a_flat_masked_mean(self): + result, _, _ = _run_single_rank() + assert result.provenance["global_active_tokens"] == sum(GLOBAL_ACTIVE_MASK) + + def test_normalizers_disagree_on_unequal_sequence_lengths(self): + # If these ever agreed, the normalizer choice would be untestable and + # the sequence lengths in this file would have stopped being staggered. + token_mean, _, _ = _run_single_rank() + seq_mean, _, _ = _run_single_rank( + reduction=LossReductionSpec(token_normalizer=TokenNormalizer.PER_SEQUENCE_THEN_MEAN) + ) + fixed, _, _ = _run_single_rank( + reduction=LossReductionSpec( + token_normalizer=TokenNormalizer.FIXED_CONSTANT, + fixed_normalizer_constant=NUM_TOKEN_SLOTS, + ) + ) + values = { + _scalar_bits(token_mean.loss), + _scalar_bits(seq_mean.loss), + _scalar_bits(fixed.loss), + } + assert len(values) == 3 + + def test_fixed_constant_normalizer_scales_the_token_sum(self): + active = sum(GLOBAL_ACTIVE_MASK) + token_mean, _, _ = _run_single_rank() + fixed, _, _ = _run_single_rank( + reduction=LossReductionSpec( + token_normalizer=TokenNormalizer.FIXED_CONSTANT, + fixed_normalizer_constant=NUM_TOKEN_SLOTS, + ) + ) + assert fixed.policy_loss.item() == pytest.approx( + token_mean.policy_loss.item() * active / NUM_TOKEN_SLOTS, rel=1e-6 + ) + + def test_kl_estimators_differ(self): + k3, _, _ = _run_single_rank() + k1, _, _ = _run_single_rank( + objective=ObjectiveSpec(beta=BETA, kl_estimator=KLEstimator.K1_LOG_RATIO) + ) + assert _scalar_bits(k3.kl) != _scalar_bits(k1.kl) + # k3 is non-negative by construction; the plain log-ratio is not. + assert k3.kl.item() >= 0.0 + + def test_mean_only_advantage_skips_the_std_divisor(self): + std_normalized, _, _ = _run_single_rank() + mean_only, _, _ = _run_single_rank( + objective=ObjectiveSpec( + beta=BETA, + advantage=AdvantageSpec(normalizer=AdvantageNormalizer.MEAN_ONLY), + ) + ) + assert _scalar_bits(std_normalized.loss) != _scalar_bits(mean_only.loss) + + +class TestNegativeControls: + """Prove the bitwise assertions elsewhere are not comparing constants. + + Each control changes something that genuinely regroups or rescales the + reduction and asserts the compared surface notices. If one of these ever + starts passing trivially, the corresponding positive assertion has stopped + meaning anything. + """ + + def test_regrouping_the_token_sum_moves_the_per_sequence_vector(self): + # Split each sequence's token sum in half before adding, changing the + # summation tree without changing a single input value. The scalar loss + # absorbs that most of the time; the per-sequence vector does not, which + # is why it is the comparison surface everywhere else in this file. + import rl_engine.kernels.ops.pytorch.loss.distributed_grpo_loss as module + + baseline, _, _ = _run_single_rank() + original = module._sequence_totals + + def halved(values, contract): + view = values.reshape( + contract.sharding.local_num_sequences, 2, contract.sharding.padded_seq_len // 2 + ) + return view.sum(dim=2).sum(dim=1) + + module._sequence_totals = halved + try: + regrouped, _, _ = _run_single_rank() + finally: + module._sequence_totals = original + assert _vector_bits(baseline.per_sequence_policy) != _vector_bits( + regrouped.per_sequence_policy + ) + + def test_vocab_tile_count_perturbs_the_logprob_it_consumes(self): + baseline, _, _ = _run_single_rank() + retiled, _, _ = _run_single_rank(num_vocab_tiles=NUM_VOCAB_TILES * 2) + assert _reduction_fingerprint(baseline) != _reduction_fingerprint(retiled) + + def test_sequence_order_matters_to_the_scalar(self): + # An all_reduce would combine sequence totals in topology order rather + # than in global sequence order. Permuting the assembled grid is a + # stand-in for that mistake, and the loss must notice. + import rl_engine.kernels.ops.pytorch.loss.distributed_grpo_loss as module + + baseline, _, _ = _run_single_rank() + permutation = torch.tensor([5, 2, 7, 0, 3, 6, 1, 4]) + original = module._assemble_global_vector + module._assemble_global_vector = lambda totals, contract, group: original( + totals, contract, group + )[permutation] + try: + permuted, _, _ = _run_single_rank() + finally: + module._assemble_global_vector = original + assert _vector_bits(baseline.per_sequence_policy) != _vector_bits( + permuted.per_sequence_policy + ) + + def test_clip_epsilon_perturbs_the_loss(self): + baseline, _, _ = _run_single_rank() + clipped, _, _ = _run_single_rank( + objective=ObjectiveSpec(beta=BETA, clip=ClipSpec(clip_eps_low=0.01, clip_eps_high=0.01)) + ) + assert _reduction_fingerprint(baseline) != _reduction_fingerprint(clipped) + + +class TestGuards: + def test_reference_logits_required_when_beta_is_positive(self): + contract, tensors = _single_rank_setup() + with pytest.raises(LossContractError, match="ref_local_logits is required"): + DistributedGRPOLossOp().apply( + tensors["policy"], + tensors["action_ids"], + tensors["old_logps"], + tensors["rewards"], + contract=contract, + num_vocab_tiles=NUM_VOCAB_TILES, + ) + + def test_reference_optional_when_beta_is_zero(self): + result, _, _ = _run_single_rank(objective=ObjectiveSpec(beta=0.0), with_reference=False) + assert _scalar_bits(result.loss) == _scalar_bits(result.policy_loss) + assert _scalar_bits(result.kl) == _scalar_bits(torch.zeros(())) + assert result.provenance["reference_model_used"] is False + + def test_row_count_must_match_owned_slots(self): + contract, tensors = _single_rank_setup() + with pytest.raises(LossContractError, match="token slots"): + DistributedGRPOLossOp().apply( + tensors["policy"][:-1], + tensors["action_ids"][:-1], + tensors["old_logps"][:-1], + tensors["rewards"], + contract=contract, + ref_local_logits=tensors["ref"][:-1], + num_vocab_tiles=NUM_VOCAB_TILES, + ) + + def test_reward_count_must_match_owned_sequences(self): + contract, tensors = _single_rank_setup() + with pytest.raises(LossContractError, match="one entry per sequence"): + DistributedGRPOLossOp().apply( + tensors["policy"], + tensors["action_ids"], + tensors["old_logps"], + tensors["rewards"][:-1], + contract=contract, + ref_local_logits=tensors["ref"], + num_vocab_tiles=NUM_VOCAB_TILES, + ) + + +# --------------------------------------------------------------------------- # +# The claim: bitwise equality across the mesh +# --------------------------------------------------------------------------- # +# Every reachable (TP, DP) combination with at most 4 ranks. Larger degrees +# need a bigger node and run unchanged there via the same helper. +MESH_CONFIGS = [ + pytest.param(2, 1, id="tp2"), + pytest.param(4, 1, id="tp4"), + pytest.param(1, 2, id="dp2"), + pytest.param(1, 4, id="dp4"), + pytest.param(2, 2, id="tp2xdp2"), +] + + +@pytest.fixture(scope="module") +def gpu_baseline() -> dict: + """The TP=DP=1 answer, computed on one GPU so the mesh comparison is + device-for-device rather than CPU-versus-GPU. Built once for the module.""" + + if _cuda_device_count() < 1: + pytest.skip("needs a CUDA device") + return _consensus(_run_mesh(1, 1)) + + +@_requires_gpus(1) +class TestMeshBitwise: + @pytest.mark.parametrize(("tp", "dp"), MESH_CONFIGS) + def test_mesh_matches_the_single_rank_baseline(self, tp, dp, gpu_baseline): + world = tp * dp + if _cuda_device_count() < world: + pytest.skip(f"needs {world} CUDA devices for TP={tp} DP={dp}") + baseline = gpu_baseline + actual = _consensus(_run_mesh(tp, dp)) + label = f"TP={tp} DP={dp}" + + assert actual["global_active"] == baseline["global_active"] + assert actual["per_sequence_counts"] == baseline["per_sequence_counts"] + assert actual["advantage_bits"] == baseline["advantage_bits"] + # The sensitive comparison: per-sequence totals, before the scalar + # average rounds a reordering away. + assert actual["per_sequence_policy_bits"] == baseline["per_sequence_policy_bits"], ( + f"{label} per-sequence policy totals differ from the single-rank baseline" + ) + assert actual["per_sequence_kl_bits"] == baseline["per_sequence_kl_bits"], ( + f"{label} per-sequence KL totals differ from the single-rank baseline" + ) + assert actual["loss_bits"] == baseline["loss_bits"], f"{label} loss differs" + assert actual["policy_bits"] == baseline["policy_bits"] + assert actual["kl_bits"] == baseline["kl_bits"] + assert actual["grad_row"] == baseline["grad_row"], ( + f"{label} gradient differs from the single-rank baseline" + ) + + @_requires_gpus(4) + def test_pure_axes_agree_with_each_other(self): + # Transitively implied by both matching the baseline; kept because a + # direct TP-vs-DP comparison names the culprit when a shared drift moves + # both away from the baseline at once. + tp4 = _consensus(_run_mesh(4, 1)) + dp4 = _consensus(_run_mesh(1, 4)) + assert tp4["per_sequence_policy_bits"] == dp4["per_sequence_policy_bits"] + assert tp4["loss_bits"] == dp4["loss_bits"] + assert tp4["grad_row"] == dp4["grad_row"] + + +@_requires_gpus(2) +class TestMeshGuards: + def test_preflight_rejects_a_dp_rank_that_disagrees(self): + payloads = _run_mesh(1, 2, scenario="preflight_dp_mismatch") + assert all(not item.get("ok") for item in payloads), ( + "every rank must abort when one of them declares a different objective" + ) + assert any("preflight" in item.get("error", "") for item in payloads) + + def test_preflight_rejects_a_tp_rank_that_disagrees(self): + # beta is not part of the logprob contract, so the logprob path's own TP + # preflight cannot see this; only the loss preflight's TP-axis check can. + # Without it two TP siblings would compute different losses for one + # sharded model and nothing would notice. + payloads = _run_mesh(2, 1, scenario="preflight_tp_mismatch") + assert all(not item.get("ok") for item in payloads) + assert any("TP axis" in item.get("error", "") for item in payloads) diff --git a/tests/test_grpo_loss_contract.py b/tests/test_grpo_loss_contract.py new file mode 100644 index 00000000..2a17bce4 --- /dev/null +++ b/tests/test_grpo_loss_contract.py @@ -0,0 +1,392 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Unit tests for the WS2 deterministic GRPO loss contract. + +These are pure-Python contract checks: no tensors, no collectives, no GPU. +The distributed behaviour they describe is exercised in +``tests/test_distributed_grpo_loss.py``. +""" + +from __future__ import annotations + +import json + +import pytest + +from rl_engine.kernels.logprob_contract import ( + DeterminismScope, + LogprobContract, + LogprobDType, + LogprobRole, + MaskSpec, + ReductionSpec, + ShardingSpec, +) +from rl_engine.kernels.loss_contract import ( + AdvantageNormalizer, + AdvantageSpec, + ClipSpec, + GRPOLossContract, + KLEstimator, + LossBackendCapability, + LossContractError, + LossReductionSpec, + LossShardingSpec, + ObjectiveSpec, + TokenNormalizer, +) + +# Global batch geometry shared by every fixture below: 8 sequences of 8 token +# slots, in two equal advantage groups. Tests that care about variable or +# straddling groups override group_boundaries explicitly. +NUM_SEQUENCES = 8 +PADDED_SEQ_LEN = 8 +GROUP_BOUNDARIES = (0, 4, NUM_SEQUENCES) +REAL_VOCAB = 30 +PADDED_VOCAB = 32 + + +def _dp_bounds(dp: int) -> tuple[tuple[int, int], ...]: + """Contiguous sequence partition in DP-rank order.""" + + seqs = NUM_SEQUENCES // dp + return tuple((d * seqs, (d + 1) * seqs) for d in range(dp)) + + +def _logprob_contract(num_tokens: int, **overrides) -> LogprobContract: + kwargs = { + "role": LogprobRole.TRAIN, + "dtype": LogprobDType.FP32, + "mask": MaskSpec(num_tokens=num_tokens, active_mask=(True,) * num_tokens), + "sharding": ShardingSpec( + tp_rank=0, + tp_world_size=1, + vocab_shard_bounds=((0, PADDED_VOCAB),), + real_vocab_size=REAL_VOCAB, + padded_vocab_size=PADDED_VOCAB, + ), + "reduction": ReductionSpec(), + } + kwargs.update(overrides) + return LogprobContract(**kwargs) + + +def _sharding(*, dp_rank: int = 0, dp: int = 1, **overrides) -> LossShardingSpec: + kwargs = { + "dp_rank": dp_rank, + "dp_world_size": dp, + "num_sequences": NUM_SEQUENCES, + "padded_seq_len": PADDED_SEQ_LEN, + "sequence_shard_bounds": _dp_bounds(dp), + "group_boundaries": GROUP_BOUNDARIES, + } + kwargs.update(overrides) + return LossShardingSpec(**kwargs) + + +def _contract(*, dp_rank: int = 0, dp: int = 1, **overrides) -> GRPOLossContract: + sharding = overrides.pop("sharding", _sharding(dp_rank=dp_rank, dp=dp)) + kwargs = { + "logprob": _logprob_contract(sharding.local_num_token_slots), + "sharding": sharding, + "objective": ObjectiveSpec(), + "reduction": LossReductionSpec(), + } + kwargs.update(overrides) + return GRPOLossContract(**kwargs) + + +class TestSequenceOwnership: + def test_single_rank_owns_every_sequence(self): + sharding = _sharding() + assert sharding.local_sequence_start == 0 + assert sharding.local_sequence_end == NUM_SEQUENCES + assert sharding.local_num_token_slots == NUM_SEQUENCES * PADDED_SEQ_LEN + assert sharding.num_groups == 2 + assert sharding.group_sizes == (4, 4) + + @pytest.mark.parametrize("dp", [1, 2, 4, 8]) + def test_dp_shapes_partition_the_batch(self, dp): + total = 0 + for dp_rank in range(dp): + sharding = _sharding(dp_rank=dp_rank, dp=dp) + assert sharding.local_num_sequences == NUM_SEQUENCES // dp + total += sharding.local_num_token_slots + assert total == NUM_SEQUENCES * PADDED_SEQ_LEN + + def test_bounds_must_be_contiguous_in_rank_order(self): + with pytest.raises(LossContractError, match="contiguous"): + _sharding(dp=2, sequence_shard_bounds=((0, 4), (5, 8))) + + def test_bounds_must_cover_every_sequence(self): + with pytest.raises(LossContractError, match="cover num_sequences"): + _sharding(dp=2, sequence_shard_bounds=((0, 3), (3, 6))) + + def test_bounds_count_must_match_dp_world_size(self): + with pytest.raises( + LossContractError, match="exactly one \\(start, end\\) pair per DP rank" + ): + _sharding(dp=2, sequence_shard_bounds=((0, NUM_SEQUENCES),)) + + def test_empty_shard_rejected(self): + with pytest.raises(LossContractError, match="end > start"): + _sharding(dp=2, sequence_shard_bounds=((0, 0), (0, NUM_SEQUENCES))) + + def test_context_parallelism_is_rejected(self): + # CP splits a sequence's tokens across ranks, which this contract does + # not model; it must fail loudly rather than silently drop the rest. + with pytest.raises(LossContractError, match="cp_world_size=2 is unsupported"): + _sharding(cp_world_size=2) + + def test_cp_rank_must_be_zero(self): + with pytest.raises(LossContractError, match="cp_rank must be 0"): + _sharding(cp_rank=1) + + +class TestGroupBoundaries: + @pytest.mark.parametrize( + ("boundaries", "match"), + [ + ((1, NUM_SEQUENCES), "must start at 0"), + ((0, 3), "must start at 0"), + ((0, 2, 2, NUM_SEQUENCES), "strictly increasing"), + ((0,), "at least 2 entries"), + ], + ) + def test_malformed_boundaries_rejected(self, boundaries, match): + with pytest.raises(LossContractError, match=match): + _sharding(group_boundaries=boundaries) + + def test_variable_group_sizes_accepted(self): + sharding = _sharding(group_boundaries=(0, 7, NUM_SEQUENCES)) + assert sharding.group_sizes == (7, 1) + + def test_groups_may_straddle_dp_shards(self): + # A group split at 3 crosses the DP=2 shard boundary at 4, so no rank + # owns that group alone. The contract permits it; that is why the + # operator replicates advantages instead of merging partial statistics. + sharding = _sharding(dp=2, dp_rank=0, group_boundaries=(0, 3, NUM_SEQUENCES)) + assert sharding.sequence_shard_bounds == ((0, 4), (4, NUM_SEQUENCES)) + assert sharding.group_sizes == (3, 5) + + def test_population_std_rejects_singleton_groups(self): + # A one-sequence group has zero population variance, so its advantage + # would silently collapse to zero rather than fail. + with pytest.raises(LossContractError, match="at least 2 sequences per group"): + _contract( + sharding=_sharding(group_boundaries=(0, 7, NUM_SEQUENCES)), + objective=ObjectiveSpec(), + ) + + def test_mean_only_allows_singleton_groups(self): + contract = _contract( + sharding=_sharding(group_boundaries=(0, 7, NUM_SEQUENCES)), + objective=ObjectiveSpec( + advantage=AdvantageSpec(normalizer=AdvantageNormalizer.MEAN_ONLY) + ), + ) + assert contract.sharding.group_sizes == (7, 1) + + +class TestSpecValidation: + def test_accumulation_must_be_fp32(self): + with pytest.raises(LossContractError, match="must be fp32"): + LossReductionSpec(acc_dtype=LogprobDType.BF16) + + def test_fixed_constant_normalizer_requires_its_constant(self): + with pytest.raises(LossContractError, match="fixed_normalizer_constant"): + LossReductionSpec(token_normalizer=TokenNormalizer.FIXED_CONSTANT) + + def test_constant_rejected_for_other_normalizers(self): + with pytest.raises(LossContractError, match="only meaningful for"): + LossReductionSpec(fixed_normalizer_constant=32) + + def test_fixed_constant_normalizer_accepts_its_constant(self): + spec = LossReductionSpec( + token_normalizer=TokenNormalizer.FIXED_CONSTANT, fixed_normalizer_constant=32 + ) + assert spec.fixed_normalizer_constant == 32 + + def test_lower_clip_bound_must_stay_positive(self): + with pytest.raises(LossContractError, match="must be smaller than 1.0"): + ClipSpec(clip_eps_low=1.0) + + def test_asymmetric_clip_bounds(self): + clip = ClipSpec(clip_eps_low=0.2, clip_eps_high=0.28) + assert clip.lower_bound == pytest.approx(0.8) + assert clip.upper_bound == pytest.approx(1.28) + + def test_std_eps_must_be_positive(self): + with pytest.raises(LossContractError, match="strictly positive"): + AdvantageSpec(std_eps=0.0) + + def test_negative_beta_rejected(self): + with pytest.raises(LossContractError, match="non-negative"): + ObjectiveSpec(beta=-0.01) + + def test_uses_reference_model_tracks_beta(self): + assert not ObjectiveSpec(beta=0.0).uses_reference_model + assert ObjectiveSpec(beta=0.04).uses_reference_model + + +class TestContractCoherence: + def test_logprob_token_count_must_match_owned_slots(self): + sharding = _sharding(dp=2, dp_rank=0) + with pytest.raises(LossContractError, match="token slots this rank owns"): + GRPOLossContract( + logprob=_logprob_contract(sharding.local_num_token_slots + 1), + sharding=sharding, + ) + + def test_determinism_scope_must_agree_with_logprob_path(self): + tokens = _sharding().local_num_token_slots + with pytest.raises(LossContractError, match="stronger or weaker determinism scope"): + GRPOLossContract( + logprob=_logprob_contract( + tokens, + reduction=ReductionSpec(determinism_scope=DeterminismScope.FIXED_TOPOLOGY), + ), + sharding=_sharding(), + ) + + def test_global_token_slots(self): + assert _contract().global_token_slots == NUM_SEQUENCES * PADDED_SEQ_LEN + + +class TestFingerprint: + @pytest.mark.parametrize("dp", [2, 4, 8]) + def test_every_dp_rank_agrees(self, dp): + # This is the property the distributed preflight relies on. Each rank + # holds a different slice of sequences, so their nested logprob masks + # genuinely differ -- the fingerprint must still match. + fingerprints = { + _contract(dp_rank=rank, dp=dp).cross_rank_fingerprint() for rank in range(dp) + } + assert len(fingerprints) == 1 + + def test_dp_degree_changes_the_fingerprint(self): + # A different partition is a different logical invocation, so a rank + # that joined the wrong one must be caught rather than merged with. + assert _contract(dp=1).cross_rank_fingerprint() != _contract(dp=2).cross_rank_fingerprint() + + @pytest.mark.parametrize( + "overrides", + [ + { + "reduction": LossReductionSpec( + token_normalizer=TokenNormalizer.PER_SEQUENCE_THEN_MEAN + ) + }, + {"objective": ObjectiveSpec(beta=0.04)}, + {"objective": ObjectiveSpec(clip=ClipSpec(clip_eps_high=0.28))}, + {"objective": ObjectiveSpec(kl_estimator=KLEstimator.K1_LOG_RATIO)}, + { + "objective": ObjectiveSpec( + advantage=AdvantageSpec(normalizer=AdvantageNormalizer.MEAN_ONLY) + ) + }, + {"sharding": _sharding(group_boundaries=(0, 3, NUM_SEQUENCES))}, + ], + ids=["normalizer", "beta", "clip", "kl", "advantage", "groups"], + ) + def test_numerical_identity_changes_the_fingerprint(self, overrides): + assert ( + _contract().cross_rank_fingerprint() != _contract(**overrides).cross_rank_fingerprint() + ) + + def test_to_dict_is_json_serializable_and_stable(self): + contract = _contract() + first = json.dumps(contract.to_dict(), sort_keys=True) + second = json.dumps(contract.to_dict(), sort_keys=True) + assert first == second + payload = contract.to_dict() + assert payload["semantic_operator"] == "grpo_loss" + assert payload["reduction"]["cp_is_merge_axis"] is False + assert payload["reduction"]["dp_is_merge_axis"] is True + assert payload["logprob"]["reduction"]["cp_is_merge_axis"] is False + + +def _capability(**overrides) -> LossBackendCapability: + kwargs = { + "backend_id": "test-loss-backend", + "token_normalizers": frozenset({TokenNormalizer.GLOBAL_ACTIVE_TOKENS}), + "kl_estimators": frozenset({KLEstimator.K3_UNBIASED}), + "advantage_normalizers": frozenset({AdvantageNormalizer.MEAN_STD_POPULATION}), + "determinism_scopes": frozenset({DeterminismScope.CROSS_TP_BITWISE}), + "implementation_kind": "reference", + } + kwargs.update(overrides) + return LossBackendCapability(**kwargs) + + +class TestBackendCapability: + def test_matching_capability_supports_contract(self): + assert _capability().supports(_contract()) + + def test_backend_id_may_not_shadow_a_dispatch_policy(self): + with pytest.raises(LossContractError, match="reserved dispatch policy"): + _capability(backend_id="reference") + + def test_unsupported_normalizer_is_reported(self): + contract = _contract( + reduction=LossReductionSpec(token_normalizer=TokenNormalizer.PER_SEQUENCE_THEN_MEAN) + ) + reasons = _capability().incompatibilities(contract) + assert any("token_normalizer" in reason for reason in reasons) + + def test_unsupported_dp_degree_is_reported(self): + capability = _capability(dp_world_sizes=(1,)) + reasons = capability.incompatibilities(_contract(dp=2)) + assert any("DP=2" in reason for reason in reasons) + + def test_variable_group_sizes_gated(self): + contract = _contract( + sharding=_sharding(group_boundaries=(0, 7, NUM_SEQUENCES)), + objective=ObjectiveSpec( + advantage=AdvantageSpec(normalizer=AdvantageNormalizer.MEAN_ONLY) + ), + ) + capability = _capability(advantage_normalizers=frozenset({AdvantageNormalizer.MEAN_ONLY})) + assert any( + "variable advantage group sizes" in reason + for reason in capability.incompatibilities(contract) + ) + + def test_variable_group_sizes_allowed_when_declared(self): + contract = _contract( + sharding=_sharding(group_boundaries=(0, 7, NUM_SEQUENCES)), + objective=ObjectiveSpec( + advantage=AdvantageSpec(normalizer=AdvantageNormalizer.MEAN_ONLY) + ), + ) + capability = _capability( + advantage_normalizers=frozenset({AdvantageNormalizer.MEAN_ONLY}), + supports_variable_group_sizes=True, + ) + assert capability.supports(contract) + + def test_asymmetric_clip_gated(self): + contract = _contract(objective=ObjectiveSpec(clip=ClipSpec(clip_eps_high=0.28))) + assert any( + "asymmetric ratio clipping" in reason + for reason in _capability().incompatibilities(contract) + ) + assert _capability(supports_asymmetric_clip=True).supports(contract) + + def test_every_incompatibility_is_reported_at_once(self): + contract = _contract( + dp=2, + reduction=LossReductionSpec(token_normalizer=TokenNormalizer.PER_SEQUENCE_THEN_MEAN), + objective=ObjectiveSpec( + kl_estimator=KLEstimator.K1_LOG_RATIO, + clip=ClipSpec(clip_eps_high=0.28), + ), + ) + reasons = _capability(dp_world_sizes=(1,)).incompatibilities(contract) + assert len(reasons) >= 4 + + def test_to_dict_round_trips_declared_flags(self): + payload = _capability(dp_world_sizes=(1, 2)).to_dict() + assert payload["dp_world_sizes"] == [1, 2] + assert payload["implementation_kind"] == "reference" From c2c99c1ab9a63ce324b88608b0dd6b9e29b7216c Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Thu, 20 Aug 2026 00:35:35 +0800 Subject: [PATCH 29/48] feat(ws2): record cross-topology logprob fingerprints --- .../testing/distributed_logprob_comparison.py | 30 +++++++++++++++++++ tests/test_distributed_logprob_comparison.py | 5 ++++ 2 files changed, 35 insertions(+) diff --git a/rl_engine/testing/distributed_logprob_comparison.py b/rl_engine/testing/distributed_logprob_comparison.py index d3884194..3e37c88b 100644 --- a/rl_engine/testing/distributed_logprob_comparison.py +++ b/rl_engine/testing/distributed_logprob_comparison.py @@ -175,6 +175,7 @@ class DistributedLogprobReport: environment: dict[str, Any] ranks: tuple[RankLogprobReport, ...] aggregate: dict[str, DriftDetail] + bitwise_fingerprints: dict[str, Any] passed: bool def to_dict(self) -> dict[str, Any]: @@ -590,6 +591,33 @@ def _aggregate_payloads( } +def _tensor_sha256(tensor: torch.Tensor) -> str: + """Hash the exact CPU tensor bytes for cross-topology comparisons.""" + + data = tensor.detach().cpu().contiguous().numpy().tobytes() + return hashlib.sha256(data).hexdigest() + + +def _aggregate_bitwise_fingerprints( + case: DistributedLogprobCase, + payloads: Sequence[_RankPayload], +) -> dict[str, Any]: + representatives = sorted( + (payload for payload in payloads if payload.report.tp_rank == 0), + key=lambda payload: payload.report.cp_rank, + ) + if len(representatives) != case.cp_world_size: + raise RuntimeError("missing one or more CP representatives in rank reports") + candidate_logp = torch.cat([payload.candidate_logp for payload in representatives]) + candidate_lse = torch.cat([payload.candidate_lse for payload in representatives]) + return { + "candidate_logp_sha256": _tensor_sha256(candidate_logp), + "candidate_lse_sha256": _tensor_sha256(candidate_lse), + "dtype": str(candidate_logp.dtype).replace("torch.", ""), + "shape": list(candidate_logp.shape), + } + + def _create_tp_group(case: DistributedLogprobCase, topology: RankTopology) -> Any: if case.world_size == 1: return None @@ -656,6 +684,7 @@ def run_distributed_logprob_case( report = None if rank == 0: aggregate = _aggregate_payloads(case, payloads) + bitwise_fingerprints = _aggregate_bitwise_fingerprints(case, payloads) rank_reports = tuple( payload.report for payload in sorted(payloads, key=lambda item: item.report.global_rank) @@ -697,6 +726,7 @@ def run_distributed_logprob_case( }, ranks=rank_reports, aggregate=aggregate, + bitwise_fingerprints=bitwise_fingerprints, passed=( materialization_consistent and all(rank_report.passed for rank_report in rank_reports) diff --git a/tests/test_distributed_logprob_comparison.py b/tests/test_distributed_logprob_comparison.py index 8309e469..8bb9d4ad 100644 --- a/tests/test_distributed_logprob_comparison.py +++ b/tests/test_distributed_logprob_comparison.py @@ -175,6 +175,11 @@ def test_tp1_cpu_case_writes_116_compatible_artifact(tmp_path, monkeypatch): assert payload["ranks"][0]["dp_world_size"] == 1 assert payload["environment"]["materialization"]["consistent"] is True assert payload["aggregate"]["dlogp"]["worst_target_id"] is not None + fingerprints = payload["bitwise_fingerprints"] + assert len(fingerprints["candidate_logp_sha256"]) == 64 + assert len(fingerprints["candidate_lse_sha256"]) == 64 + assert fingerprints["dtype"] == "float32" + assert fingerprints["shape"] == [4] assert payload["launch_command"].startswith("torchrun --standalone") From 842867672a841cd5752eb8373af77a7d8e934ed8 Mon Sep 17 00:00:00 2001 From: lamentropetion <3051000145@qq.com> Date: Fri, 21 Aug 2026 17:56:57 +0800 Subject: [PATCH 30/48] feat(ws2): add Vime logprob provider for CP metadata --- rl_engine/integrations/__init__.py | 5 + rl_engine/integrations/vime/__init__.py | 9 + rl_engine/integrations/vime/logp.py | 229 ++++++++++++++++++ .../ops/pytorch/loss/vocab_parallel_logp.py | 138 ++++++++++- tests/test_vime_logprob_provider.py | 93 +++++++ tests/test_vocab_parallel_logp.py | 23 ++ 6 files changed, 486 insertions(+), 11 deletions(-) create mode 100644 rl_engine/integrations/__init__.py create mode 100644 rl_engine/integrations/vime/__init__.py create mode 100644 rl_engine/integrations/vime/logp.py create mode 100644 tests/test_vime_logprob_provider.py diff --git a/rl_engine/integrations/__init__.py b/rl_engine/integrations/__init__.py new file mode 100644 index 00000000..29bcc68f --- /dev/null +++ b/rl_engine/integrations/__init__.py @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Optional framework adapters owned by RL-Kernel.""" + diff --git a/rl_engine/integrations/vime/__init__.py b/rl_engine/integrations/vime/__init__.py new file mode 100644 index 00000000..44470314 --- /dev/null +++ b/rl_engine/integrations/vime/__init__.py @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Vime adapter entry points without a Vime runtime dependency.""" + +from .logp import ProviderResult, SelectedLogprobProviderUnavailable, provider + +__all__ = ["ProviderResult", "SelectedLogprobProviderUnavailable", "provider"] + diff --git a/rl_engine/integrations/vime/logp.py b/rl_engine/integrations/vime/logp.py new file mode 100644 index 00000000..6a47e725 --- /dev/null +++ b/rl_engine/integrations/vime/logp.py @@ -0,0 +1,229 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Runtime selected-logprob provider for Vime's Megatron backend. + +The adapter intentionally accepts and returns structural objects: RL-Kernel +never imports Vime. Vime remains responsible for constructing locally owned +CP token rows and response masks; this provider owns only the TP-vocabulary +reduction. CP rank and layout are recorded and validated as row ownership +metadata, never passed to the numerical merge. +""" + +from __future__ import annotations + +import os +from collections.abc import Mapping +from dataclasses import dataclass +from typing import Any + +import torch + +from rl_engine.kernels.logprob_contract import ( + LogprobContract, + LogprobDType, + LogprobRole, + MaskSpec, + ReductionSpec, + ShardingSpec, +) +from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import ( + BACKEND_ID, + DEFAULT_NUM_VOCAB_TILES, +) +from rl_engine.kernels.registry import kernel_registry + + +class SelectedLogprobProviderUnavailable(RuntimeError): + """Request Vime's native provider fallback in ``auto`` mode. + + Vime recognizes the marker instead of importing this class, which keeps + the dependency direction from Vime to RL-Kernel at runtime only. + """ + + selected_logprob_provider_unavailable = True + + +@dataclass(frozen=True) +class ProviderResult: + """Structural result understood by the Vime provider boundary.""" + + selected_logprobs: torch.Tensor + entropy: torch.Tensor | None + backend_id: str + contract_id: str + provenance: Mapping[str, Any] + + +def _as_positive_int(value: Any, name: str) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + raise SelectedLogprobProviderUnavailable(f"{name} must be a positive integer; got {value!r}") + return value + + +def _metadata(request: Any) -> Mapping[str, Any]: + value = getattr(request, "metadata", None) + if not isinstance(value, Mapping): + raise SelectedLogprobProviderUnavailable("request.metadata must provide vocab-parallel metadata") + return value + + +def _request_tensor(request: Any, name: str) -> torch.Tensor: + value = getattr(request, name, None) + if not isinstance(value, torch.Tensor): + raise SelectedLogprobProviderUnavailable(f"request.{name} must be a torch.Tensor") + return value + + +def _tp_coordinates(tp_group: Any) -> tuple[int, int]: + if tp_group is not None and hasattr(tp_group, "rank") and hasattr(tp_group, "size"): + return int(tp_group.rank()), int(tp_group.size()) + + import torch.distributed as dist + + if dist.is_available() and dist.is_initialized(): + return dist.get_rank(group=tp_group), dist.get_world_size(group=tp_group) + return 0, 1 + + +def _tile_count(metadata: Mapping[str, Any], padded_vocab_size: int) -> int: + configured = metadata.get("num_vocab_tiles", os.getenv("RL_KERNEL_LOGPROB_NUM_VOCAB_TILES")) + if configured is None or configured == "": + configured = DEFAULT_NUM_VOCAB_TILES + try: + tiles = int(configured) + except (TypeError, ValueError) as exc: + raise SelectedLogprobProviderUnavailable( + f"num_vocab_tiles must be an integer; got {configured!r}" + ) from exc + if tiles <= 0 or padded_vocab_size % tiles: + raise SelectedLogprobProviderUnavailable( + f"num_vocab_tiles={tiles} must divide padded_vocab_size={padded_vocab_size}" + ) + return tiles + + +def _contract_for_request(request: Any) -> tuple[LogprobContract, int]: + logits = _request_tensor(request, "logits") + targets = _request_tensor(request, "target_ids") + metadata = _metadata(request) + if logits.ndim != 2 or targets.shape != (logits.shape[0],): + raise SelectedLogprobProviderUnavailable("request must contain local [T, V] logits and aligned [T] targets") + if logits.dtype not in (torch.bfloat16, torch.float16, torch.float32): + raise SelectedLogprobProviderUnavailable(f"unsupported logit dtype {logits.dtype}") + if targets.device != logits.device: + raise SelectedLogprobProviderUnavailable("target_ids must share the local logits device") + + cp = getattr(request, "context_parallel", None) + cp_world_size = _as_positive_int(getattr(cp, "world_size", None), "context_parallel.world_size") + cp_rank = getattr(cp, "rank", None) + if isinstance(cp_rank, bool) or not isinstance(cp_rank, int) or not 0 <= cp_rank < cp_world_size: + raise SelectedLogprobProviderUnavailable( + f"context_parallel.rank={cp_rank!r} is invalid for CP={cp_world_size}" + ) + if getattr(cp, "layout", None) not in ({"single"} if cp_world_size == 1 else {"zigzag", "allgather"}): + raise SelectedLogprobProviderUnavailable("context_parallel layout does not describe local CP token ownership") + + tp_rank, tp_world_size = _tp_coordinates(getattr(request, "tensor_parallel_group", None)) + declared_tp_rank = metadata.get("tp_rank") + declared_tp_world_size = metadata.get("tp_world_size") + if declared_tp_rank is not None and declared_tp_rank != tp_rank: + raise SelectedLogprobProviderUnavailable( + f"metadata tp_rank={declared_tp_rank} disagrees with TP group rank={tp_rank}" + ) + if declared_tp_world_size is not None and declared_tp_world_size != tp_world_size: + raise SelectedLogprobProviderUnavailable( + f"metadata tp_world_size={declared_tp_world_size} disagrees with TP group size={tp_world_size}" + ) + + real_vocab_size = _as_positive_int(metadata.get("real_vocab_size"), "real_vocab_size") + padded_vocab_size = _as_positive_int(metadata.get("padded_vocab_size"), "padded_vocab_size") + if logits.shape[1] * tp_world_size != padded_vocab_size: + raise SelectedLogprobProviderUnavailable( + "local vocab width and TP group do not cover padded_vocab_size exactly: " + f"{logits.shape[1]} * {tp_world_size} != {padded_vocab_size}" + ) + if real_vocab_size > padded_vocab_size: + raise SelectedLogprobProviderUnavailable("real_vocab_size must not exceed padded_vocab_size") + + bounds = tuple( + (rank * logits.shape[1], (rank + 1) * logits.shape[1]) for rank in range(tp_world_size) + ) + active_mask = (True,) * logits.shape[0] + contract = LogprobContract( + role=LogprobRole.TRAIN, + dtype={ + torch.bfloat16: LogprobDType.BF16, + torch.float16: LogprobDType.FP16, + torch.float32: LogprobDType.FP32, + }[logits.dtype], + mask=MaskSpec(num_tokens=logits.shape[0], active_mask=active_mask), + sharding=ShardingSpec( + tp_rank=tp_rank, + tp_world_size=tp_world_size, + vocab_shard_bounds=bounds, + real_vocab_size=real_vocab_size, + padded_vocab_size=padded_vocab_size, + cp_rank=cp_rank, + cp_world_size=cp_world_size, + ), + reduction=ReductionSpec(), + ) + return contract, _tile_count(metadata, padded_vocab_size) + + +def provider(request: Any) -> ProviderResult: + """Compute Vime selected logprobs on the explicit WS2 TP/CP contract. + + Top-p replay is deliberately unavailable until it has a separately + validated fixed-order mask contract. In Vime ``auto`` mode this signals + native execution; in ``strict`` mode it fails instead of changing sampled + distribution semantics. + """ + + if getattr(request, "log_prob_keep_mask", None) is not None: + raise SelectedLogprobProviderUnavailable( + "RL-Kernel WS2 logprob does not yet materialize Vime top-p replay masks" + ) + + contract, num_vocab_tiles = _contract_for_request(request) + dispatch = kernel_registry.get_logprob_op(contract, requested_backend=BACKEND_ID) + if dispatch.provenance["actual_backend"] != BACKEND_ID or dispatch.provenance["fallback"]: + raise RuntimeError("explicit WS2 backend dispatch changed during materialization") + if getattr(request, "with_entropy", False): + selected_logp, _lse, entropy = dispatch.op.apply_with_entropy( + request.logits, + request.target_ids, + contract=contract, + tp_group=getattr(request, "tensor_parallel_group", None), + num_vocab_tiles=num_vocab_tiles, + with_entropy_grad=bool(getattr(request, "with_entropy_grad", False)), + ) + else: + selected_logp, _lse = dispatch.op( + request.logits, + request.target_ids, + contract=contract, + tp_group=getattr(request, "tensor_parallel_group", None), + num_vocab_tiles=num_vocab_tiles, + ) + entropy = None + provenance = dict(dispatch.provenance) + provenance["cp_row_ownership"] = { + "cp_rank": contract.sharding.cp_rank, + "cp_world_size": contract.sharding.cp_world_size, + "layout": getattr(request.context_parallel, "layout"), + "local_token_rows": int(request.logits.shape[0]), + "cp_is_merge_axis": False, + } + provenance["num_vocab_tiles"] = num_vocab_tiles + return ProviderResult( + selected_logprobs=selected_logp.unsqueeze(-1), + entropy=entropy, + backend_id=dispatch.capability.backend_id, + contract_id=contract.cross_rank_fingerprint(), + provenance=provenance, + ) + + +__all__ = ["ProviderResult", "SelectedLogprobProviderUnavailable", "provider"] diff --git a/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py b/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py index 06eded1c..9d6cc462 100644 --- a/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py +++ b/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py @@ -267,6 +267,28 @@ def _gather_target_logit( return stacked[owner, rows] +def _gather_entropy_partials( + local_entropy: torch.Tensor, + contract: LogprobContract, + tp_group: Any, +) -> torch.Tensor: + """Merge per-shard entropy contributions in TP-rank order. + + Entropy is not part of the WS2 selected-logprob contract, but the Vime + adapter needs it for the existing training loss surface. The collective + transports independent per-shard contributions; every TP rank performs + the same explicit rank-ordered sum afterwards. + """ + + if contract.sharding.tp_world_size == 1: + return local_entropy + + dist = _require_distributed_initialized() + gathered = [torch.empty_like(local_entropy) for _ in range(contract.sharding.tp_world_size)] + dist.all_gather(gathered, local_entropy.contiguous(), group=tp_group) + return torch.stack(gathered, dim=0).sum(dim=0) + + def _merge_tile_partials(m_all: torch.Tensor, s_all: torch.Tensor) -> torch.Tensor: """Fixed-order (max, sumexp) merge over [n, num_vocab_tiles].""" @@ -280,7 +302,17 @@ def _merge_tile_partials(m_all: torch.Tensor, s_all: torch.Tensor) -> torch.Tens class _VocabParallelLogprobFunction(torch.autograd.Function): @staticmethod - def forward(ctx, local_logits, target_1d, active_mask, contract, tp_group, tile): + def forward( + ctx, + local_logits, + target_1d, + active_mask, + contract, + tp_group, + tile, + with_entropy, + with_entropy_grad, + ): z_masked = local_logits.float() sharding = contract.sharding global_ids = torch.arange( @@ -299,19 +331,40 @@ def forward(ctx, local_logits, target_1d, active_mask, contract, tp_group, tile) selected_logp = torch.where(active_mask, target_logit - lse, torch.zeros_like(lse)) - ctx.save_for_backward(z_masked, lse, safe_target, active_mask, padding_cols) + if with_entropy: + finite_row = torch.isfinite(lse) + lse_safe = torch.where(finite_row, lse, torch.zeros_like(lse)) + probabilities = (z_masked - lse_safe.unsqueeze(1)).exp() + probabilities = torch.where(finite_row.unsqueeze(1), probabilities, torch.zeros_like(probabilities)) + finite_logits = torch.isfinite(z_masked) + log_gap = torch.where( + finite_logits, + lse_safe.unsqueeze(1) - z_masked, + torch.zeros_like(z_masked), + ) + local_entropy = (probabilities * log_gap).sum(dim=1) + entropy = _gather_entropy_partials(local_entropy, contract, tp_group) + else: + entropy = local_logits.new_empty((0,), dtype=torch.float32) + + ctx.save_for_backward(z_masked, lse, safe_target, active_mask, padding_cols, entropy) ctx.local_vocab_start = sharding.local_vocab_start ctx.local_vocab_size = sharding.local_vocab_size ctx.input_dtype = local_logits.dtype + ctx.with_entropy_grad = bool(with_entropy and with_entropy_grad) ctx.set_materialize_grads(False) - return selected_logp, lse + if with_entropy and not ctx.with_entropy_grad: + ctx.mark_non_differentiable(entropy) + return selected_logp, lse, entropy @staticmethod - def backward(ctx, grad_logp, grad_lse): - if not ctx.needs_input_grad[0] or (grad_logp is None and grad_lse is None): - return None, None, None, None, None, None + def backward(ctx, grad_logp, grad_lse, grad_entropy): + if not ctx.needs_input_grad[0] or ( + grad_logp is None and grad_lse is None and grad_entropy is None + ): + return None, None, None, None, None, None, None, None - z_masked, lse, safe_target, active_mask, padding_cols = ctx.saved_tensors + z_masked, lse, safe_target, active_mask, padding_cols, entropy = ctx.saved_tensors n, local_vocab = z_masked.shape finite_row = torch.isfinite(lse) lse_safe = torch.where(finite_row, lse, torch.zeros_like(lse)) @@ -332,9 +385,13 @@ def backward(ctx, grad_logp, grad_lse): grad = grad + g_logp.unsqueeze(1) * (onehot - p) if grad_lse is not None: grad = grad + grad_lse.unsqueeze(1) * p + if ctx.with_entropy_grad and grad_entropy is not None: + entropy_input = lse_safe.unsqueeze(1) - z_masked - entropy.unsqueeze(1) + entropy_input = torch.where(torch.isfinite(z_masked), entropy_input, torch.zeros_like(entropy_input)) + grad = grad + grad_entropy.unsqueeze(1) * p * entropy_input if bool(padding_cols.any()): grad = grad.masked_fill(padding_cols.unsqueeze(0), 0.0) - return grad.to(ctx.input_dtype), None, None, None, None, None + return grad.to(ctx.input_dtype), None, None, None, None, None, None, None class VocabParallelLogprobOp: @@ -375,6 +432,58 @@ def apply( num_vocab_tiles: int = DEFAULT_NUM_VOCAB_TILES, validate: bool = True, ) -> tuple[torch.Tensor, torch.Tensor]: + selected_logp, lse, _ = self._apply( + local_logits, + target_ids, + contract=contract, + tp_group=tp_group, + num_vocab_tiles=num_vocab_tiles, + validate=validate, + with_entropy=False, + with_entropy_grad=False, + ) + return selected_logp, lse + + def apply_with_entropy( + self, + local_logits: torch.Tensor, + target_ids: torch.Tensor, + *, + contract: LogprobContract, + tp_group: Any = None, + num_vocab_tiles: int = DEFAULT_NUM_VOCAB_TILES, + validate: bool = True, + with_entropy_grad: bool = True, + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """Return selected logprob, vocabulary LSE, and full-vocabulary entropy. + + The method is intentionally separate from :meth:`apply` so the WS2 + selected-logprob surface remains unchanged for existing callers. + """ + + return self._apply( + local_logits, + target_ids, + contract=contract, + tp_group=tp_group, + num_vocab_tiles=num_vocab_tiles, + validate=validate, + with_entropy=True, + with_entropy_grad=with_entropy_grad, + ) + + def _apply( + self, + local_logits: torch.Tensor, + target_ids: torch.Tensor, + *, + contract: LogprobContract, + tp_group: Any, + num_vocab_tiles: int, + validate: bool, + with_entropy: bool, + with_entropy_grad: bool, + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: if not isinstance(contract, LogprobContract): raise LogprobContractError("contract must be a LogprobContract") tile = _tile_size(contract, num_vocab_tiles) @@ -389,8 +498,15 @@ def apply( if contract.sharding.tp_world_size > 1: _preflight_cross_rank_agreement(contract, tp_group, num_vocab_tiles) - selected_logp, lse = _VocabParallelLogprobFunction.apply( - local_logits, target_1d, active_mask, contract, tp_group, tile + selected_logp, lse, entropy = _VocabParallelLogprobFunction.apply( + local_logits, + target_1d, + active_mask, + contract, + tp_group, + tile, + with_entropy, + with_entropy_grad, ) if validate and bool((~torch.isfinite(lse) & active_mask).any().item()): @@ -398,7 +514,7 @@ def apply( "non-finite logsumexp on an active row: logits over the real " "vocabulary must be finite for every active token" ) - return selected_logp, lse + return selected_logp, lse, entropy __all__ = [ diff --git a/tests/test_vime_logprob_provider.py b/tests/test_vime_logprob_provider.py new file mode 100644 index 00000000..021f4e9f --- /dev/null +++ b/tests/test_vime_logprob_provider.py @@ -0,0 +1,93 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""CPU coverage for the optional Vime WS2 selected-logprob adapter.""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +import torch + +from rl_engine.integrations.vime.logp import SelectedLogprobProviderUnavailable, provider + + +def _request(*, cp_rank: int = 0, with_entropy: bool = False, keep_mask=None): + logits = torch.tensor( + [[0.25, -0.5, 1.0, 0.1, -0.3, 0.6, -0.7, 0.4] for _ in range(3)], + dtype=torch.float32, + requires_grad=True, + ) + return SimpleNamespace( + logits=logits, + target_ids=torch.tensor([2, 5, 0]), + tensor_parallel_group=None, + context_parallel=SimpleNamespace( + world_size=2, + rank=cp_rank, + layout="zigzag", + ), + with_entropy=with_entropy, + with_entropy_grad=with_entropy, + log_prob_keep_mask=keep_mask, + metadata={ + "real_vocab_size": 7, + "padded_vocab_size": 8, + "tp_rank": 0, + "tp_world_size": 1, + "num_vocab_tiles": 4, + }, + ) + + +def test_provider_runs_locally_with_cp2_row_metadata(): + request = _request(cp_rank=1) + + result = provider(request) + reference = torch.log_softmax(request.logits[:, :7], dim=-1)[ + torch.arange(request.logits.size(0)), request.target_ids + ] + + assert result.selected_logprobs.shape == (3, 1) + torch.testing.assert_close(result.selected_logprobs.squeeze(-1), reference) + assert result.backend_id == "pytorch-vocab-parallel-logp-ws2" + assert result.provenance["cp_row_ownership"] == { + "cp_rank": 1, + "cp_world_size": 2, + "layout": "zigzag", + "local_token_rows": 3, + "cp_is_merge_axis": False, + } + + +def test_provider_entropy_preserves_vime_semantics_and_autograd(): + request = _request(with_entropy=True) + + result = provider(request) + reference_logits = request.logits.detach().clone().requires_grad_(True) + log_probs = torch.log_softmax(reference_logits[:, :7], dim=-1) + reference_logp = log_probs[torch.arange(reference_logits.size(0)), request.target_ids] + reference_entropy = -(log_probs.exp() * log_probs).sum(dim=-1) + + torch.testing.assert_close(result.selected_logprobs.squeeze(-1), reference_logp) + torch.testing.assert_close(result.entropy, reference_entropy) + (result.selected_logprobs.sum() + result.entropy.sum()).backward() + (reference_logp.sum() + reference_entropy.sum()).backward() + torch.testing.assert_close(request.logits.grad[:, :7], reference_logits.grad[:, :7]) + assert bool((request.logits.grad[:, 7] == 0).all()) + + +def test_provider_rejects_top_p_replay_without_changing_its_semantics(): + request = _request(keep_mask=torch.ones((3, 8), dtype=torch.bool)) + + with pytest.raises(SelectedLogprobProviderUnavailable, match="top-p replay"): + provider(request) + + +def test_provider_rejects_local_vocab_metadata_that_cannot_describe_tp_ownership(): + request = _request() + request.metadata["padded_vocab_size"] = 16 + + with pytest.raises(SelectedLogprobProviderUnavailable, match="cover padded_vocab_size"): + provider(request) diff --git a/tests/test_vocab_parallel_logp.py b/tests/test_vocab_parallel_logp.py index 87da2ef0..a0dfe5d8 100644 --- a/tests/test_vocab_parallel_logp.py +++ b/tests/test_vocab_parallel_logp.py @@ -257,6 +257,29 @@ def test_inactive_rows_grad_asymmetry(self): logp.sum().backward() assert bool((z.grad[-1] == 0).all()) + def test_entropy_matches_full_vocab_oracle_and_backpropagates(self): + contract = _contract() + logits, targets = _inputs() + x = logits.clone().requires_grad_(True) + logp, _lse, entropy = VocabParallelLogprobOp().apply_with_entropy( + x, + targets, + contract=contract, + num_vocab_tiles=NUM_TILES, + ) + + y = logits.clone().requires_grad_(True) + ref_log_probs = torch.log_softmax(y[:, :REAL_VOCAB].float(), dim=-1) + safe = targets.clamp(0, REAL_VOCAB - 1) + ref_logp = ref_log_probs[torch.arange(NUM_TOKENS), safe] + ref_logp = torch.where(torch.tensor(ACTIVE), ref_logp, torch.zeros_like(ref_logp)) + ref_entropy = -(ref_log_probs.exp() * ref_log_probs).sum(dim=-1) + + torch.testing.assert_close(entropy, ref_entropy) + (logp.sum() + entropy.sum()).backward() + (ref_logp.sum() + ref_entropy.sum()).backward() + torch.testing.assert_close(x.grad, y.grad, atol=2e-5, rtol=2e-5) + def test_dispatch_resolves_reference_and_leaves_legacy_untouched(): registry = KernelRegistry() From 189c22212a660457d999611134c5d804b5f66212 Mon Sep 17 00:00:00 2001 From: lamentropetion <3051000145@qq.com> Date: Fri, 21 Aug 2026 18:04:33 +0800 Subject: [PATCH 31/48] fix(ws2): keep TP entropy merge order explicit --- docs/operators/batch-invariant-logp.md | 23 +++++++++++++++++++ .../ops/pytorch/loss/vocab_parallel_logp.py | 5 +++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/operators/batch-invariant-logp.md b/docs/operators/batch-invariant-logp.md index b060c10f..5bbacaa7 100644 --- a/docs/operators/batch-invariant-logp.md +++ b/docs/operators/batch-invariant-logp.md @@ -83,6 +83,29 @@ op = result.op # rl_engine.kernels.logprob_ logp, lse = op(local_logits, target_ids, contract=contract, tp_group=tp_group) ``` +### Vime CP=2 runtime provider + +The optional Vime adapter is owned by RL-Kernel and can be selected without +patching Megatron or vLLM: + +```text +--selected-logprob-provider rl_engine.integrations.vime.logp.provider +--selected-logprob-provider-mode strict +``` + +Vime passes the local `[T, V_local]` logits, shifted targets, TP subgroup, +and CP row-ownership metadata. The provider builds the same `LogprobContract` +used by the distributed report, dispatches the explicit +`pytorch-vocab-parallel-logp-ws2` backend, and returns selected logp as `[T, 1]`. +When entropy is requested, it uses the same fixed TP-rank order and returns +full-vocabulary entropy for the existing loss surface. CP rank/layout are +recorded in provenance and never participate in the vocabulary LSE merge. + +The provider fails closed for undeclared real/padded vocabulary sizes, TP/CP +metadata mismatches, unsupported top-p replay masks, and backend fallback. +`auto` mode may then use Vime's native path; `strict` mode reports the +configuration error. This adapter does not import Vime. + ## Benchmarks `benchmarks/benchmark_batch_invariant_logp.py` compares Native, Triton, and the diff --git a/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py b/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py index 9d6cc462..0de98102 100644 --- a/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py +++ b/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py @@ -286,7 +286,10 @@ def _gather_entropy_partials( dist = _require_distributed_initialized() gathered = [torch.empty_like(local_entropy) for _ in range(contract.sharding.tp_world_size)] dist.all_gather(gathered, local_entropy.contiguous(), group=tp_group) - return torch.stack(gathered, dim=0).sum(dim=0) + merged = gathered[0].clone() + for partial in gathered[1:]: + merged = merged + partial + return merged def _merge_tile_partials(m_all: torch.Tensor, s_all: torch.Tensor) -> torch.Tensor: From 5f8d65631f5e899882bb6e5d04a984b311c44fc5 Mon Sep 17 00:00:00 2001 From: lamentropetion <3051000145@qq.com> Date: Fri, 21 Aug 2026 18:52:07 +0800 Subject: [PATCH 32/48] feat(vime): add Qwen3 TP2 CP2 validation example --- examples/vime_qwen3_8b_tp2_cp2/README.md | 69 +++++ .../qwen3_8b_tp2_cp2.json | 43 +++ examples/vime_qwen3_8b_tp2_cp2/run.py | 261 ++++++++++++++++++ rl_engine/integrations/vime/logp.py | 17 ++ tests/test_vime_qwen3_example.py | 126 +++++++++ 5 files changed, 516 insertions(+) create mode 100644 examples/vime_qwen3_8b_tp2_cp2/README.md create mode 100644 examples/vime_qwen3_8b_tp2_cp2/qwen3_8b_tp2_cp2.json create mode 100644 examples/vime_qwen3_8b_tp2_cp2/run.py create mode 100644 tests/test_vime_qwen3_example.py diff --git a/examples/vime_qwen3_8b_tp2_cp2/README.md b/examples/vime_qwen3_8b_tp2_cp2/README.md new file mode 100644 index 00000000..30d3734a --- /dev/null +++ b/examples/vime_qwen3_8b_tp2_cp2/README.md @@ -0,0 +1,69 @@ +# Vime Qwen3-8B TP=2 CP=2 validation + +This example is the recommended reproducible entry point for the Vime-side +selected-logprob integration. It keeps framework glue in Vime and keeps the +numerical provider, contract, provenance, and report in RL-Kernel. + +The example is deliberately strict: + +- Megatron training uses `TP=2`, `CP=2`, `PP=1`, and four actor ranks. +- vLLM rollout uses processed logprobs and `top_p=1.0`. +- Vime must load `rl_engine.integrations.vime.logp.provider` in `strict` mode. +- A native fallback or a missing provider marker is not reported as a pass. +- Attention and FFN are not declared consistent from configuration alone. They + require executed Megatron and vLLM readbacks, so the report marks them + `unclaimed` until those artifacts are supplied. The readback must use + `rlkernel.operator_runtime_evidence.v1` and report exact-zero comparison + metrics for both sides. + +The Vime companion must be installed or checked out separately. This example +does not modify `vllm-project/vime`. + +## Dry run + +```bash +python examples/vime_qwen3_8b_tp2_cp2/run.py \ + --vime-root /path/to/RL-Align/vime \ + --rl-kernel-root /path/to/RL-Kernel \ + --output reports/qwen3_8b_tp2_cp2.validation.json +``` + +## Execute + +The Vime script expects model/checkpoint/data paths through environment +variables. Override them before adding `--run`: + +```bash +export MODEL_ROOT=/models/Qwen3-8B +export TORCH_DIST_ROOT=/models/Qwen3-8B_torch_dist +export PROMPT_DATA=/data/dapo-math-17k.jsonl +export RL_KERNEL_ROOT=/path/to/RL-Kernel + +python examples/vime_qwen3_8b_tp2_cp2/run.py \ + --vime-root /path/to/RL-Align/vime \ + --rl-kernel-root "$RL_KERNEL_ROOT" \ + --output reports/qwen3_8b_tp2_cp2.validation.json \ + --run +``` + +When the Megatron/vLLM launch also emits the operator readback artifact, pass +it explicitly: + +```bash +python examples/vime_qwen3_8b_tp2_cp2/run.py \ + --vime-root /path/to/RL-Align/vime \ + --rl-kernel-root "$RL_KERNEL_ROOT" \ + --runtime-evidence reports/qwen3_8b_tp2_cp2.runtime-evidence.json \ + --output reports/qwen3_8b_tp2_cp2.validation.json \ + --run +``` + +The evidence file is intentionally post-execution. It must include training +and rollout identities for `attention` and `ffn`, plus `passed: true` and +exact-zero `out`, backward, and (for attention) `LSE` comparison metrics. A +configured backend without this readback remains `unclaimed`. + +The runner writes a JSON report and a sibling combined log. The report records +the exact command, both repository revisions, provider backend identity, strict +fallback status, and the claim boundary. It does not fabricate numerical drift +when the GPU run was not executed. diff --git a/examples/vime_qwen3_8b_tp2_cp2/qwen3_8b_tp2_cp2.json b/examples/vime_qwen3_8b_tp2_cp2/qwen3_8b_tp2_cp2.json new file mode 100644 index 00000000..93332cbb --- /dev/null +++ b/examples/vime_qwen3_8b_tp2_cp2/qwen3_8b_tp2_cp2.json @@ -0,0 +1,43 @@ +{ + "schema_version": "rlkernel.vime_qwen3_8b_tp2_cp2.v1", + "model": "Qwen/Qwen3-8B", + "training": { + "framework": "megatron", + "tensor_model_parallel_size": 2, + "context_parallel_size": 2, + "pipeline_model_parallel_size": 1, + "world_size": 4, + "dtype": "bf16" + }, + "rollout": { + "framework": "vllm", + "top_p": 1.0, + "logprobs_mode": "processed_logprobs" + }, + "selected_logprob_provider": { + "path": "rl_engine.integrations.vime.logp.provider", + "mode": "strict", + "backend_id": "pytorch-vocab-parallel-logp-ws2", + "real_vocab_size": 151936, + "padded_vocab_size": 152064, + "num_vocab_tiles": 64 + }, + "operator_evidence": { + "logp": { + "training": "rl-kernel-provider", + "rollout": "vllm-native-processed-logprobs", + "required_runtime_marker": "Selected-logprob provider active" + }, + "attention": { + "training": "runtime-readback-required", + "rollout": "runtime-readback-required", + "status": "not_claimed_without_Megatron_and_vLLM_readback" + }, + "ffn": { + "training": "runtime-readback-required", + "rollout": "runtime-readback-required", + "status": "not_claimed_without_Megatron_and_vLLM_readback" + } + }, + "vime_script": "scripts/run-qwen3-8B-rlkernel-tp2-cp2.sh" +} diff --git a/examples/vime_qwen3_8b_tp2_cp2/run.py b/examples/vime_qwen3_8b_tp2_cp2/run.py new file mode 100644 index 00000000..4823dd58 --- /dev/null +++ b/examples/vime_qwen3_8b_tp2_cp2/run.py @@ -0,0 +1,261 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Run and archive the Vime Qwen3-8B TP=2/CP=2 validation entry point. + +This is an integration example, not a synthetic pass generator. A dry run +only records the exact launch contract. ``--run`` executes Vime and records +whether the strict RL-Kernel provider was actually observed in the log. The +report deliberately leaves attention/FFN unclaimed until both framework +readbacks are supplied. +""" + +from __future__ import annotations + +import argparse +import json +import os +import subprocess +import sys +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Mapping + + +DEFAULT_CONFIG = Path(__file__).with_name("qwen3_8b_tp2_cp2.json") +PROVIDER_MARKER = "Selected-logprob provider active" +FALLBACK_MARKERS = ("using native path", "fallback=True", "fallback=true") +RUNTIME_EVIDENCE_SCHEMA = "rlkernel.operator_runtime_evidence.v1" +_OPERATOR_METRICS = { + "attention": ("out_max_abs", "lse_max_abs", "dq_max_abs", "dk_max_abs", "dv_max_abs"), + "ffn": ("out_max_abs", "dx_max_abs", "dw_max_abs"), +} + + +def load_config(path: Path) -> dict[str, Any]: + with path.open(encoding="utf-8") as handle: + value = json.load(handle) + if not isinstance(value, dict): + raise ValueError("example config must contain a JSON object") + return value + + +def validate_config(config: Mapping[str, Any]) -> None: + training = config.get("training") + rollout = config.get("rollout") + provider = config.get("selected_logprob_provider") + if not isinstance(training, Mapping) or not isinstance(rollout, Mapping) or not isinstance(provider, Mapping): + raise ValueError("training, rollout, and selected_logprob_provider sections are required") + expected = { + "tensor_model_parallel_size": 2, + "context_parallel_size": 2, + "pipeline_model_parallel_size": 1, + "world_size": 4, + } + for name, value in expected.items(): + if training.get(name) != value: + raise ValueError(f"training.{name} must be {value!r}") + if rollout.get("top_p") != 1.0: + raise ValueError("rollout.top_p must remain 1.0 for the strict provider contract") + if provider.get("mode") != "strict": + raise ValueError("selected_logprob_provider.mode must be strict") + if provider.get("path") != "rl_engine.integrations.vime.logp.provider": + raise ValueError("example must use the RL-Kernel Vime provider") + if provider.get("backend_id") != "pytorch-vocab-parallel-logp-ws2": + raise ValueError("example must pin the WS2 vocab-parallel backend") + + +def load_runtime_evidence(path: Path | None) -> dict[str, Any] | None: + """Load post-execution readback without treating configuration as evidence.""" + + if path is None: + return None + with path.open(encoding="utf-8") as handle: + value = json.load(handle) + if not isinstance(value, dict) or value.get("schema_version") != RUNTIME_EVIDENCE_SCHEMA: + raise ValueError(f"runtime evidence must use schema {RUNTIME_EVIDENCE_SCHEMA!r}") + return value + + +def _operator_evidence_status(evidence: Mapping[str, Any] | None, operator: str) -> str: + if evidence is None: + return "unclaimed" + operators = evidence.get("operators") + item = operators.get(operator) if isinstance(operators, Mapping) else None + if not isinstance(item, Mapping): + return "unclaimed" + training = item.get("training") + rollout = item.get("rollout") + comparison = item.get("comparison") + if not isinstance(training, Mapping) or not isinstance(rollout, Mapping): + return "unclaimed" + if not isinstance(comparison, Mapping) or comparison.get("passed") is not True: + return "failed" + required_identity = ("implementation_id", "backend_id", "contract_id") + if any(not training.get(name) or not rollout.get(name) for name in required_identity): + return "failed" + if training["implementation_id"] != rollout["implementation_id"]: + return "failed" + for metric in _OPERATOR_METRICS[operator]: + value = comparison.get(metric) + if not isinstance(value, (int, float)) or isinstance(value, bool) or value != 0.0: + return "failed" + return "passed" + + +def validate_runtime_evidence(evidence: Mapping[str, Any] | None) -> None: + """Reject malformed evidence before it can affect a report.""" + + if evidence is None: + return + for operator in _OPERATOR_METRICS: + status = _operator_evidence_status(evidence, operator) + if status == "failed": + raise ValueError(f"runtime evidence for {operator} is incomplete or non-zero") + + +def _revision(path: Path) -> str | None: + try: + result = subprocess.run( + ["git", "-C", str(path), "rev-parse", "HEAD"], + check=True, + capture_output=True, + text=True, + ) + except (OSError, subprocess.CalledProcessError): + return None + return result.stdout.strip() or None + + +def build_environment(vime_root: Path, rl_kernel_root: Path) -> dict[str, str]: + env = dict(os.environ) + existing = [str(vime_root), str(rl_kernel_root), "/root/Megatron-LM"] + if env.get("PYTHONPATH"): + existing.append(env["PYTHONPATH"]) + env["PYTHONPATH"] = os.pathsep.join(existing) + env["RL_KERNEL_ROOT"] = str(rl_kernel_root) + env["TP_SIZE"] = "2" + env["CP_SIZE"] = "2" + env["ROLLOUT_TOP_P"] = "1.0" + return env + + +def build_command(config: Mapping[str, Any], vime_root: Path) -> list[str]: + script = vime_root / str(config.get("vime_script", "")) + if not script.is_file(): + raise FileNotFoundError(f"Vime entry script does not exist: {script}") + return ["bash", str(script)] + + +def build_report( + config: Mapping[str, Any], + *, + vime_root: Path, + rl_kernel_root: Path, + command: list[str], + status: str, + returncode: int | None, + log_text: str, + log_path: Path | None, + runtime_evidence: Mapping[str, Any] | None = None, + runtime_evidence_path: Path | None = None, +) -> dict[str, Any]: + provider_active = PROVIDER_MARKER in log_text + fallback_observed = any(marker in log_text for marker in FALLBACK_MARKERS) + strict_provider_passed = status == "passed" and provider_active and not fallback_observed + effective_status = "passed" if strict_provider_passed else ("failed" if status == "passed" else status) + attention_status = _operator_evidence_status(runtime_evidence, "attention") + ffn_status = _operator_evidence_status(runtime_evidence, "ffn") + return { + "schema_version": "rlkernel.vime_validation_report.v1", + "created_at": datetime.now(timezone.utc).isoformat(), + "status": effective_status, + "claim_boundary": { + "qwen3_8b_tp2_cp2_vime_training": strict_provider_passed, + "attention_train_infer_consistency": attention_status, + "ffn_train_infer_consistency": ffn_status, + "reason": ( + "attention and FFN require executed Megatron/vLLM runtime readbacks; " + "the evidence contract accepts only exact-zero comparison metrics" + ), + }, + "config": dict(config), + "topology": config["training"], + "provider": { + "configured_path": config["selected_logprob_provider"]["path"], + "configured_mode": config["selected_logprob_provider"]["mode"], + "backend_id": config["selected_logprob_provider"]["backend_id"], + "active_observed": provider_active, + "fallback_observed": fallback_observed, + }, + "command": command, + "returncode": returncode, + "artifacts": { + "log": None if log_path is None else str(log_path), + "runtime_evidence": None if runtime_evidence_path is None else str(runtime_evidence_path), + }, + "runtime_evidence": None if runtime_evidence is None else dict(runtime_evidence), + "revisions": { + "vime": _revision(vime_root), + "rl_kernel": _revision(rl_kernel_root), + }, + } + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--config", type=Path, default=DEFAULT_CONFIG) + parser.add_argument("--vime-root", type=Path, default=Path(os.environ.get("VIME_ROOT", "."))) + parser.add_argument("--rl-kernel-root", type=Path, default=Path(os.environ.get("RL_KERNEL_ROOT", "."))) + parser.add_argument("--output", type=Path, default=Path("qwen3_8b_tp2_cp2.validation.json")) + parser.add_argument( + "--runtime-evidence", + type=Path, + default=None, + help="post-execution Megatron/vLLM operator readback JSON (strict exact-zero contract)", + ) + parser.add_argument("--run", action="store_true", help="execute the Vime script") + args = parser.parse_args(argv) + + config = load_config(args.config) + validate_config(config) + runtime_evidence = load_runtime_evidence(args.runtime_evidence) + validate_runtime_evidence(runtime_evidence) + vime_root = args.vime_root.resolve() + rl_kernel_root = args.rl_kernel_root.resolve() + command = build_command(config, vime_root) + + status = "not_run" + returncode: int | None = None + log_text = "" + log_path: Path | None = None + if args.run: + args.output.parent.mkdir(parents=True, exist_ok=True) + log_path = args.output.with_suffix(".log") + env = build_environment(vime_root, rl_kernel_root) + with log_path.open("w", encoding="utf-8") as log_handle: + process = subprocess.run(command, cwd=vime_root, env=env, stdout=log_handle, stderr=subprocess.STDOUT) + returncode = process.returncode + log_text = log_path.read_text(encoding="utf-8", errors="replace") + status = "passed" if returncode == 0 else "failed" + + report = build_report( + config, + vime_root=vime_root, + rl_kernel_root=rl_kernel_root, + command=command, + status=status, + returncode=returncode, + log_text=log_text, + log_path=log_path, + runtime_evidence=runtime_evidence, + runtime_evidence_path=args.runtime_evidence, + ) + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2, sort_keys=True)) + return 0 if report["status"] in {"passed", "not_run"} else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/rl_engine/integrations/vime/logp.py b/rl_engine/integrations/vime/logp.py index 6a47e725..db436d69 100644 --- a/rl_engine/integrations/vime/logp.py +++ b/rl_engine/integrations/vime/logp.py @@ -209,6 +209,23 @@ def provider(request: Any) -> ProviderResult: ) entropy = None provenance = dict(dispatch.provenance) + provenance["request"] = { + "logits_shape": list(request.logits.shape), + "logits_dtype": str(request.logits.dtype).replace("torch.", ""), + "target_shape": list(request.target_ids.shape), + "target_dtype": str(request.target_ids.dtype).replace("torch.", ""), + "real_vocab_size": contract.sharding.real_vocab_size, + "padded_vocab_size": contract.sharding.padded_vocab_size, + "tp_rank": contract.sharding.tp_rank, + "tp_world_size": contract.sharding.tp_world_size, + "cp_rank": contract.sharding.cp_rank, + "cp_world_size": contract.sharding.cp_world_size, + } + provenance["execution"] = { + "role": "vime_training_selected_logprob", + "strict_backend": True, + "top_p_replay": False, + } provenance["cp_row_ownership"] = { "cp_rank": contract.sharding.cp_rank, "cp_world_size": contract.sharding.cp_world_size, diff --git a/tests/test_vime_qwen3_example.py b/tests/test_vime_qwen3_example.py new file mode 100644 index 00000000..007d4ba1 --- /dev/null +++ b/tests/test_vime_qwen3_example.py @@ -0,0 +1,126 @@ +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from examples.vime_qwen3_8b_tp2_cp2.run import ( + build_report, + load_config, + validate_config, + validate_runtime_evidence, +) + + +ROOT = Path(__file__).parents[1] +CONFIG = ROOT / "examples" / "vime_qwen3_8b_tp2_cp2" / "qwen3_8b_tp2_cp2.json" + + +def test_qwen3_example_config_is_strict_and_explicit(): + config = load_config(CONFIG) + validate_config(config) + assert config["training"]["tensor_model_parallel_size"] == 2 + assert config["training"]["context_parallel_size"] == 2 + assert config["selected_logprob_provider"]["mode"] == "strict" + + +def test_qwen3_example_report_does_not_claim_unread_back_attention_or_ffn(tmp_path): + config = load_config(CONFIG) + report = build_report( + config, + vime_root=tmp_path / "vime", + rl_kernel_root=tmp_path / "rl-kernel", + command=["bash", "run.sh"], + status="passed", + returncode=0, + log_text="Selected-logprob provider active: backend_id=pytorch-vocab-parallel-logp-ws2", + log_path=tmp_path / "run.log", + ) + assert report["status"] == "passed" + assert report["claim_boundary"]["qwen3_8b_tp2_cp2_vime_training"] is True + assert report["claim_boundary"]["attention_train_infer_consistency"] == "unclaimed" + assert report["claim_boundary"]["ffn_train_infer_consistency"] == "unclaimed" + assert report["provider"]["fallback_observed"] is False + + +def test_qwen3_example_fails_closed_when_provider_marker_is_missing(tmp_path): + config = load_config(CONFIG) + report = build_report( + config, + vime_root=tmp_path / "vime", + rl_kernel_root=tmp_path / "rl-kernel", + command=["bash", "run.sh"], + status="passed", + returncode=0, + log_text="training completed without provider provenance", + log_path=None, + ) + assert report["status"] == "failed" + assert report["claim_boundary"]["qwen3_8b_tp2_cp2_vime_training"] is False + + +def _runtime_evidence(): + return { + "schema_version": "rlkernel.operator_runtime_evidence.v1", + "operators": { + "attention": { + "training": {"implementation_id": "rlk.attn", "backend_id": "rlk", "contract_id": "a"}, + "rollout": {"implementation_id": "rlk.attn", "backend_id": "rlk", "contract_id": "a"}, + "comparison": { + "passed": True, + "out_max_abs": 0.0, + "lse_max_abs": 0.0, + "dq_max_abs": 0.0, + "dk_max_abs": 0.0, + "dv_max_abs": 0.0, + }, + }, + "ffn": { + "training": {"implementation_id": "rlk.ffn", "backend_id": "rlk", "contract_id": "f"}, + "rollout": {"implementation_id": "rlk.ffn", "backend_id": "rlk", "contract_id": "f"}, + "comparison": { + "passed": True, + "out_max_abs": 0.0, + "dx_max_abs": 0.0, + "dw_max_abs": 0.0, + }, + }, + }, + } + + +def test_qwen3_example_accepts_only_exact_zero_runtime_evidence(tmp_path): + evidence = _runtime_evidence() + validate_runtime_evidence(evidence) + config = load_config(CONFIG) + report = build_report( + config, + vime_root=tmp_path / "vime", + rl_kernel_root=tmp_path / "rl-kernel", + command=["bash", "run.sh"], + status="passed", + returncode=0, + log_text="Selected-logprob provider active: backend_id=pytorch-vocab-parallel-logp-ws2", + log_path=None, + runtime_evidence=evidence, + ) + assert report["claim_boundary"]["attention_train_infer_consistency"] == "passed" + assert report["claim_boundary"]["ffn_train_infer_consistency"] == "passed" + + +def test_qwen3_example_rejects_nonzero_runtime_evidence(): + evidence = _runtime_evidence() + evidence["operators"]["attention"]["comparison"]["out_max_abs"] = 1e-6 + with pytest.raises(ValueError, match="attention"): + validate_runtime_evidence(evidence) + + +@pytest.mark.parametrize("bad_path", ["", "other.provider"]) +def test_qwen3_example_rejects_non_rlkernel_provider(bad_path): + config = load_config(CONFIG) + config["selected_logprob_provider"]["path"] = bad_path + with pytest.raises(ValueError, match="RL-Kernel Vime provider"): + validate_config(config) From f79602c8e56e40d28d318a8f9ff2da55606ac551 Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Sat, 22 Aug 2026 03:18:38 +0800 Subject: [PATCH 33/48] feat(rocm): add deterministic vocab parallel logprob backend --- .../rocm_logprob_implementation_summary.md | 137 ++++++++++++++++++ csrc/deterministic_logp_kernel.cu | 88 +++++++++++ csrc/ops.cpp | 13 +- examples/vime_qwen3_8b_tp2_cp2/run.py | 24 ++- rl_engine/_C.pyi | 6 + rl_engine/integrations/__init__.py | 1 - rl_engine/integrations/vime/__init__.py | 1 - rl_engine/integrations/vime/logp.py | 36 +++-- .../ops/pytorch/loss/distributed_grpo_loss.py | 4 +- .../ops/pytorch/loss/vocab_parallel_logp.py | 68 ++++++++- rl_engine/kernels/ops/rocm/loss/__init__.py | 5 + .../ops/rocm/loss/vocab_parallel_logp.py | 16 ++ rl_engine/kernels/registry.py | 37 +++++ tests/test_distributed_grpo_loss.py | 24 +-- tests/test_distributed_logprob_comparison.py | 66 +++++++++ tests/test_rocm_logprob_backend.py | 80 ++++++++++ tests/test_vime_qwen3_example.py | 26 +++- tests/test_vocab_parallel_logp.py | 55 ++++++- 18 files changed, 640 insertions(+), 47 deletions(-) create mode 100644 _dev_notes/rocm_logprob_implementation_summary.md create mode 100644 rl_engine/kernels/ops/rocm/loss/__init__.py create mode 100644 rl_engine/kernels/ops/rocm/loss/vocab_parallel_logp.py create mode 100644 tests/test_rocm_logprob_backend.py diff --git a/_dev_notes/rocm_logprob_implementation_summary.md b/_dev_notes/rocm_logprob_implementation_summary.md new file mode 100644 index 00000000..3d0d2047 --- /dev/null +++ b/_dev_notes/rocm_logprob_implementation_summary.md @@ -0,0 +1,137 @@ +# ROCm Logprob 实现说明 + +日期：2026-08-22 + +这次工作是在临时分支 `work/ws2-logprob-rocm` 上完成的。它基于 issue #241 的 PR1-PR5 integration，再合入当前 PR4 的最新更新。 + +## 一句话概括 + +新增了一个 ROCm 版 WS2 vocab-parallel logprob 后端：每个 GPU 用 HIP/CUDA 可移植 kernel 计算本地 vocab tile 的 FP32 `(max, sumexp)`，TP 之间仍使用 issue #241 已确定的 all-gather + 固定顺序 merge。这样不会为了追求 ROCm 速度而改变原来的数值契约。 + +## 改了什么 + +### 1. 增加本地 tile partial kernel + +文件：`csrc/deterministic_logp_kernel.cu` + +新增 `deterministic_logp_tile_stats`： + +- 输入一个 TP rank 的 local vocab logits； +- 每个 vocab tile 输出 FP32 `max` 和 `sumexp`； +- 过滤真实 vocab 之外的 padding 列； +- 使用固定 block reduction，不使用 atomic 或不确定的全局归约； +- 同一份 `.cu` 源码可由 CUDA 或 HIP 编译。 + +这个 kernel 只负责 rank-local 计算，不负责 RCCL/NCCL，也不负责全局 LSE 合并。跨 TP 的合并顺序仍由已有 WS2 Python 实现控制。 + +### 2. 增加 ROCm backend 注册 + +文件： + +- `rl_engine/kernels/ops/rocm/loss/vocab_parallel_logp.py` +- `rl_engine/kernels/ops/rocm/loss/__init__.py` +- `rl_engine/kernels/registry.py` + +新增 backend： + +```text +rocm-vocab-parallel-logp-ws2 +``` + +它只在 ROCm platform 注册，放在 PyTorch reference 前面。backend 继承现有 `VocabParallelLogprobOp`，所以以下逻辑仍然共用一份实现： + +- TP contract 和 preflight； +- tile-aligned shard 检查； +- all-gather transport； +- global tile order merge； +- selected-token owner gather； +- entropy 的显式 rank-order merge； +- backward 和 active-mask 语义。 + +只有在 ROCm native extension 已经加载、且提供新符号时，registry 才会注册这个 production backend。native 不可用时，registry 不会把它显示成可用的 ROCm production 实现，而是明确使用已有的 PyTorch reference backend。这样 backend provenance 是准确的，不会出现“界面显示用了 ROCm，实际却悄悄跑了另一条路径”的情况。 + +如果调用方显式要求 native ROCm backend，但扩展或符号缺失，会直接抛出清晰的 `RuntimeError`，不会在 wrapper 内部静默 fallback。PyTorch reference 始终使用纯 PyTorch tile 统计；ROCm 子类只有在 native 能力已由 registry 检查通过时才启用 HIP tile kernel。 + +### 3. 增加 native Python binding 和类型声明 + +文件： + +- `csrc/ops.cpp` +- `rl_engine/_C.pyi` + +注册了 `deterministic_logp_tile_stats` 的 Python binding 和类型签名。 + +同时修复了一个 ROCm 构建隐患：`deterministic_collective_*` 使用 CUDA IPC，但 ROCm 构建不编译对应源文件。现在这些符号的声明和 pybind 注册在 HIP 编译时都会被排除，避免 ROCm 链接阶段出现 unresolved symbol。 + +Prefix-Shared Attention 的 NVIDIA PTX 注册也改成 HIP 编译时排除，与 PR4 的 source gating 保持一致。 + +### 4. 增加测试 + +文件：`tests/test_rocm_logprob_backend.py` + +覆盖： + +- ROCm backend 继承并保持 WS2 operator surface； +- backend 只在 ROCm registry 中注册； +- native extension 可用时才注册 ROCm production backend；不可用时首选 PyTorch reference； +- native tile kernel 和 binding 存在； +- CPU-only 环境可以导入 ROCm wrapper，不要求 native extension。 +- reference/native 两条执行路径不会互相静默切换。 + +另外补上了和 PR #319/#325 思路一致的验证入口： + +- ROCm native 路径复用 TP2/TP4 的 TP1 对照、重复执行、forward/backward 和 bitwise 检查； +- ROCm 多卡 `TP2 x CP2` CLI 测试要求所有 rank 的实际 backend 都是 + `rocm-vocab-parallel-logp-ws2`，且 provenance 不得标记 fallback； +- 显式请求 native backend 但扩展缺失时，测试要求直接 fail fast。 + +这些测试在非 ROCm 或 GPU 数量不足的环境会跳过；真正执行需要带 ROCm native extension 的多卡机器。 + +## 没有改什么 + +- 没有改 issue #241 的 TP/CP 语义；CP 仍不是 logprob 的 merge axis。 +- 没有用 RCCL `all_reduce` 取代固定顺序的 all-gather + local merge。 +- 没有把 AITER/Composable Kernel 强行接进 strict path。 +- 没有把 SM90/TMA/WGMMA 源文件加入 ROCm build。 +- 没有修改 Vime provider 的 contract 或 entropy 语义。 +- 没有删除任何仓库文件；只是在 ROCm 编译时排除了不适用的 CUDA IPC/PTX 注册。 + +## 验证结果 + +在当前 Windows/CUDA 开发环境中： + +```text +70 passed, 11 skipped +``` + +通过的测试包括： + +- ROCm backend dispatch 测试； +- issue #241 logprob contract 测试； +- vocab-parallel logprob reference 测试； +- Vime selected-logprob provider 测试。 + +CUDA extension build 没有完成，原因是当前机器缺少 Microsoft Visual C++ `cl.exe`；同时本地 `nvcc` 是 CUDA 12.6，而 PyTorch 是 CUDA 12.8。当前机器也没有 `hipcc` 和 ROCm runtime，因此以下项目尚未在本地验证： + +- gfx942/gfx950 HIP 编译； +- MI300X RCCL TP2/TP4； +- ROCm native tile kernel 与 PyTorch reference 的真实数值/bitwise 对比。 + +建议在 ROCm 机器上运行： + +```bash +PYTORCH_ROCM_ARCH=gfx942 RL_KERNEL_REQUIRE_EXT=1 MAX_JOBS=16 \ + python setup.py build_ext --inplace + +PYTHONHASHSEED=0 pytest -q -ra \ + tests/test_rocm_logprob_backend.py \ + tests/test_logprob_contract.py \ + tests/test_vocab_parallel_logp.py \ + tests/test_distributed_logprob_comparison.py +``` + +真实多 GPU 验证还需要补充 RCCL TP2/TP4 的运行命令和结果；在拿到 gfx942 结果前，不应把这个 backend 宣称为已完成性能优化，只能称为 strict semantics-preserving ROCm implementation。 + +## 当前提交 + +实现尚未推送远程；代码位于当前工作分支 `work/ws2-logprob-rocm`。合并基线提交是 `b811abc`，本次实现文件仍在工作区，待 ROCm 环境验证后再拆分成正式 PR commits。 diff --git a/csrc/deterministic_logp_kernel.cu b/csrc/deterministic_logp_kernel.cu index c238e11d..8ae3f588 100644 --- a/csrc/deterministic_logp_kernel.cu +++ b/csrc/deterministic_logp_kernel.cu @@ -358,3 +358,91 @@ torch::Tensor deterministic_logp_forward_indexed_fp32( auto output = torch::zeros({logits.size(0)}, logits.options().dtype(at::ScalarType::Float)); return deterministic_logp_forward_indexed_out(logits, token_ids, row_indices, output); } + +namespace { + +constexpr int kDeterministicLogpTileBlockSize = 256; + +template +__global__ void deterministic_logp_tile_stats_kernel( + const scalar_t* __restrict__ logits, + float* __restrict__ tile_max, + float* __restrict__ tile_sum, + int64_t rows, + int64_t local_vocab, + int64_t vocab_start, + int64_t real_vocab, + int64_t tile_size, + int64_t local_tiles) { + const int64_t tile_index = static_cast(blockIdx.y); + const int64_t row = static_cast(blockIdx.x); + if (row >= rows || tile_index >= local_tiles) { + return; + } + const int64_t col_begin = tile_index * tile_size; + const int64_t col_end = min(col_begin + tile_size, local_vocab); + float local_max = -std::numeric_limits::infinity(); + for (int64_t col = col_begin + threadIdx.x; col < col_end; col += BlockSize) { + const int64_t global_col = vocab_start + col; + if (global_col < real_vocab) { + local_max = fmaxf(local_max, static_cast(logits[row * local_vocab + col])); + } + } + const float max_value = deterministicBlockReduceMax(local_max); + __shared__ float row_max; + if (threadIdx.x == 0) row_max = max_value; + __syncthreads(); + + float local_sum = 0.0f; + for (int64_t col = col_begin + threadIdx.x; col < col_end; col += BlockSize) { + const int64_t global_col = vocab_start + col; + if (global_col < real_vocab) { + local_sum += expf(static_cast(logits[row * local_vocab + col]) - row_max); + } + } + const float sum_value = deterministicBlockReduceSum(local_sum); + if (threadIdx.x == 0) { + const int64_t output_index = row * local_tiles + tile_index; + tile_max[output_index] = row_max; + tile_sum[output_index] = isfinite(row_max) ? sum_value : 0.0f; + } +} + +} // namespace + +std::vector deterministic_logp_tile_stats( + torch::Tensor logits, + int64_t vocab_start, + int64_t real_vocab, + int64_t num_tiles) { + TORCH_CHECK(logits.is_cuda(), "logits must be a CUDA/ROCm tensor"); + TORCH_CHECK(logits.dim() == 2, "logits must be 2D [tokens, local_vocab]"); + TORCH_CHECK(logits.scalar_type() == at::ScalarType::Half || + logits.scalar_type() == at::ScalarType::BFloat16 || + logits.scalar_type() == at::ScalarType::Float, + "logits must be float16, bfloat16, or float32"); + TORCH_CHECK(vocab_start >= 0 && real_vocab > 0 && num_tiles > 0, + "invalid vocabulary metadata"); + auto input = logits.contiguous(); + const int64_t rows = input.size(0); + const int64_t local_vocab = input.size(1); + TORCH_CHECK(local_vocab > 0 && local_vocab % num_tiles == 0, + "local_vocab must be divisible by num_tiles"); + const int64_t tile_size = local_vocab / num_tiles; + auto options = input.options().dtype(at::ScalarType::Float); + auto tile_max = torch::empty({rows, num_tiles}, options); + auto tile_sum = torch::empty({rows, num_tiles}, options); + const dim3 grid(static_cast(rows), static_cast(num_tiles), 1); + auto stream = at::cuda::getCurrentCUDAStream(); + AT_DISPATCH_FLOATING_TYPES_AND2( + at::ScalarType::Half, at::ScalarType::BFloat16, input.scalar_type(), + "deterministic_logp_tile_stats", ([&] { + deterministic_logp_tile_stats_kernel + <<>>( + input.data_ptr(), tile_max.data_ptr(), + tile_sum.data_ptr(), rows, local_vocab, vocab_start, + real_vocab, tile_size, num_tiles); + })); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + return {tile_max, tile_sum}; +} diff --git a/csrc/ops.cpp b/csrc/ops.cpp index 2a9f7ec2..8ca3dcaf 100644 --- a/csrc/ops.cpp +++ b/csrc/ops.cpp @@ -88,8 +88,14 @@ torch::Tensor deterministic_logp_forward_out(torch::Tensor logits, torch::Tensor torch::Tensor deterministic_logp_forward_fp32(torch::Tensor logits, torch::Tensor token_ids); torch::Tensor deterministic_logp_forward_indexed_out(torch::Tensor logits, torch::Tensor token_ids, torch::Tensor row_indices, torch::Tensor output); torch::Tensor deterministic_logp_forward_indexed_fp32(torch::Tensor logits, torch::Tensor token_ids, torch::Tensor row_indices); +std::vector deterministic_logp_tile_stats( + torch::Tensor logits, + int64_t vocab_start, + int64_t real_vocab, + int64_t num_tiles); // Single-node TP=8 deterministic collectives. +#if !defined(__HIPCC__) && !defined(__HIP_PLATFORM_AMD__) std::tuple, int64_t> deterministic_collective_ipc_meta( torch::Tensor& tensor); int64_t deterministic_collective_create( @@ -102,6 +108,7 @@ void deterministic_collective_stage(int64_t handle, torch::Tensor& input); void deterministic_collective_all_reduce(int64_t handle, torch::Tensor& output); void deterministic_collective_reduce_scatter(int64_t handle, torch::Tensor& output); void deterministic_collective_all_gather(int64_t handle, torch::Tensor& output); +#endif // Batch-Invariant Deterministic GEMM Declarations torch::Tensor det_gemm_fwd(torch::Tensor a, torch::Tensor b); @@ -382,8 +389,11 @@ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) { m.def("deterministic_logp_forward_fp32", &deterministic_logp_forward_fp32, "Batch-invariant deterministic logp fp32"); m.def("deterministic_logp_forward_indexed_out", &deterministic_logp_forward_indexed_out, "Batch-invariant deterministic logp indexed out"); m.def("deterministic_logp_forward_indexed_fp32", &deterministic_logp_forward_indexed_fp32, "Batch-invariant deterministic logp indexed fp32"); + m.def("deterministic_logp_tile_stats", &deterministic_logp_tile_stats, + "Deterministic local vocab-tile FP32 max and sumexp partials"); // Single-node TP=8 fixed-tree collectives. +#if !defined(__HIPCC__) && !defined(__HIP_PLATFORM_AMD__) m.def( "deterministic_collective_ipc_meta", &deterministic_collective_ipc_meta, @@ -412,9 +422,10 @@ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) { "deterministic_collective_all_gather", &deterministic_collective_all_gather, "Run the TP=8 deterministic rank-ordered all-gather kernel"); +#endif // Prefix-shared attention uses NVIDIA PTX and falls back to PyTorch SDPA on ROCm. -#if !defined(USE_ROCM) +#if !defined(__HIPCC__) && !defined(__HIP_PLATFORM_AMD__) m.def("prefix_shared_attention", &prefix_shared_attention, "Prefix-Shared Fused Attention for GRPO"); #endif diff --git a/examples/vime_qwen3_8b_tp2_cp2/run.py b/examples/vime_qwen3_8b_tp2_cp2/run.py index 4823dd58..72e3744b 100644 --- a/examples/vime_qwen3_8b_tp2_cp2/run.py +++ b/examples/vime_qwen3_8b_tp2_cp2/run.py @@ -16,12 +16,10 @@ import json import os import subprocess -import sys from datetime import datetime, timezone from pathlib import Path from typing import Any, Mapping - DEFAULT_CONFIG = Path(__file__).with_name("qwen3_8b_tp2_cp2.json") PROVIDER_MARKER = "Selected-logprob provider active" FALLBACK_MARKERS = ("using native path", "fallback=True", "fallback=true") @@ -44,7 +42,11 @@ def validate_config(config: Mapping[str, Any]) -> None: training = config.get("training") rollout = config.get("rollout") provider = config.get("selected_logprob_provider") - if not isinstance(training, Mapping) or not isinstance(rollout, Mapping) or not isinstance(provider, Mapping): + if ( + not isinstance(training, Mapping) + or not isinstance(rollout, Mapping) + or not isinstance(provider, Mapping) + ): raise ValueError("training, rollout, and selected_logprob_provider sections are required") expected = { "tensor_model_parallel_size": 2, @@ -163,7 +165,9 @@ def build_report( provider_active = PROVIDER_MARKER in log_text fallback_observed = any(marker in log_text for marker in FALLBACK_MARKERS) strict_provider_passed = status == "passed" and provider_active and not fallback_observed - effective_status = "passed" if strict_provider_passed else ("failed" if status == "passed" else status) + effective_status = ( + "passed" if strict_provider_passed else ("failed" if status == "passed" else status) + ) attention_status = _operator_evidence_status(runtime_evidence, "attention") ffn_status = _operator_evidence_status(runtime_evidence, "ffn") return { @@ -192,7 +196,9 @@ def build_report( "returncode": returncode, "artifacts": { "log": None if log_path is None else str(log_path), - "runtime_evidence": None if runtime_evidence_path is None else str(runtime_evidence_path), + "runtime_evidence": ( + None if runtime_evidence_path is None else str(runtime_evidence_path) + ), }, "runtime_evidence": None if runtime_evidence is None else dict(runtime_evidence), "revisions": { @@ -206,7 +212,9 @@ def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--config", type=Path, default=DEFAULT_CONFIG) parser.add_argument("--vime-root", type=Path, default=Path(os.environ.get("VIME_ROOT", "."))) - parser.add_argument("--rl-kernel-root", type=Path, default=Path(os.environ.get("RL_KERNEL_ROOT", "."))) + parser.add_argument( + "--rl-kernel-root", type=Path, default=Path(os.environ.get("RL_KERNEL_ROOT", ".")) + ) parser.add_argument("--output", type=Path, default=Path("qwen3_8b_tp2_cp2.validation.json")) parser.add_argument( "--runtime-evidence", @@ -234,7 +242,9 @@ def main(argv: list[str] | None = None) -> int: log_path = args.output.with_suffix(".log") env = build_environment(vime_root, rl_kernel_root) with log_path.open("w", encoding="utf-8") as log_handle: - process = subprocess.run(command, cwd=vime_root, env=env, stdout=log_handle, stderr=subprocess.STDOUT) + process = subprocess.run( + command, cwd=vime_root, env=env, stdout=log_handle, stderr=subprocess.STDOUT + ) returncode = process.returncode log_text = log_path.read_text(encoding="utf-8", errors="replace") status = "passed" if returncode == 0 else "failed" diff --git a/rl_engine/_C.pyi b/rl_engine/_C.pyi index 08d460e3..ac5708b3 100644 --- a/rl_engine/_C.pyi +++ b/rl_engine/_C.pyi @@ -161,6 +161,12 @@ def deterministic_logp_forward_indexed_fp32( token_ids: torch.Tensor, row_indices: torch.Tensor, ) -> torch.Tensor: ... +def deterministic_logp_tile_stats( + logits: torch.Tensor, + vocab_start: int, + real_vocab: int, + num_tiles: int, +) -> list[torch.Tensor]: ... def deterministic_attention_forward( q: torch.Tensor, k: torch.Tensor, diff --git a/rl_engine/integrations/__init__.py b/rl_engine/integrations/__init__.py index 29bcc68f..ec3e322c 100644 --- a/rl_engine/integrations/__init__.py +++ b/rl_engine/integrations/__init__.py @@ -2,4 +2,3 @@ # Copyright (c) 2026 RL-Kernel Contributors """Optional framework adapters owned by RL-Kernel.""" - diff --git a/rl_engine/integrations/vime/__init__.py b/rl_engine/integrations/vime/__init__.py index 44470314..3c35c88f 100644 --- a/rl_engine/integrations/vime/__init__.py +++ b/rl_engine/integrations/vime/__init__.py @@ -6,4 +6,3 @@ from .logp import ProviderResult, SelectedLogprobProviderUnavailable, provider __all__ = ["ProviderResult", "SelectedLogprobProviderUnavailable", "provider"] - diff --git a/rl_engine/integrations/vime/logp.py b/rl_engine/integrations/vime/logp.py index db436d69..68f5a6f1 100644 --- a/rl_engine/integrations/vime/logp.py +++ b/rl_engine/integrations/vime/logp.py @@ -57,14 +57,18 @@ class ProviderResult: def _as_positive_int(value: Any, name: str) -> int: if isinstance(value, bool) or not isinstance(value, int) or value <= 0: - raise SelectedLogprobProviderUnavailable(f"{name} must be a positive integer; got {value!r}") + raise SelectedLogprobProviderUnavailable( + f"{name} must be a positive integer; got {value!r}" + ) return value def _metadata(request: Any) -> Mapping[str, Any]: value = getattr(request, "metadata", None) if not isinstance(value, Mapping): - raise SelectedLogprobProviderUnavailable("request.metadata must provide vocab-parallel metadata") + raise SelectedLogprobProviderUnavailable( + "request.metadata must provide vocab-parallel metadata" + ) return value @@ -108,7 +112,9 @@ def _contract_for_request(request: Any) -> tuple[LogprobContract, int]: targets = _request_tensor(request, "target_ids") metadata = _metadata(request) if logits.ndim != 2 or targets.shape != (logits.shape[0],): - raise SelectedLogprobProviderUnavailable("request must contain local [T, V] logits and aligned [T] targets") + raise SelectedLogprobProviderUnavailable( + "request must contain local [T, V] logits and aligned [T] targets" + ) if logits.dtype not in (torch.bfloat16, torch.float16, torch.float32): raise SelectedLogprobProviderUnavailable(f"unsupported logit dtype {logits.dtype}") if targets.device != logits.device: @@ -117,12 +123,20 @@ def _contract_for_request(request: Any) -> tuple[LogprobContract, int]: cp = getattr(request, "context_parallel", None) cp_world_size = _as_positive_int(getattr(cp, "world_size", None), "context_parallel.world_size") cp_rank = getattr(cp, "rank", None) - if isinstance(cp_rank, bool) or not isinstance(cp_rank, int) or not 0 <= cp_rank < cp_world_size: + if ( + isinstance(cp_rank, bool) + or not isinstance(cp_rank, int) + or not 0 <= cp_rank < cp_world_size + ): raise SelectedLogprobProviderUnavailable( f"context_parallel.rank={cp_rank!r} is invalid for CP={cp_world_size}" ) - if getattr(cp, "layout", None) not in ({"single"} if cp_world_size == 1 else {"zigzag", "allgather"}): - raise SelectedLogprobProviderUnavailable("context_parallel layout does not describe local CP token ownership") + if getattr(cp, "layout", None) not in ( + {"single"} if cp_world_size == 1 else {"zigzag", "allgather"} + ): + raise SelectedLogprobProviderUnavailable( + "context_parallel layout does not describe local CP token ownership" + ) tp_rank, tp_world_size = _tp_coordinates(getattr(request, "tensor_parallel_group", None)) declared_tp_rank = metadata.get("tp_rank") @@ -133,7 +147,8 @@ def _contract_for_request(request: Any) -> tuple[LogprobContract, int]: ) if declared_tp_world_size is not None and declared_tp_world_size != tp_world_size: raise SelectedLogprobProviderUnavailable( - f"metadata tp_world_size={declared_tp_world_size} disagrees with TP group size={tp_world_size}" + f"metadata tp_world_size={declared_tp_world_size} disagrees with " + f"TP group size={tp_world_size}" ) real_vocab_size = _as_positive_int(metadata.get("real_vocab_size"), "real_vocab_size") @@ -141,10 +156,13 @@ def _contract_for_request(request: Any) -> tuple[LogprobContract, int]: if logits.shape[1] * tp_world_size != padded_vocab_size: raise SelectedLogprobProviderUnavailable( "local vocab width and TP group do not cover padded_vocab_size exactly: " - f"{logits.shape[1]} * {tp_world_size} != {padded_vocab_size}" + f"{logits.shape[1]} * {tp_world_size} != " + f"{padded_vocab_size}" ) if real_vocab_size > padded_vocab_size: - raise SelectedLogprobProviderUnavailable("real_vocab_size must not exceed padded_vocab_size") + raise SelectedLogprobProviderUnavailable( + "real_vocab_size must not exceed padded_vocab_size" + ) bounds = tuple( (rank * logits.shape[1], (rank + 1) * logits.shape[1]) for rank in range(tp_world_size) diff --git a/rl_engine/kernels/ops/pytorch/loss/distributed_grpo_loss.py b/rl_engine/kernels/ops/pytorch/loss/distributed_grpo_loss.py index bbec450d..6add5721 100644 --- a/rl_engine/kernels/ops/pytorch/loss/distributed_grpo_loss.py +++ b/rl_engine/kernels/ops/pytorch/loss/distributed_grpo_loss.py @@ -282,7 +282,9 @@ def _normalized( reduction = contract.reduction normalizer = reduction.token_normalizer if normalizer is TokenNormalizer.FIXED_CONSTANT: - return per_sequence_totals.sum() / float(reduction.fixed_normalizer_constant) + fixed_normalizer_constant = reduction.fixed_normalizer_constant + assert fixed_normalizer_constant is not None + return per_sequence_totals.sum() / float(fixed_normalizer_constant) if normalizer is TokenNormalizer.GLOBAL_ACTIVE_TOKENS: return per_sequence_totals.sum() / per_sequence_counts.sum().to(per_sequence_totals.dtype) diff --git a/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py b/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py index 0de98102..69696d38 100644 --- a/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py +++ b/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py @@ -202,6 +202,46 @@ def _local_tile_stats(z_masked: torch.Tensor, tile: int) -> tuple[torch.Tensor, return torch.stack(m_parts, dim=1), torch.stack(s_parts, dim=1) +def _native_rocm_tile_stats( + z_masked: torch.Tensor, + tile: int, + *, + vocab_start: int, + real_vocab_size: int, +) -> tuple[torch.Tensor, torch.Tensor] | None: + """Use the HIP-compatible local partial kernel when the extension has it. + + TP transport and the global merge deliberately stay in Python. That keeps + the issue #241 reduction contract identical across CUDA, ROCm, and the + reference implementation while making the large local vocab scan native. + """ + + if torch.version.hip is None or not z_masked.is_cuda: + raise RuntimeError("ROCm native tile stats require a HIP CUDA tensor") + try: + from rl_engine.kernels.ops.base import _C, _EXT_AVAILABLE + + if not _EXT_AVAILABLE or not hasattr(_C, "deterministic_logp_tile_stats"): + raise RuntimeError( + "ROCm vocab-parallel logprob native extension is unavailable; " + "build rl_engine._C with a ROCm toolchain" + ) + local_tiles = z_masked.shape[1] // tile + if local_tiles <= 0: + raise RuntimeError("ROCm native tile stats require at least one local vocab tile") + return tuple( + tensor.contiguous() + for tensor in _C.deterministic_logp_tile_stats( + z_masked, + int(vocab_start), + int(real_vocab_size), + int(local_tiles), + ) + ) + except (ImportError, AttributeError) as exc: + raise RuntimeError("ROCm vocab-parallel logprob native extension is unavailable") from exc + + def _gather_tile_stats( local_m: torch.Tensor, local_s: torch.Tensor, @@ -315,6 +355,7 @@ def forward( tile, with_entropy, with_entropy_grad, + use_native_tile_stats, ): z_masked = local_logits.float() sharding = contract.sharding @@ -327,7 +368,15 @@ def forward( safe_target = torch.where(active_mask, target_1d, torch.zeros_like(target_1d)) - local_m, local_s = _local_tile_stats(z_masked, tile) + if use_native_tile_stats: + local_m, local_s = _native_rocm_tile_stats( + z_masked, + tile, + vocab_start=sharding.local_vocab_start, + real_vocab_size=sharding.real_vocab_size, + ) + else: + local_m, local_s = _local_tile_stats(z_masked, tile) m_all, s_all = _gather_tile_stats(local_m, local_s, contract, tp_group, tile) target_logit = _gather_target_logit(z_masked, safe_target, contract, tp_group) lse = _merge_tile_partials(m_all, s_all) @@ -338,7 +387,9 @@ def forward( finite_row = torch.isfinite(lse) lse_safe = torch.where(finite_row, lse, torch.zeros_like(lse)) probabilities = (z_masked - lse_safe.unsqueeze(1)).exp() - probabilities = torch.where(finite_row.unsqueeze(1), probabilities, torch.zeros_like(probabilities)) + probabilities = torch.where( + finite_row.unsqueeze(1), probabilities, torch.zeros_like(probabilities) + ) finite_logits = torch.isfinite(z_masked) log_gap = torch.where( finite_logits, @@ -365,7 +416,7 @@ def backward(ctx, grad_logp, grad_lse, grad_entropy): if not ctx.needs_input_grad[0] or ( grad_logp is None and grad_lse is None and grad_entropy is None ): - return None, None, None, None, None, None, None, None + return None, None, None, None, None, None, None, None, None z_masked, lse, safe_target, active_mask, padding_cols, entropy = ctx.saved_tensors n, local_vocab = z_masked.shape @@ -390,11 +441,13 @@ def backward(ctx, grad_logp, grad_lse, grad_entropy): grad = grad + grad_lse.unsqueeze(1) * p if ctx.with_entropy_grad and grad_entropy is not None: entropy_input = lse_safe.unsqueeze(1) - z_masked - entropy.unsqueeze(1) - entropy_input = torch.where(torch.isfinite(z_masked), entropy_input, torch.zeros_like(entropy_input)) + entropy_input = torch.where( + torch.isfinite(z_masked), entropy_input, torch.zeros_like(entropy_input) + ) grad = grad + grad_entropy.unsqueeze(1) * p * entropy_input if bool(padding_cols.any()): grad = grad.masked_fill(padding_cols.unsqueeze(0), 0.0) - return grad.to(ctx.input_dtype), None, None, None, None, None, None, None + return grad.to(ctx.input_dtype), None, None, None, None, None, None, None, None class VocabParallelLogprobOp: @@ -402,6 +455,7 @@ class VocabParallelLogprobOp: op_class = "logprob" is_batch_invariant = True + use_native_tile_stats = False def __init__(self) -> None: pass @@ -444,6 +498,7 @@ def apply( validate=validate, with_entropy=False, with_entropy_grad=False, + use_native_tile_stats=self.use_native_tile_stats, ) return selected_logp, lse @@ -473,6 +528,7 @@ def apply_with_entropy( validate=validate, with_entropy=True, with_entropy_grad=with_entropy_grad, + use_native_tile_stats=self.use_native_tile_stats, ) def _apply( @@ -486,6 +542,7 @@ def _apply( validate: bool, with_entropy: bool, with_entropy_grad: bool, + use_native_tile_stats: bool, ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: if not isinstance(contract, LogprobContract): raise LogprobContractError("contract must be a LogprobContract") @@ -510,6 +567,7 @@ def _apply( tile, with_entropy, with_entropy_grad, + use_native_tile_stats, ) if validate and bool((~torch.isfinite(lse) & active_mask).any().item()): diff --git a/rl_engine/kernels/ops/rocm/loss/__init__.py b/rl_engine/kernels/ops/rocm/loss/__init__.py new file mode 100644 index 00000000..b7fbeaca --- /dev/null +++ b/rl_engine/kernels/ops/rocm/loss/__init__.py @@ -0,0 +1,5 @@ +"""ROCm logprob operators.""" + +from .vocab_parallel_logp import RocmVocabParallelLogprobOp + +__all__ = ["RocmVocabParallelLogprobOp"] diff --git a/rl_engine/kernels/ops/rocm/loss/vocab_parallel_logp.py b/rl_engine/kernels/ops/rocm/loss/vocab_parallel_logp.py new file mode 100644 index 00000000..272ce3f7 --- /dev/null +++ b/rl_engine/kernels/ops/rocm/loss/vocab_parallel_logp.py @@ -0,0 +1,16 @@ +"""ROCm WS2 vocab-parallel logprob backend. + +The native HIP kernel computes only local FP32 tile partials. The reference +operator owns TP transport, fixed tile-order merging, target ownership, +entropy, and autograd so ROCm and issue #241 share one contract. +""" + +from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import VocabParallelLogprobOp + + +class RocmVocabParallelLogprobOp(VocabParallelLogprobOp): + """Contract-preserving ROCm implementation with a HIP local reduction.""" + + op_class = "logprob" + is_batch_invariant = True + use_native_tile_stats = True diff --git a/rl_engine/kernels/registry.py b/rl_engine/kernels/registry.py index afa5e8d9..0f34f964 100644 --- a/rl_engine/kernels/registry.py +++ b/rl_engine/kernels/registry.py @@ -98,6 +98,9 @@ class OpBackend(Enum, metaclass=_KernelEnumMeta): PYTORCH_VOCAB_PARALLEL_LOGP = ( "rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp.VocabParallelLogprobOp" ) + ROCM_VOCAB_PARALLEL_LOGP = ( + "rl_engine.kernels.ops.rocm.loss.vocab_parallel_logp.RocmVocabParallelLogprobOp" + ) # Deterministic GRPO loss on the TP-aware logprob path (WS2 #241 PR5) PYTORCH_DISTRIBUTED_GRPO_LOSS = ( "rl_engine.kernels.ops.pytorch.loss.distributed_grpo_loss.DistributedGRPOLossOp" @@ -139,6 +142,18 @@ class OpBackend(Enum, metaclass=_KernelEnumMeta): CUDA_SM90_EMBEDDING = "rl_engine.kernels.ops.cuda.linear.embedding.SM90EmbeddingOp" +def _rocm_vocab_logprob_native_available() -> bool: + """Return whether the strict ROCm tile kernel is actually loadable.""" + + if torch.version.hip is None: + return False + try: + from rl_engine.kernels.ops.base import _C, _EXT_AVAILABLE + except ImportError: + return False + return bool(_EXT_AVAILABLE and hasattr(_C, "deterministic_logp_tile_stats")) + + def resolve_logp_op_type( logp_backend: Optional[str] = None, *, @@ -402,6 +417,28 @@ def __init__(self): prepend=True, ) + rocm_vocab_logprob_capability = LogprobBackendCapability( + backend_id="rocm-vocab-parallel-logp-ws2", + roles=common_logprob_roles, + dtypes=common_logprob_dtypes, + tp_world_sizes=None, + cp_world_sizes=None, + supports_vocab_padding=True, + mask_modes=frozenset({MaskMode.EXPLICIT_ACTIVE_MASK, MaskMode.IGNORE_INDEX}), + exports_vocab_lse=True, + determinism_scopes=frozenset( + {DeterminismScope.CROSS_TP_BITWISE, DeterminismScope.FIXED_TOPOLOGY} + ), + implementation_kind="production", + ) + if _rocm_vocab_logprob_native_available(): + self.register_logprob_backend( + OpBackend.ROCM_VOCAB_PARALLEL_LOGP, + rocm_vocab_logprob_capability, + platform="rocm", + prepend=True, + ) + # WS2 loss dispatch starts empty: the existing single-GPU GRPO ops # declare no LossBackendCapability, so they are unreachable from # get_loss_op and cannot be picked up as a silent fallback for a diff --git a/tests/test_distributed_grpo_loss.py b/tests/test_distributed_grpo_loss.py index b0a2321e..d4f8755f 100644 --- a/tests/test_distributed_grpo_loss.py +++ b/tests/test_distributed_grpo_loss.py @@ -822,18 +822,18 @@ def test_mesh_matches_the_single_rank_baseline(self, tp, dp, gpu_baseline): assert actual["advantage_bits"] == baseline["advantage_bits"] # The sensitive comparison: per-sequence totals, before the scalar # average rounds a reordering away. - assert actual["per_sequence_policy_bits"] == baseline["per_sequence_policy_bits"], ( - f"{label} per-sequence policy totals differ from the single-rank baseline" - ) - assert actual["per_sequence_kl_bits"] == baseline["per_sequence_kl_bits"], ( - f"{label} per-sequence KL totals differ from the single-rank baseline" - ) + assert ( + actual["per_sequence_policy_bits"] == baseline["per_sequence_policy_bits"] + ), f"{label} per-sequence policy totals differ from the single-rank baseline" + assert ( + actual["per_sequence_kl_bits"] == baseline["per_sequence_kl_bits"] + ), f"{label} per-sequence KL totals differ from the single-rank baseline" assert actual["loss_bits"] == baseline["loss_bits"], f"{label} loss differs" assert actual["policy_bits"] == baseline["policy_bits"] assert actual["kl_bits"] == baseline["kl_bits"] - assert actual["grad_row"] == baseline["grad_row"], ( - f"{label} gradient differs from the single-rank baseline" - ) + assert ( + actual["grad_row"] == baseline["grad_row"] + ), f"{label} gradient differs from the single-rank baseline" @_requires_gpus(4) def test_pure_axes_agree_with_each_other(self): @@ -851,9 +851,9 @@ def test_pure_axes_agree_with_each_other(self): class TestMeshGuards: def test_preflight_rejects_a_dp_rank_that_disagrees(self): payloads = _run_mesh(1, 2, scenario="preflight_dp_mismatch") - assert all(not item.get("ok") for item in payloads), ( - "every rank must abort when one of them declares a different objective" - ) + assert all( + not item.get("ok") for item in payloads + ), "every rank must abort when one of them declares a different objective" assert any("preflight" in item.get("error", "") for item in payloads) def test_preflight_rejects_a_tp_rank_that_disagrees(self): diff --git a/tests/test_distributed_logprob_comparison.py b/tests/test_distributed_logprob_comparison.py index 8bb9d4ad..887b53bd 100644 --- a/tests/test_distributed_logprob_comparison.py +++ b/tests/test_distributed_logprob_comparison.py @@ -316,3 +316,69 @@ def test_tp2_cp2_gloo_cli_emits_per_rank_report(tmp_path): assert {rank["cp_rank"] for rank in payload["ranks"]} == {0, 1} assert payload["aggregate"]["dlogp"]["stats"]["active_count"] == 3 assert json.loads(result.stdout)["case"]["tp_world_size"] == 2 + + +@pytest.mark.skipif(torch.version.hip is None, reason="requires a ROCm PyTorch build") +@pytest.mark.skipif(torch.cuda.device_count() < 4, reason="requires four ROCm GPUs") +def test_tp2_cp2_rocm_native_cli_emits_strict_report(tmp_path): + """Run the production ROCm backend across TP2 x CP2 and require provenance.""" + + from rl_engine.kernels.registry import _rocm_vocab_logprob_native_available + + if not _rocm_vocab_logprob_native_available(): + pytest.skip("requires the compiled ROCm logprob extension") + + script = ( + Path(__file__).resolve().parents[1] + / "rl_engine" + / "testing" + / "distributed_logprob_comparison.py" + ) + output = tmp_path / "tp2-cp2-rocm-native.json" + environment = os.environ.copy() + environment.setdefault("OMP_NUM_THREADS", "1") + result = subprocess.run( + [ + sys.executable, + "-m", + "torch.distributed.run", + "--standalone", + "--nproc-per-node=4", + str(script), + "--tp", + "2", + "--cp", + "2", + "--device", + "cuda", + "--dist-backend", + "nccl", + "--backend", + "rocm-vocab-parallel-logp-ws2", + "--real-vocab", + "13", + "--padded-vocab", + "16", + "--num-vocab-tiles", + "8", + "--batch", + "1", + "--seq", + "4", + "--prompt-tokens", + "1", + "--output", + str(output), + ], + check=True, + capture_output=True, + text=True, + timeout=180, + env=environment, + ) + payload = json.loads(output.read_text(encoding="utf-8")) + assert payload["passed"] + assert {rank["actual_backend"] for rank in payload["ranks"]} == {"rocm-vocab-parallel-logp-ws2"} + assert {rank["fallback"] for rank in payload["ranks"]} == {False} + assert {rank["contract"]["sharding"]["cp_world_size"] for rank in payload["ranks"]} == {2} + assert json.loads(result.stdout)["case"]["cp_world_size"] == 2 diff --git a/tests/test_rocm_logprob_backend.py b/tests/test_rocm_logprob_backend.py new file mode 100644 index 00000000..f592cd09 --- /dev/null +++ b/tests/test_rocm_logprob_backend.py @@ -0,0 +1,80 @@ +# SPDX-License-Identifier: Apache-2.0 + +from pathlib import Path + +import pytest + +from rl_engine.kernels.logprob_contract import ( + LogprobContract, + MaskSpec, + ReductionSpec, + ShardingSpec, +) +from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import VocabParallelLogprobOp +from rl_engine.kernels.ops.rocm.loss.vocab_parallel_logp import RocmVocabParallelLogprobOp +from rl_engine.kernels.registry import KernelRegistry, OpBackend + + +def test_rocm_backend_preserves_ws2_operator_surface(): + assert issubclass(RocmVocabParallelLogprobOp, VocabParallelLogprobOp) + assert RocmVocabParallelLogprobOp.op_class == "logprob" + assert RocmVocabParallelLogprobOp.is_batch_invariant + + +def test_rocm_backend_is_gated_by_native_extension(monkeypatch): + registry = KernelRegistry() + registry._platform = lambda: "rocm" + candidates = registry._logprob_candidates["rocm"] + if OpBackend.ROCM_VOCAB_PARALLEL_LOGP in candidates: + assert candidates[0] is OpBackend.ROCM_VOCAB_PARALLEL_LOGP + capability = registry._logprob_capabilities["rocm"][OpBackend.ROCM_VOCAB_PARALLEL_LOGP] + assert capability.backend_id == "rocm-vocab-parallel-logp-ws2" + assert capability.implementation_kind == "production" + else: + assert candidates[0] is OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP + + +def test_rocm_native_tile_kernel_is_hip_guarded_and_registered(): + source = (Path(__file__).resolve().parents[1] / "csrc" / "ops.cpp").read_text(encoding="utf-8") + kernel = ( + Path(__file__).resolve().parents[1] / "csrc" / "deterministic_logp_kernel.cu" + ).read_text(encoding="utf-8") + assert "deterministic_logp_tile_stats" in source + assert "deterministic_logp_tile_stats_kernel" in kernel + assert "atomic" not in kernel.lower() + assert "__HIPCC__" in source + assert "deterministic_collective_all_gather" in source + + +def test_rocm_backend_import_does_not_require_native_extension(): + # Importing the wrapper must remain possible in CPU-only CI; capability + # loading, not module import, decides whether the native fast path exists. + op = RocmVocabParallelLogprobOp() + assert isinstance(op, VocabParallelLogprobOp) + + +def test_explicit_native_backend_fails_closed_when_extension_is_missing(monkeypatch): + import rl_engine.kernels.registry as registry_module + + monkeypatch.setattr(registry_module, "_rocm_vocab_logprob_native_available", lambda: False) + registry = KernelRegistry() + registry._platform = lambda: "rocm" + contract = LogprobContract( + role="train", + dtype="fp32", + mask=MaskSpec(num_tokens=1, active_mask=(True,)), + sharding=ShardingSpec( + tp_rank=0, + tp_world_size=1, + vocab_shard_bounds=((0, 8),), + real_vocab_size=8, + padded_vocab_size=8, + ), + reduction=ReductionSpec(), + ) + + with pytest.raises(RuntimeError, match="rocm-vocab-parallel-logp-ws2"): + registry.get_logprob_op( + contract, + requested_backend="rocm-vocab-parallel-logp-ws2", + ) diff --git a/tests/test_vime_qwen3_example.py b/tests/test_vime_qwen3_example.py index 007d4ba1..91d9e2a2 100644 --- a/tests/test_vime_qwen3_example.py +++ b/tests/test_vime_qwen3_example.py @@ -2,7 +2,6 @@ from __future__ import annotations -import json from pathlib import Path import pytest @@ -14,7 +13,6 @@ validate_runtime_evidence, ) - ROOT = Path(__file__).parents[1] CONFIG = ROOT / "examples" / "vime_qwen3_8b_tp2_cp2" / "qwen3_8b_tp2_cp2.json" @@ -67,8 +65,16 @@ def _runtime_evidence(): "schema_version": "rlkernel.operator_runtime_evidence.v1", "operators": { "attention": { - "training": {"implementation_id": "rlk.attn", "backend_id": "rlk", "contract_id": "a"}, - "rollout": {"implementation_id": "rlk.attn", "backend_id": "rlk", "contract_id": "a"}, + "training": { + "implementation_id": "rlk.attn", + "backend_id": "rlk", + "contract_id": "a", + }, + "rollout": { + "implementation_id": "rlk.attn", + "backend_id": "rlk", + "contract_id": "a", + }, "comparison": { "passed": True, "out_max_abs": 0.0, @@ -79,8 +85,16 @@ def _runtime_evidence(): }, }, "ffn": { - "training": {"implementation_id": "rlk.ffn", "backend_id": "rlk", "contract_id": "f"}, - "rollout": {"implementation_id": "rlk.ffn", "backend_id": "rlk", "contract_id": "f"}, + "training": { + "implementation_id": "rlk.ffn", + "backend_id": "rlk", + "contract_id": "f", + }, + "rollout": { + "implementation_id": "rlk.ffn", + "backend_id": "rlk", + "contract_id": "f", + }, "comparison": { "passed": True, "out_max_abs": 0.0, diff --git a/tests/test_vocab_parallel_logp.py b/tests/test_vocab_parallel_logp.py index a0dfe5d8..10639e2b 100644 --- a/tests/test_vocab_parallel_logp.py +++ b/tests/test_vocab_parallel_logp.py @@ -28,6 +28,7 @@ BACKEND_ID, VocabParallelLogprobOp, ) +from rl_engine.kernels.ops.rocm.loss.vocab_parallel_logp import RocmVocabParallelLogprobOp from rl_engine.kernels.registry import KernelRegistry, OpBackend REAL_VOCAB = 27 @@ -372,7 +373,16 @@ def _tp_inputs(device, dtype, seed: int = 2026): return logits.to(device=device, dtype=dtype), targets.to(device) -def _nccl_worker(rank, world_size, init_method, result_queue, scenario, uneven, dtype_name): +def _nccl_worker( + rank, + world_size, + init_method, + result_queue, + scenario, + uneven, + dtype_name, + backend_kind="pytorch", +): import torch.distributed as dist try: @@ -382,7 +392,7 @@ def _nccl_worker(rank, world_size, init_method, result_queue, scenario, uneven, backend="nccl", init_method=init_method, rank=rank, world_size=world_size ) dtype = TP_DTYPES[dtype_name] - op = VocabParallelLogprobOp() + op = RocmVocabParallelLogprobOp() if backend_kind == "rocm" else VocabParallelLogprobOp() bounds = _tp_bounds(world_size, uneven) logits, targets = _tp_inputs(device, dtype) tiles = TP_NUM_TILES @@ -467,7 +477,9 @@ def _nccl_worker(rank, world_size, init_method, result_queue, scenario, uneven, dist.destroy_process_group() -def _run_nccl_scenario(world_size, scenario="correctness", uneven=False, dtype_name="fp32"): +def _run_nccl_scenario( + world_size, scenario="correctness", uneven=False, dtype_name="fp32", backend_kind="pytorch" +): ctx = mp.get_context("spawn") with tempfile.TemporaryDirectory() as tmpdir: init_method = (Path(tmpdir) / "nccl_init").as_uri() @@ -475,7 +487,16 @@ def _run_nccl_scenario(world_size, scenario="correctness", uneven=False, dtype_n processes = [ ctx.Process( target=_nccl_worker, - args=(rank, world_size, init_method, result_queue, scenario, uneven, dtype_name), + args=( + rank, + world_size, + init_method, + result_queue, + scenario, + uneven, + dtype_name, + backend_kind, + ), ) for rank in range(world_size) ] @@ -558,3 +579,29 @@ def test_misaligned_shard_bounds_rejected(self): results = _run_nccl_scenario(2, scenario="misaligned") for result in results: assert "not aligned to the vocab tile size" in result["message"] + + +@pytest.mark.skipif(torch.version.hip is None, reason="requires a ROCm PyTorch build") +class TestRocmNativeCrossTP: + """The ROCm production path must preserve the same TP contract as reference.""" + + @staticmethod + def _require_native() -> None: + from rl_engine.kernels.registry import _rocm_vocab_logprob_native_available + + if not _rocm_vocab_logprob_native_available(): + pytest.skip("requires the compiled ROCm logprob extension") + + @_requires_gpus(2) + @pytest.mark.parametrize("dtype_name", ["fp32", "bf16"]) + def test_tp2_native_matches_tp1_and_repeat(self, dtype_name): + self._require_native() + results = _run_nccl_scenario(2, dtype_name=dtype_name, backend_kind="rocm") + TestCrossTPBitwise._assert_matches_tp1(results) + + @_requires_gpus(4) + @pytest.mark.parametrize("dtype_name", ["fp32", "bf16"]) + def test_tp4_native_matches_tp1_and_repeat(self, dtype_name): + self._require_native() + results = _run_nccl_scenario(4, dtype_name=dtype_name, backend_kind="rocm") + TestCrossTPBitwise._assert_matches_tp1(results) From b776227c847983a38a0af406eb470e5768b697a4 Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Sat, 22 Aug 2026 05:24:57 +0800 Subject: [PATCH 34/48] style: apply isort ordering to setup imports --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index dc185132..a0c589ed 100644 --- a/setup.py +++ b/setup.py @@ -3,8 +3,8 @@ import importlib.util import os -import warnings import sysconfig +import warnings from distutils.errors import CompileError from distutils.spawn import find_executable from pathlib import Path From 9a05c1e2c7b85e9e272b78a05ba731854d5253cf Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Sat, 22 Aug 2026 14:31:38 +0800 Subject: [PATCH 35/48] test: isolate ROCm vocab logprob dispatch cases --- tests/test_logprob_contract.py | 8 ++++++-- tests/test_vocab_parallel_logp.py | 2 +- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/tests/test_logprob_contract.py b/tests/test_logprob_contract.py index a74f3b4c..49a84e20 100644 --- a/tests/test_logprob_contract.py +++ b/tests/test_logprob_contract.py @@ -248,13 +248,17 @@ def test_ignore_index_must_not_collide_with_the_real_vocabulary(): def _restrict_to_ws1_candidates(registry: KernelRegistry) -> None: - """Drop the #241 PR3 vocab-parallel reference so only WS1 backends remain.""" + """Drop both WS2 vocab-parallel backends so only WS1 backends remain.""" platform = registry._platform() + ws2_backends = { + OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP, + OpBackend.ROCM_VOCAB_PARALLEL_LOGP, + } registry._logprob_candidates[platform] = [ backend for backend in registry._logprob_candidates[platform] - if backend is not OpBackend.PYTORCH_VOCAB_PARALLEL_LOGP + if backend not in ws2_backends ] diff --git a/tests/test_vocab_parallel_logp.py b/tests/test_vocab_parallel_logp.py index 10639e2b..02aab528 100644 --- a/tests/test_vocab_parallel_logp.py +++ b/tests/test_vocab_parallel_logp.py @@ -286,7 +286,7 @@ def test_dispatch_resolves_reference_and_leaves_legacy_untouched(): registry = KernelRegistry() contract = _contract() - result = registry.get_logprob_op(contract) + result = registry.get_logprob_op(contract, requested_backend="reference") assert result.capability.backend_id == BACKEND_ID assert result.provenance["fallback"] is False assert isinstance(result.op, VocabParallelLogprobOp) From 1c73a427881252871b46936f6d14be36a22e307f Mon Sep 17 00:00:00 2001 From: hihaluemen <1596916766@qq.com> Date: Sat, 22 Aug 2026 14:45:00 +0800 Subject: [PATCH 36/48] style: format ROCm dispatch test with black --- tests/test_logprob_contract.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/tests/test_logprob_contract.py b/tests/test_logprob_contract.py index 49a84e20..60ced1cd 100644 --- a/tests/test_logprob_contract.py +++ b/tests/test_logprob_contract.py @@ -256,9 +256,7 @@ def _restrict_to_ws1_candidates(registry: KernelRegistry) -> None: OpBackend.ROCM_VOCAB_PARALLEL_LOGP, } registry._logprob_candidates[platform] = [ - backend - for backend in registry._logprob_candidates[platform] - if backend not in ws2_backends + backend for backend in registry._logprob_candidates[platform] if backend not in ws2_backends ] From 8e793c9b0d69e04be8629cf09cc354e83487b89e Mon Sep 17 00:00:00 2001 From: KJLdefeated Date: Sun, 23 Aug 2026 05:43:54 +0000 Subject: [PATCH 37/48] logp benchmark --- .gitignore | 7 + .../rocm_logprob_implementation_summary.md | 23 + benchmarks/benchmark_rocm_logp.py | 1609 ++++++++ .../distributed_logp_latency.png | Bin 0 -> 62146 bytes .../results/pr328_rocm_mi300x/report.md | 154 + .../results/pr328_rocm_mi300x/results.json | 3366 +++++++++++++++++ .../pr328_rocm_mi300x/single_gpu_latency.png | Bin 0 -> 72942 bytes .../pr328_rocm_mi300x/single_gpu_memory.png | Bin 0 -> 120156 bytes csrc/hip/hip_deterministic_logp_kernel.hip | 424 +++ csrc/ops.cpp | 24 + rl_engine/_C.pyi | 16 + .../ops/pytorch/loss/vocab_parallel_logp.py | 7 +- .../ops/rocm/loss/vocab_parallel_logp.py | 167 +- setup.py | 11 + tests/test_rocm_logprob_backend.py | 129 + 15 files changed, 5931 insertions(+), 6 deletions(-) create mode 100644 benchmarks/benchmark_rocm_logp.py create mode 100644 benchmarks/results/pr328_rocm_mi300x/distributed_logp_latency.png create mode 100644 benchmarks/results/pr328_rocm_mi300x/report.md create mode 100644 benchmarks/results/pr328_rocm_mi300x/results.json create mode 100644 benchmarks/results/pr328_rocm_mi300x/single_gpu_latency.png create mode 100644 benchmarks/results/pr328_rocm_mi300x/single_gpu_memory.png create mode 100644 csrc/hip/hip_deterministic_logp_kernel.hip diff --git a/.gitignore b/.gitignore index 5c38edec..2ad55aa5 100644 --- a/.gitignore +++ b/.gitignore @@ -213,3 +213,10 @@ _dev_notes/ # Local C8 execute dumps; default output is under TMPDIR. ws1-c8-ci.json + +# hipify outputs generated during ROCm builds (sources live in csrc/*.cu, csrc/cuda/, csrc/hip/hip_*.hip) +csrc/*.hip +csrc/*_hip.cpp +csrc/hip/** +!csrc/hip/hip_*.hip +csrc/hip/*_hip.hip diff --git a/_dev_notes/rocm_logprob_implementation_summary.md b/_dev_notes/rocm_logprob_implementation_summary.md index 3d0d2047..ab699b4b 100644 --- a/_dev_notes/rocm_logprob_implementation_summary.md +++ b/_dev_notes/rocm_logprob_implementation_summary.md @@ -135,3 +135,26 @@ PYTHONHASHSEED=0 pytest -q -ra \ ## 当前提交 实现尚未推送远程；代码位于当前工作分支 `work/ws2-logprob-rocm`。合并基线提交是 `b811abc`，本次实现文件仍在工作区，待 ROCm 环境验证后再拆分成正式 PR commits。 + +## 2026-08-23 补充：ROCm 性能调优 + +在 MI300X 上做完基准测试（`benchmarks/benchmark_rocm_logp.py`，结果见 +`benchmarks/results/pr328_rocm_mi300x/report.md`）后，对 ROCm backend 做了两处调整， +TP contract、tile 顺序 merge、selected-target 传输和 active-mask 语义都没有变： + +1. 新增 ROCm 专用文件 `csrc/hip/hip_deterministic_logp_kernel.hip`（只在 ROCm 构建时编译， + 共享的 `csrc/deterministic_logp_kernel.cu` 保持 SM90 调优版本不动）。其中 + `hip_deterministic_logp_tile_stats` 直接读取 BF16/FP16/FP32 shard（kernel 内部逐元素精确 + 转成 FP32，并自行过滤 padding 列），不再先做一份 FP32 拷贝；每个线程固定处理 8 个连续 + 元素（向量化 load），累加顺序只由 `(BlockSize, Vec)` 决定，与 rank、shard 偏移和存储 + dtype 无关，所以 TP=n 与 TP=1 仍然 bitwise 一致，BF16 直读与 FP32 上转的 partial 也 + bitwise 一致。全 padding 的 tile 现在返回 `(-inf, 0)` identity partial（原来是 `-FLT_MAX`）。 +2. 新增 `hip_deterministic_logp_backward`：从保存的输入 shard 一次 fused pass 生成 + `grad_logits`（`g_logp * (onehot - p) + g_lse * p`，padding 列为 0，非有限行 `p = 0`）， + 替代共享 Python autograd 里约 9 次 `[tokens, vocab]` FP32 elementwise pass。 + `RocmVocabParallelLogprobOp.apply` 走这条 HIP autograd 路径；`apply_with_entropy` + 仍沿用共享路径（带 HIP tile kernel），因为 entropy 梯度本来就需要完整的概率张量。 + +构建期可调参数：`DETERMINISTIC_LOGP_TILE_BLOCK_SIZE`（默认 128）、 +`DETERMINISTIC_LOGP_TILE_VECTOR_ELEMENTS`（默认 8）、`DETERMINISTIC_LOGP_BACKWARD_BLOCK_SIZE` +（默认 256），通过 `setup.py` 的环境变量注入。 diff --git a/benchmarks/benchmark_rocm_logp.py b/benchmarks/benchmark_rocm_logp.py new file mode 100644 index 00000000..5830eb1a --- /dev/null +++ b/benchmarks/benchmark_rocm_logp.py @@ -0,0 +1,1609 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""ROCm benchmark for the WS2 vocab-parallel logprob path (PR #328). + +Operator-only: seeded logits, no checkpoint, tokenizer, or model server. + +Single GPU (TP=1), Qwen3 vocabulary ``V=151936`` (64 tiles of 2374 columns): + +- ``native``: ``torch.logsumexp`` + ``gather`` on FP32 logits (plain PyTorch, + not batch-invariant by contract). +- ``ws1-pytorch`` / ``ws1-triton``: existing single-shard batch-invariant ops. +- ``ws2-reference``: ``pytorch-vocab-parallel-logp-ws2`` (PyTorch tile loop). +- ``ws2-rocm``: ``rocm-vocab-parallel-logp-ws2`` (HIP tile-stats kernel). + +Plus a component table for the HIP ``deterministic_logp_tile_stats`` kernel +against the PyTorch ``_local_tile_stats`` loop it replaces. + +Distributed (one process per GPU, RCCL via ProcessGroupNCCL): + +- ``native``: Megatron-style vocab-parallel logprob (all-reduce MAX, all-reduce + SUM of exp, all-reduce SUM of the owned target logit). +- ``ws2-reference`` and ``ws2-rocm``: the contract-aware WS2 operator with the + fixed tile-order merge; CP ranks shard tokens and never join the merge. + +Every path reports latency (GPU events on a single GPU; synchronized wall +clock and slowest rank per sample when distributed), peak device memory, +FP64 accuracy, repeat bitwise stability, and batch invariance. + +Usage: + python benchmarks/benchmark_rocm_logp.py \ + --warmup 5 --samples 20 --training-samples 10 \ + --output-dir benchmarks/results/pr328_rocm_mi300x +""" + +from __future__ import annotations + +import argparse +import json +import math +import os +import statistics +import tempfile +import time +import traceback +from pathlib import Path +from typing import Any, Callable + +import torch +import torch.distributed as dist +import torch.multiprocessing as mp + +from rl_engine.kernels.logprob_contract import ( + LogprobContract, + MaskSpec, + ReductionSpec, + ShardingSpec, +) +from rl_engine.kernels.ops.base import _C, _EXT_AVAILABLE +from rl_engine.kernels.ops.pytorch.loss.batch_invariant_logp import NativeBatchInvariantLogpOp +from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import ( + VocabParallelLogprobOp, + _local_tile_stats, +) +from rl_engine.kernels.ops.rocm.loss.vocab_parallel_logp import RocmVocabParallelLogprobOp + +REAL_VOCAB = 151936 # Qwen3 tokenizer/lm_head width; 151936 = 64 * 2374, so no padding. +NUM_TILES = 64 +IGNORE_INDEX = -100 +LOGIT_SCALE = 2.0 +SINGLE_TOKENS = (1, 8, 32, 128, 512, 2048) +SINGLE_DTYPES = ("bf16", "fp32") +DISTRIBUTED_TOKENS = (256, 2048) +TOPOLOGIES = ( + ("tp2", 2, 1), + ("tp4", 4, 1), + ("tp8", 8, 1), + ("tp2_cp2", 2, 2), + ("tp4_cp2", 4, 2), + ("tp2_cp4", 2, 4), +) +DISTRIBUTED_PATHS = ("native", "ws2-reference", "ws2-rocm") +_DTYPES = {"bf16": torch.bfloat16, "fp32": torch.float32} +_SPAWN_TIMEOUT_S = 1800 + + +# --------------------------------------------------------------------------- helpers + + +def _percentile(values: list[float], percentile: float) -> float: + ordered = sorted(values) + if not ordered: + return float("nan") + position = (len(ordered) - 1) * percentile + lower = math.floor(position) + upper = math.ceil(position) + if lower == upper: + return ordered[lower] + weight = position - lower + return ordered[lower] * (1.0 - weight) + ordered[upper] * weight + + +def _summary_ms(values: list[float]) -> dict[str, float]: + return { + "median_ms": statistics.median(values), + "p95_ms": _percentile(values, 0.95), + "min_ms": min(values), + "max_ms": max(values), + } + + +def _relative_l2(actual: torch.Tensor, expected: torch.Tensor) -> float: + actual_float = actual.detach().double() + expected_float = expected.detach().double() + denominator = torch.linalg.vector_norm(expected_float) + if denominator.item() == 0.0: + return float(torch.linalg.vector_norm(actual_float - expected_float).item()) + return float((torch.linalg.vector_norm(actual_float - expected_float) / denominator).item()) + + +def _accuracy(actual: torch.Tensor, expected: torch.Tensor) -> dict[str, float]: + difference = actual.detach().double() - expected.detach().double() + return { + "max_abs": float(difference.abs().max().item()) if difference.numel() else 0.0, + "relative_l2": _relative_l2(actual, expected), + } + + +def _bits(tensor: torch.Tensor) -> torch.Tensor: + return tensor.detach().float().contiguous().view(torch.int32) + + +def _bitwise_equal(a: torch.Tensor, b: torch.Tensor) -> bool: + return a.shape == b.shape and bool(torch.equal(_bits(a), _bits(b))) + + +def _mismatch_count(a: torch.Tensor, b: torch.Tensor) -> int: + return int((_bits(a) != _bits(b)).sum().item()) + + +def _gpu_event_samples(function: Callable[[], Any], *, warmup: int, samples: int) -> list[float]: + for _ in range(warmup): + function() + torch.cuda.synchronize() + events = [] + for _ in range(samples): + start = torch.cuda.Event(enable_timing=True) + end = torch.cuda.Event(enable_timing=True) + start.record() + function() + end.record() + events.append((start, end)) + torch.cuda.synchronize() + return [float(start.elapsed_time(end)) for start, end in events] + + +def _peak_memory_mib(function: Callable[[], Any]) -> float: + """Peak device memory allocated by one call, above what was live before it.""" + torch.cuda.synchronize() + torch.cuda.empty_cache() + torch.cuda.reset_peak_memory_stats() + baseline = torch.cuda.memory_allocated() + function() + torch.cuda.synchronize() + return float((torch.cuda.max_memory_allocated() - baseline) / (1024.0 * 1024.0)) + + +def _seeded_logits( + num_tokens: int, vocab: int, *, seed: int, device: torch.device +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """Identical FP32 logits, targets, and active mask on every rank. + + Generated on-device from a seeded CUDA generator so multi-GB inputs are not + materialized on the host; the same seed yields identical values on every + MI300X rank. + """ + generator = torch.Generator(device=device).manual_seed(seed) + logits = torch.randn(num_tokens, vocab, generator=generator, device=device) * LOGIT_SCALE + targets = torch.randint(0, vocab, (num_tokens,), generator=generator, device=device) + active = (torch.arange(num_tokens, device=device) % 7) != 5 + return logits, targets, active + + +def _fp64_oracle( + logits_fp32: torch.Tensor, targets: torch.Tensor, active: torch.Tensor, real_vocab: int +) -> tuple[torch.Tensor, torch.Tensor]: + z = logits_fp32[:, :real_vocab].double() + lse = torch.logsumexp(z, dim=-1) + safe = torch.where(active, targets, torch.zeros_like(targets)) + selected = z.gather(1, safe.unsqueeze(1)).squeeze(1) + logp = torch.where(active, selected - lse, torch.zeros_like(lse)) + return logp, lse + + +def _contract( + *, + num_tokens: int, + active: tuple[bool, ...], + tp_rank: int, + tp_world_size: int, + bounds: tuple[tuple[int, int], ...], + real_vocab: int, + padded_vocab: int, + dtype: str, + cp_rank: int = 0, + cp_world_size: int = 1, +) -> LogprobContract: + return LogprobContract( + role="train", + dtype=dtype, + mask=MaskSpec(num_tokens=num_tokens, active_mask=active), + sharding=ShardingSpec( + tp_rank=tp_rank, + tp_world_size=tp_world_size, + vocab_shard_bounds=bounds, + real_vocab_size=real_vocab, + padded_vocab_size=padded_vocab, + cp_rank=cp_rank, + cp_world_size=cp_world_size, + ), + reduction=ReductionSpec(), + ) + + +# --------------------------------------------------------------------------- single GPU + + +def _native_logp( + logits: torch.Tensor, targets: torch.Tensor, active: torch.Tensor +) -> tuple[torch.Tensor, torch.Tensor]: + z = logits.float() + lse = torch.logsumexp(z, dim=-1) + safe = torch.where(active, targets, torch.zeros_like(targets)) + selected = z.gather(1, safe.unsqueeze(1)).squeeze(1) + logp = torch.where(active, selected - lse, torch.zeros_like(lse)) + return logp, lse + + +def _single_gpu_paths( + device: torch.device, +) -> dict[str, tuple[Callable[..., Any], Callable[..., Any]]]: + ws1_pytorch = NativeBatchInvariantLogpOp() + ws2_reference = VocabParallelLogprobOp() + ws2_rocm = RocmVocabParallelLogprobOp() + + def ws1_pytorch_fn(logits, targets, active, contract): + ignore_targets = torch.where(active, targets, torch.full_like(targets, IGNORE_INDEX)) + return ws1_pytorch.forward_with_lse(logits, ignore_targets, IGNORE_INDEX, validate=False) + + def ws1_pytorch_train(logits, targets, active, contract): + ignore_targets = torch.where(active, targets, torch.full_like(targets, IGNORE_INDEX)) + return ws1_pytorch.apply(logits, ignore_targets, IGNORE_INDEX, validate=False) + + def ws2_reference_fn(logits, targets, active, contract): + return ws2_reference.apply( + logits, targets, contract=contract, num_vocab_tiles=NUM_TILES, validate=False + ) + + def ws2_rocm_fn(logits, targets, active, contract): + return ws2_rocm.apply( + logits, targets, contract=contract, num_vocab_tiles=NUM_TILES, validate=False + ) + + # path -> (forward returning (logp, lse), training forward returning logp with autograd) + paths: dict[str, tuple[Callable[..., Any], Callable[..., Any]]] = { + "native": ( + lambda logits, targets, active, contract: _native_logp(logits, targets, active), + lambda logits, targets, active, contract: _native_logp(logits, targets, active)[0], + ), + "ws1-pytorch": (ws1_pytorch_fn, ws1_pytorch_train), + } + try: + from rl_engine.kernels.ops.triton.loss.batch_invariant_logp import ( + TritonBatchInvariantLogpOp, + ) + + ws1_triton = TritonBatchInvariantLogpOp() + probe = torch.randn(2, 256, device=device, dtype=torch.bfloat16) + ws1_triton.forward_with_lse(probe, torch.zeros(2, device=device, dtype=torch.long)) + torch.cuda.synchronize() + + def ws1_triton_fn(logits, targets, active, contract): + ignore_targets = torch.where(active, targets, torch.full_like(targets, IGNORE_INDEX)) + return ws1_triton.forward_with_lse(logits, ignore_targets, IGNORE_INDEX) + + def ws1_triton_train(logits, targets, active, contract): + ignore_targets = torch.where(active, targets, torch.full_like(targets, IGNORE_INDEX)) + return ws1_triton.apply(logits, ignore_targets, IGNORE_INDEX) + + paths["ws1-triton"] = (ws1_triton_fn, ws1_triton_train) + except Exception as exc: # pragma: no cover - environment dependent + print(f"ws1-triton unavailable: {exc}") + paths["ws2-reference"] = ( + ws2_reference_fn, + lambda *a: ws2_reference_fn(*a)[0], + ) + paths["ws2-rocm"] = (ws2_rocm_fn, lambda *a: ws2_rocm_fn(*a)[0]) + return paths + + +def _single_gpu_benchmarks( + *, warmup: int, samples: int, training_samples: int, tokens: tuple[int, ...] +) -> dict[str, Any]: + device = torch.device("cuda", 0) + torch.cuda.set_device(device) + paths = _single_gpu_paths(device) + bounds = ((0, REAL_VOCAB),) + cases: list[dict[str, Any]] = [] + validate_overhead: list[dict[str, Any]] = [] + ws2_rocm_op = RocmVocabParallelLogprobOp() + + for dtype_name in SINGLE_DTYPES: + dtype = _DTYPES[dtype_name] + for num_tokens in tokens: + logits_fp32, targets, active = _seeded_logits( + num_tokens, REAL_VOCAB, seed=2026 + num_tokens, device=device + ) + logits = logits_fp32.to(dtype).contiguous() + oracle_logp, oracle_lse = _fp64_oracle(logits.float(), targets, active, REAL_VOCAB) + logits_fp32 = None + contract = _contract( + num_tokens=num_tokens, + active=tuple(bool(flag) for flag in active.tolist()), + tp_rank=0, + tp_world_size=1, + bounds=bounds, + real_vocab=REAL_VOCAB, + padded_vocab=REAL_VOCAB, + dtype=dtype_name, + ) + row = min(3, num_tokens - 1) + row_contract = _contract( + num_tokens=1, + active=(bool(active[row].item()),), + tp_rank=0, + tp_world_size=1, + bounds=bounds, + real_vocab=REAL_VOCAB, + padded_vocab=REAL_VOCAB, + dtype=dtype_name, + ) + outputs: dict[str, tuple[torch.Tensor, torch.Tensor]] = {} + for path_name, (path, train_path) in paths.items(): + + def forward(): + return path(logits, targets, active, contract) + + def train_step(): + leaf = logits.detach().clone().requires_grad_(True) + logp = train_path(leaf, targets, active, contract) + (logp * active).sum().backward() + return leaf.grad + + try: + first_logp, first_lse = forward() + second_logp, second_lse = forward() + row_logp, row_lse = path( + logits[row : row + 1].contiguous(), + targets[row : row + 1], + active[row : row + 1], + row_contract, + ) + train_step() + torch.cuda.synchronize() + except Exception as exc: + print(f"{path_name} failed for {dtype_name} M={num_tokens}: {exc}") + continue + outputs[path_name] = (first_logp.detach(), first_lse.detach()) + forward_times = _gpu_event_samples(forward, warmup=warmup, samples=samples) + train_times = _gpu_event_samples( + train_step, warmup=max(1, warmup // 2), samples=training_samples + ) + grad = train_step() + torch.cuda.synchronize() + active_idx = active.nonzero().squeeze(1) + cases.append( + { + "dtype": dtype_name, + "tokens": num_tokens, + "path": path_name, + "forward": _summary_ms(forward_times), + "train_fwd_bwd": _summary_ms(train_times), + "forward_peak_mib": _peak_memory_mib(forward), + "train_peak_mib": _peak_memory_mib(train_step), + "logp_vs_fp64": _accuracy(first_logp[active_idx], oracle_logp[active_idx]), + "lse_vs_fp64": _accuracy(first_lse, oracle_lse), + "repeat_bitwise": _bitwise_equal(first_logp, second_logp) + and _bitwise_equal(first_lse, second_lse), + "batch_invariant": _bitwise_equal(row_logp[0], first_logp[row]) + and _bitwise_equal(row_lse[0], first_lse[row]), + "grad_finite": bool(torch.isfinite(grad).all().item()), + } + ) + print( + f"single {dtype_name} M={num_tokens:5d} {path_name:14s} " + f"fwd={cases[-1]['forward']['median_ms']:.4f}ms " + f"train={cases[-1]['train_fwd_bwd']['median_ms']:.4f}ms " + f"peak={cases[-1]['train_peak_mib']:.1f}MiB", + flush=True, + ) + if "ws2-reference" in outputs and "ws2-rocm" in outputs: + ref_logp, ref_lse = outputs["ws2-reference"] + rocm_logp, rocm_lse = outputs["ws2-rocm"] + for case in cases: + if case["dtype"] == dtype_name and case["tokens"] == num_tokens: + if case["path"] == "ws2-rocm": + case["mismatch_vs_reference"] = _mismatch_count( + rocm_logp, ref_logp + ) + _mismatch_count(rocm_lse, ref_lse) + case["rel_l2_vs_reference"] = max( + _relative_l2(rocm_logp, ref_logp), _relative_l2(rocm_lse, ref_lse) + ) + # validate=True production entry point overhead (host-side checks + .item() sync) + if dtype_name == "bf16": + + def validated(): + return ws2_rocm_op.apply( + logits, targets, contract=contract, num_vocab_tiles=NUM_TILES, validate=True + ) + + def unvalidated(): + return ws2_rocm_op.apply( + logits, + targets, + contract=contract, + num_vocab_tiles=NUM_TILES, + validate=False, + ) + + validate_overhead.append( + { + "tokens": num_tokens, + "validate_true": _summary_ms( + _gpu_event_samples(validated, warmup=warmup, samples=samples) + ), + "validate_false": _summary_ms( + _gpu_event_samples(unvalidated, warmup=warmup, samples=samples) + ), + } + ) + logits = oracle_logp = oracle_lse = outputs = None + torch.cuda.empty_cache() + + return {"cases": cases, "validate_overhead": validate_overhead, "paths": list(paths)} + + +def _tile_stats_component( + *, warmup: int, samples: int, tokens: tuple[int, ...] +) -> list[dict[str, Any]]: + """HIP tile-stats kernel versus the PyTorch tile loop it replaces.""" + device = torch.device("cuda", 0) + rows: list[dict[str, Any]] = [] + tile = REAL_VOCAB // NUM_TILES + for dtype_name in SINGLE_DTYPES: + dtype = _DTYPES[dtype_name] + for num_tokens in tokens: + logits_fp32, _, _ = _seeded_logits( + num_tokens, REAL_VOCAB, seed=99 + num_tokens, device=device + ) + logits = logits_fp32.to(dtype).contiguous() + z32 = logits.float() + logits_fp32 = None + + def pytorch_loop(): + return _local_tile_stats(z32, tile) + + tile_stats = getattr( + _C, "hip_deterministic_logp_tile_stats", _C.deterministic_logp_tile_stats + ) + + def hip_kernel_fp32(): + return tile_stats(z32, 0, REAL_VOCAB, NUM_TILES) + + def hip_kernel_input_dtype(): + return tile_stats(logits, 0, REAL_VOCAB, NUM_TILES) + + ref_m, ref_s = pytorch_loop() + hip_m, hip_s = hip_kernel_fp32() + hip_m2, hip_s2 = hip_kernel_fp32() + rows.append( + { + "dtype": dtype_name, + "tokens": num_tokens, + "pytorch_loop": _summary_ms( + _gpu_event_samples(pytorch_loop, warmup=warmup, samples=samples) + ), + "hip_fp32_input": _summary_ms( + _gpu_event_samples(hip_kernel_fp32, warmup=warmup, samples=samples) + ), + "hip_native_dtype_input": _summary_ms( + _gpu_event_samples(hip_kernel_input_dtype, warmup=warmup, samples=samples) + ), + "pytorch_loop_peak_mib": _peak_memory_mib(pytorch_loop), + "hip_peak_mib": _peak_memory_mib(hip_kernel_fp32), + "max_bitwise": _bitwise_equal(hip_m, ref_m), + "sumexp_rel_l2": _relative_l2(hip_s, ref_s), + "sumexp_max_rel": float( + ((hip_s - ref_s).abs() / ref_s.abs().clamp_min(1e-30)).max().item() + ), + "repeat_bitwise": _bitwise_equal(hip_m, hip_m2) + and _bitwise_equal(hip_s, hip_s2), + } + ) + print( + f"tile-stats {dtype_name} M={num_tokens:5d} " + f"loop={rows[-1]['pytorch_loop']['median_ms']:.4f}ms " + f"hip={rows[-1]['hip_fp32_input']['median_ms']:.4f}ms", + flush=True, + ) + logits = z32 = None + torch.cuda.empty_cache() + return rows + + +# --------------------------------------------------------------------------- distributed + + +class _NativeVocabParallelLogp(torch.autograd.Function): + """Megatron-style vocab-parallel logprob with RCCL all-reduce, no fixed order.""" + + @staticmethod + def forward(ctx, local_logits, targets, active, vocab_start, group): + z = local_logits.float() + local_vocab = z.shape[1] + global_max = z.max(dim=-1).values + dist.all_reduce(global_max, op=dist.ReduceOp.MAX, group=group) + sum_exp = (z - global_max.unsqueeze(1)).exp().sum(dim=-1) + dist.all_reduce(sum_exp, op=dist.ReduceOp.SUM, group=group) + lse = global_max + sum_exp.log() + owned = active & (targets >= vocab_start) & (targets < vocab_start + local_vocab) + local_index = torch.where(owned, targets - vocab_start, torch.zeros_like(targets)) + selected = z.gather(1, local_index.unsqueeze(1)).squeeze(1) * owned + dist.all_reduce(selected, op=dist.ReduceOp.SUM, group=group) + logp = torch.where(active, selected - lse, torch.zeros_like(lse)) + ctx.save_for_backward(z, lse, local_index, owned, active) + ctx.input_dtype = local_logits.dtype + ctx.set_materialize_grads(False) + return logp, lse + + @staticmethod + def backward(ctx, grad_logp, grad_lse): + z, lse, local_index, owned, active = ctx.saved_tensors + probabilities = (z - lse.unsqueeze(1)).exp() + scale = torch.zeros_like(lse) + if grad_logp is not None: + scale = scale - grad_logp * active + if grad_lse is not None: + scale = scale + grad_lse + grad = probabilities * scale.unsqueeze(1) + if grad_logp is not None: + grad.scatter_add_( + 1, local_index.unsqueeze(1), (grad_logp * owned).unsqueeze(1).to(grad.dtype) + ) + return grad.to(ctx.input_dtype), None, None, None, None + + +def _tp_bounds(tp_world_size: int) -> tuple[tuple[int, int], ...]: + tile = REAL_VOCAB // NUM_TILES + per_rank = NUM_TILES // tp_world_size + return tuple( + (rank * per_rank * tile, (rank + 1) * per_rank * tile) for rank in range(tp_world_size) + ) + + +def _token_bounds(num_tokens: int, cp_world_size: int) -> tuple[tuple[int, int], ...]: + quotient, remainder = divmod(num_tokens, cp_world_size) + bounds, cursor = [], 0 + for cp_rank in range(cp_world_size): + count = quotient + int(cp_rank < remainder) + bounds.append((cursor, cursor + count)) + cursor += count + return tuple(bounds) + + +def _distributed_wall_samples( + function: Callable[[], Any], *, warmup: int, samples: int +) -> list[float]: + for _ in range(warmup): + function() + torch.cuda.synchronize() + dist.barrier() + timings = [] + for _ in range(samples): + torch.cuda.synchronize() + start = time.perf_counter() + function() + torch.cuda.synchronize() + timings.append((time.perf_counter() - start) * 1000.0) + dist.barrier() + return timings + + +def _slowest_rank_summary(local_timings: list[float]) -> dict[str, float]: + gathered: list[list[float] | None] = [None] * dist.get_world_size() + dist.all_gather_object(gathered, local_timings) + slowest = [ + max(float(rank_timings[index]) for rank_timings in gathered if rank_timings is not None) + for index in range(len(local_timings)) + ] + return _summary_ms(slowest) + + +def _all_max(value: float) -> float: + gathered: list[float | None] = [None] * dist.get_world_size() + dist.all_gather_object(gathered, float(value)) + return max(float(item) for item in gathered if item is not None) + + +def _all_all(flag: bool) -> bool: + gathered: list[bool | None] = [None] * dist.get_world_size() + dist.all_gather_object(gathered, bool(flag)) + return all(bool(item) for item in gathered) + + +def _all_sum(value: int) -> int: + gathered: list[int | None] = [None] * dist.get_world_size() + dist.all_gather_object(gathered, int(value)) + return sum(int(item) for item in gathered if item is not None) + + +def _tp_replicated( + logp: torch.Tensor, lse: torch.Tensor, tp_group: Any, tp_world_size: int +) -> bool: + if tp_world_size == 1: + return True + payload = torch.stack([_bits(logp), _bits(lse)]) + gathered = [torch.empty_like(payload) for _ in range(tp_world_size)] + dist.all_gather(gathered, payload, group=tp_group) + return all(torch.equal(gathered[0], other) for other in gathered[1:]) + + +def _distributed_worker( + rank: int, + world_size: int, + init_method: str, + topology: tuple[str, int, int], + tokens_list: tuple[int, ...], + config: dict[str, Any], + result_queue: Any, +) -> None: + name, tp_world_size, cp_world_size = topology + try: + torch.cuda.set_device(rank) + device = torch.device("cuda", rank) + dist.init_process_group( + backend="nccl", + init_method=init_method, + rank=rank, + world_size=world_size, + device_id=device, + ) + cp_rank, tp_rank = divmod(rank, tp_world_size) + tp_group = None + for group_cp_rank in range(cp_world_size): + ranks = list(range(group_cp_rank * tp_world_size, (group_cp_rank + 1) * tp_world_size)) + group = dist.new_group(ranks=ranks) + if rank in ranks: + tp_group = group + bounds = _tp_bounds(tp_world_size) + vocab_start, vocab_end = bounds[tp_rank] + ops = { + "ws2-reference": VocabParallelLogprobOp(), + "ws2-rocm": RocmVocabParallelLogprobOp(), + } + results: list[dict[str, Any]] = [] + + for num_tokens in tokens_list: + full_logits, full_targets, full_active = _seeded_logits( + num_tokens, REAL_VOCAB, seed=4100 + num_tokens, device=device + ) + token_start, token_end = _token_bounds(num_tokens, cp_world_size)[cp_rank] + local_tokens = token_end - token_start + targets = full_targets[token_start:token_end].contiguous() + active = full_active[token_start:token_end].contiguous() + local_fp32 = full_logits[token_start:token_end] + shard = local_fp32[:, vocab_start:vocab_end].to(torch.bfloat16).contiguous() + # The FP64 oracle sees the BF16-rounded logits every path actually consumes. + oracle_logp, oracle_lse = _fp64_oracle( + local_fp32.to(torch.bfloat16).float(), targets, active, REAL_VOCAB + ) + full_logits = local_fp32 = None + torch.cuda.empty_cache() + contract = _contract( + num_tokens=local_tokens, + active=tuple(bool(flag) for flag in active.tolist()), + tp_rank=tp_rank, + tp_world_size=tp_world_size, + bounds=bounds, + real_vocab=REAL_VOCAB, + padded_vocab=REAL_VOCAB, + dtype="bf16", + cp_rank=cp_rank, + cp_world_size=cp_world_size, + ) + outputs: dict[str, tuple[torch.Tensor, torch.Tensor]] = {} + for path_name in DISTRIBUTED_PATHS: + if path_name == "native": + + def forward(x=shard): + return _NativeVocabParallelLogp.apply( + x, targets, active, vocab_start, tp_group + ) + + else: + op = ops[path_name] + + def forward(x=shard, op=op): + return op.apply( + x, + targets, + contract=contract, + tp_group=tp_group, + num_vocab_tiles=NUM_TILES, + validate=False, + ) + + def train_step(): + leaf = shard.detach().clone().requires_grad_(True) + logp, _ = forward(leaf) + (logp * active).sum().backward() + return leaf.grad + + first_logp, first_lse = forward() + second_logp, second_lse = forward() + torch.cuda.synchronize() + outputs[path_name] = (first_logp.detach(), first_lse.detach()) + forward_times = _distributed_wall_samples( + forward, warmup=config["warmup"], samples=config["samples"] + ) + train_times = _distributed_wall_samples( + train_step, + warmup=max(1, config["warmup"] // 2), + samples=config["training_samples"], + ) + grad = train_step() + torch.cuda.synchronize() + active_idx = active.nonzero().squeeze(1) + logp_acc = _accuracy(first_logp[active_idx], oracle_logp[active_idx]) + lse_acc = _accuracy(first_lse, oracle_lse) + entry = { + "topology": name, + "tp": tp_world_size, + "cp": cp_world_size, + "tokens": num_tokens, + "tokens_per_cp_rank": local_tokens, + "local_vocab": vocab_end - vocab_start, + "path": path_name, + "forward": _slowest_rank_summary(forward_times), + "train_fwd_bwd": _slowest_rank_summary(train_times), + "forward_peak_mib": _all_max(_peak_memory_mib(forward)), + "train_peak_mib": _all_max(_peak_memory_mib(train_step)), + "logp_vs_fp64_max_abs": _all_max(logp_acc["max_abs"]), + "logp_vs_fp64_rel_l2": _all_max(logp_acc["relative_l2"]), + "lse_vs_fp64_max_abs": _all_max(lse_acc["max_abs"]), + "tp_replicated": _all_all( + _tp_replicated(first_logp, first_lse, tp_group, tp_world_size) + ), + "repeat_bitwise": _all_all( + _bitwise_equal(first_logp, second_logp) + and _bitwise_equal(first_lse, second_lse) + ), + "grad_finite": _all_all(bool(torch.isfinite(grad).all().item())), + } + results.append(entry) + if rank == 0: + print( + f"dist {name} M={num_tokens:5d} {path_name:14s} " + f"fwd={entry['forward']['median_ms']:.4f}ms " + f"train={entry['train_fwd_bwd']['median_ms']:.4f}ms " + f"peak={entry['train_peak_mib']:.1f}MiB", + flush=True, + ) + ref_logp, ref_lse = outputs["ws2-reference"] + rocm_logp, rocm_lse = outputs["ws2-rocm"] + nat_logp, nat_lse = outputs["native"] + mismatch = _mismatch_count(rocm_logp, ref_logp) + _mismatch_count(rocm_lse, ref_lse) + rel = max(_relative_l2(rocm_logp, ref_logp), _relative_l2(rocm_lse, ref_lse)) + native_rel = max(_relative_l2(nat_logp, ref_logp), _relative_l2(nat_lse, ref_lse)) + # Count once per TP group (outputs are replicated inside the group). + mismatch_total = _all_sum(mismatch if tp_rank == 0 else 0) + rel_max = _all_max(rel) + native_rel_max = _all_max(native_rel) + for entry in results: + if entry["tokens"] == num_tokens and entry["path"] == "ws2-rocm": + entry["mismatch_vs_reference"] = mismatch_total + entry["rel_l2_vs_reference"] = rel_max + if entry["tokens"] == num_tokens and entry["path"] == "native": + entry["rel_l2_vs_reference"] = native_rel_max + shard = oracle_logp = oracle_lse = outputs = None + torch.cuda.empty_cache() + dist.barrier() + if rank == 0: + result_queue.put({"ok": True, "results": results}) + except Exception: + result_queue.put({"ok": False, "rank": rank, "traceback": traceback.format_exc()}) + raise + finally: + if dist.is_initialized(): + dist.destroy_process_group() + + +def _run_distributed_world( + topology: tuple[str, int, int], tokens_list: tuple[int, ...], config: dict[str, Any] +) -> list[dict[str, Any]]: + name, tp_world_size, cp_world_size = topology + world_size = tp_world_size * cp_world_size + context = mp.get_context("spawn") + with tempfile.TemporaryDirectory() as tmpdir: + init_method = (Path(tmpdir) / "rccl_init").as_uri() + result_queue = context.Queue() + processes = [ + context.Process( + target=_distributed_worker, + args=(rank, world_size, init_method, topology, tokens_list, config, result_queue), + ) + for rank in range(world_size) + ] + for process in processes: + process.start() + try: + payload = result_queue.get(timeout=_SPAWN_TIMEOUT_S) + finally: + for process in processes: + process.join(timeout=120) + if process.is_alive(): + process.terminate() + if not payload.get("ok"): + raise RuntimeError(f"{name} rank {payload.get('rank')} failed:\n{payload.get('traceback')}") + return payload["results"] + + +# --------------------------------------------------------------------------- report + + +PATH_DESCRIPTIONS = { + "native": ( + "`torch.logsumexp` + `gather` on FP32 logits (plain PyTorch, not batch-invariant " + "by contract)" + ), + "ws1-pytorch": ( + "`pytorch-batch-invariant-logp-ws1`, the single-shard batch-invariant PyTorch op" + ), + "ws1-triton": "`triton-batch-invariant-logp-ws1`, the single-shard Triton online-softmax op", + "ws2-reference": ( + "`pytorch-vocab-parallel-logp-ws2`, the WS2 vocab-parallel reference operator: a PyTorch " + "tile loop for the per-tile FP32 `(max, sumexp)` partials, all-gather of the partials, " + "fixed global tile-order merge, and a PyTorch autograd backward" + ), + "ws2-rocm": ( + "`rocm-vocab-parallel-logp-ws2`, the same contract, transport, and merge with two HIP " + "kernels: `hip_deterministic_logp_tile_stats` reads the stored BF16/FP16/FP32 shard " + "directly (8-element vector loads, no FP32 copy) and `hip_deterministic_logp_backward` " + "produces the gradient in one fused pass" + ), +} +DISTRIBUTED_DESCRIPTIONS = { + "native": ( + "`native` is a Megatron-style vocab-parallel logprob using RCCL all-reduce (MAX, SUM of " + "exp, SUM of the owned target logit) through ProcessGroupNCCL" + ), + "ws2-reference": ( + "the WS2 operators all-gather per-tile `(max, sumexp)` partials over RCCL and merge them " + "in fixed global tile order; CP ranks shard tokens and never enter the merge" + ), + "ws2-rocm": "", +} + + +class ReportStyle: + """Which measured paths a report shows, how it names them, and what it compares against.""" + + def __init__( + self, + *, + paths: tuple[str, ...], + names: dict[str, str], + baseline: str, + table_tokens: tuple[int, ...] | None, + show_tuning_baseline: bool, + command: str, + ) -> None: + if baseline not in paths: + raise ValueError( + f"report baseline {baseline!r} is not among the reported paths {paths}" + ) + self.paths = paths + self.names = names + self.baseline = baseline + self.table_tokens = table_tokens + self.show_tuning_baseline = show_tuning_baseline + self.command = command + + def name(self, path: str) -> str: + return self.names.get(path, path) + + def keep_tokens(self, tokens: int) -> bool: + return self.table_tokens is None or tokens in self.table_tokens + + +def _fmt_ratio(numerator: float, denominator: float) -> str: + if denominator <= 0: + return "n/a" + return f"{numerator / denominator:.2f}×" + + +def _median_ratio( + numerator: dict[str, Any] | None, denominator: dict[str, Any] | None, key: str +) -> str: + if numerator is None or denominator is None: + return "n/a" + return _fmt_ratio(numerator[key]["median_ms"], denominator[key]["median_ms"]) + + +def _field_ratio( + numerator: dict[str, Any] | None, denominator: dict[str, Any] | None, key: str +) -> str: + if numerator is None or denominator is None: + return "n/a" + return _fmt_ratio(float(numerator[key]), float(denominator[key])) + + +def _range_text(values: list[float], fmt: str = "{:.1f}") -> str: + if not values: + return "n/a" + low, high = min(values), max(values) + if math.isclose(low, high, rel_tol=1e-3): + return fmt.format(low) + return f"{fmt.format(low)}-{fmt.format(high)}" + + +def _lookup(cases: list[dict[str, Any]], **match: Any) -> dict[str, Any] | None: + for case in cases: + if all(case.get(key) == value for key, value in match.items()): + return case + return None + + +def _yes(flag: Any) -> str: + return "yes" if flag else "no" + + +def _write_report(payload: dict[str, Any], output_directory: Path, style: ReportStyle) -> None: + env = payload["environment"] + cfg = payload["config"] + single = [c for c in payload["single_gpu"]["cases"] if c["path"] in style.paths] + component = payload["tile_stats_component"] + distributed = [d for d in payload["distributed"] if d["path"] in style.paths] + base = style.baseline + base_name = style.name(base) + others = [path for path in style.paths if path != base] + lines: list[str] = [] + add = lines.append + + add("# PR #328 ROCm vocab-parallel logprob performance analysis") + add("") + add("> Operator-only benchmark. No model checkpoint or serving engine was used.") + add("") + add("## Environment") + add("") + add("| Item | Value |") + add("|---|---|") + for key in sorted(env): + add(f"| {key} | {env[key]} |") + add("") + add("## Methodology") + add("") + add( + f"- Qwen3 vocabulary `V={REAL_VOCAB}` split into {NUM_TILES} tiles of " + f"{REAL_VOCAB // NUM_TILES} columns; seeded logits (`randn * {LOGIT_SCALE}`), " + "random targets, every seventh token inactive." + ) + add("- Measured paths:") + for path in style.paths: + add(f" - `{style.name(path)}`: {PATH_DESCRIPTIONS[path]}.") + dist_notes = [DISTRIBUTED_DESCRIPTIONS[p] for p in style.paths if DISTRIBUTED_DESCRIPTIONS[p]] + if dist_notes: + add("- Distributed: one process per GPU; " + "; ".join(dist_notes) + ".") + add( + "- Forward returns the selected-token logprob and the vocabulary LSE; forward+backward " + "computes `grad_logits` for `sum(active * logp)`. The WS2 operators run with " + "`validate=False`; the `validate=True` production entry point is measured separately." + ) + add( + "- Single-GPU timing: GPU events, median and p95. Distributed timing: synchronized wall " + "clock, slowest rank per sample. Peak memory is the per-call increase in " + "`torch.cuda.max_memory_allocated` (distributed: max over ranks)." + ) + add( + "- Accuracy is against an FP64 `logsumexp` of the same (BF16-rounded) logits. Repeat = " + "two identical calls are bitwise equal; batch-invariant = a row computed alone is " + "bitwise equal to the same row inside the batch; TP-replicated = every TP rank holds " + "identical bits." + ) + add( + f"- {cfg['warmup']} warmups, {cfg['samples']} measured forward samples, " + f"{cfg['training_samples']} measured forward+backward samples. Raw medians, p95, " + "minimum, maximum, and every measured path are in `results.json`." + ) + if style.table_tokens is not None: + add( + "- Tables show " + + ", ".join(f"{t}" for t in style.table_tokens) + + " tokens; the figures cover the full token sweep." + ) + add("") + add("Reproduce this report from the repository root:") + add("") + add("```bash") + add(style.command) + add("```") + add("") + + # ---- key findings + add("## Key findings") + add("") + for path in others: + name = style.name(path) + speed_fwd, speed_train, mem = [], [], [] + for dtype_name in SINGLE_DTYPES: + for case in single: + if case["dtype"] != dtype_name or case["path"] != path: + continue + if not style.keep_tokens(case["tokens"]): + continue + ref = _lookup(single, dtype=dtype_name, tokens=case["tokens"], path=base) + if ref is None: + continue + speed_fwd.append(ref["forward"]["median_ms"] / case["forward"]["median_ms"]) + speed_train.append( + ref["train_fwd_bwd"]["median_ms"] / case["train_fwd_bwd"]["median_ms"] + ) + mem.append(case["train_peak_mib"] / max(ref["train_peak_mib"], 1e-6)) + if speed_fwd: + add( + f"- Single GPU: `{name}` is {_range_text(speed_fwd, '{:.2f}')}x faster than " + f"`{base_name}` in forward and {_range_text(speed_train, '{:.2f}')}x in " + f"forward+backward, with {_range_text(mem, '{:.2f}')}x its peak memory." + ) + d_fwd, d_train, d_mem, d_abs = [], [], [], [] + for d in distributed: + if d["path"] != path or not style.keep_tokens(d["tokens"]): + continue + ref = _lookup(distributed, topology=d["topology"], tokens=d["tokens"], path=base) + if ref is None: + continue + d_fwd.append(ref["forward"]["median_ms"] / d["forward"]["median_ms"]) + d_train.append(ref["train_fwd_bwd"]["median_ms"] / d["train_fwd_bwd"]["median_ms"]) + d_mem.append(d["train_peak_mib"] / max(ref["train_peak_mib"], 1e-6)) + d_abs.append(d["forward"]["median_ms"]) + if d_fwd: + add( + f"- Distributed: `{name}` is {_range_text(d_fwd, '{:.2f}')}x faster than " + f"`{base_name}` in forward and {_range_text(d_train, '{:.2f}')}x in " + f"forward+backward across {len({d['topology'] for d in distributed})} TP/CP " + f"topologies, at {_range_text(d_mem, '{:.2f}')}x the per-rank peak memory " + f"(absolute forward {_range_text(d_abs, '{:.3f}')} ms)." + ) + if component and "ws2-rocm" in style.paths: + comp_speed = [ + r["pytorch_loop"]["median_ms"] / r["hip_fp32_input"]["median_ms"] + for r in component + if style.keep_tokens(r["tokens"]) + ] + comp_mem = [ + r["pytorch_loop_peak_mib"] / max(r["hip_peak_mib"], 1e-6) + for r in component + if style.keep_tokens(r["tokens"]) + ] + add( + f"- The `hip_deterministic_logp_tile_stats` kernel alone is " + f"{_range_text(comp_speed, '{:.1f}')}x faster than the PyTorch tile loop and " + f"allocates {_range_text(comp_mem, '{:.0f}')}x less transient memory (it writes only " + f"the `[tokens, {NUM_TILES}]` FP32 partials)." + ) + if "ws2-rocm" in style.paths and "ws2-reference" in style.paths: + mism = [ + c.get("mismatch_vs_reference") + for c in single + if c["path"] == "ws2-rocm" and c.get("mismatch_vs_reference") is not None + ] + relr = [c.get("rel_l2_vs_reference", 0.0) for c in single if c["path"] == "ws2-rocm"] + add( + f"- `{style.name('ws2-rocm')}` vs `{style.name('ws2-reference')}`: tile maxima are " + "bitwise equal; sumexp partials differ only by FP32 summation order, so final outputs " + f"differ in {_range_text([float(m) for m in mism], '{:.0f}')} elements per case with " + f"relative-L2 {_range_text(relr, '{:.1e}')}. Both paths are equally close to FP64." + ) + ws2 = [c for c in single if c["path"].startswith("ws2")] + if ws2: + add( + f"- Repeat bitwise: {_yes(all(c['repeat_bitwise'] for c in ws2))}; batch-invariant: " + f"{_yes(all(c['batch_invariant'] for c in ws2))}; all gradients finite: " + f"{_yes(all(c['grad_finite'] for c in ws2))}." + ) + ws2_dist = [d for d in distributed if d["path"].startswith("ws2")] + if ws2_dist: + add( + "- Distributed: TP-replicated and repeat bitwise on every topology: " + f"{_yes(all(d['tp_replicated'] and d['repeat_bitwise'] for d in ws2_dist))}." + ) + add("") + + # ---- single GPU tables + for dtype_name in SINGLE_DTYPES: + rows = [c for c in single if c["dtype"] == dtype_name and style.keep_tokens(c["tokens"])] + if not rows: + continue + add(f"## Single-GPU logprob ({dtype_name.upper()} logits, V={REAL_VOCAB})") + add("") + add("### Forward") + add("") + add( + f"| Tokens | Path | Median (ms) | p95 (ms) | Speedup vs {base_name} | Peak MiB | " + "logp max-abs vs FP64 | LSE max-abs vs FP64 | Repeat | Batch-inv |" + ) + add("|---:|---|---:|---:|---:|---:|---:|---:|:---:|:---:|") + for tokens in sorted({c["tokens"] for c in rows}): + ref = _lookup(single, dtype=dtype_name, tokens=tokens, path=base) + for path in style.paths: + case = _lookup(rows, tokens=tokens, path=path) + if case is None: + continue + add( + f"| {tokens} | {style.name(path)} | {case['forward']['median_ms']:.4f} | " + f"{case['forward']['p95_ms']:.4f} | {_median_ratio(ref, case, 'forward')} | " + f"{case['forward_peak_mib']:.1f} | {case['logp_vs_fp64']['max_abs']:.3e} | " + f"{case['lse_vs_fp64']['max_abs']:.3e} | {_yes(case['repeat_bitwise'])} | " + f"{_yes(case['batch_invariant'])} |" + ) + add("") + add("### Forward+backward") + add("") + add( + f"| Tokens | Path | Median (ms) | p95 (ms) | Speedup vs {base_name} | Peak MiB | " + f"Memory vs {base_name} | Grad finite |" + ) + add("|---:|---|---:|---:|---:|---:|---:|:---:|") + for tokens in sorted({c["tokens"] for c in rows}): + ref = _lookup(single, dtype=dtype_name, tokens=tokens, path=base) + for path in style.paths: + case = _lookup(rows, tokens=tokens, path=path) + if case is None: + continue + add( + f"| {tokens} | {style.name(path)} | {case['train_fwd_bwd']['median_ms']:.4f} | " + f"{case['train_fwd_bwd']['p95_ms']:.4f} | " + f"{_median_ratio(ref, case, 'train_fwd_bwd')} | {case['train_peak_mib']:.1f} | " + f"{_field_ratio(case, ref, 'train_peak_mib')} | {_yes(case['grad_finite'])} |" + ) + add("") + if "ws2-rocm" in style.paths and "ws2-reference" in style.paths: + add(f"### `{style.name('ws2-rocm')}` versus `{style.name('ws2-reference')}` numerics") + add("") + add("| Tokens | Mismatched elements (logp+LSE) | Relative L2 |") + add("|---:|---:|---:|") + for tokens in sorted({c["tokens"] for c in rows}): + rocm = _lookup(rows, tokens=tokens, path="ws2-rocm") + if rocm is None: + continue + add( + f"| {tokens} | {rocm.get('mismatch_vs_reference', 'n/a')} | " + f"{rocm.get('rel_l2_vs_reference', float('nan')):.3e} |" + ) + add("") + + overhead = [ + row + for row in payload["single_gpu"].get("validate_overhead", []) + if style.keep_tokens(row["tokens"]) + ] + if overhead and "ws2-rocm" in style.paths: + add(f"### `validate=True` production entry point ({style.name('ws2-rocm')}, BF16)") + add("") + add("| Tokens | validate=False (ms) | validate=True (ms) | Overhead |") + add("|---:|---:|---:|---:|") + for row in overhead: + overhead_ratio = _fmt_ratio( + row["validate_true"]["median_ms"], row["validate_false"]["median_ms"] + ) + add( + f"| {row['tokens']} | {row['validate_false']['median_ms']:.4f} | " + f"{row['validate_true']['median_ms']:.4f} | {overhead_ratio} |" + ) + add("") + add( + "`validate=True` adds host-side target-range checks and a non-finite LSE check that " + "synchronizes the stream; the cost is a fixed per-call overhead." + ) + add("") + + # ---- tile-stats component + comp_rows = [r for r in component if style.keep_tokens(r["tokens"])] + if comp_rows and "ws2-rocm" in style.paths: + add("## Tile-stats kernel") + add("") + add( + "`hip_deterministic_logp_tile_stats` computes the per-row, per-tile FP32 " + "`(max, sumexp)` partials that the operator all-gathers and merges; the PyTorch tile " + f"loop is what `{style.name('ws2-reference')}` uses for the same step. Tile maxima are " + "bitwise equal; sums differ only by FP32 summation order." + ) + add("") + add( + "| Logits dtype | Tokens | PyTorch tile loop (ms) | HIP kernel on FP32 (ms) | " + "HIP kernel on stored dtype (ms) | Speedup | Loop peak MiB | HIP peak MiB | " + "Max bitwise | sumexp max rel | Repeat |" + ) + add("|---|---:|---:|---:|---:|---:|---:|---:|:---:|---:|:---:|") + for row in comp_rows: + kernel_speedup = _fmt_ratio( + row["pytorch_loop"]["median_ms"], row["hip_fp32_input"]["median_ms"] + ) + add( + f"| {row['dtype']} | {row['tokens']} | {row['pytorch_loop']['median_ms']:.4f} | " + f"{row['hip_fp32_input']['median_ms']:.4f} | " + f"{row['hip_native_dtype_input']['median_ms']:.4f} | " + f"{kernel_speedup} | " + f"{row['pytorch_loop_peak_mib']:.1f} | {row['hip_peak_mib']:.1f} | " + f"{_yes(row['max_bitwise'])} | {row['sumexp_max_rel']:.2e} | " + f"{_yes(row['repeat_bitwise'])} |" + ) + add("") + + # ---- distributed + dist_rows = [d for d in distributed if style.keep_tokens(d["tokens"])] + if dist_rows: + add("## Distributed vocab-parallel logprob (BF16, RCCL)") + add("") + add("### Forward") + add("") + add( + f"| Topology | Tokens | Path | Median (ms) | p95 (ms) | Speedup vs {base_name} | " + "Peak MiB/rank | logp max-abs vs FP64 | TP-replicated | Repeat |" + ) + add("|---|---:|---|---:|---:|---:|---:|---:|:---:|:---:|") + for d in dist_rows: + ref = _lookup(distributed, topology=d["topology"], tokens=d["tokens"], path=base) + add( + f"| {d['topology']} | {d['tokens']} | {style.name(d['path'])} | " + f"{d['forward']['median_ms']:.4f} | {d['forward']['p95_ms']:.4f} | " + f"{_median_ratio(ref, d, 'forward')} | {d['forward_peak_mib']:.1f} | " + f"{d['logp_vs_fp64_max_abs']:.3e} | {_yes(d['tp_replicated'])} | " + f"{_yes(d['repeat_bitwise'])} |" + ) + add("") + add("### Forward+backward") + add("") + add( + f"| Topology | Tokens | Path | Median (ms) | p95 (ms) | Speedup vs {base_name} | " + f"Peak MiB/rank | Memory vs {base_name} | Grad finite |" + ) + add("|---|---:|---|---:|---:|---:|---:|---:|:---:|") + for d in dist_rows: + ref = _lookup(distributed, topology=d["topology"], tokens=d["tokens"], path=base) + add( + f"| {d['topology']} | {d['tokens']} | {style.name(d['path'])} | " + f"{d['train_fwd_bwd']['median_ms']:.4f} | {d['train_fwd_bwd']['p95_ms']:.4f} | " + f"{_median_ratio(ref, d, 'train_fwd_bwd')} | {d['train_peak_mib']:.1f} | " + f"{_field_ratio(d, ref, 'train_peak_mib')} | {_yes(d['grad_finite'])} |" + ) + add("") + if "ws2-rocm" in style.paths and "ws2-reference" in style.paths: + add( + f"### `{style.name('ws2-rocm')}` versus `{style.name('ws2-reference')}` " + "numerics (distributed)" + ) + add("") + add("| Topology | Tokens | Mismatched elements (logp+LSE) | Relative L2 |") + add("|---|---:|---:|---:|") + for d in dist_rows: + if d["path"] != "ws2-rocm": + continue + add( + f"| {d['topology']} | {d['tokens']} | " + f"{d.get('mismatch_vs_reference', 'n/a')} | " + f"{d.get('rel_l2_vs_reference', float('nan')):.3e} |" + ) + add("") + + # ---- optional tuning history + baseline = payload.get("baseline") + if baseline and style.show_tuning_baseline and "ws2-rocm" in style.paths: + name = style.name("ws2-rocm") + add("## ROCm tuning: before versus after") + add("") + add( + f"Baseline commit `{baseline.get('git_commit')}` ran the PR's first ROCm backend: the " + "HIP tile kernel on an FP32 copy of the shard, with the shared PyTorch autograd " + f"backward. `{name}` rows only." + ) + add("") + add( + "| dtype | Tokens | Fwd before (ms) | Fwd after (ms) | Speedup | " + "Fwd+bwd before (ms) | Fwd+bwd after (ms) | Speedup | Peak before (MiB) | " + "Peak after (MiB) | Memory ratio |" + ) + add("|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|") + for before in baseline["single_gpu"]: + if not style.keep_tokens(before["tokens"]): + continue + after = _lookup(single, dtype=before["dtype"], tokens=before["tokens"], path="ws2-rocm") + if after is None: + continue + add( + f"| {before['dtype']} | {before['tokens']} | " + f"{before['forward']['median_ms']:.4f} | " + f"{after['forward']['median_ms']:.4f} | " + f"{_median_ratio(before, after, 'forward')} | " + f"{before['train_fwd_bwd']['median_ms']:.4f} | " + f"{after['train_fwd_bwd']['median_ms']:.4f} | " + f"{_median_ratio(before, after, 'train_fwd_bwd')} | " + f"{before['train_peak_mib']:.1f} | {after['train_peak_mib']:.1f} | " + f"{_field_ratio(after, before, 'train_peak_mib')} |" + ) + add("") + + add("## Figures") + add("") + add("![Single-GPU latency](single_gpu_latency.png)") + add("") + add("![Single-GPU peak memory](single_gpu_memory.png)") + add("") + if distributed: + add("![Distributed latency](distributed_logp_latency.png)") + add("") + (output_directory / "report.md").write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def _write_figures(payload: dict[str, Any], output_directory: Path, style: ReportStyle) -> None: + import matplotlib + + matplotlib.use("Agg") + import matplotlib.pyplot as plt + + single = [ + c + for c in payload["single_gpu"]["cases"] + if c["dtype"] == "bf16" and c["path"] in style.paths + ] + tokens = sorted({c["tokens"] for c in single}) + + for filename, keys, ylabel, title in ( + ("single_gpu_latency.png", ("forward", "train_fwd_bwd"), "median ms", "latency"), + ( + "single_gpu_memory.png", + ("forward_peak_mib", "train_peak_mib"), + "peak MiB above live", + "peak device memory", + ), + ): + figure, axes = plt.subplots(1, 2, figsize=(12, 4.5)) + for axis, key, direction in zip(axes, keys, ("Forward", "Forward+backward")): + for path in style.paths: + ys = [] + for t in tokens: + case = _lookup(single, tokens=t, path=path) + if case is None: + ys.append(float("nan")) + elif isinstance(case[key], dict): + ys.append(case[key]["median_ms"]) + else: + ys.append(case[key]) + axis.plot(tokens, ys, marker="o", label=style.name(path)) + axis.set_xscale("log", base=2) + axis.set_yscale("log") + axis.set_xlabel("tokens") + axis.set_ylabel(ylabel) + axis.set_title(f"Single MI300X, BF16, V={REAL_VOCAB}: {direction} {title}") + axis.grid(True, which="both", alpha=0.3) + if style.paths: + axis.legend(fontsize=8) + figure.tight_layout() + figure.savefig(output_directory / filename, dpi=180) + plt.close(figure) + + distributed = [d for d in payload["distributed"] if d["path"] in style.paths] + if not distributed: + return + labels = [] + series: dict[tuple[str, str], list[float]] = { + (path, key): [] for path in style.paths for key in ("forward", "train_fwd_bwd") + } + for d in distributed: + if d["path"] != style.baseline: + continue + labels.append(f"{d['topology']}\nM={d['tokens']}") + for path in style.paths: + other = _lookup(distributed, path=path, topology=d["topology"], tokens=d["tokens"]) + for key in ("forward", "train_fwd_bwd"): + series[(path, key)].append( + other[key]["median_ms"] if other is not None else float("nan") + ) + figure, axes = plt.subplots(1, 2, figsize=(max(12, 1.1 * len(labels)), 4.8)) + xs = list(range(len(labels))) + width = 0.8 / max(len(style.paths), 1) + for axis, key, direction in zip( + axes, ("forward", "train_fwd_bwd"), ("Forward", "Forward+backward") + ): + for index, path in enumerate(style.paths): + offset = (index - (len(style.paths) - 1) / 2) * width + axis.bar([x + offset for x in xs], series[(path, key)], width, label=style.name(path)) + axis.set_xticks(xs) + axis.set_xticklabels(labels, fontsize=8) + axis.set_ylabel("slowest-rank median ms") + axis.set_title(f"Distributed vocab-parallel logprob, BF16: {direction}") + axis.grid(True, axis="y", alpha=0.3) + axis.legend(fontsize=8) + figure.tight_layout() + figure.savefig(output_directory / "distributed_logp_latency.png", dpi=180) + plt.close(figure) + + +def _environment() -> dict[str, Any]: + properties = torch.cuda.get_device_properties(0) + return { + "gpu": torch.cuda.get_device_name(0), + "gpu_count": torch.cuda.device_count(), + "architecture": properties.gcnArchName, + "torch": torch.__version__, + "hip": torch.version.hip, + "python": os.sys.version.split()[0], + "git_commit": os.popen("git rev-parse HEAD").read().strip(), + "native_collective": "torch.distributed ProcessGroupNCCL (RCCL on ROCm)", + "extension_symbols": "hip_deterministic_logp_tile_stats, hip_deterministic_logp_backward", + } + + +def _validate_environment(require_distributed: bool) -> None: + if getattr(torch.version, "hip", None) is None: + raise RuntimeError("this benchmark requires a ROCm PyTorch build") + if not torch.cuda.is_available(): + raise RuntimeError("no ROCm GPU is visible") + if not _EXT_AVAILABLE or _C is None or not hasattr(_C, "hip_deterministic_logp_backward"): + raise RuntimeError( + "rl_engine._C with hip_deterministic_logp_* is unavailable; build with " + "PYTORCH_ROCM_ARCH=gfx942 RL_KERNEL_REQUIRE_EXT=1 python setup.py build_ext --inplace" + ) + if require_distributed and (not dist.is_available() or not dist.is_nccl_available()): + raise RuntimeError("PyTorch RCCL/ProcessGroupNCCL support is unavailable") + + +ALL_PATHS = ("native", "ws1-pytorch", "ws1-triton", "ws2-reference", "ws2-rocm") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument("--output-dir", type=Path, default=Path("benchmarks/results/rocm_logp")) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--samples", type=int, default=10) + parser.add_argument("--training-samples", type=int, default=5) + parser.add_argument("--skip-distributed", action="store_true") + parser.add_argument("--skip-single", action="store_true") + parser.add_argument( + "--baseline", + type=Path, + default=None, + help="earlier results.json; adds a before/after tuning section for the ROCm backend", + ) + parser.add_argument( + "--render-from", + type=Path, + default=None, + help="skip measurement and render report/figures from this results.json", + ) + parser.add_argument( + "--report-paths", + type=str, + default=",".join(ALL_PATHS), + help="comma-separated measured paths to show, in order (default: all)", + ) + parser.add_argument( + "--rename", + type=str, + default="", + help="display names, e.g. 'ws2-reference=native,ws2-rocm=strict-hip'", + ) + parser.add_argument( + "--report-baseline", + type=str, + default=None, + help="path used for speedup/memory ratio columns (default: first reported path)", + ) + parser.add_argument( + "--table-tokens", + type=str, + default="", + help="comma-separated token counts to show in tables (default: all measured)", + ) + parser.add_argument( + "--topologies", + type=str, + default=",".join(name for name, _, _ in TOPOLOGIES), + help="comma-separated subset of " + ",".join(name for name, _, _ in TOPOLOGIES), + ) + parser.add_argument("--tokens", type=str, default=",".join(str(t) for t in SINGLE_TOKENS)) + parser.add_argument( + "--distributed-tokens", type=str, default=",".join(str(t) for t in DISTRIBUTED_TOKENS) + ) + return parser.parse_args() + + +def _report_style( + args: argparse.Namespace, output_directory: Path, config: dict[str, Any] +) -> ReportStyle: + paths = tuple(p.strip() for p in args.report_paths.split(",") if p.strip()) + unknown = [p for p in paths if p not in ALL_PATHS] + if unknown: + raise ValueError(f"unknown report paths {unknown}; choose from {ALL_PATHS}") + names: dict[str, str] = {} + for item in (piece.strip() for piece in args.rename.split(",") if piece.strip()): + key, _, value = item.partition("=") + names[key.strip()] = value.strip() + table_tokens = tuple(int(t) for t in args.table_tokens.split(",") if t.strip()) or None + # Always show the measurement command: rerunning it with the same report flags + # regenerates this report; --render-from only re-renders an existing results.json. + command_parts = [ + "python benchmarks/benchmark_rocm_logp.py \\", + f" --warmup {config['warmup']} \\", + f" --samples {config['samples']} \\", + f" --training-samples {config['training_samples']} \\", + ] + if paths != ALL_PATHS: + command_parts.append(f" --report-paths {','.join(paths)} \\") + if names: + command_parts.append(" --rename " + ",".join(f"{k}={v}" for k, v in names.items()) + " \\") + if args.report_baseline: + command_parts.append(f" --report-baseline {args.report_baseline} \\") + if table_tokens: + command_parts.append(f" --table-tokens {','.join(str(t) for t in table_tokens)} \\") + command_parts.append(f" --output-dir {output_directory.as_posix()}") + return ReportStyle( + paths=paths, + names=names, + baseline=args.report_baseline or paths[0], + table_tokens=table_tokens, + show_tuning_baseline=args.baseline is not None, + command="\n".join(command_parts), + ) + + +def main() -> None: + args = parse_args() + output_directory: Path = args.output_dir + output_directory.mkdir(parents=True, exist_ok=True) + + if args.render_from is not None: + payload = json.loads(args.render_from.read_text(encoding="utf-8")) + else: + _validate_environment(require_distributed=not args.skip_distributed) + config = { + "warmup": args.warmup, + "samples": args.samples, + "training_samples": args.training_samples, + } + tokens = tuple(int(t) for t in args.tokens.split(",") if t) + distributed_tokens = tuple(int(t) for t in args.distributed_tokens.split(",") if t) + selected = {name.strip() for name in args.topologies.split(",") if name.strip()} + payload = { + "environment": _environment(), + "config": config, + "single_gpu": {"cases": [], "validate_overhead": [], "paths": []}, + "tile_stats_component": [], + "distributed": [], + } + if not args.skip_single: + payload["single_gpu"] = _single_gpu_benchmarks( + warmup=args.warmup, + samples=args.samples, + training_samples=args.training_samples, + tokens=tokens, + ) + payload["tile_stats_component"] = _tile_stats_component( + warmup=args.warmup, samples=args.samples, tokens=tokens + ) + if not args.skip_distributed: + device_count = torch.cuda.device_count() + for topology in TOPOLOGIES: + name, tp, cp = topology + if name not in selected: + continue + if tp * cp > device_count: + print(f"skipping {name}: needs {tp * cp} GPUs, {device_count} visible") + continue + payload["distributed"].extend( + _run_distributed_world(topology, distributed_tokens, config) + ) + style = _report_style(args, output_directory, payload["config"]) + if args.baseline is not None: + baseline = json.loads(args.baseline.read_text(encoding="utf-8")) + payload["baseline"] = { + "git_commit": baseline.get("environment", {}).get("git_commit"), + "single_gpu": [c for c in baseline["single_gpu"]["cases"] if c["path"] == "ws2-rocm"], + "tile_stats_component": baseline.get("tile_stats_component", []), + "distributed": [d for d in baseline.get("distributed", []) if d["path"] == "ws2-rocm"], + } + elif args.render_from is not None: + payload.pop("baseline", None) + (output_directory / "results.json").write_text( + json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8" + ) + _write_report(payload, output_directory, style) + _write_figures(payload, output_directory, style) + print(json.dumps({"output_dir": str(output_directory), "status": "ok"})) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/results/pr328_rocm_mi300x/distributed_logp_latency.png b/benchmarks/results/pr328_rocm_mi300x/distributed_logp_latency.png new file mode 100644 index 0000000000000000000000000000000000000000..900ea3b82cec9590833317cd5ff860c4bafd4f0b GIT binary patch literal 62146 zcmeFZ2T+u0v@O~*<~)w5fCLp(K!OB8vJnIo1q4J1f&wBrOU7}OQIa5nWEGSs2uMyU zN)D1GG^ixWLX+cNACBjo_inwabE{scS9QB)6q^qJ|AoERUTf{|{sjfuP3sxgQz(>8 za;Hx!Q7G$9Qz$F0u3e31#xwp{iND0GPhGNBHq*DZ)3MZ}oYS#3H#W02Hq_m1t7mCt zXl5$N%YTShlzaCLYin~WF+M(%|9AzjnWX{WewjLTe9AiW)9O|f%C02x|8H81zDyL# z7_Z#PV;66Q4pQx$l)LAaM(wf>@Z=;Y3IDbBucyXkbiW-7c(musv{Tzt7b>0fbGMj( z9Cxpxvk+L>>{!aOw}e;O_IB;@z9R+Fv*pRpVr@kcr?U`UP%41JNkBRQSX5-_Ulh5u<`O% zdFp(5d)xH>9_d^C1D1g?F&ygLJ5HTCl_gGhrahG?S8ZLipo{4;=gV-+Oer>4@uMZkCCD#gnhY<`l}QlRb~jCrxkP zz8!br*>m5f1EXwBlnT-x>0KjoBe z-O60Ok^bpcHW?0X?uu9Z-t_#({pz!AjJTo&MVD^{zkchpigNUsd3V`%W@#6VVjtF~ z=chczCtJ;o%KSOAX;tDhn><$Mao=#4**>SfsRlJ|mE*_%s_cwDAJN!X7d_D%;mRc} ztUA%(pnPZbrZJ}l3gs=oC8G$hB_qY{{Xe)?%BnlRb5d66&yM%?I2RtW{`!8^$Kqn6 zfyP&tlXdTY`}WOxao*l}`m6i$I4w+d#Gmrmv3X8#W907lRU(*Z&#sk}-8;h_`Z?1~ zS{r+kKYTd3fR_b}*~D#TI`UM|pen|3YPcoSBC@|BKF}m0$VZteR`u28fa2QPTH{4l zdiof~!#8itO%63r_Lk79rIV)G9nbsGF*3%5OFH&@iw|EY_1kAdhTw?#(eePEx;T}1 z%RKLa(T_W|W6lN%D75+?R7-Sw{u|}#)rWPjPbsXRxV7zxK8C-)NYS1CjpI<2OgF>5 zXIWOg{mlju4efc3HsjwFZH8J@;?u* z9VX)19R_$7eiUwLo(yVjUh(|-^YIVc&c`T5%G|7v@$GMX70)K?9V7pkE31EGAgMrj z+kHE#O`z_zYbx2+ecVI$wy-8z^;VU3bV#cWWU(-d+g`-ucG6A?#cppjV(i%OqMIxk zIc}|3v$;M>?$O08OTCXDKW2$DQA|%_3mku^oBTa7XIRN&2T%QrGdr;)<#+OGT!@fz zesisOv%S52miS2u#eBt&si~&#HIawS+9l*_$GXdDHqMd`<0YwQQbQ%|l?v}|dc;n( zZ8ZrNwR-8~#I$t%pyKnB;R(154OXlL%5FZgV0XKY4lo^krj&4re_^`QXZ@Bff$T#q zY2{znuJib~*W=@JVUw04gColj40*>4_v{=vchN~~2;FJHzPH>V&Fa$QL`4v4)s)ST~}Gc)Tr(ck@= z`q!!r^-0>&Dz7f996NSw{iaPS7@i3Xl~LWxbD|F{#=1)T>!N+IO6ZuFxLsXcRpKtn zX=Yp3WLk7u_t$ggR4>ep6if`~jP&Dw;ZDq&T9#$Eb#Itmf8BZMhQ|lhG`v4@Wy~2U ze0EIItajkj&QHKb9=LxX{L@fIN21@pGi6OpYC1YPaaS@D7ikeoH(Jw8nkVt~yF#kH z-@Fm}SX2~;yChN1$;m0j^KAC5(dYSd!{VD*BravNJH)>)EbJZ_i2rk+e2h-vJw{>U zCzB%`5uGO1c|DaO(Fpr-GG2`2)-GIlh^_V#&t#kQ8yWcd`HjC@!^$Bhc6sa8tz)?I zAOXGl=chL0Ar57mfBoysf<~T$N@tN*9kX5Avu__g4zjbK6rox_A|abu%IT8QWJK$v zl(brs)~huvw~i_*Dz0C<_QJ`Ne`ni`Bw~Atc^wexIH-!)Qy1wYMHT?3kkG|<@7~3! zz6vJ6a^IPN%r}w(;?A>g+HZAkYKT`mj|HThtoyM(Mv={n@vwX#pLUgn>#AqALoGed zYE!r0e<|lHsQ>7(bD!Ku-c)$q$!}i$ZCO^c9ZQSRFVBUV!b6JQWB$>b=n83>8;c7!#yVk z)A=`^z()jp)K$b1OXarfweayDz48S~Q#DM?rlP`nAg7!wQ`tQqI|1nV1G&`|RMkOi%Lv zFl)2@Y+rPxlnX+NacjYk_Z#%*u*VG4!yi7pYCX_c**B1JiMt5<(Io1{3sKFcdz*H7 zVm|cd>~3HX0(y&zin7N=60j(ft$G#Q`MbIogSOe{wFAHerWa$u4GJYTzPb`(QXMX7 z^yWs*-ZQM&*ut#`4jd5oYIUHEj`B+_PG1Xb8Y&aE>Z*@bmcbrQ^)meNzWF&F6?yL2r*ECgkI-E??P{g?Dr(QPxYk(_B)EAFG1}`)tMgm8-()N;Q|dk|DJgjp82DpyvMRj*X+YFj(_70C z%cjzHB$wu0Iff|0;OD9P@c~0@Y^-{?YflA3p>Fb%>ZHqQanuRpj(n=|wUN&sHH#(h z>ChWBCh!9Xbeo6S3^X1Fa;m5fZW}h>pbuBe-|&`A;>X(+jDq@}?1uOPc{w@Xq@<+s zYH8cSC&&a+R^wWMyqfjTPON#%r>)i3*C%Gv@1Jhc`tt6Eo%#%S`JHF2~b|V zH4fM>L+O=+K)?0K+y~{>uV1e;(w^6lpf32BTV06&+gy8^kWnLd7|;fGJ=rrCqZPuq z7ZSDdE<4Xns>sVfV2`z@^<)Fysk}VP0F+}`8zqMz|9JH#Mr&$PhIWAqqqIvddta+r zfl6;x7<2l$^XElmCrJz>psI2V>CI-it(hry2MbH$u{*0{5eVhaoohgv60#q=PQqAc zu}>3r%>)8ei)IskQJ;KG*6MHNw3`j}8D`p4>$-DDT}`8%#c8HgvC1I9;^7-^<$-+V zy2;C+4&y^YFSgbI{zu*L&}I7TnxW=Y*}16^HC$Di&48-acq-d)i!@C)5BWy*{-68E z#8ELR%URdi*=F!Fze#(py7c07Bv#pYmG}rjsy!})6(_bxMybTB#=0!bW-l$yZQs8A z1%mQTz+e54@p0>Y2M)wL&5ZSbdUnS&t9!>|o-3+aT3TY3UH@QPMyjXka~?jda$agC zsw|Mt%w>PKJHTbEVcpA~?^UxC1M$c`aawuV2n7Tk0t)nhPQDg=NarslW#v@ntgsGX zvFBJ;!6SFCnB}Q#k@(@ZmyONl+lO@o?_JJwNUI8$j8D|ex%uv|RkpId+WAiE%wpD` z@C>^ghPCPGu{#$tOf~oI-+!~_`Kj5_V%CP}^Q`37d5cA^6>Y2x6;(ziVU)P_no-0w z0TcG>Qu6Uf2hOV^b>+|YAKZRW^(-nao0h=%j)M!c6LoJ4>)7S6Vi^xzbwd{0Z)Rp@ z(w3>XZU5N|C}mvcM`blL&5z+W(}Fa=Tqf8TQI!lZ{#OLVy@26=<2YqXmb3d7*2aA2 zx%xy+aT98`3b28d{ajbZx^4TPOtog1jTD{MumkjGmbi7n4`_RvTrDtwkjwlno-65R z4=99Q0FuC}Wwse<*YuHI%>3)GzuFiqz2_I_Mo94@`d1>RuAiEdQ5L`x50GVu%`ywT zYBT*q2g~X5KvPl%!(pRpt7^$;;2DBo`kRs>$byQBVyFGN@?5ZxB48Du)EnzQr97k_ z_1R0=)`kH~Qv*%aJ^Y1-Dx`K{+>rT0x@ShI{rz)`ebEt(s0`TU0BFe|U4C<84<8@j zEA0X;4AQ5ig^s1}=6%~Jl*B#JIw-f7p~aX(UYC#n37I-uM7{nxrEn)yMn62f?g$q2 zaNd-bX6`LT=h<;ZY*9Kox+v_}0aP!IL4pQv5E^K;t&{r7IjFFYA3sh;^YG33lF-W*o^$roZ=0E6>3Y%hZn@b2#z6<^&=Kfhgo|jhc8X zBc!#jWSCZtZd7tzoQWRJr74i2w!OW57Ug#;Q@o6GujBP=fA z;u(&vwo9R_p02&XE6WdT-{Kk5nr#zbm_OvDduxhnSJ@pY>tmZ4j;(hM%SGXuou2>6 zp8i{MGJvI{P1>%a&I$@LSL6kbpE%)#jI3{%VNlHyQX?b1G;b9kY|r0PTlVC(LpUMfz#pzD<03XJ~Wg1#|XD2qZdW$g#xa|~KAC`Om{NL=_GA(Rd+1c6YQQHd51&iC6CN_3I7@URXc&aYt}6*6ug>^KPQCD@AMuFo4sIClVZ-_f3FPj zRUl!p@bj3wZ6puwmEe(G)eUeqn8%>7l@cl?Ek<_o3eNahH;x zCmYu7k9OA2sP6eN;hCO`T1F8WiZ+9iK`TJG$$LlsPBom z0nYk$(-z13>U#2PQ4Hgm?b{!}@G)sk4;paneyopge(UD(_e zwLK#ROTqx@QM;Vx`Z9A#-<)1W*hac;M+Nly25B z{JPp}mk22=HfuMf6&Ham5i&TpVeQ)2a}Jw;&|Y2Eo8(9J&RrhiG$wb#PqDzonT>;^ z=0WJqCW*dje|9zb7PSLcNaqGh~vqte}7tCT2X z4HHaJ1`13(8kwJTIwm*QvN{g(Tg@OAT5fajc3G+6_(R01r@}bmQQh@#*r>y=lkmIX zq2bIosmxZ<)qTE|fgz}q)nYPt zqVbBpwD-&C!y5~4fuLe83JVQ2_Q(2=xM6GSHVgb7v>*gNqb794BCnVF z_4wo(x$f2%cA7R1ajt%R{dwNs`3-=b`sq0cqoJlN)biC#^CnbNU!|V8k#HrW>TLMM zM2+U*H^o6sMts8Y+YTtuBQgoHGl0vKadOJ7qfZJ<<;+OR11@?`f5iN$#r(8Iv~zY# zM0|(6YL?u7{q_R_tSgYisM83nUO)#7Y%-p&<@)d5y~_X;W7jv)ttY^r)0&nv14Jhj zIDz`$s~8}H*A>&OkH$tL(|j}+B%Otg9uh4$A z?(v+%IaAJ4!_><3$v@AirW#bI_XUDZG}r8@t`6M`4vtNbVD+{%Elb9wU;p;@t$XcU ziS*R=UDPX=_n|}Ii$JFI1?17n-`_phez0XB!SW-bdF8~yi0e{$io~8h@3=2a(P&l< zp*{aZUh71b3JyHIal-~dFigX*eNZw5-#I}LvtCx()j<*mJr)xe&dLKmYJ22$o>uFg zu&74ls7OC-)M$O_)06OUQz66J55{M1U;Ns$kHQi4U?-nxyR_q^v3Gcgy+dQc;_Sg{ zUCUG5Wse(PUCEFbN=3cpKo%Kq0m(RhtxMCLTf&Z5>^61 zzdv)YnA@;}`?b1$-dQ*HO}m6EbXY+dIefc+z;at6^;UxNVb{5#4BJ+XEXz{`^J5-S zp0Qs;4jbvGI-x!gvmc9Mj|J#KjTT-wF)>WNg2FJ+pMTR(ta7w*fGbj$A3L^sPT=X1 zGmTBju1hXpn2QXr*bPOjN^7rhtwRx_*SZMa=wYB*JCkm=H8qPLYh2FnqprnlyPrUM z)`aB-#H9a4Z(8~eUoqp3QewUDHb?cEa^ zkDH1mgM7Y5fPJ5tnQ1oZEwfm=eoK?jKa{6#dW*k(8FYUxN$~p`R;S%^{mrS-S}rqs z6Mc2EfZ-K&^h1UK@!p-43ZT`QT}Cg1!B-Wx8}?^M&A6v}TX0sg!^BP6$oHY4R}6ys zM@%AS-WoO~Y6_Pptno!z(#*0Okcf9@jlK%PWZI3I<47&_KrW=92r;^TrZz!6wZdqN z^z@fIz_|r3dG-a!Sj~y42%{|-W)V@ID~=3wjDnEs>6pTpsiKmzntHxi0CLN)<&9y> zYn?lrS)~$?-`z_+@z&Uju}YigESnaC)CMH6-}j#nPgu2K#|_kF4}5&ol$89ai{31f z{Yn~%4?H~T0Gr={d?tE?7$lWMjV!+bo8F?b#|QVxef|1XcrKo;aiuQE04Nn-0$5`j zkMQguHQ$#vCaL6OT>b@{FZc+`18^W7nO<}2SLe} z+b-CD1sg4}?WQV7(T3z}#e^-Y3=>x)wBk#JusBe8@d)4$A7<7hN&5QwHf^`EO+4qc z8DuVEv1@=Wsx6|mfFg}UjRDYe9so)aF@$zFzaoG7+lI%S%0*wtDmobM={)K93L1x~ z0b!G2_N-OBzF*6FO~Bx7XJSf#aKodSeUYzVJ|(LnnBCLcI{*|#u!vbA8DYZGmBs-k z1Cwhg$p}atI`rvl_ShK701MdSMB|cE0@K$$G&J=3mH%jv^-%J5Csa8(Iqry zhMJ_r@N}2Z;Buu>R8V60*)FnI2s3>xU)5kUvzY%-a-t5sLSWOYD{3HQY+7CdOU+{X z_USq0w^$fjD)4zYe0bFtcg$r3%>M_@;=bfFL2!oBpDCb8O8dR7k&IY_s$~5S2 zz4}XPxKyJpfNFP5i@A_(Npbh8r=Q9s`k$V>|EH*fDVE&C+Z9_#h!O)IeQz^!9H0>4 z&Tlp*1flYKfudPV@{g)lgEDkI_iR>%$D4SwN{9FIcgbjIgxW8GNT~x3=oz!>?CdmI zG;=-&TD{pj!CcQVD-K);5m6wKky>UTt2aDVJ=~t|WuEWmrFoi=%{$7@CRX8i<<>S)?y6I75uhyIt4@ct^!TzX$!qO+k z#*{#&yfJAN#-dlix;^~pQ_$2pHJ9&@MH;%V>zYoUmo5D;I2oHe8=9qx*QT7K&m72! z23gZz6X{OXtCpM!1!Rgvz1WAnPoBipJswK_J92S-S=!tf*#G|5Ce0~&4S9}sMEUWH z!hYI$LjZT{&Fj;iixM#?hVV7%L*?MBUcWvBlHgt6B4^g5eTb-BCJ9@DUx1N&O7{IOfA!>e*=*> z^=MhPv@l5cJ$v@_6oB8Wg?v*nl2joWrYRm7`&iuD(K00|00bHdMg=PrkB~KXK^~RU zj3pmrBk#Fsq}GOXlPk_Y-mU4O^M;a2^vlDLyzCtvr5>2pz=mWhdaol_MF8hoA})aV{|xLJHOLg zd~or!{OQvriW<0D+lY{vtGrerp@i%Jan8=!>28!^O09o-Yz4Sq1t6Io>*vVmbtnXO z%PB;c1|*s-X{21gU&q-E7AEn@lP40|N&fs{vn#epT=h1;`T6xTMEoRBA)MUYXTjv_ zUorCI$uZfJHOcUUYS+;`cm}xPA_54Z#6EqxAlPgJ!Ax@U^EKWiagnhx=gSuoMmDK+ z3^e5@$p>(A$_cYsj?Jd-6RMtZ1k=Xu?zBq5;?YWxjw9z7vuc|#w3fL%iE6MtK)`qjCP^K! zsSz=<;^sVp(!0>(1qgb!5fS#|o(C=Tq%Kv>w}E+YHhHv-RL`!15MH%``mCSCXoIIL zU)a&L7!)|*b^kv9)D)7Il8?mL>2&9=$H0f0N=3IPfIaH7tPFC75)K?6Q$BtApQ;Gy zq>_@;z(X7zO@TWR&1~Rp2wW2OZdVI(jp>;%ICCtYZ+RsRJDBx8RtOU#%gFDI`PcWy zBhDtM#0S_*A31VKN~+2=PCYs*AaNw>s7pDb&O(llzDRJ3N|0RU=H)Ob7&4y-2mnw`9j;3}rRhhL z&V>l;-}d*nDE&C7FJ{$b;WJ?t7#tku@9%Hy5miz^rO$0+rcgc$+Vo$fxd1AqNItDn zK=N(CesW%%>1}}mNjN-j-6No`+nuMV1O3tkz^#0;CW(?)OEo0JsNXpZR3=3q+dIgs zB~A@^P8=rXRO_9MYUyeU*TeOtXydfTIY>lAFc^{&p`FI=cB)oPkk*|)_j zSK&2rlC0Un;_uJDkQRAT*CnBqWngX`iif^s?(5Y2B!$o;zC%9JU}Jr6Kywz7O_5KZ z`+()2qM3BiBq4FiO8mg`IzcrZ2)r^^7T@%cH6ilmi}zMRI}G3oIS2|=j^TblMP&M5 zY3R|4&`aFL*naxn30B?C76i|ybvH=aw3=iDt=nWY*4?tzxV9kjb3%C%^Sc21l@Tf~ z&Kdd#Xz8VWy_+bzmryYi3Ks-Xt|kZ~fU5so9E9*q%&M_sY?@;*qz>zq`PYED=zD#H zYtNozKq*$4-uUL=DJw@On^(|120$nWu&16bYpC8muy_kJ#R6#j9+&>!UV|#@fGjGg z%}CT=trhsk?oVZ5#gd}UY9QUq2Ngxuu8*S7i4h1w$b8xo!a^0zM#%W{1?87#`Iib1 zw75+Y*6eFuc*wODp`mP2)}`mpKOhWRxyvO!n%J%pM2NS_ZWDVE=f99{2)QMT*~)U0 zkt0gQy`0yCVr;tZut5>YPqfh-fbm7Z)gSv%<(|%W%qle{L59cM4xA4c8c~Z^t<=fE{I?9AZ_a985X?w4$Ri2QHAcL3 zho6w^;{3or0teNWHqRuRI$q8&p4czo$bx(H;)M;K{n;iV;rE&NYu! zZ0F{ojio{L4ZOCT?Yas6k=0tYW=+V*NLxu?@7bQ}PNNJteFznw5PriZAG3}O!?cqq@f6DD`yXAc#4-`-Pdju9g#)5%IA@LNoVa!)6t`<# zLMeoG0r}sa82B@cpiteutO&5J5Z%;7C>W3(<@&A7orAQGvu#`|ai5t&w2V*~Dj3M7 z<@d4^N?)O~^gc5E1;VFBdG3wH-p;(pZZl+T&}d>lq(?)PyI$pz22F$Z`N;0wZ->ts z3`-1`&P!Dd=On)#iZ=HID|A;Lf(|gM5lbeisbQ2kpPz7$i>oZx%7$kOtW43bqYg4T%WBpk`heC`yVp68DAMaiCupiUu% zQK%llkx!wiJi~{5=Fae*J&Q~U1~0g?$#NYSHTkL>(eAhm` z(5?aK2KmLdk^%)&p_S&o0Rb<8qg$NYCsOvWc=YJe1d!mF6zPw-X;c^}jJ*#q?b}8h z$T=(yl$MP>$gVfICCGp)s}We8X3V~a@>XGq?S1a#-Ow_-c3b^5x)}4BIGBTm;-;O-&>0)O9yK z472Dg>cfaX_UDHq`IJ&)xrBWB-SbVuu272fEr2h4N) zcEcLcL!IBgdEy-mn7cRj5{C~r!~k%ANK8TFDnc!2Mq?ZtV$Dj{uVhTEmSuIW zZ&NFL|K1bK%Ms^cH~=eq8)KD=k3iH65Qb(W2up%qdWfW>ZGdoZb%Y4fjnnPno%Hhe zr^^{QD=#kqN0AxrGYqc=QW4GBTx_qXW!}bmfUqFg+B~<5@!pVK274!p^77gqcm-v0Y=X-wISemfOOd;m1*`5%ScCWLV>W_4{du7gJ<$63#AzN-Y_xT05m z$_o+pvsa-*=rxm_^z`3<-hR`=R;4n$aQ|4dNF&c>0W zjw6-I(FzPuTLfFZn8k(9gn*BEQVvA8z+H}syD5Qd79Oyov+L}z4J`ak%&l;pky^qfMO;-L zP*zC14Uc-tNtnmfKuU`DScozO{eV>_ zM0pN7@(f(nN`N`n>F*I|1;S@PC;(ysg(3Su^e9*oRiwEls2MgW)Z+j?gq7Q|G+ROi zQnAEmp|p;}vc(}Kr3E%c6^Oh6HH%8D(kUQcLKH79&Y2O@FN8yX_Ea%yS!?SR;*9_q z46RoPsuJO}$etp`9o!}o?d-rabna}Nz&4U$wb-MI023vE;(|I)VRqC z+}q0zm$`GKj6=1lf_=M*QRp;i>x-x^RbkacNZ=F|Rre#;83SHHeRkc34W#aWiUsl= zONh9dTxPm=5{5pz?`23=Z*Mem;tkM@!7%;70bwTidvCboup$DbHJls7x`5)o2HrlE zw3{kn(ym(2=5k!HsQ{#a*R+V)T%$$UKH0UWXd-cnaHL7mz@%o9ppxM{xr^d0?O%rr z+{pbM1qGmjLk%qEW@h@5i&^Th3zHDw5N+kj6oKL-o>#D=AtS}PS{ZB|PBbsb=aGoo z%Giu|h^rFLk^b~nQ)PG%Y=9r2XP<=yxaR%c4cl2+BlrT>TRaX749s7cyaG^o7S5kW z+%T!H`(cZ{h{b6$)+I-LG(9h5p!*+&BWcyjmDl?Q9;<29phA7FnPY2+=E5=bGbl>sKF{3_TPxX6u*azaEri?mFr15!(nJxciYk^^Zq5Y>oPw=K zq_ua#Ai`#L7-d5+TlA3E)A7kkRV?UnV8x&z;NwSQBRLJ*Q9OTgU7FK&hMSEPyTqkT zb~(g`v9YmQ_@~H#k>G&nC0zWC;ugvl?d$k2XT9vGStM21J;^R14V3FZLGbQ2z=&60X1p-h_`X7^cX{ zVQM^hNDH@=Q;WH4pSQX*E5om_m-313}+AU=CIvQfS16e(SE41y1IJ0KR(G=W`k&4QAdiN$5!OO*L;pi1{eq#aKf6i zIsV-0wMRg@Q*D;u)9yKa*f9ge6eDr>s^0IFIeYeOzy$FQz3AN!V+yvN&1^Q5ozZ&b z^P=$5X^S7ETPbG`4Loe=CW1(g*`8$qjzCUpVpat z1{=+W&Y#u=&gjU^yxRH3CuL>DHK(rmNa_!E6m-nHx_GAxZ7j?L7jJ=ZpFPg<_dJ9ZDjBp>PWO3X;(1VT9R8}h77FVHI|lv+Iv~v|J;-o z8K3B$I&w%h12IIDd257Pa<>q#?1(`;5+}8f{~z{FsBN~{LrGrcVUawqh>_R(y0x_j z3?l#@U9WxGe%=Vszj0%l?#J~xL+!3h&LOr-a~(^elOft2uM$t6J9ij+rq`SbDX<6# zT0<;<1?BqVx{+J|iv5LKL8zxf`cEhQP_N5!3m(;(pHt}E8~t_K2BkyuT>fq=Mtjja z^=`kd@Px{Agvtunpm8XaY#mo3MAbPH%V-hSUxjm%dK)RW7jXq*3l-_@iIS7$rmH#Q zp}GVQK_?$J-ye&0+nSIB&B(U(#fujVpw;zr-o8D`26XmymTFHh2az;Gnw66BKyNk? zJKPypzRU=kfCZ)G0O@2=Pl>{^e~zt)9C(&EIHB4kBHbM-U;7{w4#fbMg$XrDT7@9V zL-5#k4GbJaiva4>7_2l#aoYrJ<$)iJP20IH5(;1f7O3w;+yn(FT3gFZ%(ldni%C@k z`rQ7Vyocn8?Q&`Jx^?Sn$fw{% zeYe-JTmZ*L?u8J?3F^v(sfH=;DF`P6@dpDaF`LlNHIBR<2MhrYG8df-q_YkHA|7CD z7I||5AFqDYU$!iO_z|vC28(gecFN zOF$Ck!ud;c1xt0f4Qq&dVcMCcq&YcD zb4R{gNhYDpiGk30vn?~3WL}7IrvN)kY)?Ar5LSJtElb%fZz6!0eE;o90$3)l?ZXz| zPU6)c`_lP5fn-cwT-pHadIpFSl<41tM#4B`kj5jvfoh3Ac@7=AfQxWUABytyym1lDpKY69Y3M+-`@ut^*Wypu5QE271MS!xMBO+s=s!1aNq z2YC_>P+XC#PRgH^N5!R#G^Y9BBRa5PmAenw`=vp+y$ot5W#}5XXlP7{&}&GGSuN(S z9#^W7*A8@kez^k=1o(NR#u^Y2DQ}D*iB;NpMyfx9U7Iw1(w2}Q9$}vrPUO%siy1or zkH%}y@WnzL#SRwA7($SJ~b(A%)+NN&~t;l zi^AamtKw$SO%#g67B~X&L@mK5a2L3jBYwXkZAeN*;{`;nId<3f#2L_SKq>p=z{q6o ziN3pre8|&7m{#c$Lki_fkQ0#_@Ph(0qb%YFN9iC(H~=oy1P)lB2pmOsWGyC6FOO^D z)3Xz#h!K%>y7n1PU*cy6(ux6^AuY+|;e#sJb@`?Qp|?^fZhQH_loS5OWpTzJryNLu zEO{>$$!s^a91GGGdiMQttGnAg9Bm`MWITSob%^xbB1*k+nz1Cla=_gNXk8>)ptg(! zfO9|-&cy;3XFGRtWB`S7bua1r{37OYY@N{Z6Tkl0(%n7$>*t?;?=FBO_g|lSdu{oO z|MRK;d!Crd6sL4;@%Z>{_>Z3-opNYnA$1R(T-Sr;e;?Jkcp+=A{{Ok(|H-kkLVhkd z&+(vJU2*l|k%tF=9tm*w*s@2M{eiQzjrziX^orNr_e z3U6ls!$T4UP1eoF#|JfG<%s-lX*lIkMZI|WQod}O{{o8p;uOTyJ%0G`DZ+h@ zI5&Ae9&T$obJw=Lcpb5G^ykS9Go1YVib_Rm*SV~Gx*J5*b^uN$W*zFzUk5#u@P+QY zokKTC+l!`Y?2>?hfcycldw4O1`F!PKKb?U!)AjHwzV35$vTLXy)!@PX`Ser(fm&Gy+1eOg+l_`17f_q3V>1(*wHWC+$hV9)L=Hw z&7lnb4)cNvh*lN&W#Pe#g`!UyN7HPFE{CL}KORk$eo(k2?x`Xe9fJ32q+1Ihmbp7041mZqmxWZY)YVQZXn;GTzE_f)O3?iPj917 zomVbSBP#_zn6?8_b{+~qZG74fv5dSv6%a&pT(WWbC=>GvHB6@HU_ENBs# z-E|H}GQ=spc)-rNe}9pLd0-U!9tR-tTm-V10I74PjGoii&+j?VY4f-hdU;@lISGFq zMk%mT08kKV-UvnF<|gT@Zz*VilJ|o-<~cGUsF?34M8`&Frq+KVy*_|hDhSKS$E8lW zfN_T&H2)$$zq|~6uPst zeoODEw)YLoyY=eoYg5!jJ0xOZwAFysGc6D_Pct=Am ziBnH~31IeOdioaPk_K^xfGVnwl-&|i!-$mnn4Pek#QpKP7Yi>@ocGr~n|C74u|t(c zIDQLlf)}M2=^rC~=anhNERJU}bk(Ug6C)kkNqLjOq??bl)sWsL_U5x}2ldXH9h3v8 zc}C9AkhroutK`EuT8{92YhLy_P$=u8Q7FWM+EI5|85JFE1h8xy(b?Vo;?0{kOi6?) zAq*0Bi9gmRso6k%qy$}7^+RLBW;wJ79kj~b{Ogh_vgDJO!;hOLn&4aD`az9pDy2_< zPghs>4bxYioztW>9TI$O#8tba zlbXLcPdg8bDpxW8ukUkv?m^od?saAmQ}#9yrJe%mZ6kaVfa@86T+;VvJKPpaDo3ol z^081v4N?XXiXAwe17Ag$RHUjhTjv$4H+}-}8!tPU^kgZ+088m+Yx=9zTUZi^xd4cr zczi$;#R2Uap`)0H_8_dRoOk`YAPOBr=+05&ie+q*!-5u(bD>8R5etZ6cO3CF7TX)q zJq+J{V@i}v&Tzq!3!#lk&7tMGKxqFDFD51io0YMRmC;*jnm?oOJ`jUu9uOE&X!syR zIq``Srxo~G4gmorZKu(@q>pc)2^sb0;{i*R$1PzG9r}AYcaFBFB328$bX-As%0Zet z(6~mL<1l+U;!JeM2JcWP*LSOZ)iQP^Z{}bukt?KXP}gh_Q+u~>a{2R)KHE7_%Vg*N zbU91?PU%`FrM$dRPHhVOyyE9q{`~WQ>9nNv|G>0AvxAv=XxRMczLWUNC+FAaJ$L|A zAf_3Pr3X1x&JQ)2#vglrzV4R9^47q2y8UhmS9WxHhvSH(B=};#CE~Fv89e=k-Z)l*iHii+A?F9);oHBspCYklm6D&ZS$hJG zwE0d*TVz$0l&j37wA#^i_AhZuUJ~MawP{UM&yG!kTGu{gF+xzfCotlf~r;8 zW~^1x81vPVm35J>L|T!r(!y@-gIx_Dse6_`@WKrkE9D5U^Rr(A?U*5KKU3=uriJ{v7Zx zTM4lZibJl9+C2f3a%PCQi8}MY+V}4`Z)dl5&;yroR7y@>UJ0<2KsP_XZ^PynR=$Xi zKFGs!4w><8e+DbYa&~IOY-;r3!#_yuf;SjD?GZxI3wXi@pg3@US^Mu-mJ;$@P6>^2 z4LAlvME2*x_@d77(DEIjdfTNzGwx!;t2x>bV1#r(tR*%zWIoOx6SAjH$@u|2XxwD| zN%-GDgZu+8ugVOPS3+kHQ{O;JmCFi|l=lNZhIOa#;>nXI6KCBGW)8NOQ}?{)Bf~et z4q_h{k3i@LSv#k2^cGTMS$X-H_V#vE+*}t67RUu*H&xCdep+2)$AL52AG+qpvQ}~R z9KZc4jwIoHC;RjRpT~d(6=4+0l0X#W5J*UsCFvWShNacdn#rfwe(Sd0Qx7(;k77El zpwNiwEs~6jmq`0VcA|BGt}a(`bzPVUB$0s_zL0lFdp2=q!zfokY@jp&Yg~DT{aEP% zn}2B*bf82Xzq_8`iTCf<&-ffeUP*>x7DqTHzxsGJzPs-3IU>99KKw?Y%A@TF$HQzH zq&UAvk5VOYXpq%8_?2Lt$m*Yh|AJ_E;G~H3OPa|4)mcdJ{0{b|OwqN`uqM?CCt0-p zy zN$CUCX;ps&YqyF@N@^M!8bVn5Hunc5aYcdAmsn!6#t}@X%NDvPmM_`O(I3nP(vzhf zJumGS_Puu{b4izW_bgOY(%Ot7iIxlo6g!JoZM!R!flq{bK$kW8;Z7kRlRi8mnStvl zJ+zU~T%fzPwDLD1&TtbQkJRWO20${AxQ>k^PHchb$sqiOYQm*GayWKH&ggyfw*aStuiVnq{8(U-f09jMu>1_X9f497WT4E6c2|+Z!u%Lot1 zmE_Y+z9eM+hfs;TTavASTQ#2&sNBy|jO5dnQbbJzZ+M~ls zcdujJ^1?`p1?UH%)X90noQC3L_o+5wNFrIDP~@O`F}}pp_k!R9a$piVs>kCc;Saa|XeuL?c7Ohy_Xo+w%ms%8U38>-!l$P_oA1SlUdaoeAGciYgoMs{=sp zVC+KLNuGnvigezPk4ILD0Vo$<(@VrDd-15Ks2DKB#3q0}%8eeh$|w9MP{G7u!x?LZ zq*WVDZQ)Du9I4u=?W9N=<$%u#zlVy%2*xG?n$S9ndeF_da}+@@BO?Rn1x!n=bQ^d( z!g%XBZiKk#{G!0oXGvT;vb_DwtD5b7iK+?4+}uHnJr$swaKFO6r=U7~vtFpG@_;NW zvourfO3e1ry{9ygAc$bBC(1VlHn0x?R5*!8oHS9`Yek_WAXwTpA8KzjOc~iuT2Zf` zA1%8NGuw^NuXV^0A~fd`Iw?tF%eEZ~KCplk`W;lKD%#Xht?uK|%Dn_{Va20-WC(#d z>?T6F9JDd5101L#z6QdGlQG3z zo=2!5W)O0S9McrpyreU?d(w4&)P30iE2`C$Hs}>j+J7spGstmZu%-mqg0=Qz$N7CZ zxl)eO8l)!-lm#c)3s}DhEH7X9jU)KER&eL=A9Nyd9yuo{=3bD$`+j^a^ot@pTNG!( z;UTf1_tX(SOP=*n8rQ?9IE72-{9IZ-Kf9izmpn$Vjhs+L$Pm)ZMowJ8@n#5lgv2Iu zl#OnAjbRePKy2`4NTc;G%5_6kQt`5pt&tp6#`?SW<_c#gfz8f!wY3^r+E`gbQY>hI zgoC9Im8Qtx7A(duS|E4*34QPr()fI^-&*CN5nrOM%g4jR(|VmW8RDxxxeg`v5R`md!XzD4(S1&QCq&aEISR#;ZB|6+{Mv_$l82;mj+J+~ zfsYQox~ZtBRCz3NN{jP6$Z|gc8NKC!!5?n#-D!J8drN^noNm{n(y$(QmC^6Pm;VW? z%lJVss#Z0=9uP#NaSVLQC{?LRY>F z{g)JT(2(T(aKM?@uJkfgjcu0*fN0XR>w^qY3yutD;E0lARs6b!hIlv)BG;u@6*JqN z_Uow%8&GoO<4cmzT$;Fqj1*8ze8V`Pkb{fsL8r*CJ6JDT<&jzSsM>>W=66mx5-eF< z=PgK{#5pWPy~3#DTUEl2nm^Z-SHL(24lT(FoB4Ho#yI?$?_#}Z&rn3PUP7}}RMyg?#gX-l+To3Z3-7eH%fCU~b^J+|4;hxy;My;+@@ z3#bHcIppjl>4ot}Fax~AAojr;MA`$1FoWA62jpNUzT7+-|=O97*156Xlid>^PWm^&p$}S!mg>*%1-?8H*$wpuwk>$3oS|&=#xX>vf z%rKvq;#2C{5Jl2BoKyH#akXln`}Y;05K;Be)=>Xir!bLC3@bX9A65f$J_{jUKwl(+ zFV0O=L@I}@&#BbyG(Gd>_e}W`g}a z*2|T<{yNXkHTLhHUsf-Z%K!P?k4yg-Pc;4G<@H#HzR5Yv+b2`9_bVZ9shl|+g~!KjKfNorX)#y?Hb?AYknPrqoQhXqIYez!ZA(0gjV z8Y%i_cBAY^;i!x*0yO+S4hZNOet2Q!El0`2VOgXP`Nz(;;Q zRSB2Ea)F7qa=-t<-g`h*d2U^!7?1J9dSWb5v2T=4L=iz+^dN3g>Afk4C`A+m1Zjye zCJ+&~6f0msnt*hXrcp$?g${y+q7+f8^gGwPHwJT(?|%3G{~hDMV|WB^M47 zMuEN|)i?4hLl|S2di3hRg&=*CHhmY^*Sj-FJAogiNdy%l-y%)2d~cZ+c_bEhg@r9A zdsiSd&RRclKN)gz;dK_-4nj%s7*$CuK&%)buOM-Bu&|Z`nxZ-%$Z1BL6VDY|3lJ|Q zDWpIHmpTYJrRBFh8-gEtM!WT4W;QfTsHT?~Vst zhf6F`uI0En$g<-B43SYII@hTSUmIR}ia&EUOA${zr#90-cX z@P|C2w@Cd3T~KqWFE`gvy5bK57(CbQ9O~Ug72ozmWtk2TGpdS-e+Ps?c0U(aahJIF zqD9LEqP)gNotMo9+=VwqMlLAW-{O`@0Ke4z{rAB`OCby-EiL4T)WXG(oVaNkp_4YOmtDXJ`SD+WY8V9s^Cm zRBiQkSZ6?d-ze*O)bCC%R105l*Yhx?wZ{XdQ|8(|im~t5W89YfmN;z#{=jN2HTiJh zX^>35g?UCswi2!O7`OY86kP0YfCD-*z*Z=HF$IF>;;ADKi~_sIxehBkLsA-NZ00=- z!J_3&KcaHK&<`J*=B{ocQ1J{^IRIH~({>_zro0;dI^;09Xs!8!-YL_WFZ13kBrpHG zc@*RW@qXP2Xw@cZJ0Ye}FOlgz)46(Dtxrhq{&$ze(@MU>gFch-;-$u@kIUF?gFaA%fqu7K}^$^QM3 zt~mfTH{@!N?U;s}Omg9cxbnXnz%722eg>zC+9#ZBwlr{wo4Gm4%MB~-R zle?~Sg(Zop>3P8=?)Wd3ed zB>RBUgFSYF2}>sN4V>=0ka^GfA9CNI3H6@yPix3HP*^)M41k)9UGRW|Yh1H8_pV-i z8(ft^&LlKQ@zrk;t4Pe6dhmHP%{d_Jhg^3;)T!OshWx=@qMNpSr7H&`uy7R?oL z#dvheHhWlUMHmG`k69F6*v5TFFCLa9TB=f!##xVvH3APcc+3W}$ibW%7tr@4&So5U zGq*Kx4>6Y5=kGw>Ei5+5V8;O&VAn~mULZdSb#70s#gJnWeO&qV3g(LV4*mnu5TKg@ zz{0OUJ6ebyCDfg0OOUReu}r}jB*OH>*_t9N}`3_k7j@$smyw z=$)EP|y5AN(rG;tM^var4E)CZztr zkoFIBHIbB+ND~y0tx#>~fu5^dvXr?UZxx}gDViwp8TF||_Q%aYt8WPfk;g4&UO3#w z3cs5{p1*`}7I$OI0606LIus3uo0eJ<^t8tKA5=UeH}n}AT*0C z)@i{O5Kz{97~ZRZx;69I_n%Hre_&}W569`+C`w|LAkJ(F>`h^fd)@d^pOO-+?OB_$?)&+^iF0Bw+c&;SJ{ zSyE0<2*CVVy9|77EAakGJn_&NF5NSJ=p&gMVxK*c|4ttNe@Z>^Ki4t-8tLEvzkYMP z(vWvQcXdCnZX3V)Ke?t=at7u_Tj~(0O`>YB`x#rfv&wtw%Gtan{Cu6l@iIiQUZfb- z;I|A=9Hj05xIuvgnA;ZW?u6$2@D^UQzWwCPN$yz$FVvYXC*k`kYz71yrBqirln7aF zy-z_Ip~rN_eKJBOc{;pOe&6|od(?}q%1gPT$*}zn$9^aSyvkzZphQ>q-+%gweDSbA zq#Sy=nNn_~LC-*QNTEO6$#efW+F ze|i@uYb6r6dU@{40RR`SUc4Nz+YKx`!WfreEOynV;_AJxa-Xbs#0uUdP(L!)=Zh#> z{fP^}8W)W^&8oKEJOK=03g1d@%HX}`)8~6`XDPJ;rzMpT)?I=7-x<64#dPq#TIADJ zoej>Id=$>gM77g^IJVrS~6e+nX@F@rXmM?=!+dgmbXYwxR%F!8Ak2)nvr`U#ReLkoOGUL*YiP*fyT(m_-y0u{-@Hh{{Wc(u>u7mF`(`#{1&NIC-7VhHdvG)!;&xo8W9=aK zK%Z%8dMn9(3WJ`yu<7k^0w{T;-nWxW-L|1h(l~VJ77wB!Q+YJhmO3?Xh1J#d;H3gi z#`A#DO5qv2H_A6xF0FJqJqt)#AYXZ8w-%BmjJ@( zs!@Oq1w%P{M1$I#w4vpM_W00I(kPo|alsV6DvfW)2e{zTgA)G4S3*sML|-&5_ZEx< z!3OUlMSwQC<-0#XA>siReBYij5($IdyDhSVd&$Np@sbPLVMtD@2V(Isn&^d4HyEyw~V| za@v-q!bz}3A)pBhi5^roFe8nG4Mr=lgJ&3CH_44nK_M=@b6oy*SboL*EoEXM@812Z z3jGF$CN)&^z4$it%p)Pp9Y%YC4muE9ni>J)JG8bkf^;XlI znY;8}thFb93sHT*708zgOO+TTGbElvEPDnn8|PR_l2WW2WPa9l~GD4)Rs0|i{s z*9U*zvmdJ>h&7$mQjterL=`=|PziCDo`-(Kjt0bhIB&vMQ3mf|2Q_M9T6C>e=bmMw zwtkm7#*Q`?O1q_CUUX63z)Z@Wy?C2ffT- zM&iYHcZ$2!WN!=itt%t2>Vw!2369`%If<3E4eRY07TH8WdsCpJ=qh6gmX3ZL;%>EE zVdY~IqJS^Q8W9L)lwl3_a^hB^~ zG@HEVULxC33 zZFX#-tmPfnjUC)o?fs$ZnQrT5Y>~)(|Ks)F?m0orB{n|FS@jkHQrY(kwcIvEejh3njzK$;RCvqSlx{N)e5lK&*yQ!+FRv{ zQtMcuF+aB4s**b|gT1@JzC{$Cmt=b6_`==@8l~P;7<{uT)+9q0KxH{DhG0^_sFZf1 zG=7FHT*?3ikGG?}SbO_H$aQF(o{0KUW2>V!|GdSEMY#L@0TQkjwHS|xd*>sZp#0Y) zT{YU;C5Q_|7C_)DdKJkd#pfFL5q0UG3H`nmQ_&50gMekxtM%Z9(NR~kX9x+jDF$R_ z;mEr2Va+|}g@SUn5*{Q#KrCdJQ*(~IeP38eo3lKbYdB#Q+7uJN|3qn`pflgrYJpneJt zHO)MS3nKMBySvvd<)vG4|D2G4REHwTN4C8mZk;r+#^)Q~#5|?&`6K|oDxCEo?4?z> zLyJ#s>mLd2&AA{be}twyS$9A5fC2xVCj4*i`#H)tzAiG46E11U z%TWihNU%f<^7GURXVwkwhyAta9gUsN+B7>YI<9}}X#L5_lB-&G@=KQSYwjo(TDPO@ z(&H|x^D>%l#fdxkYTaC31umP+XVomnJ?C(L`X^OF;f^tN_3bj zwd&;z>Oi^C6;d2|Vn{e}yHOB@Kc!Q=&nSk^-Zs)3?^V3#Fi_m0u7Xon_fwr|jKVLAPQ*3jmriwArp`jV%zR$L2k2+cK&~ z!xW?fOzr{@(~8n|Xg+V1-F64lMw%{Tv?oHbiK8lh;NmNMkS}uKAhRC#M)gdJLb(%> zkg#^y;Nz2qsXD@eSQ)K;O;K6ux|c9kXy?MkRH=dr`|=G_7?edk%`uNBl)Emy+q~AT zul+nQeA>qGfbDS39PCgUg0i~_J4x8KNzw~>^P&fXi;u5W>#d%@A$B8M2NmghY3VfE zwuT%ppR13|M!SM@E3zRf$F7*$p~R}Ku%zkv_}<&`Z=8C41N!=kLKPLghH4uOQ#%q1 z#f5xUB8xGz?U2qHC}!pFyJ%N&Ly9@>!-V5tf_!?e@)=0$;w@`~bB3EB$ZK4X^HCX_ zV)`TFl3OHzs{oy>+#p)=XjVn{bOB=LNJ22B!ukFg{iZ>KHg7n zyhz_BH#Ds`SyiA;-jtT=tf z+LyUDzUe7%y|uNpv@p0g)Y>KmJuLg(PGq|UjV5o)C(QTkzOrG7e+?!hMpfdyFBgj@ z(9q?JR%XVW_lcS#O4dDZo)o4qf316hcVY}h`WW-0WV4oOS7QF7?B<3MA9?&y)^F6R zPRseLg~vu;B&ssM3p;M}Y*G})`yJuDC~e{N!Pb}<2c|36`t%R7esq$R|qGVm?wUN-h&+1uwC z{pLfEKZbOA_Gb*lj}69SuP12`7mkay3FlB!31=qu-OYBg%Tv~ap0xDDezOr#Q9^L7 z@6)X8X)PCN7$|W`tg(&>$KN8V7r>t*C+#pQXeE^wlB0a0ctuJ>uSYgvEY4q4ApQcC z;Vy7L5Wx?I*g&au;pd-!Hg+o4i8W*DYANmu2~!AESl*kKGa`m9%GhXotar4-Kc8Y+ z4tLml*&HvRCmks;ApkTtH$!eJ0KukbX$t;X0`B!$U0vOmf%Ht(RX@$%$(l4*wB{;j z_r}_1wba;SE*>~|ute9$uKX`PY&KA!r#V;I^W*(x&Aw*C=4dDg7pQL8wM)UvICJBS)o^9q4dEza~m@JhhyO; zcq*8)Kc9A?&TE$kvdqQ?D;=?KLK7J0QgrKvj17d2u{oKY9-W0CR72G~yUxA^lb-+i z!NJb!aBxg)s^P4SFTD8=-JN6^Op{t^CU4#@`+VNkl+U|P?UTQKyLKgtKL_Z!18w5c zZ&)^9SI92R{TrH5doMdx2qkQ%+mn^$J)|e%UtUMc?FCvLTBadkF-g!h%d=;2i(>_Q6XkUo8s58)lcnPyZJRuG$&w*D9 za?W8?40i?Ri2mu_(3%a~hRce>FK1WpzjmsN%tWBo-i}?hgYNXL3mw?OX++6vT{wqE zPvoQt11)r<-HN6)R=OluS7o==*rh$+FmvY2f!c=Aimm0$a1)s@+)yU%Kxl$5dwjof zEwB?>Rv|XG{)H`DrIf_GX2QxXi#i#T zq6GaWnYh$KV;69wEwEbh#I#Gt-n&uDXzg+ze!Z+TPjU1^Vj=x5>>=;EPCFl-B1C^f zKi(C)$2%V}77RLCMf#RvqiDI2o@cI5eoNZdXS$di!@88Lp+lc9TQp&GW3R<}eiVi@ zUmY4$xU14UZe&&WLdryHL{8EPZI;cL@dQx$C4?TZ=baGdtU#VciwBkWp8U2Mc7M$7kHnU6yL+$E<90|UomDMxtf2Aj|fWS(6s zjYujC(9zJ0045avGo;#9(CHVQ_8NYjiTH6Iro1@88E88i&sq4fId$+-kby);x3y|2 z{~@J^eoHRmgBUnkgG%JzK&sEmweOUJN=WFwwiq_^eLXe@xkwZg=h}`m& z_x#gOUYp+wa8$wSvXd^LnzM@!-oEF}q^%;t8M6afXizCV5OFyOu_2}B5`EWxd6YfR zF?7YuOe(=rwBybUj&-CwtI~HnAciKX8i!+$Clz|+AR;%pt)|(D!&-T1$9DVha^f&a}Y_<~~N;eTHZA*0AVZEl%O=EEG(@wzp4_qA;Ho+6+wXRtZ=3(_BI-i2Kb5M`w`iI+1y{8De9e|+tv4{6>&BlCB-7`hC<3tLJP6_U49Jam2eW48E}uTY8H8B$xmBCploqW4jvY@_}XhwF<9 z-fbb!cA@N%{Zx91#AhZ(>%|HCyfW#RXcvG#U%YjTtxaiGxw`XLOJsAD^ErJFa(HLL zzbqBgY?@q6oc-7xdkrKIAJHY@NYw=|DR#&H$q;UnPhH-!|4PZm*T3z!uwIDrxI_pmtS~@)J3_HbX<^5SA&5vg z;4G34T&4q3ZlIDxQ?Bf6`5yHfYN7S-;+8uB&u9h+xQk^e92Lahp`Y^OI=<&tI;G)$$TD4?aE;%}YoNWP|S_+H^wIxtjHwvPxe845^j=XssC|iOc zQ0s6EU2TQNX>YQXJ84}|>ZOZCb?1jr53)5VVDRZnj5Mc?MbT((Y`0eEpmmCVVFKgSfY*2mCqD4>f z(lDAgpnmPttdvJSz^+f%9$RE{GvR3qUBK?@8Y6Bgma2i=4CViVp>Etm_k(ZLGTS_m zg2a?Lz_u{eSf=&I>Ee$tn^U*vwISzouC@#&W9oEx=LflEUKNxyfA9+{-xQO1`}nD~ zc&FCFsjmjll6ZFYz6KTD;UX$xv^9<^0^&LlCc__2dll{4)R9~@iu6-TBCAJ}(-?Wp zFSZSBICaK00U0d+Mn7N6?$7FMx_)+-;kzMIpaM+R$yz97t&Nl@Ew^v> zLP-_^WPUjlO2?Ppcsi4Phfk$8Phq2K1b{m?$ExoTs3I zrJ@_XOjdZQ=~(F@vO9;}3uOSNSk98~d3@>6?_W`(I&Imf3jU~zlj3r00W$_r>S_Pj z?+dd1CEQ<5?@W(FQoC@2tv*Qh)Tb?UkIawmj^|#vTvZtUTRmf9`CrIHaOPv`N&l^1 z&?4tK(q`7;J$aV?l_7o<6tr>~MXjW>Sf8Q!a<|~E*F$;BVE=@c+2R@h2%9W2GEEaa zAnir8T!UGZ*&5ABlY{a?!(-7-izjv;atg z{}iaWBb9kb>ARG%60do!@TY~k)==+xh{_*|7Y?Q$%!$5*)euIK`WzoYyDr8`F zIglnc5Pv44D@9wFWiVkZN>gueZ`*58q4@4A*dMu(wl>UD=u4o&Y~YcK0tbOe7RVG~ z2!0!1{gU`O78i_cJL*IQENtpvqHF-oVmB}@0Xo3=uyXeM6A}R?3}7zxZ*FDdnf!L% zb=at0X)<&v5Q=EL*f6}0hNNv`{%X~%22^r}9bI&9kzaH3WzjO=C;&~2gaZp4Vq>+1 z2i~fAT?hrDsQO$QyTDinPXt97v}ik^nVMXgp?!Tm{|KUs79&q~FT%Rl|5>?SZc{ry+PJam< znzEUDm)Xg_lcESZpz0oz?@3~;ppQTWlMh{4ODDd&oeOV85{e7S(8@f1#~nUbx1pFy zLff!jJS4z zxvAvw%7zMP=;ogxS*K_ICvbpVT%oQXQq@Y`;J!@(ZOG<@SlNiY!jOF z#WI+6%z7fy2#4%y0MZVS=(j8z`>0agL=Q&&+W0M4R8L_BI!5Su#}b(}gcc}ZO6`ld z?7W@uqFzs+X4;E&KEuQ5x;$P^H`^Aj$>sn~rMdukS9eBzgk}^Me7m)I{43s0)?93R z>w8tEV&gO9VvSrJ6bfU@VAOdDpJRWK(Dk7h^)V=lobkt;NX0K}n0JdcZ7O^`<<+8;3>F;$j(7y>{^E4SMBp)|UV_ zy3^wIm`PDIprP}KQFClyR^LIDCR*5!7zHr{W#)vVVl&>|p~}#_wgMj{&>s1~Q}f!E z;)~2m3M|#mfLSD@l(_R%K`OhSQ_=gwZ1?3@K=Yx2xqvE3k)3#d)~3A}GwG0mI}l88 zGYiW!iBD1!dM1utPKXHv+Xmw}H-4E!%UUQ>x~W=%&Y+awKlgz` zvl8T|L70!0fMCqppSjbpyCwJO6lMu&C_52?)~ed(Acm${wbmnX3N~gS`IHHm7W`Om9l_o@C_ndAhts91-~V)}lQl7vbrE&^v6)4%2@M`T zsU71l2-H%Ak^N=pqsZKD;-w(U(^!F9(qT%?`jOX-BOVj5DqvlujkJ!#9wroOA9-}z zh-R?pORUEsTWiq_Z-A2W?58L!jgLH0CaAFc-fYkBI*&CBJq67aPVr_hcC52ASWSm6 zxisNf0kHBNaYgaz3i$y6lcL=UAQuB;=&^-We4?+WUN1pu$K@QtX7T=d4R`S=n=%g4 zea1EBPE4t4I=st}%bx|%lr!mHIJftr2<|#wTjSIcA?SlsdppOBE$k0)8>MWicNh

f$^xK{mFOR&(g}hJn_M5} zJCw)c@^epawpn(8c(9z&-gqi78ev|1tR4nqxGXVJ)Du^Qgja&886PgP2`~}Hd6bIZ z9Ga{NYGD_a*Fteh0lXJgF7k*3(7ig?)a^iFC?v8M05Vu&VR!9$qS6iN5fyv+CN9qd zvCgJPw@`43)T4^Z=Qd{lI`A8|x9)wZBmWKso&&5%nm08J>>^~!{l(;Ev{2!HT3XM9 zmt;gmMK$Sn_H`rkN|K#TC-T;*F!YGTkX;LJggu^kdM#urg}RBzNdnbi>FzPw(J(58 zaash&nR6I@y1jKcmJi;K@ymN*3!_N&$%5KG-1WX|r#2WidmzLF2x&>;99aElES0+* zEk)a~R)aW_EKX*equgI9qhjfhI?+^`Y5nch>uBkP*FrF$&rM_9 zieqQN3T`4UM5mOh4)xPD?!s2kCobIN+-_crxq9nCs%14QA&QRLS#fYhXTU*jmm)A{{8y3$bwmnF57hxv~OSw5c;vK7yO8 zQGOPRGmk3SZW-++?K;$vwhrr3(IuI6Y)J%g<3snR5KON7Mn7=7WqQTVbKmVb(Pk@= z+HN%JZKP6SuaS6j7U3HQ>J^ z(slmGZS8;cKhiOJDUSSj7qLV zxau}hCSTn|D_B`(U&TrEpV^Z70GrP@qW|m>V2o~TTqCpT-EE6Py$<@^tyW422xJBy zEsgxo5Ew{kPzQm~pd^ARuTs`vu|aY+SmL^hP>@{&a1gB1HLZH@*_{serYvZD7CL%l zL*1eKRGLu+0sx|$vMtzZLq2*98YGGHR#&bEJY;|y}~ytCXqD}-zs!gK>Yr$oU*Wi@*P66sue z=N(|jWI8ju9RSs`*cHZ(#qF5?=_q7K&yngBrBjJU+)F^<2O8HXp2w@DrBLQg*zusY z?B4fLcOR5lr4WQh<18dGDZrw)Z1bl(R|#Ss`>OKC8iVRI;yI0gNUbi0P$0tP)SFTd0H)p-w7muA2{svI-{9@N<+O~02HftUhpXg>G_Nn}+)c4*iJ_tS2% z%*cEM+;Sooon!-#*bay^wiI??Ufp`IovzN>sM>=Xg$1yct(`dF@l3o@->F4ML#b-c#dR4Z!6=?bhzO;9)PawhX>!BWl7f`wa_*5=w;~eLW!J9xXJI(w>D;O zLmOXgVt1W6?S8%0i71!zc%y}CpcOcKfx-CNg@Q0K@3*nGhUSW@2sHFAsO<*}$jooV ziF33fSPp@EmUL)*1d-r(fL52CWdR}GMPC}N$OuU4HSnGgFpCjaFT{KH#t?^@1cj3& zEN&2YkI)q;JB+mh^7xKCsCvjnYyCZJ&VzNFA%D_}{{OQ><)yI*(ETf3{| z@e@k2h;}6`B?)MJ#FafCe2t2s?=TyD8tec7avyYF0y=lhQz70!zo{HWVFD@lqHrSt z6a2$N#8RtPQrrp#|7)94uFJ)H)E8K%aD^d>DF7%MFUN>heetOs`xJS_ZVRx4|eMnL#4 zv;?@oOGvfkQQoLfN!^SRVHe(G&>d$yW;Ta1tUniv#KE+gjS(!i!1&Pjix$;qT`JyC zD~eqJOOU@-{e_Z%0BK?@K<6~e6KK=}ODCA#+Kl{8cM~Zzp67xQKytBu!v@GM0967M z$HI@o;@V7M2Du8FP(`2~$||}bB|?c>cZkgxo}VQg|Edq&pIIBmo7nYmw+BYxMJgg4azM^utYQH!39kGRm4RRF@6)H zZ-6M^%}g10WG`-@Ec$a)1%n+BErHHhJ~Jt)1xe~c3k}T>Hbe3wnB`AZc>hD4L!lU( zgLHm~5}Q8@3u&oh5p17?t7;}r7Z^%)BJG6d3eb#m6Yl4XA`W`aK%LJjvylS}CTp^3 zMI(kYpMd*@k#{bHb`S;M(^e^w(VnPEvPM7(5$&H#230c_`fumJF?b)yhJ})QxBaPV zRrw4+3_`GmV3PwDa|O076!|kehJxrI!fqXQ2w-#2ZDRE}?0YdqEUyRi|5JQIq6>5x z2Li1P#Y($_=ux0S8`xb`kdj)DHoZmWB}T>R_Bb*^@b;TfY&#)ntzpAMJ9O&Oqsw7) zRsgrwE-U#kxFQY) z;UfSm{F!sAoGwAu5&~Jc0x_H7c#?oyavFVTNWJ-mH>6bYkVFesa*d9>^DS!OTsu{b-K&i1MDw_#6A(3Fol{@ zBFs4=jQDi^gn5TMV+WXxXzoxy1Pq zX$k*?_bF1f`;0lGOEIy}C@Zo1_(*x3zHL>DA zqfrX4Zzes0sm(Uvq>12fnjJ%jj4@{XAdPT7Pp3I@h&SEBp^Q`IP?#mD8etHt^UY^Q z!pl(x;iQmamx*(=K+eZpiKp>qIf>h+;OMZ)yirF?0_6!5WU&d-dBEh9<%AA5u1P`} zv4#z!qVnOsw47q91%}_bQ)?=?$f$8gr}7eZ{&cb3qT3ScWm(gRCI$eQOQQ zyFR}LJOGuqE$6L5^8Z3}8k<@Z-8hYcd*qbyG5dVRHnC*j9S1hk-#>2Uy~-^a9~qf7 zeaWA7cy@#Hj!|gV@q?(F^89Fik}Z#`8X4I#3@P74^8;Q^-rs*2t)NTnU7fN2vX7lYU#zOqDFwR(ay0DK;JEB94kR7q`x(aIeho0 z!zV@2O3Nm=1&2o{jvT;<8hx!ex{7MDxZAIMj2-d?@ah#u-l`P}&ZDcJHIATD)=F|- zuqbb9(>l#-RdA84fqr!SaOnw7qlDLhYVACRPuV$vPbmiOG;^kJ@LDvkWr9O+1zZ_Q zm96D5ifK^ETLCt;Gx;pfF zOyAfMFfP#*88=dfE-9X!+J1Bgw~$OC0}0WPzge=^TavJ!L{8LLMch8U0C^8)i0uq)KSP= zY@d0Cm}`gxT;MhvOi(*zLsUm2HOC5Zvr7hpH9AVIdLC=1R$&~jMx4?M!V5`9uonn> z$=1bo0W4t8wxH2E8_|(af>PunsrP6CV~ltmiY?$k+bYYX0B@6iTQnmbL)= zJ`Xv&KkqsRyj)bdpU3@~(k!+D%JH-rDq^iaeY0yDrVnMvb)Z>dbt&fZbu^$LH9BYy zV4;b-fNzu?C8j3378V*dyQ0j3AncSv|O}H zHOQYw!@BHCvxG_?80b|8)xtnazxfWfFlz+FRZM*lse;V<;Cv%JU51!c0_cx))JQ?t zds_F+=qC`|vcP6RYJ06QPoKyYHcmjiSE%w-f8|=vz$HTr^Ak2eUXX^JabjKNN)51btkd z=e)n~OhEI8FKT~nskau)_-WSVhO2uGxN!a-sIousY?628wF+&*t*XUBh&f}MlX~yt zp6!{DIy~{`_-}LKyu|;DdyqxF?FiwK02_NSgkX#Y84=m2G*dVyL;pUZtPaNj>P-|Q zcF1A^IG}`C%@)7?;zuY6g#`2{?^nM_0|Z4$T2miX2)OT;*{Gj}+^7@FTlNqHg8N8; zSijJZPO&|X7NIcbg?Kj@4g1qasHk&%{<{ZB(PJu^QX+j93>9;;bxRLXKvim!Twd-& zqnubdHiuS#C>xq-D1$b#6QLPM;f=v8(7#!h8(UE}hs2i?tg?Udm)8w7fJc(*n?*ux zNvw4O6(k%zzt&4|Y7k;~ zg-dt{QB~8XunpJBw>{}rL-PARgwzjlieok0F1e=lL5M>Ml)XR=Z)!e--B_x+^0Qe4 zS|GsPX;L_d78wVONh^w^G^=cLM*65npdQ$15tJ7Bo6KTeAcc?bvY_t?8-bYx`XdE1 z7>V4S*V8oZb`u(e>?`=}E?J?Awux*wKLi72ZbWH%G%mBNfOPN%b}8ir;ygc4Am4r> z+eYHUJ>=`V_5=6@C3-B*2rv{QbqC4PuiGj|-$y@peH{Pj-U!>5{^MM}-iQfn0`s9T z@!R-&-p_wiVfg=CllZ@Q1Qd0o4Y@EFVDkOb^(pPsxYz6LTr%(a__cdqXd3Sme!TRy z^(H>0m_vWM{l^>JTWO`GAHg4(C{YuiIJIMDHH}-oBS*8%33_0DA`QNraB9nmDewQr zUm)GVx2Ze+>CVDTzFSTgdP{DGA}zy8C->3?d{i85!M=QUSK9k8*E`}DWr05@QZVQK z>=%n$KIQS5fRVQa57vnC@|!p6#(yXjA`S2fd^n{fkm)e>mTew?X84j~hbclwB@_-Y zN)vqQ@g5p7ydU+X@$-1=yskx;;{LqxzrZK{i>JlCMV%*Q5$)4BH=-={xF@`u!@Wh^ zgv7Gjlf3=zc_sazGoiys_uti9j(ypu`^zUC#eOl%*&3`bg}m=reR>^XOTVqUw(IX3 zRCy^JedmA55V=>PyICtVQDobN8$l*`W_OR_DR6&*S^Jq#?QazwfBOrPw(-vOc$QSJ zP{xnf4_H^v14{MTv*W%Le6TDiG2Xo~T7~a%-2dXNWM&M`7`A19tCsT=_Or4?5&Wn3 zth|Xc=KuAuu6gud>NK=?+%;>mrtdlAXpnFxqN)uCz(y)LXLO{IBIx`kmv`HjBtg*; zp5J>AFlLKl%G(>j$EQesCa_|%uWtL6-3=p zosC|5IqKh~mQ!db;>teYdzhw>^Fxalk8qpQj{k?cBEW=DUyPE#vwP71FZ05CGA&#; zlS*VsDx8Q7BPnG{t9P>@5!ZOV1?WAp`Z;L!=mimspK9Tw>3fkzpC8|IQ0FAl}OGW-PHBRT>_Q$AefL2EWpSWu!uy2s#suiv^7VWVc9E)T~=%Q&MW4OjaO=21mX+Ec2^DK(=Rb9|QU z{`7Z7vvq!byqC>d>lzQE9v>6On%N_xABF`y7&q`~To z5%R<2_uJYQk<1Pdk0ri|%qa+AfI~+Mb~@YDwd-{&if7X+ujebD!NsP|Fvj*eVnEa% z!6<-$Ru&t|DhgSb07jR4FxDdU7Q81#oxnz%p#)(QtdFyWKz~5H(p7_Qbi@rcQ-O<; zS@+F3bWi9|p+k%>&~qzEh@Hrn;CY5iDRoppv)rs|HmFIoyRMryeL=GHnA<$2KsOAY z6s1hA3eYYVLaPxpq=2s21J>4tt=WJH*I_Ph4~%9~HabvO`!n9~ZyOK(HWm1vU*IWz zo*n(qBl(|4LUW6L%KrBbhX1>SP$mV1c*mrXh|&sdx)G*N0vrR-x+fA4U3owzL0`rG94=Hr9~ab`W=`pUli=^OGVfM9t!^$qz1&BBS4raryE zrHg{%-c4l$Oa$hzIash=qHIWH2SF+;9~f>n)WoF~Keu+>jpBClenG^fe4U=X!HSBBamQ^1Y z?D9UugI{?n=hLE_Lhe9d{O8mTPZU$_SbDZ8BiitbkQ!oA22rEuXflHsX8V@=o7R#B z54FG{$VNs;)G1n-Fm=Yaw;qf%bW{O^%yn48!_p@njXeV7NlnC|K1!q33ln>q>+X@Rc2cP<~iY#M^Bt)r002`#$hbRAqPCx3N{C3)J@b_K4%D8kqVNFvN_lMWUbRr z8%1dwRx}`m!E|n)V?z)#8hln!gC`iS>^Tgk`B)=ampe-=xwC{49PrrW`a(hwHt9(r z?U5p(5#y0iOa*L}$Hfa5z`9a%nk=6afY}>(1ZbzyC668=Q`nL&@9K<^*;1W*mPN8e zk|U?Sy2exiwlO?Lwn-bx0dKI4o2G1<=i0MynI31BbWhY}BKUS6#OwH*bO>0$JBqM3NnY4>Ei6|H(s8u+`y(>TZ(lsf9 zf^!DTz0r47*lJY%|^os{zvjJosy(z zT3e_|lQbfvFDLquULhHm_=?{VL=NszIaK&zNrzaT-(aTc1yZq+O98Hx`ixynMDi?+ z&hSs-3llp!HtN?`(GXe5|Bbxgi5jERug&rgHytT!>3jNSp3X_rCq8SJD;IbFXh84L zUYLTxlSMMS5H-;yYYH*MMWSyB#i88&5;!JTtgSK5A$?neL|Wj3=YZyRa!_mxJ3gy` z-Xn2OSQYc(C=h@_m(@NexRshFA(0Cp@f6kG-Q;vcx~O&7=7{lCpnHQ4XzGETiMC5< znWPd9G*wFSqM$q?`pHX&)UoLNaDY)C(+mnW#F5w!5RTO6p&Fy+ClYZ@A%2W@w{nEZ zA|&sZ=KsY1gI9Qt2Kb((@_rPq-`Jy^*E3b|7^|-zTC-9j@Xzf^Z`_hyA zZTgd>4k9WjbIcJdGC2wmilxmF{y>%3%m6+)RWD;IZcHe-ow^0y!uKalEso90cp zB;-O?5$zDycf&OCX5Z^!=oQr=;vUmCA0l@jmJoFom_w~%1{Gq3n-{S70h z$OGlgVM0f)nGPD5u2kCFEjtd(!9+bv3UuCGI5!HgLR{NZp!6X^paoA z*rw5)xtrFf`xgu7g=lZe1Hm1(P17BUCu8#lFxx(?T%<3|gIJm<*I(!8%TgUlhmP47 zp?x0Phg2*_(QM`mXztn@%(5p~x0!cBewne42R5L~WhMIqURV5)V&sORq?7lCD_|^;;JzI zg3`AJ6}Dt0+7|PpEmn`5sB+1xjf|mmhC1&d=K9#Xzh2Dz#u)AfCLIos>@$J1jU zPv>6n3jO$C4bDYwQPgdqjJ7RbG^5O2Y?rL@q^vA?bLW|Z*#pfh*$LDy{E-D(sW%%A0DreN78 z9dtw2QFNNTUsJeV*#}4C3w~X76(UbXgwIYAvNlw4zh@hdD8J;xPRnDgyw?4v_(`d- zfbE0p^Z3;R+GnFPnf+_rWv{yBJ3w2Ri+oxi(*d%~SJ0;04Y*tiJBI6MGmim zQo{%B5J?-cUyzZ4Z$uWXDZ=Pg@>8h3x&EkTo?^W-@uG(U>eCcO{#8kzXaHCE{X9jA)My| ztEqc>6gnEyxb9Kd>Irh@Lb&@vv%fH5q~-SdB`;kGTr)>FlptNUIS`$~@|$NYoE-iU zwDYr<5j?upCME(e!+vb!RCrW5zwhSL*H-mxY}9WkwaZ$Wbv1FJ|J55#$qS15b_A6_ zc~Y;Ps^@Uiuj0`Ma{W4UePi=Ei=d4Sa6Zjl$wn8IOLF-Z+jYGGO&K0vIJ|tlLvxr6 zdTbjYS=3+&DSW&)dp?|Xj+SNWF$d1IPwDCCA8o}Bhf-lhq$#mqvddMqXp z+cZf5Y0$h#COa+BfktK~IkK9PMm`s>QEzVUnB^_u;GMy*yITG!dU~D38vIg z8#z^>Z7<_CHdaCM)DE!^*-$8wa57E(}kLJs{@YMDy zeO(9aRXJQ!h-5^$p7%@_8DwWP6(`onlw6RCsRG9C1F-VWIWlSL(;4Z!2Xey{+UkPC zuJyCUZrUA~dH%#{8_dRIrLRS8uiC>J(^o#kOCZlhPuEZNnMO!u&Pa#7FSK)Hkz-e_ zTYMv!Un*<24zu3HsGO$wu%!2C!+yKofNTAoopCouA3iPH;-`Ei?}KTNkE}CumRdJu z)u$rX$7W7q#PVQQhRtggpuTDOCiLUK8RUrff!o6@8SPR40ioRVz zAy z^{_lmqL`uw3nQW4UFS}?^zRF>Np6}PRF$fB8`bY^XmUy?yoG}*T5(Jki(tqLY)Cwyul}YvR z8;ias7asxVTGGW;!@3EAj)oKFa=Ut7KUrLGh-0L2hyDO#{=&Bq$s%0=8uTzBc8p};0+@)Lm zHoYTVFb8SWJ}@t&#GqzgD>R1|pv$;IH#}_oH4 z)=%)6a6jChj}|%qRIBwoz9h)q(W(FP1%DTCN%d zjmSgIG_`Zzp)V9fjX)pJ0(p7}*x2+Q!R6_1uKMam{S1|*JX_Lp_#-{4vQc{E7%01- zhynHyifX|!*C*CNKGArpB4F zXmdIi)t`<dnN6NeW2a$ZN5|JBnZ_Vw}7o9)w^$wC8n*Mb$eGIGES>29E_ zmMdIV#d{T|E57W*1TBB=^)GZvi*#k8h|f5Cy{WkB>imPpAk2w2F+Zs>-9UU;`}ZHG zOb<$LCfzUGx8{WY$d5h-&R%$Qnb-~?*8(>*)JQF$jOkj-rUrv1a>>mjqeZyW7QpE- zc*r4@MjiEi=cb~BS4GdS9+kd9W{2jJx2VJ2NmKVCi!1!)>s91LBI z(nw~dDu0?^Pc5-k=8KrEpJfXtMzQxP3y3PDC;#x4=$ME2Wu3N3}B{ZRrO(!pG%@ilpO#t z%dJz=q^VeG{zxojE)c8{du$NK299PF4JPFIVy$rF*olvVGvS zs*_TIS8@~iJ|v{3EBjGiE`YZ20Mi?aV=yrdep|9B)p7_!#fKQms9yLyU8M>p0bOkK z)etPERjw%@n3w6tJJeV- z7hV~j<_)?YlsIL2 z##F-y!Ypf3EHfn82MTH#Hvq4)Gt^2#9yS$;9cc0^qRCAjK%`YI>;s)H0%l#0dT+>^ zmC%EjiiQxZQV0=bat9w2$EduIa0b6fMge3GYGKilRaclG8$FV_6lbAkLpm0eyj|h- zQ9MjTg~REknB;d(518!3pp9{Mq3smpFc}Y{IqoeCHup$ z_>pW^kt zte!6ZAXoX{S<)K69riB~w!XUg*OGtLRTXVGuKcf+KTRs$Ev#tg`hT@|rBO|uX?yB7 zGkUPLc1E!Xib_B$Rs;lDV~Yz06c-d(Lp2bDu!tx#Aui1I;C$Mi)6+A5zJDLb<4NAUyu9y|=eh6ux$gVEuCUpc!4cx% z;DM$mQ-ZCpW_BeFRK9vTQO2#1&BSr`?)3mp#z!#jDa%OV?&Ccza*2UnPaO(7U682W zM7y|)(nfXh@W{vP37&%5!(BWA18688d?RgW`q1uFpEQ=Ays(QRn6$2%e!h^HXm@nm z`GZ5-cC51!YF^%a_f{>#6Fs}yhYwU`7oHrW+uicdk5bSU*4Ea#;|nLBlb*%+DS_i` ze%y5p-bm7h?)pMw@BI0wlHs4|HOnn1;PZG|0IBOnP(RP_0@trUx;OaZ%+TafxFtR+ zE+G4u0I|R}ASt8pRy@S38I@jZVoQ(x>(HS?u?&^!H+7GPU-<=}nDNj(;jUb@biv&- z!-|qs)zinLR95zU8Dml=c-$)Ym1$^bL)EZi_$9_u8~$0AQqTR8H3dsc)~5Mv@O3|L z^6ksbNBRtTk!Q1%*H0+u3sqhyy&7$IflY{ksl07wZl0f)R}Zh?jC@Sm3ZsA<5nCO^ z>!lm2-sxA;-fqKAm^%34N`iuo@Saa}c7)E%X{GLG`lbSIckEvMdB4bZ+TYit+IOFu zuP;o0#Pzxxt~4`ays_s?;Cjg1x*Kn9PyY+udJV?stPnQ*s&nFdorRR668A_AOi4-CljtoPLQe&Ns z;32ZW`eQHk17s=21{cTau?(YWLG*lmQpFMaaw6F1!;_sE8^xcqZdrx8z0=lcvvy!M zyN;#pRx1JZNrb=?3X1x(+bD^<*C*}{z4Z^jdAvwdm;O>uh#c!3zkG*Lb5A_mV{%fZ zI^bo7ykzPFl^Hmy7}rC~om= zgstD~KVp3$K0X(M^=m+zN<8EstqvY^BIiy@1khqdO)pEMXXe32(P z@-RD9n;DfAGZUxj^P3Z|ah~~!d<0U>3ab1X-Ce6SXgqY=wP1&s$K#0vU{{S4=iwG8 zt>9oWaOS;+#q&hdXhn)~u# zHU}lovR-+P6DI1Y1j0?X(I2#);{^H5m^Y{B+7?kM7WBm?Yb*Pbmm7vutokQ4Y36~R zj=w?Rw+m9FNnUOvQpxv=WykuqbT_u#;&JFj)zHyN%>Ymi<#JD~O!lJk5Pn zJsmK}7`n7_Vq8Sy&Ri2(C~GXZ62lDroxkyZoHEsJp%g8dlFQ}p4St7AmG1Fhz9B)5 zJHG1_t=1i_rQYc@9%(c%iXVP{BXEMramuyImo;}wlEjGeJ=g?!sMBM@-Kd3>{JpF) zB_GihAAy4nC|r7;Z^CNiI3ZCJrk(m4O-9aaceEMBrv1zkGJUla$MA$)g^+q-u=0a9 zO!u8@izJ=6^`ytu?iH^d*Lw+>4qY^^P2dp33dQ=jBXy|Bi~A086_N{uVSf)9IW^N* z-KcNe*PF?qPw1q_v<8f&Cw4_X7;w_2!+z3x+6PUWMK=-luH68Rp-)C^C!zd+_f;HuO zL4m~&+f5*7Q|oHqIo?+8mrEH$z1c%OCaU#v3JRK0_*D#+pvZnsO~yOauCjnTkj{vn znJ{EYcCq1g*;CV?Tq^b(w!%=GV;eB|$VGC|TP|hyGPAXny~!rZPr^l}HIBlse6<|b zrARM&$%Wt}Wz{f+EML>LmCn6|F{fQR`_3(Cp7ve+AiGn`;z(50X{x^f;NFn2QEzWP zr*nMIm6$=3!So)D?j;6^8{;g3>b8VauPw}s38w@X{ln;S)?VcfAgg9-9D{(Dm(dO? zM-w5l)d;iLfsiu@sfMDJ?JX_W0eW6~NLY9L2K=yI@-jv^NfaXDSdQrN$;w+u$xI>) zb4B>!8dC-Mo}B@!>ozmqOTxBxA5~RV+ahKZ!4N)+sO7W`U0!Z5?2SrPD@#ib zvbmel*UPW|TuJXTln4hwb(0v1DiXT%4f^iRp_0Y;Eyhv|`lbw!#+ZSIG7cCtz0U&a#D;`SrdaHV?p(DRsOXO3`H6RK#T0c%!Wb*{r_Va%9p!xu?JSGp@TD=Uu|n=2Setn3!>NJ_P^}xpg*b~0hFN&W}RY5d|X`1k4x2go&GPA zE6BR@si`xE31aP3Ca?ap7=Pzr^CVX6g)HK+QdH`#yK%~sOG7#g?dIud<__gMBkBDW z6}}>tfGO3Cuc(mDtzlm+y!_R{!o`K($^{M%b5aspk<848L&DvTg85)Gjn5Kq)Mkb8 zhChYi5C8e|qwrqEh0?;++BH?18wNX%#Yqnh1xJSiSrlM_GZj?7&AL@cXR*@A#Rzvy$fm#5W`Z@qh6 zo@8*K`y_`OvUwsoKK}jS;c~?1M4PO|^)ke#(jQgtrpv6)1<4)%?%C~di11!OERcgj zFjs7NmKm!(tXeArwoO9f-*02C&vNO;=oYyc0*-TM*Rf7yL#t`CUGs@ra&nAYJ zk}D9<;i8&RHcv}{2yLKexB~G zpTmFrQ4SPZ11Jlz3=Av-Urg9z1`?i=1bOsLb?0=T2bu^SFH+Fj-p==ULO{BphB!E= zT?d2`=#8BLWVZr@__HF!ZlY9d`p%(5y9tWoi@w}+`A+{hTa4(COZM48h`^+1Sl1iSN45at0)%e4UY*^3{vQT}MVo0}tAd@-&Go>3cSx_*xc~4f3)wlPm za%k%7*Ym&$+YX*24Ver`=}&|fcv6D*2uvY6+UAoMD$6EuEKzdEd>bXYcv0T2^!OE8 zZIEFlcOvPqxp_pC;+BV$R`lIGoeE&FSnc`qhlhp?d|ym?Y6SGvE973kzKO;MoLAf* zt-RF}eK|U`DbyWp7gUK6z$U;xB7iL-OW+#kh(AoBRI{EeqEfh=57y&`@T?(>M}kf& zwa-#mAN%VQ6D47U=YD+i$%G$e0O4p7S{a(#Td^RH z!OeQiqRnKMj?L(g2wQW&Yj(id*|`nc)<8%WmGJp)HEdM{LmMu9{9EL48lT^b(1C<6 zkpxF{V##2n6Snc|*a?x8|B01VXDi@#fG+p~#?8HUZ71f*a@+`@C=a3`+5~cxTvC+S zblpm=Yn<>Ww9oN$w-J}9 zFbr(;Ql#fKLN;8Yno+eL!iL%acWl3eLY8m!PV-r1jUAsxKlm7 zWt^4ch+&?X)|d)?dB^dYqs&?_#c1?tr%9c0q?QezyAm#6pm45Hrcn)is+)8{=0Q01by;&a}51C-XvH`zLF=L?5!UvPG$9enrJovZe6z z^YaI66Y*cojoc-lzjrPF_2dXjNF Operator-only benchmark. No model checkpoint or serving engine was used. + +## Environment + +| Item | Value | +|---|---| +| architecture | gfx942:sramecc+:xnack- | +| extension_symbols | hip_deterministic_logp_tile_stats, hip_deterministic_logp_backward | +| git_commit | 1c73a427881252871b46936f6d14be36a22e307f | +| gpu | AMD Instinct MI300X | +| gpu_count | 8 | +| hip | 7.14.60850 | +| native_collective | torch.distributed ProcessGroupNCCL (RCCL on ROCm) | +| native_single_gpu | torch.logsumexp + gather on FP32 logits | +| python | 3.12.3 | +| torch | 2.12.0+rocm7.14.0a20260608 | + +## Methodology + +- Qwen3 vocabulary `V=151936` split into 64 tiles of 2374 columns; seeded logits (`randn * 2.0`), random targets, every seventh token inactive. +- Measured paths: + - `native`: `pytorch-vocab-parallel-logp-ws2`, the WS2 vocab-parallel reference operator: a PyTorch tile loop for the per-tile FP32 `(max, sumexp)` partials, all-gather of the partials, fixed global tile-order merge, and a PyTorch autograd backward. + - `strict-hip`: `rocm-vocab-parallel-logp-ws2`, the same contract, transport, and merge with two HIP kernels: `hip_deterministic_logp_tile_stats` reads the stored BF16/FP16/FP32 shard directly (8-element vector loads, no FP32 copy) and `hip_deterministic_logp_backward` produces the gradient in one fused pass. +- Distributed: one process per GPU; the WS2 operators all-gather per-tile `(max, sumexp)` partials over RCCL and merge them in fixed global tile order; CP ranks shard tokens and never enter the merge. +- Forward returns the selected-token logprob and the vocabulary LSE; forward+backward computes `grad_logits` for `sum(active * logp)`. The WS2 operators run with `validate=False`; the `validate=True` production entry point is measured separately. +- Single-GPU timing: GPU events, median and p95. Distributed timing: synchronized wall clock, slowest rank per sample. Peak memory is the per-call increase in `torch.cuda.max_memory_allocated` (distributed: max over ranks). +- Accuracy is against an FP64 `logsumexp` of the same (BF16-rounded) logits. Repeat = two identical calls are bitwise equal; batch-invariant = a row computed alone is bitwise equal to the same row inside the batch; TP-replicated = every TP rank holds identical bits. +- 5 warmups, 20 measured forward samples, 10 measured forward+backward samples. Raw medians, p95, minimum, maximum, and every measured path are in `results.json`. +- Tables show 2048 tokens; the figures cover the full token sweep. + +Reproduce this report from the repository root: + +```bash +python benchmarks/benchmark_rocm_logp.py \ + --warmup 5 \ + --samples 20 \ + --training-samples 10 \ + --report-paths ws2-reference,ws2-rocm \ + --rename ws2-reference=native,ws2-rocm=strict-hip \ + --report-baseline ws2-reference \ + --table-tokens 2048 \ + --output-dir benchmarks/results/pr328_rocm_mi300x +``` + +## Key findings + +- Single GPU: `strict-hip` is 5.97-7.23x faster than `native` in forward and 4.94-6.83x in forward+backward, with 0.15-0.33x its peak memory. +- Distributed: `strict-hip` is 1.69-4.60x faster than `native` in forward and 1.97-4.64x in forward+backward across 6 TP/CP topologies, at 0.15x the per-rank peak memory (absolute forward 0.773-1.041 ms). +- The `hip_deterministic_logp_tile_stats` kernel alone is 10.2-10.6x faster than the PyTorch tile loop and allocates 57x less transient memory (it writes only the `[tokens, 64]` FP32 partials). +- `strict-hip` vs `native`: tile maxima are bitwise equal; sumexp partials differ only by FP32 summation order, so final outputs differ in 0-65 elements per case with relative-L2 0.0e+00-1.3e-08. Both paths are equally close to FP64. +- Repeat bitwise: yes; batch-invariant: yes; all gradients finite: yes. +- Distributed: TP-replicated and repeat bitwise on every topology: yes. + +## Single-GPU logprob (BF16 logits, V=151936) + +### Forward + +| Tokens | Path | Median (ms) | p95 (ms) | Speedup vs native | Peak MiB | logp max-abs vs FP64 | LSE max-abs vs FP64 | Repeat | Batch-inv | +|---:|---|---:|---:|---:|---:|---:|---:|:---:|:---:| +| 2048 | native | 6.4801 | 6.7318 | 1.00× | 1245.0 | 1.557e-06 | 1.382e-06 | yes | yes | +| 2048 | strict-hip | 0.8964 | 0.9366 | 7.23× | 2.0 | 1.621e-06 | 1.318e-06 | yes | yes | + +### Forward+backward + +| Tokens | Path | Median (ms) | p95 (ms) | Speedup vs native | Peak MiB | Memory vs native | Grad finite | +|---:|---|---:|---:|---:|---:|---:|:---:| +| 2048 | native | 12.8275 | 12.9236 | 1.00× | 7715.7 | 1.00× | yes | +| 2048 | strict-hip | 1.8781 | 1.9318 | 6.83× | 1187.1 | 0.15× | yes | + +### `strict-hip` versus `native` numerics + +| Tokens | Mismatched elements (logp+LSE) | Relative L2 | +|---:|---:|---:| +| 2048 | 65 | 1.098e-08 | + +## Single-GPU logprob (FP32 logits, V=151936) + +### Forward + +| Tokens | Path | Median (ms) | p95 (ms) | Speedup vs native | Peak MiB | logp max-abs vs FP64 | LSE max-abs vs FP64 | Repeat | Batch-inv | +|---:|---|---:|---:|---:|---:|---:|---:|:---:|:---:| +| 2048 | native | 6.0009 | 6.2081 | 1.00× | 58.0 | 1.747e-06 | 1.271e-06 | yes | yes | +| 2048 | strict-hip | 1.0045 | 1.0516 | 5.97× | 2.0 | 1.747e-06 | 1.271e-06 | yes | yes | + +### Forward+backward + +| Tokens | Path | Median (ms) | p95 (ms) | Speedup vs native | Peak MiB | Memory vs native | Grad finite | +|---:|---|---:|---:|---:|---:|---:|:---:| +| 2048 | native | 12.5812 | 12.7740 | 1.00× | 7122.2 | 1.00× | yes | +| 2048 | strict-hip | 2.5475 | 2.6220 | 4.94× | 2374.1 | 0.33× | yes | + +### `strict-hip` versus `native` numerics + +| Tokens | Mismatched elements (logp+LSE) | Relative L2 | +|---:|---:|---:| +| 2048 | 37 | 8.004e-09 | + +## Distributed vocab-parallel logprob (BF16, RCCL) + +### Forward + +| Topology | Tokens | Path | Median (ms) | p95 (ms) | Speedup vs native | Peak MiB/rank | logp max-abs vs FP64 | TP-replicated | Repeat | +|---|---:|---|---:|---:|---:|---:|---:|:---:|:---:| +| tp2 | 2048 | native | 3.5356 | 3.5972 | 1.00× | 650.3 | 1.452e-06 | yes | yes | +| tp2 | 2048 | strict-hip | 0.8436 | 0.8830 | 4.19× | 3.0 | 1.452e-06 | yes | yes | +| tp4 | 2048 | native | 2.3684 | 2.4618 | 1.00× | 353.0 | 1.452e-06 | yes | yes | +| tp4 | 2048 | strict-hip | 0.9375 | 1.0703 | 2.53× | 2.5 | 1.452e-06 | yes | yes | +| tp8 | 2048 | native | 1.7633 | 1.9199 | 1.00× | 204.3 | 1.452e-06 | yes | yes | +| tp8 | 2048 | strict-hip | 1.0407 | 1.1309 | 1.69× | 2.3 | 1.452e-06 | yes | yes | +| tp2_cp2 | 2048 | native | 3.3620 | 3.7359 | 1.00× | 325.5 | 1.452e-06 | yes | yes | +| tp2_cp2 | 2048 | strict-hip | 0.7731 | 0.8305 | 4.35× | 1.5 | 1.452e-06 | yes | yes | +| tp4_cp2 | 2048 | native | 2.2469 | 2.5062 | 1.00× | 176.7 | 1.452e-06 | yes | yes | +| tp4_cp2 | 2048 | strict-hip | 0.8380 | 0.9284 | 2.68× | 1.3 | 1.452e-06 | yes | yes | +| tp2_cp4 | 2048 | native | 3.5902 | 3.9379 | 1.00× | 163.1 | 1.452e-06 | yes | yes | +| tp2_cp4 | 2048 | strict-hip | 0.7803 | 0.8550 | 4.60× | 0.8 | 1.452e-06 | yes | yes | + +### Forward+backward + +| Topology | Tokens | Path | Median (ms) | p95 (ms) | Speedup vs native | Peak MiB/rank | Memory vs native | Grad finite | +|---|---:|---|---:|---:|---:|---:|---:|:---:| +| tp2 | 2048 | native | 7.1279 | 7.1917 | 1.00× | 3857.9 | 1.00× | yes | +| tp2 | 2048 | strict-hip | 1.5376 | 1.5811 | 4.64× | 593.6 | 0.15× | yes | +| tp4 | 2048 | native | 4.4086 | 4.5749 | 1.00× | 1929.0 | 1.00× | yes | +| tp4 | 2048 | strict-hip | 1.4017 | 1.5074 | 3.15× | 296.8 | 0.15× | yes | +| tp8 | 2048 | native | 3.3382 | 3.7182 | 1.00× | 964.5 | 1.00× | yes | +| tp8 | 2048 | strict-hip | 1.6933 | 2.2111 | 1.97× | 148.5 | 0.15× | yes | +| tp2_cp2 | 2048 | native | 5.4500 | 5.4998 | 1.00× | 1929.0 | 1.00× | yes | +| tp2_cp2 | 2048 | strict-hip | 1.2361 | 1.6392 | 4.41× | 296.8 | 0.15× | yes | +| tp4_cp2 | 2048 | native | 3.7984 | 3.9900 | 1.00× | 964.5 | 1.00× | yes | +| tp4_cp2 | 2048 | strict-hip | 1.3963 | 1.5687 | 2.72× | 148.4 | 0.15× | yes | +| tp2_cp4 | 2048 | native | 5.0433 | 5.2447 | 1.00× | 964.5 | 1.00× | yes | +| tp2_cp4 | 2048 | strict-hip | 1.3849 | 1.9775 | 3.64× | 148.4 | 0.15× | yes | + +### `strict-hip` versus `native` numerics (distributed) + +| Topology | Tokens | Mismatched elements (logp+LSE) | Relative L2 | +|---|---:|---:|---:| +| tp2 | 2048 | 54 | 9.958e-09 | +| tp4 | 2048 | 54 | 9.958e-09 | +| tp8 | 2048 | 54 | 9.958e-09 | +| tp2_cp2 | 2048 | 54 | 1.069e-08 | +| tp4_cp2 | 2048 | 54 | 1.069e-08 | +| tp2_cp4 | 2048 | 54 | 1.162e-08 | + +## Figures + +![Single-GPU latency](single_gpu_latency.png) + +![Single-GPU peak memory](single_gpu_memory.png) + +![Distributed latency](distributed_logp_latency.png) + diff --git a/benchmarks/results/pr328_rocm_mi300x/results.json b/benchmarks/results/pr328_rocm_mi300x/results.json new file mode 100644 index 00000000..e9c27234 --- /dev/null +++ b/benchmarks/results/pr328_rocm_mi300x/results.json @@ -0,0 +1,3366 @@ +{ + "config": { + "samples": 20, + "training_samples": 10, + "warmup": 5 + }, + "distributed": [ + { + "cp": 1, + "forward": { + "max_ms": 0.5072727799415588, + "median_ms": 0.42468123137950897, + "min_ms": 0.4007662646472454, + "p95_ms": 0.5017554387450218 + }, + "forward_peak_mib": 222.5634765625, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.2246337153243303e-06, + "logp_vs_fp64_rel_l2": 3.358224717653215e-08, + "lse_vs_fp64_max_abs": 1.2246337153243303e-06, + "path": "native", + "rel_l2_vs_reference": 1.4190225998557055e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 256, + "topology": "tp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 0.7819016464054585, + "median_ms": 0.703359255567193, + "min_ms": 0.6873710080981255, + "p95_ms": 0.769756268709898 + }, + "train_peak_mib": 296.75732421875 + }, + { + "cp": 1, + "forward": { + "max_ms": 3.5062278620898724, + "median_ms": 3.322325646877289, + "min_ms": 3.264498896896839, + "p95_ms": 3.463823697529733 + }, + "forward_peak_mib": 81.8603515625, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.192615400213981e-06, + "logp_vs_fp64_rel_l2": 3.3594778807350763e-08, + "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 256, + "topology": "tp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 4.906749352812767, + "median_ms": 4.854129860177636, + "min_ms": 4.76876413449645, + "p95_ms": 4.896933306008577 + }, + "train_peak_mib": 482.30419921875 + }, + { + "cp": 1, + "forward": { + "max_ms": 0.8099130354821682, + "median_ms": 0.7396885193884373, + "min_ms": 0.7042759098112583, + "p95_ms": 0.7940537761896849 + }, + "forward_peak_mib": 0.37548828125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 9.63235819995134e-07, + "logp_vs_fp64_rel_l2": 3.3267799957748666e-08, + "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "mismatch_vs_reference": 5, + "path": "ws2-rocm", + "rel_l2_vs_reference": 1.0160969201330157e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 256, + "topology": "tp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.3461587950587273, + "median_ms": 1.1561571154743433, + "min_ms": 1.121405977755785, + "p95_ms": 1.2802621582522988 + }, + "train_peak_mib": 74.19873046875 + }, + { + "cp": 1, + "forward": { + "max_ms": 2.130434848368168, + "median_ms": 1.3161206152290106, + "min_ms": 1.290236134082079, + "p95_ms": 1.9846681272611024 + }, + "forward_peak_mib": 1780.5078125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.3102978989626796e-08, + "lse_vs_fp64_max_abs": 1.3112566623618704e-06, + "path": "native", + "rel_l2_vs_reference": 1.261659473422803e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 2048, + "topology": "tp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 2.777086105197668, + "median_ms": 2.7411116752773523, + "min_ms": 2.6900162920355797, + "p95_ms": 2.7740074321627617 + }, + "train_peak_mib": 2374.0498046875 + }, + { + "cp": 1, + "forward": { + "max_ms": 3.6533367820084095, + "median_ms": 3.5356120206415653, + "min_ms": 3.4904242493212223, + "p95_ms": 3.5971936769783497 + }, + "forward_peak_mib": 650.3125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.314942529180228e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 2048, + "topology": "tp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 7.222007028758526, + "median_ms": 7.127888035029173, + "min_ms": 7.024975959211588, + "p95_ms": 7.191663910634816 + }, + "train_peak_mib": 3857.9091796875 + }, + { + "cp": 1, + "forward": { + "max_ms": 0.9156097657978535, + "median_ms": 0.843642745167017, + "min_ms": 0.8321460336446762, + "p95_ms": 0.8830241858959198 + }, + "forward_peak_mib": 3.001953125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.3108948497165656e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 54, + "path": "ws2-rocm", + "rel_l2_vs_reference": 9.957929192632568e-09, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 2048, + "topology": "tp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.5859352424740791, + "median_ms": 1.5376345254480839, + "min_ms": 1.4071101322770119, + "p95_ms": 1.5810548793524504 + }, + "train_peak_mib": 593.5771484375 + }, + { + "cp": 1, + "forward": { + "max_ms": 0.4723607562482357, + "median_ms": 0.4185016732662916, + "min_ms": 0.40345918387174606, + "p95_ms": 0.4652172327041626 + }, + "forward_peak_mib": 111.2822265625, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 1.2246337153243303e-06, + "logp_vs_fp64_rel_l2": 3.358954781712721e-08, + "lse_vs_fp64_max_abs": 1.2246337153243303e-06, + "path": "native", + "rel_l2_vs_reference": 1.3632370710873585e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 256, + "topology": "tp4", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 0.8185659535229206, + "median_ms": 0.7887370884418488, + "min_ms": 0.7499642670154572, + "p95_ms": 0.8185255108401179 + }, + "train_peak_mib": 148.38232421875 + }, + { + "cp": 1, + "forward": { + "max_ms": 2.1284017711877823, + "median_ms": 2.066188957542181, + "min_ms": 2.0460491068661213, + "p95_ms": 2.0982032641768456 + }, + "forward_peak_mib": 44.40966796875, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 1.192615400213981e-06, + "logp_vs_fp64_rel_l2": 3.3594778807350763e-08, + "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 256, + "topology": "tp4", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 3.6968719214200974, + "median_ms": 3.3774820622056723, + "min_ms": 3.315543755888939, + "p95_ms": 3.6180405411869287 + }, + "train_peak_mib": 241.158203125 + }, + { + "cp": 1, + "forward": { + "max_ms": 0.848931260406971, + "median_ms": 0.7611459586769342, + "min_ms": 0.7448559626936913, + "p95_ms": 0.7933490909636021 + }, + "forward_peak_mib": 0.31298828125, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 9.63235819995134e-07, + "logp_vs_fp64_rel_l2": 3.3267799957748666e-08, + "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "mismatch_vs_reference": 5, + "path": "ws2-rocm", + "rel_l2_vs_reference": 1.0160969201330157e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 256, + "topology": "tp4", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.4106249436736107, + "median_ms": 1.3137920759618282, + "min_ms": 1.13757885992527, + "p95_ms": 1.3786639086902142 + }, + "train_peak_mib": 37.10498046875 + }, + { + "cp": 1, + "forward": { + "max_ms": 0.9429808706045151, + "median_ms": 0.7553219329565763, + "min_ms": 0.7309648208320141, + "p95_ms": 0.926045305095613 + }, + "forward_peak_mib": 890.2578125, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 1.7274325472271812e-06, + "logp_vs_fp64_rel_l2": 3.3315658627415676e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "path": "native", + "rel_l2_vs_reference": 1.322253230879159e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 2048, + "topology": "tp4", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.5383749268949032, + "median_ms": 1.5194814186543226, + "min_ms": 1.4623119495809078, + "p95_ms": 1.537352125160396 + }, + "train_peak_mib": 1187.0498046875 + }, + { + "cp": 1, + "forward": { + "max_ms": 2.864705864340067, + "median_ms": 2.3683831095695496, + "min_ms": 2.306436188519001, + "p95_ms": 2.4618386523798113 + }, + "forward_peak_mib": 352.98681640625, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.314942529180228e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 2048, + "topology": "tp4", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 4.653112031519413, + "median_ms": 4.408588400110602, + "min_ms": 4.365826025605202, + "p95_ms": 4.574884986504912 + }, + "train_peak_mib": 1928.99462890625 + }, + { + "cp": 1, + "forward": { + "max_ms": 2.37527908757329, + "median_ms": 0.9374623186886311, + "min_ms": 0.9183650836348534, + "p95_ms": 1.0703487088903796 + }, + "forward_peak_mib": 2.501953125, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.3108948497165656e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 54, + "path": "ws2-rocm", + "rel_l2_vs_reference": 9.957929192632568e-09, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 2048, + "topology": "tp4", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.5082103200256824, + "median_ms": 1.4016875065863132, + "min_ms": 1.3047782704234123, + "p95_ms": 1.507394341751933 + }, + "train_peak_mib": 296.8271484375 + }, + { + "cp": 1, + "forward": { + "max_ms": 0.45378413051366806, + "median_ms": 0.39185211062431335, + "min_ms": 0.3761090338230133, + "p95_ms": 0.4420726094394922 + }, + "forward_peak_mib": 55.6416015625, + "grad_finite": true, + "local_vocab": 18992, + "logp_vs_fp64_max_abs": 1.192615400213981e-06, + "logp_vs_fp64_rel_l2": 3.3019515775307595e-08, + "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "path": "native", + "rel_l2_vs_reference": 1.2835542362061829e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 256, + "topology": "tp8", + "tp": 8, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 0.8295630104839802, + "median_ms": 0.7557619828730822, + "min_ms": 0.7264092564582825, + "p95_ms": 0.81349175889045 + }, + "train_peak_mib": 74.19482421875 + }, + { + "cp": 1, + "forward": { + "max_ms": 1.6495096497237682, + "median_ms": 1.6037765890359879, + "min_ms": 1.5587848611176014, + "p95_ms": 1.6448863549157977 + }, + "forward_peak_mib": 25.68408203125, + "grad_finite": true, + "local_vocab": 18992, + "logp_vs_fp64_max_abs": 1.192615400213981e-06, + "logp_vs_fp64_rel_l2": 3.3594778807350763e-08, + "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 256, + "topology": "tp8", + "tp": 8, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 3.228455316275358, + "median_ms": 2.7676159515976906, + "min_ms": 2.665730193257332, + "p95_ms": 3.08224034961313 + }, + "train_peak_mib": 120.58544921875 + }, + { + "cp": 1, + "forward": { + "max_ms": 1.0527628473937511, + "median_ms": 0.8622754830867052, + "min_ms": 0.835892278701067, + "p95_ms": 0.9595537558197976 + }, + "forward_peak_mib": 0.28173828125, + "grad_finite": true, + "local_vocab": 18992, + "logp_vs_fp64_max_abs": 9.63235819995134e-07, + "logp_vs_fp64_rel_l2": 3.3267799957748666e-08, + "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "mismatch_vs_reference": 5, + "path": "ws2-rocm", + "rel_l2_vs_reference": 1.0160969201330157e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 256, + "topology": "tp8", + "tp": 8, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.5623611398041248, + "median_ms": 1.4463732950389385, + "min_ms": 1.2542130425572395, + "p95_ms": 1.5456543304026127 + }, + "train_peak_mib": 18.55810546875 + }, + { + "cp": 1, + "forward": { + "max_ms": 0.9086602367460728, + "median_ms": 0.504503957927227, + "min_ms": 0.47396402806043625, + "p95_ms": 0.5674065789207818 + }, + "forward_peak_mib": 445.1328125, + "grad_finite": true, + "local_vocab": 18992, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.363677481837659e-08, + "lse_vs_fp64_max_abs": 1.3207212923305178e-06, + "path": "native", + "rel_l2_vs_reference": 1.6072921840575048e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 2048, + "topology": "tp8", + "tp": 8, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 3.279380965977907, + "median_ms": 1.0833695996552706, + "min_ms": 0.8646342903375626, + "p95_ms": 2.7472038753330694 + }, + "train_peak_mib": 593.5498046875 + }, + { + "cp": 1, + "forward": { + "max_ms": 1.9366671331226826, + "median_ms": 1.7633400857448578, + "min_ms": 1.7227185890078545, + "p95_ms": 1.9199213478714228 + }, + "forward_peak_mib": 204.32373046875, + "grad_finite": true, + "local_vocab": 18992, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.314942529180228e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 2048, + "topology": "tp8", + "tp": 8, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 3.7340661510825157, + "median_ms": 3.3381988760083914, + "min_ms": 3.2053212635219097, + "p95_ms": 3.7182031897827983 + }, + "train_peak_mib": 964.537109375 + }, + { + "cp": 1, + "forward": { + "max_ms": 1.1442299000918865, + "median_ms": 1.0406605433672667, + "min_ms": 0.988238025456667, + "p95_ms": 1.130919624119997 + }, + "forward_peak_mib": 2.251953125, + "grad_finite": true, + "local_vocab": 18992, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.3108948497165656e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 54, + "path": "ws2-rocm", + "rel_l2_vs_reference": 9.957929192632568e-09, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 2048, + "topology": "tp8", + "tp": 8, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 2.4306410923600197, + "median_ms": 1.6933211591094732, + "min_ms": 1.5776939690113068, + "p95_ms": 2.211051783524453 + }, + "train_peak_mib": 148.4521484375 + }, + { + "cp": 2, + "forward": { + "max_ms": 0.9187641553580761, + "median_ms": 0.4205044824630022, + "min_ms": 0.40137581527233124, + "p95_ms": 0.5561909638345245 + }, + "forward_peak_mib": 111.28173828125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.2246337153243303e-06, + "logp_vs_fp64_rel_l2": 3.5790710063228845e-08, + "lse_vs_fp64_max_abs": 1.2246337153243303e-06, + "path": "native", + "rel_l2_vs_reference": 1.8103588949760993e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 128, + "topology": "tp2_cp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 0.7929271087050438, + "median_ms": 0.7523719687014818, + "min_ms": 0.7147910073399544, + "p95_ms": 0.7845988497138023 + }, + "train_peak_mib": 148.37939453125 + }, + { + "cp": 2, + "forward": { + "max_ms": 3.39422095566988, + "median_ms": 3.2698858994990587, + "min_ms": 3.200352657586336, + "p95_ms": 3.3097355626523495 + }, + "forward_peak_mib": 41.2568359375, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.192615400213981e-06, + "logp_vs_fp64_rel_l2": 3.580900305363989e-08, + "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 128, + "topology": "tp2_cp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 4.5003630220890045, + "median_ms": 4.432037007063627, + "min_ms": 4.371511749923229, + "p95_ms": 4.489488760009408 + }, + "train_peak_mib": 241.18994140625 + }, + { + "cp": 2, + "forward": { + "max_ms": 1.3963528908789158, + "median_ms": 0.7118615321815014, + "min_ms": 0.6896951235830784, + "p95_ms": 1.39130181632936 + }, + "forward_peak_mib": 0.18798828125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 9.63235819995134e-07, + "logp_vs_fp64_rel_l2": 3.519814568537689e-08, + "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "mismatch_vs_reference": 5, + "path": "ws2-rocm", + "rel_l2_vs_reference": 1.4312143726175079e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 128, + "topology": "tp2_cp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.4072302728891373, + "median_ms": 1.2146991211920977, + "min_ms": 1.0791430249810219, + "p95_ms": 1.3531540054827926 + }, + "train_peak_mib": 37.1005859375 + }, + { + "cp": 2, + "forward": { + "max_ms": 1.6678068786859512, + "median_ms": 0.7275603711605072, + "min_ms": 0.6844867020845413, + "p95_ms": 1.4014774933457377 + }, + "forward_peak_mib": 890.25390625, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.322330169050237e-08, + "lse_vs_fp64_max_abs": 1.3112566623618704e-06, + "path": "native", + "rel_l2_vs_reference": 1.32910346411907e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 1024, + "topology": "tp2_cp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.4916150830686092, + "median_ms": 1.4505994040519, + "min_ms": 1.4055166393518448, + "p95_ms": 1.4872347004711628 + }, + "train_peak_mib": 1187.025390625 + }, + { + "cp": 2, + "forward": { + "max_ms": 5.079873837530613, + "median_ms": 3.362047951668501, + "min_ms": 3.308674320578575, + "p95_ms": 3.7359460489824423 + }, + "forward_peak_mib": 325.482421875, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.318216084432305e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 1024, + "topology": "tp2_cp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 5.5114347487688065, + "median_ms": 5.449994234368205, + "min_ms": 5.38027985021472, + "p95_ms": 5.4998258128762245 + }, + "train_peak_mib": 1928.99169921875 + }, + { + "cp": 2, + "forward": { + "max_ms": 0.8522761054337025, + "median_ms": 0.7730736397206783, + "min_ms": 0.7586758583784103, + "p95_ms": 0.830516847781837 + }, + "forward_peak_mib": 1.5009765625, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.33456531944267e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 54, + "path": "ws2-rocm", + "rel_l2_vs_reference": 1.069130383334331e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 1024, + "topology": "tp2_cp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.788655761629343, + "median_ms": 1.2361111585050821, + "min_ms": 1.2153349816799164, + "p95_ms": 1.6391555080190297 + }, + "train_peak_mib": 296.7890625 + }, + { + "cp": 2, + "forward": { + "max_ms": 0.9403661824762821, + "median_ms": 0.4830381367355585, + "min_ms": 0.45273173600435257, + "p95_ms": 0.5635019624605778 + }, + "forward_peak_mib": 55.64111328125, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 1.2246337153243303e-06, + "logp_vs_fp64_rel_l2": 3.573556089056118e-08, + "lse_vs_fp64_max_abs": 1.2246337153243303e-06, + "path": "native", + "rel_l2_vs_reference": 1.693435683024808e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 128, + "topology": "tp4_cp2", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.6028108075261116, + "median_ms": 0.9448430500924587, + "min_ms": 0.8031027391552925, + "p95_ms": 1.380775030702352 + }, + "train_peak_mib": 74.19189453125 + }, + { + "cp": 2, + "forward": { + "max_ms": 2.4561178870499134, + "median_ms": 2.218499779701233, + "min_ms": 2.147639635950327, + "p95_ms": 2.4312858702614903 + }, + "forward_peak_mib": 22.36865234375, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 1.192615400213981e-06, + "logp_vs_fp64_rel_l2": 3.580900305363989e-08, + "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 128, + "topology": "tp4_cp2", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 3.5232328809797764, + "median_ms": 3.4358128905296326, + "min_ms": 3.351978026330471, + "p95_ms": 3.516418603248894 + }, + "train_peak_mib": 120.59912109375 + }, + { + "cp": 2, + "forward": { + "max_ms": 1.11051881685853, + "median_ms": 0.7829479873180389, + "min_ms": 0.7545412518084049, + "p95_ms": 0.8756248280405999 + }, + "forward_peak_mib": 0.15673828125, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 9.63235819995134e-07, + "logp_vs_fp64_rel_l2": 3.519814568537689e-08, + "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "mismatch_vs_reference": 5, + "path": "ws2-rocm", + "rel_l2_vs_reference": 1.4312143726175079e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 128, + "topology": "tp4_cp2", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.538364216685295, + "median_ms": 1.3768249191343784, + "min_ms": 1.2114690616726875, + "p95_ms": 1.510734111070633 + }, + "train_peak_mib": 18.5537109375 + }, + { + "cp": 2, + "forward": { + "max_ms": 0.5847080610692501, + "median_ms": 0.4740641452372074, + "min_ms": 0.4566791467368603, + "p95_ms": 0.5416566971689464 + }, + "forward_peak_mib": 445.12890625, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 1.7274325472271812e-06, + "logp_vs_fp64_rel_l2": 3.337932457263568e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "path": "native", + "rel_l2_vs_reference": 1.386501020113035e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 1024, + "topology": "tp4_cp2", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.1372589506208897, + "median_ms": 0.9921537712216377, + "min_ms": 0.900637824088335, + "p95_ms": 1.1346454732120037 + }, + "train_peak_mib": 593.525390625 + }, + { + "cp": 2, + "forward": { + "max_ms": 2.624067012220621, + "median_ms": 2.2469221148639917, + "min_ms": 2.2091111168265343, + "p95_ms": 2.5062156841158867 + }, + "forward_peak_mib": 176.65673828125, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.318216084432305e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 1024, + "topology": "tp4_cp2", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 4.030606243759394, + "median_ms": 3.79836093634367, + "min_ms": 3.7597129121422768, + "p95_ms": 3.9899602998048067 + }, + "train_peak_mib": 964.51611328125 + }, + { + "cp": 2, + "forward": { + "max_ms": 0.9333761408925056, + "median_ms": 0.8380450308322906, + "min_ms": 0.8166632615029812, + "p95_ms": 0.9284091182053089 + }, + "forward_peak_mib": 1.2509765625, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.33456531944267e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 54, + "path": "ws2-rocm", + "rel_l2_vs_reference": 1.069130383334331e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 1024, + "topology": "tp4_cp2", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.585574820637703, + "median_ms": 1.3963484670966864, + "min_ms": 1.2736623175442219, + "p95_ms": 1.5687244711443782 + }, + "train_peak_mib": 148.4140625 + }, + { + "cp": 4, + "forward": { + "max_ms": 0.9446828626096249, + "median_ms": 0.46018860302865505, + "min_ms": 0.4228181205689907, + "p95_ms": 0.5898057250306012 + }, + "forward_peak_mib": 55.64111328125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.2246337153243303e-06, + "logp_vs_fp64_rel_l2": 3.730282272543457e-08, + "lse_vs_fp64_max_abs": 1.2246337153243303e-06, + "path": "native", + "rel_l2_vs_reference": 2.3950332323559945e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 64, + "topology": "tp2_cp4", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 0.84735918790102, + "median_ms": 0.8005395065993071, + "min_ms": 0.7707341574132442, + "p95_ms": 0.8399948477745056 + }, + "train_peak_mib": 74.19140625 + }, + { + "cp": 4, + "forward": { + "max_ms": 3.747745882719755, + "median_ms": 3.4810248762369156, + "min_ms": 3.437324892729521, + "p95_ms": 3.573427628725767 + }, + "forward_peak_mib": 20.970703125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.192615400213981e-06, + "logp_vs_fp64_rel_l2": 3.646037610209301e-08, + "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 64, + "topology": "tp2_cp4", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 4.93944576010108, + "median_ms": 4.78887720964849, + "min_ms": 4.680632147938013, + "p95_ms": 4.929828573949635 + }, + "train_peak_mib": 120.63427734375 + }, + { + "cp": 4, + "forward": { + "max_ms": 1.1552250944077969, + "median_ms": 0.7531433366239071, + "min_ms": 0.723003875464201, + "p95_ms": 1.0587618686258795 + }, + "forward_peak_mib": 0.09423828125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 9.63235819995134e-07, + "logp_vs_fp64_rel_l2": 3.531619957105821e-08, + "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "mismatch_vs_reference": 5, + "path": "ws2-rocm", + "rel_l2_vs_reference": 1.8104749470140388e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 64, + "topology": "tp2_cp4", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.505616120994091, + "median_ms": 1.3656336814165115, + "min_ms": 1.260161865502596, + "p95_ms": 1.4661509776487947 + }, + "train_peak_mib": 18.552734375 + }, + { + "cp": 4, + "forward": { + "max_ms": 0.5427957512438297, + "median_ms": 0.4715598188340664, + "min_ms": 0.4449011757969856, + "p95_ms": 0.5398092558607459 + }, + "forward_peak_mib": 445.126953125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.339807416706557e-08, + "lse_vs_fp64_max_abs": 1.3112566623618704e-06, + "path": "native", + "rel_l2_vs_reference": 1.368420826566803e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 512, + "topology": "tp2_cp4", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.897196751087904, + "median_ms": 1.007936429232359, + "min_ms": 0.9225797839462757, + "p95_ms": 1.5796012012287965 + }, + "train_peak_mib": 593.51318359375 + }, + { + "cp": 4, + "forward": { + "max_ms": 4.713290371000767, + "median_ms": 3.5901865921914577, + "min_ms": 3.531333990395069, + "p95_ms": 3.9379455614835033 + }, + "forward_peak_mib": 163.0673828125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.323407831042024e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 512, + "topology": "tp2_cp4", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 5.251959897577763, + "median_ms": 5.043254466727376, + "min_ms": 4.988599102944136, + "p95_ms": 5.244744545780122 + }, + "train_peak_mib": 964.53271484375 + }, + { + "cp": 4, + "forward": { + "max_ms": 0.8831419982016087, + "median_ms": 0.7803190965205431, + "min_ms": 0.7526376284658909, + "p95_ms": 0.8549518883228303 + }, + "forward_peak_mib": 0.75048828125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.347644666288935e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 54, + "path": "ws2-rocm", + "rel_l2_vs_reference": 1.161601588696792e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 512, + "topology": "tp2_cp4", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 2.1002087742090225, + "median_ms": 1.384851522743702, + "min_ms": 1.292981207370758, + "p95_ms": 1.9774960353970525 + }, + "train_peak_mib": 148.39501953125 + } + ], + "environment": { + "architecture": "gfx942:sramecc+:xnack-", + "extension_symbols": "hip_deterministic_logp_tile_stats, hip_deterministic_logp_backward", + "git_commit": "1c73a427881252871b46936f6d14be36a22e307f", + "gpu": "AMD Instinct MI300X", + "gpu_count": 8, + "hip": "7.14.60850", + "native_collective": "torch.distributed ProcessGroupNCCL (RCCL on ROCm)", + "native_single_gpu": "torch.logsumexp + gather on FP32 logits", + "python": "3.12.3", + "torch": "2.12.0+rocm7.14.0a20260608" + }, + "single_gpu": { + "cases": [ + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.19532999396324158, + "median_ms": 0.11733400076627731, + "min_ms": 0.11172600090503693, + "p95_ms": 0.154381200671196 + }, + "forward_peak_mib": 1.1611328125, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 5.384272654396227e-07, + "relative_l2": 3.365204219985256e-08 + }, + "lse_vs_fp64": { + "max_abs": 5.384272654396227e-07, + "relative_l2": 3.8718812743187373e-08 + }, + "path": "native", + "repeat_bitwise": true, + "tokens": 1, + "train_fwd_bwd": { + "max_ms": 0.46933698654174805, + "median_ms": 0.39827150106430054, + "min_ms": 0.3791230022907257, + "p95_ms": 0.45484338849782946 + }, + "train_peak_mib": 3.1904296875 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.2908720076084137, + "median_ms": 0.19056300073862076, + "min_ms": 0.1811089962720871, + "p95_ms": 0.28451651185750965 + }, + "forward_peak_mib": 1.744140625, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 5.384272654396227e-07, + "relative_l2": 3.365204219985256e-08 + }, + "lse_vs_fp64": { + "max_abs": 5.384272654396227e-07, + "relative_l2": 3.8718812743187373e-08 + }, + "path": "ws1-pytorch", + "repeat_bitwise": true, + "tokens": 1, + "train_fwd_bwd": { + "max_ms": 1.1102089881896973, + "median_ms": 1.0265445113182068, + "min_ms": 0.9868659973144531, + "p95_ms": 1.0943811655044555 + }, + "train_peak_mib": 2.03466796875 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.5845879912376404, + "median_ms": 0.5019449889659882, + "min_ms": 0.4906879961490631, + "p95_ms": 0.5795643866062165 + }, + "forward_peak_mib": 0.00341796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 5.384272654396227e-07, + "relative_l2": 3.365204219985256e-08 + }, + "lse_vs_fp64": { + "max_abs": 5.384272654396227e-07, + "relative_l2": 3.8718812743187373e-08 + }, + "path": "ws1-triton", + "repeat_bitwise": true, + "tokens": 1, + "train_fwd_bwd": { + "max_ms": 0.630295991897583, + "median_ms": 0.5488154888153076, + "min_ms": 0.5224959850311279, + "p95_ms": 0.6256272315979003 + }, + "train_peak_mib": 0.5830078125 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 5.795327186584473, + "median_ms": 5.484604835510254, + "min_ms": 5.387600898742676, + "p95_ms": 5.649418306350708 + }, + "forward_peak_mib": 1.96630859375, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 5.384272654396227e-07, + "relative_l2": 3.365204219985256e-08 + }, + "lse_vs_fp64": { + "max_abs": 5.384272654396227e-07, + "relative_l2": 3.8718812743187373e-08 + }, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 1, + "train_fwd_bwd": { + "max_ms": 6.590147972106934, + "median_ms": 6.508185386657715, + "min_ms": 6.449540138244629, + "p95_ms": 6.570390820503235 + }, + "train_peak_mib": 3.91943359375 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.4892059862613678, + "median_ms": 0.3573305010795593, + "min_ms": 0.339944988489151, + "p95_ms": 0.4387818902730942 + }, + "forward_peak_mib": 0.005859375, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 5.384272654396227e-07, + "relative_l2": 3.365204219985256e-08 + }, + "lse_vs_fp64": { + "max_abs": 5.384272654396227e-07, + "relative_l2": 3.8718812743187373e-08 + }, + "mismatch_vs_reference": 0, + "path": "ws2-rocm", + "rel_l2_vs_reference": 0.0, + "repeat_bitwise": true, + "tokens": 1, + "train_fwd_bwd": { + "max_ms": 0.8597570061683655, + "median_ms": 0.7872090041637421, + "min_ms": 0.7644150257110596, + "p95_ms": 0.8305177897214888 + }, + "train_peak_mib": 0.5859375 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.18343199789524078, + "median_ms": 0.13664299249649048, + "min_ms": 0.13035400211811066, + "p95_ms": 0.14256014302372938 + }, + "forward_peak_mib": 9.27685546875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 8.398243309670761e-07, + "relative_l2": 3.9344983122904206e-08 + }, + "lse_vs_fp64": { + "max_abs": 8.398243291907193e-07, + "relative_l2": 3.693892459584331e-08 + }, + "path": "native", + "repeat_bitwise": true, + "tokens": 8, + "train_fwd_bwd": { + "max_ms": 0.5676829814910889, + "median_ms": 0.5142639875411987, + "min_ms": 0.5031470060348511, + "p95_ms": 0.5522160500288009 + }, + "train_peak_mib": 25.50439453125 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.24724699556827545, + "median_ms": 0.1849144995212555, + "min_ms": 0.18062900006771088, + "p95_ms": 0.21668834313750268 + }, + "forward_peak_mib": 13.91552734375, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 8.398243309670761e-07, + "relative_l2": 3.9344983122904206e-08 + }, + "lse_vs_fp64": { + "max_abs": 8.398243291907193e-07, + "relative_l2": 3.693892459584331e-08 + }, + "path": "ws1-pytorch", + "repeat_bitwise": true, + "tokens": 8, + "train_fwd_bwd": { + "max_ms": 0.9568219780921936, + "median_ms": 0.8782845139503479, + "min_ms": 0.836562991142273, + "p95_ms": 0.941102597117424 + }, + "train_peak_mib": 16.234375 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.3518419861793518, + "median_ms": 0.3047724962234497, + "min_ms": 0.30160799622535706, + "p95_ms": 0.31199708580970764 + }, + "forward_peak_mib": 0.00341796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 8.398243309670761e-07, + "relative_l2": 3.766025511891362e-08 + }, + "lse_vs_fp64": { + "max_abs": 8.398243291907193e-07, + "relative_l2": 3.535840565806921e-08 + }, + "path": "ws1-triton", + "repeat_bitwise": true, + "tokens": 8, + "train_fwd_bwd": { + "max_ms": 0.5473729968070984, + "median_ms": 0.4635085016489029, + "min_ms": 0.45010900497436523, + "p95_ms": 0.5311666876077652 + }, + "train_peak_mib": 4.6396484375 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 6.244194030761719, + "median_ms": 6.082473516464233, + "min_ms": 5.9791998863220215, + "p95_ms": 6.183037853240966 + }, + "forward_peak_mib": 6.22314453125, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 8.398243309670761e-07, + "relative_l2": 3.9344983122904206e-08 + }, + "lse_vs_fp64": { + "max_abs": 8.398243291907193e-07, + "relative_l2": 3.693892459584331e-08 + }, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 8, + "train_fwd_bwd": { + "max_ms": 7.186433792114258, + "median_ms": 7.1299498081207275, + "min_ms": 7.041217803955078, + "p95_ms": 7.173869466781616 + }, + "train_peak_mib": 30.29052734375 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.43849000334739685, + "median_ms": 0.35905349254608154, + "min_ms": 0.3502410054206848, + "p95_ms": 0.39031168818473816 + }, + "forward_peak_mib": 0.0107421875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 8.398243309670761e-07, + "relative_l2": 3.9344983122904206e-08 + }, + "lse_vs_fp64": { + "max_abs": 8.398243291907193e-07, + "relative_l2": 3.693892459584331e-08 + }, + "mismatch_vs_reference": 0, + "path": "ws2-rocm", + "rel_l2_vs_reference": 0.0, + "repeat_bitwise": true, + "tokens": 8, + "train_fwd_bwd": { + "max_ms": 0.933987021446228, + "median_ms": 0.8225814998149872, + "min_ms": 0.8035140037536621, + "p95_ms": 0.9159964531660079 + }, + "train_peak_mib": 4.642578125 + }, + { + "batch_invariant": false, + "dtype": "bf16", + "forward": { + "max_ms": 0.19561000168323517, + "median_ms": 0.16456449776887894, + "min_ms": 0.13676300644874573, + "p95_ms": 0.16843809783458713 + }, + "forward_peak_mib": 37.10302734375, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.8589741301866525e-06, + "relative_l2": 4.183495189450554e-08 + }, + "lse_vs_fp64": { + "max_abs": 9.292139733219074e-07, + "relative_l2": 3.427046566031066e-08 + }, + "path": "native", + "repeat_bitwise": true, + "tokens": 32, + "train_fwd_bwd": { + "max_ms": 0.6035760045051575, + "median_ms": 0.5357359945774078, + "min_ms": 0.5176489949226379, + "p95_ms": 0.5917504400014877 + }, + "train_peak_mib": 102.01025390625 + }, + { + "batch_invariant": false, + "dtype": "bf16", + "forward": { + "max_ms": 0.25301599502563477, + "median_ms": 0.21045250445604324, + "min_ms": 0.1858759969472885, + "p95_ms": 0.24114195480942727 + }, + "forward_peak_mib": 55.6513671875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.8589741301866525e-06, + "relative_l2": 4.183495189450554e-08 + }, + "lse_vs_fp64": { + "max_abs": 9.292139733219074e-07, + "relative_l2": 3.427046566031066e-08 + }, + "path": "ws1-pytorch", + "repeat_bitwise": true, + "tokens": 32, + "train_fwd_bwd": { + "max_ms": 1.0021679401397705, + "median_ms": 0.9008379876613617, + "min_ms": 0.8616790175437927, + "p95_ms": 0.9823387175798415 + }, + "train_peak_mib": 64.92529296875 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.3513619899749756, + "median_ms": 0.3042919933795929, + "min_ms": 0.2994439899921417, + "p95_ms": 0.31357196122407915 + }, + "forward_peak_mib": 0.00341796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.165070511177646e-06, + "relative_l2": 4.043209774336302e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.0317191794229075e-06, + "relative_l2": 3.629703680141769e-08 + }, + "path": "ws1-triton", + "repeat_bitwise": true, + "tokens": 32, + "train_fwd_bwd": { + "max_ms": 0.5292260050773621, + "median_ms": 0.5104179978370667, + "min_ms": 0.47462499141693115, + "p95_ms": 0.5285591006278991 + }, + "train_peak_mib": 18.5498046875 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 6.187510013580322, + "median_ms": 6.039510488510132, + "min_ms": 5.880495071411133, + "p95_ms": 6.155314278602599 + }, + "forward_peak_mib": 20.78515625, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 9.292139733219074e-07, + "relative_l2": 3.3261220288821645e-08 + }, + "lse_vs_fp64": { + "max_abs": 9.292139733219074e-07, + "relative_l2": 3.229389879068821e-08 + }, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 32, + "train_fwd_bwd": { + "max_ms": 7.214517116546631, + "median_ms": 7.1387834548950195, + "min_ms": 7.0237531661987305, + "p95_ms": 7.204205203056335 + }, + "train_peak_mib": 120.70654296875 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.43793100118637085, + "median_ms": 0.35977399349212646, + "min_ms": 0.35096099972724915, + "p95_ms": 0.39481238871812824 + }, + "forward_peak_mib": 0.0341796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 9.292139733219074e-07, + "relative_l2": 3.3261220288821645e-08 + }, + "lse_vs_fp64": { + "max_abs": 9.292139733219074e-07, + "relative_l2": 3.229389879068821e-08 + }, + "mismatch_vs_reference": 0, + "path": "ws2-rocm", + "rel_l2_vs_reference": 0.0, + "repeat_bitwise": true, + "tokens": 32, + "train_fwd_bwd": { + "max_ms": 0.8513439893722534, + "median_ms": 0.79117551445961, + "min_ms": 0.7693020105361938, + "p95_ms": 0.8505870968103408 + }, + "train_peak_mib": 18.552734375 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.3250420093536377, + "median_ms": 0.29465749859809875, + "min_ms": 0.2627910077571869, + "p95_ms": 0.32371008992195127 + }, + "forward_peak_mib": 148.40771484375, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.4593867163625873e-06, + "relative_l2": 3.7283481956277905e-08 + }, + "lse_vs_fp64": { + "max_abs": 9.801408431542313e-07, + "relative_l2": 3.050237461618506e-08 + }, + "path": "native", + "repeat_bitwise": true, + "tokens": 128, + "train_fwd_bwd": { + "max_ms": 0.6414729952812195, + "median_ms": 0.5895554721355438, + "min_ms": 0.5013440251350403, + "p95_ms": 0.6334868520498276 + }, + "train_peak_mib": 408.03369140625 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.3740359842777252, + "median_ms": 0.30306999385356903, + "min_ms": 0.26948100328445435, + "p95_ms": 0.33719690591096885 + }, + "forward_peak_mib": 222.59765625, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.4593867163625873e-06, + "relative_l2": 3.7283481956277905e-08 + }, + "lse_vs_fp64": { + "max_abs": 9.801408431542313e-07, + "relative_l2": 3.050237461618506e-08 + }, + "path": "ws1-pytorch", + "repeat_bitwise": true, + "tokens": 128, + "train_fwd_bwd": { + "max_ms": 1.0661829710006714, + "median_ms": 0.8456965088844299, + "min_ms": 0.8178949952125549, + "p95_ms": 1.0009622424840925 + }, + "train_peak_mib": 259.6923828125 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.3670249879360199, + "median_ms": 0.3225190043449402, + "min_ms": 0.31927400827407837, + "p95_ms": 0.32923495918512347 + }, + "forward_peak_mib": 0.00439453125, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.2187595608281754e-06, + "relative_l2": 3.6275552663920587e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.2187595608281754e-06, + "relative_l2": 3.5171515827779054e-08 + }, + "path": "ws1-triton", + "repeat_bitwise": true, + "tokens": 128, + "train_fwd_bwd": { + "max_ms": 0.5265020132064819, + "median_ms": 0.46553100645542145, + "min_ms": 0.45211100578308105, + "p95_ms": 0.5206794470548629 + }, + "train_peak_mib": 74.19091796875 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 6.214951038360596, + "median_ms": 6.065549373626709, + "min_ms": 5.840994834899902, + "p95_ms": 6.191926646232605 + }, + "forward_peak_mib": 79.03369140625, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.4593867163625873e-06, + "relative_l2": 3.674698683930475e-08 + }, + "lse_vs_fp64": { + "max_abs": 9.801408431542313e-07, + "relative_l2": 2.9901949672921016e-08 + }, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 128, + "train_fwd_bwd": { + "max_ms": 7.209747791290283, + "median_ms": 7.14607310295105, + "min_ms": 7.076991081237793, + "p95_ms": 7.198733782768249 + }, + "train_peak_mib": 482.3720703125 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.45359399914741516, + "median_ms": 0.37147149443626404, + "min_ms": 0.35677099227905273, + "p95_ms": 0.38836511075496677 + }, + "forward_peak_mib": 0.12841796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.4593867163625873e-06, + "relative_l2": 3.58337193479546e-08 + }, + "lse_vs_fp64": { + "max_abs": 9.801408431542313e-07, + "relative_l2": 3.0008002695414004e-08 + }, + "mismatch_vs_reference": 2, + "path": "ws2-rocm", + "rel_l2_vs_reference": 1.2665006080681408e-08, + "repeat_bitwise": true, + "tokens": 128, + "train_fwd_bwd": { + "max_ms": 0.9247739911079407, + "median_ms": 0.821900486946106, + "min_ms": 0.7889320254325867, + "p95_ms": 0.906566995382309 + }, + "train_peak_mib": 74.1943359375 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.8934069871902466, + "median_ms": 0.8458364903926849, + "min_ms": 0.8043149709701538, + "p95_ms": 0.8792500913143159 + }, + "forward_peak_mib": 593.630859375, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.4498686553565676e-06, + "relative_l2": 3.3782278450435274e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.265310253018015e-06, + "relative_l2": 3.100727931388369e-08 + }, + "path": "native", + "repeat_bitwise": true, + "tokens": 512, + "train_fwd_bwd": { + "max_ms": 1.9239779710769653, + "median_ms": 1.8965969681739807, + "min_ms": 1.8372089862823486, + "p95_ms": 1.919651246070862 + }, + "train_peak_mib": 1632.1318359375 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.9607880115509033, + "median_ms": 0.9033215045928955, + "min_ms": 0.8575940132141113, + "p95_ms": 0.9529467195272446 + }, + "forward_peak_mib": 890.38916015625, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.4498686553565676e-06, + "relative_l2": 3.3782278450435274e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.265310253018015e-06, + "relative_l2": 3.100727931388369e-08 + }, + "path": "ws1-pytorch", + "repeat_bitwise": true, + "tokens": 512, + "train_fwd_bwd": { + "max_ms": 2.2761409282684326, + "median_ms": 2.2405279874801636, + "min_ms": 2.1743900775909424, + "p95_ms": 2.2733644127845762 + }, + "train_peak_mib": 1038.76806640625 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.38609299063682556, + "median_ms": 0.33629900217056274, + "min_ms": 0.33293500542640686, + "p95_ms": 0.3548104375600815 + }, + "forward_peak_mib": 0.01123046875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.767553932552346e-06, + "relative_l2": 3.649871630209298e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.2392332031652131e-06, + "relative_l2": 3.379189024884559e-08 + }, + "path": "ws1-triton", + "repeat_bitwise": true, + "tokens": 512, + "train_fwd_bwd": { + "max_ms": 0.599049985408783, + "median_ms": 0.5090160071849823, + "min_ms": 0.47718900442123413, + "p95_ms": 0.5664758443832397 + }, + "train_peak_mib": 296.7607421875 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 6.331644058227539, + "median_ms": 6.183424472808838, + "min_ms": 5.945109844207764, + "p95_ms": 6.294121098518372 + }, + "forward_peak_mib": 312.21923828125, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.4498686553565676e-06, + "relative_l2": 3.3724263980110736e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.265310253018015e-06, + "relative_l2": 3.053555163043888e-08 + }, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 512, + "train_fwd_bwd": { + "max_ms": 8.401599884033203, + "median_ms": 8.282042026519775, + "min_ms": 8.15567398071289, + "p95_ms": 8.375352001190185 + }, + "train_peak_mib": 1929.0439453125 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.485960990190506, + "median_ms": 0.40726499259471893, + "min_ms": 0.39743098616600037, + "p95_ms": 0.4436423107981682 + }, + "forward_peak_mib": 0.5107421875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.39654445874271e-06, + "relative_l2": 3.323884866087576e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.265310253018015e-06, + "relative_l2": 3.056490829487832e-08 + }, + "mismatch_vs_reference": 16, + "path": "ws2-rocm", + "rel_l2_vs_reference": 1.2006765443781159e-08, + "repeat_bitwise": true, + "tokens": 512, + "train_fwd_bwd": { + "max_ms": 0.9189249873161316, + "median_ms": 0.8466775119304657, + "min_ms": 0.8301529884338379, + "p95_ms": 0.8978874832391739 + }, + "train_peak_mib": 296.77001953125 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 2.8302440643310547, + "median_ms": 2.3364105224609375, + "min_ms": 2.2821099758148193, + "p95_ms": 2.690348017215729 + }, + "forward_peak_mib": 2374.015625, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.5566917461740104e-06, + "relative_l2": 3.33818552584321e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.3818783255459266e-06, + "relative_l2": 3.089300583262462e-08 + }, + "path": "native", + "repeat_bitwise": true, + "tokens": 2048, + "train_fwd_bwd": { + "max_ms": 6.796535015106201, + "median_ms": 6.494866847991943, + "min_ms": 6.18959379196167, + "p95_ms": 6.789486598968505 + }, + "train_peak_mib": 6528.5244140625 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 2.8184258937835693, + "median_ms": 2.304742932319641, + "min_ms": 2.2484591007232666, + "p95_ms": 2.7830338239669796 + }, + "forward_peak_mib": 3561.103515625, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.5566917461740104e-06, + "relative_l2": 3.33818552584321e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.3818783255459266e-06, + "relative_l2": 3.089300583262462e-08 + }, + "path": "ws1-pytorch", + "repeat_bitwise": true, + "tokens": 2048, + "train_fwd_bwd": { + "max_ms": 7.514641761779785, + "median_ms": 7.046125411987305, + "min_ms": 6.996592044830322, + "p95_ms": 7.323990345001221 + }, + "train_peak_mib": 4154.619140625 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.6434760093688965, + "median_ms": 0.6005325019359589, + "min_ms": 0.5902770161628723, + "p95_ms": 0.6080828070640565 + }, + "forward_peak_mib": 0.044921875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.5885667394854863e-06, + "relative_l2": 3.6268790641540615e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.5157620829597818e-06, + "relative_l2": 3.420512007493752e-08 + }, + "path": "ws1-triton", + "repeat_bitwise": true, + "tokens": 2048, + "train_fwd_bwd": { + "max_ms": 1.178589940071106, + "median_ms": 1.1376099586486816, + "min_ms": 1.098431944847107, + "p95_ms": 1.1706946969032286 + }, + "train_peak_mib": 1187.0400390625 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 6.943714141845703, + "median_ms": 6.480145215988159, + "min_ms": 6.294229030609131, + "p95_ms": 6.7318147182464605 + }, + "forward_peak_mib": 1244.96435546875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.5566917461740104e-06, + "relative_l2": 3.363042993957078e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.3818783255459266e-06, + "relative_l2": 3.0918720395375866e-08 + }, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 2048, + "train_fwd_bwd": { + "max_ms": 12.976513862609863, + "median_ms": 12.827512741088867, + "min_ms": 12.742406845092773, + "p95_ms": 12.92358765602112 + }, + "train_peak_mib": 7715.73779296875 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.9866660237312317, + "median_ms": 0.8963714838027954, + "min_ms": 0.8853949904441833, + "p95_ms": 0.9365820139646531 + }, + "forward_peak_mib": 2.04296875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.6214356648447392e-06, + "relative_l2": 3.334502491966002e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.3182267863953712e-06, + "relative_l2": 3.0632826213818444e-08 + }, + "mismatch_vs_reference": 65, + "path": "ws2-rocm", + "rel_l2_vs_reference": 1.09810589785962e-08, + "repeat_bitwise": true, + "tokens": 2048, + "train_fwd_bwd": { + "max_ms": 1.932750940322876, + "median_ms": 1.878069519996643, + "min_ms": 1.8332430124282837, + "p95_ms": 1.9317951619625093 + }, + "train_peak_mib": 1187.0771484375 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.17421899735927582, + "median_ms": 0.12049899995326996, + "min_ms": 0.11060500144958496, + "p95_ms": 0.14784510135650636 + }, + "forward_peak_mib": 0.58154296875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3678638666192455e-07, + "relative_l2": 8.545758893188667e-09 + }, + "lse_vs_fp64": { + "max_abs": 8.168879297443254e-07, + "relative_l2": 5.8742217369343824e-08 + }, + "path": "native", + "repeat_bitwise": true, + "tokens": 1, + "train_fwd_bwd": { + "max_ms": 0.6362249851226807, + "median_ms": 0.5021460205316544, + "min_ms": 0.46448999643325806, + "p95_ms": 0.5911399483680724 + }, + "train_peak_mib": 2.900390625 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.24876900017261505, + "median_ms": 0.1910640001296997, + "min_ms": 0.18271200358867645, + "p95_ms": 0.20439544916152957 + }, + "forward_peak_mib": 1.16455078125, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3678638666192455e-07, + "relative_l2": 8.545758893188667e-09 + }, + "lse_vs_fp64": { + "max_abs": 8.168879297443254e-07, + "relative_l2": 5.8742217369343824e-08 + }, + "path": "ws1-pytorch", + "repeat_bitwise": true, + "tokens": 1, + "train_fwd_bwd": { + "max_ms": 1.1947740316390991, + "median_ms": 0.8466980159282684, + "min_ms": 0.8064379692077637, + "p95_ms": 1.0594113111495969 + }, + "train_peak_mib": 2.322265625 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.3544459939002991, + "median_ms": 0.3016285002231598, + "min_ms": 0.29928499460220337, + "p95_ms": 0.310148461163044 + }, + "forward_peak_mib": 0.00341796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3678638666192455e-07, + "relative_l2": 8.545758893188667e-09 + }, + "lse_vs_fp64": { + "max_abs": 8.168879297443254e-07, + "relative_l2": 5.8742217369343824e-08 + }, + "path": "ws1-triton", + "repeat_bitwise": true, + "tokens": 1, + "train_fwd_bwd": { + "max_ms": 0.5243780016899109, + "median_ms": 0.46805500984191895, + "min_ms": 0.446382999420166, + "p95_ms": 0.5172576487064362 + }, + "train_peak_mib": 1.162109375 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 5.4307861328125, + "median_ms": 5.160525560379028, + "min_ms": 5.13234281539917, + "p95_ms": 5.209868025779724 + }, + "forward_peak_mib": 1.38671875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3678638666192455e-07, + "relative_l2": 8.545758893188667e-09 + }, + "lse_vs_fp64": { + "max_abs": 8.168879297443254e-07, + "relative_l2": 5.8742217369343824e-08 + }, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 1, + "train_fwd_bwd": { + "max_ms": 6.593995094299316, + "median_ms": 6.322952032089233, + "min_ms": 6.285376071929932, + "p95_ms": 6.512369465827941 + }, + "train_peak_mib": 3.62939453125 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.6988779902458191, + "median_ms": 0.3520229905843735, + "min_ms": 0.3469559848308563, + "p95_ms": 0.44055113494396225 + }, + "forward_peak_mib": 0.005859375, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3678638666192455e-07, + "relative_l2": 8.545758893188667e-09 + }, + "lse_vs_fp64": { + "max_abs": 8.168879297443254e-07, + "relative_l2": 5.8742217369343824e-08 + }, + "mismatch_vs_reference": 0, + "path": "ws2-rocm", + "rel_l2_vs_reference": 0.0, + "repeat_bitwise": true, + "tokens": 1, + "train_fwd_bwd": { + "max_ms": 0.8288310170173645, + "median_ms": 0.7904340028762817, + "min_ms": 0.7575650215148926, + "p95_ms": 0.8223415464162827 + }, + "train_peak_mib": 1.1650390625 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.1482599973678589, + "median_ms": 0.12208149954676628, + "min_ms": 0.11745499819517136, + "p95_ms": 0.13984964415431023 + }, + "forward_peak_mib": 4.64013671875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 7.819555989385663e-07, + "relative_l2": 3.607215816086566e-08 + }, + "lse_vs_fp64": { + "max_abs": 3.7962424670467954e-07, + "relative_l2": 1.7082649106274754e-08 + }, + "path": "native", + "repeat_bitwise": true, + "tokens": 8, + "train_fwd_bwd": { + "max_ms": 0.5449299812316895, + "median_ms": 0.488986000418663, + "min_ms": 0.4679349958896637, + "p95_ms": 0.5280747920274734 + }, + "train_peak_mib": 23.18603515625 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.2609480023384094, + "median_ms": 0.19470950216054916, + "min_ms": 0.18827900290489197, + "p95_ms": 0.20846810042858127 + }, + "forward_peak_mib": 9.27880859375, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 7.819555989385663e-07, + "relative_l2": 3.607215816086566e-08 + }, + "lse_vs_fp64": { + "max_abs": 3.7962424670467954e-07, + "relative_l2": 1.7082649106274754e-08 + }, + "path": "ws1-pytorch", + "repeat_bitwise": true, + "tokens": 8, + "train_fwd_bwd": { + "max_ms": 0.8992949724197388, + "median_ms": 0.8240644931793213, + "min_ms": 0.7896119952201843, + "p95_ms": 0.8870369821786881 + }, + "train_peak_mib": 18.55224609375 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.3505209982395172, + "median_ms": 0.3025895059108734, + "min_ms": 0.2998049855232239, + "p95_ms": 0.317638635635376 + }, + "forward_peak_mib": 0.00341796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 7.819555989385663e-07, + "relative_l2": 3.418514829490082e-08 + }, + "lse_vs_fp64": { + "max_abs": 6.202060394144837e-07, + "relative_l2": 2.1631422949575374e-08 + }, + "path": "ws1-triton", + "repeat_bitwise": true, + "tokens": 8, + "train_fwd_bwd": { + "max_ms": 0.5011450052261353, + "median_ms": 0.4633080065250397, + "min_ms": 0.4420959949493408, + "p95_ms": 0.49667425900697704 + }, + "train_peak_mib": 9.2763671875 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 5.932651996612549, + "median_ms": 5.814716815948486, + "min_ms": 5.754907131195068, + "p95_ms": 5.861942601203919 + }, + "forward_peak_mib": 1.58642578125, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 7.819555989385663e-07, + "relative_l2": 3.607215816086566e-08 + }, + "lse_vs_fp64": { + "max_abs": 3.7962424670467954e-07, + "relative_l2": 1.7082649106274754e-08 + }, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 8, + "train_fwd_bwd": { + "max_ms": 7.035409927368164, + "median_ms": 6.977062940597534, + "min_ms": 6.87232780456543, + "p95_ms": 7.014931797981262 + }, + "train_peak_mib": 27.97216796875 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.4312010109424591, + "median_ms": 0.3580314964056015, + "min_ms": 0.34767699241638184, + "p95_ms": 0.36920684874057774 + }, + "forward_peak_mib": 0.0107421875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 7.819555989385663e-07, + "relative_l2": 3.607215816086566e-08 + }, + "lse_vs_fp64": { + "max_abs": 3.7962424670467954e-07, + "relative_l2": 1.7082649106274754e-08 + }, + "mismatch_vs_reference": 0, + "path": "ws2-rocm", + "rel_l2_vs_reference": 0.0, + "repeat_bitwise": true, + "tokens": 8, + "train_fwd_bwd": { + "max_ms": 0.8861969709396362, + "median_ms": 0.7877904772758484, + "min_ms": 0.7638139724731445, + "p95_ms": 0.847366926074028 + }, + "train_peak_mib": 9.279296875 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.18960200250148773, + "median_ms": 0.1570734977722168, + "min_ms": 0.1322769969701767, + "p95_ms": 0.18153364658355714 + }, + "forward_peak_mib": 18.55615234375, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3984818885148798e-06, + "relative_l2": 3.32171929410509e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.1600633094133173e-06, + "relative_l2": 3.572127579605161e-08 + }, + "path": "native", + "repeat_bitwise": true, + "tokens": 32, + "train_fwd_bwd": { + "max_ms": 0.5316699743270874, + "median_ms": 0.4779900014400482, + "min_ms": 0.466933012008667, + "p95_ms": 0.5286594897508621 + }, + "train_peak_mib": 92.73681640625 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.2554590106010437, + "median_ms": 0.20001699775457382, + "min_ms": 0.18471400439739227, + "p95_ms": 0.21884885057806971 + }, + "forward_peak_mib": 37.1044921875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3984818885148798e-06, + "relative_l2": 3.32171929410509e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.1600633094133173e-06, + "relative_l2": 3.572127579605161e-08 + }, + "path": "ws1-pytorch", + "repeat_bitwise": true, + "tokens": 32, + "train_fwd_bwd": { + "max_ms": 0.9258949756622314, + "median_ms": 0.852606475353241, + "min_ms": 0.821461021900177, + "p95_ms": 0.9069850802421568 + }, + "train_peak_mib": 74.19873046875 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.3556880056858063, + "median_ms": 0.3040114939212799, + "min_ms": 0.30184799432754517, + "p95_ms": 0.3192687973380089 + }, + "forward_peak_mib": 0.00341796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 8.62301916981778e-07, + "relative_l2": 3.133177230624908e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.0085128359094142e-06, + "relative_l2": 3.172992466391412e-08 + }, + "path": "ws1-triton", + "repeat_bitwise": true, + "tokens": 32, + "train_fwd_bwd": { + "max_ms": 0.5708879828453064, + "median_ms": 0.5329720079898834, + "min_ms": 0.48099398612976074, + "p95_ms": 0.5704015374183655 + }, + "train_peak_mib": 37.0966796875 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 6.055755138397217, + "median_ms": 5.980923891067505, + "min_ms": 5.832943916320801, + "p95_ms": 6.0501996040344235 + }, + "forward_peak_mib": 2.23828125, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3984818885148798e-06, + "relative_l2": 3.32171929410509e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.1600633094133173e-06, + "relative_l2": 3.373302046539746e-08 + }, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 32, + "train_fwd_bwd": { + "max_ms": 7.136520862579346, + "median_ms": 7.03747296333313, + "min_ms": 6.882863998413086, + "p95_ms": 7.119196367263794 + }, + "train_peak_mib": 111.43310546875 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.4266740083694458, + "median_ms": 0.3478565067052841, + "min_ms": 0.34162798523902893, + "p95_ms": 0.39109080284833914 + }, + "forward_peak_mib": 0.0341796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3984818885148798e-06, + "relative_l2": 3.32171929410509e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.1600633094133173e-06, + "relative_l2": 3.373302046539746e-08 + }, + "mismatch_vs_reference": 0, + "path": "ws2-rocm", + "rel_l2_vs_reference": 0.0, + "repeat_bitwise": true, + "tokens": 32, + "train_fwd_bwd": { + "max_ms": 0.842290997505188, + "median_ms": 0.7901535034179688, + "min_ms": 0.770704984664917, + "p95_ms": 0.8351881861686706 + }, + "train_peak_mib": 37.099609375 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.3078170120716095, + "median_ms": 0.27308548986911774, + "min_ms": 0.24384300410747528, + "p95_ms": 0.30477319508790973 + }, + "forward_peak_mib": 74.22021484375, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3963434604136182e-06, + "relative_l2": 3.895801114451707e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.2196775962536321e-06, + "relative_l2": 3.098601801905746e-08 + }, + "path": "native", + "repeat_bitwise": true, + "tokens": 128, + "train_fwd_bwd": { + "max_ms": 0.6033759713172913, + "median_ms": 0.5621750056743622, + "min_ms": 0.5227360129356384, + "p95_ms": 0.6017537444829941 + }, + "train_peak_mib": 370.93994140625 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.37095001339912415, + "median_ms": 0.278654009103775, + "min_ms": 0.24876999855041504, + "p95_ms": 0.3066359490156174 + }, + "forward_peak_mib": 148.41015625, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3963434604136182e-06, + "relative_l2": 3.895801114451707e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.2196775962536321e-06, + "relative_l2": 3.098601801905746e-08 + }, + "path": "ws1-pytorch", + "repeat_bitwise": true, + "tokens": 128, + "train_fwd_bwd": { + "max_ms": 1.0451929569244385, + "median_ms": 0.838605523109436, + "min_ms": 0.7780759930610657, + "p95_ms": 0.9953847378492354 + }, + "train_peak_mib": 296.78515625 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.369828999042511, + "median_ms": 0.3232399970293045, + "min_ms": 0.32083699107170105, + "p95_ms": 0.3327618941664696 + }, + "forward_peak_mib": 0.00439453125, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.2674927774014577e-06, + "relative_l2": 4.306072396111767e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.3520369215314076e-06, + "relative_l2": 3.688548182438791e-08 + }, + "path": "ws1-triton", + "repeat_bitwise": true, + "tokens": 128, + "train_fwd_bwd": { + "max_ms": 0.5017049908638, + "median_ms": 0.4668534994125366, + "min_ms": 0.4521920084953308, + "p95_ms": 0.49999229311943055 + }, + "train_peak_mib": 148.37841796875 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 6.2147908210754395, + "median_ms": 5.9756762981414795, + "min_ms": 5.825572967529297, + "p95_ms": 6.2053526639938354 + }, + "forward_peak_mib": 4.84619140625, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3963434604136182e-06, + "relative_l2": 3.8421598950283275e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.2196775962536321e-06, + "relative_l2": 2.9435109902863417e-08 + }, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 128, + "train_fwd_bwd": { + "max_ms": 7.188317775726318, + "median_ms": 7.1406660079956055, + "min_ms": 7.0864057540893555, + "p95_ms": 7.186334228515625 + }, + "train_peak_mib": 445.2783203125 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.43035900592803955, + "median_ms": 0.3568100035190582, + "min_ms": 0.34992000460624695, + "p95_ms": 0.3875843018293381 + }, + "forward_peak_mib": 0.12841796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3963434604136182e-06, + "relative_l2": 3.8421598950283275e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.2196775962536321e-06, + "relative_l2": 2.9702644822176375e-08 + }, + "mismatch_vs_reference": 1, + "path": "ws2-rocm", + "rel_l2_vs_reference": 6.051339139114186e-09, + "repeat_bitwise": true, + "tokens": 128, + "train_fwd_bwd": { + "max_ms": 0.9075480103492737, + "median_ms": 0.8009094893932343, + "min_ms": 0.7760729789733887, + "p95_ms": 0.8808863967657088 + }, + "train_peak_mib": 148.3818359375 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.8196970224380493, + "median_ms": 0.7682015001773834, + "min_ms": 0.7305650115013123, + "p95_ms": 0.8118575990200043 + }, + "forward_peak_mib": 296.880859375, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3502658759989572e-06, + "relative_l2": 3.7873685403982327e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.1845403324883819e-06, + "relative_l2": 3.04023501035505e-08 + }, + "path": "native", + "repeat_bitwise": true, + "tokens": 512, + "train_fwd_bwd": { + "max_ms": 1.847985029220581, + "median_ms": 1.784870982170105, + "min_ms": 1.7256829738616943, + "p95_ms": 1.8451365232467651 + }, + "train_peak_mib": 1483.7568359375 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.9186049699783325, + "median_ms": 0.8245844841003418, + "min_ms": 0.7842050194740295, + "p95_ms": 0.8577517896890641 + }, + "forward_peak_mib": 593.63916015625, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3502658759989572e-06, + "relative_l2": 3.7873685403982327e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.1845403324883819e-06, + "relative_l2": 3.04023501035505e-08 + }, + "path": "ws1-pytorch", + "repeat_bitwise": true, + "tokens": 512, + "train_fwd_bwd": { + "max_ms": 2.225145101547241, + "median_ms": 2.170043468475342, + "min_ms": 2.116302967071533, + "p95_ms": 2.2047388553619385 + }, + "train_peak_mib": 1187.1376953125 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.4085260033607483, + "median_ms": 0.36115649342536926, + "min_ms": 0.3568499982357025, + "p95_ms": 0.3712318629026413 + }, + "forward_peak_mib": 0.01123046875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.8262701786397884e-06, + "relative_l2": 3.9251382435907614e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.2793943788835804e-06, + "relative_l2": 3.244174275029511e-08 + }, + "path": "ws1-triton", + "repeat_bitwise": true, + "tokens": 512, + "train_fwd_bwd": { + "max_ms": 0.6068210005760193, + "median_ms": 0.5617550015449524, + "min_ms": 0.5477340221405029, + "p95_ms": 0.6044418543577195 + }, + "train_peak_mib": 593.5107421875 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 6.162874221801758, + "median_ms": 6.032660961151123, + "min_ms": 5.892111778259277, + "p95_ms": 6.1107371807098385 + }, + "forward_peak_mib": 15.46923828125, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3502658759989572e-06, + "relative_l2": 3.7865335567684935e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.1845403324883819e-06, + "relative_l2": 3.002057868447724e-08 + }, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 512, + "train_fwd_bwd": { + "max_ms": 8.313629150390625, + "median_ms": 8.26219129562378, + "min_ms": 8.17958927154541, + "p95_ms": 8.310960674285887 + }, + "train_peak_mib": 1780.6689453125 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.5036280155181885, + "median_ms": 0.44291800260543823, + "min_ms": 0.4307999908924103, + "p95_ms": 0.4818976894021035 + }, + "forward_peak_mib": 0.5107421875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3502658759989572e-06, + "relative_l2": 3.786183308895877e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.1845403324883819e-06, + "relative_l2": 3.0413377637587183e-08 + }, + "mismatch_vs_reference": 21, + "path": "ws2-rocm", + "rel_l2_vs_reference": 1.090919925911375e-08, + "repeat_bitwise": true, + "tokens": 512, + "train_fwd_bwd": { + "max_ms": 1.139691948890686, + "median_ms": 1.0942450165748596, + "min_ms": 1.0628589391708374, + "p95_ms": 1.135095661878586 + }, + "train_peak_mib": 593.52001953125 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 2.281989097595215, + "median_ms": 2.052648901939392, + "min_ms": 1.8154170513153076, + "p95_ms": 2.1600565791130064 + }, + "forward_peak_mib": 1187.015625, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.7474648927873204e-06, + "relative_l2": 3.8132201080718383e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.2706277345841954e-06, + "relative_l2": 3.087959234643079e-08 + }, + "path": "native", + "repeat_bitwise": true, + "tokens": 2048, + "train_fwd_bwd": { + "max_ms": 6.306046009063721, + "median_ms": 6.086260557174683, + "min_ms": 5.54748010635376, + "p95_ms": 6.2793127536773685 + }, + "train_peak_mib": 5935.0244140625 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 2.301619052886963, + "median_ms": 2.1072499752044678, + "min_ms": 1.874743938446045, + "p95_ms": 2.2768060088157656 + }, + "forward_peak_mib": 2374.103515625, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.7474648927873204e-06, + "relative_l2": 3.8132201080718383e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.2706277345841954e-06, + "relative_l2": 3.087959234643079e-08 + }, + "path": "ws1-pytorch", + "repeat_bitwise": true, + "tokens": 2048, + "train_fwd_bwd": { + "max_ms": 7.16956901550293, + "median_ms": 6.578951835632324, + "min_ms": 6.511311054229736, + "p95_ms": 7.140474343299866 + }, + "train_peak_mib": 4748.0400390625 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.7294430136680603, + "median_ms": 0.674140989780426, + "min_ms": 0.667352020740509, + "p95_ms": 0.692109915614128 + }, + "forward_peak_mib": 0.044921875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.9093045153795174e-06, + "relative_l2": 4.012909908798742e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.549798071209807e-06, + "relative_l2": 3.362522490350142e-08 + }, + "path": "ws1-triton", + "repeat_bitwise": true, + "tokens": 2048, + "train_fwd_bwd": { + "max_ms": 2.0440359115600586, + "median_ms": 1.8701984882354736, + "min_ms": 1.8029580116271973, + "p95_ms": 1.9687563359737394 + }, + "train_peak_mib": 2374.0400390625 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 7.918561935424805, + "median_ms": 6.000913619995117, + "min_ms": 5.853054046630859, + "p95_ms": 6.208111476898194 + }, + "forward_peak_mib": 57.96435546875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.7474648927873204e-06, + "relative_l2": 3.797537729833619e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.2706277345841954e-06, + "relative_l2": 3.092466091771656e-08 + }, + "path": "ws2-reference", + "repeat_bitwise": true, + "tokens": 2048, + "train_fwd_bwd": { + "max_ms": 12.87468433380127, + "median_ms": 12.581206798553467, + "min_ms": 12.432106018066406, + "p95_ms": 12.77396845817566 + }, + "train_peak_mib": 7122.23779296875 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 1.0888580083847046, + "median_ms": 1.0044524669647217, + "min_ms": 0.9957990050315857, + "p95_ms": 1.0515999376773835 + }, + "forward_peak_mib": 2.04296875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.7474648927873204e-06, + "relative_l2": 3.790029424792315e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.2706277345841954e-06, + "relative_l2": 3.082868776934362e-08 + }, + "mismatch_vs_reference": 37, + "path": "ws2-rocm", + "rel_l2_vs_reference": 8.004053417128302e-09, + "repeat_bitwise": true, + "tokens": 2048, + "train_fwd_bwd": { + "max_ms": 2.634474039077759, + "median_ms": 2.5475040674209595, + "min_ms": 2.4890170097351074, + "p95_ms": 2.6219990849494934 + }, + "train_peak_mib": 2374.0771484375 + } + ], + "paths": [ + "native", + "ws1-pytorch", + "ws1-triton", + "ws2-reference", + "ws2-rocm" + ], + "validate_overhead": [ + { + "tokens": 1, + "validate_false": { + "max_ms": 0.46549201011657715, + "median_ms": 0.36105599999427795, + "min_ms": 0.34206798672676086, + "p95_ms": 0.38701819032430657 + }, + "validate_true": { + "max_ms": 0.5614339709281921, + "median_ms": 0.536176472902298, + "min_ms": 0.5264220237731934, + "p95_ms": 0.5588081836700439 + } + }, + { + "tokens": 8, + "validate_false": { + "max_ms": 0.4257519841194153, + "median_ms": 0.363740012049675, + "min_ms": 0.3520819842815399, + "p95_ms": 0.38267235755920415 + }, + "validate_true": { + "max_ms": 0.5631170272827148, + "median_ms": 0.541905015707016, + "min_ms": 0.5269420146942139, + "p95_ms": 0.5520039200782776 + } + }, + { + "tokens": 32, + "validate_false": { + "max_ms": 0.4319620132446289, + "median_ms": 0.3632794916629791, + "min_ms": 0.34891799092292786, + "p95_ms": 0.378378190100193 + }, + "validate_true": { + "max_ms": 0.5548239946365356, + "median_ms": 0.5365969836711884, + "min_ms": 0.5251399874687195, + "p95_ms": 0.5512092620134353 + } + }, + { + "tokens": 128, + "validate_false": { + "max_ms": 0.4474650025367737, + "median_ms": 0.36904749274253845, + "min_ms": 0.35268399119377136, + "p95_ms": 0.38824863582849506 + }, + "validate_true": { + "max_ms": 0.5527009963989258, + "median_ms": 0.5418049991130829, + "min_ms": 0.5232570171356201, + "p95_ms": 0.5510261595249176 + } + }, + { + "tokens": 512, + "validate_false": { + "max_ms": 0.45796000957489014, + "median_ms": 0.3971495032310486, + "min_ms": 0.3919830024242401, + "p95_ms": 0.42089290469884877 + }, + "validate_true": { + "max_ms": 0.5918390154838562, + "median_ms": 0.5795609951019287, + "min_ms": 0.5639979839324951, + "p95_ms": 0.5884522527456283 + } + }, + { + "tokens": 2048, + "validate_false": { + "max_ms": 0.9521740078926086, + "median_ms": 0.8911034762859344, + "min_ms": 0.8809080123901367, + "p95_ms": 0.9250400990247727 + }, + "validate_true": { + "max_ms": 3.0691590309143066, + "median_ms": 1.0742555260658264, + "min_ms": 1.0614559650421143, + "p95_ms": 1.2030188441276564 + } + } + ] + }, + "tile_stats_component": [ + { + "dtype": "bf16", + "hip_fp32_input": { + "max_ms": 0.06269299983978271, + "median_ms": 0.020930999889969826, + "min_ms": 0.01818699948489666, + "p95_ms": 0.036663001589477086 + }, + "hip_native_dtype_input": { + "max_ms": 0.032607998698949814, + "median_ms": 0.013680499978363514, + "min_ms": 0.011978000402450562, + "p95_ms": 0.01689119935035707 + }, + "hip_peak_mib": 0.0009765625, + "max_bitwise": true, + "pytorch_loop": { + "max_ms": 5.524566173553467, + "median_ms": 4.910352945327759, + "min_ms": 4.4162797927856445, + "p95_ms": 5.317995333671569 + }, + "pytorch_loop_peak_mib": 0.08154296875, + "repeat_bitwise": true, + "sumexp_max_rel": 1.457063660836866e-07, + "sumexp_rel_l2": 5.770375928877574e-08, + "tokens": 1 + }, + { + "dtype": "bf16", + "hip_fp32_input": { + "max_ms": 0.025117000564932823, + "median_ms": 0.019989999942481518, + "min_ms": 0.016665000468492508, + "p95_ms": 0.02466100063174963 + }, + "hip_native_dtype_input": { + "max_ms": 0.02619899995625019, + "median_ms": 0.022133000195026398, + "min_ms": 0.013178999535739422, + "p95_ms": 0.025628050696104767 + }, + "hip_peak_mib": 0.00390625, + "max_bitwise": true, + "pytorch_loop": { + "max_ms": 6.1993279457092285, + "median_ms": 5.494100332260132, + "min_ms": 5.012845993041992, + "p95_ms": 5.790409994125366 + }, + "pytorch_loop_peak_mib": 0.28125, + "repeat_bitwise": true, + "sumexp_max_rel": 1.9626060066002537e-07, + "sumexp_rel_l2": 5.3845673395049095e-08, + "tokens": 8 + }, + { + "dtype": "bf16", + "hip_fp32_input": { + "max_ms": 0.07238700240850449, + "median_ms": 0.020330000668764114, + "min_ms": 0.015623999759554863, + "p95_ms": 0.06005695164203645 + }, + "hip_native_dtype_input": { + "max_ms": 0.0392180010676384, + "median_ms": 0.022373000159859657, + "min_ms": 0.019509000703692436, + "p95_ms": 0.02867680042982102 + }, + "hip_peak_mib": 0.015625, + "max_bitwise": true, + "pytorch_loop": { + "max_ms": 6.249682903289795, + "median_ms": 5.507360935211182, + "min_ms": 4.994658946990967, + "p95_ms": 5.8171706914901735 + }, + "pytorch_loop_peak_mib": 0.93310546875, + "repeat_bitwise": true, + "sumexp_max_rel": 2.358648032441124e-07, + "sumexp_rel_l2": 5.510006778248904e-08, + "tokens": 32 + }, + { + "dtype": "bf16", + "hip_fp32_input": { + "max_ms": 0.06762000173330307, + "median_ms": 0.0408409982919693, + "min_ms": 0.03881800174713135, + "p95_ms": 0.04695559833198787 + }, + "hip_native_dtype_input": { + "max_ms": 0.03449099883437157, + "median_ms": 0.0323684997856617, + "min_ms": 0.03172700107097626, + "p95_ms": 0.034224050119519234 + }, + "hip_peak_mib": 0.0625, + "max_bitwise": true, + "pytorch_loop": { + "max_ms": 6.158267021179199, + "median_ms": 5.456785440444946, + "min_ms": 4.993456840515137, + "p95_ms": 5.7379707336425785 + }, + "pytorch_loop_peak_mib": 3.54052734375, + "repeat_bitwise": true, + "sumexp_max_rel": 3.4588379094202537e-07, + "sumexp_rel_l2": 5.557826589309245e-08, + "tokens": 128 + }, + { + "dtype": "bf16", + "hip_fp32_input": { + "max_ms": 0.16484500467777252, + "median_ms": 0.12496599927544594, + "min_ms": 0.12105999886989594, + "p95_ms": 0.14391364902257922 + }, + "hip_native_dtype_input": { + "max_ms": 0.12794999778270721, + "median_ms": 0.11775499954819679, + "min_ms": 0.10916200280189514, + "p95_ms": 0.1253622055053711 + }, + "hip_peak_mib": 0.25, + "max_bitwise": true, + "pytorch_loop": { + "max_ms": 6.2457170486450195, + "median_ms": 5.5467188358306885, + "min_ms": 5.0723748207092285, + "p95_ms": 5.867206406593323 + }, + "pytorch_loop_peak_mib": 14.16064453125, + "repeat_bitwise": true, + "sumexp_max_rel": 2.514437369427469e-07, + "sumexp_rel_l2": 5.398326236955934e-08, + "tokens": 512 + }, + { + "dtype": "bf16", + "hip_fp32_input": { + "max_ms": 0.5361970067024231, + "median_ms": 0.523857980966568, + "min_ms": 0.46677300333976746, + "p95_ms": 0.5341791957616806 + }, + "hip_native_dtype_input": { + "max_ms": 0.5962859988212585, + "median_ms": 0.46587200462818146, + "min_ms": 0.45467498898506165, + "p95_ms": 0.4989755868911744 + }, + "hip_peak_mib": 1.0, + "max_bitwise": true, + "pytorch_loop": { + "max_ms": 6.220038890838623, + "median_ms": 5.557474136352539, + "min_ms": 5.107827186584473, + "p95_ms": 6.041820120811463 + }, + "pytorch_loop_peak_mib": 56.642578125, + "repeat_bitwise": true, + "sumexp_max_rel": 3.6352432175590366e-07, + "sumexp_rel_l2": 5.424340271221068e-08, + "tokens": 2048 + }, + { + "dtype": "fp32", + "hip_fp32_input": { + "max_ms": 0.03917799890041351, + "median_ms": 0.021391499787569046, + "min_ms": 0.01742500066757202, + "p95_ms": 0.02791384961456061 + }, + "hip_native_dtype_input": { + "max_ms": 0.025918999686837196, + "median_ms": 0.013819999992847443, + "min_ms": 0.012338999658823013, + "p95_ms": 0.022494250815361744 + }, + "hip_peak_mib": 0.0009765625, + "max_bitwise": true, + "pytorch_loop": { + "max_ms": 5.71785306930542, + "median_ms": 4.998143911361694, + "min_ms": 4.4156389236450195, + "p95_ms": 5.3919374465942385 + }, + "pytorch_loop_peak_mib": 0.08154296875, + "repeat_bitwise": true, + "sumexp_max_rel": 1.5778303463775956e-07, + "sumexp_rel_l2": 4.05095450038561e-08, + "tokens": 1 + }, + { + "dtype": "fp32", + "hip_fp32_input": { + "max_ms": 0.05628399923443794, + "median_ms": 0.020610500127077103, + "min_ms": 0.012218999676406384, + "p95_ms": 0.033030851557850856 + }, + "hip_native_dtype_input": { + "max_ms": 0.03176699951291084, + "median_ms": 0.020770499482750893, + "min_ms": 0.013019000180065632, + "p95_ms": 0.02826625006273389 + }, + "hip_peak_mib": 0.00390625, + "max_bitwise": true, + "pytorch_loop": { + "max_ms": 6.11968994140625, + "median_ms": 5.53972864151001, + "min_ms": 5.055549144744873, + "p95_ms": 5.798912191390992 + }, + "pytorch_loop_peak_mib": 0.28125, + "repeat_bitwise": true, + "sumexp_max_rel": 2.2257805198933056e-07, + "sumexp_rel_l2": 5.588434866821146e-08, + "tokens": 8 + }, + { + "dtype": "fp32", + "hip_fp32_input": { + "max_ms": 0.07563299685716629, + "median_ms": 0.02119150012731552, + "min_ms": 0.015943000093102455, + "p95_ms": 0.05310280136764051 + }, + "hip_native_dtype_input": { + "max_ms": 0.02711999975144863, + "median_ms": 0.02153200004249811, + "min_ms": 0.01886799931526184, + "p95_ms": 0.025712099205702544 + }, + "hip_peak_mib": 0.015625, + "max_bitwise": true, + "pytorch_loop": { + "max_ms": 6.234379768371582, + "median_ms": 5.507981538772583, + "min_ms": 5.032916069030762, + "p95_ms": 5.821695160865784 + }, + "pytorch_loop_peak_mib": 0.93310546875, + "repeat_bitwise": true, + "sumexp_max_rel": 2.452264595831366e-07, + "sumexp_rel_l2": 5.466176954859379e-08, + "tokens": 32 + }, + { + "dtype": "fp32", + "hip_fp32_input": { + "max_ms": 0.06401500105857849, + "median_ms": 0.042402999475598335, + "min_ms": 0.04058000072836876, + "p95_ms": 0.049896100163459786 + }, + "hip_native_dtype_input": { + "max_ms": 0.04562800005078316, + "median_ms": 0.04182199947535992, + "min_ms": 0.04033999890089035, + "p95_ms": 0.043649150244891646 + }, + "hip_peak_mib": 0.0625, + "max_bitwise": true, + "pytorch_loop": { + "max_ms": 6.129784107208252, + "median_ms": 5.4718873500823975, + "min_ms": 5.026266098022461, + "p95_ms": 5.79066231250763 + }, + "pytorch_loop_peak_mib": 3.54052734375, + "repeat_bitwise": true, + "sumexp_max_rel": 2.3305682361751678e-07, + "sumexp_rel_l2": 5.403736616575617e-08, + "tokens": 128 + }, + { + "dtype": "fp32", + "hip_fp32_input": { + "max_ms": 0.1773039996623993, + "median_ms": 0.12897150218486786, + "min_ms": 0.1261879950761795, + "p95_ms": 0.14754240661859516 + }, + "hip_native_dtype_input": { + "max_ms": 0.179066002368927, + "median_ms": 0.13676300644874573, + "min_ms": 0.13492000102996826, + "p95_ms": 0.14382479414343835 + }, + "hip_peak_mib": 0.25, + "max_bitwise": true, + "pytorch_loop": { + "max_ms": 6.199489116668701, + "median_ms": 5.575060844421387, + "min_ms": 5.104261875152588, + "p95_ms": 5.851804351806641 + }, + "pytorch_loop_peak_mib": 14.16064453125, + "repeat_bitwise": true, + "sumexp_max_rel": 2.839864805537218e-07, + "sumexp_rel_l2": 5.3810339225841154e-08, + "tokens": 512 + }, + { + "dtype": "fp32", + "hip_fp32_input": { + "max_ms": 0.6945109963417053, + "median_ms": 0.5459109842777252, + "min_ms": 0.47338300943374634, + "p95_ms": 0.5795429408550263 + }, + "hip_native_dtype_input": { + "max_ms": 0.5639979839324951, + "median_ms": 0.5364370048046112, + "min_ms": 0.5297470092773438, + "p95_ms": 0.544399482011795 + }, + "hip_peak_mib": 1.0, + "max_bitwise": true, + "pytorch_loop": { + "max_ms": 6.186348915100098, + "median_ms": 5.5582358837127686, + "min_ms": 5.091322898864746, + "p95_ms": 5.904921054840088 + }, + "pytorch_loop_peak_mib": 56.642578125, + "repeat_bitwise": true, + "sumexp_max_rel": 4.4772318119612464e-07, + "sumexp_rel_l2": 5.43710522076467e-08, + "tokens": 2048 + } + ] +} \ No newline at end of file diff --git a/benchmarks/results/pr328_rocm_mi300x/single_gpu_latency.png b/benchmarks/results/pr328_rocm_mi300x/single_gpu_latency.png new file mode 100644 index 0000000000000000000000000000000000000000..de6471223270db8a20ede58250efa7db0dd518a9 GIT binary patch literal 72942 zcmeFZXH?eL_BDzAkn8K}C=*MVgABbOGtj5-?H)=~bmmlP=vxk=~0m z73sZqxO1EHf8X_V(3@7}+E&r+C!!{mSc1iP8V9gaUPHmT!PcHg_KZb?CLERFnsi}q1pI*Kh6 z6jFbkQ??Bs>$Y)Jn%^j&wrMQ7`|wC*&oNrcU-svy{`u}9CHuSgR+_mRbN0s{XwSc1 zWKN(i+PZcBcB*pK3BAqfNuCDAv={^S734`+37|t2Qme#+aw%fl{vXX<2+`Tp5(uOzbQ7q0XDw z+*p*D6&mU(D%kt+thGbA54~zlD4$-s+Tq`RdzEd}7MM`P_-!q=ritzB*~{kU=7w#l z5f{CV#?m`4y=)hbkb7hAFgv%Rudi=7Khk197-7R}@#Au~NvF#D_wU7p>_l8P9DhDQ zM@H-|U0o_Fs@O8GV@WoXJ@(V3)Y&&bx+%74g+DoAU(xksFv_L))94+2eP(`sl_MUK zcm-kGsT)FB8F$afhSx9lx}>RU+euz<+pB=j>+V$Ips2aYV#{Lsr~C;g{r<{;qe2!j zp?pShLqm5ZFI`Hqn;lpg)!BR<<+732K3_E}ct$FW-=tAVPAt~@xCry_zn{B#^CnqS z_M$mOv5lp^Pze|3r=zT1e#L=!`@S--S?SJu-;VO=dzN|9e%wt*5wKNv#NCyPPuFl) zS+i%N^ke*W~je!|%42_SWT@5&d*E&$+q#x%LZV z9lC+ET(|z-w29e!(dUFoj#Ed5-a}LGn>20>FV1ZX8Wa;0Y!`G~dZRj;W7gXyOa=^3A>GbcvzYpfQBbz>6RawdHw(pqAZLO=Jyj=XIU24W!)&Z1E8Z4!TAAaj@;N4u+CJXnI9sE?d?(5&r+r56+gi`ef_2kX0opsY z;Y?YwgW{9pa-lsZVP;CYL=o^(Ap)tzB0eB zpj-RZyu`}TY0LF3+jjaZvFyn;Xo|ZPC1Dnjwdr+2EWcvKvExo-Qa(kq=yP)+JnV zZn=(paqJ#UN^!v^!Q6UR{`?|;g1vd2a=VV#?m#w;f%!Yq>eoJ{Q#hVhqu6>%KtNSh z$0?ik>T!ho`^URR-C0uQQ?6&2`3GtSOSi!jNdv znPdOWmAcT5(&-e=3BBXg!zx|=u&daL7I!KZUZYml{{H@94~}B_d-vYN$wg@!M<5Ti zyt(<&osF7`s((RETRXC96YHlcE9;G=)HHUA9BWU%P~@=4pO!k@ocMg_zN0b7But{B zn&f4X7UsKcdIMfNn`t>tmy$fgZ}KGuKW{kP7(3Dw@6$FaqGRBU6cc!1kB7>X)ow~k z#i~HIY==eD2&`H2{K80!^Z|*rd%k+tbBxtre0DRpBZRH4AQJ}Nq&X_#?C5K?G9wri8JYTU+b+XWcd7yB{j}VFVKuG= zFkkcc^LsVknQQd^(T*=AE-p_Ox~WBvON-_Wz4BPAKT1JStK*r-B2E%R1Q+6Sd>{F% z$H(BK6!J{lnJu;_jP=-$vhi@IBI}i511Q;yayz}~UDg`jt}k?Yd3x@Z$jMl{_UV^% zcnGDuOcodH4 ze)xO0e#b<2fdi|<+>ioEG0$ewzOIbt_EM(QsfI_J9^<_$f4hTGI*{hvxpVrqv#;rC>Q{T#}I7*CJ)!w`lv>pq-Nt4}n$;m5;3i>g(Iysmv$y`bB4 ztTQ*Qd3?cbc6!=dWMay`=*N%SnK~+I+T9hkwe008_6z0oE;JvX91tk;p*Ksuo?R)b zFQVf*#HOROdn8G`Z_OERo|%a+K?6N_iM*1z+@_a|9aGt#%-;U>WR%ei) zb|2g2usFe%!LGyfIek0TAzDF-7vFEnMEH>8{qcs3)=|&IZmu}?qq>GRo{8Vsuyx1F zA2t?8aWuK#nI3YnJiwd)eWhxmWf8H z(9i32+3eQM8%-^!Wu5Ka+-TFK#R}aYZY*V8#M&RpKY)DBSTnRV-Pc{&W!vX9Ff$`N zJ0jn><@{IU!c5tg5K!$9lvA|GVf-H=Hh07b#Z5y`M+Fy8fD~R_v!w zf4RE4CZ2n|lhkkNPIrqR6$i!Uo0SZ^^Rr*QdZi4|Vl~_-xw%~D;vW*i#3VtLQhG}P znYb#;$It)y^XSjd&lpX8eg9fDE%L+1k5tsu@z&#ChM%5^lvh+#49Qif-zU)Zg8AC( z%YJcJ!&L6(nU$=oAg2b!oY#3g^BWSseobh|ro)x)w+Jnh_BZGJnv>Oe_R(;@)73h7 z@E}l?|8mLc?AMuDS-e;p9Y5vRD*@*h+is-~y?kL(z2E zF~6ALV4C&Ws7==S1?CJSs$mpbh0Yuks$<8J{`uz@zPt5*3iLW=>eoeTjjimE2+GXJ zm|N&H`&s7*(96Ewl6gl?N)fq+fWBkLMg!T~zcpN<=PS|SHTjZNKi-4j%TQyb+`U`D z-afyI`y*$TMgJAC)v?Tx?t)z8>6ABbI3*<|6BVK_0@(%+)JABG6?+k+T?M4gC?y@p z+Q63Ebt^74M~&6Am)e*=tjB3lOUt@L@AdWU*RP_ZTQc;d#ils$~LC&<9?*Yx!C$d~LSZQF0Irp16FH7Ea?>= zX?cB337KvF!yb;I_H?l^gmMk$EyC*Dw{^GtMl%w<(t8Z10TqeMM% zo>nP{?9Khrl-$=y?vw`(sNTBuDz#|g`mAOC*~G-emANK4tHo}cB;9gvcMlJq@1*27ku`G5t>hv_nUY!z16P4WL<2hu?K0H|LheMGYiPI59^db6BG_Yu z4AzL$^tv`eghgDuxPEf3WOJinqIlJmgvYmU-|*~#8#6&AcY$+DO`F;(SBk9)b;`Yu z8;Q3jsdD66N_Y=eJV#?Qy0JQ`ie({ZY-;LWOPY=javPg4%947=W@o4)SvaR`o@ISFfh!aG?jVj6i7HWy}7#UGTy7gsfAiPE}XDePkdYT^UyK=OBxC zUAR!6MfRol8ft36TxLCmr*iY|4=YAvi(OmI8ziY|Q4IJ_>+bRJUf@4J)u!XZheNGA zw7@|7^5x4i6wk^2OclYwB-gzkIkE01F8Cd!r#JiYSPw49v=lBt7b`m^yrbNj>~#>yS-y<%)yQ5i(cGSa#7;lx*_h?+4WflP1RPw zX=6KjUIDjf91HoOo8&e zm>340ZZoOI?bKyL*Nc-?mzG8hUs7w^zGqCyiq}By^*VG$_VD#5vx74z`(6N9Z(F`q zv`^tIg!rtZeLOL01n^arAE`L>W!!pJuqQ=BAh#c_GOI-tT`r0jEfW)y>qOY>G}3Zx zS?|VzfX}?33p>4##V2y+UUqsS#E-CD|B<_w4tVuWdRn@_3uAw>dZuXWw;D_Jb_eI4 zLOWnTK6g!yKmI6NS{Y3%`f^FfaZpB1PQbKiUL|b-Z}C}k(v@19lRiqwQs1|e6XhKx zw0g#anu9$@^uasb|5Woko{3d%q!IJGC8H%sFc)L5ccN3o!@d?(9lm50qn9T1$l$&~ zL4)b3V`IjrSy_|Y2Sw3p+2TZcm~@g&G8l9(pvu*WWYv6I|BtsfKLc$;LqpjsNe|5h zMsDuLXSesF8uYE`IL!x0ADhS@W%udtab7jatp4D-M=NmoyW^9`vkoTXU$QS2+0S>4 zFty7e-f!r~&)hHASQ+&~=fCvB@?+RuIz6Xe;SwE>e7nlZ(8x&By~l;K+O5*^29BQQ z<7-KB3YTyxK1fZidRI$T%PPP(AmG}MS$_1;(I`=+IYyu{285AM^m|Ra^R0a6kGb6( z>n-UWd!kokdadWtj*}-tc@19Ggb7Id_1Fy_=lRe^mLBsSD=-t)0-hmRZ?XkxKO{ZEv7 z{>LyLAXLOAm7MvYe&iXRXp2jpG~Y7E?LTtv6VZMeRaR9swWet3(YRGz(`QU%L5D*6D+`MZ(omsae2v(04J1iqzCTFE5MB1l zw;LrhP$(cX0~0H&4EP1T8^?YScH@0vZ>%)6 z-km$6>x;c@bHh#9y5;-(2L@h)pBVvNf)^7<{Fic!kMx$57`7%Sk~Kn;MY`tl;&D#A z{OpFSZESsI082}rVi}5+7~VFl)Ig(EY>0IB6)-Zh#iGW7Cge$BrEa z?Qd?9s*D6VhpaY=l+lcQljXQ>4>TfaW%KV<2J#eEMpwgu{g_FQQKwX^4J8 zdd@$thMq+WitTK|L(Zc}jRF&0e{6WWCCae@>Z)m)N8jsoBx!BG=D4fJd{pbe zQMLAOs%PpO4As9xL@9x@+j-zbGHAfpIBY0GKqXak%Y#uYY;0FiGK!GVV$gt5=Sk?T z*Mr)>OtPDl_+|KOaHUIwCwM4~IT@}iLZj$L*QnfTl8UfXC zNB>4gu341x%J3Pd*%|>b7xt*N*9xpnEIqaB({;=5ql_4$IQYf1x2ttqRI!b8EB zPJD&uy$h_p0??T4vbpiwl8-*zb%Jwq5(kU?SZQgg!Ro?zsHk0rDy^+!V)A4swlxvRBa}LGARiai>Qc1|s_pcnU%m1K%Cuu0^m?Saht};T4o)@d z7pUfxwJ@_{gUj5>DoN>Pz2YA^D+diDmKVZ1ZUigd7Q z1=cC$RuO|A<_Fv<28A4sul~3P66MITmeFKjD*fdE7S$sf&WjV>q}TlYGb3}()iC~x z*4EbPqpOPM2RFck34%T2(kf@<9G_d)Tjn;&=jsO}f+J>fWY?xTL@ za#pvi-OT4RHRHuoANIHgqF3U)b?a8OxKjJ7gpg2&}u*K92v(8w2c&X9^Bh?M_w{4&8X`NMx?dVeV1b9ik68>f2M z&dPVTMgF+(eNJzBi3xN1J=(u3M$!r*wcZ5J@w!4dkX_5rSr!y=X1#^?xNZKIS1-Wz z9KH5hnTvrIG@1Z#+K==u@CnAB%F2%VdFES<+84Ti3F;L>j5=q^;DC6DF0iX-d`h7w zy27*Dpj~ybF|MhrD1M=x=qXew*>=6SbyH?s+ndX(fkufZj(xwbVOcZdzj?-R(<{*_ z^-?bWx`LDB~t44yC9hm5(uDD}%w1`%ZrIIwlYrVs=!(RP!#s&&K*{mla%=B6KfUq6lginSy$`+y3U|@BKa=U~KoWX&n5?1nKL? znIlc+J1u8lPq~Q1;wdum_T$o|%chYr^o{oXEA| z9ly~#z2l8^5PKO~3HRit&N=;9+GhWO+s`da40{UmgdW&ofxYU)SF3}04gAcZ1BB=- zFw%k{de&yb?7^my6WnPGH1SfWD_p6y=ibMCuC1!~UZ@0y37oke5zv2I37^dC61`J% zu%P)Xq(vPXA_+}{G$S_X=lvR>rrjE4uQIO{ znvaU+c=F`QF?^nqX;2QdQI0cD$twhJ+j=RCQsk+YHpsL*D!MKq`b%eIg5`&-{rvo@ zpI-nrKe6jkto7BlkUDrvhi^PDUz@9liK@ ztKbRyq2Fg`t%1QL10zs_)EDEUoJVrHRC|RGY&GYc0j<0q?>ca#fPRM)T8aVwNT-F4 z^HroCy7_O4O#-nn_UwP~?#6#Q7CCbA`1g`lg;q&~Gp?AkthspqSG6}c&b`#@b(j>( z>;jQ~C0p#{y&|=DO1};OQ&jgOukt}pkuLQ5uvHN6pguc|oIta>ILVQXZ_hORb{Q+R`-97MGIU9H@1 zYYpH2@c4QMkPYs}syFG*-2>O_x--%VAa=ldV|wYr*IXSdXhR|qwe=pr#=)$1qP~^T2aLTjMTnwGzJAG%+D z+-8Uf#&6C0NL_77RJ!Ch*J)NVoRZr!qoxC3r}tx*2n!QaVzUWH60jk9L^(z0sJE*V z1$)>!p*|AK=QrciQtEU>e6IdNscFfbx=4K|Zv$33x6pw=FSMlwf ziP0%>HmrZ~kfDt?q&5d#Dt=W;(OHL(G1Jx*+jjyzbH=fs0D6ANl?@;b3Fh5T{A{R- z5NiM%pfxm8Fp)2f*9%|(@!i&^)_AN|*G8_cK$n$W3jIbn_~t9w$M4wL*$w!=^5#$0 z%xiS)wQocpWZAhR{*rpJW8U76MDtpi8`hPT`<58~I6l$2@LJ>S@zSCL5%Cx8zXAg*_{59H%CHjfS=#3$cjYu%sAEAS?K7}e$c8^;!LQ42gT+mx;035g{m@# z^D8YZP~qj}#diOMh>m&nH3J&A#>U1hyIFlF0J)W~j}P4Mby^$-Pu74yO+ka`ZzNI~ zc9SN5Fw)_6Zf8w-@y3eLI^gt{?p(BQg#Tmrp>=!N+^mc(lZtX$2qAU|Y?uL9wNPOz zL*FyHv1|12e~RBBuXeT-e_a!O^7r3_xjZrGj1J-jfqm;RIQTLYK~^h0A?^4rwP1p~ zBv~#(xMlpUUwF9j{$GDPbO_{S$@x)e4!7UQQU+Z_6boCAsSVA5HNP4ust$pv8DJh# zPBflf3v!EgUd||*+BUM6sE>C+nG->EqHtE0=o2bQ*N*WU*T+abzd!Rs3a7bUl(xU- zj;gx1Wv!FCAN0*Z{>f*B-bbar*QH zD56SW6RQ`ZMC}@lU2bM%X3mp-1-)-8p1}&@N^E?*L-^@0VM00{VLaW-pVXtoo!nV9 z5VIs@=@qZcABlOFKT-)-DKvIVYVR}Ucz7BM|~j`(bP)lz*m;FwRnmyoKR?Ps-*G-!9+ntP4MS$ zkdO>yOw!fdO@bK6`=XsmBuKe)vqs-MP^5hV)FfvLKRscNHWX2lWFa48FfHBQjcoe&|wXmse4+Qof z7rtE`Dc1QGC9Y<<;&?7TNkfXGR8!Fr{@&2|j2%LXVOM4ZC+*I`q$3$u$--RPgvLPo z)tqf@?z3mlZgr~A$PH`n`^foIyWD`XKr199G_YoXV6kA*I5DifC-cVpt(NTXL7v&k zWLt~JI*`4qnL(33unt<}h=@Q!TEK1Z8l28(z+C<1AZ`24x_R#WI%eo_tF3nKrGZk6 zWEQLklt29V2=yd*vx|TI_4n`n@n$m8(w6GtIAWPs<;3Rwb-pMcjg?v)Nr9{oX4QrO z{%bTau8szbW@m1lo~fGLPpTOzA&3Cf%-xG)bY?h8v$JJoW%l788{V7d9s|ziuQG#9 z*laR;QGH+(+PCb}#$Qr(N_tpbs@QVhvf0d)g6g-;UhnGCLuoCS~+G)6Gh9v?@t0qk|utY%YA9a;6N(vm6XuSq+Tm4J}@&wF52=tEvqbQo1cB zDRh76nGjXgi*t`P1NHRuLM-Q6k%8J5whFrZ6jr~8fXF`Z^K3}}l&Q^p7>7D=LVdnnnVzvp(AwOw$ruo&^Uk~Tkt}d<3)&;FBFA?OU z2UhjI@BHDx_q*tMN3Vl9sLFuq(L0a<3Nz3^WJ581=E#<3{`Aw%u!I@FctEJq)VJR$ zEjy5%QhDyw`X_E=Xqx!@`u;9OiDoUD(?c;ZO9xs&thRuFKyLHskpZ`3rt&ajF!bwW zbJ|R39+z;|RQ#$~bfppX-Ne)dNQcm2U6r3oOG6fqo3K~n%|%m3NAoGaz)eM)O%Dqi zki%!sb}bb19&oO=a;eX9adCk;_oHRoKsQ%VJ$cRLp7T}qydQq2s`$-%x(ZL#9XzPn z6Q1^oD7X+Zs`cD%zMWg#!_;CVMMQgg8ym=rM>}OMnG_I0y+_x z0kzQ4w-O@{kZWF6ENjzm{&RwDsahgDV(DkVp%YDTd6PJMMsflK26;Pr9OHz0qIa~h zRR~epC_H-nxMuQqiS@bXcGv#ytE3#{4fAI42v7TPuj{~`J+rnu%Fx5MKp#RAE3LTE zl1KU~I4RPGo8m{I|22YK;B#0o&NgVe=(iehGqyuwurEk}>8_J+S@W^9!WnmMmvt*p zs34MuN0QQ7k&1vB4ijiQU^z|np;3SSfTRoR#b#920iO7`rEu!IUnnV=m6+!t<7+vL zXZyvF52qO*fj63UsMNCQzV@wgW$Rh7vgTzPC5I=FP2UCQJN^sQ96b|fL2a6DR>Hz{ z_s&g%0i>4eloa|mAod?ANb$?b`#QY>vWBz{`ua9`GnFj!U6yqQjWJThC)E+SxjyMK zARJjX;ccE~S;DfXoHs^sgR6Jy+2@y>J2FczFRMGzsdvy-9S;!ujZBq zlEJ(=HcPY`v$iy0Bw25cnutYwwb>?5C(e{ z!__EpE$~ME{t^@%%xu9DbE9o_GLyMr=5r4#0F4Wj$WsYS#?E{HxU^8}l4|CSR0E!!3M&=mm9h;kOU2 zL@R1dR*!}hSy0_FswwNY|Ac5lgow>+RCay)G894kwSyboHobgTbf|4nAo^p$vR)10mL%vu#&X-rix($}VO7tJ_W|WoE-U!CBz0x9xDc@P zaC4Of3)Ab4Aa0A4&}bmzrdIU(h_56tBtjQ04P=v*>eQT4TSr!ut~^kWtv73iRm{hV z@Iw_Q%881SXXGO1)UhYf<7s*Hhd?F#JeP18)PxH)C0YLB{f^j00Wf9IPZbI|x$9n` z9oCd@ga(pj+`)|itkkPKDz}A>>YL*wNukBlJ2sq*($ydUFn^L{^|qAob9kXFMJ%{z z2J#4eCg)#bbJ+)tLN<2Tq_?&IQhz}Z2bit!T}0v=s0jxuN0QL9-TxpG%2u1yvDy7pY=7hL1UDRA zg5!KeTDnPRb43+So05w0or-8>YE~aQrr9TH6<80<4%a;B6k=`-+fu|ahh_xSXd6VC zZ1_PK85t9y3xsm(xvQ6jvABX#9m!;IkCO>eyg9Iu-8s%ayX%O}wMdB+we$~#I$djt zrA!logQ7Ms1)MtaJWEzA44PkGt16uQr<%jvG6bJuHD=$Av=4V%lS2rDnKOpUr+_|z z^f=fF!00fi%r|-V)EwHZOq<)^q4{<%tgKot_sFntHie=1fR zi@u1e-$N^L#BN7CN0f*Ak`sXb#^}<+8z4j^Yjw24#hq3X5cw?H{TyqoUXHjY;1-0zw7H zSKw{fADTebCIMLeZr`8`7xuv_Nko27_n&#wFr&k2R7T;X6v>f|3)mE~RUEMhq^zKO;#nhF%cN6R+o*ldeOD6ZE zNWA6jC!v`PUbyR8@~w=d8o@wm?#z2=0b5lPWV?Y)WFiQX2X1)G2)X3_|7942RxBkCzCvoWcsON}mGbSnmr zA+_wJ4-L`TKw5H+*gz&fVilJ9ul#O?Q!fdUHYsN9d0A~@m2~vmK24mdn#C>{(KH(` z9)ZmmAkb_>r#Zm$VQaCI!)$PYWRtF*@MCe*Y4Vb{XYCs&bRn!ZFIv zQN-G&u*dD@U3Bk~f%Ac<|M}RE3#BOq+Db>#@HE}S^XlsA*Ng14J5<5qjVsnR&>uK1 z97E{mkXz_-2e_0x)9P$&$y4>ZfZe=qe9<3XR1k%d)BUThLLL{*2VKH5Lxt!^wH8ijMXZQs*)6K$Tj>6LAn0I@T;6np{& z%O$haJ6$ZRe`l8>h{7ZMiSgRfv|dl7?gKa&5{WxYFSU;Pgv*+_mi=%nVF0aAG2Z^Q zd*`EWQvs!uS7w6kb4XYQBo$bXGtU`@p(+P0zXy+~pUVxx`Y7j(2hff#QHo6K(zp== z-0bED*`0!og~E!riMD9?B`Z-n1my?Ss8n>A(4(VZj!%s4iNS6J8$4i4#L8|l_Oc;R zJ@Wv^^{j-7!^Om#ZC8@;{7L^!;tttefpT%pA_M8_v{|1gKf#`y3TI4SH6wVeB#h4?xGt4 zz3))C^@6a#u3^}0Jm_KwXR6ovbSr7UXDM6FikIUmgu*ZaqdHcYgo^&^ZNfq__*#gs zCh7A+&cB{z-;r?ID%0OZ@5yYy?uWc6l(UpCZF=nDtD{!QzwEv*V32^PR~osDyH}T@ zmIKxKV{pDL+}N1q>3_nN4U16SP2yK&VE&GO0MRE)dK{t3cN}R>)L(cG-#^RQvlW^H zFD*OttE^aiU#`ey7I&YUS= ztwWzloJg6~UIIO8L)L968h4;i!e@8vn2|AHc}jIKmu_`FTn(CcwGRCu!}(J_kt{BT zmd#hCZPRV#8aPWexTtc|sr`3YrH0^gQQts8o>r^Ps{>Q?C4L<6@GsxV_zyCXq8*xP zpjOL(e7Y;E!Swzs{h?J~0?XpmZZRoH9|m`zUXKn83%#9+ir#(hxjIFt?4GbUw8G4; z_EHaxd*4jtzTLaI;3vNJ${OJ_`xBS0k~x@ z68&mibos~g6QdS$rG7bm*=^c9qt>G3m^U+LSp=n@7chzye>v|BAjrB2`$qv=Zs&NJ zWu4gIOo)2>YQo#e!#jv2n70aPptKDAZ&xLuj=Frli? z>21Gmv9v={?J6!1)IdLNjuUk%chHg5ylZ3H$K`ORECtmeUBg8S&qOWwS;~X+@8Vdp z|0{htsDzX?dk+@BnKUd70cw*LA&a#Jj@7i^8NR{jD*?;{(<^|n=IW79>i&|H3@P^_ zN^XibD*rLkFIO#48ZAE9vaOmMPddLOdi3Ik(K1dTb)Jq=oQ%18AnflmMYA|;d=?fK zly+`=VKpFY&c4neD%w?1S!ux{f(e))3qMaH(t4bgXfs#Gc zj^2=L?M6B><5kmw1j2n>A{X8VO?IXguT*M+)oQ=hxqvyKw~obhP%-K=pdb+YpeHD@ z#80bgei;Qell-tcqxJ(PQm%K2092==)}PAFg|RGmkR3L|zQs)Us$cmx1qB7-f3u$(BD6a(-mmm-Zp33QDFGy* zySKLr5S#U6PYM}7g42Wy^oVaP>C#VTZ?fqWYx;;S1e2*dpztKf-R?dHeu%C%7WO>Tv&&pOso}@CDMl z>Fd-F7jy$L(7~->h@(urLXZsxR7_DPp)ZCq7|abdV7gD{rIkGkT#P7q=62Pbdy$6` z>sLu}t6sMI`0?XB)+4vNleH3_vhK!pq4>0L5b{D%ksm?v40Ew{;!qG`#c6ds_j;zj z*Y#Y}6!M&Co>hPmkFaYdVE*ER%*O?AD0$VLW5!5!M@6iY@j2HWLqZqPh4T)&SZT7g zA`j>CIzQycWRfMj%&x&@y(otXp&&g}eqvPrHWOT<-vrY@GE75P0SlDTtGB=<3E!8b zfq?-TK+~@deu>i&HYyX$`F{M0$erRb9@6W|Mv5jHrRH5Er-oSELqlKhJH}63ox=8W z2{0WM-Kh@##y^?+IL*hB?Rr){c^kB+*U3yM*ueX+OeKPOM*yzC%EgOEoV}8XKK43# zIkT5_>XHrFMyiB}AdPrHU|`${G5aA@HU${MsSX}|36jGIyv;yB1qiCy{$&<`F|Fup$G zw^XvZD&d62;LDqvkBPpD{~JMrFl>tRBJM4;HV`4D$@nVjdq`6|H9n7U{x`Q{b0}!T zAVBcnizXoP5I>i7ffgaz#4+3mp4I?EPJ~f(PfmkM`2_?qUXk~1{YftxuJf2n7+6h5 zi!`tXu-e*!`C?w!)gj=JNj`gE06u&PGEfFg5D8%zBeV5uj$*>_3NnObaZd46)aKeH z(j*X4E#KI`o1PLD!OY7?L$FOu#1H`PWe7nYh=7fOwLyT49?StVg<$`up<_ zP*j3Eh5`!N)01mnh$t*PB;%q@lv2*ZKQuwlt-vwS^76v2ZczKg;RjSOho#2n6X)i+ zCc4nBOa5zBJ`r57twdL3hR+Xm0=wTGOC|^~j)9)O$)8E-{>q#|TG<57iV`VX!{T8u z9}{OD2%y)BvF=#oSfW3oXkEi3PI}XAK@V^5R*X{d-FQclqM4^i9Pq>|MFt}E%cSqX z+?x-I5A2m_<|Y=jrdoeE8PXPUAJ$l44PEu!r{|Zw{r>UrXd;e!qN0KqBQzF~J_(Y; z@GzLKdKA-a5Lj9+>+_+Qi>k}fqR}Zf1Iu~E_3yulchRhH+6~hlX!GwB>%$^y88S13 zXlOx>$buL^2pD9J)@Q$8AQ=<$7iZ6$x%A`!JyFrJ;Rie}?x=?0TJzPh`B+FM1~WfC z=l2I{K-Qeca*hz|EwD}k_^JxS>=6*bq|F(qi%JVK&Ht^3Ci zT3xZ8iwmZ`#oe&qN2}Hmq|W8Cv36@QlamT_0!C>?t)m+aNE2zP&ni*zo_=4vkE8gs zJjsPJ1K0t(fBm*ZszS6|`t8qt=SPR&s8wG;73HuFbMtkH^_y*>fShl|(&xD2FZI+4 zb7wO<2T%l&>EOQ4kIuEUoZpdFF*~55{I>%!NkMO;%5z2)n1ruRrd&qZm!K;UOuphZ5?6vvR8K)bQU=YHC3w`rC`7~EH!;lIDbdOfS6 z-!it4`yK=-Itk~lx0v#zH1Ppq)3zUGOiR7rUgA>X`|KIKFp-hb0A$AyEDIxIw8xK| z7^9Oz&2P*=MlM4P3|$LzudM%8PsV%J;On|(Y)1j%lhBBJTq0?e9Lm7moG>%+(3TNUofp%zi`>VgaYECkAxGO=$w-i~@p5s9 zo@35-WUWE1b+ZcCPJKPkV_pcO7<(tcAhlQ^GI)e^9%!b+`7UnXI%kj9PO6%%FP3at zRqa%o3NCqb%<%+66R#GW{%85d60g&4WHGrttKgIHiE_DDI%6eG>Dm<;Khr! zBB@iXDd3v#qOOnx$9L<~FN6Sxc~SvENLwo6xODGogzz=OhpcM8>c0siE|Cw2OKBLs zY*hrp;iE@ifBkv`S0r4+%QXFS@HBCpKH>p{|k+ z76bSRh$AYEpwe3nO8my=eZ4-b`C1oYl2NYJ(cl3WVQx@u0+s5o2nS{?70r9`x zMIHJ%_K#>j`^O&_$uGe&25b=*A8CJuz#h=R9w~Lp^P-aj5<9*=;zC zYvID*@IPH6C0`V1jEuhU8n<(jVLg;}@bC2~Mx?}E4HrzWPgcPI;0osY_|T)0k~=01 z(^frsa@Rf7BRIQ@!jG>bEV3b05!aYam-!QT0+V5)oXsA?><6qj(fGjrhC1+`^Wt@_ z!0P+W;T47uGsnvz8(tfgxoykXnsPBZgpB zV#;A3?32aT5)2k_r>V-dl=I3nvt1Ndz9P&7%{y+N&BH=6u(~fOgwnj zGjZD!E--ik#vG-PCq7ii9B=rPIICDD8sZn}Xs&h67G4 zWB;AY$HX0NA805EZ@))zHUM7!O4D5G{58lhY2r&?w@EEC6{1ZXN{f5xrvajbArXxc z@osJH3a!+`SQVL@(bNnN@uQ}A;C>R*EhNSLyaD5Y=+vJNG`7FMoVb^}JD1g0&N$o` zbr!tu@-Q05qX zQ)yps>=&3iP)_%bqiF88_1ZeRx^*PYaqeBGC@3CIk(nn1H|aRzWFuaa2@XGUAr0nS zPrggpCf~socTo~!yZ3Ns3k8KgD(RgpC`V9lAq^`KM99Fz%d3nbZkY!fs`f`da(ezL zu=U+?JdtE(>m)o_5229(qcQ@s<*ZT?E(&_!h`Fz3_~A597-45yQ?(p=v>$Hy`r$n; ze!!o~KR_`f6#B?n<=jJ6NdoI1X<_^;9UC?x7^4h=OazLh8ULw zlWIQPKG!n!YyK3V!b=6Hp_+}frKLe@B)u!ySL`XdV(P&p{0@X4wukY~>f=6v?fF7_ z=Xd^}w^3#8%*TCN|NBn>Xz>626$QnXixj*5>+3%s9DMiRKmY3uM9F{s*-rES@asEv zT0*(dM|jyDLIUeDqi-g50e3VvY0rzrXv=?i<4LOMy+=n6<-;u&o?IkBsRa(hVlz z@&TgSL8DXs>#x6FgppfRjwv1UctY+Z1!h>{l$#9pdvj$S+?v*qz)K(c?7J8m} zlcut!CZI34sn~4H#|%2$@}jc%YP#T^m_{AEoGFU-93T_ehds;@xx6L zTf6)kdG=la@*=m%pu-{EJ%HQJd~$6Y1^b`d?`eL;{YJyn(_kJ}bAO)-15DeIzdL~e zyLD#e*w4*-nf5+v$n5Q}w{PFRTv=CmZffTmZs$rs;w1B6xD^16^}iRS$+Oqib-Dig zSP9DLQ5l{ZDE<2D5s%X9Q{RU?$Tz+k-TwOa9r&lNewXsy-Xu=4c5Zw!+UxS~+x&c2 z*J2Vf-_HLzwR5X$F^Pp2g}(o{c>7O`3rba0w)8v?sjR3~QdWC5qUxb6V|)2U*bee` zZ{PjT=0CVn$GO*}-R3%dnpA6U*nXyReSsj?of>=zS|;5+*~P-m9!Yab2Dnq~=YJdnUrboH=!lufh`F72O{)6nf>CS!`bg(%x3Ybs2}5u8 ztY(Gcq6=`_nBScFQ{d-=FnjHBh}lhE>`%s^z<{KPOFRJ~Ve$40zr;PFpuAj{Gu-;r zD9gabUA3RuOysr3nbd|k1uc%HlinW)UIA`YGB!3wT;UbIihi_ne|4=^;osv){pLgb zqxR3rzdkthk#layb6$IwnV|8Ps8^TnB)5I~7Qx)_04w}oXpHyX=OIrJenwF4o0!r6 zM7`q~vCcRJ`t`WcGixzvt;g>PGx^nVQGJ>}UESQs^<8M` zS}{4KRSBY@Mf7GqGPyeIky(EIYMR_}5g2Ub@T$iIOy}AMkmxIMT?fVi?%S<_8nDOM z0?1Gn$-n->jW{ak;~P-gb;|CLrTtU#ot?Yi@BiN2_h8T8y3D_rxjeI$*`d%O=lx=4 z#rCR2N2`uj&U_WKkdQiIBk|Z1Z~OKWHi8U(5|j=VGNk_}_>OWSrkOTPORt~uA)5Q^ z`S%m^Z6*n757rZbg9qA}v>%2U$X~*8A+H?y5{MDY%YKA}@~Miy8uo`2`#E~t>68D% zPkIjiZ)tQ;WU!j(VWdd`ENsFgv-B#typAG)LQqb@;y>kUqwnpbM?vc!td^#3N-09> zf1vSb&vB=H8-KYxmT2hx$}z{c6pH(g<{QjnEB~m^z2-&9L$$rHwX(dkO%xvx0w@8% zxN}W2uzRUTPT1e?+N<#|`=URM+A6Z}YqtJYXk%FFHd?jH!P3P4+|Lip+i9srqUSu= zBSo%W!t`7`#>9*;_3t;sa}Q~SSzNp)HWE5S638=mFE7jjcWMS=_u#0y0Fs&G{_5$A zp7jV>Mk$~IObjcdb6A+{P5b(FEhNm+(vo}{KEvkA06A^oaBA$4e-gp`p2H+s8=SS( z<)G_Qh|35A$*@5w5(aSd_OiT!)2~$9@H*sUlUjjQ;4qJTb-&`@V)t&xSH}p-ty8WS znAC;ut;jqzE&iqFY)y_+wic7bg@0w1cJM?g_)w$hg1G2Cyu7UT^gN3J=o zgnS`~{Uj%LQTY8sQ?jK<)gn4FI~rbUhnpSU_Kt%XzaSRpmO~ASN54`anu94*FM0`W z5HqcaBRJRlGS+~GUSn_~h6?1T%FN!2FI+vbeo>e8P}-0 zypZV(Ef{`Lw%0Tzc=RQ_*VDf-YM&|BZ%dyW<2Ea}C=-1CQK6iLMrh8FKiJu8 z`})x102L2xVhFXBfNMBv8~d29T(iarqf@R_)x}9Cz*>e3S<+v3mU@omZb!DqqqQlRNjH z!r9Nz8EC^_{~@lKC-psb;+@Cblsk1}^ak^jbu(A*c(>4nX|2jeDvW*!82$7M`(B+M z7xkxYq*O3&Y9`Z#6)k;m{bf&%;panQ%xk+#(a&n5$s*Thk@OOzM<^PVWVLUZCi`#+ z8Znva*M=t&FU-#&raYObQ~h$Ir=)NPSsN(BvCMH&*MBK2I>S@N+v>+WX-E0*>c6XN z%&^Fwy&K1EIjdZE;^j)qDV8-KmjGY3I%oE0r&!vUR$U!9FJ52h__~Kj@U%-rMr88T z|Ha;S$79{Te_v^bc1BiiBax9kG8%{?dkfj)A{k|qQkkWYEqi3=lC4zscG;_hGLpS{ zj+5%H@ALgVe?EWxuGj0nFY%e@d7Q^O-^csN_odW~3CAqST)X>d4=ybsvpq<}slMDk zuhC~{5gy%R0J>?k{VZCGB4fGv83dkaIdo1R5L7cfK=k8m* z&FLP8G!ys)Uqo`&sev~1 zLQ18=5Vl}2qF2GPHj|I%-R+SNsI{@Muh+=e$!(uh9Z=!nwp5c8f2iF536y9}RkaOa zDU)r@k)%2@Q{&or)WH z_C{CYy<}C^cMqKil$@!?KBQQsoqRwY?bDL4Oq`cVp?NTUrlgYE=^EO|LE#Z3MUX9q z+)E4Wb)fJD)iYruA+Lu@VZf9;LEr>PbteO^2KXwRvU@1N7a?K4@ycy&h$b)|m9FX+ zz#al3<17S$oJaL2utB8^0Uc7bkhwPn@H=F!dA9uga)i-cEpu*98s}>86R@4%trc|60@y88Q1Ko@g#wmc33h0Z;Q-*w|?t_O*jN&GyuLhi? zJFYLKw2e0|_Y9B8u{YcyW@h-99u$zD=>VIeP{4eGoIGU@v9pk!lyzue#M3h=22k+A zP>nqhG%N40aA9eWzT^_&Z+}2e<8({ze6+X_&rnE-ND)&~PE?XExSUKRrUOT=7&u>W zU^y>_J-x$>DVKOvHiY>u6mq}Mwsw<#yeGB5!JcdIUBvD9N4iAULpdTmxf@Tb_#CGi z>3dG+uS?XyNqgW>c2vjIYz^m$U2@d1758|o@1Dl?M#l!Io;LrU=*d=Dk%bgRCMXJC zxz)n#(fMrTs$!fOz=TMqJ>zz*W=k+ExNb~Aelor;qWEZsW@w}0kMWb3m@7f7lS0gU zT;(2?9ggvuS(py{!kNC=eQ&bxTVUb$3*>CAfgpYS&opOtSc^ZUasH%N^K@Uwpbzt> z?~49PkN3$dnaCw2{1{fOwhvh{Ztk>-Zk>Kz<--EHW zI+fF91^%p?=PCt5yLep+EXx{OBvcHJ&|Y9uXRRyu{z0JVf1Hz5%2qAw-YaM0A3?M0 zxzDZ5hZT<`%3WC2J8)v@*yx8py&b*{{;&);*L>yLJCrt>MtN8AZ)oTQ=-@BnV=pp* zClt@z5VaG-OzD>7vbSo9!0jB-rGN%Co!CKQsqWXJ;_&lz_b^hBcZz*_OdUq-hxh5d z*(1%*T5HYh_T9RI;nBPFjC&VIcrSH1*bjG4?7nwa%ahF_h&JaP;mfxwjq0oiLT|r! zP!IZXSp4B$7W%4_U;EsTLG-fqDGk2Os5(SC)dF*UJ_je9WxVNMr(zto6D|AvPNItux6^VUTi#QlAhMnc6N?Zc=SxMBzO|#aj#!# zxfFeq3mtor`^kMwg;3StO4chxdRT3i%~WOf6d$$<{ ziy+pjQuMFq+4ruF{?kku-4bmhHq|fo-YG3l<_RMeLM7@$i*yR`b!Px$VD`U_~QFaA!HWuN+#iOf6!wivMwViv95C z^Wx1Ak-JM*S@PMgT+5J+Hy&4W#L32we!c97^G|{=37mhzu*pCQQOI$Xgr~-Q+^*9! zB4&0U^aLp89`?qM72a?y$bHK!>%pRORd2YVkDYApb8-5;3v3PZMSDw* zlaLMR6=**3aS|BCkd2m;U2l|sJ31bjU^vcV#om=5xyt)Y?}~lJlb35}cYD#2*Gw&X z&4g4>cXkc?&v5qAaQWuW^9S9@=vh7|+1D^$nD>FhfL-q4%iMs-k|~7-(}FrXsvdjR zsp)ee>vaQCqxM2IqyyhCMyV!^UT7G1bi7e=_n;ra_{rVD0^R1KPIqrROjrmR_KPld zEa%nbQq(XHO=G+K^)*KH{3VqrL5d&jRzGpnLix%QaoP(i5dEJxX`p;XZbSd&+vP>7 zfY8monhnCP3)vh=jtA0e3!F`!C>J<56n4WhsoqmJ&M&`e<{0z)c>}>p}%g9x32DL_ljAcM@!~F2JhRSnXDf^z82w(XE!e$ zQ~f@D_vzG!`5|scIf}f~6!rxuKxSsD;CLQp$hDJ`(u1rr@*cOWLBy&kYc0uF#+?}F z7+u}_NikkF&MLarT^^*Byzbn1S>lk$f)9;AA2ci&^Xcdrh+S2bqv(d=6) zZHnh-+=u)viJd#QvZs(UiR`m9SyfTDF~fV!FrL*!1meI4s(+HgfnH88a~27*Gu5jTC&gxtUV8He6tM zrQn8&ij-lvuGoQWjk%kf5(kKJVT$x$1Y`nq^RL<`rJS0OZKe9s)hP2=H=iP>_M3IZ zwA%(ZtG_Av#DYyL8~ml+3AbUh8J#{g@tn}eW%uSCHr1Fuw)tr74u_h3MXj%o`|~=m zDgF=;C<@q}*oA4hLo$3_*dY4~No1tR=mde!Oj6TR@ti1a`;6k&&6?=jlJIxh__Yi7 z@Sl}bcH^w=wbjq&2;xH{%~e}cUU5!!I# zYD});Kg%K7mXgkw0$DeWLTpDSh#5l2ZhrOtn7v|sNBVI`Nv!oNvpvzxz>qMNi5c3K zz`^Rmk|a>__L z+`u-gmhodjt&iaF0o+?2wYL@lNjSmXEd2SEy{;Y$$1U!>UB7Kw(N#`(6@FOD1Z(B7 zPlD-&gniVCTb8_l;l4mZ!kn+4SjNvVpRIbxTJZq&5U?ndaJ)rQ9vU2cuvQo>@!TwSb zr;Y9Bi8gwF)2p%(k+?5+L9-fv`Fd9aCu?B9!jk{Wfmyoo=F?PhFJ=+JYxuZJ%EE%z zrkj^HDNFHq6o_K_B{QzxgOlppxgT}G3exbY)fM$_vYN5<(_`n7nZelJ+j)a?8h6KNoGeqU32a3H;x#My$$;a zl8?hNS_J1B`Wg>a@jrci5pl9WTHTg_Mon<4TPtZb&9Mk$TE} zSv#}`5*#bI)KB;TnceePKY-JB>;{OHnK)eI3GAqb$VLCJm5e&+d-)No|T`mp(O4D~wo-VS;p0V0?t_ z8E%c#Zn*=QN586|m{lwVXa|WBVdw(+}BL1Gh`^C4muu^iY5Tj9d2iXcOGVeTD0)NO=pOApe^f7YT*5~|tZ-UwH>?-r;#9%3CRseXSk^Q&88rBy*HcV))?yC2{Ub)~L2`^b)@c9o>_bMz+v0qfU0U{G z+~&_;+mwzDJ9M>t{I;AwvYtSDEFp_*6PJA-CG!A|Tg5@ZXAi8wh;R`?SIR72B!59& zkZoFDU5yNz9>3vm^5h(dN#l)H4ZqKEYXkgu-)EX*YW(!bq-$}*d$svxG;jBwIXrY% z!{6W7U>uf}LM}l#10CbpPKHx-o0+Yz8RTDj2kcjLR(g?od6#M`&D53EgKmR+?UpEZ zz+S1Cg6ofuuP?N`o$n7C5D+a}dsJ{j$Rqw-(679(T@62!?(;Z}|A31v$BZ zU{rAW5V-)G=*((3{vTx@K~nc4G?AhYr0kM5sVuIz@ zMb9AdL(lQqj-ugQLVLj=$e(|xtlV2jeuEHz5Ks`gvk!$Vrv`(uqCZ0xuI-RzNR_7p zt4+Z~|8=dN+KqOo;~=J{F1z5Gd?3w>PhqSTM_yPnb5-hlGq^q8w_m}zuFF)~yE<5& zhf_&q{DH$itW8%|{b$}E8vDUV0Ga3H_Xl6GI(I++JQ2Z8rFI%$@;)Cs^A2*LannPU zs1y+G%%W0S2(c0GA}Wdju`~#_+bHdglRSj*#DR!2L8=A7Al#71cL6LZVlqL5N5Cm9 z0WkPxs7nT}3{I%*YBgL1Lfj5iJ)Uqi1W`jmE?F58e_QN(kl5*f46+?yJ){F&mp<)< zeS@%ZR78ZDw>au>0gw}J0d<#(lKzzPfa~ElYhywPXtq_vErC#ds38M1fmvQc7+-8R zHzF%RaA~wX4h4s2L_}`z_1QlA=uef#tW4GcvFhO3Vr4UT$=oDwdZ$AjHWdF4Yzui% zkN0RP$6XYD<>!45b8wKrfR8A$?v}wv{t&7P0Zj>4F^LWRPyvS>=4l`PK?`Xj08VCC)Ng=G7I~gM&aod}u2fa_$^zjH2M|Tnz7+ zit;yrb1DRa0F?MZ_$SyuI4SJ-x*&tjq=FXEj3`kCHRJOkWozO9ma^Wz@#9O;R4Kss z`r?5j2t@mOe(!A`M63bmwlM$%b^snE6Syc$5Of+4%xKR7GD(*%Tv%3TC*0$CoZ)9j z4OGPoDSs9~VgRnX84%QW08Cpjp=ph9Hi)$n6-L{k#uQKm40w_VV9rQ^l%zbsQEqgg z^tTfInb3lh`36ZMHzLo5_pLKe)P^Z_5PE8)F^mc|60RNfK5&MWMb+uL5i_08b$PuD zsXDAVEt_f~LE%Zq$VU^;R3ECj)fjAjcHUPuI{OT}yE^AZ&sVZL$VLgyMQ|A@iBdd~ zXXR9S8*f4&x64%ywg<(s8Fg3_sLyr@dTC4d>EI?UBiJUFFXTMe)zVDB*)G0Qu$lqX zY4pWQm+~Ngk587bFzf=!Ms(OH7YRfph{Bw+=L7J!w?Wk+N?QZI{)6QzFy5k>08Xmc zz)TIh92JJk8|%xB#(+s{13)g9`w);F%U!ur-8fEp{;nL_)`0o}$ZFA^5%rXWL>!!L z^^In8>0!rw2Ed0wx68q7(@Q}GLh2TV7FLSmkRRV1z?(wexWE94o$8vI039i^KDtUJ z7!*2=D|XGhBJ3o?vYuck!>JxExo=SH0VL@D$kl4K3_tv9GqdQ_K~66{rg>kPN-nJ= zR)JtwwktyriO3pk`6w+p*d17_pdY$iLK1dh_P%m39IpyKa*V|JLa_uq1IB&!^4Ntl zhF0!4?T@FPC_o|30@<6xG&DM&3xN-C^X`daS}BAehGZNnM$3mRLn>etALr{t0DDCb z5$uhChWH9a19)JYi-&|RB89zyN9_q-yrl?)4T5>tbuj^i{|ZRe{z15Bxx;XhsQ>r} zt9-N7Pu%&2@-HtqX5HcHvlGAcVE5kLF?LLZ1pIQe@y0!9n^U8BGP!aX6t+=%PbTcb zra|uYB;TvJ*b3$wi}m%nLNeN)Pd>7)BdV%upMJeoRo_R`2eUpz|e7^9W8{G1I)d&mMZLG!EV*4-7;*mUkQY6PCB2l zC%&BFssjC@o#=o7bub!zH7JiAXgk(+CNqEZ%wRmw+@kgdK!1lo*FcEspiKjJJG{^7 zy2046gJ#~X+9LtAR)z5y8&hK>sqzob5HzZum?FKUB=|q%X3Y!sT9bLb^o=+lVtTc% zj!brw=6YlMV_(Y1g(p!}(crl#@%*EvOy>z3EUyI~Jho}~UCM0@c`tFl%8cRaR#JSB zKrX?)x}&=GY0IGym!9Uw3UbMd!&h5XPD4G@8z|gGkTOKVgqAlztx<^yqFIFk-65?ef*nz2SorJK~MvP(y-0m@s{FX z_(IZG$l*iEG3n@aJR~b@nP;_5c_mqpkLZ(k)zsLD?pyIewzuZcn@0iFo8y#^p3=NH zVr10vXdiLQr;6p%(NKf1vweF%Zf)%Q0|uSbEXs~?DbGW+S0?v7yn1MZ`GM*g7pepf zK%AltAD{+mhg5Pr;N7;!n9;@|EN!Pqzqt5OCCVKK0bbRFI=r+P6>$)48Zih%ZI8xl zlTQn8hR#b{SiI8dLEgmO3QN8@JOsfmC-%6-X@ zjwKbuN?QY@Ak}YGu?;+F@WE}*vWgOZoPIsAW-8sH?a58)cHL;Cs&RCm*8Sa@vpYF% zzB^8H!fcPh-I^`4e!-v`$uXvU@m<;#tkn}2b1_h&K^j1BQr8)|Ig z9xvNu8hzQdU?8St@F;Y5MMUD4iS;#IUz-ajVU=Ir-hYxiVq2PIv4b*lYFb;f7}e?{ zUtzNMU31(rY1L&33F$ysO>9yKWdmaPR?Ski?KzbO0&DF9^1gQbgAo*ZBiX%oTr3KrB6&U{5OTd4a19neW>|0-Q;^l96P|#i=f#vs^(4g{GL$Ew6vl8uv z_S>f0s3j&zJGQ4`iy`(6?8}Aqt5@yV7F@!;t=H!}SO8B7TVcw8_5~7jWH_OaLK!i< zuPjWmNFhRORPNjdtet4SfDOD23XA%zE2$Ft?ZG{OhD{k1R>jEUQGnM3o@k+j#6+l; zk?97K=&e3wQ*aGbNdy!Zu+3_x07z{>EPN+f`g8~>Gqn2*HV$=aBP3;OL&Rd$3?W}r zlA>muoM0H-?}6may**Sk-#;U2$JnvX*Mzf6{rbQI+m;}ohk1Q)<07n0qp9j^!<`eZ zJvXkYWkDQUca7X1RBFBjxnAl~>TAJUX?6^oEr}1eK`b}+iS4bD#c}-i7iBjv4ODAD zdfoyIZNOP3h;XUkHz0aIM3>wL)$@P=eoBu@V?blHLtV}s0K);5l>&7Ph6e_p#G_KE zvh{^OYB}>Z;`cMys4PFEb0LC?_E81$Qqoi5@J3{Z5=Nul+QvBW%TC4Q@(arxAb~Q$ZSgH^6L5BE1wVo9TxJ&V}cSf&=Uu>6!VZ#{SIxnobdE?C`F1Mc3 z*R6NBXyV&)J6HbjO6tX3*?8tg=7=`7(4yJGB;_kk}| z{{H?$6Wt;Wlm=DkIG3e8Nyuh+7@dvTm-T~L;j_4hCGt!G6QoC`QMjJGvE0WyDG>Du zcb@m3%dmx_m9MP4v}KSU8K8spw$NGb(qjQ}rYFx0tM&;`^5qZuok|m@E97aKIqhWj-jH~I zKL!R{?dMi8xG*MG}gtSpc zzecelfY?-``fh2uinrbs7R}1jp#O*lBN+xm4A3|z^OG;D6(_T*SbQUMd+$}A#3zAL zb4x{u4Flqo6|uO30hd=%2?k9=&_|EnH2H^xgT2c_?TwNGlO$QgkwR80IbnrH?N$w# z)4_b%H;_HvK}tpP>RVjl7#QV}n)l7WQ%$xy=n|NktEdE^E#=r(BlOR52YozE<^1Q~ z^loeVdaOEZcu|k_DbcZ8C1(xBfKD0R>AFMm!C|=Q;|d|up^f7%#e(WKd{jz6Im-rC z#Y}FsyRgK3Cv2!+<=##&2R+w&lSI8|%>$tW-QUqiS2fiRU|Sb81KzOas>O%1rKFwC zgjD>wPdiMg8cy%J74t3&(h+j(vM}J!Z(DdDAXX_cI|50n{kF%pFc3e| z0P%_Ze?U6}fywx5`$-K{b{ohI!5BociJgK_Z9a#p{{Z0I0#48IZfLDu|_YTnAqY_|k%&Kc|;M2rsCU zLQJ6Guvww@AbC)q0_8AxNme#d2x6cR@fj4K0(~>E#wg$8A0&msI|#3#mq6gm>OXL8 zX1=1nrdM{W^DFPD1cQSoam6YYZ=%SXq&>UwH+%0L{fVf<#1Nzb=lK?S0u&qud;_HK zhYVwAU;r=%MB)Q=9H`>54XDh3jSSTWB3}`RQZk{EP#YLUBWV`mEda=73su^11S)Pu z*y$@*_MN`O!e1o3$$U8te{wMRxj)^k7v0?ovpi--M#l1VkbSXJ=`tcI**5uRkcrDT z{6wqL*Ae&l`(*#+BQFgsFIE47dCFZtO_)%+qE;V5zJU^rJAf5ff$*0>3g$}zdIZo9 zZ(>k)D0qTC$MJO32mxAd#sUOm2{$c50zh1Fe^B8Q*bW$^k$|z}HweqBP1fqKmac;8 zG2Xt7PymIJ?0YaFqPYybh7O8@j#?sbT(IjUZddXCK0j<~Z_(<#-jXPc+=ZZF$ti%M zO9cWqvf&;peh30zpv(v&`GM{R@(&JyWkYShOj|BP@fozv;PBijXAQh`(lybgqs}Kl z#Gr2D!~6G#i{YAb`y;EXTlb1iw6C&kv_V?)T$Q-p&G~mR?PI-J8jz?WB&Iuvqy{t| zsz!o|VUR)y{Yp+qX(GrWRBtB%&18RcJ1I90R2{;;yVlz3>DMOr4M69(tvd}8O0Sn8cUOo5jBP@dVTaDd(@Q&1@Wd4@HGPD*zAYL9opYd!Q@L$M~b^2B(2{ z6L51SAa(3gVEMVXvF>b4l`T2*QxAmZmb+azi57=d^D8}nGH7{DfS0OB8AYNo7s0mq+i&aAZ7Fy z{R6@LHk#SStv0)Be%UK-bO%R%*@=hw>H&2Hp6-|O{={fRn*T~cWE5(M2XfSMC= zyg>fR>1;l*5kr-oFob2X6UtlD65?Dw zBw(l^`Ms_$c9qrnRnZDALkh6x-b6p^EJ$Ne0v@n)sO%0=Tp%(bw%Rgq)+hT*Gy}Pv zmwt$GA3;-Bquer0RC3-boCGWU8CQ4nsh53f(&>$si~G;%fh`(D*?IJFKQIAG7*Km$ zn?M5+7!OtlI5B&j&ja`rs3Ar+5sntJPxDY4V-6*?i1`2ttLPwlLCtRf^cENo1C<77 zS=a8ZU0)_Q2vHpGbkT&(=m*}OnXs%C;(G;Qb$2p0vJ=)16Xh=fJD~%ealyt=hs-95%x6Ms6#FCJ4r* zX;~nS+801_ZfD;$G&D@P1A($;6TRsx>tfzLIX@1kQGe}Ha+~Z|`$T&Ijo~j~D~35M zK0cl^o#xoFIOv6A22N9{>6YK8bPC+}EzR8P2k~4P?h;nGFEhcc62)_3zpVwj8-_nK z0<xc;bRvsHCcZ}5XX>UN7keslRZWKK|hY#ihl zZMNAq6kFFn(cb*}?7GvZgD%;#m3!d zepj-f<4?uN8h-hj{y}K|TccBuF<)=+9jmCA(M;0>BOZN%Cc$#i?MjiKPqgc@Wt4Vw zhE%(z&z&>F*D`u$B&OvZU3FRJ&EzkU_%L8tcdb4AReFKWaUV$UwYvrxhg5CJ*1q5m z_jiwf5~=6@oZ|%ufITsG?wh|H0ZiGqpXdUSi2<*2cCZyEzB0xfeYstVgo<0Lqt~$m z-_Dk-SGcD4M#5tK$Mjau-99IEtg;VLbM=tqdOO?F-b&oN-`7Nsz;uVa%K>Tgu!4SX(L2R9JR$*?%3*gsM&IKYFfotW82`!7#2EFyc|) zI}|$ZyrN6PA9W`oIDs%Qzd!B@J+C)kqixH)?PZ|JGgc$v7-Os`{$P5RTkd#2?eAkT z?ni~ff)n^iCYngi7oD{FGi72o&cy_csRqp1)Vj|?-t6CFF`0%vGvG=~Io|okX87d` z#E79}ain|l+6`qs?`#qrT>DE#m2QHb>~`Q*3@va-H4czlH+u^Hzcq_7nOEt6aS)i8gY(qLY))q`+5fBEyya@CNBh1}gZq`1)VLPo?mWJ1XV z?Oo$x?|M89fQS%`0robLw5%^y*)8H$8cXNa0-C=PZRG7|duR>wuQEfIjX5gyKLC~& zjiytnNMANWfL-+I22OHz7mEAB+8?ovs_Y~c`BN#x1pZY|y1do$OD4dC@an#DWU**R_v!W=+|?Y)!m7 z#1481Kh6%DA{;ch1MWL%<ix~^9jaS}(03Pdex<4;5NPr5UtoW#y6{%2^ zr=gM&ps_3FRHEF!S`hqJ70W1lYN^&>&y;TGQ{o`?5Dwl*6a)zI|y7|MUd zi2gkZGnx&kCmfYDk0%c=%{6=mooP-N@qiUe<;bkyjxT@dyN=%~Xb`C$dL;CAwU`tC3+{e6SU2j}iD38MHsV|`l9mY?Ug82gg z^*W~P!*q!l;|-ayf6v(UM2}4HMPNexUWM1gBItk~zX1U>BF>#<#2y2{YQ@|n zBgSyyKfO}qZn=9yuKBq$XKP*Zr>YFt{LNRIhkOgVeqM^UIM%!}6&uNHc7sa-jHspI z)YtBqYda|K*qY0Y+3h$U@TZbx+KW41v~fOIZ;!c!z>8GCOb10_S(IHMQER3sq>|of zgg4Pl@(!cF3@Fes{K}unjQqZz$oJ3ln7aWOf5KkqC9J)XuXc93DD;0S_IvryDNU2% zi+fIO)%HMxXK+dz*kNC35x zepf*ErW9K1uUHy)q7!zS2#0)_AzAXx_mvBurk2^?FwbFK>Yks-3 zz_T>gy^61g#Zo~^n<41wb~%4Z;@eU4WJ~~Y%|3oZ!@L+i`cl5AJfU* z-ETz2X)9$1LR8Id$wY)8WHu5lCVDvETnv{!$>cvz|N$6U`tC zh!uGe>vFpBc`BQaxY==oK=;#CN0)pKWSmUP28lRIG#)9&n~7hQ zj4V^e2#Im2$~S$x+aB}ensaB8$NhM6iYA{@%<$45Yt5YyAcGmA0fr{C9=WYCaEW%$ z6zN7>P-_deh zY%aXco2tV%mpP z!!7cUkB;2KAgZh1^HRyMXCfh$9WuAr@h`@0w+uQ=l!T>K(mqtO?scEt`Fn2MXEngI z*i+YoC`FBR(J7T_#Ba&-SuJ-|j4rmAzddtcW6;EdNBKP5_<3q6V4^^Phb>h6UMV_C~RvSSG)N% zE{-~6v~waX&$0|{beE=^ne=A)dmkLK<(Kw{Lg^V*H=$nwp*z+!aWbXbi8EITi&Kl2 zz{j;E%Dz&x_ha_HZf4K>NlpQbYQ86GD93<%m<>L<(a(Y2P zlZ=8={Vu5ZgyA2lfO=Z);u+go;B$8{KWP1vk)XiGefFLnnWODT$4s|D+s^H>gG@;4 zp8K%77+rEXwv62C^`b!{FAd1itzFoNx~enE;Xg&G{SQ%!QNZuT{P>*|*=+2OtnARn zFAur(ahtZ5JRnGTK%zt`cQ`91hCp0NOq|kQj>-ECv1bE+L-LTqNNwF6t+FxS+Uu9q z4UX;D&38QhjR#$A7IeDZ-K~K0vNR2!+}OC%I)al5=dE7vrEeS^vZh(=w-K!sDoDyN z7%lxSfFatyxw>Ri!p54#lYf3r>AU9Z^Lr;m40U~Os{a(?Z$IxwAl2>NNt~GX4&`2Y3~Kg{M_YpyrPZm`BUQ#Wz%Vrjv`Vt>z38tB342U78PFPJU3p8 zyvnZhAL2@F@I8mg6Wg{m?qT`2vF4?2o5$7#KN2spNndqx8qKe9T6&T zXD2aM*!hX(G_9orVE{HqX0cr)vgqMtjj;3Q-qwfp6bn5X>*2j$zg3JyHT1PM;-=L! zR|FjSH&jF(O-+&sZ)P|xohs>>780y+uuz{hs_ZA}=*wj)+o{uE$oN7>N!EqP#G^lGfHKli7vz=-kggTm%|OqVns}}+v$a70glIptvEsF>Vj`dW^FnR|3Akl zDE4G=dc}E+Srtuj*kjTNVwkokQK=s2)}1VL3T;j82wyUXg@t*u>8XKin_7OWvN7*z zY3mIh-uf!sC}eF+H?4(V&S=ek99L$Y6b~tRyVPtKDu|)kK?dqGmITXZkb)?(q${%@=h}KW1~^^V>@0w z{&dwXPGzab&#bw_@$^q{K@86Sc5tRTO|hDrsYYwquz7dO>%{cWg)9;BEV=r=saF%Z z?RRsW})t>78j(>?xiyU+5CZ?B8^tvsXWi z^DC9$`DWD?_H-_v;zLUZ@6Y#Y@&OS$pPy)VsWUd|s-g`noxBZ4yQ>=Ag5GRGQkV;y760>z!PBzMpRWd`}Ghwp5iL#`nuN*wlyDJ!&Iu~2VVK0NWub2#jE&eIr^ys| zd;c>#^&Sma(9hc&q4%%WC3{NI#8Gw5LUuz=OWTkByh0LB8F_gPG9y1SNC7-V=@t)> z@ybbjeAy1)^V9SU!;H;S-gMc7t|JHga zXH-cyuU*Y7-Iqcql40~8&+0@(W3;_4qc6GOB`mz`E47|5V<)v>1NB8Y ziTp0kFwJfs5ZBQOGR2R)77;u2(Tu*FEnMYf(Dv}7sthsdpVARXqj}4-WyU*7Q-|lytFL60&Wn19ZJX5?m z=48{XzIWb9AjseUwn_%JHaD$9`97zkMn?Sj!q~}BDmp2~0Bq;sDl)!QI>nDX1|zSl zDU516xt}Ai;7Il^sN5@W-xjlS$8EnNJm*ipA|cjAThl2_CExPJj-f96IPWfwU@Whu zggIz+Gh=&SdAk$2I!bkc%@s!(6)Tk+{1J;MeKl;V`K+L=2QQ*+2~(Sz5u%yw*qACB z>p$-{y8lG3!7GgpUNQXqK>WuQSSHsJ{mrWm4pkaFoG#JIj6S1Z+;tAlPnFoVe1WgD zdcvj#HZPa?Pk+rBIbZn7!JWA~$4fq;A(W%A-o3fNw)*3#k8RV$T>xrovyN+R!rG%~ z!OUknOrZzwzy{RM0in(JA6JNLpV_j??U{Gzgj6&5bwNcmMR(4Lor~M^g}~5;L)aA>5zn&d)_y4{u#C(|c*<{IN=;)m5XMt}yzh(0r+Z zqsSVH)6fo=yUaB*#b}=fL8e7kji}nGyWD(zW%R6ADhg^EMV*kLh3KdG)SN=AQ@mYu zXv=3Ve0#v>Vw9MN-TRWhz<&lC;WwUFhNM@I1><~aw4?j(kcB_?BNh*Vw=L)RXZxWw z&t+%RvWL-TLGrA8FomTMF1Gb?f7(DqS<(BaAvfDHrUw&yTetN>V)(OaSuvko zQ4G0)bhjrO{rZ0|{r?ycHJ4^UPJkxA4gDSRD;J>t zLnZ%kYHUhM%GaZ07ky*ka6JOIOaiOo@5Ref?Z0(?vDao9n)F2h0_09^Wz^lzk0Webq(aX-+y=h0 zFs{b)T+roHX@@e(63}^ABlufPw;5x(HNbgB0lQzvS32$Iy@12vioe@s))MpeJ^}to zw9|!}{Moe0O9CBTLW~(`7WYqe{AjM_usp{Gm#VM32GR=vu_;S;X2D3<+V?wUSM~Mv zH|BLVd4RG;4|N%aa+Ag7?v7HG%PDYPIfrosdW>Y;N-W{m8R!ixAe95$kjfA|_YBCU zDAnvH)q3K#lH?{GSUk9!y~;ExGI+{)R7e2WKvHk8z^NR?tCd5o7mlxNx&BO$^m5;nt1A8^di1An|9y0|B73Z$pdfSi#!{sy zdebgo=FS0xt)&SKU`x@sLCu>|5!$@g-Nhjb=VO#K>v!?n_dASo^YIx212{#uWx;ST zeKpICY43WRPOgy*hY3Yv#=ygIO?YqyM zj<|L40}l0m4>SPmEK95soTvMO>)$8!2g0(tXi?Dwg)@j|CKu?$fz6v0+R%tYiIisc zWRU8OP7`%#(?T$VdIHeKVSVs=5jqpK75a}BWIU!h_tHsr; zR}Tu$H5EfI zjM9~{UO*U|nwd#!YHDWogh(f_ipb{ZzP;zY3NJ3j>9R6Mkoq$0eC6K;2X2DH{(N60 zt{wgWNg1BcbaQc?Pt!@zVk6Z?c^!^>za@=;a(1+_5s)Ue(1DBDIKo;24KMUhXG_V< z3_Xg(>&UxbQ_eMZ3yP`0VHhUVS-RN2+P_HhAJ>ETDLfCwZE$j{T>868zhP zn1`<;qJ-|~;3bA}LZ!yPfAutm?p?w-c4#S9_(1N2jEszUNcq0l|FDVk8L%?=0k@ZX z^Mm)=r$cT`g=lhg@BfmH6fL$8AGUv_p#Ccx7%ormrM|1{`oohZ9{94p26`ocbke6h zO^UqGH1CI67Lsc1uZaB9UMscAjalW*l;Zit=i6#*8V1RvGewD9wk-0NVMb>9Qc9-s zs<3(e*_WKqgLv)7L(#f3NfCk2D&@;gnkBJ=f?s1`*;=QqqIV~v|Ml+IK!7$ztTq@c zsqScP0IFZe3Uot$p$m8dFvlmDk(VNI@|V0_a~}eXcr~~;v%n0*@U7=KePTQEy!Lqh zq3Pc4CjA0H6&FCqha7Sq+$n%YrXx8XYaM@7Mk7k}II-?m3KsmGcmV!0>ZL zZ#J|GNcQ&E`cSpqp&jKomI5uD1W!lZz1f@qf5y;7{ChvBK=bA1<~9IGk9Y_YUEf=O zEbQ?9xkP9{Ci)U{d!aiU5NYXd42f>g zO*sv$&cu?IFaZ^?C0e6~&}2=scM9sSG8G=cdF5C1erjAW6=0wF(6Yz@7E7(WpB`xT zB`Gl{q^D!Eu5OJg*{?xBlTb4}6JrUUGp0%P=dDviF_Q3UW`JK}=+ABI8XfIz>yLit zYy1D*WkSw}g&E+umNyp5q5)YY3N^M@&CSg-dxE7Cw18<}`4w8sWKut;PeE~q|5dgE zdNgu5z<5fG9lq_5I|-40&2Wwfin?^0$hIv^iV*=NRLt5m2#iL6Cg$el9RV4uhY|(_ z9UU%)Eqi+Z*Q^`0eNv2GQGzHF=|^X{d@XQcd9xwVCE3eAW#>CELO+LRgO*`?#Nn^* z7yr;RSS&UadI&+Eoy>pNJsc#KhX+vyGw%v4V<8I*3;*S>=~gc08k(B3aWw16Ulq`m zk2L&JO2-E80+=$|^N>exeM(x|RphP1ND=4f$Nv&63YEf5fAT130UAah zD&pe!Hi)D5+h1CqIAKGDWlGaeQR#a6`B5VOmvbu>O-X+#HO}WN8hHb`?VuO|@;lb5 zixAi4y1x_}7-negqh^c78uM$SRbic?f1?jwWF-8{Ej5?5&885zK(7$MK z^=f|3o1Hjm(=ZfgIz!EU1QZ7kFW2g91VXDHK`Bm$nJ-#!-DTv4K_kX5o-}K^Q4S-a zP`k&A%r;>LLe`V{|B^i}u3Kw^dquB+m*uur>1F`!uY@hRT-TPIXP!kvPxz~)OCQ)r zkV6RMtXjj;=s`aH79g&jm1vAWa4GFO^-fH(va*I_S6?1-VUn83%XCR+&O@V3=ksel zFqSuWcZUH+J+ahvDE<_1!_`8|556Az5N-Qv;C;0L-gDL5%W#hF{XZNmsC0TdGBVHiLqC8cQyY<=;o+;Wy_@82Ctjfyr*3s zJBy*u~FFb@*F4qQcV6B2P+2e7s7%Q(@l>03f?t=~CWZ*OcuTa4Qtt3JkF} zusG0V(1)Yte~1U?#UtjxO{2)tkF;x7-)Vmq4*u&Ep~;{3adENlwXUSEMvEIFCi-ds^m0MFjycL5aO+jQzZOn_rHXV^Cb;k5Wi zGh3SA{kA3`Jh1=P+!tffRdwjFR7?r2AJnsSFWgw|?tm3@4zfuQwru?iD&z!!0r@rr zDu(d$hZ87|N2K|O;KD_^KuJSyfyGS))ea|hP<7Gk6UMF0%}*PhN4S(M##Ikdkd$;XCJtCC@ts#m(pUAA*+?d(~ z&)g8$I~&1gpempG-=Q(r{=Xr@ZuS{^85(zY=R(i84c``8=~S z~p~OYmPh%=f$FddT2%v3|-!eZ-!^+)F1~De9r+6CiF5y2maMrbW$iZ z1(FSsGT_a$0iJ-v#Q*rs=}o;WowV^|jy&-eW=Q|Ig4YuJ-L-nh4O4 zZ@!|T#ca>8vizTxQd_2>9MI}dVWx$k3!q6@c(_fwy739^qi_PpFIVN|h|%Cu=H^P~ zMB#Xbz?UbNZjYrTjbN>uG(QYMSkeXkF>--$cNCQ5*<@)rQYazq%lWmy z1G*pi_^+?DRBz?_jNSoq5+Lr-ojU?L{TM>b@2O3KMuFfDhYj4#S}33?%xXnQwfz{A-4;P^}JhH z3&v8@6u>mWv?~9X)d0R3tOkhk-Fy`N`(ezNze<$ zW!6JXwEj9j|FjekK73l6OA@U|Ye^NBiaIp;IS3OQ;XG5lTNNp0BFSIRh$0YTV&dhu zhg_fN`wK9lGz28@vOZ-)Gd!n?N!Xo&$~AWAQ>P7@bw?1A;)=*$j)Bo0fBH~iCwt@u zYu$RgI}}*9!E63t2yD5bqHeCP(Bat_)V_GgmdVxsdDqlXa{fYOv1nF1D`z07a2Duf ztHJSU>Fr^$@Q zgC= z$P9qnqFIcdkBIx%@(1S^IAE^AkfOW?@)Z6vG=w37t|#H(U!FXU9{s?tY#YRW69uDD z?cPWunBDs|6(AC@Y>mKoDgaIsA2#$?vQK|VEsB-9Ai*-*aHo_j;QY*$=3zJlL-*q6 z6~=tnVrqc|$flMx{|o5xdL+BgWChRcrE$K+C>!zS!+L|@_$y{*9dX9SrU$ATb?N>LCH5fEuATNJ4lK)Q&40wD_0OR#|r z&>+332-1<32!x`HVCcOCh)5@d5;_6$uO~Qr&pzjT=llL#doJ9Nym{ZJtY?+`zL(0B zXCd^7bN=(!vv0GqvPN^^n*P6j`P&XhPCv{Ykk3Wo5I9tX$_Q6~_?K_xL_-G6vyxXB{hVCuvN%Ak!XU^m{o%I&rN1ZDQ<7h|QtgL) z({Dc+PYoN1{V4O&0=IkRKeP4$7cJ8lyxXIr?hZiUIMs6UzsFyYz`yg;LkHMLcB->c zW_Oy-ctZc+JZLXAgpLK(x!1f&kZp+ZF zgN8#?i0_0otnjnWkfh#aj+p%Db>!R+$L;?FoBcc+QgxmXWXDMRj%jJ)-AMy`QN~+! zv9MtXCSMPwm+rAkaOiWJ)nTtSjzJSKT{t43HOL2^_GFZdwFWX(_t=Ri^v2CG9A_0!CKoa_FSkmQj-X_$Z9_a-PVm3w z*^$Mu-6*Dm_L_N++_r%xhv?q6fl?L9=ITJGmCXdD17YX`JTa34ite^E-#$}j51>7Z zUg{?ytI>*QEhn(Z5dW;u`&dn9g%B7g?^gdOVKVeB&UNZ7LN{9oNo(iNJB$V)45lTEdtV60Wj*hj^!+=nX{Cf0yMTojqBj+#3#B4F>tXqv}HAEv3+S`KBN=7ZWDK{YI$ssvOh22hcaYiSG8ap;xWL0D(kmEjS}f3eye>;!sX1-(q@|CpqS zho*1N>K=*3ExJmQ5R%*B2azW^F@q9~kikaYgjo}`0}HW((*nl2-mT*=oN}xDrr=GD zJw36OYvV?mcyPi9RT!|Z!4P%{Ls-oStrJv(JS~I<3`o6{zA2%tgR>!j>qm5$pqJd| z-^4k;z77;T4~a4Sbznr(U5m4-P__exz$S37Tjz8qsGlXcm(Ip5Ga|}jK~8AoeZ{t# zz3x3PpcUb9(33)5neqoU6riG){%*Ld_mHpWA==f)cG28=g-mm=Yo0-gQIrpb_6+sV zhjJd}YS8MG`tbFh&r*eu;L@A#3&=luU%3nisF4(cdd=+tuih-Cb?@RB*vtf8+)~Bb z&yHXsR53(A8_-p`3Nkk;b^F*w;Qw0G{m(QC-|@}mkO@-gadL8+bS<6?ai!Y*5mqxK zsoE!}UIoAfuA$Y|Z4EL|n*CQIcmuow57sIrdyCtjPuAA)Ly5=CHwz+d{mLSe$3idv z7drmd9MDuwH8q{KY5;rXeu-1Zd?#EpuVS0h;`SohH7cWYX0jeM3|km9zgLzKREoX6k-piZ0oQm{BR@hBIu0ObLL2?a1u6Ci|jD56Gj=Fjaev`nI?CWN!j zus%`zt%DbUe7fRzN@5yQ{jZ%Y2M@I!<=5T1Z@%M+fX0CRT)m2G}iL(V69=~w)4d~+kD?_Z#! z5tPWP^1xIdQlvj0yJ{@;tU8ztU=yaI`ftXlO$1zEb)dH7Us;h@kbSxvvbKudS2F$zZHj4{Zu@sSF{~0i8DDDtnt2j+S^pcq7=o9S^bLW z=0CGD9RB^+d#eYFTtd~>QbwR@j>w{Um#1fc4*Bse5o=LHq6T`a{H2X}r<}u>c(ict zLth1d%t}kh38E#@`4$Zg);amZ^WD>bnv=ILia8klZhk>Wh7C~Q<{EMyspZ$2>?N#) z_fQI{V@AfhY^TX1+>0*DMilJcwv*bs7^NUT|4vBc>6E5hjO=?kv>Xn^Bv}w2i?9oI z$Cv0GPkWJW*efNi5+}&aal3$rL(lPrAfrw4K^j!F#EX8=*z)(2diS}caD%SpkF@3l zO^q5}KQXtGY*IVu*Vq;E-QP4aJ(QCc!u|Fpw)WvKNbKa%KZD-$ zNd6bClHilJUw030RNV7blaBo#YGmAxTycA!S7^r^7i|)Q2k>9t^C-!}Jx2CO0rx)J zuW&j4*G{j(RrsuzQF-ZALVMOZK~gEM%452ye&E@Wjp=LE={>+FmyD}Q!D=Vh3O;v6 zpZhiArd4xk7Ov`sWrXtLsQ*jZmuwhSxqip-LtRXsHKod@dZWrHZ+{Y(pRp%$AcOza z0QbJ{mz0y%ivN&}t2(~4m-7+|5{~2=kGmGLGeG6~?WDtnb2{YQGDnt-g@@zO*(6RQ zbU+<({@a_{cJz5L?$XsNpK*q6mp*?!>25l@(C)%LVj|4=#xm$|*X@q3I9gS3`Jr=YQp9gywX&4NmTNL8 zMAjC|aRYyJGzB~vwjtY8E&kHC)LJ=RnY>3V3DeydM}$6`JsSn;^uz{LBEn()7i;#?jQL-@h=(?gqW#=LsR0(qUffC4@HCu`TbGIiU#@L zzIf7l9vSZ~Ip592o*I4H0|oRh69XwRwdEVm9SY(UOJ@^1kHoC{i`l-yZ1hb*264Jl zCgJx!cqhjbccxgV(t2?&CP6`@3YRKNYKntxlHx>*L4SVOdC_Cwx%;k5XaY}0So+Es zPvayg7Sr8>e?ASpFdW)xxbekLqa#Ej$_fLgD{((#(I-DM>7U7ZYKC2eDl2kogxGxI z)6|=fwts~``J1%B?0+k*Ua(%5RUA}#_!6DG??3t)iK#7}w4v1A$G2pk-&48sQp=G) zHg5T!9%xikY|2;`jNV1zLs&gPirEmvHZtO;H zr@W1ubNeFxC#4-m9p5otXG~jOnk@_qe&T2CLz*ERzhj%s$p-I4Fb)a?o!A$(#WGRH z5MdwP^KURt^8a+iYu)T+OiZs4llsS~2Km{?=PGXY&Hj0vqpxfXjbXW) z>B>n)=5~Cz0Y9ZuhZgH;Judh#Oip0Ky8qnl?J=N!vyU}#?FllPql~$P6Lvi2#Xi=i z1A~JZQ=&>G?^DtOB$07r!>TRC!|?1FJ^Sl(!C|A?tKsIwx z+v~Gu)Uhq^TB&ele^m#QNHaH|=GEFtLeq9U(JVrnn*BaE0PH6G`D8(+r&Qr8HdJKM zkWe;B4K<22I;RPL;NZ`=f;U@9zCMHx!psMe$Be{^h0I_EV%alrxg~l8UqkJIt$q6Y zjHSm<#(EAEo>d^U$qB#nljxd#|80EFCN_fXl?h>Y-x71bIC|L58#p_~EdCRJwsP5XJi2N58|I;Au3ouIS=9>)K2lw*{A>1m?y_&;>JiKqY|o`GDfaZY z-Xnw$M$3jiEK703^*$tj9QplH%a#q8<9(Yt7x>cAeWV%eCj=}!InzzS)9N+XS6iwV z3WW%R4jWH(^~jxv3OlLhUF+RAEsN#S)92Zb_X?Q@st9lIl`g+gPx7ygZER6NlYMHpLs4mk=x!S+1vqIx{N?dl%2wllJi`6?`ZfeZgLuj_Cyw zcx+rFEX{$9>BC%Esu_$X3h&M2rzF>&uV1k^YW42h_#2~*>r0}wc~XY$Xxt%=P7YRI z3MCZqQ$#1DV_mI}dfMK0%{jC2lt%EB7HcvalP7=J_mAavs;FeTJRvBiov*Lcs^Q!C zwe*b@FAYcf_vd7#Mn=Q%aTIy4=#g!(y6sZb&i3Pqin9UN-|gCIxYPFl+fth5!HQE0 zqi@`F=8b>v+khFeX_l6oucnnv(*}h0y@-R6uu*I)JY?cu51I2;mXza6t|Y8f4nA|h z_wq_*Nf92i$7C*9f;6*)g%f41X>gi%=f;`fo}MtqjXfsW(`nf8Jl5GWcf{&p*k?kz zbP>nq4hWe4#a{WJK(7VeM)t;GscIf)HPk$@XPDLZsS$0`%CdZ?r_zBG0{%d9`!P6I z6g`La_$lXI$(p;wqJG|;OUSBX#wUEOe1{Vu=Nh|pbo zEwy##p8h{!Xwy&83`MCYB)Wu2FAk_hKR)YoWc$-p`;(){5FgRb&(h2~p<+BoK_)?7 z@$1|*`#02$TQG0W)Gl?s$Uq|#vf45D=wBm)kt7ov&hK1*le<>H5>s5-`?>W`|BapT z7VJ}f^}+#`oZ`2Hr>SULzA9cI46e8f^RnLNu07%v0ech~y?etm{SG>Auo%gi|XjIZ14N(byIMtzYW46=)y|+zFs82VematIF(ktporD7aJRZB zzT?s^&hyAm@qdvHbK1H0U+d*yL}-GaGPp_-;5XPa0vdk8avxwX+h^J(0Sc;zHw<5J z0`A1}{4NQ~f}gcBjPwbiWnQs(9e4_jc-M5oFMaQAQJqi!z~s_JXG)n7q2#L^S(AEB zqYXTX`)p9W{|lSmw81B#^1N7qnSC3Z!L4V$Zg4o{GNI1T#<_Y(b1f3Kygu8lJmNej zKU?{$(D~riTN}H~aeN&sMr8$76QW%3vRRrMAri@=n>TO<2Py)lE3+ji4>!CaMxzdy zXG-^CUnilalx$iW`Yh_o6=QBycDxXv#{TlF2R)X;2fK+PLkmF|rP4pp#_HMjx$TSr z+cQ2w$}|l#^E}VnAB6jr?K#<3eX|^LF5N78I1`aR)x+BSw`H9tc)>$vCabY(y5uSu~J)Rn@yWLV>rtf9O6;e)SsSL8mixE4Fc0Jr!cOISK=-SvY`-QZFSB<_~whX^y zm*Jq~hy0ZBJ@`vfdJD>=_qhk-*|tR8vSu7i1846+ztGC&Q`3#M`)zav) zLF<-{2_j@oSAMd*Ob}Yo+qirB6{2|Ix3|~FIK$|0I(g2-Y(4b2n+?P-Z!Acfu1r@G z-x%?Lb~)srJ@qr~T3=}p=8CnRO4a88eQ8t=C@gdtS?DZP4Y+2?fE3@@rBKgaJjVcXA>m~lzc?eaq7`CQyF zb}Xi^G+8EJIJOpsk+q19y{&W9%WR*B%v|JM>qt{5xqn`3UfERtf1pCp&S<$fw<(3{ zeB$>UV#@Z*&B~tgurA)P;#&k){G2O_@h=q!^wn>JnH?fQp+8GJeO<*)xwM9d4-byF zgr0TEcEk-a@g21a-*0E#$AJ;|Z^J+p{jQCbTGPxGTBCQ)#L&r+pJ)xq5GCd84@J8f?72PI9uAJ| zn!bP?EAH~S`xvvIB%#{l9OrW z!)TupE}pc6`{3WW4{LA~2-OQltGmy>;V}Ed$~Gf7e)_^7u&dWxLkus z1PjBM6Ot!{je~C)PnsSDkIsW_b6$CYJ@e>&_;-sicpX;FhDD%)dr}e2wl!@{Me>C! zlHcFxU>;-U>yfEuib zx*y3k^^(b=8b7e@!sZCJ8+qjNoqVy5gcxNX7qEV;Skm~~!xyTf3pZIEk>EqWkT6p#^5OHNbsrer_eO0mv)en zAk6muZaXhZi4kk{)<+z|=*+lS(#&k7Ov;{(TQoHC7-%4Z-DFr$uOQ#szCl_wbai|- z4C7rU<2h@DFS>y4mYo(TFTV*RGi$hU7~c3WP(yBz%5d~vPY(?(eeulxAUyivjq6sz z;i?A?!n)x1<=f+X(|I;ff?`{;Uvd7)TR&|XIhi+X)6|GVPd1DDezS~ncYI8rQJ4Hc zIr1)>24L`i-5C6J_B$*tgAgnZHe#N+P`^IN%VZRN-W0qX{{+JCa;Sh!AuA~NGcm_g= zW-+zLU%k(LrNG{6j=ei8_!EBnZY8Rd$Vsr14|jLU+iN?ogpo5Mb{Xw*yvyDZrtJhN zSlE9(+vOeq?q%z8n5y56!eG?<&lab~xj8GrhQ(XlI2*Qwi|r2a(M zxRV$PI1U%!>QmnjVm(^-U|LIP74A}(-^9C6PAbH`?}IiDuKdr_m3H54NpoMcnUb{U z;6n96N3EX4dj0A1_vjY@P0F41e>t#P9W5QO@4zEd=oa)N_S)2@uM=o{6M23 znWko4&h`>ii0h^dHI}6frBqA=yGqL84q)$E{}+ys%{+nSM>)$qv$GAtWvga36e?-o zG9_EoH4&y{v;PaYuK$a)Op(Hc3CMC`;aEW64pfKpL^XGS)ej;Bv=nLj6ClQRnD2qu z2;aLLHfYCsmrM6ND=LPQ;{{L`t(^=Vj0+puwn= z5mp@brGd=Ny21A>(Gu@Rz$oXM*Jd43;``aA`aGp zYEG)AlX6)urt&VCuR}Po3W5^uBZQ&)UInr&y^ybzS zD1)s6wx}99<3w)PJ-!cWkigbD57h*%X_fTZmKA!sOn zjoUKifv*Kfi;0;JK|w))*sXIzO^q7Gpg5DIEOHam*C}rmQ9oVzviE8F*3B<2oPc%H zpHWia65XUz!7m*L{(yQpl-3awZU}*_vx>&k5GPfSSpc|27~L8uspUZ-e)yf*+D(KY zhv71Vepyk+Z(o}POcZH86HC9%%6flR?>j!nX|Vw^zV8k6)>W6=e;?nXR;`>$=Gz5aTQBTjHhiD=kSYtnqa%I+X6-h7$;xn78QX=r z3AWnyBHK%2{%*AE^4%u(K%Dd73?N-d)W9sV3;Fm#0EqYe(l!kMTnIcU zd2b(R`{TLLWrvsrM)_(`a5sjc^>A%eUT*=cw7K!ZD!dl;Wrm-fM9LRFT%fn)ua0F}1FV8H61` zs8!bLh(vesWCB!IIgO4$mJ^un-HNl7PtNdDG=emB%fzGR8Q;e5iRV6m3{Cm1W=Q;x zCc~K)A4Zzl`icL@Ka}Sh)yGCh&*3KkalkG(15zf>u`721iuC+~l)u1F2>YG~THgr# zUI;h|giEo5y0RkFsks&d;Sn^#+Zse8N)(M3I)X5+^N?H=ZHO_wY@qMTJ|8R2cq2D{ zE4F+K@&A8qzc1Mfa0v+n)1o_L@_Qp*zfMiPN#p;>@o$`U@|zXYo4wT&w^~TYlxG_Q z2+pZeHI{2*8F4;Wzmw%07@aHr5NXqaS_I1P5rhqtO>xC#FjdXR zX|!1&jd{dyqP(%Mfjy?aRj5Rkr(regjqm4QciGanT3G5E=(EFEC;>u{I8b_!wCVRW zGb!xMOcvh>P_3{Caz6OzKPbG(Y4gPP0`S3pW;d*p=XsUW}{PL;0Bm$=XpdieT zemfki?!nWW-+%lsAOonhU2QBNigv`ut-7AFVO~f$XR)5OU;JM5RaEixf1L)2Bz8(_ zsa>#wABY3Sh0Zo;D*{zUeCPS5BGnX2Xe{5>paLjlR9%OrA=@Zz zdMyhEUXW-C+arVtkItPIyZ{+7QQQi62v)3@0t+Efr;+>g1=3~Z`mJ!Qi@_F9j43eo zum%pN=R{6bb5qlW&-+t%nBR?)tgAc{2@s4Z*plL4bgBU+w>8yD7$7OW2>t>vQDs;= z8|!-#5FpykiZ=9ZN)!5Zz$b)b5d8JA)j{_TBSP6~FF9S!L|aE+oqYfWMDz3qO&Jn| z8eKdRfnTtnbODeZ0RaJk>JZ#<`AR>wEZSLH^ zz!QEmPc7N_3Du>KyiNoiz(&<6P-BHXdubL%boOnenCUvL-*TQJic@%%WffOgqLqi( z<*uvex}Y|v6BDR_ukYe5QZ1+2M3^6S9ls1E#AE<4YBo1E>d(6XDh-16A;1A^140Ik z2T(!`OM+k6!sb~9^23Y2a2X6h$B@3wz(FOC0~;SIjF5D{B9_&Gx`uTrUl46^(#VvI)!DlcZVQnl4!A_i6DpC<$aknG6(u zXC8Iz82FE^@1cqaf4sUCT=25u*SWs@75wFm`2axkqpJbh8DVz|zuw!rIr$Hmh*Gbw zO5`A%Wfy_A2~wPK&jc;cY3kG1>(#!^u!6sUs4)SWS4c?2Fx#O4WIv8N1m2G7ym8_( ze9P_&>r07)0q!EGw7Pr-#UeqWTP3#IvlFT(CGDZgt>zBt#&x?^GY`c7#R=i0BB;wZ zkUrYYn>2t~jjj|4EaLe*Y5Ve=%O@{stRCHiz-;J@@|+zW!fQ!wpgebb;0h&|7f5dt zF(_Wss%LvWpQ`V@W-shl?cg(-S$37O_{cLX^Gp3f2DEO*diqqIB(F-_zyOGM6i&vU zLH~KysVa@kzS|r5lXbwNX%m3|4g??H54Fw>#%)Sg)ueWo>Vcb<#2gHAZXnikKa?f4 zdw;-U3Ll(Sk|xFb=Npt?UZjn`RhEbh?$l#)tS1?}UIwPjGA(sl0|6wnONmh2 zl3J8DiC?tz#UkCMUkhW;tE;Oo02k;@4a}JT-TA?(&>EJ!G4Yx;rE@P}8MF)kKV-O=JO(Jv%_2*8s;n z53qkjI#{nuKzs>)f{`v9#Phs<2`x%gk}lq?umQ(3Ca$bTrb-|brieLNoLv=-1PkE1 zR$J88E}dcK+SD-M%sbAU$kPKzV+^$797#zGu12AU-+DMa%~gJ7W&}H%6%CcLsF$&x zzPU9|f0o!9updRs{vY$5Y@Ek#lN0>%>CDo>{Ag~hFnP#bwYzL>YLvRZHV=Zo1PVJ%YQU`Fg%O7)B*YS-BQO5C;xX5B7Eq6| zt0TB|I|M@`F?*-#p+8tOT0B((F4|H>qi^VIZSa&QsVa*e&RsL)G%*E&+%v5{XUx%o za&VybrCpYas5!Ii)chS^{@NB*AIblb3$t6F4UnR2?~6Qsz>~#EPFc_PS%ybXMqp~7 z08Vt}NeC*9NtD%t)Spw!%!%qDOy4`PVqGWgUirg`xwOGbpAnVp(GEq#o>E!tk<~nk zTVLh5`&5`slB$)dVq0beV+4DD3&8re&~ji$72GhM5PmkmzF!7I8u@VRE?XF6O+0+P=$P53q=Tm8fxVv86AX{~Kn0xXcX1l_ zkNmnd%uw@SOqFQOdJ!ZkM+cu_#TwL{e-Xd_0ci7>HopdYwhQwycWIKo6;@5P2zk4T zbgInr_$#IIPZLb@bRzCg#=v98JXFA6 zM>|4OEbgTm5dV=!vSNTFInZ&UygV3+#!y0#2^iim@QZlz?g*287keYnPC-oeGq5j{ zwWIJ`r9S)t&64T>uiKH0szI-e>T00=V#e2(lCGy)sO`bI@|tBi1ZVRryRtPe)*=;C zU*3B9AzGLaDwH0p1EyF!ppX;a&>=830O<5L1&>dfRMEU4N>9&RgHDac$eUJLQ_75520l?|Jgl6 z9)e6z@0zdOA;o?NGF8Czit!NvcEOB004nHQ*;7;*L6Vf>La64z+qpz=@NOlIpB#1|A^2IMCj zSY0CM0<^!itgV|X*fr{@=+hDXN&ihSCsleav$Z>xU7(g;qdr*}a*?i$zL~APYOW{b zfuXSv$MTa`h)0b}4;q&qH(vM1U5bBOk{~oe%+`wcJ{DCo5y-FJty1`d#STYs)X+x)?_YFMrxq`h2!Q4Od>z3sYm-N|3tbq$-^l(P!R7`WD8D zmh(=ytbgGMi9I)tI^L$ajK|kMf1BSIRMlZA-R*&+Gd?OV7Inv*vASlh-nWD|cUe<; zN@VWF`jDSR^TMt`u^tHs%6*4Ph@OVxF_lSVDPbXFR67938-{2R=)MqPe=sdQs~UJB z;^CJy@#@x&A)A5`#2sEgw62@~-K221O*9`Rihp%*0urg(f-Qn|wQkvBiHOfEBo<-@`dJ@0Ys@pC~l;0kHoyYY)`eR;| zp!Xo6j008aDCj~6Yr+8wNt0P%A(U3J*`&%4EK_{4z&lexv<5br21$p+`>l*2s^iTY zDcnT@cN$MALQv`qT_;EQsY@DtAXW=o0zAY&iYIclkTMO1-KIwzY61!*iRA%a2?{Us zsAl$O+g*A6r|DeSWcl_ zuhZ@>^0KCRJ%OJXhcL*LUI(-5HL8-Su1 ziJDpaqnvoD@qhD@4xx;vmnGw?YZzrCIx;+$dL$Nr;r}~~7W3#(#3d8T%M0o8S8&F7 z_;IecT7@kXC9Lx*%XJA40~4D zF1yreMl%hpU0htcDHKGS1ul@^-t=O(|~ni zr7$!8d48sjpS${yA6KayiUVm8>OXAOP3@68@%erJP5%MtRi}qn24p<&wdL;kjvfO# z=YJ4o?X@XIPi|@S20X89=CKO8Na;SM&w|Cx%= zMVl&q-LA=5H72nm*)<}4)%cR&WUT4%^XE+@K z(B_5t#Jpbh--;*jtCO7=A#P3lF2R_+mW=LA%knkQtgso_DM0B0HJLf4N3Ra%#+T1_ zLb1%9k_|)d%F!{CmiTZ&Fi%JbX!+h|HheMVjfz`yk0JG|{PcPE=m^fVwXMwtWN08m z8ryLNHq-u0crq`u+`&E!W$MRl9o?nq>CN11ANK044AZcTD`tH4SNiDdPd-D5>T6ut zG4ZHl)}gm5oyc#Ui2P!_lXnu5E;@fZLs2~lu3`OJz`4%X|2M6|&kqsE+25&p2r4}U zIns+e!IxT>J@__<;;bN-LZIEyGnDE*?p^uhbMNUrG|oK^(VK|@m4{+#<7M&~>kr1r zDSJF>WNohbgeA`+it~7Zs2i^N8K4_KDzWq%j~~LcEKHJAe7$lCAc8wXGj#HNT(say z_BC>|2w3{fwS3FB0*nW3^jvql<8K+GA~%{62p?l*d+E0*FTjzy77NAO~ywxaXU00Y+Es7_(evVxDL+SdtK`SokMW#>D0r>Iy^ zygDcuJ0weflpwgYKXLYMQX6YLYJ2+MUvv3S1yhsgj)wT0nf=7OX(DsEB?SL@QMkB{uQQrs#o=HnA*@EzTT9J0ebl<~ z+VMIQnZ|Mc(10AxiVyQ0z1YQOLvaxzE)7?cflgyv-kyd{mk~to>LA}UEXvz~ChXWV zzTPPXHKki>>Ux#heZsLadCr@I9v6!y?qNUNZtd^Oj>Eq0WEsIVA3JdPca_77P4A2) zPOKO$-ZDs|#rEfCzkZTMXc}O??_Hs-IRUO;G0V2yas`~(g%ohsDHwgejS z>wbAd;mexlzC*u>NrVIHl+(oaVbRL<=4Y^)JAY}6j5?+(NSfsY5gvPNZ*?G4(}8io z*|9LC%xe}Sa7*LpZLgzVr$;8d602e!Zc8b3E}WHf`rJJDk+$NV!OS|p&d96p5;y7K zo}7qR?+q}PARM^bWEy#`g>ZqVV2xG%acb1kf8eV-x0z(QEo6~8-5iYGu15u4v^S4? zX(w`N@%aav=}|0qK4eSkr5C4mm#wV&zr1KQu`Q+{vPtEYj8(*2r`Ga>T~K3sTY{0e zM8VSe`y7btl9L3vwU}yO(>wX15v8|7E1Or2R6Wqi62mmh=P=eIAIIeDOpvmK%;ORs zUfca{cOe+9j*M`FxSCu-#`*Tl^MFm0l<1&N+M9OKb99JPZgGs0J0dbnPMoX~KX>yo zk?-Kj_O#t^jU@s?0U5YUu-Mnq$qOh*U_gAp8vR3{H5o#atD|(aW5yf8=R0Vo| zXYk&9)+(YtuE3+=csBByL(Tmwi~=Y}00jfD6+$5lJ+u?(EKKN7+t(W|8oe@uq&!_uO|$#KC_X_7H?86Z~2W*GMs|N;Z@+>{?4xpX47on&;k7F|?_W zw|7G_6<5?i&QS{38ZIX}HPWn>h^r6=A`OyvW`ZeW&xqzwSpD^^zzdMU`Q4E7nMcL+ zJmycDDurVE`my8>65T6cdw1543eH)%%mR^5qK4nepsl+1 z4dA)bPjD*8y{1h3A(1f3>emPwvl8bM)u|E4N<;emCtPWCfo5ObC!HDll+R zOxoY7V7Sg4FH*7Bwxf%hy*P>stc44W!(+U2&>8X0JK2&=E&ju1ZiFA5HJ?qZTF8Gq#*_%*{WG zqj3Ljly*hQbxlD}BZS6Xbhz*xtBN{nKn`x0V%@%aaF*y3f~0O@A!UUmBASxs$WS^) zDt|VR>;PT8f(n|ctLvSNgt#~F8vm(gmochJJ5%TLhh-u>I?J#&#d^>Q#Mp=-sBo>_7@=Y5*#bYdWHYDebiYm5#4+r8I2x;sSb zXq#=_o?kQ{3T)i5FGR9#b2AKCfxOa}TYKN#a2R7w?1=gCs8{5#*2ksuT2-c7t<|P> z+IAmaJ(iPK;!O!(S>liyDpmHF~K*~#UEbd}%HhSlYG@EvTb%1?G;8%Yndgk14ji_B^G5pa{X!Dba7UtV0JJe`|l`o~W^SRO@qiayd1) z@l04VdID(6`RA4YFKk)mhDECx>crr5YnStW+I#bDK9 zBP{pG6-B&1L+rBA2$?EmJ&W(8=ltXJnKm!XeaV}#F9|nkRDRtA3rEQ#pxCURVX}N+ z_b4C^`dDjg{7Kn<-WOO4Wy=T1TDn@Q(RPsP#b4sR_<82|w zW4h-TR6Z)0zA6hCr7{rtmYtnK@{r+@rs2TLmI2DSk*>;oD`|Z`y8-G4g;tb{#E#U- znkZDO(ZnOxEfNGpDj=ZXI)ak6vXG`kdaJw4ZkuQFQcIMLM*W8dRu7+#`;p0P#$@$F1w{7=c2+E z^$&EKG_5-{!vx2-U!$(}W!Kr{nI;Osv!fs{q}Cl##}47lRTW)^z~TpH<~j+jZ$ z7%PDf2RquZKEds*zM9_nnBaihF+|J9jfU6cuC6LqM1s&lRU=seI}`FG1*xy)67b+F8AA2qSDtvZ8G%RWxBSP zr+n&7)gG^uGJxT+0VsrGMAJr*;bU~Vn?IQ z6$i)r5$>+X@m$U{JzjOvW3|4%=A3?-)}Bf%_i4u>?Q=K)&x<4mEGIlJ?fS=>!Vj!) z7Dz3JyO+?+k<=o}#-IWUl7F6r+_n$oB5RRyp`OpT?NB4pVxXb~Xr}xb;9Go}-`vE+ zgn_h4kUSC1VzCHP3#t28n?i*^Hc%0=$9bSRA>dLggCw74As;>jvKTe*ROeGkptnLs zvRUj4Lrw4~bOT0Z9R7tVy82W+q{)Zjt56Ee8Fo-I09DYwLz*BqAU?B0jnckap|+~- zw?3ON^74`{85EilVz8x(DCsu~nT`FRd??-0+1WW_{ZaqOq`T1qmc=FA@Z^vXUFC37 z_l~qHp%*g9eBheH0FN`zJe>WTANSNsrb$Ml2a#1GI37_ z(VgPd8rf|+Y4!GCSng^{!}V63+2-lF`@VgduJ3&F;5XbMe9jtjZaOc6FOExBTIlJnW~<~0v~zdm7!t?w-BGwWRDQZGD~7V(R= z-K%S}ge-a`rcd7MG>dar#p4cnj&_N#8k?yFjptcdtsSNBKBYVI%XRb5u8T(i=$=~I#cAO& zF&(`p?$aIL79X!2$s)c`@PTYjEL8bzKolvA*TUyc?lm%D@B5w?8CP0apG4{H%L={; z#Do{Id29E-BEbNZa6{q*aNxRpwr&6cga8}8;rh53C=HZ!Eb4%=!C67LFv1co_W=|L zQZPo!eJ$Jq@LuhBm?k7+&CYK_3G~gi1e9@s6bhol++QQKPtsX%~P`$hm?!D$BENdKGP)+^X$JOb2P)WtI+X+<$5yknt}iC zD)qShtG&aR6<=gnn3nfk(X-lQO!~EqyoB?vD$-~3x_&#*OW;%S3r(^wl2p5>R_WdU zakwDpo`Floy%(4b?Y6zcmYHvpIq{^tr00Qz&&Auj)4-(o!rVv*y)Nx}tR-?f0Rr`>@Jftl%Q z@N>(X*1)1eb$s3T0^=M~6e(x|_(;Gw|7RHVv zC0l3)Jlv2_W=sL#jd@Nj(rTN%w~KT&kM0JGOqfDwchtKZ ziUgy&pcIDmb&#eUk~~ic;gZYTStY*md*mpl@=2edDbBo1HCE;kHD&S!ebv@{yo$Se zhOhs8QoTr*TTG|`lMIXpZ-`LC_PWQ#X-CB!(-ki|y^@%fCHw4%?dhYxU@N?$_r9(A zT2=jatZ8tON@po9Dyl}ENH;CN=3~Q=xc|9K>8xBA7sCU?xx49y;|J?UfBG>tKYvEO zf~Uh(B_HF}ODXmJ&=>DZnD0p{;vt%vn1;SQ>UD?sF~YCyx$1iG=H&JG1}c?mg-Zy` zY<>TEh3nf$3XL*#+dCnoZyakL=Q9ww?yCf;pTXX;^LvW>Y}}c?b6!ee^j@4hr?YqH ztmh>&{>|6Y#({wO)rIufhO%V4^^TNQZf3VB-aw0qoBXMTi0P%#Qww5{iY!~W7Si{M zkbd4Z3+P-`bFDC+m6nA1Q(SvnN37`}b8-k%;qVFkmIz@`VjU)gmqbB<@oF9cO4f!y z9QWFKxwXT68z+Eg7MFccDIUQ#E z({jHPj3>MLS>Xc#o;#xU1(0uYA7dM^KkG*m2*bC7MiIqDqkPq5tJgWRBPfBDYAFjRg z&9+R@6WyYFGG_2M_Wy9LrF%=ehi0p@>9qpU6PmQQsTjwXG50P^KKhb7vdCpKP<=+q zJKoOSv&6Q(*16tg!Ka=vT--3*?-C!V(z_()zOI-l=ds9&msq8C0)$M~ZK4yI2xN8z zy+&XAfU0u?lA=Xo#eof#IkFY}WTDOCNG}+ak9_QEoL;9mWH}LlK_%e)WuIwJ*|Jvw zNPZm37feZQA2;w>_hbITL61`XkwgEtT{qq-$GG2fuR2?Qeqqo1v6e&g<_bC&I$U+f z$3t~pV|4R~Ts+v5=}O~x=Y)p%GkrIpH-*yk(Fni zaN6Oc?vYlDB>b~Nh2@WScq3fFKm%oJASQl<`}hR+Uw4L(DjgQy2+2wY$atiEK{Nsf zf(vgJeEi#^X99{xHJzI^#cTkdph(Nea|Us~WbegEz7uzDpcrL>;jl>SkY+7~up&{_ z9O;gM5>}X1MSu;IlRZJe7(E4Pl~GeiNq`<8%kQ9-1}M5FkP$}Y&SZ~bMIdj(bbp0A zbL4#4P#71JtXJQkU!90Vvq&*!g}Rl2=D=Mtz);n5Go?q|g#T25F~(j3*n}463tT^bR$8 zehXuDwqXSRYd|$pRP_QQp$B?_fD|=#^hWxv2;RtsG(u6b8flGzwV80^DgH*FIMoz% zSi}HC5>@S&0ZEh~k@aCecs`m$r{43aKiQ}4&BAiDK0>?T{drf1k&vW{v{lRPuj|$d zvMuA3mp<=h-l>75X8)-3r@l0h+ZNW`yiMSWv&fy}ZT7iK5oDLpP)oH)M>AGn)p)Up z(B4o1MZcDJb;k~OnKLvVuSz8yJY$Q2b>#`VQD|9m?>&+4T2|=0>;}`vquH$dHunG` zHXw=aFCYaQ3nkZEM2F5M0YcY+p?z}QdYmcM_C> z$}4*CGe*6`f;=)s+|G?LPQr65UTeX$)_D#jStjRN=`8RaJJ#iWqdC*PYe(~oFZ?H& zR+fUJDH&J}$7H=t9KZj`!G)>vcDt8%%`ewar`c6n>0@L29a>zapHSbNt5Rg}+qh5# ze?mj4a^nSKDu;)(@6Q~s0=REA%Dd9yxRps0;!tOZv-y5asJ2{rmy6JY9NqB{dM24< zqOE+m7-$O|{wnv_{37&Me>~q;^b@@fM*Y`ccdB1bZ_CxX$}5UPZ=l$ra`3DB=^!+Z zhsE;|_vX?Q*h4)0^1{pNf&en-xIspxhm%~;WxYXI6M!!O9dUYe|Fl7R+hNO6Wk7Oh zu0KDEE@qM@U5tL(LyT?V2{rYW*4CHMxo8*wu-YT_ahD(iClz!LB0dpKXwtsmA`sI+ z%JK9_1Ze3PXt=$WL!ZvywHa|?PJCEdSrNEZz6lu_RHKAQWmRZ=f=}M5C-QAiO~jq? z)>3b99P=FLOb%`1hX)fG8Ixm*3^o%0kc>f6SqUZ$$J( z=;ip4P1xZ$So`SL0|YPDupB8V=rsJ ze@q*S`A^l1#KSheAtm6=5;N%>L*v5rp#5b2GkAq3`yEW3|J|DDKju^`W#YAbLOh!I zSCWnC@Z+l9$iFqlMABi3?Yqu!WZKP&g|D_GxBu;;a8LfZS8xaMqzk_rC&D+r7z}#z zrJJDm?`~maZlnOie!Z+^lgrS*PyYY>wRoF-fr%T^m1n=akJ^3R_}BlmD%pSe?~C$WFQ{q17PjL#s56<9Ho6gyB}#Rw*Z>$FGj+} zOv4(fAUAuvU@ieAjx+Du^@U4*e)87n)P=c22cs>1%+$%gMj5Y>;z%ih;QIi=8Eod+ z^Ah9}kE%frfoiG8+w379gn)Di2DG+FRfCYibA5SamI$)RNQ{#K@_fnA%grW2lS>vl z)5LbYf0CAd2xXF0SAygB=L(7oWG^LIp75Xk}eRH+M%9Oz!q_y z{>X*M69|}qpj0T)F!9^XK(}F!@*Gb69~^kjG^fAf_S(@Jt@P`O&_ z{xLX2*%mqV=fX2P4=ab>KlPGq6V~rkAkGSFd21Qn46-iK@@9R86uPb z!j<`h;y1DX`%Q{mhFuM_!Y!!$Vd7}#eV)B1j%nsXBOd3iuPaP|fdD)uBR9SYE7T+j zo@5A4G(E64HBy1wuECIu^_4JpCbTmVpbG8~bvF_*(C{Y6R&fUJ_1}NR;l|cE@f10j z*<`&_nWF~!TfKHrZ2dX?4gbIDt}`miGYexphP7alwNbI2*2go&N_Yk?8a;$#`6P0h!M-&!0z?_O7b zN!l!G|9)(D>2m}$6OJUovEKT1FDUm*NI5e1ex?4V2gVAVR%8ciX)C2d{mu9%mUUR} z{xo<7YX|?6FvrlTjo;(^UW-3aLA)YKl=K4!j2~aOUF%R9c@b6=)QMTH5j8tiD@z-8 z6?8i(3|6jx`Gc>xQH$n_pPD6SUZR_#-;h{ay z{DQ2KfGvU5@x?~g&=Jts!{J|)4We*<3U9C)(^|?JF^H4qZnj}OreF9dZfk-Pb=I4> zrHj4wq_VZ@m(zxbdiRw|*!zXGa8BK>c`geedf_tSy;}19xS&jK{ThrK!qw}x0)K(X z87~TzgkV$VS*#bEx;c0}|M{d4S~e@pDcv&`UOW`udEa2*W+x_9&J0?Ry?~dK>rwyY z$v#DcprbkFnS0rw-Rf>1Z%n9v*o2N&3lsCgzDnJh85akf0nAx4*w%_}%P{8GjGhZnOK|KSB>l9}L8H=GaNLh04g>xeWdZW)(L> z#`#&Oe=5exCqp9Dm)H}e458yhf?rp?BWY7GVVRshl*(S7+mwVolHF?P+Qa$bHm)?5$;9w&r!u?kJoqpshGWrGw1*)=u9gv8EL2+r7N;33QWK&+UEO~ zk&>Y+3tLy&jrzlAJUx;lht|@l4N#{^$Y?LvIo0HOiQZv;^Xt5A3eS&9xwu_3qAREM zG&fihrPSB2$cV~P-W94y&B|f}Y68_ZyM=&xFri62J=(Q9slL2X0E}-X@awYDhans@ z4bhOx42!>)$;poibI0?v6Q(6ATNcGD_UWh z+K3_R>%Wye=B{pdMN#D?MV}k`lQ;GEm-gDS?$p}Dn{|);JYCs+@E;Y??hMLCALlRJ zc!TM{!(CUg=sy$Y@AttwU`Viyqgf44xw{l}DC_WnK&4rRpJ(rD8>GSfe4iT@I zHSS%}m3F8hN_1I4$Md0RVoc;*m9ju{ST0hVS2P}#*jzyNyx7at5gQgsHOhDW zl>#xYV(yM)$#a_-G2P!JxM7+UFAct7!(D_n!|(S%a7y@;QY{Wny`?%Wz7`yZp|{WF zupO5A)r{kEA0O~B3Bj(As9{dcR>Y9vdK%Jt3}=K->C)4!VQSY^ZZjJN;x;j48)c0- zZlIm-Bd9DTQ6afd6rj(OdKY)H;=KAej$NsA_BSHIm!kE zMw8EsGJ;ey3V<2Y3d3xzv8RRgi)Ks-qwIIn=⪙pcje-&EG~|il<~0to--;ZqA~_ zi&DKPE!LLnlMD013xq~a91tg&IxdjvP9jisPJdt@rxiVpxzi%!f0tYCndv=E5`jM? zn*?9xx3UIh5xLwkBb!E9qX>rMR-1xGD4jq@=x{g3aK!@D~%*t?P>TuMK#b2W#! zn}6QsbH&(e^a+!RDl<|og?Cfq;|usYIZLd0X+_`A5sncRPx$90!%Aa?@rB*nU42*o@i!JmP#~6&AOS0fb$=33P^xTLpVuLMR({$Yq@=Kael4u9I0my`iZ z;cTaF>^O-Dz>U+j>Ayl&Nol@}J1!>cfKzv4go(Km)5gM{d z$!J_0GFCI0f3BzYYE=FG??g}quLH!`S22d;Bm9Zu;rH3_{Bpo706D#*hx7f#@kPO- zP6*N@e?fB6EDxRm-XP=h2yp{XlHRlkoWa3YKz)^k zB#wYCRo=rrps?U55eo!s$A)Gpa+c%U753s>l9+^G<~bH@q!*45UCX=VUQ3C5;zqx6 z+kloBR7b82-=xF`btamR-WLp%PzjxL9~j9t2%pN4Ud7RT`;Ml(9;@OBxIz+|(SrvT z968MpsYy<%hhn-!(2h{14YYK3{-X_;1y7bi?W17mqDe{AkmWd>j5+-z#fLWJH`79E ze9?nV?^MBNk5-CaVN#Cw-`d8vS5$SUglB z-Af(&66rHCWK7goql3uc?cYA5QTQGqV?2og9e6OR3v;b_<+v?W^B|It!D;rbH5nFr zv726O7!d&*IoDn0w*c~y)~CU`~s95p-$L_P3(D0>Cx2IvVCar&8&g|WK4 z43(`z&_Cf+5<11z#ibOfW*qR0G6QYVv1=>We~E@Kqen^leUN0WCu64~;bSIjYls#n zqZq7IGtMkIO&U4-h>S^qr|3DUCt1WBjb|t{mUk_-E-VoDx@N_X=%Tw;APB#cvnH)v}4$g?j)I;k{IK=Pi4yK zRQ^GB&9~uVZF@mKQbn(E--5- zo#jnIi*X&BHxT)1`whb4>(7%Q2T>}Uu6!}lG%>d*+z0r|30N2@9v{rII!d@fM5lfR z>jl6sXHC##JkL=iBFYVQOXSucex~gOExI9xJcGERSAdC6+eaHQwS%Jsir($W1nB|| zFOvb85+G1HGK*vqb|i?vt4XOuNFH-{m}CRH$ncBDk8{@P800!qW(J_uH_+2x*b}^} z?21KrC9=AN?pSe76+`ir!R077jFi@~`y1(^nqU1`YydzXEC!fd?NjGWOV zx#t`+y(oZ*Wo1_eb-Y_kA&>CqCdX1>WSJ#zMV$0MNiLe8HkRsMICY`aW-T}fiSYJYj z=TRO9%u-=^!*H{$T#f&b%Hi$zaV{kT4QkU`Ccr_b+5*v7EG>ouxXa{KW3w)t8o&}f z^HpiQ4Tm}pc?0pd_>vVPbJaU6@Fg;cH4ar=LA0~>2p@Jq7c$MyZ8-+VM%z97ZLG>iDi>i zPSNx5BaT~LhBuOan(7j*Y*v|9)Xxj*vJsWr1&U)(E^&s3pt@-^GjlIf{OTscv~l7_ zE#h)|Pn(Sw2#)w*7K@I(Glbz%E`YR?mwy;6ZHny>N(>iJd~GMow2AStzV-w!n%+b1 zTk;AD(Ml)}mVpH!JtU&oGO*9(*vy)TpOt_eTR~P#y8ZgrgD3t8R<9YV literal 0 HcmV?d00001 diff --git a/benchmarks/results/pr328_rocm_mi300x/single_gpu_memory.png b/benchmarks/results/pr328_rocm_mi300x/single_gpu_memory.png new file mode 100644 index 0000000000000000000000000000000000000000..88e871e4d8bbd0c1ef372072a26bde9eca5ee490 GIT binary patch literal 120156 zcmeFZhhL8U|37?M$%v3NB_oAOTH2JNg%S;EY7eEQjRu-hT2$K7(voOMOIkYHLzDK< z&`$UBxW3=V{kZSn-|*}4xvtA6b)M((KHl%w>$zUXv+GwcQ|)BfNg|P`WUpLOCXpzu zNTe;NC@JurzMkWo@gHGZX*Js$_Y7?vbnhFG6m@N_%}H@I(Oa?g^Ni{~tt z(3vB~wzgI_!ra^z|L;$5-MeqZ%_h^Nj#t@Xbw%BVL}JV&{@J9(5X3~dm3id|2A zYd3v!o6=qOBgfK?4=-*n>1MmTnRU}zp>}5BrKy79XlI-IHYHDZ?@Cf`I{JTnWiu8O z?YeO3|Nk|gpuc_3|M7!IMgH&i{_h%KQxN+g`%`#0-M72v@IQrU(T<)9ju!u<$vB+01=;diuIRmYN z`-w%qQYx`$?b-19r^F0c?K9ras7Q-(JF_?NJmP%WBCL9NFfO}OOXgGiBae9|2{$J0 z;-#Y+`Ii22FHeqPk#MqJxI;Kuv1l4RV77vTa<`Hr9-dlleDz7qrVR_GyA99NRG3g?6h)l zcv4a#L42@l;>%n2vWW8)OgBhuoVnv+`Sa67O5KGA_}(QQog;V8(Z7EE+G=Hf^ytfc z_FR^s-9=1HOnu|yzJ_&?y7=~am!Kk zkG(#HC6ckk5900d0{d84w09=*O*Y0~ZqG5=E516x>&L*qGd4EXP-5CUJk{Mx+h(|) zKSJB2IpWym03E#b+uQkEqGDqGqodE9{*Lu8564tJJ8-ggN;`5C#JrRGItwqV`T$J+!r829Z{a{Ss}V6Hn+ z+pg}B#j8I&po2daoUN=!NxCcecw{3nG4YvFyvz;~ZPDFx5#MTSn`?$~Vak!R!E7D> zF3IWCzZO#N*n3#iczS@6)svEns-<8mARyonp)~8H>d~3v8-%G$#4H?;081 z))d^ld2@Lv=k3~1?vp1k`>{y$+||l6Yoi@*h!K1hrjbsYW7I%(Qe}dK>@N z+2%%DQo};r@R0L~ zF3pcR3oGjH-u}IZi(;G0F^sCNt>ts(4ywL8@KRN4->H}1jl~nKV-83@O-nn&qhGn3 zHLON)XmoH;e?}zDuOpxw*znZHbhyyx`yi-g}sRxbA#n5Xy=!#<3RcaQNy{w+4O0>GVy0reqZ{mV#cCpneoqDNO_NAqm zQ~zL2AIWF(xVBbdZLwsdqOa6L>9f6shl!v?r)RI*Di!HRg6sj8rT!VzFJAUFs|BqOyr+$8(a*1Cn{Tcg5MxJ8RMb_DLf3&8Pa4YPvJ$(4EQ^}1K z72%CcrFe?j+1cmMpX&%nNM&f31VoED%f(xE=9+l0N_oFQjj*z_lU7#VUmqh8D6z4Y zqqcne-Obcn#9Cn&J^bu2J@DnxmJw{*idC~~!vY51LzLNLO}u&!ZQs6~&-e!^|6Z?* zy}fS>NN|!Hybp;$aaH&q_VE58@KZjsf^x#YnzK`T2r}CEWaoJJyzQ=cB{4 zR#5##Y^vfhA2->9;_lTL)!{=-({k3&R1i1T?| z!xMI3VN6KUpa!?<NVJHe%0Y(5(ztzfI&mw%gaz(u;ELzrP>!7|qS} zEl^@fQh0q{$z^3USr5x{R7gmjGPbLxh&|?{mn7?0Yx-#trGtaROxL{9)59$waO+*z%TqV^7ZemEs>zk1vz%bN`kI}VOOrj|u-K`eIeC{) zYJBf3Dowv>lq->0T3R|VRQJlsWMFVm+Jk~B!S=}>=5yg;dDAwMlry!w4OdJyR(m%h z)3!TI_wYxA<Q? zas3H}&}^BBw1T}fi*uT{jHuXCe4MlvzR;gPbK`$%#alg{?R)l1QZ^_4{Jv6^K?Gm>WmY&%vT3XR>RZ?Drv|*3y zpS|Nt%bCFE{%<+Cly9e(Wb<1kDLIFk0cnp>@4YNsiOVxpEL2HKsV#ALKHDpQn(WGF z6x*Jsk zL&x}F8sT${gG2VbkkAg&*RnFHyLa!Z<@R1?aXlVYCC^=?@;&iPWcCkAgDci!e)fap z4;Ed1Ng~z*o*^0XVO%FpB_}Qx03yhl0F1oc&)gK*PI>cR)|Mv*Krc__w&)LU_KTty zF(aR|=Mt){_lxrN$;z=fYtOaO{wRK4rFkxUyM5n#1@&|r10Jq8pI99&S*HP9lHy7M zs?v$?r`gyj&PD7-hZ?c1>F@7XX7K+z-WGo?Mx1iz&d0vKyHG889B1!FWkC*T+b6~kRV3!u1! zhH^k1>&2;`dMs@(R+&gHe_K>ZByMhQm)Yu*72;az_J7B27S4V@O&}~67ni=FA@3hQ zuKO{I&|+H%e0J}j{{8wj(;1Ci$~&KIF9OH?x-WzcvWZmWyy&gK<`Ebe*u(dgA+vbp z-MouZ@>S;eDrPY!zW+gc&m$sIaug{aVIlj97JH=7@s!dv_&`NnUJ?m7cCxER#rY!FDNX`fcq*L zFXIouxeJ&UCus|sYL4;B)cm|{ntBdJQ<9=oO(jTRu?}};Qh%RAMXE_GkLWzKje*eVo*b?5Po)(-FT}PcG~NxD5X5poAXmY8Dm@* zB07wI5p9iFIowNXYHE^^kK8lUuiL{qplCv*Ig(3jAF_cekc*Kn$AKwq3Mu@H%9%%HCcU)c1-fEN)MK z{kj{xdMF^V*_1z&TRY&-A1i==>6HmL?3QoezWwMB1j1o)`LijK-;Cnky?aYbOK)!H zThdDOlD?o}pVj@kwfWu6T_7iaW@jZ)de7PnDZCbU?fg~jtyi(SI6X*Q3~oi=@bLBs z1#fUDq9rJ6*fcem6|T+Jki{eEO3@MzG3y(Z(TRjH`$eUDWZ867w0HcY6Jb<6tY>5< z0wNRbwqgQuRoc>W@Z!abbOPq#(hZr3`?y)oTaiFNL>UCk*?%9Rrhb@l=kowaL=}ja zkMp~w)#(znEmDL6#i2uoMnCswmi&9svxt9p_7!t*a6GU1t^BDaRh3b}VU&D=qwe!` zacaJpkPv;z+U%}q*9CPGbvEn{l?+XKW?|beSTf_rY?aPQ3exk)NT21o;drd%Tg`&( zMxzxpOFh6Bz;mzE)YI>7rTYLD>aDN*H4Idz@zz^>(ai7vqB4^NN@U`)Hd8gyTOwiF z{B{#|=f{EqR#c?2Cwr>^2{}zazxT`%R%Mc;Ow%f2D02EM`TP6Je(bx>JTv`DKl+a@ z%lCukPEX}&<{Agh_qc7WFVZrb{rVO}7C*$T)!cC_M2%Zq{4kh}bQsqiY8slY*hdc> z9b-W#!iDWl{aYH`8$5Z(G>@TVeSr~B?Fnx0SS~986Ee*!ZZ6G5gob(({Eky2mmpJX zYimESwTMq+nx|pZ3lytA)|`R@n9&?6c>mXxoJqz`IWjH7!GkAWx=(c%kRCD$T83DC zB;BO5w6s*_&idM9P5bhM7zOq>S7LR2eJlMo<)z;P14r#A+I45cx9?%1c5`!+nKK4% z`slov(Ge6Lx)8dG+b3NY$?t!7??Wqlds>P0`Cu>Ay7QAtm(bX|Z5nov;_maR)hPgPF6M6)>8@%bNKBqNwrqCor&`~=Q9HMr@R7i`ARXfKyWjA z<9f%7RZRs2KRzlerKG(AL-+u<7?<_al`gTkEydQ`HhF8I;SHdQ%gp`wr-d!_!MOWl zme%1&^_}AXmh?v*PF{=V3)jY(vR&Zk%gfM651krh7O`*c5=Tc29!_xlJKd`->bBNC zQee`ny_f0yk$-bB8+5goSDl4;c$7e0$s;_juC5h5Mb7C0Tb?ZZ4Se=&9~GkjpVP#b zFBfup9U~$l^nlq7Gk?Ch=yA!(=~JobeH$AuK+Xd`5^Fj$%k(dzqS9@&>BL<= zeygwlFw#k%?Ko$M53ieBi{7y>B_+kluIu>=E&00pBvP&Tt>EmhVx@?;vt)cpw=RZF??7DXC+NKNWZa(O& zW3PQbfBsB5Lcj^Djb8VSJIT9mBwytZw``1)R*uAvkDITeoY%Vjd-yma{^+q|idx*0 z4V|M+$#N5=|6W%+WN%B+>2%B(ySg&^;3K%V7KjqVcn z-hnyZa+Rpl@4&Ac^Y3ZL|F*NUI|!DMT4Ajun_+;BZ)IysL%M_(R2wSB&wuliDxFht zwryo6aaE{RInzIZW2vUx$6Gn0I<`_3{kmU3XfL(V!9txe4K-ohulz28Qyt}8|9H20 z|7a(}r?ALZzv~$+VmlkSY=&z0p>6L+&-e;*ZHgWSNV$nryz$RDWjKeE5PNQaFu#Zc z#pAa2fN0!Ml*&<|tR~udehm!`bQr}{01he@+&3s%&(th`HWX@X0~2s#2V20~Wu*AbU$BOo} z7qFA($tGhAq+H9Nop+;&!s+3$Mdj4N49$Ygte*8FIdQ)ZqCWv_l~-1R6Vc1c$`Tv5 zaUMtcYm(wCt3Q1gnZ#Y_2<#IV7niV4<6g>*Yr=-6ee z)ddR=yW3LEp7*7hF53S3whu*-R`6cWB`gQfjVGI8Db|CPRBHhHS7c>Nv504_2QH0jnSEE)cJ%_AIm*sX2@ypm zm`(AG$|>vFU$OwMBzC$eoErDnV$QUr>xps)MHjnm2=JVpo&Dadjq8$(j2^DmxG~PF zL!VaK%}pE)+9OppGY-PegRzukv|rEmb~T87eK<8(|4Y5auGUlCLIkIAIE_5h zk*4ImI2vsJ?5NIVS^8D)?Z$7hzmzLPyL)>podLx_>6gt_dHFI37uLAXBtixHGuI$d zR$3ZMr}|z;rKUYgA6sTGX{5V=mjDB}|3^T;(B77CD$d%CspL4S4{BTmZUsHvOd?>w z7K+_xoka6A{DEXQvC{>anA_%a2wHYus0v_CjL;y^4;DMtac&3z`T}H}t5L#U1Ya$3 z04ZREQcWT)t*q#+EzKBrec&uCEd0|~x`o)`#O3rhap}6!RH$oX^g+3s91sjSYre1X5gbVs()PyZHVW=A03rv;gcAyx`mF89bE&{f9~834t99F-Qd9mIqKH)GK`m3ro}P8@|oWD%h6; z5k}EGk-aD_0INIS^zq(V`lrX~7sXlC-fV z;@?9Y7yOzxXg|S(u=UjPOb>?0oxpM52^!#$qpZfOg-t56x-e-yT+d7*P-3*x7za^C zdU|>u%#RS-{1bvh17=8zw_%^?qHca`Xt;dg0tu)ucJ55y#DpLAg)R=n`TM^p@%l{6 zb{m?GIN-q>+-uH2-nLWS>PELg+;N{Exq4|8IT0up%Hugv!K zMIjHpTa4hC0Db~Vpqu*n`N^X#+x33Dxd@bgZmPE=)5d*eGQaoF*InFKHdxN;kkBV6 zw6(Q0Kig9gSk9OtdVt;J&X%-8C5A>uWsQydfUPb;k0-e9183)BCr)g0nCdRYid%2| za|PF7AYeP*um9ICm0vajs>42}6YW`q-SF@+eH^ij*tka|Bv>FyeZ{ZwSakBuk2Y1A zYn>g8*x8HsMf}PVA z089n*t`Bv_fk$rm2>Vd7t8Ks8G$<>Ts66V&wKCw#y?dx7QT`A|-7fAzo%d~T?77c} zkEk1FH=JVVuobk5p26p~o{0S!*YX6BA zdi|IUG>1`M6|$<*W%p29nxK?0NG$E|b)|y7$161pA7ErWow&HT*tU9@Lt)jX@w3Cd zoaL}z;)~x;XI>ca%)&%R zeEvF)|51Ma7Gt4vyu20RJo*Wueqy_5+!jX2iN4iJmpkJVG;)nu4~x4P|6$q{$dpyN zn<=vG_PwISMWOFCH3zOqtmlSQwmhQ1=Wu&{)~e4BSH~3mga=eK8iY+K?~}6ZGlF;G`S9$ zwWc-JOyho&({>jZ7vm^g=B`5(f|!#}+G+R%{@~-%?5OkBl2A%0Nk`A0Z|_Un~1Y@SC@c>0==5oUf>DgGsfvb z%7*VX2??eC%cct|skaot>ZG{u*xKgWUJI%DO<{B`%dk*X=L;!C0nOx`@D@Lfbb<4m zf2zFxXxw--W$KByH$&9Lnc!>UL9^%G);{(u02fle(V?JIB>eHBS(GnX5;asYa?K!0 zAkm8{0yC}qzHAz$XA>3eHWr>-Y$QlD>X}SaQc{u}Z#!)AEOf&nK6KhRKvv;p4Ks>> z7p?qem6_nSG%z~PYD@taMNMzk&k<6KuU8*y({_Iua5G+p${N76h|hifZ&=TnH1Fcz zbUV(r^xH;_b1$hS->IUul~+}bCv z#s)b_$;W(BPjFMAk(_hXN-{3l3rU#5NL%RsucueVQlQF@{D}1^oz6yUmRqLx%}j4` zOJ1MYe4!b2v9BY2B<#_tqi{4}|7I>;KND6XUFJ1%F~;Al101E3k&;{mtTsS zkL37+mVRtKgO4E=!Xg!ga(#Wh9^6)nl4ic#2cHmpQK4!D%@?^y{{ndJyOfJj9U5~ zrd>`PZf*?{A$4YdR5)S1JZmr-4ntGMO97$c*h>(6a8g>>aFzsK+Uw7g|j~O9&Ttbegc+Zw~mFJ$oxvx+_~6T;d_Wt>?)L# zX5(MPInY)6%oRY#^VPg#6-)G6btzVjx6YK%z^7%U+T`Z8b3ycqcfTueQcUcQ(=-~F z;aUHM^_H@-vR{19;VhB)e6X4nTg<5@DQ4n&!ge(k|3k0mbJDJ23hrovYGe{u?f}Lc-$-$i)swhswuOLd z;}+a&%(-wo;~h&zs%o=_9k4g?RK2~8f97Kgs} zJhQOIT=~#5L1cdxiDI>?17BSVUBJCfoKYESD|Ni-d=Ks$+3yXxq)a#HcL=RMd&HY&VQ?@tJ#jhfwbDHpF6^F36y|N}$ zcF)OA0|Uc^Z-gX7@6i4E$*^vzWj|a_joCQ3rf|DZG31;(_UcD9Krw=5`=em!e@ex{ zon9)cP#i4y_D&{N=MFMiBnex=3y!)d=M{D$`_kRJ3Ar1f3JOU@C>hQDmNli&J#IZMo&8Zf3LOxNfJkI~8o%U0t*R{WQ;LnZ z+0Ztk@~Zi?O%v1Mz!^5kI)}VY@vfh3rF~&_-}bX)?2qMRZvg13Wf-J>9wJEoSSX>r?=EO6YXN0Z?kcmRGtQ> z|DI_8jgG+xa;(($O*DM93~{dRXt8nCg?k`wCNlGzhE(SQCkXe-wvK(tvL2KPHi3RG z&tcJj!5E1anchQEJ+6yC_lIpj3sF9-?Y^E@%ICF(42A94!-u3zG;6HW=rEkES-YC1 zWbaMw)zo&G@d|1EF&E39nP~r`2Qrz*D$0rcv5S8Er10@CJXU=dOg^W0Q|ae2b`zIS#u@->5w>|-H*a|5 zt&OgpJH%ZA`a`zY&tVR~asDcL;sD`wxwT<81-*EocP_B7P&CEkTL1ac{9d;M7cNdj z2L-W;RncS|>*Se-BJE?xB}thD@q;4<#hhLte%(>fp|>@oaKH>#7YNqWnp_~UkU zoinM4m4PhWBuY4t9#InMO2cP}4$jd4aQ#Z*6*hy5tE<1;$=j~@#|>4j=h{wSZ_ z6>Y8NB~6w#jTttdeB~Ms)WHNpVuiB@g~z-zcL$iNl&vivK>>rO;Fi;g*q=eD;OW8B zw?ZCR^zUI74(|rLv@5TukR7Kt6R{f$na_#MTgvr%^5jse3j@sAO0QjqNQ8<;o8C1% zI=X}O(dn=0vw%Blh}Jboq9Zvrez~F?FCBdx5eRi=vkB$!5 zb%;v`N1KTDf$93d$k2r~1aa+b#4rMUXCasdn<%6^n^SeRbs#keC0+0%+XPI88SsTZ z$nTp-@OHmV6qPd&bPjUYX7;7RgTPnGeZ{c)}6u6xxneM1~s8F z`JW34bUU&QPv7H6v2#f+zW9ph&XIMOe`}5=R27hv9{*iZ80pIAM#rNdowpv?S{cA>b~XxIIzL$Kf=3MO-;bqutDcjj@-HNWCS|RX(37#r%UImk9)Wrs zXJ5MkkLDam29Rn`7;fSjoAqnPh=7iu(@*O%7yFKpV$9`q$};@Buwffq#egI z`u|qOxk(Q*Gr3`#-1@bi*B~S)*ft?#R>ju(pwhCpxTw(rK|use_T#IszyA>&40=UI zIy&}s&GVK8F3Z`r(=)TP+2;cYfZ-gF^gZ)V5^*-yrg(3<)#dEHZlx^<0YRKlBOg9? zE*TQZc#kzlBB0FNP0VTj0+z-KswyFGf+jSo9}t4l0k}zeh~?6S?qWE1hNH4R32D0u zpa&Zh)F*o$d&7y{uu!gL45LR8$n9vYGc5Goz<| zy#ISAa!&8UG7@I8*}Q|R@5;oq_!2?{?9ftlqfMkO+xL_cp|hze%li`lBOK8?FHDqz zl|bC&78^FpFd)(b5IG<>>%a|NqIKw^mS9ifn*5PF!@ zwX_gzEao5)s&!rAd-!;Glhh)g1ui{%#o>bNn1)GtCd^dAwYYotcs*i%OG^(R9*!V) zcR@ymg518*B&I8jI~+bma& z+cKi63(-;tUCv|1&Lr1s^Ct_@i7ywO2w|U|*H8x*ik1%6)|*J9v8uOSTnguHfL#@a2I1pFElWsDJXrLcG%zrbz#W23 zI^uGnj%R)ocM7VuKIA+iCrhjY$PQ7jNDqNgLPywCX^`rHJUG&vascrTPY6)N7J;9A z4oJ%b4*D2>ttrDWiO%(7b7)i)=laC&-#1Mi>9Zfq=#4(&IdX)A)H8xTM4hw(%k*WL zO(zOHkg(tX_a9V{CZXXoknfCZ?&l1d4zT z__xd)3!gDH;b~x#fUOQvG2>Khv~ z6_A*$OvEnlE7@4*e(1jCKXr7JEWP3YKz-h-kA!;uE_lz`e^seO5&`Z4AtrXrlAcHV zDx_!?eX5@9Y~ZyUYvy^S4OpJm`KSy*5a?lid=B`O0XyLvKGt(-^`FPo4z ztu!Qb4%5-oKl}FWif@h46qDjh?avCK(I?+%|091|eDzt-6#9WE!cz|*Xgx>xi|Bzw z^kE_WcK%Tgjvb`4P$YBQALOz9m-ZF#BQTPg-h^f`kNv+x`bO(~eo_884ilQnp`q%> z#KpBTw>;q=Fi_m?h2s%!vFm-g$aJ9x{Fp6Ei+gKBd)iRyvWyxI6U_rQ6GX3F58Xu! zjmKh!dcgy_KiTcVqCSEejj$M?WMk70-YDwL(}i<;ivO*cc|>j7&7-PX*V2TSJa+o@ zt_zL5Hf9{McXI64oX|LSvxsYsHS6uLVLmUXJY36B?^;P`rd@+1&erYQ3GJNds_%75 zH%FCHK@bWO9`&c=ksR)*QSM|8_nBjUg=h~HzD*=pzFNPo9*66G!l%K9yr65MA+>nh zEWelj)QxIy_a6sXVQ&#qHM}ZB%Nl28muQKo=-vQx!3p-uidyoZDmnCY_gz&joBpAsv(=Z?4T8t((PHjr&$l-`& zbnACbk5bizMjqre=ltM7A4se6wVtew~^~RHZ1NAm#%Iqy>%I_Pg=}8Yd;1zv69a>b$`hJr&rbAT}gy zY(S*0;JR&6^X=Pnw-3KWZi-w7(myNDQp?QDoH+bkzbZgB{siBnqNJdmOM)NU4oAf< zci^r2zWD(=z_hz<`f%rt9i<495h>xcJD+I@M^xZmk01Q2c-UX0`OAI>hu3nawL|@A z*{kRLSf#}b`&TVwql0z^J^)arGHi(E+;$m7y#m3MKyA0>ZHxlu&oefNiHga1^RfQT)I87IsonlCE1tDI~7`3cgl*I5ihElz-9lsQ(^@P3rXwTbnUu&+qy`ctJQ>D~=h5hh)SjnCx;KCtRMaQ% z2X)c=h&`TQwgAj4zZ!0)&pvJ;S|OBSACXR9GBi9KkDaDG><f8PMZ;5B7_NbUUL!W4kLf9TBc$xxwzlxFtACj(z1CHhHQfH9O8TNz`uko4mtf1LYRecf#?wX zl$4aLJ=}H}io$;rDOk4$YfD*oBvM1)K`7=^Oit1Iz#uLzPL^sJtO`u0eE{VBorc(t z7cOr3gOLFejUQJt-(lO49~X_r6r><9_`>K>OH@Ehmqc(j?S-mr_3!)Oh_e3CCMG7g zB8uyokfIlztL0@2BmJ1aK(UkIwA|Nj*9`PjazkjVf&UBfkx;hDMx{`fWYg_pp?=-D zmoS@TKi%U$F=0;3FJW1m7$ah3x6{nOF0me@z^yyxPnseWRV8_396b1ZoM8wOmB3Hn z2&9eVTS6=s7U|1gcz3-JZJ<+b)~79QGyTUMTn(EE#mYn`SwvWv2wKU62@%k>!jWRz zwtipk;xD`nqJ$|*bSb!C$t}eP(3xkJX7p1F9a;-RFI@Ckf;03UHpgor8yTaJ0YV&> z2%!~x-?ny<=EptF+z@+LVXMA}9f{1}k4@UFL3#2OZF*Yt(#zQ${r9ZXyfKj(B>*MV zehlkSfiLL+M3Nq&M8A6dy}k9n?6^6D{&4oAESpiC3F419LP!dIP%tU4A2PhUzUV$1G2w!6m)e=V4e?B}MGqq+DwI-fpyu1epTYNi< z%aJ-#y!z0VrTiOYkVYahg5dGb2?ZV?mv9Mp@rREaT44S8+dJhZq=jZXFkZ6`NWEhS zX&vhpTzrFoS*T8olYic*uaxaSkzzBXHn)D-|HH!_2iW?smZa-Hbft23zrYL!Hrghd zT#vsXduq#Jxlxx_Uw^h6_a<_7sZ0h@d0o@H`*$;`+(qCH4 z=H_O-YnyhBT=h4oj&%L-X0#&XXV=vO)Lp^$Z*)f@%`#UFEyvQp3uVQ)82L>T1Sd_E zX63G44TaFXd9y->>}Mp0INpx7rZW=hH4;1J8F;X}-n_XC*d}ogiI^m@-!H^)1_+^T zKIg=vkM@Dr?&|`XLPmK;u$WWhC*RshZJ)ACHt^Z|yP}*@X zfq1fpT$A_);=tYFAsUb#0?)}dJV10fEVR!F`5B+0+_yvjNEae^B(o!mlu%D)aDw&> zRbr!?DxHYc^{n$EBJvVlac#x$)pLW~TBC>Rs3#9cR1onnKGSAOk`)kwZ|me~3#@o+ z#FPed60jZZ&nRu0n0RyoWOVkuPc%cuw)E-8gnJN^Vc8)1nZ&2UbM?_1>!|!3l2+&`=OGKUqr%N^{uKx&66H9co z4*%Q$S6gwVmI#v@nU;J7{d8VPn|GIQt;!1n>y=;Z2CT|?)QBi{LQEHOUN6tlSZb2@ zwDTY1efU_0q^LiLV!)|UCJ0>v&7e4iqM1l*2?-S(PmYyZ?F=_laYGHs+RScK%dEO8 zl=_EVbaLOj&S^&_lMD$GgsHB(8a54ky@>$3D_EJ!kM`2@hQpDqNk7^ z&@OLe?^tAjB!cr#0|G+FOp!{mLw;`%IFit?Op?NZ9T{G8B;|I|`C&SdJa-N5Z3n$l zErk5bvSX3_gR1hq9vF*y4d4HYV^~6cgmTEL0^)0gC==f$pUQcG*VokE{zI-p6o2To zO|~MI7-Y&*bA*{pb>U)FPIhZhy@q*XXvH3;trr$$Okr;A=F%!;U)Nq8vNY@TwKp6 zAo2-vAu%Twg)kbnn^~CVLMde|Iv`ygFySGKdK3sK^W&Bm2U9Os9WqQpQD1 z&DZtc>f#_Er|}(lA{qMy865ewCvubHx?{o!4e4X%ift~tHLp2ItV{+smDmue+ZG;9 z?PkIGb_-%Wj#THdbtO0W#nKvwlM8<++}uOscWMP#XhJhCZWQv!}E*Ml*E*10JVG1N(`&TfWr5tN@u%tsKBd9SG`dHEI}!*b13Fz0nqv@X{K><;o{g z%v$}lIbHVx(rBjz9I8mSNy6xcEU)HkVLQ1o_PgI^f7dA1YpQ8jY$CED^U>+xZ#%s< z!!M$^epc!Un;UxoG)VcHE&%C+sWy)n6ky^l>1Nd^hrX0qjXHesxF=@e_o0*?aNk(Z zuePd&m+??00ldAc4l$$5HTGeZjV%%@bqO2T-MuTCKD>n zssEX|9E=7Qg_yDF0+gYr5TC~rBT!00RN>|O59LAe;vhyX6^ctBmjsjdu!tj0Vb=K1 zoIbUTfyho^@>qEV3&p6Ad}nyLHbTE;_G%^E{>yZG z!UV*GNUmNrXT0mZcN%%`=I7w&m}`Z`k-JsTxbDWx4>hK!qOV-NDiFzr%=<9!_^)5b zEqe;p6YgVf7Wv6bh%tI}K^#3pI6J?);xF%wy_HIpl(*nIMc8X%=@^IkXgm5sD-vq; zv%WqZjuOJVuN{ZGC0Vokujb!Vd2ZK}|jwWMz zN;NYThqSLanq|y%7uftIL$#9Qz7u#@%x_4Hzmu38GNY9F-vOjTUg6JX!Ut z`1Gx`=L>gGX-nts0nCOBLEPU5)th4Odqcw^$i!`{>$uH{>t@YwS+7Qmnq(`4ack4! z4Dv-(hQzsu1CpBbCR2&sfI6`Q>g;HNI3^hHy5U3)#0QF>ThcT$Ib$HJnWQ@vY251o zynP(8wZG)o67E_UZBmyr8Y$wEi`XqIFMj}DqtjwPLdP^hnFC0N#sp;LZ~-Sm7QlqV z1d%B!IPS<}pFx+EGt_x#xZ@_}&j&jnQtsJ9i-8WZyaPth`zI$ke`fBgy3`sF27Q%1 zaTJrvrQ__EN}ZRHE({mV{A#TyBja6fGmL5WO9*2HomQc~;pFHz+AM(q1COjR+!Z_W zChGs@c&Bmr68I3e8bHT$GO_Ib$olo`SKEw;tZBzOd;P4!56HW62)wAiOQAmsn{D6W z!MR+%d1!l6XQ#y?KX!*8H3a1{_`$EOly3I#hNB`*B~DxfTLn;mqw1yHOgX zu&aL{X_#x=xEW3#2GWR#IE=~<<{d{Br(rXu@G>-Mx&2!-g+xd= zR?v*TV5>J~>0jWN zu|iNde`{am=6rU ztnw=H+)R4nH69(H;R861MW=TN$Rgf9nTR;9uZ*1+yb#)(auvs1fjsmY%g*a z6tL{ZNRL6^V4y^+dV_iwN)!*yJ!zYmM7of{G(w~aDU1B^-!Q8DH)$~^pD4*@cTJ5K zcN6n^b|0{f`v10Rw=G90r<5al=!2-}W=u~ZbG8N6**3(iVIlRcEzhZyxDjM9!IhR< zD@}0owO9P(wY3qvu<9RQ^q`1Ep+l7L0&GDdaSh0H1SF)hz2=EC6{O zY`P{DK{D)uX05ej@s_!lRgII{`9b4wmBdiBhDKzyEdzJqpUoI6-GiJv zs9ir+h)(GKaUdJ&Lx%{qJAhOVv9fTnYOku^p+OvujR^S@b}=rH_^YIAF#({4gm8|{ zAqjg8K35|~u=h7G#6rF@eXCU@OlT({LZW}0Z&2>ueaYPX44$%+K)(-Ti+*k~wJ{Pb z&<@I=%@Ly`#JnoTXC*PHfXUE@IElxPA19WXi1}lge_(7Z%K3Rt4ll8Pk(H&zof?4- zLHOl3>tA8369Wct?4Dyv63>Rw1K6z)AI3J9!ed`B(Ex*r4j#{kd%c#~ZghiveGjoL zu~6xeey;-Xc#k;_BEV0Czwu#jWoSNwO-(2iLro=2WvHLlL$&ls^HL$uNH)IYk0)yM zmF<-%$9%shCNPQ76v8~m6oNncGhwJW%~PVM$f&5$5c5jV{&ILDLry^1AvPcJ%oF4> z6(ad|5Qz%n{{lIGt*fIVLJ^?41jS|&b3)quotj-XUm-er`v(g~qOofh+~0H|;A9X2$z|MhllL&>;#G;5kv;brAFMbl!yfYZ|flTO<`1L zgYpMIA5DBu%0Ih__C}qwPQV}h`W?C9p&>)tq;ObX`f#I`!0dytJ3!KjfnDOEB^a>= zqx8mUWj%Q^x!hv6*Ye8B0NPgfF=Ebzc))?-%{MyqMX2J$9z{2*LfnRU z+J`PNDAiLa1Q~}Y^Tez>wncl1y99=*{y_Z?`#AAIg<;Y`Sizn<_M{Ak5)zc?LJJuL zEzNDqgVHbW+re6nq*#KMxwjS(Jw@i~Ae`Pqq)Uj9?jF)N3G9U;{|pPKVVe%FSdNNV z7oE&wU_jG|{&XgT1)n(E_?ODns{vQ1-J~)BSYJ(AuwS}(uMc#p<*rx;vX=NjC5@s` zWlkingPYbJzVY<&K84QstuJ1@m?s+t`mtAEVV-OVP0U1IlgK*9J8GEGDD?+9g%&FO zbx~!fAtu7u79FgtsEM)uir=SX_ZmvUUGPFryBZrK+Up+2=Z9?%@Ft zd0p$SIx{F}PL6malrO>*;obMJM~G(+@i@EZC%8$=$c)vN>yC*uy(T7E8TJ{BwB(to zMy;l#C~a!}80uvb__nps&tHXXI6C49sQDBqA(GRK81x<@qTx{uA5v6hr?eNsA4we( z7S=$?u8FgmpP#2W#HIP8rtC1U6{XVIce1<#b?x$O!Oz09oT^8zti7oJ!g{InDbJ*S z1`US#!Q}d*H?hV|JeS1l{nTzI;BRVt_=4*31q^M` zyM}%+EIf0m)Q>r|dle7l>XCPYo7pK z`XNx*mY{0*q6tT?7ixU2j|h&4;DAoApWBV{t%t|Q(Ds%QO(87PiZLW5A~;ulC2F?u zGRqDm5_5j@3#48_KI3Y)&e-q1z9X2>GF4ipB5-FN{dv##S68F5R`-fnJwMLHbrqOX zK_TLtG%-Gj=R{EA1u(UA91l&A1{AG;5mgR^gdxV_Mwf|3$`hALw^T9SD`XvO%j8Cr z_RP%8G;T`rUjkc_zI7|~dTI-rfFg++u!Uld{ri@9!r4abq7n zy~%yy<5Koh-5l6v$KZx}BEXMvuI!Qata!o>lr}Up44MU)2`5q)Z*QiCVwChC2m>)t z_a6hIfA|lw7I`_}tgU_|dD-&+@b=&FShw#VIDSbYt5jB`kcLoJc1cD^MhMv>*&{0>log?}GPC!Vku4b^ zduB)W&Wi9oPU^nz_viO`{J#HvugCj->+Zf>*X#8<&+{11<9R#}(M!INn7#xqtpwm0 zMW&qODjU0eFrms2?SH1A~Ji;0J0Tj!U>qB^ih9c^s!|7`zD!L}m=I^zLAU z5V^$;sN25xk__u1ZKwf$1wHgkT!li98m{U-sIfcyS7eS7jH#L~lOO?QV z$fiC3Q=|a|Gk|9iqew zKcSzLblCByF9bxWu&hGn;UPYaqQx zY8A-JBe(^)H7L+Va!=ZT^ods7pYL+LtpgDRf{4iAzR>d7+1Yu)d3gy>O&tKT217kI z(F{cbh={OG-K{e|-Qw#+W zA7C}3?pGA4odmg~a0V|&e*7WqejKF7Bq55{K(KvLNeKZUv`7^Wwf-Uc1+AOjvZ}21QxmYx z$H9z|(bCeQ=X&_a5pSrD`aq20pJ@fv3%7&;B}A!iH%DxT*OAZcI-VKt`fwQwSvOdsPEDZ5b&65^^i%4mt!07@bqXc;8+Z?6?O3;9e z51}sF$)^Dmfzx?0m`PdNJhVGD8|>#iG4*K%g9>(=y*E31FG&>a06If~2IwhO{AL4G z{_*6AwgA+HZMQUnh1hwp+o}c!2Zu!LwNdB;V6-QY1pxJT2iw+Haz|KLPgF^XLR&|N zaz7 zx&*RvP6Wcj405G4cV9Nd062PU>tlL41MFI;_$6?vL#%fDx?rCSB!#=X+h!OXq84Kp zq<71>2X{!f^BbzD{W<4J!5`-0);6zK_l6=((_~{UT;kN@e$+gN@>NW%w!w3G?d>c& zV_Z;N3mw`8jHd-g0fhnatNTgBC?n*zD4{k<)?vD})rA)Ir$9uU}`13D+Pl?#F=LWvpl2%=IiDux4-@JhlC zoSBiS`c&zH9H<*%D#5`v+3V|s8SQ}ws0NXa<^&*L)5U&=b`)|5wgDz`g>{mfaYU@W zZL;(T?YX;pq&oWBT61>+CJ>e{s0XE(A@_5PDZoV>LE%eiE#x$i-PQo1KqpQPvJ{da zHRuF_7H<(?hEkvs+iK}99N zD)SG|(J_+P7$o6qfg=Y4{23B8)HsYXF2E;1^ak~^qbCG>Alhr!T8Bg!&_dD0(ygqn zA~~CDM%_#xiqQocw1t1Z2QJGb0+iPN_f_9KzRrR{*YTfMTY%pg?*DzCm)G2@{}2AI z305|G<<^COz7jn2yy1nx(7KiR2wwlRj<>$sV2xtkq*mH z4O&xpF&1)T1G)Rb zlS>{LBcwjK1=u&bz_CN@!zC~_bmz~1{`Vp`;3EG%d(uAebJ!3Cm4b%NIt%{6-^;pJ z694l_KVCuc2u{iJmy7Q;A!|g^Q6RB)_d+#5CCm-<^z_(Zp+&%eLFeaD@CYc+1;0cD z`T}31PILmQ|t*#<+FaDF^Gaq7Y{yM1_aLlUMB|_4R8_!FMyFY zU!PM(B9})tG>Xh=(a{CDGNonXzqiD0{`wYZ#qsWie9^^dTId`Pl6G|dLZ;MGuc*{h z|DT@?A+Uky7#Kt*ZGnO5C$cRHgl=FwVqE^DqnL9cjjVuE0ttkFXSD-dHtzc3us?Ku zh;QF`_L~CaX{sQ@CIHL|tT`42ay|?OT2d>4UMiFKNQ#@AJ8+Z#<-vtTod;TV0)%-W z$b*e;C`BIhWz(b7?$D?;#(i5rdFL}BO~vpZwT>jrKu`7 zI7zt|L`ik6fdmBNSuEf@%Y1~$+BpmjPTy0qzPfXALCA0jLe3%p61 zAn@FTWdC63GQ*=#Ops@^J}n6vQ7FV^t`gD0DxFG$PAO>^`E2>1H!+e6^OF)k$~e4_p|86h`-0)=S>eE@(ShQHNZ z_n9a8asR2Azvjck`v*&zD(+}*D%|PiO=3bqxA*VQQb{(rf)ByC0AnUnQjSdsFFrAm z4$yL7>H``KROUR}oP%X-c(u7{H$4d#K_!(`#tfX`nMy5_-~viO2Hy=tMMnAp`UPnT z(i(#_S+Y2&X)b ze`3iq!yxraBx+4f-TU-~vBZUds-7NlDAB+Eq6_d~p6#3cdr79gQZd+{a<@HQt-(Oz zi~4~sBV295->=XMIR0teMyg*38cM!c!7ha?4&qx^=wcLSYS9HMa9Ko z8Ckk#9X0-Y#L$tmvN%G_NIF+~YyI<<*#4F4zC-N#JfWCtVfP%u*Kg>Z8M~EB-Yu|i zA##t{jDV=-{>xqn#oJrIAkY@=Ffi*F@;q1sFNtFzGBK7{=r5kNy??5{L7}R;28;9e z;OGyP-K2r@2fxh742cg^F+G894Fzq~JM&^1Z_$`Du>=tTHn&ifTv5Li)F5~Hy#OpS1a2Ut zmbVsUd<8@vhD4$-VrD7(@Mll=9oqjHre*0s?oxjO?uox_wWzI`%6+~h%q=N|M(mMk zjRD@JR5&-Ni2;?#!0hZlIX!vNdpAtaUp8NOc7&zwrl_c&_1=lHKc_u|LD%V}RXJx|IcD~`jsZjMPk4jp#Z zJIbm>8%1}v)rvMKvNCV_B%JK$f721d%fk}`<&b|E9pl2KTT@;>1nbBgb4IUqS+QzRl{o7%}y08 z8lGq-&l&T~hf5QLq6GBvdA)pD1(hsBi-U8G64Tytr;>YHkfe;IWtF4a4hc0iAwWWa z3~53Y3V2c7;w2znI$aA)0$Pa3ra2*!E*mpK@Gms-AT@cV2{@4O4WqtXL#@A@purD1 zZPk#36(rY7%geU_8OsKu!U^?*s5UpDNtGcni`SjR$A5H#eIx(0B>v;z=M*`c6p2~( zx%wo@Qvko111OI~#zAN63ut(u2Tf{E^^KS4$1gDwzq z6{Ktg^vjn({U3z`frOp_j#C+syuE*=*FPs*z7hplA0BN}`3L>s)(K!&Ld^!Lp@Hf@ z2rcqLA=Di@kkh*Xz(hff z(YnA_-IDMa&E%=+B12m5Qh!(Mke!2O@XmDQlYsKnfv`@3BN!`-%_S{Vf`Afj$g}rQ6M@zi z)rx1R-~z=urrnxcw)<*Xg8f(<*GuGH)puuqK865 zlicy(+C+JaY=PLRUK*{l#y*!yHu**Vz8n4QmM8f}Z#;-niuy-1{LNK6ALi)F(%Y2e zB6J?k7712)1@3c2Y*FJ1|Kf0kKj0UE$@~y4)V3!`!p?MsT`gX* z2L)z`#N42~tg;oE=m_Py;O^Y6;D`u9j6MDS7!%0(b86*|GYdIp+^3*B`HHzZh7#v7 zBqlBr3D3crL4q2S@x!2&Oe^}gNL*h9WT_A^$fQ~K1ViwFj-<8U6Hpv-grYX&O>U5E zfuNCmhKfRA{3ml|&WbC))lp;*3L%$~q!bk44q*Z%5;|#Ep^|ZV2b%G5qBRSAkKNc5 zT%-W!a2TTsW&Y1SjVh4DgX-5QeD}UBiGra2-=HepV7iKr;oF`wZsut9Gp+8+cX_Ia zseh=u$UCg-BtG4ZpAeV=9L>yB{<)*Kaq5tV^n7` zx6F3@6#6}|ivk!lFDh)UtU3*Z`v84__>%}QUqA?P|0oH9BsK)a=3o@ZCxU;gSj}`E zh2*{(tN@5?A?!K=gzKQ3S_{fa)}n(b)yJTkFQh#^kE6ijF|d#$A43x4_bS5CxMC?< z_^ENpasQ9Esj3>gzDOj{B-B3(BpigHvMmc@<)|!~C7=zK{764NU^bpm^`-n8FnpcJ$ zxosMle!I1t>av{x|6v-ItrtJ8FxrenqyTV52JX zLu2$h@mH1ViSFU%uV0bf1Gg9uU@GA281v*!qgL6vmw9-h*yHBm)_lxvSz?$8oJaOP z8f1PTbf5Xb5b%X?!fa*})Ez&eW-Gut&!9vWD%lte2>Z2(&w`S8v}3*)JWZ6f4Gu=m z)`ZCZZ}6KOkHVR#R9$wuiIl^Zk)<+xQ(u&=KdsPWhVy)}KZmDSW-i zp=F+Ds8{~;2nO{Dg28DwVgmX8_Rbi0Xt+t1Po+DHo(-^mAP5!&;|qdbE>I?!wO*eF z9tdWn{tdmzjXx0yk@80oe@eoMfTRx+H-vv+{p$l6s@J`~YvKCgqn2otjHs$o169ET z>CA&4o`SaOG9N|lle})h#(RDY{lkjL$^M*>f)ayd#YrJe=$wTfYws#2)FKS3mvm>Z zz|nz9FoLU)y*YpJq7Edf_VhYHU3~#L_C5m#su4sJ)}Xow`6cN`!hJ6%ztvomFq(9` zF#T+P{nUI95{zqKq0JZ8T^L|SG+PBwqEXcF4ISn%MCmyY1E%LYfC7S8 zMYlZ&ZvlmW1S}|ExG{hSJo4v)5FO&^V+a<7ko>~kYwt`!5gNKqE{^q)|p0S)F^Tiy66)THC`p?5(8K05_ksJGK z{>flS)uQBl*PMrIo;z_#MDzso(tv$aL_Q1n3E92db)eb6&rc1W6oUaPfV2#to{p>? zbf3KdX*UFmKBfUah!{w-FF`*PHJ=S3F!BTq4GnV|<%1!nW1<*7A$?UqJ-)W?QPmBl zYvlO+h3?u*q`D*)Ij0R827$HL7`fiwug@%jS@gTN%nT0qwm@q19EiUS%qK2J*T z?;eJeA+eP+V>!L%jKl`sj%gEW>cP*?K)593eH7eSj+T30SB|Z65#@d{`R~NFf37 zX9XrHY%@(5P_zjl95YJ?`b(IAgfJ?28_P0};psC?3L)UtAWwqw2R%n;@qPD46O79< zTX|%@j?;tL_++olj!KiS%jOC{dv%Dnwxu)5O+bkhekh4L^!7ZbojzUW17UZfx4~zX-g4(!ao#^T4%;A(`1mPed z&KCv@4iY~F4~vBL(Ha82CkC-PQ-dw6A(T&%B0+k)YaG*(mBr`wmnP=BLpr}tpYbsv z|NbWFcOh0g*nQfsX`_9_F^_5H?wg7hA0)A)3So4|sH6w0XHMo^5XTr-VDi=Fv83fF&Xmzp3a%J62BWDPR;H!G4q#T7uR zs!C5b%kZ7Aj#}6R1dNjGE@c2SKdVM)#TgiRX18Rf+4lD}hVqHo{&N z{U^EHiTB;QfHKRV^DBim)?WJ?G(5jN^gCZSx;cK0LIJwo79#abV{KS8r~+IR6SAk-B|qaI`QU6-Yw0Pi-Aucf zC)QROKd?p0PhS-ss_YVO3{fzknDTqNV|3%qJjpAT`8zQ(NW}uTS6h6+l z<phn=q$$_dm4B+@a((1kQYRSwyzdtB?8I#-}Yb zREEB_A1BYDr+TWp%G~O;WX15N(7Aqkrd8c2Yf1|kVv^j2Ls|l40|1c<7|x$R2fD0K zaBVLDH5aM}nX0!Sor8IH-c3z+p{l;`7$D|HDmZ});(zd#l%n^+zMOSBIiri*{0$B4 zVVNRcS1;HQ)>I}cjLzuVd)#4>65lsV!ong!0a>3KbXb)fm&n?mtcD1!T3UQ=TKe|x z+s6w75o`SyAS2;{p*C^<7fMXRt^+5&4C2G^N#}TYD3Krw6wij7_nbe|6Ar8_gP_+FH(*IVkBtr*SU%?XlFSC6$1u2@;<; z`97WV(I5Q!PKX5a(sroBrp+P|zg_z9y^-;A)o#REb~*ZIHXXRbn}A&t1J6XoZBu)p zRg~M7omlxP%&btsq1`oIvSi+4|EY}4XWlCdWbW0xl9Juk0*?ZNMQjYPj%t98mZ8DD zp9s8p=Nzn|CKSBOO-EUbzR0dZJDPJw&AZ54CXf>Q^^5_ zBrJ?>nOI%=6FFbjn7y_~6-V_$`RZAxS^CcvQ}ajFGw&r+$z*HDyw*E;g8wp19)gA$ zJgLh*M$K>66aqJ1%{B*{I7sdMuMqVqB!izFF>wP+io9`f)gV;Zc?^xba z*Md>61#8vFdTm$f@`8)Ulw4u1ujINe)M5Fcq8zsR6WlnW!n~LU9333LDbV@X~c#P71FcpT_8m@w|O|BE#OXo3pyt zwb~C*Go&MgKve*BK0z><5+aJ{3djdG2F7w2&C;FNH`%hRx<68~f6mYgM@iR422#gO zKQE17%~i&DaAY37X?u|>`1PM%5hW}UR2eM!MakwnmK^C;WU3mqo?WWKxhRt}ho#Az z2@Q?rDz)@cmI0(Y)Sw-~ZE*I_AG^yxXkQGyNz!cn`POB)h6$MTvHh$Wqq*DH=2C$L>9KYq3?Y`Q_kGWV+aY z>A2aNnG3H%uez2QJbMO+I)MY8KD1Z?SS=cW)xLL8D$ETUbtU#7t^a>TZ5wcCUAB{4 zE1a=dc8HTL_Xa0M4slT8e26?EZ$?z#P)fW^cQQd@;*>l}wsgcs(zSG~0zHq5Jrv4N9N>|ke8g7YV^sLZiGF&kKf!IX zV^Q#HzpFOt)m5`!juaLWuT_^|pN^J#SK%uuTATeNnGi;qRYO185&%16s0RXino)WJ zF)JeIfBKUag$I98T8f0(ik3J3b66c`$r!n?iJhC=eovTI_bYs?{FM*?lj)hq?aF?S zA-_5sT-AV>2#_$)L*-m)+vt%KOt7pe9|Amb>RbL5o5kyvYRsC`Ih@lH3C|uwCJW&M zRB<28GWFg2Q)I^b_r⋘ko`?R`%k5?}l9{7b8b?Z}L(|jJ}%2D?@dhuc`Kc8`BE> zplvsKsOpm)MH2cZ9835fuIU^{sal%Ap|vBJxZvPl>s|mDA{Q=%R}utl0+fdJ7Mi^u zY}g;_EH67K-B?@bU^=pqx1l+eQ(@zRJG3y2aXO77A^AYA56bO|+6Mm4I#6Z42K^bZ zYAw7tsgEt$N3tIs9zWKs9H{#22^aiXzQvg3r|$)ZWsru~gpe4ZJr6X(7nL_4-vKZ? z$5wXV2zU#>N%|Bud%4%@24x}5BPuEKk!d#)DePeljX-BVMuPfY~}?g zROIBH+leJ#7sV^+Y@aJbX-P1_@m|0V%2rNhupfzO35sQh{X@^v|@4(1|R$zN-pNYE!U>2Io z3%Y3q&2Q(I1!(_aM`)?#z3=}Lib`kKBa1y>sV1~pMH)W%^w}?FYVcb2z;zc`G*UX@ zswS#no1s>(n-8-!C{6t@r2v2t2c^<=7LOYAMWDXrK)z2_N$wmUPdjeGbmS&YsrxGR z&N{(Waw!qfcM^s)!lRO9r(lGT7%1+%$0vlY3%1cX;7_j_FgT574QL3=7-} z%poE`@>ziK=`4S*0fgTb7!yFR#GJBXD3PG`cHBq}OVgm>i*3WK2?-BSfWZqGW1gI5a05gRCEFI{gx+8tn=cE(@ zw`4^U0V^E(Red zP~SjCl5tGl&{ZmFF=nzy_Fcu#F!f2Ii@21DHkmJPo2^wk+jnYB`dYfMo6GcVY+Pj+ zzFCxS-Jl(VUga3%D@K+^b;0V78tQ&y$He+aB)j)8j$F z>IYE)AX4A}nt@s+KxV^WWfPeG7g0X|)WcB`NLpa+6pZ08-~Ij6@z;-#cbrCKNV^6; zM-wJt0G$P?VULW+n0IuWeqg-_%76%WcXZ?j$^k}9ObpHYKtfrdj%d~VH3BmBF?i}o znG-1L9^j;P0DwTFAy6j?RIOAZ_Hk2LS%HS8nyrel?DfxT1Jdl}cO}XMrSDErc#trs zRd={+amu>atmFNI88lp?Jsg{Pam+`p$DEnAS~F~vIdycce2wJ$qUl(>tax_9OJjZI zo96_|A8+GvzJ8XaTPILKR`i78HL2G52wP6gZwuOISAd}D0bN089xb3`Q^pQ3cNZwG ztl$b@FaZow=pM%l0EZ9_y9AW?k3;ue6U{b=NZ3IGA|#NgQvh1=)bzK@9B_K_04YL4 zV>&fgV6+37(a|7C95~E|sDXnItXV1yn)-~WVW_tl&En1H9tO6?cVGs=bH%}+t|laY z2)*Hcpumj};Onh7jr+BDUGL+E^vBwljI5Y;OP4TD8F$vSuqtn{+}^9b^{$om)V$G_ zWL(|LWNN^fFa=`o{;;tar4exVGxC82c;KpewOPg5{vE1laqF{GA+ORQ;H|f5A9Jm< zowu4halQD|3!v^+c|8A~PO=Spv`U>yhY}AY2r7%LF+)RpFu|^Mbb}y_;8?m zgGT(0yGxwTttT3zvm;|;^gH+RKHnMTaCh4MH3{duTQW&51%m+@>2C#~>>Zz!GzP|6 z7#grw*N+ksc5Xuxp+v&&35DP-m|!q6HFfF7?Bt{;bXI8s_K*6VJ85G2)Qg@$UK+Ws z?zk+1Xb$@QFp=s)zF6tcX1+&_vHt^FK+~!NxoZ^WU)VzLfvO7eGv3qwJ6uE+Vrz#6 zmfo{eJw7`_bdKui(OP|SW&))iLLFRUwq*I!chAYF*Cq#jsM1B;A!v|n1->^@>42iA z2wXKxkBoMpCOCZ;N_5csa{Gk=A?G^;|6y7|05Ok<_1lkYu=jv=JH2BH5HxQ`>elAw zISvl_MJ4bR*cd>3CHtE=-yx-htM8%H1qMo^u|Oce2OW8Acg;>PFX2`s#yBq zlz;kK??UkJ-Y8d>7kNLXeeNqV!vSJM$Tx66DN&mfs5l;mX_Wx_j-s;eYqF3i zqLXTYr>%q8vx&a+66PZcFWQ2U>M^*++U*iUxg0mGtNrp*M~NXYpMfzz2IaO#FOH)8 zf;3cu*_{9SnegU|qjqecyiitHbj`Vb0|C?<10hE7P-t%^JiR2pG+1|>-4pvE=x`EQ)SR#58H zj2#@2c^7l!BoyB;Fm6ijCDmOHcDHvrQ`{a6k20&EK2f8o{_-VGq#cyEjoWx~+TxA> zPd>$;qWpAoUY=UU}Yr%g|D9N zvPX*2XU}Gyxrd;of`BYJot-ZwN=17YXdX>Wo7h^v)lmrvd)HRBesa)HRyJtWtW!9l zv5V?4TdwX?4jO}>wI@u<%gceZiJE=9L0N?!+%#lmN0!Vkdd4=%Y2Aa$GzJ4=EgvqG zsj?K#EK7a|Fvh;3#A*9Fq7agi)%J1iTUa{sg@jRG-^n(u-Dge1PrqtT5`83`f{ZFH zDZg&Fr&vc~GR%1OTBKy3Y=61F^0>3%;Q#oBUoz&rcfLOJWFhw2gjE|$ZdLvyZN z+nkzwqBxK^%CF};HJ5LbBfbTZADOs>#32lz=hzrHXooRq#%GtG02EU|S`rDS$W^~8 z0M|GDy8;R&_7!;{kmVrO%~l2A^pLhA+8~g`dvkQrQ?Wh-Dc28`E0{jSpq2E|Af|S5-w#=d{+By!$^Etd8@b-&fP0JsrgL3EyQvA(*EcIQ2h?tTS0Hf8zvm0P8t*pMnvEza@d1D(Q$|_ zk-8mHe}qxVKzx(z)I5ZNs?;YSwl0CEMKafDW)>PX3kgH%_Cg7-0W_500R%|H?*(2B z$^<$P6WJJC`vt`7Y?w8Q*w0Y>Lh{xyhny5OG@4eCc-jTaJKSTvoy%x<9(Rs|V(yj`u zt?T&XO)=6Gq1QOF=meI5(PF)urA)92+xo(Wq?D99U|yb{xX)j_$at7MK*;@@_2f!; z$=3@C|4FN3p35-?VIbm#Q6)g^I*n8oKnfTeS_>GOK}FD^zM%m=02^*zLR~P|8$tuj zh=Wjq-#_e8sR#CcC!E!v?gOYH!2qCUXAy{AC#W0|4H}tmuppQSOtM#m>K5i&k?>)K zyI~#@PZpm-b2&iqfPn$A#BfjuQqnga%;7KNhFqREQyKeb-+_uV=FS;OV-rLW^YW?$ zI6-27MmE*@wZwTu$N@EY-O0ByBDDvE9I*0=#WYt|a=2&66sdh~MY$$uq}hGcw0Un2 zXbpuQu_a$%t4)KXS{?^WK3RdT?61-9I9>0$Wa{3#|8BCZXEF1*yJi-bVQ%IAnB&ll z;{9`}`5zkZkfQ~o%Ppr8$2Ts1{c5m>hW3F0w^A_Hlza!SP8e>qS~A!8D_pO771dIg zq;9N@7L4=WAyx0gAHLC#OyMJJCR8!o7=Q65+WErrcc8oqNYc_+mGcrepN3gUk`MhO zn=JPfHN#JXq0XP2-!2bcd~r~-m{!)!#FhW2W-;v>g=eI(bb6EYH5tM9qeCoFuYY+6 zP0aXu50YnC^%;Hk0Wmi}GheoUJYZSV#y5u(|IM49o|sjE!fj*-ji!PX)4jUH(^8X@ z3vM}3J2Bpt0r<5)+l{m(0D>T<3jtVF83gLs<*f$(E@qcTW=s}O>;sarl<{%|#y1?! z4ljK4+pOcU?+2{ZuHg11^%-UIU7BV91%fz%as4ACjq7-4=TO1{z4nSXfFa*b+UD#o zme{=u)#p(Asd*M9Mpa)^KXrYn@?bb0dy%G}9vOeHdcyyc7HOfh=*F6C+zH(H)(QaB zz2;1z@92OMyU(_>0Y+AiGmVyl72&$NxZAh-x3SHBmV^e1L;yU-KK06yY6K=T>9K=v zNal#}m#qH%Pew5#U&{w@)nv+EItcA|G>y13>>+;kR)>nG^Sxj!khP56;T{PQW6R6Ln7Iy^Hqodly(==Dky(z zl8|^>!`ruB<=_4Q@2f1#xHr%E*6oISWNDRQxPB)|Ui;>0k6jO?K$s4oRGXbiS^Kv& zL-&5qNj4~a9SLt4@}!g2<51hM$9mVxdTMkn4HBumf6#R=rhwS`x|Ng1vVUo_$s6v> zE0jgD#@**!%~(A4@zp$)?rA>rzs07;2Prw*mWLEY>b{SkslM-Z$fYoi9wDxIXM{); z`>N5}IzXH^WVZ%Opl=3>$ zbU+$iCp#U{etVDfrG45rlrGC9fWA$ z5&jx_8e}1q2EKk@iIO~C(kHrqa5Cs3VrGBv@~i%$j(Z|uS88B_lA0@ zOkYP0EbaXUNdbDc&(I12%?B_M5L!(291i!rN@NydFG=yl4cS9BFxBhcz%#dBP;>$& zZ`C}2nBOy=t;uq=HWuK-;d?^JoC-MR>%hRn5DtMZJEE2Wwgsu!Sm(@h+BZL**5ZGC zOfL1}C9tDH$ScmH$6!3FD0k$Uub;?x=yrV*XF0z{2yYN;jLs!{&g4U4o-5mP;=+1x zWE+xJhW<2^#Ck8<3mqx%wI;&NWS|@vO7*xpbf)*XN%e0NRkqY*v}z478WWmmD%D8z~$PgqUuM)_}32gY!P_Cffo_quNKFsdw;-!F! zRjiFm5|QK`Vd)dlQ6f#6^0%XeRMJu9dU?MC_8a|$Tg0LFGMH?gQCp-kLMXccLf7OKVa^CPL&(l4T0zN$F)(Y?XbcE z+)&oA%v-O?FIW-QERl7Mj(*Z-r(<+7_P^xJDCBUspALS*i!r#)+mY#p6~Pt%+H((5 zy8CHHemw6ho7FN7*KB41$1jT$gbz3t?20O&+6;-%55wJZ6S?Y93m_`jMY@FhIRmz23~8}$9_ zOXg&8gZS0vWQ6S7v;TIkO&9&VMgUQh_SiJH?)W9_ifv#6{j9Vf_b@l6J8Q+P36 zbUW@<{N@!ZR^+=8zA*5(p5?){)h*^L@x%7ZO-5(rKK5VHas<>x!EQi@;I{l3(`H4} zOiaU`WbnZ@6?a1vsg2ZMM4M~U+fRXkI#Qiv-AhC(2tup z*BAJ2G;9s*?Kxc779|s^r?Gk7P%hbaE?U#aTV6z$M<8gOI)uhpGCyhOa>3-oe3pcy zg6?k}_;`lQEY?_u56kNk91Zt~oI6jeOY6YKzOi@=Ur7Iu>)nM=arKtEYBBaFP7_f^ zyyUiQ?>{cIOnz-Tlra#sXqnRbWUI#AXm^w2D|J-7*9hh$`(X@vJ+2s7;T70krgbV> zjBniJJUY1AyJ|IfE!y)-D3(T?D>jRsu5M9jMST3{0bxz=h6V}gBo`JJ-WSTrobO!W z2XAUOVRdT=!N2{-QtrC7N!@g&dd@u}lGn-i|ZCXZv&6@75o4ws8!}6+WJ4PJxANzXzCAShl{CBd z&5wVFIZsewjXA@QYVeB6&)SXl4@P68Q)VVTPd;s!-@`^*GBIX9sR zkE29m;Sa-~qxacV7vILd#@rlar+0?Xw2K_pwQq@E2@x1E^fs%#Sej$8j z7^D0kwR-^u(K^kpa*DpJ1S992^hCTAVtEh|jARMzq3?`pt z@#^KvtXw4TVsmrbJ1$2PL^o?5nIU`>)2(?#BO`C_)9JF!r*F!cEKCPp$G>!|z;vaq`pZ2-jOqrpp84O)hD-|2zvNlK`nF>V26`?wJ>)K|*z)bS}9Ey@T$VbRiy9W2*inv3JibC5ufz7U-v?z5B#x z=a3QVdxRKW?9w?{lPTZ1>s)3Ld0E*iW6$zsnQh4PPVI5zn-{%fG{H#zRWfmy}?Q3TxS%lZ~NTm=dafou88q3=gTP+!v2L% zUG6i3gTD2cN{&DabDg;F(XIiDlwu=Y@7U@?gO+#Ef5Tv;bNL9%!qwu*yU!}hD!jPQ zU)d77U7I*P8org{VD}o%?10ao+kG8YOXifls9WNvm4D~z5T8&-rcX#$LsaJe2UTuR zdc-h$N>5L7;#z0N*Zz3d?{Q&9jO&<;%Eq;TJZxU#KWB#a7wMXzO^(KFkU521R`0bi zuKnK^Ov=v6mXLO`lIiz*Z$tUMHLHXs{JK|$PTS=_JB>!JN7uH+BFw~;XaD(QT$08k zGduETiSvt9rlZ|V?Tzh=O{dOY@o^_mZsmdB3V{wr)gcWzIc-zAYa)7j2}$<_$ps^; zm^x24!H+Z49x-a3jf0lywPB1wfgB}2l&0*tllHeYe@bbActyXR@DB51Z7HK)bYhzzEnnLqbEyA*`i1anle(Sm*7Ivpl9v6;t=1L?X6s!{%|dH<3H}GG zYLcKHUX6hh<7QnI&T)miW3}bpgh!X)t0r3yPVJfaoSMvI8`|spxys4HI z#J93q*mOvtyYK^uuS*0@_(^q%ZA%+=#N2=IoB#fi8iU7{x(;&tyAd;G&L=HDmBPu> z(41vS`JV04y_=!InOFk!O;UtQY#iq3V_m3v&v+D2vvhcyJpJal8Aw>YR&>re&6FxE z8SFmCc^-+k98TuXCrWeGw(-;Ye>>iOOP_DGzVx&I%z+)T3$IS#+*0Ka;7KEA$h`2vZGM!!WBl2rC24q z*A1F+y%!&5LL4%dPBxvm8WqL`CUCo^!O+A>hz?G0R@2#p zGWRZ(9A@do#Vh4gMR5ARQiOqtZ|C6PdHaQE|EZ5yr&6h`womA6&32zOv}Dz2r>DE) zlepS57$3Q7p<`8Vtz_8;mDTTwajB_h7T%sX zThrTI>s>o19j#+v7Qy{*u8}% z?aF`o=E5Vq{Tpe$0Zv!?R*<4`O9$>Y%fN;8>_a;lVXWt6pVTE7DoJ#T^nW}J5vG6O zESR#OC+%yD3B#w~M%NS)%7ZC8seZJT=g%o3=T_*5=d%4_)aB<$`hcr|H&fdG+E=QO zL}mJbF5e8P{qJ+}Pcg|gXrt=e--|=WPaaz<#whBO8a708Ge-)jj^5h+rlFLP)IC1s zqm!g|V_~a&a1G)$B5}W&$~o8Omj~uoElP``p((8sj)cmSk$9Ckk4B4Uiav?NGbz8| zOq82#_+;`QJjf-|e5vcmC9aby78?9U{?UG#VC^MQM;JB(OC6!aRAOCX(yYgBPE#o6 z!QanUR11(=)Fq$JhmE2n*D+H>LvpLw_HOFJnSfZg`s06tOY^J$dc7yf@uHrpP4CZ( zxKDwlZ#`4}RyV)B20vjQxOozkFG0gPib#<&~ihnc7#6~S`WBHjfQcRhdd&dma{v47O+!z z2~OVc(e3f5p5OjH&|=tD;d@U;>3EtcW}!CyJp5;udv(_cP57RvWSO*d6knCpv!!6E z`}L4H%tozr!dJ56-4JW8Qg0F|hTi~MgUAPO*sA;a`)>$dzeccl75S!vYmN~pXUbH} z8(iZTf0*A(#EB!EZ05+trRSdc>dCj7KNoBeVPM?Grz`?~URtct@Zp(|gHIwJg+KJe9?^!VnPEL*UZqzD{}F_LWQIiWqqL=M$_vV5cJ#SZyaW03;!5W< z*s;&JIxC~Ua+LEKjFA#c{a`~9CIajH002>XYA zPQ9t>&UHpGbP1o`ide5;kdF%KeNtfbea9WXU(bXjlBCE^QoT7iPQ`Sd)vXQ=KC1)8 zV1o?J8(AlPrJpofV7L!#tV>U7rdiu?9{$?5b^gdpSw_~Xa`IjaR3WJ)VhNY7^Vg%T ziC(4tS`=HWcRog`Kat_N*itlKbl@A~`~7gyf}r;-@#1=?=4(HU8tQ)-UrQ#IHfBoQ zdK>Zv13CMM-|6WE2OpJPk*y+~@uiKt?q10&|HTfqo9_msr89Vvo!q-jZY;bWzXPTv zf>hd-Sok1l&T8$|?mQw`nIWZmr2ZUzJB0I0dsbypJDabE zh9AzcMbVs%L8hV>OF%bA>4V`YH?uwcLn_M#7Kgciz2O>L>zhq-w5XMF_>43AS(eEI zSH{_gDrdOP^T?sAuWc2^!QBk~yB9Zj)%xRyoh;=cyh^Dt<*o^c6Nxp1<+n?DMMURteWOIp=_TFVyT5mk!si_bwAYbk;(={nQ zwY=S6<>(%p@s#M;m@W&0!P}g+#$n|GY|3p~u791)$e_v`F-kclxv1Mx!XTA=U7;`> z+WK%WwC%wgS`)KMFFazIJ(`%e@td2brj6u+{Rh!*^leIywi&qy@?53(FVhxbd@NF2 ztBV$GZ=()wMJ-Z`LS;km<+HMNNF954YBI>C`{@JnA2(naMBO1o!_159ue=L!TdQ%; zHYlHIJ4CIMD-`mwTws#fcy$l0yu{0+oPSg8;p{7gLefrwap}JqLzdC%ZU$Qsz4E`t z#fmXK_$*8zudI4fQ2)=>?gb$Qt`89I2^$7Z_)2$Vyf_Kzxcynh{i}1yoV#P-8>Q?Z z16xJ4)-yqPI;o?Bvq0K5fE~M)#q8j82&KpUN1-v7~5TzLBG);@BuCRl=f9M$q4SC=%zu+hnRdH3=9RP>5FGTTzhj zYMm*9ccKTECwk!u+xg1B$Iwv>@}=FPAMbWoejVlPqrSJ?yL65@tX%^himc0iA~tpB zV3~{PAXczdPe8E(9G=vgh3{;~#T9)x*1{lk%M7`&I@{cJ?qEeS9_RLw(hvhN!8ko=}a&bMiBjf ze2KNheQ+!z9@r$xW-T(crba$x6zJ%jDcZFFS9(K=-SgsQI_`g04FF?ZCQNtv@8s znbZNXr^~Txgtz(b$v*}wm%%4P7hY}7(>!e0#@th51h&9;{!F#-L53hm6_IJuzIcNt z6|$F3ZQCeYac}%Ke^Z)h@F!nd8XO6VU254^uC}~MNVvx)moTL+cP(vOg2Q~XRAX4ZKD62z{|r5=Dy(NT z^~bNh(Yhm2lAz`sOkA3ww2hOtw}m6#^YyK$^8<;~m#s$$>K>0QS=`*LItIdsY&bYL zkrj);w#UJiE^mBfI|Xnf^PstKl3_$)GSN!m4Btun*_-TJjBIWh25mSTU3b#RB|LkI z{QAS=%hH;6xipJ8W=St!`ZPdYc9@77!r9p@+hPEurJ>Qa6B>bG@*l15*_%Vyq2S6Y z!#Vn?ksVVsZtQ3t@K&r7IsD70yIvQZaOyJ)wV$_7>|aV{7cA^-Ia60w&za8U`_}g? z(nPk_>vfsK*aTn7(!v8;a!EXG&_|fUU1BnJEQ^QlS=F{$?#mZP{O+X|6)C+vMNU3B ziZG86@r!9x9}LCH)rXa%S5zji+-Mf}_iz2ZTxPYME$oi7%zEHqMkI6~Nw^4TAk1{Lgj_1}Tr=+w2jVlWn*R!xTD?fjBKw}(-!@98m4En<- zBs6|Y;QDyrmxXJpegsQ-(2;gDjEhBz zz9Uv|TKnvuh5I&pV1MsWSxTt=ANyM>*tXCO*iuBbQG6`Z<)`ruPgX2B%!{f({LLD; zd+KXrKUY=)-@J(dVy(Cw45bMR3kwVlO@Zz;7^fl!5MXUX%XJa9|+BkADNT4U}hay_2jqPI7)byZd4>)?a@`lDWSq8fRN2>VpA z&CbUi|0GxJ2S}NjUi20B-Y~sCe=O*vHQs1`lxG0+;D)bX6@-MIgYmhrj3p>ufbnUhMpTNPEk;D%Y)h7)51o6*wkJZUsczpi@Pp1f-=!q`Nyz zwjdy_w9+lojiMlpYtfD5TC{Zk=e5}GbN2h3=lSybJ0JE3kLzCRzOET#jxpvVJHD^_ zHBa-6rM(8O3y4EQME>YE=a;qDF^`M-oL&)UB;i^mKWR^1JV43U{{X06>8Yv3QkYGjp8#=4DJ^KluqP0*E;89u%zP^2nzVJlDvy_@MsG)=lXy#z0A_YKEn>Kxj!%#8TN&x9~&YMIxQg~qH zB&;gic)wK%9Bwm^Cu@{D=XU1k=2&?yMa4=XG-QfMmZuEeV!x;WK%u`f)1IA5nIGy_ z8``;J1=s2fTYQ67{a|~ges@%2x)5{aG@Dj+PYo&1OtLp~;m(9Gfb$a0eW~_2GR%+L znK2Ro3lDg2a~~hqSQeQ9-NqGc>6TMOccg%HRDpL#k3RqfXMw3vCCdB0r4y=kq;kIP z&=R74{=#Fmy)_mhBDIU`?6SeiDk|eWx8eUo#B0NMyrQT`vNcOE^L_e}l;Wxmgn8me z>UVn5&lcaa%W@0AjwO$-|3=9tlv-J-11gU&23go`IeyOX>wBQlfyQ441E!xdr=9qy zHDKG~E?B3S;V?-W#nyU06Q~PW;-`KgB5VtY{WvZ}%;1{VOa_b8Uw}OVu>j54Q8_?F zlbxHZP*6}{YHZvFB===(+^#ZM)|P@8Q#gha{+%|C7kX1z`f*p+=9}ZuLZh}QZ%@_n zk}Gqn3LlQ{(Hz2W_#S#*HpwUQXLEU3e3L0A47u~Z>Em_zfrDyc|J^ZZ==-}e3e> z-HQ$lg*xMyKCB<2;I;-jm}W3IRq}wBLNhMcYiC-wuf6(vXU26IY3W`rvdW}?Pe4!$ zJT)b-m7XY}W0{U+%|+biU{EA?qEGD#{!+%*_Yfn%3ua79 z+CXk>hoIux5!l+;s08v;TRS^ODX>%qX9Gn;!(<2s=x4X+FIE8ag)G|90KqwM!!1qE z$XEfk*;I(U$h&0t*28%}<-4YP0?BbZ6<;|Gih&FZG(=hqRUIs-{ZmQBzalxi? zrIT;E2AK~#*|p}nLyo0YR$5w_aWytJ&KTu*rM-Sms<+%FuWt#83-1j|3o;8n<#!!h za?C0DD;+!+pYGWAIpjw27yF!~?@>ELR9XuAuV9b&k8Q6&p`Vgb9F-q>%pPh=q)(?LRk-B1PruvD_qDP$ zXA$osWRe19fn7<_(-&U_2J+I=IRCbfj3DrSI6$CLP&=gyreRiZpQl?In>h3@v|mQh zn(Y>cB-vj}Ns zraga>fapWmX{sa29-Dv7BXnV0aQ^fKDZHTiM{a2s&7AFS;$#^q`re2*^7n95VQAg5 zr=>fzsS-|6%i)d!Mw1#8i!}x-=~lU{Ur)Sme{&DeSFP_?XUN2&_EAk3DyDbXd^(^y zAi@Qs#1vd=7ESvv|!3pCT3-L))@DKUO)vqcAETGWoI>$@vV(DYVAPjGbNuSe|c#VQEM7` zY7SW!MZ8@?yD~j{)P*xfMLX*+Ym(G+B(<$GE<%*NeD&($b}Nv^DX%^Hv(X0rgM$uX z!FK0fyc0gGf}kn78NoEesL{Bfnxo4_rnRqcqY_V#Ss|i-Zy8R=U=v$g!|VnCEXEF6 z^W9ev3S$*0X$X#|!;UK}E4`&FGad5S6nQ89T0pFURqhH=co`C!;8z(LnT|Xo`IIeK zlxNU0>*!A-*jXWkKHu`exfh)t>gl>Uf~lh^_~;+dyP4hoNc&dcSJ`OG&4`&uNz6PG ztmhOUu@-zultcTv(npQfL5kBC*qwDXD((rCPyB4X`e3tX!aqK{o2<*wu8p!F z%W7e1hcP2SYl*SZotk=&b#+fzoO?kQorxWS_`@W@qCJsn{Fm)CoUs>n{4?ek8_WFwNd-t*Jd%Le$^uCc5>4+% z13x79M;lW2R&@6|ayij0ExuJ|%evey!Qaq;#xl{gDRFmYx~kW+Htk1 z?%g!C$<4cMoM}IrfcfSXvP~Q_dh-OPB1cAC9g{68L!v6PdQ7!ykvu#*iqU^_uE#>D zbhv>E0KJi$5Dp{3BOO^^WoehG!s0MTN_}`aumnr06il6(HKKW2;NxC8(Ui$u)=+Fqtz>$4yrReJV2Cf zNqd6|#g8Y!0JdL4w1Iutu}pwTj(3V&({EXwm10thiZ0owGAMDR{zNcdS@B2tH3_v_ zF(VgO4WE0iy!h<2ceC@(WUuwQPpapXlQQpe*|mc;Nk4sM=vqakDEFSR(R(G9e$IJR z(7`Np>E^0sy%CgbNl7#dx7-0Or!9Q&>7d7Ij}e^f5b(bn$^SB?X;qN~_Chkjuu{rO zL+7snAwipkwe{X`_R`y**|+HcNHOxW3C*EHlr(a@+;(2GOh>8ojXga7$&2!zGzY8F zsWHA3fS1*~6qJ4~+EnYb&y_7)4Yu#ID8d!?S=xEo!in&X3hm&T4zu{WxsZXi zEx!LRZblZ#J^l+)A)DH+GFtI6*Qk$*YzBRH+P$;O{k%obFOKraKvJ&)dHspr(FSt5 zbJ@vLyTOT|PI64?dkaV4Qdn5n$G)oAc+sNMCLSIY4ULVhLv#>Y7Zw(liYVCb#}DnU z7WKly8e);$dO!tQ4Ts1q(?ZxSF^Q{(J)%Mn>j?cG1Ded_>y z`5wAkYGm(waJx7)nY&$`!qGi1fA@FX7;V?$&7%5i8#d2tSJkE?4&~DnI9`Qv{L%Wt z2z&v%(mmLx(E%pSZo8{b4j)^_XXM0I&Cbu~cx{i;MRJ)0Ll)(R-2~5Hyl5D)PR-3- z+|C6@ke-no)Mn~x<_>IFm1T5ra%dsErsaeb0FBmtHV24`DNmfxoyTcIj*wRRl_Q4z zBhCMw`m1=r)E~f^p|W>5b8+1WV-i2g$6q4fyYS3@=2X{W`*c!T)oaB&QBi6XH9ht4 za2+yW?~jEA1S#N5q6w+t-i;gH*%hlA1K>Z2im@~2zIjf;gBhm5Oo3!eu5gP_Dd#1& zN`5NA#^d3A-%~`x+$T?+YTeUPR!#)HSuwSmvhvtZ@O&cj{zqBOGO41llXVubJQZ^J zxg0>)P;XPxAY_8w3t8Qf4fazx>2^FgnJwr@|HZ3yJtM}$)*-Og#Zeg6Mcx8AQ(~<9 zPpLJAcDetBuL9NDoHbBynt3J}d2-sfM+Q0sGsTmVk}SU+)jmO}%%)wBkWw;abg=l; zyo2~`3=11L>L6apy(hbQeMnQ88Q1m1SIzt#HLeysccU4gZ9u*vaM5D3WwF z^EdT92Snm4Z*4B~n{=@TSM>1H23?ljv@+?YuE!gBdTO60fM(TIsaCmV<9rUyvsMQW z5LFL=2PQWxxIhZq>W!`PK zbSfd;pXA{Uat@M|?citfix0KcA3U_UxJdi$h~dbuQHq$`7p_Ca<9unq+xP~~PlW5g zGaHRY1Q_i`bbR43Lwh6Jo2&0r8$jcxu#K8!uw7%6f}OByw^7IF9r7N3sbX6^NJM08 z({IQ#_k!yJl@w(}r~j<^W3TZK%wuhWM_adJw(-$w$9Xajq|*t>7t~ZS(ew2Z|9)#9 zv3za6Dm@CL3XKAK-T7H&)3wDX+gSic+lUzw5#?up{HW--)x|2oPi>^;m7bHM=Hjxk zvGw%nQ`mm$xqr|4hIUm!)6@kHUI_4lT1}nIM=V1vrxRLxMk%Nhx&gw+UuE&% z*IzrgQ?82#=M+cX(Lm#3noA|O2>8T~HJ%9ZvpiKPdHdh7cB-csoWs21#?&$;DUoO& z{~#$g7c`z6$IQl}Vra#8J4^c&VJK^F0z-ryefeC~xo_Brui0xd6_kWo(}T6lu>(Xm z%wCYwWy4axlr7jLqb>~2lap4k-}wee{O}un;1IM~C%XF%DmG?bUNx}&Qvk4qn#_li zimH{)4an{^yQl;7?ri_x3(mhNBoVCL%voMq4MP(GgI2y=1}{&GcDLuZ2@r7DHb$Du z>7D5@8?KyXk`B$QjiSRxo2lX^R`m}O4a>n+2MictF2lMh9Bjn@ewnG}V0sFW2k-LduuT1>7Sc9`gc*xI+U4mKac8)G4^KkJzWLjDC$*+vElaUddT zbEWjDfMh# zi5n?SndLXelLoxP3Wex9;v54=o=K0=1!sHRKRRdV=~?`+ zi4BVCcqM)f)r>BuiRt;WV?b%c;EM-y^?)ki3&St*t9n^ZF+L<4(_tG4`9&Wz5dN{_ z@zu1DzoqW!Ht;OC5YGY_K@6-`=vA5HdJ8^0KlN+ojp;GX$}CjRF=y@c(TxeOu?|SN zeHbUo#9*RWby=N_eOuNzxJA_{CVb@HLN52wp$)2yIA&`y&F%IpQyf6UxEJt&FD+e1 zu}|$h4{^Z+B_Td9xBxbV+5i|`<&WML&lrVS_JFNL`5aYLx{XI^qp5{v-t+%^`e)rG zv;S_V+D$1fTrHf^2ETCtC8orV{!+W{5KUBSUyff`6&*zb(H_eO>!^#4LK)slx}~pp zS1Z-I_346y(*x_@exe~9FYnd{_8crC8K6O0J2${A`~tVv0;D>$6N1lBoaoP=qMktCpvZRSX$Qzas*F^K0A%@BtpYYzyZ z>&t?ICs*#>octqI)`wYL5kxLGAMX2o4k^(3+JktG3t8f(A4U* z>y&1fqneg;JVO~!lLsP?5&rgHhc`aK-E}<_P0)2?ghsFJHedK%!+q(-iZAJPbb8y> zB;$lRXSYKhIpuCz-+s@9t4eyeChh4yJEXoK2aQg(Sr7G#Nqsb8?qvLXuX0bY(#C#|`|O8cTm>2xj*M;Uq2fN+ORUKds|cvLEz zro$}}mma#$;J-u8e5HM6rtK4=^Ud)0asQQeZe?v00#D3U-{>(yDTA;3X9f8A{f3$O z&RhyNH~9UnyG;vhf6;FV42BzS<8j$tc64+YV9b`iN~x2W`i{djoNF81c9JLBt=EfK z%?#$*kxsW+wED(*k8(-atX&f^)C?WL|kL<4U znRq4DF?Q4D=t5p()aCsb%VApzT*s}kRQTIUegiIwOw25|MWY37I31UNIIcXbd;W#J zb5_3IOM860-+{9x-v%&=aZ=w+42?)a_NUc>EP~inUZ*ot{@RE4J9u;F_&Lh3_ihp? zECqrx#0QDm&i^l@bJS~nr83KU_cgmKuVJ}&?qmnAVDyiHC)@cC?mnp$&i8@ZKtM$m z|Cr5&u{^g$AkP~k?AyJYc~IUEZda1K7`MI`o94_PX;O+zo94H(O7=RF8*pNM{8wkxWLu&2L6pJvS|3+ zz=oJZVY&)K_~NNHq5*Ce@jQu?RQW(_0@W$@uh8B{A^eZfj)tDPPQ<}@9mf8^92s!Q zE)st8cjNyXjOyKQDEa5FcqLhIO0#otm>-+v7+6o``Y9Of6%Qg4GXe|Fn@mByD-)5v z;j`)*V{*RnrD%Sr0>XJGqLvJWkgxB5(BRnrcP54IQ&&nEh!RAqJK1JES7y?wR>oTv zz3-W9KgFzwzJ5~nHkP)WoRr#{nV!C8dZnJg|v7C2cVK%AwUK8~f%ln=YdLS+4G|P6YPC5j3 zk6B;+`X8euFVAT~qP;O|N!<#1i|r(3m(To`iy?QKBCgSfTGSX>~+)OHbtEA!-sPFMX2&OrxVXqUol;`+fmn1okT3OM(IR z%BRYQ5p+|n=Yy%!V3J`9Y@6pt9Aiaqrv0K%!SlwOnf)h2fn9F#1JJVOUsRTQJ%kC3 zL+l1kwS_cUv_(H(=_dyvhjOkJ3dM$V=rKlSjzS&Es))wfYo`(25k?U}5p265Iv~O( zwdOC|;UM)l!_vFZmhbqEzEy7Q*81zCtW?oIc2)gQs`X)7Q!^)!%FDG>Dwpe_6Rcpa z|LQ50OPs_kllU0RSV;BRVEcP5_;xHpv8hy`C-3+(i|E2%84{K32+ zj3v#%?dZa_Xl!o_=6=$`OvIT|ujg!JlPYF2?`+a(c8Ix;M>DJMBw-ZEAK@xex%c^j zH!|>%=B{2=sO0b3eUDeIJ4|w)TVL}1#4Fj3522|`eT2pyg31;(TNZY{=&XsFr+?%6 zx3S-Z*Zmo^Pth6L8hVp|;AS!3vwd!`C)H^<=_M9WZQVh-SBcM#VmkcH`H!3H*9e9C zFvH~7`K;cc9N-Wr_dPP(ry^dzqyl^l)@#pDV!p8j78D!r`z9lAjsAl;j&f{%N++5q zl)yxR&dpD|7CTD6Jhg0ZaD3|1VF7LGNiGU%N0a$!OtP+&k#F><=VQ#&^N)iI`r^-O zFMr-X?s1ZUb->a2L&_9NPY|9I09X3sQicW%#gtVCFA0-zOSiJWwBJMx8kNEki4goR zysnxwV&L^Kz_4`s0LQ~~m=Ooe3r0*DH3cyn=5<+;lxWC%Y*Pza%)Y8dyFUm+Nnj7> z9d$^~g^|VPmlr1RIn9(onNuHb{34B3ccfYs;yB~t+USa?ok|)^|IOuM=BsW6IfYL4 z6(xQ23gwdkJ`tnV*x!~LJKgx;iaH6HxH&NyIhwko{6z~!VqpV&uIhmY#A zSG#NMop?4ZJUlWYH6q~`WfMK*Jn+j`Pb(9c&mZQ2UmoCfhm5rik>ySTSuP#8-7!^D z2!Pm6FZWP-p}{^3suaGc9(b1~9+8f8Nr&l8rIXm@Wy&I?UifWbQ@Kojj$#U&pfB3k zc!wnfz4#EbT*3VD`(c9vy4g)nB30`))Dx*T@8CeY#U;RfOI5XR|5sc1v-eWe*@~bn zQho!D(PkbW#CQhcbMk&q;;=AwUbtW)-f^X>2c^JF|f*FRXtiKFPY>5**le%b{ek2~V5`~hC{Z~ie+SV38>2L0 zAr*@Z47|P6#*g}<9BIwiAbexY=z({bb&{lMuRjhuT^AO%7uM;k);sq?E>0wCbJH~` zIoVYJJWq(?+8#O?>&vB~P+=sY zyeAUPh`L5B5OgX|dYUf6H&43`Tg52rc?*BFy1$&rMV+_~47=@N$3h0+aUEA-=!y0r zgLz0IEJXv)`#wTyEcY4EGsb?pmeu<lI1oJeJq#B1M-cfPc%f{ zj{0N?`rwNG-vo>YuZ% ztf|9F5sMS+C(_Rs-Dh8B(K4GI;m$+Ll)?aFP7Mz~yE9B8dJl%SuoqQ3 zKpRH;@K(STB#U+oxFo}$2iO5CngMq*N2~Kl-j2;)B}ZTyc-O5WP%;%2Q5@x;h&}* zw+2G-H)fO~{7D)gSy_~jN}bxOGldW)TawJw_z3Fm18-`k`2HFmRP>IH_SltPAKixo z9}9@Nd8lh;8MfteSzgo^$&BWVA51E(+x84=g*#{#>ZDbYk~(9O36%CfV+ra ziP>PuG>6IF+J_q+Zpt6$XkyZJa`g~R_C{9?;iL)N(heCAt}`ht8RzqEH8pwdSngiv zl91cWPuniLS$olB#!LF~yW|uSE#EFKYv6OSva#I-1279pI!elPyeY-rFnpqWbj^O{ zTS6~S=cUfl$EezOu8ZjolqlHdu2tpNRnx4nE5Y0Cst>gD-U0tQI~H3K1CZRtCi#_! z8je{H?cQZp+}$9%e#$@m)i29#=br8y*xvT$S%|#u3~TBDoHICAyD@%M6WiXNn4Zoe zE+G*W3fKKc9m;)G^CD%vcjj;)X%&Bd#2D)L0l`ZuJ3l_lHUTsxxqs7l0D?3bU)^@3 zV**Aq=xlf&9a{fa&FcVnHjDcg!#u`an}*AhqC78&uK{;p*w;mtSOP4>^I>fZjfnew zz_QE7y1^u?-m+*KOeMLQ1<_Z52;>-@+LlhUq2U&PyV+49Lti3Ee7e#mEz(o_$rIxH zsNIp*BSW?8oZ>&2(6d9EV}N>}IYB-AqKSx3;SGR^w;Hzk_wy#B;^%ux*n`>O@VoZ| z37|L6C=lG9(qYwoT89lcH@E9dZXE<1G(RFRyuG)SJ9y0;RD)D!zhQHg93Z~@wklfH zTU)*U8zFsX?LWn=L7w2Uq2c3j(Ag?+%UTM~!3f<_@a%Ah99;;H2(!mZ?6mXtzfAC) ziIpKu&Bu(Ha!iKr=Nkuk0!3?D-fJ@iZs|$_-Oki|XECv-uxKKIz#FokdJTIUmn-Hy zTE%*x>DS_~xe^+tn}#aok;)L$j|G*zwZQa7v@}SOXHI@Y$=>pKGS>7bE_03r?9@1J(5Lm{TBUT~~|&GkQq@LENlKlir(}hq;nJ1JbO26c^%{kP7kMo%HMwM|Aj6MaWpZ2aQ59>&^_>MIr< z!eYg?Tx7Gf??mW49=!$5gwirHOZI4(U|}1ds*&M1FI37cT(m>4sRAk=^@l#R%2U)RGHBdkIJ{OcpHHz=^#|q8zCnX=_BWa{% z|5TE``{%_snzxJ`eq1-%8CA3ipLX6XO01Wc6h87MisHdRxW?V34jGs3z8S`G!Bh2S zR`w*5ug11F#B3lrDjvx_E4e|E2hHql)6;6e+Z!H5iHtA}tg<$vw5RhjWbE8{8z=+Hw-8u&VlJPC2 z9H90CVU2L!>g&IUqaX}N=rUJN_9 zOP0d;3oV(1O<4^!>W=JMiXry05v0#x93mT~)76kq~7EEjWlxOoM2^=T<3u|oN8jOusU+Eqo_M9 z_+Kdm&RLO%?sxDHsi?fJYc+M4&^bYIiV4j3L6ClaB zEEy?HToHJ|TQzAcsw=Qp?WAA?aD69tC`?+v$_JQ^2M85($L#U^l2{17V07pf7Xcz) z+XV;1bC{qN00b0!BC_j_SIU&)CKc^D*MvFCBXgp$#>2+qDk@=v?n|;2tG#Bu5H%9> z_KYBTk{fk$U(thC((wDqN`C<{2~f9yj5Hr)P9oh-ip@NA8~5X@RIa~ZFr6z_22q!- zONST;6IXP@OaZ|1@XRmom|V=$;4-)u4d4y-i)U_T;rFjLas(TcPZ9VgHIb-mJ|iNP zKVwJn>zECaNue+ea6`&v#HSW6b&EYR1z8}YA03hm{YgWshUj5xjg{FJp?9??>a_!!NeA2bJ ze!+-}cXRNOsfKsa(#sqe?~~*9VoId3nJ}DLwxLJeg0#lGA0BRWmXF} zh0PyMigsCcLJW6G^>Vp3%Lrx90g=xclzi#w>B~h`JF@5+g5ZV0qavaSyvMn`cHIX& z)&_iEy;^=03HhJ~Fq?u>i-s`J_`Te5L+E^DG%b^tXlW&oNX7{rN}CHcN7nc2W@M(L zH@lW(=@k>e%jYH~Rq^Q%g$`(UI>(R=VJwBM-q>3A`VLqb~qA^epJo z3bxMWjyx`PQ}zY@H?%1Ccprg(L+Me%X%g^3^Q@848Zdgf?Bkc^xV8KnR^C#oi@j?# z^y}xdWmAdA#aMdTXE*cghb_MLMpXuE9F7^lKU#K5N(?f}c~Q1tqg%mbvHe`xrfH-2 zYzCKmcIcu~n&!j0Dzf;Hg~%fTh@TXEnuaE;Y`#(xr3P3sF+Y+CeNNFYYSQ#MfR#mb z?4LgTtey^&OfCNXqps=T;kOI>DRc%RGr%i@dGIfD1mOh5`&nmVG-0OU`p(|jf4aCr zs<#Y3;|0vzJe{B?7ZdwilU4%bEViu;hIx~iQMKZzHFtw)oST%oXuvTFrT+8R4KH80 z(uT!BLg$1*PmUo4U1rX+X1&z0yQk3XWjLL1godM#uA-7%t!l{rE{Hzl78m~T^4NPJ zvN;FSS<>amoHp-(@4wiz7TY7-n79`s%v}I1+u(w z40JLJ2Yh7A`fa022`X-QA+-D7*>3=B5@5URhuseV?L_}L;EAYLvgAYU@&hHqB-)Sl zqP1NTvf%Vb3plSO zay3(n784H@jrnkWo=)%qO6*~;yVX?#6zji**D+x*kN~@mSarL5Br;mMr|tXNZX*LI z=~-G+TiAthX4cZ9FO>BJEvM{e8kc$YSY20(x8MRNTu|qvOHpERbUB@oPg4rmi!@eO zXCrNc{t)#$S!Tn%?;LW&sGulbT0VzA)uK+m?3Mgfmv*2mRpO`mpi4$aoF8j@Sq_b6 z?JCyl6Vm&YwXs7tf_*6YKw!ckg~Q=CVL2mb>AduPwzECvY+i#Wc&rB9zgsklMhK!_ z=x^!$h$8fV+5OTLA=;_rt1-&5`gPl}d;4}P_}BUgW?**(NFyY;2}-+ja11ILi^}C% zJ>4bg<^*xqJY{e3BC!I%FtK}c#8Zltt&${zmP1$BR85-JOa_Sryd^04j@J}VdF`#h zB&w|-`f?6dQ2n{9k$limmguz$759c{_0V7paKkG6WE8K+Y&zXBc-Lnqt~7lO&(=Ps z$@dp;%*Drd^X!g6ofPVv8pE_sl^(`#bur;59$)>aF$0WWZqa(j_npIJyW{-61-u3G zf2|HAW1yFNgw2AtMtL6`(X7K1x*l20q*p_Kxukfb>Kl0#y5m=?=I74;_n= zU&Ts>8ldWtuCnq2{6bQ^$EV&ARr!fN-~`i*efy^l3(wsP^+6f}ZhVcvW%B0HC#%r{ zLdmbm`;C~l89;8JDm%Iipq;YLba~_ru<}BRB#*t?O6YfnA0HPaD|q2}RdK-f&D`%M zCkuY#9gcRq8i=U=-q|%*jvI`t0d5^6((j}Q*EoV&G}-CZ9Qdu_K=FJYv*|Mp_xe}k ziIt}QJ3;?wrGv8pe{>CibxBqyv z6(IPaG?eSJM;)>C2{;<(vOOCi7{>wPhHQO6MLRg8mYKd8E)zf(b~Z~;f?+xe9ghZI zc#G9egoN;*5)W5ZZ%JWi#joG5uu+(a5tBxcLtUKEkR5y~UXIIGQgo};jMZcsP!<8A4^iLV((hr}h< zDml&=NR{of(cEY6fzo1&L*}ifbg1>WjV)p5UfhH(tk!1 ziTYVe2V$E>>LWu3#cuA-^)T)G@DasCq0BV}nad3_m#J1jiO2o-p1>u@48 zxZAuSI%ABoTMGH?CVgHs>_uzHw#J4=_rsuIiP#;h&Fd_pk&H4jS*1*gQVJ%D)>U5S_Af>wm6q^_;SD2r<^G1di2`eT|-)NghU9}>44N0==9$*zul!Xmt) z5CA}~_4&ANQPn^>nJzS1H#)NJ9e&4TGA^+-Hi9xEdi3!L4DYTC<(l{DbDgWBN1Q^~ zd3I)67Z)w1&QyoJ1A>A#NA`!0I`f7UxY?PDG@>d_^||0<5y>y44woa%Y39uK2%>+% zA4%luU+Q7jJXKPfy!6YO=H_EJe1LfYEFRe8UdQD;D zkSpwrLrT!Lv&w=^=u~#NQHh}Cq1YcxvwYmBmFY{l&vHcwojMnTht@M|Q zH3)0ZpI*z;*iSeZUE#9BZisEt5gRPBQxLjAQLkqgAdwt9z+$kV4PBL|Cy@Y9+C_l0 ztc5uEfTDB3UTV}Yptf}Q0=GZAhF$#a`A26=wq9bdMHihE=DgUgJ478F4^hudS%YiK z_phIRppZr|E2V49*h;UUalN@~xm`XvDMOc<;=}>DNYf;Xs))*U-L#&s;w~4yV@Umy zzOTvb-h@L{eHu&_il03@1d3`FW@agWfB&Pu|GsDs0pwKA44H>xz7~8uZZt<4#2dyA znIPUpGsC76QjynEQ=SlaDCZJ6UuN5Y=Nem6k%S3O0AV5cP`gCGqB7;2G0fP6!xy_I3uLfIK5Fsz4ybu>W z3V1laWkd?`SqExxO=wn%V(n=a?y2Vh=FHCAipco1>UXRw0gK@Vh=J{_s`p`FKL8;h z(ZA}o9GydeW+vvtR-)7%yMqiDtO}tWaUi7z?Z4~DbBcV+0fF)H@iY(z^YnFX396{7 z;z4s93>;Mu@s7{WJ8;4fi_GR@=}oxd(=ju*XgC~lpA8;9N&l1^3N`SmGL(W}%$w{p z0?4M))A3O&K0rajQ?K0NEeK-p<#*|oW zZ?|-C*hYuj?t|x!D{5T@>?U5l@+vglHIp#3x6dE+Sks;bg>9FPTb5Nzwof;C44Aij*@v$^Hz=}CFzfj828 z`_OsbfBg6{%{B`W(O@driUKrI+-iVEA-Q z<}Jj*AZ1@wt=m**M=+luc$l@{6yA6M!{P@Abz>1!TucdnA2@Z$TM#BskRq4!Ob6Gw zg!$6q9)2$V;81pSESzYG$siZqZhNM7kZ2(^$G3F~P9Ji!0U|*JwVSpgFI1?RVE&m7 z{-#pM;pE289n>9o7Xnoe)oX9LVf3C~%WRn|>2zEML!Lr*Px@{uSdErziG?k1SS8YU zg_^fC%O9O*HW0c*8ppZe(%e-@1Ym8;Ay_OD0+#?hRvH@vW(UfOV40l#k1xNUx>i;W z%7ygz?|;1CqIH|euSJuFg_F_^$oEJ{(&a)x@-0(`97Tu{wOTQt8N;GuLTqqY8# zLM|WOLP&_5m2q$Hz!#AOBp>(S<&w|vF~-|RrdRc(2(8cZt1E;MyUhykXu**%h_GrD zzqTO!AA<<{P60vdk8)Le<@b8T3phqcF{QZ2ze`d6i-LtaPC=^ve8}|~R{^su2kF8~ z&pk^NN7TJS3(|U-XMX!lY2d?_is;zTqeIW6QyPR{Cz=to5hW9VK5T4kKnzrXwIqF9 zB0@r19UU!9Lh=hsWt-#p2zJd&QlK=$Y)tqc19F9?=vGfF`bpSzE2BZCGIa1IWy7<^XY>2$h40%w|7_i<`X~AAX-=2Ax@>}H*~p*Z zENv@1)9>HrqJ@sh(B7NY`6^-f8<6;7(3LuwLcCB7{Gi_LJi)}q~7Av7Q zVd$hn;0cwz>2jwJ4G&L_j;01D7glcWm0*3*on@th{QUe4X$q^bkPv#GnhR2;U|s^M zQw~){#rM9xN56j2>JM6jwTyrTO7ld?2IpH1iR`rQZ3i;oUQk)Kk@EJpoxl+j^$tI% zKM!>ws^HU>?_wG2bVAx>!PXFCA$ICa7khUi@{Y4kD*ZtSMl? zmWuwrA8RMK#HaA+>o)&_yTVw)dIU5kUfl1j0m!FOQfMie&&#|TM`#of3Q;%5V;Jy&bQ))I@?Tafh zZ0h$5j@;-&z$G$viO}^_Zx!7uhi%Cj^PX#Ea{7q&^o>5yxGsPOq`XnI&cg@DAK^tH z5>I-Ok&;?C^gqu%MrS8yX=zzzmz%#xApt8V%i@j=0x^nUeLXef(AoW?EnHtkBg9Ta zaH;j%SC){;=ZEfxedm^vrJrqQo|Rf*VNT;^A8N_W>!K{*7cc+>DNdde$=g!Bx7Z2IWlV%&1o@K^ijgp$GBm>S)IzLLc$F?SYa*|S2 z%{pLh0P&h0#MxfEOK}BA20ZUJg!S56^U8oFlfm|2Eyc>oDGx6N!Y63PP*eumgK#uA zQ9LW?>pvbf%fFOzD=I)4Tvxv1?`McphMjzeds;eOTd4p@0v{llC#-y`kKG$L$W@;;mC^C*niw1J! z>qM}Sk%zrdt_z1yFnLHuD%jHU(3jG zrmKt9{!3k*)3>D>K_tkHdDnu3NYI@i18Oi#ji8Q#VKlU_;Kao6RpO^a*;@gztWR4o zO+u5Vdq!%>8OWupm~2`JO3OVoBx)) z$>)pA3=K0gME2KoI;t;7DW%ADz>d~sTuPiNBoyR8G+?Rb?p_W9>3ERi)wkx@X%$4j zdE>l0`TqSy((~88j2FXyLF9==kX+;%jkdz*;d2Krw0aTY55j=I# z{UklR#8-Ra7eks_$#0~LK|P4+>7WIO><*#RmP-;NrR0)S%}@cYBMBLG+cFNT9Uy2u z?46jWCieBqmp5-O64{%8Gq}gLO4?vOOM8k$gMm|irG^5)Ug&K@#~ImHRJOg--3N4} zc-AQv!m%h)?&O|ciwZE^C=}xEiz(Xi2R2+@$5NQ*O zZ%#}cj>j$mQvdn$Ek-6LHvv*5&^ViZe)jw3OkQigtIA1s1*eQi6^%ZQZ&0oW4~notgrk+r;VGJy(Ox8n&lP_?Fg2MUD46_DdHoG9?mSA0}#lfN_5P8oGPnZ_>P zxFag=b2aWTUm8a+&z^K|p7b>>t@f^pI1mGvv>ksr`+~^+4?n%)-jybI)r^(xT~n|L zisZLTMpFr}ci8+=X$FPJ$G+J~tH5(tZ-E6IGHLSs84?#=CCbNle`I82(yGs#2drLh zfpQi|K*8bcn#sT@Yy{sED3MC{cDEcRn$CmKkv5&ja!^_h?$+cWF()%caZgE@3i2h7 z{$1d@Rc0QxKo+&bs1qIhjF$=UCnayU5DT%N{F6aDs=JxVlh;9I@V?oU`(IpJH2ogW zOUr51TR&;#()#_XJ8qkJNE}>lV>7SA2=Hpk-PTOuRCl!bmFAjAMU#`eLcmKJ*WjIU~@%`>e@PB6t^d8@a^>lxNxDxk?4Dj-X0l^)o}Z8i>Z|hVD_9h z4`k&7zq8{8&J6i`-0;|Udiq5SV7qYPX0Vgz+g9p3b@7iA?3TiQ*Cenhu4J?{v~Zc? zYUy3=)E>`psD{bH)`cHXbpjmxsNiFkrqLJvEE=df3l%h~ZnF%`yEXy8K4ednIXN&b z&4mgW8H-^W36B%b@bmMdL=Ov!lb$>|C}p(Q3OAd!CDZQx;{>zCB=|s%JbY&Qr({we zR+RpA!`rn1lK-_bP7!3SDx1IILyCMKD?QC*xIO;f0pHpkqpqi z<=pWCgIh;T8bTB?zu7JQRD=`!gSFreah(Bj+uGZR+e21?mqhP=`E_^-12XeqxDyBT z0LbOn*LxMHC*Zu4(W z?>w;e0tfDW$Sq7p)05p`SKKzAuZ~_RG0KyJv{Lhwat{U}&M7{5mbH#lc@hU?P)qLr1r(J!g=Am-K(j^8(V!|4* zmnQ^F`sS0e_E@>Q`0Ab}BvN^@`RiOh+_rtFTF^y>*atFsm1hMCElxuoYT$Z0$s`2Z zZMlFV)f^|P3#{a2>trJD#{a>?5vBKm4LcpL01y`BP=OIq3#VvDWcsFIMb&&FR4K}; zo-k&-<8#CqbZY>ussqKA)6E>7d)SmPJL$(Y{xC)U93p~>bJs**0*AsyjZ(s|de7~7 z{)Wk~dT)muM-?b}K>?`ZL~5VSDEU0Wie(!31EN)3{(9$p|NOHJTq6)00~D#wJYaAb z)xpyPn_SE-zA@XKw<_e%2c`{7g%E5P-a?zb5bx*3$lWv3YrhZ*zjQkYYTF-Ld06G% z5Y0%|I)#j997erqk1iYa01=zW{;wbVv~v42Wcii3*o)^mlB((uy zX!Wl4-bdJydS-i$edHlVQaewZJWC`q8Y5$6W!nOrp@9{j*5x0W2aWEH$khK{CR zQC&zQxA^Sa1Efbqs9{`KEorsXAP<^YjmUK^9>5X^o;g=}If)2F?Q>uShKduEeZht# zR*nS5L~yN6naSsuxQs>v;n!nwg}R64g>QNk42<|$B(Ye1E{l6kjN(TMn6%OR-zQVF zAj-$cM^{8s`|+EC>D|WhQQcbw=lMij+ahJ7JS$S*`eaQi%;r9hGXFVkB5*C=MgVsz75 z1J!gp^p{wpH2e{}3&`;((UHo(d{`ox58DF%yFx$mLJ$O08Wcp3M!LJC1f)y4yZg=s`<(wd_ug-eJI1%hVC&vnVg1&6-}yfC zna_Nts;z=^3~cpu@k^7C{M!xXKtzJeGfi&jMK^R3<&!mTvse6g@0G>M-S5kdbX%>1 z-dt)*i;qV2LZF(RPCCl{R9w!q5tYLzCA5I?6EwFE5X4UES|M@@zx`0{#k4UL&#bjy zs6k6hrWtDa5{Ppv-IGI6Sq+UCkzkSxh*8S5zSz5#J}90Zi0?5x1=C$C@M&KF>&S!N z50^6$9|6xkeP}@N%uED&mov+>ucD?P<7p{b@-uype7Ws5`|$NZFP^QrJX=hOxtaDy zUhUPrlprJ8n$MLNOtRr-)+jkaLwc{iSb}`h6Kg{8JaQIUenljL;s)VlPY^q_c=v-J zaULg52p{Hg&>epp7kB2z^9v}ab6rR(gP3ON}XyAFs zL@;T|u>}@1a$x-E$GbmuD?=qm05STJZHQI>-`rYqj_}u*FluGBUWV=5=`+^0%Gkl} zOZ}4Cyxh^PqL36C`l^r=@BbD=es8Du$Mt14V$1kf?9b2UPDLLF8CN0iwYX;2iXmY(XNq@YR>ZnzI3jTQ4+{4{fiy6NCfuK&wm5 z2M-!JB`y^uj|x@&(`GJ3Hru5Ie13OaD*NjTzh{x1Fc`pz@c>sWm5v#qovs?wY@E#GLGZ)7 zx>PollB)q$e?YORsH}`|G?0je6AghIa#NESbmd4v;fDHUe}1fbdz<91E9Um* zFF!$70k*+K8JL^X3#elGVD|HUm+NUT#tjS}$zLBUb~|oMkpRcJxL@iE2Fw@y0k`p= z52`|N8$-NQWbFr}Mgs2V#*m;A@VXWe5D;vExwgq{d+eE=o%YE_xVaO_e9ap<{v*R5a7t38ZKTua!g<}42Dd#z6YDuiBFUu=A|FkepW9k#B>!M_ zNhfj3h{->NJ2KW^o2<*!GPoC6sA z1t4X55w0&H0v{XN!#oeq$Ck_lX^!BoIk(w$74V6vd;D)8<#Kc!35w$-SUZ02-z!2t zILK6?MGR`*FJHa{4tC?foCt)CFhr(;mN!we_|B~I!2K+yyZe%4P8-)ATsrmfi(Q-U$4j2Mf%iv52|8 zfk&3hJs5~G%tYVBLx=L_igTOtL(k$R_6Pr9lRC~7Jj(Zn=h&nZ>YONAbtWfj_)wP^ zbw4x%cW@z(y;@R6&K-Z6xDUGch*_fezy(hWPKc%HWxqVwO`aQC@aGe#WiCm0X^jYW zJT?hjxRb$~GyYSJsl4M0R7Jq-hwYzc)|puX@B&E6^^Mo|xdu1B_NrAkFB5S>XB_GW zsZ>@I_@}}Ej=YgZ^o>GVs+o;6hYPXqbj8s4z`Ydcs0lq!VNmfunnys%{QL#a#~T1j zpo%1mm~6r9^z^7G0-p7{TMK_YFPchiwBQ~Ir#UVw-^mKZp@(6Dsw4FVO{{NS^`r0*!yUy!jiE7X^$gs1s>U5ex z=8qWh3BK|SZW?0*XD!lX3I?zr_>BZ4+j)89OiWYCke85n z2lnHXv9Xu$)6;YFoq+Lv=ljQ_b3be^ROtiv1A|{;Ec*K)-v`E1**4Hm#{>3AZ%STn z8v#J$+c)RplI|ttN)i3{C@`hmHY;8d80Z%G6+zKb0FhOlF2RH0tyiC6wqF_k0R_qJ zRMPnH@MIv_re|g(LcxU&%9-CF8H3`bG5Coge&7j-OqGFl&@G3UEJD9)`Nl6gYY3CGcf38Cu+{eBE0Kkyygr>L?c^R8(_**+x?}{U2FmN^aUDtQ}}VCScL4 z+dn07nhIfevo9R@e>OR0a!Q9z+!3eQScocJW2Yw$m?Yf&noMwVbm#RGJL@ZZA(bYW zei}331?P{@+xx4w$9G6wV8R!sRFJJ`l;(37bupnLuj4sWvW@ooEamQqRD+NOxwAiHR| zzS{;LPQmBG!u7(tJUjjRX3k6CjGE_h>Mq{3(38gIe&i?)LXaMw-C?L}edV;70XJmc zu7>x7Y)%_5!W*HNtaHR+EF+ljLMfd_4Bv(<&6#{Lo=#5NpNXia?`$7xwoJ5{sxLhP#>G6n7pNB z_N&&2uTYV8dln{ zBS&K-qQkDj$CP|8@AyGEZj=f#f9$g|!oR8sjbS$U?(WK!m=o{_D1rBv;ue515CqH^&+)vBN7rz%O%@l(JGpg-a=M+Jm-&|&Qjl4O zDajS1|{9bL&C5}1ri+t)F|Gf%x9o$v#f zcXWhOCAzMB2_*utrU2XI*=V+(8?<0YU@zS02aJ43c>)_3gIc0<{e75`a^l{L^$UtR zC{KZ8)6Rl6Qi;&-GVv?c$Q+|QqwcLPNhSna+P`~3gv9{1gUDcmdqexeoRS75MW`^_VCYlT-QT}Qa~GsGQ^v0=!VwZ>R-&N)N%v2PtPUr zQFRtpJnL0(eo`vy6Y!wqKXjhqF&?=p^q^i%0v~EM>mjy5n{Q9;+OU$S>wr5E!l^dB zovSupJWj*alv+l4+@8C4`<-S#Ka~a zRA2uozOD;n5`m27GbOa&NI|EZjq?&RdWN0NGB2OA9K=adlYiA|=``sJv5M~6p^B%P zXB!P<-aK}*h>X(I(CDF{f4Uni^XGBaJflb2b2T+*;l|)?3;FcQ+h!Tcnu_I_rXn{C zpUA{^AIZ4=0FQ(RDRbAu*T2`XARY+?MP%{;lqdF^Mn@}eRKfkGEY)lBy>L`#m~t^`7G~R_}~2!WyPmt^9$1;JPZ{j zFB%%x%T5j!XK7dw^$HAb(14O7T@fp_J!B9tv8{?*9?;fL}yH?Qr1;ytrI2SL^vC8P7`tiSCr8Ri|Fz z3y?iN&}rg%mvdle%D8skYiG{_@D?EA$vqfCfgBf=(6Fc(G!IRwP{dFa77=)4l;O0@{#ushyad2>E zJuC>mo(yk}h0eZ`Kr@PhL4XSITxz*=$DUE)ST+MT3;Y<5Sg7=-wUzZ>hKa&i7e1Qaa zZ5u^2LqamkG~iR__uRg_WoZ9(QnvTaGXFn#8gJrgUY{ieYHYFWmvQv=>@v-Eb@;S( zQN)~N5JneRZ~d7%={)6gaUda8blUMaVr3oJeZxTc^n+U%v78)6^y}W zo0NQ+JEk5xa#nqV$$ww>-s$X3_4K@qgClnJ@Mghfv8y+Je&C6W+&t8$ykl??Ju)Uz zykZrU609~OHrDmU4`YgFxpv>z>sfSqNIo~v!?oyDTMK!XQ0eyWL!v6lWX}j=qU#ov z!MESievz2fH+!qqT}PkF)^P;lIhfD!7fzcASgun*6x7~rtboImg9vM-Xc8;z=dF;Z zdJY6jbm@@I_s3|ruWyu46I0OEHSCYEnP7;&(|ejMR`WW8;bV>;a$OGXW%dpE{J8<} zm1?my6(|n$K|GOMkH@Id1iRrv(Q?6%Ye)k8QPGz@_9u}}A9sI(Uddr~*f03%M0;oK ze)E@|%R`UiX*L84CftOwT%xjL?vPC86Qe2c9_;?14&K4X!wLC3{TP#|@@Se#HQ)FS zLVBfSWK2WIb|`%ZQ%*vmX?#TxV1Y1@6!ml{~_m&mg3ymw^2fC-?Hg zrn;Oy&7uza-D2SvGq|}dg%R%tzca`TQ}LDo$O;*mobP5KNK+j8n)l{u1}i#uDjSRU z!n4){wOgzy?F1e1Gt4lBMP?|je%`kC^r)t$=HRflrbfDjg@NJyY`jMtpcHFRwB&-$ z1;iGf-SaOx6NFo1IB$6Z&CRS^^9Hwh0Sa$^)SV!1EQ2MhM{Rj*~oItpM%tX(C$lu5>~{`R`F z)UC~wUTFgsjKl|}R~_pkGr9I56mdFjP(npKkccyMYU%~b#>PfdQ`48!)OY>;Dw)MI z1oXJq!KFD>)u_-YnfjTlETi6Qg%&i_U)Nvo8?hk|XXle7ml-@cFujd98jus26Huw;8XdXY_O`q)Q z>Vh7M$I_)tG&H_elMNwZVV8;ul&rX7pDQb0Vm9q(hKdkB0N%oG^GEPp%)Pb_D7+-? zaQ9-Qy*(`2^;fz~HzZPrC-{}FcZ)bouB?~O-ymM_#!xiSi?d11UyxVX>oMx z@moR(=ebH@j;5aNqA^HR{@q{oVp4CbyY@mDDpp$0yLvsVLn@vT2Ws4a>AbXO-=+;~ zzS(Wg;)x}9-#b)^OjD&u_6r==!@_gnXy*2Ax zHz&4Zq6Ic_GSu&tpNRdHkG%rc11g`~VrV`uy_yXPBWcuND`L%MT$SA2fp!Q7uAaz< zgagXAF046!6n``35Y{>PW=Hrm4w>lGrmgiT$SO{eUJ46*v!AHujZ|hW~Q-o zPsr}kbl(k}kcLh1OTNi;yh7j1Xz8;!&FtQrIVJT+XA{4xzv%WvxGyV{<);gGv(>me zPc`k?J$eb+;KOfbnjfV9{lR3vIq;c+Wb*h)nZ3MFbnU1dn-LSh%~#wmD?v8NI}V`NMHWTjYd<)ZP$q2z{&n z(@$*5qah(;&T^Nt9>S5FhHBq8{k-67$L4gm`|R&u3e&sVYY%iJa%IRH8>coN$}GR9 z&yWOmJxk#O`%}A_FqPiF(LBOA7q=(xRWnldG22R$4%oOlV z-7EEZPhK$fN#4Z4W73YgoJ+*5FFT~F>7%3kps*pQ5TL5FD@5sA`}ow@&^0H!5uA5) z%jWC9?_F2$wVN=YxaEH>fB&8!&KNpx!Q1TNPi&ypSn>aL@ajG6(H#;xi!K%c zQzcXf{|t*>{E(Y6t}n!{zob8VR1=y}wQChyTU*`c@y2s9&z^1Zv+LpVRO+GjM0!|q zVe$?+zbV#UqTUrsQXe+h*${U*tn)BS=cpKlNnNvA8{ z3@wm2dp?8nN02B$)=AC4u$+F2Bw4S7R=bYd4vS`JQ>G+}%~~VP^fnkt&Ev1*R0oQQ zTblWX-t71?rBd^{=a}y7?z}|LRGU9z2C%tAyrI)0;_@t3HJ4iE^8vQwE6gdje#zFU1m3r5 zw-J`0kkHf;)*Z!-)}N*)`46yV0u;9w@aw(94q0+ML^ZdltR*tSXMUSJ!eb)8nF0Wb zf`^9~79J_28mvLl4_{uB;X8g*dC-&P2<=M|Y$l(@@4m|;u}`iZt)VTL3eV-tiji`n z;gp*<>|#j}FpY`7-3)zKK9s^NUlb^Ht9x;OsR$ugorfWEGv9QD@8p-2Su zt6UsByhf-dNE527s*0PM(tD#}Q?s&mFFhanra=7LU7oQ^=nk9xPpr=$Lezysq*+wa z0)C{Jsnv>v@2wqV$coqpC(AS%rYqSsNKVG8RnxGrbT&qcnIxa#90Vw`2;S?7tSo!n zL*(OhG|d_Q8AmLO?`F|+gP9+xc58!CmyUScNMEwq7dqOi+;B+Rnb-*^y+xw8$?wB} z>>#U+tLYICe9cC!Uf2BHb-731@We?xINocuNPZh&izfqfMDg|A=oN`PR8Wg9E{%aVieG(j)F%JUn($oHZs zHj(d?uy1=`-wikm&X=*;bn}SJ<_8N5uhRf>OP?NyMU6LSdT~-HdOtnICID@Sewg_i zg%KX*O+s8DqfRRcw+{xA^0DGJD#pM3h9AW@WmxFsC_g@QROIiiCJU@qUsj~{Q~e#ibO3=n2X z|D|cu18UZPkcBeaJU&jf5~^L$)C8}8;4^U@!$OV(f#aEJ{__=*b+$t4?CvZmA4=>G zMR8jPjPThOpeP&^>dy$r&J*1-c7IKN(k>@ix8vY9xOc60;bs;{F7yk?lyAix|L70NIu zrtPL>M8|2vlWr|#pSryc2q7eLasStVmD`iGswwxa)!~8yOW9cQM2^^OGaT1_gZs^P z!Rn?rqA{q@qTQEx6$~Fc-c-puE`Ah$mZ0Q!??W;OZX$bAVKOQ|Ux*)1H6jCtPu7Dy zpy)!HZMUf8gK@V88Z8DY+@9bDynV7#RNJT;kXJVOlI?|iA@MJtVZDOcPkI(L_V;nG zZ!O<$Yg${r!v0`@H9^q<)+>XpJb&m*amY6_Ys{<1L?VM`z}cC-EagtKhaV7I9@v9oGg(|G@z76V7hdBO%v=CM4j z(eJa_qp4$uhlZ#>6OqP=EUv$;+zYFDIZ>c;=vsX~@vo|H_^-`+sfiQ6d_nnrQ@HCV zl9v8AT)sG7{$qx@jP6F94X49DNWYgmNztw>v?92#kLXLoifuq)@U`hU|MFs^=Y7;2 z;sw;HtVXg{Sg!kc{UyI`$|g}FIMAFQXD|EfJ{=Y5XCBL6ewR2?ne&Ai-{h6+ z$u#CV!GA#E$8?r4;+5{l3%h}L&|lQw$wv}DvO>Ss7(F<8ApoB=%sz9r3)FkQ>nQS# zlKCN3+(&yHFx)&l4A*ZVhha$Xmy-5sbrTyWqiHRPv|%QW0F0dVfEH)#S*lr_y975# zS^Xl?CN8^fe@t7inIBJRi+F**D~kW{PI`e4_{cpC$H~lkAR}AkK=HZ6deBbVtLXOC z0HfkD*!Hdh21?`%b<7ynGS7dGmwsYcr{1$ZWmhtAW$I*?)4C|e`VZx zoINz?w8Q>MnHK}iZ;WY%RRORti>NhPvr;1IaLtK6>M|~L%xxy|oG z?w(n2v*MUA-H8N7Y&O>FjdvZ6zm6BQbnFfy#`P|DNyFcFODu%XVnfs~=6g(=r7t7+ z5{`Gnbqn5>eAXqgqoEA`{je3e!8f}2-&T3sZ%NwcqmefX4yMq{=3b_9QrZ6zj}0;Be?rB#oE`qv_* z{(iSSj9SAZme=XHs8o#)VU322+(KZluOS2Ld32zLk9I{_&|PV=!GOvBz06^76bHN) zwXfdwJx9mU+!ezL6D*_2f*h3<->xSr zw1ih^Ms{7f!{)NTtSa?c0%0_NQ`FI~K=3?OAZ5nr`$dQ(Ayj-V+eOi1J@&ME@6Mrm z$qfp*!?Hr-b86(2OG&BiJjW8`h*?oivGWH@v^#Fa&XzZo(oT*ouJx!ATDd-^kH1vP zw@D;JfE7kUu~a6BZu0f^8T5;c5YD&ff^}0xo0Y?AHd<)PsWF(c-r_)_KUikrWH9sA zT#9I#&7Pf@URUreqP-xd|DMp>`OUSY&KDB@z$Fry_@GW_Z$M#iosco*j+-g8Q@gNC znZ048D2oP6?iZ1oUJ~S_y=*YLaOrtZafe%j=w+MTz0qN7-Nypnwq#L$(P5(ZjlutK%ViA5s~m3e zf2UUH=$QTfZg^Z{YYvYe`t_Z&08nzfmmAyU6m^Go*HOqt%yq#E%xW2h$@v@YGvs!pZ(n%cxe zLWGVthKYBIDv;RLAp=X5%IzYSM0os*R5|wp^uT6V#k0Z~2G(!Rh z8WCID6Q4v>32U--*Q{?qP>?e$F7muRS`CdgZFw@!!t10%o=d%(WZ-)rvK4d`>Etx8 z{T9B`b}x@6J)4XIs{!vDBh6b|0LJaE`=MebcQ=E=#Xm5Rl2(G|<0~;KsclCOetxfF ze@9ul$-mug`&v)puFMVa##D>TArz>kJf(jpnaaX$N?O8Nk?+e+wUCJkJ?-RsW}g41 z3lSK-?(_Gxciv`=l{8UvQV^CFhy3Ch;8;%rqJul<1zy_jZRbGk-#lKvkX)^quTvXP z-@LcuhTcN*_BBJ?$H-ZSIE#RH-6;=>abCG&M-ro!beP-`eFsIbbpt+1FvFx{`Og}RH5Cts#!!$zWsl| zi>T;uKfgRq-s3-9baVmGgZ&oZ=Ovxo0YILpOqE9up`7t7g4qkmQ9bQURP-K5Ki|F4H@ z^s|O1TbI|=8O@lRDRz{SUpH@+ji^R1$|D^m2|-HhqBOru9O1}5**9afBVlx2BqD5Z z3p=nJS!3)^Hy*xWr4VyT`{YUi;}+2>2MTq&&uF-JR6IO_%e*FkIjklwLq{ejO#9sq zBeXA#Jw2>ErJ$mE3)R@%mo2z@dU_GuP7fbEcz{qAFwj#@MWv}*hOZ5@*d247iEi6; ze!*~3mD$AH$S5H@TfG^6-4_NVb&kt?vWU2qFJ+l@w-KIY+a3z4CbK;~XnU1eI|^F* z4grmCuBEE6v!|#FTn5U>c)8t%SH$qcr%d#5tyYQiv_zEln(pMqCP!ik?EKaae73}wL4@}n z`Rc~~`SUz@7rZ1QqWvNgii-OC9*7bC9$aW$VczT55t#9a?(mn`ZFqkp`YoZ4J*C-m zau!|wk1-r8-tSIp zb0!xxs6GG>OnDu_Um=pEmJP?YM^QN>BBBX;uya)C%MKROlBHZxpcgsLAl~hBXQJpP zyoy6|yY-y8Y#sMUMXaP|txsn4j|?qucIsCD`_l>eTyToa>|rnPnA%U}M+v|@&|WsK zQaKDYc^mW{079tmW2P~b0+NgfQ5D!^eaUQ$`6e!?2N=ySy=-9hm)7mykmNJwfFQ=kg2hw#&a5ctI|a}>SjCb ze=(G&o1s4ZgC6%DmE09~3Cwg|lF3^K zc2i!>$8*thq^DYi!Wm!N)fj3$N8)QYIAx25YEiCZ6yI&&!LZVllF9n<&oN9T3+-#W zNxbS_lkjQl8XDr**Du_-dskdTg9rvFkkIf}ZSy^#ruLa^JT{r9W$E~xY4vPrEM)pg zFx!3(K#dn-;r+Nx=S%V1+uO5MKFVZ^3@Kc-Hxow~UtfPEP?<(Hho(H{BT^d0g4_0H zsovl;((QZI_U6%dr`qsGzOeTyvP-`N(u6PR)iZY3d;w;E{r8l6;7VuvRIAX%x@x=Z zB^-{ej9JGlab_!aO00)$3^+I;tm~?me7ol<>FHbb=^s7XN)P7inI|^`hlX)LKN0M! z>X1R0ZGkuQi^^>c{BPJ4UV!cjv&M!CD;x6uufzP(9x`rj|#ESmhfb$eyR zFDWy|K33r#;}wXf$8-QU#JWEz^{y9gm6x4JVddK=tCtDd-rs$;)b^gD^rmU<0kFJ$ z4jWg$o%ZA#$Bv&o7CLfb9>>9V%pmv#iKfMTm?^rU+*khAQZveE?C=e!dY$A_o#s7$ z{FPrRlO(h*1MfTYH_3FauJhaUARd@dGt+Mo$W~@!Zo243U0NVM&w`cScCsW%mJv)2~Zj!x5B(n>=k6Yx`xHhz)}TQxfgy)l%~u9LERknVJ$w_<6Z zn=66m8}H@X?%~3_x8ZeWogKY>U(KScDzbreoEab2UkrW*8&Hx`t)z<`CTH|FQ zpUa(WWHAs%w3E-__C#bkuKm$aPmz_OX6C;F{A)3fo%lg^!t3rp(u{&}1`1Wk^ZN<8R_!Yk{4#ftXKi}oI8W?Y6R zaM4qSkRvrMO*LDG-+P_49^_}n*CwdB=p!zO8~2|d-h9eYXpvchGZ#}hYCnGvuhmR5 z!zcr`xV1%Q_>4y?1DSutuw5Y3n8*(eiyK+`Fr+3>282z2WtR>2ON?jE7$t$Ay$7_zv^kG^ zUPgj?N^g#4=J)q)s{$Fb)14RQI7t}yIJYW@(%CmRZo#U;by#}A){)U)dq{{iZb@Li zZjO#~bn>p^Xzr+8B|~jlyiN*XB~En~(c&hF(W>eOGHr}9BB z+!rqg{iKH^keojHX6#N|<+V<1Cm)}y6|W8IWWDM?D5t;#-?nhz8xB0V84Deb%Zs{S;D$Z?(O#Bt6#j$ zAo|!biyWa=mD2oZ|BHTs6e3%?=~{O$99$S#4L_WEUHp=$u`w;|WB22O9Jj-r?iG`3 z0rjs(%q{$HpK}#L+-VtaGOp{^a^LSCTq~MuXV&8i6!V@Ff|sAwMdVZ5)!i*q>b7ud zqnB{vqs9uwBcIRdxmiDhH<2pnWRrOCI$ej%)Nc50lBjPAZVUgBw^m2Hmu}TchweVn z{o!R@&nrH|fiH(^g&ORHP5O(G0{Pk5yll%mBmf&TMml-OO^V0Ivvyo(Ww^LC`f;$J zFIH!gBp#?7+Xbz%VyNIewjLzv)5eqrJ-m&(pnqJv(WvOdZP4#Jygkx`%FjWF#` zbsv*TJ3q3Gf7A}D*EYJWG66L;f^ZFM^fC8C21I0m)Upw!Kvp+w3b@cUq|uf%R|ctH zu5S1=UJ->Anc47O`Vzz(wHPFOnDPZ4i+MM|^dy&aOux(`*^nfvEBH?Zaa5UJm{_zW zH=$Cimu9?k5DliUxXoW~Gd>75X2G8gu3D-WnoqBKoIY_FvBN{s1{AQlz6WHaL7bD1 z_VI%9w#yc@X&_69LJ_xEX|R8d_Tv;gccvRDK3g|%nz;H}$otnTso5aVI)M60R!dU* zc*Re7c~ksd&CzfZUn>3FR4c){wq}YCC(3;}I*iX1YPJvN|`{xMqNVAUw92bY7 z*$di1VPZ5-@`wj;xNEJf9cwG4tEot$4cE7vja;2$a2qA-PvZ&BQTe+1$+G$x$Ld2V zMyNyNSwrb%CX2U=q z(!Kk2X9)4g5AH!jLcF2Xh8%C+$$ObN8d&eYeNy zg88BhH7)K}x@B?=JIS$#I6T{4CBpoOXR%zLfa{L_aHB&r z89Es#!A2txyfnhX!yVcy&!d7^nv!)B+t;#nsT6usM~(37pbx;`_NQxv=zh9ofI5R^tzYIa>yO8=`nBgFflGIYd~c&8u8<K-~{hvyqONJ32FP^@_)*wd$!l_)H1v7PnOg2^2UwV`u2H@z+%09 z`*zH>6=bMsSrt)W<;3N%(h8M^X2=FXX+y73^dXYfAoSCxdKf_Q3sR$*ojrv%k~omy z+#<=uvPAYZ4o<41Zg(lb(wP44wW7Th<_&3ldph7>KgaD=p1p{!D4?n!3n<$*V~jWF z6v#M!p8JX&QpQY@7(j3K`i6Z)-BFwNB%I9~hzqF<gi&e(?*4Pq^sbc*{)KO%(-Fs$%w|Hw zW$Onfgx{rOo!g5c4}j}3r6}>zQLst=`(ZZB;OhZ48F?%V(qePEYR)$u)w+p%{KQNY zwe$^u2pzC4%9EQ6U*#5AP@sE|k)GH-KU&<$!u8RGW_#QRNAv2>AH@oFQfElSuKhoe zhz`$ghb(-qL4x>LC-vb!ixE5kE+OmMq~Z-PZ^oP8P#+83Lka1EWBrK=Dh+$ypgEK34-1ccEZ|9 zZ3K;Nc(w|gK26v?N;M0~<8QyNEsqUHKd#Z!oggZDa?58VVFnX^4D|+}Loh#_fu76> z_|j{OVnakJ$o@%^4Br5Lma8%8_rdFgwZF@IqIy$L&5NuWa{P>1!Q@&N3DG<=ry_$y8WDOyR z{4mz#Uy*0{LHT6VXRA_R0B5&|;kL*r5o0kn`OV=;I4`>H^P2tLt6t6EXDT$2#yDL}c5)=QIdPJ@onc`4fX5cqRTr2K`GLcm{nk zbU&9|tJyj7^)s`$rSWm(FNf$0@=uq(%lDk3#^5Y#I*{-f%YXLbx9hbvvjV9^U3~er znRzbB7z5esLYO>gBJd{s;Y}l9j7#<&SS_%M$PU&S$@sEb@)O-euG{mfQx#2vHr4sM_}_E2gIO@C2O5xSE#mVuyNEnOgh1hMYD zZF(ixz{McPwd5e;-W|62)6=uhm88_d9LU-rlkppvEkhBm35v6N$p5f9dHZTMT=!hi z)8o+&mD#x&DG7W=F%n?M1ZQeeY2{i(y_ZpyB26Wyl;>6ySDOtHYmqP#4 zkzK4>H&tOlvv*tIjC({x-R8B~a2|vS|Au1{!9nK=b#~;5fr$jFB^!Q>$H_jBx7CdU zPBON(w#U2hM=*m9-Ior7kENR#AKqtrZ#Eo6kuBhd?D5J#iha3W<;JAhal}M!rC|#*A?(t>+gQFCmK&Tx1!@$TBwHeDOFvQOQt> zk*0yJ7kwBWN74u_FqoDM2GJir=t!8COeC4f`9}X+X9{)n&+LjOJH|D;?$UKTh3LE$ z8P&Q$Ydg;|i~N6`JRT2Rm@3(vlM4$qjU9X*%ch_nSIaDUCeg@GG8y_ZtK6;#5M=BH z+v!-&?|XEDE)G;&&(u@IHZKn^MwU0^4~{6y@K2+k!N4OzQ@k;$XNHehW(@9QUM*4! z^Za%u@0FefNQlk5=h>(25et1|leS!-nFNms>%-;3u+N`G!WlJPKSdeAmceIzg#lf( zubM-tTBF!n%)lg#UbE!3Cv?>gFa48%seN3YBCByI6Rm5<&guR5h$EBT>hdRV^REYG z;w1|*&(0{6Q&5So?G&ORRrhza1S%P6=@3b(bKMi*{rUpjM^T3D zhs?fDC(&sQQlHBOp@w*_CR5YQ;Ec^)mF#E6{uzGJXV$73$MOUdP`|*<4pB2ghP0lZ zNIZlCspI2_K_xYqMg$t5TrRU8j@SG+k+1K+SMElqyPs(JB6oJrGrnq1^otvl&E`$m zK~>lVvT$zS}(EY*!+U&!f~gp6Iuz|Jrh?<{9F!8nj>5fu)7)+;3wdXRH3(mFIVAzdPvE zy51igTPu1K$83y>5sNx*^*xEknmyws+&txPe4px#*hWX=dB&f?pIwN$$!On9PD^RO z*Vj7@f>Q3#@NMCvF`!pm3(`Uc!#=Vw&$G-u?D~@jKVixiP8ntlZzzx09PO>!I3N8` zBs;3k)~#)7v(q+S`jH$r0bi+=vCu34J4DXo$^H9&;KTjqTo_&pa1|ay)x52|rlzLH z63oact@l7X3o;?NRxbho&M;iB9{t_(6L*gFE@Q+6o(At8W8-(^2HjCe>UpvG;F2&> zR0DMi$s}G(cok^;Iv=w{8Esa)Y#HwoA$73~SBZ<}dY0AMRdQ#xCfvg-gVZs|VGcWN z=ILq+zsGf0J-o<5hw);6l=%i%ZC6KQ8Pp4y!32o+V2%Qko2QnR%-{ty1=_|LH~^_B zg#xaeiO_DNrw-C(JjV5@6~@41}^JmsI~;( z%-Q6rad7p_NX@Rc$|r75w*JlVe7Ef8YZ$XB>wf>kLOMvsJWbdzj0mOO?x)R`+0ov= zk86j2|@OgP=EP8W&D;YVV51CeLd%VN4)kc)B!!l^|m2@&OV>}%QmB|=U zCB2UM(|Y>@rFl8?TQW_({0$$+_uA!gfXGUe|1Qpcnfd0z{Ro$_99!h{;_F%|8JVcY zEMqA(29Ux3Q$H|f!aZLKGb|Va8+|xA$HW5I^o@EP+}OEqWk)t8XwgOmG?S~TX&%5^ zJXvy7#bk0T_I2nb1$8nn=O{1j$1$n7da1+`oST2vluZ7c=Kap&o04MJm0=@VLjC3z zG460^rH|h#$5QsZnb;}aN2j72-MH5hJ+Q)u(QwdbW`VOHf1iMgwSJ$6=QwXDnJeI^ z{E|eJXh+t6hHc!+jQpsF{YP;osxyfdv}@3~v4Vy_fUagz%~vBwe<<33x(!a<>=Zg8a1#;MfD&pxzK z;YGXV;DWE?MW&!RlvHO2le&@9m;kgpXO~zv(HQ39Y;n|M;}!yvJ{#`9D0H{X-xY&! z>()^VTxh39?BU_cD@jE^+Zjl1zMxu8xV<8=EbRIMIndBi(E6%>ivImw7&lo?U0pZu zfuresmkR;q$SN4Yfhhmh6+P+~Jlr$h5fv-4a;Elv3}hWldL2_AdJ6`#Y; zf_3p|puF(vt8o}gK=7B2+$CSfkLUd8>x(wGa1_8$;t*xtO+(oSx&GqZw zVb?tj*Rv{NGQrYO@lN4gwWdWI5zw5t+DB<6ch}O#OtEftj!g=%#bDu~%vxQjCa6i) z@hh`MFJeotG-jKx&rnm}yq_vl6E&;5+@h0Z9W0vX`^Q(uk>IjX3mK#zPeB-Tw!m?$ zex#ARt|kkax~L37rCt zI@Wv*JLbW{X;=J;a{Hb(m>^vuPBmvLi&Ip_3POQ=>-X@%O}_lkvK@U&MNNHhkB}wc zOvL%GMyEP@1}zhsonmUA+!OR3D$HQX;>(N?UKwM-eg#(&A58QLXTdo?!B9*?>v#}g z$~zFV$}(E6d3jfRG(4?+HLxeO0Tbu3G>z%$RV8jzs^S%^08CJtKzB}O>={fKQVaSQ zM?OfNd})W#cY9N3_2)l?BAiRM1UhG2-c8G(Pl}rp@5;+cPwv?#kK?Z1ApW&Vy*2Y) zP7}RA71$p!l>Wk-pZTtdvbzLf@1X~JTprwgc*gd#yahS>(nx8zR(X_t%{l~ZYnb4i zFDn9%53UT^mpCuVuMlc&yRwaeS8DAQR?Rj-82m9!a}x!L*VZLrJ$6U>J?>h^2Plem zSZ7k_%CdBt?nYZ0@;Zy}--M4^LKuMlbklYYN)9}0@Cccip@MJzeIq(|nl7{Gb-qbo zSC0PVjtuY`g~glgPuTX)6TU9|ebsDtUem)HOFeOoD(Sy>C6p%{oWi|!67bf=5aNij z#|~>$Bf561YZn-YSzK(0pojG%`-;Xg6XXJ}$Vb25{+8+V;u+QOn-t^=yvlP&8juBH zu0a+ArxY`E{DQ}#@bl+?pveymP!W2CBUT!k{LYpd`qLeP{u3quHCPsgjv7KP_#vni z-vu?4ySr|oKZmp?q@aDp@+VLqaq1Pm6iF-ZzbaiH=J-ih8QmYz<(IezqV?n?H{#k7$c8f)jRO}yZcA#Tn?QpC>E5H;? z8K@t(v>bEWjf{CY^iQ$k|8p~D|M{%oPkraX_jnXh`b>??2#;KJ=L$eQeP~^k= zA#iF*W(B=}Vu6G1_+PS0;u<{y5^Z(#MJ=zac<-pNmG$2$MO(c1$JehaUHS+cNqrLJ z;PGxdcl8?r#{t_;!C-|+pwaCJH`g0U!BzJW2=Vp+PmVqeZx-T|QHv{~xJn9t z0(XfRwP4DEqRSY8nu7@zJ9jqfLPtz_pHHy3TXooTa}<*bmNZ00h5O0iqfWgnQR?X5 z!C59|JAFSOp!<898Au?K^|4KRUt;!h%z2NuKPgrk^WN6X#n)2p@(aD@tVCqdd&HGS zf5pVabO69>xUSbD8KLdMAAfLD&E!34rio{^TX+s){mPT=ESUiRU6{ZcZ$Gp#BL3>1 zKFhiNJJ2q9)}rH4xgG45|V$Pjwa2mQ2C)m*#ncgrxFs^0`Qn#E%oOh z1_!n)!;zq#`T$)XACX+4q=d_TJwSU6?ClS@6wgEQ>VJwWefGn?w1~lj{o|xnLqz}E zgj{3Irq3foDJzM-tJerC@ui_;UxBbX$9^HAz0^13ZOP@~lhK?M*q^R%4qPPm?U`5` zs#NHSbPP)>h zs$}VOvIS?uxPf%ZM)=3}vV1Nv$}dTD@r@qdGUh-Y%iZ#rglyzwhD%CrJzXnD!fnf6 zej}|4a1$-uAZQhq7r8P8oBAQ7D=GY+0@gP*brtj~daS4EX9U*|hZWmy$IX8uCDFQ} zsTJ`Ljw_46q;3y)L_lpqAJScTJIub1BhQ;Ah`Zih>)lL?tA8wTu#*tvbOODg_~4$F zYV)Z2&&hS3Q|Y<(z#qmvJE5LYUA8~5n}gjv-DUqvls}SxIiPCtqyC?`Yz86E+iSI_Z~qK!olW2HN&@tMaF;l9+FEH!Dz|+ zKtiOTeoXU8q7Prq=ke&@9#LvN<&6?P+48jZG_@7kR|<$ff;DVVv=goy*DGHhmT$%U?<~*!I?v52un3_ueb8CdL$sb|XiA{v8Z9L1-&r>^& z?A?TLMN0mn#Z-(7c=}$#ae_@k`M3t9vzN1o|TR$k;yYPK>&}YcfT*PPr?4}vn%^%w1*z~84I~tz|C;+ zT_^TZf3TD|`OSHrA8v;>+eST|{Lsd+));nVGFQ)jZ@u-&iG$HFewX*`?5ES;dkPrX zZzEQL5-`1=BJVh^ValEmSGXWv&~GzL*^{B+&g|e@L2Cf=hjSrB)2&(z(D(rx85YU8 za~>~4Uf=!?e;v)}j~(zGj8XpN=vekzr>)PY$D`)_zKzba&S>L#@a|CS{a#fG?`vCT z3Uj1|o7@}%AAXG_8Ll$u!g-sB0to7!?~VF+$GC+;stM@!lAq=nAu zh1vSAIU`~RbqkQYhA+HlzM3<$P3pEqH%UYQ4vu0l?j+D3w&w#mPV{I09|tV&q4UxE zNUaRwghE4L+pQG?(D=QEx5;CA+sJc*fP||PR4ib z&j!cYF2S(oeSs9$RuH;X@Swzu(>nX}TZDPh?0?E$-Aob9v814p-vLF!6W#-ITN~E- zGcM@@^VO3-s&tSme_W>I$K;Gb7Yj&QPTvd2f=*TrTipV)jr@d+If!qNK!y4!J$VLa zbSkdlXCLiI^@R`QeEYLN$NgO2C89@cN`|BWYzo|nlzTgkE9?d$wM z)j20HJP)sXvo^nOHqj^P)dUzl;;~w#%ks-0cbG1nbnCv()Q$`Z)_;M!joTj03fVLt z6?F{|evlIr6g^hP189|%l(5AO4H=+1hvc0hZ`qxt&N)zO-SvW$)kbjm6*eRf;pe)a zLEYQ+{KrhfLRf0TGvzMzhCT;k{=nPG>n#V7-?GgokBeAGOynj#9g~@JSKof^I24gY zeMUUhni&U5_lnEOZOQvDKya_aXsxM7B0|G+^wHjQ+*(f+J^H23=qK~Y@oLkgcs|#= z&-6U;duxu_>O8^haQ04XwK0UIpAQCVyUS~)Vj)+E-MCmwR!)uq>J*?il+?yi8El?g z?hi#Z(3hW13C7?kPs7b-)gMMhiO*Mi)`d!^d8Fht_2}+3 zrmJ`W7rM0Uz+Ripx+Uo@1^ah`6?hP?iu8&c!;K|Xz=03`+5PlX?pzxXW+MyF;nFob zRjLj3?Z?9Yvw6*Zs*`GgrW-TqMReezw6xl>3pfJNG|BOhFz?>;7cVBDI$_xU`noU( z_W_`ytBCsOG4X{82zjf143Jk=6ph;sbNHV%76;JaRg|e%z2Q@iOWJz(lit`YOR+tf zE?P)9X3M<9>J%8Lh}e90gYLH^N!lNRtTz%@C}`0d+FKO*zDgxjXHh3jj)S8{+e#P+n#SJ z`w!eakQwwPJQAN3VgH@3F_%anxJ8)DV7NQzOS4lkZg7)WWwSi4^=YZTb#Lzz69c6} zFZY6+bA5tJ(!xKnA|ALKPT zK#9y`Lo-6p6%dF8ols?6T?G}D^H}@4nhmRkfaN{a z$>{_WYIpy`-sc1xi0d*z9c57Lm~_spEg9?YY7&muvR_&} z4Gmx=i*{-u=3MfaE+BdLiFn%sF2zG4sh?O1RnFvURQWnOF>s3>?5ND0^z7obojB@) z8Z%nZSt&*MA3~f0Vaa;meO~7?fh|MFEb3sl(fU2h0boJyQ`|7|%A~Y3BwR&Pcy`y> z0-E}|^@uPThqi{;_G*^i{tFQ4ehOiv4R}i5!+7v~#>Vsypu5rziw?$S;X2&j)oEik z**}+g2U1Myx4?t@eR%zZAAkk|w>S~&&etQ5P*plk#F^b!zxsi))cIx4r=qXJ2rbPa zm;k}TU^93{X4x8fRm?iKUdE)8Keb~HvA70^9doOG$*JXP%fb-sdnhR>xo=OuLW0WFl|jLU|&cQcSDRqbzfvHNW0NG&jLD{nSR&|dM56Y@EC zI~5qcRnC7>S1ecSvHxD)Y*bOVCH847zRnD^=iut$_G9oUCfxuSt8n-#+I_SKPq@S) zS%n8D2sST&#?D8eNf7k$bCPDOadDjwS3@!j=$9Y`0Qcmm8P=P+E}H#-8VRQSLDLRlbV1qg zpVYw-e3I8>x!?wv&3H~n>UE0c;p;Cn)qg)idn;%aOQYxsgdp=We|YvUs^3X%S6biZ zc!cT~7)ZXQb6on-qsySwY%-5dotl|ZR#k0;a@vncfQ!2W&;Yt5Th(kE(;6R5xo0YW zbFsVK_Xqb>hgQmRQ(*FsFhsBli$sxeF8s`zF;V`gpKZXI;{7;{;tTxA5m}EiD`#>G zH6E)#=Z6&e3-=+Rs<60vt?rn_`;TP-^lGg7kYAV4U(3P?1)b~F$mBg=IKho5a-F!v z7Wu=TKOYBX@>!6uOM`|>!L(u{KcvGwLD7z;XThb*o9J9yyJ;T!GXm1nOIK_c1GR zwY-%XvQ1p41o!xG^sH(2h2xVoBO2?eP@*&BeyPmDD84L#K=WPf# z8TyGv0y^h61(eq3ACQNu-Gb%%O9J4k%a;5tzy?&-(48bBEYU1`%gbG zsGC>T45lX`RexU%ZoLQ5rOW_y2nAt2g|a-??0w?KH_B{)pxIRYFn8>>RpMsVpM zm8^3bh+1-l70o#}0jsOMfdv{u7&}Nzi$Fd=7qw>sxKV@{o$x;!RqbKj@Fe zN9hbkCM9_e{JbBj{QUWI(H5!pceCn$t5#)lab~hTySpsjly$GDcHDM#7SQGec-4MT z!Wd}1HHDD?%U*?kX9zY3wi@6V@9XH?b#^YxyH8?j+b`Yb4E8%_S@_5Gk<$Czh2X#X z9$cjA8M};Tx`4Cz0K3f?LcXMCz?e22;ts5H^Y9>p<%O8X3q32{xK- zkM)6*AB~b_dW002fZ534bq=^B56E3c7AUdK{w_Zz)!~~o`O%Dr3rMwmarNHNnl&Oa z(j(D7kePs7{nz3L5%;Jk6!rBJ`kY){;!tpK%**$rb`-C&ez|*eY>~QTf1>H z?dY&!!(@*kZ9R?;C^JRxaWS88CvkVDLP#$Qu)^_QgpUyLT&6`40$woza7J_Ec?WAtNCJ8#SYe*+VcCV zrX{j4yn^fYe`&YX#_E!;ijn%||B&rb+bnmH^+IgQ_Ws;Po-h6yy&EpY zl6u}TA`Ej#iHpvqtX$wlV8U^)*Yz5;o+9eHF4F>HqI`U>z^vHW*v6wd}AS5$02Br$fi{k2=36!a*P!&48u&?K`A;9c9$PeJRq6VWP9 zK=;^a{4Zx?P2PZ@tf2|Av)V#zv2Is3-U5IO^FpD5y=HD%N#4IwmR3PIV*gIr6wPUM z`!~q(eZ{l44x5OS^fs7Hkb|yKz&y8P+(Zi*)JaO%PE_umqT?W&6Z!n4Q77-YesNqJ zXdCiv+YwBHB5`VZ`dra2jqBW7DG^8LZw{C2 zo<{1g;PXvP+=6^<4JqZ|(s}mR9Tk?a8+4BgiQ4j5QETh7y6EAvb{_@#?d?h0!xr%LondUDgJcf$*kM3oUVH%nQlwNguZ;m& zZZVhilVs=BO)7*;Fft}EVsb{_`Gg?;HB7ZF9BaB1pL(#=`U)I8keR~%QeY%46-xaX zV5=UGq(^?R?%AEbq36UNscl;|ZYx`lfW0e#u!Mlt8yKJRVzy-*k2%(B=y2A29V+yo zL$U@q?G?aYzK%_j2^vtfiFgl+d5yV7Ap1s=0{K3@ruCnJjfU3|zunH_3g`j!2Z=OC zN`FE%zuBcYLfGKiz@)B1>xEx{7c2DGbqrgDw*@KRJwIsd*$kv>j6tKoSyT+bUo+U8 z)#n!pEA;#HX2IYa-rS8(-vG7LPLggACC%Hhru%Rx+StvmUk0XWqjuQBQXGE~Z{$AD{ypZDpx2yC?l97w)cMw9e+76I?Q zovg*40zmoeqtH)=kAG>m03gJrIH4e;84(}G{d{0AN`*w$ zDQhpf?2=fsWM)r>teEr2Hz$eJVfV&lsWpz{m_tu^6%j`AKxc^Zb78v;0W@yuB)T^Bth-bPTcN*eBR-~$M z_3E;fu-v1>T^4L&dNr$y;VpOT1xYTvvnLP7={FDTJK$BKIt(pN`^LVA7y8C<4v*#3 zXecrDybr?Jlhh)I5Z`$w)z6waKNt2c*o>rUh18cZ}iM+oqsP5J@4t|z(7T&RuR z!+@Jjc@~TF*Umw<^i(f^IoSyOhtH%e`TEiTD2rjj9E9v>>brHxM(uwbT0(X0!hB$u z^V)V!fzZ}0;NpPs_=3f3c!HjnRjl%=U!}4enuFz=xtRK1vaWPo%e)PbZQh>OJ=Ycy zD}4WOC;f{;L38N@lJUuJ^_2gTNITNWK+pt2Chqj?Y2A!pdnm#0B-l^8~8lG zfC+7i>|l{@3qIuVeqjWF4B->tirnm#kf~R<3JNSzdny`0d~ts?te=IIN>^uow_sUJ zs)1JE-*~v83AMA@d{t+GIC}S$b7E~Ecgr>XWjhT~s!*#nVPhy!9(a9gV2AC9J_fm- zl#-GdkU8L5s&r%yEay*4ka2#4+8)CHU0-c<00U9jr;l1 z7#^+0i@+jBc%bnAh63Gt4v@rAm*Dp?OGm`WNE`=45Eq&6mJN|%Fo`N9juAN~kR{UG zMRoSvtoSdRBQs(ufjp9Q7!7m+j0dS=aOOMtOW`KV z$mU7A8{dQo?I8@1;0s)x{2M^~c$b?SYTBI^i#Wso33_o$nj>0eg8CQ&bv!*Kc$TG= z$#8Jc_mS7Lo{^Og+dMj)Z|%m|zd2OmW%FJwl!)y-*+JG)%=A9p;@l4)%{uMO(Fsc0S-_eguRQI*rKjVBLNY*0nB ze!;hN&jJ2@q26oeJ-7^9hf)opoB8}dt!$-LZagC23;i*o?$dZiI&6oot?r+g2w3ib z1U4Nj15^WoD%k#M?R5vRd2nRYG*<%>cyFV$b{bA(L?f6_03&S}jy08jX|e7GLrJYf zOE^A0fxUmjy`E)3mQzE)mvSp1*b6-I`u|bQIKB!^c~HQc%;`vK;2e0R*?(m%C?fV7 zA5E4TE|KZW)ZAx*9eL-~xN@Cqo%iFf;dos! zX7be!_toDq0*Z(KoN31&=n}pDkHv?g4{OGESj6gk?4J9KKa0IK_nwHvK+O?-8vDLE z$rli>-J7n?5bH32Iv(`N`e98-E1CcV#OvnY*`Uq%Q;e{2;ENZTwU(s_*?}L}d6yyc zdh#q8;;{gF8E80$O~Drugk2(l%drAjR6f?`nUz-Y6m(!1|8qCQRFhX7JriFW4%v6lXi^7_fm+#q9?FzerUpn`xr(7!{q_ zvEH6V7eshjdqU9#;Hl6+RI$uFL^{43!g4)6VGeFGaDlPYPoC&o}(?boQZK;|Ur@Wy>=SY^GECi|w%?{vZs z7S#iCiO-ia{-OzYaQy*tZgJIn5Ao!CCYkr(zaL)XH@}9r^&HNY+>|Fgi7+u+#ly`k zF19_yH-2RbH|;u{p3S}fIauNO$9HR#;r@uzvZ46y!vUfB-HZo)7pP?$zr?B#DAb=0 zUAg%T8CmrCKRl~dX`V;<(;mR8r5=Q+`Sn8zfs6j{Z>mURM%EPjNTO%`qX!1%6CU&O zzQ#v{2KUgurrCMZk$A|#Dd@m>r z47CX3rM)4YmBqFC;BR?#p~OFB^1aWi!w{|W)}AVU1Rh5C8OYp2uZV8bYUEW*jUM$> zTv4AI1YUA~JE~YT1pUsHK|GTN_14f& zO($Qj3P!o;N>O`<9v^>p!R8=@i1HBz*2plM4w@D5=Js0;c0}b^7BB=|PeIi2(hxb6 z^(F^W_V-TcJH<-lbMf#OwW9DS*e;~CCxZA6-28*UMKNb8lBkrO0|_UJVY(V2-$w3( zRe$VLy%lPMecQPFKa-j@ee~-TwFglSm8aPTjw5c?XeLu>M}M_JcHy;`NtsXAF20*(%|M*YR4~AQu55U`0;f z_V|p?`rYiUPZy$hk*p6P@El_D#}{C2@HM0-%}K90GbzqPLKfZe9E^ReK>X;(s$^Jbn1y_*4T z`0XgvwPS(>`Z_fpur0|(%jO_uz5 zZUn(_teMa4tc&5i6)!CdKy=GYglH0m!svA-$j`9D1ur%BC7)xknv(KmTca{3> z%XDnt5wS1`#imV-F=Dw^S0)SyjXta^0scCgjbn94qnat?;b(qMUK^doG?TDm+AhbE83M!e!tp+A}{m` z5tvI1wmm4&eIlYz^oASVDn6KDT28H`7ocui0k;5ogc zho546tF`eBAs5_X2=xaRSo`WI>*_f-_t`8-uOMKPkNVR%_t3<}6p3;ZLs~0}en0*A z;;+p;YpljAm4d8&IBh2NBaf0Mb)@T2&8HU}=?&7Dm^#Lto-pOa{H<-I@cs{8T9LPX zv(xsSP<`MaWZWAZHaf2(!I-?tf1t%G8jU_Z+fp^-w5lx=8mH$)jD@%y!ACd@)+)W- zE}j3m3mP7P!sYQm=p02i&BjjKfY4fdqR^|O#2LTq@2lWlR;@Ns)hGbcTawX66(iHV zMn_gQLkegy{j(c#nA6HQGuM8liBAC0H3fOfQfJW9nm6j4RDZS3$a`y6h`~ff^Msz* z@r48IP|8hb2NRk7ws0p&d3_8Q|C-ah)dsnm zGqrusSm%&f2BCCm<%0&!jHj3C{QUhR z5)u{{xRL8=g;lV$Xig>?`Tg)gN=UePddtjHpHl%R)>nT9yDqhQkPFHacuOQER)!2u zo3811k>|VfQV)*1v~Tl$+7i3v?!?7~WWd6)0M`z`iHW_9D(TkndsXdvE_aCO-lJ$p z z2MPbTx>XTiJ!WeazX13PBJgW$%yf#7=ET@nY1&7ivTh5%a1v*3mP zT=kGyD~I;bxVX5ckPv9{F{`5KGtzo^fXyB+j1j69S|?3W`Tsy@*Cw806R)LhK_!&wuh z!D++Zi>`OgpZs<`YE$tCHwv7|r1gbf`{`))q_s(VdwYm6K#bv(HwcTHnVCWGPX~!R z;9SK+P7-*|%Nu)re96(ev5*ra|huNa{>0Gu1 z-b3vet#$-mHnim-LP99I`0uHx4>9U$KPN+Y1itk_S+;LQA4oB>1C7JDXG>z^Fk;v0 z3)yLSH7>+8F1SBfh9<{xZTgR&2&wWZdvkAjUwhTfRph823MLx)>41zplEHfS?p^ni z+uYj#%Jn{oq7W%-EHZYjhTG&iC%mCdim>139c}<_T=a0y9{Qqu1RZPzH+KLTT*o7P zB?6-aI-T5(#UW)@pj-r5^WRR^lodLE#HbIePzE;p!ufMARACdWR%dFR%JZ`Ku6@5e z`LKAHorcU}I`sN1B3evBMYD)hECZ{*ua6k^oV%wvM?-if02+xHd7d>Wz{o$r##h9r z9yKaV-0gvfWy8`RDzjbhpc>@^>^Fh|u^X$yMtV}09__l-cWJpL*C!n3uG3_TX?*AB zrbHhc4>G>BF;RFpBzj-Fg9-p^pAiQH)&GN9L9~UxVXbY`COf-{o#tWjwe@GYpnYU5 z*yIbC)^Mx|hfkql293#Yx4hkk41J)k)JF5{b>|;@mE%vzU-bqANubt%g-{&AjG*cr zc|~z1T3r`WSGdN7P@%$0QUtOrxf!`Btvtzp@m7?m^kblCM2vqy(JbTNMf1TUH3Wcc zX;%r&_&ld$r2RWIMw{675GCqhO>1iB$bdapr=GtU?yo+*uzBhA%2kIsBP(5GC!kp} zp4Oj*_N33d=LduwKwuF+REU)I>eV;t>4Cn!C-V08fb;(xHqxSF?~T_FN-Qv#0W8Rm zKLYf*?zRAO70=HPvMdvxHy+cGDyvBozBxV;qVc2Ev{&HdgfbmlT6ry$A_HX#ftvY4 z#80@~fqd|nf-5ywiQ*}mvUi+55yizmEt;ejKA;&gG7YCDr!TY`DFsPm1lof^pjOd| ziPawLg7PCyvxitra`JNV|B>otQK+v|LY90}9J|$}WV5ytKSsm^gao4dUs@d~`fTGSO ztaSM-P~S(HwLFjuNmeP~Q|m&*9JNCa_&1-)8rt0hqd3+pSKjwG=X3w+kOF{Bp;ydS z@_%TmnolqG`2N(p)euT}nz_Zo=CwhAQn$yk=;KzXk~Ru}x*SVi=`0Wa8(J=Ttarp{ z8exZB9u!HQgbo*oOuB7jokB>`F0efi)yGjwuPx%AwYB?i*}oO3d|D6USD+rv$=^rbVlOG@F(>@srT=b$wJ_O^p{x-!F?b(Ng#a zIPF5Jm?cD#{oF|A{9iPTVzoDzV+Y!=e=NoiE9`qS8VyLzXRg6l2Fx0>xe%7+AENVC z?ZgW$UCrV@FgE}N-#)Rn@J)Ab@!k1P&nyv$dw^N%5qh?4wLfS)I_RVB*=ZWJf&#y~ z^w?_~n?&P@Ir+#yNp?!XG{5|B`B(UG$_;QNA!qfwh3@G9bhOOsnlL`(zQ68#0p>ov-9JwX0tPEiuVgGkP^+OW4#~XH1(bMxuoB~RRPY(lnQq~$J;Os7KLzeVX0EpE=a(ONOJEK-mF zhB^wS@MH@~+2h*kJeL{M%^$#{q5ylhxOL)&p7n1ts^1iZYV5cOFh1wiu))f*rC(JE zNAkI7N5JNv=}j7aI#2G9rt>f#BB@ovkIf}SF_jjwApU8Z&VR2|gkeiyw5Zl0+pvlEA_ngg2a@vVslE=AwVV`Q|9*6sR> z$phsXBv7wn1W<5KmU^Mxv>Y^IXa=oY#UM^lNdfTNaT}y!&49_L9O~yWSsW@eI94gj z+%)#7#|sLHTRN9cZF8wur^)+vmL+1=7C^Wq3uM)%B8C_F?ZEVcR2ZV$g2U7pmyj5k zocV_uVa7^tTITc%wiGzXI6O9c;`18!Xu^ZnOI6Y|=Qa%7An|7Jk0Ni;itY&w}ge=(iI)LC1b zk0QzcQD2Vl?JFe4TF>sj|I_U(lZ$wCz&RfycR2i%A#BxPc*5DnPK;v_BPzP5ZA-v! z7UzT~jUP32Yhqhh7Qrw|PHAU6ojm1_PbFn#l{>hvD6)PY{-?74v#5xB1Tfe;gJApI zq2=nI^dpl(0%R0UK(YJBj~_cmwV*u1ejtRn0$U*!EPboE4{HdJ%2B7GkEi;lz`?i8 z&iQ7=7&I0Y($gX%wQ?KMVPUHy7gLa7Wx}6R@p>8(=&op4?i6}|GIbKURv@JX>2h;Q=Idf!~IXZ`7U3Y1D&{o}Ru&0Wp54X&&2R%zmx-g=~?5P(-ox z0Y6vaes2J@ z8*{!wPy4@Pp*sRF=3fq~1GxW-`9GbohWC#t<5mRKi&H@?8R@n}7tB&X-WWuG@k2Q? z)2FTf58$0hE_tc$>suQ6IhXH~7}TN2>X87BceR&@(x)f3IJ2gD0x_ljmC8FB0ThIwv&Z?C1Itm!A7( zB>Yv4_3O-|{)vAQ>u7#Zz+ZCV3<9xTm*C!kL_kQs2DK|L=!}%KH%|m2e#syxFJa)4 zGq{Bam`c500nwp3qKgCQ`JfXu=XS^1#OgP~s-4=ExT>7k{+e7&cNOlD0-bzYY|!UM z;KAg*`2O!d=0Bjyo3J>3PK5NzgLWFspxetZLP;S!~e z#3Kr-sC8yB(5mxp=c}8qd4Ye^KtIoF$97l#>C||K!rvz%2LE85N7b`|E2jjxk+OC7+X$S}yQbODKnvkrg5JbfYPO`-b+;T4s!&xFfArQ( z_$oEW9%=CQMc?M~r8~QXY;m~#L0z7EM-Fk0X{0nVV7PD;5QY6m6RiJqe3;yLfPGF< zFzIpY%nd4PV-1UM4$0!v-c%VsZ{V=0rDL%SG?hR5m(AZ=b8o1MJ{7mVMXZzJG9R~= zuVJ=7vAKH6b)Ps4?~1`L*}38a&1VWUc7zYIRn~Wzr!RT>aAls0Qj?h9JVlH6DvH9-zuDV~%ty`CpLb?2p5EE0vH!L?MqyLt3s<^C20d77%gBC0a=~m(S&bn_v1Qyk@W7Gdqu-!NQ&y!S%Hp>N z|AZT_J5i8dg@OuFkm%F)-tn;k|7J~0TxctFQ>5vfW%0Y){yP8ci-S77*-Is} zuapOHtj&nLGh#fOe>)}P#c3jSX)qZEh>WQi! zcr_kR{-$I|gof{~HT`n}0NmIg(;>KNg!;|37{G z%L7$xgHdXlCKIoYgnL{1Z~Nh_cDA2|z9ZREISoihJP^nUPQQ2eqb$|EgtM%Nw4jxE zZD5aQX+m?2xO$8HtJpvuS+OeN1x$ke}o?8=m}{NWV^a?KfGzsUU{^G9~ZO z#0D#FA}sF1fYd1SH9JAX4bO4`)FY&7mHt!8<-aI)qkoyPjt$m+j3fU`Xr}xW5>GMw zSwf6NM^o33jHFlktgcP)0&MQjvX+2ljVGgA?_We8DhLw1AVt_3R?%!pZ}YJWdMME| zd^}2TN*;TCv?1K@Vq9Bzv|27aD8#_RRajg{hy_;uwq7=t%SQ_0cmD`}=zTRi(wZYV zZCB&}DD0N5|E%*etNujF7t=QwWp{YT8~?W>Vq&w{_`kT?lNr=D=MY!B@eHZB-vRr^ zDNj8UwAqmoF-*?AkG!A+x8Xi6kR(h3N^ncBULVzZWmr>>vO4GbUThM>v4cb#n3Wuw z{*YyNR9v-o!S-)Qhd6vWiuze}SpAF>HsIQF$>p8TLn(kPXdWP)$o!}w&yEpPw%lF{ z(J*`$V`<&-d+!B?(q5Zh4A}cs>lXLp1fJb1{Lpu z_&;?H+Q+xQl-y>;+NQnpo;TWrU)Wk zoBtKOtjQlK%Zxa>mu8t7{Rut)n(n~w0hsCr!^EX0s*+Z2K`q%{`Z?_OXN4r*rBet- zwys(KQ#EC<1dkc9omJ!1YJ+dU>(;t^2jSc|jGDp|%68=u3j8G=bkR0!@Rt5J`RrFC ziVhDey>*%IJ^5FU%D);FA!EfIpf-3UdMOSfKMPziKW7R>@bc@%s0@Z^=|vll%21L6 zEJxL0?&!b2A6-cXprwFbHxL>;eU8Sj)8~&}lKgXAJl3*2k}p2iQ;?YNPJqBkD52oagEHOafKcMULf92(GU%ZyEwCYQPNc!90Yq0%E5;7eASt*MD z{6VJS){dserk8%6W0>^cJP1sO9~hq@FxK!(E9^()3DTtoWp!*RzQAPygX0p03_kUQ5~f?%*)wO_?@G!mX=^VooJZk9 ztHGe_c+bKt=?!;9WBc<2u!@y6=8R5$zUs!LWXVrH220aQ>X+ zr>e-}q8rdf9Mi!YV%l&l2b+s*UgrsN*Qu8UXaE8tD{=QOqRRvcosgq^px#Gv^%x2L&0a zO$O!p)bC|VgF7j5XI0He(@RvJnq`r(-XkmaWF4!KppzkvY9u3a!o71#yrt#}*XCxk zxJ(Ptu~-2UxkyfqhXV}fJ>>DDcu-_wI5gOQyPQXNsQyubEOVPRp3 z2T5SV@6*El@u_*EYs1?FT_tY|y;vH5Ef~SaSnMiFp;6l=^M&M>+lKQ+Y67??=ra!2 z&!djRhuDF~2(Yk-oOwH~4+LIYBL7MJ%~j5q-$gE4zGm%F;kF1{x~k(^%e}?R%spgd zgIv(v5A!z~EH6+$^LUr3=@sT*CCgoO+h$@yK3)mYXlS@J-6<}{8^2mCYp~G70rb@p6s^Veduc1u$)RD z65@{oe^YmObqs#7>nVkr^ha6VN%c5f?-00T=x$E~{~2q=uR0z@+MTL%xH{rFaLx3a zJo7Ylp)EFibj=@m(Jwcmr)7mkUY2ma`fzaR^RMA@iA$-D!yZne$e;78o~jMMaX{y> znzp&o@HFXDNYW+7fO-oHl?|9=bkVy6^}blVknBaL6cG)JjGd93HE=zMoR zRPac(l1S}16kaz^a_Y^XJ&RndLrLjs&Tf3s`0_~Yp=9wZj{O{mrHjgseYdxekNe<- zoTH&s|HKQDGyUTFlAfM5!}M=R>ALQPO{+O~O5OI{En%oV6sM2&3vqO6M5XTqXQ_N$ zkn3@nyQCR2aB(Q3MJG#f2TeL0kX0AUwrWio_tcG2p1xQAmMZ z%94QFcO(vz9mW}r@lL?awV2QF!`xknN&VuJ42{T3NwNUn0U!hk;(Oq(onbl znR1+SEnGz>+D{KuoY=<`k4aq-F(=*qOzn4Z2W|f9MR(r4)$Pb>Gp9HemW(Lv&Ehv* zTt%m+F8mpt`z<`nn3Y(g{+iw6AoKlWqtQdk<%Qsvu6i8!RW}7E?U>mz=X;+AMq7Dm zwvGTjh|^M!qnstIBtS~;I2vWY9Hrc8FoTbbfs`*zUuGYEA!+#9^WKe2)S9aM$MdRA zqHQ)0k&m@*V8+@y3RbS}s}lBCjyaJJFV^N=k`xqCrM#x_!!7#!1*adqPxw(BJ|bb) z!xqL$A55ilCM87<<22t*ULRp+qRt)%e$k;+Q_&in$%K~Iq#fi z5>M!XJg7YL2&czy-y)ar&E?y+I;q69o9?t{Y$*vbQ>p(3**^&haV+zL(ddmwNfRMI zJ~ZuK`TYtf%_Zi>^JN1@N#rWm-o2C2It>23%BB3ow0qDqP_>t8hx`!F} zD>IHRF}M3mR0dYlRFu9rA36>T*RoN;Xbg491R~ebm%kfb$$Kfu*1<>}Z_O$clt31xZ}+O>g~4N4FZ< z99PNKF0;O+)8U;58Bv0F&Eg*(nZTzZh-JAqN~q@+BMnc~x?8haLu{E^qwe;C*A*`^ z!MzB%ArY$P!R4IS9Me|oBc3y==$3iq_anxW$B>V>xAe%cu;@saqDyLjFc0vL=uZ)M zp3dlc-1Y0KiFT)8Cvt&j4?0cItryUnu_J?(I#*%xez?|rPDegEL65x5;~Y6c%o5>* zC1PHcSI?Z7^dr`EvONpcQo_W5>|3neDG$yZ*KccypX0&kC~=`i)bo8Akx!D5`SJda z`XIUUL#IogYBVQeh$m!oSx}PF+2`A`7IRu;I1yq=>(nlb$TkgEyT$Z=Ij~XYG zGZ|*|;V|&&r6Z(P73Ixg?eQBhH7N$5Gg(Nzwp;xQ@KE_b^LP>Ux(P ztr)v%G#pJ9S9E?+&F?)|)`(hNfzPo{h#@a^TFUBG)H*FZkX{wD>UrLgS{)}CmE%pa ziMz4SiIFdA9aArcMOQRqoAEoq|L5Cq^vmQsIy+-- zM2mSM2Ql|2Wsi+wH*YpqFifCeo{b0RY@bg}r66DOR-4=9LATB;cbbe0?rJlKxNFGq zcUx#_l2r45SL5;km1b4`o9R3`*jmY+nkxQ2$4_KXx3zL8gsJ@m1pZ{c|@p7im& z9X``b6Rc-i^Ebg$KsRkiR8||VkgO-?y6 zDQWM%F7;g8Nq^U?3O7g{Dq!}1h8Y2}>`WD{LnqT=z0}MJ|DVLR=*y9*Ys)d8QjV4x zmhQbe?7SOXTni#j)JcLjKfecM6jHtTBaS907FU1 zuV8@dLCVDZT`~dNSK9~oZ*xv}wJbBWOZVjs+DN$#J_{!)83|7&ojr5rit3}o_6;%O ztLT%rMSLiAe!(m1)w4ThsAEY2D;iyAtK6#p{So4xcwb_>GqCDk^n-Z*OR26PKZ^G5 zOXc^nKP!%GAXeNn#8k)&t)v!t#1>h`M%5OAsjyY9%}RaBCnLfUekG0O-W<2|}(e9+kWbM#&n?;q^o+-Q;PH-eiC-=Ea$QMuCP!IV% zN}k%@jGFMkA50uayPQikxR09KI67#sXj71lmiB>-*{#sTC8|pgSsNo_X@w>+A=s0s zD0>Yz#A;y0fnDJojZy22BJh|@^^u_D)}=fUVy>w?GGldm$WV_-61b0R(mdL|!7P@w zS`%)oJZg@@%z4bM1gTq?CI2RW!N|0+9z7B=GU%qm1m#D~I1=Oz&+txCV_`W**4_cD zvDQ~x4MsyrIr)0hWo6fph+CK18;~*XYnZk(1$d?{+_Fb7RUL)n+XB9-oVf9JvEU1D_cdR_N93n**yACeNcplDS8E$D>?B zezS$y&1f()e8(KB;&-l@q8U39h13CFT>)x@#Rl~_NWYVTa{KpWY#RDb&Zp_C=N26-3pFUY8VsZc#Ht5-A0Jg_Bt|ARK1GLCOe(n36Y ze9xFk8I}J1(6>9gUIoE+KD;Kd;vB1&T*fJ# zQ!X#Sm*Fg4Fk_N}@4fNcsXg2k(~L<499-VW0sK?ZYr9SFH&nOp*FBxg;&>&x9q>E3 z_G;R@d;>Syd2eb$cfQ1bcEXa2Y48P{VM>0ox&lQ!f|v0$S%CGNJuU1+ad( zVV?a00^CG6u3V9Zrm?z%%F4=7iHRPqd1r67d_d)mMBXr#6LX)h?y93o9j}cXNH*r> zMF#t^0kYmtB<~9ex20G-U?N>KtFnHW@NKy{sh_t(5AhQ)%ayzSeX?gJNFXHrS_P>l zWk`?!HJVWtDy*5=+24Q{2CR9e>fh))tt~D0+}+)MjM`dTzrB?>l{Xs_9IWEvvd!G= zM_1_PQ)F0CWa861)0#`3<=kTZ&9?4z{ZqtK5pR=(gP)FnYFSXbs6bnA#bwe%A_dM! z;oDTzQgUyJkBqJRRZ&sh*#vuqpGLo?KDTysh>Y9v@bFYXhgJYmrlL@c?3y1WQ=~#7 z27i7>r{3YQosjeZA&83Um}f1jm3=STv-!&xmaKecG^$*szRdc@g)7V}DZ*eYRKD zen)2cu4R>T%Ig)v-J?3kneeRtIe?nT_2USYFclm}< z-3mf?BZ?k=j==c|O)|cN`qb|ZzR_=kwI*FO(^FHASH?cRr=~il;_}oaBkFxuxTxD( zgH@?`1$19OS|k~wF5yNhWCpx(UIVfhTC~-oCmMrDiESNr4rL>m<|FQKd^HLaNdp>p z7WCocl{)>V2V?z?w5-7VD1XX^+cWpanfZ7^*y^Y}v6tE- z@6B_g9b&lk6Tq=#&b_myhXnFOjKJb8vKkhYMFaD&YQvVFno`8IXr?nUi%@JY|K{%2 z>g0N#QP?~v&!7vtL?2Wh!vBkyV#d13s-#12nQ+B}`$6W_$9**4r7Gss(O=w9ff|$J zn8|qa*85VOj`O%5{)RxJf)4be)2j8XMj9LD)uv%%)n>BA#3)A9cy_r}W-0HC)(VG> z7@>+>{=fFV1e(hAeS4Ei4y6-L$(W&tN+B~RZERC)^IXX+B=ek-j*Qz>WQa{>nKMgC za?E7PROY#4w)yUNYdGut&bNMNt^Zp8wZ64J>zp*$@B57Ra}U>ZT~zy*^aa|R1;Fcs zlZ*SxIV;)CcyJ%Z6z~o=-kCz9Yhuq|^{vS%Md$xvAFhY$LIFz?#D^;X9$ zE+v65YqqgDw3E}`jy14v;6UCBCqUxYuMQx~>;dBVLZSeja>(lm9 zXSr9YKk`35C^U&|=(~o7Fm?|1MUG=W*COlMKW3&pX^DBiGqWm2=orG)8sjUEr$C+u zIeop&w|aWmTNIffejY8jA{LGu_^t04kDU$0M?^$?TFLB>3J>qN>eK=nd@^9^y0_Ms zmce$P78ce4jfqY$C?3rGx(6F^RqHCc!qOtoU%4lM&0O67Bf*4v=|dVUf=rlc6aCG> z3LRvlf?Y0hU$w!sf1T!`W4}BRA=)jd(rcFWwBN&rN-yIYU&*pKI**b2NrZ7B@~iOMh}5Kq;w!@vFoijiBbqn zuYm-f0NmPBlkkE&oKTR2(;|w!@v@4nv^^DVf>2tJ_a9G7c$PgVJ@A5G?;wOc%6wBUx z;K9J&wx{#R?YR=KKH7W;n(I87`SF+3w2Vgdp8-TSvFS2NJs-Tzu~~ScQ-bqK{0F>| zD@a95!bN&OU`BziAQ*sOgXjh1ZA$b2dDaCo9Lv3C2~$yeHf^E{6&=HCzP_Yzi?0B~ zx3VxaWJMyYp%Ix=z3q0hHy_;?#oGaZ_gDuB)Z*iyB~Ry^P?hxVQx-KsS3b*J4qRwP z@-cXoQS0*%!*xP^)uy#O`v9>Uz@Hb>8})Sn7qg zVfI>kN_W;U3t@wJ=5Yok#6gR0U(t&7j?ei`&TL>*hm)g>@W&>kTTtftfLrRDo1L|$ zqUV2WRSmz%txu#)I~DPspaJn!dTPZrcZdB`?VBktOD>3>KqhJJ=w1d#p}1x>38&8= z>u`8PxsT4a(-G9`5HsJ|94A_YSmC`4O(XWD{BXav5^kBeZLd!_Hu9FN&3-_sTv19< zf1$mmLrv)80&Q>)sDQ+Z^3d=ALn6Rj&O7EdP|RJMvq!$@tQICf-y zV=w()cRcRMS+240qr}dm9UKbJlue$9>F+=ufRlr0u-#_L?XE_$4p>O$!rp%C&wH19(D=c3rDj-DHSK?p7!8z1Ke zS*MfLS09TwW+0rjRE#3(j3N%L@RQoPgxX)ub^__g;Oq>FoHU9AqAXG<+5gS-nYo!D)CW8 zQ67)qf=*`771PtRty+fV5eX8y91}w)waPp2SOxCoxlJIzS?E|I84NYsU9a>IK|>yJ zoN+IML}+-pT6r7}XaB*=u^%orcwlStfbkL24g>;4pjW$Z4#bG8u_XSlM$FrnkDfcA z{DI}}JrgecR_K#DVSBm-4LMBn$egvXf|&Px_2`YcFF9!kDY#m~rQdTUyt0>>l+uXc znHMnMH*fLt<)m6*w)D?v%`LG>p5Z!uy5r^D{Z^~8GBVtpT}oLb zR7YhWhyJPP>+6>@GBSE1{hMa$V0irav4rp;AJniD7k<{~8B4NIzuFU3J$#;4GtSfg z1vkOoqVezv=h(}o*TcsGl)_Gr_eh50TgmIP%ZyZws=eNIr%wuNY6@Kd4;_V4x&>|` zIQIrrD=jes3V@i+@QjD`2oyVA5{JN^1M5LLY_HR&Pb+|biG`*2!KMT#k}X>ZKnAc* z5MY?;xfRSNDTGl}2YWEH^{;K97LEw zP3FA}p`$uO#e9^H3J+FP3udO{0#=v|-``kmGf|>AboMqjg00kQ>-J5v7c|}IfEn&0 zq;fDxpIvt9N3D&+F@sro;aVc5LRLYkc703Q>va8q-a z^dR$tprGu5GpA3B+Eka_bPOSBl~<2sIs5^(>3M`&H$LFR z)Td{Vc$Dihe3O;*5kmeHh_4`s-VNc>{GlZvZ<#aD+}!*=@%O>*{Y}lyD{hoNEUJdB z&rc;gaRv=wds%l3M90IcXf9rG+az!_;Z|M0;%g3JNt5v-k~>A|mz0qB#cczAetG&0g{dTW@u8E5BZuN8h1J*!M1SD0znyck0y{{5 z!8EYj3pk3RM;}KOgG0(GEoK1}Y=*0#0vkg`CIfnh{fw6y&2%Wz->S*oDn@o()aYBC zCv&b5$hYLsjO*0~gS1`0=RCSmdR=WaczJ6oKq-B|%&(qdXZWLKi$m!?VVM>!{Wmn| zd0o%iDx?x*H)^f3o`RDnFf;fvt@8iO?@3YLDo>efOwkB&G4&FQg6#5=+En{~d@<%* z!h}BnnZ)4W_i#rD4^kr2V3E4vR@w2dc{~ch7k7t!3W13;p71%6_kn_o@O$&e%A&o; znsKl8()<4+$RI$wU0TRsUoyqh9abCoDB@LrLlK&uBIs*l5uuJEMD^ulDTU~BrM83} zg|MMYHvi0N^YROq#wv5vU8`d?;?9LWzK9H^+{gVKO^2%{Bm)M{>0R@$TI@5t)i=g1 zu_ICX(OupI_ilDaG{uMV@4JO)?{Kg=q2-o9(Ypbni~Kj*jEQOn_m9=e~A*;iL( zsH62d>}Wq?qLTX2CGVabVv(qF(8^t-y_CD^YDR|W&RgrNNT+?mWx#X#va~d2Uuu~E z{N@;tE`5BENld$3^89%$AD?VAkKC0j-9|ebTA<&j1Xn|KD_rt`@`|gZKN}SCtbp|g zhX>n<1c&zxsagZnqZRzz5B+97n**E8rQ9XqMk(R;T(*%CY`?W1E^ci*6vivKm5i|8O z`>OV~Xyeqd&>L{$;LxSdIV=LK9w29$x#9}89xk1!Z;J8rtHXu5R0s!ifanSF{~%8Q zQEf(ATAKJOTv!`j?M9KM1SA;;oS$yS=jm}1UV3*2oA+ORAJ{kd!Uh`~%tL}4pr~`n zqx&tIKDCeg;mT95x%Z!Gf3QF0_fPA5#HN!Sg`+zSNOSti95X>KI$O)gQv-Ok)U~xA zZLBWjg2C1SA0G>RiUZe3c|%MvCa$s%;gyN1E-o@b0he|RB+I1Zk^)7%nI8YnmJzp; zL#**hsj1T`%p^ogc zFgc#{!5ADZo#)mO8cjLB(aN)H;w?U}z6SzG+Ft4xEj5mtIT4%~a zm|2j)lJbFcozzmP)hJIxHt!p|CjuiZG7>axm<5aaj zzeMaVOU;Xc_TX-Ope%kXj;f!LwY|4QE?R=6F69xiu!|bLO=DM+`RAX1y0tJK^k?VV zP2az}H8eC$k#4up#-WUyPxd#;Ps!%%Zy$Jf?3M#pc%O;V!JXcmScW_dg^ALTPpv1} z4=)?>T72k6Hx(`F94v;sD@#^OvGf84)1CD<)RCT`B%voh3|%SqbQ5yuoh2Gmn)%N( zW(>V&dKao6`q#hbY8BGQAJ&>^5sbr!QtucLGl#9$3s*P{qz}!j%-$Cc=-Fh>ozwVZ zBF*J3jmAl;D1991)`B0$6C}8TYBE$!zd8PcPy!(q%JiW2*Dmh)AmP?6-1v^f((yxn9Iw9x^P-kF=eP zk3z^t%1j5_aAzO-*s^JB8u#a+Wn+&^Wzs3UJ=HaL%x`s@hx?+WMnx0|qAZtV{L!Qyo&rcUW1CKc1l7p?Vxi z`(7re*ucbZzYEEoav9c?{jdbW3WPQn*(N5Ja16rw8eXM^W8`7wN=f>8T=sRGjkMt& zE~0e>jN$BlG7KkeFzXz+Ep0JhpIUY8Z&vJE^_*V~9Tt%=w*L9DOCZgVbbxjL8Xv$a zHmVh7wz)NEFItBLhM(Ck;vco);(5ocio`J}T^0%<>NOsPYv9yvEO~Q%DR-Wsy`bG} zr}4D{YeKNqbgWH`5^hS%lm|!6A9f^-InBf=JiA^qPG6WO8Y(4z_Zn!!v)-jCX9O$b z3O-rb*D;)V|3C>kV8(T?AIZR@P|Mm;5tTZ9LPmYt8KW6uedB?l`|y~F0j?Cv97(#7 zjHqIssPoLx@`w1vaxc`X72f)M1J3;&BVy?x5k2BtYM|iC`_Erviqh$9%D>XBnaGQE-=kR1D-%HY$ z92OV<dD4+U^ji%s!h%tJeOTUu#iJJW$Dyc?uUMdG`UK_a7#6Ao z&N|1>wp&P-eCQsM9wf_ocirN17t|z`W_Fg{66_kZEsTb@!|iB&4KHI}SBe4>uCD?uRHsXW%~UFY*tJNm?=A0Q9L z1JRvbc_sYzWe($!f5sd@`h~cAWT`<*vCdnK7hMC`0`fVWgKV#hXv#gJiMD@nG+8h1 zG5q ztE!Q7Z?3T`YbY&($=r!S(;TnQ*FfATu_v??)AZ`=&tG5Fitw;WicPz;K`AyC+zZ zot&OCj>Y0F*>7D^MTjNDMB2qV?R&2@ILz7%yBx$iWL1gd?@P6pVUhd7g@VD`^&^-3 zUcbJ4JV{t)Y!%>Hm!EWwd=+VoaS*MT)0t*x&xpIkUX+Sf)_uMCFj4k<3HaMK*Y%|n zWg1D&dk~y=7&nlcvNxYVKxh6#T7+d0uI7%5l~l8F9Q@i;iVP!4YVk{#_T_sayJ#?S zc~+Xa>o|V4t;SEgq!}jL&?&r=NzfUQM$;APN&6TrA3_J~$&E zxmvE%bzskW5fMcOsrm=nQ{&Mov%FB4Z(kY1WqvVtH^=j{D2c2px8!qu8;KYI54uD? zeyp=l`f2g1G_WR7wekOs#W}nm!=l5~pFZ+>AxqsBq zce9I6Imjc9{my}T%GKUFWly;s1gFVZ+LJ!C*Ea1+1T6&;Mg$&z<-~<_vgpqd5`YwF*G5@ z40b~7m~Sxy5ykd1Crd<75I?nYeIL=Fe1Amj-&-H0LR|Gw;0ygn_}|}hy{5z8pM-pW zerz{20o6`45K05EZp2Ow2MSX+AM8tA+^&TK2C*u;Q3ou^Y2*LWn!k-DouxC1oJaP3 zJkijg&iQ>NCU&K)YAA^r;O}h_ysaH&(#uGOa%WuUI8w4_?Z{se)WpG16?$* z*d(aOPG)50U}57i9Q3y-pA#&9b&0yK(!Io{bc`;}b^I|Tyt09Wzq8ugyXg_1G#ZU| z^7m&6i;U!e?A!9oN~$VEF_iIQ3w$0f;+i`@AV1;0G7GrtG7;zbEV#T4j4D^WH~WWf z=tK=XWktpPY35yZ618}7{=&ICdE}33UC0!#ijk(Cqa!i8&hv5Y(i~LH8y$eq5%TG3 z7z{>a^~3FDD0ucT{_5KaWc5YP3x-I|X}Yf{^NI2^q-uaj)wuyLGXe~N`f)!ar|)aW zZi8o_%2YL26)*1IM?>Y?1iF*is>w0}J=Ji(848kb;h_2~A z`cOy$Cfy}6A_8COzRJzPf!;05=8LkErGupD6-ea_o_~ zVBX3yVBg5=JPxI~n)@YmB?Ih})*?#rO(aTzjePX2dU@}M#WlP|Eg zcj$`uh8iS3E4RKh3Zm~zIi61-tjD8jhF-MPD8pbNgMxvXaJ76^02Nk=nbhVS9b+6Q|e>(`S zQGEs7)pSU~>@y@G{r&bA9(MN2AlT6f18;^9r*70sZa-aHb0L;%^KZC?qH& zi;Rn_4T?FA6LHS+xIy;?ab#Br$_7YHJk+0QPkAU4M4tjT{hMG#RBsL}H~=}m`PjS2 zRKtC`LPkR!u;u5gcf0^D1HE0l;6O$VeMmw-q+w#pIdFt?P1Kq!aeTDz!C|&SU_gTM zhh=A)>U3MO|9sV^4dHW$x4MT%1#rbGRWlT#JXZF%;_R;YBAb}z4ck>xWD4@g*XR>A z39siVeno8;SaxN|^h49<`zT&Ob(gus$STcLdTEWjdpaXMJ?{h?TQ>6A6PL+IyhnQ{ z61;6N`M_<@YIN@b%W$UHCL|wuG4Eg8`sBfQ!uQ@;`|SMu_P!Ull8c}=pZWZGD>pB1 z5s)NjL5Z!6nKjw3;8TI*&=C~Fq1~o@w~w5NeKh8&m4suR{5pv2*cKAxmSfp(_7ye^ zjj%jeXX^%vDs`yUZEbCt*4-Z-2@%@_AQlG(AT|=qHPq@Jk!G+PBG?B&8mj65s zmD0hO!Sri^hSk%);+vTuVfhZyz+WfZB!Xz~+-ccH>Ha#8`Ky^GHsjOnYHz?n7G4hv z3oGhB6W#bmj(V0J@}--r?&9N6AN1Eh6MdiZ*vIZ3YBBo8f{KcYeT388++W|ag}j|| zm<~9dv*_+Yeo$$>q+`r}%I`Nr6)g7GQa?4P{DclDKrcgu-*BMJu>&v)gk$uiGzkQf zLiP{gS4oWzBS-tA8syC2A+35dv8{Flj(*f^IQWp|;&#I|`{r82mI8tOU0qF$4TI#< z5XA~i`}1qB_^o@+vE4ecC=g`@iPdVTLO%t`K1G;REez%lAZ1PlqAUf^jjwH$tAkcT zx?~w|zRUu&1Z!cDe&)q6spC{T8iFj{{2PB%5mxk+5+n>N-ImWY@RD}@u{{@ohxK?0 za7)OuVUa$4OKlBf0=z~Bb}#or@o1*2(%l(Zk05!l0jh^z;eT!skv_XZak+D8l zMb``*yEHWAkl4 zo+HqY<4I_i#bx5zZhjEiN`Id#?bqpu_qASjicG3;+^KvS1QoPxs|m%hg_c2x$sDMm zoc#S*E)(-v-`v66Ftu-M5GiUCi-eH&GGcocbMo~(3iZet2;`^dN`|7?#W!aQKJbrR z{P9B&8m!R^?@iv7&w_HADhL8v5SM~QQ4HTmN!B?%4Q5JwQm(=Mm{X^TD0LchrZ;BV!TODvts3c)E4&7 zR--gNQTN<`-K}pDGjE2HqnCIa|97tfmn}B!DHX z|9yKU8KO*doc5E{tRjxp{6M(^4on{uYo5t$UU_^Tt(L0DcbMLnl6Xz#Te1+wnq>Eg z{DE$Ok@X0A@`V{<4Bv?Uxtx+p@11R#&FF}THE_4Mh`8`?HA(n6(p?MxBV>fN0PA8* z63(Ze9y97{0d@IbU22s1+}{GSzq{%Ga~J$Cum5k+;QzwQYcP;Shr58l!N*vg4=3-gQms0L=3!t=*)Wtb zyB(|sj|}h~qzY+&=`iY_V-(*vaFmbQ30Xeli4(q^%5YG&QWhXMH@%}9QxG?pp6~iS z5vSQSh${dhIeGuMjf$!1I9Gk1%jfHUetr!jc9Q@Wv2NXlfB}TVlmJAE_&&q1QESwVHhh*u4) zxq{X`iB!i0C!v~z&Mv~+YC;8X)H$_>1F@fpbjF8~ z^**eCge*Rq6X91R5U?Ew26#&g5scmBq&Wr&mJ!S$_^6n6=|kdp=p&S~H8fIUZOSl6 zqziHZ&k!%*CClr+BCF+(56^P5vi=ScAcBSkKkGCmATY2~pj$r~aMEcImbej?yO%-V zw0ouqimA(h#pMzgH&9ho#o+-@Zw@*U0Mw2G9PsJf3hct85Xo(AoCApZPQxnCB5;Jh z8d14)$rnMv@*0#Crh5}F#-WDU`RHCEJ5venGXf~fahtHE`-u04w??`C4;)UI8j5S6aAn-h+**!tK*>F?e zp$%Q+ISu=(l~zli6>+w#DvO=w`Y<^A+qXkDMTmpyvx6+1Z*-6F*ttkHT>?m z>+Fb=mkMPlBVt_>vn>U6)0Eagy zcXAbP31yWI-Yf%EHx7)etLySg2&1UxL4RqaU<*-FGGL~3K%$^t{1CAji@a}6AUV@S zE+oMKK(Im(ZI#2$CBFR(qO$1T-rgcY$TEmzD11>AEwdkc_h~x@jh$mbtEz^<1S=mk zhRSv~0YG~7;+*`QOKmx2Wo3ig2#5-%EGxKL)D+23r z5$|ph!uQzj4r!$!?vTXWH+Xd#^DBxIfCCqYhsd^Z&dGglX)M}^8{pkIhe6i~##LQN zs^maW*;(c34hcz1knDvR(4uuA>ERh1dnM4dDS5Mc3EB-yHN{@c6)Xlnx`%;y#}0Mp zkGuz|*Wmc@*}cTQ=AyAlT-0{3vhL|fSd~tX1XU25KZ)5lgrH# zo7UO7Me-2ncETnU0E8ZF{_dHjf&_vht`Jt+k?k^kVWGN)hDOke@G*#TgNkL*XpD$R z53#m{Qa)&Wcp1s{*QAda!9@*fhkcYsSuk$kF0Wzb9Xuoy#5X3B;hsV$ zyMgOiq(nI{_`1XO(y@ov*xaNwQd;ZcOsZaG!OQHViea}zENbc0w@5ya&fS#Mf(XB9JG z%q_BNB|;>8!e41$_C)Njoh1ahP=AgCVIJvBt=#vBsZ@>=nQDD14RaX{uS|i&0tO5F z2i8^L)t97zJ>&++6__r0MPN18je#^0B=Jn+z7}mR&pd+@39)LR*3^}znhYn>Sq#OY zL){gwOJ70Aai+N6Mhy-RK(VMpQWQvO;c?Rtp1lWZ#7G?OV?lKYoJYFIMhCUBOr%*+ zCg~Lw`V9>YW#chcg&ynnV0j}f6o@XR5E^QJM4#x{n`?|*GrQ}FI5{XE-cvXK`se5W v3!dHXo8j00C^G#2mwy`@!T*Q7`GTE8W1@N3;z3ghq&FC}^5vXMSAG8v0xT5V literal 0 HcmV?d00001 diff --git a/csrc/hip/hip_deterministic_logp_kernel.hip b/csrc/hip/hip_deterministic_logp_kernel.hip new file mode 100644 index 00000000..61753f59 --- /dev/null +++ b/csrc/hip/hip_deterministic_logp_kernel.hip @@ -0,0 +1,424 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright (c) 2026 RL-Kernel Contributors +// +// ROCm-tuned kernels for the WS2 vocab-parallel logprob backend +// (rocm-vocab-parallel-logp-ws2). This file is compiled only for ROCm builds; +// the shared csrc/deterministic_logp_kernel.cu keeps the SM90-tuned CUDA path. +// +// Every per-tile reduction below uses an element-to-thread assignment and an +// accumulation order fixed by (BlockSize, Vec) alone, measured from the start +// of the tile and independent of the storage dtype. A tile therefore produces +// identical bits whichever TP rank owns it, wherever it sits inside that +// rank's local shard, and whether the shard is BF16, FP16, or FP32. The fixed +// global tile-order merge in Python relies on exactly that. +// +// Tuned on MI300X (gfx942): 8-element vector loads straight from the stored +// shard (no FP32 copy), a 128-thread block per (row, tile), and a fused +// elementwise backward. Build-time knobs are injected by setup.py from the +// environment variables of the same name. + +#include +#include +#include +#include +#include +#include +#include +#include + +#ifndef DETERMINISTIC_LOGP_TILE_BLOCK_SIZE +#define DETERMINISTIC_LOGP_TILE_BLOCK_SIZE 128 +#endif +#ifndef DETERMINISTIC_LOGP_TILE_VECTOR_ELEMENTS +#define DETERMINISTIC_LOGP_TILE_VECTOR_ELEMENTS 8 +#endif +#ifndef DETERMINISTIC_LOGP_BACKWARD_BLOCK_SIZE +#define DETERMINISTIC_LOGP_BACKWARD_BLOCK_SIZE 256 +#endif + +namespace { + +constexpr int kTileBlockSize = DETERMINISTIC_LOGP_TILE_BLOCK_SIZE; +constexpr int kTileVectorElements = DETERMINISTIC_LOGP_TILE_VECTOR_ELEMENTS; +constexpr int kBackwardBlockSize = DETERMINISTIC_LOGP_BACKWARD_BLOCK_SIZE; +// Logical 32-lane warps on a 64-wide wavefront keep the reduction tree identical +// to the shared CUDA kernel's topology. +constexpr int kLogicalWarpSize = 32; + +static_assert(kTileBlockSize % kLogicalWarpSize == 0, "tile block must be warp-aligned"); +static_assert(kTileBlockSize <= 1024, "tile block too large"); +static_assert(kTileVectorElements >= 1, "vector width must be at least one element"); +static_assert(kBackwardBlockSize % kLogicalWarpSize == 0, "backward block must be warp-aligned"); + +template +__device__ __forceinline__ T shfl_down_32(T value, unsigned int delta) { + return __shfl_down(value, delta, kLogicalWarpSize); +} + +// Fixed-tree block max. Threads without a warp value contribute -inf, so a tile +// with no real-vocabulary column reduces to the (-inf, 0) identity partial. +template +__device__ __forceinline__ float block_reduce_max(float val) { + constexpr int WarpCount = BlockSize / kLogicalWarpSize; + __shared__ float shared[WarpCount]; + const int lane = threadIdx.x & (kLogicalWarpSize - 1); + const int wid = threadIdx.x / kLogicalWarpSize; +#pragma unroll + for (int offset = kLogicalWarpSize / 2; offset > 0; offset >>= 1) { + val = fmaxf(val, shfl_down_32(val, offset)); + } + if (lane == 0) { + shared[wid] = val; + } + __syncthreads(); + const bool has_warp_value = threadIdx.x < WarpCount; + val = has_warp_value ? shared[threadIdx.x] : -std::numeric_limits::infinity(); + if (wid == 0) { +#pragma unroll + for (int offset = kLogicalWarpSize / 2; offset > 0; offset >>= 1) { + val = fmaxf(val, shfl_down_32(val, offset)); + } + } + return val; +} + +template +__device__ __forceinline__ float block_reduce_sum(float val) { + constexpr int WarpCount = BlockSize / kLogicalWarpSize; + __shared__ float shared[WarpCount]; + const int lane = threadIdx.x & (kLogicalWarpSize - 1); + const int wid = threadIdx.x / kLogicalWarpSize; +#pragma unroll + for (int offset = kLogicalWarpSize / 2; offset > 0; offset >>= 1) { + val += shfl_down_32(val, offset); + } + if (lane == 0) { + shared[wid] = val; + } + __syncthreads(); + const bool has_warp_value = threadIdx.x < WarpCount; + val = has_warp_value ? shared[threadIdx.x] : 0.0f; + if (wid == 0) { +#pragma unroll + for (int offset = kLogicalWarpSize / 2; offset > 0; offset >>= 1) { + val += shfl_down_32(val, offset); + } + } + return val; +} + +template +struct Packed; +template <> +struct Packed<16> { + using type = uint4; +}; +template <> +struct Packed<8> { + using type = uint2; +}; +template <> +struct Packed<4> { + using type = unsigned int; +}; +template <> +struct Packed<2> { + using type = unsigned short; +}; + +// Vector load/store of Vec consecutive elements in 16-byte (or smaller) pieces. +// Alignment only selects the instruction; the values and their order are the +// same either way. +template +__device__ __forceinline__ void load_vector(const scalar_t* __restrict__ ptr, scalar_t (&out)[Vec]) { + constexpr int Bytes = Vec * static_cast(sizeof(scalar_t)); + constexpr int PieceBytes = Bytes < 16 ? Bytes : 16; + constexpr int Pieces = Bytes / PieceBytes; + static_assert(Bytes % PieceBytes == 0, "vector width must split into whole pieces"); + using packed_t = typename Packed::type; + if ((reinterpret_cast(ptr) % alignof(packed_t)) == 0) { + packed_t packed[Pieces]; +#pragma unroll + for (int piece = 0; piece < Pieces; ++piece) { + packed[piece] = reinterpret_cast(ptr)[piece]; + } + __builtin_memcpy(out, packed, Bytes); + } else { +#pragma unroll + for (int i = 0; i < Vec; ++i) { + out[i] = ptr[i]; + } + } +} + +template +__device__ __forceinline__ void store_vector(scalar_t* __restrict__ ptr, const scalar_t (&in)[Vec]) { + constexpr int Bytes = Vec * static_cast(sizeof(scalar_t)); + constexpr int PieceBytes = Bytes < 16 ? Bytes : 16; + constexpr int Pieces = Bytes / PieceBytes; + using packed_t = typename Packed::type; + if ((reinterpret_cast(ptr) % alignof(packed_t)) == 0) { + packed_t packed[Pieces]; + __builtin_memcpy(packed, in, Bytes); +#pragma unroll + for (int piece = 0; piece < Pieces; ++piece) { + reinterpret_cast(ptr)[piece] = packed[piece]; + } + } else { +#pragma unroll + for (int i = 0; i < Vec; ++i) { + ptr[i] = in[i]; + } + } +} + +// Per-row, per-tile FP32 (max, sumexp) partials over the real-vocabulary part +// of the tile. Two passes over a tile that stays L2-resident: max, then sumexp. +template +__global__ void __launch_bounds__(BlockSize) hip_tile_stats_kernel( + const scalar_t* __restrict__ logits, + float* __restrict__ tile_max, + float* __restrict__ tile_sum, + int64_t rows, + int64_t local_vocab, + int64_t vocab_start, + int64_t real_vocab, + int64_t tile_size, + int64_t local_tiles) { + constexpr int Chunk = BlockSize * Vec; + const int64_t tile_index = static_cast(blockIdx.y); + const int64_t row = static_cast(blockIdx.x); + if (row >= rows || tile_index >= local_tiles) { + return; + } + const int64_t col_begin = tile_index * tile_size; + const int64_t col_end = min(col_begin + tile_size, local_vocab); + // Columns at or beyond the real vocabulary are padding and contribute nothing. + const int64_t real_end = min(col_end, max(real_vocab - vocab_start, static_cast(0))); + const scalar_t* __restrict__ row_ptr = logits + row * local_vocab; + + float local_max = -std::numeric_limits::infinity(); + for (int64_t base = col_begin + static_cast(threadIdx.x) * Vec; base < real_end; + base += Chunk) { + scalar_t values[Vec]; + if (base + Vec <= real_end) { + load_vector(row_ptr + base, values); +#pragma unroll + for (int i = 0; i < Vec; ++i) { + local_max = fmaxf(local_max, static_cast(values[i])); + } + } else { +#pragma unroll + for (int i = 0; i < Vec; ++i) { + if (base + i < real_end) { + local_max = fmaxf(local_max, static_cast(row_ptr[base + i])); + } + } + } + } + const float max_value = block_reduce_max(local_max); + __shared__ float row_max; + if (threadIdx.x == 0) { + row_max = max_value; + } + __syncthreads(); + const float tile_max_value = row_max; + + float sum_value = 0.0f; + if (isfinite(tile_max_value)) { + float local_sum = 0.0f; + for (int64_t base = col_begin + static_cast(threadIdx.x) * Vec; base < real_end; + base += Chunk) { + scalar_t values[Vec]; + if (base + Vec <= real_end) { + load_vector(row_ptr + base, values); +#pragma unroll + for (int i = 0; i < Vec; ++i) { + local_sum += expf(static_cast(values[i]) - tile_max_value); + } + } else { +#pragma unroll + for (int i = 0; i < Vec; ++i) { + if (base + i < real_end) { + local_sum += expf(static_cast(row_ptr[base + i]) - tile_max_value); + } + } + } + } + sum_value = block_reduce_sum(local_sum); + } + if (threadIdx.x == 0) { + const int64_t output_index = row * local_tiles + tile_index; + tile_max[output_index] = tile_max_value; + tile_sum[output_index] = sum_value; + } +} + +// grad = g_logp * (onehot - p) + g_lse * p with p = exp(z - lse) on finite rows, +// p = 0 otherwise, and 0 on padding columns. Elementwise, so the result does not +// depend on the launch geometry. +template +__global__ void __launch_bounds__(BlockSize) hip_backward_kernel( + const scalar_t* __restrict__ logits, + const float* __restrict__ lse, + const float* __restrict__ coef_logp, + const float* __restrict__ coef_lse, + const int64_t* __restrict__ target_local, + scalar_t* __restrict__ grad, + int64_t rows, + int64_t local_vocab, + int64_t vocab_start, + int64_t real_vocab, + bool has_lse_grad) { + constexpr int Chunk = BlockSize * Vec; + const int64_t row = static_cast(blockIdx.x); + if (row >= rows) { + return; + } + const float row_lse = lse[row]; + const bool finite_row = isfinite(row_lse); + const float lse_safe = finite_row ? row_lse : 0.0f; + const float g_logp = coef_logp[row]; + const float g_lse = has_lse_grad ? coef_lse[row] : 0.0f; + const int64_t hit = target_local[row]; + const int64_t real_end = min(local_vocab, max(real_vocab - vocab_start, static_cast(0))); + const scalar_t* __restrict__ row_in = logits + row * local_vocab; + scalar_t* __restrict__ row_out = grad + row * local_vocab; + const int64_t stride = static_cast(gridDim.y) * Chunk; + + for (int64_t base = static_cast(blockIdx.y) * Chunk + + static_cast(threadIdx.x) * Vec; + base < local_vocab; base += stride) { + scalar_t values[Vec]; + scalar_t outputs[Vec]; + const bool full = base + Vec <= local_vocab; + if (full) { + load_vector(row_in + base, values); + } else { +#pragma unroll + for (int i = 0; i < Vec; ++i) { + values[i] = (base + i < local_vocab) ? row_in[base + i] : static_cast(0.0f); + } + } +#pragma unroll + for (int i = 0; i < Vec; ++i) { + const int64_t col = base + i; + float value = 0.0f; + if (col < real_end) { + const float p = finite_row ? expf(static_cast(values[i]) - lse_safe) : 0.0f; + const float onehot = (col == hit) ? 1.0f : 0.0f; + value = g_logp * (onehot - p); + if (has_lse_grad) { + value = value + g_lse * p; + } + } + outputs[i] = static_cast(value); + } + if (full) { + store_vector(row_out + base, outputs); + } else { +#pragma unroll + for (int i = 0; i < Vec; ++i) { + if (base + i < local_vocab) { + row_out[base + i] = outputs[i]; + } + } + } + } +} + +void check_logits(const torch::Tensor& logits) { + TORCH_CHECK(logits.is_cuda(), "logits must be a ROCm tensor"); + TORCH_CHECK(logits.dim() == 2, "logits must be 2D [tokens, local_vocab]"); + TORCH_CHECK(logits.scalar_type() == at::ScalarType::Half || + logits.scalar_type() == at::ScalarType::BFloat16 || + logits.scalar_type() == at::ScalarType::Float, + "logits must be float16, bfloat16, or float32"); +} + +} // namespace + +std::vector hip_deterministic_logp_tile_stats( + torch::Tensor logits, + int64_t vocab_start, + int64_t real_vocab, + int64_t num_tiles) { + check_logits(logits); + TORCH_CHECK(vocab_start >= 0 && real_vocab > 0 && num_tiles > 0, + "invalid vocabulary metadata"); + auto input = logits.contiguous(); + const int64_t rows = input.size(0); + const int64_t local_vocab = input.size(1); + TORCH_CHECK(local_vocab > 0 && local_vocab % num_tiles == 0, + "local_vocab must be divisible by num_tiles"); + TORCH_CHECK(num_tiles <= 65535, "num_tiles must fit the launch grid"); + const int64_t tile_size = local_vocab / num_tiles; + auto options = input.options().dtype(at::ScalarType::Float); + auto tile_max = torch::empty({rows, num_tiles}, options); + auto tile_sum = torch::empty({rows, num_tiles}, options); + if (rows == 0) { + return {tile_max, tile_sum}; + } + const dim3 grid(static_cast(rows), static_cast(num_tiles), 1); + auto stream = at::cuda::getCurrentCUDAStream(); + AT_DISPATCH_FLOATING_TYPES_AND2( + at::ScalarType::Half, at::ScalarType::BFloat16, input.scalar_type(), + "hip_deterministic_logp_tile_stats", ([&] { + hip_tile_stats_kernel + <<>>( + input.data_ptr(), tile_max.data_ptr(), + tile_sum.data_ptr(), rows, local_vocab, vocab_start, + real_vocab, tile_size, num_tiles); + })); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + return {tile_max, tile_sum}; +} + +torch::Tensor hip_deterministic_logp_backward( + torch::Tensor logits, + torch::Tensor lse, + torch::Tensor coef_logp, + torch::Tensor coef_lse, + torch::Tensor target_local, + int64_t vocab_start, + int64_t real_vocab, + bool has_lse_grad) { + check_logits(logits); + TORCH_CHECK(vocab_start >= 0 && real_vocab > 0, "invalid vocabulary metadata"); + auto input = logits.contiguous(); + const int64_t rows = input.size(0); + const int64_t local_vocab = input.size(1); + auto check_row_vector = [&](const torch::Tensor& tensor, at::ScalarType dtype, const char* name) { + TORCH_CHECK(tensor.is_cuda() && tensor.device() == input.device(), name, + " must live on the logits device"); + TORCH_CHECK(tensor.scalar_type() == dtype, name, " has the wrong dtype"); + TORCH_CHECK(tensor.dim() == 1 && tensor.size(0) == rows, name, + " must have one entry per token"); + TORCH_CHECK(tensor.is_contiguous(), name, " must be contiguous"); + }; + check_row_vector(lse, at::ScalarType::Float, "lse"); + check_row_vector(coef_logp, at::ScalarType::Float, "coef_logp"); + check_row_vector(coef_lse, at::ScalarType::Float, "coef_lse"); + check_row_vector(target_local, at::ScalarType::Long, "target_local"); + auto grad = torch::empty_like(input); + if (rows == 0 || local_vocab == 0) { + return grad; + } + auto stream = at::cuda::getCurrentCUDAStream(); + AT_DISPATCH_FLOATING_TYPES_AND2( + at::ScalarType::Half, at::ScalarType::BFloat16, input.scalar_type(), + "hip_deterministic_logp_backward", ([&] { + constexpr int Chunk = kBackwardBlockSize * kTileVectorElements; + const int64_t chunks = (local_vocab + Chunk - 1) / Chunk; + const dim3 grid(static_cast(rows), + static_cast(std::min(chunks, 65535)), 1); + hip_backward_kernel + <<>>( + input.data_ptr(), lse.data_ptr(), + coef_logp.data_ptr(), coef_lse.data_ptr(), + target_local.data_ptr(), grad.data_ptr(), rows, + local_vocab, vocab_start, real_vocab, has_lse_grad); + })); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + return grad; +} diff --git a/csrc/ops.cpp b/csrc/ops.cpp index 8ca3dcaf..f2716c48 100644 --- a/csrc/ops.cpp +++ b/csrc/ops.cpp @@ -94,6 +94,24 @@ std::vector deterministic_logp_tile_stats( int64_t real_vocab, int64_t num_tiles); +// ROCm-tuned WS2 vocab-parallel logprob kernels (csrc/hip/hip_deterministic_logp_kernel.hip). +#if defined(__HIPCC__) || defined(__HIP_PLATFORM_AMD__) +std::vector hip_deterministic_logp_tile_stats( + torch::Tensor logits, + int64_t vocab_start, + int64_t real_vocab, + int64_t num_tiles); +torch::Tensor hip_deterministic_logp_backward( + torch::Tensor logits, + torch::Tensor lse, + torch::Tensor coef_logp, + torch::Tensor coef_lse, + torch::Tensor target_local, + int64_t vocab_start, + int64_t real_vocab, + bool has_lse_grad); +#endif + // Single-node TP=8 deterministic collectives. #if !defined(__HIPCC__) && !defined(__HIP_PLATFORM_AMD__) std::tuple, int64_t> deterministic_collective_ipc_meta( @@ -391,6 +409,12 @@ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) { m.def("deterministic_logp_forward_indexed_fp32", &deterministic_logp_forward_indexed_fp32, "Batch-invariant deterministic logp indexed fp32"); m.def("deterministic_logp_tile_stats", &deterministic_logp_tile_stats, "Deterministic local vocab-tile FP32 max and sumexp partials"); +#if defined(__HIPCC__) || defined(__HIP_PLATFORM_AMD__) + m.def("hip_deterministic_logp_tile_stats", &hip_deterministic_logp_tile_stats, + "ROCm-tuned vocab-tile FP32 max and sumexp partials read from the stored shard"); + m.def("hip_deterministic_logp_backward", &hip_deterministic_logp_backward, + "ROCm fused vocab-parallel selected-logprob/LSE backward on the local shard"); +#endif // Single-node TP=8 fixed-tree collectives. #if !defined(__HIPCC__) && !defined(__HIP_PLATFORM_AMD__) diff --git a/rl_engine/_C.pyi b/rl_engine/_C.pyi index ac5708b3..19edd7fb 100644 --- a/rl_engine/_C.pyi +++ b/rl_engine/_C.pyi @@ -167,6 +167,22 @@ def deterministic_logp_tile_stats( real_vocab: int, num_tiles: int, ) -> list[torch.Tensor]: ... +def hip_deterministic_logp_tile_stats( + logits: torch.Tensor, + vocab_start: int, + real_vocab: int, + num_tiles: int, +) -> list[torch.Tensor]: ... +def hip_deterministic_logp_backward( + logits: torch.Tensor, + lse: torch.Tensor, + coef_logp: torch.Tensor, + coef_lse: torch.Tensor, + target_local: torch.Tensor, + vocab_start: int, + real_vocab: int, + has_lse_grad: bool, +) -> torch.Tensor: ... def deterministic_attention_forward( q: torch.Tensor, k: torch.Tensor, diff --git a/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py b/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py index 69696d38..88584c30 100644 --- a/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py +++ b/rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py @@ -229,9 +229,14 @@ def _native_rocm_tile_stats( local_tiles = z_masked.shape[1] // tile if local_tiles <= 0: raise RuntimeError("ROCm native tile stats require at least one local vocab tile") + # The ROCm build also ships a gfx942-tuned kernel with the same contract; + # one kernel serves every ROCm entry point so their partials stay bitwise equal. + tile_stats = getattr( + _C, "hip_deterministic_logp_tile_stats", _C.deterministic_logp_tile_stats + ) return tuple( tensor.contiguous() - for tensor in _C.deterministic_logp_tile_stats( + for tensor in tile_stats( z_masked, int(vocab_start), int(real_vocab_size), diff --git a/rl_engine/kernels/ops/rocm/loss/vocab_parallel_logp.py b/rl_engine/kernels/ops/rocm/loss/vocab_parallel_logp.py index 272ce3f7..5e4fe8e5 100644 --- a/rl_engine/kernels/ops/rocm/loss/vocab_parallel_logp.py +++ b/rl_engine/kernels/ops/rocm/loss/vocab_parallel_logp.py @@ -1,16 +1,173 @@ """ROCm WS2 vocab-parallel logprob backend. -The native HIP kernel computes only local FP32 tile partials. The reference -operator owns TP transport, fixed tile-order merging, target ownership, -entropy, and autograd so ROCm and issue #241 share one contract. +The reference operator owns the contract: TP transport, fixed global tile-order +merge, target ownership, masking, entropy, and autograd semantics. This backend +keeps all of that and replaces the two large per-shard passes with HIP kernels: + +* ``hip_deterministic_logp_tile_stats`` computes the per-row, per-tile FP32 + ``(max, sumexp)`` partials straight from the BF16/FP16/FP32 shard. The kernel + converts each element exactly, filters padding columns itself, and reduces + every tile with a fixed tree, so no FP32 copy of the logits is materialized. +* ``hip_deterministic_logp_backward`` produces ``grad_logits`` for the selected + logprob and LSE outputs in one fused pass from the saved input shard. + +Both live in ``csrc/hip/hip_deterministic_logp_kernel.hip`` and are compiled +only for ROCm; the shared ``csrc/deterministic_logp_kernel.cu`` keeps the +SM90-tuned CUDA kernels untouched. + +The all-gather, merge, and selected-target transport are the reference +implementation's own helpers, so ``rocm-vocab-parallel-logp-ws2`` shares the +issue #241 definition with ``pytorch-vocab-parallel-logp-ws2`` and differs from +it only by FP32 summation order inside a tile. ``apply_with_entropy`` keeps the +shared autograd path (with the native tile kernel) because the entropy gradient +needs the full probability tensor anyway. """ -from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import VocabParallelLogprobOp +from __future__ import annotations + +from typing import Any + +import torch + +from rl_engine.kernels.logprob_contract import LogprobContract, LogprobContractError +from rl_engine.kernels.ops.pytorch.loss.vocab_parallel_logp import ( + DEFAULT_NUM_VOCAB_TILES, + VocabParallelLogprobOp, + _gather_target_logit, + _gather_tile_stats, + _merge_tile_partials, + _native_rocm_tile_stats, + _preflight_cross_rank_agreement, + _tile_size, + _validate_active_targets, + _validate_invocation, +) + +BACKEND_ID = "rocm-vocab-parallel-logp-ws2" + + +def _native_backward_available() -> bool: + try: + from rl_engine.kernels.ops.base import _C, _EXT_AVAILABLE + except ImportError: + return False + return bool( + _EXT_AVAILABLE + and hasattr(_C, "hip_deterministic_logp_tile_stats") + and hasattr(_C, "hip_deterministic_logp_backward") + ) + + +class _RocmVocabParallelLogprobFunction(torch.autograd.Function): + """Selected logprob + LSE with HIP tile statistics and a fused HIP backward.""" + + @staticmethod + def forward(ctx, local_logits, target_1d, active_mask, contract, tp_group, tile): + sharding = contract.sharding + shard = local_logits.contiguous() + local_m, local_s = _native_rocm_tile_stats( + shard, + tile, + vocab_start=sharding.local_vocab_start, + real_vocab_size=sharding.real_vocab_size, + ) + m_all, s_all = _gather_tile_stats(local_m, local_s, contract, tp_group, tile) + safe_target = torch.where(active_mask, target_1d, torch.zeros_like(target_1d)) + # Exact: the owner's logit is copied in its storage dtype and widened once. + target_logit = _gather_target_logit(shard, safe_target, contract, tp_group).float() + lse = _merge_tile_partials(m_all, s_all) + selected_logp = torch.where(active_mask, target_logit - lse, torch.zeros_like(lse)) + + ctx.save_for_backward(shard, lse, safe_target, active_mask) + ctx.local_vocab_start = sharding.local_vocab_start + ctx.real_vocab_size = sharding.real_vocab_size + ctx.set_materialize_grads(False) + return selected_logp, lse + + @staticmethod + def backward(ctx, grad_logp, grad_lse): + if not ctx.needs_input_grad[0] or (grad_logp is None and grad_lse is None): + return None, None, None, None, None, None + from rl_engine.kernels.ops.base import _C + + shard, lse, safe_target, active_mask = ctx.saved_tensors + n, local_vocab = shard.shape + start = ctx.local_vocab_start + if grad_logp is not None: + coef_logp = ( + torch.where(active_mask, grad_logp, torch.zeros_like(grad_logp)) + .float() + .contiguous() + ) + owns = (safe_target >= start) & (safe_target < start + local_vocab) + hit = owns & active_mask + target_local = torch.where( + hit, safe_target - start, torch.full_like(safe_target, -1) + ).contiguous() + else: + coef_logp = lse.new_zeros((n,)) + target_local = torch.full((n,), -1, dtype=torch.long, device=shard.device) + has_lse_grad = grad_lse is not None + coef_lse = grad_lse.float().contiguous() if has_lse_grad else lse.new_zeros((n,)) + grad = _C.hip_deterministic_logp_backward( + shard, + lse.contiguous(), + coef_logp, + coef_lse, + target_local, + start, + ctx.real_vocab_size, + has_lse_grad, + ) + return grad, None, None, None, None, None class RocmVocabParallelLogprobOp(VocabParallelLogprobOp): - """Contract-preserving ROCm implementation with a HIP local reduction.""" + """Contract-preserving ROCm implementation with HIP local reductions.""" op_class = "logprob" is_batch_invariant = True use_native_tile_stats = True + + def apply( + self, + local_logits: torch.Tensor, + target_ids: torch.Tensor, + *, + contract: LogprobContract, + tp_group: Any = None, + num_vocab_tiles: int = DEFAULT_NUM_VOCAB_TILES, + validate: bool = True, + ) -> tuple[torch.Tensor, torch.Tensor]: + if not _native_backward_available(): + raise RuntimeError( + f"{BACKEND_ID} requires rl_engine._C built with a ROCm toolchain " + "(hip_deterministic_logp_* symbols are missing); it does not fall back" + ) + if not isinstance(contract, LogprobContract): + raise LogprobContractError("contract must be a LogprobContract") + tile = _tile_size(contract, num_vocab_tiles) + _validate_invocation(local_logits, target_ids, contract, tp_group) + + target_1d = target_ids.reshape(-1).to(device=local_logits.device, dtype=torch.long) + active_mask = torch.tensor( + contract.mask.active_mask, dtype=torch.bool, device=local_logits.device + ) + if validate: + _validate_active_targets(target_1d, active_mask, contract.sharding.real_vocab_size) + if contract.sharding.tp_world_size > 1: + _preflight_cross_rank_agreement(contract, tp_group, num_vocab_tiles) + + selected_logp, lse = _RocmVocabParallelLogprobFunction.apply( + local_logits, target_1d, active_mask, contract, tp_group, tile + ) + + if validate and bool((~torch.isfinite(lse) & active_mask).any().item()): + raise LogprobContractError( + "non-finite logsumexp on an active row: logits over the real " + "vocabulary must be finite for every active token" + ) + return selected_logp, lse + + +__all__ = ["BACKEND_ID", "RocmVocabParallelLogprobOp"] diff --git a/setup.py b/setup.py index a0c589ed..07853c51 100644 --- a/setup.py +++ b/setup.py @@ -143,6 +143,10 @@ def get_extensions(): "csrc/cuda/activation.cu", "csrc/cuda/attention/deterministic_attention.cu", ] + if is_rocm: + # ROCm-tuned WS2 vocab-parallel logprob kernels; the shared + # deterministic_logp_kernel.cu keeps the SM90-tuned CUDA path. + cuda_sources.append("csrc/hip/hip_deterministic_logp_kernel.hip") if not is_rocm: # CUDA IPC and the fixed-tree collective implementation are not # part of the ROCm extension yet. @@ -206,6 +210,13 @@ def get_extensions(): "FUSED_LOGP_ONLINE_MIN_BLOCKS_PER_SM", ) ) + if is_rocm: + for tile_knob in ( + "DETERMINISTIC_LOGP_TILE_BLOCK_SIZE", + "DETERMINISTIC_LOGP_TILE_VECTOR_ELEMENTS", + "DETERMINISTIC_LOGP_BACKWARD_BLOCK_SIZE", + ): + nvcc_flags.extend(_cuda_define_from_env(tile_knob, tile_knob)) if not is_rocm and envs.env_flag(envs.KERNEL_ALIGN_NCU_LINEINFO): nvcc_flags.append("-lineinfo") if ( diff --git a/tests/test_rocm_logprob_backend.py b/tests/test_rocm_logprob_backend.py index f592cd09..8cacaec7 100644 --- a/tests/test_rocm_logprob_backend.py +++ b/tests/test_rocm_logprob_backend.py @@ -42,6 +42,13 @@ def test_rocm_native_tile_kernel_is_hip_guarded_and_registered(): assert "deterministic_logp_tile_stats" in source assert "deterministic_logp_tile_stats_kernel" in kernel assert "atomic" not in kernel.lower() + hip_kernel = ( + Path(__file__).resolve().parents[1] / "csrc" / "hip" / "hip_deterministic_logp_kernel.hip" + ).read_text(encoding="utf-8") + assert "hip_deterministic_logp_tile_stats" in hip_kernel + assert "hip_deterministic_logp_backward" in hip_kernel + assert "atomic" not in hip_kernel.lower() + assert "hip_deterministic_logp_backward" in source assert "__HIPCC__" in source assert "deterministic_collective_all_gather" in source @@ -78,3 +85,125 @@ def test_explicit_native_backend_fails_closed_when_extension_is_missing(monkeypa contract, requested_backend="rocm-vocab-parallel-logp-ws2", ) + + +# --------------------------------------------------------------------------- GPU cases + + +def _native_rocm_available() -> bool: + import torch + + if torch.version.hip is None or not torch.cuda.is_available(): + return False + from rl_engine.kernels.registry import _rocm_vocab_logprob_native_available + + return _rocm_vocab_logprob_native_available() + + +@pytest.mark.skipif(not _native_rocm_available(), reason="requires the ROCm logprob extension") +class TestRocmFusedPath: + """The HIP fast path must agree with the reference op on the same contract.""" + + @staticmethod + def _case(real_vocab, padded_vocab, num_tokens, dtype, *, seed=3): + import torch + + device = torch.device("cuda", 0) + gen = torch.Generator(device="cpu").manual_seed(seed) + logits = (torch.randn(num_tokens, padded_vocab, generator=gen) * 3).to(device, dtype) + targets = torch.randint(0, real_vocab, (num_tokens,), generator=gen).to(device) + active = tuple(i % 4 != 2 for i in range(num_tokens)) + contract = LogprobContract( + role="train", + dtype={torch.bfloat16: "bf16", torch.float32: "fp32", torch.float16: "fp16"}[dtype], + mask=MaskSpec(num_tokens=num_tokens, active_mask=active), + sharding=ShardingSpec( + tp_rank=0, + tp_world_size=1, + vocab_shard_bounds=((0, padded_vocab),), + real_vocab_size=real_vocab, + padded_vocab_size=padded_vocab, + ), + reduction=ReductionSpec(), + ) + return logits, targets, torch.tensor(active, device=device), contract + + @pytest.mark.parametrize("dtype_name", ["fp32", "bf16", "fp16"]) + @pytest.mark.parametrize( + "real_vocab,padded_vocab,tiles", + [(1000, 1024, 32), (13, 16, 8), (151936, 151936, 64)], + ids=["partial-pad", "full-pad-tile", "qwen3"], + ) + def test_forward_backward_match_reference(self, dtype_name, real_vocab, padded_vocab, tiles): + import torch + + dtype = {"fp32": torch.float32, "bf16": torch.bfloat16, "fp16": torch.float16}[dtype_name] + logits, targets, active, contract = self._case(real_vocab, padded_vocab, 24, dtype) + ref_op, rocm_op = VocabParallelLogprobOp(), RocmVocabParallelLogprobOp() + outputs = {} + for name, op in (("ref", ref_op), ("rocm", rocm_op)): + leaf = logits.clone().requires_grad_(True) + logp, lse = op.apply(leaf, targets, contract=contract, num_vocab_tiles=tiles) + ((logp * active).sum() + 0.25 * lse.sum()).backward() + outputs[name] = (logp.detach(), lse.detach(), leaf.grad.detach()) + for ref, rocm in zip(outputs["ref"], outputs["rocm"]): + assert torch.isfinite(rocm).all() + torch.testing.assert_close(rocm.float(), ref.float(), rtol=2e-5, atol=2e-5) + # Padding columns never receive gradient. + grad = outputs["rocm"][2] + assert torch.equal(grad[:, real_vocab:], torch.zeros_like(grad[:, real_vocab:])) + # Inactive rows only carry the LSE gradient. + inactive = ~active + if inactive.any(): + row = inactive.nonzero()[0, 0] + p = torch.softmax(logits[row, :real_vocab].float(), dim=-1) + torch.testing.assert_close( + grad[row, :real_vocab].float(), 0.25 * p, rtol=2e-2, atol=2e-5 + ) + # Repeat is bitwise. + again = rocm_op.apply(logits, targets, contract=contract, num_vocab_tiles=tiles) + assert torch.equal(again[0], outputs["rocm"][0]) and torch.equal( + again[1], outputs["rocm"][1] + ) + + def test_logp_only_and_lse_only_gradients(self): + import torch + + logits, targets, active, contract = self._case(1000, 1024, 12, torch.bfloat16) + ref_op, rocm_op = VocabParallelLogprobOp(), RocmVocabParallelLogprobOp() + for which in ("logp", "lse"): + grads = [] + for op in (ref_op, rocm_op): + leaf = logits.clone().requires_grad_(True) + logp, lse = op.apply(leaf, targets, contract=contract, num_vocab_tiles=32) + ((logp * active).sum() if which == "logp" else lse.sum()).backward() + grads.append(leaf.grad.float()) + torch.testing.assert_close(grads[1], grads[0], rtol=2e-5, atol=2e-5) + + def test_tile_stats_read_input_dtype_exactly(self): + """BF16 input straight into the kernel equals the FP32 upcast path bitwise.""" + import torch + + from rl_engine.kernels.ops.base import _C + + logits, _, _, _ = self._case(1000, 1024, 16, torch.bfloat16) + direct = _C.hip_deterministic_logp_tile_stats(logits, 0, 1000, 32) + upcast = _C.hip_deterministic_logp_tile_stats(logits.float(), 0, 1000, 32) + assert all(torch.equal(a, b) for a, b in zip(direct, upcast)) + # Same tile maxima as the shared CUDA/HIP kernel; sums differ only by order. + shared = _C.deterministic_logp_tile_stats(logits.float(), 0, 1000, 32) + assert torch.equal(direct[0], shared[0]) + torch.testing.assert_close(direct[1], shared[1], rtol=1e-6, atol=0.0) + # A tile that is entirely padding is the (-inf, 0) identity partial. + tail = _C.hip_deterministic_logp_tile_stats(logits, 1024, 1000, 32) + assert torch.isneginf(tail[0]).all() and torch.equal(tail[1], torch.zeros_like(tail[1])) + + def test_entropy_path_still_available(self): + import torch + + logits, targets, active, contract = self._case(1000, 1024, 8, torch.float32) + ref_op, rocm_op = VocabParallelLogprobOp(), RocmVocabParallelLogprobOp() + ref = ref_op.apply_with_entropy(logits, targets, contract=contract, num_vocab_tiles=32) + rocm = rocm_op.apply_with_entropy(logits, targets, contract=contract, num_vocab_tiles=32) + for a, b in zip(ref, rocm): + torch.testing.assert_close(b, a, rtol=1e-5, atol=1e-5) From e9f1d2a5c67a283bd987978614b4444a9416c1f0 Mon Sep 17 00:00:00 2001 From: KJLdefeated Date: Sun, 23 Aug 2026 09:06:31 +0000 Subject: [PATCH 38/48] modify report --- benchmarks/benchmark_rocm_logp.py | 42 +++++++++++++++--- .../distributed_logp_latency.png | Bin 62146 -> 63649 bytes .../results/pr328_rocm_mi300x/report.md | 28 +++++++++++- .../pr328_rocm_mi300x/single_gpu_latency.png | Bin 72942 -> 92977 bytes .../pr328_rocm_mi300x/single_gpu_memory.png | Bin 120156 -> 131045 bytes 5 files changed, 62 insertions(+), 8 deletions(-) diff --git a/benchmarks/benchmark_rocm_logp.py b/benchmarks/benchmark_rocm_logp.py index 5830eb1a..6758bafe 100644 --- a/benchmarks/benchmark_rocm_logp.py +++ b/benchmarks/benchmark_rocm_logp.py @@ -840,7 +840,10 @@ def _run_distributed_world( "ws1-pytorch": ( "`pytorch-batch-invariant-logp-ws1`, the single-shard batch-invariant PyTorch op" ), - "ws1-triton": "`triton-batch-invariant-logp-ws1`, the single-shard Triton online-softmax op", + "ws1-triton": ( + "`triton-batch-invariant-logp-ws1`, the single-shard Triton online-softmax op; it has " + "no vocab-parallel (TP) path, so it appears only in the single-GPU results" + ), "ws2-reference": ( "`pytorch-vocab-parallel-logp-ws2`, the WS2 vocab-parallel reference operator: a PyTorch " "tile loop for the per-tile FP32 `(max, sumexp)` partials, all-gather of the partials, " @@ -922,10 +925,10 @@ def _field_ratio( def _range_text(values: list[float], fmt: str = "{:.1f}") -> str: if not values: return "n/a" - low, high = min(values), max(values) - if math.isclose(low, high, rel_tol=1e-3): - return fmt.format(low) - return f"{fmt.format(low)}-{fmt.format(high)}" + low, high = fmt.format(min(values)), fmt.format(max(values)) + if low == high: + return low + return f"{low}-{high}" def _lookup(cases: list[dict[str, Any]], **match: Any) -> dict[str, Any] | None: @@ -972,7 +975,9 @@ def _write_report(payload: dict[str, Any], output_directory: Path, style: Report add("- Measured paths:") for path in style.paths: add(f" - `{style.name(path)}`: {PATH_DESCRIPTIONS[path]}.") - dist_notes = [DISTRIBUTED_DESCRIPTIONS[p] for p in style.paths if DISTRIBUTED_DESCRIPTIONS[p]] + dist_notes = [ + DISTRIBUTED_DESCRIPTIONS.get(p, "") for p in style.paths if DISTRIBUTED_DESCRIPTIONS.get(p) + ] if dist_notes: add("- Distributed: one process per GPU; " + "; ".join(dist_notes) + ".") add( @@ -1055,6 +1060,31 @@ def _write_report(payload: dict[str, Any], output_directory: Path, style: Report f"topologies, at {_range_text(d_mem, '{:.2f}')}x the per-rank peak memory " f"(absolute forward {_range_text(d_abs, '{:.3f}')} ms)." ) + if "ws2-rocm" in style.paths and "ws1-triton" in style.paths: + ratios_fwd, ratios_train = [], [] + for dtype_name in SINGLE_DTYPES: + for case in single: + if case["path"] != "ws2-rocm" or case["dtype"] != dtype_name: + continue + if not style.keep_tokens(case["tokens"]): + continue + tri = _lookup(single, dtype=dtype_name, tokens=case["tokens"], path="ws1-triton") + if tri is None: + continue + ratios_fwd.append(case["forward"]["median_ms"] / tri["forward"]["median_ms"]) + ratios_train.append( + case["train_fwd_bwd"]["median_ms"] / tri["train_fwd_bwd"]["median_ms"] + ) + if ratios_fwd: + add( + f"- `{style.name('ws2-rocm')}` runs at {_range_text(ratios_fwd, '{:.2f}')}x the " + f"latency of `{style.name('ws1-triton')}` in forward and " + f"{_range_text(ratios_train, '{:.2f}')}x in forward+backward with the same peak " + "memory, while carrying the vocab-parallel contract (tile partials, all-gather, " + "fixed tile-order merge, vocab-domain LSE export) that the single-shard Triton op " + "does not provide; the gap is the operator's fixed Python/launch floor, not the " + "HIP kernels." + ) if component and "ws2-rocm" in style.paths: comp_speed = [ r["pytorch_loop"]["median_ms"] / r["hip_fp32_input"]["median_ms"] diff --git a/benchmarks/results/pr328_rocm_mi300x/distributed_logp_latency.png b/benchmarks/results/pr328_rocm_mi300x/distributed_logp_latency.png index 900ea3b82cec9590833317cd5ff860c4bafd4f0b..27d9a71de7122c9aa92f425078089d9ba8621f33 100644 GIT binary patch literal 63649 zcmeFZby$^a_chAic6U&sf`JkOQc@DRl>Wwzd%wT;oO7M?-}lFNU59IjE|+UP&;87M&N0Ur^L}tiPHMwihP8BbbQ>tr zXBFt^)=1OQExWpUC0-fN+`JtBJ7IH9%|_8&-^Tu$l^)%NYc>`p<~AmVI(zN(tgH>q z&G@)@j&TWd?7d-QV_|)Qo7?n1@8B}GGT=TaS*MOqSz{rsZcRtWm_+{lLyN(OiH`2q z2Flq}N;g9WY4$g_E6*~(#KYaWc z2_5UH4yNiCd(aOLZ=`#1>(gAAMaG`L z{`%{PVw6Wqy7B9a67xzwfBtN4?28uH@a_q7Ytt9Q{mpa*Xpi^DCB?eTP9C|O7JdG9 z$|1Rs&r&Qd5j#0AN6QBBj!sRfKG=G|ri9r(+HG;+nW#g`R;tGszPcO=4aEdmu{sfsHp|c zOb*3$7Tr5~{CItW3zN8W7Fx&L)J1FMBO2W#s15H)~h9o;SG&i;nD6RvYvUMv!c z=N|3s%b#ki`(FCQc8X@-W0<=*KjJnqm|ERU$xkX=$lU#BXc@uzBpUE3`fPIWjaAmHy4LPz)geaE{y z4LP>PaZOvdZXKU$cQY>YV?SZhajPmsI2@OB<6E(3ta6-6hOK^qT=Hv$NEvml^A|3p z&xjm5c1bx_sb+GBnwn7ke3!PfJd4|0G8;P^+gVzUtzx*Or&8Jt`78O3%BwaqeNNFU zPcv$W)y%YHEtu&)`t-2;`)fh?G+IO(xs}azZzS;mZnf@1!iHZvpbF`eNuL5cOLVk zm(J^X?zEkeF%Cg%z%FR=YVEz?4y4DAQC>a(jbQnGy zw;00*Ufgy_PHB3yYsao#J;TFv7OYz%=)SMRmYiy}_I}2193>Ze!muG$Ime>&w&Ubr z46}&cr}FZ0+7*nPq_hqRS=lB`?a$0TdIX_?OVcPGAiOB~_gko8S7ZsN*i zWy97ayE8m|sL_7zmu3D$qlQsyR`SE`?EAH~weQ}$7bo-dh*4W!w)=YeS5J?sw!{k- zjdqn+Ht|aMb86`Ot7n)5O*_iUdaxNz4bvPKrh9CurkODoo$piBI8^GYr;u+nOPW0< zBo^lsy(Jd1jwr`QynTCe2e+1{DK(>Elf+zyMgEsBYN-i>QExL<1-Q86c0T%OH2&$s zgKef}j7}px6+wIqJUV}41rqPyUp+lN9dq;NS8_?>b3e{pDR6F%8M5g8yl=xEQ8iqk z%GIkcWc&`l?5+$Rh)XI^kdb+)Xe;FF>-)<0!1*$r{Hl>B&X$m zN!EFuX7zavWA#be63Q_O=l1Q}w|4DX?5n5bmCOq+L()1ruf*ra71NCxWVUSC@=s2z zk5}i*UtE|aA*nc_nQM0;%w;l&f!`?N#}5l^3h7LmT`)=6%;L^!3X>(K8YR{E+JVk{ zEGo$NfByOB_AOgplKe2-n!R!MN${AAJ1(;CX9{}7D#$A#T->l{GL@1{L*ovzuVei>aV%0tyz|21=#u! z&vv_|^OVVM9LXE=Vh}Wo#}+NGuD*!dRG?;7K? z@p*q7DVR-Ew?h&W7zpOkb#LIWs0o)c^`Av0s^mrGsZMF#(bLE{-G;}nfv-?Gd-iW~ zcZdY3Jq~A)PHev%qPNM-PHUA849IlZhDO0ruP!Raii=38NbJ2VPU6FpH@mYQYQK1s7m}N zdAG^I=I{V*SM?1H$1nZ-`mxgZHddpr?{!!sZr9o+c%<(2#kf;tajnRm9?9?cpUak|#`9f!zuY>sz-9JX5{u}~W9Ju> zlB{bZDDH)Yg|r2m{`%4^l>SOn9=t8sU{tj&OiU)3Uiy{K=t=b-W@&bqY&NJo;kr0KWl~-^UVC1D8r9N3-LN_`5UfjFbbF)Lo)Ik>esyO!N=PbLSmJR7onGFWibL|r3b8HOn&2s{i@VG5F-y32UbF7b6PmQ9T zcwbcH`SH;%1EgRbq?)>;%fIe|S+gUKcmMOyie^ba09 zu*;O{{?>zJ=gF&A_9cP9sp7wutzt6MRxQn&PZbG3HL2jxKqTdM@fPkkl^3dklhe8;)3+sAdtP# z+0~`zj4~#ivvWD!+|kUYwFF9iIU{=~(ETX&K-5oevT=B?i@R>E`^2onkg=sMD_xg? ze(1V*Sl8L^k#p?@E}qk4J&Z43zOo-r&!@k3vBezJ($jJClREG%bs&VYJo`1OG zXp4yfih@49MibX?&M=xKwN!mi`FJ&+T+_yQ^}73y9`SkeXVa=JtZ!<~aM5}Hz!D4; zZIbX~OHfahEG;b^otUt7oYyV!>U+0}<+#*{ZGSyws3l#ouQu|jzkihCYdJ&Yqsv(q z*Zk#oW4i{h(NH&hBLD*WQ8+4Vi-$7X%~FSuN{v6<-!eKm+4%OaWesVDiYt~czk&Mt z@cw;^8L{gb*NRpGEIRo{{IP6hG@v7Sr~3xRMcGKt&(s#5`SUf<-I)$D1Tg9dC^1E%wWCR+FCz9zlhq%^Kq!%Ip4fQlv1u!92^|_ zKKZa-MsOi}Mf&+|7aObKJ8~r_%|W(oT~Af0BI;3PO(^;gl}t05Qnr;I{)%CV7c(F~ zxf1f2BO@ZNi&%*ynKq}KOVG#^C|rm7+JIn;|MW>ZRLJtQ#KNy|l_af%RD z+bh@*Y4tqmk*Z|jxTQ>lOfOBwqJ>&aPs6W(ghv7@az+n zs=+8|R{!hgH(O+4vVsIvbXSEM^XI;u;L^-`VMfd8Ln$-h zEaLG%2n2?ar-bhs9G@&T{#N2G7?x>DZS)kgY(v4e{n2rXqEO_%;Q%FAz_d1yM>ox+ znU}1>-F3TEJQ?`wBhT*=oBsX;;K#}!!1_BH8C85m%Z0A_4bkcvS~ojaRECPEK0TtWK!9SxW&Q&P4?fG>d*jQSm$z@<_6rP* zjyxaG9~q!krci?CojOO$wKtDzT3>W*zy!tg_MJNk{yiVF8=6PvQ>(?}9&9@#<2yGu zSD$IFO^QD9Jo&C-Z9>1kdT8bpeI3!6QufUZ&8#L}4wrY~> zdA+-j#}h!W@2^#RKBH_jq#9hb?frZTKP?3=xdqeRjHEjybnyO@(UqC^4G3in?cuUGWqe?dfSnR*M+!7byjcP=Gc$`l83Vd5r(+Xe&bygR_5+?^ zP|2q z0R({UD($yO`R`I6nPsI}o9wI@g?(fYcT&t;tAo_-| zS1&VZ%WFQ2oTF}rtfXERBH`9{7{Elmo4_sKCr?Z#+4t>x7bs}1<=NIxSL_?ru%nV1z$gumYo+E~0iy)UHFIoAZ(u{JH??M4TQ~UgrNkWq zY%;R>Ze1;AU_F~RKS|XO0@`S1a+pPD9jG^)Gr;u?I99!^B3O{Sy!XqXq=pPHa*QcJ z;-T}|NN3N&tMl{nJb-CMT31$usG{>V^5+Dqt4skv?!F;Tv6C9rH*3<$J|DoP?rG!L z9b_;#S*e`DXd3kR@l~=;H&He!5lxo>2Ri_<nitur;A(RGN&WH))lph$dAfG`ozb#;aZP->_~SpMY6w zU}QzR+oI|7^C9{Fjczo(v0Xj$ZUh8Mo>)RRxeJ+mJf0z}QP(^J-3>k**A%|v=O=#t zs|P(_ik}FoYy*#LeYCv5fJWL4pQ7d;sKL5bCf@;_rub$$?rf^az>e`mWuYcmqeMt* zYKG9JZ0FN|qGna9*P$;lDbOxD=7DV&Wb45s%v(Nc@Z6#J@%8$|E7a9%*YdK_CnqPX zm+4hRcMl90v$3&VS07Fsc)urfzANy{{yW6YaFqb}Y+T3Y%T?#g_jvUr*Ya^JQhL@K(@xUaYT z5{69?q_`>*lt_uG@X=gHR6WCq<`Zkn?9i8R%uY0EYjEZ|PQ=QCvKo_Wn~>PdhqM~l zP|zcLtVZReL=pCJrPU_!UoVPGJ1QI^QPK3(r7}OOWtxT3i?X9ns5JEE%QTKNr0AkaA}|Yl7+7G9w=`j@pa8JA!0pxpDrBZ858%J zW{Ad?C`PyD0Tjo8@|3x@|E{|`!|c?EegAwmZ(tzp+su@x+JcU@GtusW;oKt<-W5RcQnE!eV%zGZ?c++0 zYliNfC_9?;g>x2J``Mb?D5B{O@m9qkBPu7`9C|`I%DKU09*mxyo)*rNOD$^7BaUnpU5>jiY>GL^{u`t2LPK@}6F_RJ=+2sBkf9E&I`&#HHEs9jCx z_$Kg&?{ReMxwgw!tRUTY)dhRg$wRua-JlYx^n>D5llsx6`}m3TJA*cg_fJA=aOmQ5 zKQ`OJFY+iBDO4?jHvsA=eRZFogFG^t##7*?jE;_`I6PXeT6U8K549Dc#;YZtrcfvjwXX-~ zW(JbF#{#q*OFRAglk!F%R!&5WQ$?Me)ov*}yla5FdkpNi^#Hd6}?{j)46e+wX28g>4*;D6EFYV z1aG_h`^Ne)U!_7%M6E`daxJ zwEa+PwxVwuy2XH1$E|h2gLn7z4h<#XAqa7VO}{q;ldMwC%e5a>IVJxjfZ+u$$6#f! z%wQq;5;U_F&@nXvtHuxea8iH-c}K+NKCyaVkI@Ad>mCESZ$Q=0wHpHeEl%m%WkDbI zE!(=MqIznQYOeb6;STGmsD(nLXG379762-O=aI*nH|R5o*vaAzsTu*flNVLfo%-w9 zFFZR=0TcVVDpX_#3ri9}X7B{zMq*rrkAf8jkyMW)Mns4Kz>FMl{E5^mPf)%xzF;ik zCsmzz|Gba*C-Q`b>kIwyIyLLw{_J8W!;g&4R@M+Q9Z!)032;CJo z>n-9-vpmdtmTdOB28P&~i2;%LI5b>@ z++~Y&E7=2X!-Q&o_uZXU!GEN$$FkOuj`2p#e$q7n9A}f*M?Q+P{8rrD7&N^qi4**a zV13V^Cd)4-A&VD#9D84KbfHb)*DKqu9Iqx1ZinnB(x-wo>^5BForRjE^y=)I+;YA! zJ~S59omfCc^vg_*p$;rnT#q{|erA9GT)T0j%IVX$!NbL>YP&RO`fp)Fo_q>+m@o;P zYHv>B5%4dj9^vG?fDjGv+vBlm{rWVj|IwxNkG?7jSfAp1Im3kA+}u1@FLGSBcA`lI zGQv68Z`!2w*j)E17bReG-D9Nx)3QYjJf3b_>fQ{gcpILTk_sMa@Ihaq=agW)+r6lZ z2 zv>-HaXOsA`C)P)y9jsWlQ^=Oq7^{4gQ#0$*nl)=kTm?n@C5O%Y`UM(If9-}1PuU1B z4PEKjKvl@Gr@}lx90g}{4q01i*!4)NUMXT7@!7j#Z+n?&{|nSy;mes|0UUpI-Xm*I zSZIKPn#cRjC`HQb;7917NBj)BW#GqYq0AIU`9MZ`)*m69){sjAruGSb{ra_{2OmJ# z+)yn+aq+hP5sxYG-#$a1)*564q&iLwCm>3X@bfE6%>B4Sddax1AgOCD41GV)fCW~E zPQMJumhHCnW){&GFZAiCMAur6Lc$TQ&Rn|Mzr~t$wg-A*ItYl|0p%pEE8vIh9dhXb zV4?uWPVeH<{FK4PK5M#9*lqnUG;iamz(Qg=H9;BHtlyj%u$e$uNJtH6EsEu<^A<1y z_OD&D#!{`Zy!<>`+FEE0HppYxE0ya{8?+q?nSQAXE#&o`2ieUMLo2qWQeSJ@hGc7j zP*2&gJ8Pg-F-c3z7X3nZ$}X*&r=UG4p_?YD5HBhtACygAey`;sSUxd33P!;!(*)1I z0X_IU7&G4pbVBX*S9QR(h3?6BKn(C*lE+7zlm~#Q0>MZ4fueautKpl$!qQpT=VQAq zM!!?WB!YPt_+BE7>L24NLhh;54b5M)`Si%z*lvV#Y>M-B2_xOx%2WS|Q47;t`lVPs zDa(3Phe2=`kEhLt1S8dOp1KOZ06>S|CC_;RT=H)={u0NrR1 zyxM=CwL%>fii+hi9`niAUPqLoA5V`Ul@PsmZ0@N;RO*1i@yZUbf#)HEaY%MVifI@* z`TZh9%1;3bii!zx!?@*YOXhZ$snE-rW(la{^&#&q!6dZg+9w^93ptE^hU|QvNTgt( zyHW~7%HCHeRMZVnp9sITn;z98krtAFDfxiz@>A{}Q+{=(mf=J!M8-MF&Q2kptbFVz zI<4;Uau6OV)2tvJ&kOgfzC3-`Br$kehs!!Bq`xNo9>i}p_W66p$Z^V$j`8?qKp1q} zyaGFk{7}bM+IhUW)^jNAyg!GfP~_a3Yen0)ZTmKA7dhTn51sg$_1`4t)MuD#5d8*^ z3!0H&x&HFz6zWc2>yld|g^O)Q+C-m20*Q%^K79A@zXRVT;^X@s9C~h%Gt_F)H8rkZ z`pZmc_DbP`lQJ+hC56G~Tfgq3qu$i!A;rWKRhlCzVpGz6nY(&vxtVYufJtR{neGa3 z97=C$rE%Uv!msFovVk0It5)DTKcj!la&ckap*fw=w{Bihrd^6#Gpoa5(J!R1yKME= zg9i^9fuAN9MSrt&wSd0gliFpnPE*6mxc~S+A5GqV;9zN1Ih?PzP(T|rW~DnhdEZbi zS2^h@Nphb~E@e2zGfV%Azi;@R<->8a(7<9a9s@rb-E@mRYOvVKc{eRR83nnhGfdev z-^$zaof={k4_(Q&I_Il|UZ3DUteCvd{&U+mZ@%j9GWWBDX!*g^R^=vDdIbfAiXO;< zr8U_i)?e%Jc@t=>rDp_B9noDZ52WDQ9WLkC#H`%F5P>c&0a#CAhCe1@zT`@i7=2i% zx3@Pkd};!>kWk?A7!5j@c19y1y4F=xY(hbRx^Jy{@1sr)xEFqlj+4wn7ANCW5^6y` zry-7qXt_;HVwqrWfpOmT$=%lA007$p3OqgP0zLah z8d43?Q8GqrgG;WPXsej7dpcD7;X~l;EP9;rFS_1(8u#Mzrn`fEYq{eMK$8;om?PjP z%RYw~H0dYY7V^+Gg6?`LYF@nqVwnxR6|W^2o!a9nB$TeWm}CdFuTy0gf$sdI z`VEZRD&jZYtG=>8{HOXwTGHhrC|J6x(1+mlctHEKji`4_@O2d zAp#KSp|^JvC<)He6Q{UOoM@4K>|1M|9Y~WkfUZ&cYY?&K7ZMU;>bw1CrpBFsToF3D z3=rKPo!cTPPfWO@Qx%c@C;IB7v|Xno2?8PPUHEPaNWMaN%@NwCg=Waq^zPTDu9-)4d1g4l za6?e3uMOCn6LT*+`{}3>Gkj_))e;Vv| z7v7q1qvln&v`VRCm=JbA#?T4HgGHWuGOgb$mjn$AwHIo;UlWQiS035NecdDW>1eg- zWu_SDa?p|vM6SBb>)n!az?zQ7`YKN$yJprK;gl+LqmURYr@CWZ`m01njxS@Ne-|(y z^@g(OrBklvI=?!jg7p{g_l9hKk)X^Aa^8Y~P-vDJe#^v~RIVl11cLX!%2S9nx@fcU z-9gE;^PpNmT2Rfbz+VTD3Y}}}$Enfi{G2)vH{eOnCs{2%8_yO*QK;H;hj+4s^q#O- z9Gg;mtIzSV*DvIW<}}>G{En&rKHc!N79p@p8$e%p z#}7K00~CmpX)`s?)Flfe$7{r!OEnLU<-ouqj|#>{5wROQc1K>=l9YZ2;DbSbAd6=W z3x1)DGzseg$rdno@zZB$H5rd1P^;Tda;I8NT7P|nsE~k6r9Vn^t8+rrQd^o-U2OcwzCyEgf18TR_SyUzLsr!W| z8J`i#>E*>uKcEtQ>0N)zxFa5nm5_*0vAvuxl%B3P62d&7iNx?kd~2UX!1@y^gAyqzt&G6aBtLXz$*)-AEdF!=-a#Rl}`;?URaMq`(R~cOe17y%ylmX-Din zkc=E^iSuK+weh~YNBrE}u>)U8C&+GVs{ZEcU&5xgAS`!)ZB7>*^H!2N^b_iSKVsYZ zOF%{4U;3G*L{=aIf`Vg5kK~^Z>8CdMM-Z4~mQAF;!7s_#W+na=;q2eOeuYu&NSu7Y zWYnvGTB&sg&U>u8D=c?={6gBSl)r?ustBq#-rvAuM)v&qI;_a^Z&N$dEUPnC5ma~B=Jhd+!@Io z5x4fcVs`f2xfo~yzO}E}?COU>DOZl2M1QRe-r}CE+i2+_y8v_?EuuSDu2>ONGCQ|_ zY|~c{>izHbo5aQ+Pdh{Nv?~u4bx5?jJa*-!Yei(F?O4BxiY*3rL;IbfsE2VGtOm~n>q98bt08h5%bg-=y36NUF zro)hE60lUBHrHAF9as#HCv>M&vxz`Ss0|Qdb?v9onpcikWLtHmuu1EgOf6fr@r&3h zT*R}mbTRB@C14x2$V2ZIfe7RZ?+rRbN+6c71CiCOJ+i(^#in}>419P!FeL%J%a?5o z*OUHjLS-lDdsTK+4Lq-C`azmy+IPF6W14LOqmTcfbCC9(wqWDrto(&0k$4TCbpbh; zAR<_)#Do&0*ZiZi$P;bhZfMDNO^awTDo?Cjzx%P?G_!r{HECWn)t(>|?#+YyicxO@ zpk?x}{0!mEI+S=!;`wx);18W{%5`aH4%J(`Ek$A>{(`qj@vBP)>bJh2CZ^|V~4^;m_4oArai7!-pTNzj^wbEMK1 z1oe&7c0$@f7>Lj4#Jf}dtS1)wI8-w5;HwCZ0!Jm6?Afqkbt*3`i3BIHfMa?u}emlXq+3{Fbm3JCr0fpQ|M zYAVc{>xjn36X^1pp8G~-qp_P3Iv-Z2$|7_;u%WNBkNrP2qv{`1Q<*cOCd`?KWCO1% zM-dqZ-AV+FqrSR98rT8)O8!7o)Wh8p@Wewz%99tg_@l1aj_f8K*RHeKQ(kr2!W2)Cp-|+@}83ZE% zF(HpFWaC*jz-kj8)x@Qd5%2n|a}}W)RZXD+o`ry~2!GuPo8A|oY6*i15}EiqP`32i zQXAXbwZY7kDo8=(Pji{I163D|{-d9onS_tjmrH5`@Zb{>Nuoicgv;h7j2^LwpjcGt z|8mG!$95s5Q<;N)fW3RBIBO4;n>bHUC$H~z$Mbjv$5}6S65&=|XZjA^n1?DZ;anb`L!GW26OS&DVvC3mT-~ zLIpTbS|L~60UGw_w~{!p8N^V6q?BgSd0Gq`>WKWSzp)=9;723$E$GfdeD(9-3#O{b zl*$kePy;Tan<3s9Hd1;8Qd{^KPuN7e&JGEpF_3|OO$lX?|LR*hFhv*fOfI9#0?iPO z3PGvW^%f(RW4)L|$B!!^b14E=5r?V^7_cSRJnTHZ8C0_;a3KQ6zl5@K15E(zrx^{a zL%<|yBT3c}e$#c{u_0Ri49b8Ks7PCMVDm^lGY}BAGcmsH2 ztv`(5q(DX@&F0u_Z-2(nGI zWoPWW%DyTnElY+ZxKnx|T+ATz61*ce`ib=jf>Q3eXbgy$n!JYlczAeI#I0MmIBfGL zTa3x_5aTo)H&t_5sJv&8bje4%&i#}k?J(5IZs~JUbu8x<&?X+Y?v@1yAVn7F&dLpb z&z{A?l5hjQrtTY0z?{81P?t3Y+J~64P+=yJJmDE|1Oy9%jg89&%t4_@e_9 zJol!>`SwNO_*6VU(gHts>}RhGis{*#_E9C;y}F-&WRut1X8TiNWLoRO_9UKD+SvFsRigIXGYu>gIA4wN=Ivc z9F|V#Y%VevE;L|gzDCa!hs1J3D=&)(0;Nw5k#gqUlLMKtN-Xb`tj0(S7?) zfzC|N*}01DI}?v*E9*Z}PyfHlCGpV+p;6Z*pjs1M@@0~mVAy_*`-la74X;ow?sNjT z)CBx0&rCU>850qe&7;H)58NJuHxj$taN}Ge(BRNtsVW$#6g)KS5Vg5Cc2&slmHOX6Gd7CL)+ z44mg78HC>*)cb}s`EfM_291D(DL&&y4x;GGxp09g<(CT+H%%RajjVwwMt`6t&%v4% zK3Xcr&(`}79+cL$VKG84PXW0UFY)g#8XP-ykwi91tzmS(2L6miC0uJzjCca|Xz1P; zIy*Zz7T!TWzyroc-&(u6qkq0mJ<&uE`0?g zY%@rmqE3!KgQlh+vZ{G%7Az19ql@}%H<06M)8J`gvoUpQH&of8Ek6_cHRNXXMrK1g zvDxi!ZrER%1q%+d+3dhYz*h+lM*Un(}))7^@=`w(fmqLU&xZ!OTU)h&znW|ZpB~~eVy-V+^psB;d$36jtzOamOWa*CYl>KlgXYAS? zN65ZoAY~D`nN$~S$SU63`(G2&B|=WQT5Ku=05ysXmCVePVPF8d2rOrS0r zBU>==>C0w>XAyrF;Vl3{z~zT3!Nm@lx)h-7A6vx)itqqbys8v4Y=|^ds=(5t@r35300mc$1E8^kEKduBq zz(bv#C_NN*1N+PoW(V`j?SqCGc_KL*ApR`&q7s>}7(0(UhDssp9CV&2{Y0`E}K;|;)c>7nZW_AjC)g8NcMJ*$geX!^P`58qGne%Cp~SGjz#HP3_L}yP&{@cw30%9MNzuh^-I)9J zv)Su@wKr;iG=^D-=Uf>vOa}3=17eYvi5L5zY~TyLTc~w%S{@*#;#xs-O7nkKjMeX=!zKk`O6D*X9Im(TiY{C1rS6^Oz?m&0bn4& z*iXQ;MTXrR`yvBIBTVTIe}bHX0u5mfY7cch0`xFZ;eFn2VuOTo;gW9!gavRB?B5seRuY9=U3336{v`F$~Km#?R zcTs{y^LpkhMy<$rC0y;n&z**aaXsgeXV4?4K)VC{Nz(NL7aP?@%kCS*^31>)AGpDDL&fdROE{?}4XGM@OfXx^Me#v_5eAe+E{v z#enJ!%nhiQeZUZy+v0+61R9iR*xCfM1E#B?@kbsCb5`&jM^N|P-KAw=?M>#D--_Kg ze0v3}i)kb}I?+vh(tqI9=0Lz?hsB#xQ)lj_qXbiY$wP5loN_Z8#@q0-ViVxfKFpEc z>e+XSjKi;JLBe?o|E2vjC_gf?1_#i>6kI&H>Q{1M%8E3E*614niMrjl?)@_IA&Ev`2b8X0gqd)h3z>q8*-AUKe@v8+jT)lUw!x+;_Ta)$tqmB>T z&!6PpZ}q_R`H*xPAnWYySFCu9J^@jWMBDp@DS7Rb&MbLmC*PflUp?%nt$U75+TrkR zE%LdSj)B@Cg+l*rc}a;B6s+!%(|fshyT8Qj3q~-cWi~mZq?C5{p48bf{Bd#>`MTR! z!q!fe`#ZV{g%il9(A}y7>b$-(PsX2vI1~TW%3{X^9&k2kHE~ho27wNShkwWD3WntU zS?6&x|B0r(Qy9Nyi}YT})^RtuR44hcsAwzGn!sf4-_LexpUQIk(vjr-|2&=-MgT_LFp>lnP{XqooE`^VxBOen(+ zL!;4_{*>;d_{sRgZa=N;$z#Be=Dww2;&bq-DT0oLN$V7_k}MET&RCTQKXJE?Vy7>i z34j_weB+o3RYdV8pvw??wu+vvt}ezT*(k&)1xbg6hlfYKrP>0%q;j2Pl9dH!MNM0d zqgTuS_4ScTQW(Zi$*`bV?vTKWl`D^cVliz=tHJb*nZ?oHcd>o_hlh{p|2b6SZ8-3& zG$#6du}Py9;Gh9j-{Xe0Y{BI?`rt4{<*yX9t1)O18HBAO2$CdqgETp$ zQW5n9u-Mj2@OMr-`PZ5F)xOsOw=L`sbvV)oJ#@|^tJ={_*eMX|1pfd?3Y4GC=j}N- zb;RJSq@C3ToV`H!1~zsKC#+YX)PhGN2O5w;D*}BEM0&54m6KCR*7+FUk14fiVk*Vh zGcgYs1tE^W)qscQ%^ku}pX!ok4_WgFxIjF%6L|j;Ym3{tgyB9|jZCq+rOsk#-{CCy z?8mN1(AKl0FdS}x9=ip`Q~0wm-Ba3XdU%($K8xRd(imn%I zFb;(S6Dk#u&@jQeWCi(>e1_033_&+BEla7z*QbNt%(c}5Q|&(c&uwqUZT~L9baaPb zv^kGmKto{&)SUz6CElM9fR_uLv&&oh>t1`e-TbzJ^a6xuCyptKiel9HKE>?wwOAH% zPDW0=%kP^Kl8pb9`k&NEH~a?XRKgOSe|r>;N06{7*hkCt=(=FmN~c+Pdx& zKvfMstWs93DqvoS$U6_Bf}m!w9SQeXT-8uURJw-q#^m_T+ceR%#TCp?Y6RO75)-_) zlGp6-`=Fb>zr#VmEctgfTN^>FcI55}<_EW0-qfT@q5{2QL$EqNJ!9RKuP7W^; z)PvDQWk9QXjP{vO2f&kqNR4RHa;lu8`m-xWEx~!n{$#>3cQ}V&V6w$B%-a(I%5Gp! zf~5N3Y5LFg1d{8~{|C;jsNHr^^d$kJP+^`S+zJ*KoM3a5BODVbag#(oqB4(vK0^lE z+FYit+iGJ^%BLj)Tc+9C8w96uX9pf0yf<%m0IK=nfwSDCo0rz*69u z9io5}cvsnro&Z0$N?8A?hICmJF!7xh)1(t2XQ}uex9E5qTyx;}*Ljz4PX7MYEoU7x zdA<<<(4;s07t`M|__waoeOd*%krY;nFBpgK-8f%`{M_8QHG&N;2X%~=G!YgQP(Kb$ z6^S4;K}692pL`mw!E$99iXcw+Xck91)brK;_inzWK`9=ThaHfpCc#d>7q>$gxb&~N z4s@&yiwjc{V<7Dp!BhdNiUf z%O`X+VhNDZt$l_{Jb-2JV&*t&EL}$S<|ktsAKR69c4;b9 zCbQnNz4)XhC#mF&V^1Borh);Egu)z6(m(wuSIo97EkprIHJLyE=T>f&9HmQsg%gLs zo}Gfmu54m5=s89C@W9NLO}g0!1JZ5_(^9b2G-{aVAD|E#=qOrA`IFC(xoAuc z0X&?JcEr>cJp8_>e*MMrCyvd9;~bYUZG##0&FGogVFw2fJttRj7hEZEzWZ9aIsMKv9hY0LRE&ME ztn4EPvO1H@>_F#xH)ez`Q zZp*KKA-MvD7%?WNFk}7;GH{?RfJ~#T8vXGbkHO_tNEM&(wSg-tA5SP24momG2?i4Y zLrc4*i+~Nt=_}@i^BIH}L+cX_6Bumy%0#yXmdK+@e+u9=OHzfFNjwLYzms=IoGtLyltdL$KJu`Cx=F86el94Xu zo(aoW;6#)`>L`LR1R`m8d1e)`^5OFmd{eq8Pvk?OmXQ+=fMtxJ964louHCQyI7hCV zc?t=D90exnvHK-Ngnm!Kw$Dgkq2V)o9KQQ-Bw=M@5X4}H?p{v2LB!F+2M_-QU^s4N=j0 zs>7Oq$Otqi69Sqlz_R41F<(6S%j8TUaP~oj0EY$RQ+0K)A|kv*ez?5uez8`o0lh zLbKC|yDYeR6O&L!ZO&5KyOnOlZyZ!AFW3*!Wo9^cL8h)ori-CrZ_eV2^un)(7k8rqc6R|HYthue%N61p|Zum zQ)Lu*0({trJFQY|=^d>yue5@m9GWP``6Dj-u3I{VI&#GsBU{NJVZwEQld60;%AvU? zKp9pAwOyJr2-&`3^ci7oL6rRZbu(aM%XAa4tIe+;*C@46&|3c~j%%Oe)^p;yzMh?C zLDeb?q~Xx650MiKK-%47NFiBiW=pl4vw-(dS{s_e3SHJ9|ZU5A^WpPzcQ9%Ou_VL9!5onH!=mFuor64Nipv9300OcQ7y zX-h+b2Et+9!_wgWRV^&u{BwE?PPIEv zN-8XCU82tSmcH<2!TXoj047gZ4&Io&8%m@IN-db=RE{yY4OqQJUtw6S4;p;h z8^B`uid_l`Irc_lGwF`DKXi|1C#L|U?4T_SYc@4!S?Yow>BCVUb-;U-e<{Lgr~+>v zIm`@AQa^0hO$gz72>Rr^$YCw0lM$e>HL;!+4I2Mo!)=J=2-2FtG*?en1COr^hz%Tr zcH#h=oQ?6(A8Qn#IgrsFjHvp=k$@p-!&)7%9cPMtX6$8NTIo*@wuHIFc(=J;hzsCs zYz->>bZq@o4&}>{<>$I%3qVWGLv1AV1d1-${=F0e;Sk45=dqv_bi|PRa8= z^vU~ib8oG8_^E9fO`2mkI-0}&n{)jg&S;_-aSrCczajsBtBc{k|I=UShydLHSVVLq z&LOFD=h%-NDeD`B*#~~^D=`S6O7QMS= zT}$4c<~> z3jUJpTt>Mu^qHX1XtdvGmCpMt-r3oOBVU((vfihxC>%&$l^cCwzCR~bfs&r!7QAx2 zH^OjRVGY_6;ni!9BANH2p{P@q}hcyfU(|{DO!Eg#u<)R@Ku3f#FP?i(#*Kzlvk_h+X$X5KE zbS0LUbi+F0CicY#l0%t*O8a3PI;MM&(9UdeLK~9zS9| zC$%l%&5!*j>?b`Y2WQUnxrMy)y5iJdx@#d^UE)H`&v2>n@9P$Q*q(AQkz0(*Atj$k zqUgcNkxS$NCuA8R6E`GC(!Aps8yn;oa+*~wY{nR`7a-O^$NretAQzc|cx8e8RIxJ? z{*(_|X5Nfb=~Ms}i5G#m9^v452E(V8L)Q*+yergl>zqWA6Tr}3f<2EM#zguQa-^h6 zqGk->B|8QU`f`>Qgl_Z^&arYk7dOj(b!vhp@VQ4%iaJE7}j*B4*w4!Z-Rz5M8k-7Q<6mVMwWN?!>P*_~XeL#Cw zYEJyt+}?(g!*2-*imFTeVGSuj$%oOH{xA041FFh%>l!5|Cz@lTrdVPD+eVRU1EgaN z>INy&rARC&QbeiJIWbX#5pYY9UIb|Z(nTqzC>@k4MNnyiVgW(v|6K0|#hje`op0Pb z#()2Rd<=&o?EUWdeab3x&b1!E3W(-|@W34#)V5zO-h4o*d9XcgA@q_n?`mc7SIf%U{G$qNwi(N?bP{zNNyUf(y8uCy0y) zC9fHjMTkegfD?n^Ur_>QO%;DCqn|VLa(pwzsk=WHME@`Er({%$ye>__Q!V|^dz!RX z6b_tzZjCk|j?b@*oDWV^J9Se##;o#Vl9W*Ivry4V;5NHBq&#fTPB~%`^F2tTt5+Q6 zBQ@+S_0RBwlM7|7M+J4ykmV;m_@sD>BanxZFGk2#GssvXo{rf*vi)bFD^CPI@&vv1 zw80t>M1`)hvZMZ~+yP?kT4nNqy`$$i?j!G`kozSDCVli+-+`0GObK2xJgB`4g{Nu}h#5n4E*Js?` z@Qbj|lP6D>r8I=2eOZ(1I4VD&LRrZLj9R=X9z%e!9ZKQ4P$^p=hl*4^3U-6k1z{;? z^e-Iw!)qGX=8HXU79X5*EOq}1?|L`674<(2ylTojcAiyI)@-O*xueoF4(;AJG0=@l z>$HNEitTGViz_SF2wjGpKRAZ)#>82_+`gBf*>eXVe zbw%$!Hz#k$U78WI*2v3x3#9emN&c3zGfm=xzDAZ+?&A#o?v>X5Zw zPHheQ%={G&dz=68>Rr9#@l6E9Z-$>e< z$x7H?v&dkB%jwgyt%enbSz5!KM}lba1NJWLY4$nzG=kP4a3v)uNo{1FU*As165vvb z`*~$g-W6^`%Agu1O>Jv+ftGhbkpgUd={D!sgZJ(>Y^yUrcTNbUMK3MrXaVeOMC_>0Q_m@$yWzR8m^4l z0YBOD)$0++NW|Sn?A#y|Mc+s*1gnEBwpBgQwCO4;-co(RHEy0l=wqcx`Yy!agHX;2 zL25@V=j;`(k#XV9rN1nMz-bqzz8WrK5P;ZMLj4c*FWkN+9{0^LCv0qvrP>llp`T~O zJs~D(ydUvg7e8J(OIB;2N6!2Y=uFhHxnCQ9GAx_$#yfq#lcu#hqA3OHj5m8epSHGk z?cBP~XYxnOR<`hCQQgD#&B{7$KORo9Up(P>K#=%_3r=9fdnUcNFDLX0*!x1 z_#fL3%mZoH zZ(4}@bA0{Q4f09bty1qDv|Dodqs@Oqcf@_vu);Q#xuW8>pBJw3$X$27ee1o zTx>h2G@QDVx%SE;U<=f76ls;CBfq+Y-O)NVVB#`zIL85NisULo* zMWsq%lY5VubB2bF-*&doHiE zHX-o<41n856OsZkJrFKY=XKm?D6S&R03|Y1@I@MQI%9;sG*)gnWmF(Thp!>1#p$94}O zy{-mjNh5_05XMx3#zIEHG>QitH`rv63bpeI0@roW7CSI+Y8pDkkiQav#_B;Fw^a7z zHDm`Y)M25_JtQp-D3%4}>L^PJg``Z&UdE6LjB}qW$nKm$jPm%JY&`XzwKVgtuec;J zPR+0{Uf#3}*CaR#EEqL?lB}Ov#2toMqg?|80|VMa#Y==8zPb~UuWvu zyd{!l6U_E0P6g`i4Myko?b|(o^Yl>q!_Nk`r;CXTOM%>pw`%~z(Su{TznG#TuiVs{ zo4ok8QA3=nB<+L){>%M;MVbXYI}S2{b{u5@H5I*l%#C^$9hmi(jO7bEV|3Y-j0jw> zb%(A^ktHG`kgHM4Uw^}W$ebU_UqykzZx+#`e*VKyMc6eMJ9ZxB(hNMZy#LT__!;tI zaOA{xqmq)_6L2Gq$?iBC~<(=`B~7;WRl&=HcO^$*P@HPJ`9G3#pnWvS+;<4jEi z79Shz<7ip#?P$C3RQ_|61TjsK}SH`yg9W?4D~_(@Uu~&m~MC&1IvK>9ZnJ4$CTyW zyh}<0%p+MDw7sJO$wA~)@i=&Z%GtsU6hI5OhC>j7G`Cqe5aM`S-kqC#Seg6>?e-&M z(;Dp5^SIrOUWUm9#x`DkpYjx2gjyJ#>ihKM+JfekUmbpodjgV;a0ifnMaTmbc0)E)mC84*@F55C#Nq9xNo_>r+j56^`zWIO7!m6 z(qNEo4!dHhkz&d+pgHZ}8CC+yS0wLl=Z;Xz9E6YX3hOE7l5>9T3xo;%su!$SVI8mp zYI8K{etKv6=I5P`l%mbg2`&62oJFcGECo&s?y^z%qOWfx7Pm47&DT9n<#VS}(S0%M zJjrKKb_bymm3;z8NkYaBXOo;1hwj*;%nV!Xk}0T@NaSxyAoZmZWuOhk{S$PSY1^�L@VmR^G#w z^bZXcUO!&%jE=VnPg_4cvW?B!m4FpRW|EqmqY_mO=78FTl5z^o-|Ns@3C`yp^@6{3 z>y}X#q`8!3nNSPiEfBrZ!A1ySsh+70T@cW!Uvb~QPuw?gquHtFxz#zx^#>|A0;v#% za{)Tc$ALIwG7lN1piYDea|zk(NaT_W)1x3&l&LiRJ@fDTplIsH|MP?Y3j^N&zs+p` zIKUap@Y6r zhFrOA=6DZ9@yMAi2c4naz2W#G1p;Ewf>5&p>`Vf-Llkz1GilzCvKlTtA;A6ahmyX3 zXre*j1r|x0dk3lFT3&u10W&1^MS~v1{3;Zg;4FrgdpgX!cv^ltyPbP03w9q<_az*d zMY>|$R7kM2fYlH-ju_}Nsq)a^1?U4ZCC8n5Q}_Sb2lwBLqU<@JeMVghh7BN}l(4+N z5;bq=LfPn=4@pBms_W4F;op5angCFWUuQv8b1oerg6Ht;%m!6J7uUr&f9+1rP;Xdy5!7tBnXHDQzOZdvVbW;@! zEJ88pkF@AG3sR1N!y{Zy*A!hSl2KIRFKXPRn$Ml$TghHqxxv5UuN^~wPrzGCkw{`p z^e}ANFzaDxR!X`-#79?4D?=pJIkjkk;b=Eihmo`4u?F{EKDpTZbqrDeNP-e@_z#8_ z;|*4~26;?|Ul__%SwzH0PP^*sx8L79)fTZ){}e{$Z;*vk6Fgx0XpRE$0Ww8S3`9dM z&qTQod+XybrXTxcztRudr;#l+8>8WT5*BFdA5(hMTUzg~zqQ5vPx{(op<(k`3CI)B zmQarpST!{V{cOTU6ZE*x^K#1{QQYt5c)MO$Nd@gpe+IZYI9s$5MWr}Upq3xQxohXv z#rHRbxu70gR2hppR?hx!)RNmRrvw@+KYl;iWe<2yqTQq&vREn`p*9Slf#92|j_AyM z?#HiOH)BmlY@>ofFbBcX_K94@FZWfqw!eGYOSEm_<57LIOM_eM!sGPeJR?cMCUkV< z#xXp6-XE+2DiA>neX0$8$E%+FaH5GBK~zMVg$88v52g!oP71tV53rD5AV19>nk#6Z9T1tt5WS<#l3ymkKT_8Ha-VZlsHfg5WDWdKlNyF#2b*KFZ}l_ z+?V?v99}hVA&yFfDDQ~jF$6#)Sy5@`3SiS@^$7u#27=FNd2Zu@RucE8Rk58z!Z8T%dr+-MWzGc?BNm|OG!YX{mrwD??sh-7 zouuf=d>UIew%@!2wke*y*TLR)Q^WT2_2T2{M6*LUMG*pA+5cu?K zqXyn(%Y@F|=JXxJ)QY<9S@y|v%dr-R=w)Mns&*>Pa6z3^-cx{!!^u{?p z04I=$m>V6BsC9Tt{TaXxW)RCTOfNb>4h(`XcEgzshY!>uJCDQh8rPGiLiy_-VUGP9 zK_F2pGom1HBr8n_pWEv_q)K=&HJ2|1juqZ+r894EGSibOS{E>aYt86gs9(;Mp&4L2 zDf~H^C)#)5S)U)ahhrZ+dBMq z>*50FN!0vhPCWkOyxkJM=))%s!fPo}_p~@sIg4j>>yFiNSanp*O)9|UN9P&!ocj!t9f;~NH)=n6K_ zQLG?LKGU2tFq;b{YkYyx6+{ix@<)fiIXy*lo8D}#*zH>?o9u+`W}~z7j$hAB#}~Sg z4^x>kc|9b8UTC|ax|3_RzSndAd3zQP7lp_s$GHO7Gn%B+VS&(&NmjsV(mJ3c?MR;0 zEIpU-igvf1-+k9n(EjopkJ~ONX?SMB!*hL6wamTG-H>u?) zTNC4pbT7-OuDx`94fmscZHMLN@v|w{5;ymwW1wL5sK|uQF}!*66P{a>2VYeFo}5@V z@v896E6?(?dmn#uNo&qLDY?Ub@d;;jeM~$5?iFMj6mKRGZzd-{Z|jyzQdOp>Os^y? z{7v#q=_-{iHJV>2ZTu+cLeJH<6%$*1{#f}?{zzEHo2T~1qc2|z|Xf6w2K+=d*YYC@EY0p)=NzTG6O{3!*A4q0OcYNs?(B-IV zU%aMvZG!{;8SKdI4ljc)ox=jyAR@ICH|9B1HAl1TMAOl@&ggTZ%A8-8iVGq(vre-W z*}F;ZXMSe3%v+k>dXfdaLa}q%tz$M1(4*@9()~1%wOi(DPClyDQm;T|_$5~E!yi7g zDP14ux13QNWIe`Di*W#Huu#hN)c{FdZGYF+($boE9j9Q5wK&`<<`Kk_dVak^#sl9Cl*#t5dad?XHjNz>2_nglhC)3A60R>? z+RA9C#|Be9DlAg-15ocEM|dX=iW>NJzWWA@ZW#_l+~W|BH}Q!7E;7Ukb!a+TQLzV% z`zv$toyIs|B0*aaf0m%CQgc+8x%@3dYF2JC(I04ACX&;U5Eour2)seq==uiOg;KO! z2(w7Z-C;}7m!f~&_#90H#^p4E9ThsEm9XHUs96wNj;3~Jf5A}rcjp|EraPc7zqsei zVz5R7oF^cWOtBjSvZv>COl8X0ZG@62SHcAu(6*ZG0=6_Z=lWnX;<^gRG| zRVl|UV>L?+3; zGq>R~lS)KJw%Divyi9l{zMJ_sQr^FHW#l_b7#qsLofzxD3|q4?#8%^HyG6ztfE*~4wqt zir&T99lCWR){D{e`~2+-n3zc4czQTq&q_dpgBZ;OKVBvr5?6bFclJPO0(qGz+Fo($ z$KrnP0TxA-^=_Z6$23ONyfDAuv@TTgx>6T6Rp5zq%b1eY(T!y^mCw;cXjTagQV0UZ zm16m?HM`1#E=CIk$Yg6KkfE{Y{FD|){3lbl7OT2t*LZ(mVL4Fuw(M`R4w=Mi9<6wZ zm+-`YIrW-2+&*m-sb|-I-{dhd_Bo&IR(Rt|1ey`3o9TjyDn&Fn0KP^ZU|9v!$bl$@ zr1K<9&(P9Ql({^;bgX@EbZ}JYD$7s&@g*(Di!A7aT&ea40UkfQU}7+}8yr*Rsx!+u zSbwWfsjL{ZiY)EjdDCm;bo$5N$T(PH#tERTL zHZK|R)_Ski9;3}|0ZT_g$Q^+?2OV;^{mUDzeVc5`L^4ovS6^^Jf@?}3tG9!gv- zaeDLcqxl|}lw&y&eSb&1mu)2vyCNbzydz z6BBO=yV2dvWuRdDe7P;_LXUsDB#IX8UGM2h!Nw)?O{SsDauRDQ6c%6|DkQ1~&~ot# zSh7ZlE;CSw>i-2l^Vwn31D(Ik+%&t@`KjwbNSm+>BRxa!C}tM=%mv&?&*j@zO08q3Oo+IC`)9^-Y6At^eqrd+dS}U6kj0C%PdRWSyWm zY|CJal!Q_v`b8}tV&8d zsL(z18`;(zxGlYep9k;jV*FGUOV=f7|h+quHcE*uep z>78FzD9BTgLV=IAp#O4@u~+=xkzZ1DxiV!VmX^(Xh&uf$NTqDy0lLu{w;WYxmsBMh zS<9?4_izpYsdEA)aghb*jnnz0(3-aFbGlhihh6X8hd&D&%F5yX{xfvHK@wI5;Hocy z3l0>D73{g=%)=yDgZ;DaqUe)Q|Pn(Ti6R9AlV88UK*W$J=7v4YGztHh;sApBY0 zK;`h0oeIIIUiDx8!JFNVZpND_)a#BsHLrBrY+h{#1|qE8Fy-R!X*kfJa=%1kRg+f^ z;t1jnoJd>onOL4LRD@k%q^(+Wa3XUau0tF1`%`n3f1wg+&!H*r=$`rWtOT4|#e^M$AlZ%l<+qm|9!5Wv;&5B_xA1Eu_!$< zHSL%;%$e~9kcFBF+UYn(uAuKeXlyc*LDKf-Lf_FARTP;KpP>h&Bp7dX01J9Y$AGxj z+`oVSw>`|Zu$ZweD9=IW76kIXutLTEa9o?=#&4PT z{B!C(5b?|xeDfF-LiO?uKp<(sQ4 zW8>-R;>D+es>=wx9-zut3TKmU27;j@zhggH2+uku%;nMWP)4kTb-@JUK$+DzmN?vZ zFEX2&`})piJSsi5BgP_|Ui>&PE2o%xvix|FIzZFRvJnS6vv(yg|e&Yu+Fpp#%Lk zM47Sh4AyZzMaKtLqhn&;3%3_SGUTjz%=HbgZb#jmDHe=V`+3Q99P1m%OR1frs$l-M zUdq`a1`4G3BJ&%xtO&;96WzJ|#CO?fWp0OqK9KG?NTBP7>*KE@ip6j@j;`K7uT9~- zd)f*;fW`zM)FLQ7Q&QD``ZH;0>l zqNpex7N-T```Fue0m6K~OhP!P#YQC@4WGA&i^rl-PSBR4n(l*1CY|6-D?ggOoQze< zy)m)8m){aHKrxLID&r7pA-|1)$=|YO$1lYm;|(62nD5m7BnpZiCO3V{+nbDoog=JW z+m|^t+-&s+ED$qwGrG7Lq|1lUPL_kJw{Dcci}0TYreq4aM#X&hm#0XUAB3ZSsqH&< zaQCHy-fT__Ef-j#db8)J=1B{Dy>@r;tYwzIZR5`!K?JLz1APc;>BV6qs-P+gU*o+s zx=D;#);VUM*~jNdqgOR53qp-tU)^?kbFJ1Uy*5r1FCtCygyxj?DzGp#ZUZ|9_g z-coDzj7}(VI^fHU`(`EZck5R#AJmN)e|BWxytV5Zp!rpBvhhq18TpC7F}nMKH0lIW zq6(c!h=m&6NMiD2xfH#db^#%gB-&bl!fhygA%9{YnQX`;V$=n5_0n!1e5>uWk(S zmGr`tP_O15csPms-=9j~zIO-UD{rK7dHtnAWcmvb$E4Q30<2^1-U@%e(A(u{K@J~f zk&1)35$W;QHJdS$uCLCSlw?Dc}|OiTjBsXfxLaV>x7_Du(eT zh=-e6=4hgf{s0QUD_JF`fyM#Q;iPn&olyS|hXIG&yK*`(bXAc56jZ0FJeEP%BU?1b z%(gS5i4adVO+1(M2u`FN3vJ-kim}LN8UdStJ9RC*ESlTFf#I^u>ON@^WrcH>g_ggm zce6@nQ_LHwn_0(=%0-eenLcjM4oV>*vIU^gf!i%Z=djXw>*K;43`B1{iDs2Rg0%?~ zA|8AM39F^M{L$)|KabT?WZGWQ`j6VLW>lK<-8>XhcbV?7HMiP2EeF9TKbr)EWaozK za|-;eV9H~K*6RrRV;kR4dMq_qtF;;Rw5*a(7906$sY#ih?{0l*cnMS6K7o>V3McW8><9h+>Ezb&ngYgenfVSe2JB?vurFvp6V#iP(N2vkl5U$k-fG)5}0TsHg-7;+@Fp~YqnKntb z9N#{K@R$Y!Zl(<6TnLMzSNdUZ-b#E7PryCwT%pZ29mU9{(w}XhpP8#|Z~Vy53deNh z!X92^&7C_}3Z>^Y*h~xMAqmSe7?6~Z=<=3%z6-W}6J{4aT)qc1A7O(@D*+l-02$))6Y-b5OZV*L9n5Zr^dKB}BLn5-(9(!0`UieM&feoa zH5}=1^nEK$^MF}Q($3uB;s59tx(6FS8XIXStVEiUX;8G>vE@WAqE`cf)pQYa2L8Ae z{9!CLnecjH%sg}&8bjaWeVwmN#vU)Wh3U%qo&ktzSI>x{5mMUXd^4O`AMVtgW-?r= zXLEa@UJT6BUu2f{7-&=Y4+264a9ror<1JY9p}pd`_HQ#jgCYVZ zAfn_6t6BlAk@XLk1HcX@eCGW5^SS7`EH-Lexm_>ev{Dq&)4Q$1mNr9pZujpc+ui+-+y?T2g$}OEo(5C0om}%IeblyW4Cz=HccLivw70=DcQmN$* z*yOTBhL(H&gbHXQK^|>=Q&1kEpn!x%bSh`K^ghw-J-f!c7%7u2ZZnX{W_V@-`Qq?Y zl+jSmG(Zg8pJ@k-CGDYZC`Ze&A%jp?CLb|U8<&pAJr=Z-klZmKDai{~;jVeIrqOm# z1opLicE)NZu-JHfk(?He;jvuP*dlGgg;OBmx%c;Ij*QewlmX%tRc1W6>ZJ`)&%WUC zl&=2}@6)%PTAoCTy!$fn`gfL$3|3wpKvus$L-;9Ntng^(6S_?N!s%sC5I$T~`TjG! z0U(mZRIJ1pS}nKvPdo$x`B2&(MmvsDR6(R8rtMy{VtAMPNKM2Q$mWMT)+q#`2&Efi zYl~)b{s6_rfSZWQ9HLKbT1DF?USXP&3NKvD({vG7{B=~uZzkUY*X0BR+6EBBS~lvP zgV4BqW8rw8^#FpMZsH@sb5S!mEftyvm!wQ6UQM-NsNlP|-e@nA@v9SKN^829l0J)KH*1<7%*8BV>7ql|m^@96Ylr zV@Va~PI)MGfcV#dFA^(5*jr;14VA|4@7C5_O#(?tfB-!9FYMAFg>cB5Y*2-eM@%#h zkWg-1MYpJNYWgbJ*3Ur9u{r3(z))uVfkqtCCA@H|av=vQ@`k7xOh(S(OLPO7?+gdF zEwVh(!56syZrHUFS}wm)`1{#T#sDYJutQ=WWyss;lwX7ca8;_#l z6smo*>DiojXc*56zkg@rJ_zH)19)$Fxxxfhf-ftZM_y^hhgzPhbu1VN!^h$)a^94; z%0}+tSSi)r0cHsS8bKgHzq>R4B0z(PgVGNjJ`cc$1YsTf^(IP>N#IsuAMO4Bnm=Zfbo zewMB!61GhA=GEagBD;YuP$Dji?v+l!2W^n+QB+R8c3+(>J#%dmA(CQ1f69r_ff4I?@WS1Zcon@+xbn0V(k7>sVukLShrhECFYFzs?G0?qUX85Hu~K%RJh2Gk$y5p&&D=xns@Xv}gQ%`+^5JZSOV}B@R`R%Y z9^I7W`Y7G5!ilGa!4ZYI0yXI^skZHXly())?7ZqrcGYD6x|f?a_) zDgtpv!EhTG9~GKsBzN`mVjB%Cl69JLX+fAR!FzEAYIjwYP--3rMk2%Mb|UTl*}Qfd zB^T&5wWoEheMi*6NKJ2EGYk883fa&Gpvv4?p_IB*Hw+ek0NkxGM| z$atnve_u;o&L6$LJ5hG14$MuW$?UQ#B438A$H|y!&tBrgcy>LlkD|!{5W5Bp8hXIM zxRUv%93AkuT~-&;hNLaW`a4RJc#jySS)~RXJ#JAYwO4Bz&oQj|O5x(*+kq!dDf67J za{ur_di8r1D=7&EhKNUU|M1FHn?`YFSpu;D6jP=HdO8e5$q5IAs+{c)3xTyH1eUr) z&=?^nWMFxjzD%ux;R@>jUMh}LtN<1ekDjB4a8E&UE)M&EKawztR(=LB1Bo_s1~YI> zLH33~;zlSeA)yZ$XnrSY9RWCX-d6M!#lpZ+X7>Z&J&eq=1T{Ofzx@{Wn>QXfCt~yK zCyP5kzMn*uR0w6Z2tA-8vSfq_@Kj}!V^xzeM6GrDY3Vw(I=|KZKKf~fK}VW2^T)X( zo%_v)|Gi*tR%RdnCa+tL-nR+f`x($w1~I`wE8>Xt9t!dnvYs#8tEtla;N}|{?<`^U z{MJovgO>mCsN=cafvl`!6VfW2HBK-rItpEaDUhU~;1~!5iaiM976o;(>09G9XsmNm zO>VH(5)2FFpqv<>ER7I;4&XpQMgzkk>K1X0z-DbsgJ*C(U#|w?iC{egqG%xhjpb!r41%a7di zISi(1iJY??Svno*Y`%hpm!6DVUG#2w-h#_X;FhTbXk_PwgBNraW%ZGx?8VSMY94v@=8?HDq{GSL}j1F$vFAL!>GY9 zIl#gdc(Ri7Yoh{c&jP4u?d>K)E?x;jDYU%UiggOH0pOcWD0;R8#nl4?H84aQvA0eL zXOSK4+rVu7?$VU)c*9#Oy-!x)$$YnyN62nQ>9l4M4E=1XD*+kR?J;Aq*@|FpRSVK|q^Ofx{{xT$u=&^XFC0I09Z-%Z6H8 ze{d_kGmq#C%R>B3PFiBp<;mTsTs)uN(aLCnO<1zTfv_o6B32B$kJ4f;z6a>q79eB( z0Mnl1)*LJY9N=_N9x}I1t%worx-MEC5Bo5@{&|gduY1%zq|rhC5FseU)v0X%z51M? zi2k*eMBUmxODj5zbH(_TZa9NtAtuH-KPtq}eg4e`<}EdlLU2Kz z_|LKoxEhk%P2Zi;?(gbVCPsG78%MM*!0~QJ(*`EH`(EkRQ4j{;vh%qMQDFoESDd0@ zkL<@g8z6{A39FTezWnBqUf z=WO=#hq*VkL^mO@CyR88TBb ztcNm`>!1)eBf_CXX86`@d0&i5EPDHt{Q}__z?6d^ok$Y!SJDBn)K17a(*e9Nd9<4R za0fh_Q|>{uk_lxE!?oNG{t)TX0!~v1-5cU6@;)(9`#T?5X4UDEJ%*{s)%7C4$@?geNG~-%Kb0nu763rsMv|#i;yuqYYt*nc-2#>jKq%XVlF~}p-7rjq3~4l#&k~dD zIRyPC0=EyLl*lZA(zO5?LVzVol4ch=kiamCYk-q;)XG(G#Fs=wz<3-6#{cKM6)Kk(NZ?BeL101or4IXYf9yVlCo10m zvSJwa7w1Kii!6Qns(rKj*js{O*h~`Y(wC)&cVD7OrG!-3n@%P298#Xv6u?#k*Jd$# zGO+q$IH{ooWqxF?J77nSVEKA<0@B{i$%C1y=?c7g#-oi!r(1WC7kff(%da zPcV-FJR~x(VUNidlj4h`Num;45S9ZAXiCT#2=L7~hv7BUZGXBk+Q%=P8*ZjUd4W7b z#v1chiIOLn`mNiaeCfJ=}u!zl?

__DN6Bmz;bcAp}r|wFuop;Q5n{01nIAcvmn%PY$&I&g1l$6KspIu;jaWg ze2_j1C~n5%;Ep;FGh4Wm6EHnPyHauPl;X<`yV4w5XNuHqSD|TTxOuDI0ws=Qpu&o+ z$G?ZhEcNr{Td!T!U{+Gm)Jh_NoI+VX2=w6N`G*U?t!2b-03es(ZL}i*2d9@$#d1?` zqj~m`A2H)l8ZcbEQ@(nvFAtNE-SMwh5}v~p$2>LyoW)RoAZ6!aEV(rJ_cG?h0#E74 z_b5A!8}k(zn;A4K0TRP=CS$A5EVtQ3c?mEil2`ZO1W5Vu9`$ZSZt8m$n0}~3Hi0MG z$H^P%O&YF^llOsWh~sE~=zV3ehVWR4+`uc9zQ_U*${#8*;$i~KjLqnH_J^%)}u<$x1asbSHq zApycX*_XFW@3=lSY%8^f9E7Ylfv){va5m7Y{>MEAG2Aze@d0RlOSg(B54ehDu2KR& z<#R?3ji$)w$leR>pa?=yWE%EDtS*OWiQFhLN0?!AqnUhI771t3x0ti(+WB9jqNoE}+#3IgY@_1N3}N zYo?zMB^qqH+fKr$bPcWiu{VJOle~fnZty+U1F?Y=w~(YG{db1-3wx))OkJ7v2P%>- zeLP=w13I*np|Mghc+NI9hmzn%m~4t8IcsPeM=qMe^#UboR%U&N62FB}bVFt*rVL<| z>U85%dNse<3`$rxsr7(0>~dnMEFusLR?1&!sR<-9MUJEC&wx4cs@&@e-%FgUc_hqU z*CL$pm(pDA-L0>t08^>ymwljT{ea6uEM@>DKZ$O^oQP#uQwobRDH4o4Lx@NMTnb#j zie?@(e;RWJD|#H1a`%s4s}+-rN9l@amNf!#C$PzIW1J4~dyIquhL$Qvz7(Yq>GPXL z-(lMsO@j@jzam-VG*KjY!0tc~5ZQ>)mhNhlj<vV#2k}Y7i(cVK9VJbFWrT#ok8>iO?;u7RK0pMuJ1gcoAfEVC& z*hY*R4P>JjY!kabr*J%Z@NdjBuYE@`E{maZl+&kUf4{V0|F=+uXIZ9~tR*WpFa(oE z`B(|MAsNW*WatO4ngEf7=1-EP}2fd$!z#9(>r2Z!qFm`0k+;1tt22qoe#i} z%|T7qT6T49tsi}ZB?6es$ov`IK1IS*fzP*&KCX|;=PJ5EUStjv2BQKe>5M9OKQeq! z_O7Tw;I`bRc$La|I#A&vi-`NOzZlt|D%Z7C;Y9%lz&jQtQ)Hc)ugnEA^{dq@X%{nJ zwZhzA+*SUMbeiegX|Cg?u&$Z64@P5T_ku(pD3r19^&=n+g7#^3B~xYyaYZ;*kl#1- z6pDR!SNb2K3HP{_7`Hvq{of!VQUtLIHd* z;Y>($cq2aMiFX^fe$uvjzg2@SqYk!XT zf{}<|%Wy;BNC`7v<^Cz0+bnH(h0C+FXXwC0L=h(jxHgqK0=)YtyxGUY^hjvlu?7kl z%QP~}JOWTZJ_rPDDVmWFePu4ZF?8FPsPdAWAZ0|f2l&4uo?XLmY3aPkxpE|@83U1mu4u!m8>6C+PKehfr9PJ?sRlU-lVeFixBKns1{NovbR!pzhif(fe>r$u16s zi&j}oUSVcmrXxfxrl@4>XTY$p)87W;;BDz z{5I=->a?tL{fc^hnLn>?UXdQ#oql7>uB^zJ5=oUI6l`yvDr9VRcp@s5E`0 zouWyX^E*!@-3i#M^f2sBb8G*7-I9a$#RmtAP4PGCgR`EK3=m0f5Gxnh3I*%_OdT)n zn7SW8)7OpVp>B~;0U{?hh~=_w=D*`yyCV6u0*!PpMQ zjG14Ym9QxD(dlIp3K2NwbWV{-V*`3va2u?}e@tp@h#EP0jmzzk1_VqERW*hR-7p)* z#vr&ddUTzCln|{&K8(7>29?_f5}JFDdQ-${ws&ca!QBAEV?hL9vTYz+5_ZG}u(g36 za2tYh9p8)njwO-ZyE}�DLgYJz4Bypv`x~(*Y;5q{eSoisye_|@7~E}BPd{LKkaQXRUR3ce~$@6G)#k6&;p8t zp@YS~`sL)n3T(Q&Uq#5c5yl-a2H^%>gZa9viAU79e6@Evw3#_Jmzc5fJgm#qdI6lA z9xE)BVAD_5$FFY7B;1s~%nWfM?zdQU!qc@rcgH54=Pndmt)%EXp zH%-3}*H|#urIv$)n7JbTWroHI)Bo`dj-6R0JM$NH2r!v1#j;+H?URdw99!<55TPc3 zisH0MroQ4~0;j+Kf4JIz{V^B<>kp!^$O8SDAMZ z)jLz85-jA)sOQdWL5wUXY!Hrm|fd({2rRt=>wb;^)h&JS$k&p|avWfjk%yLh50?1~I z_`U;Wv`i5cdJCjI^QDJ1-v=F0=0Yk*8UnEn@3M*wdM1!?dV-|Ti02+jxrs%^`7>EN z1{|N880R%9ZB0dHWj;O z8!#9rZkFekjMkgEnglmP;KgB~my;od2WU?n3Jpl|kP}H4UO#ORqu$QwMnOGMrpKV* z@IT*urZpL~(jf>PB^BDtE}z%k`aF4*$|b4zCdodW$TYOM0|{qOl%gyOoJatHO4Ei8 zOulmgWRj`wOcLuWsi$bN5Yuw((Jq5ZZfQ_JX^HN6LwK&bvwEVPX~C^W^ZMVeb-L=E z?QB(ca4<8@sPy3ANgpD}-~TAPyYN++1|ILvg)jpSwM~JbZyuBSaX;-I6mji zE~l&E*}*DKu39dWGE5p$O1B`=ha}46uwjumZw6+s3ozetd6d*JX$-c(KyHPp^`M?v^&wRZm^NrcRq zXzIt86eriI$(K~*a)^TGcRxy(n>=rF`i8Y{FGAme0i-x7$6E~F_rtxn^Hx+0H0R$(IXgZ8z}vlXTv?jjT8?#$2MiE+=YGb(KlX&P-C zrd!uI9grZJy@r~n3=E*WEP48~%#nduyW^<2Dm^PEccND-p3Il|17yung>Q_l4>I_8jCnzEUI+=P! zHIzq?nzPPr4|gbGAyHEZ1nMHnzgAPiOjt_Sxne2w4v_|0%j%TJi&4$p$1i* zCb^dX3yU7Be2>P3`C>#@)BoZ5_xkb=w3q*duVXIvS5EMMela7v_@6<NBLVF_e zKPyV3y-hVgWDm)10!Ab~DwwQ5rXK6s_rJ;nkD~2K#rn8VPI7jWAWn>`BnTWQFLyvT zp|)p%jx&>_WX=}Z%a!091@wAX7wP(ERAjh$|8GCC=tvvtwIn<8jDArxWbcq8Y zK)@T3bkA{YGZ^r^5wXwlA)XSk{7*IL>E?wGBP&>Eg zTe5Iib%zgMzJaX|RH2BT)PO0W25f9C1h=+~u5q|Ju)vSBDojyG7i1{O^(vH>Mb0%Z z;Q}CKaoYr~7hT~TBGQ(i+r~+FF$rj3R5>jt+s`M7zu~;H4r~82n3;YfibBD=-kFCO z$%JPoRXxI2cKvICIKo`?f?2T`c? zPwfHELZE&|*SBnCTLmJImlvkzlA=1bnt?Sk6{c6jZFHRLg}CpSdjXU^(_x5tr5xhV za)?v>yC~<4bUQlr>_WQhp`MUAy9pex0@Fc(H4L#_5cDOkx$W*~8DI)DTa`m9b{d;D zhvy{3;YtH>1m5n02it9mjHwY2Bo+}J710f7Q1CL_R!`7BhnE?i`dZudiI`S%C491? zORYgx^bM048KE(Vl6Y_Q9e@xM+IKw)qFbHId&?hy*-NZ2u4>D+R8=W(Ti^ELmuUzR zC?ilvE*v@Id(F8RhyIMl{=E6xGk-thCGRlIYz6zmA8MhOy+7s(nu5Vnt>vx++r9x= zjh$f_%N25)7^E{4{VlILHx#qAO-^jtk{5KE$XuG3r|7*vdP|+)n^2Q86GO5QonVp) zdJe@+t!HLdadtaDxP+aY4pM_YM-w3e@7FT;qE5IAc4|4aJvXXzyRk#i_!P}9|2$iW zs@a>swzC)|4ra*I31``bInG2y2#ZZpUUIc7sk~G=x+wW?3I4X>@zYZ zAWI}9rD%K*LlUUSVIi11h`k-QFW*_jz6GCZi_~cyn~IN7SS-Bx4Wi5d!XpvQt8%C> zn|0o+9Vnwj>&DLN7vN#3gjM8quf7%l(~^*~=8Uds!8UZ-Gd0`O-(N5{sF7aK*)Ojl zHzj%J_BMFe(cs;G8z=O)^-eZG)o8Wu)=0F1-> zQ3%%=b5X4nHIf(5xsj9{!*|cf`%xh*l-D_30<>JD?{OGfkS72D6%7}Caj{J9_vnE3 z{|=`47yL%w6xA0b*+zl3j{h*}dJd!W3W@5{p$BH9;Ye+5P{GoL<}4~YmG{1)L8btl zWRcZ)ZjyIUsth{EmNd42K&)lX7!~w{b)`2sjj1c`=|CNW8MF`hLo$Tb-A-{t3sVId zwFV*YK%!UbV}gRJ&~u_U1aQKje$T7^w^y}!Q zfH%#GXltvj?_D++T=d=fgFk2Mw!c;HzkaysX;!|hxl&z}Zl?2=U3&)d_A^FfzCU`# z*4_Kc3HbhDMxF>kCFSVUSnM_8cc$qPi0Z_!xa13E;sZa4xvqkG(Sf?Z5cWd=Hi(5l z68b_0Zb(`zKiT zq;(a~FGGFeg_v)o?aXw#fX_VeV%-K#uTAaCZe!i*l8Qu$6P=cpb{#%A(Iz~57w9XR zR4Twa2XZ5c4*0teqksMQ{&S&mmgogiSJ{q~=NzS5yEI8teH$k>okQ1~#Vg+4nvG9D z<1?e(OcxY$i(dY~I`=TAdrSe>>TU%O~^D4AV1ZS(i5j+m~R6_(;*3Js`m%BrStK1sbEcZFE*>nePO zQl^kLNoteuTYKj#RVxvnzQG$4>NLIgpcr{f>d^}^TOoVXlitS{Xl+06>D5akJBXc& zKcNjjvNOL{(!fBj^IrEG$*ioA9r;~p0<~KM%q+ycb2vrf4f|62CS!E9dCirQ)K4uC zuUdAPXuY=@OiL7u461E3SnIvzN1m7t;WVSnU7D2X;VBKZ=I6>nC*Z{@b0d-FZAdLK zR#=!i zP)vI+Ij8joXxOg<1ec4J^uT@VQN(#08Xp}pk+-2M)TS5=A9zPFVgSL5FlPY+ihYx7 zByJs87Es;}HPSg_hsN0Z=!tQTdO4XF&WA|Az!jA^V$_Z;8a_Y?T5B(Aj6%bIL;KYW zsx-b3gAhv6i~0sA!E$>+p|A+mWqXOxsT=vl4qk5o+Y6GRv8ZjNJ}21rTKa+WvZ8IhU0>=;4W1_gc29de;a<-1)|scBOD{GFTMn`Obg(b}%zsJcX1{!%oIl2|b9cvL%I}@rbs_;Q@n>TDv6=6yMwa$G!Z)@aR zDn2EJNJU%vI1sLDrQOYkL#!}$#GZ?LIRXJ>uE1KK-DIBhSjc2pc_aMh{rYc>p z0F?+XUA8PFwaR=<{v;IQA>Gsn(hC(%x!=gUo;(3ec>ySWVff89SqFAn*TEH|`f0H2 z3rKKx0)4F(qY3b6@MmK1y!em^^bb|ypcU8KKWyHV9c->@@6SB*)ERr~Hi{dJneB|l zpqHS(#s}8-+JW9>WKJ*N*+V^l1~uEvjt=_mjA(0icRT20F7Bfgb>|+R`SO1KwuWqj z3kG`{VTo&+nwK3mTpL(-uT@?1J zA7on@ES_`;V!eUFz_C?Slaxeiq=6w+*Di*X`#82dij8%9@>IM6pa2?CE-LA4HuN{j z{*DhP6+R;S_ZA9?#=PXEh4ZPq)oS($RK1I4ynu+^u-E4S|843y5q$}r90rCUA6z&& zXfmn8T7L|+S31%!a3rn?YrQ-5qd5MHY(ai{|8?2E&Knb@dIg%y9z+eF&Vt*Moe#r= zLqvU&!gqc@Tq{XtoUz(z(!B`<_c*;GKJtZy`RTpO%Cw$33C|Jt)``-sEf}C{HR%dz z3JHeHT5rD3#rb#3rzlBlsyS1dy_=8BbtpX0EA-L6hV_G-@-};2m^M`i-G;X{LaW5s zw-J-W5drS(*n~i%l+ve0_eRG#3i4RvH>+)The>A*FMCYkOoKorj{D6rGcyyQ3Z~cq zjeQsRJ2#g*+c!sou$d|IPS_Flmu}<-dk4qXIT__n4;>oI znIPbL3~1H5RktEKP*%|SSDB~+T8DDx%V|;W5gR-opsVimdf2VAG|^gEU9N+Sb*s0* zp3zFI>p%;`WV5~IO3A6K6eZeR2gvKGSd9`q?&I3AW*7w}{uGAC7O96wh@iNI&7n4r zjL=26K3h#R>7af8Rw(^Q+ujJvNYPKWL%db6tZq?nM;bLTgr&AYPTh&-bd^;8O|9aI zwILlVHFz=We9|1?T+)durRuxao)wjo9G~5f&bFy9@8cu|rY>;;BBL%8QIw&?AbrP1 zN~8(cJfkr72w{`Gzt1QX$?4_U)VngoGMy1?qc{LpAO$cD9`*i#U`hl>P#avl8@L+g zqqDc;6tZw39Tq7BK|s4Rzn!D8>uH@K)d)2+`eFk3+yE&C! zwWdi8UV6r0k4r_nhq)N6#J$q2SlN2%xthM&uTJk3qLuvHnNow5K6Po*PN4u3u=xu~ z4ND!INhX0)==dq$A-l{pqyU*_DD+w)Zl4VmScy5hAi2EALa?R(d;(*7Xi9Tw1MAm~ z^<&chO~BZ%*C=7GKVh+Slxzvwy9uDR5`pgN;L=j+ohA#1p37}b?4Q^E_TrDMV)3u$ zB$e^I;kT8XIdi@d{YT-%e#vXk510JLPA|-8tJ;0I<5{@e3wc?W%Knk( zYoEOG?QA?B8y)>E1XT5Kl951zLI+N$FFxdr15vebXd<*6CUv=OfH;znJF+p08ct5R z|Es<0j%qSZ_v)(S%;3z}MnDjkA|v2PS1^JFg(#r(CP5|?5$RE+y1!TVuFg4g?_c+?d#=ZGgyhR--sgLt=Y8t$M=~+( zeq+18j*?;}e8C-q8|Q#N;z%oN>&_Ez<ee`9Oi>hcuj!zu~>V!OXKT6#EUoty1>S5@;TCa&V= z>8i|7$3FNKdAYgw5I0;w0c4xLz7yC;L5s^wRI$5ESf(l}D$IBp=UNpS;T!+<=nYaa zR3_NYHwE;dYjp+@g3*Qz8^8(JiIjSvDZQ?-ktGkhPDRe*UAOVE&cZf$KMC>imuzUE z2T{5*@h)B(_p?e%gbHC-P2sTF1`-kyX8PbrV#>R^yDQ7d9RSf(ggJecI)8NZ6qcT6 zlBVr97YZ#6#5{a1#$NH2kCDxx>!wbRuT!%VPFD67JAa>-m)DNnoO|mnpp6eR(e;`a z9WAz6YS-j==H9h*9ci6qMk`-zy1j+}Se3s<%wOzKWs{HQ^}q%->E2seMV#TCriQV@ z+M9=VF=c)+h^;^Tvp_rGU{sEIth1ni^T`tS+0s4EG!y~}y9j8}Q*vMHmx%GYt2rv$ zwnbGP9a!g$MHyFpE~~RrE=BMXI=X{e=EF1H*iFhbEA|-dymQ`lz~WO3dIj|kBU{ss zR%E)~3OgF@TWTc=CE4YXuLqMPHQb^*xlUYU8a?4m4^!^SEIj_Cp{c1IMUhJY5@;8V zomN&-BFw3|ULa1mhtn?)QC8=hKPi5P9X1abse*?Oj~CKv?2~9=9yNTU#89&v8D5F| zhF%d9jM0ubkvPFQ4}3N04EdViAvCnKa0Rbq0hlIBwXX>)LnwFbId`E5`vcCXWZ4A1 zuqFJBroLM<&H877(VK^kLJ!bXy?nENlg^HU`LNFx57dd~s&jkHog?IAr2{S`_rEH)Ozz)e zNb70ZpujtBCa1n-08@JFSS|z%w8;zcxE8fD-)2WXhZA*^f$^}d^$m@U+2t~nkb79i zcg)i$f34P@wLFk`8~cfn-lG$7^W0UCX>F}XYR(OkxPksd_U$YOx-9?1^2N&70ni`K zCX0M}%A4iwurJRt)^3w^ETIIBoOg9`=|WgU_x^=)lsbz#so|qpAXTKwkKtKat|WXf zSbCRyH{ko0Nsl3FG-fsfzk2H&wthE|c0J!xKxfNxftHoS_kJ*etq_db{BP2IU@DzJ z6i~Xb5Z(itu-1OLMvJ=&jTZw7P!V%9W`ZtAmNS*{j1;b^p6*xydSkdR>K=5{3rO_@rf`7L}4^KFJL+_4Z@2?DPA>2t%_ zsE(b6OM>VT#GXhE?IOm-DCu>~Pl={F({C0pSf=(F38ducCt~|IaWwg68~u&51i_T1 z4Qe*+U9VPd`=J*A23)8a17*Bu%khUhjB!%^+t4mbUCl$4a(OWlvI-lSa+EJDU8 z5T7d> z5T@{2M%yaLwJ#B9jR3XO3g#M#uEO+mDF*09iN07D+Qa5w&SemSFRYYM_z`L?Yc8!2 zDkg*%5jCGT4?n^Pb7qCHoA=})cs&I|H2a(Wu1(@k_hhXqS>WAtlL(AmEyQG$xtUo! zPREGtPPNAm9z3Y7;q|D}3Qe%*w0(LDpzenz+Q1NsX33-Av#<;fE+<4r;367J^(=~- zztDs7Flf{?$vR*(EIm9tnxe8cN`;+%738DS)3ni(G=i9&0R*S^`7|TPl_MFbg!{+Rf{{mb*wqZ^54b zD*u^_%5`|EOPB2?l0o8OIlRL4Y1$CY(_m6sb2<_nA zJo%?~ z3H(EQo(Z!KQ)dn&#;{z^>vSsQj_q@G&DpSVql2^arNN;gd#u9?kQ%{gQSHGgb?W)X z*P1;jZF|8`b(VSBG}hN&!z#LoEOBz0#W~p9hw;P+$FuM0f#fiqFovT*A?0XkJpIut z%Pn7t{mzX3+aJJ#vjBWbMu3=ZW2Gv8Uo25q%OuOYYLBk%{mkQb+ou{G@vf?r8ji6m zyoied4?#`52o#87%t|_#n#PiEii!OR4676)y|;6^)bqG7b5PjQmR6^4pa)pDE1S=y z#Je7&42!1}P*)6yi?!%XSAKCwX>iqFU&mZLPxnpkU$qjC@6qtiv-@MKBftDiXC*%% zvB_`w_fp@-ALe@b8>ruzTjcJQLMur-+>ni>HPzMCv$u9sjjJQS;prY;fLhwhUFwl_ z>f-x`Y?A`(l3)e`W0B==!;F$-`{eOIA3=ikAD?_Ig7@;k%bXm)h$+Q?Ct%LgZnMy_$MYSzgVe5YFiPLZ}xmJ!jH@F{~D;P7yP;B zeLv!VEyd)UQurdj`?LDT6(7)?$UFOg`GXgXsN|5Zt2>4R0lS$tyoj=<-I+cpdW!{hMArRu&u`&ItD0QYf3+>YFq1AA~gaB-81;%rpw%3DP zux=`;sI()cSQh#A%@6dK=}Gboxbu~WBReh~_=K>%!x^xqQloHOjRuVqpy#p{Pbh=m z^B16}@g&sUZCB&$ub;exRMRN11KUWY!1wf;OpPn7Cu8j4a3THG{5P-fCY%1mWo5p| ziF-1ZJ+Ql<_l!MUATPsMPI(a?`&d%uZb05}^i5tH-?AY&z~z;$P{(OryjJgU?Z>xC zP82iR$3zj;ZEHRH@;A)sIh3D{s&s;dyhsnh5o62*Z|B{jTW`Wh1KT(%Ha9n{U!N>r zg&u93xJu8uppf4wC@84u$LYTbc}NEprU11VALeNcBn9ID=|PFq6I3=OSk4173`nFR zK!&&w@ZpIW8D6M+mOp*^9f5Q6`Wl!3CLs^ul%&XecH|{cOK^Vi9uC7km4lYk9uEwo z7oBi9y&$Mgj)fx2g+%nQ?CMYaDDK9J{}fPLUbpfmU*2!}3olSTJYyz(2SdzBra9dG zVHbW$NYY!D8LDUK8}sF`PDcRmc)yUY z2L`%8aQdCMe&{sY%Em@9T#Orve;qKrV!|v$f*|;eiAfCM1KV1UN`>vP8k+0_0Pvg* zPQCg=(c)&qzd3_-77kuR!{u@rz3?soBg?W}5Pf`_l_iLlqhDoR{oxXe_97-aLoajf zPkO1Zn|G-P>xLc{*A;5@)pZH`w>ghwr}W`cpC;3=<^~Iyr7hK&ab)|Uk*SkjhPIz&#T=aWaL}x zu`?1UbwZlEqD^`IHCVV-cHC6RbF#wb5f@7DZlRWw#=Lmk9FasbHze!T&Sv3`kU{eZozg2xj#Xd5E=J6f z4~+*9+?(}f3{VNrBN`9FoYv0zijvY&1u7ztU(Sf&?GiGPljxtduMW9}fW~QwKG&VHMz}G%i z&s0F9ax`QC14C3ME#%3Mxf7V?Ekpe(8#a6dwj*BDYlAnP^V5@`mX3?sUp$Vxqx3%=_lkakR)nQ5^KVzH@ z;dJrq=phY_4vIB{oA$|M|CkfUycI1s;&QzhpQ~Lnqh@bApKSVc=O&{y(AiE*qOeO! z)F*mt9Kf7$NnQx!W&?)$KxEU&khb64_8mKHLPJB%`Ds8+sd+XB5gDC}G-t8lB6=VV zXV!zJ+>0nwn=gvy;vT#~uFr_s8zJ&~M<|JnGU3(T-JPo89!rMV!eSa7XflkMczI57 zvFfu3WQtm`FRr^be*F^eL$Cu<@@cLKOqA^{T}MMp;!(IZUi z*$}7_D@VsdJX_(}&%e$6UlQWNp~{z#bkdFg=~1W^YQD6wT{<=L)+x!|tvq5ejO7CS za1D)8I03}|5$-chjrC{qtfxjjIm&IFoN zW+AyQG;NqLLD31cre5iwx8Sr$lgCCXA+WjSbp98+LCp&H`k`*%$T2Q=*-Way%Er-^ zIga#9Ox5#u@f+E)Wj&xJI*s@;7_ct}4963fxf?Y#dTP$xKwP78Y5`A#z(MgAAKdnN zk$O!c!+7hC9T_vJTnPsGiJ|!-?T=ry>;Frur*-B$w%nsPph`p z?mpjl_I?132Kxm~PBx~|=1&goq-*G=;Kz?%d~bvT=j(wQhy%DY=g3Ax#MgjT_7C}P z(4aOIJYzW?u&oWh2~%RXZ{Od%`fB%Z7+johEEix6MnA}8vFOaxs8p$m3hc`oTDTjW;^Bx-~q847wdVu=Ec4g&8w5+=V zK*#vi@oKm9RT(oqRPO7o`(+kq?>G7IqV7>|K`cJ9GxvRDnit&CjSd8TSbzK~K}g@Q zC09c6j+UMzgYp+nm!Zb3_eOZ#rML(Y7T<3$!tNvXcg_6BjX2@oSC|`7`}DdOfr+6? zorNKCKwe@Tp%$m6@S16`vUNf6`1olNAZb0*cKVoxy(KvS`EGPZTrM}Xu0sJXTTNtX zVXl3KLb^*_{d{f>rFt4OYmk(2bDKrLXXou*go>G$+Faq-Q{Sh*eu|L) zDhU$G5eFnM0({bgS-@t?KY1~g>L(kho=7YyzNL8Mk#{Ryl@UFd10!wngC1|X73-2v zY&`~9BwQsY^BzNatU2*&K%O9a-4jWf7`@Ci%L$nB>gvE&&)TSaeA{==9?N%8Av+5q zR7*_KyOQm;OsWDI)gY8NJm4I_lE$G(AHZXrU0__sY-TjDBQKvlyR6Wz^boAW;ZZ*Y zA?l^E@x_12dFYP+)0gm4Gat1D76JpN|DH*P&5>7gE1#jjlwar((@O+A2o$U|Ae9D|+1}*0t*3)+6s1D*EPi{jPZ4C_AZuGPh>3 zsZmFEP0`dfN|r^9OZqCotCKTRO?U1Y)*q$Ob_*5y3`XcGT*ie3yaySVv1rrpWc+|1 z_>X1VD$XY*T?Nb5D8>mb8xyx{g-#D>JhPzDiV%k?1f_>4R-w!H6sr07%jzt0%tr{M z_%eCTn1GwaPF^&Ok<3BJC`1dvJwP#E_Qni+7V<7Q_LN~)YJ2(9OaYixE)i7UFeLA^ z04bK+15%1hkXSBKH)U}ku(vS*V&*eCf%KTiA@(kj2}zGWD9e{8jeATs+?oo}MBgnZ z<^>juHF+~4E$NTM@&~f>C@>AHZwMgHOuF1!Y@Ws!Ivq!NQNA&8hh$n*#2}i1kHiAG zK@aO~Ka!pC=O&9AtoKspI|YL&DF~h$tqW|OeQIC3wa@;vP+5%=t+A=;I*Ch>#A^w8 zi?7}dx|@FQmX;PIqfm>=?a^DDvGBZq`Ns?}tC`)(zFt}R0CSbrto3taGp55@TqJWz zEKMx|i*%>=yVT9-ZFJRij&%hwY91p7#?ja!-;I%#%|SiLpA=vk5zZh+^Czcj+~Ayc z!<4iVlL$)^sGbfjfRF1=wCmyek2DEB?sJ_E&ksKC$5(!I8#U>v)#kq0Nb!s!+Rm2t z!NOCuy;4mxDI9EC-lA?*bu|UYZs}|b@!(&gu;qaq)I4CTt|Gfib~+GyN^mp;!u#PD zxRA&D5}4*AXfbaCo{*OFo#hB5Oa})C!w~RW+1loko(XDP-vOGPot@J5?Hy$6V0)f| zbB%9NN<6_Dj8sbM2IfYfN}ttM;U5rCcJjU~u|1Wjz7fDQE70d&`$B5hMMMl9=-&b9 zvr$IA|GKI3Y8l2^reSq33`b=YWF*?q#?Jm3q0>B^op`^zz}5rSy~NjfA{DPRY*BAW z>uup|RyR$+XU^$uvo1_t#TGXc z71&<8gO;!pAoeRptAhiQQ3#m2D*XB{TmQ@}4W6*yv?uI3?M}= zKXKy6mwt2~x^9E2#=75y+CLm~vv>!u{oz%~&p!Xp5Z=32fB5?UU-%%NB}@KK$)oe_ Yl0kaO+N7$PPO?VoK~saG{U^@;3x3sT-~a#s literal 62146 zcmeFZ2T+u0v@O~*<~)w5fCLp(K!OB8vJnIo1q4J1f&wBrOU7}OQIa5nWEGSs2uMyU zN)D1GG^ixWLX+cNACBjo_inwabE{scS9QB)6q^qJ|AoERUTf{|{sjfuP3sxgQz(>8 za;Hx!Q7G$9Qz$F0u3e31#xwp{iND0GPhGNBHq*DZ)3MZ}oYS#3H#W02Hq_m1t7mCt zXl5$N%YTShlzaCLYin~WF+M(%|9AzjnWX{WewjLTe9AiW)9O|f%C02x|8H81zDyL# z7_Z#PV;66Q4pQx$l)LAaM(wf>@Z=;Y3IDbBucyXkbiW-7c(musv{Tzt7b>0fbGMj( z9Cxpxvk+L>>{!aOw}e;O_IB;@z9R+Fv*pRpVr@kcr?U`UP%41JNkBRQSX5-_Ulh5u<`O% zdFp(5d)xH>9_d^C1D1g?F&ygLJ5HTCl_gGhrahG?S8ZLipo{4;=gV-+Oer>4@uMZkCCD#gnhY<`l}QlRb~jCrxkP zz8!br*>m5f1EXwBlnT-x>0KjoBe z-O60Ok^bpcHW?0X?uu9Z-t_#({pz!AjJTo&MVD^{zkchpigNUsd3V`%W@#6VVjtF~ z=chczCtJ;o%KSOAX;tDhn><$Mao=#4**>SfsRlJ|mE*_%s_cwDAJN!X7d_D%;mRc} ztUA%(pnPZbrZJ}l3gs=oC8G$hB_qY{{Xe)?%BnlRb5d66&yM%?I2RtW{`!8^$Kqn6 zfyP&tlXdTY`}WOxao*l}`m6i$I4w+d#Gmrmv3X8#W907lRU(*Z&#sk}-8;h_`Z?1~ zS{r+kKYTd3fR_b}*~D#TI`UM|pen|3YPcoSBC@|BKF}m0$VZteR`u28fa2QPTH{4l zdiof~!#8itO%63r_Lk79rIV)G9nbsGF*3%5OFH&@iw|EY_1kAdhTw?#(eePEx;T}1 z%RKLa(T_W|W6lN%D75+?R7-Sw{u|}#)rWPjPbsXRxV7zxK8C-)NYS1CjpI<2OgF>5 zXIWOg{mlju4efc3HsjwFZH8J@;?u* z9VX)19R_$7eiUwLo(yVjUh(|-^YIVc&c`T5%G|7v@$GMX70)K?9V7pkE31EGAgMrj z+kHE#O`z_zYbx2+ecVI$wy-8z^;VU3bV#cWWU(-d+g`-ucG6A?#cppjV(i%OqMIxk zIc}|3v$;M>?$O08OTCXDKW2$DQA|%_3mku^oBTa7XIRN&2T%QrGdr;)<#+OGT!@fz zesisOv%S52miS2u#eBt&si~&#HIawS+9l*_$GXdDHqMd`<0YwQQbQ%|l?v}|dc;n( zZ8ZrNwR-8~#I$t%pyKnB;R(154OXlL%5FZgV0XKY4lo^krj&4re_^`QXZ@Bff$T#q zY2{znuJib~*W=@JVUw04gColj40*>4_v{=vchN~~2;FJHzPH>V&Fa$QL`4v4)s)ST~}Gc)Tr(ck@= z`q!!r^-0>&Dz7f996NSw{iaPS7@i3Xl~LWxbD|F{#=1)T>!N+IO6ZuFxLsXcRpKtn zX=Yp3WLk7u_t$ggR4>ep6if`~jP&Dw;ZDq&T9#$Eb#Itmf8BZMhQ|lhG`v4@Wy~2U ze0EIItajkj&QHKb9=LxX{L@fIN21@pGi6OpYC1YPaaS@D7ikeoH(Jw8nkVt~yF#kH z-@Fm}SX2~;yChN1$;m0j^KAC5(dYSd!{VD*BravNJH)>)EbJZ_i2rk+e2h-vJw{>U zCzB%`5uGO1c|DaO(Fpr-GG2`2)-GIlh^_V#&t#kQ8yWcd`HjC@!^$Bhc6sa8tz)?I zAOXGl=chL0Ar57mfBoysf<~T$N@tN*9kX5Avu__g4zjbK6rox_A|abu%IT8QWJK$v zl(brs)~huvw~i_*Dz0C<_QJ`Ne`ni`Bw~Atc^wexIH-!)Qy1wYMHT?3kkG|<@7~3! zz6vJ6a^IPN%r}w(;?A>g+HZAkYKT`mj|HThtoyM(Mv={n@vwX#pLUgn>#AqALoGed zYE!r0e<|lHsQ>7(bD!Ku-c)$q$!}i$ZCO^c9ZQSRFVBUV!b6JQWB$>b=n83>8;c7!#yVk z)A=`^z()jp)K$b1OXarfweayDz48S~Q#DM?rlP`nAg7!wQ`tQqI|1nV1G&`|RMkOi%Lv zFl)2@Y+rPxlnX+NacjYk_Z#%*u*VG4!yi7pYCX_c**B1JiMt5<(Io1{3sKFcdz*H7 zVm|cd>~3HX0(y&zin7N=60j(ft$G#Q`MbIogSOe{wFAHerWa$u4GJYTzPb`(QXMX7 z^yWs*-ZQM&*ut#`4jd5oYIUHEj`B+_PG1Xb8Y&aE>Z*@bmcbrQ^)meNzWF&F6?yL2r*ECgkI-E??P{g?Dr(QPxYk(_B)EAFG1}`)tMgm8-()N;Q|dk|DJgjp82DpyvMRj*X+YFj(_70C z%cjzHB$wu0Iff|0;OD9P@c~0@Y^-{?YflA3p>Fb%>ZHqQanuRpj(n=|wUN&sHH#(h z>ChWBCh!9Xbeo6S3^X1Fa;m5fZW}h>pbuBe-|&`A;>X(+jDq@}?1uOPc{w@Xq@<+s zYH8cSC&&a+R^wWMyqfjTPON#%r>)i3*C%Gv@1Jhc`tt6Eo%#%S`JHF2~b|V zH4fM>L+O=+K)?0K+y~{>uV1e;(w^6lpf32BTV06&+gy8^kWnLd7|;fGJ=rrCqZPuq z7ZSDdE<4Xns>sVfV2`z@^<)Fysk}VP0F+}`8zqMz|9JH#Mr&$PhIWAqqqIvddta+r zfl6;x7<2l$^XElmCrJz>psI2V>CI-it(hry2MbH$u{*0{5eVhaoohgv60#q=PQqAc zu}>3r%>)8ei)IskQJ;KG*6MHNw3`j}8D`p4>$-DDT}`8%#c8HgvC1I9;^7-^<$-+V zy2;C+4&y^YFSgbI{zu*L&}I7TnxW=Y*}16^HC$Di&48-acq-d)i!@C)5BWy*{-68E z#8ELR%URdi*=F!Fze#(py7c07Bv#pYmG}rjsy!})6(_bxMybTB#=0!bW-l$yZQs8A z1%mQTz+e54@p0>Y2M)wL&5ZSbdUnS&t9!>|o-3+aT3TY3UH@QPMyjXka~?jda$agC zsw|Mt%w>PKJHTbEVcpA~?^UxC1M$c`aawuV2n7Tk0t)nhPQDg=NarslW#v@ntgsGX zvFBJ;!6SFCnB}Q#k@(@ZmyONl+lO@o?_JJwNUI8$j8D|ex%uv|RkpId+WAiE%wpD` z@C>^ghPCPGu{#$tOf~oI-+!~_`Kj5_V%CP}^Q`37d5cA^6>Y2x6;(ziVU)P_no-0w z0TcG>Qu6Uf2hOV^b>+|YAKZRW^(-nao0h=%j)M!c6LoJ4>)7S6Vi^xzbwd{0Z)Rp@ z(w3>XZU5N|C}mvcM`blL&5z+W(}Fa=Tqf8TQI!lZ{#OLVy@26=<2YqXmb3d7*2aA2 zx%xy+aT98`3b28d{ajbZx^4TPOtog1jTD{MumkjGmbi7n4`_RvTrDtwkjwlno-65R z4=99Q0FuC}Wwse<*YuHI%>3)GzuFiqz2_I_Mo94@`d1>RuAiEdQ5L`x50GVu%`ywT zYBT*q2g~X5KvPl%!(pRpt7^$;;2DBo`kRs>$byQBVyFGN@?5ZxB48Du)EnzQr97k_ z_1R0=)`kH~Qv*%aJ^Y1-Dx`K{+>rT0x@ShI{rz)`ebEt(s0`TU0BFe|U4C<84<8@j zEA0X;4AQ5ig^s1}=6%~Jl*B#JIw-f7p~aX(UYC#n37I-uM7{nxrEn)yMn62f?g$q2 zaNd-bX6`LT=h<;ZY*9Kox+v_}0aP!IL4pQv5E^K;t&{r7IjFFYA3sh;^YG33lF-W*o^$roZ=0E6>3Y%hZn@b2#z6<^&=Kfhgo|jhc8X zBc!#jWSCZtZd7tzoQWRJr74i2w!OW57Ug#;Q@o6GujBP=fA z;u(&vwo9R_p02&XE6WdT-{Kk5nr#zbm_OvDduxhnSJ@pY>tmZ4j;(hM%SGXuou2>6 zp8i{MGJvI{P1>%a&I$@LSL6kbpE%)#jI3{%VNlHyQX?b1G;b9kY|r0PTlVC(LpUMfz#pzD<03XJ~Wg1#|XD2qZdW$g#xa|~KAC`Om{NL=_GA(Rd+1c6YQQHd51&iC6CN_3I7@URXc&aYt}6*6ug>^KPQCD@AMuFo4sIClVZ-_f3FPj zRUl!p@bj3wZ6puwmEe(G)eUeqn8%>7l@cl?Ek<_o3eNahH;x zCmYu7k9OA2sP6eN;hCO`T1F8WiZ+9iK`TJG$$LlsPBom z0nYk$(-z13>U#2PQ4Hgm?b{!}@G)sk4;paneyopge(UD(_e zwLK#ROTqx@QM;Vx`Z9A#-<)1W*hac;M+Nly25B z{JPp}mk22=HfuMf6&Ham5i&TpVeQ)2a}Jw;&|Y2Eo8(9J&RrhiG$wb#PqDzonT>;^ z=0WJqCW*dje|9zb7PSLcNaqGh~vqte}7tCT2X z4HHaJ1`13(8kwJTIwm*QvN{g(Tg@OAT5fajc3G+6_(R01r@}bmQQh@#*r>y=lkmIX zq2bIosmxZ<)qTE|fgz}q)nYPt zqVbBpwD-&C!y5~4fuLe83JVQ2_Q(2=xM6GSHVgb7v>*gNqb794BCnVF z_4wo(x$f2%cA7R1ajt%R{dwNs`3-=b`sq0cqoJlN)biC#^CnbNU!|V8k#HrW>TLMM zM2+U*H^o6sMts8Y+YTtuBQgoHGl0vKadOJ7qfZJ<<;+OR11@?`f5iN$#r(8Iv~zY# zM0|(6YL?u7{q_R_tSgYisM83nUO)#7Y%-p&<@)d5y~_X;W7jv)ttY^r)0&nv14Jhj zIDz`$s~8}H*A>&OkH$tL(|j}+B%Otg9uh4$A z?(v+%IaAJ4!_><3$v@AirW#bI_XUDZG}r8@t`6M`4vtNbVD+{%Elb9wU;p;@t$XcU ziS*R=UDPX=_n|}Ii$JFI1?17n-`_phez0XB!SW-bdF8~yi0e{$io~8h@3=2a(P&l< zp*{aZUh71b3JyHIal-~dFigX*eNZw5-#I}LvtCx()j<*mJr)xe&dLKmYJ22$o>uFg zu&74ls7OC-)M$O_)06OUQz66J55{M1U;Ns$kHQi4U?-nxyR_q^v3Gcgy+dQc;_Sg{ zUCUG5Wse(PUCEFbN=3cpKo%Kq0m(RhtxMCLTf&Z5>^61 zzdv)YnA@;}`?b1$-dQ*HO}m6EbXY+dIefc+z;at6^;UxNVb{5#4BJ+XEXz{`^J5-S zp0Qs;4jbvGI-x!gvmc9Mj|J#KjTT-wF)>WNg2FJ+pMTR(ta7w*fGbj$A3L^sPT=X1 zGmTBju1hXpn2QXr*bPOjN^7rhtwRx_*SZMa=wYB*JCkm=H8qPLYh2FnqprnlyPrUM z)`aB-#H9a4Z(8~eUoqp3QewUDHb?cEa^ zkDH1mgM7Y5fPJ5tnQ1oZEwfm=eoK?jKa{6#dW*k(8FYUxN$~p`R;S%^{mrS-S}rqs z6Mc2EfZ-K&^h1UK@!p-43ZT`QT}Cg1!B-Wx8}?^M&A6v}TX0sg!^BP6$oHY4R}6ys zM@%AS-WoO~Y6_Pptno!z(#*0Okcf9@jlK%PWZI3I<47&_KrW=92r;^TrZz!6wZdqN z^z@fIz_|r3dG-a!Sj~y42%{|-W)V@ID~=3wjDnEs>6pTpsiKmzntHxi0CLN)<&9y> zYn?lrS)~$?-`z_+@z&Uju}YigESnaC)CMH6-}j#nPgu2K#|_kF4}5&ol$89ai{31f z{Yn~%4?H~T0Gr={d?tE?7$lWMjV!+bo8F?b#|QVxef|1XcrKo;aiuQE04Nn-0$5`j zkMQguHQ$#vCaL6OT>b@{FZc+`18^W7nO<}2SLe} z+b-CD1sg4}?WQV7(T3z}#e^-Y3=>x)wBk#JusBe8@d)4$A7<7hN&5QwHf^`EO+4qc z8DuVEv1@=Wsx6|mfFg}UjRDYe9so)aF@$zFzaoG7+lI%S%0*wtDmobM={)K93L1x~ z0b!G2_N-OBzF*6FO~Bx7XJSf#aKodSeUYzVJ|(LnnBCLcI{*|#u!vbA8DYZGmBs-k z1Cwhg$p}atI`rvl_ShK701MdSMB|cE0@K$$G&J=3mH%jv^-%J5Csa8(Iqry zhMJ_r@N}2Z;Buu>R8V60*)FnI2s3>xU)5kUvzY%-a-t5sLSWOYD{3HQY+7CdOU+{X z_USq0w^$fjD)4zYe0bFtcg$r3%>M_@;=bfFL2!oBpDCb8O8dR7k&IY_s$~5S2 zz4}XPxKyJpfNFP5i@A_(Npbh8r=Q9s`k$V>|EH*fDVE&C+Z9_#h!O)IeQz^!9H0>4 z&Tlp*1flYKfudPV@{g)lgEDkI_iR>%$D4SwN{9FIcgbjIgxW8GNT~x3=oz!>?CdmI zG;=-&TD{pj!CcQVD-K);5m6wKky>UTt2aDVJ=~t|WuEWmrFoi=%{$7@CRX8i<<>S)?y6I75uhyIt4@ct^!TzX$!qO+k z#*{#&yfJAN#-dlix;^~pQ_$2pHJ9&@MH;%V>zYoUmo5D;I2oHe8=9qx*QT7K&m72! z23gZz6X{OXtCpM!1!Rgvz1WAnPoBipJswK_J92S-S=!tf*#G|5Ce0~&4S9}sMEUWH z!hYI$LjZT{&Fj;iixM#?hVV7%L*?MBUcWvBlHgt6B4^g5eTb-BCJ9@DUx1N&O7{IOfA!>e*=*> z^=MhPv@l5cJ$v@_6oB8Wg?v*nl2joWrYRm7`&iuD(K00|00bHdMg=PrkB~KXK^~RU zj3pmrBk#Fsq}GOXlPk_Y-mU4O^M;a2^vlDLyzCtvr5>2pz=mWhdaol_MF8hoA})aV{|xLJHOLg zd~or!{OQvriW<0D+lY{vtGrerp@i%Jan8=!>28!^O09o-Yz4Sq1t6Io>*vVmbtnXO z%PB;c1|*s-X{21gU&q-E7AEn@lP40|N&fs{vn#epT=h1;`T6xTMEoRBA)MUYXTjv_ zUorCI$uZfJHOcUUYS+;`cm}xPA_54Z#6EqxAlPgJ!Ax@U^EKWiagnhx=gSuoMmDK+ z3^e5@$p>(A$_cYsj?Jd-6RMtZ1k=Xu?zBq5;?YWxjw9z7vuc|#w3fL%iE6MtK)`qjCP^K! zsSz=<;^sVp(!0>(1qgb!5fS#|o(C=Tq%Kv>w}E+YHhHv-RL`!15MH%``mCSCXoIIL zU)a&L7!)|*b^kv9)D)7Il8?mL>2&9=$H0f0N=3IPfIaH7tPFC75)K?6Q$BtApQ;Gy zq>_@;z(X7zO@TWR&1~Rp2wW2OZdVI(jp>;%ICCtYZ+RsRJDBx8RtOU#%gFDI`PcWy zBhDtM#0S_*A31VKN~+2=PCYs*AaNw>s7pDb&O(llzDRJ3N|0RU=H)Ob7&4y-2mnw`9j;3}rRhhL z&V>l;-}d*nDE&C7FJ{$b;WJ?t7#tku@9%Hy5miz^rO$0+rcgc$+Vo$fxd1AqNItDn zK=N(CesW%%>1}}mNjN-j-6No`+nuMV1O3tkz^#0;CW(?)OEo0JsNXpZR3=3q+dIgs zB~A@^P8=rXRO_9MYUyeU*TeOtXydfTIY>lAFc^{&p`FI=cB)oPkk*|)_j zSK&2rlC0Un;_uJDkQRAT*CnBqWngX`iif^s?(5Y2B!$o;zC%9JU}Jr6Kywz7O_5KZ z`+()2qM3BiBq4FiO8mg`IzcrZ2)r^^7T@%cH6ilmi}zMRI}G3oIS2|=j^TblMP&M5 zY3R|4&`aFL*naxn30B?C76i|ybvH=aw3=iDt=nWY*4?tzxV9kjb3%C%^Sc21l@Tf~ z&Kdd#Xz8VWy_+bzmryYi3Ks-Xt|kZ~fU5so9E9*q%&M_sY?@;*qz>zq`PYED=zD#H zYtNozKq*$4-uUL=DJw@On^(|120$nWu&16bYpC8muy_kJ#R6#j9+&>!UV|#@fGjGg z%}CT=trhsk?oVZ5#gd}UY9QUq2Ngxuu8*S7i4h1w$b8xo!a^0zM#%W{1?87#`Iib1 zw75+Y*6eFuc*wODp`mP2)}`mpKOhWRxyvO!n%J%pM2NS_ZWDVE=f99{2)QMT*~)U0 zkt0gQy`0yCVr;tZut5>YPqfh-fbm7Z)gSv%<(|%W%qle{L59cM4xA4c8c~Z^t<=fE{I?9AZ_a985X?w4$Ri2QHAcL3 zho6w^;{3or0teNWHqRuRI$q8&p4czo$bx(H;)M;K{n;iV;rE&NYu! zZ0F{ojio{L4ZOCT?Yas6k=0tYW=+V*NLxu?@7bQ}PNNJteFznw5PriZAG3}O!?cqq@f6DD`yXAc#4-`-Pdju9g#)5%IA@LNoVa!)6t`<# zLMeoG0r}sa82B@cpiteutO&5J5Z%;7C>W3(<@&A7orAQGvu#`|ai5t&w2V*~Dj3M7 z<@d4^N?)O~^gc5E1;VFBdG3wH-p;(pZZl+T&}d>lq(?)PyI$pz22F$Z`N;0wZ->ts z3`-1`&P!Dd=On)#iZ=HID|A;Lf(|gM5lbeisbQ2kpPz7$i>oZx%7$kOtW43bqYg4T%WBpk`heC`yVp68DAMaiCupiUu% zQK%llkx!wiJi~{5=Fae*J&Q~U1~0g?$#NYSHTkL>(eAhm` z(5?aK2KmLdk^%)&p_S&o0Rb<8qg$NYCsOvWc=YJe1d!mF6zPw-X;c^}jJ*#q?b}8h z$T=(yl$MP>$gVfICCGp)s}We8X3V~a@>XGq?S1a#-Ow_-c3b^5x)}4BIGBTm;-;O-&>0)O9yK z472Dg>cfaX_UDHq`IJ&)xrBWB-SbVuu272fEr2h4N) zcEcLcL!IBgdEy-mn7cRj5{C~r!~k%ANK8TFDnc!2Mq?ZtV$Dj{uVhTEmSuIW zZ&NFL|K1bK%Ms^cH~=eq8)KD=k3iH65Qb(W2up%qdWfW>ZGdoZb%Y4fjnnPno%Hhe zr^^{QD=#kqN0AxrGYqc=QW4GBTx_qXW!}bmfUqFg+B~<5@!pVK274!p^77gqcm-v0Y=X-wISemfOOd;m1*`5%ScCWLV>W_4{du7gJ<$63#AzN-Y_xT05m z$_o+pvsa-*=rxm_^z`3<-hR`=R;4n$aQ|4dNF&c>0W zjw6-I(FzPuTLfFZn8k(9gn*BEQVvA8z+H}syD5Qd79Oyov+L}z4J`ak%&l;pky^qfMO;-L zP*zC14Uc-tNtnmfKuU`DScozO{eV>_ zM0pN7@(f(nN`N`n>F*I|1;S@PC;(ysg(3Su^e9*oRiwEls2MgW)Z+j?gq7Q|G+ROi zQnAEmp|p;}vc(}Kr3E%c6^Oh6HH%8D(kUQcLKH79&Y2O@FN8yX_Ea%yS!?SR;*9_q z46RoPsuJO}$etp`9o!}o?d-rabna}Nz&4U$wb-MI023vE;(|I)VRqC z+}q0zm$`GKj6=1lf_=M*QRp;i>x-x^RbkacNZ=F|Rre#;83SHHeRkc34W#aWiUsl= zONh9dTxPm=5{5pz?`23=Z*Mem;tkM@!7%;70bwTidvCboup$DbHJls7x`5)o2HrlE zw3{kn(ym(2=5k!HsQ{#a*R+V)T%$$UKH0UWXd-cnaHL7mz@%o9ppxM{xr^d0?O%rr z+{pbM1qGmjLk%qEW@h@5i&^Th3zHDw5N+kj6oKL-o>#D=AtS}PS{ZB|PBbsb=aGoo z%Giu|h^rFLk^b~nQ)PG%Y=9r2XP<=yxaR%c4cl2+BlrT>TRaX749s7cyaG^o7S5kW z+%T!H`(cZ{h{b6$)+I-LG(9h5p!*+&BWcyjmDl?Q9;<29phA7FnPY2+=E5=bGbl>sKF{3_TPxX6u*azaEri?mFr15!(nJxciYk^^Zq5Y>oPw=K zq_ua#Ai`#L7-d5+TlA3E)A7kkRV?UnV8x&z;NwSQBRLJ*Q9OTgU7FK&hMSEPyTqkT zb~(g`v9YmQ_@~H#k>G&nC0zWC;ugvl?d$k2XT9vGStM21J;^R14V3FZLGbQ2z=&60X1p-h_`X7^cX{ zVQM^hNDH@=Q;WH4pSQX*E5om_m-313}+AU=CIvQfS16e(SE41y1IJ0KR(G=W`k&4QAdiN$5!OO*L;pi1{eq#aKf6i zIsV-0wMRg@Q*D;u)9yKa*f9ge6eDr>s^0IFIeYeOzy$FQz3AN!V+yvN&1^Q5ozZ&b z^P=$5X^S7ETPbG`4Loe=CW1(g*`8$qjzCUpVpat z1{=+W&Y#u=&gjU^yxRH3CuL>DHK(rmNa_!E6m-nHx_GAxZ7j?L7jJ=ZpFPg<_dJ9ZDjBp>PWO3X;(1VT9R8}h77FVHI|lv+Iv~v|J;-o z8K3B$I&w%h12IIDd257Pa<>q#?1(`;5+}8f{~z{FsBN~{LrGrcVUawqh>_R(y0x_j z3?l#@U9WxGe%=Vszj0%l?#J~xL+!3h&LOr-a~(^elOft2uM$t6J9ij+rq`SbDX<6# zT0<;<1?BqVx{+J|iv5LKL8zxf`cEhQP_N5!3m(;(pHt}E8~t_K2BkyuT>fq=Mtjja z^=`kd@Px{Agvtunpm8XaY#mo3MAbPH%V-hSUxjm%dK)RW7jXq*3l-_@iIS7$rmH#Q zp}GVQK_?$J-ye&0+nSIB&B(U(#fujVpw;zr-o8D`26XmymTFHh2az;Gnw66BKyNk? zJKPypzRU=kfCZ)G0O@2=Pl>{^e~zt)9C(&EIHB4kBHbM-U;7{w4#fbMg$XrDT7@9V zL-5#k4GbJaiva4>7_2l#aoYrJ<$)iJP20IH5(;1f7O3w;+yn(FT3gFZ%(ldni%C@k z`rQ7Vyocn8?Q&`Jx^?Sn$fw{% zeYe-JTmZ*L?u8J?3F^v(sfH=;DF`P6@dpDaF`LlNHIBR<2MhrYG8df-q_YkHA|7CD z7I||5AFqDYU$!iO_z|vC28(gecFN zOF$Ck!ud;c1xt0f4Qq&dVcMCcq&YcD zb4R{gNhYDpiGk30vn?~3WL}7IrvN)kY)?Ar5LSJtElb%fZz6!0eE;o90$3)l?ZXz| zPU6)c`_lP5fn-cwT-pHadIpFSl<41tM#4B`kj5jvfoh3Ac@7=AfQxWUABytyym1lDpKY69Y3M+-`@ut^*Wypu5QE271MS!xMBO+s=s!1aNq z2YC_>P+XC#PRgH^N5!R#G^Y9BBRa5PmAenw`=vp+y$ot5W#}5XXlP7{&}&GGSuN(S z9#^W7*A8@kez^k=1o(NR#u^Y2DQ}D*iB;NpMyfx9U7Iw1(w2}Q9$}vrPUO%siy1or zkH%}y@WnzL#SRwA7($SJ~b(A%)+NN&~t;l zi^AamtKw$SO%#g67B~X&L@mK5a2L3jBYwXkZAeN*;{`;nId<3f#2L_SKq>p=z{q6o ziN3pre8|&7m{#c$Lki_fkQ0#_@Ph(0qb%YFN9iC(H~=oy1P)lB2pmOsWGyC6FOO^D z)3Xz#h!K%>y7n1PU*cy6(ux6^AuY+|;e#sJb@`?Qp|?^fZhQH_loS5OWpTzJryNLu zEO{>$$!s^a91GGGdiMQttGnAg9Bm`MWITSob%^xbB1*k+nz1Cla=_gNXk8>)ptg(! zfO9|-&cy;3XFGRtWB`S7bua1r{37OYY@N{Z6Tkl0(%n7$>*t?;?=FBO_g|lSdu{oO z|MRK;d!Crd6sL4;@%Z>{_>Z3-opNYnA$1R(T-Sr;e;?Jkcp+=A{{Ok(|H-kkLVhkd z&+(vJU2*l|k%tF=9tm*w*s@2M{eiQzjrziX^orNr_e z3U6ls!$T4UP1eoF#|JfG<%s-lX*lIkMZI|WQod}O{{o8p;uOTyJ%0G`DZ+h@ zI5&Ae9&T$obJw=Lcpb5G^ykS9Go1YVib_Rm*SV~Gx*J5*b^uN$W*zFzUk5#u@P+QY zokKTC+l!`Y?2>?hfcycldw4O1`F!PKKb?U!)AjHwzV35$vTLXy)!@PX`Ser(fm&Gy+1eOg+l_`17f_q3V>1(*wHWC+$hV9)L=Hw z&7lnb4)cNvh*lN&W#Pe#g`!UyN7HPFE{CL}KORk$eo(k2?x`Xe9fJ32q+1Ihmbp7041mZqmxWZY)YVQZXn;GTzE_f)O3?iPj917 zomVbSBP#_zn6?8_b{+~qZG74fv5dSv6%a&pT(WWbC=>GvHB6@HU_ENBs# z-E|H}GQ=spc)-rNe}9pLd0-U!9tR-tTm-V10I74PjGoii&+j?VY4f-hdU;@lISGFq zMk%mT08kKV-UvnF<|gT@Zz*VilJ|o-<~cGUsF?34M8`&Frq+KVy*_|hDhSKS$E8lW zfN_T&H2)$$zq|~6uPst zeoODEw)YLoyY=eoYg5!jJ0xOZwAFysGc6D_Pct=Am ziBnH~31IeOdioaPk_K^xfGVnwl-&|i!-$mnn4Pek#QpKP7Yi>@ocGr~n|C74u|t(c zIDQLlf)}M2=^rC~=anhNERJU}bk(Ug6C)kkNqLjOq??bl)sWsL_U5x}2ldXH9h3v8 zc}C9AkhroutK`EuT8{92YhLy_P$=u8Q7FWM+EI5|85JFE1h8xy(b?Vo;?0{kOi6?) zAq*0Bi9gmRso6k%qy$}7^+RLBW;wJ79kj~b{Ogh_vgDJO!;hOLn&4aD`az9pDy2_< zPghs>4bxYioztW>9TI$O#8tba zlbXLcPdg8bDpxW8ukUkv?m^od?saAmQ}#9yrJe%mZ6kaVfa@86T+;VvJKPpaDo3ol z^081v4N?XXiXAwe17Ag$RHUjhTjv$4H+}-}8!tPU^kgZ+088m+Yx=9zTUZi^xd4cr zczi$;#R2Uap`)0H_8_dRoOk`YAPOBr=+05&ie+q*!-5u(bD>8R5etZ6cO3CF7TX)q zJq+J{V@i}v&Tzq!3!#lk&7tMGKxqFDFD51io0YMRmC;*jnm?oOJ`jUu9uOE&X!syR zIq``Srxo~G4gmorZKu(@q>pc)2^sb0;{i*R$1PzG9r}AYcaFBFB328$bX-As%0Zet z(6~mL<1l+U;!JeM2JcWP*LSOZ)iQP^Z{}bukt?KXP}gh_Q+u~>a{2R)KHE7_%Vg*N zbU91?PU%`FrM$dRPHhVOyyE9q{`~WQ>9nNv|G>0AvxAv=XxRMczLWUNC+FAaJ$L|A zAf_3Pr3X1x&JQ)2#vglrzV4R9^47q2y8UhmS9WxHhvSH(B=};#CE~Fv89e=k-Z)l*iHii+A?F9);oHBspCYklm6D&ZS$hJG zwE0d*TVz$0l&j37wA#^i_AhZuUJ~MawP{UM&yG!kTGu{gF+xzfCotlf~r;8 zW~^1x81vPVm35J>L|T!r(!y@-gIx_Dse6_`@WKrkE9D5U^Rr(A?U*5KKU3=uriJ{v7Zx zTM4lZibJl9+C2f3a%PCQi8}MY+V}4`Z)dl5&;yroR7y@>UJ0<2KsP_XZ^PynR=$Xi zKFGs!4w><8e+DbYa&~IOY-;r3!#_yuf;SjD?GZxI3wXi@pg3@US^Mu-mJ;$@P6>^2 z4LAlvME2*x_@d77(DEIjdfTNzGwx!;t2x>bV1#r(tR*%zWIoOx6SAjH$@u|2XxwD| zN%-GDgZu+8ugVOPS3+kHQ{O;JmCFi|l=lNZhIOa#;>nXI6KCBGW)8NOQ}?{)Bf~et z4q_h{k3i@LSv#k2^cGTMS$X-H_V#vE+*}t67RUu*H&xCdep+2)$AL52AG+qpvQ}~R z9KZc4jwIoHC;RjRpT~d(6=4+0l0X#W5J*UsCFvWShNacdn#rfwe(Sd0Qx7(;k77El zpwNiwEs~6jmq`0VcA|BGt}a(`bzPVUB$0s_zL0lFdp2=q!zfokY@jp&Yg~DT{aEP% zn}2B*bf82Xzq_8`iTCf<&-ffeUP*>x7DqTHzxsGJzPs-3IU>99KKw?Y%A@TF$HQzH zq&UAvk5VOYXpq%8_?2Lt$m*Yh|AJ_E;G~H3OPa|4)mcdJ{0{b|OwqN`uqM?CCt0-p zy zN$CUCX;ps&YqyF@N@^M!8bVn5Hunc5aYcdAmsn!6#t}@X%NDvPmM_`O(I3nP(vzhf zJumGS_Puu{b4izW_bgOY(%Ot7iIxlo6g!JoZM!R!flq{bK$kW8;Z7kRlRi8mnStvl zJ+zU~T%fzPwDLD1&TtbQkJRWO20${AxQ>k^PHchb$sqiOYQm*GayWKH&ggyfw*aStuiVnq{8(U-f09jMu>1_X9f497WT4E6c2|+Z!u%Lot1 zmE_Y+z9eM+hfs;TTavASTQ#2&sNBy|jO5dnQbbJzZ+M~ls zcdujJ^1?`p1?UH%)X90noQC3L_o+5wNFrIDP~@O`F}}pp_k!R9a$piVs>kCc;Saa|XeuL?c7Ohy_Xo+w%ms%8U38>-!l$P_oA1SlUdaoeAGciYgoMs{=sp zVC+KLNuGnvigezPk4ILD0Vo$<(@VrDd-15Ks2DKB#3q0}%8eeh$|w9MP{G7u!x?LZ zq*WVDZQ)Du9I4u=?W9N=<$%u#zlVy%2*xG?n$S9ndeF_da}+@@BO?Rn1x!n=bQ^d( z!g%XBZiKk#{G!0oXGvT;vb_DwtD5b7iK+?4+}uHnJr$swaKFO6r=U7~vtFpG@_;NW zvourfO3e1ry{9ygAc$bBC(1VlHn0x?R5*!8oHS9`Yek_WAXwTpA8KzjOc~iuT2Zf` zA1%8NGuw^NuXV^0A~fd`Iw?tF%eEZ~KCplk`W;lKD%#Xht?uK|%Dn_{Va20-WC(#d z>?T6F9JDd5101L#z6QdGlQG3z zo=2!5W)O0S9McrpyreU?d(w4&)P30iE2`C$Hs}>j+J7spGstmZu%-mqg0=Qz$N7CZ zxl)eO8l)!-lm#c)3s}DhEH7X9jU)KER&eL=A9Nyd9yuo{=3bD$`+j^a^ot@pTNG!( z;UTf1_tX(SOP=*n8rQ?9IE72-{9IZ-Kf9izmpn$Vjhs+L$Pm)ZMowJ8@n#5lgv2Iu zl#OnAjbRePKy2`4NTc;G%5_6kQt`5pt&tp6#`?SW<_c#gfz8f!wY3^r+E`gbQY>hI zgoC9Im8Qtx7A(duS|E4*34QPr()fI^-&*CN5nrOM%g4jR(|VmW8RDxxxeg`v5R`md!XzD4(S1&QCq&aEISR#;ZB|6+{Mv_$l82;mj+J+~ zfsYQox~ZtBRCz3NN{jP6$Z|gc8NKC!!5?n#-D!J8drN^noNm{n(y$(QmC^6Pm;VW? z%lJVss#Z0=9uP#NaSVLQC{?LRY>F z{g)JT(2(T(aKM?@uJkfgjcu0*fN0XR>w^qY3yutD;E0lARs6b!hIlv)BG;u@6*JqN z_Uow%8&GoO<4cmzT$;Fqj1*8ze8V`Pkb{fsL8r*CJ6JDT<&jzSsM>>W=66mx5-eF< z=PgK{#5pWPy~3#DTUEl2nm^Z-SHL(24lT(FoB4Ho#yI?$?_#}Z&rn3PUP7}}RMyg?#gX-l+To3Z3-7eH%fCU~b^J+|4;hxy;My;+@@ z3#bHcIppjl>4ot}Fax~AAojr;MA`$1FoWA62jpNUzT7+-|=O97*156Xlid>^PWm^&p$}S!mg>*%1-?8H*$wpuwk>$3oS|&=#xX>vf z%rKvq;#2C{5Jl2BoKyH#akXln`}Y;05K;Be)=>Xir!bLC3@bX9A65f$J_{jUKwl(+ zFV0O=L@I}@&#BbyG(Gd>_e}W`g}a z*2|T<{yNXkHTLhHUsf-Z%K!P?k4yg-Pc;4G<@H#HzR5Yv+b2`9_bVZ9shl|+g~!KjKfNorX)#y?Hb?AYknPrqoQhXqIYez!ZA(0gjV z8Y%i_cBAY^;i!x*0yO+S4hZNOet2Q!El0`2VOgXP`Nz(;;Q zRSB2Ea)F7qa=-t<-g`h*d2U^!7?1J9dSWb5v2T=4L=iz+^dN3g>Afk4C`A+m1Zjye zCJ+&~6f0msnt*hXrcp$?g${y+q7+f8^gGwPHwJT(?|%3G{~hDMV|WB^M47 zMuEN|)i?4hLl|S2di3hRg&=*CHhmY^*Sj-FJAogiNdy%l-y%)2d~cZ+c_bEhg@r9A zdsiSd&RRclKN)gz;dK_-4nj%s7*$CuK&%)buOM-Bu&|Z`nxZ-%$Z1BL6VDY|3lJ|Q zDWpIHmpTYJrRBFh8-gEtM!WT4W;QfTsHT?~Vst zhf6F`uI0En$g<-B43SYII@hTSUmIR}ia&EUOA${zr#90-cX z@P|C2w@Cd3T~KqWFE`gvy5bK57(CbQ9O~Ug72ozmWtk2TGpdS-e+Ps?c0U(aahJIF zqD9LEqP)gNotMo9+=VwqMlLAW-{O`@0Ke4z{rAB`OCby-EiL4T)WXG(oVaNkp_4YOmtDXJ`SD+WY8V9s^Cm zRBiQkSZ6?d-ze*O)bCC%R105l*Yhx?wZ{XdQ|8(|im~t5W89YfmN;z#{=jN2HTiJh zX^>35g?UCswi2!O7`OY86kP0YfCD-*z*Z=HF$IF>;;ADKi~_sIxehBkLsA-NZ00=- z!J_3&KcaHK&<`J*=B{ocQ1J{^IRIH~({>_zro0;dI^;09Xs!8!-YL_WFZ13kBrpHG zc@*RW@qXP2Xw@cZJ0Ye}FOlgz)46(Dtxrhq{&$ze(@MU>gFch-;-$u@kIUF?gFaA%fqu7K}^$^QM3 zt~mfTH{@!N?U;s}Omg9cxbnXnz%722eg>zC+9#ZBwlr{wo4Gm4%MB~-R zle?~Sg(Zop>3P8=?)Wd3ed zB>RBUgFSYF2}>sN4V>=0ka^GfA9CNI3H6@yPix3HP*^)M41k)9UGRW|Yh1H8_pV-i z8(ft^&LlKQ@zrk;t4Pe6dhmHP%{d_Jhg^3;)T!OshWx=@qMNpSr7H&`uy7R?oL z#dvheHhWlUMHmG`k69F6*v5TFFCLa9TB=f!##xVvH3APcc+3W}$ibW%7tr@4&So5U zGq*Kx4>6Y5=kGw>Ei5+5V8;O&VAn~mULZdSb#70s#gJnWeO&qV3g(LV4*mnu5TKg@ zz{0OUJ6ebyCDfg0OOUReu}r}jB*OH>*_t9N}`3_k7j@$smyw z=$)EP|y5AN(rG;tM^var4E)CZztr zkoFIBHIbB+ND~y0tx#>~fu5^dvXr?UZxx}gDViwp8TF||_Q%aYt8WPfk;g4&UO3#w z3cs5{p1*`}7I$OI0606LIus3uo0eJ<^t8tKA5=UeH}n}AT*0C z)@i{O5Kz{97~ZRZx;69I_n%Hre_&}W569`+C`w|LAkJ(F>`h^fd)@d^pOO-+?OB_$?)&+^iF0Bw+c&;SJ{ zSyE0<2*CVVy9|77EAakGJn_&NF5NSJ=p&gMVxK*c|4ttNe@Z>^Ki4t-8tLEvzkYMP z(vWvQcXdCnZX3V)Ke?t=at7u_Tj~(0O`>YB`x#rfv&wtw%Gtan{Cu6l@iIiQUZfb- z;I|A=9Hj05xIuvgnA;ZW?u6$2@D^UQzWwCPN$yz$FVvYXC*k`kYz71yrBqirln7aF zy-z_Ip~rN_eKJBOc{;pOe&6|od(?}q%1gPT$*}zn$9^aSyvkzZphQ>q-+%gweDSbA zq#Sy=nNn_~LC-*QNTEO6$#efW+F ze|i@uYb6r6dU@{40RR`SUc4Nz+YKx`!WfreEOynV;_AJxa-Xbs#0uUdP(L!)=Zh#> z{fP^}8W)W^&8oKEJOK=03g1d@%HX}`)8~6`XDPJ;rzMpT)?I=7-x<64#dPq#TIADJ zoej>Id=$>gM77g^IJVrS~6e+nX@F@rXmM?=!+dgmbXYwxR%F!8Ak2)nvr`U#ReLkoOGUL*YiP*fyT(m_-y0u{-@Hh{{Wc(u>u7mF`(`#{1&NIC-7VhHdvG)!;&xo8W9=aK zK%Z%8dMn9(3WJ`yu<7k^0w{T;-nWxW-L|1h(l~VJ77wB!Q+YJhmO3?Xh1J#d;H3gi z#`A#DO5qv2H_A6xF0FJqJqt)#AYXZ8w-%BmjJ@( zs!@Oq1w%P{M1$I#w4vpM_W00I(kPo|alsV6DvfW)2e{zTgA)G4S3*sML|-&5_ZEx< z!3OUlMSwQC<-0#XA>siReBYij5($IdyDhSVd&$Np@sbPLVMtD@2V(Isn&^d4HyEyw~V| za@v-q!bz}3A)pBhi5^roFe8nG4Mr=lgJ&3CH_44nK_M=@b6oy*SboL*EoEXM@812Z z3jGF$CN)&^z4$it%p)Pp9Y%YC4muE9ni>J)JG8bkf^;XlI znY;8}thFb93sHT*708zgOO+TTGbElvEPDnn8|PR_l2WW2WPa9l~GD4)Rs0|i{s z*9U*zvmdJ>h&7$mQjterL=`=|PziCDo`-(Kjt0bhIB&vMQ3mf|2Q_M9T6C>e=bmMw zwtkm7#*Q`?O1q_CUUX63z)Z@Wy?C2ffT- zM&iYHcZ$2!WN!=itt%t2>Vw!2369`%If<3E4eRY07TH8WdsCpJ=qh6gmX3ZL;%>EE zVdY~IqJS^Q8W9L)lwl3_a^hB^~ zG@HEVULxC33 zZFX#-tmPfnjUC)o?fs$ZnQrT5Y>~)(|Ks)F?m0orB{n|FS@jkHQrY(kwcIvEejh3njzK$;RCvqSlx{N)e5lK&*yQ!+FRv{ zQtMcuF+aB4s**b|gT1@JzC{$Cmt=b6_`==@8l~P;7<{uT)+9q0KxH{DhG0^_sFZf1 zG=7FHT*?3ikGG?}SbO_H$aQF(o{0KUW2>V!|GdSEMY#L@0TQkjwHS|xd*>sZp#0Y) zT{YU;C5Q_|7C_)DdKJkd#pfFL5q0UG3H`nmQ_&50gMekxtM%Z9(NR~kX9x+jDF$R_ z;mEr2Va+|}g@SUn5*{Q#KrCdJQ*(~IeP38eo3lKbYdB#Q+7uJN|3qn`pflgrYJpneJt zHO)MS3nKMBySvvd<)vG4|D2G4REHwTN4C8mZk;r+#^)Q~#5|?&`6K|oDxCEo?4?z> zLyJ#s>mLd2&AA{be}twyS$9A5fC2xVCj4*i`#H)tzAiG46E11U z%TWihNU%f<^7GURXVwkwhyAta9gUsN+B7>YI<9}}X#L5_lB-&G@=KQSYwjo(TDPO@ z(&H|x^D>%l#fdxkYTaC31umP+XVomnJ?C(L`X^OF;f^tN_3bj zwd&;z>Oi^C6;d2|Vn{e}yHOB@Kc!Q=&nSk^-Zs)3?^V3#Fi_m0u7Xon_fwr|jKVLAPQ*3jmriwArp`jV%zR$L2k2+cK&~ z!xW?fOzr{@(~8n|Xg+V1-F64lMw%{Tv?oHbiK8lh;NmNMkS}uKAhRC#M)gdJLb(%> zkg#^y;Nz2qsXD@eSQ)K;O;K6ux|c9kXy?MkRH=dr`|=G_7?edk%`uNBl)Emy+q~AT zul+nQeA>qGfbDS39PCgUg0i~_J4x8KNzw~>^P&fXi;u5W>#d%@A$B8M2NmghY3VfE zwuT%ppR13|M!SM@E3zRf$F7*$p~R}Ku%zkv_}<&`Z=8C41N!=kLKPLghH4uOQ#%q1 z#f5xUB8xGz?U2qHC}!pFyJ%N&Ly9@>!-V5tf_!?e@)=0$;w@`~bB3EB$ZK4X^HCX_ zV)`TFl3OHzs{oy>+#p)=XjVn{bOB=LNJ22B!ukFg{iZ>KHg7n zyhz_BH#Ds`SyiA;-jtT=tf z+LyUDzUe7%y|uNpv@p0g)Y>KmJuLg(PGq|UjV5o)C(QTkzOrG7e+?!hMpfdyFBgj@ z(9q?JR%XVW_lcS#O4dDZo)o4qf316hcVY}h`WW-0WV4oOS7QF7?B<3MA9?&y)^F6R zPRseLg~vu;B&ssM3p;M}Y*G})`yJuDC~e{N!Pb}<2c|36`t%R7esq$R|qGVm?wUN-h&+1uwC z{pLfEKZbOA_Gb*lj}69SuP12`7mkay3FlB!31=qu-OYBg%Tv~ap0xDDezOr#Q9^L7 z@6)X8X)PCN7$|W`tg(&>$KN8V7r>t*C+#pQXeE^wlB0a0ctuJ>uSYgvEY4q4ApQcC z;Vy7L5Wx?I*g&au;pd-!Hg+o4i8W*DYANmu2~!AESl*kKGa`m9%GhXotar4-Kc8Y+ z4tLml*&HvRCmks;ApkTtH$!eJ0KukbX$t;X0`B!$U0vOmf%Ht(RX@$%$(l4*wB{;j z_r}_1wba;SE*>~|ute9$uKX`PY&KA!r#V;I^W*(x&Aw*C=4dDg7pQL8wM)UvICJBS)o^9q4dEza~m@JhhyO; zcq*8)Kc9A?&TE$kvdqQ?D;=?KLK7J0QgrKvj17d2u{oKY9-W0CR72G~yUxA^lb-+i z!NJb!aBxg)s^P4SFTD8=-JN6^Op{t^CU4#@`+VNkl+U|P?UTQKyLKgtKL_Z!18w5c zZ&)^9SI92R{TrH5doMdx2qkQ%+mn^$J)|e%UtUMc?FCvLTBadkF-g!h%d=;2i(>_Q6XkUo8s58)lcnPyZJRuG$&w*D9 za?W8?40i?Ri2mu_(3%a~hRce>FK1WpzjmsN%tWBo-i}?hgYNXL3mw?OX++6vT{wqE zPvoQt11)r<-HN6)R=OluS7o==*rh$+FmvY2f!c=Aimm0$a1)s@+)yU%Kxl$5dwjof zEwB?>Rv|XG{)H`DrIf_GX2QxXi#i#T zq6GaWnYh$KV;69wEwEbh#I#Gt-n&uDXzg+ze!Z+TPjU1^Vj=x5>>=;EPCFl-B1C^f zKi(C)$2%V}77RLCMf#RvqiDI2o@cI5eoNZdXS$di!@88Lp+lc9TQp&GW3R<}eiVi@ zUmY4$xU14UZe&&WLdryHL{8EPZI;cL@dQx$C4?TZ=baGdtU#VciwBkWp8U2Mc7M$7kHnU6yL+$E<90|UomDMxtf2Aj|fWS(6s zjYujC(9zJ0045avGo;#9(CHVQ_8NYjiTH6Iro1@88E88i&sq4fId$+-kby);x3y|2 z{~@J^eoHRmgBUnkgG%JzK&sEmweOUJN=WFwwiq_^eLXe@xkwZg=h}`m& z_x#gOUYp+wa8$wSvXd^LnzM@!-oEF}q^%;t8M6afXizCV5OFyOu_2}B5`EWxd6YfR zF?7YuOe(=rwBybUj&-CwtI~HnAciKX8i!+$Clz|+AR;%pt)|(D!&-T1$9DVha^f&a}Y_<~~N;eTHZA*0AVZEl%O=EEG(@wzp4_qA;Ho+6+wXRtZ=3(_BI-i2Kb5M`w`iI+1y{8De9e|+tv4{6>&BlCB-7`hC<3tLJP6_U49Jam2eW48E}uTY8H8B$xmBCploqW4jvY@_}XhwF<9 z-fbb!cA@N%{Zx91#AhZ(>%|HCyfW#RXcvG#U%YjTtxaiGxw`XLOJsAD^ErJFa(HLL zzbqBgY?@q6oc-7xdkrKIAJHY@NYw=|DR#&H$q;UnPhH-!|4PZm*T3z!uwIDrxI_pmtS~@)J3_HbX<^5SA&5vg z;4G34T&4q3ZlIDxQ?Bf6`5yHfYN7S-;+8uB&u9h+xQk^e92Lahp`Y^OI=<&tI;G)$$TD4?aE;%}YoNWP|S_+H^wIxtjHwvPxe845^j=XssC|iOc zQ0s6EU2TQNX>YQXJ84}|>ZOZCb?1jr53)5VVDRZnj5Mc?MbT((Y`0eEpmmCVVFKgSfY*2mCqD4>f z(lDAgpnmPttdvJSz^+f%9$RE{GvR3qUBK?@8Y6Bgma2i=4CViVp>Etm_k(ZLGTS_m zg2a?Lz_u{eSf=&I>Ee$tn^U*vwISzouC@#&W9oEx=LflEUKNxyfA9+{-xQO1`}nD~ zc&FCFsjmjll6ZFYz6KTD;UX$xv^9<^0^&LlCc__2dll{4)R9~@iu6-TBCAJ}(-?Wp zFSZSBICaK00U0d+Mn7N6?$7FMx_)+-;kzMIpaM+R$yz97t&Nl@Ew^v> zLP-_^WPUjlO2?Ppcsi4Phfk$8Phq2K1b{m?$ExoTs3I zrJ@_XOjdZQ=~(F@vO9;}3uOSNSk98~d3@>6?_W`(I&Imf3jU~zlj3r00W$_r>S_Pj z?+dd1CEQ<5?@W(FQoC@2tv*Qh)Tb?UkIawmj^|#vTvZtUTRmf9`CrIHaOPv`N&l^1 z&?4tK(q`7;J$aV?l_7o<6tr>~MXjW>Sf8Q!a<|~E*F$;BVE=@c+2R@h2%9W2GEEaa zAnir8T!UGZ*&5ABlY{a?!(-7-izjv;atg z{}iaWBb9kb>ARG%60do!@TY~k)==+xh{_*|7Y?Q$%!$5*)euIK`WzoYyDr8`F zIglnc5Pv44D@9wFWiVkZN>gueZ`*58q4@4A*dMu(wl>UD=u4o&Y~YcK0tbOe7RVG~ z2!0!1{gU`O78i_cJL*IQENtpvqHF-oVmB}@0Xo3=uyXeM6A}R?3}7zxZ*FDdnf!L% zb=at0X)<&v5Q=EL*f6}0hNNv`{%X~%22^r}9bI&9kzaH3WzjO=C;&~2gaZp4Vq>+1 z2i~fAT?hrDsQO$QyTDinPXt97v}ik^nVMXgp?!Tm{|KUs79&q~FT%Rl|5>?SZc{ry+PJam< znzEUDm)Xg_lcESZpz0oz?@3~;ppQTWlMh{4ODDd&oeOV85{e7S(8@f1#~nUbx1pFy zLff!jJS4z zxvAvw%7zMP=;ogxS*K_ICvbpVT%oQXQq@Y`;J!@(ZOG<@SlNiY!jOF z#WI+6%z7fy2#4%y0MZVS=(j8z`>0agL=Q&&+W0M4R8L_BI!5Su#}b(}gcc}ZO6`ld z?7W@uqFzs+X4;E&KEuQ5x;$P^H`^Aj$>sn~rMdukS9eBzgk}^Me7m)I{43s0)?93R z>w8tEV&gO9VvSrJ6bfU@VAOdDpJRWK(Dk7h^)V=lobkt;NX0K}n0JdcZ7O^`<<+8;3>F;$j(7y>{^E4SMBp)|UV_ zy3^wIm`PDIprP}KQFClyR^LIDCR*5!7zHr{W#)vVVl&>|p~}#_wgMj{&>s1~Q}f!E z;)~2m3M|#mfLSD@l(_R%K`OhSQ_=gwZ1?3@K=Yx2xqvE3k)3#d)~3A}GwG0mI}l88 zGYiW!iBD1!dM1utPKXHv+Xmw}H-4E!%UUQ>x~W=%&Y+awKlgz` zvl8T|L70!0fMCqppSjbpyCwJO6lMu&C_52?)~ed(Acm${wbmnX3N~gS`IHHm7W`Om9l_o@C_ndAhts91-~V)}lQl7vbrE&^v6)4%2@M`T zsU71l2-H%Ak^N=pqsZKD;-w(U(^!F9(qT%?`jOX-BOVj5DqvlujkJ!#9wroOA9-}z zh-R?pORUEsTWiq_Z-A2W?58L!jgLH0CaAFc-fYkBI*&CBJq67aPVr_hcC52ASWSm6 zxisNf0kHBNaYgaz3i$y6lcL=UAQuB;=&^-We4?+WUN1pu$K@QtX7T=d4R`S=n=%g4 zea1EBPE4t4I=st}%bx|%lr!mHIJftr2<|#wTjSIcA?SlsdppOBE$k0)8>MWicNh

f$^xK{mFOR&(g}hJn_M5} zJCw)c@^epawpn(8c(9z&-gqi78ev|1tR4nqxGXVJ)Du^Qgja&886PgP2`~}Hd6bIZ z9Ga{NYGD_a*Fteh0lXJgF7k*3(7ig?)a^iFC?v8M05Vu&VR!9$qS6iN5fyv+CN9qd zvCgJPw@`43)T4^Z=Qd{lI`A8|x9)wZBmWKso&&5%nm08J>>^~!{l(;Ev{2!HT3XM9 zmt;gmMK$Sn_H`rkN|K#TC-T;*F!YGTkX;LJggu^kdM#urg}RBzNdnbi>FzPw(J(58 zaash&nR6I@y1jKcmJi;K@ymN*3!_N&$%5KG-1WX|r#2WidmzLF2x&>;99aElES0+* zEk)a~R)aW_EKX*equgI9qhjfhI?+^`Y5nch>uBkP*FrF$&rM_9 zieqQN3T`4UM5mOh4)xPD?!s2kCobIN+-_crxq9nCs%14QA&QRLS#fYhXTU*jmm)A{{8y3$bwmnF57hxv~OSw5c;vK7yO8 zQGOPRGmk3SZW-++?K;$vwhrr3(IuI6Y)J%g<3snR5KON7Mn7=7WqQTVbKmVb(Pk@= z+HN%JZKP6SuaS6j7U3HQ>J^ z(slmGZS8;cKhiOJDUSSj7qLV zxau}hCSTn|D_B`(U&TrEpV^Z70GrP@qW|m>V2o~TTqCpT-EE6Py$<@^tyW422xJBy zEsgxo5Ew{kPzQm~pd^ARuTs`vu|aY+SmL^hP>@{&a1gB1HLZH@*_{serYvZD7CL%l zL*1eKRGLu+0sx|$vMtzZLq2*98YGGHR#&bEJY;|y}~ytCXqD}-zs!gK>Yr$oU*Wi@*P66sue z=N(|jWI8ju9RSs`*cHZ(#qF5?=_q7K&yngBrBjJU+)F^<2O8HXp2w@DrBLQg*zusY z?B4fLcOR5lr4WQh<18dGDZrw)Z1bl(R|#Ss`>OKC8iVRI;yI0gNUbi0P$0tP)SFTd0H)p-w7muA2{svI-{9@N<+O~02HftUhpXg>G_Nn}+)c4*iJ_tS2% z%*cEM+;Sooon!-#*bay^wiI??Ufp`IovzN>sM>=Xg$1yct(`dF@l3o@->F4ML#b-c#dR4Z!6=?bhzO;9)PawhX>!BWl7f`wa_*5=w;~eLW!J9xXJI(w>D;O zLmOXgVt1W6?S8%0i71!zc%y}CpcOcKfx-CNg@Q0K@3*nGhUSW@2sHFAsO<*}$jooV ziF33fSPp@EmUL)*1d-r(fL52CWdR}GMPC}N$OuU4HSnGgFpCjaFT{KH#t?^@1cj3& zEN&2YkI)q;JB+mh^7xKCsCvjnYyCZJ&VzNFA%D_}{{OQ><)yI*(ETf3{| z@e@k2h;}6`B?)MJ#FafCe2t2s?=TyD8tec7avyYF0y=lhQz70!zo{HWVFD@lqHrSt z6a2$N#8RtPQrrp#|7)94uFJ)H)E8K%aD^d>DF7%MFUN>heetOs`xJS_ZVRx4|eMnL#4 zv;?@oOGvfkQQoLfN!^SRVHe(G&>d$yW;Ta1tUniv#KE+gjS(!i!1&Pjix$;qT`JyC zD~eqJOOU@-{e_Z%0BK?@K<6~e6KK=}ODCA#+Kl{8cM~Zzp67xQKytBu!v@GM0967M z$HI@o;@V7M2Du8FP(`2~$||}bB|?c>cZkgxo}VQg|Edq&pIIBmo7nYmw+BYxMJgg4azM^utYQH!39kGRm4RRF@6)H zZ-6M^%}g10WG`-@Ec$a)1%n+BErHHhJ~Jt)1xe~c3k}T>Hbe3wnB`AZc>hD4L!lU( zgLHm~5}Q8@3u&oh5p17?t7;}r7Z^%)BJG6d3eb#m6Yl4XA`W`aK%LJjvylS}CTp^3 zMI(kYpMd*@k#{bHb`S;M(^e^w(VnPEvPM7(5$&H#230c_`fumJF?b)yhJ})QxBaPV zRrw4+3_`GmV3PwDa|O076!|kehJxrI!fqXQ2w-#2ZDRE}?0YdqEUyRi|5JQIq6>5x z2Li1P#Y($_=ux0S8`xb`kdj)DHoZmWB}T>R_Bb*^@b;TfY&#)ntzpAMJ9O&Oqsw7) zRsgrwE-U#kxFQY) z;UfSm{F!sAoGwAu5&~Jc0x_H7c#?oyavFVTNWJ-mH>6bYkVFesa*d9>^DS!OTsu{b-K&i1MDw_#6A(3Fol{@ zBFs4=jQDi^gn5TMV+WXxXzoxy1Pq zX$k*?_bF1f`;0lGOEIy}C@Zo1_(*x3zHL>DA zqfrX4Zzes0sm(Uvq>12fnjJ%jj4@{XAdPT7Pp3I@h&SEBp^Q`IP?#mD8etHt^UY^Q z!pl(x;iQmamx*(=K+eZpiKp>qIf>h+;OMZ)yirF?0_6!5WU&d-dBEh9<%AA5u1P`} zv4#z!qVnOsw47q91%}_bQ)?=?$f$8gr}7eZ{&cb3qT3ScWm(gRCI$eQOQQ zyFR}LJOGuqE$6L5^8Z3}8k<@Z-8hYcd*qbyG5dVRHnC*j9S1hk-#>2Uy~-^a9~qf7 zeaWA7cy@#Hj!|gV@q?(F^89Fik}Z#`8X4I#3@P74^8;Q^-rs*2t)NTnU7fN2vX7lYU#zOqDFwR(ay0DK;JEB94kR7q`x(aIeho0 z!zV@2O3Nm=1&2o{jvT;<8hx!ex{7MDxZAIMj2-d?@ah#u-l`P}&ZDcJHIATD)=F|- zuqbb9(>l#-RdA84fqr!SaOnw7qlDLhYVACRPuV$vPbmiOG;^kJ@LDvkWr9O+1zZ_Q zm96D5ifK^ETLCt;Gx;pfF zOyAfMFfP#*88=dfE-9X!+J1Bgw~$OC0}0WPzge=^TavJ!L{8LLMch8U0C^8)i0uq)KSP= zY@d0Cm}`gxT;MhvOi(*zLsUm2HOC5Zvr7hpH9AVIdLC=1R$&~jMx4?M!V5`9uonn> z$=1bo0W4t8wxH2E8_|(af>PunsrP6CV~ltmiY?$k+bYYX0B@6iTQnmbL)= zJ`Xv&KkqsRyj)bdpU3@~(k!+D%JH-rDq^iaeY0yDrVnMvb)Z>dbt&fZbu^$LH9BYy zV4;b-fNzu?C8j3378V*dyQ0j3AncSv|O}H zHOQYw!@BHCvxG_?80b|8)xtnazxfWfFlz+FRZM*lse;V<;Cv%JU51!c0_cx))JQ?t zds_F+=qC`|vcP6RYJ06QPoKyYHcmjiSE%w-f8|=vz$HTr^Ak2eUXX^JabjKNN)51btkd z=e)n~OhEI8FKT~nskau)_-WSVhO2uGxN!a-sIousY?628wF+&*t*XUBh&f}MlX~yt zp6!{DIy~{`_-}LKyu|;DdyqxF?FiwK02_NSgkX#Y84=m2G*dVyL;pUZtPaNj>P-|Q zcF1A^IG}`C%@)7?;zuY6g#`2{?^nM_0|Z4$T2miX2)OT;*{Gj}+^7@FTlNqHg8N8; zSijJZPO&|X7NIcbg?Kj@4g1qasHk&%{<{ZB(PJu^QX+j93>9;;bxRLXKvim!Twd-& zqnubdHiuS#C>xq-D1$b#6QLPM;f=v8(7#!h8(UE}hs2i?tg?Udm)8w7fJc(*n?*ux zNvw4O6(k%zzt&4|Y7k;~ zg-dt{QB~8XunpJBw>{}rL-PARgwzjlieok0F1e=lL5M>Ml)XR=Z)!e--B_x+^0Qe4 zS|GsPX;L_d78wVONh^w^G^=cLM*65npdQ$15tJ7Bo6KTeAcc?bvY_t?8-bYx`XdE1 z7>V4S*V8oZb`u(e>?`=}E?J?Awux*wKLi72ZbWH%G%mBNfOPN%b}8ir;ygc4Am4r> z+eYHUJ>=`V_5=6@C3-B*2rv{QbqC4PuiGj|-$y@peH{Pj-U!>5{^MM}-iQfn0`s9T z@!R-&-p_wiVfg=CllZ@Q1Qd0o4Y@EFVDkOb^(pPsxYz6LTr%(a__cdqXd3Sme!TRy z^(H>0m_vWM{l^>JTWO`GAHg4(C{YuiIJIMDHH}-oBS*8%33_0DA`QNraB9nmDewQr zUm)GVx2Ze+>CVDTzFSTgdP{DGA}zy8C->3?d{i85!M=QUSK9k8*E`}DWr05@QZVQK z>=%n$KIQS5fRVQa57vnC@|!p6#(yXjA`S2fd^n{fkm)e>mTew?X84j~hbclwB@_-Y zN)vqQ@g5p7ydU+X@$-1=yskx;;{LqxzrZK{i>JlCMV%*Q5$)4BH=-={xF@`u!@Wh^ zgv7Gjlf3=zc_sazGoiys_uti9j(ypu`^zUC#eOl%*&3`bg}m=reR>^XOTVqUw(IX3 zRCy^JedmA55V=>PyICtVQDobN8$l*`W_OR_DR6&*S^Jq#?QazwfBOrPw(-vOc$QSJ zP{xnf4_H^v14{MTv*W%Le6TDiG2Xo~T7~a%-2dXNWM&M`7`A19tCsT=_Or4?5&Wn3 zth|Xc=KuAuu6gud>NK=?+%;>mrtdlAXpnFxqN)uCz(y)LXLO{IBIx`kmv`HjBtg*; zp5J>AFlLKl%G(>j$EQesCa_|%uWtL6-3=p zosC|5IqKh~mQ!db;>teYdzhw>^Fxalk8qpQj{k?cBEW=DUyPE#vwP71FZ05CGA&#; zlS*VsDx8Q7BPnG{t9P>@5!ZOV1?WAp`Z;L!=mimspK9Tw>3fkzpC8|IQ0FAl}OGW-PHBRT>_Q$AefL2EWpSWu!uy2s#suiv^7VWVc9E)T~=%Q&MW4OjaO=21mX+Ec2^DK(=Rb9|QU z{`7Z7vvq!byqC>d>lzQE9v>6On%N_xABF`y7&q`~To z5%R<2_uJYQk<1Pdk0ri|%qa+AfI~+Mb~@YDwd-{&if7X+ujebD!NsP|Fvj*eVnEa% z!6<-$Ru&t|DhgSb07jR4FxDdU7Q81#oxnz%p#)(QtdFyWKz~5H(p7_Qbi@rcQ-O<; zS@+F3bWi9|p+k%>&~qzEh@Hrn;CY5iDRoppv)rs|HmFIoyRMryeL=GHnA<$2KsOAY z6s1hA3eYYVLaPxpq=2s21J>4tt=WJH*I_Ph4~%9~HabvO`!n9~ZyOK(HWm1vU*IWz zo*n(qBl(|4LUW6L%KrBbhX1>SP$mV1c*mrXh|&sdx)G*N0vrR-x+fA4U3owzL0`rG94=Hr9~ab`W=`pUli=^OGVfM9t!^$qz1&BBS4raryE zrHg{%-c4l$Oa$hzIash=qHIWH2SF+;9~f>n)WoF~Keu+>jpBClenG^fe4U=X!HSBBamQ^1Y z?D9UugI{?n=hLE_Lhe9d{O8mTPZU$_SbDZ8BiitbkQ!oA22rEuXflHsX8V@=o7R#B z54FG{$VNs;)G1n-Fm=Yaw;qf%bW{O^%yn48!_p@njXeV7NlnC|K1!q33ln>q>+X@Rc2cP<~iY#M^Bt)r002`#$hbRAqPCx3N{C3)J@b_K4%D8kqVNFvN_lMWUbRr z8%1dwRx}`m!E|n)V?z)#8hln!gC`iS>^Tgk`B)=ampe-=xwC{49PrrW`a(hwHt9(r z?U5p(5#y0iOa*L}$Hfa5z`9a%nk=6afY}>(1ZbzyC668=Q`nL&@9K<^*;1W*mPN8e zk|U?Sy2exiwlO?Lwn-bx0dKI4o2G1<=i0MynI31BbWhY}BKUS6#OwH*bO>0$JBqM3NnY4>Ei6|H(s8u+`y(>TZ(lsf9 zf^!DTz0r47*lJY%|^os{zvjJosy(z zT3e_|lQbfvFDLquULhHm_=?{VL=NszIaK&zNrzaT-(aTc1yZq+O98Hx`ixynMDi?+ z&hSs-3llp!HtN?`(GXe5|Bbxgi5jERug&rgHytT!>3jNSp3X_rCq8SJD;IbFXh84L zUYLTxlSMMS5H-;yYYH*MMWSyB#i88&5;!JTtgSK5A$?neL|Wj3=YZyRa!_mxJ3gy` z-Xn2OSQYc(C=h@_m(@NexRshFA(0Cp@f6kG-Q;vcx~O&7=7{lCpnHQ4XzGETiMC5< znWPd9G*wFSqM$q?`pHX&)UoLNaDY)C(+mnW#F5w!5RTO6p&Fy+ClYZ@A%2W@w{nEZ zA|&sZ=KsY1gI9Qt2Kb((@_rPq-`Jy^*E3b|7^|-zTC-9j@Xzf^Z`_hyA zZTgd>4k9WjbIcJdGC2wmilxmF{y>%3%m6+)RWD;IZcHe-ow^0y!uKalEso90cp zB;-O?5$zDycf&OCX5Z^!=oQr=;vUmCA0l@jmJoFom_w~%1{Gq3n-{S70h z$OGlgVM0f)nGPD5u2kCFEjtd(!9+bv3UuCGI5!HgLR{NZp!6X^paoA z*rw5)xtrFf`xgu7g=lZe1Hm1(P17BUCu8#lFxx(?T%<3|gIJm<*I(!8%TgUlhmP47 zp?x0Phg2*_(QM`mXztn@%(5p~x0!cBewne42R5L~WhMIqURV5)V&sORq?7lCD_|^;;JzI zg3`AJ6}Dt0+7|PpEmn`5sB+1xjf|mmhC1&d=K9#Xzh2Dz#u)AfCLIos>@$J1jU zPv>6n3jO$C4bDYwQPgdqjJ7RbG^5O2Y?rL@q^vA?bLW|Z*#pfh*$LDy{E-D(sW%%A0DreN78 z9dtw2QFNNTUsJeV*#}4C3w~X76(UbXgwIYAvNlw4zh@hdD8J;xPRnDgyw?4v_(`d- zfbE0p^Z3;R+GnFPnf+_rWv{yBJ3w2Ri+oxi(*d%~SJ0;04Y*tiJBI6MGmim zQo{%B5J?-cUyzZ4Z$uWXDZ=Pg@>8h3x&EkTo?^W-@uG(U>eCcO{#8kzXaHCE{X9jA)My| ztEqc>6gnEyxb9Kd>Irh@Lb&@vv%fH5q~-SdB`;kGTr)>FlptNUIS`$~@|$NYoE-iU zwDYr<5j?upCME(e!+vb!RCrW5zwhSL*H-mxY}9WkwaZ$Wbv1FJ|J55#$qS15b_A6_ zc~Y;Ps^@Uiuj0`Ma{W4UePi=Ei=d4Sa6Zjl$wn8IOLF-Z+jYGGO&K0vIJ|tlLvxr6 zdTbjYS=3+&DSW&)dp?|Xj+SNWF$d1IPwDCCA8o}Bhf-lhq$#mqvddMqXp z+cZf5Y0$h#COa+BfktK~IkK9PMm`s>QEzVUnB^_u;GMy*yITG!dU~D38vIg z8#z^>Z7<_CHdaCM)DE!^*-$8wa57E(}kLJs{@YMDy zeO(9aRXJQ!h-5^$p7%@_8DwWP6(`onlw6RCsRG9C1F-VWIWlSL(;4Z!2Xey{+UkPC zuJyCUZrUA~dH%#{8_dRIrLRS8uiC>J(^o#kOCZlhPuEZNnMO!u&Pa#7FSK)Hkz-e_ zTYMv!Un*<24zu3HsGO$wu%!2C!+yKofNTAoopCouA3iPH;-`Ei?}KTNkE}CumRdJu z)u$rX$7W7q#PVQQhRtggpuTDOCiLUK8RUrff!o6@8SPR40ioRVz zAy z^{_lmqL`uw3nQW4UFS}?^zRF>Np6}PRF$fB8`bY^XmUy?yoG}*T5(Jki(tqLY)Cwyul}YvR z8;ias7asxVTGGW;!@3EAj)oKFa=Ut7KUrLGh-0L2hyDO#{=&Bq$s%0=8uTzBc8p};0+@)Lm zHoYTVFb8SWJ}@t&#GqzgD>R1|pv$;IH#}_oH4 z)=%)6a6jChj}|%qRIBwoz9h)q(W(FP1%DTCN%d zjmSgIG_`Zzp)V9fjX)pJ0(p7}*x2+Q!R6_1uKMam{S1|*JX_Lp_#-{4vQc{E7%01- zhynHyifX|!*C*CNKGArpB4F zXmdIi)t`<dnN6NeW2a$ZN5|JBnZ_Vw}7o9)w^$wC8n*Mb$eGIGES>29E_ zmMdIV#d{T|E57W*1TBB=^)GZvi*#k8h|f5Cy{WkB>imPpAk2w2F+Zs>-9UU;`}ZHG zOb<$LCfzUGx8{WY$d5h-&R%$Qnb-~?*8(>*)JQF$jOkj-rUrv1a>>mjqeZyW7QpE- zc*r4@MjiEi=cb~BS4GdS9+kd9W{2jJx2VJ2NmKVCi!1!)>s91LBI z(nw~dDu0?^Pc5-k=8KrEpJfXtMzQxP3y3PDC;#x4=$ME2Wu3N3}B{ZRrO(!pG%@ilpO#t z%dJz=q^VeG{zxojE)c8{du$NK299PF4JPFIVy$rF*olvVGvS zs*_TIS8@~iJ|v{3EBjGiE`YZ20Mi?aV=yrdep|9B)p7_!#fKQms9yLyU8M>p0bOkK z)etPERjw%@n3w6tJJeV- z7hV~j<_)?YlsIL2 z##F-y!Ypf3EHfn82MTH#Hvq4)Gt^2#9yS$;9cc0^qRCAjK%`YI>;s)H0%l#0dT+>^ zmC%EjiiQxZQV0=bat9w2$EduIa0b6fMge3GYGKilRaclG8$FV_6lbAkLpm0eyj|h- zQ9MjTg~REknB;d(518!3pp9{Mq3smpFc}Y{IqoeCHup$ z_>pW^kt zte!6ZAXoX{S<)K69riB~w!XUg*OGtLRTXVGuKcf+KTRs$Ev#tg`hT@|rBO|uX?yB7 zGkUPLc1E!Xib_B$Rs;lDV~Yz06c-d(Lp2bDu!tx#Aui1I;C$Mi)6+A5zJDLb<4NAUyu9y|=eh6ux$gVEuCUpc!4cx% z;DM$mQ-ZCpW_BeFRK9vTQO2#1&BSr`?)3mp#z!#jDa%OV?&Ccza*2UnPaO(7U682W zM7y|)(nfXh@W{vP37&%5!(BWA18688d?RgW`q1uFpEQ=Ays(QRn6$2%e!h^HXm@nm z`GZ5-cC51!YF^%a_f{>#6Fs}yhYwU`7oHrW+uicdk5bSU*4Ea#;|nLBlb*%+DS_i` ze%y5p-bm7h?)pMw@BI0wlHs4|HOnn1;PZG|0IBOnP(RP_0@trUx;OaZ%+TafxFtR+ zE+G4u0I|R}ASt8pRy@S38I@jZVoQ(x>(HS?u?&^!H+7GPU-<=}nDNj(;jUb@biv&- z!-|qs)zinLR95zU8Dml=c-$)Ym1$^bL)EZi_$9_u8~$0AQqTR8H3dsc)~5Mv@O3|L z^6ksbNBRtTk!Q1%*H0+u3sqhyy&7$IflY{ksl07wZl0f)R}Zh?jC@Sm3ZsA<5nCO^ z>!lm2-sxA;-fqKAm^%34N`iuo@Saa}c7)E%X{GLG`lbSIckEvMdB4bZ+TYit+IOFu zuP;o0#Pzxxt~4`ays_s?;Cjg1x*Kn9PyY+udJV?stPnQ*s&nFdorRR668A_AOi4-CljtoPLQe&Ns z;32ZW`eQHk17s=21{cTau?(YWLG*lmQpFMaaw6F1!;_sE8^xcqZdrx8z0=lcvvy!M zyN;#pRx1JZNrb=?3X1x(+bD^<*C*}{z4Z^jdAvwdm;O>uh#c!3zkG*Lb5A_mV{%fZ zI^bo7ykzPFl^Hmy7}rC~om= zgstD~KVp3$K0X(M^=m+zN<8EstqvY^BIiy@1khqdO)pEMXXe32(P z@-RD9n;DfAGZUxj^P3Z|ah~~!d<0U>3ab1X-Ce6SXgqY=wP1&s$K#0vU{{S4=iwG8 zt>9oWaOS;+#q&hdXhn)~u# zHU}lovR-+P6DI1Y1j0?X(I2#);{^H5m^Y{B+7?kM7WBm?Yb*Pbmm7vutokQ4Y36~R zj=w?Rw+m9FNnUOvQpxv=WykuqbT_u#;&JFj)zHyN%>Ymi<#JD~O!lJk5Pn zJsmK}7`n7_Vq8Sy&Ri2(C~GXZ62lDroxkyZoHEsJp%g8dlFQ}p4St7AmG1Fhz9B)5 zJHG1_t=1i_rQYc@9%(c%iXVP{BXEMramuyImo;}wlEjGeJ=g?!sMBM@-Kd3>{JpF) zB_GihAAy4nC|r7;Z^CNiI3ZCJrk(m4O-9aaceEMBrv1zkGJUla$MA$)g^+q-u=0a9 zO!u8@izJ=6^`ytu?iH^d*Lw+>4qY^^P2dp33dQ=jBXy|Bi~A086_N{uVSf)9IW^N* z-KcNe*PF?qPw1q_v<8f&Cw4_X7;w_2!+z3x+6PUWMK=-luH68Rp-)C^C!zd+_f;HuO zL4m~&+f5*7Q|oHqIo?+8mrEH$z1c%OCaU#v3JRK0_*D#+pvZnsO~yOauCjnTkj{vn znJ{EYcCq1g*;CV?Tq^b(w!%=GV;eB|$VGC|TP|hyGPAXny~!rZPr^l}HIBlse6<|b zrARM&$%Wt}Wz{f+EML>LmCn6|F{fQR`_3(Cp7ve+AiGn`;z(50X{x^f;NFn2QEzWP zr*nMIm6$=3!So)D?j;6^8{;g3>b8VauPw}s38w@X{ln;S)?VcfAgg9-9D{(Dm(dO? zM-w5l)d;iLfsiu@sfMDJ?JX_W0eW6~NLY9L2K=yI@-jv^NfaXDSdQrN$;w+u$xI>) zb4B>!8dC-Mo}B@!>ozmqOTxBxA5~RV+ahKZ!4N)+sO7W`U0!Z5?2SrPD@#ib zvbmel*UPW|TuJXTln4hwb(0v1DiXT%4f^iRp_0Y;Eyhv|`lbw!#+ZSIG7cCtz0U&a#D;`SrdaHV?p(DRsOXO3`H6RK#T0c%!Wb*{r_Va%9p!xu?JSGp@TD=Uu|n=2Setn3!>NJ_P^}xpg*b~0hFN&W}RY5d|X`1k4x2go&GPA zE6BR@si`xE31aP3Ca?ap7=Pzr^CVX6g)HK+QdH`#yK%~sOG7#g?dIud<__gMBkBDW z6}}>tfGO3Cuc(mDtzlm+y!_R{!o`K($^{M%b5aspk<848L&DvTg85)Gjn5Kq)Mkb8 zhChYi5C8e|qwrqEh0?;++BH?18wNX%#Yqnh1xJSiSrlM_GZj?7&AL@cXR*@A#Rzvy$fm#5W`Z@qh6 zo@8*K`y_`OvUwsoKK}jS;c~?1M4PO|^)ke#(jQgtrpv6)1<4)%?%C~di11!OERcgj zFjs7NmKm!(tXeArwoO9f-*02C&vNO;=oYyc0*-TM*Rf7yL#t`CUGs@ra&nAYJ zk}D9<;i8&RHcv}{2yLKexB~G zpTmFrQ4SPZ11Jlz3=Av-Urg9z1`?i=1bOsLb?0=T2bu^SFH+Fj-p==ULO{BphB!E= zT?d2`=#8BLWVZr@__HF!ZlY9d`p%(5y9tWoi@w}+`A+{hTa4(COZM48h`^+1Sl1iSN45at0)%e4UY*^3{vQT}MVo0}tAd@-&Go>3cSx_*xc~4f3)wlPm za%k%7*Ym&$+YX*24Ver`=}&|fcv6D*2uvY6+UAoMD$6EuEKzdEd>bXYcv0T2^!OE8 zZIEFlcOvPqxp_pC;+BV$R`lIGoeE&FSnc`qhlhp?d|ym?Y6SGvE973kzKO;MoLAf* zt-RF}eK|U`DbyWp7gUK6z$U;xB7iL-OW+#kh(AoBRI{EeqEfh=57y&`@T?(>M}kf& zwa-#mAN%VQ6D47U=YD+i$%G$e0O4p7S{a(#Td^RH z!OeQiqRnKMj?L(g2wQW&Yj(id*|`nc)<8%WmGJp)HEdM{LmMu9{9EL48lT^b(1C<6 zkpxF{V##2n6Snc|*a?x8|B01VXDi@#fG+p~#?8HUZ71f*a@+`@C=a3`+5~cxTvC+S zblpm=Yn<>Ww9oN$w-J}9 zFbr(;Ql#fKLN;8Yno+eL!iL%acWl3eLY8m!PV-r1jUAsxKlm7 zWt^4ch+&?X)|d)?dB^dYqs&?_#c1?tr%9c0q?QezyAm#6pm45Hrcn)is+)8{=0Q01by;&a}51C-XvH`zLF=L?5!UvPG$9enrJovZe6z z^YaI66Y*cojoc-lzjrPF_2dXjNFdz8Ip%dQlem05~1v$Hd!LP$o)$j)9V zqs;rbdVfFj`v=_j54T6OUM|<`c|FhbJdWc$&gX-xS1#_NV4xt8NV_gyl2Rg($ZwKJ zTiD6T@Gt$n|7^x@XC0)~9h7a19h`628Ij~~INY?faj-PgKkQ^=XK!X>eVU7hn@fn} zu&IN?P5ZN_PFeluE4XazOir<0_@aTU?6`SJ!=6N9%pyKF=`uWGB5fj(E=x(MxICNc zaduH|+t~iYxh0{*;rwl`f~h-KBhDSDd-#%fpU>taaf65Syr?QfX{Bhl8TwN`Vmp6f zr@FtTzxNa!Mc8Jm@wlV4f7WR2E_bCDn6=D#iYmmg7LCTZS-aYW+W#twJ-SDdoQC#) zemr{r@n8?>$p8F^n+V>-B=x^OsH#a+|MO)>j%+s|ZU3LwqkVLj$NR3||NQt^NnauH zzb^OvI%WUi|8+e(ZnB&I`v!&Rw+Q~vtNHmc1^)lH_y6ANy`s1zC+8!_dRLD1yn{nt ze6;eBFG=#DU56MMC&xyJ=eKXq!GoM0dY9$oI2=V46&004rKD7BZ_3GI52$!ro=M^g zkPe``Y^9!O`sjSu)kCQhod7Wh_r>r1nI2Hm^&?-QxjpUB<6y=A**+03UW zr+2lVSxQuS8^|L`{-LVs#>l59^k>ZXb`@9!JbXy=rl_d0r{`d0W#!n{Eavi!bvHvp z!!T3kLhBxX+M^c^b8t{*W@f(9$Ym{d7{6olc8yL*Na$GVNinfQLp326?~?86pP8W& z7Z(@#-GBG@@1YRe$#%}%$@Td!3N02*8CnA5EF9Le?HO9HuBFi#@*W72IYz}SrX9B! z;jnmCjQPF|%@dwpGj6p5<_iE%XEkJ8pTRQl`C6mh2$<=xR3kv zBVBKkXAONYH8rKbpk3^6nfaT z9ji)p)GD;rnc8p~Z#fokv`|lPzcizhwsPV6^%v@e*3YXJhlhtBJUV*uX-1mFU9!R7 zb3=5KEv9^fZEAY^c%OVS)X6t*+b!%o!>6T?rCXNp^R&t5yf-}q?|o&5`#hF(0*31) zH_D8rtvX)c7@X-TVV!K9{ng&v^5siF+tBE!iPQ6z(pWCt(i5)pV>hxz?{25&vh8U< z>;C(7-t(rWt1p{PW%AY^ACpa3SXr9YyOP1j$-+`OIBAmcc=2tDa`MxR#t08qVd3t< zY$?3+NIg%DS8e8@3t=_3SIAlH?Cd@yTn_co<`U3KE-Wl;ThXRV6pCd0={sBRrCK2z*Gt2<~QVK!dyrOa)~u8)rPQT$E02A=$)#%sYTK|@iR0S4wnf=T>A z1`~q%t>-|eg(9|@PaXH*|Cu<%t+Kx=)hIbP6nRgiS<<<)bEN}UM@9VgqReW?yMrCvMf zbMIbdef`d&qN3lkZ#Q1ME|@HYAK<@{@-dODHSAMaTw8HAlUR5P`AkbW#J z)#Nsc1w?Ra4I%{93oH*%F$sF_+|TvGrmy_>Xsj2_?%n!1na;hpw)c6iaFQzCzu&Tl zRqmXj;XgUKx!J}I43>T62L-IZRrFWh&Bpp6chP5a5@t}C%(RypYx z;%RZoLV8ixypOfD*#;-;^`rx7T6H+T4-Ou2U7Fz}`QgE9YR@J{>OPKZEODGP=1HY@ zRUD{B_}KpZemS>WB}GO5=l5DAQEHJN-nBWC?HQ-vP#k4rOX*ldt~(xn;H<-|+Tvg) zp(8rSzE4kE%}*Rz+pt<$oNmxv;4%5klwyfG5f~vaFTW4jZ*!AwrQ=u=OXxzUzh|qJ zx_VS+4zkAHD_5>4{ZUT7a)2{=qd4r?X6H{)JcFSMVzkIg`>=cWO447*jr_hA79D+( z;f(n|ByyCuZ1XnuP@^*pL2vAaWm3AWcC8H!eNhm%T5!F5`Lfd6`K6`(fByU#?vNS! z94mgRI`F_zmCE7z$cA>KlgET2u2y}Z1p3=YUDK*b%WJT{xEy3Hk%&MdTz;^ly^M3*K@_ntSsN+Yge9p zA_9m#OeT?ZSVTlbN>w%D+N8(t0pAu+xhK4d&$Kjn_YD|^@+6L*UOxVa-};+r&NtlG z(Qvo1m%eqaS1dy8QRW`im9GA*H|k2_oHJA&uE7$-`TH7}a|WjfCAEhSM-@*Id%HM6 zFsXn#IOU``+0C%n)MCBh3KCW7rMx1_2OnqFS7yb>J(p&B>Q&VA^^X86;8$FHd{D## zfB(mIW9jPIo2kWDx$oS$(?2+P=k;rDsf!mG^BSHp3Rw4N6!uCY2>S;HHUW!BUAQn9 zZBs5`ZOuipV%G#1`tjoj@AOg9-o1OTBYDyDnbJr~O7>4oP%twy=b5)rWW8NKyL-=` z`qH4d7?;5vM97j}yUgi_4)a@e`Cy-J@~+*GxTVM*4mj{=zC(SY+J5x&Ku5L#$@@YO z!%MG?^$U}B_V({TJlK<1Q{0-moRgD7WMtC0Aciw?Gg+wrVQk6?HN||nNp)Q6ug}R? zEPocB?MHPXoBqAF)VHZayusq~SaNr%{b=wOE&Z}*&zM$$r$~8h(#&pCJT?5}T@F!O z$S6rKtYWi0O#|26T4_^s)9y}w&616!_4M>~nQy)*9}yH77}mST*4Flo`yacMLW>4l zDpL9K#7UBOWaR$8Tfrvlh_J^ppRVU0qde2*&Aq+7H|_1utEU@P}pJY>PJJ~ z6X7z3CMJv=dnB*V_LjLU^*A>pX}6{*Z|CCT!ecmB=DNUb_x+OPw>Qm$z(qo1F<{H#5m*xJVXB)wy48>%A8$)V@M&iAvjP62_)MDJxNkwBzWvDc=jpKa|S&E8C%uz!2tcN^* z-x&8A7#;OpobDdND=To7hJv>u<@TE^UY_^;(IR{5B`Pi=(N`(OCpMP(FP78u(s?Sm zsVOOuii!$v|NQmqhbbs1INH6wf22+6IeH=Rb+L}rcR?oQNT!D!?%nHCd3U>P?3WQkW8=ZEdR|pNJN7gs$uj~F zUm9vHu<9NpAi-~rf@);0A5*!Fae$3V9>}y~UHK<_-DX9fopnsjEAa=SI>OCO!~291 zYusY;zRzoebFJ^~`m#m*WKv?H1XrZ&FiL@O-nX|K>nFo+jXrnw=R7s}hw5Mtzr$?) z{JewvVyB5|U%4pKlALF=iS$@Ukwj}NbHR~r5AC;KwkQy79)$xkA4ZoEM3vi?k$I( zJ%3*Ht;jyZy-n`RTg^-4EHSr64s1HNb?a8YJ9jq6Pf{}QT_Vv0F$&O0l9%XmCa9)K zU%mRU^Taafifp}}`*ZRS>| z|Nj0iGLOo~;5?Ux32eWpEL*y_z(Xd#0x7!v`t09-GT8 zTchKcIrZ94x?4KtH*IHjzy46QtAmrBUFKW4N4Md@<=FcDr*!usAWo_Jav8k8$n5ct zsB?eCruYTm|GMis25b(f#|})hi015JP$T1|%g&SCg;{TqvT8lPh?mcF<;#}-c(?Py zWb=l0x$V$>(=SOuwN}f&fBQ{$7j|=QskygvKgW8=rs?o76e=D~-!oZ>DNCz(F4BO6 z4Vk0E!^hgQ^laM1x(0TOG#TJ|*KNR)Uz`UZsz+B6u|2G1NUA{;zQG;)oNC-+PGegAHl!W?m?qvoO^DvX#{7cfV3n7Ps0Hc6_x13bVl+xB_x)4Zz!_n}$(A?bISWk%nN*_yT zZBw!Wv&henM_*|ba?SLX1^oW)!l6^V7ktHKrf3vZTmQ>RCzNy|ZGS6v+==U{WF$QO7oBI19y!wg{rgrRa{x*S z@*6k9H7-w3P*O^vem#Et_~x$>`GlmTIHV!gdW9H~14wwn`uh5>O&SlmZ>%l(1qFp? zesBdjY|hjX&h2&x>n^l0=qYw2k@oN3?=aofmc1*dB}UXOVdc@IM;{v-Ymfu7QAwW( z*^}M7ckh)>34gKU=wX12WoW#<&EzML_1?&%6w0ce`J4^Arn`uz+#>Pwu%)uT`v7L&^J0M^uY5LknkCnx$ z=J<=7NM|hCKb&q9xl6~XdHDG8?MQ4@PtRCzTXt}RcdCC&;sky>m!TT(xwbskmbRZD znH3f1O!*iHq9^P$Re?JBFf{Zo*5U{kS7yhCv~yd>P>47c$+Wk$s5xyE-*MxI3o;Eo zkN(zH^D|JRaQ8m1UY%%7y|xo^=Z80&dM)i?c=$c6KNB{`#l{%Xh8YEoxAr5$!!JP+ z1zmm|CSfu6z4xWQ7|Es6KRFrj>66?l@<&JMt<#@lL@jOxK82Vi_UHQ^Fs)C~0@Qe^ zK<8%wc@DjDCeqcno*n@!(o*}j)U>t_MRoB35mN=HSV4X`PjHO}9>)cGWi1PAOZB&& zqCn-`H{NZ{v*|lzp3mU6I8_O)=Fgu$k9{kG^3-t~fKW0~N#Z5<>bM{) zYxUGnT;s`b8He$fs_RcUKRfSrTv@cDJS0MotBv)&6+h%LuZSAjKhc&(R+9{^VBpVa zY~-P($*q}?UhN~J=z#sg9W;|IHj?QYr9Nk*Sz*var!u%bbMaW zX^Nfz!{jX7W-b3XOtkI*U=nl~qZ#@9JOF^3P&;DXf9?=l{c%3tF_ zSCt4A27XKOCvvQ1Swyb?p)oN$d@ni1l>&U|^p)pl@4b2RM&CoBk5G)P#!MR09y0Cb zmy{x?E{`7rqlnAvjyJaPwSN5eGOqbFFP_h;#5dE2DW8ol zZNAhRN9&zfrg`G@X=VQx^K)~w5QZ;I3OUa_<7-U!_4R#o>sL~eCDk{ zBU`ppaj1F%8g2G|SU)^i7k*OG?`M&wX?+BzKC~#}lfaq#z9cK`hY}Gn0hqfVm5RSO zb6xvQyEg3$ag!|TSO;VR6P{ExJ&^l^pU7bO>g$750Z^9Dleqo-tq@DdjD>H?IW_!$ zsHLb3Le|o+^4~{hA9W>~zrpbNzG-uoH8BV5qj)u{u|8dZp_fW2mR;7$y)Hw$%u1Nu zf84IMgOL2LzFv0XbW-mIIDL!TtMtOhy2BdnY4<+$l$Moc6c>Msdc61~A>p5t(%4wN z?8ZlN^L_iV;R@8fG=lA+eCMcG0}ddc#c;0_J0g2enMg|)Ya)Glw1i(hbm)*ia#U3v z@QqfU#n&rcpBZgRZ{%^Um6M-?2J!y8(@qMCgf#Bu#c3_eKVTbNr0MWy2!T^{+pa1Jcoj=!x9+x%JEVQ;->?i?B)?YJ)=!HjHYdDCyA>uD9M-tjoj+5MF#fP*3kWz+GpExzJ!i;T+Myj!-D6qBzUw1vp7KRh@zlxPqK zIP_Fh8j2;5aFho%v$&^rox3v9Q)s8i^1|ft{1mk;UDneTNU_;Y)0(*}*XG@-oclbE zFRXzZ*p7TU5bx*(RHJkd%I&FwQk1>TO~&5M;a?iPQ!HZNzI{tNeAanJJ3aWPVBbr( z4MbF(VJG+o+ua~J);+&Q8cwRRVuKTf_eJl|&lQS?X0nYaqnNOSDZfvb>9)%g`@C;oA| z_1rRNMAhMDvKP~H5WU*ze^^;%j{5>s2T^TZSQ$itw$7eL;d-s2*DUdz>$$n1VZywS z*;IbZ+QZ?gDGo^d9M)f=P-Mrw1W*a%jpjqL5Zr?(BCM13jg65>6t+%I{AWM5HjB^c zv<%rPMr;;LPZSsR%idce8OGslP#4D5GCB>A(Ga2Y`uc5^B?vy6mPHxm_N>Q_UB2t? z<>h5N-E}5lMX@#3Yn{z%2#EfSMkbnw+({iJNx(-AqBrz+K7@H zpv*1{ZECw(vd#Dyo{4!d$VPHi?L1ZL4d5ORy@c2|4FzjI>rX{%x4*t|9?eNoQ<5X6 z`P%YA%Nnk?2_kS}S8#p=mktHV^xGRAGAbt10!x4^A}5h5^t{%tjaW@Go;Ie0)TF|` z4}g(Xl+aiH`R5;!x4iPFh!2Eb-&^JuEg%0BZ?8una$TXOp+UYlHaz^Awd0ePEIskN zdD_5QS+C7>9sf2P6R%}mmVW%`5lfG_+ad>{<$yK}yr&lb09e6;y+qV_Mt;kOEu#%F zqFOS!sxd&EtclZda&jLCl{g|THAQ!AZOs;U@=VZnGuEDyPy#3qp4~fC9Yi$sY=NWh zu6cmrBsc^NUluR>_PsXL7Ia!o#1G07Y0XrSBX?HNry6_vd{{2d1#Ja~|3?WQq2I%hQy}o%j`0AzD?GelB8s+wx+C@rF`L83&K#&SLtqy!C zw9!MiY7@6yXbK@0bQIb!ldSZ}DJa8; z1G_(HL=H?&?j#YgsX060%zVvEwXSoIIunTyzg%2g!pw!2Tg~(N5?^KR0!|69#_;iFktaP$M_u4vK(g%4FAHz-xiWD<68c{W#y`uc^KYIk6{t8i^2R@vua+G zr_6GBzF}MHwG$^!kVvI-)r`jV?BOzj^gNqMH^JX>Yvv#ZM?GJ@;`OvEZs=&fCa3(S zAwWR1&uT17Ia2g}1JPafc=R~Nf&`~m{bLlHpj@mx7&WWcc1L!!}owP5@$UG(^mynPc zgRoRv47K2(lhPYLV$(tz4Xy9__U-lc%FWs(PP@>TJCF8Y`RY$uLMb2a$|T-xnrg;7 zlp@>Vx_w}cgl6Fv7zp8~CflHf8^Qy~=}QpEmf8~*VkLEU&%Vsoleq0iKI#7w+uX%` z4NTzVn)e)TZD6{vk85u7swO&h(xN{qwzRCF&3IH?oY{NPz72_dsyIaCCpjI5I_aG7 z8%8nrH_dj)a$~J2p;e1?&vVrZub7*s@{LU(6!|(DYilLs|>(Wdj#i z&o|!(y$gvx1lfe~uy;K!7`<*-3P#(J?rai~6 zd~lmB^MwgwfasU1My`{}-!*#4NL3*Yp;X`h+out^2?_fzcC6rWG-6P$H{=bN-s}x= z!vJ}WHPU&ux94zF1--15latd0kl@#5Us%6?|K5*KZ=oJvMk^&7-IEX1)q@aV&R@7d z_Vee@z;?4K$FD(dM!Zj|&S_bEA0EB|!+=tg zNv5c5e#0C^h&3**u5qB>9pxTw*(9nS1Hh=ehXdm*Zvo=&8Dzni|Vg5qq|nE6)TKv4Imhn7g zhqr9mlDGm(5#1%~y!i(LJZu{yYzAV9^faGRi1wbhJ=^Dk+Ut zXko9b?8jldG(BpAzFqfTds2i0en%{?L}c?lEm;2CSzx7Q)`^x-L%YFe`~TWNCJ_Lhu!4NxaU3V&AV0t6aU5FoA_fyJo+fOrr_YeJ z$C`gHnJv%l7*W%`CCzSQSsV^~lqxg;^`1<~!u!+CFC9>dXxzHC_H%JDVg@o?*VM|J zR|?{5Dw=9u#gXXRGL(AFhdexEGOV^s&92S=5urE$6OlIGBPiEE${9sjZNS2`8DVcyn zj2tV=lUa#I1l{`LG?l17%z5MpsmpJv=8zjDGG8U)damg__2qgZl|rU0lv{`#HjjrO zLP7z37{}efCfk*ThzN7-flo>)%c2gl?QMWcQj>1}YuljO~5qO4MDxv%ebh*QUd)~_0gTDD;zzctE(F& z?p3~*a~&*s(l|na;H9eJ=_J#0k%<4o*em&roJ?#)US_` z;=@|l^@{Aa0j&;~H$bc=lpzGxd+$wKpge9xy7;}a5|7eI*kJf=dM%qGa^z&MwhU>b zVkFFes7CWL&#GJ1)qUXIJ*C2o((&Zlf|1kLmJV?iTPrT7ylGH8vA1REA=JvtYSLqw z=}ii`DJ40VHrOe9JRly&t6e8b3sR$?^9&Pdx~(YmxI%ThM(%6K$Kc4!$P}FInK}W0 z%v(-)8mXHtDs22VlBdwrO5n+_PLB2ThoZ5YA&879B43iJlN4fu(Js+P%4xBfLu26b zWEg*V+7*p*n4*V-1^(^M=i^V-8+8V985YSYT4-yMrY8+Ezd`%7wADIhW#2rs z0H)**-f^vNrX}A(t#ql^4G&2S{vx*gVchiNb{>8wWc7s=9gVRhmTh@8?l=Aky~`$@ zTSe+h7b{M;EFLbFV_F^#W)|l%QsoZ6kty=oS*6`xe0ZTeKcV#VM#tbc>Z^-Om2hz- ztSon`Zo0m{`lp4T!^kG>gy202#ciX|z6lv}VWRD>Orj*19^qmFIu}k5EvDuPmoa)M zQ@^*THl?LW{CdcN)76&dd~Spg{$t=h(P|(>S3LJ^FAjP~%kDI$#uE&0S*fb7rjw;5 z81(qg@Wflkud#>E4}mCO%>FaSAYjeWQ)o<85A7*_lJ1yK1zNKSV!xm##-rIyFl1Of zZ4h43rZ#_(slUTJ-N6Aq?Vit*e?nW0AWqi3L9dORsG85jJTaCy&odA^56GRKyv zpGmQ2o1QkRtW}zseW9Y;w{PD)tL}mUEMCW3FLAUojJegaJcUSaoTeEdT?atik{Kto zbkq5Zgvm_zsQ`ggiMXuYbFcW1mX*Xthf%|ht{nH+>~>_+Ie^?vP3yC0vdFEIUvHrH zY;W=66#L!3MJ`sX$JjCQ-pqC}v}=G&c(>V#EPb;_=p>wRoijCkRh1JFme(L1(uSKA zBUyQW#ye;N6O4BxG@L1Z^VV0l^YZwjx6ON?$E}{Nzl{*ZL7eqJ_|d8t*(F{ZrE$1@ z0C0}b{vqO@l8=pxRw8@D9G2%esruo=hwFnM9*`-w)9>tmMLsIbURT6iF*mCPGikWY zH5M-Y=I1M${I|=5s{p??CCct6Jhp#+4*wpBwt+P^7;+_nfwquD{(KL0`ZHs+6Qy!>z_O9$UnK{jaZRR`eMQRm=6tIL9aU=!K0ND#PU2 zS!4p0?xNX`78r7g{wyhP%SU(#2D~;_m|;{QyaoEmmBxjWm(ZbV@wiy7mVvZ$a(p4r zxFUmjpGAXrFC7rZ1K2hHn(CZqFTkS%ML7HBfMiPG88r%M%k{=;dpWh=yngp-Ev?y< za)dh1;aZI7bZ?nfv5ncfiY&*$(fLuXyrff;k=bf`TtT2-vP+JS{)~)80yG56cU_%l z4}>f_08(8QAqqn=Fqx#PDjoLLLoogYd1dkp-+3MzRVtAwhFj!gbBCgpD6W9&;bnjPnOA+)N}*0dhG zA@4R{KQ=py8cGNOq^=ixlq(s%hrfyl3euaInSobJYicsAuC8(v)H9trb&7a?$UvO7 zG`^nKFTgf|PDbMFK7dQI+g75k^A!ZjfhCZl9(@3}D-vXOMN;!hOy-9o)?UW?KYkpTEGae0`ZBPOpAF3a=3q(r2 z&DqkjVF!!NRm)pMiC+QvI;|Bs^AE#|HHr9F=%4}kkVu4qn?EsSy&$kHF*GXbMB*(+ zgz6cJc<6Ep%XI`q>_xXmdx;b>xbXB)V@cfT4wwxDwkEE!6R+1 z$SV^QN~I$p6`-oyUre7SY&No5>~ZK5$;`e->gpPnVX;7z@7_**2ttTidTeniA2u2O zi?wWY_wL=4EfnD8!&&5dqQfI?|LEpV5_P4%EI=M z;*;E@=?Z0MB_8y$2KUi>^E>jr5LL99ks8NUY=NM1rtWWA`)Uk9%(!0nt!EGG-Al3# zS=&f_rcD)=8nUs&q=ZIASvKYFJhP)ZbFb}3uS6#d=WSct(+ZjCgt89r6FG(i0v{3m zO6=Ceug1tPMyauW?Ar<70>7`pgg@M$-DnJo1gD@tw+}{jo!y6gxI3at`|4G1Fe6e| zGOGhSU-4e6a*?hH8fN~#Md2Jwj6-q1nui!)64VQMSguc zO{RD!_(ij6!d^d8U{Z_~Sb)RuM3McdoUTt9VqSr;dArzQlO8UOtYAH90xCw z-zG|ytL22DH&;U%mvsB_Sob?92n=C_-4y|vC0yNHp7$6;jjq^JB= z6BX_1$=G{J%)R9vC7pZAA&%ub^nI&+|C0uD9M8SAa}0N=X6bS+JUJfY@-8(^6v_mP z&+FQZIBm9nd@4YE;rm-kice`XFP(3ZZ5a~ib%~c-(6E1I@9bRjLgFsTG;6CS>P_wP z`syzQwv2o(d3)($I2qxy+3W2T>|lSqJ4?Ih4uN#M7u7R1&Y#y9YwRpRK#rgOpiZII zb6AX@UzK4DT02A8<}Gi;x0^v<=#=P!i%7lT`H47avMu}4vCX)JO@1Lia(*r`-?@C; zzaf7VYG+5DS!mU=gE+am%%)V?9!k@!siwEQm5<=$V9{aWLyL>oYbJzI;L2Is6-e?j zvx!80p5-}`s{LPT9eiznc~P*z#L8c#;2-wEsN#PhGw1<-^nCwG7#}(n&{W@w$qut8 z`36fcjL;jNvF#M-KFQTxQs(o11#Hq1GD8U5 zVQn=kFCJwyI){D0-Ej59@)+E&wIi(P0Ho$$tfKYnmohxLUnQ=ehMRD_q<_Dj`;9_2 zgHObhM#psr>0@2p$@ zp%%^c-=(G?ZT5zMqnyo4l!AzmgTn2mqCh#3b?W*lQ|GBg|Kapn|1)||GlNUj7{&>u z10-V7!c=E@7)8pTv+?t1x*Ioc{M)31(8X6Q`aCd@k_WxSY&4+*R?PgZ-OzJR%#2rv z6>IT?+EHtIVC`#76u})o)YL@xTroCIB3w@iuBaIw(54NH(7}*KA z6;;GUGuy%-i$|&J)Z6We?_K$mFuLKM4hmrJ#_BsJBqaF3!N6eIs8T{S zZo|iWN(!65Dw0oc*r8!#ykr3iOjZwMNLlVobx1@VzMCkQdp=iFRIt|dFE)zM?gOCL zH`-1~fBh+hZ^7DnI(BWB^q>qb=>cHo5sqbDu`2|T+y<{MSv~|7qCh_$H z-P^UwZP4(fHv2O_{)Mw(`mfE^y$EVrZH{@qCNhbRJIu_-dTSnsmATBt{g|H*>$QiD z#rLNuGA4!#(td;WEP)brg`uj)_cgNfe%Uigqgcd#;zR}ZEujz-*bW)yyp+^d7*|*# z>oNSX3lT=Ry_9^Sl9P|amp9ZOu|<4seot4SO$h8cxa4;1D#8f|fKGfvx^~fSbVraA z_HgNy2Lf#|B-pXhEOR}6 zXD@@)a}4`32<=u_0=AIy)Z(88T(W{wkGK_gcu8Q%q(bi| z&?M#-8K5PEO(qB}V6rg4j_hbUR zd=IG)KJAbC!zgrwm`fsR9FJ8OU#O$zNdbZVc+F{-E9AG+(>c(h>o4r{3>_JDbabpj zC->@z)ze&5Q~hxa0;&;q_ooU9qd@|_+)pDoG0TVls)GDQfK38*Hj4bB#NFK|95~$t zyRgU+Ew4SW1IT{yPe$`wF+d8TI_H)j%c=Y;7!~U2#x>`Bb$n#=c?K8YS(|am9bEfk z)Q7InU8C~ya+kG*ue)NDQ&fI`4^{X9>k~042{U}Yh4*d zEECLSGuSnnN>sS&dqS3b@#3&HgfOTJHNdRR=^Cdo16`9=m0WrPuJK&j6|R|e=?imOo5S;n1>+bEBw%Nc;-AXP7xCm(5X;l z-ot72Ff!7Y$DsOp?c-y{laQ+S19=lJJb01yAst|MKkN;`9&A}H21_B6PM9+~oj@gdiHVi?e+pfVptb2dm?T-^*d< z+X0L{pv)8fo@@+Tzz$D=y3HHy>3|?nFLy7RS7!|!cw!R6_SNHX+|Tzs`HufFW3}{u ztT9!@?~a#M^6-W5sbQSrB2orne}_3QPCm-QvI)Ce zQeQu=%8YmZma8j4!U>n$;!IBfBx(cnM~KdP*ql-<5(I={e+b~RYgs7NO8l}O1Y1NH zCD>V8TeF_7>R5Kaco~-ZegMJ>Snm(}RJ^er`TkZy$sbbQ)AV>4p_Doi zZ7qZR_nLH561>iwGdDNS=3Q1h7dGtA1W1)%hqj36V`+oPQLKQ&gGh`7S`37U6<3SB*5Ncr0n zo6a$cHG|=<%F&ndfc~QSahAGK!Q(zOgw^OOd${o=;C>Xw|9CVm!uWP+6fkH%^kd_t zG@=e?CGC!zSKFDQ!ah#1m-G&1iSMJMjGV%zO^5IDPopVj*#rImQRaPKCyTDU6bRSP zs%DQLE5p2l73&WBzEke*?jH~ayTsJ$PHyp~k%Z1KOAIX(>KVFTW@ zHWXjN+x%%U@p35Z10W!EM>&Pdm$N52j)&HA%tZ1)(<4GtsR;egC!apdLA7PD{DmLP zr#S;cL^kZYFkxi3;q!X>57pXITD~Xvseb2m|3mf&d5gA<`rQ_bm8!E^EIuxnnE7<* zi+^2Xij7tHP6Bk6}i!H~XZ|9%tO=l)3! z11nN0#_-kbC-4KNMv0Cz42q%vB?D7alo;m6#2S&wTo4?*WkWun3H}5H**&T$80ZJS_gd<4U7;QAK zz)Xq%;6d1GAiDSz^6JQJY+TcfjCDD1n@T<;@+CP!oF&sp=ckqO(P=Fu&_;C}%O2DT( zkt1tlPnG$=5i#`KxtB*v;a2$~{9Eq$8G;h=V74E0XqsX*?DHG>x=B_$FGZ+XyQx&Y?|HqCt$eN5i9rfcj$ zx`G4v^i2%n=Bm95C+2GA(CWVqX>P0|`zZAAO~A~=b|PRXfQ_-^+uUu)H$=lM)?@Ji z2DP@}6>myP;t@FX0@h5(04599sL{@?g0=n!a(FyoD^NM#t>Ks}c&CIDltZh468rBW z=KK1`$9LS5BqlMjk*NsJ0eH}($B*y8)YYY5*zj|H{sW8Axo7mv}+M_50dB4bG9dMY@|E;No)~S{$4lLB9- zB9J)fDET?JJ5P>dGXB+(Kd?;`P9IX2Q2&8a38{StYg)gPW9Sm@GPr-Fj0E65CUsGo z!C7T{(?Z4?S~O?4W)EGKL?Xx|SV`brE#)$_ zfG~9O9yDI2Ru*Ts&;phc*^ETOX(Es>dn-AQ;IJSA^VC;@Nlr|`DGzW|Pn|JF^aS01 zbCa&aY-tqZ-=erT*qTIp?FxK8=b_5NUhNBHau{=)m@9g7b6_*k8;-viB4Y%_X6L;- zzKl84JxmqM9`m#-&!3O$u7F{6U#F773SSh&fM%v{nLip^9GZFL&`md!Fb#baQdDKe z(f{jaWC#@o99;muO0;2v35ZAPM|=>@waSc>qEBcf1&e8=RQ?d}8$N1ot!g|~sNL_8 zu%=qs{~l44f#Dqv-O`T!BBWX-G50quy0;~yu5JH!rLoo!{QXc;_qh=x2!vhqVAGNA z>-1-6!|O2mPNXv;#Q;cEp~3PfI(ny`o*vOJMT@K&&sh;;R2K!piF1xpd4QGB_>99b z6;W`T+!gS9CD~6LhJcTcPDDuoh#)L_#4r<6&okZ|WdHZMOrgTXn}|*D-#^+8{=amR z62(OU?xa0cTEjP ziI5JTIPtNP^JQwVN6^%RwSPbF;m;|&3j)RlugYCA z_oIP!Vz0<^nZIi|?0fV-pL*(EV&V{__k4V_U@@EFF`b7&To2=O>gwtw&CS_~f`Iwr ze~)IORoqwgU*~wG{!n2Lm-umC@})QGzZ|Rm&H9f_JsYxIS|IZH`c=lia_9!O3R8sp&z4VoG1{_vGzF_Ygnby+W@b6;9vH!Po z)l{Z;u-u%IA&Zz%c$efedE%;Bg3)IRA+mzhyUhpB)6xzqPbatFq%A4p)BqfGglzY* zxd8{J5Npo+-wsr3({w7Q$VwOVKbWi>#C7Vb+3S%@yzNYvFU%&L;juSlbSYMu{$dVZ zOGtJ&St&kR^56BhQkj0l`tO@jJQEqN=9imN^pR8LsMJt(%3(kMkDt{BQf{q0dgxx0 z`cwEy@*w__?XF+L&ivtq49KP}y2)tI3Kz5ed%4;lBKK~!*@s>J`&TKIdrx90204=N z9Bw)4Q*rWSu;#(im!S{cw-dK|%b@F8{nA6F6vElwKW6;*)HlUrmB043|F_g-DBSsn zT~6DMJfyMHS3Fp|O_(h%F`8mxfM=}3*R)3Nv8^VlM$1fTm49iv=* z9gdl4Sel5kz{5jJC=Y<_ICp^XQWA34QKKWoUMUI*ZclIC`F~eJ`0<6<*~tFh;zAu zbjw@^(5qn|O8a*$_HAlZ4x{|f^qWfFqe9cZemEnVBCa@=>E(pQ_)^=1UN)=KTOjRXBu*cF&%p;l%M3y*PLZZa)zW1iDP;fBo^}J+%QujIs6% zMoeL_hX?uj5rX9=6525+A*l{TjCDr91iYCP!J(n=@CWkjri#O3XshhNZ1G`&Tmn-K z;;_K?5Kjo30E__p3DexPEL3EC>?7eFJd-0w-X_ms?%B6@wNOomjg5`u4fwsi&?oim z;+dS|ivoW?_v6oJShliq38|!7ZZ3;%Pv=&axcN7k{gGk$G3 z7P4}4?TYS$f;8H=50muD-QD|s?k;sUcPklOs(oIc*MMi)nx;lg;)edp*1wqq9r!-Z z6^O%hG#p>2NmgXR@7_s5aHnq#efBI6JrHclA+R?xSa%N)0wbEV&)%Nw$^Ee;$Q7qYRL)vshE(J(Ou;#3a8K=U5z zPJJsKnt6m#p6CN&4Jacx;q5}ppsr;TsV0QE1~`Yu@FUIL{#_(6C-to{07S%eFMxJ+e25sk=v&M_gN2DKv?{mQ7cce{*>A@|ekWB4 z+v=tl>c%wx<}D_dm^g?3%v~y*;HJ-NIea>crt=oN;c&{s=UHpf;c6#(8b7PCa2{u# z4A$lRvdGV>{XS2kp?^3vulW#7r-GHu*WFplFIxJIojjiu>U^o+CUbl;^jkzyMiL}I zdJkQk9~R$yDFiVQfF*o~fn{Pgm=NB;rx7`zmc(@A$Hr4D7R^QWCIk|Lm_$@6cMlKS z`7uSpT0tB)BtiEOgV0+@1vY)bKyK0Lr-g+}K2869Ge=hH>P~(@>NZM^FOEsR{vStS zJ__2GdZw%2UnHELbe^K%MCsBqF8U$Qn-^5e3*1%L*G5$sRz77>n0kFGf9$lsVL~Nq z&ry{#t71j=55Csoig%f3&)t?WD0;nvi$ch$ytnLK>{VibpI~+H3A}`RiL4&lyASet zx1|;_I6?T@A-((!V32s`>AX6#x${r;-IER>vnG`%Vs@Ng()K){pd=`B z<#t`O{NWtNQ)3 z&vE>YXZ-$ouH(4xr0crQ^ZcBj^Zj{$-tX5iEiC;oKmnB)1RMv6G#OIxfsHYMYINBW z@le2*B|_x@{r{K#r}9<4ZHBYA$TE{D{yI-;+`ku0r@M5K-uYtw9+|VcHjQ#Hk3}HG zEXWKEBE&4gHf35m*R0z_RM%K~iBzK5U@A3GQTOd>{Ls*+9I~h7)>mvAbJ(%11c#VD z`(!c+LyHiBt^$9|2ljTTHX5E}*zdwNnOv)JWaKP_d#2_0pv;1(up#Uf*gMw3%wC2D zt5_;0z72?8Q6n{K;r%VFs=bxt2&69~3%36;u+aqxNuy^2hnbVVV#@m#@5^1wf)eWk z_YUWgKr^hgp(Ikd<2GsS6WGiIuy{8_(0`g9D!5?eZ-ylsHhWd`v>2^U;#K^tk1XY3 z%VOU1&NZa4IF7f@7}n&Bo6%1+)b$7|5?~2z{=vUVgd@^o( zVZ0_Q&kquGQohk|2{wIFzP-R*7KIGq4o!5+3>--PsB_r$bhAZR73&|h3LzUb-H(BE zP-q2`OC5mJ*nkNI%0;=o{E(2;A}uob1)zq{0CNFpHUb0ql7%G<4#K}d*MU$=P-CLJ zTm&(6ZNBd`a?b&220J$XYS766R{V_{H>zQdAYvK6Z~Q4H`<12Ds25x7(I41*l(kwF zl>#_C#l;O2Z|};8E0Y9RD$(SvDm$1CKR$ncBsXo(zatg-%cbbv+}fiLiPN~ghMWtR zNG4@s$G+LFd;I5TQt#QOvyNkNfpyKwOxT*!m@I-?&#&1F6V^3LJY1~gCamj_z_{o* zlb?-mtRNt?$2D|n80&=Lg(yyTTlW-HNe>Ko3MS6nJvWbOW9#T<& z&e1(i_j~XMqzk8j1qy0iI;|$vfoU>qHe!E^IbJZZ(gU3=Nm`Y`?ZCu{_--PcL&(b*IuMOKn5(ZmlAAFSW| zaUck}1#Dt~cqgU$#ypudH-zb?j%{;@k_znkQF3N@p-m}*yR>~ctuKNqat#K~C$MZrpWM@anQl>l+b zeY$b^y8sTBGTo7dGifi50$g79ivF7kG2zobk0)O=$y}(bsDAF;$MW2ugzfcMobd(x zD|fLqiC36c)fBYgmNTD>V#WJR*>R%r)uOJ6%PClKIB;ieQ6AJ9BIS2ac&*rkfAUoOkh%wl;o{gvf)GZyc zLEV;gIt$-qBvt=gSAfA4!OEb_V{|dyxWsALc%2k5X&u~#s(M!}rBmnDHmxLM!d{4u zdiOWn<>2)!%U4|1ec>Fio=nvq?fGF6Q^;?nQ$@|+aK&(YY9ov+~UzI&`!;0a!~u}?w7XDH6TI9 zgInHaSlA{bpueTsWKJ+AIhhud_b9mexnaQ)`y0CihwkG)nWStyNWb8f#M^skR6|~O z_)5umLg-s%MVW8^PHL}yK(e`*=^b0?J;n=mnE?@8ntrdx9G?fHH>u$THf{>x-O-fJ zG>g~z@!7*Sp3;}x#tFv{Y&O`maM#STKNz;Y*!MH9>!(*Kf9is8$}88<^{w}u*HTs8 zI*ixyuj5vT9Ia^F^dwd)%}YDiycX5b${G9BErVHM_{00>_d|*gRF$!lMpI-4Lu8h0 zdkpi~@G}@Aj$Ie-6(fxtn}~aouJEO`(^T@pE4p(ZjuO#bd&8YV?*FdLGZK?P>_N|W zB|>zj03kP+5Ht6_h5km?BVKF1I%D!Hq}UC1CmOe&&epWn_36J8l+}aDLP(_iNDS9t zGBxnEqWeJKcJVt!_ku`F&8kPFa{=0y6^ArrhpSaiug0j&J+77tsy?Ze^OTq@PNzX! z*O$a_J&f;@Ur1M>XckMl>u{=Zg>~HqMF^Kd_WbZhQ({KqM$?Osu9F2pq;5kmDx)NH zc(hz6nNRsEiME6Q?TnbjQt+4ZW{f#iY2iJeA-S^B{Zdfl&cX#m}(r+ye8{b8m(= zDt2GbGRZy2Ssw{eSSNYMB^8`IS4G+Xl4O%zErpy|-td!wd;Tk`FEZAQB-V`Y4CMGf zEeG#62}?dUOqtAz=qSbN z&Ml=e6C1S)W`lP*R+YFj15)zjKN>2aJh@Y9*kZ=^A8leW5TEg4ENGn(<|dujHPhG; zJ=MgHBcXHwQXhO{WlA?HCRfRHjLca!@AJRQG<}+}>RyFX+20y_j~CmpG#ZEZ4`;pU z%)AD&(zVJn22*ELWbsE+zJbW&%WKiW3hxWXlpZD~CkvDwGw_hag>6ZYT@fL933I}T zGTnF<9*o^r*Po5{3u$s5rwvp)3oCibH6vrrY$m(6-La2p-%YQ&t!2P}+^Dy4vjz{>t9N1pGOk zjqmpMdB^-u|GFEvs1>{}Raai}NB6Fbnl@;~8;b1>F1aXO7i@pes!5DLm~#<%bVGyG z=>x;7inE&m9aMdj^+5Vw`{>Vr!S2b%pRe z$70@3Y?-n`4#i4YC++E;w7b8GtD&P9;-rq)tvSEbVVAP*DRvt>@{LYXk?XYi-S~rI z2gjefA>S}qH@(ynHs<=%ubNe*a+BkXPInPt6i)7Qzbxu;7%IW0IG;2Xy6r|fKaP3V zn~EL9PEPyk6RnqB(;1Q7t$o)YoUEnMWfSb=HdU?rvN?Twd^RCMPd&LH6W|2eL_f3_~A18H9!?gtx%tzkVHcY=?Ux@!W?)-ysM_qdJ zp_|+DsxN___}|%2@1_jA&h7d7;b3iefy_)Jt0-%KybRk^uwu_9JSo8XVX)pzic#po zmgGi{;796{U7HOfq>CXALr-U&24?iMGRcp9Xn09vB zgVdSxLrk}Aj8*VJiT4pU_xPs6DkKaVl(sQ4>72hE6t8tsc&rUNcf7jM5%T@kW`{MN z6^npQX>^o^9903J-f=hBjF1oz^6_HgilANT#4o$2W6*+Q^glw&Whxa%cI^ z8#mrx^Nm-|;hFURfh+rkNt(OiA?D#F$*%{)mn81K%F$h982rM-ratLGd*${_AFiv$ z5#<-D0W9oM1n1+JWfQw;g^J|xKD1#Q!P`XoD;kT|JUPBTI!?Pg&gl_)>}RyB;)&!_M^Tm*K+gI5KTB)&^o4WQ#e0`@ zTf|4_&x|*BZC5W(^lOYit`2l`e-Y+7;2;a!E$F{Lg<`0Fdz=J!>Ld6+(&$Gxc~I1P z+j&^N-gpRmF5!hr_mPtf@gnAeR`K>75e~6Z*9(rV7e1OxNpRQ*_r1gnMSG^3Ygkvm zE(moSHnbVGeq_HI@Va8Vm?iNuSwHQ|-SDexURU>oBFE;&@Dml+aX&H=B)ML69)m56 zrC=rUXszVC*JF7+kji9nvwQJ=uh;Ly@o$VTpg_(4t6gS*MQUj@{`AdpRr`77P6~N5 z$QRcvAecga;sUpB07JNbceCvxGT%Hw+6k4vGL?3!!r27|{x zF4>5>L#$XZY+ACtFXEGV^@l*;aW}b(&g08aE>591E%w&2u78x^-2r6a9AM5C!?i*aFB#~a!0w~z8Ohhn-k7Ij{CTc+dQ`xutsdIJ2vsn$lZokE`l2(H5K zq4-ZPBt#837euHSurL_L$@&=XfZqlL#DMWT(8?SInC2FM*8vskMhjb9z$F0z`vd58 zJp+9n$_xa{UN{bn<$9>V=K}Kr>ZUP(W_8 z1X=IxU_+FnjBHe&`w~VoyA|4M$L?rXOH^`GHdIu<(BRwt`g342iSuSNtB8z8fBPMu za#azZ=AWt3f&}?S)9ePkpVBuDf_%-OMDk41V*x4ZQ$Qw(OiyK|qf=q7t^zn#;t~>c zPz8t5ml%LymwWF8+-!zm6&MtRf$|LqXaqcq&r=#FT;OOl0(S$N>}CaD;E{rwtS*Ag z5_DjLw?VU|TH^tKOPB0|tZDg(Xx^5)3Pe3az%IvJ?{W2aAt+PA= zMgC@f7*t9~&j{HY_CDnAnqI*S#d+X={$5Lv!J38oX$XlP$+sqED(#PV7H10Ic{W_Y z(7%hVP%Wu0gJI2b12*2M>y!?$VdUe(_l8`BPu|~Rqy4q!P#KDzrCnA3)J;S|%TZ!B zAXG_JTyQinl=hy1AAj`|^Td8l5H*4sh0tMOf29jOC7kAa_wHe!5PTdcuRP#PiA#ae zLRAAi0zJ3^>lMJT=Cz$Aj&j{l1BlMu(ws>IhlPOP5V3Bc(I~t>@eb%(r&p(18$qcw zM#(gP8nl&U(YiKpC@5AXi|?Xqze-xTbcAkW+?4~iCw8wSkzXEPn19kvUAOf4`np`c zPx?EJ8xx5udzdoHD$q}qZ5arFy|YT_$U}Y-;ND(c_5mN z698)e0Ab=KAY0C80K}zCzm*AqY9O}?Bxmjih_)w@yy5@VCXgo)?#c$(Dp;r7yUTuC z=B@rGsCV*4xgh&?oYv!X&@d?#R{m5`jX(4gHr<1zVSMyIv?pJ>C`0&X#N|b=dyL~kDcb%}p-s3P^`gql}lWCiO7L!&WaeiO}A2U+gs!PM9UtR3I;6V&xy8K&J&{%};#>|B{Zc)Yli?EL^=`@Qs#1zz8 zGAnW->!QP_IT7*Beg5fzo^=Z3j!gyoZ_@+fCA2IDo4Sp@v|*bQE4jPlCw5f0zjRg% znZ{(tT)@6ZOzAv9j+st+leDtwImM9w4>_rYqlX+8c{j` z@5Zq@@C=GWxQ%lD=<0(5T&m8E@3hAyZJ#eYhoO$ zz&e8o5`D$mb@r}@tXu;IoDrFP0-p#luNJ7W7u1lKz9jG8WnEge@d-PQ#eO(mDJxAt zC;hTYft;$v9hCg+hTH)r0&Ebh@Oh?C5Qpam@xBs=Xh&UY1EVKiMf6-%i1C zh%;w{+()1&kP5=scebz&Yw8H5IyE{eWK|bj#MYnoWEb>1df$Ql_pqe}0X$R>L~c?* zLkVu-Q-~8_la*Jo(He5uJ^m-h`v)&Vq1^@I)mGaphHtxOz+Vd`IcEa^e?(jXBzDUI zboWNc`F+7W@JF{6$`)gkm=T@GfebM8f|v3g3lF^JgOZogo0{ULua_^m6JQE0>9_ij z^`@O5mcbuP$%0I2TEkLh^^^6UJai7!WcnUSJ(FpgB<-G;^G%Nv$QCo8-Pe> z0~{U1?&GHKX z=>&!r^e<0?-ijnNcu|NZ8-n}?oEln+0LTE1pg#x1kduH>JPs{fgkX>OenEe^Ri*3( zaz09i$c(t@a_Y>U>oM2~NH%k3YGFa*Egn57Y8AX%)QouJeP^R#eIK4~qG#hQ-o5H2 z@OyIjnl;qdV*^;!)`p`g>^2U5_|Z!=xS9f{Kb>s*je8!oIr;(@3L#U}!)mt-sA8b8 zgJ26_e8bYN0VWLktvG<)Z3J!<0tP}MbS&UEBCZY4BtfhSU}Y3W$QRImpkrr8^IDJ< zlX8I?9Zn{~f;8N9zMaZt{Im60t_h(bXiqn}v)^#@6f);wPQlE%(!t3L4aR=&-P$^D zt?*xaC+V8{<78z-svPa0T;jKD;>Dv`-c)6N0YS%sW;tgm2+!CJK~FxubrT>0klApR zM1gd1<<__32>I&F04WIxLt%VQPTMT^;heVP$zQp8u=qmtO!SjT%<+687_z#MD1v3V zIuwqD+||9Vw^R*~cwl6jjt9`+y`|L%y+owhtC#-kq?rAIpav-zCA8)n07q)IwWNn= zF`(FNXO(V)?yf#HqU#1+83btu)?mkg2Y@Io0>$zflVVbJUtdY&X51;OWz48Of?}s5 zyu4?(>;rF-24iyjLQRThy zbo`y0MUeWlLm`*1vxB;>iMV>%CYg=$z0a@Epw9=Kw;eF@nq7Yt983rUVJL_+KpsE7 z22HOgNWg%=Q4c93X!|{gKXX@~ot$tlW}|!YI4%9LkF7YGf*8?k)rCIs8tBO=PV#wT zpk(XeCidRrEj+xghWxR0NK~$Dm_30V+Ag<0Geb!{rAU{CxbI0Po=mkgTVyW`y$Ws% zWHP`@zWy6%bR%$-7M_t0K(s~B(&^nm4_xG1@)cm0H5DWMV=wEszPyUb+TcJ+xc|bDn2RlwHZAN7A1{l*n!UX`PJOl_Xsv6iQZ2*Txxr2HgKvIAYztpKHs|S!2q``);7X}&-C@mzYSCg~v ze|!37b!+v5EiU~rp2+X)_!e46k?oNWSH=8@NZJ}83#q z7k7@;d+~(GunUrtdunCdH@+ky!XJ)N7{4;AMCVY-$Yyz4fJ(N~D{=W|?;9%uY-YcL zK?%CX>TX-;4Uj>@m17kLeF;QCe3A0CD*S$-@VLY#X*t9==T( z@960cHs>D4c|PZkX=_q%Dp^UNZ%|{EZ)|M&k{O={{SzfU1o`x2ix!Ie9XMQdp#^17 zKd7lw^MG=#LkTnW5CWrxSDf@1U_{!C3rHnFm%i43ZzvCv1E3Mtw6?bDZUX~30FrcI zE29vnKd{fahZ2DvZyTicE5_H*ClpkNOP8})Ywl9CO0{1SVpQ<5EdO*Aw9AfZ&BZ2# zAt|LYITmk<*ow7soEkNb89E>U{<#5BZ{3AB$)zUa)(3xij&0=v#6{p71k;=qaw3LX zfQf|SbJo5>}c;Y!$`FVSF|dQ|7+{axqKannR zi|~1dlGZMW?Ei-U9L^eX@vi66^w6iLICJ(!O=5vyyy4y;3E8?XA&n(D%KrXESlH9( z|Goea1pE3E&gJj|?%fi@&2P3P=LZL<&#yuDR|+6hAQl@UEmNl6r{sMQuF3io)AY7UYZ{KHuetZ5iNMaMnTgFPrG64N#dzy86m z07Mk_#MCqUUFmnxbg~J+TbtTYAI_Sz3)ymh%1^U&E#|Q`%kBw|OTy+}d%7JJP;|^E zU(90a%&C72U{jh*02Bu&3LH3u+lijtC1Bk~>b{Y75l9lw+__@StRE$jRy@E8Ce^;0$x>=DcNfTsW#mnWqc5HQ1TinSaww=Ajw5J+Y!S-ZF z!tX0-2SU-z*Z!ISq9Aco4;2Lf{ds}x!cj1`VN5=!aYLAMTvl?2yiH`2>iHH|4Q+V*Rg>Mqu+nLQV9*X<6r;8{B%xCt0&&s z5AtV#igpfJ-9eQ}il?3#LPF`a}&nINwOqY1s5;ajSu!p8s(6mQ7iiDVsq zZHckMSUf3W`oG~?MDp0(Ag_9!fq}$%AMxt|#Ww~aR(@L$SVu@{ZICaXzi=TA)a_s` ze+mQ-ZSAN-743yxIr(FBOI1P#`TP#or{F}lMxqYJ%A!V;`(9OIU|;|N*vf_@BMH)G zaPZ&61%7Ta4ex_b9_!`+LK|N9{4Yz;6V6mc_C8Ph08Nk z9|2Aol1#%gXWkJ5_`>i|Lv^?Ma1vwk-mj9GxnF?)@)8J?bc~F+z*a{7N$Cd;tdVm`Sp@KU406|Y+8pgmF8x8 z1%h*0g|{OW^ntRPWE=@N?T-vkUQ?A5#gQWOiw zR*)M(9E*^wi0;ajR^5?HjxZj%FtP{)6d1e>009N*-rJxkLJIMe2jHxkSucgOpo8RuyKh!|UB2;)FBcQ=eljGZl0Vui#U)-B5w&4S8$^I~ zP=39b3n}?{tS(PQTz{|CO&jOU1!A*^PwI1QOO%L} zmDRH%%TEAkyNLW1YLZtSmUJ(vmXQWL5>-M!L4@!hJY#a$JAkKxj3LG9d z1ryLqSR;UGp-%QYYL3)p>W6o~nUfe^;{1qBN~M0J6j4f34ruQw%d?vqC@~kc&mS2A2ctt=SyHs_d=ae#2~@*-Uvxj z&wTIXTxd$}Hss~$83=kHpvn4sBnic67xo|vouh(N=TL7L>N_TDk5|WWvP%k}RFH)# zRsJ=Yq5a5wE|;+r}PpxBAVnYI_qVCZ<8F)fixcxl@V?1&X79Lmwza zS4s?S;Cg5sBm$IXl^4hg0HwK$-C%p7v6yvzbZ*RS1m{%Wv%z+PKjfr59@+b{YneSo zw{6#sE#BK4C2)nRIUX%{0=;?Xsk!v7b-hl9Azdm|tAfndg%V=X!7kJjxOpK|2GH;(&Y*#%ikiHL{OTmW2Z1gMbU; zLE+9zd)ZP}J5ENq>>k>z4vXzczPmcIXvjwP+oswO zNRr6J%HQ^8T<&&)Xo2{d15E&mF4u8A z;Eqy4QArH+j@ltO!N<>mU<#-+G(e%nA9i6N9Y6q%aZrgV5Oq!J&u)$cL!Sh8c&I=c z;~0{I&_JMqag0hjPzN96Rt4V!2@Qa!DBbxD zoa~FQ0?>J{H9x_v6G^n&GL`M@h2H_hV+ryJt^*zD|cNVyQ>9q7-T z3X%Sy{0#a%;7fW66yp5g*5ZJ{89c@;XR75DARI}7%3~$CaYCh27aBdF<0=6fCQIEK zmgFJtuHM5z(Sj%tj6GQRA}IE|ai|j-+aTGT`}n65BA@~8NQjj;7|0^E9$>lmSD>G$ zt~*iEUUx29>dnLw4}N5)5u}{|;h7sGI=$M>0%Tc-NN7P> z|AfUcW!wMm3qn(?9{!(^Q#!c66Hku1IqyI1{jU1LFy1qM$+4()EOsr&gffj;%jme4 z?z0N8BSeJZi*063L81l*zK$EBd45f6j#8nN(4RcLOWCLBxK*0*`#QVjVV&(lxTCjf-W8(b`;|}DG{?z#BD%6Xn1m?fWLIfFPVD*L z0l>+hB@Y*puFF^&afZ6L%1>{f$THHAkkLCK7($aWr$!p}CCcC89P((&dxrt4&--Cm zj?Wm zzJWn@e=H;Lo^Con+0a#DXrt7#8);U)dvVr2x^eO8EBTZ&l5=jwYL@}oLYV34+GH=~ zn@0H0iiZpeL@av1-k0ZnW3-Eu;VLZ`Nvr|5zUrg%!+SBm0f?Sru(4|sUehlaq!Tsx0vqv39vpaZb zU!EVd3T|Ff71t6RiN~Ng;?!SUFWzrQCkUrlv(g|Ul|#Ri~?xlChD;j@6Nb;y;v4ztG` z$lPBMf9_wev?Y*TKxkLc{eXo7DqS;DwrRzn&+MLd^@pz}tCX=ZG`gY`7?5jc&`Fg1 zmZ|w6vIJ06Nw;lDQn-rI^c|SRDS}o7zj}e<;-Q`(WCdI!cVN4MFT{-+45f!)&5MbG z;tcg*@YhGy)}9~%4;pf0z}g39UH=d!7wh2XhV4$X$XhM7C6xWUH%&_99z1Dz|4Ol1 z#58WDmgo=ZRGH#XsRRP4t&QSGyuYv!$aT*GY171u)52ircy}9)+4zMmO9*;X7&@Rc_@!f(!gg{u1k)D`I-TbvlUCa?V0a2<}? zQNSM9$<$QL$mFega^*c_5vcmSUM3(AD}P58Vw$%Pmr(B1{GAUvO1Zovz%dBUs-n*mNCg=5~Ds_ULPs>r3Av@_Ak9 zduG=ox~&`G7tfD8BMi#?Yt-g1aC3K$<}Mtu0-6Vw`t^<4gDdFe6kKwtZM!vpdBm$pWb}&m;>4+jEjgDZ1C56lwHfa$QpU>2mp3 zAS+aT4~6kJe0czB18|21K{)U{(NNG&ICU{I9m`XvmM$eDByO3f&ujFmc$uVV>~rrG4RoahY% zUvg-HAcZS)ISwiTdu%>ziYMJ!&(J0B*Oq4p&;8B3KAifAgC!citEDIcCRKXMyBrIz zODyhR00!nj1@-=Zx>|4wU zNu$T%b9RdSv7&cQaRxMvIAt6ETdX0ghzLJ2h?53=&z0K*A2wfd| zXMMNB|MU)jMy@RGxp*eg4Vl&o(wK+BH^#+GxvlX!4YP!+J0z}%jB8g*UY6Z*=H-y{ z@@3}y&z%fvfs7O;k9weNd`f_$X zMH9)Wl|Ss;6^ZyA=P6W-n)c*kO)#QgwmZRT_@cvI6X(gU^WgEeHlr?M)sEB9?;e|+ zH+V}8ApOyQ-J4KCVg3u_rd+e%DQ;w4j%uJ`|LBvZqkZO%ZrPfNXskH;wsq4_vFzs$M|BC zU>WoSw(+r}6V-4|0O3$A5@R`rA>< z?B3(@^BjoUbK<@8R3VSqM*Ak74S#33g`St>n=h%PS!dSkIt%Q4_kL25ToXnqN$s&< z{yLhRVS?e5N!!cC!U+-S)D4}=@f@6%E$y=eRZny>;%~!I6b$`&Iyh}PFEy)`3Z1WRjTem{onzzj#|w(~QZ_-wT-qW%^6QIZq}lE$#!p%@uV3LkZq64N zC%?*L(lF{j_zg3$R@E!CoNM@Yhtk(i8NwxMhxdmT>Ww^_pr8Vj9*`!a_VuvTigeYbM@JSjdzT+< zuH0HT{leLzSa~f``|CBD_1cKkz7Yl`#>U-|h4!lb;fC=&H`^bI**b%giuG}{`q`mFhmcj7@L!f9(TXo`ydzxoe?(dt%zn@`O~x(qBcpL8qf6A(4^}8f z5uF`&%w-k0*uOxAa>t7Ut=Yqusr!6+$KBekITE(PfR~yR-^=Fx7rBT;W%zzl1)s0u zwS6IjS}HGAku&-5+qX@jx^2O1*QSlQpj+j3YOB#RP7^KJ3Wm&1KfivzGS;kU z9kQhUQ)0hhUeLVXUW}xRiCcP1bf5pHl;&VD%eA|o8p+*k2X*G}er-)3Ek4!INB$_( zC364UlxD3(z1i-Iapv9oGxAsVtLF3{g^oE3y?&i3@s{~<*}IfLcf4pUl&qwOFloOqL-z6UvjM>h&F-{GBHd2~QdquS;xKNryOvg0iW z-X?(!h1)jM<~{O~9sR~->-%$#7Uqv*{WEMLoGR0w z%l8d#)I_^o!`$EFPtZLGL{(Lp)u#B431P0+`!6=R?rGhPOn&p`#v-;@VLgNuGRq~d zuH_Mnr8_SRt|X>lerC^ z9NlhiJ^Gf7xt0z84vj-B5_L6BB!uh~zxSi(bR6{#eb0`YcD8{cU8x_O)ZoF1AZl5B*(kkWEgVXdaW zV#lUgG~hrI>)nZU%O1HR?XS0gP)IF}FgE$+;;>vDMxlI6A}XsSY0HzA+$PKCyY4C| z1fOSuzVPJvlm(c1r^X3X^vi3Qk#&F>grJMM1uJ(Vk`1T2dS$C8%WiKate z&{g>)ny8Du>#CW|753s?DxnP#U5jezjb+k@&rU#UqEtjNc zbrbBN5%2EiG}SYuYe$xTtDhObqu{iL-`%(0{cN;q zmQStMURRujFI->J>lPEacUQZ085$^2or0Ci{-+L)w}g>oVv5{Di($rs36bYW+Cc6l z;FQT%&3Pwr>ie8{1JF1lc@42&)+%3B$hV$Y$&|SBoDH8zur1Y>@PLC`OFXJnU(&a# z7V{ma9`dCrc7L9l{iTlm(}|_#(hpxR@@Bv->a4t(P-Ya7#QtbiNTrxJXvT3M=v=yP zWS&rKPZ`^JZ)|=cOHpfSEFbzSw;*lwN7sO<@QCGh_1+hIDf*z5_N zHIu9R7$Y3PhG(G{{AzJUp|t7rsnMwtOv$SRwJzIS+B@!e9qFGQfZpX z?ZJQ?tXlu`J^NMz3@yxdE@989PaFDZ-=A}j6mCRAgpa%6^;G#x%RsVB38P)PF^^J|rf;{-wgNebt=Q@SzvmP=?G>}D76+W2G zb{M{p6L4SD$Ntyd`CmQX|E+`dzj$z&vR~P3%p|}L|BNtV$PObcN@kgZcMrX49nKjQ zc9l`|P5Sk}oIew;c2Q|2)3An7@MArPF86E-nwU z;onraefuSWi7tt@&RTc|ZE{{x=7||4%0w47Ersh*cQ_m;vwP$sSNEn$GOa{vvAJws)zX_7(+zB3`4t{zh=^UTWZisuq2IPg>tY>S0r+6u;y{mxdxVE09{O{-T+t?@h+$Xr z#7fGK!4gI2W0<+YDy!i*{(yvMqNl$O zJNkuw@J?m88K&)I67-XUg9D@p^wh4Q-zxWr?VnL3paK|R@cD|J+4SH9fVkWNQF$M| z6hO4m>|VHVftiD&W4wd)@DRhbg4pPvaw1wetp;5;hoE4RUB2=h;6{m{5_T8F@$N#m z-xjdj&hur~6Vf*oz}92l_wdrv`2HE2BoD;)3#9oY@$1@v0bvWSCeS;707~#7m+j!PCLlNg$ad+j~@a|zh@_tJEj&;hbwB@vESDm>SAO> z2X^;&wQNHXrSI@06Z^ybNzUuf*-df~P=PFV2f)ne0S`?;!FdDha^!(8yS;Weh&fdM z%ubHErB4^5KSDwr&MtwNLJAyaEGadACdH#(>X^2%x~CI1{I+1-&`*7MO`TSs6%Vf~ zefu_+lot*G@u^dB?(V3vc=WH`Dg6JLk0svHV05|mMPP9nkc;h(Q?MmOpLvQM5VS%f zBGLdvHZe6dKz;N~)5nj2=MC@Q*LZ9W=kM*~!z?JMDU;lK`+6kNUnl;E9U#j;LBG8l z>?hVi559S1WMtt6a%?$saEvkq#H4kJrk#`r0Uo6t9wlGKYd8!@q@{!0Axx0>4A<5E zHE3X)&;`DCv2J_gXJ^}Dy$}+V?R>91Ks4_QqDMt84?;}9Btdcq6G;4%`{+GlS4~_h zwCByJc$H=n(bUqS3`kyZ+XSplGT^H2J`3gjGmNXLzLbuhJC&D&g>{09&GF3c zk@CU2pW|l_e$6N)0Zz+m8D;amf5toq)(E&CEVQ&D zJcCwVL-NQVphO57e_vHZYesc@j^1_ZYc8r_o?|D44%{J%rYcz*)U9$(c9mjSKaq+* z)DV97V~3FcGd=P#8@YNk<6%~1>QoIcFe2aV&JMNuw;K#|V*zdoPp87>W_zmIg@?7% z({&CcsiSnVq^=ET+}RB_j7S=iHxCO8q^)D)1dF?}_)+|SZeutJZ`9)*6W5hb7fYw2 zoC`pU5Q0ZP^)ciSR^S-#NWtay3Q7r@^Ba+ZVP`A^W1zE<0wuKw!6h+o?48dV?b5LC z10C5c*ix0n$6&rmP7#DVSKkdBuM7grTLAVNpvK_Bg)+~-b{5i;9Qj57ZR)7nU%!G_ z`tpM}9hb|Nl`Xhgz%+Gp)I>BBrP1pMuQD`PewW)!5B@6Eq`97!x(OiT1fakJZ>n~I zkr(TCi0N`MYPiJ8Due9vK*4qmgbAXT$(?{JfeUMP_omR<(QM2g|q8 z9d#4debnfUwP`~1Ml|E{d#3w&E{%HTtoO#B7J>N<#3%yrNj-M#7zM9Y4EU%4YB~WJ z`?FyDwc0Dpz<|KOzLMP4j0GGZGN%B+--XNpAOktg3jlR}^Ge*^Om&7jD6)ew5=qJ9 zSODWSvEiG8B~b6@<7X}f>XA8tQJT;!Agp_U{0_uw7VM)(S!q zX7G0iwX9+3nkSKAabLf-_1LoBUq>T%oW&G8$Y%#j^1;2!X28721pE+yl>0WTVnaPs zv*c^V2%PvD(9e)G2!O`jU|o{Il3lI<+^j+X^(FzQvvd=ZkqU&Ue^UV{4r_o-E*cqYT-%)NPD;&H~Jb-IY$r62y$N3(w`K*J-a=S zWgZ5Y@fk4j*hpEQ>5xKVJAg>1v*5VI&MvS%+K&72%tGz74aNu7@(%0!$MEpJ3hytU zYGpEEn;@!giqBeHq2`Cq;g!qT{;CXdse1Xw zu%CVPYU*oH!91Q$nYKd=^g$Dcs|3|G^*u_z|pZU@LcxLnh2?>cjrf>^OUpm@80qrC>pCW&(vP$Fl+NyqTZ;Zae$ zR>w(@A4S3o|1@gyc#f@^D$7FaKuL^w(Rn)dsSWroPS+m4(n-KGNDCPXGShY-GwP~> zp8;Si;Xl)tpFG(boxT5acP{P)Yj9On6%uZMRP}Dz33=oo<6j4wNoh3c6r|7SaH?15 zGXoA;Js^~k0U&2;UMmqmg}1{@D6spX4x6`Zec@lHsp|8q{AXxH=7Tix&6|6D+uuKY zg8b~Gr=-BXSPH(u>%gl>2Q8oonBX8(k`9_H)8HP{1`of|{8eOhfLok#)@@H7_%!7p zg^&(Jg;J%Svi+SoaB;j0;I}l0?hM-T5Xbv-^t`kk#<1ZbGl7WW1RIBS;0B~ZmgfZ5 z-vzWfYqycZ6U>Q%blb!pS2fuxI3%oM2OgcV2 zON8TQH^bq&u9&LwZTb&D;S!EXA;83MGFuCrVF|UPTB$6Y9OKF!-YO#bRRZ|as!=w{ zQ2QAbs>Xr7izQG-=DKGkWqvI%?qv5uSF*+l=8LTp!!*-2DQ;yxrR%{)NdK)Vqx^8I zic)QvqTq$>aJ}z-vOX8yNY7hyg9mn3;l_X>xjy)1Gj=i`4bszNoet+xV>(%qT zs^fD$pZjy)*Lc6**LD3$rzkk_^6n`v*|#r=p!4)4#2A;KA*067{f>TSAwe{#4K{+} z56IgrBv)gGX)sugP{h_v~y5)XYB2-Ae--6mBZ$H0$fo)a?6Z`wB z#0&!pV`skEQyOMO7``4JlZ@-@*Lv& zq%QMfSv1iIK9)pnQKATGLlz|HTeBaa9fVfkQQ{A_ocQf-PbV6M*?GNY@Txe2kb^{{ zqVi2P&%4Z=oEh11s%%M{uM~wC85t#s(7N1{F~5Id^7n>rcNN=bKtT;kuV8spR~LRb zqr5blk{M>fs&0fl(y-yjJoRVov;a>E4(Tw%dEkL30;P)+~obLMdCq2ZyKr(bUOzG!-qaGQfuCpv6 zYYk>@kCh12@p2XM;nx*rAeH^JdE#Yg?-l`>5~n1xU@-m*;sh{hY%ozC$}<`vAWcb0FOq^XhBnwC zK@kzD5I!=)zk(VqfYiM5tR%#7$i*>?!kD!-=)g27Y}kJ(ATS`{Jxo!DvX7gc@Xlds zG9DUy2`z#jfw^N6_28fF{RrvBa`Hk8fd~K?8XUwx`JWTC$*_fkriPP0Sb29ho%@tT zbrs4qQPVXe2zcmho4E~6iD&UJiKEB*xS#MqZQa3#`YJ?KHc(-L005Xs)(pVXaBnL_ zEs<9(uLmF#UkX#di(uF&N?Au>o+#;Ge=#pcK<^3F`}gBFAp;-lG|5nfT8y49C{JXR zJTDi~Xb_^4i1J84hH%jP#ReuIxI&@K7|cHegPaJJ`iGySj7&^es5cwyEiFc!1RzW&cv71y z08@_O9F!vS^J_xkZC(#F69dw1%(=kdX7)~p-z6Pf^uPsN|$m48Z}&paW*P)a(-}i<-zt^j^fl?V&7=(yU!#wCxZ`eVp~z*`h9;c ziH`?hOkBv_Q>$&|n+dO9zrMG0kc>4Xp@QBw!sF@h%|vI zYyj&hK%X%c_^Rr5a;m*6442sRuJbMc#Pk;I1FuL{@ zPSsgSV;I_-)pdvfdy_evor^2GT|YuDX?AX|G=QA(Uw#B&!Rye=!Q0zApn1hG9!3ko zPCD6906Pt;02%?mNqt51PWXmQ0|OBzLOk~<4U0(edYkQj$)$|9PmF`MXo3b?x*}9m z;Ds>&@|xT}0o7o$?c`-oclR+*w8rB&2&-U8rZE7o=s7w%cEZ$m2_arybwFlq+yjVc z!kyR{O(g<>&Cc&odO?C?MMHypKuOV)O1;P&aKfYq4>o}_^3gHSAF2roi1*0X-WeSr;P!;4B&n)N7yBOPq4Lca11RTqPxH(Q`x?*pE$o^ zA=%+CR1n?rEsEm&=g*&Qz`&V6K`gor8^{D;bgX=vwzX_wLkIM7d;aX%!`Z=VIC1YF zjbe}_7o7G0r7`#2&wr+AjR4WjfM#=`T!Ap~H%r!>PT%oql=MmhdEUHlV;Jfd|EPT|Nd$9HTb zJPns}*RD8p$nWy2np(U4P>1}O&S~TfyG{|xvfjc^!ng}agKKTepcWM+jmSqpi>AP+ zC=SUYnA8avdo1`26R^a{szbPQ7UAlESE|7oT7VXb$9bGK5fk+0< zFiMtSl81ocaiSOl7I~`o45-g1t)m5k}WS2yb2nC2j#s>H_ zE#8cwZBB`Ye0LvEw&Om+L{(r|YUHP+5U_FrWZ(} zrHs66=+{<{GUJR$qPzx_QvbRU>Ah7zvVog_-k^VK|8e5cCW*QJuWtl`06FA z!EeVP!MPOxW)QN}bCT%z02a?638kqhOlP`)ZX%`yXN9x*XTk$#-Apk?kNtYrm*V8n z2lf|!aPGojE{`kxe3%#*9s<#JlMeYlh)!*RS2O9zHjt8)#lyrT8yKLQkdR;me>2vZ zmjWMHfFUO6C+xlu=t}QDnaaK-6hZ1XpOK(^FHYw70h_O-1jvg|>qvx;UWL_jIDb2~jQO{P;jQOkL;t3d3ZQ>dBZ+N)T4>bK8?pxP+f`^ZW zu?k{b#CHKppzF3`aOgO3>&XR0g{R-G^<>!;bxB_I7de^%e^7nNIh(J z9h1xn5KNt$XwMQuJui9`T}XE~EaDkag)8E*fVDL2E`a>k`_-!!NbfJ)BJRAakw#%u zrB1o|^j*4%1MuJUrW6TvLRT_OwIjf|IptfoUi2tRRf1IqnkNd)Z?``Y{U3Z?JcL!i zKS66R!*1U5=Ov&o4!q#Je~7s44;w9ibM}QA>Leeye=~^!M_vUMcHNTO5d5kjdg#AdVa!%4H()S2Ip`RR;|3-9*|g|e(-$55&DY6l;LCY$#HFk*n#1xh>RQ_8>LC3#FI#y)BvkpV;t^-wcuAkaQq`4NgBS)e@(A$yJ+?*MRN0AnX#1mtn~w zbYw)R>Cn*7C~)s;@P38z=>fM2Xo0*Y&!SHk+^)K*scEQs4>Llpfg+Ft6f+1iE`$I= z8x>7T?MIK%kri4Qd<(mf6JLlDF4PGZ8cc*Ti~<_gf>s}jjL{q?V1P{vxmN#;x`#j+ znl=+LJs7(<>L)NsObkDKxC+(yw*ms+y~6;OWdtlGASo~jt_|=uV{2n%bTOT7@>@Cw zn#nt@CcLja4)kgsy^wDSW8jOv4?BT1k-`I_2^euJbb#RjLk^&3KnolbF1;-oUH1)J zNQR1|+fvl!VIX`ON*bW|Hv-HO_#2?h&xm<1LLy?3xItuV#UBvIQsb_Vzqv$2%){I$ zBWT?R$j-aXx#+EjaF?qP-U}|Wl*msZQUwGT8J!%>L`Obdre$7u69pRREdW}t0KK8w zPzDIbCwN60U?O8+t&*3(lua>80u#+PmnWy*jio_{@IxKUfxeMdFb91uUNW%o@fEhT zv8DG90^gwqj1E8qF(4#ihLF8wIE8iBt|O_!5lM(vaR8daj79#sXc0YtklTQ$W~6XY z!a%nx#DMVIPnud28GH=gm{%+3+@->mLjC~T_|I+W)GF$mZVAP{UMgj&Nkqv9Eo~Z% z*cwAs-YRs`C@(a#zpgZllme6 zzYl(!2<{G?jq)U6|LQ*DnLbLk3;(2c4>SA2aW$F#@rfV4ylib!U<2X!MCkzV8!#bG zI$3>yS$fWM-cao3y?gILZlc`UPDiTlnpuNFZm={blg~3!bpHV(z&_&y?S9j~pLU`U zQT*R0k0zEGE~GW`t$ta|Dsdc80g9t&44c9Y?LiUwhIlc6A{ZrsjSZ;Dm#PfawyI_f zh)4YljorN}`tbzyFVLOByTxv*Hw}PqfJE{;)62Hu9MGkh{&-ZVtyy3mxw$drZc-nS z?M_7mV-yPa=hA@1JdjT_lE9EqU^P^8Pd$UVgi#!v5<@beaslyx4meJ90!Xb02b~*c zciP8&OFlw#^~X#J{Cj7)dwE;B-Z6w8!=)-5qHAWA45p|g-{5CshG>z1{1w|yN+r(p6X zTfVM?7M%*`j83`evzLD$(T7eKq4VnGH)XB<=Y~&)I(xB!9Ezo4p-z)}o3IU?4BcPgfH%ewZ z{lJn?)p=(Wmw@&*^w~@2v;H-LFf52tY?i0{&n|7ScQ1vRq&K;5&e1#np|1G+t*r2D zcHG}5-0uAkbr@=XdC>lk6;x9on?NsEy&)3fIS8*pj2Znte6I+dpPmT&-I?d9e zXu0Tu??lQT_5&DO8xt+HUXq$!V4{SR{k&{>)VgZuN+|!aN7sETsKEGqpS&$x0UjRV zYLYkzy)3`FLfh%NP1!LyBZJw`sVHIzm>wOj{ zb#DigoB{y`1c~@3k%zopmd}4Y7E@+O;j)}0YZFej)lZM$Js>t9U4@sTi`}3-;-$@@ z&X`0zhk-Y5i@wurH8-+99P@d{4aEGcw^{3pHPXKd!Cpr$slKcTggVTxA94=0+mXWwPKViCcy zDO%#mPMtX4Jzqois{&B3IG~WZWSdKGjOvB*T10^$2l7Yu?Z}$Mm}cpx02OEV21%598M7;{AV`lgSOfN6jwz7e9>IAeL_!$&E?W6)8VY>uM zjRm18EkW0Dz-H#a&%B9|D^vTiXW>@#VT&T=M;@r-bR36*R>4(*YK&77fP)a!2+J4d z3j?m23<3_sV2&h)n`iU!Y(Kl-?$WLQj!~x)7*u@_wE=M(5>ll=>mkiTs{}YoJzW5^ z2`;rI8e!r2H9tIpLW#^XEa*}yvOKnv%?S|sW0ix*y~M-hYt|)c z8x?XWHvq#J-bNr)4ABAocuXO-3-jmdK6KbP6R}h-R;>)6(gK`S2|U^w zT_NLWYg+itWOU0hV+cuteTN<FF}^NN@60}R0Wf%XM{eSQ4|6%`iR?6G#0?8bu$ zln@p4W|b~==l5T7Zjs08*X-pK-8FHtOACy70|dsvE7OD@pXU)7GaK5WX@C~bs2X=35FMu}nv z#KQrbi?o17VTLXzdMgTwpbcUh zsYYxEfN>VdkZ+*fG+DN0UK~e^Vie+b?L`O8EuomM;O<0OFW#Uv8p@VJ(c?w3pdr$g zQL|kJflV-{9tSAM060V<)XA&CT}8Gjz$2u?2=);GMgtgnKrDrDW$5LHMZKy(kd@2F zh+Y&4kC<%qHf|Mj0p;pz!F@^k8dZX0d+TXq;$JbO@n?>`B0k)~9wz1%k=*QitAg9Q zUnS@y=o@(=RK&eZrnv>!7r`-@gMmX|0@{Wh{es6$G zhlM(46v@i&WWo>s7w6&j*zNHcb(i{H%_9ngkyakNA>Zo&JXp4B7#cy6WmlD12fYGS z0XIfGZCr1>!1t>SU0mJ4@SxD>WTu614D9gd`<``B^bJGZlfpw#Rn^qg;Go60DGc3u z6ANMJ2!fbVxCJJ@`XUSti3N#3&=?JL{6R`#h>D?+{6$a!5qUGnr2?T%_fJb;R8(M% zyn9*!Ujb?MUq-W3mlvIK! zH#;XM4*(pHE9!y}84nKyU3OIm+d)u`hoq!^ZvVc0ESE3mw3`w4$Hz67-TYfGkWhoS zHi>R_ciZrnD82hoOqReSYbX(ACZB2i5j;1fwRs>@MDQypmYcu{Vc>kR2nw!q`cn%! zWU5=kBpwvyS&1OH8kp$FnGawH7pD63KyL@!$zzm70@PAHWHpKcTF|EC88tKYI1$oT zK_ciwqt`yJ81Z>+a(mTGmTPwQRvCj3ua5s0k@ptlZUjXO)#9Muv6bL4&z3wEjfEee zp9??gOm;uW^g7G`0W6c?}Q`7KqLu4c)$Si zIkX8t<_l=mYcw$+?RFN@>djv*12YZYMl5SsPDVxrx@Xp`P7oK)gEZ55CmhkUVi5o7wV z7nrs7#k{9@U6)oyWJIxiGJ-HK6Jj_xHC+IU^z(Ag5rhZfa5x))h)h7she%JFBVIjO zQj8FD_PJ?g-jIPd$KE|v7Y`#Ss5w!#8Hp9|uR&6PXb=S6Sd=aV7YjSi-k01}p&wEm z+QFzxj4Gz6xn~J)_(QF6ac&mH&bIIy;u6_vEQ-p|qz(YW5j2GlLuJnD5M9;R^>#3 zi{yt|&@eHYAejy}I9NMB7LU1*!M`b^$ii*i!58NIti=*7x9Cgk?9Q(2 zy=b^bQ(CfV5X#k)cgHYOcXZRc=*pzAe)d=`oELB~GzjB^$RQ*;I=aWQhJl9%zhMj0 z#b2&P7EQG1lk!E%!KD+glTjz=L@@!&*WKn}0sSfS1^zJp>%WzQ=%_~R*z&X5iPCRd zWpQ2t!t%Dy#CuaX3KKrf=H)V zyrGFJAzg>mM_#;qnH)%{u%D)?Kt<#`Y|XZ2gsl`SxLw;J>KK1N>EU48xxATaaet?^ zqKSpC6c)@}%($pKJTCwxDw+O#2nm3$n0I3!SJQ?)R|tTM%U5wP!zjnE@1HT6v}eYy z5rE6iGwaH4-+=KDK0@0x$a3)SuhC#kWE|@qS~8Zeq4RgjNGzSd*G5S(csQFyqweQ+ zHG_4=5m0V{LL?5@ykD=XRz82Z1u*a6qpQ%d4fw8Qg9$#WFm#@n9b8N9Nvh|@OEWcZ)al68p;)p_g=qx<>@(mWaj2j_3cA@C& zK!`h*RYk{^?GNdFzG)}}qTkdc{v=-8Kw)?HGNd$kApRTM2;CmP=Chq2yHa zrxQZ<96SXJEFyd%j7^xdZio~pz?Bnu9^$%@kjRDnhs(P5a4&LMb?3snjvo%V{kS3p zNYXrm+LH)T1{0HaxZ*V^1o&-6Vvt<}cToxj661c`Mud1x%w1rp7Cw3(@aDG2ePx_X zp0WD+7a9FWQko>mTemQcGoX)0YF)@arXodmZUg;K<>Q-I=ZY*UT{|sZyd+AcpK>_y zOsb+VBk*w2uBjw`-ow~<>19um{YokSJGaI-H`zH8?n#{Gto?Eqd~}C%^f12g-;(IL zWGdnXj9>VcTeR%e6N!1tcV1}BzMxcVDX(1K^KN_K;mX+UT4N96ufT)}6{#Mw<7mVQ zNh5P;0nrg=O2d^WW{4#3rFq2-qU?bL-x6|A&-i=TCAzsk+&rVB$_9;k9439QFp2G! z)5lOj_R=uPf@_+VCvn-S>Tlx-g$9DY*zD|>xi0F{0V9DJ!MZq=&ur%u& zvfED>n&eE;8Zf`YRH0|tPD^|M5(f#OydW`FyEpfB^e6cjlrKv)$YylkA^!Wvzd4!{ zLRw4X;rC!uiJ|}|zjta*xQ<$Qm)XSb`JWSH!`|)qa5;5+-Bc)l?2_yB+yT0U2YF$m z6)D0k`^CdRy9XavWhi_*azMOnf zDi&L>z?rZe5e1o5WzkY)-FxQufNlZP=h9o}^uGD{to1VNvY}7gA~zBhOE0b?ENwwr zvDSk{RzZQ+a<+xFs+9#V`;13$u{Xnb=|-ntKvXO?CrN1K^U=iwl`PJzc*IxhfHGZFN{iVzyQO<;I%(>Twdk!Oh zfj*YD%HE-c)9BGlT_0y(^7|_1Qnwo=h*AUtRJW$qofPf2t}0m*xHJ_$KY0_F-VfYW(kt^3AdcCy0pD`H7 z4o4qn&?`vUc}r}h=i85NE-uaKmQn5Y2EfMXhgNZ$c4PvJ^HlSQ($d!+X{+Nie&b6RPZM!6iInRnmBcU7(BaaW!$2i7?{l3(@_<07hEvpZI%P~2$ zNaRw-I(PP*lkBv!3I7FDWP)=rE3nwT=NmZ{T=mzCMQisM)AocNq$ghA<$E`+HP1}d zOD;$PKh@un9lIjg6COAzMvV#E-4di7?pEcV?mXHrjb73%NC*gH&rlw=6tRFzt+T}} z?owpcL+j5E41O76B{!IiPqe;R8z64~A5S6d$w+rs&2f*KP|ToJ(Xv=@kxq-#0>i1c zYxr1y791}(S`MkOWte+W{NP(-JBPk)b0IHXE0U=0sGbaq?oS%#bk}tkShKjo zL}d9%DBggUv3!!Buj5u?{MNQyQAtRd{|BRj1EepIy5X{VrX`d%Vq&;E_` zma>yfQ-Uu?-tJC$q35^t{#dPqLpQ5x!2Wz1OjD#pZ9~7IGFSUn)qJa+#)x$H-z6)(g8|Hr<@ETC9pfoH z;?tuyQf(2Xf?|rB`^Ak{gPLuxMIQRsMas)J)d$L`lLRj}=vebdIQ3^!+Tc6XJKi0( z+wIUnpEK=#h{tWh0fo>bKOWY7u=1>b$zS#L{`PJy0c#iAO2G8 zDgO!OsQEo{ZkA7d7Bss}NXh>yU^(q_j0|k>|F3X8_I0hzs4uO>NX|yIV@re2Re8=_ z_iK(Uj#5&|?ao)B+Y$QJDt$|`QK)uG@t)>;i46%V@MB$hS?WQET%Z7*C{pe{7xy5B|a_wag*a7EH zxlo~-MGj-`bNOHQ?tQSgvaK>XywkE|$Bpr*%enc{>1{Z?`Q=*+Lps$5bX7Aiw-eg# zZ`UFytf-Ca#GY>zj0Hz+Uw8NFd`qSLC_VeuU45=){+$LanYSP4CW@!gBupJ~8&)jO z?g$ENw>>M3AsPdPo36P7=VX^HpU8wxYQENc&14%L|AEPJai3U_=*}KJDOD@k?8-LT z{I!BwXsnI9Tzr6&mNV&T!5Hx=Rd^uIsHSEjY)ni!x0oEjw}z{iB!koff&|rrIoF9F zx;=QK^GcEp&KA<5`ioyG5;Lt>@=?*&ODNvS2wQcMS)n>)T~tw7<|EWN$y6%A4-+w0gS~ z9vdU59;Lh7zi(sBeQOcZb|^Us4993U#@bBEb8Pp}mkg$q>W$9e{n;88hPkl)d-~DK z@5GxK>hByRy3>WHolF<>HC^xINo+)Xv*u1~Z!?g4+b=u=mctC8?PIBQv8n#vIQaW96h%V3R z%mR1Z(&0#2%lzH9>j7XC3L=1vFW>-h2<(+;I4%nPsdnVy}n{b!PvFR4yVK1Ri3KP3qlcDLpH!0xb(?nzJPRDtw%n@7_y9hC#~?FjIF^5J_7N#zR!%HFIY=4*vWIER zJ5jy;T`E;IKg)0vEtgd@V@~o;K}t}=?+bskA0iY^MK{0K@ev%GAmq1cmG&y7kuYs1 zCKxG%tMJOTu66Y;VI=m6kNQSgV03!yAhl5 zTCbg#_VM)n!3QH{6ic8RERiVHGZiWA98lf2y07#2j&+h&atiRgaQeNiBtdWtC4D(d zGWdh(`f$UaxeuxJUj2}4T1DHf@+iVcO^${-C;toApO(=AYIcE?Z_ zmodA0mH*xi$mkTfxcRsV&7?@ULM-W&R%Mc!JxcbHTB;l}v(17VQr#mxf~wDeS|t8j z#tvvy*)FNu6G8^c9a=2RY*Xl_=06al6XFM${in!XGZfh*ScG_c;x>QWBsYe=TGF-@ zYK z*r)DecN3!@+>oV>&Mf$|u{7tbxGon~EcoYa-c@GZev5Pfx%8M#8P|Mep3otBiq?3b7!RDu}}8<8FL5=_YOGr2Z;PHEoJJK#)Y#0 z`Eyh8x=db-7#i2==2eMWY){E2=$Z+?TDTq41xuGJd{&p;>782$&s&;XqLlI3cGB>* z`g}&ceDe7r^~$h#M2ql|-j1d#wz#}VtGdIz*Xj#+@pXfSoWSizP?Zrk_l&x|@eC4t z=az<-f)dS6vdS{d^fs6id+x%eV*c$?9`Vsfw&EddW#-|IO&f?Xc)*P7w2p95zuh*D*_|%;bIjL%UYU^ZnNp;0b$X{e zzWV-;^*>Cd)GD|~wX+5UdPXB@R1X&@5PzWMw)D~k{5u}~K*L{!CdVEh3+pv00!dK* zP8`pilCk8Pc3w>+Hg)|13Xr%H=T4CcTEzHP8^4>_dQ;}`e!VKXOWxS#P1{cu#@*YD zzgc|JsM~(vevM7r*q`{OSG|{8&SIbY?igQ zNq6jK%PFWl_}Y5fNxzwzZXvN>mIsUs&tE2Yc4JaCFq>3s6SrNnh7awUO)7d^rj)MV zeBWAp8ohHXA>v$5k%G4=6V%wX)&=IX2hQefbyuBc$Sb=xb~W`!J`0Zxkd3>m^(UZ6 zjst^dqrdT;pQWiA5fW?q)O3Y$uZyyqTN%<-qauD#P^qIzf2db60RqX2KU4_?pm#C= zeOqX(@tW7i`O_`5+G-N&qAaj-k7!{>Ueq|Bu&+SMS3ezLJ ziW2TU%;$JzZK=4~9!~y+HE=bjuPi3AaiV#{Tso_5 zlTF3MBe^;)KDv|^wGgZ>$y>s<@5CJcpLiwRdB=TvRbN!xzOwE_HllgK!!^x0eL^W9 zREK0#W$J7xevW+`&z#l)h#L>X0(tIz`KuHd?tpk`ZKVveU5NVUsnR7ZXc(>}A(Tkr z_y*^SI}z=SuxyWWgG_w1+9x`&`BTP6cFRF`oEPscmNh{Ft*4ZF>DiST{y?va9Ye1Y z+`Xw{>eNbpLKUeI~F-|AxmQgI5O z8^UB4jv*;02@lv)W&t`nta6f{X70m&MGMfBf`}DEMtx;FU>B7|ItK2=uB;mCp+Xb6 zisWGqXSt2=rGiTuWH|$ujruRLa1t8A`p+*!pVhGmcP+#oRQo~vRuuswJ|naq00pf^ z_vpPXl7H{GSM|EMxM%PMjrY(eOc!Qxmx9iOsp2BAWgt>k$6AAA9*TY}jhhpFp^ynF z^niLD4N})tgh7ojt7rus6ktJE?X-Y?22JqoC<`K}UGtq=wyv9BS-?4hg$KV-o0qsSKsP3<#p{=;+WacPmAzyF%+>W6=D9W`otr6dnp_I^}Ukn2i?* zLo+9}(HbaduB5w^E)V!gxows!if>^ZrGq!l$UHi0VC&R1*Xp^56jz_&GQ^ zlMH~i5x}tm z35A+DxB{hvs>Xm=8MSl)1~Lyw=z9&3AoT`;OyEKnav&Z`0#UZwH2vBL;g^)#VR&7qZAmziw!25E|ke7f-~e>$S?!Irq5Uo!fJJA;5=2%RR) z?Ro2{F##alsc}Vj$0!K}$d7m}B8;J_96up7he6W%*LRleQczl>0v}*zhZOrjDtm-Z z5Oaif`5Ede;Zc({H7MM(SZRiKcI9I8ylyjWHL1?KMy+@#B&{!0qMAKjM;-KRI};QQ z%4c&7@w`C$UToB>n@7y^s~*2@L8N=QZ?(&fR!Khp$=6%43%FQzrB=z}`3Z+TyPV5%qIVmNnBh|S3PhSlcAHEs z{iF}*jJw-zgseuYu5+fF{Mj7%H}5UN1F5c)Ed2jo$xb?{Tsy{quZPY&@H!bJp#=wr zW%*|_XoQC}2+-eSq8D1W2oURhp?(F|+>FmNQBTt;Rzocq5W5E_TXm8by{VY58if`K z0Hb`!p3Ol^ByLZab}emx$2++BOx@KHdZn+I8k`EM66oZ){1vKqJgJH#wOd~d0;54c zY0nP&NcEq(irR0lu082s|G)AJ6)cF|iptGF?L%a!flQbra0@KR?!x%26H(s+ zkgpl9FV9e2e(-8Z3BEHdF4}aRdpUW*suv)VNH8 z!)Hj*S`@IW$zIB1^S1|eX7Nc+3*51`_Id=vLcHlb_F zU7B=}ewIa+27k-g7`sM*g5kfH1Z=yKt4@GE3~of`IrWuSN^bmYz@cAndu$=K&7mJ9 zwj(AMC&K)XI&Rt00)kLo-{rnI-P-ctO~)n?Telx8CF+^Crv2nqOcpM4$r-)B6nTxm ziJ)r0JIUfWWBX$&KLEs_@RXph*GO$IzP{w{S}rjs{=xl2Z{ciYSWn5_uzhS6f%%Sc zjh|%qRfRqn8YCiAF{cXCwPm~cvw&!qdGsn#Tkz(vM3e>i;ji2j`IBTfQ=K~F94jbJ zBoDZUCyd`#6V3X2ryVW!mAw0!ihen53Q{%skiNoosh6dx*=lyY`+k7IoOuP>TFsU- zD*Y5swmdNP;0gQZiY;wfka5Hj#SCMYdLKl^@?jo5_j~v5om4H!+RT4;S!VmHWh{q9 zzE6Z^^;4W1z_#-b8h9yaX>>+U|HC38ziJwoJG!^Q>Ao_bxlTNP6Fcj|?0S#dZo6FU zGk<%x)TA=UnmETeu-~P9ORizO4A9xZ+25-HhPo2z`Y${0h`4kHQd07`*9l@ktlF4# z%T6sgLJPcpHJUZ>?`eb1e*?U!i`^gG+VG zX%WmW+_n!!UGKdk^}9YlnTx40fl68$zb7owAP)%opx2-O?%gSTx=vRBQ6poVuTx!Y zL!c}$R-fhYUnOHVZw#bU-VKR1FxFeVk#XgMXDEK6EpBI8q@~78F_> z#p7J4pjbMD0}o|oYWL}zx${glKHbm(r<&%6xciv>t22%MRC!i`O4iCkV+eZnEI2(` zR(h7TW2Of}rW5Oq*!;Z-CcUpC_ACxQVc3Z+mO&M~uvlNbd5EeF%w*0L0L9JyO85mG42NVpysU1>+0=czrkmse%PoWzXtqYv0X?sS0{^4f* z3f)4{CtIGpHkC;Aa4iST2hG}}&2IwOrW(TZVr(j8M%PWoYoDL}@SF(xel)y4Y)E*s zEP+myMYoQ{f|JiL-=A=*YO7RJ9mN7TKIoEShk|ZRlR+6|XSRV4Dmc$G@6q(U`wc2U zP_s8w9s#mrT?nDP#m4ek`~3VVgJlulefJCzBw)l{-AA1Lo%-6#R@K|wtNZ#bl)32b?mW$X_`znqz&kpnIg&!;Kncg0 zGqKmJyu7?jkQV_ph5a9B89tt7Fk~cbW=yp~iLT%n?U+3}rytm{s@f|!N`Fa0-$H7e z@dZfmuoB08Id^c+JH?QP+4jw@%{RY^794XI8J=>=ugyWg_3`iOL&K6Ly1{Ct452F& zB7KD$@#2)K>qPms2bI}Y^+u6>Wof4~ZcgnrP?=-TYxNX7^gLuC#qO3d>W{>6Mmx3= z3Lp%Q=DPTGLAC zE0FTK60+$+#u;e-?#yY=B`%_CFphx2)=@y*Mjcu${Z~;Wt4J$8v)Rjfx+=kc|E&XV z^Ql#KLHp~2p4{G38M1@ZW!Ck-s-W5&dNd)|F`-D````oJhv_}1xhdAGxf95yq-&!B ztS$={JZ}w3ca`QZ@DGe+WvAiTu&54gaNI7vpRTDUB6^r(Y7=@1Rca!Mh(@e?9ebW% zBvwna=bkGpg@h^H9=$Uu= zfif$7&}wn+@*bnM?Dt)oy6if0MJK3qrUfr*7db`+eo)lpRqo7JsA<@=b~k*DaMOXz zCtZn+3`E!ot{O#Fr$!mG3wdAk?Nie~{8c>J#Hoi)`EI0DDIN)SbbZOiWkt0EQ6G*}#13<~%C$vVr-7@IvBT!t zo+CeO9i zUp%9M&IhCe-fKyq!Yz|Q1GZU0)rnl^PzgR>?DK$(s59!f_l7_D zM7l^aKgRdTbjTwACWtqO_-X2VMo;oAfITn;!Uc z8&%Zi6U#c5wfJn7vFoH{v=DDm2Hc`}23;DIQdXKb zSPng$vGB=TE~vbj|NN@hJh19TxlIW{5r3PPTZ9OgC`t*9Xp}G0B{%p4RNR&4aG>3x z-7>-pZ4(nsBYrBH{M~y4=>{q?J+CCp9t%Am&*(3$b@D{d$u89*^RN>gUv}0Z>{8l8FuS?`X6Nzml!FG+Y*BAp%NVQYyVa)t7+tlGf7hCSV=s)Nxdn82 z#(8(u2X2XSe?}+_WPT+*^jA(vOs6^ki*F=vTvHJ^Nuu(XdqFIhsZdegm#)MJNpW3I zKSSE2#}%w5eF4EjqZ)_)3fb?*97$^^&7dN@<1Z-?n)qSr>u7oTMq8YlVyO=?-Jm;a z4gtdJTv<(<5S!(BdEV2wge7UAsNnUdrZ4xCh^m67BA>1YyH9}fA(VpESZ=ITw~$D9 ztVZF9#UoHs%XM3mMwH@HpBcvVkljgZfd;O3<>h_FaN6&vgSp3mx3aw3nJEO42BIU1!uFj*7-%W&S%Sy z_z3B#j8GBL|9bt~WQo~>29(`g25V;nyp)ZO;|PoQ{{SICt?|2Ti=qO#~E*#u!sL?3hakx6!9>-zG3`r5c|Q$&u%Iz?BRWsa`N20J@z+mHhGhp zo8dc+eabu4$~(Q&)O^&^u2;5_vYP*R^5j%oXHm;O$_uPD_dG1EsljOg) zJNK-{7iDRxrxsqH3M@IEzy3@}zOhcC_d>ztedK&(Vyh#Hqbnu}tKQSLo0~XEdntYS zocBP+2Rs3MXsNS=PVa~5Otqgp_f(#v=0(Rn%120UT^;ZGS#j}IE6I(u7_qL5g~zKL zZMLVQ9#X!4*uiw?&s4WHOVys+X0`{blkWMMqs>VI>C`+L{6m~diwe!P ze~|nW>H?omQa)sp1nzoaJr%D!!%gPz+gi-g`5gf2WjDJ1)=MrA^mH~CVCu?;$3)v}_m7KJ6lqz_{ z<2sdeipG>&6&G)|&vvZUwKkmJ6yyI zoNapI6ouSIixLjHzBw*x%%$J=;FLHam+o%=OS=6GMFi0%)uVAN?I-YSVpj_)*fVO` z%Dd=S$4Ffl3SDR|CL7y=Cm2R8Otwy{dL50fO?vYE$?Z~Fm$~5F!STj<7Bkv4>QPLz zWU_-FW96|QqCYPlHkxzB2^>nulb14k`-UT$+uYOU8%^?^KLps|89tt6#4{ExC-~&W z-VBkWAV85c5WDOKr7bgSL%~Pn3US4@-#S<2@uJa<$JZVZ(X(JY}cc zS4*f}SXXb5Sm5V$*?3alwIoO;MIFy=ABO8TO6HrOUQNZuN)gyvQ?4kaSXmU3UOp5w zT9;dNpLA!R*+}+EzX|uOeGe@s$FI4I!-a5Px6ieGe8M*%EZ%l}=3xf@{kWSc)$6zg2fjB(>Kb2;w?U+K&I(+aL z``y}%NQbm~dgrp6su~Y#-4o7pzLBaBc=N6Qu&dOs5j`pM44%9_FAA}qtY+BG__43X z1`b>w?s&!HPnnL-zHONh$?EAW5^i76@ASO%xwGehy~x+scWc%J2pu_}gB>HHS}#g; zX%(FpT6eZzUNVaE^EWCltr~i{=6rn}>f!vQMOxzsboiPKb?lk;#-e({ym2R8Gq?43 z8Q+G}(N2$lF;H?p!J1%|&6Lx|{;;4`e5HwpQ8Lo4~n`mlN~s?O2tQ ziK;cHvC8$NDI#2NdV7RoqT;x2h`O-h;Opl)Nk=029z0Ol4Q#>P%9ebO~sg)tErt^Lx zuS+Y0Z?MtAZ`rs0l&k0SAdP<-8A=_!R0Q>m`$!!=-WtsVGWcCP2Jj9>5QoBZ=WF`3EA zIy-jlQ8TTs2LHkH$BMuRh!R}WXba8EsbFm5R1_UH9> zGBHgnx#39OA}*&QoBF>Km5q|(8Mue-fbpyh`lkctCNh1c{tlyyG{(7Gg*#?ma%kCx z)(~p7LJA?5oW%5vg2By9zsNSeF6A`FWi(JOkS;Ldh0E3Yo9m%@(oFE9syv?wukaX;IPfr;v+6L zfB!XlbpOkv|I8GxYadk|vHui7IXv=~HhO;flKuf-{QG2|<;X^HYcAseemM3)%Yk^b?^aw4({>Z*3Na^zR%Q3BbMQPA*AQOP8 z-^LexADAD>kF#hV?0?Ii`u+a@qwc+-=@@{}f1O=APO6+Sk7J zo?w>aJtj=sQg(3Rx+^vyJN70Nm-bTK7V@+d#a{JJgwne6owH~r17YwJm&By3%2je$ z&>c_7RHfvVr~LzrwV$eL){4I&EYm;R9hk;2?k~K~=NUlL^Qcc+cn)2htwtq%Pa4pi zWJhO^>$+NeDx@p2I2q%XDGL2UrYZ6uibwoyc5UyhJI?n0ir-(yW(pgNlbf0c+pgFD zCy~u{f4GD?)ZodRq}(J%+d&`W-8`0+Q>lHJPtGgMjHBoFZUTx@x4f%s#r4$=CJZUq zEz>PLO*+uF{_JH>pTdQN4qXElsa<(&(Vz9%i`TOaBb8~y65@is%7cUEY58P3lKo5t z^7Wgs@dQTG>;V3F^fxf3tCDjX2>XRSF0e>?qdAlY%1dcWpY7HPFdXix?_bRgUS#Ic z#`?oF!S2sBj|HZo{;F&r;i8_AY2kPCEXicaOxs(5_t964z<-wb;FJq&p&!_77f0=T zk*)h<(lrb{sG3RbPYQocYI^znmBaBzjZ1E|2}C0+GRKoXNkO43Q9O*jPLD>lp1Lwv zbw!a6Vp)%PeDhvMrny|kTGc*wCD{MGJuR)y%N*p7uvQ(k@?PCf?88vJux9Xj9br+y zqcylPv|KeB1g_C|u`XV(vX$*hpPnX|QzIVHTD~ILL6GCtZnTOsY2XP^ZSr9%^Gbvl|Peg~7z&)GP-LEZS<7@4jp z6=_m4Y0o5=?~`o2U#OQA!O|8KMGZ*Pj5`-23y2oxWFtULmX0WtD@AH#>(EPUdp?M9hP+H1#iBnwsd)08_;=QXSumuN_GtoEFZzj zyUtyyq$)@;SDd~>NXm>utc_NSA83xqpwUDU(u)SoBCZ5oJPqvv_OXQLJ^x({~zvVpjd}b_kpOR!JmC-66_NU0D?Zw(|bql-*n6@!%M zmoo~7PnW0)iTXIoeA702x8kQdqt0PKyPXoA-fELsGkBA;^K9uDeRbW7$x7Xg<4F|> z&f9uMIw?VIZBI`{WIr-DMbA(iWQW#n3(P*malDnS2mD4^ZpwZNbp@-N2Op6I$qR6+ zu3DH#;_c|xktA+wm!%P7Nq(!i$HIaW+mAoYAdxQ7`pvOvwAEnM_VIG-!NO?o#*MdE8?d^}!aBb!;&b!ZEUn zB&V|rk>FbX?R+49fzznv29MbcvjUD!A&^fI*XohNXspH(9lDD}MUr+&nb_O=)(%EW zN$xp>V@2+e>g`0dOuel4ptlB<^5C5#y0a>csUF0P$lB`m4kqT@AIb*@@*Z7DJ@xMH zE(#~8vGR*W3H9X-bZL}V)j3CDjOQlIn?jF&dDPsOW|cTxjfq2Iq0b~ewS&BzcA^TAxSs?SRbM)n5tcuP8;RvAe*sUy~j<(;qu5lVs-#C3ttrtgYBzMC$@6Q6?v7%+K)AsNCD@>D{)L{G8Y8tob zy3dUji1Bt}`*oG@-X*nlynvXDyDh7S`em4;f*O_1FPB=xmc>Z*iqg@(iS)rZLtgnt z?J+}c&YLP&wsE_g4^jqx{^kqw$>4`Emwl5*C+549yG+@HKE2Ovn~Kpu>lhNImvJrK zQ@tDZDMEBQ?rh(LP2GQ$^nhuM^SKGfijcyej9NXpqp)ZSYHPmh(`Q?U>y6+5^B+udXQtj*>U+vdVsL^XJIG3)Fy+CfJ z_fpzqN52VXEwh^xPBEv-nx?mG->erLym8KE8e@AUcK<3{?IN`+QMjV&zN) z&BGzBpX(g0RVVpZ^nEn09`U>`;FqKU;ofU}e1&mU{j|9h|Ac@G#3XRmJosGVds?JG zL@7l0w;MWswnIBXWY=*RcwU$e?Cu535Q6%d&g>njPbUTWW_x{)@(THi_=;l{RFUsU z$GIgl%Yy(e{vr_!tFS+B)5(`@;6ruj05t}vz%WuZ=z4JhQMDYf>>HM<(;7%=6N=>d zq9HcQQ&UgZ20_mx!Krg-C%lOt2c2P2jb62khufLuvsr{;1IMl7Wx%Cy75-uiW~r}* zdTw|Y6bO66=FG8ijZiNpOMo}jsERbp(;6oaXcL6EoGjn+_){|Gg%I7BEWK0Xb%Sd)+D zKQNI#-t0ExC+)Fb9)%=d6;$CYjgj8B+}|W%&DlzeyA=L0n`?-Sfb{WFR87xCQSV`G zgu*^Q>)jGBX_qWc{K4)nkmburK1Ea}AS$?Q4o<{yOKBW?Uvq@q`(OHz7oRd_nJnHpS~7nU&Wb`L)-4K9|VVn_Tc{IFpF=;5>HUf8m*@fRJq@ z9H8k$t&?qvVqO)erlV8yBxlh_ant8$S)e z#mdxeV4bjOtm45Ub%q#@J*WWFojQW-j07ULp5Oiq5Vnx_7B3Wc_VM9KmI?%NrNk<^ zYCkYyD!e`yTA6;k?zJgv6M?lDJtmA$?n>1JwWrpC!@M_k$4KU?BJDmr$mKuGS+%AJpWBHnsH*V+=yc zEhTaQ8=X7G3|lvOX=eoC2q$ubK$M+C>62vO4c|#=M~hm+83gsW*gKE`vV0IWl=@mw zGu?B#W~3%sA8;fw?H0<017ofhew9kal#z?i3M+o5FJ`6^(2SPs=b{`Offz0&>s3f+Y;Sb-!HI3UvZe`_?eXVnbc-(=18!97vgFTU z_~eV^Y5lKiA5MN1(G>{+rt98>?$}B-*aTqRrKy&0mG_Ui z8iY}3^;eV`sH0)+ll>tnGmu)@u!QEE#!eV>A=aU-Vr9l?r(!SDMG-aeeQ#{WR`zmc zMG-#n!Z*B|w1R%tf7{5cLE7eEY{S`g_JP5jdF45Ct;0P;W+&b2!*y3f`JCR>_+74# zfXWk&p8DAhLY&X$gpeqpoC90n0KU3kb5oDfvQ}7C(~JjkY}M*I{Q}B9#wh;f9Ax~Z?-x7g z{ug>8`kXDXUoJ{OppSr+4R2!v9Tk`o_tR-FLB7gnD0{A2VsKp`#Gnz4D!7w_(y__0A8VT7PxQ3YK8R9xW<`;_MJPa4IRmMx#XB&JlZ+x#FACo~4 zQP_^Y#Mc(ZJoxeYy{9eAc}}HlR!nS`CSxk@{yt72k*p#{P;HENb z#M?R;cs~-~Ra85Hj$HZbrCav+9^j8(j(55y)jPR8_GrO6ES@&(`TnGt$mfnt=!i~B z?7d%C(aMHZvL~MazY88I1K2$&7pKEa6n;UBnXg zgy?1-reKd8Orq!Y`Gts9-WA^$3)RO&*y9j!Y#VvhfD{$F=WJWt^g#43q0#$RAOMk9CHgzX{3M3_Mt=qb5Gf zdzypgR&1OC%dOmc%ru3U>i|b9!Ne8K_@0Jw!r>shlJQm@K`(d_3q)tAg88*%6!mluqYB->u_+bhNlgA9DCNz!JUdjgF-XP;_r1|Ww| zZPi@v`s|4hMocw_e6li^C!0;&0n+o&dRJ|4<)BYGmRg$W0(uFpH>$qQfO-vsptAT zTIcXjnCs3tUFjrdX(IaKdyWNWF)7BcORoKf{K?GWY@IPvO)rVUTca(-))gbP(~T~q zqGs$J)4SnGTw<_XBwiPR0|?Chc#lVFedGL-cBLJ<9GtI1<0fW{tB~va3ChTGPD#6e zV10H~5KAbc9P%q3^yafV-)#vlLnTKd}mlnvvXFu(ba71xH>i7mDX z{tjxN9Y35bpWTrs(>v(%de(=|Hi~%k=XU&eBYSc%&NpW+E3_hmypL((AmHR+Sf(Xm zO~S$Kc;|DYm7lMlLi{%^`S7a6LBNrNEP`j0%gAwNx8T_d$nuJDidg>!(JRmCJ)2i6 z`O}}QH%nsLtsgyy5Rf)WGN7F#?;^Hdw5}|i?K6!F{siOeWpvxyhj}dtOmXf}Cr-;| zakCvCbITu(9Hni~+{Q%gOtmv;U~z*`rBgYlSH%S+vTC3=eF$Rh3n*nfkGs8xn*XhR^b7&Lmr8c-0ahNKpL8r<7 zlsdB!VQ?Z-P$BLO^NeuWgkhvxjdlK(X_w2iL`@hmmmI3^JDF9x?8@Xw-7(am>b?Sw z*PC2km{>xZD*hY%*s6+rkUXZYjv?#c!|E*;60+*mw@jlI7jc7>T!fQ zu?(_f_)~FYEMcCxru)u2p3_|dlgLL~8wSvl=K}K~3>ZtDoOk{#Lt%HkYbnS}dWX zGDKz#YdryYYgu-1f3o5o8aySCZtcX(x6mfyjhv9e0ip8 z>L9kiN-7tdpQ(f;TRVD?<64_{)A2_pK7D8zF4q3`V;0rRxVQ7F;8gJYPc2^{=M;_q{q3;f2V7F|K-7Bpa(YY{Ca`;17fs7|-O9kmbxj64-Q0sPU+}n8ch1 z{QIq+$#&{6Sz_QOuVDa`9h!uub0g2(Iq+zBcRqP5{5_BQXDwRSuo;*4rXb@lTMU95 zfI4{sXpF6^3#$tTwnso7Mcuc<&i44B=VnXTDJdxxOiZ=4pW3nTse^`RdvrUGWw1@0IgZG-)BblVr_1+WHG1Vuzo>192P-!n zZ>%fxhq`}RM293u2@}uy^P5{S0nFlzdiH|9A9!XwE*w7)Z{>CDW?MGOh&Dsvd=TZ* ztDl~s1>&c>9+k=%)SQ?+_xiZAAKXgGcMjpBQP#`p9&UV_WZ?98wRr4%YGGG1!%rL=^~21y%%5b`MB7OMl1Vtv3v3E|QTMi3eXd=}+r=+78j zK3Yrb9u#!HdeGJYjwq-ev>V(#)Dq1CS?1$d1&;RA9D3gfrz5ept@)tIy&yR{_-Ykt zJZmUDNZMZa&Ye z>r)zJQSkzl@^s~BrY(QAi_Dsp+%er*GcfmoEePz|;-}&lXi1%kHJ<#;>itTwRB-8j z!EbYL>eXcZ&dW~z;{t&tU*k@Y_Q;+-vQU&gJR0sUnRI~Tcl5mSuzWEd10*uPywK7@ z!~Hre!9mekE9?Fc^`6lo?r@FtS@yiOgQABPE3Z^p*Hl;R>s z)(};{j#fdzT za6StM#k(owaHH!2;Od!tIHqxJA%0KNN@;jV`F$uD23%Cb`mkyRYc_+Fi4YsLSI$g*lB zp@+5`+GSv>nC8u6Hnzm46U;78bceTl1oO!VcoKb|dTo#a9N3XYYW6B>^!~apAjD&t za*!(GPk`-^koS%Kw3deP{S4v@OepKGlCxq`NOmxw-^tQ>;(h5Fjxv~3*e5t@K4*vb zeR)ZE%?xiA*+g6TwjHspohcHySV-&K%7xc}5XOvPcqA-s%7*P! zL@lCxwUE6@#2QRoWt>71qW>qtx(Qw0-C35C#}g~uFzz`1U|w0rKKf`FpH7DVx8+nY zG<@HGn>pye!mna#`?M0KmrX>-w&7LuqyE73e5*g2*(*7@gMs@NlLKO4rGNESbGGJn z1vb;;=kYinU4d?ga|1f;U**R0UL(Pa$;~Y^D-xE1Ap++VnM++oXSy@09xzB)R$Z~e zPa?AY`k;7>LavBT)<-4fm`%btW1YY!&{M=?BQZ^@{Htb2X{b213)8@E_)io7^=u&F zEjnFH|JkEEi2mgus<&}Wr&l{^cvJt<;}8fTWb4%tKsxd$+jP}6nTOy!_+fzy6Po`f zcKEE7U23EONXF*sM3vs<^1L++&AOfPNE?R1R^1u3MsdEpV6o|$3P5B^RZ3?74NX8i zGC@g{^}Q|lBK>E;&q)v7Nyj=Z1Xl_215 zKRgW?%Q3wIGCI5`CRSb(?WuQK-@og(|0THd&i5h|vOFq&tBIh==oPLTP$cH!pj|oO z9HmI@tM+B(fNHSn5D2gN`Rb$l{5HjOnh z)EAz$zdAfQm=!mbYuO`%!pqHO)wzpat2SXHAvcKxU5cVPdxL?LV+8r)_RJu<7XjiNs%pUO0wV)8hl80nS z(p*j@Nq~p`&$_M2m8x^H?$q|7Y}bhMrM1dNa*g^gF6s{Ud0RU-3Tx3hzJg`&Y=6!b z*Avpmni-!w$9g%6nY5{Qtz7%@lCFTka!F)-LW%h++xKZZl8V0-BoOuP}7ggReKV+ahdF_kZPbxm2j_adMJr^l}{E%S~ z?6t!$9Wlsb%fj+rN4Z-}s9VxDTLNW8QEkFd61^A^9a=u^O-o~#UUTJN8a=@9ANu2M zyT|B84qz<{mY3FN2~${8y{Q-;=)|S&+5(E4;y3jE8&B=Pcdiu{c?_m~=*704n=r8t z+CVJ^645>E64KZW3y>8lQqZB=Si4dG-Q!(J^_RD+&E&RVN&%GIk?*#L5=g1suIPUq zd2$c}Z+w7ry=+CiO~#h;n(@u(q%Gj=a3a{8rJV0c>#j7f@S2;fyzje0(d)}DDN*@A znw1`S@}5rd?&ZBsIrvJfc81TMI8u{VL|*(^gf8EMKP`92hvBE^aR5hhg$V)pa!KHu zcb%=LrsoV-K`!vyd%isCCqlmRd3+Kpqd;$3O0kiBnQG~2yM{zvBD^?W8pczjBb&g*G9B-Mc3JrjMRP15u1@EB*H4d*P@AbA>z|->TFTR=M&Im2mkHcNDbW zJjgI1wMxz6QbYxsc<8aoi5>8ccI5pxr#l~mKjrekz zEl}A1=3Nf2>4MK_Pj3zlS@)sXIf)ftb6)_!z{NKL;11E;#0V5G5L%^=ueRJK#qP_f zgqYGFZ?z08SJ?~T662LWJpuR2;mub4ReLA~yow$WJPt2DW%%hr$$pe(WXZgJHgm4X*a`%WM>J=VuIIUhVDf^6#|a<;pb8Ri#Oe^K!UR;Snd zU&AL~*H2`!DGh>3+p8RKIW$B$2+fbBJWZD_J2Yv(ai}0wzGKe>AK}JSG-A=5N(U%R zQgY6pgUnL;>pU5pN*5way#xzed10tH->e1l7Ct=B`$($Qc5NazRw;Gt$zmk*%iB)0 z@wMUwBb>RyV}}p0@(;Id$T=QQ0U=5tpHahJjklA;^eOuyAT{8CSHK1V`8e8e*u%nEClTR zB2)IC$q?SXJ>(0idn7VrF&`lkAkI*Abo^`y+B&mX>>1EpJ#HL#YS99MQ(_um2d^44 z?Mm7re}?YUMpv(GyN>d#%?KS<*)myO8FiCQ3*%^vs<4jTo~5OsGe!1jam!yFH!@;G z;G7l`(pW7b*V?H!q<&78$#X|fs9K)ddL2)Gs!VrYP+&bNp5B?Pem)v38<2K-c#!{n zwbD$ws4ZBfCUuq6H88w!Ze)~~$V)Bxlu-(iR=rknY|+d(_-+t>nz)+AZc}S2shK@* zgOhx+Rx|#w^MtMn5s@aXiCkip(hQupIhBl_(BZ0UTmdSUmE%O|JXjTpeqa#m$VE=0 z0fnnTHTk3pw&sij%B*6aqWc2QzkcPPqqmF{lL!W#koM=yFgi$;*~vg&cXW=$NxI97 z=Iy-p7!2Umg&PMe;X1bt{#=A_;k2g$hX(NFo@{8@*zKdPoV}YNASa~;C<6{;LFnr= z<|cx6T_Iig>>y{BX9T5LX*cq@SsT`6_pWF+EQ;mDm)8?Eqw06h-xp+tX?qTi+W1Pj z$g5qT?Bk>}1E;r%pZm*OQXSuim({oLfPH~$US-!QZ|3?(yQ-Pm^gq(obDmiR5AHQ6 zcy4L@yc}HCPQwbgEx~PB&U@z}IIsVa8v5yCv{<=o^DVA~P7 z7drj9`NsWp{M$b}HJ|E|LzwkGr>_2(3a#$-e|TAcJG=Jcm#bb5f78x*yp+DYa4<=M zm;D?mZ_Q?Kix*d|ID^JCIWdvqr4q~h@HUbd_Kxk35N3`(J*ahh*s9ooG>dwp-lk=1 zE?ynV9Idw&#VDpaa%8u^DHW0M;zWLP59CLha={tn?+@?SM?#nt#BQM1N1O--dX5_B z?A^-jD`B`+7qm2H>*3?cta(S5O6lgUDum_=Q}2ls%2#dF_EyOncI#nree!&PZE=5| zgp6KQsp$IEl^);7&~fg|`Wf|jX`wnS8^}DD9GB?#3&43;?QCmcfPQgW9fT`lAdjcsR1{)#!G$<5J$579HY#~ z(=KPZwpNg9bBYAHHtn0{mKQ|L-#>;OIY^8-iKmoM(YNsYsOGY6NF4Gp>>I7F%^!U- z>eenkD!eQtF)FN$AZMJ^nq&@XVQK7|#8yZ1?0-M1-*Xs!T({Th6Or1F!B)o`&+hM< zIoTXV(uWH<Em&^#`3UdIOqhrWQ>Aer+#@VJ=%DOLKB2b5kA`s z`QidV%F;37p1@!K{(dp>Po9?8%!{#9FKw|#+%S|bXQL^NFe&1H$$0PHF4y>W6N_b1 zHd9DUvRQ^>uK31a2$RgHAZ*FOg+r2EBdzkNYVYBx*AcrS=kR^4%VW8jpr?{Lg%dX$ zj=T(1bnN0%G*sKpEk{5IxpWc@?*h%)&NuX*%C|L-2Nji-!3~u%GLwH)Y^?rJYFm_S zV+Al??{e%<`yG=Dt>B}qnA%8X3sAzv6SbR?66YJHHfe!Q@U5xqicA-FN(z2X^6rTO zR3BuQ)kiIibf;|>e-s=^%clRI0w_Pnp#A)!C?H!Dqb<5^@mbLP*?h_Fpfvd*n>*|k zNJ7Y914rf3t)~2$AYT}gwYQy%ApnOk_?;{Cahf33uM!`xg!t1l@nAiA*oN2;)wf)hWK6_CTBVyTr}&G$ zE(4W7>#wlI$%SEy&WI;MP`5A+F>JGBY{YM<1^!bDaxc&JozH(j3X99@ z(rOdW6C><&tGM*wk2Tf23CPo124YfMsfSg;hPBSN*_t5GK~v2*@=y%E=gAnArFEFarQu z07t%e_^*IwSAhk7e*(hx>0itdS|_c=3-T%p!k%csC)IeG=55E+XiXh(uJNR&(JQB& zfS#G@O8B12cy(j1f2bGapKrzcUKiK5@wF~o8Ql`rRT-7OGJXpy`1`)vqKv;E-hqFx zpvKU-LwP5u8^#7T#plVv`2g0AjV9y?b#T8zlO*?6HG5km(2T)q=Gvsqij6@nlXDNM z<;J}`3x%9Y#>$xZEC7}Nk|HinF?{|N0pG2Fgart^zgFC?-1IrefA)VJ315$R<)cfYBDBo}LSdXpH+d!+3- z|9ZXk85KTixwK~KSyL_oVB|097q^?DpHi%5hkkxcW$iXHHEe2ue;Y=k2Cr=wARj`I$x#RER%UHE0MMgtp+1>bE8%$6W9Tp9Ww9Gk56NH6mA~*b z6hZf=kR68#=!`wtp@NS)Pepc&^4^$OFfh04fL?VQL4kTDliOp;(JxGj-!Vw6tjDNW z8$>8&qhYD`meZy5fb;v0i75y+mtPY9U&V%I2cIiB7d(R8$&QMQzZ(rG{J zz^}U$Sa-IWt~%cF*kDGs4pWwckADo1@dRX=ih~4^+po!ajYj?%GY>kMw4&q;_I}-S zC(tIv34c|$-?d)}2Z0wZUEAn15gF8Ho`;|#ApNQ*mv#gG{Tq6yyqA=hV+Y(28-z(~ z9V+Ei0`~HMBc9=6;{X)E_ciUWeLBdXT-Z{Pqb1am>n%(QY6#Zg_NF7}AJ!+nb8dC! zRQJKY={UAmBlC#6#DC+*z3$)9xXl8Wuc@g%lV9#o2z7_VH8+qrl6Sb2;ZpCV?i&1@ zj(lYZW%ze3mzAU43TX@RRe*rv0OtMoQyA|MJOsI2jkr_4r-he#MBVw7Kq}d#4-@+z zxh%_z#e3N@tCSNyRVWi_?8wfqg7AF20tyKfP#SWlzewaqj(nU|vFy!zSUcbRob9h- z+F!oxUS`Mdgvw-6#{P|0wwj(I43d|IryE%)lJMibRGgws^fCZJ_mn5;c+@W%qd9>z zDaEfO*jIm7GX2H_HT>}SW2!g_|BAi^T`BLuPSfba`)T83Bwe)Hg0}i7h(XM#jnLa4 z9Kgk|41Dr78+SBB1l@SKa3WOeJWL1lQVZ#X932{ss$zEPQq>3V2lWE|$#(Q&cm-~V zL#rFFU-oF4Ti(Wz3SV1s33VQX23>z@K&E{Yyk~lx2pi<@`EgC&ya$PDlZ0akEJN&I zuCtegmD5WukK2p@M#knRUv|fuknhJ`<=Lv^XC)xuc(%ggob7+bL=gJ$i}`|_wsro8 zkFmcMs0hM>g(D*kj1OM+85qI{AhK73gkq$$<{h9k0dgq+0Eto&;)3epEkC7U5d@is z-~h;Yl8xVjUja^H;Qm;T@8`JamPsqf)1}m)9+@?KD^$g0+E_adPjz2o{vFMaU!W`b6&i&{8ihl*RzPjiZeAvt z$xeLlZS_=Mk};jOHsr?;75Mofs?5P56iaSlwW|9v6|7Y&*GqwT2tKJu2nsyKKKSABQl?K`Rp)GuJUs?tn6aow6#8Wlcv>{n4+8<3boS_rHw2Tq< zFBbL?KU?1}9;1&l8$r-gem+EAg^3zXA_a}n9p3$F^SH^vB9Mo1a$D+mk1l6?DG^vS z5)YSwFRaLSg|Z*{i3>_i3SW|1>e|5@8f(PLd%PRnEN#q0sH03j3wytBr@QL>j08Kg z)ZJ}hb`!@2sqfWd)r*J58_VOW3rn=0D=!SVy@I{kE~CsRJ0AZYqVD$@Ab5y=`=|lX z^(1YMY8gbhGsWTA(^?=)M4oDT&hA|q#qSOdi%bATUV)VlxcZiTjT+3yK+8H&j`^-uZ+v=lFg>NS_% zvS-1{LGojHE=uV|!;HE}o*iVMpv>xebLh1n`6IMwv&JIa{99KTVQN842MWA>!Rh;ho$EjYlT&wPX?X@FTTq)D%gJ4 z%}s%lksRu=f9%o3DiG;Mc+HwbXyqIYxwxfnJXEoJQN^OH!f52+`N9CDZ=(kK74h&f zV$S^MiV&EIlm#`=M(>UD7gpa{?W<@|E8>L2+yqo5^x-R<3Q&RNj&;8)rEdjjF9YPs zx4%jdWqBR>t|Bi@Py}EfCZVnff1g9#zJ+RhXLr-~JT(pZIjWUrkZ<_;yiz{#ueh0r zI6RGqlAa!y1ZjGDrba^jdL1?c8V%jVvAfQxp5MxO9TLndLM%pmZ>DgJ9<67SP~;SA zqQ^aHA5&R@wl5pj9em!@;&5=|2W_FO{)4TO@{Z$k7kiy9dn7XROp{btQakk_+x0%_ zbuN}T>Dk8@p|f{~E| z4omqUlm1b;wW1fvDgC1}#p5hWT>k8?wQ{JHboUy=CK;_SVUibnw{HCSp~J_j`Fgu< zZja80QH_pG4AwIkmI**rF2pJ^pVI6LG2A|ffl`L)>Sh^GUWxxX)Hu)hKJ)pPZTg>ILheUhI2Z0TkkF>5 zhv7-7C=p9}bjx5zSnS+RZrt_lJp*Qo@^dr?wJkKZjW6cf*Iy_d)sE@_Ef8YECEcvn zuI##=#7Ri%jR7{xm56aFcjHDok{~Io>Y#>!Z*zYVitO@CTqSZCQVFGqLv+4WrD%O- zN{5P=k-Out~(ZgVU612dV67-veg zF546L5%0#?Nr$byV9UKY)#HI9e)REZ{`7MXY!kuPcP&8Iym-r;QE7`w|7a-Ipq)Ko zb8Cyc^nzpW<54^@tKn`^>jbVb0{MblR)u6i_pREK8+U)}8%*54VCk;mKI5S9HG*md z^`<;x0KM_Zj6=%Nu}j=BERFH1{r<;4W>nh2sXj=ju_Vv^_v`==`mzO`REG&Fs2-p) zc@lRhH(tSznu{`A-bo1wSVO>O^KULx51+55vv<=m>z8!>IoG{?JHA6FxvJjmlX}b* zoO1}LpaOLF8i!YYoKV$)&zIF%LwPx-h{YoHH5H=B72NcE=^LNLj@(0G9ZqkmL!*k@ zN4PeMpV@Y|W8^YVe;v(XjW%0Oy-=xb->Af>112mcZz_Gs(A60^*uGuTj!K%860$Yl zYRvmGbvnmSh%%J^Y>%sXIMm28i-#V&_c<9vAx(BbX~+DYTJh!E=v|g~p#))W?R3_~ zr49DDEuU9`*1^@YJYE@wUhvTYdO-Tvj5*MAIQ==)x}%a{`6 z?n=^RYANnU&6z81aa;db#nIGkq_47)=gPa_+)k6%ROEM7i8|sZKV5OJIxyNu?bD@0 zsK34hA4Fvu8%Sb7bX$CxC9;?$Mx=gep5X9SH=CaQ+O4Rbx;m?(XBX7Cez82Csk-v4 zip;XgyMyQ$7l{N5K(KIwK~$SK-AdWVO{JJ6m}{Z!%UK3V_!@Dr^hPiI_lLTzQtB7J z{w(^6^pxU`)D_7J>;%O2ig!a>zr8vO9&jLdfOkicXA<$9*c27XQB?j92AFw~cdV&a zMdGnMLe02(_f*(t{I|)(_>%}IgQPaw?`Ws#680ooH?!_Y>du#|8BYcr4Lvn#Wfi}y zu99%?T`zE!M?9a*l~1X^wIRMpqj)#47bX&^a=kTx?ci{w!6Q{lP;MdX;DcMC+q>c- zE$rdm%**0|jH4-xUvvRtnffKM>mi)vi3>(7%d_^`g^r^3DPq_5#gE2{&U~zoa+B0^ zkLhfFYWSkwBYvs)v3_&V2L1owAt9nzd`Q;gGW)xU08|IQ=jl71=9q zl$XjC0p6%Qvc-mECB@Pi760#B!7R2uWi?*Yu~Xx8um-KY@A7kj=(}txLmWmWy%$}3 zxU)8EwNT6gT{2=O9V}rQ=V-XwI~K0qAS5RM%dUq z`6m%+E@#OwrAPDZHrg9a1Ord%6k7S@aDEzjx3D&zF-`@Um_L7Nr#h@ipKebw{J2~@ z`LpEMJ3|w8^D26$pP(|kcQt%26>>CwNoN>d=lCDnKp~e$5IJWId*y992yQK>bR%E? zq4~>C=_8}WrGU)e3aXd0D&gw1g}(|~k>m7W7sT{T`8-(o<9RD0`tg0@TzMN_>dWE1 ztLRsKaY+U`)>MH<oL9 z^+9(L$zM-M68>+~=Ia^I%-LgF^fHKeCgC9a)S_->05eh53s*)bCS|XD(QkHiVM4Df z8BNN#+1?SU=j4{GUW-hxRlJvSE!imQ<9;p4i;uk?)1QTQ)&(@}+(u4^dl>JUxS z$$i8gLpwti6{XX0cUEd&0=)6%&h@vBfg=Yfw@&JjRUc=<@Q3?r3D88+3il-15tF*L z>|~``_l+gl8_Mn9aHZ&1&t_FmeCABct9#>alJvs}`Hd2i51ixWrs`MG&!5eyV$WY; zeb{_La_8g!`ZmjD&CpK7sel>0;YBanbh7M{F+%q`x0}Ek)f@G2;q8+Pz?eB}Xo{rr z9x9zo40auA@=^y^&7`EIF5B9l1>E!ttXAL^3jD9HFveexfGkgGQKq;^K<3 zY_8*W?)Z*mpd@KlHSMi<%p)Jv}Oy(P!=_h~d-Y zv<%3fyiE`$h#dL2Usu}OTNdCF!%O3Hx5L z08L~oB~JKV)dk{vtU;0^2ok|W1A}%!d03F-$IaWQvaR`0%RH@?%c-X;rN)Z0`UMr} zHKFYwk)kD8#daqU7zoCe|D74&RL){TzpnxwGVsF^OdK70wXFxa9(+SQ`r`p2ms++9 zoP2xs-`Nx4A}yCYu8$4(v8Gk#Y4sAP917zK z1HHTMN_O`M&umQmS+IKS99wm2n5t=w(WM&S+Dj-i+wxlCO6l>QM8fT05cBgcAi3&Y zc>7?Oqayf1UvDYBlxq_gw#qBZmoYmykB2C{zG0*iD!vdRb@_SRdqJcFlh`;ckQ<%& z;p-%CHTN5xH%|gJ{xno-8Q}tN|Fp#muR{qUg;67^ z0G{M|xf4dhlCx9Ic8+TW+NvC z+J;(EPhD^J{x~BT{123_8-WV@^(97Rd{L!GX?fE8rbM_Jye(t6TsMQo8!47o1G zAy475(Ke7FO?9xbndhLd#+Ukt-Qf2F~RB!1Jd203~i=FzZK+c3M!L(}8g_0lMU9P3%yFazyVk?x)wlX?llwDL+WIr*s`o5K@oy!`({Mis zn}e%=3|?+XkEZgB1e`1o$3dy7{u>K-TR7HX({(>jh8-EBl(*f0(q?))x#RsK1Muld|}Ucy@>XMN5r%B z<8;-4R>i~wHdQC_l2V|21C>3LX1Q@F`-Pi7QTD37$+X8yCZDxweMEJVAbja9D;psq}+4@FweaQ99olm^RBp@HP}zVa!AyL%MNTc;C|cCcH=opxlImE&PwzJwkt{& z=uHz-v2!E{AJUfD-s)JUX!0Vy%fEV%efFV| zvf6Aw+O#{zKXM?@PAUqOKC`Ld8|}RQxS0+@5h7zu>yF-@ifNE(9qI-3KC&^tv+^N+ zp~UvNi;4_P{my;i;%V!_3}W;hB9;qtE<%%InaSdz<3{v?ub{B**7N^3>6C@GymKpt!jbJel)PTLuD%%^IjzMyRB|1H{y$FBjmzOTDA= zJRJfAs(*lm*>y-0h0AXU#HDn$DU(;!LDn9S{rG2mYKR;t%|)|m$e{9Mi{sP5j5j># zd*ujZF?gC~fu4{QU%!w4)oz9e1_Cxtvjjmeg)Vukth|uwz31OhJP^OL4rb`@ug`;$ znKQkyONfOMi}3nITRLE=yLoon$|ug zkC7Q&yh;FZ1XVbYZUNxZ0Re{}v8=nO+l8p*e-0m5%@HFAB=>AyQl#|wunzxQ84~mo zK*hs+MYMzm|L=bv=6;~c{;#JTuju@*=iUE*ew;YE_*WVVi4q4lSlr^_<>0uFx|aB% zxcJv1MU)&6!u?l&kD!VHF;K4aU(e$(Qhe1hc=*)MGltV_=sf8@N$~U zzj5+EMNe`PvN z_m+LnS~vY8*uyuIcv9>d@$Z2Ce%MV=ivFLEg_z$5+QkHY|=_n+tg&5t=U3JX<4#vB4PBsU(;&d;|`IV|`S@sy*v z*fQ-KkM-wM-P3l5Et#HMT5<{q=zyDO`bk8_GTy$uTGTEdGhF^8fSy$8>;Ki>bw@Rs zZF?NA&b0s{3W5qaMwtOAiYU_UM5G8(q$2`iq!WZtEff_*qXGf}L3$t{y+;M%UP2Ls z1YrOLrA1&UQ3xj7eTXyf-u2!e^VWNBt^3FO{$MR2U(R>-+2yzQch3H8Us7r4T_bAs zX57zPWk=P*(A|7_A?zg}`bp3TC|fHiu`i`Pi}R(=V1vj$V3shATbnr1I>Xl_z}NDk zZmrXh)tZ)u%JuPKb6XU*kIHPqC6ULHkn%Lxy13-H)y$mY7dvo5SX#f!yIsQ>kJT|k zlsTDti9Wq9SuQ<=_Fk1%gs=VASWAP;Hncbe1~6O56)jUqp z@Be_vCx83E&q;d$$x%1~sX<4Ze3hV?a)m$&!3@?B1Dm7hd{V6KSx-l-*i7q;qi z5$#c8B3W&!;yCb%bkr?P;XEG~CJyWn^!Uv8V&i9=-QG5n7wePA+BKfZKOkWbUH9Hf za#+7rYRy8R=GHA+%&pfN^+XEo#$Z2v`a~DGk#1sXN$%~nAk(z(T&N|K9&+tX2g#dB z#57HK-N)Dju}m^DGVlu}1A-w7$Jg9EDtDo%A+6cYx%P;}`<_*yVm%Bgp%tOPqDIec zi4<)!k&;j`>p+`>L*vP0vN5ddA-7+%x5}E1RWT{X78VKjG})KQy2m$Hfu1~0PNvMy z`%J;fTMjv-$Zp z+y+tk^KNdGa6w7RLj%nVnkfddrg?_;_NnWp`FuCG`gyC8RKN5%;AwK@mltP0v6;5Z zxvN&M9cz{?=mK4F9jXqsZkW9`PO+|@pMPt>ghA=trD@5gT>H}{*ZsLaZW?MuoE1@*e__dZ@qeD zp!=mumu|@=Y6exEv$pR4G}lF^ixZ>X6+0&yt6Y^9ac~8@Ys@tvM>*1B#k7{*wMVTM zRDa|5ZKg>tP_*!_`DCBPL1ksJhkNx7>=1~U8P7^-tR(d4?Q80Tr*qahFjd8r7nX_W zKcPfNv|aTdJhlN{SITn&>xu&q=rj&4bFV!MS>}fKO&&TGE$;YI?buI+Lw=zSKk`|B zQ0JL4n1bSl(O=L+(8>!*0uH#EmXm{xkRW7FRgOvof#K!m=bM6`rH}IOCie!&dL6nrUf>x^uAEN2EyQoSLHm- zmTLrWh`V!5X7=qow1qZP<~MWEzt*NTL0NXdu|7Oa4EIplfPPa@Gn+`# zb!EiaFI?JP9Jt^P1t&O^MeyZ`v)`^KMEfw z&VZ7ritPF6cEgpLAnxL5IA%CXsV(xm#<5teMqvDMs979IYy`E5nwYp!Kkg&$Ov0P0 zp9$wT_e9<%l;XiN-akAg=*cF*Byyx6M9o8MsG>=a*X6)_p+(d9o%1{xB5K7@%jW-4Mx=26N84DxMZ z>4|{BrR1XG;!;?1uy62Va+#&CG*u+{?qBXvc!^|#JYVtmXBxWhaAN@w4rU#7aFC$2 z>96MI>xkB74z5gkzL4Y`gUU6zuB{!h)7RVzcYJpIO3mc2mBsa6z%N#fXC@-1Cj3NK z1ta2P=flAa_~i0Jlh#4t)@a;wHSJ+;D@mWPh1&g~dGpv?JU7D&kv9Y0KX70syo|X(sPjs)lKs@-J zW8#sjeTa2*sV#RljtwS@|JAd}r6E!%H#fJFTu_$_KI(Ey{QjzmT=H3ph%cktFH!9p zC6`RbFMNm~7Zw`s*|UcYk%t0=8O(AjC;v+?PT?UrtRMfi`aBVQdxo54*;pL)TZWZ< z9QN_9cRtE=_--G#n+d})!110pGDsweVl( z*=92Tpxe8GFVY4MY3v7Kfu3=uL$rnt?LHNCndF?ynM4m$xu(9XYCEo7eJ;v#=(e=@ znl)>}8BAqhN{Z%u9UNR>6@<8c?k%O=xMT0!A6*`{K~x+$hS}NKjpco=^Grs&%1FEe zawQiG0qP=(eBeO{;gP@kNU8Ji?2AwXMoKbqMP;sBZb;g*rsLtR$~H$wAO*j}y!Rxgz)J zj=!LD?E4OC*nbSY0EVu6u94~}w7r4OQ}E!~${X<2!!oM+fblnx$dsxx?V3w@m~n-W z#SLoO4g9=)1?V7I*z<5dJF8VZWB=ZvlLrE4UR2x>fV0+(l*$qDesG9D#|M#B)Y|94 z>#e&k{8LBIZMeuE8@Gm@s1IKEdyhKa_UKK5QRFdz=~pL2sS9ClRE{sU)3v-mGle^! zMHji1{+f3Pg!j<9B(|CHZRTIvv$8v!H#dN@$JG@n7weH86Q+A{1>JD}UM#y_lKV7e z`P-<7*{w@|`di@Si=7g~%3BXdn1pzJlT!P|;;!tmqKhkA2ELvx&OMn>n_O@+j zWodD+X0}14{~ZA?c<|tFC?yDjkNEzJw=jPn_mMe#xSTdw083---Ky;LC>&G&Cxtl_ zA+RHq|1URR7A^sJob*}(6ZDI_h555$MECy(wC3Xz@wTx3=ijw+|4);Ff4eUyLPqz| zL&c|_|FF6B`V~p&cqU~*Myp)8lF>eu7E!Rt1Cqz=>}(x)i*6dnI}~iRZS)sNlGUM? z2p?ZrA%yB$_5_*Oa9e2QOYDAWGlnCWes=2-4c;Huf0_2)e*;78i>UADjh%1Y?*6F$ z!(Xdifi4Tbq%HN7%9(UW6n0)-ULWTc&l?&(Rcco^2J9U~8BUFmAT|v@fOba(!7Q=> zGbLc^2x2D%=Qz6>etw~pVweHso+wR%WAFf`p4JPfodudk2&|n# zr+1`ANCsEDO@PY>AUqj9edNfISwD3kO1$3~CHCxzlJk|0w39)>)Rp->-Q8vkffFX_ zWr=i1+pM#*^Pt~vb`TYfqobpNRMF7M$*JcMVs0wIG2}^viJW|CZ3rI#ROb7y4Z6H5 zazqIL@==_bn2^sV{Bxu23_QbHI_qst^>m#j%3mxQ-E{iIx4Cvkkt59_4pKNAa3>Nt zECvgrDRVkxxjH$7>(9SUQEF)J#~)Uae0yC8BTt+^`c@+V-rRhq7O*%0%#TG~<`^6v z@NiMl$lk4P+?8y$DYN6leRoa*Ojk4L5r1pn;bBe6T+QK0dO4$ ztcn>43E~=aElRzRyRZPNmDdGhLKbPZ=dG;rUf&fZr==aM_MPkmPEdR&RvJrmt6Ow} z2)eu>poO?cx8zE<3`8JAhxPdJV;KbnGgsIAGPnL*>6Abt=OoWl?cCif8XYPDpqDpO zl>bZp@(l$oVFPu@%^-R4dA-%_XAKQ&RJkZXK^`=Mm~OkwW&ng>&Mu0Ji~B%^jeB`x z%><($#gt<8?!$?CW&xD~T4fG_R7PH&4zctRi6n=eUE-)NetgK9AQ#B&-hJFai&LVx z@cw3G0YIe8tYpHOV&`^KNTx73fJE+~gX=x@ z*}u_Ud%@5WfU*%Y{wzjwy(NPh5`r}XjmVnjxe4=|b6e2s5mey_P5*7Gdio6U-jR`Q zM@SEr8{Pp$uZf}Irm~=b!Jw}slseWzE+wu(C>bCy=R9mC$(y@tdvKpu1PC8v!2u{-G zW@nF;HG*S^q!kwxsRl*$?PPoBKYzae>f*W6J2NcZYfTV8Ei8J@V_CB^>uagM7-x>b z3dJnAh^>siDI_WDy*ijw&{5b^#}|4C6>0!!8X)le+p7&~8c%j%0WuSPfGN0HG0M&T zuBgI2mwZ%LK#5hqXVXqqR+sT(SZG)&7pQq4uh^zO*CHUzD~p8&t$3p@XsZbaSi)** z)UlmVT&en%;uu8xGN~7@c$92bKnG6M`B|C*^Omr_Aq?K&IyX1B2S4D|wGoI+yA49v z#u_kAZxx=b3Hm~>#K*~5W`Yl86!8wM>zMp86EslIFivUVqy~s-rjD7G}Y)I3Q0&mH_(~iwMd#+_ZR{H%>dHUB0KaHipTYg}W@hBn z)Kr=);aOc>9gXGU;zIOIG33=epV#}}@mPt6hZgkU#P?NuU>qi<10!)A>n)Hq5XMpy z_B%l3;*Kok4Y>4vRfVf?ltCU$L<7{3n2yXWm`d$qsJ{Wd8Ow$ckO!%Xg)LNOm~1wifE+jbwL);J(C$;0`^sE{0>AD=QKV`t z?Jml)Z;|%(R(fY4k5~I`mPSk$)>ql(RKPo#eoVzc=pM4ZhHzd5mv9kiu~A*7G?+u+ zm(;YXtgOtJhIlZ{4;2&?+GOP9x>2qqOQ|@9kRHUT^ENgEy`A&m=sO(atm(syoxn{8 zeIS(mXD@LeA?^gl%!0si4p@cbFd%^-sNz{vc%)9%%<1<-8LYXbR z+>7UY&Cbv7Mrl0CT05r0;C2dN)mNOS4=C*uTS@5gSv<-CZ)j+^>zL|~x zNpRXlP(dqoXc9nvfej?AU0q!mEONQ1$RTG@fSC+= ze8*`?r~^tlEo-3KN_uL^5i`gI_HrnnJ;^cC<$D}KE}*kEj?Nis_r0w-LaFaA)X z9jA0#gjfFf{YR|S1Y)=R{rX=nKL6X{&EMzA_dl$v|Nq30B>$7PEG>rzn~d(3{Tmnk O8+%g!1m&3Bjeh{4yycz% literal 72942 zcmeFZXH?eL_BDzAkn8K}C=*MVgABbOGtj5-?H)=~bmmlP=vxk=~0m z73sZqxO1EHf8X_V(3@7}+E&r+C!!{mSc1iP8V9gaUPHmT!PcHg_KZb?CLERFnsi}q1pI*Kh6 z6jFbkQ??Bs>$Y)Jn%^j&wrMQ7`|wC*&oNrcU-svy{`u}9CHuSgR+_mRbN0s{XwSc1 zWKN(i+PZcBcB*pK3BAqfNuCDAv={^S734`+37|t2Qme#+aw%fl{vXX<2+`Tp5(uOzbQ7q0XDw z+*p*D6&mU(D%kt+thGbA54~zlD4$-s+Tq`RdzEd}7MM`P_-!q=ritzB*~{kU=7w#l z5f{CV#?m`4y=)hbkb7hAFgv%Rudi=7Khk197-7R}@#Au~NvF#D_wU7p>_l8P9DhDQ zM@H-|U0o_Fs@O8GV@WoXJ@(V3)Y&&bx+%74g+DoAU(xksFv_L))94+2eP(`sl_MUK zcm-kGsT)FB8F$afhSx9lx}>RU+euz<+pB=j>+V$Ips2aYV#{Lsr~C;g{r<{;qe2!j zp?pShLqm5ZFI`Hqn;lpg)!BR<<+732K3_E}ct$FW-=tAVPAt~@xCry_zn{B#^CnqS z_M$mOv5lp^Pze|3r=zT1e#L=!`@S--S?SJu-;VO=dzN|9e%wt*5wKNv#NCyPPuFl) zS+i%N^ke*W~je!|%42_SWT@5&d*E&$+q#x%LZV z9lC+ET(|z-w29e!(dUFoj#Ed5-a}LGn>20>FV1ZX8Wa;0Y!`G~dZRj;W7gXyOa=^3A>GbcvzYpfQBbz>6RawdHw(pqAZLO=Jyj=XIU24W!)&Z1E8Z4!TAAaj@;N4u+CJXnI9sE?d?(5&r+r56+gi`ef_2kX0opsY z;Y?YwgW{9pa-lsZVP;CYL=o^(Ap)tzB0eB zpj-RZyu`}TY0LF3+jjaZvFyn;Xo|ZPC1Dnjwdr+2EWcvKvExo-Qa(kq=yP)+JnV zZn=(paqJ#UN^!v^!Q6UR{`?|;g1vd2a=VV#?m#w;f%!Yq>eoJ{Q#hVhqu6>%KtNSh z$0?ik>T!ho`^URR-C0uQQ?6&2`3GtSOSi!jNdv znPdOWmAcT5(&-e=3BBXg!zx|=u&daL7I!KZUZYml{{H@94~}B_d-vYN$wg@!M<5Ti zyt(<&osF7`s((RETRXC96YHlcE9;G=)HHUA9BWU%P~@=4pO!k@ocMg_zN0b7But{B zn&f4X7UsKcdIMfNn`t>tmy$fgZ}KGuKW{kP7(3Dw@6$FaqGRBU6cc!1kB7>X)ow~k z#i~HIY==eD2&`H2{K80!^Z|*rd%k+tbBxtre0DRpBZRH4AQJ}Nq&X_#?C5K?G9wri8JYTU+b+XWcd7yB{j}VFVKuG= zFkkcc^LsVknQQd^(T*=AE-p_Ox~WBvON-_Wz4BPAKT1JStK*r-B2E%R1Q+6Sd>{F% z$H(BK6!J{lnJu;_jP=-$vhi@IBI}i511Q;yayz}~UDg`jt}k?Yd3x@Z$jMl{_UV^% zcnGDuOcodH4 ze)xO0e#b<2fdi|<+>ioEG0$ewzOIbt_EM(QsfI_J9^<_$f4hTGI*{hvxpVrqv#;rC>Q{T#}I7*CJ)!w`lv>pq-Nt4}n$;m5;3i>g(Iysmv$y`bB4 ztTQ*Qd3?cbc6!=dWMay`=*N%SnK~+I+T9hkwe008_6z0oE;JvX91tk;p*Ksuo?R)b zFQVf*#HOROdn8G`Z_OERo|%a+K?6N_iM*1z+@_a|9aGt#%-;U>WR%ei) zb|2g2usFe%!LGyfIek0TAzDF-7vFEnMEH>8{qcs3)=|&IZmu}?qq>GRo{8Vsuyx1F zA2t?8aWuK#nI3YnJiwd)eWhxmWf8H z(9i32+3eQM8%-^!Wu5Ka+-TFK#R}aYZY*V8#M&RpKY)DBSTnRV-Pc{&W!vX9Ff$`N zJ0jn><@{IU!c5tg5K!$9lvA|GVf-H=Hh07b#Z5y`M+Fy8fD~R_v!w zf4RE4CZ2n|lhkkNPIrqR6$i!Uo0SZ^^Rr*QdZi4|Vl~_-xw%~D;vW*i#3VtLQhG}P znYb#;$It)y^XSjd&lpX8eg9fDE%L+1k5tsu@z&#ChM%5^lvh+#49Qif-zU)Zg8AC( z%YJcJ!&L6(nU$=oAg2b!oY#3g^BWSseobh|ro)x)w+Jnh_BZGJnv>Oe_R(;@)73h7 z@E}l?|8mLc?AMuDS-e;p9Y5vRD*@*h+is-~y?kL(z2E zF~6ALV4C&Ws7==S1?CJSs$mpbh0Yuks$<8J{`uz@zPt5*3iLW=>eoeTjjimE2+GXJ zm|N&H`&s7*(96Ewl6gl?N)fq+fWBkLMg!T~zcpN<=PS|SHTjZNKi-4j%TQyb+`U`D z-afyI`y*$TMgJAC)v?Tx?t)z8>6ABbI3*<|6BVK_0@(%+)JABG6?+k+T?M4gC?y@p z+Q63Ebt^74M~&6Am)e*=tjB3lOUt@L@AdWU*RP_ZTQc;d#ils$~LC&<9?*Yx!C$d~LSZQF0Irp16FH7Ea?>= zX?cB337KvF!yb;I_H?l^gmMk$EyC*Dw{^GtMl%w<(t8Z10TqeMM% zo>nP{?9Khrl-$=y?vw`(sNTBuDz#|g`mAOC*~G-emANK4tHo}cB;9gvcMlJq@1*27ku`G5t>hv_nUY!z16P4WL<2hu?K0H|LheMGYiPI59^db6BG_Yu z4AzL$^tv`eghgDuxPEf3WOJinqIlJmgvYmU-|*~#8#6&AcY$+DO`F;(SBk9)b;`Yu z8;Q3jsdD66N_Y=eJV#?Qy0JQ`ie({ZY-;LWOPY=javPg4%947=W@o4)SvaR`o@ISFfh!aG?jVj6i7HWy}7#UGTy7gsfAiPE}XDePkdYT^UyK=OBxC zUAR!6MfRol8ft36TxLCmr*iY|4=YAvi(OmI8ziY|Q4IJ_>+bRJUf@4J)u!XZheNGA zw7@|7^5x4i6wk^2OclYwB-gzkIkE01F8Cd!r#JiYSPw49v=lBt7b`m^yrbNj>~#>yS-y<%)yQ5i(cGSa#7;lx*_h?+4WflP1RPw zX=6KjUIDjf91HoOo8&e zm>340ZZoOI?bKyL*Nc-?mzG8hUs7w^zGqCyiq}By^*VG$_VD#5vx74z`(6N9Z(F`q zv`^tIg!rtZeLOL01n^arAE`L>W!!pJuqQ=BAh#c_GOI-tT`r0jEfW)y>qOY>G}3Zx zS?|VzfX}?33p>4##V2y+UUqsS#E-CD|B<_w4tVuWdRn@_3uAw>dZuXWw;D_Jb_eI4 zLOWnTK6g!yKmI6NS{Y3%`f^FfaZpB1PQbKiUL|b-Z}C}k(v@19lRiqwQs1|e6XhKx zw0g#anu9$@^uasb|5Woko{3d%q!IJGC8H%sFc)L5ccN3o!@d?(9lm50qn9T1$l$&~ zL4)b3V`IjrSy_|Y2Sw3p+2TZcm~@g&G8l9(pvu*WWYv6I|BtsfKLc$;LqpjsNe|5h zMsDuLXSesF8uYE`IL!x0ADhS@W%udtab7jatp4D-M=NmoyW^9`vkoTXU$QS2+0S>4 zFty7e-f!r~&)hHASQ+&~=fCvB@?+RuIz6Xe;SwE>e7nlZ(8x&By~l;K+O5*^29BQQ z<7-KB3YTyxK1fZidRI$T%PPP(AmG}MS$_1;(I`=+IYyu{285AM^m|Ra^R0a6kGb6( z>n-UWd!kokdadWtj*}-tc@19Ggb7Id_1Fy_=lRe^mLBsSD=-t)0-hmRZ?XkxKO{ZEv7 z{>LyLAXLOAm7MvYe&iXRXp2jpG~Y7E?LTtv6VZMeRaR9swWet3(YRGz(`QU%L5D*6D+`MZ(omsae2v(04J1iqzCTFE5MB1l zw;LrhP$(cX0~0H&4EP1T8^?YScH@0vZ>%)6 z-km$6>x;c@bHh#9y5;-(2L@h)pBVvNf)^7<{Fic!kMx$57`7%Sk~Kn;MY`tl;&D#A z{OpFSZESsI082}rVi}5+7~VFl)Ig(EY>0IB6)-Zh#iGW7Cge$BrEa z?Qd?9s*D6VhpaY=l+lcQljXQ>4>TfaW%KV<2J#eEMpwgu{g_FQQKwX^4J8 zdd@$thMq+WitTK|L(Zc}jRF&0e{6WWCCae@>Z)m)N8jsoBx!BG=D4fJd{pbe zQMLAOs%PpO4As9xL@9x@+j-zbGHAfpIBY0GKqXak%Y#uYY;0FiGK!GVV$gt5=Sk?T z*Mr)>OtPDl_+|KOaHUIwCwM4~IT@}iLZj$L*QnfTl8UfXC zNB>4gu341x%J3Pd*%|>b7xt*N*9xpnEIqaB({;=5ql_4$IQYf1x2ttqRI!b8EB zPJD&uy$h_p0??T4vbpiwl8-*zb%Jwq5(kU?SZQgg!Ro?zsHk0rDy^+!V)A4swlxvRBa}LGARiai>Qc1|s_pcnU%m1K%Cuu0^m?Saht};T4o)@d z7pUfxwJ@_{gUj5>DoN>Pz2YA^D+diDmKVZ1ZUigd7Q z1=cC$RuO|A<_Fv<28A4sul~3P66MITmeFKjD*fdE7S$sf&WjV>q}TlYGb3}()iC~x z*4EbPqpOPM2RFck34%T2(kf@<9G_d)Tjn;&=jsO}f+J>fWY?xTL@ za#pvi-OT4RHRHuoANIHgqF3U)b?a8OxKjJ7gpg2&}u*K92v(8w2c&X9^Bh?M_w{4&8X`NMx?dVeV1b9ik68>f2M z&dPVTMgF+(eNJzBi3xN1J=(u3M$!r*wcZ5J@w!4dkX_5rSr!y=X1#^?xNZKIS1-Wz z9KH5hnTvrIG@1Z#+K==u@CnAB%F2%VdFES<+84Ti3F;L>j5=q^;DC6DF0iX-d`h7w zy27*Dpj~ybF|MhrD1M=x=qXew*>=6SbyH?s+ndX(fkufZj(xwbVOcZdzj?-R(<{*_ z^-?bWx`LDB~t44yC9hm5(uDD}%w1`%ZrIIwlYrVs=!(RP!#s&&K*{mla%=B6KfUq6lginSy$`+y3U|@BKa=U~KoWX&n5?1nKL? znIlc+J1u8lPq~Q1;wdum_T$o|%chYr^o{oXEA| z9ly~#z2l8^5PKO~3HRit&N=;9+GhWO+s`da40{UmgdW&ofxYU)SF3}04gAcZ1BB=- zFw%k{de&yb?7^my6WnPGH1SfWD_p6y=ibMCuC1!~UZ@0y37oke5zv2I37^dC61`J% zu%P)Xq(vPXA_+}{G$S_X=lvR>rrjE4uQIO{ znvaU+c=F`QF?^nqX;2QdQI0cD$twhJ+j=RCQsk+YHpsL*D!MKq`b%eIg5`&-{rvo@ zpI-nrKe6jkto7BlkUDrvhi^PDUz@9liK@ ztKbRyq2Fg`t%1QL10zs_)EDEUoJVrHRC|RGY&GYc0j<0q?>ca#fPRM)T8aVwNT-F4 z^HroCy7_O4O#-nn_UwP~?#6#Q7CCbA`1g`lg;q&~Gp?AkthspqSG6}c&b`#@b(j>( z>;jQ~C0p#{y&|=DO1};OQ&jgOukt}pkuLQ5uvHN6pguc|oIta>ILVQXZ_hORb{Q+R`-97MGIU9H@1 zYYpH2@c4QMkPYs}syFG*-2>O_x--%VAa=ldV|wYr*IXSdXhR|qwe=pr#=)$1qP~^T2aLTjMTnwGzJAG%+D z+-8Uf#&6C0NL_77RJ!Ch*J)NVoRZr!qoxC3r}tx*2n!QaVzUWH60jk9L^(z0sJE*V z1$)>!p*|AK=QrciQtEU>e6IdNscFfbx=4K|Zv$33x6pw=FSMlwf ziP0%>HmrZ~kfDt?q&5d#Dt=W;(OHL(G1Jx*+jjyzbH=fs0D6ANl?@;b3Fh5T{A{R- z5NiM%pfxm8Fp)2f*9%|(@!i&^)_AN|*G8_cK$n$W3jIbn_~t9w$M4wL*$w!=^5#$0 z%xiS)wQocpWZAhR{*rpJW8U76MDtpi8`hPT`<58~I6l$2@LJ>S@zSCL5%Cx8zXAg*_{59H%CHjfS=#3$cjYu%sAEAS?K7}e$c8^;!LQ42gT+mx;035g{m@# z^D8YZP~qj}#diOMh>m&nH3J&A#>U1hyIFlF0J)W~j}P4Mby^$-Pu74yO+ka`ZzNI~ zc9SN5Fw)_6Zf8w-@y3eLI^gt{?p(BQg#Tmrp>=!N+^mc(lZtX$2qAU|Y?uL9wNPOz zL*FyHv1|12e~RBBuXeT-e_a!O^7r3_xjZrGj1J-jfqm;RIQTLYK~^h0A?^4rwP1p~ zBv~#(xMlpUUwF9j{$GDPbO_{S$@x)e4!7UQQU+Z_6boCAsSVA5HNP4ust$pv8DJh# zPBflf3v!EgUd||*+BUM6sE>C+nG->EqHtE0=o2bQ*N*WU*T+abzd!Rs3a7bUl(xU- zj;gx1Wv!FCAN0*Z{>f*B-bbar*QH zD56SW6RQ`ZMC}@lU2bM%X3mp-1-)-8p1}&@N^E?*L-^@0VM00{VLaW-pVXtoo!nV9 z5VIs@=@qZcABlOFKT-)-DKvIVYVR}Ucz7BM|~j`(bP)lz*m;FwRnmyoKR?Ps-*G-!9+ntP4MS$ zkdO>yOw!fdO@bK6`=XsmBuKe)vqs-MP^5hV)FfvLKRscNHWX2lWFa48FfHBQjcoe&|wXmse4+Qof z7rtE`Dc1QGC9Y<<;&?7TNkfXGR8!Fr{@&2|j2%LXVOM4ZC+*I`q$3$u$--RPgvLPo z)tqf@?z3mlZgr~A$PH`n`^foIyWD`XKr199G_YoXV6kA*I5DifC-cVpt(NTXL7v&k zWLt~JI*`4qnL(33unt<}h=@Q!TEK1Z8l28(z+C<1AZ`24x_R#WI%eo_tF3nKrGZk6 zWEQLklt29V2=yd*vx|TI_4n`n@n$m8(w6GtIAWPs<;3Rwb-pMcjg?v)Nr9{oX4QrO z{%bTau8szbW@m1lo~fGLPpTOzA&3Cf%-xG)bY?h8v$JJoW%l788{V7d9s|ziuQG#9 z*laR;QGH+(+PCb}#$Qr(N_tpbs@QVhvf0d)g6g-;UhnGCLuoCS~+G)6Gh9v?@t0qk|utY%YA9a;6N(vm6XuSq+Tm4J}@&wF52=tEvqbQo1cB zDRh76nGjXgi*t`P1NHRuLM-Q6k%8J5whFrZ6jr~8fXF`Z^K3}}l&Q^p7>7D=LVdnnnVzvp(AwOw$ruo&^Uk~Tkt}d<3)&;FBFA?OU z2UhjI@BHDx_q*tMN3Vl9sLFuq(L0a<3Nz3^WJ581=E#<3{`Aw%u!I@FctEJq)VJR$ zEjy5%QhDyw`X_E=Xqx!@`u;9OiDoUD(?c;ZO9xs&thRuFKyLHskpZ`3rt&ajF!bwW zbJ|R39+z;|RQ#$~bfppX-Ne)dNQcm2U6r3oOG6fqo3K~n%|%m3NAoGaz)eM)O%Dqi zki%!sb}bb19&oO=a;eX9adCk;_oHRoKsQ%VJ$cRLp7T}qydQq2s`$-%x(ZL#9XzPn z6Q1^oD7X+Zs`cD%zMWg#!_;CVMMQgg8ym=rM>}OMnG_I0y+_x z0kzQ4w-O@{kZWF6ENjzm{&RwDsahgDV(DkVp%YDTd6PJMMsflK26;Pr9OHz0qIa~h zRR~epC_H-nxMuQqiS@bXcGv#ytE3#{4fAI42v7TPuj{~`J+rnu%Fx5MKp#RAE3LTE zl1KU~I4RPGo8m{I|22YK;B#0o&NgVe=(iehGqyuwurEk}>8_J+S@W^9!WnmMmvt*p zs34MuN0QQ7k&1vB4ijiQU^z|np;3SSfTRoR#b#920iO7`rEu!IUnnV=m6+!t<7+vL zXZyvF52qO*fj63UsMNCQzV@wgW$Rh7vgTzPC5I=FP2UCQJN^sQ96b|fL2a6DR>Hz{ z_s&g%0i>4eloa|mAod?ANb$?b`#QY>vWBz{`ua9`GnFj!U6yqQjWJThC)E+SxjyMK zARJjX;ccE~S;DfXoHs^sgR6Jy+2@y>J2FczFRMGzsdvy-9S;!ujZBq zlEJ(=HcPY`v$iy0Bw25cnutYwwb>?5C(e{ z!__EpE$~ME{t^@%%xu9DbE9o_GLyMr=5r4#0F4Wj$WsYS#?E{HxU^8}l4|CSR0E!!3M&=mm9h;kOU2 zL@R1dR*!}hSy0_FswwNY|Ac5lgow>+RCay)G894kwSyboHobgTbf|4nAo^p$vR)10mL%vu#&X-rix($}VO7tJ_W|WoE-U!CBz0x9xDc@P zaC4Of3)Ab4Aa0A4&}bmzrdIU(h_56tBtjQ04P=v*>eQT4TSr!ut~^kWtv73iRm{hV z@Iw_Q%881SXXGO1)UhYf<7s*Hhd?F#JeP18)PxH)C0YLB{f^j00Wf9IPZbI|x$9n` z9oCd@ga(pj+`)|itkkPKDz}A>>YL*wNukBlJ2sq*($ydUFn^L{^|qAob9kXFMJ%{z z2J#4eCg)#bbJ+)tLN<2Tq_?&IQhz}Z2bit!T}0v=s0jxuN0QL9-TxpG%2u1yvDy7pY=7hL1UDRA zg5!KeTDnPRb43+So05w0or-8>YE~aQrr9TH6<80<4%a;B6k=`-+fu|ahh_xSXd6VC zZ1_PK85t9y3xsm(xvQ6jvABX#9m!;IkCO>eyg9Iu-8s%ayX%O}wMdB+we$~#I$djt zrA!logQ7Ms1)MtaJWEzA44PkGt16uQr<%jvG6bJuHD=$Av=4V%lS2rDnKOpUr+_|z z^f=fF!00fi%r|-V)EwHZOq<)^q4{<%tgKot_sFntHie=1fR zi@u1e-$N^L#BN7CN0f*Ak`sXb#^}<+8z4j^Yjw24#hq3X5cw?H{TyqoUXHjY;1-0zw7H zSKw{fADTebCIMLeZr`8`7xuv_Nko27_n&#wFr&k2R7T;X6v>f|3)mE~RUEMhq^zKO;#nhF%cN6R+o*ldeOD6ZE zNWA6jC!v`PUbyR8@~w=d8o@wm?#z2=0b5lPWV?Y)WFiQX2X1)G2)X3_|7942RxBkCzCvoWcsON}mGbSnmr zA+_wJ4-L`TKw5H+*gz&fVilJ9ul#O?Q!fdUHYsN9d0A~@m2~vmK24mdn#C>{(KH(` z9)ZmmAkb_>r#Zm$VQaCI!)$PYWRtF*@MCe*Y4Vb{XYCs&bRn!ZFIv zQN-G&u*dD@U3Bk~f%Ac<|M}RE3#BOq+Db>#@HE}S^XlsA*Ng14J5<5qjVsnR&>uK1 z97E{mkXz_-2e_0x)9P$&$y4>ZfZe=qe9<3XR1k%d)BUThLLL{*2VKH5Lxt!^wH8ijMXZQs*)6K$Tj>6LAn0I@T;6np{& z%O$haJ6$ZRe`l8>h{7ZMiSgRfv|dl7?gKa&5{WxYFSU;Pgv*+_mi=%nVF0aAG2Z^Q zd*`EWQvs!uS7w6kb4XYQBo$bXGtU`@p(+P0zXy+~pUVxx`Y7j(2hff#QHo6K(zp== z-0bED*`0!og~E!riMD9?B`Z-n1my?Ss8n>A(4(VZj!%s4iNS6J8$4i4#L8|l_Oc;R zJ@Wv^^{j-7!^Om#ZC8@;{7L^!;tttefpT%pA_M8_v{|1gKf#`y3TI4SH6wVeB#h4?xGt4 zz3))C^@6a#u3^}0Jm_KwXR6ovbSr7UXDM6FikIUmgu*ZaqdHcYgo^&^ZNfq__*#gs zCh7A+&cB{z-;r?ID%0OZ@5yYy?uWc6l(UpCZF=nDtD{!QzwEv*V32^PR~osDyH}T@ zmIKxKV{pDL+}N1q>3_nN4U16SP2yK&VE&GO0MRE)dK{t3cN}R>)L(cG-#^RQvlW^H zFD*OttE^aiU#`ey7I&YUS= ztwWzloJg6~UIIO8L)L968h4;i!e@8vn2|AHc}jIKmu_`FTn(CcwGRCu!}(J_kt{BT zmd#hCZPRV#8aPWexTtc|sr`3YrH0^gQQts8o>r^Ps{>Q?C4L<6@GsxV_zyCXq8*xP zpjOL(e7Y;E!Swzs{h?J~0?XpmZZRoH9|m`zUXKn83%#9+ir#(hxjIFt?4GbUw8G4; z_EHaxd*4jtzTLaI;3vNJ${OJ_`xBS0k~x@ z68&mibos~g6QdS$rG7bm*=^c9qt>G3m^U+LSp=n@7chzye>v|BAjrB2`$qv=Zs&NJ zWu4gIOo)2>YQo#e!#jv2n70aPptKDAZ&xLuj=Frli? z>21Gmv9v={?J6!1)IdLNjuUk%chHg5ylZ3H$K`ORECtmeUBg8S&qOWwS;~X+@8Vdp z|0{htsDzX?dk+@BnKUd70cw*LA&a#Jj@7i^8NR{jD*?;{(<^|n=IW79>i&|H3@P^_ zN^XibD*rLkFIO#48ZAE9vaOmMPddLOdi3Ik(K1dTb)Jq=oQ%18AnflmMYA|;d=?fK zly+`=VKpFY&c4neD%w?1S!ux{f(e))3qMaH(t4bgXfs#Gc zj^2=L?M6B><5kmw1j2n>A{X8VO?IXguT*M+)oQ=hxqvyKw~obhP%-K=pdb+YpeHD@ z#80bgei;Qell-tcqxJ(PQm%K2092==)}PAFg|RGmkR3L|zQs)Us$cmx1qB7-f3u$(BD6a(-mmm-Zp33QDFGy* zySKLr5S#U6PYM}7g42Wy^oVaP>C#VTZ?fqWYx;;S1e2*dpztKf-R?dHeu%C%7WO>Tv&&pOso}@CDMl z>Fd-F7jy$L(7~->h@(urLXZsxR7_DPp)ZCq7|abdV7gD{rIkGkT#P7q=62Pbdy$6` z>sLu}t6sMI`0?XB)+4vNleH3_vhK!pq4>0L5b{D%ksm?v40Ew{;!qG`#c6ds_j;zj z*Y#Y}6!M&Co>hPmkFaYdVE*ER%*O?AD0$VLW5!5!M@6iY@j2HWLqZqPh4T)&SZT7g zA`j>CIzQycWRfMj%&x&@y(otXp&&g}eqvPrHWOT<-vrY@GE75P0SlDTtGB=<3E!8b zfq?-TK+~@deu>i&HYyX$`F{M0$erRb9@6W|Mv5jHrRH5Er-oSELqlKhJH}63ox=8W z2{0WM-Kh@##y^?+IL*hB?Rr){c^kB+*U3yM*ueX+OeKPOM*yzC%EgOEoV}8XKK43# zIkT5_>XHrFMyiB}AdPrHU|`${G5aA@HU${MsSX}|36jGIyv;yB1qiCy{$&<`F|Fup$G zw^XvZD&d62;LDqvkBPpD{~JMrFl>tRBJM4;HV`4D$@nVjdq`6|H9n7U{x`Q{b0}!T zAVBcnizXoP5I>i7ffgaz#4+3mp4I?EPJ~f(PfmkM`2_?qUXk~1{YftxuJf2n7+6h5 zi!`tXu-e*!`C?w!)gj=JNj`gE06u&PGEfFg5D8%zBeV5uj$*>_3NnObaZd46)aKeH z(j*X4E#KI`o1PLD!OY7?L$FOu#1H`PWe7nYh=7fOwLyT49?StVg<$`up<_ zP*j3Eh5`!N)01mnh$t*PB;%q@lv2*ZKQuwlt-vwS^76v2ZczKg;RjSOho#2n6X)i+ zCc4nBOa5zBJ`r57twdL3hR+Xm0=wTGOC|^~j)9)O$)8E-{>q#|TG<57iV`VX!{T8u z9}{OD2%y)BvF=#oSfW3oXkEi3PI}XAK@V^5R*X{d-FQclqM4^i9Pq>|MFt}E%cSqX z+?x-I5A2m_<|Y=jrdoeE8PXPUAJ$l44PEu!r{|Zw{r>UrXd;e!qN0KqBQzF~J_(Y; z@GzLKdKA-a5Lj9+>+_+Qi>k}fqR}Zf1Iu~E_3yulchRhH+6~hlX!GwB>%$^y88S13 zXlOx>$buL^2pD9J)@Q$8AQ=<$7iZ6$x%A`!JyFrJ;Rie}?x=?0TJzPh`B+FM1~WfC z=l2I{K-Qeca*hz|EwD}k_^JxS>=6*bq|F(qi%JVK&Ht^3Ci zT3xZ8iwmZ`#oe&qN2}Hmq|W8Cv36@QlamT_0!C>?t)m+aNE2zP&ni*zo_=4vkE8gs zJjsPJ1K0t(fBm*ZszS6|`t8qt=SPR&s8wG;73HuFbMtkH^_y*>fShl|(&xD2FZI+4 zb7wO<2T%l&>EOQ4kIuEUoZpdFF*~55{I>%!NkMO;%5z2)n1ruRrd&qZm!K;UOuphZ5?6vvR8K)bQU=YHC3w`rC`7~EH!;lIDbdOfS6 z-!it4`yK=-Itk~lx0v#zH1Ppq)3zUGOiR7rUgA>X`|KIKFp-hb0A$AyEDIxIw8xK| z7^9Oz&2P*=MlM4P3|$LzudM%8PsV%J;On|(Y)1j%lhBBJTq0?e9Lm7moG>%+(3TNUofp%zi`>VgaYECkAxGO=$w-i~@p5s9 zo@35-WUWE1b+ZcCPJKPkV_pcO7<(tcAhlQ^GI)e^9%!b+`7UnXI%kj9PO6%%FP3at zRqa%o3NCqb%<%+66R#GW{%85d60g&4WHGrttKgIHiE_DDI%6eG>Dm<;Khr! zBB@iXDd3v#qOOnx$9L<~FN6Sxc~SvENLwo6xODGogzz=OhpcM8>c0siE|Cw2OKBLs zY*hrp;iE@ifBkv`S0r4+%QXFS@HBCpKH>p{|k+ z76bSRh$AYEpwe3nO8my=eZ4-b`C1oYl2NYJ(cl3WVQx@u0+s5o2nS{?70r9`x zMIHJ%_K#>j`^O&_$uGe&25b=*A8CJuz#h=R9w~Lp^P-aj5<9*=;zC zYvID*@IPH6C0`V1jEuhU8n<(jVLg;}@bC2~Mx?}E4HrzWPgcPI;0osY_|T)0k~=01 z(^frsa@Rf7BRIQ@!jG>bEV3b05!aYam-!QT0+V5)oXsA?><6qj(fGjrhC1+`^Wt@_ z!0P+W;T47uGsnvz8(tfgxoykXnsPBZgpB zV#;A3?32aT5)2k_r>V-dl=I3nvt1Ndz9P&7%{y+N&BH=6u(~fOgwnj zGjZD!E--ik#vG-PCq7ii9B=rPIICDD8sZn}Xs&h67G4 zWB;AY$HX0NA805EZ@))zHUM7!O4D5G{58lhY2r&?w@EEC6{1ZXN{f5xrvajbArXxc z@osJH3a!+`SQVL@(bNnN@uQ}A;C>R*EhNSLyaD5Y=+vJNG`7FMoVb^}JD1g0&N$o` zbr!tu@-Q05qX zQ)yps>=&3iP)_%bqiF88_1ZeRx^*PYaqeBGC@3CIk(nn1H|aRzWFuaa2@XGUAr0nS zPrggpCf~socTo~!yZ3Ns3k8KgD(RgpC`V9lAq^`KM99Fz%d3nbZkY!fs`f`da(ezL zu=U+?JdtE(>m)o_5229(qcQ@s<*ZT?E(&_!h`Fz3_~A597-45yQ?(p=v>$Hy`r$n; ze!!o~KR_`f6#B?n<=jJ6NdoI1X<_^;9UC?x7^4h=OazLh8ULw zlWIQPKG!n!YyK3V!b=6Hp_+}frKLe@B)u!ySL`XdV(P&p{0@X4wukY~>f=6v?fF7_ z=Xd^}w^3#8%*TCN|NBn>Xz>626$QnXixj*5>+3%s9DMiRKmY3uM9F{s*-rES@asEv zT0*(dM|jyDLIUeDqi-g50e3VvY0rzrXv=?i<4LOMy+=n6<-;u&o?IkBsRa(hVlz z@&TgSL8DXs>#x6FgppfRjwv1UctY+Z1!h>{l$#9pdvj$S+?v*qz)K(c?7J8m} zlcut!CZI34sn~4H#|%2$@}jc%YP#T^m_{AEoGFU-93T_ehds;@xx6L zTf6)kdG=la@*=m%pu-{EJ%HQJd~$6Y1^b`d?`eL;{YJyn(_kJ}bAO)-15DeIzdL~e zyLD#e*w4*-nf5+v$n5Q}w{PFRTv=CmZffTmZs$rs;w1B6xD^16^}iRS$+Oqib-Dig zSP9DLQ5l{ZDE<2D5s%X9Q{RU?$Tz+k-TwOa9r&lNewXsy-Xu=4c5Zw!+UxS~+x&c2 z*J2Vf-_HLzwR5X$F^Pp2g}(o{c>7O`3rba0w)8v?sjR3~QdWC5qUxb6V|)2U*bee` zZ{PjT=0CVn$GO*}-R3%dnpA6U*nXyReSsj?of>=zS|;5+*~P-m9!Yab2Dnq~=YJdnUrboH=!lufh`F72O{)6nf>CS!`bg(%x3Ybs2}5u8 ztY(Gcq6=`_nBScFQ{d-=FnjHBh}lhE>`%s^z<{KPOFRJ~Ve$40zr;PFpuAj{Gu-;r zD9gabUA3RuOysr3nbd|k1uc%HlinW)UIA`YGB!3wT;UbIihi_ne|4=^;osv){pLgb zqxR3rzdkthk#layb6$IwnV|8Ps8^TnB)5I~7Qx)_04w}oXpHyX=OIrJenwF4o0!r6 zM7`q~vCcRJ`t`WcGixzvt;g>PGx^nVQGJ>}UESQs^<8M` zS}{4KRSBY@Mf7GqGPyeIky(EIYMR_}5g2Ub@T$iIOy}AMkmxIMT?fVi?%S<_8nDOM z0?1Gn$-n->jW{ak;~P-gb;|CLrTtU#ot?Yi@BiN2_h8T8y3D_rxjeI$*`d%O=lx=4 z#rCR2N2`uj&U_WKkdQiIBk|Z1Z~OKWHi8U(5|j=VGNk_}_>OWSrkOTPORt~uA)5Q^ z`S%m^Z6*n757rZbg9qA}v>%2U$X~*8A+H?y5{MDY%YKA}@~Miy8uo`2`#E~t>68D% zPkIjiZ)tQ;WU!j(VWdd`ENsFgv-B#typAG)LQqb@;y>kUqwnpbM?vc!td^#3N-09> zf1vSb&vB=H8-KYxmT2hx$}z{c6pH(g<{QjnEB~m^z2-&9L$$rHwX(dkO%xvx0w@8% zxN}W2uzRUTPT1e?+N<#|`=URM+A6Z}YqtJYXk%FFHd?jH!P3P4+|Lip+i9srqUSu= zBSo%W!t`7`#>9*;_3t;sa}Q~SSzNp)HWE5S638=mFE7jjcWMS=_u#0y0Fs&G{_5$A zp7jV>Mk$~IObjcdb6A+{P5b(FEhNm+(vo}{KEvkA06A^oaBA$4e-gp`p2H+s8=SS( z<)G_Qh|35A$*@5w5(aSd_OiT!)2~$9@H*sUlUjjQ;4qJTb-&`@V)t&xSH}p-ty8WS znAC;ut;jqzE&iqFY)y_+wic7bg@0w1cJM?g_)w$hg1G2Cyu7UT^gN3J=o zgnS`~{Uj%LQTY8sQ?jK<)gn4FI~rbUhnpSU_Kt%XzaSRpmO~ASN54`anu94*FM0`W z5HqcaBRJRlGS+~GUSn_~h6?1T%FN!2FI+vbeo>e8P}-0 zypZV(Ef{`Lw%0Tzc=RQ_*VDf-YM&|BZ%dyW<2Ea}C=-1CQK6iLMrh8FKiJu8 z`})x102L2xVhFXBfNMBv8~d29T(iarqf@R_)x}9Cz*>e3S<+v3mU@omZb!DqqqQlRNjH z!r9Nz8EC^_{~@lKC-psb;+@Cblsk1}^ak^jbu(A*c(>4nX|2jeDvW*!82$7M`(B+M z7xkxYq*O3&Y9`Z#6)k;m{bf&%;panQ%xk+#(a&n5$s*Thk@OOzM<^PVWVLUZCi`#+ z8Znva*M=t&FU-#&raYObQ~h$Ir=)NPSsN(BvCMH&*MBK2I>S@N+v>+WX-E0*>c6XN z%&^Fwy&K1EIjdZE;^j)qDV8-KmjGY3I%oE0r&!vUR$U!9FJ52h__~Kj@U%-rMr88T z|Ha;S$79{Te_v^bc1BiiBax9kG8%{?dkfj)A{k|qQkkWYEqi3=lC4zscG;_hGLpS{ zj+5%H@ALgVe?EWxuGj0nFY%e@d7Q^O-^csN_odW~3CAqST)X>d4=ybsvpq<}slMDk zuhC~{5gy%R0J>?k{VZCGB4fGv83dkaIdo1R5L7cfK=k8m* z&FLP8G!ys)Uqo`&sev~1 zLQ18=5Vl}2qF2GPHj|I%-R+SNsI{@Muh+=e$!(uh9Z=!nwp5c8f2iF536y9}RkaOa zDU)r@k)%2@Q{&or)WH z_C{CYy<}C^cMqKil$@!?KBQQsoqRwY?bDL4Oq`cVp?NTUrlgYE=^EO|LE#Z3MUX9q z+)E4Wb)fJD)iYruA+Lu@VZf9;LEr>PbteO^2KXwRvU@1N7a?K4@ycy&h$b)|m9FX+ zz#al3<17S$oJaL2utB8^0Uc7bkhwPn@H=F!dA9uga)i-cEpu*98s}>86R@4%trc|60@y88Q1Ko@g#wmc33h0Z;Q-*w|?t_O*jN&GyuLhi? zJFYLKw2e0|_Y9B8u{YcyW@h-99u$zD=>VIeP{4eGoIGU@v9pk!lyzue#M3h=22k+A zP>nqhG%N40aA9eWzT^_&Z+}2e<8({ze6+X_&rnE-ND)&~PE?XExSUKRrUOT=7&u>W zU^y>_J-x$>DVKOvHiY>u6mq}Mwsw<#yeGB5!JcdIUBvD9N4iAULpdTmxf@Tb_#CGi z>3dG+uS?XyNqgW>c2vjIYz^m$U2@d1758|o@1Dl?M#l!Io;LrU=*d=Dk%bgRCMXJC zxz)n#(fMrTs$!fOz=TMqJ>zz*W=k+ExNb~Aelor;qWEZsW@w}0kMWb3m@7f7lS0gU zT;(2?9ggvuS(py{!kNC=eQ&bxTVUb$3*>CAfgpYS&opOtSc^ZUasH%N^K@Uwpbzt> z?~49PkN3$dnaCw2{1{fOwhvh{Ztk>-Zk>Kz<--EHW zI+fF91^%p?=PCt5yLep+EXx{OBvcHJ&|Y9uXRRyu{z0JVf1Hz5%2qAw-YaM0A3?M0 zxzDZ5hZT<`%3WC2J8)v@*yx8py&b*{{;&);*L>yLJCrt>MtN8AZ)oTQ=-@BnV=pp* zClt@z5VaG-OzD>7vbSo9!0jB-rGN%Co!CKQsqWXJ;_&lz_b^hBcZz*_OdUq-hxh5d z*(1%*T5HYh_T9RI;nBPFjC&VIcrSH1*bjG4?7nwa%ahF_h&JaP;mfxwjq0oiLT|r! zP!IZXSp4B$7W%4_U;EsTLG-fqDGk2Os5(SC)dF*UJ_je9WxVNMr(zto6D|AvPNItux6^VUTi#QlAhMnc6N?Zc=SxMBzO|#aj#!# zxfFeq3mtor`^kMwg;3StO4chxdRT3i%~WOf6d$$<{ ziy+pjQuMFq+4ruF{?kku-4bmhHq|fo-YG3l<_RMeLM7@$i*yR`b!Px$VD`U_~QFaA!HWuN+#iOf6!wivMwViv95C z^Wx1Ak-JM*S@PMgT+5J+Hy&4W#L32we!c97^G|{=37mhzu*pCQQOI$Xgr~-Q+^*9! zB4&0U^aLp89`?qM72a?y$bHK!>%pRORd2YVkDYApb8-5;3v3PZMSDw* zlaLMR6=**3aS|BCkd2m;U2l|sJ31bjU^vcV#om=5xyt)Y?}~lJlb35}cYD#2*Gw&X z&4g4>cXkc?&v5qAaQWuW^9S9@=vh7|+1D^$nD>FhfL-q4%iMs-k|~7-(}FrXsvdjR zsp)ee>vaQCqxM2IqyyhCMyV!^UT7G1bi7e=_n;ra_{rVD0^R1KPIqrROjrmR_KPld zEa%nbQq(XHO=G+K^)*KH{3VqrL5d&jRzGpnLix%QaoP(i5dEJxX`p;XZbSd&+vP>7 zfY8monhnCP3)vh=jtA0e3!F`!C>J<56n4WhsoqmJ&M&`e<{0z)c>}>p}%g9x32DL_ljAcM@!~F2JhRSnXDf^z82w(XE!e$ zQ~f@D_vzG!`5|scIf}f~6!rxuKxSsD;CLQp$hDJ`(u1rr@*cOWLBy&kYc0uF#+?}F z7+u}_NikkF&MLarT^^*Byzbn1S>lk$f)9;AA2ci&^Xcdrh+S2bqv(d=6) zZHnh-+=u)viJd#QvZs(UiR`m9SyfTDF~fV!FrL*!1meI4s(+HgfnH88a~27*Gu5jTC&gxtUV8He6tM zrQn8&ij-lvuGoQWjk%kf5(kKJVT$x$1Y`nq^RL<`rJS0OZKe9s)hP2=H=iP>_M3IZ zwA%(ZtG_Av#DYyL8~ml+3AbUh8J#{g@tn}eW%uSCHr1Fuw)tr74u_h3MXj%o`|~=m zDgF=;C<@q}*oA4hLo$3_*dY4~No1tR=mde!Oj6TR@ti1a`;6k&&6?=jlJIxh__Yi7 z@Sl}bcH^w=wbjq&2;xH{%~e}cUU5!!I# zYD});Kg%K7mXgkw0$DeWLTpDSh#5l2ZhrOtn7v|sNBVI`Nv!oNvpvzxz>qMNi5c3K zz`^Rmk|a>__L z+`u-gmhodjt&iaF0o+?2wYL@lNjSmXEd2SEy{;Y$$1U!>UB7Kw(N#`(6@FOD1Z(B7 zPlD-&gniVCTb8_l;l4mZ!kn+4SjNvVpRIbxTJZq&5U?ndaJ)rQ9vU2cuvQo>@!TwSb zr;Y9Bi8gwF)2p%(k+?5+L9-fv`Fd9aCu?B9!jk{Wfmyoo=F?PhFJ=+JYxuZJ%EE%z zrkj^HDNFHq6o_K_B{QzxgOlppxgT}G3exbY)fM$_vYN5<(_`n7nZelJ+j)a?8h6KNoGeqU32a3H;x#My$$;a zl8?hNS_J1B`Wg>a@jrci5pl9WTHTg_Mon<4TPtZb&9Mk$TE} zSv#}`5*#bI)KB;TnceePKY-JB>;{OHnK)eI3GAqb$VLCJm5e&+d-)No|T`mp(O4D~wo-VS;p0V0?t_ z8E%c#Zn*=QN586|m{lwVXa|WBVdw(+}BL1Gh`^C4muu^iY5Tj9d2iXcOGVeTD0)NO=pOApe^f7YT*5~|tZ-UwH>?-r;#9%3CRseXSk^Q&88rBy*HcV))?yC2{Ub)~L2`^b)@c9o>_bMz+v0qfU0U{G z+~&_;+mwzDJ9M>t{I;AwvYtSDEFp_*6PJA-CG!A|Tg5@ZXAi8wh;R`?SIR72B!59& zkZoFDU5yNz9>3vm^5h(dN#l)H4ZqKEYXkgu-)EX*YW(!bq-$}*d$svxG;jBwIXrY% z!{6W7U>uf}LM}l#10CbpPKHx-o0+Yz8RTDj2kcjLR(g?od6#M`&D53EgKmR+?UpEZ zz+S1Cg6ofuuP?N`o$n7C5D+a}dsJ{j$Rqw-(679(T@62!?(;Z}|A31v$BZ zU{rAW5V-)G=*((3{vTx@K~nc4G?AhYr0kM5sVuIz@ zMb9AdL(lQqj-ugQLVLj=$e(|xtlV2jeuEHz5Ks`gvk!$Vrv`(uqCZ0xuI-RzNR_7p zt4+Z~|8=dN+KqOo;~=J{F1z5Gd?3w>PhqSTM_yPnb5-hlGq^q8w_m}zuFF)~yE<5& zhf_&q{DH$itW8%|{b$}E8vDUV0Ga3H_Xl6GI(I++JQ2Z8rFI%$@;)Cs^A2*LannPU zs1y+G%%W0S2(c0GA}Wdju`~#_+bHdglRSj*#DR!2L8=A7Al#71cL6LZVlqL5N5Cm9 z0WkPxs7nT}3{I%*YBgL1Lfj5iJ)Uqi1W`jmE?F58e_QN(kl5*f46+?yJ){F&mp<)< zeS@%ZR78ZDw>au>0gw}J0d<#(lKzzPfa~ElYhywPXtq_vErC#ds38M1fmvQc7+-8R zHzF%RaA~wX4h4s2L_}`z_1QlA=uef#tW4GcvFhO3Vr4UT$=oDwdZ$AjHWdF4Yzui% zkN0RP$6XYD<>!45b8wKrfR8A$?v}wv{t&7P0Zj>4F^LWRPyvS>=4l`PK?`Xj08VCC)Ng=G7I~gM&aod}u2fa_$^zjH2M|Tnz7+ zit;yrb1DRa0F?MZ_$SyuI4SJ-x*&tjq=FXEj3`kCHRJOkWozO9ma^Wz@#9O;R4Kss z`r?5j2t@mOe(!A`M63bmwlM$%b^snE6Syc$5Of+4%xKR7GD(*%Tv%3TC*0$CoZ)9j z4OGPoDSs9~VgRnX84%QW08Cpjp=ph9Hi)$n6-L{k#uQKm40w_VV9rQ^l%zbsQEqgg z^tTfInb3lh`36ZMHzLo5_pLKe)P^Z_5PE8)F^mc|60RNfK5&MWMb+uL5i_08b$PuD zsXDAVEt_f~LE%Zq$VU^;R3ECj)fjAjcHUPuI{OT}yE^AZ&sVZL$VLgyMQ|A@iBdd~ zXXR9S8*f4&x64%ywg<(s8Fg3_sLyr@dTC4d>EI?UBiJUFFXTMe)zVDB*)G0Qu$lqX zY4pWQm+~Ngk587bFzf=!Ms(OH7YRfph{Bw+=L7J!w?Wk+N?QZI{)6QzFy5k>08Xmc zz)TIh92JJk8|%xB#(+s{13)g9`w);F%U!ur-8fEp{;nL_)`0o}$ZFA^5%rXWL>!!L z^^In8>0!rw2Ed0wx68q7(@Q}GLh2TV7FLSmkRRV1z?(wexWE94o$8vI039i^KDtUJ z7!*2=D|XGhBJ3o?vYuck!>JxExo=SH0VL@D$kl4K3_tv9GqdQ_K~66{rg>kPN-nJ= zR)JtwwktyriO3pk`6w+p*d17_pdY$iLK1dh_P%m39IpyKa*V|JLa_uq1IB&!^4Ntl zhF0!4?T@FPC_o|30@<6xG&DM&3xN-C^X`daS}BAehGZNnM$3mRLn>etALr{t0DDCb z5$uhChWH9a19)JYi-&|RB89zyN9_q-yrl?)4T5>tbuj^i{|ZRe{z15Bxx;XhsQ>r} zt9-N7Pu%&2@-HtqX5HcHvlGAcVE5kLF?LLZ1pIQe@y0!9n^U8BGP!aX6t+=%PbTcb zra|uYB;TvJ*b3$wi}m%nLNeN)Pd>7)BdV%upMJeoRo_R`2eUpz|e7^9W8{G1I)d&mMZLG!EV*4-7;*mUkQY6PCB2l zC%&BFssjC@o#=o7bub!zH7JiAXgk(+CNqEZ%wRmw+@kgdK!1lo*FcEspiKjJJG{^7 zy2046gJ#~X+9LtAR)z5y8&hK>sqzob5HzZum?FKUB=|q%X3Y!sT9bLb^o=+lVtTc% zj!brw=6YlMV_(Y1g(p!}(crl#@%*EvOy>z3EUyI~Jho}~UCM0@c`tFl%8cRaR#JSB zKrX?)x}&=GY0IGym!9Uw3UbMd!&h5XPD4G@8z|gGkTOKVgqAlztx<^yqFIFk-65?ef*nz2SorJK~MvP(y-0m@s{FX z_(IZG$l*iEG3n@aJR~b@nP;_5c_mqpkLZ(k)zsLD?pyIewzuZcn@0iFo8y#^p3=NH zVr10vXdiLQr;6p%(NKf1vweF%Zf)%Q0|uSbEXs~?DbGW+S0?v7yn1MZ`GM*g7pepf zK%AltAD{+mhg5Pr;N7;!n9;@|EN!Pqzqt5OCCVKK0bbRFI=r+P6>$)48Zih%ZI8xl zlTQn8hR#b{SiI8dLEgmO3QN8@JOsfmC-%6-X@ zjwKbuN?QY@Ak}YGu?;+F@WE}*vWgOZoPIsAW-8sH?a58)cHL;Cs&RCm*8Sa@vpYF% zzB^8H!fcPh-I^`4e!-v`$uXvU@m<;#tkn}2b1_h&K^j1BQr8)|Ig z9xvNu8hzQdU?8St@F;Y5MMUD4iS;#IUz-ajVU=Ir-hYxiVq2PIv4b*lYFb;f7}e?{ zUtzNMU31(rY1L&33F$ysO>9yKWdmaPR?Ski?KzbO0&DF9^1gQbgAo*ZBiX%oTr3KrB6&U{5OTd4a19neW>|0-Q;^l96P|#i=f#vs^(4g{GL$Ew6vl8uv z_S>f0s3j&zJGQ4`iy`(6?8}Aqt5@yV7F@!;t=H!}SO8B7TVcw8_5~7jWH_OaLK!i< zuPjWmNFhRORPNjdtet4SfDOD23XA%zE2$Ft?ZG{OhD{k1R>jEUQGnM3o@k+j#6+l; zk?97K=&e3wQ*aGbNdy!Zu+3_x07z{>EPN+f`g8~>Gqn2*HV$=aBP3;OL&Rd$3?W}r zlA>muoM0H-?}6may**Sk-#;U2$JnvX*Mzf6{rbQI+m;}ohk1Q)<07n0qp9j^!<`eZ zJvXkYWkDQUca7X1RBFBjxnAl~>TAJUX?6^oEr}1eK`b}+iS4bD#c}-i7iBjv4ODAD zdfoyIZNOP3h;XUkHz0aIM3>wL)$@P=eoBu@V?blHLtV}s0K);5l>&7Ph6e_p#G_KE zvh{^OYB}>Z;`cMys4PFEb0LC?_E81$Qqoi5@J3{Z5=Nul+QvBW%TC4Q@(arxAb~Q$ZSgH^6L5BE1wVo9TxJ&V}cSf&=Uu>6!VZ#{SIxnobdE?C`F1Mc3 z*R6NBXyV&)J6HbjO6tX3*?8tg=7=`7(4yJGB;_kk}| z{{H?$6Wt;Wlm=DkIG3e8Nyuh+7@dvTm-T~L;j_4hCGt!G6QoC`QMjJGvE0WyDG>Du zcb@m3%dmx_m9MP4v}KSU8K8spw$NGb(qjQ}rYFx0tM&;`^5qZuok|m@E97aKIqhWj-jH~I zKL!R{?dMi8xG*MG}gtSpc zzecelfY?-``fh2uinrbs7R}1jp#O*lBN+xm4A3|z^OG;D6(_T*SbQUMd+$}A#3zAL zb4x{u4Flqo6|uO30hd=%2?k9=&_|EnH2H^xgT2c_?TwNGlO$QgkwR80IbnrH?N$w# z)4_b%H;_HvK}tpP>RVjl7#QV}n)l7WQ%$xy=n|NktEdE^E#=r(BlOR52YozE<^1Q~ z^loeVdaOEZcu|k_DbcZ8C1(xBfKD0R>AFMm!C|=Q;|d|up^f7%#e(WKd{jz6Im-rC z#Y}FsyRgK3Cv2!+<=##&2R+w&lSI8|%>$tW-QUqiS2fiRU|Sb81KzOas>O%1rKFwC zgjD>wPdiMg8cy%J74t3&(h+j(vM}J!Z(DdDAXX_cI|50n{kF%pFc3e| z0P%_Ze?U6}fywx5`$-K{b{ohI!5BociJgK_Z9a#p{{Z0I0#48IZfLDu|_YTnAqY_|k%&Kc|;M2rsCU zLQJ6Guvww@AbC)q0_8AxNme#d2x6cR@fj4K0(~>E#wg$8A0&msI|#3#mq6gm>OXL8 zX1=1nrdM{W^DFPD1cQSoam6YYZ=%SXq&>UwH+%0L{fVf<#1Nzb=lK?S0u&qud;_HK zhYVwAU;r=%MB)Q=9H`>54XDh3jSSTWB3}`RQZk{EP#YLUBWV`mEda=73su^11S)Pu z*y$@*_MN`O!e1o3$$U8te{wMRxj)^k7v0?ovpi--M#l1VkbSXJ=`tcI**5uRkcrDT z{6wqL*Ae&l`(*#+BQFgsFIE47dCFZtO_)%+qE;V5zJU^rJAf5ff$*0>3g$}zdIZo9 zZ(>k)D0qTC$MJO32mxAd#sUOm2{$c50zh1Fe^B8Q*bW$^k$|z}HweqBP1fqKmac;8 zG2Xt7PymIJ?0YaFqPYybh7O8@j#?sbT(IjUZddXCK0j<~Z_(<#-jXPc+=ZZF$ti%M zO9cWqvf&;peh30zpv(v&`GM{R@(&JyWkYShOj|BP@fozv;PBijXAQh`(lybgqs}Kl z#Gr2D!~6G#i{YAb`y;EXTlb1iw6C&kv_V?)T$Q-p&G~mR?PI-J8jz?WB&Iuvqy{t| zsz!o|VUR)y{Yp+qX(GrWRBtB%&18RcJ1I90R2{;;yVlz3>DMOr4M69(tvd}8O0Sn8cUOo5jBP@dVTaDd(@Q&1@Wd4@HGPD*zAYL9opYd!Q@L$M~b^2B(2{ z6L51SAa(3gVEMVXvF>b4l`T2*QxAmZmb+azi57=d^D8}nGH7{DfS0OB8AYNo7s0mq+i&aAZ7Fy z{R6@LHk#SStv0)Be%UK-bO%R%*@=hw>H&2Hp6-|O{={fRn*T~cWE5(M2XfSMC= zyg>fR>1;l*5kr-oFob2X6UtlD65?Dw zBw(l^`Ms_$c9qrnRnZDALkh6x-b6p^EJ$Ne0v@n)sO%0=Tp%(bw%Rgq)+hT*Gy}Pv zmwt$GA3;-Bquer0RC3-boCGWU8CQ4nsh53f(&>$si~G;%fh`(D*?IJFKQIAG7*Km$ zn?M5+7!OtlI5B&j&ja`rs3Ar+5sntJPxDY4V-6*?i1`2ttLPwlLCtRf^cENo1C<77 zS=a8ZU0)_Q2vHpGbkT&(=m*}OnXs%C;(G;Qb$2p0vJ=)16Xh=fJD~%ealyt=hs-95%x6Ms6#FCJ4r* zX;~nS+801_ZfD;$G&D@P1A($;6TRsx>tfzLIX@1kQGe}Ha+~Z|`$T&Ijo~j~D~35M zK0cl^o#xoFIOv6A22N9{>6YK8bPC+}EzR8P2k~4P?h;nGFEhcc62)_3zpVwj8-_nK z0<xc;bRvsHCcZ}5XX>UN7keslRZWKK|hY#ihl zZMNAq6kFFn(cb*}?7GvZgD%;#m3!d zepj-f<4?uN8h-hj{y}K|TccBuF<)=+9jmCA(M;0>BOZN%Cc$#i?MjiKPqgc@Wt4Vw zhE%(z&z&>F*D`u$B&OvZU3FRJ&EzkU_%L8tcdb4AReFKWaUV$UwYvrxhg5CJ*1q5m z_jiwf5~=6@oZ|%ufITsG?wh|H0ZiGqpXdUSi2<*2cCZyEzB0xfeYstVgo<0Lqt~$m z-_Dk-SGcD4M#5tK$Mjau-99IEtg;VLbM=tqdOO?F-b&oN-`7Nsz;uVa%K>Tgu!4SX(L2R9JR$*?%3*gsM&IKYFfotW82`!7#2EFyc|) zI}|$ZyrN6PA9W`oIDs%Qzd!B@J+C)kqixH)?PZ|JGgc$v7-Os`{$P5RTkd#2?eAkT z?ni~ff)n^iCYngi7oD{FGi72o&cy_csRqp1)Vj|?-t6CFF`0%vGvG=~Io|okX87d` z#E79}ain|l+6`qs?`#qrT>DE#m2QHb>~`Q*3@va-H4czlH+u^Hzcq_7nOEt6aS)i8gY(qLY))q`+5fBEyya@CNBh1}gZq`1)VLPo?mWJ1XV z?Oo$x?|M89fQS%`0robLw5%^y*)8H$8cXNa0-C=PZRG7|duR>wuQEfIjX5gyKLC~& zjiytnNMANWfL-+I22OHz7mEAB+8?ovs_Y~c`BN#x1pZY|y1do$OD4dC@an#DWU**R_v!W=+|?Y)!m7 z#1481Kh6%DA{;ch1MWL%<ix~^9jaS}(03Pdex<4;5NPr5UtoW#y6{%2^ zr=gM&ps_3FRHEF!S`hqJ70W1lYN^&>&y;TGQ{o`?5Dwl*6a)zI|y7|MUd zi2gkZGnx&kCmfYDk0%c=%{6=mooP-N@qiUe<;bkyjxT@dyN=%~Xb`C$dL;CAwU`tC3+{e6SU2j}iD38MHsV|`l9mY?Ug82gg z^*W~P!*q!l;|-ayf6v(UM2}4HMPNexUWM1gBItk~zX1U>BF>#<#2y2{YQ@|n zBgSyyKfO}qZn=9yuKBq$XKP*Zr>YFt{LNRIhkOgVeqM^UIM%!}6&uNHc7sa-jHspI z)YtBqYda|K*qY0Y+3h$U@TZbx+KW41v~fOIZ;!c!z>8GCOb10_S(IHMQER3sq>|of zgg4Pl@(!cF3@Fes{K}unjQqZz$oJ3ln7aWOf5KkqC9J)XuXc93DD;0S_IvryDNU2% zi+fIO)%HMxXK+dz*kNC35x zepf*ErW9K1uUHy)q7!zS2#0)_AzAXx_mvBurk2^?FwbFK>Yks-3 zz_T>gy^61g#Zo~^n<41wb~%4Z;@eU4WJ~~Y%|3oZ!@L+i`cl5AJfU* z-ETz2X)9$1LR8Id$wY)8WHu5lCVDvETnv{!$>cvz|N$6U`tC zh!uGe>vFpBc`BQaxY==oK=;#CN0)pKWSmUP28lRIG#)9&n~7hQ zj4V^e2#Im2$~S$x+aB}ensaB8$NhM6iYA{@%<$45Yt5YyAcGmA0fr{C9=WYCaEW%$ z6zN7>P-_deh zY%aXco2tV%mpP z!!7cUkB;2KAgZh1^HRyMXCfh$9WuAr@h`@0w+uQ=l!T>K(mqtO?scEt`Fn2MXEngI z*i+YoC`FBR(J7T_#Ba&-SuJ-|j4rmAzddtcW6;EdNBKP5_<3q6V4^^Phb>h6UMV_C~RvSSG)N% zE{-~6v~waX&$0|{beE=^ne=A)dmkLK<(Kw{Lg^V*H=$nwp*z+!aWbXbi8EITi&Kl2 zz{j;E%Dz&x_ha_HZf4K>NlpQbYQ86GD93<%m<>L<(a(Y2P zlZ=8={Vu5ZgyA2lfO=Z);u+go;B$8{KWP1vk)XiGefFLnnWODT$4s|D+s^H>gG@;4 zp8K%77+rEXwv62C^`b!{FAd1itzFoNx~enE;Xg&G{SQ%!QNZuT{P>*|*=+2OtnARn zFAur(ahtZ5JRnGTK%zt`cQ`91hCp0NOq|kQj>-ECv1bE+L-LTqNNwF6t+FxS+Uu9q z4UX;D&38QhjR#$A7IeDZ-K~K0vNR2!+}OC%I)al5=dE7vrEeS^vZh(=w-K!sDoDyN z7%lxSfFatyxw>Ri!p54#lYf3r>AU9Z^Lr;m40U~Os{a(?Z$IxwAl2>NNt~GX4&`2Y3~Kg{M_YpyrPZm`BUQ#Wz%Vrjv`Vt>z38tB342U78PFPJU3p8 zyvnZhAL2@F@I8mg6Wg{m?qT`2vF4?2o5$7#KN2spNndqx8qKe9T6&T zXD2aM*!hX(G_9orVE{HqX0cr)vgqMtjj;3Q-qwfp6bn5X>*2j$zg3JyHT1PM;-=L! zR|FjSH&jF(O-+&sZ)P|xohs>>780y+uuz{hs_ZA}=*wj)+o{uE$oN7>N!EqP#G^lGfHKli7vz=-kggTm%|OqVns}}+v$a70glIptvEsF>Vj`dW^FnR|3Akl zDE4G=dc}E+Srtuj*kjTNVwkokQK=s2)}1VL3T;j82wyUXg@t*u>8XKin_7OWvN7*z zY3mIh-uf!sC}eF+H?4(V&S=ek99L$Y6b~tRyVPtKDu|)kK?dqGmITXZkb)?(q${%@=h}KW1~^^V>@0w z{&dwXPGzab&#bw_@$^q{K@86Sc5tRTO|hDrsYYwquz7dO>%{cWg)9;BEV=r=saF%Z z?RRsW})t>78j(>?xiyU+5CZ?B8^tvsXWi z^DC9$`DWD?_H-_v;zLUZ@6Y#Y@&OS$pPy)VsWUd|s-g`noxBZ4yQ>=Ag5GRGQkV;y760>z!PBzMpRWd`}Ghwp5iL#`nuN*wlyDJ!&Iu~2VVK0NWub2#jE&eIr^ys| zd;c>#^&Sma(9hc&q4%%WC3{NI#8Gw5LUuz=OWTkByh0LB8F_gPG9y1SNC7-V=@t)> z@ybbjeAy1)^V9SU!;H;S-gMc7t|JHga zXH-cyuU*Y7-Iqcql40~8&+0@(W3;_4qc6GOB`mz`E47|5V<)v>1NB8Y ziTp0kFwJfs5ZBQOGR2R)77;u2(Tu*FEnMYf(Dv}7sthsdpVARXqj}4-WyU*7Q-|lytFL60&Wn19ZJX5?m z=48{XzIWb9AjseUwn_%JHaD$9`97zkMn?Sj!q~}BDmp2~0Bq;sDl)!QI>nDX1|zSl zDU516xt}Ai;7Il^sN5@W-xjlS$8EnNJm*ipA|cjAThl2_CExPJj-f96IPWfwU@Whu zggIz+Gh=&SdAk$2I!bkc%@s!(6)Tk+{1J;MeKl;V`K+L=2QQ*+2~(Sz5u%yw*qACB z>p$-{y8lG3!7GgpUNQXqK>WuQSSHsJ{mrWm4pkaFoG#JIj6S1Z+;tAlPnFoVe1WgD zdcvj#HZPa?Pk+rBIbZn7!JWA~$4fq;A(W%A-o3fNw)*3#k8RV$T>xrovyN+R!rG%~ z!OUknOrZzwzy{RM0in(JA6JNLpV_j??U{Gzgj6&5bwNcmMR(4Lor~M^g}~5;L)aA>5zn&d)_y4{u#C(|c*<{IN=;)m5XMt}yzh(0r+Z zqsSVH)6fo=yUaB*#b}=fL8e7kji}nGyWD(zW%R6ADhg^EMV*kLh3KdG)SN=AQ@mYu zXv=3Ve0#v>Vw9MN-TRWhz<&lC;WwUFhNM@I1><~aw4?j(kcB_?BNh*Vw=L)RXZxWw z&t+%RvWL-TLGrA8FomTMF1Gb?f7(DqS<(BaAvfDHrUw&yTetN>V)(OaSuvko zQ4G0)bhjrO{rZ0|{r?ycHJ4^UPJkxA4gDSRD;J>t zLnZ%kYHUhM%GaZ07ky*ka6JOIOaiOo@5Ref?Z0(?vDao9n)F2h0_09^Wz^lzk0Webq(aX-+y=h0 zFs{b)T+roHX@@e(63}^ABlufPw;5x(HNbgB0lQzvS32$Iy@12vioe@s))MpeJ^}to zw9|!}{Moe0O9CBTLW~(`7WYqe{AjM_usp{Gm#VM32GR=vu_;S;X2D3<+V?wUSM~Mv zH|BLVd4RG;4|N%aa+Ag7?v7HG%PDYPIfrosdW>Y;N-W{m8R!ixAe95$kjfA|_YBCU zDAnvH)q3K#lH?{GSUk9!y~;ExGI+{)R7e2WKvHk8z^NR?tCd5o7mlxNx&BO$^m5;nt1A8^di1An|9y0|B73Z$pdfSi#!{sy zdebgo=FS0xt)&SKU`x@sLCu>|5!$@g-Nhjb=VO#K>v!?n_dASo^YIx212{#uWx;ST zeKpICY43WRPOgy*hY3Yv#=ygIO?YqyM zj<|L40}l0m4>SPmEK95soTvMO>)$8!2g0(tXi?Dwg)@j|CKu?$fz6v0+R%tYiIisc zWRU8OP7`%#(?T$VdIHeKVSVs=5jqpK75a}BWIU!h_tHsr; zR}Tu$H5EfI zjM9~{UO*U|nwd#!YHDWogh(f_ipb{ZzP;zY3NJ3j>9R6Mkoq$0eC6K;2X2DH{(N60 zt{wgWNg1BcbaQc?Pt!@zVk6Z?c^!^>za@=;a(1+_5s)Ue(1DBDIKo;24KMUhXG_V< z3_Xg(>&UxbQ_eMZ3yP`0VHhUVS-RN2+P_HhAJ>ETDLfCwZE$j{T>868zhP zn1`<;qJ-|~;3bA}LZ!yPfAutm?p?w-c4#S9_(1N2jEszUNcq0l|FDVk8L%?=0k@ZX z^Mm)=r$cT`g=lhg@BfmH6fL$8AGUv_p#Ccx7%ormrM|1{`oohZ9{94p26`ocbke6h zO^UqGH1CI67Lsc1uZaB9UMscAjalW*l;Zit=i6#*8V1RvGewD9wk-0NVMb>9Qc9-s zs<3(e*_WKqgLv)7L(#f3NfCk2D&@;gnkBJ=f?s1`*;=QqqIV~v|Ml+IK!7$ztTq@c zsqScP0IFZe3Uot$p$m8dFvlmDk(VNI@|V0_a~}eXcr~~;v%n0*@U7=KePTQEy!Lqh zq3Pc4CjA0H6&FCqha7Sq+$n%YrXx8XYaM@7Mk7k}II-?m3KsmGcmV!0>ZL zZ#J|GNcQ&E`cSpqp&jKomI5uD1W!lZz1f@qf5y;7{ChvBK=bA1<~9IGk9Y_YUEf=O zEbQ?9xkP9{Ci)U{d!aiU5NYXd42f>g zO*sv$&cu?IFaZ^?C0e6~&}2=scM9sSG8G=cdF5C1erjAW6=0wF(6Yz@7E7(WpB`xT zB`Gl{q^D!Eu5OJg*{?xBlTb4}6JrUUGp0%P=dDviF_Q3UW`JK}=+ABI8XfIz>yLit zYy1D*WkSw}g&E+umNyp5q5)YY3N^M@&CSg-dxE7Cw18<}`4w8sWKut;PeE~q|5dgE zdNgu5z<5fG9lq_5I|-40&2Wwfin?^0$hIv^iV*=NRLt5m2#iL6Cg$el9RV4uhY|(_ z9UU%)Eqi+Z*Q^`0eNv2GQGzHF=|^X{d@XQcd9xwVCE3eAW#>CELO+LRgO*`?#Nn^* z7yr;RSS&UadI&+Eoy>pNJsc#KhX+vyGw%v4V<8I*3;*S>=~gc08k(B3aWw16Ulq`m zk2L&JO2-E80+=$|^N>exeM(x|RphP1ND=4f$Nv&63YEf5fAT130UAah zD&pe!Hi)D5+h1CqIAKGDWlGaeQR#a6`B5VOmvbu>O-X+#HO}WN8hHb`?VuO|@;lb5 zixAi4y1x_}7-negqh^c78uM$SRbic?f1?jwWF-8{Ej5?5&885zK(7$MK z^=f|3o1Hjm(=ZfgIz!EU1QZ7kFW2g91VXDHK`Bm$nJ-#!-DTv4K_kX5o-}K^Q4S-a zP`k&A%r;>LLe`V{|B^i}u3Kw^dquB+m*uur>1F`!uY@hRT-TPIXP!kvPxz~)OCQ)r zkV6RMtXjj;=s`aH79g&jm1vAWa4GFO^-fH(va*I_S6?1-VUn83%XCR+&O@V3=ksel zFqSuWcZUH+J+ahvDE<_1!_`8|556Az5N-Qv;C;0L-gDL5%W#hF{XZNmsC0TdGBVHiLqC8cQyY<=;o+;Wy_@82Ctjfyr*3s zJBy*u~FFb@*F4qQcV6B2P+2e7s7%Q(@l>03f?t=~CWZ*OcuTa4Qtt3JkF} zusG0V(1)Yte~1U?#UtjxO{2)tkF;x7-)Vmq4*u&Ep~;{3adENlwXUSEMvEIFCi-ds^m0MFjycL5aO+jQzZOn_rHXV^Cb;k5Wi zGh3SA{kA3`Jh1=P+!tffRdwjFR7?r2AJnsSFWgw|?tm3@4zfuQwru?iD&z!!0r@rr zDu(d$hZ87|N2K|O;KD_^KuJSyfyGS))ea|hP<7Gk6UMF0%}*PhN4S(M##Ikdkd$;XCJtCC@ts#m(pUAA*+?d(~ z&)g8$I~&1gpempG-=Q(r{=Xr@ZuS{^85(zY=R(i84c``8=~S z~p~OYmPh%=f$FddT2%v3|-!eZ-!^+)F1~De9r+6CiF5y2maMrbW$iZ z1(FSsGT_a$0iJ-v#Q*rs=}o;WowV^|jy&-eW=Q|Ig4YuJ-L-nh4O4 zZ@!|T#ca>8vizTxQd_2>9MI}dVWx$k3!q6@c(_fwy739^qi_PpFIVN|h|%Cu=H^P~ zMB#Xbz?UbNZjYrTjbN>uG(QYMSkeXkF>--$cNCQ5*<@)rQYazq%lWmy z1G*pi_^+?DRBz?_jNSoq5+Lr-ojU?L{TM>b@2O3KMuFfDhYj4#S}33?%xXnQwfz{A-4;P^}JhH z3&v8@6u>mWv?~9X)d0R3tOkhk-Fy`N`(ezNze<$ zW!6JXwEj9j|FjekK73l6OA@U|Ye^NBiaIp;IS3OQ;XG5lTNNp0BFSIRh$0YTV&dhu zhg_fN`wK9lGz28@vOZ-)Gd!n?N!Xo&$~AWAQ>P7@bw?1A;)=*$j)Bo0fBH~iCwt@u zYu$RgI}}*9!E63t2yD5bqHeCP(Bat_)V_GgmdVxsdDqlXa{fYOv1nF1D`z07a2Duf ztHJSU>Fr^$@Q zgC= z$P9qnqFIcdkBIx%@(1S^IAE^AkfOW?@)Z6vG=w37t|#H(U!FXU9{s?tY#YRW69uDD z?cPWunBDs|6(AC@Y>mKoDgaIsA2#$?vQK|VEsB-9Ai*-*aHo_j;QY*$=3zJlL-*q6 z6~=tnVrqc|$flMx{|o5xdL+BgWChRcrE$K+C>!zS!+L|@_$y{*9dX9SrU$ATb?N>LCH5fEuATNJ4lK)Q&40wD_0OR#|r z&>+332-1<32!x`HVCcOCh)5@d5;_6$uO~Qr&pzjT=llL#doJ9Nym{ZJtY?+`zL(0B zXCd^7bN=(!vv0GqvPN^^n*P6j`P&XhPCv{Ykk3Wo5I9tX$_Q6~_?K_xL_-G6vyxXB{hVCuvN%Ak!XU^m{o%I&rN1ZDQ<7h|QtgL) z({Dc+PYoN1{V4O&0=IkRKeP4$7cJ8lyxXIr?hZiUIMs6UzsFyYz`yg;LkHMLcB->c zW_Oy-ctZc+JZLXAgpLK(x!1f&kZp+ZF zgN8#?i0_0otnjnWkfh#aj+p%Db>!R+$L;?FoBcc+QgxmXWXDMRj%jJ)-AMy`QN~+! zv9MtXCSMPwm+rAkaOiWJ)nTtSjzJSKT{t43HOL2^_GFZdwFWX(_t=Ri^v2CG9A_0!CKoa_FSkmQj-X_$Z9_a-PVm3w z*^$Mu-6*Dm_L_N++_r%xhv?q6fl?L9=ITJGmCXdD17YX`JTa34ite^E-#$}j51>7Z zUg{?ytI>*QEhn(Z5dW;u`&dn9g%B7g?^gdOVKVeB&UNZ7LN{9oNo(iNJB$V)45lTEdtV60Wj*hj^!+=nX{Cf0yMTojqBj+#3#B4F>tXqv}HAEv3+S`KBN=7ZWDK{YI$ssvOh22hcaYiSG8ap;xWL0D(kmEjS}f3eye>;!sX1-(q@|CpqS zho*1N>K=*3ExJmQ5R%*B2azW^F@q9~kikaYgjo}`0}HW((*nl2-mT*=oN}xDrr=GD zJw36OYvV?mcyPi9RT!|Z!4P%{Ls-oStrJv(JS~I<3`o6{zA2%tgR>!j>qm5$pqJd| z-^4k;z77;T4~a4Sbznr(U5m4-P__exz$S37Tjz8qsGlXcm(Ip5Ga|}jK~8AoeZ{t# zz3x3PpcUb9(33)5neqoU6riG){%*Ld_mHpWA==f)cG28=g-mm=Yo0-gQIrpb_6+sV zhjJd}YS8MG`tbFh&r*eu;L@A#3&=luU%3nisF4(cdd=+tuih-Cb?@RB*vtf8+)~Bb z&yHXsR53(A8_-p`3Nkk;b^F*w;Qw0G{m(QC-|@}mkO@-gadL8+bS<6?ai!Y*5mqxK zsoE!}UIoAfuA$Y|Z4EL|n*CQIcmuow57sIrdyCtjPuAA)Ly5=CHwz+d{mLSe$3idv z7drmd9MDuwH8q{KY5;rXeu-1Zd?#EpuVS0h;`SohH7cWYX0jeM3|km9zgLzKREoX6k-piZ0oQm{BR@hBIu0ObLL2?a1u6Ci|jD56Gj=Fjaev`nI?CWN!j zus%`zt%DbUe7fRzN@5yQ{jZ%Y2M@I!<=5T1Z@%M+fX0CRT)m2G}iL(V69=~w)4d~+kD?_Z#! z5tPWP^1xIdQlvj0yJ{@;tU8ztU=yaI`ftXlO$1zEb)dH7Us;h@kbSxvvbKudS2F$zZHj4{Zu@sSF{~0i8DDDtnt2j+S^pcq7=o9S^bLW z=0CGD9RB^+d#eYFTtd~>QbwR@j>w{Um#1fc4*Bse5o=LHq6T`a{H2X}r<}u>c(ict zLth1d%t}kh38E#@`4$Zg);amZ^WD>bnv=ILia8klZhk>Wh7C~Q<{EMyspZ$2>?N#) z_fQI{V@AfhY^TX1+>0*DMilJcwv*bs7^NUT|4vBc>6E5hjO=?kv>Xn^Bv}w2i?9oI z$Cv0GPkWJW*efNi5+}&aal3$rL(lPrAfrw4K^j!F#EX8=*z)(2diS}caD%SpkF@3l zO^q5}KQXtGY*IVu*Vq;E-QP4aJ(QCc!u|Fpw)WvKNbKa%KZD-$ zNd6bClHilJUw030RNV7blaBo#YGmAxTycA!S7^r^7i|)Q2k>9t^C-!}Jx2CO0rx)J zuW&j4*G{j(RrsuzQF-ZALVMOZK~gEM%452ye&E@Wjp=LE={>+FmyD}Q!D=Vh3O;v6 zpZhiArd4xk7Ov`sWrXtLsQ*jZmuwhSxqip-LtRXsHKod@dZWrHZ+{Y(pRp%$AcOza z0QbJ{mz0y%ivN&}t2(~4m-7+|5{~2=kGmGLGeG6~?WDtnb2{YQGDnt-g@@zO*(6RQ zbU+<({@a_{cJz5L?$XsNpK*q6mp*?!>25l@(C)%LVj|4=#xm$|*X@q3I9gS3`Jr=YQp9gywX&4NmTNL8 zMAjC|aRYyJGzB~vwjtY8E&kHC)LJ=RnY>3V3DeydM}$6`JsSn;^uz{LBEn()7i;#?jQL-@h=(?gqW#=LsR0(qUffC4@HCu`TbGIiU#@L zzIf7l9vSZ~Ip592o*I4H0|oRh69XwRwdEVm9SY(UOJ@^1kHoC{i`l-yZ1hb*264Jl zCgJx!cqhjbccxgV(t2?&CP6`@3YRKNYKntxlHx>*L4SVOdC_Cwx%;k5XaY}0So+Es zPvayg7Sr8>e?ASpFdW)xxbekLqa#Ej$_fLgD{((#(I-DM>7U7ZYKC2eDl2kogxGxI z)6|=fwts~``J1%B?0+k*Ua(%5RUA}#_!6DG??3t)iK#7}w4v1A$G2pk-&48sQp=G) zHg5T!9%xikY|2;`jNV1zLs&gPirEmvHZtO;H zr@W1ubNeFxC#4-m9p5otXG~jOnk@_qe&T2CLz*ERzhj%s$p-I4Fb)a?o!A$(#WGRH z5MdwP^KURt^8a+iYu)T+OiZs4llsS~2Km{?=PGXY&Hj0vqpxfXjbXW) z>B>n)=5~Cz0Y9ZuhZgH;Judh#Oip0Ky8qnl?J=N!vyU}#?FllPql~$P6Lvi2#Xi=i z1A~JZQ=&>G?^DtOB$07r!>TRC!|?1FJ^Sl(!C|A?tKsIwx z+v~Gu)Uhq^TB&ele^m#QNHaH|=GEFtLeq9U(JVrnn*BaE0PH6G`D8(+r&Qr8HdJKM zkWe;B4K<22I;RPL;NZ`=f;U@9zCMHx!psMe$Be{^h0I_EV%alrxg~l8UqkJIt$q6Y zjHSm<#(EAEo>d^U$qB#nljxd#|80EFCN_fXl?h>Y-x71bIC|L58#p_~EdCRJwsP5XJi2N58|I;Au3ouIS=9>)K2lw*{A>1m?y_&;>JiKqY|o`GDfaZY z-Xnw$M$3jiEK703^*$tj9QplH%a#q8<9(Yt7x>cAeWV%eCj=}!InzzS)9N+XS6iwV z3WW%R4jWH(^~jxv3OlLhUF+RAEsN#S)92Zb_X?Q@st9lIl`g+gPx7ygZER6NlYMHpLs4mk=x!S+1vqIx{N?dl%2wllJi`6?`ZfeZgLuj_Cyw zcx+rFEX{$9>BC%Esu_$X3h&M2rzF>&uV1k^YW42h_#2~*>r0}wc~XY$Xxt%=P7YRI z3MCZqQ$#1DV_mI}dfMK0%{jC2lt%EB7HcvalP7=J_mAavs;FeTJRvBiov*Lcs^Q!C zwe*b@FAYcf_vd7#Mn=Q%aTIy4=#g!(y6sZb&i3Pqin9UN-|gCIxYPFl+fth5!HQE0 zqi@`F=8b>v+khFeX_l6oucnnv(*}h0y@-R6uu*I)JY?cu51I2;mXza6t|Y8f4nA|h z_wq_*Nf92i$7C*9f;6*)g%f41X>gi%=f;`fo}MtqjXfsW(`nf8Jl5GWcf{&p*k?kz zbP>nq4hWe4#a{WJK(7VeM)t;GscIf)HPk$@XPDLZsS$0`%CdZ?r_zBG0{%d9`!P6I z6g`La_$lXI$(p;wqJG|;OUSBX#wUEOe1{Vu=Nh|pbo zEwy##p8h{!Xwy&83`MCYB)Wu2FAk_hKR)YoWc$-p`;(){5FgRb&(h2~p<+BoK_)?7 z@$1|*`#02$TQG0W)Gl?s$Uq|#vf45D=wBm)kt7ov&hK1*le<>H5>s5-`?>W`|BapT z7VJ}f^}+#`oZ`2Hr>SULzA9cI46e8f^RnLNu07%v0ech~y?etm{SG>Auo%gi|XjIZ14N(byIMtzYW46=)y|+zFs82VematIF(ktporD7aJRZB zzT?s^&hyAm@qdvHbK1H0U+d*yL}-GaGPp_-;5XPa0vdk8avxwX+h^J(0Sc;zHw<5J z0`A1}{4NQ~f}gcBjPwbiWnQs(9e4_jc-M5oFMaQAQJqi!z~s_JXG)n7q2#L^S(AEB zqYXTX`)p9W{|lSmw81B#^1N7qnSC3Z!L4V$Zg4o{GNI1T#<_Y(b1f3Kygu8lJmNej zKU?{$(D~riTN}H~aeN&sMr8$76QW%3vRRrMAri@=n>TO<2Py)lE3+ji4>!CaMxzdy zXG-^CUnilalx$iW`Yh_o6=QBycDxXv#{TlF2R)X;2fK+PLkmF|rP4pp#_HMjx$TSr z+cQ2w$}|l#^E}VnAB6jr?K#<3eX|^LF5N78I1`aR)x+BSw`H9tc)>$vCabY(y5uSu~J)Rn@yWLV>rtf9O6;e)SsSL8mixE4Fc0Jr!cOISK=-SvY`-QZFSB<_~whX^y zm*Jq~hy0ZBJ@`vfdJD>=_qhk-*|tR8vSu7i1846+ztGC&Q`3#M`)zav) zLF<-{2_j@oSAMd*Ob}Yo+qirB6{2|Ix3|~FIK$|0I(g2-Y(4b2n+?P-Z!Acfu1r@G z-x%?Lb~)srJ@qr~T3=}p=8CnRO4a88eQ8t=C@gdtS?DZP4Y+2?fE3@@rBKgaJjVcXA>m~lzc?eaq7`CQyF zb}Xi^G+8EJIJOpsk+q19y{&W9%WR*B%v|JM>qt{5xqn`3UfERtf1pCp&S<$fw<(3{ zeB$>UV#@Z*&B~tgurA)P;#&k){G2O_@h=q!^wn>JnH?fQp+8GJeO<*)xwM9d4-byF zgr0TEcEk-a@g21a-*0E#$AJ;|Z^J+p{jQCbTGPxGTBCQ)#L&r+pJ)xq5GCd84@J8f?72PI9uAJ| zn!bP?EAH~S`xvvIB%#{l9OrW z!)TupE}pc6`{3WW4{LA~2-OQltGmy>;V}Ed$~Gf7e)_^7u&dWxLkus z1PjBM6Ot!{je~C)PnsSDkIsW_b6$CYJ@e>&_;-sicpX;FhDD%)dr}e2wl!@{Me>C! zlHcFxU>;-U>yfEuib zx*y3k^^(b=8b7e@!sZCJ8+qjNoqVy5gcxNX7qEV;Skm~~!xyTf3pZIEk>EqWkT6p#^5OHNbsrer_eO0mv)en zAk6muZaXhZi4kk{)<+z|=*+lS(#&k7Ov;{(TQoHC7-%4Z-DFr$uOQ#szCl_wbai|- z4C7rU<2h@DFS>y4mYo(TFTV*RGi$hU7~c3WP(yBz%5d~vPY(?(eeulxAUyivjq6sz z;i?A?!n)x1<=f+X(|I;ff?`{;Uvd7)TR&|XIhi+X)6|GVPd1DDezS~ncYI8rQJ4Hc zIr1)>24L`i-5C6J_B$*tgAgnZHe#N+P`^IN%VZRN-W0qX{{+JCa;Sh!AuA~NGcm_g= zW-+zLU%k(LrNG{6j=ei8_!EBnZY8Rd$Vsr14|jLU+iN?ogpo5Mb{Xw*yvyDZrtJhN zSlE9(+vOeq?q%z8n5y56!eG?<&lab~xj8GrhQ(XlI2*Qwi|r2a(M zxRV$PI1U%!>QmnjVm(^-U|LIP74A}(-^9C6PAbH`?}IiDuKdr_m3H54NpoMcnUb{U z;6n96N3EX4dj0A1_vjY@P0F41e>t#P9W5QO@4zEd=oa)N_S)2@uM=o{6M23 znWko4&h`>ii0h^dHI}6frBqA=yGqL84q)$E{}+ys%{+nSM>)$qv$GAtWvga36e?-o zG9_EoH4&y{v;PaYuK$a)Op(Hc3CMC`;aEW64pfKpL^XGS)ej;Bv=nLj6ClQRnD2qu z2;aLLHfYCsmrM6ND=LPQ;{{L`t(^=Vj0+puwn= z5mp@brGd=Ny21A>(Gu@Rz$oXM*Jd43;``aA`aGp zYEG)AlX6)urt&VCuR}Po3W5^uBZQ&)UInr&y^ybzS zD1)s6wx}99<3w)PJ-!cWkigbD57h*%X_fTZmKA!sOn zjoUKifv*Kfi;0;JK|w))*sXIzO^q7Gpg5DIEOHam*C}rmQ9oVzviE8F*3B<2oPc%H zpHWia65XUz!7m*L{(yQpl-3awZU}*_vx>&k5GPfSSpc|27~L8uspUZ-e)yf*+D(KY zhv71Vepyk+Z(o}POcZH86HC9%%6flR?>j!nX|Vw^zV8k6)>W6=e;?nXR;`>$=Gz5aTQBTjHhiD=kSYtnqa%I+X6-h7$;xn78QX=r z3AWnyBHK%2{%*AE^4%u(K%Dd73?N-d)W9sV3;Fm#0EqYe(l!kMTnIcU zd2b(R`{TLLWrvsrM)_(`a5sjc^>A%eUT*=cw7K!ZD!dl;Wrm-fM9LRFT%fn)ua0F}1FV8H61` zs8!bLh(vesWCB!IIgO4$mJ^un-HNl7PtNdDG=emB%fzGR8Q;e5iRV6m3{Cm1W=Q;x zCc~K)A4Zzl`icL@Ka}Sh)yGCh&*3KkalkG(15zf>u`721iuC+~l)u1F2>YG~THgr# zUI;h|giEo5y0RkFsks&d;Sn^#+Zse8N)(M3I)X5+^N?H=ZHO_wY@qMTJ|8R2cq2D{ zE4F+K@&A8qzc1Mfa0v+n)1o_L@_Qp*zfMiPN#p;>@o$`U@|zXYo4wT&w^~TYlxG_Q z2+pZeHI{2*8F4;Wzmw%07@aHr5NXqaS_I1P5rhqtO>xC#FjdXR zX|!1&jd{dyqP(%Mfjy?aRj5Rkr(regjqm4QciGanT3G5E=(EFEC;>u{I8b_!wCVRW zGb!xMOcvh>P_3{Caz6OzKPbG(Y4gPP0`S3pW;d*p=XsUW}{PL;0Bm$=XpdieT zemfki?!nWW-+%lsAOonhU2QBNigv`ut-7AFVO~f$XR)5OU;JM5RaEixf1L)2Bz8(_ zsa>#wABY3Sh0Zo;D*{zUeCPS5BGnX2Xe{5>paLjlR9%OrA=@Zz zdMyhEUXW-C+arVtkItPIyZ{+7QQQi62v)3@0t+Efr;+>g1=3~Z`mJ!Qi@_F9j43eo zum%pN=R{6bb5qlW&-+t%nBR?)tgAc{2@s4Z*plL4bgBU+w>8yD7$7OW2>t>vQDs;= z8|!-#5FpykiZ=9ZN)!5Zz$b)b5d8JA)j{_TBSP6~FF9S!L|aE+oqYfWMDz3qO&Jn| z8eKdRfnTtnbODeZ0RaJk>JZ#<`AR>wEZSLH^ zz!QEmPc7N_3Du>KyiNoiz(&<6P-BHXdubL%boOnenCUvL-*TQJic@%%WffOgqLqi( z<*uvex}Y|v6BDR_ukYe5QZ1+2M3^6S9ls1E#AE<4YBo1E>d(6XDh-16A;1A^140Ik z2T(!`OM+k6!sb~9^23Y2a2X6h$B@3wz(FOC0~;SIjF5D{B9_&Gx`uTrUl46^(#VvI)!DlcZVQnl4!A_i6DpC<$aknG6(u zXC8Iz82FE^@1cqaf4sUCT=25u*SWs@75wFm`2axkqpJbh8DVz|zuw!rIr$Hmh*Gbw zO5`A%Wfy_A2~wPK&jc;cY3kG1>(#!^u!6sUs4)SWS4c?2Fx#O4WIv8N1m2G7ym8_( ze9P_&>r07)0q!EGw7Pr-#UeqWTP3#IvlFT(CGDZgt>zBt#&x?^GY`c7#R=i0BB;wZ zkUrYYn>2t~jjj|4EaLe*Y5Ve=%O@{stRCHiz-;J@@|+zW!fQ!wpgebb;0h&|7f5dt zF(_Wss%LvWpQ`V@W-shl?cg(-S$37O_{cLX^Gp3f2DEO*diqqIB(F-_zyOGM6i&vU zLH~KysVa@kzS|r5lXbwNX%m3|4g??H54Fw>#%)Sg)ueWo>Vcb<#2gHAZXnikKa?f4 zdw;-U3Ll(Sk|xFb=Npt?UZjn`RhEbh?$l#)tS1?}UIwPjGA(sl0|6wnONmh2 zl3J8DiC?tz#UkCMUkhW;tE;Oo02k;@4a}JT-TA?(&>EJ!G4Yx;rE@P}8MF)kKV-O=JO(Jv%_2*8s;n z53qkjI#{nuKzs>)f{`v9#Phs<2`x%gk}lq?umQ(3Ca$bTrb-|brieLNoLv=-1PkE1 zR$J88E}dcK+SD-M%sbAU$kPKzV+^$797#zGu12AU-+DMa%~gJ7W&}H%6%CcLsF$&x zzPU9|f0o!9updRs{vY$5Y@Ek#lN0>%>CDo>{Ag~hFnP#bwYzL>YLvRZHV=Zo1PVJ%YQU`Fg%O7)B*YS-BQO5C;xX5B7Eq6| zt0TB|I|M@`F?*-#p+8tOT0B((F4|H>qi^VIZSa&QsVa*e&RsL)G%*E&+%v5{XUx%o za&VybrCpYas5!Ii)chS^{@NB*AIblb3$t6F4UnR2?~6Qsz>~#EPFc_PS%ybXMqp~7 z08Vt}NeC*9NtD%t)Spw!%!%qDOy4`PVqGWgUirg`xwOGbpAnVp(GEq#o>E!tk<~nk zTVLh5`&5`slB$)dVq0beV+4DD3&8re&~ji$72GhM5PmkmzF!7I8u@VRE?XF6O+0+P=$P53q=Tm8fxVv86AX{~Kn0xXcX1l_ zkNmnd%uw@SOqFQOdJ!ZkM+cu_#TwL{e-Xd_0ci7>HopdYwhQwycWIKo6;@5P2zk4T zbgInr_$#IIPZLb@bRzCg#=v98JXFA6 zM>|4OEbgTm5dV=!vSNTFInZ&UygV3+#!y0#2^iim@QZlz?g*287keYnPC-oeGq5j{ zwWIJ`r9S)t&64T>uiKH0szI-e>T00=V#e2(lCGy)sO`bI@|tBi1ZVRryRtPe)*=;C zU*3B9AzGLaDwH0p1EyF!ppX;a&>=830O<5L1&>dfRMEU4N>9&RgHDac$eUJLQ_75520l?|Jgl6 z9)e6z@0zdOA;o?NGF8Czit!NvcEOB004nHQ*;7;*L6Vf>La64z+qpz=@NOlIpB#1|A^2IMCj zSY0CM0<^!itgV|X*fr{@=+hDXN&ihSCsleav$Z>xU7(g;qdr*}a*?i$zL~APYOW{b zfuXSv$MTa`h)0b}4;q&qH(vM1U5bBOk{~oe%+`wcJ{DCo5y-FJty1`d#STYs)X+x)?_YFMrxq`h2!Q4Od>z3sYm-N|3tbq$-^l(P!R7`WD8D zmh(=ytbgGMi9I)tI^L$ajK|kMf1BSIRMlZA-R*&+Gd?OV7Inv*vASlh-nWD|cUe<; zN@VWF`jDSR^TMt`u^tHs%6*4Ph@OVxF_lSVDPbXFR67938-{2R=)MqPe=sdQs~UJB z;^CJy@#@x&A)A5`#2sEgw62@~-K221O*9`Rihp%*0urg(f-Qn|wQkvBiHOfEBo<-@`dJ@0Ys@pC~l;0kHoyYY)`eR;| zp!Xo6j008aDCj~6Yr+8wNt0P%A(U3J*`&%4EK_{4z&lexv<5br21$p+`>l*2s^iTY zDcnT@cN$MALQv`qT_;EQsY@DtAXW=o0zAY&iYIclkTMO1-KIwzY61!*iRA%a2?{Us zsAl$O+g*A6r|DeSWcl_ zuhZ@>^0KCRJ%OJXhcL*LUI(-5HL8-Su1 ziJDpaqnvoD@qhD@4xx;vmnGw?YZzrCIx;+$dL$Nr;r}~~7W3#(#3d8T%M0o8S8&F7 z_;IecT7@kXC9Lx*%XJA40~4D zF1yreMl%hpU0htcDHKGS1ul@^-t=O(|~ni zr7$!8d48sjpS${yA6KayiUVm8>OXAOP3@68@%erJP5%MtRi}qn24p<&wdL;kjvfO# z=YJ4o?X@XIPi|@S20X89=CKO8Na;SM&w|Cx%= zMVl&q-LA=5H72nm*)<}4)%cR&WUT4%^XE+@K z(B_5t#Jpbh--;*jtCO7=A#P3lF2R_+mW=LA%knkQtgso_DM0B0HJLf4N3Ra%#+T1_ zLb1%9k_|)d%F!{CmiTZ&Fi%JbX!+h|HheMVjfz`yk0JG|{PcPE=m^fVwXMwtWN08m z8ryLNHq-u0crq`u+`&E!W$MRl9o?nq>CN11ANK044AZcTD`tH4SNiDdPd-D5>T6ut zG4ZHl)}gm5oyc#Ui2P!_lXnu5E;@fZLs2~lu3`OJz`4%X|2M6|&kqsE+25&p2r4}U zIns+e!IxT>J@__<;;bN-LZIEyGnDE*?p^uhbMNUrG|oK^(VK|@m4{+#<7M&~>kr1r zDSJF>WNohbgeA`+it~7Zs2i^N8K4_KDzWq%j~~LcEKHJAe7$lCAc8wXGj#HNT(say z_BC>|2w3{fwS3FB0*nW3^jvql<8K+GA~%{62p?l*d+E0*FTjzy77NAO~ywxaXU00Y+Es7_(evVxDL+SdtK`SokMW#>D0r>Iy^ zygDcuJ0weflpwgYKXLYMQX6YLYJ2+MUvv3S1yhsgj)wT0nf=7OX(DsEB?SL@QMkB{uQQrs#o=HnA*@EzTT9J0ebl<~ z+VMIQnZ|Mc(10AxiVyQ0z1YQOLvaxzE)7?cflgyv-kyd{mk~to>LA}UEXvz~ChXWV zzTPPXHKki>>Ux#heZsLadCr@I9v6!y?qNUNZtd^Oj>Eq0WEsIVA3JdPca_77P4A2) zPOKO$-ZDs|#rEfCzkZTMXc}O??_Hs-IRUO;G0V2yas`~(g%ohsDHwgejS z>wbAd;mexlzC*u>NrVIHl+(oaVbRL<=4Y^)JAY}6j5?+(NSfsY5gvPNZ*?G4(}8io z*|9LC%xe}Sa7*LpZLgzVr$;8d602e!Zc8b3E}WHf`rJJDk+$NV!OS|p&d96p5;y7K zo}7qR?+q}PARM^bWEy#`g>ZqVV2xG%acb1kf8eV-x0z(QEo6~8-5iYGu15u4v^S4? zX(w`N@%aav=}|0qK4eSkr5C4mm#wV&zr1KQu`Q+{vPtEYj8(*2r`Ga>T~K3sTY{0e zM8VSe`y7btl9L3vwU}yO(>wX15v8|7E1Or2R6Wqi62mmh=P=eIAIIeDOpvmK%;ORs zUfca{cOe+9j*M`FxSCu-#`*Tl^MFm0l<1&N+M9OKb99JPZgGs0J0dbnPMoX~KX>yo zk?-Kj_O#t^jU@s?0U5YUu-Mnq$qOh*U_gAp8vR3{H5o#atD|(aW5yf8=R0Vo| zXYk&9)+(YtuE3+=csBByL(Tmwi~=Y}00jfD6+$5lJ+u?(EKKN7+t(W|8oe@uq&!_uO|$#KC_X_7H?86Z~2W*GMs|N;Z@+>{?4xpX47on&;k7F|?_W zw|7G_6<5?i&QS{38ZIX}HPWn>h^r6=A`OyvW`ZeW&xqzwSpD^^zzdMU`Q4E7nMcL+ zJmycDDurVE`my8>65T6cdw1543eH)%%mR^5qK4nepsl+1 z4dA)bPjD*8y{1h3A(1f3>emPwvl8bM)u|E4N<;emCtPWCfo5ObC!HDll+R zOxoY7V7Sg4FH*7Bwxf%hy*P>stc44W!(+U2&>8X0JK2&=E&ju1ZiFA5HJ?qZTF8Gq#*_%*{WG zqj3Ljly*hQbxlD}BZS6Xbhz*xtBN{nKn`x0V%@%aaF*y3f~0O@A!UUmBASxs$WS^) zDt|VR>;PT8f(n|ctLvSNgt#~F8vm(gmochJJ5%TLhh-u>I?J#&#d^>Q#Mp=-sBo>_7@=Y5*#bYdWHYDebiYm5#4+r8I2x;sSb zXq#=_o?kQ{3T)i5FGR9#b2AKCfxOa}TYKN#a2R7w?1=gCs8{5#*2ksuT2-c7t<|P> z+IAmaJ(iPK;!O!(S>liyDpmHF~K*~#UEbd}%HhSlYG@EvTb%1?G;8%Yndgk14ji_B^G5pa{X!Dba7UtV0JJe`|l`o~W^SRO@qiayd1) z@l04VdID(6`RA4YFKk)mhDECx>crr5YnStW+I#bDK9 zBP{pG6-B&1L+rBA2$?EmJ&W(8=ltXJnKm!XeaV}#F9|nkRDRtA3rEQ#pxCURVX}N+ z_b4C^`dDjg{7Kn<-WOO4Wy=T1TDn@Q(RPsP#b4sR_<82|w zW4h-TR6Z)0zA6hCr7{rtmYtnK@{r+@rs2TLmI2DSk*>;oD`|Z`y8-G4g;tb{#E#U- znkZDO(ZnOxEfNGpDj=ZXI)ak6vXG`kdaJw4ZkuQFQcIMLM*W8dRu7+#`;p0P#$@$F1w{7=c2+E z^$&EKG_5-{!vx2-U!$(}W!Kr{nI;Osv!fs{q}Cl##}47lRTW)^z~TpH<~j+jZ$ z7%PDf2RquZKEds*zM9_nnBaihF+|J9jfU6cuC6LqM1s&lRU=seI}`FG1*xy)67b+F8AA2qSDtvZ8G%RWxBSP zr+n&7)gG^uGJxT+0VsrGMAJr*;bU~Vn?IQ z6$i)r5$>+X@m$U{JzjOvW3|4%=A3?-)}Bf%_i4u>?Q=K)&x<4mEGIlJ?fS=>!Vj!) z7Dz3JyO+?+k<=o}#-IWUl7F6r+_n$oB5RRyp`OpT?NB4pVxXb~Xr}xb;9Go}-`vE+ zgn_h4kUSC1VzCHP3#t28n?i*^Hc%0=$9bSRA>dLggCw74As;>jvKTe*ROeGkptnLs zvRUj4Lrw4~bOT0Z9R7tVy82W+q{)Zjt56Ee8Fo-I09DYwLz*BqAU?B0jnckap|+~- zw?3ON^74`{85EilVz8x(DCsu~nT`FRd??-0+1WW_{ZaqOq`T1qmc=FA@Z^vXUFC37 z_l~qHp%*g9eBheH0FN`zJe>WTANSNsrb$Ml2a#1GI37_ z(VgPd8rf|+Y4!GCSng^{!}V63+2-lF`@VgduJ3&F;5XbMe9jtjZaOc6FOExBTIlJnW~<~0v~zdm7!t?w-BGwWRDQZGD~7V(R= z-K%S}ge-a`rcd7MG>dar#p4cnj&_N#8k?yFjptcdtsSNBKBYVI%XRb5u8T(i=$=~I#cAO& zF&(`p?$aIL79X!2$s)c`@PTYjEL8bzKolvA*TUyc?lm%D@B5w?8CP0apG4{H%L={; z#Do{Id29E-BEbNZa6{q*aNxRpwr&6cga8}8;rh53C=HZ!Eb4%=!C67LFv1co_W=|L zQZPo!eJ$Jq@LuhBm?k7+&CYK_3G~gi1e9@s6bhol++QQKPtsX%~P`$hm?!D$BENdKGP)+^X$JOb2P)WtI+X+<$5yknt}iC zD)qShtG&aR6<=gnn3nfk(X-lQO!~EqyoB?vD$-~3x_&#*OW;%S3r(^wl2p5>R_WdU zakwDpo`Floy%(4b?Y6zcmYHvpIq{^tr00Qz&&Auj)4-(o!rVv*y)Nx}tR-?f0Rr`>@Jftl%Q z@N>(X*1)1eb$s3T0^=M~6e(x|_(;Gw|7RHVv zC0l3)Jlv2_W=sL#jd@Nj(rTN%w~KT&kM0JGOqfDwchtKZ ziUgy&pcIDmb&#eUk~~ic;gZYTStY*md*mpl@=2edDbBo1HCE;kHD&S!ebv@{yo$Se zhOhs8QoTr*TTG|`lMIXpZ-`LC_PWQ#X-CB!(-ki|y^@%fCHw4%?dhYxU@N?$_r9(A zT2=jatZ8tON@po9Dyl}ENH;CN=3~Q=xc|9K>8xBA7sCU?xx49y;|J?UfBG>tKYvEO zf~Uh(B_HF}ODXmJ&=>DZnD0p{;vt%vn1;SQ>UD?sF~YCyx$1iG=H&JG1}c?mg-Zy` zY<>TEh3nf$3XL*#+dCnoZyakL=Q9ww?yCf;pTXX;^LvW>Y}}c?b6!ee^j@4hr?YqH ztmh>&{>|6Y#({wO)rIufhO%V4^^TNQZf3VB-aw0qoBXMTi0P%#Qww5{iY!~W7Si{M zkbd4Z3+P-`bFDC+m6nA1Q(SvnN37`}b8-k%;qVFkmIz@`VjU)gmqbB<@oF9cO4f!y z9QWFKxwXT68z+Eg7MFccDIUQ#E z({jHPj3>MLS>Xc#o;#xU1(0uYA7dM^KkG*m2*bC7MiIqDqkPq5tJgWRBPfBDYAFjRg z&9+R@6WyYFGG_2M_Wy9LrF%=ehi0p@>9qpU6PmQQsTjwXG50P^KKhb7vdCpKP<=+q zJKoOSv&6Q(*16tg!Ka=vT--3*?-C!V(z_()zOI-l=ds9&msq8C0)$M~ZK4yI2xN8z zy+&XAfU0u?lA=Xo#eof#IkFY}WTDOCNG}+ak9_QEoL;9mWH}LlK_%e)WuIwJ*|Jvw zNPZm37feZQA2;w>_hbITL61`XkwgEtT{qq-$GG2fuR2?Qeqqo1v6e&g<_bC&I$U+f z$3t~pV|4R~Ts+v5=}O~x=Y)p%GkrIpH-*yk(Fni zaN6Oc?vYlDB>b~Nh2@WScq3fFKm%oJASQl<`}hR+Uw4L(DjgQy2+2wY$atiEK{Nsf zf(vgJeEi#^X99{xHJzI^#cTkdph(Nea|Us~WbegEz7uzDpcrL>;jl>SkY+7~up&{_ z9O;gM5>}X1MSu;IlRZJe7(E4Pl~GeiNq`<8%kQ9-1}M5FkP$}Y&SZ~bMIdj(bbp0A zbL4#4P#71JtXJQkU!90Vvq&*!g}Rl2=D=Mtz);n5Go?q|g#T25F~(j3*n}463tT^bR$8 zehXuDwqXSRYd|$pRP_QQp$B?_fD|=#^hWxv2;RtsG(u6b8flGzwV80^DgH*FIMoz% zSi}HC5>@S&0ZEh~k@aCecs`m$r{43aKiQ}4&BAiDK0>?T{drf1k&vW{v{lRPuj|$d zvMuA3mp<=h-l>75X8)-3r@l0h+ZNW`yiMSWv&fy}ZT7iK5oDLpP)oH)M>AGn)p)Up z(B4o1MZcDJb;k~OnKLvVuSz8yJY$Q2b>#`VQD|9m?>&+4T2|=0>;}`vquH$dHunG` zHXw=aFCYaQ3nkZEM2F5M0YcY+p?z}QdYmcM_C> z$}4*CGe*6`f;=)s+|G?LPQr65UTeX$)_D#jStjRN=`8RaJJ#iWqdC*PYe(~oFZ?H& zR+fUJDH&J}$7H=t9KZj`!G)>vcDt8%%`ewar`c6n>0@L29a>zapHSbNt5Rg}+qh5# ze?mj4a^nSKDu;)(@6Q~s0=REA%Dd9yxRps0;!tOZv-y5asJ2{rmy6JY9NqB{dM24< zqOE+m7-$O|{wnv_{37&Me>~q;^b@@fM*Y`ccdB1bZ_CxX$}5UPZ=l$ra`3DB=^!+Z zhsE;|_vX?Q*h4)0^1{pNf&en-xIspxhm%~;WxYXI6M!!O9dUYe|Fl7R+hNO6Wk7Oh zu0KDEE@qM@U5tL(LyT?V2{rYW*4CHMxo8*wu-YT_ahD(iClz!LB0dpKXwtsmA`sI+ z%JK9_1Ze3PXt=$WL!ZvywHa|?PJCEdSrNEZz6lu_RHKAQWmRZ=f=}M5C-QAiO~jq? z)>3b99P=FLOb%`1hX)fG8Ixm*3^o%0kc>f6SqUZ$$J( z=;ip4P1xZ$So`SL0|YPDupB8V=rsJ ze@q*S`A^l1#KSheAtm6=5;N%>L*v5rp#5b2GkAq3`yEW3|J|DDKju^`W#YAbLOh!I zSCWnC@Z+l9$iFqlMABi3?Yqu!WZKP&g|D_GxBu;;a8LfZS8xaMqzk_rC&D+r7z}#z zrJJDm?`~maZlnOie!Z+^lgrS*PyYY>wRoF-fr%T^m1n=akJ^3R_}BlmD%pSe?~C$WFQ{q17PjL#s56<9Ho6gyB}#Rw*Z>$FGj+} zOv4(fAUAuvU@ieAjx+Du^@U4*e)87n)P=c22cs>1%+$%gMj5Y>;z%ih;QIi=8Eod+ z^Ah9}kE%frfoiG8+w379gn)Di2DG+FRfCYibA5SamI$)RNQ{#K@_fnA%grW2lS>vl z)5LbYf0CAd2xXF0SAygB=L(7oWG^LIp75Xk}eRH+M%9Oz!q_y z{>X*M69|}qpj0T)F!9^XK(}F!@*Gb69~^kjG^fAf_S(@Jt@P`O&_ z{xLX2*%mqV=fX2P4=ab>KlPGq6V~rkAkGSFd21Qn46-iK@@9R86uPb z!j<`h;y1DX`%Q{mhFuM_!Y!!$Vd7}#eV)B1j%nsXBOd3iuPaP|fdD)uBR9SYE7T+j zo@5A4G(E64HBy1wuECIu^_4JpCbTmVpbG8~bvF_*(C{Y6R&fUJ_1}NR;l|cE@f10j z*<`&_nWF~!TfKHrZ2dX?4gbIDt}`miGYexphP7alwNbI2*2go&N_Yk?8a;$#`6P0h!M-&!0z?_O7b zN!l!G|9)(D>2m}$6OJUovEKT1FDUm*NI5e1ex?4V2gVAVR%8ciX)C2d{mu9%mUUR} z{xo<7YX|?6FvrlTjo;(^UW-3aLA)YKl=K4!j2~aOUF%R9c@b6=)QMTH5j8tiD@z-8 z6?8i(3|6jx`Gc>xQH$n_pPD6SUZR_#-;h{ay z{DQ2KfGvU5@x?~g&=Jts!{J|)4We*<3U9C)(^|?JF^H4qZnj}OreF9dZfk-Pb=I4> zrHj4wq_VZ@m(zxbdiRw|*!zXGa8BK>c`geedf_tSy;}19xS&jK{ThrK!qw}x0)K(X z87~TzgkV$VS*#bEx;c0}|M{d4S~e@pDcv&`UOW`udEa2*W+x_9&J0?Ry?~dK>rwyY z$v#DcprbkFnS0rw-Rf>1Z%n9v*o2N&3lsCgzDnJh85akf0nAx4*w%_}%P{8GjGhZnOK|KSB>l9}L8H=GaNLh04g>xeWdZW)(L> z#`#&Oe=5exCqp9Dm)H}e458yhf?rp?BWY7GVVRshl*(S7+mwVolHF?P+Qa$bHm)?5$;9w&r!u?kJoqpshGWrGw1*)=u9gv8EL2+r7N;33QWK&+UEO~ zk&>Y+3tLy&jrzlAJUx;lht|@l4N#{^$Y?LvIo0HOiQZv;^Xt5A3eS&9xwu_3qAREM zG&fihrPSB2$cV~P-W94y&B|f}Y68_ZyM=&xFri62J=(Q9slL2X0E}-X@awYDhans@ z4bhOx42!>)$;poibI0?v6Q(6ATNcGD_UWh z+K3_R>%Wye=B{pdMN#D?MV}k`lQ;GEm-gDS?$p}Dn{|);JYCs+@E;Y??hMLCALlRJ zc!TM{!(CUg=sy$Y@AttwU`Viyqgf44xw{l}DC_WnK&4rRpJ(rD8>GSfe4iT@I zHSS%}m3F8hN_1I4$Md0RVoc;*m9ju{ST0hVS2P}#*jzyNyx7at5gQgsHOhDW zl>#xYV(yM)$#a_-G2P!JxM7+UFAct7!(D_n!|(S%a7y@;QY{Wny`?%Wz7`yZp|{WF zupO5A)r{kEA0O~B3Bj(As9{dcR>Y9vdK%Jt3}=K->C)4!VQSY^ZZjJN;x;j48)c0- zZlIm-Bd9DTQ6afd6rj(OdKY)H;=KAej$NsA_BSHIm!kE zMw8EsGJ;ey3V<2Y3d3xzv8RRgi)Ks-qwIIn=⪙pcje-&EG~|il<~0to--;ZqA~_ zi&DKPE!LLnlMD013xq~a91tg&IxdjvP9jisPJdt@rxiVpxzi%!f0tYCndv=E5`jM? zn*?9xx3UIh5xLwkBb!E9qX>rMR-1xGD4jq@=x{g3aK!@D~%*t?P>TuMK#b2W#! zn}6QsbH&(e^a+!RDl<|og?Cfq;|usYIZLd0X+_`A5sncRPx$90!%Aa?@rB*nU42*o@i!JmP#~6&AOS0fb$=33P^xTLpVuLMR({$Yq@=Kael4u9I0my`iZ z;cTaF>^O-Dz>U+j>Ayl&Nol@}J1!>cfKzv4go(Km)5gM{d z$!J_0GFCI0f3BzYYE=FG??g}quLH!`S22d;Bm9Zu;rH3_{Bpo706D#*hx7f#@kPO- zP6*N@e?fB6EDxRm-XP=h2yp{XlHRlkoWa3YKz)^k zB#wYCRo=rrps?U55eo!s$A)Gpa+c%U753s>l9+^G<~bH@q!*45UCX=VUQ3C5;zqx6 z+kloBR7b82-=xF`btamR-WLp%PzjxL9~j9t2%pN4Ud7RT`;Ml(9;@OBxIz+|(SrvT z968MpsYy<%hhn-!(2h{14YYK3{-X_;1y7bi?W17mqDe{AkmWd>j5+-z#fLWJH`79E ze9?nV?^MBNk5-CaVN#Cw-`d8vS5$SUglB z-Af(&66rHCWK7goql3uc?cYA5QTQGqV?2og9e6OR3v;b_<+v?W^B|It!D;rbH5nFr zv726O7!d&*IoDn0w*c~y)~CU`~s95p-$L_P3(D0>Cx2IvVCar&8&g|WK4 z43(`z&_Cf+5<11z#ibOfW*qR0G6QYVv1=>We~E@Kqen^leUN0WCu64~;bSIjYls#n zqZq7IGtMkIO&U4-h>S^qr|3DUCt1WBjb|t{mUk_-E-VoDx@N_X=%Tw;APB#cvnH)v}4$g?j)I;k{IK=Pi4yK zRQ^GB&9~uVZF@mKQbn(E--5- zo#jnIi*X&BHxT)1`whb4>(7%Q2T>}Uu6!}lG%>d*+z0r|30N2@9v{rII!d@fM5lfR z>jl6sXHC##JkL=iBFYVQOXSucex~gOExI9xJcGERSAdC6+eaHQwS%Jsir($W1nB|| zFOvb85+G1HGK*vqb|i?vt4XOuNFH-{m}CRH$ncBDk8{@P800!qW(J_uH_+2x*b}^} z?21KrC9=AN?pSe76+`ir!R077jFi@~`y1(^nqU1`YydzXEC!fd?NjGWOV zx#t`+y(oZ*Wo1_eb-Y_kA&>CqCdX1>WSJ#zMV$0MNiLe8HkRsMICY`aW-T}fiSYJYj z=TRO9%u-=^!*H{$T#f&b%Hi$zaV{kT4QkU`Ccr_b+5*v7EG>ouxXa{KW3w)t8o&}f z^HpiQ4Tm}pc?0pd_>vVPbJaU6@Fg;cH4ar=LA0~>2p@Jq7c$MyZ8-+VM%z97ZLG>iDi>i zPSNx5BaT~LhBuOan(7j*Y*v|9)Xxj*vJsWr1&U)(E^&s3pt@-^GjlIf{OTscv~l7_ zE#h)|Pn(Sw2#)w*7K@I(Glbz%E`YR?mwy;6ZHny>N(>iJd~GMow2AStzV-w!n%+b1 zTk;AD(Ml)}mVpH!JtU&oGO*9(*vy)TpOt_eTR~P#y8ZgrgD3t8R<9YV diff --git a/benchmarks/results/pr328_rocm_mi300x/single_gpu_memory.png b/benchmarks/results/pr328_rocm_mi300x/single_gpu_memory.png index 88e871e4d8bbd0c1ef372072a26bde9eca5ee490..53a27857845ffd5e4075ac3f50a502f64756e4e2 100644 GIT binary patch literal 131045 zcmeFZhd-8m_&$D@LPM#@$ZS~2mh4g}qcSQhM45%McO{ZpA(4^nkc7(CK(cb%%7|={ zz4;y2^ZERKuh;MUH+*~bJdcOleP7r6{XWn0IFI8vuSXZpE7NRe-cBNsXwIEIqe&uB z+mT2VeAJZqPH*?&&G?U$gQAXumaUnC^R+uSNb1)d>~7jR+_W^|aJq5F-qO}aR6tlr zK$4%s!ok7LUP@5#*8lek0oyy~f?P^Xx_Fmuc4u|%Nu&eW#6O!1nV+zbHjzl@&YaSA zi5c&5)}gzyar~cqWOv}JG%@$d`&V-o*sM+37-y2A%%)ZMG^vF$Csrp`?sb18o^v#y zWWkU`ud(N%aeC3vnve5P#|>%zqL&Lph4Xu(M9u4K<6>hcf73*-UPx z-ShwdQ!?Cs&*%Sm;kWw#`@R3W2LAs4twE;1@#FhNqITl1X=PB3^KJgU^z=s}WTm7! zyDK;k9y~}EHmhYEaXoVUcx!hBZ_ek>pE<&+Y4`6pn|x)VQRe*Q$&-130Nwq9hWrt( zQ+*Zl0=pXReRD){3^k^Bx8=;tFkg_(9=fqRS(58*@>9Jb6;lIk&7U=A4@lcR4oOrF z?F*LZKG~F_u92d~bV5?{Q&$)3r%#{!gMz;Pv@HvcIC`aTs=sPpV06bRPti+d%d;b< z^Q8!c=FYbEFR#s*Js;4Xa2TdFtB-jUY2?ME9xZmt&Q9ppuV1|r6Cp2Oa?B1lQrHjH zp0cuftvK}2bH!;Vs}x;ALV}&6qhi6GfggpA)~nMsg5SPswdii|6 zy1IT(PKKtazv#=zE>+OfeA?)_vDm+RggMW;ZI=RdSe2@Zio(+L0B2Zr@1H*pv{KLO zY8$Fr$kmRxOP)AU-qyxI!z@<*_0hiDz2%$2T&?7cpRbS?*7@kRvq;|AQntQ)v@(dv z$8B!(Q)6Qz-)_O5?{9tic)R~%#^qy+i;Lx7zEE6xt$nI0gmtQd!Kin!~PE*JTT!&zI`E1=2LyF^i-WhkMWO}r>JRY#Ft## zsG@{*)YNuFp60`CKed*o1cVFLL`w}#GDc|bS?TKHKVXo=ge8M-r(_w#h4qhttaLqAq zU`t$b7;QeMB5yya7FNY7a@{|CJ?w&|L z{gacn4)LEo;%s_Kk4ZTU8^4oTTb|23VQU{N<)|Wc{_I)*i|N|?|CTw7Xj*Q>$-3{? z;-!)HUN5d5GZ#=D>&dfh3K}%8lM0KAvzazL7*to?6$)7v|`sKJE&@wSGnUIst zDF{R!sB5aKsv5T!2>AR<)oW>h;YE0Gt*G0q#Yl{ReNc#${VI%AAXd$JF`eT z?L@MmoLOG%51pzJEF+O*WMqWAR$b1VIm00<%R1JUUG?T_(Z7GcPtT6E?G_dmwrEa! z((SQdz`!4&$BCD0EO>7`>@IOnv}$?n*7QdI!!GOA%uS@`j4Qk2z1IsQBY*I1-n{u= z&&K+dF=tm-SC`r0J@b=2p>tzxr_;4f{&wcuIXHX`pg*cMDCsbKap!^Kwb!L~@7^8T z!yBQJwej%50}gI(sz;9=am_Rh;eLnPa}vcyQ*?99hMQ7%^Idtf^U9SgGH$cfL-ny= zkYj#(nXmlwylrQ9GD|Pt zZe{+~bu+X5Qa!s^rFyeIbc;l%T+Y(lACdLeQ3+RwJkXwFN_BU2S<-1-N6faHc;0>6~A^$%yk##FDh<>Ndwdm_%Ddi)l9bB38wI?{uPe`}NR+V&0>;>#DG z+oSyb;}{G=3c}xBY1qT3OPucXNF5!W=O6D31boWli%?;aak*GL{r%{{Lx;ZjCG&e7 z$@vgdC=nAF7&!c^AeY2}6JUF%>_fMk>S#pwo2$}cCBX@e`UdRDn!Ey63$M>U$KIGs z|EboL%eQI~dG6F-8LXEm+Sk|T5F&y5=B-l|EV(X}-IRJkHSr-5=WK`+*~9kBmoLE~ zlu^>oZ^GYSEpl>el5?9qyJgE3Y7Xz~ZPj7iG=t{HqDyY8&xQQTH4nA3vwKW7D)rE{ zm8u&vv1)!OoTwDM*T{Pnqs;Bx+}8=Nz*-W*^H- zd=Ver7iHqfRP+hm6fWZ0JkvG1F@9F61sL)Yav*|A8RSIT4VpbP@1eRY#Atg&$?%lih47NVW0o&B}A~Ju{UV(ZFWlYs_V{&7? z!9vzQG*luwGGFJ8V8YY%!$Al6XjDf1H_y3W(KGAiICv{4c1GCAY@C0K>Ck86@yx)S z+}lC1j>y&ax`&)D_fTzLaTQ3B+znla%vXtU=SEqZ!7cm=B=|WTT77$R!QCq7@=8+XlIt8 zicm=wUmMQU&2=Cb&yrQi^L=sam+Hvnx6AKt(p+|(85~hJ%~Im=-2RhV+;LRfvW(`y zah-fM->KooL%;w0IZUEPR6cq3%&gVR$YaXafz7sb$xJwsEOI1ak4pUKW~REzK>NJ?q&;pjj9k1n|5H&aMLcKE>seE(T{CYfsP@oPRn{*tN{c#O zPQ5X1;6R*o{eYa@TS86(Ywh@pYaXy)-g}1kS*l z-QKSA?Yap5`ZOI=R#H;2Y)0hz;yiN*3DFW&mxiX|xv**8ceC0r4|h-fe#rg`iQwPA ze|O8uQDm2i>tE&kPIt}sr1{>-yO)FeqsNu7^&oXhin1ABXoKNc2#{$DiW zK3Q^wzloNHn?eQNpC$(~ zGx=-dWDnfx%sWNExX7A4OibHINH(I^MVRnA4k>#syAQ=WCk+LnG%(3bAc1U&QlsSN z=1xjYwZ64afrE>SGTvi)o0QYIKMKI-ii)jlY;0);1tNYl2mElBp38hlOXErB2f`!W zm35ytUi|l)9NsDtr8eZsDJZy`0Hzt2vwDYys4iT%(ER$+j>(b*HsmZdwzj=nv%Qs+ zC`&ii-Nme1cdoCmW7|)bxXt~m=8or(l_Ib-lNkC8#KiS#Lz<{OO zV@6#&<1&qnjSbJ0H!Pj+Zv|o(6EA8_GYskt4h$rCEKNVIzRqJ*TKuUYFlA<_LA8y5 zntDHp7T{J>Ot%3$=a%C(zx{dK;kZtY5xn`QsJc_7IyZ~3vx?a z5Z=_()OYdo%@h=kDE+lBBzEKYay3-ZosMrfn>j$7xbaHyHTZJ{Os*MsQ-e-gjz%BIOKRb{0r6D3QH?Hld)Z~lU?!CUm zjS{2r#z1l;cctZu5$CaE>efA9zn<}!?=ZLc@p2Q2-S7T>AE&>aQEMI)Bx&b~ugKOK z$tv_8i;BJgyP&>&w$3i`1LSAwx?+^+va#;zFnoXh`QQHtA4?E)5>&aWb<}EgJ1O;i>}ND&k0!c`4jn$c<;|NncJ}tC(V8tU zOormUqFS7;?Kp&ik88QjVh!P(MD`)zJ{!opdhDZiIX!aOUWEevGX9``PaG z<*{rtR_osKdt^BV0fQf%X6G^^A3wf}_yWYHym#-Oos*MaWF*6AOU6SS_-kk>m3|g! z^U=3v>iQ&{e)#Inz<0e{)M4BgT6y)a(XV=>q@?`U+M$H^_V-iB%gg6kwQR3>cFad5 zncTSm1j{I4!$MT?`~hUlAJ

QeNfxDk&?g4ul2-Y$h3$d6oEo**aYCdvK6zZtu|( ztf|ShBf@|J?!4;gPg;2%-xp+HV3?XL+hCR*1jh7@kB|Q(10*BZ*Wa%m@>T4^hhLBO zai1CZ9znyl`!4Qaf7!-*QPh<>p*oE-3e-e!ZaRs5x%}tPpW&`lfbz|E(=-zOXcgKW z=$>e%)8W3A+}k#=IOKVWZkI0>Ufkzn9+UkZJ{-H*^X^7qUWxN$x6+K;?C_KD;~X62 ze+pLTf7)ug%{jc?%`O+NWPx7Qe;@bx=e1RO{C}6^k==RMkjrLMcr6qCpYiJ+{F`g{ zTOs3FD$jA-?#xRJYLa%ncf(83ja3+(+#xHN=pVu^FOY28U0hH-JL-Pu?!m73x`Kv^ z%1V_clD>%NTh5jFzqy^?*sX&PF0alse3-gF2ce#S`gQ+vUpkkfni%KD&Kabc(Ws`dhhs`}Qrdw}~)M9<2<0+2@|SEVOv zxc}hBiebrkOnPMc4A~sa#pk9bEfQ^9T$~ARS8u3qVnTrAj~02fYXi5`%17`WvnekS z-{EF>x!zBkuEHDBGgN;XSn|Er+!EVVXL6lJn|b%Juw;f^_~a(yJYitz#OIm4iGtGK zy6uhbfcuwdNA)`9DQj)*a9zht%chjR^|ckV_;|U>(U@cr5s@?aWSVDfEiIQ;=h}?S zsuUFqo)JeRBqU@r!JrE1&WDH7jTk0OE`9&SWNlvioYUayRrONmKnB6qqKČUEP znirnd4D{tT$YIM;EWG3d(Qk-RvPwH;=FRpOyUrw8XG|JeSv`}s&hxdBD^h!LTs>3& z!+Dg89m=$u2sTN;u$Y>(h$~2rYj`hY^Xmjyi#_Tuj#q%QpxwLoKGE75m9jkC&93C+ zbeu@Z21QOZNRurc8`A?d!(Bxm#5B{8)?5Z6uF%raJ-{nX0MrQT%WoH}UfPq;t*oGx zK#%w-B-~ZwIt|!OWM=Kx+S{U{qE6iTLyZh7$V0cic)rux?)L4R(X(-gCgV@{x1Lo| z`Hb#e$gcNf*56Qe`DbnyL)c_Ht*XQy(r%9TT25~^%?C%6rkNaYUdr(*#a$3J%yqf= z7W$#Pn(1E$Y9cuw{9yJpJ7>6Q(W@44f83I)}!v+~O5<)x($ATvr|&eXYw zjpqbaLC>dsvdHQ0zfS9H(3GFK21XkbslG=FhGJxM5AM{|(jwZI2M-^be)ZqODC6?BwInNPh2S0je(2duVqswsy4fz|Fy6ig(CMSy z@6GG0GZ$n$7HM&#!#@%zLRh6K(dqjv-7q&dMH&0(FtQf`=mU(CU{VnPj3Ue?=Rx$H z37uFbT6$mtIu# zk6vq1#6}A6_HHNv!S;oKsAokc8-mXJ6Maft3}6SVq)xO2G#km z7Sgq48&#nb^g-mBB3vKWch(yt}?_7n_Vf;wq}PhPmy|pKni|%eo&Y8F#$9G2EE=@KZlX#7u*N zy(icG)lMfC1s7m>_1@p)t!vgyP}oXP$!IchCaBnC-5y}GOz;7elU-UVYM*e%H<3z9 zOBs*fWJJaIJuu)4SY`}bqWyy{yUnkU_s|0N<5ko{^krau7mC%VCJ-_C{$8nHFR$r6$H7Q93fnmF{&?fKRW=wdAL zRzWXSAVm|rIJ%`X%j4^~F8{#5FHoQU)k{wrgMlAz%{p-B&K=^M5ZJ4}zTW@Qqfe_# zGfyHTA0lCF>nd_q2A$;?lBmtsn5ev);C9?X5!?^C)T24j_+TZzl9N@RgoXKZT4Kudia>ysM>v zc~7C^7{0NYV?*@nzjc#iL9umJ?Hxj(qp3{I$4tgEgL6a)YP8|XxPdlOz*i)&-rv9P z;rt&0vIWgRwTq>kxE@^p(#+78pFi~{GwE7`BH59|a7q2>Tu3B#cJ`g@axA}p|3*(! ziQTrFAJ@IvRY;HE$z3!+L_EUH9zt$H7FJwzrBW( z|G3^fMfG0G-#ypnk|pzPd6`4^*3hfivAfKG}%~RBfvZnf8a2l*Jh-( zy+i&d&I{Twb{2Ur773eQKcauXekpF_JOSBfYz$8g8OuG80vxIk>4#$r8*i15X7 z{OW++! z9etIUI5%hyM%;2u7i`tk;*@F0NUeq)-4j{`?QFxVBwEkq*~W7lGY#^pvTYOHB}X6d z#A%!X4YLot1}*VVO{5?L2o?q;rtoPVt(2e>j-%n-p5kra#LbiL4(gST+G`#Fbg7Pu zj7B@jogDh;k#N*Hk`dWLude$Y?wlSen`_|%mzM0t%=Gjb1%>-M`Bo9Fohcd`8k95q zXbi;0ejsk=LU?|7luT6~I4CIC@Ot3*pS6Xa$dHw#q4)!8kx!m*78e&MNtq7Murn}l zd-??|zjK&1*HGTWt>Z+@sD49Dc5OT-Vcp6<;(|lm+~EaC0w&z#Ez8Kr zh{?a+Pnv9gaa*^`xg)z(4yPTe)6fiN7TpEuOh?87dJF1yM(PsE5l%~pxv zqf`!JWY&26_%Xq43r6%OhRTV!@Y_Q&c@!;ft+F(7K=N^H>@gItSB@n#b>-KxtkX+} zTwPrJ!O{B+^nj2KZvYXIlH1kH6Iom=lQp)400;~U+F4Td9V9FYW$V%!q9}i&VR^KS zR$+C0Wm2;c%`meJx7KzqIkZt(qD4NM>CJ9_=mye36qtZfkXiy2t{xrnsjcnG^&2-v zk{*SIG9V9c{ua;Lhcxe2;{=KnB*`I?KP1)?5XC{srR&Sqv0|ZYvU}-|UJjoK#QCE` z4(mf7EoB)gaFuVM1(l#=U>Wr!Sq$OdEG;sMGWHoeuR?H;v0_`}2|04kD>Us07ne#y zZ>3q*o<8q}CDd^#VIQ;~t?J0@Cx~7|{TiVzXcC;OsMGta<2UP=PLK!^E&!6xSd#Re zKmHCE!fOWH%rE4a75~B~r>jA*6FET&fn0eyeZp-v zt$rSy)iw@qN=C;*u(Pq#5xiPTBwCQ|58rNV$e0JC%g5(vb8oebwYs!#Ut>V^|*^sXocC*O{=}e4|-^`la zd^%TBC$T zttsVMG%8Fwcc2kVTHRUE)@oJOmz6<9G7prpPJJzo2;sUlI4$-x)V35h*#L$VG^X^J_Rhf<{EqfuwN|NWlP z)OfF_@5ih?T9Y~$-(+jF7%VdUA6;D=uHXEv675Xk=*1F)%*P zZ|F91qy&0#wAL^B;sX6>FF+LXdxB7{ts&1USX|K0@H&}0Fq>^Pczkmj`_4lL4}N`m zL^WtoSLm$B+`Gf=ylRKS%}QQ_DVcq{)L|a~2tI;#2UR&8;2TqHmZn~_c!JGhO#og^X#47B)?vdm4S8odJSOD%` ze<NuIN731`)3RyN3Sq zSry>82mv^2?qv^ZrcdzCiWce{^%=$#`GcMgP{S!h_Z>KJRGit>&27|^u{ARg=oURH zs_EG!dJmq<7jl^i`uoCY(dAKjJ4v)ftA}!V0qDZIivVZF)DvC3=l=Z@lA^f|N>nWP zRc>x>H%VVT_tY=)%AHfuZDnXGDJQMb6CUK}S5t9^7?JfCqV8wp{6C%-{8!79k(xGt zGuoc`mgr#8W&GCDk_0(Qyw(aG1i)MqZV6ujpH5KRN$(h~$E&}6`?35bx>R0Ce@4&h zxZkm>h0Vrt(cV9r+TPxHO;LsRHTf^<9&htI`q7~_2oj00>JMm%Q$(J5UYZ_Pm$0GvNK9);wMUO&IdY0PFW7csNiEa1s7StrSto^ zrdu9gJy0lbbzb^c#udRZ*Zo`nBRn9Se`$}($))|DymHxeeANTY>_mAEj3YjlLv=5> zlTl-J5<<8ijJPZz6_Dql4MYWfniZzw_5ue?oSG9u{Tj6F7hN5YN8ovC0b@*E#h+g_~OLvR~1#7O&;vqyDx}Y z3Iz7|^_l#e@j|mTA~hPAoz>xW=2the9Ld`Ax>qWM$XhGQ5l9>sS?Ve4 znn!IQIB+$bvk#BS@vA4dQqPTdh~8eFwe@}C#7epI-uiUXVz4%fvWLM_ zZhf);VB`i8mL?k^*>sC-r68kA34k)N>CXn|Bq|JFbIuO(_&M9r-3C3L3$fvsdu9^E zvy;t!bb|^hSpfy8a_H1fankeA^2S{A3zkJWel(}W4*V)z8axm-0%25Rcv4=xu3-pD zqPfsj8=E(+JaFb{|^E6wHm-a>7wyY4tUOJwBe%!ID6do3n(i?MtByBA)m?O{nn zgkpx#pz-)k_n_E~heRhQURnRl6!l-1dd#UeP5;B2#yo`lXHcl5rC8nV()Lf&U6w$$ zMZP}}{kJ^){%sRO!?@={*Y3|2L0*0I=g;*-R}kuFSv|H_o`c(C&q=TGxM$C}WUJ^h z4@C>Z{!p~TQ9yy3hRN5?U&v0XmRT+L5_uf!I;S!UkP5j- z)Zk`IV&$_mhH7uak-^(Quw?2lA|j4>czV)!aqF+>S;@&WT8Wa2Hx4k@nr2c|Az>%4 z1=DDXD0OZ6fu7U+y_RQl?l@g?e{x|hK#!97L$U{(+emU`&|vz6c9LM}d$W~TvloG8 zu}y~B0frV66BCO-+fz^kjX|Nem$*xyqq+}MfOWF8_qq(hdw{mNhE_+>Q2!%*LP{U5 zfDwcy5YZ~y>M;H1+=#iS^^_%{|7kBZ$H{pHK+&Th5!=;T=!uSrboc0$H*_QITenhC zC4w9vJ_EAAediuehVqTpU2{`Yyv=U^o?Xno#{1x;o*px(PJh&x-(?$Xf`;yY#aE$C zwC7pw1EkQ)9N@80pjLY>vTSy>7 z-dDp=8+1To7ebKg@rLVB7o$gkfu|iD#7Km=4vIqJc^kC#%)(0N+h}tX(EZWaINRHQ zA$&1VmVjyq=MK6+Sx^91G6&oiy8iQW7!*1XE2s$JNC?gnSdC5!ME1j>N62YU!@>xu z@K>ql0aESq1mAbh^SJ^P<=}ZVvJGWp9a>&r`U3JhM9y=W^ewvv1b+Ua6y38@_j9M9 zD?ari+Jihl6EvvnZXQYmkp^4D782ofI_#B{swTQ9v{aD^L5z^Ifz zdE4&Y55Yr$IC*Q^Bd7hwK$p)O?Yg$t>36xggahSfXWll#Vgw(-leoAy&g#nct@p+A zpDlw@Ns5_`!Xy&3 z9M~@F`_{x8fO&rYk+iPlYIUhZ1t|D*3t$p`?oOE0K+W7kqGnXnhu?2x=>F}RWYx$d z_t415$d21*;6JRk7rX3LCnA2Uvw7G#K4&^3-t+G2N+IvnLWfT%7MuR`thDsp-YRbM zixJ%=*UT7*TdU@@duS?5tDhcO9L>niYduq6S62bJNg|=sIg@QzqBf|j9TXdE`aeX| zP+xg_!5_l?#AG~f5N%=}rDpcdfb{ly_1=*vt#~xrste+$)gq9xm_TV%0-3->dk=kk zA{YdzkpO1F4AT?%rGL6zdv$en2|n!j&2|!rApBb2T-||HO4*(Af&y-?zR^?W_15;Eq#_Kp{VXj z9}vY9-$HeF>>Em(+)erc<~DGFYQ3#q3$*IZp3<)CZgaWLP!Zt9gt~e;t$*`La9Y2| z#;om%%zi=wTn!U^lH&!+rqfiH$UE>^_m^l82?qU z533u=?23*h0C?gzq@1JVsqfE@SvwR32JRr8-CYt5OI1>QhIYD%7(1lk#pUIL;^HmS z3Y%ftdNb>b!<9INz)rYMA10H@LI97k4p@LHWX~kO%UG@4P>ts+ zF@tLrRaI1AoW-AeV`JUJWasy6xfMONJdfVI0f2gW`T?NQ8L*O6v^RmU`a%#eP zO&yA)_D?M=NVs{8I^3NQO&^pb$a!QHi6W1b`bM24@tW%suA2)71mC!qrMo9at8slH z{2m~Y*&$bc{a<;`as`!$@UxN1Vd+#~|J@SJNO*m9rsyVNM-wyH@=s0ZT#l* z64RkqA|Gn45UEnECo=a4W=Qiu`l!FIe^*j&dH9fw=Q3US#(Me92~SxOk-dneGjJS1 zsPa30tCO^7RJGG_O=|-r(bZqYuKQ5A;EecukAjNyXQNT8M>LjpGrQ;P9Gxg8Uh5_s_ zI%m7dZdtG=l!U9F`IjBwI653uki58(+T@OV=|Vj@IE60Lf3}f&!E@|)J_sg`SJT7n z+|P#vmUCUUSzTVK*=mXh)=vpOz<$k6_OROz)mPxyCJSni7D0OD-(UXB^PT7L976)y z!q(vLwDw9aOt01zc!J<*Q<${=R(_A-_us!X9NuDIu-K2o!?%!}SfhTR#T;35p7`|% z&ATJnV*ajHWC!CMO<8YgYkkH`tk~I-C0Q96I|%RT+_nH0UK?x9VO6;dK!f}`Z%YYN0GBF#5T0*OoW1oD9yRqA+owMS*CY`_r&(Dz0bOb z;F>TN^M^yrf}Hedb^~Th`ofzP$6#_=?uePxt~+@O@mA|8q1-*WZG-c;}-F8IoG~W-fVxn2?c2D zOKFo#p`ldQ4H5IalNxd6vpKWRw@!XgbXDRz(RTxE;&YK30dy?+w|?%2G{n1~U*|5U zU)8n^NOX_qHTWWwLOCBiCL{6~S#D_QjE-FkUZ{_5+e;$M_*-M5W5w^Tnjf4y>yPq5 zI1%@=Yvo?c5Tc>DJJ-;mIaqm6@sV!849hJIFZZ66d#j5|S0jNSv-8#0yrb0o0I^@h zn%_1aj|t7mb%i&1XHaY)RG}tD=^@kq`R33SmYm$e7(YV0CQWCDc( zzg(Od(!c${7y9tXNba?mCr=WbDfge=UcaB>k8{A~2Dz8rZ{i-E7VRpqA4-ycz&vU( zi^+>#tk6T1*zDpb|NpDW(qhT`oo z)~Y^QtnvP)E&*AghKdsAJt$nNllKPQEky>^|Wp>y|T&U+gbOb1(hJ^e3aP^XxNM**Qtr!0?6rw6OA-YVQ<>x+o6Cy zHra9mz>knkvi^!&H9uy{6LWt(5{5NWR~-NSJE7^zrQC;w>BVJuHCe` zdim(6kIii|3+s(~=htRGOzXd?+ao`kAnd-?)_#T5*9aSW>=UL@hm&3nOGxI4s5xZn zUL4e6)_gJZdKODM=x>QR#-BF((D)N7(WQE-A6}J0slVUzG>9(ixYgcY(3{)1A;L60 zJsm(tN>@&mmXb1ou1*;#S>Q18w8NE>lCrO8vh)kCGYG`~>}bmlqB^yd?vs_3B~1Al z#m=q8&rW*zxWMZ3)}~8}un}2@_W_n9%ZKkj{`ShoDY_rG48$VMJUfe=v#XO?RSwnJ zufmIUEY9+4x60>|GPNiX7j&6#J(lheU}c z!W1yOeg^MLs-sgj$U6o4@?*`yY|NvWik>R;LUbHuCx5G-Li4il}w zjDrcF7in7l3Y-n1?IVpb@1kqEZ^`&dzA=l+Vk$##qPcv2gq*wsaM}3n7azBg-S%|J z^-(5u3)v?+H!Fp(Jc10j>!jC1nmLiPkbQAEn;-Y=YDM@e3>Y;#vDMkzw7p@)c!yEi zop6U%o3f^dqIQofP^9wiFWhgk_^F8um>l17FM|_uqi@{i{XgQEYhuJ#Zl3B2yqfzV zR*K_2ciqX;J>qQ!3w6!Sx@cu0+zeXNwUo^}FR4zu!HL>TO;1Z}?4aHW%lf%f>V$^{ z?lr2DUdBxH^jkT~lvh#tt!AIkr$`Pwm6gvJwEFS#-fXVqy^m(07wtED{x?C=+?(sY z;gvq6RkF@v+)e$8z=MEjmt8xom<{aHwE40u%hiHgbC1+fDQS(pjy4&UGq=3js%Dx| zXut^T%ww%7(VYs%$!|2Ynq6Xfev_a>z)9__ zjgf2`9u};Vx1x&}%r9`ANvWP4Ov*Wf-L2*w6A#t>SE4O{Xd7peJPUm#YVYl#dREd> zu!SFUBB>^O17s-?t|z3VRANtL(w}*qt!!fQs@ji1Yce+fO)^(qvWA$lX__}I=ps^= z@`Viz4au@&sQ13ibDDeObF#AX;!V$eT%ah2;H;FH$DGr$cU_O!(3VK*CFjBz^7Vc@NcP3Z|cs)i?pO(1TguIfg)kdz}2dSz#jsr;tQbavAtGfOAwHHK8XDSs z?vvqM9_XOxhUroFtt5gZS2E9d`{#f2_Ci#jf8c0qRv=s*n@I0(b{v%L|J#wPtaslU zmT+G~J#9~K5=Y$w(`_&rq}!i9_h@V2>Dzfd^j~+eX!F2eL2h_IoNs$J*Q_onUOd&c zNjX>9>>6R%bsTNpL}HP3YtD3B3D5`moS#wzp@uO2u&Rb%m{C>jk*Xucy1XuDFIf5y zid_V9Z|h)VV&YX#^?>l1s1<9)rfOtpg^r#)n~lVe*@1-1{cWgXf-NNk_QzPKQp2Y> zvDA`dAtUXX>Y|H6Eom-m%Z`?M)!XU$+Q$tKLv*h+oNLa8XNWMd++CH@YL|Z?yBp4A zPS0IOFTK7Vw&%)NZ#d=_$yN2ZURjRIL!+}iB5bsMK0Tu37ugPEb$ zf`o!SQ2_fC7eykBZ7PHjOecBg)&r4Z6i##Bc5;;MIoZ|z5p+jx>p#!Mf1Gz$DLqVS z(umn6)SJLN%z9$k!Rbl^aGTKa-4&^5zAYY~yFKJ{6HCJDr8s>R&$Uf2QI3?FqTRfH zn16fBsZ;7v(EKP7L3?@RYl7E?HeW>bP#xrX#x5IB>*@FbnLQaMv`w5_Akzh-^pd`9 zOo-=CqzaF&Sb|{4TYq~bpg%@68jZ4QqR#wyhv|H0%@N;0SryrzzQWO#nU0fG(__(Y z+b2-u_dzGtR1Sf+GXYkP%hJ!Fahi?a+zDs6PfqR}oZ%t=<=iDk=@+Sozt$^WdtHmh zpXlE2u1+4Do1ruAa?xG$+QIquM$P_2%ev%spTa_$TYzRDi-gB^kB){YfI@)%&tLnB zw9|N6#QCf2>@qHQ!2_HVt;oCCp*g80T+vl&?61ZZ7UvBc)Ac(_bXtG6_-cd~n$<0~ z{8}Ia1zKsQqANpkJ(Fp*Cl^#FEUDPXl8>F6n{(65z#7*y>zXdnoh<4fM2E5?LK#_& z^v7A|s3q%2X2t$ny5r@iNO`tNDPJoq37wL>A#OYIKBeRA0>m(;s|P^B>MF{`N&VC2 zp<9GN92gTXUc53zjK?r|<=fR3mKQ=gm+&jW^3MmMR;0@JffR?oo# z@`8;y+7gA1C|Z2`b%;H*3OWNf8uE+8w{Z&H>J-DAOhZZu#sG`wTJ^~rU=r634w%<_ z)?>yQ8FuSqc?AoYI4NgVAOj`E6^@R)Q00SRoG9`q*Qqa?2LA&zssM5WF_j^{LDHa$ zP9%(Ue4Y=7UvyrwcknCRR1RaUSKzULl>w29T03S~b#Ak8X6+Ylc}gx~hlH!c%SP(Cl@M zAaG8ud-ldUcRx{@4T2f^TrtlFfl|%G^;NfJ+6v_HlYLl3h{k7ZqTQSn1;MRbTq6DRbl!jTYgww3@bV?Xd zNFMkL9KEAO@|1nb7yL@`j~l$z0f}qpA9Czf*g{1Wi1sBA?tDgdXEfoBd6rt6PLgOr z|2CgPYsPB10(3by*hCB*1oNhl-&-~*brORTJCZfi!_2^(Xfl%yiiv648b*%*UZv&& zGhW36?sk#_9O0m$4JW6X;cTry`~4MjvTAXuoTL2(A3xGbg~E${ZpaazQer8V9V%D0 zIvq%)k;Z<4=_79Hdlj4;7#tf>FS^~Yl(ldO(~Q( zu~2gaiLYKckYi@e4gBo6XE{v-WQk0Y>;4Vk( z!}LiASQ?1b zD{H8)-w(2zEc*u?mDCC$cA{spi1_(qCM9P%A6nyxUSUy@zC!~qkD%a%=1g5Jz6L~x zBhj<_BP6UNijd9RW^ya@`G=v#YQ|w=`!zA9GW#n$BI1Ta4}_<4D*}3XXD62=VC7Yr z`G0Lnt^H>+6O)?SE~-49Svv3Q>O`*~9k-bS zO~6-Z1de0~LuxMW?y3u8_l2M2_YXk6P_dM9p2%E~+Wut25Jsx%VORQ~eHg#Ui<4fN z8{>u(Y^OoO>T_~UjcUev0X#Utmk94w#IC#TW!`1o9+O=~+{sgObM%C3{hU3Z73S+x z9P4&^He`G_5HKK)XAzvud7{SEbU-yVIr$Ncyt&_Rna#Be{ zp!e$3Lx*g!dDnciSkW7xCp8T*d~)#k@ypzG*sNqX>v1--y7we?c)TLI2`$Qy?+Z$r zT3mekWH@2VT*4ow5(z@v9&*=D*ABoK9d9V%dAV+GexMfBz#kh*SZbpAi#P|p1V9fUvz)^yP|s77c!U+C|Z->pM42iDPa6VKwnNgO9I2^1fz-Qc#E`Q=_Ze2 zoz?h!ei~D0#GDzP3_y@~`1!x@N3%J=ei1K6*b#yt0Lxx(}o zNVxVz_H~^SH*wfXe_)W3C{!S6 z2%r92JOF|)xSm|6i?4KhjR{CbNZFhK7SaUI3i;*pgKhipe4cx7?`0a58IJM}kBp>? zFQ3suqfJ~nX{n?Zz5Rtm$ASJ(`AWcrV64P7q*K^0>5+<3!fXIC2-iBv7|ES1Rj!zD zk%w5ute$P0kDopBgLrF-VKQ=I2?p*KA%_NdsF}h-kRh7lUdVtCQG=P`Iba1l z3^r_scDm-)?(S|VNMDdT-&(cs9fs#`r8ZXjiusl}F6-8)HbZwB3@;A1WH4Zc>ONcy z#9ji2RNyfDM}jYnd4c9vF_*Oyx9%Z$n@7ZlK(kWR)Z7aTr5_w53DBZpYa}=?!a5X8 zi~zh4d-w6Mb>pk8Dd%GYi1Ef_$7s;!R3Uc}73X6?!EY$eP;>8zDHLEn<1)KrqGX|h zY{i?>Jq$Q*esEJGkp!n`xx-{lIKnVSWrBi5j7wAQVj+~wUnTAgM=xiV19uUUD&8|H z=1u2b_!>45>I+OjeK=yoVso93@dr;xu;v8efxHBnD-heg-?f!mosbye$PzNG+<8V> z*%aL(MsHEXY9~2Fg&aX zWax(zj8Vw=t{C5ET5E%v`%k!-qNxdO?Js zLJ5Kc0cp|xmJ^TWc@iG}08)b~rZq7u8=-XQ_1s7k6`t22S-RM_<7-t_BFL5D&itcD z%R#8mn2eQio!$cdPXQb~1oNHvgfP+Q@WXhd0uf!rbObtnKuuNZjANOOWc0w(4qWil{_{WQ~1wq##LFrL3{yc%hmtT?B}=~dkuW%ZajlPfp8gk zt-4^y1TL;V$P_w-4(Vdnd&;6dm`sUyr8@L2?4I+NH&BEHU@`-F*x z9QbR|da3jr8T>ap`;~ak3hX_AI>rE1guskWCd0W^2CI?>qJCq2C0ly~8rWsF4=PXcLu8oR8cH>BnY2t&|Hgb3mv9IOaO8nIf$#5{UyvLh_O zL;@kE4}4tdujrR!lsm_M=sXy*5hg{huydkbbQQ1|z{GBAj1dH^*D2sH;^8u&U1%DiLm6G<^5*$Bn zObIJuy|<8vC+CzbJ^$IifN#G-l$u>VibgT#*3aW%LzZXHF)21@8_DO6DPw@7mRT|) zA|kn^$fzd+@x|qSVEdDv1B>XTg!MveB93wsq=Zl?RQ_}SdC0=5=ni^s2^-|O2WUN# z)X2i55&lFfI{uiw5uUAM0`y2Zlh?}g>i(g*T34pAr8gIOzGB6KCnkG)IS6A|Zq21| zw%TPp6k!KZ%SAk~+LvYh#4Wl{*SU_2xxRq4etw;J%n8UIaa{rAcI3P0U(R@JfRgg5 zZ5lEQlAP1-FYwqQraD{L?WX23dT7A<_g9xTRfnt4{4ig za&oXa8R>S+8D;g6<^Uv>{{)+?o3XWT|*u>P-)Dz&r&a)BU_g~deAKhc? zX9&w}@zUT6O65KVN6KBC)5<}XZ1#}Ve7~o@{ZGWYxhN&-)JhPYa)YIe(myFphgKO( zhc*a@JREZMjQQmstgw&u(|WiXg%z;Dj{D;w4l0@icMXScqwPr*S6#L;Ku|B#W9LHI#B z7KDg`FmSY;^2W`?57Iem$vMhTqj#`vUuMJdX>O zIvR+@$1wBxAUiu7H7SAMTT#V`Iqc)Ez?HCpqT9=Ho3r|FG(FFzOHUiU=m}}*9e66% z@1Y@X2(=tap=`%s$hCElI_!y@%tc6^kPUtD&?6qSm=>6MC;WYi*d9e!R!l(=)Dw`C zB9?VKJW<56Jc#y(k>(`0CFhY<7~yxG=S^=Q8r;9qn4pXO8y(Dx$Bg6*dx%9fQ4jq7 z%?YC7Oqti(*Z+sN_m1bfZU4vLlI&3S%1BAFO3A83iW0I{RwTP@LPj=aq!L0_*<1D| zd#{X0_R7xs9w%M*eSJQU@8kRX@8@wpZr6QZyuHSG9_Mj9kLUAwxZx1q)hY`fz>CiV zo+A=+Ld|go)X$!ft|3;X-E3-p(|e9>5VX7sG<;0^Ew=5$y26=|=}6z!;goi~cm<9c zXwOuGDIF^W_49ylp9fA3_lFN3JYX&35fYvR_WA*%w?A4Hc0Djb{h;6Hi-4&OGB^X| zifl|#7qO%>bmqZ_jY7yWd{CXym8Pj~XK#-WoDg(P2UM_t@h5DVWK!NcD-OW;mH;sa zDy%`tOa@0dGP%+Prp=iRZWGo}QZYc#qy@#t6~H$@I0LJ zEO5AaMnpt_m^E(`WWdBSiJw~`IYg|I(|}oWfkTN0kPR$sZKvya%ak4U(Hr z8a{KEV)k{uz&})M|6tPD{IQzJy*Eemp{xfd4k-k^u$(?E4gnJ1(9lp0csNKzyF6q& zf(|RXW>z*fi5+kciG_F^3sO5Uu8{!37D`)DOLMsb_mnqW?pu?`e6>*w0v`40tldb2*jAM^z@mTpHTR~h$sE+o=^WURN4ecz9n84h7#VQ~;o zRYOmU1Hdt(Tt1pu5TruS1blz=uH+KQLUX?9Ap4oDM5H;O--8zwKzk{TN}1WYLKa|& z0ICh7lotm*G4jZOt&MudV^AUSzl+Wtdvp&aEYw$@enJ{`u*rEf+z4=&AJi))Bdot_ zeg5oyOiMS56tEPV6*Wl0M#dRXo5%}3X!IdqhMw+AU^2aY3^oerWdW%+7S2swNLR)H zUxSx2c2!!}{vNOjs6aVJ2Imn9EFc)v0>y@Eib|#!o=zn79lY6I{@mEwbg*IA`u%8{ z79x)iXmh=RchW5sCxIQ%KRPqI2J zXckyt0Rta{L1^_`e&aWh(&F5{AgLOI$Dbq{a|GCy_`p4Mq?-0@hspw)PUJ}HZohJd zLn-C)A`2N=7&Sl*AQry)_%XY#p5DABJqt@V(!K(}5(|U;nBHD}mhhZ#m0ox5?s{ck zzcaPaN%%E0a9gMa^a%^&M|q(VUdEg{6AMRhyxW@C+eYpWCJ6dh$MQ3(8yzn4fbRZ zm?|Vac@jHJ>XF4Zn2f#L-`gm|xWe{|fd!SvL0d9vvp7tQL72svyKl88JBn3cCE>=) z<3qRw1zGo@DQYZ>Ae4GgulkS`eVA4Lwb61K|(vixwo)#%c zK@@idwb(ym^?CU$?9+QG2S?M zVB%~qQU_QB{(FHG0}v*W;R5iI;5`n4W#{2nJf-0%lb=Os=gK)=UNQ{0G(4URZd>tR z^EdId4lsG~_j}Bx?u#w8vV8a8VZs3hHhbst^fE-owiMnkTvKq!jCy?`BkFZ(M%v-` z&^KQVX7lh`%Ku)DxVUE)!l-~r4xnFYiJ#cvUsss1Ja+t_hnl4#B}Fjovj5x(a5Nz0 z$_$7BP4xlNhPmm)U2zZ%5 ztPP!jLvyT0N{rxM1ovUh5~mS)PiC%GJ);SK}PBenVue$D#)v(lIEXx z94`C!Z?SMZ0^^}Tz+NdG_6rIGAc{jmPl-XMU{J_^5Vk)73i)cl98mqWxwSPt_eFsN zLCP^u(Mxb(|NTTGaP#Xp&&MafxOnp9NmF&wEFaKt!dunCq!{X-Pchaf{O|X8aLPG_ z&i)c0f9fiIj{;l^i2O|Ohk;Xp90b~7u|sW%zzc}x)yOmfP|Q(4CffU5ph`sk901qL z(kG&L1z)R`y14&5{aE<~5Q{>3*tYd3CmdoKm;K#$V3w;swgfWkQ^|s(3=|tlm(HiAZ7J+iaPw$;@l@}Mwv>W z1C_hS9nyfofSHuRd^Fq>@VTL4y`HR~R%EAe7n+6hnjS2`mU0D!0f@Rsr|JjQ?>t&~ z?va$F(pyid(ps-{_+2VI)++D-p!^6|DL+Yk5AcC1$VT0ODya?B95n_3Fiiq&Kp=X( z;H@IK4W=x|F&5C@I*Nls@zvk6X5G63a7_&R&^(!m-{F=w)>XKn6#nj=&*r8bJYGjC z&!$giUGi5xgc`s?LYA!Hd?4 z^Nl}+P=y`qG%}OEefyQ5g-S%B=`j!p-^vHk7YlUsk%~e+j}x-menvS05TRpTPhMDJ z^0@xw2}f{nFeuRjK-uw>GVsJ1Hny5!iH}s6S(<)=S=#<12jAa2$7eeVg{c4o1gL`M zHp5KnAr1O)287T$cdflXd=$UIH1f_hXu#0jO)R|k?9@#lbYU}lXc~-QTmhPgcnU~j z#>%n&(7R{c0cC~d3K8Q~NQfRvFW^h=oRULP{sW&+T1BCqp}{5SH?CdeY*JPw`4I8R9*l>6Y9@~KT_)z8=n6O zvWLO;9-ZOnN8O4RI{o`8E{C7{2bhv~uAL_*aG$=EZ?y2BgO~BNqs|aiT<~SA{ohSY zC4RsWTO27f=ZXAvx%v6u0GhOEfltAa+mZE@EIQ@-t-KSAT*vX2ZN0*cr=y>S9R5&J z0hB&;u*U&QgFd!6#KF~sBD-Z4*PUdO-Q8VN)#?*L^JIVTR0Dkcr6`z>whV($jO2L6 z+79;5pUIz0TLPc18p)ES=iuFl(;lI6p7{o+J4k|PfG1fC$o4^AhAx`}3rd`JuzEpA zAdu=dN8L%RI`Ehg_>Z{`#0OK&>JzXQf9W|Mu0y#WaJYuiPWRi}8FtgmmHxiHxc)(y zq9?l4Yb|jx=Gpwyw^vhmg*BrVIk-X^+MqEF`wNV!|6Z^Z!dHPoj~s9i%^ZS~vFXDi zn+rguA;zE@`d%X?4NdaK23naL57@~ekbw(IeRFf}lXo52D@(%zgnf?lnZxt%=gAn1 zkBuf)tVM6H<3X8&Y}sLf_Vg!|?)@sJg11#8FX{-tj6Q;${LJk6ekCa>-x9y;+?bw= zSrk3|SvZH^cYm&PVP0Q_kLS_dlXH)}LNM@rg6vk)HQ&{_$wcOIc>vWaq3}T%JVc&@ zHzk#nlcJ&F@~OOk+o&LPiQGuFXSeDKMLL`v&=SAH9me6Jgy$(1v$hT0p443^^aCJg zyNUM#+6g)ku`$2*F(uGiMfOveHn1lENWLNOtYYOS!%3a4*pOpF=HxjW1A6{yR=+*X zg8&y2F8woe{=a|T--OMK&2W6I>aK=MXqha<)~9tk`nAe_>#8rz8=* zrlwQfbDkecyBZqsq1&Ei%X@GKNglk#aq5PSfl$EWZIn!;;!^6`yhrxDHl7FH8+ab-q2am17ahS}4LONHSopu} z#*}6Y$7<|6$;YEa^a^8BsjbO5 z5h0S3;ZfDDg@mA2jvvXnSX1J%{z9;FM=OR~AgXS|Ff%}Kb_cXY` z`9c*H3p_H|UD2Tnz#7{88FHT~ys;gGEr?7t19oX#h190ByKpzp~3bi^i zt#hDJr3^AmDrwP`e;eCnE16Q4cUH0MOj2)H*0n zShAS8?rrV(1O#}JR{oTJwoZHxSe$P~p9><}D%c39frqj(P!Hq6mK}_`i(d_AtmMHN zE7zVWQpv~13|TaD($5D$X*`OWP@H^}Wk)$3TwhSHR}Z$pa`f$9zP~&{ws2+*l?Il)H$b1W4N$=%*~KnjT7=18XM;Oel|{ zy=>wQjmA4$kar$5WioYf0Qo<&E6UGL0oY-I%{(ovN4Zj`t;&a-=PxReG&qBX49jAG zpTqfIN#zBQ!)iEW$!6nQrD>*BbK>ZjnMk#gL3&rlIN(vMps$O9tXCyTYml(G|M61OcNAbKl(2=_B_5|MiOQ4!}&>q1h?^}H{`$i zdIY&%U1OO5)9CI$n&1$KtOtQ9DP6X*yxeZYiRB8dGK3x#IIM9%@4-)j5)e3Knv7cT z5U_LdQdKgK0L(;7fsH|8QZR<%1r=6m?FkIP4K>iwMCM5VkW(dV9K?KuVyRhqQt_Vh zb?<$hzlTWwi9uG*$;XRFWZRKH9=x9_Q2UV5)1#&wO59Nc2t^(^uAUA&5CSkB93bhf zoKJxdWcHi6n44Za;NpPBXu$c13WlC_s5-%B_S5YNXy+1v_eu~5+djgCfsRG+M&*VS zK?hv>Iu^SofRU7(^CmR`di>a~Ew=%nKEDk4H%+t}!Rro_~_vPWIpKw&|>E{?GEfeyePk!wtkB`9h9f#FX9Fka^%BN=QN zV8%WUSD51d^2Ziu#1FkD=+4*0O86n$LCkf?CA%1T?UuC9?E=1n^|Li^8H`G%=~>U6 z!^ePHFrMWhx3S#+*YTCGwY@-Yw7DrU0N0b@nYrG2q*x9kzbpV+KuWMB9^b z6bCNWTAn0ApCB!`{Pyf1>*2|r21lpOLvA&%k3+p7%Ub$8!Wve%jn~rqUW{Lj*D}5c zvd>bScGeneM{Gp)#6su+2*Y#)cUngouTH@?Ob5qN!DZPwjMl0)hYg@BLl;!(r z$qs|C?E>;7=>;q>ki-kxLm(^Jq7&K!@Dgz`$Dl~8MSj%~<&Izgi~17QGct1O`t*|^ zbXFf6yh6Qx+%cP_ul@xf$jNC++4<5xYBvyXk|WR;vX6Q4F+WU7@%OuHM_r|I0MCL% z=LZy3s0|L@6M6tQ$3T;GolEVtU7=u%tvf-&N!s?J8#O0J?6ENb9{K>oWa`ozsNalc z+T4CVy$a4~HFY2ZC5L}To&unQO3^Bz1;$4AyVQ6>ptxC z^mj@yiJVdSzfO(}kywkq7gYqnN`OAfOW>aofOCvEz>pxuh5y9H#x{Lp1)2xIWl0aq zL?Kclk8dD0Ask$AHgF~zL^_Bo0bbQ`6VQPn1eU(r$B!Rbq9S(#Ovu!7cGW3OUDWXj z39A6aL#|!BGI%clV%C6p5R0PF-mHME;ex0V$mPIF(Hnx7E)@L8*Aotggo}J=yQiNr1Ohi=s(TJ!1M^Kda-jWnGMpbkKcXqE{8BS*kS93|pVT3@s;hGSO`qWfmq z{`NHt>b$QdsKYslo(|Bl5V(W*-B58_f&%ozH5W!Gc@Qy77L+`(nwW!|&hnil>C6~Y z5#AGhPSTo{a#5<;s_z5(;V=B5r@HA^k$ALzEbU>Y9dn{XX>i!DFKlNT~)f zFLa73Q3QnL|GtDv0UZ6n!qOd8L`kwpBrgVDVacajI@qv*xR@L+*YVt?xQVGc&(Dq03 z5FJKL_PE9*PK^RmXdh!R^kBh?D98h)PbZ|1VLc`T=9-}TWsVPQt8~HJRt0nvIEHc2 zU)w7btig4|9XyGlb`ZOnDx~<>!G>{I)w(w2H8)ywO*#QTrYr9lG{KF#U|$e;WMriH zVeJ(GH$p<27DjI=sQfEk`WE12q+%(|=n`ETl=$u?-K}r-MWgN6b7H3q$6eIoN(oW{$R+(td| z71EqEwPPV~A{`dO-9C13AK;PQ0rfAzuo+CdA!H^$aLp~Ke^FRPH@35DsN%fCwO9Bm zsgtq&i!;+L@uP7!*v#woD`y_rX~D!L^8_Zx2mekTdRDqbrP!$c3)rZR)#~9m3)!8L z9FLkF8vf{ptIs#8A@;JyK0kTP{deHI+jpBdp72Uj*odE>Qu~?05F$E zX+H=6pzL{IxDC>INOSEYNA`hv3B@+-=|XElQ$v^y7U1?A#TL2ev!iBD&+t9{J^EwL zTiYd{%8Y>FV9h#Mz7-n0^(va~rKE?pMJH3}icOgFHP}}EZ(X`#+j-(m&*8<6!PKla zo_btjcc&%QR<(lg-(~CGOxLd16=wMa97IkndXlI|<7JX3GIpi-R-2{e`bQ|z^OHR; zhduvTulOTit=DoQrgCn_INY{wbs_9w{W{*Qa2qR_5t5{^#d;YyHUATt<2H?ez6&62 zh@!)E=@L1xd4Q;U4IK$U5Euh81=ny$1}o$&UCH&w=?+re)tztO6muDF4b!ehvOF7tuh>qXer!F8Q28iu42D!0 zSw~w~S&2g-fzp>TAa{A?=i4rmM;$z0Z{Ox-N!WOET!8c_C-q+@f-xqCn2xx9GK3vU z2H%?RQxUftJ2VN}8@tIuSik6+eEDYa%%-is<)C`?V)$Rs9|l$6KpRm1ne}#?Ib-sD zwt`bsBEKP4nvlA@<&f~d5Sm;nX`3hlNApkGoi(RFlti2HM7k3_eo#Y&Jrgb=o^<6W zAB}>q!s++dPM$jw=}IyXRyW(+I5tyKC;OA&=$qvm66REP1p=7*$|^7J9@^}`7yjv9 z@(C4}8Aj(}YX+ZIt3mt^3_q#6`N;RDWr>{`TW+lcgNUAd&hJ)5srQUJuA%vNiCcjL z%oN|4`$mYh(&AkVME?c*34IOu;B${Ecxf^0tlgXVOK-W_8fkXn`l7!+lIy7w44?(L zFKKM?2xe*JqHxAP(>t~3#qj0|f8&jqmif{JS>e8`*T24hD{RRuoEC3AzgVQAu^Yb| z`}YPR(NEtK?;*trM5e0I6a-Z_;}Hz0cTWs2ZcuvceYXDd)<(x#YeVVFrDy-$w`h%q z_mlVVZnf|DvVim1er-YALLwA-m%K@HQW-J0_RIH}zd79c`&1IgA1uw6DO(CFxj0G` z(PF(FC^<8uw3=zzphX0;j3j{(7wTLJ{A)Cx;mQz{0&5F54}Uklu~~9j#a@_k_WsDt z&y`DAAz@BKJi;v0?h!a~#ra=|rEmZJBiym-L7mH0>&AV~=j0aG|AaX$$pxuB)=W(a zkY3jX8yBI^U=a>$AJ8V`Ck08;)WPf$%vM|s%&;+)&jSX0N!k6oUHdN-{{<8Bf2Y2~ zCJm)kiJvx%jUWJxFM(&Jek9uXe1 zh!dQ)eK`smhekXWra!P0WU8rtsV1dV}ADJ*{kI#c4Jw zA&%gGW$7gXd|6>CHXLN`GWN|7$Rh1cEtKeK0>uL8B&4_jhrzA)wPhD}dI;q~I_c z>Bz}{Z`2Of^U6L9;U_Y8RD=UZ+RKzUF#`3ErWC%=zq~?nqi6=pf*p%hMWyxdVR}co z6iQKC*0^06t4H!GIC(I6pj$4G>|W#e#2H{JjcI$GDD8taF9j{5j|zL{;$c!@Bhq(b zWSP#LyYr_6<{d#EAHNSE04+4DK@fCc3B#$Hbrfym>!jwUPS~E-N*sZ#-J-k9hNK!j$P=U8)c>E*?r2{ANFx*6e0|59XVy+<~GMpn${` zLOG-TBT7#_072QVJN4PY5JQ37L1bgy~IpPi+7b%OBc zC+wCe*N4=jGp3932Y1j;$Rc##XP)Y0CZxuJ@7|im>7_oUmUA;0WQt|c{tL8(@p+k@ z`7vm!Z24pFLCK(F7h(1G(399Fu8G6*`a*N%az{KoJ%0cvU=vR6Wsu;FLzV;Jtr+Nq z5g!ck4lQhc3D#pU%&yyst|tjy{amoW&Fz~$v%s1mK2lkS9rpf6R>u%KEhXhK3=|-_ z##aws?d=G*Y*Qj)l3LQ1yc-ICN|^rB%5x~yfUNwv=>qnbxR%w#0GggYj2qUiUjQ{sT_4Ai9^T%F;cTgx(fW1;R6EPDon758!n*1Bcs z?FZb!rvrq!up936sAh@$&*LmFEHbfr-OmS|Gm_?K>F%Bl$6_!Jt80WjEA7_xWb;jJ z5mt`t-aGHEG|d;sysex@(}5GiDkw+` zxh^0JsHg+{2?zM5(3dp28U;>yM)RsnE~*hQ(cv%llKAU{DAE)__2v(`WMy|Z1(fH= zA|IGXsSjkV4t&=+_smZ zP>S}J^g0!{yt{5J72LvA|DMdLPJ>L z$Aj~>{`@(D0SygmJ_3M31AwI+%-RFrB@aL>KLRoF?G?;b*?|7dGCy zM*x^c7IaW>;-eb^f@Z3<(LvOa3Y`;Xv(a$y4yzlp{JDP8eYz{#)c-M~|2lu4j{B6S z?6(0{DRp&(j4VTj8wUu}UxGmvSkx4y3s-IorjOOeSw+tC(eQkfV=EraJ?6IaeO^|K zgA=F{gQ|VRc5k%%Ma6K0iTAx$t3_zG3BqN7ed!4;s`t+q4)Ut_ipWnyDZ-3F)`<(uS6S<{zxs-% zQPVQIG5@%cG&*ijb>hV9?qUeVupGvKkpiDb$n&I&-RO?;Uz)=^8xAg<5wjwB9X}UN8vUPXC*B-M#cR8ftoomB2PYF0hyD3^$k^9eULya{N65 zfct8X^UeI+0Y+(Wztu^_Z_-B!hhb-b8uuv-6p0;WI2#ZfD!|oh6ddLdKT;4Zd~jx5 z_Et^-w+tH*0f7$mSuqf#)4BlOtJx#_HS^+SWKd?1N>Hf<{u%7st zoUyOOaarqjWwN8uvCf7^*S~xdrIx*hxgb`uVmMguA#Okk z;V=br`1gDAW19G`o>zz;9e-u!DMuLf!qt6?ID>D|_v=LdD+ib7y_M~}`*F&dvD4#^ z&1_MqzEA+KpXa?HV4avz5T13|SYHLk2}~kS%Krcosruf^p8u4tW?3!4^UEU4>N=XN z>JXjbd$tC2AcVNsbSIYW=u6kCwFL`_rc@wHl=NWAGAP{XzpN#jVyJKea9w~qCJ&ds$WK@$D-Nc*p4i*tO37Rtf9({F5e zptF@Ju=91}Oz>KcX@QWCw%8yCimAmK(#my)d;tssN&3&$j?EAsZAs;HVzkB`h|d%k z5tArUGO?TlaPS@=xZ^N00kl<~pe^WG2Q(V1txSUeMSeLWg9Z~ih9_^Tyniu2e#nXm zJRWarMLZ9Cq4 zp!@mUKj;@UZPgNaCTPoJ9In)xtB6h{e|EUS-q?+sqDV;y)O7$hk3eNZE$Soy0~?P+ z9}&&41`Y^xeAC~4m3cpij8UN|Yx;2YB#bz_1p*o@%>GuZ3ygkrhnqc^00h{t#m=TB zbb%SsFfxE)k){@23Nt(V?Qah>kAwZluE}Sjc*Gt0f8~kO-xNRp6M7no4nt2P52T3? zDKAtxjk1ShsosD6cC+hc`~?OmhKQ27$dlT!3rd@RKQ~^*aP&L(;DeG1jp~M*vIB`5 zQk(!Y>LbW=JX^BiAe2|H2F>xOwPfg(-UDgjC~D5%y-NgM;O;;?MC>~>)(sL2bPM41 z`4Q5V9zh9||HMvytJ%K4JhcAByYTfVTSxInE>7R$C~3Vm`wR5-xxuQ1E&47>p5j~|K+;$VZWF-OdkF#ZFD=23|9%kKK=lR-2?jcyo~L)7 zZQ;xF5!;F$U$PQ|z}KE=rh3#tXfVS=eD(Y52G<)aFW8>DdR+XqsGMoJ=WO#NXcIK> zJdeJSgLy0iI97)aumH<~APoH8PN4y2kVBxV3m)6~^kSA=3Di7BqFE%ehNF-Mi0tt9 z%Tb>Id@$6O=g{SNN0HcuP;44a@|y@pe6)28_#Fjs2&G^T2ktwD{hm8{XAcl_H!0Dz zfPVwsjxj>#s6!)m)XoiMx6R+Q`RhltNz}2H}sn^TtT`q9SW^Y0r#f2D%QOx z2Uoga=X(c#x%9kMZo1|G*+5}YsW|xJQy#;`dz?bFUi9%&hK@L=lQ!?0bQY2K|+M80Ndq$(xM@GjSuD{zYn-5I8UUK7l_VQL|Xx0@NIswrq znLHx&h2q%--J#^7+fuE(fN>2{+^lrxy(CNxP>+d<7>h#;v}6vRoiF=ENuO&ZLluiC z$xao@PRfTQdr_8HPtCKUyP@JZ+oxbRh=x}1IzJ|Cr8ACTC0ky%5gZ-E-+Xx_ZFrrF z8U0Z*3O9$mfUvr8%cX=p>xVCPUVXgT{1jrl+wl*k?%pPzfBQcQc#zGaIQ`J(VO^DB zzu;gb!vu9F2T{7Dl4rV5xEq_z=*NzxZaTaa^F{eX!UGmR#ECUQG=LRSQLQYNphyI)8Q*f3>#d?gZC+d=3 zUVuRb^$Er%Id*aGxBvH~%ZxYF-}fdIt^*_SFdO}!fb_5-PhNg?|A$BY`~UKC3Ah1f zS~)&cx2K-$kl%Z5F>pTzU#wG`8$)pV42OwYY4FR#wB?{aiq#GIqHwSXp~&LBaWYnG@x?pY_m7;V54Np#(h$9y%N%C$!k!uRCV$%E$v#1V$9l~5I%ON`Ifhp+pJ z*0SK`tL(n{Ytqn4qIw0 z^Vd(2)A5FMknJ_%+^llv{iMp{wmW31BE~0qC*ANbZo2+S9^d3@wCgEbt*W7)cO&Nm zpVC0eG}F92uPK%X<%;@^?R1RGe=Lk@-W~bL?a-e1cLWgwJtH zTEjsmmLq=&b*sYCE%g=Yj^^6^?E*hJtR*uPAgx1yhBh=#>iaF-Q-UBcDUgYcigns?j8 z!P-{_H>fEYEBh~dPk91YYwR!QfCK+`C`XEjHA&gr*FZ^!ftw#7s19-v0Xk6GJJb&g zj*3{wLN@u;>xdW2Lbc-^A<}T;rBng%KlF6+J=sdzKe3}hoydF~%zf6md{tqRqX(SO z<#ly+f0ZLihJ$twudmPjag6ko&+f??KvUzr|cS;!KgxXgFmO z4mrR*H=cNDVujz<**Ed0C2Ee-e+rM8<^~lv1CIqHR%jM4Sokji_Ys-2fsrKi{Pv4KL`KFd1Ai}G7_#L+Xn&4F4Ts3z z$P5P+Ab#AX7!1;2gWR$ffC}qC7nt&Z$jTsL@=8oRi(VoSgGYf2i@|Wh5Z9KLtp_3? z!-x2*$>A>O@(j$qM`5C!7~pQ8@bLk6k#U$p+xE^^@%b=NS9<^Kf+G9}Sdf6}i5{e_ zz=UOl-j*!TH&KEKc0TVU4hs!Ud#M*w5$Eg-tB8rztjPR=&d$siGh?^0bkSXOP_9T= zCgv%$W=mc$+g}9ckq7Yc%4=$Hkfj5#Uy=PC zAhFVjfP#H6$yd8j7LCF39{qEh_(8maEqN7af+qOPPGr($LYKx zQIauvA)&|GXYW4u5-e)9aIh_`wHZJ;W>p6ACx9I4;W)f{RW+V-);yKFV{g_h_2xK1 zK&m|Dv)zrd{hCjALeMl|P%nK1A`eo~v_xMEf^ivWFxT75rnHpYTK=G`LcHJsz8wy{7vR}~QUwdfg}5T_x!HH``=VP%!N)^w zGNC0~d-ql2%62fCyNEb{Fm2%o5EfBO4a{_EffQ=7G7k+Q;dPgODH-iP8gM1 z=urrIpmu=oBG3u!$ zdXD4h+ zbdQL`+X|0G#G{P}pH1YQ$7g+{33{80}&Uno@~mF>;PFHE<4;;Nf)Rm^%0~5yS87*N5=SkRc^{k5P9VQGHA>f6%vAt@eT~UWo2bG?$&^& z{0={82L?GKu=g@t_MHWUMh39wa(lIIf+q zbjV2w;@a}H{grE`fvC8wS5xdUdab15PMgZeQ9%eR@GZxLvjg5tk0UmUO^Bky^yz5`X{$ zPMi=Gh^To!cCP@8f{%GCX#FH2dRC@3t$J%{(Yg0i>Sr%6To_tx#B)Dh9t$HSO}s&B z9GHm{1=KoFH#371)}u#{z^KS%Z5d_cDR&f0jyL9gsCp497B@~C>ALxezWVG0)8&p+ z#*@SzU*Z_79I(97$aOoXIqC1{!CkSa*RJ8E9q0=B`EHPAjyo|94~P4H=umUF6e=WI z@$ww&_bC~j{Pp-Hp3g!c4*ltOTwIz1JJhLKei7DJ26w+Z%Q$B!=ae!`mQ*_wL0n@5 z*TGE?y)MBn*8_SZgluH^gkA*m&6hcUQ~=&J9Weirl0t+5D*N}I3}Q6r_t*xUiI-G) zjLfjAS{8byCDMEI|7?A=;@#U|BqeoiYPt)P1)f!Sz+DQ)e|BJC+}%0N^)od8$sXLR z@b6_y|Jix3#o%XhvhDKu#JEk2v6W6XkjtJ26}#T5bMDs(8A^VB{}zXl0prB`Wf!j<{h^e2-rU?gu5JdTCNR23PC>z{SLU=E)AjV;xP*jXX*`9-=?D!6&4mg9pDA; z<2FGoyKeuRXQ^Dnk1ZrLea_E(GpMUu{av%86XP8)cLY}my@K4iFQH~IqC0-&hl>Oqe;n?OO_yRe2A z@N0N~cu_(}C#HSP+s*9=M&sZ}$kJCP_`IC1S11tj$SFWy*86-^DfiLm*lPLwOs@+w zUEMj~e{qMlUxXXP=I<-K3yThAKWpgnSHR;7W)~yoCZ3xsG*^xw1}n6Sa6HAZMtj1} z6T6tDgGt9Ij6tn{O~(uD<)W7G#tGAz;P%(Z{T%w0mkSpm&4Dr8w}8Fh{~Tc%g7H7+ zvJCR?8GU(aMz`|vGwH7?PkfdvQLx&)XLg003<%UYn+1RVICOP&0iUgExO9zy`~CQ% z#Ye$wc--;IMS6wG(CkDG5#V`<8HF^n9PC4Y7oSp__U>t6am||duRSh0MmbmhW zOTIge9nbDPDJJsc9R9vkIPb7iykvjI_tP2(J4|vMC3JqIdfLbTR+@bRT6N5IC8akF zQI7B$k%Skua$&YfgSPP5vxyC+@B{AKO%o*>!#Cmd8#lc{x&+HmjP&RZm`z;K(?_X2 zgJD%eQ_=mee%lUz#^UOZ{n+&Nv9p&t>UYK4`>yLOnQ7HQIA#eJvs4sOS}5@MSv*61 zVHoeXMde_2T*hoaFZHQ~)-mENL4a#1)62|Vsy(Win9s&E2L<5!+_*O4oIoFMkc3;q{u!GB)JM=hEJ%B-@6-);6h|x48STLiIjzd}tt9B_Wda$>b3Zb>E zLBei`t_>y^VBd$~;NYB_rE? z`QN87bDN)*Vei=`hUh&jY{fM}cS{mOM`4jtT*`nsg0qgDRwZZoa4BI>11OGF_-!Z{ zN45pHs2-ZFjsN+xTJt&cOOw26K3Ghm5mG<|Lv$BlGiMs0p58&`rX5r(H)GF_(Hc1h&pty2=mbjK-?&s$2RGQrrjr(@&g_p&8 z?w_HZ{c`&<_n9*Z4fDvC9`k!ey;LwGCx_gjgb_yi;Nj!*8r|(4cv9@Jqqq}l-1l-t zM$f%&Z2#8oDUG>)!LR&QU)95Wi5*9!=HdC&foK7>4gd|$+qMJK`J}Wogt>)>Z>`fM zBuGh1W4Wf|TMNLxdSQPn;VDz@QJ29;=M5j*<%T&6aonQ!Oe|RvJytS=L_|k1aq8l- zlU=e$E%SSYFJ3$eCl>U8!653Q5fNNb%BGRT7ePx{!P?z@ijZfqr796(0zBE|w;^Bf zf+<7Hu+)ZT$r#?bcZoj9rNwM0B?i_AvnKQO-X?exYzTpRapFjE?goL`_ms=j01ikV z^VHt@1pCG6@=cE}tvA8Oww(j5@d=g3>GkAKJtQd*}iqAS&V?k$}t)vnXFsh(yI{xu^i1qr`n_fZJvs6$X7J`n2IGox07Lg5(Vu)m$JR7#okltM2?=bhw5^M z(I+?9hQB|Yp@W61dO;ai*~8H*LWl@qp|J+tv#Wi! zg&6e~{U3d~W5?|SadD4k+PQn*7CSu4Sol(ReygPnxljzY7~JH^l9ssgkgC|uMId3F zP`gfOW>p1@N(?r@S8fd4Z!Xy$ARuR+jT>~Q4_$sLD?j9vPwa9<(tm!!%RKn$!d4+= zAW_j|-3cOk|Vz=JnBmT?LEtL&)Y|Skp4X$s8o!?&LBtOZr6Z5K3R_TjTp2%ZqUDjPPim~>n zj6-p+xA$qq^iI7hpSwO73a%%~z{&ZO=I zH-x@kE0P{18)2Iqi+pZWf8h&1b$&?WQs0s+XJj8?XV7@w7l>?!2USS)mh?x^R+SiUTK=iI_eLAjGmpLQN#D{&`OHi;8QpYH?#i*sFy)xP z=m5FX^n3zO~!!$qsjr}j(`Zy{=N7h^81h>d1b-LFnnfe6)s zIblzYmp)}SgVckABn0+@BZQ4G-|&ktS?cfVZv~>aOWy}P32<$@?vhj-`Qw-VNYp7$ zD5S<5j=nqD%YsfPqDPd<6GpUlot#;JMGI@(ZP9h2dwfxW@Via zj64qP_NCvy$L@rmX7y!(=or&E^p&=!#?v$pK4)q;{-?6`$~32wBNtWJ1*<{4b*tJP z9S%t6#i(r0hKQ6!gorq7ox{RlVJ(~y;v$nXtX{+8qe?q)AK_P2Pz)j=(X6U{@UCr! zxF{6IQE+_gPRa68`KaU9P0yrkJuKXJp&&;mAS7hQJ3>G}AgiH4ywN+kS6LRj-$ z-C8#dW%RZEddLDg-CqlZt$`9WO!j}ghv<>}Y6Nj+1VeIP z*-Ou#GJc+kaM`DwNv$r{;SlB%IJ#$+FpJs(^NMc#nS!N`+8seLBq z#0RpO)Z$s2G_?}fRKu+0;fgN>s;cHZ3y;s%ElS!}Mc1vldIhUjofsm%CikiVM#zFX zj}ZoSi(6Y;k9%2WX~2nu!NkiO{klCM86%5uLg)x<@Bq;vw&?TcUY*G;VYJp^uQs=2 zE7$WBnqz*E5Z5Fziq65c>Bkf>>O2>$`ul|H2Yh0^f6nol2)y$-#r5-M+(~DKJMQk0 z^qsgzbt=4NVF^Z96~NkcP5s@?WH&*N z$Vdicxic~GXhAlKqU#AihKY%(yy-RCxYnF<ECG3-F`f=Q+la( zf&4=w$HhMdK4g1#Mp8#Bu?tG*<?W=sHSfvlk@FlI&7w;1x zuXb@a)NW985PT+DSX`KOqgo`X3w6dSrhb_e`A>k1l!B9C`Ns{a-{WtAL-F#C7Hqnz4-DfL;j+kmN2v_4azH~n^!e+h``sNLan^;X3gO9BE zqqF(J){cck5o7u*te_yg%HkJ$vV!cgPT9LPO!|&?G8X}m&?l$5YeIPc)oGkxG&%sRsHAGT0=Bj&4g|v5YH7>$eXipgFU?UU_qOmxrkh2LNho3N~5g+thO++G($$ z>^V3c#Pj05Y6W2%&2Q7Qhd1(E`J=RajO-mHN-^Q9{0@jdn7|zUSW_+{?*_5C6G)_^nThYN-D?h=^pbHGA@xc+M|s%ed|h)wf-JE^ zI%CU<(&nMU%{Z#}k#saufs$TcV=+@q1w#GO)NF?zK=#_&!tzpc-dV2WJMZ5mu`XTl zJGgyL8K|#d4`fVOT{a`cFC&d*A4BX(t_OO|Y9eaG% z>)T+z)eSN^)_<-=0?zBRX5@Mh<>yLLUgml3s%+yrY7g^JjK#W*)|&m(ITye1Ij#2~ zeD>e(9aJpF!jK#vbs&*DR@ zLH0LAAX7b?s#X5xV!z#Dxgxn@cf`-Pgpx1c{S*+__x_vcoW6R4)N;;&3_a%3Mx3@> zi)l$Xbpkx=PkwQRmtUd}7Sic= zmA--|+qr|cZ0?w5uY9MUUs@{HZlX&N8)4!i_?t+O<@O}g)!6B6P@Falr(p;Rm~%5c zvs%9I>_S8J6b7Aq|C{K54U7*9TddGO+e$E0i_~&;<{pZ&*ylXk^4m=Hw3E)FD|N>{ z5@R)5M}c-de8u)iDJqXOizr)6K^R+p5ff$i=*el=#>X%wYJ5iya6#%b;maMNxUUe#4K;cjAxb&C`;Tct*QD31GZhGg=kkp!n3#Hq0|U1~eU zVQyG(wcmCX)bjcNETvp^xCyw^+FqajdhT>%n<%er5d&g^3dlWHXU5LxcpOFuaq-BT zkxD^pXZe=Wt9s?MAlxRHpg?7cIy%E-(v$qbQA3)w4VX0Iegp(0Xb zmQ6NAM1<@SvLZ7XzsK{X&ilMS-|zSHd;QTl=XTEFdR^D^daV0+By|g6K`DUJHs;e~ z;H}VDI^(jkRC95Pt~{XGjIF`Cp`pM{QERJA>JghFyRFcl7fI(aw7_HYjhBbT3Hjz8 zr_N-)Y2gS~685~#bx(}dU7EQv8zngHE?oRpl`%=I_8|xIP6ZVELY-0znRDY`lr!v@N{F%g&aK6G!939{}!eO)q_Y8 z(E`=Kj*J-;b0?rAgknxrDBHQ{;pEEiit+c(vSUj&jxe@AtB$@cYz!3uKWTh5wU*}J z4`9DL+G+xr^Q9W!(;%n+cyL~f(itRj{1!{%gYV5vSKi#JJZ;rMMm0^Mukh{lePKm( zVem=sI8Ujvyyu#QPyl>B8WSERL9cD-tmx^WHOr~JDM_GDhDIqz2{`*X zZ7dksT)BdcF?Px|DgT+*0+n&2^pC1wFfxQAgx0@-J1!DrzzrtE#}6}TNdPn$&6Pp? z4VAR)(s~Q#4)Pnk=!KPKztCa6CaV5871TM

VpF)x5qD{I6Fn`Jw}guMa~PXIxk# zLnmc-OqRRv6KE`bhB`)L8!Lj-KtDMyIu|#)_x`h7nWJn9O5=0X*Ov+eaEK%|#y}kg zAoW*Sc{vy-0ZF#C6+~*Fl<(>5;}8)c2bZS)GG`9h&Osm~lwZl5Q&K_=62TyYoF>3> z{5nO@`yTHOiNp%>wY}B{1(j5&rxchqlcY~kx`u|berhoKPcpDqGKx9O!e!F(u;t{_ zt|Nyh_Z(s`$-R`gN=h=WBG&UgA}An=cM$ydUi4@1-pP7)>ATbzYh=>ICp_1Tv@`py z19#uw+}woe!|@5&HVRf6Vtb`v&5_+ZS1G6Y;@ z(Y*3z^x`6ZNQ$A`-w>B<41r!K?%G&JLtnI+#pLw$F!`pnTIiBOx&zJobY~1rGn*q9 zPsXzvj8sguZTvTDW95RQt8emq~F6L2jy#3_rmg7;9muf9>2ty{)SSn#@GFC0HQ5d2m z9AI6dOx5;W&RH!jw2CK&+G%#(TV{JU1Ht>qp~(_tGngg%BuKBcoI>$B0> z4nat-sV1TQt%6DNGL+Wd+ln3kc_oJb?n zqAWUO1y;Y?4QL!sl?n7&H*F+)`PX%dm!=XDNw#NaWgP~#04^5tpPpp|;s~!^))-n5 z1@8MtoADHC;hwL+;&BVWCBc04Tv)RP3I)^7yrqmN>;0aRx!5Z)dXuB)a>T8Mi(WQP z1Utk1|EX*_Qrq~N7%KC}+`J>YRz#|OSAO0nSR>lFD=EspTA5U|Sb9_ME002MW9EIY zwI9-hjXFl+)J{n{U=@5w0SgPE(*w7Ntzj_B#l^L}vEjhz1Jt2H*h_^Dft;Ki%14i6 zJOIy`1&#;4vKG2Lrq|GoGQGbAX^03)Ls-+Ef7jCG306A`b)=50FXsBBObiUD_qO}q zvBOIZbbgp504pa+)<-)ZqtIy zEzt61^-0P6%HvCxf{e{&7<1*XqodRx9edSL95FaW2p%U$MpxqtiFo`y#0N(%$b~&R~m!|4}SXPR=AwY9oGs z=N{Wh{W~_pFm(FJgfoEkr(c{6m&wWNvv#ySQd8+gaYMombA#Ek!nm22euOEotvJgmsKe~4rj>Hbu6xv&pd}Z9==p%3BIY2o*BJKdAvN^Wdaz&3xo<7J zVA$I1Vo((-ThaY${r!%am%TC%<02&*OBPK{r3Pj(H8Eg4jemdlu|M^=1L-3kzP1*e z1E1rQ`&(5R!_g)Kk?mzm`MtjGe}MSdM~=X&h@8B!XxIKV zV*1?A%Rx-y;iADpb_Xgmm~h&cv_j0`ypo7|j)jV(FsYlX!{SgvPoVnzG!Jc>dKk$T zv^&`+vtPcf1KTB32p~@{(5AJ1_<(~|wei*N6726XjngqQia@K~Vf7xwMSvR40GNjy zU=Z_9mO8=#J!q!1gg-02z?2=9e5$wez@CY-2`wmPtNFeaTEdhd5 zoq=5fM`~Z^JgchCd?_3m5h3FJ&iM6%sbA-ehgKhAysJvMndC=>_X15b zD9E1N4~9-|sBaiIU+eo}ny}H0( z$fdm^@>uo-y=+Z`$eB|xylD4MmAg$38?yj+K(5KK4hrafXl@t*m^bXNLs1IW%c6Z1 zuvWZxL4?_F%MxCir&U#qU^v(UB()?x^cLB?)&0p|V7p1hc&_q>n=iD2X35lnb(ST) zGxH-V>h=bs3RDlk$?3q60BVby{wT_y>#muSt>w_Ec%E=SCZ_tDYQv+{}K{mJC5Q@+B*G2I$N_O35GW2Q)4^$bgzzCN1D$~PzfRjK?mF!cpah%o5!!55rWSeOj?k=xq~ z98#zSzcplZl_Kp)f&}$|7e12J1m>4@X$=e@9 zj);jczC|8plJ0Yqp`(eb#neZgf>Js=`jPK z@cnipO0#EqvlWK?1)r&AIC{h;?7qHfI8930R7+iA%4S>Zia^b)x51n`7z*U$o8Wqx zt{(@EnM~kDLI4|9<65A8#+b3ghQ<)Ud)K3*(0#(Uk|ER56cXI3)1iLx#Tw+V^ewT&B4Dnln7@BB-i*v4b;{1xR{P?`9F?V?Do1oGzr$= ztCA?(@qM?YouJJ}8t4tIZrgS>8v1uJza~9{I{O*X5Drm$gleLMm;-2 z1D zB_{z$k>XDCKTG(}7{5~D;m@4q@y_zI9by7QSOMggOJ25a!rs66u`l5D%n?R#yb)+W zSz&5=le|W{_}JqdJ#CJM5+1$R$6nGDw7T*vv24pb6?iI)e@v&2o0&Oz_&j%X-K1E^ z*-n{2shg*LWrD|kNMosI%40o!x;#?yc1hq5py3uqe{+7*+Wy?ztlKKAD4sqm`dRwm zt@@Q!Lz)pekKfi36X`-5Q;aDYv^X1=1-Q);Dn#d~7k~+a;$IhUFT7LfuP}CIWa* z=YaENLQX+u@z{rWfUu_KFI2I~$Mp`Uvis+fTp^Vr?k+i5tqPW{eA;Jp=S1;!C@(Y4 zc;C4>b`X{io||zSA^%*#28_036NsxVw){>)t3tCx+7i5r%l5D|k0a%TRocLD=#uug z7`)_f5{Xh_QC{#@n@SI-iw)dbz2o^QvkfajhTHY^A8;W4V@~#GD>WD98qN0RkQA2$ z#?M`vB#EWP=VC#JW1|koDaJqV<;xo$Ba7Ch@}Zc!bPOoMD+5{*%Re%{XNz^4_6{~- zcWTZ?0H9D;r|B?7pj@)3@zcL>dgyl|Zj;VH0Iv{K{PAx#PaWX@IS2`#2rXM}Xzc(cpCIb1W&uR@nG-R0qUR z;}bVoIU~BebzQ^GhzHzK)wRs)8Xn5+9&0g=EL;Va|NUMGX)QtAXtx3Tw>PRye~m~i`kl)OxJ5r)7HB6}b!l9ysYyCE9-|*tw7eQq;`q|V+>>~x7P4fVt{3bodGyTZ3;?VMb!XjEOtUx}hw zhC5&Hx)ihtarvD#93sWv_?7@zNs?9$!OsYarZxerAaIb9u zN+ds*g&~>Mtdy`4U7TjQ?xds*5eKz}I9FMCc^3Od)HJiOiU(tD6luvyhrVw?Vz1C~Cmxp+g$5po_ zYQmmx6gJ7s=dMW+`)z#}^GzzGunk%GGSXFANTJfM=g~k>XgB_CSn20hlHCR8E6L~` zNARuWL(@?Bn$`g7U$k5M*s<>6T-zR4heoL*rbA2K_J(TGzBShgI%e)8s?S2zUHll$ z>x`cN#ay8|0u*6gOoTSgwHi!f$T>=2_%AB6d0 zMg)t+;x{2DOjS48yg2CTe^mW*1E`8MSp#3$wUxAKE3s?VDx;YwjXFs z@E9ZFO~)*_9Y3#vGtvJ-ff)e`O!!$Bm^LL($nO!OaZA9>454G^EV8HZ9|-l-dKaN= zBx_;A>vkxm(09wci+jTIy>-TeKR``xq#4}hB zD#wd&w_zSqk=|i3q`=lPp=0UAm&ILWa>-u9?L~WghS&bAd7+m1VcHw&6uhm4x%&yU zVpF_uSzjPJ{V&1P-K_lEdjG9*f=WN$=8t@UhXQ}Kacxb}M4sg+p>wdwaa`NBXL%NVs{1N2axJkzNGQ1R_1=Nsg0hskdZH!08$ucd_qN)Igt?DMsEHX zFE#!|TJU;p_Y$e~Tv8vaVS#e(y_H@}{~x9Qg5s{zYf5P$s!4mzD^adKzfW`IJrOz{ z=F>17;pqAgj#)R*u8Sb8lXT5dZ7u5?4bW$O;to&Mj##>AsvqEv|6sDMnE#pdB};cFANl9Cn4yy~_@dG-khxxUSB_X;Tifw0mkPO5*B{ncT?Zx5{JlkI(< zq(l*N{7w;f@+@j-zzr6;7EZVuB+D7SO#KgmA)d=ZO?$AO0x8)?n(KOk? z+V9xK9w5Yu%zDS+qSNRWBG1h9^WE}Bgs|4#9>G75XfgQ#3jprOLV0eeD)%XAiBAnj z;*OHkE!6Sd*>=j;LpW17lfuovga|aTxMY`yN{{dH3|pO@-vJjOZoX~8xV+G4h9*rg z*MWRmTKq+3zP*FfKhU%DcEEWTCX1;@B}YU(Wh6HCpZSzVbmRAtw7<>zQzG7L?5 z`#qM+x}xsm`w`Tbmh{-kTP?2ahdpVstv!2F@6GkN8T=D1wb(?Nq_G;nS3#kG^GJ+l zZQy+z^af`bIw}fxvllA}`5U$~y{eB6tXp<=zb|+e^A5ltqDMfNv#N-6Iksy!%)r$9 z1*2&XD!-&q zciK!2x|XZv7hNBlcv=hGhIYDF^=A^VS+}|U{e?w>^5(#Al;thEf7#N_*LWYRO87M6 z@U970Y;^0-#D`ZVM)4n5>h>^LjIfT&l`v2xHAoh)T)jK9(>`rVk8!W_CZ>*SZk@&RGQ&9}CHmi8j`EEgFUyr! z84e~~?OytD zQOD;B0PLx^Uv3GW23DgNjYO4<7~^Ea?tpvZEQxpOVQIf@^Bw+4d`cdhfU565!c%#y z(u$H_iZ#UJM!A>6z!(x+ZGCrU}3Rte8O_1 ze>tgq?xtnLtuIE9`JUewTK>_V?|gv`NA=737p*ddPGkJbn=E&{FU)M)H~za4BksNX z-~1#P1xh`DEWoyYyyW=R$xn%jvv>`pv&0oJ$oB~+gx6AmX0zihH^%`j3r4}1fRO!m z?hQ^!3V&^=ve+xJP}57pj8vCSox&>4*d4dwmQcE*{9th|CUr||u0@i+4+r`H@*s-6 zPoBILA({{5Bt`iT`T(7rOd9>kd8h%Od{nT!0^`?|g>Ok;gvbyLxazUQWouya=;SQJ z{b~FcpK7`tvHeSmPhTpQ|2X^gE)WObG4xVtHLV4_=&n0Te(&XS$`=t)QUChi^3{>M zigXswOL%p)Eq(=Xi?}R5;fnic0`<I%`2?FWquv2Kv#aHh!admvu zaL{^f@$faP4@vuhCUyDCH|fl8qjMq38p#eCHfuX8agXCO&x*|Y7EH7+*Erz>7=C? zBja#8C702$Kesuf+m^US_F)vrS)El(2Nqk;Ivkjz4}TcF zwhKx0a@FG7*w?jili_Q`<r8~n;nv604w6UMslC_g zepvU_ew6Lk(7DG2O&0u|_!tW4f6V#oL6x!a&IBvQ3}B3c7bmZh9;pMm3fV;F{r_+k z?bN}ilLFy9Ylt&;|{w;GTsmuOHPT9DoF_u8N9Yc;5S>ZHUL znKVurEg$EPzh}Cg%zd`#`ayyz(m9=9AwNBNz5)1l!yL;zOG# zrk1>xYTjk|lpg8ES2G9(BH~32jp(l5y|!=P1m5LTKOghsvYL%Lrlxs)1$tGEzP5r@ z7xE4p{rXlF${6o$DQgHo8WGE(VZ}^XvV=r2rR{IT{<{V$y16bTTO*Ym0<3108!=xj zOuj#@y(a=9Q+v-csh>G2>C%~I!y#fV`?1#9hogJMi76LXcE)DNhy4TB>f(FHjUK>z z+IrZe+9){nVN~Iax(MGK6!NalcdcE@BmhccuQz`$j>ieB5~$oQj8gJK9}P#bB@D+Xic@-wW9St~c)xi@~+Cu9VX&3)qp*~NqW zB2k7>&b&s6_2f^jHnjk;`FWH3FEfO#;F<^3cXYw&e0y*@b-39h*)=mCj?JGO*jmx8 z>DpCifV?^pU&oeoQaL2AMi2wCDcdj#@f2S0y?yTQP^&x5q!OsuKpq*Ju~SiPQFGgD zj(iyp^5KzHSM8Zv*GJDPzUtb`EX(ZaxgEwdl!gk8lXAf;mlo(H`Kv!tX0BW2i8Wy| zM7rkFr|Bd>F{*<;*oLz2lys-2V8a~(e^d|vjlMvScHlljDli|Z#M}w`s%jxB>5dsX zCej}hp;UStWE4v+C)^JZfyvgAumh8%FNOdIr^@@EdQ1KPRH6?^NWW9fXrPgQF`2+2p{<{P zl6KRnq~-IZk40B&o>P&bo6W4U!S|hSuP?b?73=sghs|*zTCI+*O>K{ueB2q{)l~XF zi1iuwYLZI9=}GwdXWIpCIxp2~YoRY6tfS!MJ~fDuS)ea(#~I0fuehyX+~&An*4hVz24$jDb8 zJP8K_hW7tbdM(W^N91;VtT$hN|Lh%WjUa*Pe6IJ`fz(6w>-$aP_M<(0|iH; zq={GFNR5FiM_Wt%;?*fp3wvN`!HsHc6?7^%+1-XA)}TwuRR<1SoS)w@#9$TMKcmhT zy1&Y5`>1^D+COqGvo$5R7jt}KdzJolUmlt&zBcj5`TAYlEGV2Cx7niOPH%yyo0#5x zoAUxC0ZNoAlr%ITh(7%6*|TX9BsR`g`i@5~pE=wJOijtVx>&Ex>^<^64AQwC&RK4~ z3xb;E$RqH~Tqo&a;6hHG9LW<2vlx?{C<%I6{*iFqTN8=Ecjy{!xj0VlPXBe=)?Ru? zRPG~6V;=Li<%5z$kj6HeAdS@nj0l0^L{S+|?DOZGJCiK?;8%jLZE8xKtZ_J1vrp2@ z*cj+M2HflaBI>Gct#U@A1}=Md z?v3|C`SWH-t&+fsZz{hXJS5_oO-QPO*PwJmNaqe*yPJeb!z?Y(#8iJ6=Ja+SR=me^ z;)IwdfekK%Um(~gch7`<9$eq5!4}+d^N?Ji6b0NpiHEO~32d_JYTxAUjFCn%z zZEQ;|)jNh|oS(~C^hGlc*Pc!9ZajNmhz0E&YRGu}GrGR{^S4MMNOC!z6UKFS&PXW4K+6UUJVl?){KXfRGjs70tAGUw3OvXka}~5WJWnf-EO{q7Ifl<=3xfMacwR{N$jk%r;(%v zl{xPRq*f;+r*n#Sz;ywBG&V621k1n-J{s~R@(2n27|Vu38E!0;pG%ENe8mH;f;WJU z2h=pg41+$guzKgIu}cUH%01*Jj2DU*Bcg(ve0SA2UqXuYb+jZ&fF!`edF*MupP2nAe?$b&1eu}yC3-5UG!)`O+x z04ITPQ^XA2;3#)hr2myg)n|%_hn(wDs(Gwx&XWp6m=bB_J3up-?JkfNfZTqNwi_H{ z$(Wf(CWfkgHWk700qjbTsUX@=&qZSeE;N+{4!Dl)>l+?vQNcJsl^4F2O^BModBfcA%ahUCajuU<*edTO_l&3=z=pLitB zA`Rnu9X94Hf`nG9%xj!uyBYos*v}Y+yBEshp4R@v_HPca(^6+)j6ZTo13L^%=g}6u zFlKLp{hb;46<~KOkR8H0qpeLA85#MsZyuQUu!j**4nZl`)UE>^ht4(qOGaUf=gPOF zM1TTgfB~19!+u+LXj2_m-fA#odx+_WMDg=15f@eGGU&I4|L9FrLhUZN&hUmKL|EnG#J0*{e14x{H&zkIB3+uHhMUQjKF@6B7Kq9anJ1MLSrQm=Ioam~Z~gS^ivw7z5J8F(7(f zdAzaH^Bs@#`A8!)b0E(ksQ$E5lVN)Mev+(mh}z+>r%{y&y!G7hR`J)1VdT!Xuk|D% zSDKgjKRUc;U>E>C(-E~Ff(|j8H~%V7?$=i`%l|Q`;V)gu)XaXR9<#sXPr~p)FvCC) zcZp|Y`Ixc{oiMfQ@!ZgA4x*X&+eSmlRsQ>Xl3vTK;6vcN&@TdO*cI~oI9OS+fbpUV zd!s)xp*`qEWIB{Edjd1rlPe7&zSrIA^Eqxf6(k@kp#Uv;Y*XgxQb6xiWc?uBmp}JSLyC`i@qwnl|6VVOu>{hHDCq?431RrVGYp z6?l($)!9E6L(y>W}!qr7Xh7aII36`m#uw8w&(6Qsw(RPIu_ z(izPqQ;OHuLK7L})TIUJ=f)nxayk`lJoUXlz%Xj zo@Zu=@3Be0o_i}{4Q%1DPL)K`!_I2N!~g%hy?ChV zBv@Xx^=F;{4>0Lx%-(bl){UZ_8y+Za%+CnV$>e@ZIlexl8}+@`87%!#X^Mz`um&7i z4IvQ-5~5^jJ&6U_%nA~ZHqWJG(4^bUi>I{PZ-Tjri^kF*9!kHj!h|S)PH!@BK|HBo zI>>nTk3m#=wmL#$)1;2dJ+tdHsS5sh$op~aTdP)V>C%abS(C-MZ3UUzQg1{DUVmvV zUFnPRI_}0d(Y%XG_GerU^xuP{mSR$tiLu`&94Ul+tqS~U{*0*|h>v$Lgi z)GrFwB-Dbh91!v`3yc8+NA-l)>PLx>OnfNu3FJJ_G6h4Y*BcKK-K8in_Npb(2idTM ztvZ`t>!wUp*uH!F^%d1oR!@5`Us`b)xhkbdiSXQHO~bfFe6!l%$1P*k-NlES7&g+w zn?5jat&FF7`EL6gRVj#nWS4fL)iW>%eSNuf0#bB6m?B{fz4GXqU%wig>k6~^8!o?r z2Z$B0v*UYGS&8X2;NTdJ%=Fpplh$eRyw}}u#JQ=!V*b{m+!yZR26Cna+o0@L>VKVE z99&%NOMViE^DL7{q@COxzPL0o8A)Jn@0ar7+}su){Le_7>J~KabB|2u8pbh$lpv@o zxODak99@m%Yw4`4<2{vLLRVz-_>cPg+oXr%FYbaMzm$x!u_L6U*|_6|uD{^hgefD9 z+RvXUz_L>kY}&L94a0!;{zw|BDeeWBk;UKFi@5!g>ui}p=m+o5zALbQD?X+bVnsH^ z7^drtUx!+0y!eU3*r=N0Q&sCS=5L_Q>poJw+#jwj`IWaF(Z0^tyg6cbY*gW%1}^O{ zDmr4W2XWghcQ*aBx5vT-lJqH^(7VDQZeNRlTCXc)Sg#p62{yK8?-xTqu%FKl zNZ%O8kb`vW4oJ>`EryxWtU4DLjXZAw6rQ8vAtB(v$7KZvnExr}+fN-7^ESh$q_C0d zVK>-N$Kjcvv|&Xfkh#3CL^tWSwZV8 z6hKr3{W!7kYLB4@ZfJfRHJ!L+>sMmIkrKV$urmhHkgvvKs%Kunob6n~ zx0o5p8YjSij$M@QAZQxN1I#sVB?mB9WV=;^J1+oTFn+&8JxF9al#01!l}284EKJ_a zqhG!$3Z2XyBIv^HT=zYcp!SZiyx!_v`f*X|sO?1_e*OlyXEDIqn*KY2XdFA8xL7kXN^Rs~CG^t@v;)xD0+~5GE6G-6?p{R$MdY%hjpw{~w=jy3(}`1E;HAJh}i_5UwAEU&&6UuO%t8otVC`j{-|I^?-<{F2_1S1R7ZIT$$+V)Dri(a!2G(VcQ!0W*PNswkQ6am(rAO`l?DH;?d(X zil>THdq0m-328-~v41Y#7oV+{lc$T+L1)SQ7o&aEYV2%87d}y2sZGd}xo*hjJ}PJG z=&0%6T^(JiTVHI||AQoaqYRSQGQl-}cQ(TBSb9eTJeY}zi30Ga(|97s1k3ymW>6YW zHFH{?@q^YYB~0mM2qXmRnav|^El8>)%tug{)&YkJWWiZn90m4my4E>rzZr<1k zdjT6a48b9%`rWpy`2#B7PFD_+e7-%YUKgeRxm+)!?-fWH^z9 zgLQQ9%Ta2kLf=-R@HvUjQ`pA_grj&dYVUb20``upiHLlN%Ge3p3sI|;WztqqoI?rw z%-`uZA^-ooGIeq-dN*VJ7sH8P@BWSl6f%pCdpX*MJ#^91Kq;2tI|^XnorgryIq{2d zx-t5!XD?uQa@#Mgu;GEi@oRnb?0Oa$plX;xhjQh-AD%}_X9MZXYW!t`!3kfmPt%xZ z2`|fLJvu*495S>vEIgF5(AB1O1d9Ny8-A_)1+iOyLY)Y-a4)+~Dw7kr7@{v-SDlxi zi{l+HUPSc0byIf#MkVtPPJ%7FS=?}jO;VnPOFO`@yvjSK&ExR)NhIvmfX&s9hx({N zx-Mnfc5?uJdg~ixO$BiiU6&XKOMv5yeSYn8In!lrH!^ttgM*oJkm*uu#X+UtT9NWm zpy?Gj110F#y7`(o%A;E2c97Q!=|I-2555aslq0WfD%65BLeDikDE11@#!bqENo&*b z{dcRKsGl|hIeEb<{KB-(hF>W^sb{|O+tBt6k~(+Q!8s1=4kUhu&X%_`v6AtBj|#>1 zn&{(sMLH3euu&R5B*1$U3uRw=9t(vcEaXE+kZbw4P$5yyyu9Yz#bdvpklW;23M z9M;o|e8fxW%?GckjJl=swhL!F8s4v2y;+kD0a5hqbZ}UDuyCe1mP}%9)p0}`2|Orm z9=!e+TZO^s9pQ>o;}cbUOGP1}*1=zyVmjBfb9jrhC&_g5;1M7TyBy5#xzT&w4T5Mm z)BAGCdma?V)=#`0DX1m_p^Zyorn)1zq`G%$Tu$nPsjfr=p`7a zq*6a&Di7S>Nr^RnJz_D`0xa34je78uO?9|zK285t&FB`bQ}u4kk)J<4m~bXoFpD*% zO~a2ChoRImP`nl-pM>qcqom!f^{Hg`6ZGQ2ytlJ+^Xxi**9t2upQ+Z2i{UO_u6vfl z^SX;ZnxBbsYcgrc%L;=L+6kvie_x|-RUucDmg)xLcn~iBV_`ma+gkr}c#QUU!A?S| z`Ve4A6D?7;_xtGm`m0R=pJk3FREE^DeW*gh>e(P_Y6++-FbIYnpzI2$Fa`^=F-c zCn?!Ou0y#>|rB0XvObxbw;}QZ#lWG4gRD84WU2H zwLwCCyiC&3tmlaa*wD~fKfoagQ3RD35c$A9Ie`%dRN2K92A4u0_dwF8p=%O@IPI)G zw&#b9WDT=vH3f!ldAkbeo&Ne~-Z$+E+EB-(@}LaEsOVVIAJZ`awh0UOoJV@eCs{Se zE|b=TcM|%kCY!+6FL0#VA>$Z_tJ1qxqb$ON@Ffel2p!6gD{oSD_4TPn{C?m3qke*o z^yDX`t}Xq~EBj?2z~Ab>D~X|J!EFSD}=^55-ALCGy*(@F+)N!t*( zNyH~p8iHsY^8z*{M|V@te7I5zHx1pp-CZ1T8xT4;m0v;*UcheNyjLw@Vfo`Z3CNF& z=tl5=P(!@Z-&~?jGAq`()myx}Q{J_$?%oMgX2{pFb_M6J(84cAPFz+6pndbR(secB`O!@x?6&=eKQ@{Wujd>I|O4Z}a<|TFx zzjM1Jc3OKB+}cF6H)2IDxizjTnKuU1xIP~GF+m@&NwX7uo_D3+Wsd>^>>1a~h74*7 zjg&1|?J?fRcG<5Lt{#p*)UI2rkTp3w8$LE>;Ti_m&#_mZ=i={rQ%HBtfYlWCoV#Mw zDwwU3(Joi@nj=G@a3i-D&o4Zbi$OZUHD`Q2r@(o>eUtX&ig zubzvG=0q?D|8hoN|E!mmLhLulaw1Ak>`Jy2k3Y=tA03z-^%OUI+1`?QKN#Es*jnH5 zLojIQJyF2ne~7&T7lVuD27u9bV~iDwvT5gSFVdGE8}|Uu6d6jY=(1VXwWW#U|B5}H z+>z`XW`e+l$scR}XdCRDeALTEo8W=gTnww02Qznt-i`O}z8+4eOD>KEXd-SqO?CLS zdgj_SXaBJJj%U3s?(Rg&eV;C2ojHS5s?LUdBa+TsxIm1&b)oA*3VyV_64cFDW8i4s z0RF^3yrVySP_J6;kOqC3nzy$UB_W>o=28rD*TyhzukXHx6@gNqpoqV(ABtWonEg@e z)0u@hNrPRLMjR%x!NBV--K|N)yuy5JG{J}gHR)hlby7-Bq@D$dFTJpY)Y&WaE?f`+ zk5I<_6Pk*N2Y{tRP&k)ggNXmcpp(mnsNQ`r)dLx3AQ<+ear)-Y7Fbp{Ha4HlrSfmMnq$UDrA*BoT zKdvkB-^z`(Vf68Q;*5G=*UI%oG<&QKUy{QR<=nnX`956cllw3d^_Y;(sYR+Fm}f$y z&&*54gIGqU@)7r0pc2B}4F^Nk!^HGGTcba}wLLS)`LQjTsu>2mSvcTQVQbK(cYd?QnYn-yQgXZa(y#bh*Wy8R2CZxw6!ifbw%GOIm z<)D0rH3B?-J1c6)ffnfUr*1AoRU;mf=-%(n#sZ0M+~nj54u<~sS3vhpXS{J_iEC5?7j<`pR=Z*p0IAqD>kkRHuVxRCh`1ePmg0 zJ1O^AZY{@sNl*=1#!pkKZQg=c&se}1pk4=yoO z->uB6sOC$R4Z}PcMOCmoLSAmi|KWnB`&H50$^GE zcSJxb$9=^MZWcRCmPyQbYk`sj)BZcEj$Tc5i`#gW9D;lVq$C82cV84MgQ)+{XIQ$L zb>jDwNma2Baru?EhbK1(>6jUqL&4AL4#@v8#CAM9nw6Dj@5TnVWe9+zO4y6rx$!9Y zeTI1mQn)CmQh|R14;L3+kkkD6Dp>mRWacxN6#LL9lS5;|G{gLD>czK1EV_!-(|1 zh8XO&EMN%8xp>@Qu_lOrPCdglOAQa^h}GoNlkCoJh(W|oW6pRfH2;Ghsw+Gx9leOy z_ZTIai;veF@$tz7V3Yl2K|n+F|CGrcC6q*2-sb96@ew(&Bmqevbi&|rY^-0)c2f8K zwP?UudS!|P{|vVSPD~}UMwZ<*45>C$cbc{3{V;DV_;uO!ZOikKktD|P7$^Ce%QIHD zse0!4BG{Kw?AiKScmA6TwK=D#NjI{!LZ~^!BQ7QPDvT6`g#a>)7bwo7ZFPIuEgU(1 z$xzYqqjx2C^E!;kR-`-kF3H>=HZj2kejxd|j}wHFE-o(1OO-#YK(XH1-mYwFYHEG$ z$k>lB-%_a-kCD87ct5iVI28ljAi+=A#6wht4TqwbtRs z|GoL61x!-3b#$C^)H~RLf4)Y#8-)r8f%;z%tl!zl!Kw zsa&04g^X#Co#jXYk1Lfl1diR5oGHq}U^-q`7yFw+{aNu8y^;7o{NT65m~nI~HI;tv zWN@%y;mSp77(8wgLhtW2zBl5D<%nS}DpWET*<=*wQ)DShhvCH&ahQXtU7~ zm>ztNOb!wg6O&F}BY{OqI9P}s5z|o(`-rfyu63lu3)IB^OS$(iM~3uRLlE(xE>7}E zl-_r)>n|^`|_S_5{2bjcu36S`S~+OY6P9Ch48Q z_*?h9lV~xc_p@)mZddR)``?iOpUKetw%4Su-cmJXzrM?QNz~#(x$2iFjMxnUEjO=0 z&8a&M%Fgj$VMy%4>((p-QkSs&#PLb%@>ao<0z-G(0sz;Vh~vN+|1lTvsL1Nn=(QA$ z%#+i`*YwTO0`oib8a*zMxQR9_-oaU8-}`ihy;*f1k1zz6kdWvjS*A0FMfy<^ovsfb zg1}t^wypQR`UU%vc!V6wZ&zpP@g*YH`B-MzNLinau_hYxQ%T^*yRkG(gq0+7tEh+V z82P|aa7~PeCIs`5x-N!q*`MEu%Mr*!MQc=N!z3$SIuVPC6K1Qr&o!M-PuhauNodAEm4K}N=Ebd*yT>2J<|I0^Zw2TEG#rdfohGtxEtCUz{e zuECIr4H6nTwnNN}YLUi+`*uCIV0iJD^J4E#tVC!MSMZni8T58$fXW?j}F({O|fAfH97}x~# zKRjJrf&?BrHup<&TKn7Xjo71!WEh_mSViJ+AXuSfE91g4s9M_6?|mg&@^| z|00avp$?XzG@vgDs7j5C!eyo<3rC|9o}MTbj3t`Q`rn9rBB$xm?hOE%19|9UH(0LI2U26h4} zV0fsqw}1%!}G;Q6W3TDBb~VGM^dIrz6BY=?n`X%Zu52R(6H^Pq=^ z2LQ_W&2cPo(a}fI2C|nIVKqcgZ?71$50ubMdnd`r-LD!F@%fOhWA#jCYz%()CPdfQrnqhfN7k7B6?yZ!P{r<3}|lksQ;Hm**0xF`8gW#vEk=p zl0u)6T%l0ZP)1_gzpIj4o`_6m9v)f?RLv`}%4s6ib3_3g-Ili|;~3h{!46^g7f5`H z*22AfktVA2Jge`mouNsY6Sm{KUGWmf(b%;$xBkIFgw7?&`^z18#NnT_{ztrWisYW^ zAMuJstV~;ecFSImE_(fX)4x*BNb%Uv4u8Z=sIL}z^%y^8VG3)Fr{YG{6;30;1I_p! z^_-pbWi!lIoi~JHv27AIA1;e!TqdkM{dHVRM0Po>Yj|wIV(Rh@b_UPgZNFUE4fa}O}9_rSeAeZES5A_>$XitB)EBZg8MI&w&Kj^Wx3;&_!J@_ZH8t|HyL$CF z*eomSAnWCtU(*bbzLKSPO2%UCSSZ|pqtV3EGHB3 z%873Ep6JkT4S4o!DT#3@##vToe4>@~c`A12jlU9k-GVF`v>3t3?y?MUIzvyYg z-cB>C&*>T2W>r>fBgD^3=hJ``x^fQbK!|^RbH@~*)UmUXr~oaEdP9?_JAdhXp1T1Q zq03!RgjR0*Vv5kus0fWZm$6TnJmg*Z1$p#-xo3wkz$6kzEVR0*4)#lF$?u*5x0Wwn z`|lID8TN;YuW9-Ie9xUEY#DS~@eSR9zMnHIb00_NH?m1`Q%(cka^a>6HY({0?NmMy zo!IA)0=NbB8einez(VqPT$ljvzrLnL+TW7G7*L#smmyxRo_xAQd@&ujB&7)alNmvg zuRAdtE?};rLV~EXHLuEoU>7_>l+=VKI_oHeUF-S!2Uk5I*SDk-N@7`bKR*Y%(VO7v z@4)gRHX$KW*lz)<9~AfDts)HxGCWH`K2Ra`nmH_=(h5yVs&ucrJ?R)!Ib4AX{m&UV z%a(y^EMdSP~8?mr9`U$W$+OIlrh`ev}o3Vm`Cik ze=410!(~Pxz2GbQ``*`CuXh%nxk<+&etZAI%nd2lq!zYsaLH-xGv6YlrF7_zU}ttF zX-c3N4as-BV`Rmd-qB`-sfl4Bq16(w?OK(AX6E8S2Q-C7sJI;G@5cZ4+Awj4&;K7| ze;rm;-?a^+3q(*-kZuV9B}GC&5ou6L=`KOKTR@O5K~#_sK@{nbknRu!QBoQNk?xTC z#$0&c&vSh5-uvDD!*yJTvF2QJj^7yPIHPaWF|Jr)9f9y*pOJp$UPRC0>Rc!UJX``4 z5BNV^^a#-&N!Xz(SLS)FC7^99g&y({x z6pELUE_`_zAvj7)Mmf=m_q?fwmv@K&thjE|;o>gb-8YSWF?j>6)@ghE{V4rvf>|B~5jL~EJI&m!7$AS6vyFS!vxK=s> zC%6nKstPNH$_xGD6OUKjR(i-1&92NRS z)&5H_@N#-I+ePW#!jP9#$d&?z`8JrI!T)QxVv-mF%i8F@@!=Coog5#1rOjNn^0;}* z*%8S>_5?Rbirvr_OH#CQ95$P#V<}52svoQ!qrn%^vE~RFJ%9x3;P@#)M5>ta)Hm%k zn(6gMm)fl)Cjv>4mm9C|w-x?}WYR$EdHEa@lIbvVpcbob<8kQ)7+Zpjj5BEn?|4Ni zu0@(I;`j9*92}p6l>smx6rlP>#nDLE7vb^VnwFljiUd~_jCU+=LF*V^`s;Py8$~YQ zt4iqW)72iVh~^sB(Zh@tVo+dzs7M9rMhGPdB<>mDHuf;Ougsz0opZ^{i!9o;|Dhq# z-@kk&*mb*AvbO;tzZ-CuZ9h!>8I3k=LB52c3p?hy~H8b~O7G6N|nGli!5cyH%#{qyC0>o7W9c5RF=wO%XPi*c4 zgi;c~(z$y$*fEqEZyjg4O|(;wtaKf2U%(Xec!4UV|6c_g&ZUxHJ?{~^{L91F_Ybsx z?1$u108@=oJ@qI~^NG55{^W;GNiuCAZrhC@41#8{|E=S*{k3Gb)lWZle7yI_e3G3} z(zxI>0Z6gwO3ix6Q0dy*IRVM1F9Ts%D0-=9iXqDn{b8Z+k(bTSPv;>g0%dn({%6~Z zU-ltQh0*uV+MVa)ICUZ3A@IZzMV;c%)AkHw&-pG9!s8zE3_^`D|F6aX;U{q+{3K(M zfsNNcYW~YldYiGy39E$F73C)Pj?cV1$|kGar`{1#dqT$}@+lX49@(BIqmbhc4|P6Q|6@jznb9GQis_Ts|2#L#ES}^SA>_H4#=iQAEWIa|-|^;~*n|X2 zgitrHiD>QoxFJKE2E~Yp+*~Be3+ik3>9}|I?I)c;A|k^kh(h@`PcS$Jb_Rt$r{-Tl zv3ZQ4Henj?PMkxPo=&VhGRy9O^nTG1u;UO;i)_P z?on4$Q$v-pFbW}}k>e98=cQ8=Py4K^OrcEI6c-&h>ZmC;a~wJ9aR0Qp@~V#+YxrKO zo)x$(LOsm%zwTkZ82{<0qw3vnwY4{`C4!_|e)0Ne=0HDj5fPv&?a0)=xLoWwSg_jm z)3o4{L@w5^;F&X6qMQtSoWW8j20-x%U1>DI6VA@bA!cBRA)patggn;(#_MIQRFDlu z&ZYAdUaq7Y3b4&A$To=ST+j_de?V>hs4st`t&f(bjegMyAdPM8+0(L{7y!6^5@=za z841pk0qGD?AlW_86eIg_CRO3oQ>$;m%*SQ#tdC(qun}D+{y5x^>sf*Lx#8u6STRysLHA z+xwW1IM%!*0W+GyzykxU*6c>pp;xW={24h9f%l_YqU&{mxVI(;8jBlMxkX0IH>&Vw zOMeQ|H7RjR0^)=n-9r8cy@-HbVNn~bbTA;Oa9n6w}+tzLVf{CL9|54Bj+m-yO#f4Cbx2cq*b3qbs z4dxk^Ff2lJy=_y$gf=oJdcrvpQVhm6O_6nF=(|(GwN-(2kUGLaSUSk8pZ9DoX^)0!HCq8HKafOzoh@In&Jxm&@_x5%Km7jr0NIP^r*!YzGz|Nd5 zYNIzD(O_b_$Sdtm0E^yRb#8-}>_g|)o$N+6{wRUDezVNqncE7+Ni$Ih-6&Kxi2?0> z$|8VXQK|+)0?Kz>Yn6ol`-SLFpt%na7_xw3HxDvh8u^Ed27B=*^#9YWS1%emd6>!eJj6}RBs+=E3wude)nR=YG8y3kL znz&S#Gq!Ei`ZdpAQkJ}Iv3`+QTRg%?$N}=Ek#hw+z}Y#gw=cKDtRoph*mAhP$>%V6 zn_qTlA2L}fYYg)=6lu=Mr+@m?H+o404@*@_0{JPK;vmtZ1pCtTD12gKlJd}>?T63_s z=^RQN?5d>H9}#EY-RUx0gs8QDPP2&e=rNkCXeJ8mcc7wsLpl!gvF8PynkfauLmD!C z*1`K@&Dr_7YnGHoK?OU(8+sr55i9;Qey3jJG=t{vc0^N8imqPLyk2Ju#j><03p09Q z!HQG67)!U@?yN%&fp%bgJSjSN`*(^vFL<^3fGiE^qBXO?KkGe2!gQYPuZTB{ir81L zNH&ZL_j;TZ{q`Bg&7=5vhWP;BC*1HOoY5UezMMU_!)%^rG60dsDMsRII@zVmTxF_n zJ0r0Y?civ0G{A}8G`+4l`@Nu^r(^QnVh}QkTH<&1`qrPbsHvG#KM1zgUAYgs-^vy~I zeUOGN#b_MySPi7%$w_?_yLXXPl`Xx0U~)vVU^1gRko{#RXp?ulqXR-3sR66|*ocXxOc8^*`!Aqbif83Ec?hjJ7yU?Gz?5?Me1bhWBF)MCff&?B!{qkUy=W zs~hn(Nyrqm3VS0H$H8Ekx;lqA*tVO&)kDaVZ&kG8$Vy|9UFF+bmXtPN!!}yrlhCmjW3;@Q%g&u)1zR`3`wNm z=tw6mElnbAIYONsz=dcIkXWs3uzzgngk0o+*g}l z*6R<%%MXObZ;4_*4j8?6fp|E(+exX)?J>guy*Ghu^OX!}r@Y0)*R=7Yk5fK9A@=Wn zZeg~c%R2b(SkH>@m4l{+M&sA7%lUW=D9}LS3keB5P^S_vx=8GL-E9!zGcpzH!y|z3 zQXSshd0QVz#qTiPeE!}0_a?vEV;_b_!cE=EUfh0vGTb4yR z^d9-U841D*=Itv(>CZyjHMF|QpMqdbC8wXAlk+Jgyhk|W;o*VS@ZoyDuvDMtgZ@Q_ zTUA{wV*=ROzE_9NZELuN|F19mYiDQY8UL!|8W0<4U!=fc_7)xe;mA36m5NFJ#jDf) z=RVT^fYqLWqhU>`c*x!6uJs%v3%^~aN6;sBnT&HGHu6ZKJFq0ydu16QBLz4Uf$ZUk zE=Q`DIF<`6#s5h6Lysm2%|_fc#aIoM^H0VEkar^V+{=<+)%*7Y0lS+7w`4hhG>1zp zVqd)I->bf=nP0ugcjfBU^={Es#Er1N7{LUUh}LNXO!LA3qb{&k?@pu>ux z8ubE=+54yn{$kEA9W*U_ZTa#4G?PMg371FI3^k*_Gdl??!;gXJI>%n~E*)K1KK0Ue! zG|^P}ZT!-TQYm2{E3KulDvVXwtE>?P9kHw1%i>_izgv&)CJ8R5GQ3Gl41=u>xTT^{ z@d~UU2!-J!6v!FK$bdsc{}iI8^<&iGWdmPkiJboF%^vZ(-$>oylveNs{NMR($bI|Q zzdWE*1Pxvd7-gFM0c$M14;fgURyj3FAG)vl)-}!ndZ*R{CzU!r038_4_jNX}70SktBf_7xK z@q>RxT=0YN=cf)%hO1Wh*cY~lCf^L-<9R0`$95Z6jz{771vy3uDT($cT1-r-`YJb6 zSCt?<#_@ZeyTy@?`TVb6irTs($CG4MBD#4Hs2E*X)%Uf_Zo{+ApxR*g z$k$@${K1mhZdH%iOejVz_p5M0AN|SIhi(5};gkf{T;oVOQF1aeGTRm1YL9hS+bWQY zCyTh^q6&?hup_Aja8YS$A0fk=6ChrYb4^0RN-JU+Oh#tzA`VHDdrb_-=r|LIr>FoY zNc$-vxSwiD<~8ID@6Ak>8}uw)(;Ga5C`4Uci`s_cXv7~~tt=42K3kEMQSF^o=^W z6DV7V;zNYQGe#nOf{psL+d{&_+M`q7t+c&UW49q%j~18;pu_KY;^{UMZ_os_aWK`a z5p>2FXhsazc*Q8Z$?HF@3oeO@(g3B0U#I>poDde>v=Ea`vK-jDP53rgGL^%8gbTyrJ1@2DO0i0_C1m0 z_PPhTj+_O!ZJSqAKR&8``#1|LEk>CFe|akU?FrY={M2kS(F62J?V_IxOz4c>NpOci z`6DZW6y|f7052gG7_oK7o=InCf|3b+wpOVX8EDKV^7(+bo|vS>W71TFnLC>LB<{6L zprdFhE>YCN6N}c6`ljLh?cXs+z$tTo>+K|M{p3^-_D$~_79!+S`U?%(vXN_`4w;Nz*$~gYScsvM-O&I zLn1>OQtm;nbOVBoq}J7}_l}Pal9C6nX{f0MgoopLczKmYbLo6KXXtk%07v9m*xvYT z_iNgg(ZTKjStS$p!sj2OmX9l(vEdk(7YT{{4&R3a>OIj;TVt5l5&bc>os$aiS+l-b{ zGLWs@`K(j%3FSwHquLm#|97%?UfsbJPlX+FHy14&&(f~eo!mlV&>uW_KuNc|wbg8R z@>f&rV5#Jqtc=uhOY)dNFjl3hB@q$+d8f8Cl>5K7B;Cz4&ziEdv@}m#^afR`wvElw zhAoPbm9>SlNYDPsV|n>7ZTc8aMf1#c9+j}JFi#oOGyPuwm>mj})qSgsna5S4Q-4H{ z_{0kFPy`AJ*M&ZvsmIbv>)#d)hcY-$km7WfyDq42GfLub||*7T$Y6YrYB06+>x%?3EpY= zS1{kPpoazR+V3Jo=5gw*7xP zh~05&9mU{btQWt7lr#}`*fQ_UCE4px@YDOONgQq$_(?;cLedR5Gdvda$`J?MT>UEY z-rimyszj4ak@UJ=({v{wVjEEmqF%~B4c)^n*%83k)EpdmBS8Jbg3|Do+rHpjpBl0Wl<)K`k9i(w>!Ergg# zRn3QXb=BbM*PkSEG6OSU{S&Z;!)x64hB2#4^AM`qi7DTzic+c=SWm8j8N>E#=M%^l znm*c{XoJRxr8sgM4acNyxi1j`6i-NBbtFUP+gnNV@aU{f zK6m%1mz{6%@m)gj{`IP{WQ*slDS^`p0SXLmtK(6yXg6&RCAMw7NJ8RpeCUa)ZzAe+ zx7~!imyF$4@$v2CXF5I<7pwibkaf{0)m6_zVPU=D7?bvD7Lb_5zL6L_oGz%rr0?8ogI*LNpCLXS1Cr{7xSKi~YJLJ-Cl-fj&Z5v}s9b z#(t@=zFx?E)@|YYdNE(H#|V~XSy44Lw7nC9RA+qjzrl|uD?6`T2fw||FUD^bS5uX3 z6L0?4Eg)|Q(mV)t#|`kp)G^|+3l9@2t=CdTLu0_JE-+9s$>ceNr{zMZ#RsL7Mws6U zq1%#@mX;uuMnvZj^MHf8n<_@7s;at7A`0K>*Y+Hn<~m=F=KM$)AC1Ro*SAtIDwx|} z==+7`Glnrmzd6Eu^DAHWy$)tya6v3(104~8LFCF<2*(8wKOoGHi^^h85VjPL-#*?a z9WDc(b?LwoE$D-hmcD%43lGSKtqdo7vO(#r#ddKlZ__Hze~4Ig%&wqk!_AJ>|5!7% zLTyKAwRK|uwiRV)v9-K{Zz1Y>fQ4Ax0PFS==OY4Bke;`H{VGu52UK>gN|#5A%gAMF zm%QIDi;5x;v>%UzaT-Pi9H?pzzzAXwx3sab8QVoB4RCtOs96?ST2fE%e;_gP(GJIP*z! z!JUgnW>kJGw1p1MITd+5F5rMjhY|A#z8xLia8ic^k;P^()d4yg8i#=C&tz|HYHfEH z2L&(_lYjsj0Re%7vomCeok7M!ko@26{w7KQOEymGJVLgO9B8dbptW)oE5M!I{z>yQ zL;=whh(B-y!5hi+~7ZiJhBi4eS^0raxlPpW$F*S2nebo4nq6MrQRH>tY>c+D2YX2|c0hZty1ZZTm3sw=1f0qcaTjorTiuDdj6ZME&Axz@Td3@5%NV^cL;nIe|5!43OR#3 z2X4hCCgfDf@bJiv_ErIij-H!|^-qDy4HD_|(7rWp*#2qH`9>L;qmKD-j6U?N>f5!I z>9pHVr8Z)vh(B~+HaZIPW0u#&we39k&RMS%cfop)t_1sLG{)l@#;gY@|G$-ZTIHsf z1|Fn7m0_iH>35)z78^fO>I&Dv7WLNr><2~ z$LL>7UhX3|oC2v7{9+M8HxC^aelk=qSyEd5G*V~GJU=*?%uI0DTA5>VHj~=ODY)7& z9kqK&I=jZDjVy>#N1}diPR&N?F-@$j=1ktl6N&FrvwK+{LRX;cuXn9|hD z%-;1L?0+sb&CbnzAcD1km0&S2rS}n=VTGqYwpu4+Ch$mqtt;htesvLZei5V$Y|c+L z+r*fit-t9tPsih|SLsye?{lberPtQc=r9M-T#;{lP%5;-a7^Lhxh-9Toi(^tLzdWj zT96U*ZcZt0*Fj3sc|r>B+CKCa2x;rvi%B(vrY7o0jdgW`de_Gd2t5|MQIUizjf$^T zKU@l9db#p=sWL)H$5I{2oMRk6J=Gc$=3(3Gt1!J=SsZG-lDnkoHn_)CM`G?OG^lUd z?k8#ubI%h4O_x*X40>*Ih{mzCEBC3LWfd0{yj917GnZ)2QDEH&kL&vOl-~740%fC3 zSLb@Xi%kv_-%wdmJ{-0m0o@~H>&b1qCE=`iK*{-YI>;E0&umiiVR*O*GyM;E9ZJi9 z9)yto-}oKhK(er)xrC@v$+$>&M(C8FvAZTK=Q6`0f5%t4DbpnCpvZ3u81w& zHS24=Y4*2Y=#ydUVNRN~S${F)K|d>Flu;TXJv-(z6W%=;`SKRLD|40+AKG?Xn=%5X zjB+#>?=a2giLbBf{yZk^pnucK@FYzwkpsTnxaZQ3H>)1~xC|JSp>soqd89seyPAOI zwy>&-tKygd7REV?e?8eXe)%|u>?P8H4|%d`ZX}nDfT$?FeZK++Zs zwW^^X26uk#!$`m;e<9699b7y(8U((7e;E;Q+<$+!;lA?Cvw(GS8rIT6!X(0il#6Uiql9ZD{xN-7a#zQ#zR$n>&)_~o0Lf@Wlio4&{_PZFo}%IJ|3 zb@!`&n#de1qq#y9XDDIdn<}7@k!ZgRNH=_2)6!^|$Cy^Q$KI&FDtI5}XBIZ-;UPPZ z_4Vu5YcT1X0681w{~^^2H##~R2@AtO71aK{Qf4<6UQr>8H1ohh(Y$6>d?Aiys{HVZ zk0(2YcnW8*23uM5@NgEdYZ?PRE!9&P;U3X8m5YDehUyZe>OD*lmkQZ`NMMlf+Znvf<+&Z`G_(F&m za+xQz9;F^#e~@4<4xe8fKEIs2{1b>an|+~~FJcdwD#(5!)n`LKSEm#g;U#n2DVB1* zC&Ib6BkY#UB_NXDA2#HWi=v4pV+8m{<10NT#lfQT!PP2jWLO%Wrk=MetO`!b;0nh2 zZ2BJ%GZvFYzHz!HqM*tnqa|^JU#r@kZ#-p<8?T$^bdUa{n|yFG<(>0Tid*me-Zw^f z&eTx~_pPn?8xU^t``4u1%OhdYjcsWuvplShz91d*IS3x9F^GBp{zU&oDNmyb(Ei!k zS=%|Q>&O6N40r~N&{?Bz-4wX+T*>3OQQTzqUHiW-D3L!f+!Z`?W_0b%KJz_t$4|2E zQI{lMRa$2~cv&^14Q@IpNKjH0CRu3zFZ_dU=sqV`dyo(v9@gKpKY3Gi9EdT7oxqo# zWXged&GCip%gE?>i3JfVt)fEfiDg!ex`u&B=TK9VunX|bARG(C@6S>*AEi{v^Kz;j}yI%JZQ!Z zm5x^7^t%^PL}gyb;C^mGf2|){r$9E{^_Xv(S(i^Mo1>wF`@V5 zfzW*6&L9a(DtnNIeMQ;N{$cW%sI8Wx&(m7Sd z9)Y3l;m~r1ym`Y=TwDx3;Q$k(5Roe~P(ds(FZ8$rpFx3mX0oDK(FmI=6-&)rTazZ$&baAF@q(7F0=@J-Q8DI=60mM{Ml^KOZy z+NtfTLy+-#D~xyJKV771AB4|A?M)=el!jSAeI~>DYUg7JSX* z6cw93JO6eVu`S0yA>La6zM-xEka7O^@qUjVHtK0e$W@Z79IKs@pW&{?EjvnT46ox}N2ELMjqqYeHlaNs|b402Vm1-~tXK zK!EDP3(<>Tth>uBKVz52EhKI2{gH|t-o1*NpPltTuH~kjT#SSt=QavMvbVoJr+=O8 zWZ8Ax{>A!pE2*kCm~(+iJshu8Fowy@Ju^(SPd(CW4@nyM%6U+XHaAW0mWyLq64niO zkL$ja@u}NL?}GoMz@zoXW?n2W5%dLG4_G~`SXAEAbi}>#0MkL*!X8j7NxO1myDf#R z2UcB*p)FJ|jw^R3lpmPNkII}oS-t>7jse6{^mr{qd=gF)OPfQ(mKDG_i>&&uAp8~N z`0+hC+8ro5Sgm7#7C)EdZCoR6PEw?S;=0U@;cVV-t9~uiM?T3@CKESw_K{5tjK!)s zc>M6s(k!cfG^-CU zGa3@WwzZVSxUxn(HMx*>eC(in{6t%igxtQGh~sG}#Jq2a$lAs{@#mje9L~@lfXJ41Re8w$Xe043vkD+?@DA&)2MP_T` znxxb~L1qbV0iX0T;A{UzQDib5%YFEm-)(MH%EXI_kb_h)SMs@moy>izumkXILBK%B`xu2s?Rz2_h5 z|0$A7Yr9i-D8{Q^PQT?s2RRSEH%1_im`DH7xhrJYTMm{1K7V8NI-hmX+$wa{F{lA*{k&#KsG zss55JAwOsN9)|STy*fgilFdKbS+>^gm`!aS4O?C-KK|L?Lf9i6WU{j?QCKoHd@QAT zgt}tSFLP9{!%z{xw z70zFinWDy|@wTxuM?+80K8c~ZI=;5I1RvTozmL4fZqF%nqi3X-=7;2mndX*@Lh6&m zR2RaI^$!fa0VaZsgw}^(&mSML>u!;*@WY08#pXlnVJL(=+wfEjZCamAzpAb|JNcXr zZ|$D`TxMr7U|3Umuhv|QI9I}aX#QbXQUm_0pW3h7yk6YKMdl!OJv)RdxxRD&E}-~Zb3~hy7c&rpt;>0ZOOzA&oSC6W3PFau5&3E z;uLj)rrl-Tdz^Fpoa0rL9A`NOeY)EHk_;{FwIC7dEtcnKn*2jD#>k_ zPoC!`I&}QRfH^kIgAe zhd4%c5-bc@4mBhz6Hb{)Y`fYE{o57|o1%MOvvrsB6}9rHNUTG(Jv zK7D8A0x>0q?u$R2(jp@wGh^phr(K30`6TUd#T7qKBRg>~%d7cGU(`^PKz0l7sA}i8 z1zViHf`UTQ2-l7F#cFpOA76Q>1}8US4TJn_ZbnG{z8V=Fv$7o20@c9CzpDC)E2^s` zTA4(8>1Hu*j;~?h-BPTqky=A?mh<;*s-_a#Ss}}%*Upi55UL7)9G-I?7hF%j6RcVM z=3<%y>|+cy*`oh#i3)nb1p|JO=Vj%eqq!z%biehoo=t_NlYwSiegge9y4|4h5Y(Xm z;&>e_J-*qZuw7{UHCH^&vNC_&juyI4y;5W8I`h8&YrxONFm_1i_ z{y(-s0Z7k2G6FsOGw9ie&jezQ?lqjggPd_nea7|;1}yK{O#G&NpN|Qmj6O|RcHt+5 zwb!@jxQ$fMbGvcRk7Wv>>f4@MR0CS!DwSCp<>79DU3)W*D8+J*p1Hn>nCBjD+s(kk zG&M)yYAFog@Ln6Q9W2z(#g43&`WtRl&AWC!l6L3mAy8pdLm(Tj*eih1$N~7EIF&O| z-gc6{(q|S*MTAF&8NdA?&L=D{OFK2f=`#pl3@_qyW`;T;^|95(LP8lw`7TjnG?g*` zg?Var)2DFj&GP<#!cWZ0SfL`HBQon|466+4gIh7@qmpT<({NH3Z=T_EKc>1OP-h21 z>CH()(ghVgIvd9%2E0>KoOY)_Qgk~ME3$XIFyTiCd)VdQ(OYz36@|i)M6^V@0^*Bs zLwZXexiTF)3sbBvf06R6888}c&>rjY=j+K@ zXz0_QivNvlVYtj0ly{Ry93G)LbF#Cv>Fm?fEfFw221nqXP@c;EW4Adz@-(zYDvk+q zTXH8P|Nb)>_UO_NUS=2UfJ&?m%1eo7knT&4Bb|M!?YXp8Ef<|98FipzsX~43$eDGY z2ke#ZjwcMgm`h*@b9jNI|9mfy%9HjYxxfl$&WGUo4MVK z!SasP$bM=Z9v;dmQB3}(hfG&^!^Z?UDUpxaPO{v?-W2z)L|9+T)J3f94Us|kdD&2* z2)wX-mIYV2TWf;jUq}5h1%Nc$7x$Ek5CcfF*IYJN3hn}Fw(k+UPQ4@8n~JT~pMlQy z3@WSNN^EDybPpz46)^tD%3?=&0C6m(cF$8&quEkDFhMf!0er)F+!O?}flu%bZ=0f! zZxc#S++ULCn_^2hxm`!jB|V|n_iCrSG{%mA7_qD(`dU$bms6T*BOwIN9n0HfUH>uf z^>)d*_R!p>DN_d1mN26ADg~k}g0>8G{k72)`wWYo^$i^8=2GHJ&eFnwmvH?N$(1Wt zbSQ*eMCgu>R{apW!3vLcCfN6caqCwyLWgwIYYS+b7>H)ExcC|ZCjeWyK9ra_{x8|- z=MB{BPb)kVdJSn3Otl5jpSj% zXLfLLX$I+NION68faPh^#L2Nc0tUdQ7lE6DaM!)~ z(xSSy_D7dMUzQHo9w=^lOa!*ItkcOq}`5BQI_p2 ziO$`cw%=RWn}}u6xP47+v`>+!i|F+e}T9F+_%*`E2 zl+l-X4w#%9fPf3`E4Kj376kqp0^j87eI{X+iL8e`{fl$|o*0oEswC@*y*zSqiuI`h zH@_)*J$DXqXNZO}AhjJDQ1jEEmAyj`O`lb;wj7vAGR@puusnZVr-|Dzf5)nMVkIy! zux850^j!){l}+=0Qb@DmgjBj9#aHyM4e3)X%N;{$|FUK6r^3HU$|Cp5Fj-wDH8nLW zKVJq^$;l}xx~l~R1tB>(SFt2+1A!2v+5}}USk8BMcLUcWD=+T_pH=6(!NnpVJtE9a zu&k8@qo29%ln4-%Hnp~j35amn?vN@_SXpqXH^a^XSN0Aa)qeqhKm-LkomHKQCoFtp z2&iaOv@|W9Qj=YP^zQBqqR(Wv5hzD<$W@{aagh;G0TVfjejOZ}X2?>Qru{ z4rHudiz2i{QhEM?&p%)ik5i%vEHX}@$a#L?{LzF?*viUka&B(m;nW!v+=Ho=6@*fp zfyYpvdGZe4ph>8f*QBiWQy=eWBSyW5EF09qVZdmeLUu>RB_->_7H>G{=+QG#tolV( z;*fB{OP6k9aO`t@lNW(6(u+u1&x7{zPC~mK4c$3(DTR&Fr9Up`Utaz&<)dBGP@#Z& zh)+-tbc;1-_XkKNeg8wi*!jzwb%}9}`0OA>p4Xj23M6yFyRC;|Ia71=xk#;&l3~}V z2U~hb_Ku7m9x?QeRpfQDE$*I&hV^Cu`z0tzKp8QnVQ0?KKeykKUG3`H~%ti!=JUa}xRgj-y?Lm75x2 zLH$C>;u8ThoU_ckeFo5fciw){P>v&`N{f3RY`M zYSeBiR|UZPw4E-(^uS1phX?c*#e{V!$3zZM)h0Te^2Q%!z6pB)SQ&7r%!3O zRH$rdBr%+usN zFe^^lEfyL4qYR`&As%CtMk|KaVGm1P!{FarboI9uW|T}IvXh+e%@B55qC^ZGs;k9N z^+?34(74CU>jQWTTYywp1ROLo3k%6S4Q5VGd}LEu(sK(BqIWY!po)Udd15}Hu1*{m z?_T*n;t*RDoqj@X!T%D#|^ zAi7^%IQNPcVW?G<=iDTh3Yj8mI%AZffd(L|w7g&u;)Rh!jrUeaZIm<%ClxO-Oy1NW zP~q!lVpwRq2>V}(oT4IP;B*Sy@Ga$&kK=s*a}P|TD;D#s?7R*@(FIN&OtAMvP5%1z z6fke#>LQy)grnrPGI}0-eFJR;s3GGjvb@XL!Qr-~WYfL2f8z(S`gM)TkS+$~0jd5e z_Iy*%R^QQmwTc)fDU}$N=bKs1^Xv#I|6MgQ(Fs2bB)Wf1b;MHZ_QEN~WeRmecz9jW zBK>NAKl#iiOmyn~RM)exCc}5NQ`4Wa`Lr3&?YSFSujZnkTOrvZcRvXU2|BCwMo4*u zH{bTC9DHQs&P z#i9_tsHiA70jp;xqR{7A`%n3}r-q8Y($E09{aUWDDvx?HhRwvyNyyj0=kRID;>v;t z`eN`?44Q*Y&uV?Xd4^RW3VWu^3WF#Lzm@;uoyk8o)y({=7N^Zu%~QWuOOlxis0X$a zZ*GK>OphJR#LmBPaNAZ#_)O`#8Cn>!isfO^Mqsv-m6c`bJOL%HzgFhlTaR{170Ut1Ky!GLJxT$B6oY zeJ~S*W}B%Yb1z=sf5NKUb4j55`@Dyg zmQXbbL&i?6SPz@5#gKU*E?Gj&!Cfn13CPe~Ms%5J@K z_-y+O$&ZV%_n(_OrIzG5TbD=ON`3qb4|JHJB2iu6bq@JC2uA$sj9h&D>uAW8R|5Gi z>6=5!t4Q*_kx2EBW1_m-XR`3yLgd{%jQ5!EmwHxj1EOkbj1dyhoq7`wtd>wm2fIQy z3e0n1X9sm;ZEMShj^?`M1$w_jM;}-%XB8B*0ea}THYtJVYLKH4iurhL9oFgD7XVw0 zUOZU@^;!NrQt;C@qfXPWIjHTDpzzBv!BEZz|2i^-Uj$FhRJ9*fGh@Ww=t!~1L=qdb zD8*Aa0@E-YZYNSj&D3!Gf<@=;KXndzK3e!|kRck(FIp#Cr|kZ!Ms`#a7otDK|*^ zth~=Mf#F$P2-8{G*w|QDpFaM()sV<<@pIIj8B4-n>4SC51#?wh-I2c~&!7J(spDp- z@<}2W6cj{OS$0`Qr-y6%Pr10K{OT5K_m7o_yp31kynpI;zGV!<-L!huL-?og7fM#w z>(D%Fo}wKmJxCiTCFwK;TLa4iF>7G$=16|_@N>VMeLZ;fUh-0(hgdoVJ$@f_?D={~@psdIC=;b+q~2}p z@@k8-lH7HxGP@FUj(I1FaL)k61{hY5c`fbVY;eIp*n*WZ1)yJ>;!qu`58HL#mMx!#UM%}W=?j7J+%o`f*0?5-Fs4!R}!hJW`Tyr zE0bB|<>c!^{p@arII{+i#bHdYJgN0~q$V6twek3T?+X^y<`v1#B9UdqWKDVhN`So> z_VF4eKeU4ik~lWV2r9)+Q2{Wr^bi<}-4 z#bfd%j^V(|y)NZZ2KhA(?)75{lR_BXg5Cav6%JDYk-ctWVC)7}&Q8Nk`{qm555}Dz zO@JKxxXNh#*R9ZV%U{yfn3%gQRvWXA{2J_vuz5E~{_te4H>Q1XJovVco|nyxwd&t8 zvazuNw+#DWXIWBBjRb_*X1ua-EG%qnXHb{#y=HL_Jv|VX-`9So7Y%nef{$i4Xr7e( zJI(F(J7t#lSa&5VbcO3Ek$bHjKOi_gmO{0+BXo|IYlV|Iy*o2sJ|t#0we7<}w`2w^`@MySrxVY=^>8q|uw=WJ$>00J z$~t_lCHm}M@n84r+0Y|R6DZd_WjkWxGi?+%w*GzEBfXS|5qNTPQumfM2>ZaInP1w) z8pwWc2TxYNRBzXG0%&C9KYkIkSvJ;%&DP#)xmd1AqXD zMEq=h`0kZe+M@wx6&I;zA1(sscCUMILF(|g1NZpIi?9u^yoQRvJlvSEVZ4}_#UO6Y zVIozFOxx_1zbO%G>p@3ZY5=08Jff98?|c(`aw?1LKm8L-X7j8MSUL)qlS`>V-tXk> zOt4}ILJDwz`iZ_M>w(-yHMXXvfrz~CaBsc6qeH^Q<$7{*GOT{7X=rqJkG2O5%MVs> zqc6)$3BItq`&2CN*o>@q1_0AK=SU=>r$xK6@>r;H$TgNvA${>jEc&{x$;Ykf`jXX( z#YenRBA3j$cp@4{D2lM~w}6EYcH7@-N`;k4Sr1*ZDLdnSJkxy3H}pY=brPEQB1RF? zu`1#Ay^#yrII0CrJqO3ZyFK=I-cmyZlpq3wJ%R3uwbE@d13g-5-04N+s25bimY}n9KO^HVwJaWtSoGu7vwvt zG*!LJ@YJJ3rp;(Xa43&mXLKa8I9n%fOishbz-3l2Dr`jpMlg z-UuWf-xjywc&+i;ZUnDfaC9K@^Ydf9lisPPU0-C;Lyx-c=vai=tKw~~UAVorrt<%y zMiiClDHbTTy$s+QQm@8~`YZE%(@S`_KE9qKbSEM>Cod@f7LdzccL4Kh#7*|2d)ats76DpY0#4iHfKpSa=<2IKHrBM1&AQn!L1*CuG$O zEW}ceQg?1@(Ek-BZ18whGs5c4S^bz8dn<|6*^Ext0&X`0LJ{Wre0le`8WbCU#TFC{ zWi77-te>T(!q*ELrVDg*QL+q=NdcWJSW01Q7ly;vX28jxWw8~G-G2*;2!UTZkDYJ& zb{|une{^10*Ky#NnCJ^SlK|LX(T6mQVVC>1GR|sCrk1K~Adyt@4cnp7UtdRyn+-!x zcUdFFrxpg+gswCmv=WNkk0f?&k251_ZF+N*XQj!@S45fNyQcN7NTN) zw`kts>Bg$Kk6=&xT3?)%mYdL55dB--g+lFt@NDnTvHeWqjVQi2Gb5k-r30X6#5Z(-mOp0JMPb!w0d(<2fV z;KUJ4e)m&2IX>b8tvZ4@1NB`^_)B*P-etO^Qd-6i2&?*^*XGgZ_m2jkNN7NP2n|ICt^ zh*N&-EF_8RfhPACXmXi5Ge%Kxn1bI9h4b%j#Lyd2iuml$z7lp_AVs2Xfa-^sdMkwd zTe3V#&JhlI`U%UXV?5yRoY(WsfGR9z7;<;FYH?Qdsxy_)VUu5SY;9VV*&^&8F+8GJ z_?And4Qpm9eb~)pR)@!IluxDWpz_(r3!k%yO_GFr_U25^jZ3~|hhb2JKH}e7Zh;q0(3M;Q#DI*b}YckVIM#uCuxh`G1o;Jj(*OT48k( z@6SOw$%hJHOe@MJW}URav2HMZ?TNIPa#XPYuymc~T80`NYuM{Qa%l6Flq_~TPKvmS zkh>-OQ$cJ7f`c^@NW?i5=(~&EJf_>*BfO8Ez@s~v@PvVBj(9Qo9Pj@Yc9=x@D{{#r zRvoZi2mY#7}5J-3GH>2#fsM+FCu#NHIXWFd5?2aQ*^J zM1)FYeV_%L+WCRCeO91y|Mv;~KK^+i&H6UbB}w>695<%rpKsbc5KN%FP7F8`<@jTd z#ot&9lpc4h_z>P?F}~8n%B8T9lrKPRD3;lmJ}8{~<+Mb$5JhgcHJHOE&O%=vG{)lx zMnF<2k<({sVQSMVc}M^X`oP>=PPsU!5<$S9G6ARXgjTDg1Z1{ zGWmY_d%pDR`yPYk-b|c_!QIU_FYVU{cE{qdzkoPBpGW(z8y~FQczi{TYyw30} zuVw%y19k{@WB-#^DsOd7ZQDxlD3qp}^m(Qa-ZO>DgqL{FKlv?bamlMI=fVR{r(%|Y zm)2$}=;;vn^kfb`yJPNq(-Ay{rfr0w>F4C*zE!!~i*2+s!?+B1xxd<5^yEP7sufi! z12Y3NAjNFlsk}FOWXuDz(%6ws-R;fDTapSR#S{tzEap@+RzIU*tK0_X-Gxlui3wmh zXjMz3DBQb;Q9D$0!BG8;bOtQjAYAjiqQ`eXu@#y+tTMN5^uW+}15j5!?;RT`mhj3V zuEcP=C`H|J)uv8~;jWg6FB0Wo{S))sPbzN!+VIGwsp zt>aT6dU26(_D;^luVoVTc@i9M1bk zq;wK2XQ&Q_0-Au7?wQQCd)+0(HoTWCF=1iQd~SkxsZ7EQ6S+z8svuFbI4D_PAK%Qf z{tlTzKeH!i=0cZ>|I&=RyBONz<2vxmCPuvoPeK7!)5G14#1Qn1{x&=W+QIu9t zLP1)(K}ktb2}uEwZs`UI1qlVDQ@WMT`^*>D+WVZ}8E4#aFaIpYT6^=YIiLB&C;TY6 z20PYX1RLcB+e6dxpr{#?yhZ_dy&K+=(SVZsOK3r#2${lZVu za<8Jw0axKFL!r3puIStQVSX1k?`J0%;2By&4|YR%Yc$iX^yC=BVPZ#OjO4WE-9Nim z^vGe4h@*G@I#%V}Xe6_}otY}`*O^ZTZ9o61=lZN3Rn;m1Lb0?pUtY}24(?ZBz*R%c zqrqS%8%|uV6oyCR)3;xJB*n=eiBiiKi3!-id+}o7t%M|m7T))2I1SKL&g_sruv}87 z4N!1*joQH@(iGLRDpNPEUyGmb*7`)pTpIA1{qV!*ZvyVk7O7;_4W4jZwybUyRJQ;6 zYb^OFpDa*2QJ5U%^MD3VCU#8xi^Z3eKlXgh@A;>VSq9vvDUUqbdgRIxb=pg^b-$cp zt92(>ZE615oS@_Ui;lk#hX-@yW_NFOaJEGCd4zcH2i6)K@h=;%l+JCa5!v(DE8LYr zBtf6Xv%rD9KF!$pJ6|d8lBV#-&=$7;F=uM*pNX91;j*W1c^g~2U;?dyT790_krxw( zpnaAjWOOMt@@DL_nZ?l{R5)DJqb?D);gjp`kW%}>6A*h#gh70gE|QPV^%^~(MQt^J z5__Yzm~>3@bNrUw7~&+X&g)n(010S$y!ye8T`s;xbhq#mvxJ%sDHkZ4h+(1WP7?Cc z$P34ePwu~1Un_f&v2`3+N)F@{YmweOg%XFQ>U2zvbgvi7(t*m{KlxcG<-y0e0R2dS z&TM#rl@dDTKcB6CuVGYTAbl~Jgz7XF8StB?&J?Nq>g5314h=#Af1Td{Zf{3STqasf zY)ZUVHsf+cs%lu$mVn5rfo8qmYl!!^Viq~cuh{^}+j2$1-34+RGQMfajh`zab+L`0 zo$@C;w>iTiN+U{suYQ-1jqf2I{QXehqqNu}MdrA$vdwV-^9bZ;q>K_u=u0%iID7ic zkM?gVeHUe5A}!NV@%@Hj&gIRhFHJEXwzhS#I<-7;LF(Ems253lfX+Wxy8?0O+E;~= z@!e(yZU(^m$CX;zUc!Q(I$XZaL@ZS=+;nn2PpnaHz`ZbT?5%&!V|b55D!WiyiVfSu z33T88T>5l~QSKtnOaYeqQEnq{X9yUi4I)L90Wg)Q18fa@$ltvtrHS(?el2wU)G69p ztchZ-0f729tRC`<57^SLYQS(1^8bmTc~5&?$C|pBrY@O!TCp4=_P~sq>&llNeWFX3M3{o zS*m$8$!)y!0NdVB1jBcZqF_VwfaMW;tu1r6_qr4TGy z+N$FnFn6J=&v=~_Fm83bLA~@T z)#@+Bk=PZg=$H$tc%ZNiy6F9F{Kzt;mRO=+;UJp;0>7ngANLcRjX4|G7Voglor3>n z5!gK(;@r4rEXY(_l5exc62IR*8w#oY6T1l5`!>^HF^mTMFU zA`N=9AY4?_*SB3V;R?NrZ~-&=zi5oc_)k`SD0j$Wsf>t9B31HP3aavNqq^334bUXz z0y}EKIKfjx`TH3j@-hO?(I6`f;Yf4if^w+93Ncf_TKeh%pc zRE%`=Xr`MW3e?iKU{Rnzu?v)xbdPN&pKLA>!?)pQ!!B0+7E>Tu%9|eva0?Fmg8v2Q zgdB%#WyPgmXx8jQhU_yD)B`3FJkDEIJ5ai2=jZbYbMf$ypsC=BgP4}x$kBHR(AtNe zErtv&Ke}R(SK+~SngjeO#fjdty^>+*Xc(QBaY@{7lfZACQ}%|`S3U-Te|S4xn<}nH znM#g9wh-Tv`Z0$pc1|p)bxj}J%`5Hg8x9(C7-<)AtohQS6B_ffS6S19$Ktomd#U@L zsWbHWOQqn%RlzzrInGd3>Zm1^JGj9@QM<}k#{EcKmeBF|3t#Lij}E9S^Bc^4d;;P7 zHA`(qKvt_>V8}_ATOYJ92uxJrg&b#ds_T1L!_q@;7ewdHnLF+kb-)L^-Y zN`^+KaE1M!C(SWpG{7YR2ub(^h>& z)KVw8-}hk%ruZ>g{ov#BD@qbF+UEm@M;1qkDNEBHrat%i1zjJe84fE45*mnwoe&F^ z1*jnwGMcI}#sm?09`LNbWL0Dy@e>&ICQDiSPNzzyYj6}NGEzQAXw^i9UNfLlVUO%( z^pT@_srqs~hRt!;_(dr*mFFV&H(0>%xQp+mrcZ&C$(Z@h;OAC~R> zZrXsU*GWoB?}>>gJIg_?pW#0xStZkH{nNesXG*8Z5G?d>8zSBEOIlNH>rn|^$99{J z%Rj@TetwIiVcothYb?o8B4l+K^MIq=PCjTpW&H@I=;^>6RYh;%j)CucKK>&-^AqNT z3C;lPu^d@$uzHmzUM9)?VV%44N>jIapsZ$ z)WJJFXf4=LvufMN$TW*=cxQY9Cw`9B9`+Qxch}6yNzIKtrsM|$kR)L2dQJWHd7bIO zhD@M%dG7fJVi};;0{s$5l4C}{iXmN>bpTnGzR$-X+`eRd#Z1*vl!4Ayf`Hf~LnBerLnbsKv*0wu znPG~OR@&ehMb-NRl&TLe$s*hXT>uPajU5jDqe@;FsJx18v6Krcj~u^N20~DjEb4eq{u*Q2uZBqfE!M3wNuKTp0^}D7;_j zsgb})g~<|8-_%8-wPb4A52X(VAIKX4^q0X4Z*(*H&PI{Yd6X7fW-sa>R-!`$;!+;` zH7T?TWID9aT>`4?#C9p2+WSA)>dcQDOq{-(j_Bo;QD>A{aYVuMPMx|23L#*pjH@ z3|%8fElRcYo-gZk<-}>`oM)v4g5x-$f1_{Iju;Mm(9WrpD?ct$nqegUooxY?Jh+8U z>tyLXa6$k6{Vo|9+53tLsQ%tXQIe3H(LIoc(z}k{HK24{MEA$|N=<<6@EkzW=+T-3 zAz#qD0F1(s=nt3YS~)T!?Z9%?}s zt<5b0WbBYD1CmOEc1Wg>Dt3NG6_5SGO!Dm?%xVgvOdzS$D9xKl z)7=)S;0~E(+M;&(0&UK&Gb8?!9odNF!xz})Yb$LyWpt$JS7f!$xi}m%eL)E56oInE zYt`ht+b!JC34sq&t*(UA0HR&cPdD$(Q|7cS{{!~~Y5LY<<;Q3K?i|_sx)xgUg#puX zKSt(%T?9OQ$2-&n<&e5=&8)I`v%xmyXq%r`163?15lZCW)gJnb6dO;=Ds|%nIVh9y z>sWZzD50(5XNK~{&kXUQ@#;y{Wgco##m|`nUE-{9e-OneGHVwIKUqlU3 zB%cG|wptLL0Xa(o1_lQ7c>th-q!IUbhxX%AQ;#FX{*C3a5TeprUw{A04z8fARQCK8 z2Mi!Xh@K1M91!k6JL5~z^&t;u$~A2m?M7t^n^?r$xTCF9s2EPleN?)X$W!BvbUKOR zUEP7jhwl(kuO|Rao-SnENT`!@1WcSHQG^v>;(*Q@<2x6y(Ozwy{g{%MNjD;&`s|zw z)jnr$!12HRFCS8Ri&?*7MfQ;y3JfM&n$n3P%%hHGQQB9@@mqrr*@K+3rp!^APQBx{ zPzuHeHUK{lI-7?fn04JZ8-Zsxy7cjG%JAf*H;9HH(}c15fZ9(_f0(aKDMAf!o{R@V z0+eiF{Ur~KzYM*gyDD-g?NT{+??9XgE8b06e^-JM;U$d)nX|+w#NMU;qOPXbTEuMxLGtV&2v zCqSrgTU$Kyr|q?UeSPm|YXSR>ltY^%1&Cvve!o1_ZXy=B$B^j!gKCk;Jkw7HVS^#0e)6%(7s@WkU=6&%}B z6CJnE|EzZumfe@DWGu!)%Kqbh-(TI!M1}qWeD8hZ#ynoJ)(gm;|TpR(mJ9;vn2)Xe%tQva+%^_?Zy5wY3HRMc6s1 z-{Pvff9WNLAd)zS)pv&kFh2Zwgg?R+T5Tt7)r?>KvuW9koMeSp);Fky5NzeljQ z_xAP>zV$#!=@4eBuaFD;HoON)ZFRR+rZX&q=tOV>WDmn^FR=y-UyYA^rT*^aXGBmj z%+F_?+$`PDm7OBEdE#9?1bw|{sAQO*w$oxky}If@O?7;@`OT6H$64#5m&qAkUTjX~e~vo2EaMw0dwJ4i z#{1Z7)$*KkI7%C>b_DwcQGob(^W6)Sw}NDUh93j`c6QKtT@#?dx}bEMVaam+MMlO& zB&zz)i!*uzhsI1YdO;IPN}X=XBHx4UK4IeZ1Q#YJ+&upZdMI=BGxEwN8P!aPxz^zo z-@lVe;l{4TUhvvONTGa5ZRJLh_q0qpx;{ga? zyADE)px;9BYg|u{Smp>OknCxOKIH_|3x32_-|9Rg}1OeHk6B zgK4!B*g;-XP>?k><)o&j27|j!VJ%I~fQ$DFBu^gccirDTE_(xQpAP%WQ`FfP^=Y8Z zj!^qfZFbEB(O+PLc8s8mG*cFEwjVSjT}Rf~uCAVR;H{+VUM#*Tu5(+}s= ze7MpS5)@e;q&k%NCuw~?RWSx`SR&c^wg;_Kra#%B`VQFvApjP*jvqe`Ej-NW2j>dF zyvA;Oz7X-s;MCEr@i=zFSh|H&;b2+YBNx2gref zHnI}qU6f@bttS^t6UuyO^*-S+Ol3)m92RrzO z>*rK=voXfYvL^(* z6*{bagc;G6DzOKFaUi5EfuuzZh>ie%rU69D-V(jMrUGg(P9=+?;ysQB-TJ&}>Oag5 zBRM(c!#ZX>^4T&JN$;0b%ff!2+ev2!Re*TP?4<%b|8m_k9!6xhP>8ZVLh)xBVD{bJ z-4A}|Gq7C00!~dpMtba%Zux)9XS2IRjXDH$ZhxUi`CneAKD47w82Yc#2VB6p$Nkv| zU+Km15<|Khshe7NOuHhC^fW<*4{8;+tSq#1h8Guu>g#0%g@xJZNP{^dtQ-IyM?X09 z7akX_;fW`=2DU(swM}(e&+%WTOJ5zgcgod0Q<2g7qNQ=@9z(f@{tMQzg=c2cCQU!F zPeKn3;}Q6~FrR-L8j5sh#P$0jma)6+W(mDUm% zr49q-v|^?o8kwhnEj;@9B~m{8@ZrO!;sFbl;Gm$OL+!5~9O#8&(cjgt|D?kr1&Nij zPtK11s(S;-HGKZ?lb3Oz#Mkjy#7{m_z)~O+8fDb+jJq|H0rs;nF(~W)F}?s)uQT&6 zmYZ6xuFnTT%7i3{Kz?#I4v1c33#!YCQoLW44!t+LU!;BRx?Ppfkw5XO)!&%au#I>h z4mWU}5#8VUgWylg>y`U-wO)kMR#u;?qrjPL*ppJL3g`~nW@bT2u`U{a8?YRz8ds^& z&~to;qJy7i|1}N*+Hj!K-<1`3U}hGmr8bZs}cg`Ib^8Q5Yj_ zn7J%}Yk*|#W!O1yT{Ys{h?hkV{ji@S<7(Cm^1qVLg8Tt1k~|t zI8nhdd}PseUj*5bfwnsV74MPG&Q6RQ$ZRtT2v7oG|Ei_sCCnqxF2+B8{4TE^iegND zfc9h*I`6M&7dfMn5%}c&+9-eyTmQI;wyV<_mH`&&+H7 zB<%aPWnaMJi)HlUI6gv~ac?C_q8gZGT;(gIfpj8W4*OfHvw$SPAp3 z+8gNnafy9cJiDf4^cVZ*na{&Xgx%5IH#gqx;uaMZ<+hq?>vBcLbJwqb;Vb+xZ51wT z%?8%q!?Uw~*0Wt_goUX=jT+3RKp?XLlq8Y7whmx3=P4wDS-!wj6sBxW`}fc6ZS$Hk z1M8hjf#85bpJVRc_%?k`Q!Rg=>l%p>hM6nmc|7$ATrtl+Z*fj6;6{c50fI1(2FvWj z`HU#|jl10r;}N}2PR0l4Ntn5xzVw7(baWJ`lOL@Pp2yc4LaiDWyAP zXn;W{Zkr9kTv6nX1EKE!SH{u#Ke)H`q&U0yN8AVAZETryMyz^;&}eGTEj>^=|73{% zhzLzZ5)*IYubK-^McXVpZEE=K*K8x`9fmhw3CaZ76TZ;MfF>UNgtm^tQ&rW~9R;j4 zAaYbeBpkbXUa16o0f^sNsB#>#cXa2#DE16X2t%>RTH!{fcCy(>Qx=e<4q?D?^Z&p# zYB)qR^Xea&04rDi%V>0)R!&GCH03F`ICo6n(eHsQOD;YipXY`#&?7Yeh(sL0DT=qN zmFF70P`Ml~J`$UqZ+VbsGtE}rrbjm{LxH-aP&7g&Mk{EJ!EoFE)=RJiimtIS2kKh` zx9L{Pl25PZ|D2eNj0B4T1kPKg|ODfB#zIA)wYdyD8F6VfdmS|W`@Y$W}Gbe-O zGTc-PE53qZJe(zopfWfHQLhG($}j1+oTgg@0ntI1ZIgb>RxzZ|4L=<&f0HhN#=rRk zzyF0<^+1kuO{%Npe~6A#k(t9Ni=eo;IHNF#K_YOhA?Pqh%XQZpbUNx@ojum)Qt9^i z%hSg5d|I?LKe9wXV2Am0pYp_%Pdeb{1kG2N(?wT#zAHOGv@|hoWP>Fy%$txj*vj;e<*>?~qh@8xUN73Qwah?C zs40y^^!~BcZ>y}6M~iV$@3a5MHa$vGaxqlBMM&m%&v1ZP7XZ_>jXd@U(Z^Rsu3kzCXY^Q92!dlva0-T~!A#4|YggWtaGC<%v*=DEcFW-EZqIOgW( zaWG&p?6|&Ahlu`gwhm_$4%Q?qCY(HVD&bKF_T%()Je_@|{^ljF^WoIF)RV)c7)p^a z5R(0$>WOc%>JBJzO^WV^K;aDvu14&$yllfCebRYl9#Gz3o6bhVsA7&HR|SLd!_ z_cxFS0(}#Sf&oL286qpYO8W39n0`T@P2_*ogM{H_v%+KV?ij2{%;6HEq8UoeBwGvA z|G{X-rbp#xX&063Es~N@2h{&6Yii&es*lnT4iEqS+jVwq><4Gi?a=C|Rx<^4F%{$6 z4HKuX$~L%{M=gzhFuzGPd?LixlP3CD5VOF?GNu#zy*Zed5yn&gPW8)Zh}kd@PP$2< zWO)B5)2EyNMXTZd*^6pY*t^i2AZJ&3VBLQI8qq}CZmlHNNJBr_&f&Kinfk{WuP07r0!R&BN;+@wQGaJO?rqt% zdgcW85j}k#`8Y`9i;Hn+?rPP^9qm($BXwgJ?hu3~uN7f!0>CsnFJ8j}k$r^WZaTiF zsd9ZtOa#N3nwFM@RLjVS2vo_4J8w0T(V^93%@5!*P~-fmS=K%ZAg`M@!+m-uG~QSq z;AM02`*5>L6=a~#@BIHl<-DDbkAF62;Smw4hTuWmp8<%Kp6ZnUxrSA*(jT1Vm? zSkm@C%@hC4?#h)zEwK{B7&{=7KK5hB;<6zq<S;9w_=9fEzi z*zd=skWqybH)t?Y&j@NFp3PTdv(*pM%#JU|004Od4+rDA-8TgLV!0ZR;r+*txZo8B zF0IDpF~(pZur}A@4#ELwbb`$5dp@pYx$dS##j&SmW)eYuC=G5gckkVMHd1pG-h%G8 z2)-xIQK4Ma$kn)6BXtrXsscTPdH?DZ-T~TG(dyWH^8CV2vHG|Y?nO(2Hhu3Sk$Q3U zfrqiicWPO4b8a9*WeK%*Ye*?55J8d@6ct5_3FFLQ!t$PQFw$NG)!>l>Li*3-c?OTm z+qa}RKUiFzlzc*kd`C_l?F1Wg4?8fncji9dHqd|OF5aVCu9Lm@pM5!2Rtb4sw-UjZ zYufg=#g?49jc=abJ$@u4_$v|be+&B^XklvrsN+T=@5XqRs7QOqu4>6n^P50U;OK+A zf(KN;sLKltEFf5$hW{7Mg4f%UeDdVFZvVx_bI%N!YQ$>4og}eV5&d{DM<35iVad?$ zzk!2?>FDax>B)WEUG$6@6hI)foTQ-e`bRtQOU-%`;r z3@>^?Q8C=omN`$stS(`+CGt}%HSN#xbfD+P?Rg=CfE~@5kCe-AT`zwO*vB)Tiz4%B z+xi!K@h?==uNbwqj`Uq0Is#Y@=z3Ox-Q$6<0FfH%~|flH>L zpzMMrH*ZcvP8{=TBxrerUb%!3!h91YR z89Hv1X19gHjZA5z@_pMdxJ>~n5CN>w2Hb;n5Y+>St5H-x(21Uf{^43;MNmkR#MO^C zj`(*7-BNTMe&^%L$Y|idgM+fJcD@|9F<$BbTmYL@gA>}bxo#d>F$XexdwYx<%n9fq zL96Fmhs?`wyeUi<3iHO67B7tqSv9rOPg-M4NPY&QDMq|J%uZhVoy+FYWqxh(y6wFY z_w1hT3*BLzNu4fdxRA9KFuP?J&5HSXtvTu_n{qad2}xUbyn0!okSGsu>ya)Qd#*ju z4$q)GPD$?bI$iVe30%=*frubkXQeB_DSUQ8ts@h2*E=FRp$b1KnX?4L@YEsPYR89+RRy0Uur+Z<5) z-zOmtAcsy>+C0esGxfS>zxRlEZ5%0QbVK4?mTTWMAezs`Z5-t%hQom~Boy7$Z;+rH_#a_Frzp%a3l_1GNMP{1Ja%{b4d-Th( zruK|&NzZs}&<7TnH&v9FBSrlALzt~~0q2ZA|5cLz;0W5^+g?n7O#uZa#{BoM>t(xZ zeHb@zi-ZRXAAg3vI>)W+&d2!mP3l2gmbsFD=$B~7`x-!@>Nqr7`JXtA*GE%^`|1BK zGuf1}$A*1HR}_J_wAd(F)9&OWn@KS@3lqn7?IN>eiLacYUl|yK$%R(>DAhLm95*x# zaFx8q+F!88l>WK3GAE{Z(4bG&%!aSw>+7rGy0#A5B#Xau;^Ud}$p6o^`RXj&aM2+6 zM&9s`<&)~l_*W|xZ7#Z4m5$Lz(^6+CS;q`|6n#$KYFV}YEtl@f5x*PX-Z}F9UGz9* zK;dEqNz9h~;#mJD)8w>dX}TFx`&gO!CGdHA6hjg$q|{}!+#bV`zR7~85k0bf$@biR zQ97y@K!kF0pa1r-K0Nu7U|2DEn#(2)(KTzg;KzM&@q`panm*O;^q~@hXipfbatw33 z_xAR7#YRU(-E1j!0#`6oWd()6o;5V+)l11W%zdR4?h#!lqS$A=RsPn!7Y8;jx6mWr zt>X1DmRs9i`9YlYF0w_L5{Ps_$`Kz}9HetZO zz$vmC*hqW2Ia=G$AQxzU?aaTb92vUFAji=)m(l&U_*iAwzDsAZdnsZ4O53#0u|TAI z!Lj_Gyv4^m_3r# z<&QXuZHwLK8g0+iB@wdX>EOp2M8V6k0PzW`qLPw=z4Ug2zwZt$+bLJjw;CC(MRy}& zX?um;5I(QPQVr1k9%(dQUa^>E(M1|$9rRc~#ED}pp>n{J2lh@=VIp%@&4zGyx%=+Y z=x6gI_U{&y9EFe#hJ16LSG^&2F0bNn+L{esuBzf^(N=9cMwrV-mBJKo0AIfO(oFcj zs@`1`Cj^_TT3fSLCu?f15YP&?O_jkf34PdP1tjl$7J`n!2SQTPiHJ)Zs+DS4yuu#>gA8gasOf@N@QZ5PRLb91rup?9+olZo+?2sQx~g;|8Vxyb$)T z?ID+K@Ww610x#N|u)qsHs5*CN4b%6W30u-+Bs*;$ygGX`f4SzzSZEcO`CBa0ha@;& ztx6kLK|ujEl@SsId`MwN+4v|B-h+YyF6L&$epFSxjZERtFpwCco-KM3-KIz7aT!Wjn5$(uzBs1|Yw| zr5IqIL%01fhTdgEo8?N`nX(Dv$ zzU<;au626j$inHhBUM+BRZYA?&{0}dY-yThR({c!2_u=%{ysfDebohAFu@1m(_kPx z!c1q{GZ3*y2r}UQUsX~flTrNaMM#6>$uL!btAk>d<%gLx?2Zo{<;e|PS1TXgn_!)5 z86u2Mz9jS$B(vAO`u)(M*1>YE{yh2&_!zdCd^=ZFRuIfSk)d_@^tx3+?WHm4rK(28 z?!IIMCUbZj8eQv+5C&eNKHyw*2mm*;)(krJd*m-7npWUyGFHcisI;{I|CnSgR31i& z!}s5Yzn7dR);G?~;=MJz0VGkyo<~l@c1Xsz-6bJTW!VpPo|%d}1*Vs#D~z3T7exVf zal$luz60rdi}Dwo1BnpqZFh8OR#A#j3^#qfXP|^lYgUZ_IOl|5Bb@vzmU{?WQKPDO>%mg%D$&vl36M9dYzuES!&U7ErW? zXj^K5nxd$c#do?;`2?aJii(LJfr$wA84U7Q)hfXvpvaCm6)Xbk#yx4Tk_1^56E2@p ztu6M8#S)50f)jOd8pvUuWct)mRaLbN>Ownk|NV9fn~WVj(C1+)=BBz(ccE>Hk9AVG zPf+*8;*rl=4NRZp&mYkmdIw7(v+C51UvZQy#`0-Z4S0Tc9_XqSoxz0n|2zfG81Au= zVpi56=Tk1Z(xOA?lWh!O47U|3{9657WP|@|k2rP?@NBPDRt@A48}dnJcr|2Pa4UGe)Pya|#9pugPY3XqU5|4MWiDSjqj`C#S5*=bEl8}=f#iPM{$MhEO zzh0m?G;$eBsTp?fdZg##CRi@ajeDJw-DdufmeAZzq8>Z+ec%0dUe}phfMs|<$g5!? zyHj;_X_|kHvhbOZYI0ZT!R0xaSf22+OB*3lrtNdokLs2+^n%2M7P6u z5Zy2L2~hNRcR)c7TpdP!(vImD575s&hfcsFyX>Uq;N?vg`|wMxEqcb)fK?cIr9_N$ z+WZPYr_Jv)_b^VL7^^xS{eNioH4lvb;-RgS0pdQk-MqN0+ZA-)=sX~fpeLP7uZU#9 znVsY$Sx&2e!RChQ(d;SJ-vOO0s%mIe78Jz3M@AF4JTh4JnXaAh7iHpT`CCx)`E&I^ zhGSdQLqx6uU#hMZg#{U_vA>Uu3)I@+s5kyl@GbS#4P^?igQyg@n4U1}dK3YuXxJY@ z7mH8(iYS?DR>oTyCtvFcB5fqfbDkqHQhRsO)QY66_{c+gHtZ&^q-QmGv|NjO6Qos> zCzTVF_gi9&YQAvAKQM3;vDj_r3*a0f7<^y{ZWySXn#d)AEC2@pMUI)bI*p=kiEet z2lr=`V0uj7e&WvpnhtUvBh|c>mq`-2wsVwW+~n#zp!84O)kR={Vu6XPSuY~xZXYE2 z@qs?EU5&}0KgnNsKlAy_*jR$(<+GGn=PqO|Vq^Vn>&e`OZg)6^&;pgYepyJMiUot5 z4`K$_s~X5M(#1=OgxK2On@h7?FKQ^Di3!{}=d`nEu(*Vd zisVxVgiytEuK}h8ceHe`>vpm+?*Q+R)V$mE>Nng_czanh;D%JK^`gZVD#LZ7+b$iG zW~28$`|A>~4I554&t=RBRh{)!Ca0$NA6=cXh$8#s791ooHtaJgw|_KF``+U%ltDxL;_r>&sU~Dd0Ur(RW0-cR zMvH)&)J@{(!0_SIMZLve$-puaM1r`W9Nt|&p%xM41rEsxmaGF^bbdNDB)!p#lgXAT z|K(8U-B#f4rE)U6xR@;R-jM9C_D(g6Az8Ap=(?F*S1I2L>XQWPLU)vON2-#zou-`J zchTP`c}H_VBo$^p8W3SPo9^k;Zwv0M{@(WOm` zolDcSf#b^1TWtSviVH;C;-u)2h#St#>pXQ(?5F;ZP<#_u}Z z>e(!_5-uTqC+!;@C%Sg9;sGIfMiRvEm!-RiH^l!ql68>-`-!6C^sEHMk1I)MfXGt7 zcApAMR^;XU&zYv9{av1KV=C+lZF35bFn!lI66EC6NUzC=e@o@hzOwq$t=BHl_$Iwt zmT$6UO=og1GtvhTZprNZvO+){($o)6Kwn{SxSdj*b5ljr76T`v4Yd`|K6VTF+* z{S6v(^ELi^dT2e$2zVWkPgL0QSWjK|yxy(+=|p~s#)1CW%1apl+CyFDURCD|VJr8} zfkIoE82}6v8L7vSPF#8PyN!u!^j7iO&rAh?T0}|it6qCpTR#D`PXEjpw{pE3axASV zPn>t0dCv8bwJNR8bg3 zO=(~rM;S-!`DR;ACqs|P``|TP$O3*S!=e!4hbF}JEZsesxyEnDlq)b1c+Be!UD&D* z=KViBr{kR`lu764M4L~OqZQKrZRDI7ipwXGC|3m^IGv8<70}inRJiQ-jdjw-a?T5B zMxz{Yj)LPM(x#TN`avDJs%FcL2Y&Z)H*`0&T3cj2ixb?JKNk%ynHSuHFyEwnXNCP! zVfW)Z?D*$7#C^?wmV%7V!R55EeEeX~uUX$3pk5mGPG`7t_o0qOOu!@eXV1sXA5J-+ zo)@6zXRD|~6RBKW;c_3shU3|Wa=5-}G=%lahnBoYp4*8FlU0}oVqwKIH030fAV6*c0z}dKyV~K%+`N+! zvp5&zT7|UvT;;!Cu5bR4M`89n6yv5T@k%3<8TV$Y@%h^Q}eW|#T zSSyMw@T6JT8h&JQtvUaX#M@DHV&8HZ&>M}v57h?x#S<8jm&Wx`gSg;{*;RxKD&{QU z4DYw(MU*w@MV{%`NqYomqsQ4}=S7e2Y+W#SU@n=7+!Od%fVb6(F^$FBAU?cd>(zMv ze^-jT-!Nya6fWFGOdLp11HusQ>qeD&p(C*X#tnY>*lH-j(QyO;=EeSuX(E2bJ40 z>porygnjy_03&UfdX&Ez?|UIN!%qR6b9LK}xHjjI47l{A_DBPM(*3{2Cj*(4u2`y~t(Vo~V9s_vgM8}r+v@&7Cv4+x zu=HD0Yah_LA`S@P@sWVY;br?zoI<(x<<|TwFEi0}FlXh#4?l+wp-D_BjK}b}=smQ) zM@ID_F6-v4E3l=VqB$?Js)Q*gP}G5NJ8 zekTQ+2E`{ggg)GRPRv^&9e3kU^Gd`!v8iMKF3%jSBKkGlqpL?Dv#2kKH%T@~1)U(3 z?7FFi1d>?>zcX6|3d4ExP`k#k30zj`Ifl-3xJuYyyT-576hHT90sRryY6 zmwY#5&(GNEPRdvIv$i)*qsZ2vl)TvWm5Ef6jQo!JgI6ri&mO_=Syyja401?O(_D?7ng{ofLEO6M^)y(SoV7-B@@wvj|z`glUL^ zv)8y%+c=e1T&dRPr_Papu?lgexGZAj%f;pT4?Gs41n>L|yrHV*Gw|^7A*nCi1SUIk z_F{XRKel=&UOVkHo-Gk$GpudB8+uVi8n1OlIs}8kNN$r1yWQ^T=?TbWh%LFq!czCb z@DW8tCHG-x+&n8Utuz}IBjp9Q^5t(dy$*Ipe!6%p1mFE}E?L#7t0qS#L<${^9}t#~ zjWc@xv(3~#GO<73q_;D;ROQ0Lp$pC*V*L&sYSPEM5vj>Pe)F@?mDREIy^^OlKQh%G zn?012Hx^l#KV0j*lB3A7*+?4r+rx0sr?ew$pX{3^ud>Mew^j?g)H5+B#hm76b9;^{ z;$SfEEg7reAH#B9+Eag+V`l>f%na%($Wv{U1w{BcMHN(6%CB77YHb^)cfSR1v2EPx z_k->Ab5+kd7h)qV`tmCWSO;vLO6(dukf@XM+)QC(v>T^zia(d@nxfpoEDp|M@_%p# z-uuZ7{Ww`=x7SNvvc78c*5q>Z>V%Sdff~9K-@r5HPhc=^E$$eQn|5O$>m+-+88hxM z7Af%dUd)d?Iw$plL~4YE1Ubbphy|x};zZxMZl9^u z^IJkgD~vrXXuS090R6XHWlEH>`zJg-gsrB`$KE}2;d`3$WPC?;zh^gI#nD&E^*axf zs28gztD!?BO$XN>3uUDni5k46DI_IjJFq(4Jx5w3IJ3^mS^UgjJiQ>Ca#{5T)kN59H_0~- zRgxSEp{FP~;C1}#>}AFaUvDepR41lpDw(3H|^fZZ{b~?>XQHRvX-1Yq`#f`D7w2#hiKs+`UbbB z>C20^Z(SV9dr+5O_bt#5|7F_;|LYO$u3ntiiY0AmX4wv7IPN^0jvedRoEDTsE)q1F zg>;UG1z75OS%}ttUfz#byeX(tdVwXe^kE?R&Gfy!8Vv>g46XaNv3o9$LVV|cX9`%6 zdN(92Hmv;o_J-m-f7~J!y;$OzY^m64f)%RfD_MX4!6ysARLpyFwpsRkCVkJr?p?0} zx}L7l?l>_o!+JNW2!}4EI?qd;U^H-rrTDu(*Y>0u3D2Go(Mh*><|{jvQRWJDI1Toq zrWxHU;V~OqSI7GZr{9)2`e+--s=aoJT0O^-r1|JpFg&^2myI+MxE7U{w!xpcZw{90 zks2X+H}V&HpETI`O?{kpPBMZkQipT5eP+YK2zvIh%xj!6Q9nPg#hl{YB!2Dn6}r2H z{%Z_)_;;L32et>5} zU`E8eWU~%-9)^MmZNL|@kDU{3!WLBmN@WvA5^n@uSG~FM&Q+r*JM~V}I>*n*(xHy$ zzK6dB=Jm}{_Zg)>AqbC4U*Yid^vF+g!eDxntSUdP#(q!Qf6GR+wcafaO!{Uv&rJ3% zl8@81_g1&McK8`|x0bYgt(P1~H2Ye_*z_RE`eTASr?fcu7v{TR8W5nOrYzHdQe!i0oo4|{6Z$B(xy=RIrRol)hha}T*9C7$#xjTj73S^QpSEGb%i__jS0 z)37dg$&aukbJsI==1YEMo3F!ArTkWDqW$3~v#)v9zfz3GviPA-D=CAP(UEjR)%?Be zdE#Cxy*I7%3PJbMPN*1JR~u@+wF;gfj?GGLOP1ALQ<>}ZEhE>kG!OV1J^pPljQMli zGqHxJhFeO6^ZcLcxbZO((@uOi&g-Q;?j#glQ zzP{7Le5$eWoAg~V7SG?CYb}~aTozPcbg|#?Uyp|0CODbR_s#tm6`-6TJ!~`H1N%7T7Y zQK|E-!E!`t{#2t}YEEAShd1fu+Emv9^X2Va33evoo_N?u5bV8Cu*VmKV`p%6y+F9^ zS$|7{h;?VAYRZR?G77alb4B!l^d+f7$|Vj}oij8;h2`UiFm2+GH_;j|s9+2!%yyDi zioZPS_CIL>bG)(yP6zEz^s(!ATpJ)BN#XtlEQHF!TX}mI9+7N z;A2?n?5x5XIMf1NldmmJravmY)0|@sh;uv*9&8DIo05>GOnR8k?WiWgb_DZ2TGaIE zdg=GjtKT!plir!=MqU2RIz2ai*FdJJA;2f3VmtS3rSjGrX9FwJ~h3`@G zsooH&{BrIij-51Z>!9F`*vXLTScZw243nd&w$U3@0r>f|Uo9P{z5IBsOdUionZ1`^ zI!&LkWAHp??wAU1z!MrVa&KtL&(k zeB9@Y7&I#(Wj?iQ^8mufhcd~nm8*-!G>`Ej7$`fXg%=&BcOpN0-qW{X;E#MO{(YNo zK8UTiT$x`#(c#0BYXiJzxRqR%bbN+NbTItWdu=jU9WHm%1$Lwlp zkfrian$X2B9O zUmo@P``#Fi=ACrro0BU{+0Qd4{hSwPmIH{Vq;21|&NKJh>>6(bSr0b$g}0+GYQTcd zKF0UIlqXUK?+q=C^$Y%zC7cm&Bqqnfw8@U+-HtHPb^SBORO+bTciR5VTfwSNzcHAl z@v69a2aEXuhosD^$u$>FY1`HPK+f`wpShmqTU79e1y7Z|z=fBdm|O9kt0BcgWpf+` z^HC&0a=nhm1p!$@Pwfi={QYh>7m!SFXUJeMPC^yq+@D&ZECH9 zLz8bL==e*YocCAYuQD!S^NHvf?bE#~pKh^Rs?5xnn{AJWw`;e%2oYg08F!yu+ggeJ z-c@(^seGQv*=v*ZktWPf-#;CP$BCNhwYccAXnIk2<$3CS0c%s-X9v*|7MJh}$DTt% zI`=N7lEO9ciH^XXN+#)@_9~z>`S~>}L?gOX122teYrdQ049N|@>wo%|;T3!t3toH7 z5NR@2`MRNVr(0b2gQ9@EdzMZCA^e|*t$rHyc;T%yR}?B4hv}Ko_T#Ug-ik>moKV=x z8-0U&|K3@+)JZ*jpa!@p)N&KdpYM%RE>8QzrXO-V)hR2bU+W%k(s|aX3(hzgx(~mz zSEl_2d-!iyL@-SHNbvN@{_c6AjAv!2uC6X_#>K&Lx$tCo{D%*pXGn+L8V95D^6h0TEC_K&2Z6=}@{6mG16P2^9fp zX_4-(MJWhKH%Nm>cQ>4QgU^1R_x--_jCYK4#`)u{vA6EE#rul;x@P|7Z%U1DErrfT zJH#3c%1GWlyphoQ{Om5aPHgYr-<&|xp-}gGX~+ zzAXSL%Q`YO|0r*uj1h(6wdd}ruE!|MC%k~?fF zip5fMxEYFCl8u#`^SOq0TVrVE2^k+r(lnE?q4fc2*EC6C)~FR{P0(Fe)Zgz*36w5J zA~t7KO!;K;?MerAf6a=-w$+1X-!=ETRGrc8RE{am5;X4muwA}{Oc;kpp&PG&Za`Gj z&@{SvQazXaANQnZkCsi-kVeqOA#2_HLa>Ivz{Z^yQ8?o?0BMN2QX zXRfMeWNJMhp-E5OTz)TIW+mz5s&#fZ-S7dn zfM%#uMlc6m!9(4IOL8~$%*!NMeZ+1g(A@uh&|PtM~_bL+a;%%u`1%)!7R(xG9?^vv+>caKR{2;@L*psXa3o8_hno~Alg zX+F$KVIot;#NNbv+&Fh;KYGegZ1Splhi%rcR@XaNx5bsW^n<0uZL@AQKL;aJcHIM^0i~vCsYGUD|T{b2t_DN13c9zJcPp( zZi7^iS657N1!IYTR4t3PG^5^vI)~J6vNk$yj#2f?&uuI3R7_+G8IZU6Wysfy6(-h^ z8Y$mE)2e^GYa*oWm-|INh#9Oo4;ZoHh!Z7hiB>Zosjh9JCsmL93sr2t|t)CBF(EA$aL3GCPsAj z#!%-HOO1P7vZflJsEn9x;_Gv$Nb>4W7Hkdz#G3o16Md{5LShuAA)Xo(9(mk^;O9QX z7Dy}Njk{aOm!=j0ihkIH30qy;yK^CwH@>R^KZ6Dlf&8bDjTrqn&(N0LTNK( zfoEZREuf_{sEH;#=z%CryE`ok74_6QXMLw}cvPr8bF$8(POLg><1CwuSA+s73T3=f zub5@{y$;u~^clsDlH|iilEfI_-IBA|N6{x%|MGi8=dow-Bse0H50N_2xjacEt|lEN zIE`_a`{B@td6F+2Jw4HS*3EtkEWbhhF(qJyOrAnnnP`&PY5JLN7YqaGX32qCC^M8O zFt1kG2MjG*V#88UhL=a-SKbBsfwW+b;V@R4&_&*b?J?167LJ#>cXTBb3?x#|Ujw7+ zYiz(5t zD&dGiRmuGfJOWlia(sfmSL8Ymw+SuMI?Jv$>(}9m-|V^HDU|hD;||;de&Xw%R-^%A{;Vp3w^Xk{BOUIq{BnG71E|~$Lsd)OJoH(F(ftLiMlc@V&k|T81KEl zAnHfU{P6p+?TE@hivbD`T^yNvGoxXH!7n%MRWFRS0BrF6Kg%TaALyTq)*U1gQ#l)D zJpMDZ+3%wF=2=#up?P(NGblIIQ@9qPJqKb()Jkb$5#g9@FL+1~Zs~lgVAFqn8~aqx;5Zj*2Gmd2a)rJ-KOvYq?)?qL5DuG7SvzUCf;k#h`*aL;i7m!TJUtlpc8((gL zH}*tSKo^w#`2L+3g|nzHd2rGVc^ws8`!+vX{0W%%DH^~QP^1+q@^gM#epVtUQ3S?A zEj$MI$nBQRm*4!=62uue#u6=&=^!7wxP05-A%RTTG^TkVS@OrN>*YK5e??b=mKUwg ze2GZqZ#%DgD>t@Y8B&&T@kNDk#2=IP$pC@fTg_FbI;_WZaV^H~3m?68iX5efPJWwB?#Gtite z?a=IL?mesP-?~t(?zzahw}*bdJ`}+7G?GeUD+ z6QytkuE5;JlNrJ!l9(HTDTr&V*1)d)KVO8 z8^IRyhkjox?BuncG#|AvJ-#hBU>cs<8b+ntiZ{$w(qNE~o4HcjGJ;l#!Qj{Z4gTt6 zqVRfCW~lWq9g!r7Ti0~1hm(ZItKK;fJ+DuwbZjo7I+6YOf=`=D-vKiz$TU+=dra*7 zb#J;ONSuX1JM!Vnhg=o>uwd%-pO>CEuY}($9SImx9-3#@eeBvD$t2Elx{{va_VAAj z7^r1NFt>&aJskG)l7c9714liZ?=ug*>sWZ{l9vCd^F$H2&HUqi8`mWoONff*d?EVm zvBU@qWmz)Y(8_30M=XqunK5Z;E?y=ES=rX`)cPBN-Ic0Go>K;P?I(diOzo_0c`>dK zu?Rx3b9{y_Y9j8;uROsY^w=m7(PEAGEkT!Sc)o%6w40P(<6_aq6s~KMakN;>0HaNx zb+5gxUfYgk#JvPW#NH}KT!3$n>ja3?EEHxst795Iwg293DwdE$u5OqtGthWGhC}wv za@i6&5m!=%w}=%3;(xqG?vD7Paq?Tn0iWlumH8$`mmN)lMe*RUx&@(tq9&YyC^xBo+ zcmb+w0V>dkzuAQ9IXQjB zz!CFyQa=+c$ufO{*y+jI@uGw1G4b0{ z+&lRgC@eJ(t}^j>BS-mEI#U-lze5J-4aY~tqXG|7_Upt_quzHb%lhV-M$DG8U4EL0Fv7?9~H}Oy8+6Osi zU!p!3dK2sPZ^vg_$NJQV1+svYYnd^>6bs^iG&L*4reZA0m|`h#vaYW>{=1@s?!eD7 z_dYlmk818?XAG@KA860v_os2z*JYK^4dU&(5)LJ;Tk3q)yan&w%?)|)ZmiT9^u}1m zPHNjZgPx@|E9GAu?FES8Ajj8TKHgiK>5HH}))W)axs!F)OLL&h%a%mj-zNs2L%Asz zE~=-TA=~%kwcLw#Hc0PGTEpd!w@_CV#rd8QJuYT+(?K&&}>>NL7e$31CxRyEJGniZZn2Lm`L$mJY+ zUwu5#$XJ$dbKy#^+(4BP@%bEFrHHk_Kiq(?RLJp}o_7dGEkPkF;za zvwc-{F4ymSC->zRqFBGtG|V>SZ+=tBXi`p*JQE>B+%MuQ*7FB@X_dM<(6iBGJg^pY z)l|BF#dcA?wtxOnH7rXYAukIwpm|S1Qz4us6>SzO2bTq&-QF?D)Vw~3({&6j^!SZ? z8BS_nyh;#i@C?ajp-zz?N|rLXH`H*!1>QyVTe5};4oN+VhsuFW$eP|X`Dc~K`m`QW zb%(I>WvY5ITlj)i_bVE_bd=kSK@^L_&cV42qs4*L>=aQmiiYuaNxX6v-Ts#!#7JKp zN8wR#5QC0CRQ{P}`~A+^b7#mRLEKyoi${(f*s3k!ynbMl++t@pOkfNZ2T8ier{@w$ zFUYxP0U;~rIws_*C71A+&>#I_bl7bqwE8(2BBe}E)pg%0ie970_o&?Am4?CJ!7d#S zCZnvGQgE&ZxogJveo-gY_5I%d@lz0-u&~5eQtj{l;2}cp`}j`={kaz~3QJA1-&~I{ zr>igX@3Sv9|VJ7*BFjb(s{ zlSf&(6*7JHeTXz6UXv#4pN4+4)cJTXa37?|pirEj)1-D+>0z9bB{semg4;OZlTpTT zBbXrGP*&$-CeqvA_-q-br8P}$K9HJe(JGpayl66IO92D8T|zF?bicSTS$M3SB3-6iVrdUp@}S_^Zu{e$vicgzbf>yOImcdhiXZaZV4rim^EleIA| z9jKWGxsdy1%&5@JFcdA$+vnVo-VdDjA8KMe)()WpZWP~)htTqj7298dYckGxZmLubi=AKL_VF=Cn>yExGRD=R;?{FOd&nT__zB{~LmB73Jo z9}O4h>#+Sg69oyo!kgs|Ic9OQMh2@~1*HQSWHnK7OKOGuS}Wy$UB}QLr3x5}(yqIK z$W)=#>Fiy;o%hhLeSNi&(GYmZcQ>+JGrUM0e$HOO;D=ymzj}o#%OX**XDBV`l?P)F zjCnN*@>J&5dJU0N`E%flimkEtc^Up@j z&TqCUSu%-`6(Ku%+|dP(U28MjhVKqs0||AH{B5z6(as*9kquwl zeR*lIfl$yb$($Yg+r`W?`^4pW#y^twl5{{cxetu@xEAteB-tDacvwJ>DR&~zO!;&Z zlC4iy_9Syp{G7`ezHV$fO`q=+dAPglA#Y*18g~Pj1q|1x+X>{(exXB*Dab+4axo4} zZ8}Johb$>)PB@0p(ea4m`Z+2=fK{@#yVzjeG|Zn_(JpnGK;9Q7`}mWii(CqXzTu5ZSO*p zp;Rb6)`O;R0r5~VtceSK-}L>U_hoDysiOpxXLmx;!PP5Qz5>I70Tjxi$9=i3l9CR8 z_j}yX*x1Q;xPq&0H|83VE-h z^a2;23?N7hE9CvWSf*gTr!c#m6W3o!~EsFR`A2je*0&OcFO^WFS2o)cw{lJY~<(Li5zE9kk#<^+U%($G0K zX{dF=Vy1yEOAZRKp!jv8>LR0FN4)5&9k)vEtM?`)69mpcfQTORBwhyw8@9g6{Cqj+ z-S9lk;O4_u?^;p9&d%kTC6h#mTm|a5?rg1U6>DzPa;RdZOLtbC_j&JjcV2vE(7Qrg zL>bGUd}C@Lgee>RM`RJbNm6a*SP91Qn})fD{QG-~4lAP$-TjnEq{gh&TD6H^-L~(P z2UVo@i5O`U!(^ATsHX}|hU$)XV_-luP1{s2v&$(o9bphvDKrWb@*|}e%^g^P4v!(+ zq<)!|hYkn87S;{lnazLy5Zcq#(P_Bq0Ch zjqtA#CA@dOV*nnq&kTzvvi;i-aI*5tpW=Z`+tJOu%nrUf4Ad?Du9CU+hI6iMd5#*2 zVa=bU!mKDd#@i#?qQjYH@M_|H@=rtuy8z4{_P=A&nUGQAo+uHYi<5xe_^UkK)*k1r zp(M~oGeBgfcV$!$w`d>(8dPZ<9d1=_!2H$O2F>1%_~M}MS~^5>$~X%`7G?H%_oHhH z*Jht5Y*|Pt`_A%gEL8YSC2}Wey>GgOiNbmaVYp!<{UsIsR+W@_H&;J0yzvnrbZrv_ z%clHbW@Np~SJ(P!CV25rtJ@@`^Ifb~PK(`SZw?1I3|Q1(W<%N^ z)FA<*``7}+HszUhe20M*D{^}`nQHWVNhqdh#>+S)Lu8N{x&iBGHEzZJs<=J(FdMVi zC6fO6;_-nC7ca`Isigz)M$ulyQppr)`Fe*q5syQ@!}@eAp{7$#`zFxVCIcJaLIoM> zb?b~xYBqXVkUwrUUYXhV46}h6i$Z7$0P;+|h?(Ls67EEEb%Mm}NH ztM_j1ky`ltd1ttH2pZjD;8I!E8Df`==9W0#BGWdJi)+K1`K=^fx0B zA2l_#UU6hdNY8w6V9~X!SGTFYENmpPUt?B%j z?!YOPCeF_Vv3?8Pl1sg$J!9e3!>-aLYBGwqzX{hMfYgn&2wxStnnZ zv(J1quo8KesZIA2rK)7k?N#2VcEdtX{us-NbjKDpfi{X^nn0YB5%B#R?TkcTC zQQB;MbAd}hN+T(b#7^pToALtPu8g$KNL;sk2?#?AG)vZ&oKL-jNFf&QaXc7FAuhj# z+EmX*5Z=wcT=oO?d-=jPHieqLr@vUH+4ja@X?KzK_=i*6W#}bshiNZdlFt8eC&EE+ z{i(&P+yIhcKq<+@&E;t2IJ~N(P*KlDH*Z<$Hza(xG!ah>XTYMHrp(Mx;5kM5w70_2 z>^$#Nn6ey9c_W2TM9cp!E0L!GQ*EPv>E=gqXy?hOyPC~33=xTzd6Ak&Jb})B8hpE9 zgS@;}L6a6UN7j}6q0)zW^AHs~&Z3}503k+c&sRPYFwSHJ%uJu3UAkAb)k{QL9rwHc zH$f;XO(y+f8xbhSV4^68-*k86Y1(PT-itkhNOwS7z%?CQt_5?G3*{;57`_p$rOGs- z>*V`yr6Xm%|Xb@d_SY(SJ&2@7f;_Zv&Z)P zVDN!Rr-7ZcgWdkDPY1f=QU*B#cP4p||CK0<+wpkg<@668K8v-nj9xTmjUXl6%AU-O zOp6k>Ep(_P=kc5#Gsk6X+5`_@KayoQjAb)(qb1*5Pk3%?fuo7IXPRx8c$nQ@IjHd6 ze*;`NcU;kry`(=aq2HxOvj8!3v9l$*V z>NkS{7o+N>UC=>uql4CjctdC&xUxU)Gk1=+`PJ?#i0W$o)p*ZupStOC-){MN--3-j zH|YYK#^9%v1a<lJAuZk0PJO?)h#iK1~!p~in& z<%738<}+z)tfYgv7}SOAKH4_L3LKU&@^dGRH5rdosJkA_X>J6@>|6wgenAA#S0+@lOC{G-7^es37?dUkk$2Wux{Fv1; zt>kU=a8|TdaK3*Gd`ncvcvp~;`UgQrEQ^(UavSwQ>${`&UTsaoo!1i$3{4@O<3_(o zK@cm5g=Ee-QvS{tqXCpL*Z=8r9`}&p)z{K_1(v6vH%o$eI$ovKF~2`sFx}Lk%K=& zS@w!Uq z`}jZj&z&!vVz!c6SH3H+IRQxbR>xuOXTDB-yUwz$!H8h=4b5nM@&U`|sbHOazrk?; zHMW4&bT}OvIO%wQ&(oeC@9`&A?DCEJ9u8SF(g(!rcP8osKkgkKqFZ(xJW?c?yn~mU*hw4M zcPxVQQUwf;<1ZimiK6^|<+Yy?s!SWWq0Kx)XITV`x(IJ-7_ zEB-W@XoecW6LWo;$`zRr_~#xC=SqGl#IT>Lt7iZ!h(5SN{b=j7LZiN~2YXxM0gIJK zE^gRaC-3 zN)GH7Uv~PCdTA!h&^1>3iRz!#2Z8dbjR{<6pSFoWGFcI$)RW9B|I{P5@vBhm{S;{A z0|&@})}9Bt_u}+UfuZ-rn}9_P7YC<%b)u2QW#4Y%&mY8D03#rvZv2_Cv2k^dro&9a zC0A_zFHG116B`r5{GNVpwNTfISgoEPFQt9Q`>l7iL9f_pK26t3`iZ-hXX`j5jh_Th zL}3}tH-{K}P%Y9_$`7*7u?_KY!H*ZT41EH0Upw35e%8=nyU6}%a8ONKTbt(YUEu&* z`(Mxp9r3p!*i3uG8O!5Y411`**IkhU#@yNVIIq@m&YPFIySnmA7%=@B&==_Y%_=p1 zs%8*Hgzs1Q*p^yb>V+rzmT2m~Wh^|g(x(ARwEav+ClR<#5}`4?98dwFp(sBoJNrJ) zjfa9G2aC3<`|D!5f&a#azeWz2M0q86`yULvt%@hJ!A{#tvK&YJ9LqEMV7Z7rR|4#&MJ6lt;}aj&s-T&&{5FPq|J$*}rkAob z(<`n0^Nw!E)9x+OH1v9o#U{7(1&+<|LHe1t*h^`BWYa8&&7lS)K%BslOk6S99MPLv zIiSuEnpw6g25*U)g+-pRZ28R{$#6=)x;QHFkm(RrlXjpDO(ixdegyQ7Hh~9zUOY}I z85#1gUYRjodmqwWI#=*#oNvxlvoifkgfQoXv>}4G9>BL+`{5;jt$pOLqY!xeoB4UjTY5xLJi=>mozq!<2})6M^s@|Y^mdRK;f)n;0@k2_OgVb)hvR(&ZaRh%qt zH{bged$^0kP1LC&Bz*Mycde`H5cF=#=(k<@UDIFTTuP}A)MSXoPg!qLWC&h3IJ4-( z-MI~5;8cX~`ngw)vNX!`!S;@+YiVumb;CH_vOabT%MRObxUWJ)+GR6;)d%3!K9{T8 zuoc~1xOWxBc&CW!b!4xRT}6VhZbJX0^ZZ%`KmC=Kh82|snf;y<5*N}g5Yro5>{~<) zY~9V-|89~;NAoa6izYDWbFFv;!35fz8o8s?fVH z205vO8g{XwZb(ZEl>&ocAb^Sx(((cJ;^Z%1_`%sUu{6fw2B2&;&r$aq$)mKa!E3yA z5~-n!>|1wdxn_j}hDpFwtf0|4mYQbtJKwxt`Mo~zB$jqdM8d1-{JOnfS_C4#XjX_ ze}eQ%X|nDejR2!p<*IAD&cai^rP3FC?w{mMu(-=EUa%tXZkfH4r*ku%Gx>mO&l+x{ zZx@Mw=lFc~9sq2Ub>*Z@sVVLhh>B29aX!fz%K>W$@}MtFLglhI1fNqZ;n$lLnT)tM z;lK9Ds}=Ph;>?wS{04A9CO|}iQh&NF7Mz`mjO^^EH8nM}pg*e%fm2bjXxJg`aOvn! z8tB5%e*(nUp+)2@&{Oq->b3fQhR|yMt=q(A=zM#i;ZpmD52j8gOPWg^WnC-iSHG9E z$890(4d4PIj7zsfig`?~bzfx(V;TQ5r){lGq9WiVSk|h{rkCr65cA_V0v>@v_%GF5?M{&Rm63sOj6F0f-oq5)xRr7XKKZIdjJ6)XjkP zE?H?B3JQT%o~=m4f&dP2afs%8nQV<&I{MnKe^s8OeWd3=MjY4mcBo&lxU4$ELFKj< z`u8j!9VPG3{wc(wxb+RGN%}Wp@)MKXL~6(P*|&h-qJBszlkzO(!zG_W^u5Yt3;7w5 z1w4{Wj`Zq+x(Z))UKvM0FsgnI^-E_LK1!yS6ZNH};tB}988&~=>z^L~Gv2e&Ki0p( zc2h*iBbMTtFlRcil?!lWDulW*ywP}jV>)lLZ?@rTbfqYsJ@GY{=W&neR#ggOv@ZJR zP1zsIhhf6TFlIT<3L};Bc&S6q{J7&xWKRJQGc&9Ce)wQuGgv<$H9S?^*7meHhOcty zHBLmc@P%()EiSybf614oZqyc@ao8{`G%yN~+9|9l9YS+ZK9ObI&3Nn(nL)u4U?6qG zGrn^U0WSV`;OA>Mh(=b>?unU_*opUf^!bFK&gXg2&{2siZdS15l z^Cs-vq|Uu08HPfGbi)1D4L@MQTXeexkR4uaU0pV`3-|Vk3~T<+;I$3Z8g-vZD?mRl z_y^9Yv-t6*r&e;6a_rmSW=3O+;@xcS45zp_`l^L7F&}*weSCQ_-^;>Ddxy8#+jRl+ zc$Ueko|g%GW}0*l-J_KEYcgozAr0sl_KZa&c*N3-Y#rL%IQTrL4Y>fJs$JhS^P~+N0362jvbgs_-*Z zq#zw0+#S3V$ufQ%|09?cX~w%@u!AqKfOmsvhUv5UK4MpSc)8pDIsk8$;io*~g!6oq z_RCgBneVzXhMG&1msVTAaeou^K;gr4<*TS^3F!TCuXKn33G%M(=k<6TAX;56jqtmN1>wYfH@9&X!%9c4RX8lNrl*U-*L;~_&PA%c)`wl*7wUsq75CaPM8bIX1B!#2*L-=>eYO>hMtOc4F3q| zTsp-YfeY4W;q{ASalx8p5phojRZs0CqS<+hUDK~yRAb4 zq0sOsCVl7qV1Gpu(+XsW47%LT;Gptcb~|6hHq}h+8_r_jGk{cUY5`a1> zO}}&7kN;a)+YXeq@mf#f-;vLxEzetuvLT7f&&(gP{$*@KB}7Tv=bsMq2N%CA)K6X7 zhLVnx8NJiTM4J#7OhSB1pFa8)V4IIo>z;1&Oo~xe7mH}QP2+B_veN$dU|UPqvxik* zrHr_picf#2F~{gL#A07p9;KzB@gG<)DPCN$qZ9>mo0gW=pm=D^jQY;xj{pVB$)u)k(3DpYvIeo-PnDlrs*r zYS8y)FH&Xe3$WfPT9g&(xG!#SEV42I7;&lvG>wHrB#8^(gV+a>)i>(i+yJwBEJ?D_Qe|_RjlSg_zjOxE`#Yxz;#DS&TQfGVLhz(Z#I(h}z}b zRawLFy^Dx+LMnLw6D8v&9kt^wT0uX;SSJc`kz{My)F%r7nzhaE-Iw^Mo1> z`1?y0`tuY(`)=%Q-n_HX7=W08A496%8KU^V)YsSR_Z~toV#HI}fUX_8Nw=&xBQ`U7 z-r$mT5w<*x8K~X++hdz%fMFkS`uOe>Rzgku zP7WZbQTTS;!=Gx03x<_hE%<^4tuqGL?j&e=;%qbrya?dRmxGLGRMsWe%(^_+6d#rf z-3KJkraH=+W{CVZ+^#Oh;5Hybp!)sbh)IFto*2y{52cc(YYzaxF%uvj4=NKS=duhY zmuaD#b-eYShmcs*8kMh#Ug)Rfj9i7OJXTrMd$Etj>W@=@rC=c^|O ze07pRbn8Tf53^xgy3HAO9o+DRIs*tzh03UAwtZ#Pwp0)mBBLRfwfM2Kw7nq@;gQX< zDRqL0pP?BeY#VB(N&w+V+47M2T1QQ{p9gbb{F2k+k@6rLsE88OaU&TR%<$_7(Gg8~m-L2HT?yTcBa&w)1Kv zpm(XF%Zx+;b!e4n^<9m|_)l1?#n_67M~q`Nk>haSB`3-f)e<>1wf>xMwRWKL+u!Tj zjDX4b1xa~Nwj{03e(z4rz0pC#DDVO*8{!wyzu)X#4RDI4VnG}$lnQLW(*<|wvL<3? z=`#~&`sFLqy7#a@5+~cgodj=JfJ8}olzi-8(|675R-|=l9I{jT#Nel4!=V=BY#XRQ+-Q|iPezJ{Fa)S*oQOSRvJ+3nPuuM)H<)09B07-@Bl|_ z=hpGI!5m!b!}*z>tM+HN537*Udhb*;0>K|HZS#sOorNOu_^SMAj zG>|%eBexz*TqJ$d>zE$N(;E1=YGCy#oTJN-HA|~-kd!uYktm4{zcaAl8Jjk1jCTIR zo$#?(Id_0Q87WIZwJBh-kILS1{ynjz@7nucDnq0sI7soX#HNJFvKY4A!;!q%zsx?W zs=I%9d;9prc-nw9H&Gg|;K}BDkt?&0# zxD`0(uB5;XS+YwJuEcNr3OANvX^vu`;MQJfzmhW)*jdg+hW9$A3Q5_Ce0axl;x)SQ z;f?Y?eShAs{mbIpbOw+#m~W+{=RS}3&&%}x1TS#?nT&{1-5?=k+Ld-D&aKOBGOpXK zT)M%V8>Pz;h0fbM6MlajukZKgM}XP`2pa~1aYKE(ACI<3K`W)w_&2P^FNp8k?z>AI z-(Y}c`JX@J0*4QamWrijch7TVy8F$B#Yy7ZrerRfRm- zOG~7MC?#=Q@OoVP#7oWYn&AOrhENgDfujwzRZIOJI5JxmhkWqlM?lfvIJE#7{z-xt zuZa|W4i9RW91@;;q#6kYjtZp@#Bm5kCdRnNJeCno`AJXn?MU;4#o&J{8(?TFsZ0kl zjE^tiUs)5|US;*2^9PI+WiUIHKdZ(023wWuRTZJua!5Y;es$aY?o)NHb=?o#hN-XC za-zX#1|dLwTtdQMw_2PK-l5?6Kq{OjuA*bMVSw>=?JWEV7WFaF#elEJHJ~ZImz?+# zvYxiK^qu%I++T0nS_dw$R<{LjbSz9A@NE!Td%wcX`66bCIle_0h5O$m+OfqS+Ha9U zS-0mY4S9ufd*R4#d5mOwMZNaWT#hu#I9$GnY4&xMx`)^)nfpO)Pg49)wKQ5E`yYyW zc6YOB=ZX88$jI5! z+kG(;w$S_!a#H9Q*Xmzys4O3v7L*?*LqUUqI@)s72&Zg2ROWLWMFaSAu%VBUJIJ5G zVR-!lSDEB9t|O|t=%a?oOuif%O0|N6%%>ZlmKwsiBlU$Mz7xq#UryRLy0e=1-uH1< zflr>$CR(P?u<=zEoqtB5+$Q>ApWlID8hvniVfIuG;Fz^kvv@@EGyFrv5A!|kxW^j2t^V>TKmUIS?eZb@te@(p)E2$OkipH$ zeec(RrAur1C2l3}JXl8ogBy1r6V#o)UO;u1fo)KMR#koGy_;ug%kk=jqOsJOFFdEb zdGWd3+v{}#M4oXG++*f4BA-VLQ%EhGOHbYG7RcPv1R&Cv&g9g4UFBJtaYk`fj@2~e zV-69LjZt0&PHRg+1rv-8+u^t|7`0HJy4G2Xvp{8sr?LT}SyY0wS!%b&3Y zRBRf<%JuKs*D;oQu*tdkR>6Eb!W^6LmyHDo#d-(SVtn+RyQ`i_AK1k){8{gZPUy`c zBCiKT9t`Lmte2D`V-H>*tr_KsGQ}9HKGmDddwsMG_kzN+F3wT2a`Iyp6s&gDeK39i zND-oy-C=h=>eY1d137YJFK@Mhy1r&_im%5KX*Q)XJhc2{_wJp zZsf!n2&FY{GKNMfu_pLbVGE2{d(MP-yCj$hR0@nDhzwL47MMJV($R)Thh?JJ+hlwZ z_r)C@7l!r{#`2t+JfHQme;iBQ5HE2dB(mSVai1uJUX=X-17*`}!)aN;blaYg{fyz^ zfgA~o-l$)sv!6U$f`pobJ6fiE0OHwCek^?Jl6nd{#8nTo)c2CO)E&6T_dbwW#2GBa z(sEn?Vq1bgAXFqR;$-Rq=6wf+@{J>Q&#YfBN9W&Tv6vkH#$lGMWP1s=Du zOQJg1()w+&U0By*pJMqR4u)LbE4JgHun1gLA1mH%EnyTph_JkkGN z4UEOT5v-sHIzb1|e0V}AJPKpHD?#iEr7#-=_0bma2CY7&svJoysLIREhmIq!6!iul z-f8sm^)xy}ho|qzG%N>JT^}m{juWyBYx+j>mE`;(htwRwJYM;p!@PfXT4tKWli?3kjqc6;)JcEMRg}PB){EV?s4NDQi13@s- zvlw7h%#3vnLh8Z${RJPz-0PmI%pycB$aJt+!S=JeY2MLh7~iUCviHEVDd@_>eMMO} zrki!cm_EM)`SrY#7dzU*_Km)d?8ft3NKge!I>{N-g%_Yyz8yae+4Wbf zrRunvYiLiq$1Qr5$9z%Ez!$}VbNmkUZZQ3em&CYkcv=jpj0z{%H}lA!C(!TZN+RPj z3$N1QO^e3MA<6PW)AHSXdM3XL8-9d3j>E1uI3*4>&xw( z<9&-Hg`-jUPHCo;Q5>rGq@x3E?gb87-!1gwjE`kXcxIW9zqa#(w)9rA)h?B~ z-%fXYLU{0+=7H{(x5@&I%z~>*OIAbOwrGK7%hijrQkmc@%r_-y0YfPE7J%12is{9!6fg zAqCCsfNgR26U_XUgM%m8|2=R`l&7kWJnUZ|ennxZ0QmRk+Y6-R;NPEn$BatGcr>s0XyKV6C^AGBLoL1H?`4|+~t+I=hoj=#Y#R@VP*4{NW` zvQyNB9wQl{tL4QhisK(u$6zx1*WW>YS84)iiJB{ejN zApnSQUS~A;2e0z_tU4vxl+_8|v>hJoSy$t*P0#1GDyXSNzIgG1`PI+M$Pd#pG8Un^ zG2a75yU&O(hj_N@>nc2R)Bp{-+Y614#_jjW| zXPlwkEYOQmKi=`MRlYe~YX)bUl>T`dy3>e%=|RW+!SUG zQW3?QkeCR&B(1NnpL(4AkME`;7g0C+ex?BlfeaWI4JAH$dv*q}Qtd#dze91_g$clS zLuS=D3ZbuWu{f;j`H=FU+l@*6grPTpqKwRyR_)~ozWp&=PP^qVAq?uNP`x4+;IbY^ z1|d?>iv;)D21_hi0M!Q2ty>&{y@&fzP@}xUeVWOP;grs}6DH@CDT?Q3Tb_IyVJB?o z4ta_(Dk~`H-qFzk?zy^>kdP;3N`8J<|KrP6P!A&~EBn&=#`NuB=PmV#iHRAp(vhMoSS1j~xDMfjUU zq@P=*Im4;L+LCvZc%yJ@FBB9MKw!HAY7y3ZR892{_jllVT?wHY2G;smxBYlFKY#uD z_3E{232*}E>hH$9@)5F11{5#E`zv;Iyy+g@Psz$sgk~j1tg3MXadB}}qA^aZ@2_ct z5W~C)E4&9|_OX*;xQ?q~J_qU%{>xkr>&XNRYRL#NV=B_p7dmnRCv5Ecx)oBO*3-(I=apX(~6X}Rz6K=?Y{Ia74KbtryCI7&>~Z6YN`sil%%8&dHrW5U0)6liqY-3Ug$IQJYA!_UkPoMvXU2b z(8Rt}w)PPKDxbHh+xm9qI2PDL8?`?Yv$bUR8Px)QyZw7N#nbJ)*2CT2N}_;|cVnD7 z+#B%pV7>tDcwodDVGg(V`clPSCN#-){p*pe~a-7qL)@RRP=FmB|z;uMSIf_d$V7-NF zX}WO0B(GQolhbuw#lZm$S2Z3f)XJ$N?aXr7pi_5TC~yE&Mzh29=|B4ikpxiH0!fij zggZnrYw>3X9{+xyt~UWaA_1UQVb_;|7$cddoK>p|lU}X(DMW9=ne_u6GBH`hEu3*Z z+Hwv4^yv%0Z1Tu1abjrfot~DklX~;Nt|4oW>s|NxD=3mFfY}FSy{ zj(!}DvoM~e!4=jx@$lv)FY|+q-n}?M%-6otc~M-pu%#pWlWd37c#pnmI&XF>YTgL= zxa2g&b$D{<6ciP8omT7b*4FwE`{d?6gnmj5*#ir;*1hVzcvNwLdA|$zT<@avF8#g1 z$oakj=NmaH$i4FYjtys=7-}E92mF#oDD|QCi$wj>8ztfO@WzRN&KWsTQPENZf#vCU zyaVNqMU*MJ&_j+9`%#UibD`PTW6=(189X0Tb=hja-4I$S60sNrK^azDaWQ-O-eM_S z@o-M-R87Mt&qoXR+*;hHD3ODS=vNRZ!r!nXz1=ET(3LMC6otO;Ro!j|$XZpU^KaC} z%WbCH$Yb{Q7ff9P2Nrsu6ysNH0l8g%moDHuBx`$ndS+w2NM!(Qmnad|F}6X@ozb9lFZ0y+OmKN-JnmE{%WgnjlnmcM`H zo88a|Ypenx>KiA7zHmalaENBy@Dhl&CR?b^r`*y>#S2DuCH!kR5drXRRu^sPq93LQ75E#^|~u4)txHuv%-}^+pJSRBb?UNe@ z!cCA^x!m(2$`DxlucxG>cv6lBh%>Bk+@YX|(=tw#twhj`mKTM-ajIt>?2R0;CQ8Sc zKnvL$@QQ^X?~C-{9%rlE@p-18p!28(OE76J5Awe)*V#-W{Uald+&w*sfN{Kbdu7}Q z%28CF(w~wybg?D!)PiPljh|jQf!x2N;Pu=Bt&x_N#*|sQ$Qft5G%N%c5BckAt_K^n z(96*H5wa4Ola;*vKdfY+54-yQKFI})wi3u^I`0jP^tc|bk%6#tUT+sB;&lRTPL6fL z*zh;a;u@C(W(4=$|1{FTWE+`I*~~$V@3kTOsiB9N+fqnrq}r|I;aeZD>Va zxn0A!{_CZmXaK46`ZK7FQtu&f|MKL7M)f46vF{Uo>O}7UV|^kPFD!1KqHFdP>hm~a zBgFaf(u(eO#vI1S9PLjLD7ruYpFRXg9NW$eLM;@|`yxa~AsPOAnONUH{Ig(iOt*=7 z&U|4S<2e5x);t6dV25@H0X+3HYCnEwI?gY^PT+sKg@@qoV7<7T#c~WQ2YHlNd>fPQ zgqrLmNj_@2e)X>``6rA9uK=df{lLf*tVBYLZ%3O;BUv9mVv7QP4s0M9?{UyUCPT0q zogmL!0{^ppHGb^ozEiywTm;yT`2qH6URAklKx6K-xq%##sUJ5PtvueU+Fvtvyb7 zu7*(4sPIl`2ea zP=0_Jet_@&-uFG{JWrnIoJ%*B77bXdFec_A3gzA^-<6ayuEup~Q2`D;y-n{0U0XVu z-_Ur&0~bgM3O^k%DU}e?vBJV9`&=y7ty4&)QlD6XK)_=F#arg|b=-gxM~)oX?Bulb z4=exNtG(pcv~NsEcOYk^#8)FbYBGC|-Yp+Yxlc(B5UDLX0)-WcW9%YZl#j0=k0X^O zV2Z5ntLVc~S&v^Q180apkbif$bugt17>QhwmK=^;Cfghm92}3@lKeH$xpGN;>_6SS zFma~|E`g&1u!>OT`}yY>jNkNYY@0swR8x`bQ*22KJlJp|W7WArFjSS&L*P4JD+;?B zR|{EZZ0GS39z8TtWsy1vy zy56*<_75R?1@7vH03|possR<&y6tJ6p;6hFcw?D2<||Bc-gy<47~YoCg!&C3*BP%t z245~b#b`5rk`x}IQP-h_!Mv@sP_uZP_~o4>62h~5vd7noG}3G}s%K&5Pw52!F+2uX zg3L*D#29U=c#_-K-+wJjH8+<->f*o;$B*EBHpO%t+hp?>GSHwH4}z3~o)!l9M(iBJ zsvma~6oG%i*2N_p(`Rek=U;5AoCy&MFT!Qw1#L|b&=Ako%`Fo4lOX7@N5->^y2M7l zZuIa<_mjNAtl^?Fu9+!nn?pFwtVF_(?KHOw`?|VmNn~2hBlC3o{ESs#_)U%5x}T_U z@K}`%S3Ik>np->B?FH%_9n-oUnDFMkm+rog0kp|$GQI|w@;|-Rn^i(L<1>J}zlovaFr99ChJz>66hyi? z%P;gQT;Gfqwa4!3^I~U5AY0r~J||BO2&LEhIrL#BIzR1^rwDo%st@j7!N9arBW~F& zhI+lY-)d-okNLiFHGi4mTF|g18{600;fv9fuqB`T6;A z#mfZULgDfeKF=2@h>O0NRIt*jtoo>ALl+Ym7%CQv)w4Sm7Y%vkBNbm52Q!C~r|iKzx z&u#LDZClNUmAeo~ClVDHIbTdlbt+bHLw2G+!NkOb5@FI`y5*z$QMt(yiJj>`oX>dt zIe4XSizu)@+l%91&0z4f9TeD$v$H#j);bFb>cLj6jO*e=rgvBZW==ReSzCJxC-0S< zxd6W)Ij4-B+*Bk~FWt4|2z5&9``Te?A}`;Rg4b}N*mHij=>&EEr?61MD=jOV`^i0bI*AdA z#qD&lD`UkdFF$49TD_W|psjw#kb3Lit8xr&GpD3d8R;PPy|svg@s3ZyPL%SX+8kC^ z+SkhELXk*R7_0d>GBUDoVB^M())m_yEc0_3^U)0)kPT=nsZRBQ^kg5V!J}XUK{D{_ zyHbhhdBl+Rhtu1v843(E?)oS{cEySn`|sFYkN=LfW~LlP?fDi5LHi;S4XF0sM0)DQgQ)Q!}al=5B5*> zg+8Z}^&#S>j}43TGDV07aK3~`J1TAsH0@H$TN-c2PC9P7O;m0oJtu>-%Xx9-wh}`s zr^fF20y|pRxIfHW5|Pgo>if{3ppt$?#AVl)-*{1%k))*hPyPJ3HLe5}r>d+LWqmWQ zNk49(4MZBIZXWxAs1lxhU#jm{oaK$-2Gk+L8Ns!Tk;fg)9~z|6HGt-|7&Y1Rw%qA# zSL%`#LT~stX&1oTY#zIHIX}R?cCx~7;BvG@xarZ;;;`XsMmHc9bK@T2S0C+xpDn!g z3q?bV{;&7|X6#u{6(T@LN=n1$_T@ZVclRi)zX$oa5hAaS&Q3X|`6575&xrnp$U0w0 z6^|R{&t+(W;JBv74S2Xjc(F*9$@{XP;Q50+m3$zUXv+GwcQ|)BfNg|P`WUpLOCXpzu zNTe;NC@JurzMkWo@gHGZX*Js$_Y7?vbnhFG6m@N_%}H@I(Oa?g^Ni{~tt z(3vB~wzgI_!ra^z|L;$5-MeqZ%_h^Nj#t@Xbw%BVL}JV&{@J9(5X3~dm3id|2A zYd3v!o6=qOBgfK?4=-*n>1MmTnRU}zp>}5BrKy79XlI-IHYHDZ?@Cf`I{JTnWiu8O z?YeO3|Nk|gpuc_3|M7!IMgH&i{_h%KQxN+g`%`#0-M72v@IQrU(T<)9ju!u<$vB+01=;diuIRmYN z`-w%qQYx`$?b-19r^F0c?K9ras7Q-(JF_?NJmP%WBCL9NFfO}OOXgGiBae9|2{$J0 z;-#Y+`Ii22FHeqPk#MqJxI;Kuv1l4RV77vTa<`Hr9-dlleDz7qrVR_GyA99NRG3g?6h)l zcv4a#L42@l;>%n2vWW8)OgBhuoVnv+`Sa67O5KGA_}(QQog;V8(Z7EE+G=Hf^ytfc z_FR^s-9=1HOnu|yzJ_&?y7=~am!Kk zkG(#HC6ckk5900d0{d84w09=*O*Y0~ZqG5=E516x>&L*qGd4EXP-5CUJk{Mx+h(|) zKSJB2IpWym03E#b+uQkEqGDqGqodE9{*Lu8564tJJ8-ggN;`5C#JrRGItwqV`T$J+!r829Z{a{Ss}V6Hn+ z+pg}B#j8I&po2daoUN=!NxCcecw{3nG4YvFyvz;~ZPDFx5#MTSn`?$~Vak!R!E7D> zF3IWCzZO#N*n3#iczS@6)svEns-<8mARyonp)~8H>d~3v8-%G$#4H?;081 z))d^ld2@Lv=k3~1?vp1k`>{y$+||l6Yoi@*h!K1hrjbsYW7I%(Qe}dK>@N z+2%%DQo};r@R0L~ zF3pcR3oGjH-u}IZi(;G0F^sCNt>ts(4ywL8@KRN4->H}1jl~nKV-83@O-nn&qhGn3 zHLON)XmoH;e?}zDuOpxw*znZHbhyyx`yi-g}sRxbA#n5Xy=!#<3RcaQNy{w+4O0>GVy0reqZ{mV#cCpneoqDNO_NAqm zQ~zL2AIWF(xVBbdZLwsdqOa6L>9f6shl!v?r)RI*Di!HRg6sj8rT!VzFJAUFs|BqOyr+$8(a*1Cn{Tcg5MxJ8RMb_DLf3&8Pa4YPvJ$(4EQ^}1K z72%CcrFe?j+1cmMpX&%nNM&f31VoED%f(xE=9+l0N_oFQjj*z_lU7#VUmqh8D6z4Y zqqcne-Obcn#9Cn&J^bu2J@DnxmJw{*idC~~!vY51LzLNLO}u&!ZQs6~&-e!^|6Z?* zy}fS>NN|!Hybp;$aaH&q_VE58@KZjsf^x#YnzK`T2r}CEWaoJJyzQ=cB{4 zR#5##Y^vfhA2->9;_lTL)!{=-({k3&R1i1T?| z!xMI3VN6KUpa!?<NVJHe%0Y(5(ztzfI&mw%gaz(u;ELzrP>!7|qS} zEl^@fQh0q{$z^3USr5x{R7gmjGPbLxh&|?{mn7?0Yx-#trGtaROxL{9)59$waO+*z%TqV^7ZemEs>zk1vz%bN`kI}VOOrj|u-K`eIeC{) zYJBf3Dowv>lq->0T3R|VRQJlsWMFVm+Jk~B!S=}>=5yg;dDAwMlry!w4OdJyR(m%h z)3!TI_wYxA<Q? zas3H}&}^BBw1T}fi*uT{jHuXCe4MlvzR;gPbK`$%#alg{?R)l1QZ^_4{Jv6^K?Gm>WmY&%vT3XR>RZ?Drv|*3y zpS|Nt%bCFE{%<+Cly9e(Wb<1kDLIFk0cnp>@4YNsiOVxpEL2HKsV#ALKHDpQn(WGF z6x*Jsk zL&x}F8sT${gG2VbkkAg&*RnFHyLa!Z<@R1?aXlVYCC^=?@;&iPWcCkAgDci!e)fap z4;Ed1Ng~z*o*^0XVO%FpB_}Qx03yhl0F1oc&)gK*PI>cR)|Mv*Krc__w&)LU_KTty zF(aR|=Mt){_lxrN$;z=fYtOaO{wRK4rFkxUyM5n#1@&|r10Jq8pI99&S*HP9lHy7M zs?v$?r`gyj&PD7-hZ?c1>F@7XX7K+z-WGo?Mx1iz&d0vKyHG889B1!FWkC*T+b6~kRV3!u1! zhH^k1>&2;`dMs@(R+&gHe_K>ZByMhQm)Yu*72;az_J7B27S4V@O&}~67ni=FA@3hQ zuKO{I&|+H%e0J}j{{8wj(;1Ci$~&KIF9OH?x-WzcvWZmWyy&gK<`Ebe*u(dgA+vbp z-MouZ@>S;eDrPY!zW+gc&m$sIaug{aVIlj97JH=7@s!dv_&`NnUJ?m7cCxER#rY!FDNX`fcq*L zFXIouxeJ&UCus|sYL4;B)cm|{ntBdJQ<9=oO(jTRu?}};Qh%RAMXE_GkLWzKje*eVo*b?5Po)(-FT}PcG~NxD5X5poAXmY8Dm@* zB07wI5p9iFIowNXYHE^^kK8lUuiL{qplCv*Ig(3jAF_cekc*Kn$AKwq3Mu@H%9%%HCcU)c1-fEN)MK z{kj{xdMF^V*_1z&TRY&-A1i==>6HmL?3QoezWwMB1j1o)`LijK-;Cnky?aYbOK)!H zThdDOlD?o}pVj@kwfWu6T_7iaW@jZ)de7PnDZCbU?fg~jtyi(SI6X*Q3~oi=@bLBs z1#fUDq9rJ6*fcem6|T+Jki{eEO3@MzG3y(Z(TRjH`$eUDWZ867w0HcY6Jb<6tY>5< z0wNRbwqgQuRoc>W@Z!abbOPq#(hZr3`?y)oTaiFNL>UCk*?%9Rrhb@l=kowaL=}ja zkMp~w)#(znEmDL6#i2uoMnCswmi&9svxt9p_7!t*a6GU1t^BDaRh3b}VU&D=qwe!` zacaJpkPv;z+U%}q*9CPGbvEn{l?+XKW?|beSTf_rY?aPQ3exk)NT21o;drd%Tg`&( zMxzxpOFh6Bz;mzE)YI>7rTYLD>aDN*H4Idz@zz^>(ai7vqB4^NN@U`)Hd8gyTOwiF z{B{#|=f{EqR#c?2Cwr>^2{}zazxT`%R%Mc;Ow%f2D02EM`TP6Je(bx>JTv`DKl+a@ z%lCukPEX}&<{Agh_qc7WFVZrb{rVO}7C*$T)!cC_M2%Zq{4kh}bQsqiY8slY*hdc> z9b-W#!iDWl{aYH`8$5Z(G>@TVeSr~B?Fnx0SS~986Ee*!ZZ6G5gob(({Eky2mmpJX zYimESwTMq+nx|pZ3lytA)|`R@n9&?6c>mXxoJqz`IWjH7!GkAWx=(c%kRCD$T83DC zB;BO5w6s*_&idM9P5bhM7zOq>S7LR2eJlMo<)z;P14r#A+I45cx9?%1c5`!+nKK4% z`slov(Ge6Lx)8dG+b3NY$?t!7??Wqlds>P0`Cu>Ay7QAtm(bX|Z5nov;_maR)hPgPF6M6)>8@%bNKBqNwrqCor&`~=Q9HMr@R7i`ARXfKyWjA z<9f%7RZRs2KRzlerKG(AL-+u<7?<_al`gTkEydQ`HhF8I;SHdQ%gp`wr-d!_!MOWl zme%1&^_}AXmh?v*PF{=V3)jY(vR&Zk%gfM651krh7O`*c5=Tc29!_xlJKd`->bBNC zQee`ny_f0yk$-bB8+5goSDl4;c$7e0$s;_juC5h5Mb7C0Tb?ZZ4Se=&9~GkjpVP#b zFBfup9U~$l^nlq7Gk?Ch=yA!(=~JobeH$AuK+Xd`5^Fj$%k(dzqS9@&>BL<= zeygwlFw#k%?Ko$M53ieBi{7y>B_+kluIu>=E&00pBvP&Tt>EmhVx@?;vt)cpw=RZF??7DXC+NKNWZa(O& zW3PQbfBsB5Lcj^Djb8VSJIT9mBwytZw``1)R*uAvkDITeoY%Vjd-yma{^+q|idx*0 z4V|M+$#N5=|6W%+WN%B+>2%B(ySg&^;3K%V7KjqVcn z-hnyZa+Rpl@4&Ac^Y3ZL|F*NUI|!DMT4Ajun_+;BZ)IysL%M_(R2wSB&wuliDxFht zwryo6aaE{RInzIZW2vUx$6Gn0I<`_3{kmU3XfL(V!9txe4K-ohulz28Qyt}8|9H20 z|7a(}r?ALZzv~$+VmlkSY=&z0p>6L+&-e;*ZHgWSNV$nryz$RDWjKeE5PNQaFu#Zc z#pAa2fN0!Ml*&<|tR~udehm!`bQr}{01he@+&3s%&(th`HWX@X0~2s#2V20~Wu*AbU$BOo} z7qFA($tGhAq+H9Nop+;&!s+3$Mdj4N49$Ygte*8FIdQ)ZqCWv_l~-1R6Vc1c$`Tv5 zaUMtcYm(wCt3Q1gnZ#Y_2<#IV7niV4<6g>*Yr=-6ee z)ddR=yW3LEp7*7hF53S3whu*-R`6cWB`gQfjVGI8Db|CPRBHhHS7c>Nv504_2QH0jnSEE)cJ%_AIm*sX2@ypm zm`(AG$|>vFU$OwMBzC$eoErDnV$QUr>xps)MHjnm2=JVpo&Dadjq8$(j2^DmxG~PF zL!VaK%}pE)+9OppGY-PegRzukv|rEmb~T87eK<8(|4Y5auGUlCLIkIAIE_5h zk*4ImI2vsJ?5NIVS^8D)?Z$7hzmzLPyL)>podLx_>6gt_dHFI37uLAXBtixHGuI$d zR$3ZMr}|z;rKUYgA6sTGX{5V=mjDB}|3^T;(B77CD$d%CspL4S4{BTmZUsHvOd?>w z7K+_xoka6A{DEXQvC{>anA_%a2wHYus0v_CjL;y^4;DMtac&3z`T}H}t5L#U1Ya$3 z04ZREQcWT)t*q#+EzKBrec&uCEd0|~x`o)`#O3rhap}6!RH$oX^g+3s91sjSYre1X5gbVs()PyZHVW=A03rv;gcAyx`mF89bE&{f9~834t99F-Qd9mIqKH)GK`m3ro}P8@|oWD%h6; z5k}EGk-aD_0INIS^zq(V`lrX~7sXlC-fV z;@?9Y7yOzxXg|S(u=UjPOb>?0oxpM52^!#$qpZfOg-t56x-e-yT+d7*P-3*x7za^C zdU|>u%#RS-{1bvh17=8zw_%^?qHca`Xt;dg0tu)ucJ55y#DpLAg)R=n`TM^p@%l{6 zb{m?GIN-q>+-uH2-nLWS>PELg+;N{Exq4|8IT0up%Hugv!K zMIjHpTa4hC0Db~Vpqu*n`N^X#+x33Dxd@bgZmPE=)5d*eGQaoF*InFKHdxN;kkBV6 zw6(Q0Kig9gSk9OtdVt;J&X%-8C5A>uWsQydfUPb;k0-e9183)BCr)g0nCdRYid%2| za|PF7AYeP*um9ICm0vajs>42}6YW`q-SF@+eH^ij*tka|Bv>FyeZ{ZwSakBuk2Y1A zYn>g8*x8HsMf}PVA z089n*t`Bv_fk$rm2>Vd7t8Ks8G$<>Ts66V&wKCw#y?dx7QT`A|-7fAzo%d~T?77c} zkEk1FH=JVVuobk5p26p~o{0S!*YX6BA zdi|IUG>1`M6|$<*W%p29nxK?0NG$E|b)|y7$161pA7ErWow&HT*tU9@Lt)jX@w3Cd zoaL}z;)~x;XI>ca%)&%R zeEvF)|51Ma7Gt4vyu20RJo*Wueqy_5+!jX2iN4iJmpkJVG;)nu4~x4P|6$q{$dpyN zn<=vG_PwISMWOFCH3zOqtmlSQwmhQ1=Wu&{)~e4BSH~3mga=eK8iY+K?~}6ZGlF;G`S9$ zwWc-JOyho&({>jZ7vm^g=B`5(f|!#}+G+R%{@~-%?5OkBl2A%0Nk`A0Z|_Un~1Y@SC@c>0==5oUf>DgGsfvb z%7*VX2??eC%cct|skaot>ZG{u*xKgWUJI%DO<{B`%dk*X=L;!C0nOx`@D@Lfbb<4m zf2zFxXxw--W$KByH$&9Lnc!>UL9^%G);{(u02fle(V?JIB>eHBS(GnX5;asYa?K!0 zAkm8{0yC}qzHAz$XA>3eHWr>-Y$QlD>X}SaQc{u}Z#!)AEOf&nK6KhRKvv;p4Ks>> z7p?qem6_nSG%z~PYD@taMNMzk&k<6KuU8*y({_Iua5G+p${N76h|hifZ&=TnH1Fcz zbUV(r^xH;_b1$hS->IUul~+}bCv z#s)b_$;W(BPjFMAk(_hXN-{3l3rU#5NL%RsucueVQlQF@{D}1^oz6yUmRqLx%}j4` zOJ1MYe4!b2v9BY2B<#_tqi{4}|7I>;KND6XUFJ1%F~;Al101E3k&;{mtTsS zkL37+mVRtKgO4E=!Xg!ga(#Wh9^6)nl4ic#2cHmpQK4!D%@?^y{{ndJyOfJj9U5~ zrd>`PZf*?{A$4YdR5)S1JZmr-4ntGMO97$c*h>(6a8g>>aFzsK+Uw7g|j~O9&Ttbegc+Zw~mFJ$oxvx+_~6T;d_Wt>?)L# zX5(MPInY)6%oRY#^VPg#6-)G6btzVjx6YK%z^7%U+T`Z8b3ycqcfTueQcUcQ(=-~F z;aUHM^_H@-vR{19;VhB)e6X4nTg<5@DQ4n&!ge(k|3k0mbJDJ23hrovYGe{u?f}Lc-$-$i)swhswuOLd z;}+a&%(-wo;~h&zs%o=_9k4g?RK2~8f97Kgs} zJhQOIT=~#5L1cdxiDI>?17BSVUBJCfoKYESD|Ni-d=Ks$+3yXxq)a#HcL=RMd&HY&VQ?@tJ#jhfwbDHpF6^F36y|N}$ zcF)OA0|Uc^Z-gX7@6i4E$*^vzWj|a_joCQ3rf|DZG31;(_UcD9Krw=5`=em!e@ex{ zon9)cP#i4y_D&{N=MFMiBnex=3y!)d=M{D$`_kRJ3Ar1f3JOU@C>hQDmNli&J#IZMo&8Zf3LOxNfJkI~8o%U0t*R{WQ;LnZ z+0Ztk@~Zi?O%v1Mz!^5kI)}VY@vfh3rF~&_-}bX)?2qMRZvg13Wf-J>9wJEoSSX>r?=EO6YXN0Z?kcmRGtQ> z|DI_8jgG+xa;(($O*DM93~{dRXt8nCg?k`wCNlGzhE(SQCkXe-wvK(tvL2KPHi3RG z&tcJj!5E1anchQEJ+6yC_lIpj3sF9-?Y^E@%ICF(42A94!-u3zG;6HW=rEkES-YC1 zWbaMw)zo&G@d|1EF&E39nP~r`2Qrz*D$0rcv5S8Er10@CJXU=dOg^W0Q|ae2b`zIS#u@->5w>|-H*a|5 zt&OgpJH%ZA`a`zY&tVR~asDcL;sD`wxwT<81-*EocP_B7P&CEkTL1ac{9d;M7cNdj z2L-W;RncS|>*Se-BJE?xB}thD@q;4<#hhLte%(>fp|>@oaKH>#7YNqWnp_~UkU zoinM4m4PhWBuY4t9#InMO2cP}4$jd4aQ#Z*6*hy5tE<1;$=j~@#|>4j=h{wSZ_ z6>Y8NB~6w#jTttdeB~Ms)WHNpVuiB@g~z-zcL$iNl&vivK>>rO;Fi;g*q=eD;OW8B zw?ZCR^zUI74(|rLv@5TukR7Kt6R{f$na_#MTgvr%^5jse3j@sAO0QjqNQ8<;o8C1% zI=X}O(dn=0vw%Blh}Jboq9Zvrez~F?FCBdx5eRi=vkB$!5 zb%;v`N1KTDf$93d$k2r~1aa+b#4rMUXCasdn<%6^n^SeRbs#keC0+0%+XPI88SsTZ z$nTp-@OHmV6qPd&bPjUYX7;7RgTPnGeZ{c)}6u6xxneM1~s8F z`JW34bUU&QPv7H6v2#f+zW9ph&XIMOe`}5=R27hv9{*iZ80pIAM#rNdowpv?S{cA>b~XxIIzL$Kf=3MO-;bqutDcjj@-HNWCS|RX(37#r%UImk9)Wrs zXJ5MkkLDam29Rn`7;fSjoAqnPh=7iu(@*O%7yFKpV$9`q$};@Buwffq#egI z`u|qOxk(Q*Gr3`#-1@bi*B~S)*ft?#R>ju(pwhCpxTw(rK|use_T#IszyA>&40=UI zIy&}s&GVK8F3Z`r(=)TP+2;cYfZ-gF^gZ)V5^*-yrg(3<)#dEHZlx^<0YRKlBOg9? zE*TQZc#kzlBB0FNP0VTj0+z-KswyFGf+jSo9}t4l0k}zeh~?6S?qWE1hNH4R32D0u zpa&Zh)F*o$d&7y{uu!gL45LR8$n9vYGc5Goz<| zy#ISAa!&8UG7@I8*}Q|R@5;oq_!2?{?9ftlqfMkO+xL_cp|hze%li`lBOK8?FHDqz zl|bC&78^FpFd)(b5IG<>>%a|NqIKw^mS9ifn*5PF!@ zwX_gzEao5)s&!rAd-!;Glhh)g1ui{%#o>bNn1)GtCd^dAwYYotcs*i%OG^(R9*!V) zcR@ymg518*B&I8jI~+bma& z+cKi63(-;tUCv|1&Lr1s^Ct_@i7ywO2w|U|*H8x*ik1%6)|*J9v8uOSTnguHfL#@a2I1pFElWsDJXrLcG%zrbz#W23 zI^uGnj%R)ocM7VuKIA+iCrhjY$PQ7jNDqNgLPywCX^`rHJUG&vascrTPY6)N7J;9A z4oJ%b4*D2>ttrDWiO%(7b7)i)=laC&-#1Mi>9Zfq=#4(&IdX)A)H8xTM4hw(%k*WL zO(zOHkg(tX_a9V{CZXXoknfCZ?&l1d4zT z__xd)3!gDH;b~x#fUOQvG2>Khv~ z6_A*$OvEnlE7@4*e(1jCKXr7JEWP3YKz-h-kA!;uE_lz`e^seO5&`Z4AtrXrlAcHV zDx_!?eX5@9Y~ZyUYvy^S4OpJm`KSy*5a?lid=B`O0XyLvKGt(-^`FPo4z ztu!Qb4%5-oKl}FWif@h46qDjh?avCK(I?+%|091|eDzt-6#9WE!cz|*Xgx>xi|Bzw z^kE_WcK%Tgjvb`4P$YBQALOz9m-ZF#BQTPg-h^f`kNv+x`bO(~eo_884ilQnp`q%> z#KpBTw>;q=Fi_m?h2s%!vFm-g$aJ9x{Fp6Ei+gKBd)iRyvWyxI6U_rQ6GX3F58Xu! zjmKh!dcgy_KiTcVqCSEejj$M?WMk70-YDwL(}i<;ivO*cc|>j7&7-PX*V2TSJa+o@ zt_zL5Hf9{McXI64oX|LSvxsYsHS6uLVLmUXJY36B?^;P`rd@+1&erYQ3GJNds_%75 zH%FCHK@bWO9`&c=ksR)*QSM|8_nBjUg=h~HzD*=pzFNPo9*66G!l%K9yr65MA+>nh zEWelj)QxIy_a6sXVQ&#qHM}ZB%Nl28muQKo=-vQx!3p-uidyoZDmnCY_gz&joBpAsv(=Z?4T8t((PHjr&$l-`& zbnACbk5bizMjqre=ltM7A4se6wVtew~^~RHZ1NAm#%Iqy>%I_Pg=}8Yd;1zv69a>b$`hJr&rbAT}gy zY(S*0;JR&6^X=Pnw-3KWZi-w7(myNDQp?QDoH+bkzbZgB{siBnqNJdmOM)NU4oAf< zci^r2zWD(=z_hz<`f%rt9i<495h>xcJD+I@M^xZmk01Q2c-UX0`OAI>hu3nawL|@A z*{kRLSf#}b`&TVwql0z^J^)arGHi(E+;$m7y#m3MKyA0>ZHxlu&oefNiHga1^RfQT)I87IsonlCE1tDI~7`3cgl*I5ihElz-9lsQ(^@P3rXwTbnUu&+qy`ctJQ>D~=h5hh)SjnCx;KCtRMaQ% z2X)c=h&`TQwgAj4zZ!0)&pvJ;S|OBSACXR9GBi9KkDaDG><f8PMZ;5B7_NbUUL!W4kLf9TBc$xxwzlxFtACj(z1CHhHQfH9O8TNz`uko4mtf1LYRecf#?wX zl$4aLJ=}H}io$;rDOk4$YfD*oBvM1)K`7=^Oit1Iz#uLzPL^sJtO`u0eE{VBorc(t z7cOr3gOLFejUQJt-(lO49~X_r6r><9_`>K>OH@Ehmqc(j?S-mr_3!)Oh_e3CCMG7g zB8uyokfIlztL0@2BmJ1aK(UkIwA|Nj*9`PjazkjVf&UBfkx;hDMx{`fWYg_pp?=-D zmoS@TKi%U$F=0;3FJW1m7$ah3x6{nOF0me@z^yyxPnseWRV8_396b1ZoM8wOmB3Hn z2&9eVTS6=s7U|1gcz3-JZJ<+b)~79QGyTUMTn(EE#mYn`SwvWv2wKU62@%k>!jWRz zwtipk;xD`nqJ$|*bSb!C$t}eP(3xkJX7p1F9a;-RFI@Ckf;03UHpgor8yTaJ0YV&> z2%!~x-?ny<=EptF+z@+LVXMA}9f{1}k4@UFL3#2OZF*Yt(#zQ${r9ZXyfKj(B>*MV zehlkSfiLL+M3Nq&M8A6dy}k9n?6^6D{&4oAESpiC3F419LP!dIP%tU4A2PhUzUV$1G2w!6m)e=V4e?B}MGqq+DwI-fpyu1epTYNi< z%aJ-#y!z0VrTiOYkVYahg5dGb2?ZV?mv9Mp@rREaT44S8+dJhZq=jZXFkZ6`NWEhS zX&vhpTzrFoS*T8olYic*uaxaSkzzBXHn)D-|HH!_2iW?smZa-Hbft23zrYL!Hrghd zT#vsXduq#Jxlxx_Uw^h6_a<_7sZ0h@d0o@H`*$;`+(qCH4 z=H_O-YnyhBT=h4oj&%L-X0#&XXV=vO)Lp^$Z*)f@%`#UFEyvQp3uVQ)82L>T1Sd_E zX63G44TaFXd9y->>}Mp0INpx7rZW=hH4;1J8F;X}-n_XC*d}ogiI^m@-!H^)1_+^T zKIg=vkM@Dr?&|`XLPmK;u$WWhC*RshZJ)ACHt^Z|yP}*@X zfq1fpT$A_);=tYFAsUb#0?)}dJV10fEVR!F`5B+0+_yvjNEae^B(o!mlu%D)aDw&> zRbr!?DxHYc^{n$EBJvVlac#x$)pLW~TBC>Rs3#9cR1onnKGSAOk`)kwZ|me~3#@o+ z#FPed60jZZ&nRu0n0RyoWOVkuPc%cuw)E-8gnJN^Vc8)1nZ&2UbM?_1>!|!3l2+&`=OGKUqr%N^{uKx&66H9co z4*%Q$S6gwVmI#v@nU;J7{d8VPn|GIQt;!1n>y=;Z2CT|?)QBi{LQEHOUN6tlSZb2@ zwDTY1efU_0q^LiLV!)|UCJ0>v&7e4iqM1l*2?-S(PmYyZ?F=_laYGHs+RScK%dEO8 zl=_EVbaLOj&S^&_lMD$GgsHB(8a54ky@>$3D_EJ!kM`2@hQpDqNk7^ z&@OLe?^tAjB!cr#0|G+FOp!{mLw;`%IFit?Op?NZ9T{G8B;|I|`C&SdJa-N5Z3n$l zErk5bvSX3_gR1hq9vF*y4d4HYV^~6cgmTEL0^)0gC==f$pUQcG*VokE{zI-p6o2To zO|~MI7-Y&*bA*{pb>U)FPIhZhy@q*XXvH3;trr$$Okr;A=F%!;U)Nq8vNY@TwKp6 zAo2-vAu%Twg)kbnn^~CVLMde|Iv`ygFySGKdK3sK^W&Bm2U9Os9WqQpQD1 z&DZtc>f#_Er|}(lA{qMy865ewCvubHx?{o!4e4X%ift~tHLp2ItV{+smDmue+ZG;9 z?PkIGb_-%Wj#THdbtO0W#nKvwlM8<++}uOscWMP#XhJhCZWQv!}E*Ml*E*10JVG1N(`&TfWr5tN@u%tsKBd9SG`dHEI}!*b13Fz0nqv@X{K><;o{g z%v$}lIbHVx(rBjz9I8mSNy6xcEU)HkVLQ1o_PgI^f7dA1YpQ8jY$CED^U>+xZ#%s< z!!M$^epc!Un;UxoG)VcHE&%C+sWy)n6ky^l>1Nd^hrX0qjXHesxF=@e_o0*?aNk(Z zuePd&m+??00ldAc4l$$5HTGeZjV%%@bqO2T-MuTCKD>n zssEX|9E=7Qg_yDF0+gYr5TC~rBT!00RN>|O59LAe;vhyX6^ctBmjsjdu!tj0Vb=K1 zoIbUTfyho^@>qEV3&p6Ad}nyLHbTE;_G%^E{>yZG z!UV*GNUmNrXT0mZcN%%`=I7w&m}`Z`k-JsTxbDWx4>hK!qOV-NDiFzr%=<9!_^)5b zEqe;p6YgVf7Wv6bh%tI}K^#3pI6J?);xF%wy_HIpl(*nIMc8X%=@^IkXgm5sD-vq; zv%WqZjuOJVuN{ZGC0Vokujb!Vd2ZK}|jwWMz zN;NYThqSLanq|y%7uftIL$#9Qz7u#@%x_4Hzmu38GNY9F-vOjTUg6JX!Ut z`1Gx`=L>gGX-nts0nCOBLEPU5)th4Odqcw^$i!`{>$uH{>t@YwS+7Qmnq(`4ack4! z4Dv-(hQzsu1CpBbCR2&sfI6`Q>g;HNI3^hHy5U3)#0QF>ThcT$Ib$HJnWQ@vY251o zynP(8wZG)o67E_UZBmyr8Y$wEi`XqIFMj}DqtjwPLdP^hnFC0N#sp;LZ~-Sm7QlqV z1d%B!IPS<}pFx+EGt_x#xZ@_}&j&jnQtsJ9i-8WZyaPth`zI$ke`fBgy3`sF27Q%1 zaTJrvrQ__EN}ZRHE({mV{A#TyBja6fGmL5WO9*2HomQc~;pFHz+AM(q1COjR+!Z_W zChGs@c&Bmr68I3e8bHT$GO_Ib$olo`SKEw;tZBzOd;P4!56HW62)wAiOQAmsn{D6W z!MR+%d1!l6XQ#y?KX!*8H3a1{_`$EOly3I#hNB`*B~DxfTLn;mqw1yHOgX zu&aL{X_#x=xEW3#2GWR#IE=~<<{d{Br(rXu@G>-Mx&2!-g+xd= zR?v*TV5>J~>0jWN zu|iNde`{am=6rU ztnw=H+)R4nH69(H;R861MW=TN$Rgf9nTR;9uZ*1+yb#)(auvs1fjsmY%g*a z6tL{ZNRL6^V4y^+dV_iwN)!*yJ!zYmM7of{G(w~aDU1B^-!Q8DH)$~^pD4*@cTJ5K zcN6n^b|0{f`v10Rw=G90r<5al=!2-}W=u~ZbG8N6**3(iVIlRcEzhZyxDjM9!IhR< zD@}0owO9P(wY3qvu<9RQ^q`1Ep+l7L0&GDdaSh0H1SF)hz2=EC6{O zY`P{DK{D)uX05ej@s_!lRgII{`9b4wmBdiBhDKzyEdzJqpUoI6-GiJv zs9ir+h)(GKaUdJ&Lx%{qJAhOVv9fTnYOku^p+OvujR^S@b}=rH_^YIAF#({4gm8|{ zAqjg8K35|~u=h7G#6rF@eXCU@OlT({LZW}0Z&2>ueaYPX44$%+K)(-Ti+*k~wJ{Pb z&<@I=%@Ly`#JnoTXC*PHfXUE@IElxPA19WXi1}lge_(7Z%K3Rt4ll8Pk(H&zof?4- zLHOl3>tA8369Wct?4Dyv63>Rw1K6z)AI3J9!ed`B(Ex*r4j#{kd%c#~ZghiveGjoL zu~6xeey;-Xc#k;_BEV0Czwu#jWoSNwO-(2iLro=2WvHLlL$&ls^HL$uNH)IYk0)yM zmF<-%$9%shCNPQ76v8~m6oNncGhwJW%~PVM$f&5$5c5jV{&ILDLry^1AvPcJ%oF4> z6(ad|5Qz%n{{lIGt*fIVLJ^?41jS|&b3)quotj-XUm-er`v(g~qOofh+~0H|;A9X2$z|MhllL&>;#G;5kv;brAFMbl!yfYZ|flTO<`1L zgYpMIA5DBu%0Ih__C}qwPQV}h`W?C9p&>)tq;ObX`f#I`!0dytJ3!KjfnDOEB^a>= zqx8mUWj%Q^x!hv6*Ye8B0NPgfF=Ebzc))?-%{MyqMX2J$9z{2*LfnRU z+J`PNDAiLa1Q~}Y^Tez>wncl1y99=*{y_Z?`#AAIg<;Y`Sizn<_M{Ak5)zc?LJJuL zEzNDqgVHbW+re6nq*#KMxwjS(Jw@i~Ae`Pqq)Uj9?jF)N3G9U;{|pPKVVe%FSdNNV z7oE&wU_jG|{&XgT1)n(E_?ODns{vQ1-J~)BSYJ(AuwS}(uMc#p<*rx;vX=NjC5@s` zWlkingPYbJzVY<&K84QstuJ1@m?s+t`mtAEVV-OVP0U1IlgK*9J8GEGDD?+9g%&FO zbx~!fAtu7u79FgtsEM)uir=SX_ZmvUUGPFryBZrK+Up+2=Z9?%@Ft zd0p$SIx{F}PL6malrO>*;obMJM~G(+@i@EZC%8$=$c)vN>yC*uy(T7E8TJ{BwB(to zMy;l#C~a!}80uvb__nps&tHXXI6C49sQDBqA(GRK81x<@qTx{uA5v6hr?eNsA4we( z7S=$?u8FgmpP#2W#HIP8rtC1U6{XVIce1<#b?x$O!Oz09oT^8zti7oJ!g{InDbJ*S z1`US#!Q}d*H?hV|JeS1l{nTzI;BRVt_=4*31q^M` zyM}%+EIf0m)Q>r|dle7l>XCPYo7pK z`XNx*mY{0*q6tT?7ixU2j|h&4;DAoApWBV{t%t|Q(Ds%QO(87PiZLW5A~;ulC2F?u zGRqDm5_5j@3#48_KI3Y)&e-q1z9X2>GF4ipB5-FN{dv##S68F5R`-fnJwMLHbrqOX zK_TLtG%-Gj=R{EA1u(UA91l&A1{AG;5mgR^gdxV_Mwf|3$`hALw^T9SD`XvO%j8Cr z_RP%8G;T`rUjkc_zI7|~dTI-rfFg++u!Uld{ri@9!r4abq7n zy~%yy<5Koh-5l6v$KZx}BEXMvuI!Qata!o>lr}Up44MU)2`5q)Z*QiCVwChC2m>)t z_a6hIfA|lw7I`_}tgU_|dD-&+@b=&FShw#VIDSbYt5jB`kcLoJc1cD^MhMv>*&{0>log?}GPC!Vku4b^ zduB)W&Wi9oPU^nz_viO`{J#HvugCj->+Zf>*X#8<&+{11<9R#}(M!INn7#xqtpwm0 zMW&qODjU0eFrms2?SH1A~Ji;0J0Tj!U>qB^ih9c^s!|7`zD!L}m=I^zLAU z5V^$;sN25xk__u1ZKwf$1wHgkT!li98m{U-sIfcyS7eS7jH#L~lOO?QV z$fiC3Q=|a|Gk|9iqew zKcSzLblCByF9bxWu&hGn;UPYaqQx zY8A-JBe(^)H7L+Va!=ZT^ods7pYL+LtpgDRf{4iAzR>d7+1Yu)d3gy>O&tKT217kI z(F{cbh={OG-K{e|-Qw#+W zA7C}3?pGA4odmg~a0V|&e*7WqejKF7Bq55{K(KvLNeKZUv`7^Wwf-Uc1+AOjvZ}21QxmYx z$H9z|(bCeQ=X&_a5pSrD`aq20pJ@fv3%7&;B}A!iH%DxT*OAZcI-VKt`fwQwSvOdsPEDZ5b&65^^i%4mt!07@bqXc;8+Z?6?O3;9e z51}sF$)^Dmfzx?0m`PdNJhVGD8|>#iG4*K%g9>(=y*E31FG&>a06If~2IwhO{AL4G z{_*6AwgA+HZMQUnh1hwp+o}c!2Zu!LwNdB;V6-QY1pxJT2iw+Haz|KLPgF^XLR&|N zaz7 zx&*RvP6Wcj405G4cV9Nd062PU>tlL41MFI;_$6?vL#%fDx?rCSB!#=X+h!OXq84Kp zq<71>2X{!f^BbzD{W<4J!5`-0);6zK_l6=((_~{UT;kN@e$+gN@>NW%w!w3G?d>c& zV_Z;N3mw`8jHd-g0fhnatNTgBC?n*zD4{k<)?vD})rA)Ir$9uU}`13D+Pl?#F=LWvpl2%=IiDux4-@JhlC zoSBiS`c&zH9H<*%D#5`v+3V|s8SQ}ws0NXa<^&*L)5U&=b`)|5wgDz`g>{mfaYU@W zZL;(T?YX;pq&oWBT61>+CJ>e{s0XE(A@_5PDZoV>LE%eiE#x$i-PQo1KqpQPvJ{da zHRuF_7H<(?hEkvs+iK}99N zD)SG|(J_+P7$o6qfg=Y4{23B8)HsYXF2E;1^ak~^qbCG>Alhr!T8Bg!&_dD0(ygqn zA~~CDM%_#xiqQocw1t1Z2QJGb0+iPN_f_9KzRrR{*YTfMTY%pg?*DzCm)G2@{}2AI z305|G<<^COz7jn2yy1nx(7KiR2wwlRj<>$sV2xtkq*mH z4O&xpF&1)T1G)Rb zlS>{LBcwjK1=u&bz_CN@!zC~_bmz~1{`Vp`;3EG%d(uAebJ!3Cm4b%NIt%{6-^;pJ z694l_KVCuc2u{iJmy7Q;A!|g^Q6RB)_d+#5CCm-<^z_(Zp+&%eLFeaD@CYc+1;0cD z`T}31PILmQ|t*#<+FaDF^Gaq7Y{yM1_aLlUMB|_4R8_!FMyFY zU!PM(B9})tG>Xh=(a{CDGNonXzqiD0{`wYZ#qsWie9^^dTId`Pl6G|dLZ;MGuc*{h z|DT@?A+Uky7#Kt*ZGnO5C$cRHgl=FwVqE^DqnL9cjjVuE0ttkFXSD-dHtzc3us?Ku zh;QF`_L~CaX{sQ@CIHL|tT`42ay|?OT2d>4UMiFKNQ#@AJ8+Z#<-vtTod;TV0)%-W z$b*e;C`BIhWz(b7?$D?;#(i5rdFL}BO~vpZwT>jrKu`7 zI7zt|L`ik6fdmBNSuEf@%Y1~$+BpmjPTy0qzPfXALCA0jLe3%p61 zAn@FTWdC63GQ*=#Ops@^J}n6vQ7FV^t`gD0DxFG$PAO>^`E2>1H!+e6^OF)k$~e4_p|86h`-0)=S>eE@(ShQHNZ z_n9a8asR2Azvjck`v*&zD(+}*D%|PiO=3bqxA*VQQb{(rf)ByC0AnUnQjSdsFFrAm z4$yL7>H``KROUR}oP%X-c(u7{H$4d#K_!(`#tfX`nMy5_-~viO2Hy=tMMnAp`UPnT z(i(#_S+Y2&X)b ze`3iq!yxraBx+4f-TU-~vBZUds-7NlDAB+Eq6_d~p6#3cdr79gQZd+{a<@HQt-(Oz zi~4~sBV295->=XMIR0teMyg*38cM!c!7ha?4&qx^=wcLSYS9HMa9Ko z8Ckk#9X0-Y#L$tmvN%G_NIF+~YyI<<*#4F4zC-N#JfWCtVfP%u*Kg>Z8M~EB-Yu|i zA##t{jDV=-{>xqn#oJrIAkY@=Ffi*F@;q1sFNtFzGBK7{=r5kNy??5{L7}R;28;9e z;OGyP-K2r@2fxh742cg^F+G894Fzq~JM&^1Z_$`Du>=tTHn&ifTv5Li)F5~Hy#OpS1a2Ut zmbVsUd<8@vhD4$-VrD7(@Mll=9oqjHre*0s?oxjO?uox_wWzI`%6+~h%q=N|M(mMk zjRD@JR5&-Ni2;?#!0hZlIX!vNdpAtaUp8NOc7&zwrl_c&_1=lHKc_u|LD%V}RXJx|IcD~`jsZjMPk4jp#Z zJIbm>8%1}v)rvMKvNCV_B%JK$f721d%fk}`<&b|E9pl2KTT@;>1nbBgb4IUqS+QzRl{o7%}y08 z8lGq-&l&T~hf5QLq6GBvdA)pD1(hsBi-U8G64Tytr;>YHkfe;IWtF4a4hc0iAwWWa z3~53Y3V2c7;w2znI$aA)0$Pa3ra2*!E*mpK@Gms-AT@cV2{@4O4WqtXL#@A@purD1 zZPk#36(rY7%geU_8OsKu!U^?*s5UpDNtGcni`SjR$A5H#eIx(0B>v;z=M*`c6p2~( zx%wo@Qvko111OI~#zAN63ut(u2Tf{E^^KS4$1gDwzq z6{Ktg^vjn({U3z`frOp_j#C+syuE*=*FPs*z7hplA0BN}`3L>s)(K!&Ld^!Lp@Hf@ z2rcqLA=Di@kkh*Xz(hff z(YnA_-IDMa&E%=+B12m5Qh!(Mke!2O@XmDQlYsKnfv`@3BN!`-%_S{Vf`Afj$g}rQ6M@zi z)rx1R-~z=urrnxcw)<*Xg8f(<*GuGH)puuqK865 zlicy(+C+JaY=PLRUK*{l#y*!yHu**Vz8n4QmM8f}Z#;-niuy-1{LNK6ALi)F(%Y2e zB6J?k7712)1@3c2Y*FJ1|Kf0kKj0UE$@~y4)V3!`!p?MsT`gX* z2L)z`#N42~tg;oE=m_Py;O^Y6;D`u9j6MDS7!%0(b86*|GYdIp+^3*B`HHzZh7#v7 zBqlBr3D3crL4q2S@x!2&Oe^}gNL*h9WT_A^$fQ~K1ViwFj-<8U6Hpv-grYX&O>U5E zfuNCmhKfRA{3ml|&WbC))lp;*3L%$~q!bk44q*Z%5;|#Ep^|ZV2b%G5qBRSAkKNc5 zT%-W!a2TTsW&Y1SjVh4DgX-5QeD}UBiGra2-=HepV7iKr;oF`wZsut9Gp+8+cX_Ia zseh=u$UCg-BtG4ZpAeV=9L>yB{<)*Kaq5tV^n7` zx6F3@6#6}|ivk!lFDh)UtU3*Z`v84__>%}QUqA?P|0oH9BsK)a=3o@ZCxU;gSj}`E zh2*{(tN@5?A?!K=gzKQ3S_{fa)}n(b)yJTkFQh#^kE6ijF|d#$A43x4_bS5CxMC?< z_^ENpasQ9Esj3>gzDOj{B-B3(BpigHvMmc@<)|!~C7=zK{764NU^bpm^`-n8FnpcJ$ zxosMle!I1t>av{x|6v-ItrtJ8FxrenqyTV52JX zLu2$h@mH1ViSFU%uV0bf1Gg9uU@GA281v*!qgL6vmw9-h*yHBm)_lxvSz?$8oJaOP z8f1PTbf5Xb5b%X?!fa*})Ez&eW-Gut&!9vWD%lte2>Z2(&w`S8v}3*)JWZ6f4Gu=m z)`ZCZZ}6KOkHVR#R9$wuiIl^Zk)<+xQ(u&=KdsPWhVy)}KZmDSW-i zp=F+Ds8{~;2nO{Dg28DwVgmX8_Rbi0Xt+t1Po+DHo(-^mAP5!&;|qdbE>I?!wO*eF z9tdWn{tdmzjXx0yk@80oe@eoMfTRx+H-vv+{p$l6s@J`~YvKCgqn2otjHs$o169ET z>CA&4o`SaOG9N|lle})h#(RDY{lkjL$^M*>f)ayd#YrJe=$wTfYws#2)FKS3mvm>Z zz|nz9FoLU)y*YpJq7Edf_VhYHU3~#L_C5m#su4sJ)}Xow`6cN`!hJ6%ztvomFq(9` zF#T+P{nUI95{zqKq0JZ8T^L|SG+PBwqEXcF4ISn%MCmyY1E%LYfC7S8 zMYlZ&ZvlmW1S}|ExG{hSJo4v)5FO&^V+a<7ko>~kYwt`!5gNKqE{^q)|p0S)F^Tiy66)THC`p?5(8K05_ksJGK z{>flS)uQBl*PMrIo;z_#MDzso(tv$aL_Q1n3E92db)eb6&rc1W6oUaPfV2#to{p>? zbf3KdX*UFmKBfUah!{w-FF`*PHJ=S3F!BTq4GnV|<%1!nW1<*7A$?UqJ-)W?QPmBl zYvlO+h3?u*q`D*)Ij0R827$HL7`fiwug@%jS@gTN%nT0qwm@q19EiUS%qK2J*T z?;eJeA+eP+V>!L%jKl`sj%gEW>cP*?K)593eH7eSj+T30SB|Z65#@d{`R~NFf37 zX9XrHY%@(5P_zjl95YJ?`b(IAgfJ?28_P0};psC?3L)UtAWwqw2R%n;@qPD46O79< zTX|%@j?;tL_++olj!KiS%jOC{dv%Dnwxu)5O+bkhekh4L^!7ZbojzUW17UZfx4~zX-g4(!ao#^T4%;A(`1mPed z&KCv@4iY~F4~vBL(Ha82CkC-PQ-dw6A(T&%B0+k)YaG*(mBr`wmnP=BLpr}tpYbsv z|NbWFcOh0g*nQfsX`_9_F^_5H?wg7hA0)A)3So4|sH6w0XHMo^5XTr-VDi=Fv83fF&Xmzp3a%J62BWDPR;H!G4q#T7uR zs!C5b%kZ7Aj#}6R1dNjGE@c2SKdVM)#TgiRX18Rf+4lD}hVqHo{&N z{U^EHiTB;QfHKRV^DBim)?WJ?G(5jN^gCZSx;cK0LIJwo79#abV{KS8r~+IR6SAk-B|qaI`QU6-Yw0Pi-Aucf zC)QROKd?p0PhS-ss_YVO3{fzknDTqNV|3%qJjpAT`8zQ(NW}uTS6h6+l z<phn=q$$_dm4B+@a((1kQYRSwyzdtB?8I#-}Yb zREEB_A1BYDr+TWp%G~O;WX15N(7Aqkrd8c2Yf1|kVv^j2Ls|l40|1c<7|x$R2fD0K zaBVLDH5aM}nX0!Sor8IH-c3z+p{l;`7$D|HDmZ});(zd#l%n^+zMOSBIiri*{0$B4 zVVNRcS1;HQ)>I}cjLzuVd)#4>65lsV!ong!0a>3KbXb)fm&n?mtcD1!T3UQ=TKe|x z+s6w75o`SyAS2;{p*C^<7fMXRt^+5&4C2G^N#}TYD3Krw6wij7_nbe|6Ar8_gP_+FH(*IVkBtr*SU%?XlFSC6$1u2@;<; z`97WV(I5Q!PKX5a(sroBrp+P|zg_z9y^-;A)o#REb~*ZIHXXRbn}A&t1J6XoZBu)p zRg~M7omlxP%&btsq1`oIvSi+4|EY}4XWlCdWbW0xl9Juk0*?ZNMQjYPj%t98mZ8DD zp9s8p=Nzn|CKSBOO-EUbzR0dZJDPJw&AZ54CXf>Q^^5_ zBrJ?>nOI%=6FFbjn7y_~6-V_$`RZAxS^CcvQ}ajFGw&r+$z*HDyw*E;g8wp19)gA$ zJgLh*M$K>66aqJ1%{B*{I7sdMuMqVqB!izFF>wP+io9`f)gV;Zc?^xba z*Md>61#8vFdTm$f@`8)Ulw4u1ujINe)M5Fcq8zsR6WlnW!n~LU9333LDbV@X~c#P71FcpT_8m@w|O|BE#OXo3pyt zwb~C*Go&MgKve*BK0z><5+aJ{3djdG2F7w2&C;FNH`%hRx<68~f6mYgM@iR422#gO zKQE17%~i&DaAY37X?u|>`1PM%5hW}UR2eM!MakwnmK^C;WU3mqo?WWKxhRt}ho#Az z2@Q?rDz)@cmI0(Y)Sw-~ZE*I_AG^yxXkQGyNz!cn`POB)h6$MTvHh$Wqq*DH=2C$L>9KYq3?Y`Q_kGWV+aY z>A2aNnG3H%uez2QJbMO+I)MY8KD1Z?SS=cW)xLL8D$ETUbtU#7t^a>TZ5wcCUAB{4 zE1a=dc8HTL_Xa0M4slT8e26?EZ$?z#P)fW^cQQd@;*>l}wsgcs(zSG~0zHq5Jrv4N9N>|ke8g7YV^sLZiGF&kKf!IX zV^Q#HzpFOt)m5`!juaLWuT_^|pN^J#SK%uuTATeNnGi;qRYO185&%16s0RXino)WJ zF)JeIfBKUag$I98T8f0(ik3J3b66c`$r!n?iJhC=eovTI_bYs?{FM*?lj)hq?aF?S zA-_5sT-AV>2#_$)L*-m)+vt%KOt7pe9|Amb>RbL5o5kyvYRsC`Ih@lH3C|uwCJW&M zRB<28GWFg2Q)I^b_r⋘ko`?R`%k5?}l9{7b8b?Z}L(|jJ}%2D?@dhuc`Kc8`BE> zplvsKsOpm)MH2cZ9835fuIU^{sal%Ap|vBJxZvPl>s|mDA{Q=%R}utl0+fdJ7Mi^u zY}g;_EH67K-B?@bU^=pqx1l+eQ(@zRJG3y2aXO77A^AYA56bO|+6Mm4I#6Z42K^bZ zYAw7tsgEt$N3tIs9zWKs9H{#22^aiXzQvg3r|$)ZWsru~gpe4ZJr6X(7nL_4-vKZ? z$5wXV2zU#>N%|Bud%4%@24x}5BPuEKk!d#)DePeljX-BVMuPfY~}?g zROIBH+leJ#7sV^+Y@aJbX-P1_@m|0V%2rNhupfzO35sQh{X@^v|@4(1|R$zN-pNYE!U>2Io z3%Y3q&2Q(I1!(_aM`)?#z3=}Lib`kKBa1y>sV1~pMH)W%^w}?FYVcb2z;zc`G*UX@ zswS#no1s>(n-8-!C{6t@r2v2t2c^<=7LOYAMWDXrK)z2_N$wmUPdjeGbmS&YsrxGR z&N{(Waw!qfcM^s)!lRO9r(lGT7%1+%$0vlY3%1cX;7_j_FgT574QL3=7-} z%poE`@>ziK=`4S*0fgTb7!yFR#GJBXD3PG`cHBq}OVgm>i*3WK2?-BSfWZqGW1gI5a05gRCEFI{gx+8tn=cE(@ zw`4^U0V^E(Red zP~SjCl5tGl&{ZmFF=nzy_Fcu#F!f2Ii@21DHkmJPo2^wk+jnYB`dYfMo6GcVY+Pj+ zzFCxS-Jl(VUga3%D@K+^b;0V78tQ&y$He+aB)j)8j$F z>IYE)AX4A}nt@s+KxV^WWfPeG7g0X|)WcB`NLpa+6pZ08-~Ij6@z;-#cbrCKNV^6; zM-wJt0G$P?VULW+n0IuWeqg-_%76%WcXZ?j$^k}9ObpHYKtfrdj%d~VH3BmBF?i}o znG-1L9^j;P0DwTFAy6j?RIOAZ_Hk2LS%HS8nyrel?DfxT1Jdl}cO}XMrSDErc#trs zRd={+amu>atmFNI88lp?Jsg{Pam+`p$DEnAS~F~vIdycce2wJ$qUl(>tax_9OJjZI zo96_|A8+GvzJ8XaTPILKR`i78HL2G52wP6gZwuOISAd}D0bN089xb3`Q^pQ3cNZwG ztl$b@FaZow=pM%l0EZ9_y9AW?k3;ue6U{b=NZ3IGA|#NgQvh1=)bzK@9B_K_04YL4 zV>&fgV6+37(a|7C95~E|sDXnItXV1yn)-~WVW_tl&En1H9tO6?cVGs=bH%}+t|laY z2)*Hcpumj};Onh7jr+BDUGL+E^vBwljI5Y;OP4TD8F$vSuqtn{+}^9b^{$om)V$G_ zWL(|LWNN^fFa=`o{;;tar4exVGxC82c;KpewOPg5{vE1laqF{GA+ORQ;H|f5A9Jm< zowu4halQD|3!v^+c|8A~PO=Spv`U>yhY}AY2r7%LF+)RpFu|^Mbb}y_;8?m zgGT(0yGxwTttT3zvm;|;^gH+RKHnMTaCh4MH3{duTQW&51%m+@>2C#~>>Zz!GzP|6 z7#grw*N+ksc5Xuxp+v&&35DP-m|!q6HFfF7?Bt{;bXI8s_K*6VJ85G2)Qg@$UK+Ws z?zk+1Xb$@QFp=s)zF6tcX1+&_vHt^FK+~!NxoZ^WU)VzLfvO7eGv3qwJ6uE+Vrz#6 zmfo{eJw7`_bdKui(OP|SW&))iLLFRUwq*I!chAYF*Cq#jsM1B;A!v|n1->^@>42iA z2wXKxkBoMpCOCZ;N_5csa{Gk=A?G^;|6y7|05Ok<_1lkYu=jv=JH2BH5HxQ`>elAw zISvl_MJ4bR*cd>3CHtE=-yx-htM8%H1qMo^u|Oce2OW8Acg;>PFX2`s#yBq zlz;kK??UkJ-Y8d>7kNLXeeNqV!vSJM$Tx66DN&mfs5l;mX_Wx_j-s;eYqF3i zqLXTYr>%q8vx&a+66PZcFWQ2U>M^*++U*iUxg0mGtNrp*M~NXYpMfzz2IaO#FOH)8 zf;3cu*_{9SnegU|qjqecyiitHbj`Vb0|C?<10hE7P-t%^JiR2pG+1|>-4pvE=x`EQ)SR#58H zj2#@2c^7l!BoyB;Fm6ijCDmOHcDHvrQ`{a6k20&EK2f8o{_-VGq#cyEjoWx~+TxA> zPd>$;qWpAoUY=UU}Yr%g|D9N zvPX*2XU}Gyxrd;of`BYJot-ZwN=17YXdX>Wo7h^v)lmrvd)HRBesa)HRyJtWtW!9l zv5V?4TdwX?4jO}>wI@u<%gceZiJE=9L0N?!+%#lmN0!Vkdd4=%Y2Aa$GzJ4=EgvqG zsj?K#EK7a|Fvh;3#A*9Fq7agi)%J1iTUa{sg@jRG-^n(u-Dge1PrqtT5`83`f{ZFH zDZg&Fr&vc~GR%1OTBKy3Y=61F^0>3%;Q#oBUoz&rcfLOJWFhw2gjE|$ZdLvyZN z+nkzwqBxK^%CF};HJ5LbBfbTZADOs>#32lz=hzrHXooRq#%GtG02EU|S`rDS$W^~8 z0M|GDy8;R&_7!;{kmVrO%~l2A^pLhA+8~g`dvkQrQ?Wh-Dc28`E0{jSpq2E|Af|S5-w#=d{+By!$^Etd8@b-&fP0JsrgL3EyQvA(*EcIQ2h?tTS0Hf8zvm0P8t*pMnvEza@d1D(Q$|_ zk-8mHe}qxVKzx(z)I5ZNs?;YSwl0CEMKafDW)>PX3kgH%_Cg7-0W_500R%|H?*(2B z$^<$P6WJJC`vt`7Y?w8Q*w0Y>Lh{xyhny5OG@4eCc-jTaJKSTvoy%x<9(Rs|V(yj`u zt?T&XO)=6Gq1QOF=meI5(PF)urA)92+xo(Wq?D99U|yb{xX)j_$at7MK*;@@_2f!; z$=3@C|4FN3p35-?VIbm#Q6)g^I*n8oKnfTeS_>GOK}FD^zM%m=02^*zLR~P|8$tuj zh=Wjq-#_e8sR#CcC!E!v?gOYH!2qCUXAy{AC#W0|4H}tmuppQSOtM#m>K5i&k?>)K zyI~#@PZpm-b2&iqfPn$A#BfjuQqnga%;7KNhFqREQyKeb-+_uV=FS;OV-rLW^YW?$ zI6-27MmE*@wZwTu$N@EY-O0ByBDDvE9I*0=#WYt|a=2&66sdh~MY$$uq}hGcw0Un2 zXbpuQu_a$%t4)KXS{?^WK3RdT?61-9I9>0$Wa{3#|8BCZXEF1*yJi-bVQ%IAnB&ll z;{9`}`5zkZkfQ~o%Ppr8$2Ts1{c5m>hW3F0w^A_Hlza!SP8e>qS~A!8D_pO771dIg zq;9N@7L4=WAyx0gAHLC#OyMJJCR8!o7=Q65+WErrcc8oqNYc_+mGcrepN3gUk`MhO zn=JPfHN#JXq0XP2-!2bcd~r~-m{!)!#FhW2W-;v>g=eI(bb6EYH5tM9qeCoFuYY+6 zP0aXu50YnC^%;Hk0Wmi}GheoUJYZSV#y5u(|IM49o|sjE!fj*-ji!PX)4jUH(^8X@ z3vM}3J2Bpt0r<5)+l{m(0D>T<3jtVF83gLs<*f$(E@qcTW=s}O>;sarl<{%|#y1?! z4ljK4+pOcU?+2{ZuHg11^%-UIU7BV91%fz%as4ACjq7-4=TO1{z4nSXfFa*b+UD#o zme{=u)#p(Asd*M9Mpa)^KXrYn@?bb0dy%G}9vOeHdcyyc7HOfh=*F6C+zH(H)(QaB zz2;1z@92OMyU(_>0Y+AiGmVyl72&$NxZAh-x3SHBmV^e1L;yU-KK06yY6K=T>9K=v zNal#}m#qH%Pew5#U&{w@)nv+EItcA|G>y13>>+;kR)>nG^Sxj!khP56;T{PQW6R6Ln7Iy^Hqodly(==Dky(z zl8|^>!`ruB<=_4Q@2f1#xHr%E*6oISWNDRQxPB)|Ui;>0k6jO?K$s4oRGXbiS^Kv& zL-&5qNj4~a9SLt4@}!g2<51hM$9mVxdTMkn4HBumf6#R=rhwS`x|Ng1vVUo_$s6v> zE0jgD#@**!%~(A4@zp$)?rA>rzs07;2Prw*mWLEY>b{SkslM-Z$fYoi9wDxIXM{); z`>N5}IzXH^WVZ%Opl=3>$ zbU+$iCp#U{etVDfrG45rlrGC9fWA$ z5&jx_8e}1q2EKk@iIO~C(kHrqa5Cs3VrGBv@~i%$j(Z|uS88B_lA0@ zOkYP0EbaXUNdbDc&(I12%?B_M5L!(291i!rN@NydFG=yl4cS9BFxBhcz%#dBP;>$& zZ`C}2nBOy=t;uq=HWuK-;d?^JoC-MR>%hRn5DtMZJEE2Wwgsu!Sm(@h+BZL**5ZGC zOfL1}C9tDH$ScmH$6!3FD0k$Uub;?x=yrV*XF0z{2yYN;jLs!{&g4U4o-5mP;=+1x zWE+xJhW<2^#Ck8<3mqx%wI;&NWS|@vO7*xpbf)*XN%e0NRkqY*v}z478WWmmD%D8z~$PgqUuM)_}32gY!P_Cffo_quNKFsdw;-!F! zRjiFm5|QK`Vd)dlQ6f#6^0%XeRMJu9dU?MC_8a|$Tg0LFGMH?gQCp-kLMXccLf7OKVa^CPL&(l4T0zN$F)(Y?XbcE z+)&oA%v-O?FIW-QERl7Mj(*Z-r(<+7_P^xJDCBUspALS*i!r#)+mY#p6~Pt%+H((5 zy8CHHemw6ho7FN7*KB41$1jT$gbz3t?20O&+6;-%55wJZ6S?Y93m_`jMY@FhIRmz23~8}$9_ zOXg&8gZS0vWQ6S7v;TIkO&9&VMgUQh_SiJH?)W9_ifv#6{j9Vf_b@l6J8Q+P36 zbUW@<{N@!ZR^+=8zA*5(p5?){)h*^L@x%7ZO-5(rKK5VHas<>x!EQi@;I{l3(`H4} zOiaU`WbnZ@6?a1vsg2ZMM4M~U+fRXkI#Qiv-AhC(2tup z*BAJ2G;9s*?Kxc779|s^r?Gk7P%hbaE?U#aTV6z$M<8gOI)uhpGCyhOa>3-oe3pcy zg6?k}_;`lQEY?_u56kNk91Zt~oI6jeOY6YKzOi@=Ur7Iu>)nM=arKtEYBBaFP7_f^ zyyUiQ?>{cIOnz-Tlra#sXqnRbWUI#AXm^w2D|J-7*9hh$`(X@vJ+2s7;T70krgbV> zjBniJJUY1AyJ|IfE!y)-D3(T?D>jRsu5M9jMST3{0bxz=h6V}gBo`JJ-WSTrobO!W z2XAUOVRdT=!N2{-QtrC7N!@g&dd@u}lGn-i|ZCXZv&6@75o4ws8!}6+WJ4PJxANzXzCAShl{CBd z&5wVFIZsewjXA@QYVeB6&)SXl4@P68Q)VVTPd;s!-@`^*GBIX9sR zkE29m;Sa-~qxacV7vILd#@rlar+0?Xw2K_pwQq@E2@x1E^fs%#Sej$8j z7^D0kwR-^u(K^kpa*DpJ1S992^hCTAVtEh|jARMzq3?`pt z@#^KvtXw4TVsmrbJ1$2PL^o?5nIU`>)2(?#BO`C_)9JF!r*F!cEKCPp$G>!|z;vaq`pZ2-jOqrpp84O)hD-|2zvNlK`nF>V26`?wJ>)K|*z)bS}9Ey@T$VbRiy9W2*inv3JibC5ufz7U-v?z5B#x z=a3QVdxRKW?9w?{lPTZ1>s)3Ld0E*iW6$zsnQh4PPVI5zn-{%fG{H#zRWfmy}?Q3TxS%lZ~NTm=dafou88q3=gTP+!v2L% zUG6i3gTD2cN{&DabDg;F(XIiDlwu=Y@7U@?gO+#Ef5Tv;bNL9%!qwu*yU!}hD!jPQ zU)d77U7I*P8org{VD}o%?10ao+kG8YOXifls9WNvm4D~z5T8&-rcX#$LsaJe2UTuR zdc-h$N>5L7;#z0N*Zz3d?{Q&9jO&<;%Eq;TJZxU#KWB#a7wMXzO^(KFkU521R`0bi zuKnK^Ov=v6mXLO`lIiz*Z$tUMHLHXs{JK|$PTS=_JB>!JN7uH+BFw~;XaD(QT$08k zGduETiSvt9rlZ|V?Tzh=O{dOY@o^_mZsmdB3V{wr)gcWzIc-zAYa)7j2}$<_$ps^; zm^x24!H+Z49x-a3jf0lywPB1wfgB}2l&0*tllHeYe@bbActyXR@DB51Z7HK)bYhzzEnnLqbEyA*`i1anle(Sm*7Ivpl9v6;t=1L?X6s!{%|dH<3H}GG zYLcKHUX6hh<7QnI&T)miW3}bpgh!X)t0r3yPVJfaoSMvI8`|spxys4HI z#J93q*mOvtyYK^uuS*0@_(^q%ZA%+=#N2=IoB#fi8iU7{x(;&tyAd;G&L=HDmBPu> z(41vS`JV04y_=!InOFk!O;UtQY#iq3V_m3v&v+D2vvhcyJpJal8Aw>YR&>re&6FxE z8SFmCc^-+k98TuXCrWeGw(-;Ye>>iOOP_DGzVx&I%z+)T3$IS#+*0Ka;7KEA$h`2vZGM!!WBl2rC24q z*A1F+y%!&5LL4%dPBxvm8WqL`CUCo^!O+A>hz?G0R@2#p zGWRZ(9A@do#Vh4gMR5ARQiOqtZ|C6PdHaQE|EZ5yr&6h`womA6&32zOv}Dz2r>DE) zlepS57$3Q7p<`8Vtz_8;mDTTwajB_h7T%sX zThrTI>s>o19j#+v7Qy{*u8}% z?aF`o=E5Vq{Tpe$0Zv!?R*<4`O9$>Y%fN;8>_a;lVXWt6pVTE7DoJ#T^nW}J5vG6O zESR#OC+%yD3B#w~M%NS)%7ZC8seZJT=g%o3=T_*5=d%4_)aB<$`hcr|H&fdG+E=QO zL}mJbF5e8P{qJ+}Pcg|gXrt=e--|=WPaaz<#whBO8a708Ge-)jj^5h+rlFLP)IC1s zqm!g|V_~a&a1G)$B5}W&$~o8Omj~uoElP``p((8sj)cmSk$9Ckk4B4Uiav?NGbz8| zOq82#_+;`QJjf-|e5vcmC9aby78?9U{?UG#VC^MQM;JB(OC6!aRAOCX(yYgBPE#o6 z!QanUR11(=)Fq$JhmE2n*D+H>LvpLw_HOFJnSfZg`s06tOY^J$dc7yf@uHrpP4CZ( zxKDwlZ#`4}RyV)B20vjQxOozkFG0gPib#<&~ihnc7#6~S`WBHjfQcRhdd&dma{v47O+!z z2~OVc(e3f5p5OjH&|=tD;d@U;>3EtcW}!CyJp5;udv(_cP57RvWSO*d6knCpv!!6E z`}L4H%tozr!dJ56-4JW8Qg0F|hTi~MgUAPO*sA;a`)>$dzeccl75S!vYmN~pXUbH} z8(iZTf0*A(#EB!EZ05+trRSdc>dCj7KNoBeVPM?Grz`?~URtct@Zp(|gHIwJg+KJe9?^!VnPEL*UZqzD{}F_LWQIiWqqL=M$_vV5cJ#SZyaW03;!5W< z*s;&JIxC~Ua+LEKjFA#c{a`~9CIajH002>XYA zPQ9t>&UHpGbP1o`ide5;kdF%KeNtfbea9WXU(bXjlBCE^QoT7iPQ`Sd)vXQ=KC1)8 zV1o?J8(AlPrJpofV7L!#tV>U7rdiu?9{$?5b^gdpSw_~Xa`IjaR3WJ)VhNY7^Vg%T ziC(4tS`=HWcRog`Kat_N*itlKbl@A~`~7gyf}r;-@#1=?=4(HU8tQ)-UrQ#IHfBoQ zdK>Zv13CMM-|6WE2OpJPk*y+~@uiKt?q10&|HTfqo9_msr89Vvo!q-jZY;bWzXPTv zf>hd-Sok1l&T8$|?mQw`nIWZmr2ZUzJB0I0dsbypJDabE zh9AzcMbVs%L8hV>OF%bA>4V`YH?uwcLn_M#7Kgciz2O>L>zhq-w5XMF_>43AS(eEI zSH{_gDrdOP^T?sAuWc2^!QBk~yB9Zj)%xRyoh;=cyh^Dt<*o^c6Nxp1<+n?DMMURteWOIp=_TFVyT5mk!si_bwAYbk;(={nQ zwY=S6<>(%p@s#M;m@W&0!P}g+#$n|GY|3p~u791)$e_v`F-kclxv1Mx!XTA=U7;`> z+WK%WwC%wgS`)KMFFazIJ(`%e@td2brj6u+{Rh!*^leIywi&qy@?53(FVhxbd@NF2 ztBV$GZ=()wMJ-Z`LS;km<+HMNNF954YBI>C`{@JnA2(naMBO1o!_159ue=L!TdQ%; zHYlHIJ4CIMD-`mwTws#fcy$l0yu{0+oPSg8;p{7gLefrwap}JqLzdC%ZU$Qsz4E`t z#fmXK_$*8zudI4fQ2)=>?gb$Qt`89I2^$7Z_)2$Vyf_Kzxcynh{i}1yoV#P-8>Q?Z z16xJ4)-yqPI;o?Bvq0K5fE~M)#q8j82&KpUN1-v7~5TzLBG);@BuCRl=f9M$q4SC=%zu+hnRdH3=9RP>5FGTTzhj zYMm*9ccKTECwk!u+xg1B$Iwv>@}=FPAMbWoejVlPqrSJ?yL65@tX%^himc0iA~tpB zV3~{PAXczdPe8E(9G=vgh3{;~#T9)x*1{lk%M7`&I@{cJ?qEeS9_RLw(hvhN!8ko=}a&bMiBjf ze2KNheQ+!z9@r$xW-T(crba$x6zJ%jDcZFFS9(K=-SgsQI_`g04FF?ZCQNtv@8s znbZNXr^~Txgtz(b$v*}wm%%4P7hY}7(>!e0#@th51h&9;{!F#-L53hm6_IJuzIcNt z6|$F3ZQCeYac}%Ke^Z)h@F!nd8XO6VU254^uC}~MNVvx)moTL+cP(vOg2Q~XRAX4ZKD62z{|r5=Dy(NT z^~bNh(Yhm2lAz`sOkA3ww2hOtw}m6#^YyK$^8<;~m#s$$>K>0QS=`*LItIdsY&bYL zkrj);w#UJiE^mBfI|Xnf^PstKl3_$)GSN!m4Btun*_-TJjBIWh25mSTU3b#RB|LkI z{QAS=%hH;6xipJ8W=St!`ZPdYc9@77!r9p@+hPEurJ>Qa6B>bG@*l15*_%Vyq2S6Y z!#Vn?ksVVsZtQ3t@K&r7IsD70yIvQZaOyJ)wV$_7>|aV{7cA^-Ia60w&za8U`_}g? z(nPk_>vfsK*aTn7(!v8;a!EXG&_|fUU1BnJEQ^QlS=F{$?#mZP{O+X|6)C+vMNU3B ziZG86@r!9x9}LCH)rXa%S5zji+-Mf}_iz2ZTxPYME$oi7%zEHqMkI6~Nw^4TAk1{Lgj_1}Tr=+w2jVlWn*R!xTD?fjBKw}(-!@98m4En<- zBs6|Y;QDyrmxXJpegsQ-(2;gDjEhBz zz9Uv|TKnvuh5I&pV1MsWSxTt=ANyM>*tXCO*iuBbQG6`Z<)`ruPgX2B%!{f({LLD; zd+KXrKUY=)-@J(dVy(Cw45bMR3kwVlO@Zz;7^fl!5MXUX%XJa9|+BkADNT4U}hay_2jqPI7)byZd4>)?a@`lDWSq8fRN2>VpA z&CbUi|0GxJ2S}NjUi20B-Y~sCe=O*vHQs1`lxG0+;D)bX6@-MIgYmhrj3p>ufbnUhMpTNPEk;D%Y)h7)51o6*wkJZUsczpi@Pp1f-=!q`Nyz zwjdy_w9+lojiMlpYtfD5TC{Zk=e5}GbN2h3=lSybJ0JE3kLzCRzOET#jxpvVJHD^_ zHBa-6rM(8O3y4EQME>YE=a;qDF^`M-oL&)UB;i^mKWR^1JV43U{{X06>8Yv3QkYGjp8#=4DJ^KluqP0*E;89u%zP^2nzVJlDvy_@MsG)=lXy#z0A_YKEn>Kxj!%#8TN&x9~&YMIxQg~qH zB&;gic)wK%9Bwm^Cu@{D=XU1k=2&?yMa4=XG-QfMmZuEeV!x;WK%u`f)1IA5nIGy_ z8``;J1=s2fTYQ67{a|~ges@%2x)5{aG@Dj+PYo&1OtLp~;m(9Gfb$a0eW~_2GR%+L znK2Ro3lDg2a~~hqSQeQ9-NqGc>6TMOccg%HRDpL#k3RqfXMw3vCCdB0r4y=kq;kIP z&=R74{=#Fmy)_mhBDIU`?6SeiDk|eWx8eUo#B0NMyrQT`vNcOE^L_e}l;Wxmgn8me z>UVn5&lcaa%W@0AjwO$-|3=9tlv-J-11gU&23go`IeyOX>wBQlfyQ441E!xdr=9qy zHDKG~E?B3S;V?-W#nyU06Q~PW;-`KgB5VtY{WvZ}%;1{VOa_b8Uw}OVu>j54Q8_?F zlbxHZP*6}{YHZvFB===(+^#ZM)|P@8Q#gha{+%|C7kX1z`f*p+=9}ZuLZh}QZ%@_n zk}Gqn3LlQ{(Hz2W_#S#*HpwUQXLEU3e3L0A47u~Z>Em_zfrDyc|J^ZZ==-}e3e> z-HQ$lg*xMyKCB<2;I;-jm}W3IRq}wBLNhMcYiC-wuf6(vXU26IY3W`rvdW}?Pe4!$ zJT)b-m7XY}W0{U+%|+biU{EA?qEGD#{!+%*_Yfn%3ua79 z+CXk>hoIux5!l+;s08v;TRS^ODX>%qX9Gn;!(<2s=x4X+FIE8ag)G|90KqwM!!1qE z$XEfk*;I(U$h&0t*28%}<-4YP0?BbZ6<;|Gih&FZG(=hqRUIs-{ZmQBzalxi? zrIT;E2AK~#*|p}nLyo0YR$5w_aWytJ&KTu*rM-Sms<+%FuWt#83-1j|3o;8n<#!!h za?C0DD;+!+pYGWAIpjw27yF!~?@>ELR9XuAuV9b&k8Q6&p`Vgb9F-q>%pPh=q)(?LRk-B1PruvD_qDP$ zXA$osWRe19fn7<_(-&U_2J+I=IRCbfj3DrSI6$CLP&=gyreRiZpQl?In>h3@v|mQh zn(Y>cB-vj}Ns zraga>fapWmX{sa29-Dv7BXnV0aQ^fKDZHTiM{a2s&7AFS;$#^q`re2*^7n95VQAg5 zr=>fzsS-|6%i)d!Mw1#8i!}x-=~lU{Ur)Sme{&DeSFP_?XUN2&_EAk3DyDbXd^(^y zAi@Qs#1vd=7ESvv|!3pCT3-L))@DKUO)vqcAETGWoI>$@vV(DYVAPjGbNuSe|c#VQEM7` zY7SW!MZ8@?yD~j{)P*xfMLX*+Ym(G+B(<$GE<%*NeD&($b}Nv^DX%^Hv(X0rgM$uX z!FK0fyc0gGf}kn78NoEesL{Bfnxo4_rnRqcqY_V#Ss|i-Zy8R=U=v$g!|VnCEXEF6 z^W9ev3S$*0X$X#|!;UK}E4`&FGad5S6nQ89T0pFURqhH=co`C!;8z(LnT|Xo`IIeK zlxNU0>*!A-*jXWkKHu`exfh)t>gl>Uf~lh^_~;+dyP4hoNc&dcSJ`OG&4`&uNz6PG ztmhOUu@-zultcTv(npQfL5kBC*qwDXD((rCPyB4X`e3tX!aqK{o2<*wu8p!F z%W7e1hcP2SYl*SZotk=&b#+fzoO?kQorxWS_`@W@qCJsn{Fm)CoUs>n{4?ek8_WFwNd-t*Jd%Le$^uCc5>4+% z13x79M;lW2R&@6|ayij0ExuJ|%evey!Qaq;#xl{gDRFmYx~kW+Htk1 z?%g!C$<4cMoM}IrfcfSXvP~Q_dh-OPB1cAC9g{68L!v6PdQ7!ykvu#*iqU^_uE#>D zbhv>E0KJi$5Dp{3BOO^^WoehG!s0MTN_}`aumnr06il6(HKKW2;NxC8(Ui$u)=+Fqtz>$4yrReJV2Cf zNqd6|#g8Y!0JdL4w1Iutu}pwTj(3V&({EXwm10thiZ0owGAMDR{zNcdS@B2tH3_v_ zF(VgO4WE0iy!h<2ceC@(WUuwQPpapXlQQpe*|mc;Nk4sM=vqakDEFSR(R(G9e$IJR z(7`Np>E^0sy%CgbNl7#dx7-0Or!9Q&>7d7Ij}e^f5b(bn$^SB?X;qN~_Chkjuu{rO zL+7snAwipkwe{X`_R`y**|+HcNHOxW3C*EHlr(a@+;(2GOh>8ojXga7$&2!zGzY8F zsWHA3fS1*~6qJ4~+EnYb&y_7)4Yu#ID8d!?S=xEo!in&X3hm&T4zu{WxsZXi zEx!LRZblZ#J^l+)A)DH+GFtI6*Qk$*YzBRH+P$;O{k%obFOKraKvJ&)dHspr(FSt5 zbJ@vLyTOT|PI64?dkaV4Qdn5n$G)oAc+sNMCLSIY4ULVhLv#>Y7Zw(liYVCb#}DnU z7WKly8e);$dO!tQ4Ts1q(?ZxSF^Q{(J)%Mn>j?cG1Ded_>y z`5wAkYGm(waJx7)nY&$`!qGi1fA@FX7;V?$&7%5i8#d2tSJkE?4&~DnI9`Qv{L%Wt z2z&v%(mmLx(E%pSZo8{b4j)^_XXM0I&Cbu~cx{i;MRJ)0Ll)(R-2~5Hyl5D)PR-3- z+|C6@ke-no)Mn~x<_>IFm1T5ra%dsErsaeb0FBmtHV24`DNmfxoyTcIj*wRRl_Q4z zBhCMw`m1=r)E~f^p|W>5b8+1WV-i2g$6q4fyYS3@=2X{W`*c!T)oaB&QBi6XH9ht4 za2+yW?~jEA1S#N5q6w+t-i;gH*%hlA1K>Z2im@~2zIjf;gBhm5Oo3!eu5gP_Dd#1& zN`5NA#^d3A-%~`x+$T?+YTeUPR!#)HSuwSmvhvtZ@O&cj{zqBOGO41llXVubJQZ^J zxg0>)P;XPxAY_8w3t8Qf4fazx>2^FgnJwr@|HZ3yJtM}$)*-Og#Zeg6Mcx8AQ(~<9 zPpLJAcDetBuL9NDoHbBynt3J}d2-sfM+Q0sGsTmVk}SU+)jmO}%%)wBkWw;abg=l; zyo2~`3=11L>L6apy(hbQeMnQ88Q1m1SIzt#HLeysccU4gZ9u*vaM5D3WwF z^EdT92Snm4Z*4B~n{=@TSM>1H23?ljv@+?YuE!gBdTO60fM(TIsaCmV<9rUyvsMQW z5LFL=2PQWxxIhZq>W!`PK zbSfd;pXA{Uat@M|?citfix0KcA3U_UxJdi$h~dbuQHq$`7p_Ca<9unq+xP~~PlW5g zGaHRY1Q_i`bbR43Lwh6Jo2&0r8$jcxu#K8!uw7%6f}OByw^7IF9r7N3sbX6^NJM08 z({IQ#_k!yJl@w(}r~j<^W3TZK%wuhWM_adJw(-$w$9Xajq|*t>7t~ZS(ew2Z|9)#9 zv3za6Dm@CL3XKAK-T7H&)3wDX+gSic+lUzw5#?up{HW--)x|2oPi>^;m7bHM=Hjxk zvGw%nQ`mm$xqr|4hIUm!)6@kHUI_4lT1}nIM=V1vrxRLxMk%Nhx&gw+UuE&% z*IzrgQ?82#=M+cX(Lm#3noA|O2>8T~HJ%9ZvpiKPdHdh7cB-csoWs21#?&$;DUoO& z{~#$g7c`z6$IQl}Vra#8J4^c&VJK^F0z-ryefeC~xo_Brui0xd6_kWo(}T6lu>(Xm z%wCYwWy4axlr7jLqb>~2lap4k-}wee{O}un;1IM~C%XF%DmG?bUNx}&Qvk4qn#_li zimH{)4an{^yQl;7?ri_x3(mhNBoVCL%voMq4MP(GgI2y=1}{&GcDLuZ2@r7DHb$Du z>7D5@8?KyXk`B$QjiSRxo2lX^R`m}O4a>n+2MictF2lMh9Bjn@ewnG}V0sFW2k-LduuT1>7Sc9`gc*xI+U4mKac8)G4^KkJzWLjDC$*+vElaUddT zbEWjDfMh# zi5n?SndLXelLoxP3Wex9;v54=o=K0=1!sHRKRRdV=~?`+ zi4BVCcqM)f)r>BuiRt;WV?b%c;EM-y^?)ki3&St*t9n^ZF+L<4(_tG4`9&Wz5dN{_ z@zu1DzoqW!Ht;OC5YGY_K@6-`=vA5HdJ8^0KlN+ojp;GX$}CjRF=y@c(TxeOu?|SN zeHbUo#9*RWby=N_eOuNzxJA_{CVb@HLN52wp$)2yIA&`y&F%IpQyf6UxEJt&FD+e1 zu}|$h4{^Z+B_Td9xBxbV+5i|`<&WML&lrVS_JFNL`5aYLx{XI^qp5{v-t+%^`e)rG zv;S_V+D$1fTrHf^2ETCtC8orV{!+W{5KUBSUyff`6&*zb(H_eO>!^#4LK)slx}~pp zS1Z-I_346y(*x_@exe~9FYnd{_8crC8K6O0J2${A`~tVv0;D>$6N1lBoaoP=qMktCpvZRSX$Qzas*F^K0A%@BtpYYzyZ z>&t?ICs*#>octqI)`wYL5kxLGAMX2o4k^(3+JktG3t8f(A4U* z>y&1fqneg;JVO~!lLsP?5&rgHhc`aK-E}<_P0)2?ghsFJHedK%!+q(-iZAJPbb8y> zB;$lRXSYKhIpuCz-+s@9t4eyeChh4yJEXoK2aQg(Sr7G#Nqsb8?qvLXuX0bY(#C#|`|O8cTm>2xj*M;Uq2fN+ORUKds|cvLEz zro$}}mma#$;J-u8e5HM6rtK4=^Ud)0asQQeZe?v00#D3U-{>(yDTA;3X9f8A{f3$O z&RhyNH~9UnyG;vhf6;FV42BzS<8j$tc64+YV9b`iN~x2W`i{djoNF81c9JLBt=EfK z%?#$*kxsW+wED(*k8(-atX&f^)C?WL|kL<4U znRq4DF?Q4D=t5p()aCsb%VApzT*s}kRQTIUegiIwOw25|MWY37I31UNIIcXbd;W#J zb5_3IOM860-+{9x-v%&=aZ=w+42?)a_NUc>EP~inUZ*ot{@RE4J9u;F_&Lh3_ihp? zECqrx#0QDm&i^l@bJS~nr83KU_cgmKuVJ}&?qmnAVDyiHC)@cC?mnp$&i8@ZKtM$m z|Cr5&u{^g$AkP~k?AyJYc~IUEZda1K7`MI`o94_PX;O+zo94H(O7=RF8*pNM{8wkxWLu&2L6pJvS|3+ zz=oJZVY&)K_~NNHq5*Ce@jQu?RQW(_0@W$@uh8B{A^eZfj)tDPPQ<}@9mf8^92s!Q zE)st8cjNyXjOyKQDEa5FcqLhIO0#otm>-+v7+6o``Y9Of6%Qg4GXe|Fn@mByD-)5v z;j`)*V{*RnrD%Sr0>XJGqLvJWkgxB5(BRnrcP54IQ&&nEh!RAqJK1JES7y?wR>oTv zz3-W9KgFzwzJ5~nHkP)WoRr#{nV!C8dZnJg|v7C2cVK%AwUK8~f%ln=YdLS+4G|P6YPC5j3 zk6B;+`X8euFVAT~qP;O|N!<#1i|r(3m(To`iy?QKBCgSfTGSX>~+)OHbtEA!-sPFMX2&OrxVXqUol;`+fmn1okT3OM(IR z%BRYQ5p+|n=Yy%!V3J`9Y@6pt9Aiaqrv0K%!SlwOnf)h2fn9F#1JJVOUsRTQJ%kC3 zL+l1kwS_cUv_(H(=_dyvhjOkJ3dM$V=rKlSjzS&Es))wfYo`(25k?U}5p265Iv~O( zwdOC|;UM)l!_vFZmhbqEzEy7Q*81zCtW?oIc2)gQs`X)7Q!^)!%FDG>Dwpe_6Rcpa z|LQ50OPs_kllU0RSV;BRVEcP5_;xHpv8hy`C-3+(i|E2%84{K32+ zj3v#%?dZa_Xl!o_=6=$`OvIT|ujg!JlPYF2?`+a(c8Ix;M>DJMBw-ZEAK@xex%c^j zH!|>%=B{2=sO0b3eUDeIJ4|w)TVL}1#4Fj3522|`eT2pyg31;(TNZY{=&XsFr+?%6 zx3S-Z*Zmo^Pth6L8hVp|;AS!3vwd!`C)H^<=_M9WZQVh-SBcM#VmkcH`H!3H*9e9C zFvH~7`K;cc9N-Wr_dPP(ry^dzqyl^l)@#pDV!p8j78D!r`z9lAjsAl;j&f{%N++5q zl)yxR&dpD|7CTD6Jhg0ZaD3|1VF7LGNiGU%N0a$!OtP+&k#F><=VQ#&^N)iI`r^-O zFMr-X?s1ZUb->a2L&_9NPY|9I09X3sQicW%#gtVCFA0-zOSiJWwBJMx8kNEki4goR zysnxwV&L^Kz_4`s0LQ~~m=Ooe3r0*DH3cyn=5<+;lxWC%Y*Pza%)Y8dyFUm+Nnj7> z9d$^~g^|VPmlr1RIn9(onNuHb{34B3ccfYs;yB~t+USa?ok|)^|IOuM=BsW6IfYL4 z6(xQ23gwdkJ`tnV*x!~LJKgx;iaH6HxH&NyIhwko{6z~!VqpV&uIhmY#A zSG#NMop?4ZJUlWYH6q~`WfMK*Jn+j`Pb(9c&mZQ2UmoCfhm5rik>ySTSuP#8-7!^D z2!Pm6FZWP-p}{^3suaGc9(b1~9+8f8Nr&l8rIXm@Wy&I?UifWbQ@Kojj$#U&pfB3k zc!wnfz4#EbT*3VD`(c9vy4g)nB30`))Dx*T@8CeY#U;RfOI5XR|5sc1v-eWe*@~bn zQho!D(PkbW#CQhcbMk&q;;=AwUbtW)-f^X>2c^JF|f*FRXtiKFPY>5**le%b{ek2~V5`~hC{Z~ie+SV38>2L0 zAr*@Z47|P6#*g}<9BIwiAbexY=z({bb&{lMuRjhuT^AO%7uM;k);sq?E>0wCbJH~` zIoVYJJWq(?+8#O?>&vB~P+=sY zyeAUPh`L5B5OgX|dYUf6H&43`Tg52rc?*BFy1$&rMV+_~47=@N$3h0+aUEA-=!y0r zgLz0IEJXv)`#wTyEcY4EGsb?pmeu<lI1oJeJq#B1M-cfPc%f{ zj{0N?`rwNG-vo>YuZ% ztf|9F5sMS+C(_Rs-Dh8B(K4GI;m$+Ll)?aFP7Mz~yE9B8dJl%SuoqQ3 zKpRH;@K(STB#U+oxFo}$2iO5CngMq*N2~Kl-j2;)B}ZTyc-O5WP%;%2Q5@x;h&}* zw+2G-H)fO~{7D)gSy_~jN}bxOGldW)TawJw_z3Fm18-`k`2HFmRP>IH_SltPAKixo z9}9@Nd8lh;8MfteSzgo^$&BWVA51E(+x84=g*#{#>ZDbYk~(9O36%CfV+ra ziP>PuG>6IF+J_q+Zpt6$XkyZJa`g~R_C{9?;iL)N(heCAt}`ht8RzqEH8pwdSngiv zl91cWPuniLS$olB#!LF~yW|uSE#EFKYv6OSva#I-1279pI!elPyeY-rFnpqWbj^O{ zTS6~S=cUfl$EezOu8ZjolqlHdu2tpNRnx4nE5Y0Cst>gD-U0tQI~H3K1CZRtCi#_! z8je{H?cQZp+}$9%e#$@m)i29#=br8y*xvT$S%|#u3~TBDoHICAyD@%M6WiXNn4Zoe zE+G*W3fKKc9m;)G^CD%vcjj;)X%&Bd#2D)L0l`ZuJ3l_lHUTsxxqs7l0D?3bU)^@3 zV**Aq=xlf&9a{fa&FcVnHjDcg!#u`an}*AhqC78&uK{;p*w;mtSOP4>^I>fZjfnew zz_QE7y1^u?-m+*KOeMLQ1<_Z52;>-@+LlhUq2U&PyV+49Lti3Ee7e#mEz(o_$rIxH zsNIp*BSW?8oZ>&2(6d9EV}N>}IYB-AqKSx3;SGR^w;Hzk_wy#B;^%ux*n`>O@VoZ| z37|L6C=lG9(qYwoT89lcH@E9dZXE<1G(RFRyuG)SJ9y0;RD)D!zhQHg93Z~@wklfH zTU)*U8zFsX?LWn=L7w2Uq2c3j(Ag?+%UTM~!3f<_@a%Ah99;;H2(!mZ?6mXtzfAC) ziIpKu&Bu(Ha!iKr=Nkuk0!3?D-fJ@iZs|$_-Oki|XECv-uxKKIz#FokdJTIUmn-Hy zTE%*x>DS_~xe^+tn}#aok;)L$j|G*zwZQa7v@}SOXHI@Y$=>pKGS>7bE_03r?9@1J(5Lm{TBUT~~|&GkQq@LENlKlir(}hq;nJ1JbO26c^%{kP7kMo%HMwM|Aj6MaWpZ2aQ59>&^_>MIr< z!eYg?Tx7Gf??mW49=!$5gwirHOZI4(U|}1ds*&M1FI37cT(m>4sRAk=^@l#R%2U)RGHBdkIJ{OcpHHz=^#|q8zCnX=_BWa{% z|5TE``{%_snzxJ`eq1-%8CA3ipLX6XO01Wc6h87MisHdRxW?V34jGs3z8S`G!Bh2S zR`w*5ug11F#B3lrDjvx_E4e|E2hHql)6;6e+Z!H5iHtA}tg<$vw5RhjWbE8{8z=+Hw-8u&VlJPC2 z9H90CVU2L!>g&IUqaX}N=rUJN_9 zOP0d;3oV(1O<4^!>W=JMiXry05v0#x93mT~)76kq~7EEjWlxOoM2^=T<3u|oN8jOusU+Eqo_M9 z_+Kdm&RLO%?sxDHsi?fJYc+M4&^bYIiV4j3L6ClaB zEEy?HToHJ|TQzAcsw=Qp?WAA?aD69tC`?+v$_JQ^2M85($L#U^l2{17V07pf7Xcz) z+XV;1bC{qN00b0!BC_j_SIU&)CKc^D*MvFCBXgp$#>2+qDk@=v?n|;2tG#Bu5H%9> z_KYBTk{fk$U(thC((wDqN`C<{2~f9yj5Hr)P9oh-ip@NA8~5X@RIa~ZFr6z_22q!- zONST;6IXP@OaZ|1@XRmom|V=$;4-)u4d4y-i)U_T;rFjLas(TcPZ9VgHIb-mJ|iNP zKVwJn>zECaNue+ea6`&v#HSW6b&EYR1z8}YA03hm{YgWshUj5xjg{FJp?9??>a_!!NeA2bJ ze!+-}cXRNOsfKsa(#sqe?~~*9VoId3nJ}DLwxLJeg0#lGA0BRWmXF} zh0PyMigsCcLJW6G^>Vp3%Lrx90g=xclzi#w>B~h`JF@5+g5ZV0qavaSyvMn`cHIX& z)&_iEy;^=03HhJ~Fq?u>i-s`J_`Te5L+E^DG%b^tXlW&oNX7{rN}CHcN7nc2W@M(L zH@lW(=@k>e%jYH~Rq^Q%g$`(UI>(R=VJwBM-q>3A`VLqb~qA^epJo z3bxMWjyx`PQ}zY@H?%1Ccprg(L+Me%X%g^3^Q@848Zdgf?Bkc^xV8KnR^C#oi@j?# z^y}xdWmAdA#aMdTXE*cghb_MLMpXuE9F7^lKU#K5N(?f}c~Q1tqg%mbvHe`xrfH-2 zYzCKmcIcu~n&!j0Dzf;Hg~%fTh@TXEnuaE;Y`#(xr3P3sF+Y+CeNNFYYSQ#MfR#mb z?4LgTtey^&OfCNXqps=T;kOI>DRc%RGr%i@dGIfD1mOh5`&nmVG-0OU`p(|jf4aCr zs<#Y3;|0vzJe{B?7ZdwilU4%bEViu;hIx~iQMKZzHFtw)oST%oXuvTFrT+8R4KH80 z(uT!BLg$1*PmUo4U1rX+X1&z0yQk3XWjLL1godM#uA-7%t!l{rE{Hzl78m~T^4NPJ zvN;FSS<>amoHp-(@4wiz7TY7-n79`s%v}I1+u(w z40JLJ2Yh7A`fa022`X-QA+-D7*>3=B5@5URhuseV?L_}L;EAYLvgAYU@&hHqB-)Sl zqP1NTvf%Vb3plSO zay3(n784H@jrnkWo=)%qO6*~;yVX?#6zji**D+x*kN~@mSarL5Br;mMr|tXNZX*LI z=~-G+TiAthX4cZ9FO>BJEvM{e8kc$YSY20(x8MRNTu|qvOHpERbUB@oPg4rmi!@eO zXCrNc{t)#$S!Tn%?;LW&sGulbT0VzA)uK+m?3Mgfmv*2mRpO`mpi4$aoF8j@Sq_b6 z?JCyl6Vm&YwXs7tf_*6YKw!ckg~Q=CVL2mb>AduPwzECvY+i#Wc&rB9zgsklMhK!_ z=x^!$h$8fV+5OTLA=;_rt1-&5`gPl}d;4}P_}BUgW?**(NFyY;2}-+ja11ILi^}C% zJ>4bg<^*xqJY{e3BC!I%FtK}c#8Zltt&${zmP1$BR85-JOa_Sryd^04j@J}VdF`#h zB&w|-`f?6dQ2n{9k$limmguz$759c{_0V7paKkG6WE8K+Y&zXBc-Lnqt~7lO&(=Ps z$@dp;%*Drd^X!g6ofPVv8pE_sl^(`#bur;59$)>aF$0WWZqa(j_npIJyW{-61-u3G zf2|HAW1yFNgw2AtMtL6`(X7K1x*l20q*p_Kxukfb>Kl0#y5m=?=I74;_n= zU&Ts>8ldWtuCnq2{6bQ^$EV&ARr!fN-~`i*efy^l3(wsP^+6f}ZhVcvW%B0HC#%r{ zLdmbm`;C~l89;8JDm%Iipq;YLba~_ru<}BRB#*t?O6YfnA0HPaD|q2}RdK-f&D`%M zCkuY#9gcRq8i=U=-q|%*jvI`t0d5^6((j}Q*EoV&G}-CZ9Qdu_K=FJYv*|Mp_xe}k ziIt}QJ3;?wrGv8pe{>CibxBqyv z6(IPaG?eSJM;)>C2{;<(vOOCi7{>wPhHQO6MLRg8mYKd8E)zf(b~Z~;f?+xe9ghZI zc#G9egoN;*5)W5ZZ%JWi#joG5uu+(a5tBxcLtUKEkR5y~UXIIGQgo};jMZcsP!<8A4^iLV((hr}h< zDml&=NR{of(cEY6fzo1&L*}ifbg1>WjV)p5UfhH(tk!1 ziTYVe2V$E>>LWu3#cuA-^)T)G@DasCq0BV}nad3_m#J1jiO2o-p1>u@48 zxZAuSI%ABoTMGH?CVgHs>_uzHw#J4=_rsuIiP#;h&Fd_pk&H4jS*1*gQVJ%D)>U5S_Af>wm6q^_;SD2r<^G1di2`eT|-)NghU9}>44N0==9$*zul!Xmt) z5CA}~_4&ANQPn^>nJzS1H#)NJ9e&4TGA^+-Hi9xEdi3!L4DYTC<(l{DbDgWBN1Q^~ zd3I)67Z)w1&QyoJ1A>A#NA`!0I`f7UxY?PDG@>d_^||0<5y>y44woa%Y39uK2%>+% zA4%luU+Q7jJXKPfy!6YO=H_EJe1LfYEFRe8UdQD;D zkSpwrLrT!Lv&w=^=u~#NQHh}Cq1YcxvwYmBmFY{l&vHcwojMnTht@M|Q zH3)0ZpI*z;*iSeZUE#9BZisEt5gRPBQxLjAQLkqgAdwt9z+$kV4PBL|Cy@Y9+C_l0 ztc5uEfTDB3UTV}Yptf}Q0=GZAhF$#a`A26=wq9bdMHihE=DgUgJ478F4^hudS%YiK z_phIRppZr|E2V49*h;UUalN@~xm`XvDMOc<;=}>DNYf;Xs))*U-L#&s;w~4yV@Umy zzOTvb-h@L{eHu&_il03@1d3`FW@agWfB&Pu|GsDs0pwKA44H>xz7~8uZZt<4#2dyA znIPUpGsC76QjynEQ=SlaDCZJ6UuN5Y=Nem6k%S3O0AV5cP`gCGqB7;2G0fP6!xy_I3uLfIK5Fsz4ybu>W z3V1laWkd?`SqExxO=wn%V(n=a?y2Vh=FHCAipco1>UXRw0gK@Vh=J{_s`p`FKL8;h z(ZA}o9GydeW+vvtR-)7%yMqiDtO}tWaUi7z?Z4~DbBcV+0fF)H@iY(z^YnFX396{7 z;z4s93>;Mu@s7{WJ8;4fi_GR@=}oxd(=ju*XgC~lpA8;9N&l1^3N`SmGL(W}%$w{p z0?4M))A3O&K0rajQ?K0NEeK-p<#*|oW zZ?|-C*hYuj?t|x!D{5T@>?U5l@+vglHIp#3x6dE+Sks;bg>9FPTb5Nzwof;C44Aij*@v$^Hz=}CFzfj828 z`_OsbfBg6{%{B`W(O@driUKrI+-iVEA-Q z<}Jj*AZ1@wt=m**M=+luc$l@{6yA6M!{P@Abz>1!TucdnA2@Z$TM#BskRq4!Ob6Gw zg!$6q9)2$V;81pSESzYG$siZqZhNM7kZ2(^$G3F~P9Ji!0U|*JwVSpgFI1?RVE&m7 z{-#pM;pE289n>9o7Xnoe)oX9LVf3C~%WRn|>2zEML!Lr*Px@{uSdErziG?k1SS8YU zg_^fC%O9O*HW0c*8ppZe(%e-@1Ym8;Ay_OD0+#?hRvH@vW(UfOV40l#k1xNUx>i;W z%7ygz?|;1CqIH|euSJuFg_F_^$oEJ{(&a)x@-0(`97Tu{wOTQt8N;GuLTqqY8# zLM|WOLP&_5m2q$Hz!#AOBp>(S<&w|vF~-|RrdRc(2(8cZt1E;MyUhykXu**%h_GrD zzqTO!AA<<{P60vdk8)Le<@b8T3phqcF{QZ2ze`d6i-LtaPC=^ve8}|~R{^su2kF8~ z&pk^NN7TJS3(|U-XMX!lY2d?_is;zTqeIW6QyPR{Cz=to5hW9VK5T4kKnzrXwIqF9 zB0@r19UU!9Lh=hsWt-#p2zJd&QlK=$Y)tqc19F9?=vGfF`bpSzE2BZCGIa1IWy7<^XY>2$h40%w|7_i<`X~AAX-=2Ax@>}H*~p*Z zENv@1)9>HrqJ@sh(B7NY`6^-f8<6;7(3LuwLcCB7{Gi_LJi)}q~7Av7Q zVd$hn;0cwz>2jwJ4G&L_j;01D7glcWm0*3*on@th{QUe4X$q^bkPv#GnhR2;U|s^M zQw~){#rM9xN56j2>JM6jwTyrTO7ld?2IpH1iR`rQZ3i;oUQk)Kk@EJpoxl+j^$tI% zKM!>ws^HU>?_wG2bVAx>!PXFCA$ICa7khUi@{Y4kD*ZtSMl? zmWuwrA8RMK#HaA+>o)&_yTVw)dIU5kUfl1j0m!FOQfMie&&#|TM`#of3Q;%5V;Jy&bQ))I@?Tafh zZ0h$5j@;-&z$G$viO}^_Zx!7uhi%Cj^PX#Ea{7q&^o>5yxGsPOq`XnI&cg@DAK^tH z5>I-Ok&;?C^gqu%MrS8yX=zzzmz%#xApt8V%i@j=0x^nUeLXef(AoW?EnHtkBg9Ta zaH;j%SC){;=ZEfxedm^vrJrqQo|Rf*VNT;^A8N_W>!K{*7cc+>DNdde$=g!Bx7Z2IWlV%&1o@K^ijgp$GBm>S)IzLLc$F?SYa*|S2 z%{pLh0P&h0#MxfEOK}BA20ZUJg!S56^U8oFlfm|2Eyc>oDGx6N!Y63PP*eumgK#uA zQ9LW?>pvbf%fFOzD=I)4Tvxv1?`McphMjzeds;eOTd4p@0v{llC#-y`kKG$L$W@;;mC^C*niw1J! z>qM}Sk%zrdt_z1yFnLHuD%jHU(3jG zrmKt9{!3k*)3>D>K_tkHdDnu3NYI@i18Oi#ji8Q#VKlU_;Kao6RpO^a*;@gztWR4o zO+u5Vdq!%>8OWupm~2`JO3OVoBx)) z$>)pA3=K0gME2KoI;t;7DW%ADz>d~sTuPiNBoyR8G+?Rb?p_W9>3ERi)wkx@X%$4j zdE>l0`TqSy((~88j2FXyLF9==kX+;%jkdz*;d2Krw0aTY55j=I# z{UklR#8-Ra7eks_$#0~LK|P4+>7WIO><*#RmP-;NrR0)S%}@cYBMBLG+cFNT9Uy2u z?46jWCieBqmp5-O64{%8Gq}gLO4?vOOM8k$gMm|irG^5)Ug&K@#~ImHRJOg--3N4} zc-AQv!m%h)?&O|ciwZE^C=}xEiz(Xi2R2+@$5NQ*O zZ%#}cj>j$mQvdn$Ek-6LHvv*5&^ViZe)jw3OkQigtIA1s1*eQi6^%ZQZ&0oW4~notgrk+r;VGJy(Ox8n&lP_?Fg2MUD46_DdHoG9?mSA0}#lfN_5P8oGPnZ_>P zxFag=b2aWTUm8a+&z^K|p7b>>t@f^pI1mGvv>ksr`+~^+4?n%)-jybI)r^(xT~n|L zisZLTMpFr}ci8+=X$FPJ$G+J~tH5(tZ-E6IGHLSs84?#=CCbNle`I82(yGs#2drLh zfpQi|K*8bcn#sT@Yy{sED3MC{cDEcRn$CmKkv5&ja!^_h?$+cWF()%caZgE@3i2h7 z{$1d@Rc0QxKo+&bs1qIhjF$=UCnayU5DT%N{F6aDs=JxVlh;9I@V?oU`(IpJH2ogW zOUr51TR&;#()#_XJ8qkJNE}>lV>7SA2=Hpk-PTOuRCl!bmFAjAMU#`eLcmKJ*WjIU~@%`>e@PB6t^d8@a^>lxNxDxk?4Dj-X0l^)o}Z8i>Z|hVD_9h z4`k&7zq8{8&J6i`-0;|Udiq5SV7qYPX0Vgz+g9p3b@7iA?3TiQ*Cenhu4J?{v~Zc? zYUy3=)E>`psD{bH)`cHXbpjmxsNiFkrqLJvEE=df3l%h~ZnF%`yEXy8K4ednIXN&b z&4mgW8H-^W36B%b@bmMdL=Ov!lb$>|C}p(Q3OAd!CDZQx;{>zCB=|s%JbY&Qr({we zR+RpA!`rn1lK-_bP7!3SDx1IILyCMKD?QC*xIO;f0pHpkqpqi z<=pWCgIh;T8bTB?zu7JQRD=`!gSFreah(Bj+uGZR+e21?mqhP=`E_^-12XeqxDyBT z0LbOn*LxMHC*Zu4(W z?>w;e0tfDW$Sq7p)05p`SKKzAuZ~_RG0KyJv{Lhwat{U}&M7{5mbH#lc@hU?P)qLr1r(J!g=Am-K(j^8(V!|4* zmnQ^F`sS0e_E@>Q`0Ab}BvN^@`RiOh+_rtFTF^y>*atFsm1hMCElxuoYT$Z0$s`2Z zZMlFV)f^|P3#{a2>trJD#{a>?5vBKm4LcpL01y`BP=OIq3#VvDWcsFIMb&&FR4K}; zo-k&-<8#CqbZY>ussqKA)6E>7d)SmPJL$(Y{xC)U93p~>bJs**0*AsyjZ(s|de7~7 z{)Wk~dT)muM-?b}K>?`ZL~5VSDEU0Wie(!31EN)3{(9$p|NOHJTq6)00~D#wJYaAb z)xpyPn_SE-zA@XKw<_e%2c`{7g%E5P-a?zb5bx*3$lWv3YrhZ*zjQkYYTF-Ld06G% z5Y0%|I)#j997erqk1iYa01=zW{;wbVv~v42Wcii3*o)^mlB((uy zX!Wl4-bdJydS-i$edHlVQaewZJWC`q8Y5$6W!nOrp@9{j*5x0W2aWEH$khK{CR zQC&zQxA^Sa1Efbqs9{`KEorsXAP<^YjmUK^9>5X^o;g=}If)2F?Q>uShKduEeZht# zR*nS5L~yN6naSsuxQs>v;n!nwg}R64g>QNk42<|$B(Ye1E{l6kjN(TMn6%OR-zQVF zAj-$cM^{8s`|+EC>D|WhQQcbw=lMij+ahJ7JS$S*`eaQi%;r9hGXFVkB5*C=MgVsz75 z1J!gp^p{wpH2e{}3&`;((UHo(d{`ox58DF%yFx$mLJ$O08Wcp3M!LJC1f)y4yZg=s`<(wd_ug-eJI1%hVC&vnVg1&6-}yfC zna_Nts;z=^3~cpu@k^7C{M!xXKtzJeGfi&jMK^R3<&!mTvse6g@0G>M-S5kdbX%>1 z-dt)*i;qV2LZF(RPCCl{R9w!q5tYLzCA5I?6EwFE5X4UES|M@@zx`0{#k4UL&#bjy zs6k6hrWtDa5{Ppv-IGI6Sq+UCkzkSxh*8S5zSz5#J}90Zi0?5x1=C$C@M&KF>&S!N z50^6$9|6xkeP}@N%uED&mov+>ucD?P<7p{b@-uype7Ws5`|$NZFP^QrJX=hOxtaDy zUhUPrlprJ8n$MLNOtRr-)+jkaLwc{iSb}`h6Kg{8JaQIUenljL;s)VlPY^q_c=v-J zaULg52p{Hg&>epp7kB2z^9v}ab6rR(gP3ON}XyAFs zL@;T|u>}@1a$x-E$GbmuD?=qm05STJZHQI>-`rYqj_}u*FluGBUWV=5=`+^0%Gkl} zOZ}4Cyxh^PqL36C`l^r=@BbD=es8Du$Mt14V$1kf?9b2UPDLLF8CN0iwYX;2iXmY(XNq@YR>ZnzI3jTQ4+{4{fiy6NCfuK&wm5 z2M-!JB`y^uj|x@&(`GJ3Hru5Ie13OaD*NjTzh{x1Fc`pz@c>sWm5v#qovs?wY@E#GLGZ)7 zx>PollB)q$e?YORsH}`|G?0je6AghIa#NESbmd4v;fDHUe}1fbdz<91E9Um* zFF!$70k*+K8JL^X3#elGVD|HUm+NUT#tjS}$zLBUb~|oMkpRcJxL@iE2Fw@y0k`p= z52`|N8$-NQWbFr}Mgs2V#*m;A@VXWe5D;vExwgq{d+eE=o%YE_xVaO_e9ap<{v*R5a7t38ZKTua!g<}42Dd#z6YDuiBFUu=A|FkepW9k#B>!M_ zNhfj3h{->NJ2KW^o2<*!GPoC6sA z1t4X55w0&H0v{XN!#oeq$Ck_lX^!BoIk(w$74V6vd;D)8<#Kc!35w$-SUZ02-z!2t zILK6?MGR`*FJHa{4tC?foCt)CFhr(;mN!we_|B~I!2K+yyZe%4P8-)ATsrmfi(Q-U$4j2Mf%iv52|8 zfk&3hJs5~G%tYVBLx=L_igTOtL(k$R_6Pr9lRC~7Jj(Zn=h&nZ>YONAbtWfj_)wP^ zbw4x%cW@z(y;@R6&K-Z6xDUGch*_fezy(hWPKc%HWxqVwO`aQC@aGe#WiCm0X^jYW zJT?hjxRb$~GyYSJsl4M0R7Jq-hwYzc)|puX@B&E6^^Mo|xdu1B_NrAkFB5S>XB_GW zsZ>@I_@}}Ej=YgZ^o>GVs+o;6hYPXqbj8s4z`Ydcs0lq!VNmfunnys%{QL#a#~T1j zpo%1mm~6r9^z^7G0-p7{TMK_YFPchiwBQ~Ir#UVw-^mKZp@(6Dsw4FVO{{NS^`r0*!yUy!jiE7X^$gs1s>U5ex z=8qWh3BK|SZW?0*XD!lX3I?zr_>BZ4+j)89OiWYCke85n z2lnHXv9Xu$)6;YFoq+Lv=ljQ_b3be^ROtiv1A|{;Ec*K)-v`E1**4Hm#{>3AZ%STn z8v#J$+c)RplI|ttN)i3{C@`hmHY;8d80Z%G6+zKb0FhOlF2RH0tyiC6wqF_k0R_qJ zRMPnH@MIv_re|g(LcxU&%9-CF8H3`bG5Coge&7j-OqGFl&@G3UEJD9)`Nl6gYY3CGcf38Cu+{eBE0Kkyygr>L?c^R8(_**+x?}{U2FmN^aUDtQ}}VCScL4 z+dn07nhIfevo9R@e>OR0a!Q9z+!3eQScocJW2Yw$m?Yf&noMwVbm#RGJL@ZZA(bYW zei}331?P{@+xx4w$9G6wV8R!sRFJJ`l;(37bupnLuj4sWvW@ooEamQqRD+NOxwAiHR| zzS{;LPQmBG!u7(tJUjjRX3k6CjGE_h>Mq{3(38gIe&i?)LXaMw-C?L}edV;70XJmc zu7>x7Y)%_5!W*HNtaHR+EF+ljLMfd_4Bv(<&6#{Lo=#5NpNXia?`$7xwoJ5{sxLhP#>G6n7pNB z_N&&2uTYV8dln{ zBS&K-qQkDj$CP|8@AyGEZj=f#f9$g|!oR8sjbS$U?(WK!m=o{_D1rBv;ue515CqH^&+)vBN7rz%O%@l(JGpg-a=M+Jm-&|&Qjl4O zDajS1|{9bL&C5}1ri+t)F|Gf%x9o$v#f zcXWhOCAzMB2_*utrU2XI*=V+(8?<0YU@zS02aJ43c>)_3gIc0<{e75`a^l{L^$UtR zC{KZ8)6Rl6Qi;&-GVv?c$Q+|QqwcLPNhSna+P`~3gv9{1gUDcmdqexeoRS75MW`^_VCYlT-QT}Qa~GsGQ^v0=!VwZ>R-&N)N%v2PtPUr zQFRtpJnL0(eo`vy6Y!wqKXjhqF&?=p^q^i%0v~EM>mjy5n{Q9;+OU$S>wr5E!l^dB zovSupJWj*alv+l4+@8C4`<-S#Ka~a zRA2uozOD;n5`m27GbOa&NI|EZjq?&RdWN0NGB2OA9K=adlYiA|=``sJv5M~6p^B%P zXB!P<-aK}*h>X(I(CDF{f4Uni^XGBaJflb2b2T+*;l|)?3;FcQ+h!Tcnu_I_rXn{C zpUA{^AIZ4=0FQ(RDRbAu*T2`XARY+?MP%{;lqdF^Mn@}eRKfkGEY)lBy>L`#m~t^`7G~R_}~2!WyPmt^9$1;JPZ{j zFB%%x%T5j!XK7dw^$HAb(14O7T@fp_J!B9tv8{?*9?;fL}yH?Qr1;ytrI2SL^vC8P7`tiSCr8Ri|Fz z3y?iN&}rg%mvdle%D8skYiG{_@D?EA$vqfCfgBf=(6Fc(G!IRwP{dFa77=)4l;O0@{#ushyad2>E zJuC>mo(yk}h0eZ`Kr@PhL4XSITxz*=$DUE)ST+MT3;Y<5Sg7=-wUzZ>hKa&i7e1Qaa zZ5u^2LqamkG~iR__uRg_WoZ9(QnvTaGXFn#8gJrgUY{ieYHYFWmvQv=>@v-Eb@;S( zQN)~N5JneRZ~d7%={)6gaUda8blUMaVr3oJeZxTc^n+U%v78)6^y}W zo0NQ+JEk5xa#nqV$$ww>-s$X3_4K@qgClnJ@Mghfv8y+Je&C6W+&t8$ykl??Ju)Uz zykZrU609~OHrDmU4`YgFxpv>z>sfSqNIo~v!?oyDTMK!XQ0eyWL!v6lWX}j=qU#ov z!MESievz2fH+!qqT}PkF)^P;lIhfD!7fzcASgun*6x7~rtboImg9vM-Xc8;z=dF;Z zdJY6jbm@@I_s3|ruWyu46I0OEHSCYEnP7;&(|ejMR`WW8;bV>;a$OGXW%dpE{J8<} zm1?my6(|n$K|GOMkH@Id1iRrv(Q?6%Ye)k8QPGz@_9u}}A9sI(Uddr~*f03%M0;oK ze)E@|%R`UiX*L84CftOwT%xjL?vPC86Qe2c9_;?14&K4X!wLC3{TP#|@@Se#HQ)FS zLVBfSWK2WIb|`%ZQ%*vmX?#TxV1Y1@6!ml{~_m&mg3ymw^2fC-?Hg zrn;Oy&7uza-D2SvGq|}dg%R%tzca`TQ}LDo$O;*mobP5KNK+j8n)l{u1}i#uDjSRU z!n4){wOgzy?F1e1Gt4lBMP?|je%`kC^r)t$=HRflrbfDjg@NJyY`jMtpcHFRwB&-$ z1;iGf-SaOx6NFo1IB$6Z&CRS^^9Hwh0Sa$^)SV!1EQ2MhM{Rj*~oItpM%tX(C$lu5>~{`R`F z)UC~wUTFgsjKl|}R~_pkGr9I56mdFjP(npKkccyMYU%~b#>PfdQ`48!)OY>;Dw)MI z1oXJq!KFD>)u_-YnfjTlETi6Qg%&i_U)Nvo8?hk|XXle7ml-@cFujd98jus26Huw;8XdXY_O`q)Q z>Vh7M$I_)tG&H_elMNwZVV8;ul&rX7pDQb0Vm9q(hKdkB0N%oG^GEPp%)Pb_D7+-? zaQ9-Qy*(`2^;fz~HzZPrC-{}FcZ)bouB?~O-ymM_#!xiSi?d11UyxVX>oMx z@moR(=ebH@j;5aNqA^HR{@q{oVp4CbyY@mDDpp$0yLvsVLn@vT2Ws4a>AbXO-=+;~ zzS(Wg;)x}9-#b)^OjD&u_6r==!@_gnXy*2Ax zHz&4Zq6Ic_GSu&tpNRdHkG%rc11g`~VrV`uy_yXPBWcuND`L%MT$SA2fp!Q7uAaz< zgagXAF046!6n``35Y{>PW=Hrm4w>lGrmgiT$SO{eUJ46*v!AHujZ|hW~Q-o zPsr}kbl(k}kcLh1OTNi;yh7j1Xz8;!&FtQrIVJT+XA{4xzv%WvxGyV{<);gGv(>me zPc`k?J$eb+;KOfbnjfV9{lR3vIq;c+Wb*h)nZ3MFbnU1dn-LSh%~#wmD?v8NI}V`NMHWTjYd<)ZP$q2z{&n z(@$*5qah(;&T^Nt9>S5FhHBq8{k-67$L4gm`|R&u3e&sVYY%iJa%IRH8>coN$}GR9 z&yWOmJxk#O`%}A_FqPiF(LBOA7q=(xRWnldG22R$4%oOlV z-7EEZPhK$fN#4Z4W73YgoJ+*5FFT~F>7%3kps*pQ5TL5FD@5sA`}ow@&^0H!5uA5) z%jWC9?_F2$wVN=YxaEH>fB&8!&KNpx!Q1TNPi&ypSn>aL@ajG6(H#;xi!K%c zQzcXf{|t*>{E(Y6t}n!{zob8VR1=y}wQChyTU*`c@y2s9&z^1Zv+LpVRO+GjM0!|q zVe$?+zbV#UqTUrsQXe+h*${U*tn)BS=cpKlNnNvA8{ z3@wm2dp?8nN02B$)=AC4u$+F2Bw4S7R=bYd4vS`JQ>G+}%~~VP^fnkt&Ev1*R0oQQ zTblWX-t71?rBd^{=a}y7?z}|LRGU9z2C%tAyrI)0;_@t3HJ4iE^8vQwE6gdje#zFU1m3r5 zw-J`0kkHf;)*Z!-)}N*)`46yV0u;9w@aw(94q0+ML^ZdltR*tSXMUSJ!eb)8nF0Wb zf`^9~79J_28mvLl4_{uB;X8g*dC-&P2<=M|Y$l(@@4m|;u}`iZt)VTL3eV-tiji`n z;gp*<>|#j}FpY`7-3)zKK9s^NUlb^Ht9x;OsR$ugorfWEGv9QD@8p-2Su zt6UsByhf-dNE527s*0PM(tD#}Q?s&mFFhanra=7LU7oQ^=nk9xPpr=$Lezysq*+wa z0)C{Jsnv>v@2wqV$coqpC(AS%rYqSsNKVG8RnxGrbT&qcnIxa#90Vw`2;S?7tSo!n zL*(OhG|d_Q8AmLO?`F|+gP9+xc58!CmyUScNMEwq7dqOi+;B+Rnb-*^y+xw8$?wB} z>>#U+tLYICe9cC!Uf2BHb-731@We?xINocuNPZh&izfqfMDg|A=oN`PR8Wg9E{%aVieG(j)F%JUn($oHZs zHj(d?uy1=`-wikm&X=*;bn}SJ<_8N5uhRf>OP?NyMU6LSdT~-HdOtnICID@Sewg_i zg%KX*O+s8DqfRRcw+{xA^0DGJD#pM3h9AW@WmxFsC_g@QROIiiCJU@qUsj~{Q~e#ibO3=n2X z|D|cu18UZPkcBeaJU&jf5~^L$)C8}8;4^U@!$OV(f#aEJ{__=*b+$t4?CvZmA4=>G zMR8jPjPThOpeP&^>dy$r&J*1-c7IKN(k>@ix8vY9xOc60;bs;{F7yk?lyAix|L70NIu zrtPL>M8|2vlWr|#pSryc2q7eLasStVmD`iGswwxa)!~8yOW9cQM2^^OGaT1_gZs^P z!Rn?rqA{q@qTQEx6$~Fc-c-puE`Ah$mZ0Q!??W;OZX$bAVKOQ|Ux*)1H6jCtPu7Dy zpy)!HZMUf8gK@V88Z8DY+@9bDynV7#RNJT;kXJVOlI?|iA@MJtVZDOcPkI(L_V;nG zZ!O<$Yg${r!v0`@H9^q<)+>XpJb&m*amY6_Ys{<1L?VM`z}cC-EagtKhaV7I9@v9oGg(|G@z76V7hdBO%v=CM4j z(eJa_qp4$uhlZ#>6OqP=EUv$;+zYFDIZ>c;=vsX~@vo|H_^-`+sfiQ6d_nnrQ@HCV zl9v8AT)sG7{$qx@jP6F94X49DNWYgmNztw>v?92#kLXLoifuq)@U`hU|MFs^=Y7;2 z;sw;HtVXg{Sg!kc{UyI`$|g}FIMAFQXD|EfJ{=Y5XCBL6ewR2?ne&Ai-{h6+ z$u#CV!GA#E$8?r4;+5{l3%h}L&|lQw$wv}DvO>Ss7(F<8ApoB=%sz9r3)FkQ>nQS# zlKCN3+(&yHFx)&l4A*ZVhha$Xmy-5sbrTyWqiHRPv|%QW0F0dVfEH)#S*lr_y975# zS^Xl?CN8^fe@t7inIBJRi+F**D~kW{PI`e4_{cpC$H~lkAR}AkK=HZ6deBbVtLXOC z0HfkD*!Hdh21?`%b<7ynGS7dGmwsYcr{1$ZWmhtAW$I*?)4C|e`VZx zoINz?w8Q>MnHK}iZ;WY%RRORti>NhPvr;1IaLtK6>M|~L%xxy|oG z?w(n2v*MUA-H8N7Y&O>FjdvZ6zm6BQbnFfy#`P|DNyFcFODu%XVnfs~=6g(=r7t7+ z5{`Gnbqn5>eAXqgqoEA`{je3e!8f}2-&T3sZ%NwcqmefX4yMq{=3b_9QrZ6zj}0;Be?rB#oE`qv_* z{(iSSj9SAZme=XHs8o#)VU322+(KZluOS2Ld32zLk9I{_&|PV=!GOvBz06^76bHN) zwXfdwJx9mU+!ezL6D*_2f*h3<->xSr zw1ih^Ms{7f!{)NTtSa?c0%0_NQ`FI~K=3?OAZ5nr`$dQ(Ayj-V+eOi1J@&ME@6Mrm z$qfp*!?Hr-b86(2OG&BiJjW8`h*?oivGWH@v^#Fa&XzZo(oT*ouJx!ATDd-^kH1vP zw@D;JfE7kUu~a6BZu0f^8T5;c5YD&ff^}0xo0Y?AHd<)PsWF(c-r_)_KUikrWH9sA zT#9I#&7Pf@URUreqP-xd|DMp>`OUSY&KDB@z$Fry_@GW_Z$M#iosco*j+-g8Q@gNC znZ048D2oP6?iZ1oUJ~S_y=*YLaOrtZafe%j=w+MTz0qN7-Nypnwq#L$(P5(ZjlutK%ViA5s~m3e zf2UUH=$QTfZg^Z{YYvYe`t_Z&08nzfmmAyU6m^Go*HOqt%yq#E%xW2h$@v@YGvs!pZ(n%cxe zLWGVthKYBIDv;RLAp=X5%IzYSM0os*R5|wp^uT6V#k0Z~2G(!Rh z8WCID6Q4v>32U--*Q{?qP>?e$F7muRS`CdgZFw@!!t10%o=d%(WZ-)rvK4d`>Etx8 z{T9B`b}x@6J)4XIs{!vDBh6b|0LJaE`=MebcQ=E=#Xm5Rl2(G|<0~;KsclCOetxfF ze@9ul$-mug`&v)puFMVa##D>TArz>kJf(jpnaaX$N?O8Nk?+e+wUCJkJ?-RsW}g41 z3lSK-?(_Gxciv`=l{8UvQV^CFhy3Ch;8;%rqJul<1zy_jZRbGk-#lKvkX)^quTvXP z-@LcuhTcN*_BBJ?$H-ZSIE#RH-6;=>abCG&M-ro!beP-`eFsIbbpt+1FvFx{`Og}RH5Cts#!!$zWsl| zi>T;uKfgRq-s3-9baVmGgZ&oZ=Ovxo0YILpOqE9up`7t7g4qkmQ9bQURP-K5Ki|F4H@ z^s|O1TbI|=8O@lRDRz{SUpH@+ji^R1$|D^m2|-HhqBOru9O1}5**9afBVlx2BqD5Z z3p=nJS!3)^Hy*xWr4VyT`{YUi;}+2>2MTq&&uF-JR6IO_%e*FkIjklwLq{ejO#9sq zBeXA#Jw2>ErJ$mE3)R@%mo2z@dU_GuP7fbEcz{qAFwj#@MWv}*hOZ5@*d247iEi6; ze!*~3mD$AH$S5H@TfG^6-4_NVb&kt?vWU2qFJ+l@w-KIY+a3z4CbK;~XnU1eI|^F* z4grmCuBEE6v!|#FTn5U>c)8t%SH$qcr%d#5tyYQiv_zEln(pMqCP!ik?EKaae73}wL4@}n z`Rc~~`SUz@7rZ1QqWvNgii-OC9*7bC9$aW$VczT55t#9a?(mn`ZFqkp`YoZ4J*C-m zau!|wk1-r8-tSIp zb0!xxs6GG>OnDu_Um=pEmJP?YM^QN>BBBX;uya)C%MKROlBHZxpcgsLAl~hBXQJpP zyoy6|yY-y8Y#sMUMXaP|txsn4j|?qucIsCD`_l>eTyToa>|rnPnA%U}M+v|@&|WsK zQaKDYc^mW{079tmW2P~b0+NgfQ5D!^eaUQ$`6e!?2N=ySy=-9hm)7mykmNJwfFQ=kg2hw#&a5ctI|a}>SjCb ze=(G&o1s4ZgC6%DmE09~3Cwg|lF3^K zc2i!>$8*thq^DYi!Wm!N)fj3$N8)QYIAx25YEiCZ6yI&&!LZVllF9n<&oN9T3+-#W zNxbS_lkjQl8XDr**Du_-dskdTg9rvFkkIf}ZSy^#ruLa^JT{r9W$E~xY4vPrEM)pg zFx!3(K#dn-;r+Nx=S%V1+uO5MKFVZ^3@Kc-Hxow~UtfPEP?<(Hho(H{BT^d0g4_0H zsovl;((QZI_U6%dr`qsGzOeTyvP-`N(u6PR)iZY3d;w;E{r8l6;7VuvRIAX%x@x=Z zB^-{ej9JGlab_!aO00)$3^+I;tm~?me7ol<>FHbb=^s7XN)P7inI|^`hlX)LKN0M! z>X1R0ZGkuQi^^>c{BPJ4UV!cjv&M!CD;x6uufzP(9x`rj|#ESmhfb$eyR zFDWy|K33r#;}wXf$8-QU#JWEz^{y9gm6x4JVddK=tCtDd-rs$;)b^gD^rmU<0kFJ$ z4jWg$o%ZA#$Bv&o7CLfb9>>9V%pmv#iKfMTm?^rU+*khAQZveE?C=e!dY$A_o#s7$ z{FPrRlO(h*1MfTYH_3FauJhaUARd@dGt+Mo$W~@!Zo243U0NVM&w`cScCsW%mJv)2~Zj!x5B(n>=k6Yx`xHhz)}TQxfgy)l%~u9LERknVJ$w_<6Z zn=66m8}H@X?%~3_x8ZeWogKY>U(KScDzbreoEab2UkrW*8&Hx`t)z<`CTH|FQ zpUa(WWHAs%w3E-__C#bkuKm$aPmz_OX6C;F{A)3fo%lg^!t3rp(u{&}1`1Wk^ZN<8R_!Yk{4#ftXKi}oI8W?Y6R zaM4qSkRvrMO*LDG-+P_49^_}n*CwdB=p!zO8~2|d-h9eYXpvchGZ#}hYCnGvuhmR5 z!zcr`xV1%Q_>4y?1DSutuw5Y3n8*(eiyK+`Fr+3>282z2WtR>2ON?jE7$t$Ay$7_zv^kG^ zUPgj?N^g#4=J)q)s{$Fb)14RQI7t}yIJYW@(%CmRZo#U;by#}A){)U)dq{{iZb@Li zZjO#~bn>p^Xzr+8B|~jlyiN*XB~En~(c&hF(W>eOGHr}9BB z+!rqg{iKH^keojHX6#N|<+V<1Cm)}y6|W8IWWDM?D5t;#-?nhz8xB0V84Deb%Zs{S;D$Z?(O#Bt6#j$ zAo|!biyWa=mD2oZ|BHTs6e3%?=~{O$99$S#4L_WEUHp=$u`w;|WB22O9Jj-r?iG`3 z0rjs(%q{$HpK}#L+-VtaGOp{^a^LSCTq~MuXV&8i6!V@Ff|sAwMdVZ5)!i*q>b7ud zqnB{vqs9uwBcIRdxmiDhH<2pnWRrOCI$ej%)Nc50lBjPAZVUgBw^m2Hmu}TchweVn z{o!R@&nrH|fiH(^g&ORHP5O(G0{Pk5yll%mBmf&TMml-OO^V0Ivvyo(Ww^LC`f;$J zFIH!gBp#?7+Xbz%VyNIewjLzv)5eqrJ-m&(pnqJv(WvOdZP4#Jygkx`%FjWF#` zbsv*TJ3q3Gf7A}D*EYJWG66L;f^ZFM^fC8C21I0m)Upw!Kvp+w3b@cUq|uf%R|ctH zu5S1=UJ->Anc47O`Vzz(wHPFOnDPZ4i+MM|^dy&aOux(`*^nfvEBH?Zaa5UJm{_zW zH=$Cimu9?k5DliUxXoW~Gd>75X2G8gu3D-WnoqBKoIY_FvBN{s1{AQlz6WHaL7bD1 z_VI%9w#yc@X&_69LJ_xEX|R8d_Tv;gccvRDK3g|%nz;H}$otnTso5aVI)M60R!dU* zc*Re7c~ksd&CzfZUn>3FR4c){wq}YCC(3;}I*iX1YPJvN|`{xMqNVAUw92bY7 z*$di1VPZ5-@`wj;xNEJf9cwG4tEot$4cE7vja;2$a2qA-PvZ&BQTe+1$+G$x$Ld2V zMyNyNSwrb%CX2U=q z(!Kk2X9)4g5AH!jLcF2Xh8%C+$$ObN8d&eYeNy zg88BhH7)K}x@B?=JIS$#I6T{4CBpoOXR%zLfa{L_aHB&r z89Es#!A2txyfnhX!yVcy&!d7^nv!)B+t;#nsT6usM~(37pbx;`_NQxv=zh9ofI5R^tzYIa>yO8=`nBgFflGIYd~c&8u8<K-~{hvyqONJ32FP^@_)*wd$!l_)H1v7PnOg2^2UwV`u2H@z+%09 z`*zH>6=bMsSrt)W<;3N%(h8M^X2=FXX+y73^dXYfAoSCxdKf_Q3sR$*ojrv%k~omy z+#<=uvPAYZ4o<41Zg(lb(wP44wW7Th<_&3ldph7>KgaD=p1p{!D4?n!3n<$*V~jWF z6v#M!p8JX&QpQY@7(j3K`i6Z)-BFwNB%I9~hzqF<gi&e(?*4Pq^sbc*{)KO%(-Fs$%w|Hw zW$Onfgx{rOo!g5c4}j}3r6}>zQLst=`(ZZB;OhZ48F?%V(qePEYR)$u)w+p%{KQNY zwe$^u2pzC4%9EQ6U*#5AP@sE|k)GH-KU&<$!u8RGW_#QRNAv2>AH@oFQfElSuKhoe zhz`$ghb(-qL4x>LC-vb!ixE5kE+OmMq~Z-PZ^oP8P#+83Lka1EWBrK=Dh+$ypgEK34-1ccEZ|9 zZ3K;Nc(w|gK26v?N;M0~<8QyNEsqUHKd#Z!oggZDa?58VVFnX^4D|+}Loh#_fu76> z_|j{OVnakJ$o@%^4Br5Lma8%8_rdFgwZF@IqIy$L&5NuWa{P>1!Q@&N3DG<=ry_$y8WDOyR z{4mz#Uy*0{LHT6VXRA_R0B5&|;kL*r5o0kn`OV=;I4`>H^P2tLt6t6EXDT$2#yDL}c5)=QIdPJ@onc`4fX5cqRTr2K`GLcm{nk zbU&9|tJyj7^)s`$rSWm(FNf$0@=uq(%lDk3#^5Y#I*{-f%YXLbx9hbvvjV9^U3~er znRzbB7z5esLYO>gBJd{s;Y}l9j7#<&SS_%M$PU&S$@sEb@)O-euG{mfQx#2vHr4sM_}_E2gIO@C2O5xSE#mVuyNEnOgh1hMYD zZF(ixz{McPwd5e;-W|62)6=uhm88_d9LU-rlkppvEkhBm35v6N$p5f9dHZTMT=!hi z)8o+&mD#x&DG7W=F%n?M1ZQeeY2{i(y_ZpyB26Wyl;>6ySDOtHYmqP#4 zkzK4>H&tOlvv*tIjC({x-R8B~a2|vS|Au1{!9nK=b#~;5fr$jFB^!Q>$H_jBx7CdU zPBON(w#U2hM=*m9-Ior7kENR#AKqtrZ#Eo6kuBhd?D5J#iha3W<;JAhal}M!rC|#*A?(t>+gQFCmK&Tx1!@$TBwHeDOFvQOQt> zk*0yJ7kwBWN74u_FqoDM2GJir=t!8COeC4f`9}X+X9{)n&+LjOJH|D;?$UKTh3LE$ z8P&Q$Ydg;|i~N6`JRT2Rm@3(vlM4$qjU9X*%ch_nSIaDUCeg@GG8y_ZtK6;#5M=BH z+v!-&?|XEDE)G;&&(u@IHZKn^MwU0^4~{6y@K2+k!N4OzQ@k;$XNHehW(@9QUM*4! z^Za%u@0FefNQlk5=h>(25et1|leS!-nFNms>%-;3u+N`G!WlJPKSdeAmceIzg#lf( zubM-tTBF!n%)lg#UbE!3Cv?>gFa48%seN3YBCByI6Rm5<&guR5h$EBT>hdRV^REYG z;w1|*&(0{6Q&5So?G&ORRrhza1S%P6=@3b(bKMi*{rUpjM^T3D zhs?fDC(&sQQlHBOp@w*_CR5YQ;Ec^)mF#E6{uzGJXV$73$MOUdP`|*<4pB2ghP0lZ zNIZlCspI2_K_xYqMg$t5TrRU8j@SG+k+1K+SMElqyPs(JB6oJrGrnq1^otvl&E`$m zK~>lVvT$zS}(EY*!+U&!f~gp6Iuz|Jrh?<{9F!8nj>5fu)7)+;3wdXRH3(mFIVAzdPvE zy51igTPu1K$83y>5sNx*^*xEknmyws+&txPe4px#*hWX=dB&f?pIwN$$!On9PD^RO z*Vj7@f>Q3#@NMCvF`!pm3(`Uc!#=Vw&$G-u?D~@jKVixiP8ntlZzzx09PO>!I3N8` zBs;3k)~#)7v(q+S`jH$r0bi+=vCu34J4DXo$^H9&;KTjqTo_&pa1|ay)x52|rlzLH z63oact@l7X3o;?NRxbho&M;iB9{t_(6L*gFE@Q+6o(At8W8-(^2HjCe>UpvG;F2&> zR0DMi$s}G(cok^;Iv=w{8Esa)Y#HwoA$73~SBZ<}dY0AMRdQ#xCfvg-gVZs|VGcWN z=ILq+zsGf0J-o<5hw);6l=%i%ZC6KQ8Pp4y!32o+V2%Qko2QnR%-{ty1=_|LH~^_B zg#xaeiO_DNrw-C(JjV5@6~@41}^JmsI~;( z%-Q6rad7p_NX@Rc$|r75w*JlVe7Ef8YZ$XB>wf>kLOMvsJWbdzj0mOO?x)R`+0ov= zk86j2|@OgP=EP8W&D;YVV51CeLd%VN4)kc)B!!l^|m2@&OV>}%QmB|=U zCB2UM(|Y>@rFl8?TQW_({0$$+_uA!gfXGUe|1Qpcnfd0z{Ro$_99!h{;_F%|8JVcY zEMqA(29Ux3Q$H|f!aZLKGb|Va8+|xA$HW5I^o@EP+}OEqWk)t8XwgOmG?S~TX&%5^ zJXvy7#bk0T_I2nb1$8nn=O{1j$1$n7da1+`oST2vluZ7c=Kap&o04MJm0=@VLjC3z zG460^rH|h#$5QsZnb;}aN2j72-MH5hJ+Q)u(QwdbW`VOHf1iMgwSJ$6=QwXDnJeI^ z{E|eJXh+t6hHc!+jQpsF{YP;osxyfdv}@3~v4Vy_fUagz%~vBwe<<33x(!a<>=Zg8a1#;MfD&pxzK z;YGXV;DWE?MW&!RlvHO2le&@9m;kgpXO~zv(HQ39Y;n|M;}!yvJ{#`9D0H{X-xY&! z>()^VTxh39?BU_cD@jE^+Zjl1zMxu8xV<8=EbRIMIndBi(E6%>ivImw7&lo?U0pZu zfuresmkR;q$SN4Yfhhmh6+P+~Jlr$h5fv-4a;Elv3}hWldL2_AdJ6`#Y; zf_3p|puF(vt8o}gK=7B2+$CSfkLUd8>x(wGa1_8$;t*xtO+(oSx&GqZw zVb?tj*Rv{NGQrYO@lN4gwWdWI5zw5t+DB<6ch}O#OtEftj!g=%#bDu~%vxQjCa6i) z@hh`MFJeotG-jKx&rnm}yq_vl6E&;5+@h0Z9W0vX`^Q(uk>IjX3mK#zPeB-Tw!m?$ zex#ARt|kkax~L37rCt zI@Wv*JLbW{X;=J;a{Hb(m>^vuPBmvLi&Ip_3POQ=>-X@%O}_lkvK@U&MNNHhkB}wc zOvL%GMyEP@1}zhsonmUA+!OR3D$HQX;>(N?UKwM-eg#(&A58QLXTdo?!B9*?>v#}g z$~zFV$}(E6d3jfRG(4?+HLxeO0Tbu3G>z%$RV8jzs^S%^08CJtKzB}O>={fKQVaSQ zM?OfNd})W#cY9N3_2)l?BAiRM1UhG2-c8G(Pl}rp@5;+cPwv?#kK?Z1ApW&Vy*2Y) zP7}RA71$p!l>Wk-pZTtdvbzLf@1X~JTprwgc*gd#yahS>(nx8zR(X_t%{l~ZYnb4i zFDn9%53UT^mpCuVuMlc&yRwaeS8DAQR?Rj-82m9!a}x!L*VZLrJ$6U>J?>h^2Plem zSZ7k_%CdBt?nYZ0@;Zy}--M4^LKuMlbklYYN)9}0@Cccip@MJzeIq(|nl7{Gb-qbo zSC0PVjtuY`g~glgPuTX)6TU9|ebsDtUem)HOFeOoD(Sy>C6p%{oWi|!67bf=5aNij z#|~>$Bf561YZn-YSzK(0pojG%`-;Xg6XXJ}$Vb25{+8+V;u+QOn-t^=yvlP&8juBH zu0a+ArxY`E{DQ}#@bl+?pveymP!W2CBUT!k{LYpd`qLeP{u3quHCPsgjv7KP_#vni z-vu?4ySr|oKZmp?q@aDp@+VLqaq1Pm6iF-ZzbaiH=J-ih8QmYz<(IezqV?n?H{#k7$c8f)jRO}yZcA#Tn?QpC>E5H;? z8K@t(v>bEWjf{CY^iQ$k|8p~D|M{%oPkraX_jnXh`b>??2#;KJ=L$eQeP~^k= zA#iF*W(B=}Vu6G1_+PS0;u<{y5^Z(#MJ=zac<-pNmG$2$MO(c1$JehaUHS+cNqrLJ z;PGxdcl8?r#{t_;!C-|+pwaCJH`g0U!BzJW2=Vp+PmVqeZx-T|QHv{~xJn9t z0(XfRwP4DEqRSY8nu7@zJ9jqfLPtz_pHHy3TXooTa}<*bmNZ00h5O0iqfWgnQR?X5 z!C59|JAFSOp!<898Au?K^|4KRUt;!h%z2NuKPgrk^WN6X#n)2p@(aD@tVCqdd&HGS zf5pVabO69>xUSbD8KLdMAAfLD&E!34rio{^TX+s){mPT=ESUiRU6{ZcZ$Gp#BL3>1 zKFhiNJJ2q9)}rH4xgG45|V$Pjwa2mQ2C)m*#ncgrxFs^0`Qn#E%oOh z1_!n)!;zq#`T$)XACX+4q=d_TJwSU6?ClS@6wgEQ>VJwWefGn?w1~lj{o|xnLqz}E zgj{3Irq3foDJzM-tJerC@ui_;UxBbX$9^HAz0^13ZOP@~lhK?M*q^R%4qPPm?U`5` zs#NHSbPP)>h zs$}VOvIS?uxPf%ZM)=3}vV1Nv$}dTD@r@qdGUh-Y%iZ#rglyzwhD%CrJzXnD!fnf6 zej}|4a1$-uAZQhq7r8P8oBAQ7D=GY+0@gP*brtj~daS4EX9U*|hZWmy$IX8uCDFQ} zsTJ`Ljw_46q;3y)L_lpqAJScTJIub1BhQ;Ah`Zih>)lL?tA8wTu#*tvbOODg_~4$F zYV)Z2&&hS3Q|Y<(z#qmvJE5LYUA8~5n}gjv-DUqvls}SxIiPCtqyC?`Yz86E+iSI_Z~qK!olW2HN&@tMaF;l9+FEH!Dz|+ zKtiOTeoXU8q7Prq=ke&@9#LvN<&6?P+48jZG_@7kR|<$ff;DVVv=goy*DGHhmT$%U?<~*!I?v52un3_ueb8CdL$sb|XiA{v8Z9L1-&r>^& z?A?TLMN0mn#Z-(7c=}$#ae_@k`M3t9vzN1o|TR$k;yYPK>&}YcfT*PPr?4}vn%^%w1*z~84I~tz|C;+ zT_^TZf3TD|`OSHrA8v;>+eST|{Lsd+));nVGFQ)jZ@u-&iG$HFewX*`?5ES;dkPrX zZzEQL5-`1=BJVh^ValEmSGXWv&~GzL*^{B+&g|e@L2Cf=hjSrB)2&(z(D(rx85YU8 za~>~4Uf=!?e;v)}j~(zGj8XpN=vekzr>)PY$D`)_zKzba&S>L#@a|CS{a#fG?`vCT z3Uj1|o7@}%AAXG_8Ll$u!g-sB0to7!?~VF+$GC+;stM@!lAq=nAu zh1vSAIU`~RbqkQYhA+HlzM3<$P3pEqH%UYQ4vu0l?j+D3w&w#mPV{I09|tV&q4UxE zNUaRwghE4L+pQG?(D=QEx5;CA+sJc*fP||PR4ib z&j!cYF2S(oeSs9$RuH;X@Swzu(>nX}TZDPh?0?E$-Aob9v814p-vLF!6W#-ITN~E- zGcM@@^VO3-s&tSme_W>I$K;Gb7Yj&QPTvd2f=*TrTipV)jr@d+If!qNK!y4!J$VLa zbSkdlXCLiI^@R`QeEYLN$NgO2C89@cN`|BWYzo|nlzTgkE9?d$wM z)j20HJP)sXvo^nOHqj^P)dUzl;;~w#%ks-0cbG1nbnCv()Q$`Z)_;M!joTj03fVLt z6?F{|evlIr6g^hP189|%l(5AO4H=+1hvc0hZ`qxt&N)zO-SvW$)kbjm6*eRf;pe)a zLEYQ+{KrhfLRf0TGvzMzhCT;k{=nPG>n#V7-?GgokBeAGOynj#9g~@JSKof^I24gY zeMUUhni&U5_lnEOZOQvDKya_aXsxM7B0|G+^wHjQ+*(f+J^H23=qK~Y@oLkgcs|#= z&-6U;duxu_>O8^haQ04XwK0UIpAQCVyUS~)Vj)+E-MCmwR!)uq>J*?il+?yi8El?g z?hi#Z(3hW13C7?kPs7b-)gMMhiO*Mi)`d!^d8Fht_2}+3 zrmJ`W7rM0Uz+Ripx+Uo@1^ah`6?hP?iu8&c!;K|Xz=03`+5PlX?pzxXW+MyF;nFob zRjLj3?Z?9Yvw6*Zs*`GgrW-TqMReezw6xl>3pfJNG|BOhFz?>;7cVBDI$_xU`noU( z_W_`ytBCsOG4X{82zjf143Jk=6ph;sbNHV%76;JaRg|e%z2Q@iOWJz(lit`YOR+tf zE?P)9X3M<9>J%8Lh}e90gYLH^N!lNRtTz%@C}`0d+FKO*zDgxjXHh3jj)S8{+e#P+n#SJ z`w!eakQwwPJQAN3VgH@3F_%anxJ8)DV7NQzOS4lkZg7)WWwSi4^=YZTb#Lzz69c6} zFZY6+bA5tJ(!xKnA|ALKPT zK#9y`Lo-6p6%dF8ols?6T?G}D^H}@4nhmRkfaN{a z$>{_WYIpy`-sc1xi0d*z9c57Lm~_spEg9?YY7&muvR_&} z4Gmx=i*{-u=3MfaE+BdLiFn%sF2zG4sh?O1RnFvURQWnOF>s3>?5ND0^z7obojB@) z8Z%nZSt&*MA3~f0Vaa;meO~7?fh|MFEb3sl(fU2h0boJyQ`|7|%A~Y3BwR&Pcy`y> z0-E}|^@uPThqi{;_G*^i{tFQ4ehOiv4R}i5!+7v~#>Vsypu5rziw?$S;X2&j)oEik z**}+g2U1Myx4?t@eR%zZAAkk|w>S~&&etQ5P*plk#F^b!zxsi))cIx4r=qXJ2rbPa zm;k}TU^93{X4x8fRm?iKUdE)8Keb~HvA70^9doOG$*JXP%fb-sdnhR>xo=OuLW0WFl|jLU|&cQcSDRqbzfvHNW0NG&jLD{nSR&|dM56Y@EC zI~5qcRnC7>S1ecSvHxD)Y*bOVCH847zRnD^=iut$_G9oUCfxuSt8n-#+I_SKPq@S) zS%n8D2sST&#?D8eNf7k$bCPDOadDjwS3@!j=$9Y`0Qcmm8P=P+E}H#-8VRQSLDLRlbV1qg zpVYw-e3I8>x!?wv&3H~n>UE0c;p;Cn)qg)idn;%aOQYxsgdp=We|YvUs^3X%S6biZ zc!cT~7)ZXQb6on-qsySwY%-5dotl|ZR#k0;a@vncfQ!2W&;Yt5Th(kE(;6R5xo0YW zbFsVK_Xqb>hgQmRQ(*FsFhsBli$sxeF8s`zF;V`gpKZXI;{7;{;tTxA5m}EiD`#>G zH6E)#=Z6&e3-=+Rs<60vt?rn_`;TP-^lGg7kYAV4U(3P?1)b~F$mBg=IKho5a-F!v z7Wu=TKOYBX@>!6uOM`|>!L(u{KcvGwLD7z;XThb*o9J9yyJ;T!GXm1nOIK_c1GR zwY-%XvQ1p41o!xG^sH(2h2xVoBO2?eP@*&BeyPmDD84L#K=WPf# z8TyGv0y^h61(eq3ACQNu-Gb%%O9J4k%a;5tzy?&-(48bBEYU1`%gbG zsGC>T45lX`RexU%ZoLQ5rOW_y2nAt2g|a-??0w?KH_B{)pxIRYFn8>>RpMsVpM zm8^3bh+1-l70o#}0jsOMfdv{u7&}Nzi$Fd=7qw>sxKV@{o$x;!RqbKj@Fe zN9hbkCM9_e{JbBj{QUWI(H5!pceCn$t5#)lab~hTySpsjly$GDcHDM#7SQGec-4MT z!Wd}1HHDD?%U*?kX9zY3wi@6V@9XH?b#^YxyH8?j+b`Yb4E8%_S@_5Gk<$Czh2X#X z9$cjA8M};Tx`4Cz0K3f?LcXMCz?e22;ts5H^Y9>p<%O8X3q32{xK- zkM)6*AB~b_dW002fZ534bq=^B56E3c7AUdK{w_Zz)!~~o`O%Dr3rMwmarNHNnl&Oa z(j(D7kePs7{nz3L5%;Jk6!rBJ`kY){;!tpK%**$rb`-C&ez|*eY>~QTf1>H z?dY&!!(@*kZ9R?;C^JRxaWS88CvkVDLP#$Qu)^_QgpUyLT&6`40$woza7J_Ec?WAtNCJ8#SYe*+VcCV zrX{j4yn^fYe`&YX#_E!;ijn%||B&rb+bnmH^+IgQ_Ws;Po-h6yy&EpY zl6u}TA`Ej#iHpvqtX$wlV8U^)*Yz5;o+9eHF4F>HqI`U>z^vHW*v6wd}AS5$02Br$fi{k2=36!a*P!&48u&?K`A;9c9$PeJRq6VWP9 zK=;^a{4Zx?P2PZ@tf2|Av)V#zv2Is3-U5IO^FpD5y=HD%N#4IwmR3PIV*gIr6wPUM z`!~q(eZ{l44x5OS^fs7Hkb|yKz&y8P+(Zi*)JaO%PE_umqT?W&6Z!n4Q77-YesNqJ zXdCiv+YwBHB5`VZ`dra2jqBW7DG^8LZw{C2 zo<{1g;PXvP+=6^<4JqZ|(s}mR9Tk?a8+4BgiQ4j5QETh7y6EAvb{_@#?d?h0!xr%LondUDgJcf$*kM3oUVH%nQlwNguZ;m& zZZVhilVs=BO)7*;Fft}EVsb{_`Gg?;HB7ZF9BaB1pL(#=`U)I8keR~%QeY%46-xaX zV5=UGq(^?R?%AEbq36UNscl;|ZYx`lfW0e#u!Mlt8yKJRVzy-*k2%(B=y2A29V+yo zL$U@q?G?aYzK%_j2^vtfiFgl+d5yV7Ap1s=0{K3@ruCnJjfU3|zunH_3g`j!2Z=OC zN`FE%zuBcYLfGKiz@)B1>xEx{7c2DGbqrgDw*@KRJwIsd*$kv>j6tKoSyT+bUo+U8 z)#n!pEA;#HX2IYa-rS8(-vG7LPLggACC%Hhru%Rx+StvmUk0XWqjuQBQXGE~Z{$AD{ypZDpx2yC?l97w)cMw9e+76I?Q zovg*40zmoeqtH)=kAG>m03gJrIH4e;84(}G{d{0AN`*w$ zDQhpf?2=fsWM)r>teEr2Hz$eJVfV&lsWpz{m_tu^6%j`AKxc^Zb78v;0W@yuB)T^Bth-bPTcN*eBR-~$M z_3E;fu-v1>T^4L&dNr$y;VpOT1xYTvvnLP7={FDTJK$BKIt(pN`^LVA7y8C<4v*#3 zXecrDybr?Jlhh)I5Z`$w)z6waKNt2c*o>rUh18cZ}iM+oqsP5J@4t|z(7T&RuR z!+@Jjc@~TF*Umw<^i(f^IoSyOhtH%e`TEiTD2rjj9E9v>>brHxM(uwbT0(X0!hB$u z^V)V!fzZ}0;NpPs_=3f3c!HjnRjl%=U!}4enuFz=xtRK1vaWPo%e)PbZQh>OJ=Ycy zD}4WOC;f{;L38N@lJUuJ^_2gTNITNWK+pt2Chqj?Y2A!pdnm#0B-l^8~8lG zfC+7i>|l{@3qIuVeqjWF4B->tirnm#kf~R<3JNSzdny`0d~ts?te=IIN>^uow_sUJ zs)1JE-*~v83AMA@d{t+GIC}S$b7E~Ecgr>XWjhT~s!*#nVPhy!9(a9gV2AC9J_fm- zl#-GdkU8L5s&r%yEay*4ka2#4+8)CHU0-c<00U9jr;l1 z7#^+0i@+jBc%bnAh63Gt4v@rAm*Dp?OGm`WNE`=45Eq&6mJN|%Fo`N9juAN~kR{UG zMRoSvtoSdRBQs(ufjp9Q7!7m+j0dS=aOOMtOW`KV z$mU7A8{dQo?I8@1;0s)x{2M^~c$b?SYTBI^i#Wso33_o$nj>0eg8CQ&bv!*Kc$TG= z$#8Jc_mS7Lo{^Og+dMj)Z|%m|zd2OmW%FJwl!)y-*+JG)%=A9p;@l4)%{uMO(Fsc0S-_eguRQI*rKjVBLNY*0nB ze!;hN&jJ2@q26oeJ-7^9hf)opoB8}dt!$-LZagC23;i*o?$dZiI&6oot?r+g2w3ib z1U4Nj15^WoD%k#M?R5vRd2nRYG*<%>cyFV$b{bA(L?f6_03&S}jy08jX|e7GLrJYf zOE^A0fxUmjy`E)3mQzE)mvSp1*b6-I`u|bQIKB!^c~HQc%;`vK;2e0R*?(m%C?fV7 zA5E4TE|KZW)ZAx*9eL-~xN@Cqo%iFf;dos! zX7be!_toDq0*Z(KoN31&=n}pDkHv?g4{OGESj6gk?4J9KKa0IK_nwHvK+O?-8vDLE z$rli>-J7n?5bH32Iv(`N`e98-E1CcV#OvnY*`Uq%Q;e{2;ENZTwU(s_*?}L}d6yyc zdh#q8;;{gF8E80$O~Drugk2(l%drAjR6f?`nUz-Y6m(!1|8qCQRFhX7JriFW4%v6lXi^7_fm+#q9?FzerUpn`xr(7!{q_ zvEH6V7eshjdqU9#;Hl6+RI$uFL^{43!g4)6VGeFGaDlPYPoC&o}(?boQZK;|Ur@Wy>=SY^GECi|w%?{vZs z7S#iCiO-ia{-OzYaQy*tZgJIn5Ao!CCYkr(zaL)XH@}9r^&HNY+>|Fgi7+u+#ly`k zF19_yH-2RbH|;u{p3S}fIauNO$9HR#;r@uzvZ46y!vUfB-HZo)7pP?$zr?B#DAb=0 zUAg%T8CmrCKRl~dX`V;<(;mR8r5=Q+`Sn8zfs6j{Z>mURM%EPjNTO%`qX!1%6CU&O zzQ#v{2KUgurrCMZk$A|#Dd@m>r z47CX3rM)4YmBqFC;BR?#p~OFB^1aWi!w{|W)}AVU1Rh5C8OYp2uZV8bYUEW*jUM$> zTv4AI1YUA~JE~YT1pUsHK|GTN_14f& zO($Qj3P!o;N>O`<9v^>p!R8=@i1HBz*2plM4w@D5=Js0;c0}b^7BB=|PeIi2(hxb6 z^(F^W_V-TcJH<-lbMf#OwW9DS*e;~CCxZA6-28*UMKNb8lBkrO0|_UJVY(V2-$w3( zRe$VLy%lPMecQPFKa-j@ee~-TwFglSm8aPTjw5c?XeLu>M}M_JcHy;`NtsXAF20*(%|M*YR4~AQu55U`0;f z_V|p?`rYiUPZy$hk*p6P@El_D#}{C2@HM0-%}K90GbzqPLKfZe9E^ReK>X;(s$^Jbn1y_*4T z`0XgvwPS(>`Z_fpur0|(%jO_uz5 zZUn(_teMa4tc&5i6)!CdKy=GYglH0m!svA-$j`9D1ur%BC7)xknv(KmTca{3> z%XDnt5wS1`#imV-F=Dw^S0)SyjXta^0scCgjbn94qnat?;b(qMUK^doG?TDm+AhbE83M!e!tp+A}{m` z5tvI1wmm4&eIlYz^oASVDn6KDT28H`7ocui0k;5ogc zho546tF`eBAs5_X2=xaRSo`WI>*_f-_t`8-uOMKPkNVR%_t3<}6p3;ZLs~0}en0*A z;;+p;YpljAm4d8&IBh2NBaf0Mb)@T2&8HU}=?&7Dm^#Lto-pOa{H<-I@cs{8T9LPX zv(xsSP<`MaWZWAZHaf2(!I-?tf1t%G8jU_Z+fp^-w5lx=8mH$)jD@%y!ACd@)+)W- zE}j3m3mP7P!sYQm=p02i&BjjKfY4fdqR^|O#2LTq@2lWlR;@Ns)hGbcTawX66(iHV zMn_gQLkegy{j(c#nA6HQGuM8liBAC0H3fOfQfJW9nm6j4RDZS3$a`y6h`~ff^Msz* z@r48IP|8hb2NRk7ws0p&d3_8Q|C-ah)dsnm zGqrusSm%&f2BCCm<%0&!jHj3C{QUhR z5)u{{xRL8=g;lV$Xig>?`Tg)gN=UePddtjHpHl%R)>nT9yDqhQkPFHacuOQER)!2u zo3811k>|VfQV)*1v~Tl$+7i3v?!?7~WWd6)0M`z`iHW_9D(TkndsXdvE_aCO-lJ$p z z2MPbTx>XTiJ!WeazX13PBJgW$%yf#7=ET@nY1&7ivTh5%a1v*3mP zT=kGyD~I;bxVX5ckPv9{F{`5KGtzo^fXyB+j1j69S|?3W`Tsy@*Cw806R)LhK_!&wuh z!D++Zi>`OgpZs<`YE$tCHwv7|r1gbf`{`))q_s(VdwYm6K#bv(HwcTHnVCWGPX~!R z;9SK+P7-*|%Nu)re96(ev5*ra|huNa{>0Gu1 z-b3vet#$-mHnim-LP99I`0uHx4>9U$KPN+Y1itk_S+;LQA4oB>1C7JDXG>z^Fk;v0 z3)yLSH7>+8F1SBfh9<{xZTgR&2&wWZdvkAjUwhTfRph823MLx)>41zplEHfS?p^ni z+uYj#%Jn{oq7W%-EHZYjhTG&iC%mCdim>139c}<_T=a0y9{Qqu1RZPzH+KLTT*o7P zB?6-aI-T5(#UW)@pj-r5^WRR^lodLE#HbIePzE;p!ufMARACdWR%dFR%JZ`Ku6@5e z`LKAHorcU}I`sN1B3evBMYD)hECZ{*ua6k^oV%wvM?-if02+xHd7d>Wz{o$r##h9r z9yKaV-0gvfWy8`RDzjbhpc>@^>^Fh|u^X$yMtV}09__l-cWJpL*C!n3uG3_TX?*AB zrbHhc4>G>BF;RFpBzj-Fg9-p^pAiQH)&GN9L9~UxVXbY`COf-{o#tWjwe@GYpnYU5 z*yIbC)^Mx|hfkql293#Yx4hkk41J)k)JF5{b>|;@mE%vzU-bqANubt%g-{&AjG*cr zc|~z1T3r`WSGdN7P@%$0QUtOrxf!`Btvtzp@m7?m^kblCM2vqy(JbTNMf1TUH3Wcc zX;%r&_&ld$r2RWIMw{675GCqhO>1iB$bdapr=GtU?yo+*uzBhA%2kIsBP(5GC!kp} zp4Oj*_N33d=LduwKwuF+REU)I>eV;t>4Cn!C-V08fb;(xHqxSF?~T_FN-Qv#0W8Rm zKLYf*?zRAO70=HPvMdvxHy+cGDyvBozBxV;qVc2Ev{&HdgfbmlT6ry$A_HX#ftvY4 z#80@~fqd|nf-5ywiQ*}mvUi+55yizmEt;ejKA;&gG7YCDr!TY`DFsPm1lof^pjOd| ziPawLg7PCyvxitra`JNV|B>otQK+v|LY90}9J|$}WV5ytKSsm^gao4dUs@d~`fTGSO ztaSM-P~S(HwLFjuNmeP~Q|m&*9JNCa_&1-)8rt0hqd3+pSKjwG=X3w+kOF{Bp;ydS z@_%TmnolqG`2N(p)euT}nz_Zo=CwhAQn$yk=;KzXk~Ru}x*SVi=`0Wa8(J=Ttarp{ z8exZB9u!HQgbo*oOuB7jokB>`F0efi)yGjwuPx%AwYB?i*}oO3d|D6USD+rv$=^rbVlOG@F(>@srT=b$wJ_O^p{x-!F?b(Ng#a zIPF5Jm?cD#{oF|A{9iPTVzoDzV+Y!=e=NoiE9`qS8VyLzXRg6l2Fx0>xe%7+AENVC z?ZgW$UCrV@FgE}N-#)Rn@J)Ab@!k1P&nyv$dw^N%5qh?4wLfS)I_RVB*=ZWJf&#y~ z^w?_~n?&P@Ir+#yNp?!XG{5|B`B(UG$_;QNA!qfwh3@G9bhOOsnlL`(zQ68#0p>ov-9JwX0tPEiuVgGkP^+OW4#~XH1(bMxuoB~RRPY(lnQq~$J;Os7KLzeVX0EpE=a(ONOJEK-mF zhB^wS@MH@~+2h*kJeL{M%^$#{q5ylhxOL)&p7n1ts^1iZYV5cOFh1wiu))f*rC(JE zNAkI7N5JNv=}j7aI#2G9rt>f#BB@ovkIf}SF_jjwApU8Z&VR2|gkeiyw5Zl0+pvlEA_ngg2a@vVslE=AwVV`Q|9*6sR> z$phsXBv7wn1W<5KmU^Mxv>Y^IXa=oY#UM^lNdfTNaT}y!&49_L9O~yWSsW@eI94gj z+%)#7#|sLHTRN9cZF8wur^)+vmL+1=7C^Wq3uM)%B8C_F?ZEVcR2ZV$g2U7pmyj5k zocV_uVa7^tTITc%wiGzXI6O9c;`18!Xu^ZnOI6Y|=Qa%7An|7Jk0Ni;itY&w}ge=(iI)LC1b zk0QzcQD2Vl?JFe4TF>sj|I_U(lZ$wCz&RfycR2i%A#BxPc*5DnPK;v_BPzP5ZA-v! z7UzT~jUP32Yhqhh7Qrw|PHAU6ojm1_PbFn#l{>hvD6)PY{-?74v#5xB1Tfe;gJApI zq2=nI^dpl(0%R0UK(YJBj~_cmwV*u1ejtRn0$U*!EPboE4{HdJ%2B7GkEi;lz`?i8 z&iQ7=7&I0Y($gX%wQ?KMVPUHy7gLa7Wx}6R@p>8(=&op4?i6}|GIbKURv@JX>2h;Q=Idf!~IXZ`7U3Y1D&{o}Ru&0Wp54X&&2R%zmx-g=~?5P(-ox z0Y6vaes2J@ z8*{!wPy4@Pp*sRF=3fq~1GxW-`9GbohWC#t<5mRKi&H@?8R@n}7tB&X-WWuG@k2Q? z)2FTf58$0hE_tc$>suQ6IhXH~7}TN2>X87BceR&@(x)f3IJ2gD0x_ljmC8FB0ThIwv&Z?C1Itm!A7( zB>Yv4_3O-|{)vAQ>u7#Zz+ZCV3<9xTm*C!kL_kQs2DK|L=!}%KH%|m2e#syxFJa)4 zGq{Bam`c500nwp3qKgCQ`JfXu=XS^1#OgP~s-4=ExT>7k{+e7&cNOlD0-bzYY|!UM z;KAg*`2O!d=0Bjyo3J>3PK5NzgLWFspxetZLP;S!~e z#3Kr-sC8yB(5mxp=c}8qd4Ye^KtIoF$97l#>C||K!rvz%2LE85N7b`|E2jjxk+OC7+X$S}yQbODKnvkrg5JbfYPO`-b+;T4s!&xFfArQ( z_$oEW9%=CQMc?M~r8~QXY;m~#L0z7EM-Fk0X{0nVV7PD;5QY6m6RiJqe3;yLfPGF< zFzIpY%nd4PV-1UM4$0!v-c%VsZ{V=0rDL%SG?hR5m(AZ=b8o1MJ{7mVMXZzJG9R~= zuVJ=7vAKH6b)Ps4?~1`L*}38a&1VWUc7zYIRn~Wzr!RT>aAls0Qj?h9JVlH6DvH9-zuDV~%ty`CpLb?2p5EE0vH!L?MqyLt3s<^C20d77%gBC0a=~m(S&bn_v1Qyk@W7Gdqu-!NQ&y!S%Hp>N z|AZT_J5i8dg@OuFkm%F)-tn;k|7J~0TxctFQ>5vfW%0Y){yP8ci-S77*-Is} zuapOHtj&nLGh#fOe>)}P#c3jSX)qZEh>WQi! zcr_kR{-$I|gof{~HT`n}0NmIg(;>KNg!;|37{G z%L7$xgHdXlCKIoYgnL{1Z~Nh_cDA2|z9ZREISoihJP^nUPQQ2eqb$|EgtM%Nw4jxE zZD5aQX+m?2xO$8HtJpvuS+OeN1x$ke}o?8=m}{NWV^a?KfGzsUU{^G9~ZO z#0D#FA}sF1fYd1SH9JAX4bO4`)FY&7mHt!8<-aI)qkoyPjt$m+j3fU`Xr}xW5>GMw zSwf6NM^o33jHFlktgcP)0&MQjvX+2ljVGgA?_We8DhLw1AVt_3R?%!pZ}YJWdMME| zd^}2TN*;TCv?1K@Vq9Bzv|27aD8#_RRajg{hy_;uwq7=t%SQ_0cmD`}=zTRi(wZYV zZCB&}DD0N5|E%*etNujF7t=QwWp{YT8~?W>Vq&w{_`kT?lNr=D=MY!B@eHZB-vRr^ zDNj8UwAqmoF-*?AkG!A+x8Xi6kR(h3N^ncBULVzZWmr>>vO4GbUThM>v4cb#n3Wuw z{*YyNR9v-o!S-)Qhd6vWiuze}SpAF>HsIQF$>p8TLn(kPXdWP)$o!}w&yEpPw%lF{ z(J*`$V`<&-d+!B?(q5Zh4A}cs>lXLp1fJb1{Lpu z_&;?H+Q+xQl-y>;+NQnpo;TWrU)Wk zoBtKOtjQlK%Zxa>mu8t7{Rut)n(n~w0hsCr!^EX0s*+Z2K`q%{`Z?_OXN4r*rBet- zwys(KQ#EC<1dkc9omJ!1YJ+dU>(;t^2jSc|jGDp|%68=u3j8G=bkR0!@Rt5J`RrFC ziVhDey>*%IJ^5FU%D);FA!EfIpf-3UdMOSfKMPziKW7R>@bc@%s0@Z^=|vll%21L6 zEJxL0?&!b2A6-cXprwFbHxL>;eU8Sj)8~&}lKgXAJl3*2k}p2iQ;?YNPJqBkD52oagEHOafKcMULf92(GU%ZyEwCYQPNc!90Yq0%E5;7eASt*MD z{6VJS){dserk8%6W0>^cJP1sO9~hq@FxK!(E9^()3DTtoWp!*RzQAPygX0p03_kUQ5~f?%*)wO_?@G!mX=^VooJZk9 ztHGe_c+bKt=?!;9WBc<2u!@y6=8R5$zUs!LWXVrH220aQ>X+ zr>e-}q8rdf9Mi!YV%l&l2b+s*UgrsN*Qu8UXaE8tD{=QOqRRvcosgq^px#Gv^%x2L&0a zO$O!p)bC|VgF7j5XI0He(@RvJnq`r(-XkmaWF4!KppzkvY9u3a!o71#yrt#}*XCxk zxJ(Ptu~-2UxkyfqhXV}fJ>>DDcu-_wI5gOQyPQXNsQyubEOVPRp3 z2T5SV@6*El@u_*EYs1?FT_tY|y;vH5Ef~SaSnMiFp;6l=^M&M>+lKQ+Y67??=ra!2 z&!djRhuDF~2(Yk-oOwH~4+LIYBL7MJ%~j5q-$gE4zGm%F;kF1{x~k(^%e}?R%spgd zgIv(v5A!z~EH6+$^LUr3=@sT*CCgoO+h$@yK3)mYXlS@J-6<}{8^2mCYp~G70rb@p6s^Veduc1u$)RD z65@{oe^YmObqs#7>nVkr^ha6VN%c5f?-00T=x$E~{~2q=uR0z@+MTL%xH{rFaLx3a zJo7Ylp)EFibj=@m(Jwcmr)7mkUY2ma`fzaR^RMA@iA$-D!yZne$e;78o~jMMaX{y> znzp&o@HFXDNYW+7fO-oHl?|9=bkVy6^}blVknBaL6cG)JjGd93HE=zMoR zRPac(l1S}16kaz^a_Y^XJ&RndLrLjs&Tf3s`0_~Yp=9wZj{O{mrHjgseYdxekNe<- zoTH&s|HKQDGyUTFlAfM5!}M=R>ALQPO{+O~O5OI{En%oV6sM2&3vqO6M5XTqXQ_N$ zkn3@nyQCR2aB(Q3MJG#f2TeL0kX0AUwrWio_tcG2p1xQAmMZ z%94QFcO(vz9mW}r@lL?awV2QF!`xknN&VuJ42{T3NwNUn0U!hk;(Oq(onbl znR1+SEnGz>+D{KuoY=<`k4aq-F(=*qOzn4Z2W|f9MR(r4)$Pb>Gp9HemW(Lv&Ehv* zTt%m+F8mpt`z<`nn3Y(g{+iw6AoKlWqtQdk<%Qsvu6i8!RW}7E?U>mz=X;+AMq7Dm zwvGTjh|^M!qnstIBtS~;I2vWY9Hrc8FoTbbfs`*zUuGYEA!+#9^WKe2)S9aM$MdRA zqHQ)0k&m@*V8+@y3RbS}s}lBCjyaJJFV^N=k`xqCrM#x_!!7#!1*adqPxw(BJ|bb) z!xqL$A55ilCM87<<22t*ULRp+qRt)%e$k;+Q_&in$%K~Iq#fi z5>M!XJg7YL2&czy-y)ar&E?y+I;q69o9?t{Y$*vbQ>p(3**^&haV+zL(ddmwNfRMI zJ~ZuK`TYtf%_Zi>^JN1@N#rWm-o2C2It>23%BB3ow0qDqP_>t8hx`!F} zD>IHRF}M3mR0dYlRFu9rA36>T*RoN;Xbg491R~ebm%kfb$$Kfu*1<>}Z_O$clt31xZ}+O>g~4N4FZ< z99PNKF0;O+)8U;58Bv0F&Eg*(nZTzZh-JAqN~q@+BMnc~x?8haLu{E^qwe;C*A*`^ z!MzB%ArY$P!R4IS9Me|oBc3y==$3iq_anxW$B>V>xAe%cu;@saqDyLjFc0vL=uZ)M zp3dlc-1Y0KiFT)8Cvt&j4?0cItryUnu_J?(I#*%xez?|rPDegEL65x5;~Y6c%o5>* zC1PHcSI?Z7^dr`EvONpcQo_W5>|3neDG$yZ*KccypX0&kC~=`i)bo8Akx!D5`SJda z`XIUUL#IogYBVQeh$m!oSx}PF+2`A`7IRu;I1yq=>(nlb$TkgEyT$Z=Ij~XYG zGZ|*|;V|&&r6Z(P73Ixg?eQBhH7N$5Gg(Nzwp;xQ@KE_b^LP>Ux(P ztr)v%G#pJ9S9E?+&F?)|)`(hNfzPo{h#@a^TFUBG)H*FZkX{wD>UrLgS{)}CmE%pa ziMz4SiIFdA9aArcMOQRqoAEoq|L5Cq^vmQsIy+-- zM2mSM2Ql|2Wsi+wH*YpqFifCeo{b0RY@bg}r66DOR-4=9LATB;cbbe0?rJlKxNFGq zcUx#_l2r45SL5;km1b4`o9R3`*jmY+nkxQ2$4_KXx3zL8gsJ@m1pZ{c|@p7im& z9X``b6Rc-i^Ebg$KsRkiR8||VkgO-?y6 zDQWM%F7;g8Nq^U?3O7g{Dq!}1h8Y2}>`WD{LnqT=z0}MJ|DVLR=*y9*Ys)d8QjV4x zmhQbe?7SOXTni#j)JcLjKfecM6jHtTBaS907FU1 zuV8@dLCVDZT`~dNSK9~oZ*xv}wJbBWOZVjs+DN$#J_{!)83|7&ojr5rit3}o_6;%O ztLT%rMSLiAe!(m1)w4ThsAEY2D;iyAtK6#p{So4xcwb_>GqCDk^n-Z*OR26PKZ^G5 zOXc^nKP!%GAXeNn#8k)&t)v!t#1>h`M%5OAsjyY9%}RaBCnLfUekG0O-W<2|}(e9+kWbM#&n?;q^o+-Q;PH-eiC-=Ea$QMuCP!IV% zN}k%@jGFMkA50uayPQikxR09KI67#sXj71lmiB>-*{#sTC8|pgSsNo_X@w>+A=s0s zD0>Yz#A;y0fnDJojZy22BJh|@^^u_D)}=fUVy>w?GGldm$WV_-61b0R(mdL|!7P@w zS`%)oJZg@@%z4bM1gTq?CI2RW!N|0+9z7B=GU%qm1m#D~I1=Oz&+txCV_`W**4_cD zvDQ~x4MsyrIr)0hWo6fph+CK18;~*XYnZk(1$d?{+_Fb7RUL)n+XB9-oVf9JvEU1D_cdR_N93n**yACeNcplDS8E$D>?B zezS$y&1f()e8(KB;&-l@q8U39h13CFT>)x@#Rl~_NWYVTa{KpWY#RDb&Zp_C=N26-3pFUY8VsZc#Ht5-A0Jg_Bt|ARK1GLCOe(n36Y ze9xFk8I}J1(6>9gUIoE+KD;Kd;vB1&T*fJ# zQ!X#Sm*Fg4Fk_N}@4fNcsXg2k(~L<499-VW0sK?ZYr9SFH&nOp*FBxg;&>&x9q>E3 z_G;R@d;>Syd2eb$cfQ1bcEXa2Y48P{VM>0ox&lQ!f|v0$S%CGNJuU1+ad( zVV?a00^CG6u3V9Zrm?z%%F4=7iHRPqd1r67d_d)mMBXr#6LX)h?y93o9j}cXNH*r> zMF#t^0kYmtB<~9ex20G-U?N>KtFnHW@NKy{sh_t(5AhQ)%ayzSeX?gJNFXHrS_P>l zWk`?!HJVWtDy*5=+24Q{2CR9e>fh))tt~D0+}+)MjM`dTzrB?>l{Xs_9IWEvvd!G= zM_1_PQ)F0CWa861)0#`3<=kTZ&9?4z{ZqtK5pR=(gP)FnYFSXbs6bnA#bwe%A_dM! z;oDTzQgUyJkBqJRRZ&sh*#vuqpGLo?KDTysh>Y9v@bFYXhgJYmrlL@c?3y1WQ=~#7 z27i7>r{3YQosjeZA&83Um}f1jm3=STv-!&xmaKecG^$*szRdc@g)7V}DZ*eYRKD zen)2cu4R>T%Ig)v-J?3kneeRtIe?nT_2USYFclm}< z-3mf?BZ?k=j==c|O)|cN`qb|ZzR_=kwI*FO(^FHASH?cRr=~il;_}oaBkFxuxTxD( zgH@?`1$19OS|k~wF5yNhWCpx(UIVfhTC~-oCmMrDiESNr4rL>m<|FQKd^HLaNdp>p z7WCocl{)>V2V?z?w5-7VD1XX^+cWpanfZ7^*y^Y}v6tE- z@6B_g9b&lk6Tq=#&b_myhXnFOjKJb8vKkhYMFaD&YQvVFno`8IXr?nUi%@JY|K{%2 z>g0N#QP?~v&!7vtL?2Wh!vBkyV#d13s-#12nQ+B}`$6W_$9**4r7Gss(O=w9ff|$J zn8|qa*85VOj`O%5{)RxJf)4be)2j8XMj9LD)uv%%)n>BA#3)A9cy_r}W-0HC)(VG> z7@>+>{=fFV1e(hAeS4Ei4y6-L$(W&tN+B~RZERC)^IXX+B=ek-j*Qz>WQa{>nKMgC za?E7PROY#4w)yUNYdGut&bNMNt^Zp8wZ64J>zp*$@B57Ra}U>ZT~zy*^aa|R1;Fcs zlZ*SxIV;)CcyJ%Z6z~o=-kCz9Yhuq|^{vS%Md$xvAFhY$LIFz?#D^;X9$ zE+v65YqqgDw3E}`jy14v;6UCBCqUxYuMQx~>;dBVLZSeja>(lm9 zXSr9YKk`35C^U&|=(~o7Fm?|1MUG=W*COlMKW3&pX^DBiGqWm2=orG)8sjUEr$C+u zIeop&w|aWmTNIffejY8jA{LGu_^t04kDU$0M?^$?TFLB>3J>qN>eK=nd@^9^y0_Ms zmce$P78ce4jfqY$C?3rGx(6F^RqHCc!qOtoU%4lM&0O67Bf*4v=|dVUf=rlc6aCG> z3LRvlf?Y0hU$w!sf1T!`W4}BRA=)jd(rcFWwBN&rN-yIYU&*pKI**b2NrZ7B@~iOMh}5Kq;w!@vFoijiBbqn zuYm-f0NmPBlkkE&oKTR2(;|w!@v@4nv^^DVf>2tJ_a9G7c$PgVJ@A5G?;wOc%6wBUx z;K9J&wx{#R?YR=KKH7W;n(I87`SF+3w2Vgdp8-TSvFS2NJs-Tzu~~ScQ-bqK{0F>| zD@a95!bN&OU`BziAQ*sOgXjh1ZA$b2dDaCo9Lv3C2~$yeHf^E{6&=HCzP_Yzi?0B~ zx3VxaWJMyYp%Ix=z3q0hHy_;?#oGaZ_gDuB)Z*iyB~Ry^P?hxVQx-KsS3b*J4qRwP z@-cXoQS0*%!*xP^)uy#O`v9>Uz@Hb>8})Sn7qg zVfI>kN_W;U3t@wJ=5Yok#6gR0U(t&7j?ei`&TL>*hm)g>@W&>kTTtftfLrRDo1L|$ zqUV2WRSmz%txu#)I~DPspaJn!dTPZrcZdB`?VBktOD>3>KqhJJ=w1d#p}1x>38&8= z>u`8PxsT4a(-G9`5HsJ|94A_YSmC`4O(XWD{BXav5^kBeZLd!_Hu9FN&3-_sTv19< zf1$mmLrv)80&Q>)sDQ+Z^3d=ALn6Rj&O7EdP|RJMvq!$@tQICf-y zV=w()cRcRMS+240qr}dm9UKbJlue$9>F+=ufRlr0u-#_L?XE_$4p>O$!rp%C&wH19(D=c3rDj-DHSK?p7!8z1Ke zS*MfLS09TwW+0rjRE#3(j3N%L@RQoPgxX)ub^__g;Oq>FoHU9AqAXG<+5gS-nYo!D)CW8 zQ67)qf=*`771PtRty+fV5eX8y91}w)waPp2SOxCoxlJIzS?E|I84NYsU9a>IK|>yJ zoN+IML}+-pT6r7}XaB*=u^%orcwlStfbkL24g>;4pjW$Z4#bG8u_XSlM$FrnkDfcA z{DI}}JrgecR_K#DVSBm-4LMBn$egvXf|&Px_2`YcFF9!kDY#m~rQdTUyt0>>l+uXc znHMnMH*fLt<)m6*w)D?v%`LG>p5Z!uy5r^D{Z^~8GBVtpT}oLb zR7YhWhyJPP>+6>@GBSE1{hMa$V0irav4rp;AJniD7k<{~8B4NIzuFU3J$#;4GtSfg z1vkOoqVezv=h(}o*TcsGl)_Gr_eh50TgmIP%ZyZws=eNIr%wuNY6@Kd4;_V4x&>|` zIQIrrD=jes3V@i+@QjD`2oyVA5{JN^1M5LLY_HR&Pb+|biG`*2!KMT#k}X>ZKnAc* z5MY?;xfRSNDTGl}2YWEH^{;K97LEw zP3FA}p`$uO#e9^H3J+FP3udO{0#=v|-``kmGf|>AboMqjg00kQ>-J5v7c|}IfEn&0 zq;fDxpIvt9N3D&+F@sro;aVc5LRLYkc703Q>va8q-a z^dR$tprGu5GpA3B+Eka_bPOSBl~<2sIs5^(>3M`&H$LFR z)Td{Vc$Dihe3O;*5kmeHh_4`s-VNc>{GlZvZ<#aD+}!*=@%O>*{Y}lyD{hoNEUJdB z&rc;gaRv=wds%l3M90IcXf9rG+az!_;Z|M0;%g3JNt5v-k~>A|mz0qB#cczAetG&0g{dTW@u8E5BZuN8h1J*!M1SD0znyck0y{{5 z!8EYj3pk3RM;}KOgG0(GEoK1}Y=*0#0vkg`CIfnh{fw6y&2%Wz->S*oDn@o()aYBC zCv&b5$hYLsjO*0~gS1`0=RCSmdR=WaczJ6oKq-B|%&(qdXZWLKi$m!?VVM>!{Wmn| zd0o%iDx?x*H)^f3o`RDnFf;fvt@8iO?@3YLDo>efOwkB&G4&FQg6#5=+En{~d@<%* z!h}BnnZ)4W_i#rD4^kr2V3E4vR@w2dc{~ch7k7t!3W13;p71%6_kn_o@O$&e%A&o; znsKl8()<4+$RI$wU0TRsUoyqh9abCoDB@LrLlK&uBIs*l5uuJEMD^ulDTU~BrM83} zg|MMYHvi0N^YROq#wv5vU8`d?;?9LWzK9H^+{gVKO^2%{Bm)M{>0R@$TI@5t)i=g1 zu_ICX(OupI_ilDaG{uMV@4JO)?{Kg=q2-o9(Ypbni~Kj*jEQOn_m9=e~A*;iL( zsH62d>}Wq?qLTX2CGVabVv(qF(8^t-y_CD^YDR|W&RgrNNT+?mWx#X#va~d2Uuu~E z{N@;tE`5BENld$3^89%$AD?VAkKC0j-9|ebTA<&j1Xn|KD_rt`@`|gZKN}SCtbp|g zhX>n<1c&zxsagZnqZRzz5B+97n**E8rQ9XqMk(R;T(*%CY`?W1E^ci*6vivKm5i|8O z`>OV~Xyeqd&>L{$;LxSdIV=LK9w29$x#9}89xk1!Z;J8rtHXu5R0s!ifanSF{~%8Q zQEf(ATAKJOTv!`j?M9KM1SA;;oS$yS=jm}1UV3*2oA+ORAJ{kd!Uh`~%tL}4pr~`n zqx&tIKDCeg;mT95x%Z!Gf3QF0_fPA5#HN!Sg`+zSNOSti95X>KI$O)gQv-Ok)U~xA zZLBWjg2C1SA0G>RiUZe3c|%MvCa$s%;gyN1E-o@b0he|RB+I1Zk^)7%nI8YnmJzp; zL#**hsj1T`%p^ogc zFgc#{!5ADZo#)mO8cjLB(aN)H;w?U}z6SzG+Ft4xEj5mtIT4%~a zm|2j)lJbFcozzmP)hJIxHt!p|CjuiZG7>axm<5aaj zzeMaVOU;Xc_TX-Ope%kXj;f!LwY|4QE?R=6F69xiu!|bLO=DM+`RAX1y0tJK^k?VV zP2az}H8eC$k#4up#-WUyPxd#;Ps!%%Zy$Jf?3M#pc%O;V!JXcmScW_dg^ALTPpv1} z4=)?>T72k6Hx(`F94v;sD@#^OvGf84)1CD<)RCT`B%voh3|%SqbQ5yuoh2Gmn)%N( zW(>V&dKao6`q#hbY8BGQAJ&>^5sbr!QtucLGl#9$3s*P{qz}!j%-$Cc=-Fh>ozwVZ zBF*J3jmAl;D1991)`B0$6C}8TYBE$!zd8PcPy!(q%JiW2*Dmh)AmP?6-1v^f((yxn9Iw9x^P-kF=eP zk3z^t%1j5_aAzO-*s^JB8u#a+Wn+&^Wzs3UJ=HaL%x`s@hx?+WMnx0|qAZtV{L!Qyo&rcUW1CKc1l7p?Vxi z`(7re*ucbZzYEEoav9c?{jdbW3WPQn*(N5Ja16rw8eXM^W8`7wN=f>8T=sRGjkMt& zE~0e>jN$BlG7KkeFzXz+Ep0JhpIUY8Z&vJE^_*V~9Tt%=w*L9DOCZgVbbxjL8Xv$a zHmVh7wz)NEFItBLhM(Ck;vco);(5ocio`J}T^0%<>NOsPYv9yvEO~Q%DR-Wsy`bG} zr}4D{YeKNqbgWH`5^hS%lm|!6A9f^-InBf=JiA^qPG6WO8Y(4z_Zn!!v)-jCX9O$b z3O-rb*D;)V|3C>kV8(T?AIZR@P|Mm;5tTZ9LPmYt8KW6uedB?l`|y~F0j?Cv97(#7 zjHqIssPoLx@`w1vaxc`X72f)M1J3;&BVy?x5k2BtYM|iC`_Erviqh$9%D>XBnaGQE-=kR1D-%HY$ z92OV<dD4+U^ji%s!h%tJeOTUu#iJJW$Dyc?uUMdG`UK_a7#6Ao z&N|1>wp&P-eCQsM9wf_ocirN17t|z`W_Fg{66_kZEsTb@!|iB&4KHI}SBe4>uCD?uRHsXW%~UFY*tJNm?=A0Q9L z1JRvbc_sYzWe($!f5sd@`h~cAWT`<*vCdnK7hMC`0`fVWgKV#hXv#gJiMD@nG+8h1 zG5q ztE!Q7Z?3T`YbY&($=r!S(;TnQ*FfATu_v??)AZ`=&tG5Fitw;WicPz;K`AyC+zZ zot&OCj>Y0F*>7D^MTjNDMB2qV?R&2@ILz7%yBx$iWL1gd?@P6pVUhd7g@VD`^&^-3 zUcbJ4JV{t)Y!%>Hm!EWwd=+VoaS*MT)0t*x&xpIkUX+Sf)_uMCFj4k<3HaMK*Y%|n zWg1D&dk~y=7&nlcvNxYVKxh6#T7+d0uI7%5l~l8F9Q@i;iVP!4YVk{#_T_sayJ#?S zc~+Xa>o|V4t;SEgq!}jL&?&r=NzfUQM$;APN&6TrA3_J~$&E zxmvE%bzskW5fMcOsrm=nQ{&Mov%FB4Z(kY1WqvVtH^=j{D2c2px8!qu8;KYI54uD? zeyp=l`f2g1G_WR7wekOs#W}nm!=l5~pFZ+>AxqsBq zce9I6Imjc9{my}T%GKUFWly;s1gFVZ+LJ!C*Ea1+1T6&;Mg$&z<-~<_vgpqd5`YwF*G5@ z40b~7m~Sxy5ykd1Crd<75I?nYeIL=Fe1Amj-&-H0LR|Gw;0ygn_}|}hy{5z8pM-pW zerz{20o6`45K05EZp2Ow2MSX+AM8tA+^&TK2C*u;Q3ou^Y2*LWn!k-DouxC1oJaP3 zJkijg&iQ>NCU&K)YAA^r;O}h_ysaH&(#uGOa%WuUI8w4_?Z{se)WpG16?$* z*d(aOPG)50U}57i9Q3y-pA#&9b&0yK(!Io{bc`;}b^I|Tyt09Wzq8ugyXg_1G#ZU| z^7m&6i;U!e?A!9oN~$VEF_iIQ3w$0f;+i`@AV1;0G7GrtG7;zbEV#T4j4D^WH~WWf z=tK=XWktpPY35yZ618}7{=&ICdE}33UC0!#ijk(Cqa!i8&hv5Y(i~LH8y$eq5%TG3 z7z{>a^~3FDD0ucT{_5KaWc5YP3x-I|X}Yf{^NI2^q-uaj)wuyLGXe~N`f)!ar|)aW zZi8o_%2YL26)*1IM?>Y?1iF*is>w0}J=Ji(848kb;h_2~A z`cOy$Cfy}6A_8COzRJzPf!;05=8LkErGupD6-ea_o_~ zVBX3yVBg5=JPxI~n)@YmB?Ih})*?#rO(aTzjePX2dU@}M#WlP|Eg zcj$`uh8iS3E4RKh3Zm~zIi61-tjD8jhF-MPD8pbNgMxvXaJ76^02Nk=nbhVS9b+6Q|e>(`S zQGEs7)pSU~>@y@G{r&bA9(MN2AlT6f18;^9r*70sZa-aHb0L;%^KZC?qH& zi;Rn_4T?FA6LHS+xIy;?ab#Br$_7YHJk+0QPkAU4M4tjT{hMG#RBsL}H~=}m`PjS2 zRKtC`LPkR!u;u5gcf0^D1HE0l;6O$VeMmw-q+w#pIdFt?P1Kq!aeTDz!C|&SU_gTM zhh=A)>U3MO|9sV^4dHW$x4MT%1#rbGRWlT#JXZF%;_R;YBAb}z4ck>xWD4@g*XR>A z39siVeno8;SaxN|^h49<`zT&Ob(gus$STcLdTEWjdpaXMJ?{h?TQ>6A6PL+IyhnQ{ z61;6N`M_<@YIN@b%W$UHCL|wuG4Eg8`sBfQ!uQ@;`|SMu_P!Ull8c}=pZWZGD>pB1 z5s)NjL5Z!6nKjw3;8TI*&=C~Fq1~o@w~w5NeKh8&m4suR{5pv2*cKAxmSfp(_7ye^ zjj%jeXX^%vDs`yUZEbCt*4-Z-2@%@_AQlG(AT|=qHPq@Jk!G+PBG?B&8mj65s zmD0hO!Sri^hSk%);+vTuVfhZyz+WfZB!Xz~+-ccH>Ha#8`Ky^GHsjOnYHz?n7G4hv z3oGhB6W#bmj(V0J@}--r?&9N6AN1Eh6MdiZ*vIZ3YBBo8f{KcYeT388++W|ag}j|| zm<~9dv*_+Yeo$$>q+`r}%I`Nr6)g7GQa?4P{DclDKrcgu-*BMJu>&v)gk$uiGzkQf zLiP{gS4oWzBS-tA8syC2A+35dv8{Flj(*f^IQWp|;&#I|`{r82mI8tOU0qF$4TI#< z5XA~i`}1qB_^o@+vE4ecC=g`@iPdVTLO%t`K1G;REez%lAZ1PlqAUf^jjwH$tAkcT zx?~w|zRUu&1Z!cDe&)q6spC{T8iFj{{2PB%5mxk+5+n>N-ImWY@RD}@u{{@ohxK?0 za7)OuVUa$4OKlBf0=z~Bb}#or@o1*2(%l(Zk05!l0jh^z;eT!skv_XZak+D8l zMb``*yEHWAkl4 zo+HqY<4I_i#bx5zZhjEiN`Id#?bqpu_qASjicG3;+^KvS1QoPxs|m%hg_c2x$sDMm zoc#S*E)(-v-`v66Ftu-M5GiUCi-eH&GGcocbMo~(3iZet2;`^dN`|7?#W!aQKJbrR z{P9B&8m!R^?@iv7&w_HADhL8v5SM~QQ4HTmN!B?%4Q5JwQm(=Mm{X^TD0LchrZ;BV!TODvts3c)E4&7 zR--gNQTN<`-K}pDGjE2HqnCIa|97tfmn}B!DHX z|9yKU8KO*doc5E{tRjxp{6M(^4on{uYo5t$UU_^Tt(L0DcbMLnl6Xz#Te1+wnq>Eg z{DE$Ok@X0A@`V{<4Bv?Uxtx+p@11R#&FF}THE_4Mh`8`?HA(n6(p?MxBV>fN0PA8* z63(Ze9y97{0d@IbU22s1+}{GSzq{%Ga~J$Cum5k+;QzwQYcP;Shr58l!N*vg4=3-gQms0L=3!t=*)Wtb zyB(|sj|}h~qzY+&=`iY_V-(*vaFmbQ30Xeli4(q^%5YG&QWhXMH@%}9QxG?pp6~iS z5vSQSh${dhIeGuMjf$!1I9Gk1%jfHUetr!jc9Q@Wv2NXlfB}TVlmJAE_&&q1QESwVHhh*u4) zxq{X`iB!i0C!v~z&Mv~+YC;8X)H$_>1F@fpbjF8~ z^**eCge*Rq6X91R5U?Ew26#&g5scmBq&Wr&mJ!S$_^6n6=|kdp=p&S~H8fIUZOSl6 zqziHZ&k!%*CClr+BCF+(56^P5vi=ScAcBSkKkGCmATY2~pj$r~aMEcImbej?yO%-V zw0ouqimA(h#pMzgH&9ho#o+-@Zw@*U0Mw2G9PsJf3hct85Xo(AoCApZPQxnCB5;Jh z8d14)$rnMv@*0#Crh5}F#-WDU`RHCEJ5venGXf~fahtHE`-u04w??`C4;)UI8j5S6aAn-h+**!tK*>F?e zp$%Q+ISu=(l~zli6>+w#DvO=w`Y<^A+qXkDMTmpyvx6+1Z*-6F*ttkHT>?m z>+Fb=mkMPlBVt_>vn>U6)0Eagy zcXAbP31yWI-Yf%EHx7)etLySg2&1UxL4RqaU<*-FGGL~3K%$^t{1CAji@a}6AUV@S zE+oMKK(Im(ZI#2$CBFR(qO$1T-rgcY$TEmzD11>AEwdkc_h~x@jh$mbtEz^<1S=mk zhRSv~0YG~7;+*`QOKmx2Wo3ig2#5-%EGxKL)D+23r z5$|ph!uQzj4r!$!?vTXWH+Xd#^DBxIfCCqYhsd^Z&dGglX)M}^8{pkIhe6i~##LQN zs^maW*;(c34hcz1knDvR(4uuA>ERh1dnM4dDS5Mc3EB-yHN{@c6)Xlnx`%;y#}0Mp zkGuz|*Wmc@*}cTQ=AyAlT-0{3vhL|fSd~tX1XU25KZ)5lgrH# zo7UO7Me-2ncETnU0E8ZF{_dHjf&_vht`Jt+k?k^kVWGN)hDOke@G*#TgNkL*XpD$R z53#m{Qa)&Wcp1s{*QAda!9@*fhkcYsSuk$kF0Wzb9Xuoy#5X3B;hsV$ zyMgOiq(nI{_`1XO(y@ov*xaNwQd;ZcOsZaG!OQHViea}zENbc0w@5ya&fS#Mf(XB9JG z%q_BNB|;>8!e41$_C)Njoh1ahP=AgCVIJvBt=#vBsZ@>=nQDD14RaX{uS|i&0tO5F z2i8^L)t97zJ>&++6__r0MPN18je#^0B=Jn+z7}mR&pd+@39)LR*3^}znhYn>Sq#OY zL){gwOJ70Aai+N6Mhy-RK(VMpQWQvO;c?Rtp1lWZ#7G?OV?lKYoJYFIMhCUBOr%*+ zCg~Lw`V9>YW#chcg&ynnV0j}f6o@XR5E^QJM4#x{n`?|*GrQ}FI5{XE-cvXK`se5W v3!dHXo8j00C^G#2mwy`@!T*Q7`GTE8W1@N3;z3ghq&FC}^5vXMSAG8v0xT5V From dd5fe05be2252e04b0308149b46ac783bd68e42a Mon Sep 17 00:00:00 2001 From: KJLdefeated Date: Sun, 23 Aug 2026 10:22:45 +0000 Subject: [PATCH 39/48] feat(logprob): add Triton WS2 vocab-parallel backend, shared fused kernel path, report update --- .../rocm_logprob_implementation_summary.md | 14 + benchmarks/benchmark_rocm_logp.py | 178 +- .../distributed_logp_latency.png | Bin 63649 -> 62072 bytes .../results/pr328_rocm_mi300x/report.md | 162 +- .../results/pr328_rocm_mi300x/results.json | 2692 +++++++++++------ .../pr328_rocm_mi300x/single_gpu_latency.png | Bin 92977 -> 91786 bytes .../pr328_rocm_mi300x/single_gpu_memory.png | Bin 131045 -> 129020 bytes .../ops/pytorch/loss/vocab_parallel_logp.py | 158 +- .../ops/rocm/loss/vocab_parallel_logp.py | 126 +- .../ops/triton/loss/vocab_parallel_logp.py | 261 ++ rl_engine/kernels/registry.py | 41 + tests/test_rocm_logprob_backend.py | 105 +- tests/test_vocab_parallel_logp.py | 38 +- 13 files changed, 2561 insertions(+), 1214 deletions(-) create mode 100644 rl_engine/kernels/ops/triton/loss/vocab_parallel_logp.py diff --git a/_dev_notes/rocm_logprob_implementation_summary.md b/_dev_notes/rocm_logprob_implementation_summary.md index ab699b4b..d019b230 100644 --- a/_dev_notes/rocm_logprob_implementation_summary.md +++ b/_dev_notes/rocm_logprob_implementation_summary.md @@ -158,3 +158,17 @@ TP contract、tile 顺序 merge、selected-target 传输和 active-mask 语义 构建期可调参数：`DETERMINISTIC_LOGP_TILE_BLOCK_SIZE`（默认 128）、 `DETERMINISTIC_LOGP_TILE_VECTOR_ELEMENTS`（默认 8）、`DETERMINISTIC_LOGP_BACKWARD_BLOCK_SIZE` （默认 256），通过 `setup.py` 的环境变量注入。 + +## 2026-08-23 补充：Triton vocab-parallel backend + +`apply` 所用的 fused autograd 路径被提成共享实现 +（`rl_engine/kernels/ops/pytorch/loss/vocab_parallel_logp.py` 里的 +`apply_with_kernels` / `VocabParallelLogprobKernels`）：backend 只需提供 +`tile_stats` 和 `backward` 两个 kernel，TP 传输、固定 tile 顺序 merge、 +target ownership、mask 语义全部复用。基于这个接口新增了 +`rl_engine/kernels/ops/triton/loss/vocab_parallel_logp.py` +（`triton-vocab-parallel-logp-ws2`）：两个 Triton kernel，按 `BLOCK_V=1024` +从 tile 起点分块归约，masked lane 贡献 identity，所以归约顺序只由 `BLOCK_V` +决定，TP=n 与 TP=1 仍 bitwise 一致；同一份源码可在 CUDA 和 ROCm 上运行。 +registry 在 `cuda`/`rocm` 平台都注册它，排在 PyTorch reference 之前；ROCm 上 +HIP backend 仍然排第一。 diff --git a/benchmarks/benchmark_rocm_logp.py b/benchmarks/benchmark_rocm_logp.py index 6758bafe..cf9ac57d 100644 --- a/benchmarks/benchmark_rocm_logp.py +++ b/benchmarks/benchmark_rocm_logp.py @@ -63,6 +63,7 @@ _local_tile_stats, ) from rl_engine.kernels.ops.rocm.loss.vocab_parallel_logp import RocmVocabParallelLogprobOp +from rl_engine.kernels.ops.triton.loss.vocab_parallel_logp import TritonVocabParallelLogprobOp REAL_VOCAB = 151936 # Qwen3 tokenizer/lm_head width; 151936 = 64 * 2374, so no padding. NUM_TILES = 64 @@ -79,7 +80,8 @@ ("tp4_cp2", 4, 2), ("tp2_cp4", 2, 4), ) -DISTRIBUTED_PATHS = ("native", "ws2-reference", "ws2-rocm") +DISTRIBUTED_PATHS = ("native", "ws2-reference", "ws2-triton", "ws2-rocm") +WS2_KERNEL_PATHS = ("ws2-triton", "ws2-rocm") _DTYPES = {"bf16": torch.bfloat16, "fp32": torch.float32} _SPAWN_TIMEOUT_S = 1800 @@ -294,6 +296,14 @@ def ws1_triton_train(logits, targets, active, contract): ws2_reference_fn, lambda *a: ws2_reference_fn(*a)[0], ) + ws2_triton = TritonVocabParallelLogprobOp() + + def ws2_triton_fn(logits, targets, active, contract): + return ws2_triton.apply( + logits, targets, contract=contract, num_vocab_tiles=NUM_TILES, validate=False + ) + + paths["ws2-triton"] = (ws2_triton_fn, lambda *a: ws2_triton_fn(*a)[0]) paths["ws2-rocm"] = (ws2_rocm_fn, lambda *a: ws2_rocm_fn(*a)[0]) return paths @@ -398,17 +408,23 @@ def train_step(): f"peak={cases[-1]['train_peak_mib']:.1f}MiB", flush=True, ) - if "ws2-reference" in outputs and "ws2-rocm" in outputs: + if "ws2-reference" in outputs: ref_logp, ref_lse = outputs["ws2-reference"] - rocm_logp, rocm_lse = outputs["ws2-rocm"] - for case in cases: - if case["dtype"] == dtype_name and case["tokens"] == num_tokens: - if case["path"] == "ws2-rocm": + for kernel_path in WS2_KERNEL_PATHS: + if kernel_path not in outputs: + continue + k_logp, k_lse = outputs[kernel_path] + for case in cases: + if ( + case["dtype"] == dtype_name + and case["tokens"] == num_tokens + and case["path"] == kernel_path + ): case["mismatch_vs_reference"] = _mismatch_count( - rocm_logp, ref_logp - ) + _mismatch_count(rocm_lse, ref_lse) + k_logp, ref_logp + ) + _mismatch_count(k_lse, ref_lse) case["rel_l2_vs_reference"] = max( - _relative_l2(rocm_logp, ref_logp), _relative_l2(rocm_lse, ref_lse) + _relative_l2(k_logp, ref_logp), _relative_l2(k_lse, ref_lse) ) # validate=True production entry point overhead (host-side checks + .item() sync) if dtype_name == "bf16": @@ -660,6 +676,7 @@ def _distributed_worker( vocab_start, vocab_end = bounds[tp_rank] ops = { "ws2-reference": VocabParallelLogprobOp(), + "ws2-triton": TritonVocabParallelLogprobOp(), "ws2-rocm": RocmVocabParallelLogprobOp(), } results: list[dict[str, Any]] = [] @@ -771,21 +788,17 @@ def train_step(): flush=True, ) ref_logp, ref_lse = outputs["ws2-reference"] - rocm_logp, rocm_lse = outputs["ws2-rocm"] - nat_logp, nat_lse = outputs["native"] - mismatch = _mismatch_count(rocm_logp, ref_logp) + _mismatch_count(rocm_lse, ref_lse) - rel = max(_relative_l2(rocm_logp, ref_logp), _relative_l2(rocm_lse, ref_lse)) - native_rel = max(_relative_l2(nat_logp, ref_logp), _relative_l2(nat_lse, ref_lse)) - # Count once per TP group (outputs are replicated inside the group). - mismatch_total = _all_sum(mismatch if tp_rank == 0 else 0) - rel_max = _all_max(rel) - native_rel_max = _all_max(native_rel) - for entry in results: - if entry["tokens"] == num_tokens and entry["path"] == "ws2-rocm": - entry["mismatch_vs_reference"] = mismatch_total - entry["rel_l2_vs_reference"] = rel_max - if entry["tokens"] == num_tokens and entry["path"] == "native": - entry["rel_l2_vs_reference"] = native_rel_max + for other_path in ("native",) + WS2_KERNEL_PATHS: + o_logp, o_lse = outputs[other_path] + mismatch = _mismatch_count(o_logp, ref_logp) + _mismatch_count(o_lse, ref_lse) + rel = max(_relative_l2(o_logp, ref_logp), _relative_l2(o_lse, ref_lse)) + # Count once per TP group (outputs are replicated inside the group). + mismatch_total = _all_sum(mismatch if tp_rank == 0 else 0) + rel_max = _all_max(rel) + for entry in results: + if entry["tokens"] == num_tokens and entry["path"] == other_path: + entry["mismatch_vs_reference"] = mismatch_total + entry["rel_l2_vs_reference"] = rel_max shard = oracle_logp = oracle_lse = outputs = None torch.cuda.empty_cache() dist.barrier() @@ -849,6 +862,11 @@ def _run_distributed_world( "tile loop for the per-tile FP32 `(max, sumexp)` partials, all-gather of the partials, " "fixed global tile-order merge, and a PyTorch autograd backward" ), + "ws2-triton": ( + "`triton-vocab-parallel-logp-ws2`, the same contract, transport, and merge with two " + "Triton kernels (tile statistics read from the stored shard, fused backward); one " + "source for CUDA and ROCm" + ), "ws2-rocm": ( "`rocm-vocab-parallel-logp-ws2`, the same contract, transport, and merge with two HIP " "kernels: `hip_deterministic_logp_tile_stats` reads the stored BF16/FP16/FP32 shard " @@ -865,6 +883,7 @@ def _run_distributed_world( "the WS2 operators all-gather per-tile `(max, sumexp)` partials over RCCL and merge them " "in fixed global tile order; CP ranks shard tokens and never enter the merge" ), + "ws2-triton": "", "ws2-rocm": "", } @@ -1060,11 +1079,13 @@ def _write_report(payload: dict[str, Any], output_directory: Path, style: Report f"topologies, at {_range_text(d_mem, '{:.2f}')}x the per-rank peak memory " f"(absolute forward {_range_text(d_abs, '{:.3f}')} ms)." ) - if "ws2-rocm" in style.paths and "ws1-triton" in style.paths: + for kernel_path in WS2_KERNEL_PATHS: + if kernel_path not in style.paths or "ws1-triton" not in style.paths: + continue ratios_fwd, ratios_train = [], [] for dtype_name in SINGLE_DTYPES: for case in single: - if case["path"] != "ws2-rocm" or case["dtype"] != dtype_name: + if case["path"] != kernel_path or case["dtype"] != dtype_name: continue if not style.keep_tokens(case["tokens"]): continue @@ -1077,13 +1098,13 @@ def _write_report(payload: dict[str, Any], output_directory: Path, style: Report ) if ratios_fwd: add( - f"- `{style.name('ws2-rocm')}` runs at {_range_text(ratios_fwd, '{:.2f}')}x the " + f"- `{style.name(kernel_path)}` runs at {_range_text(ratios_fwd, '{:.2f}')}x the " f"latency of `{style.name('ws1-triton')}` in forward and " f"{_range_text(ratios_train, '{:.2f}')}x in forward+backward with the same peak " "memory, while carrying the vocab-parallel contract (tile partials, all-gather, " "fixed tile-order merge, vocab-domain LSE export) that the single-shard Triton op " "does not provide; the gap is the operator's fixed Python/launch floor, not the " - "HIP kernels." + "kernels." ) if component and "ws2-rocm" in style.paths: comp_speed = [ @@ -1102,15 +1123,17 @@ def _write_report(payload: dict[str, Any], output_directory: Path, style: Report f"allocates {_range_text(comp_mem, '{:.0f}')}x less transient memory (it writes only " f"the `[tokens, {NUM_TILES}]` FP32 partials)." ) - if "ws2-rocm" in style.paths and "ws2-reference" in style.paths: + for kernel_path in WS2_KERNEL_PATHS: + if kernel_path not in style.paths or "ws2-reference" not in style.paths: + continue mism = [ c.get("mismatch_vs_reference") for c in single - if c["path"] == "ws2-rocm" and c.get("mismatch_vs_reference") is not None + if c["path"] == kernel_path and c.get("mismatch_vs_reference") is not None ] - relr = [c.get("rel_l2_vs_reference", 0.0) for c in single if c["path"] == "ws2-rocm"] + relr = [c.get("rel_l2_vs_reference", 0.0) for c in single if c["path"] == kernel_path] add( - f"- `{style.name('ws2-rocm')}` vs `{style.name('ws2-reference')}`: tile maxima are " + f"- `{style.name(kernel_path)}` vs `{style.name('ws2-reference')}`: tile maxima are " "bitwise equal; sumexp partials differ only by FP32 summation order, so final outputs " f"differ in {_range_text([float(m) for m in mism], '{:.0f}')} elements per case with " f"relative-L2 {_range_text(relr, '{:.1e}')}. Both paths are equally close to FP64." @@ -1178,19 +1201,22 @@ def _write_report(payload: dict[str, Any], output_directory: Path, style: Report f"{_field_ratio(case, ref, 'train_peak_mib')} | {_yes(case['grad_finite'])} |" ) add("") - if "ws2-rocm" in style.paths and "ws2-reference" in style.paths: - add(f"### `{style.name('ws2-rocm')}` versus `{style.name('ws2-reference')}` numerics") + kernel_paths = [p for p in WS2_KERNEL_PATHS if p in style.paths] + if kernel_paths and "ws2-reference" in style.paths: + add(f"### Numerics versus `{style.name('ws2-reference')}`") add("") - add("| Tokens | Mismatched elements (logp+LSE) | Relative L2 |") - add("|---:|---:|---:|") + add("| Tokens | Path | Mismatched elements (logp+LSE) | Relative L2 |") + add("|---:|---|---:|---:|") for tokens in sorted({c["tokens"] for c in rows}): - rocm = _lookup(rows, tokens=tokens, path="ws2-rocm") - if rocm is None: - continue - add( - f"| {tokens} | {rocm.get('mismatch_vs_reference', 'n/a')} | " - f"{rocm.get('rel_l2_vs_reference', float('nan')):.3e} |" - ) + for kernel_path in kernel_paths: + case = _lookup(rows, tokens=tokens, path=kernel_path) + if case is None: + continue + add( + f"| {tokens} | {style.name(kernel_path)} | " + f"{case.get('mismatch_vs_reference', 'n/a')} | " + f"{case.get('rel_l2_vs_reference', float('nan')):.3e} |" + ) add("") overhead = [ @@ -1256,6 +1282,20 @@ def _write_report(payload: dict[str, Any], output_directory: Path, style: Report if dist_rows: add("## Distributed vocab-parallel logprob (BF16, RCCL)") add("") + absent = [ + style.name(path) + for path in style.paths + if path not in {d["path"] for d in payload["distributed"]} + ] + if absent: + add( + "Only the vocab-parallel operators take part here. " + + ", ".join(f"`{name}`" for name in absent) + + " is a single-shard op that consumes the full `[tokens, V]` logits on one GPU; " + "it has no TP implementation (no vocab shard input, TP group, or partial merge), " + "so there is no comparable distributed row for it." + ) + add("") add("### Forward") add("") add( @@ -1289,19 +1329,17 @@ def _write_report(payload: dict[str, Any], output_directory: Path, style: Report f"{_field_ratio(d, ref, 'train_peak_mib')} | {_yes(d['grad_finite'])} |" ) add("") - if "ws2-rocm" in style.paths and "ws2-reference" in style.paths: - add( - f"### `{style.name('ws2-rocm')}` versus `{style.name('ws2-reference')}` " - "numerics (distributed)" - ) + kernel_paths = [p for p in WS2_KERNEL_PATHS if p in style.paths] + if kernel_paths and "ws2-reference" in style.paths: + add(f"### Numerics versus `{style.name('ws2-reference')}` (distributed)") add("") - add("| Topology | Tokens | Mismatched elements (logp+LSE) | Relative L2 |") - add("|---|---:|---:|---:|") + add("| Topology | Tokens | Path | Mismatched elements (logp+LSE) | Relative L2 |") + add("|---|---:|---|---:|---:|") for d in dist_rows: - if d["path"] != "ws2-rocm": + if d["path"] not in kernel_paths: continue add( - f"| {d['topology']} | {d['tokens']} | " + f"| {d['topology']} | {d['tokens']} | {style.name(d['path'])} | " f"{d.get('mismatch_vs_reference', 'n/a')} | " f"{d.get('rel_l2_vs_reference', float('nan')):.3e} |" ) @@ -1356,6 +1394,15 @@ def _write_report(payload: dict[str, Any], output_directory: Path, style: Report (output_directory / "report.md").write_text("\n".join(lines) + "\n", encoding="utf-8") +_LINE_STYLES = ( + {"marker": "o", "linestyle": "-", "linewidth": 2.2, "markersize": 7}, + {"marker": "s", "linestyle": "--", "linewidth": 1.8, "markersize": 6}, + {"marker": "^", "linestyle": ":", "linewidth": 2.4, "markersize": 6}, + {"marker": "D", "linestyle": "-.", "linewidth": 1.6, "markersize": 5}, + {"marker": "x", "linestyle": "-", "linewidth": 1.2, "markersize": 7}, +) + + def _write_figures(payload: dict[str, Any], output_directory: Path, style: ReportStyle) -> None: import matplotlib @@ -1390,12 +1437,21 @@ def _write_figures(payload: dict[str, Any], output_directory: Path, style: Repor ys.append(case[key]["median_ms"]) else: ys.append(case[key]) - axis.plot(tokens, ys, marker="o", label=style.name(path)) + # Distinct styles and z-order keep coincident series visible (for example + # triton and strict-hip share the same forward+backward peak memory). + line_style = _LINE_STYLES[style.paths.index(path) % len(_LINE_STYLES)] + axis.plot( + tokens, + ys, + label=style.name(path), + zorder=3 + style.paths.index(path), + **line_style, + ) axis.set_xscale("log", base=2) axis.set_yscale("log") axis.set_xlabel("tokens") axis.set_ylabel(ylabel) - axis.set_title(f"Single MI300X, BF16, V={REAL_VOCAB}: {direction} {title}") + axis.set_title(f"Single MI300X, BF16: {direction} {title}", fontsize=11) axis.grid(True, which="both", alpha=0.3) if style.paths: axis.legend(fontsize=8) @@ -1406,15 +1462,17 @@ def _write_figures(payload: dict[str, Any], output_directory: Path, style: Repor distributed = [d for d in payload["distributed"] if d["path"] in style.paths] if not distributed: return + # Only paths with distributed measurements get a bar (single-shard ops have none). + dist_paths = tuple(p for p in style.paths if any(d["path"] == p for d in distributed)) labels = [] series: dict[tuple[str, str], list[float]] = { - (path, key): [] for path in style.paths for key in ("forward", "train_fwd_bwd") + (path, key): [] for path in dist_paths for key in ("forward", "train_fwd_bwd") } for d in distributed: if d["path"] != style.baseline: continue labels.append(f"{d['topology']}\nM={d['tokens']}") - for path in style.paths: + for path in dist_paths: other = _lookup(distributed, path=path, topology=d["topology"], tokens=d["tokens"]) for key in ("forward", "train_fwd_bwd"): series[(path, key)].append( @@ -1422,12 +1480,12 @@ def _write_figures(payload: dict[str, Any], output_directory: Path, style: Repor ) figure, axes = plt.subplots(1, 2, figsize=(max(12, 1.1 * len(labels)), 4.8)) xs = list(range(len(labels))) - width = 0.8 / max(len(style.paths), 1) + width = 0.8 / max(len(dist_paths), 1) for axis, key, direction in zip( axes, ("forward", "train_fwd_bwd"), ("Forward", "Forward+backward") ): - for index, path in enumerate(style.paths): - offset = (index - (len(style.paths) - 1) / 2) * width + for index, path in enumerate(dist_paths): + offset = (index - (len(dist_paths) - 1) / 2) * width axis.bar([x + offset for x in xs], series[(path, key)], width, label=style.name(path)) axis.set_xticks(xs) axis.set_xticklabels(labels, fontsize=8) @@ -1469,7 +1527,7 @@ def _validate_environment(require_distributed: bool) -> None: raise RuntimeError("PyTorch RCCL/ProcessGroupNCCL support is unavailable") -ALL_PATHS = ("native", "ws1-pytorch", "ws1-triton", "ws2-reference", "ws2-rocm") +ALL_PATHS = ("native", "ws1-pytorch", "ws1-triton", "ws2-reference", "ws2-triton", "ws2-rocm") def parse_args() -> argparse.Namespace: diff --git a/benchmarks/results/pr328_rocm_mi300x/distributed_logp_latency.png b/benchmarks/results/pr328_rocm_mi300x/distributed_logp_latency.png index 27d9a71de7122c9aa92f425078089d9ba8621f33..d56ef80b54e6a8e953ba4549958a730a9e6977a4 100644 GIT binary patch literal 62072 zcmeFZ2T)bn)-`%M7#bB@#e@U{Du{xT1SDu15Rf22L{cLHl5Q%i{x373S?6da@bB;O2nCqdewAdyZdKwCa zvPt~nIXMdDw~G|YFMqCIi+83ne*YDJ2wI-MYAJ7`V`-ylu1%5Bv@|s|u{6}XzSmmY z+(OU9n49CoaSnl_dv97=npy~QavJ^n2@Vr;UCu)nYL#)7-%KwmTTm!_lE{BQsndHg zP$=Ww;^)pN*ai)^**MDgEUb*%WUY(T_E)4?&w7@xME&Re``2e`cYSlFTle|X=6X-o z{j{c=BX0eopzyo)*`_zH>r0NV|Hu)UT4m?#GUcyz_9@RXJ^7l3$?^WXPb~UGl%%D- zpAM@hPl>z#P^?AdNy#d~P!Bbu_C6 zyG_EQrP7pdi@)Z`7#haUwR=2%JZ;HOp{Oktsdn?wG}Pz0(Af-!7OoT6l({D~^wumJ zt-#}N-)y<>?>l?nmgWB6pA#xe7QjrH^t@lC6sb+&Dy$|lYv*vmOH(2G4<<5pMISeQNF)({?r@E}1oOk(Bv~+OI zEBWZDg|U2gKE5lD>3QXz9+GO!bEZ%}?-F<%w^Lvf<+Ry3>0c&<{t3AZ4U&mT0V!;&6 zFNbTW&T8xEG%t$s^Ix?ZtgH55JT);~>2*TuLw`a>yEn_dd-q;x_Ss7Hp^V7fx$W#%ESsaudrHpJZK6=l-wnEd4C79r+qgRZl*$*U zX-<2Y@g1o=zOg)1R6|B+{I8kOwzvag9{rzGOzNNW8q|fIx%+a%R{V}b`Eyat7?xR}%9r$V=9PaMtmBi;HZt%t7$3p-r7ef##BYn8tJ?YY^w#3RRCtdXsl z0?KFFZk~^NVf~wSbDgV=l9I>j` za)7h*$0z%GwNw+Xt}M-VOn<(+Io5ilnTD43$~~IxmbFqR2e9(Y<}(``uUe?tN!=cc zF>-XwIlALfdB2xlZB$bmpGQdLKy5UyaVvkCNr$?Zmlw_Y^|JWox;!Tbsb|N}?Bq~2 z)jP!?Xn7HnU&q^^8vgn7pNjEUFORfjG#n0feCh7K(SP3b(18PIeUB)5<6$Eue9fhJ z4J)xS%lbO8bu~!nV(fsz?<*QepCFagt6JGpU(41puk#{{v#3^eu zvy1k;m>4?`j{;^rRy{Y%etIy{ro&0;uuMqmLZj%)qP$7oT$FM42p_ixiEW9h$|pM< zQ$!ZOp2adR?ztLIljYLunmgO<8$bJ{@X`Eu5i?nJ{rC46%TmpIy2r+nzJ4|3y;*g7 zW!7aS8c)#IB!jsz6gkDk&AlCq&$un?6$019e=rNl>bs<;#-Wu;XaelYM6uqNN$J|JYbq&ZQYNe8M|IeA8*%Y;0`y zeZe+$R14kn7cLyZio^C(z*e4`A4+0BaiTiXnQWNR?6JWQdrTApxU>~w71-2l+N|sc zUqxGf`>H8!=VIKlu&A1%^LR9OR-FtJE9*r})?jV4XK!Wb7*EI@$E2$X8u{v*MZSd% z*2hJ^etrDj{rhA!kZ9?5Or^JG=Hmcz)clysO5BCV^sg)jYK~yVK0j%aINnt>gaMly zO=4Ab7)k3jQkuQ}<#Vl!X#Lz|-$8Mc+-bksb6@HQ5%K3cR+ghbC8)&xagT;9m7Wq` zE3(dVXWL6B7mC^!zPd6yR~corMa9IhuKD%XXQ72jdGoPN*4U-i3kw6$p@trs9t;B4 zGujYiP12P!O*IpHM!y;7de1K9*iF`D+nA71!t+&ES7(joEuAOJ}I``qvF7s|4H(C<34jzl%!pux|(JS>_dzwv~6v;fL8>u)=Rg2j%xoF?G zakKO9d-THgsm*By(Qez>kL=xhT2oUqRwX0x{>I%eec6?I*+YaKX8LiDS4#bkDZcn) z&E&H%# z-hK9$-FH!BAFu?t%#Bag$^?354@3n9&9ot{WtaAq2kQD9Q7k#M*1@H-vy)cKdM3kh z&V2GjsXyn*G3Ha&@h6O$c@FUN^EYTkNeAnT=zRVBu6Jo2(%|-$9w+Hgk+$_4H(KtJ zK_qm4w3D;BZHIicblRS?&dp3hwpBewT4QsUf(7f>Z`fclpDE#cWVb;>q7{9Ir|9y{ zzwX}`91d5@w!ZMkAAf`_h^ZrEo=EZ3a9QH5oRJZp_Gwx!VsXJ zzUp4Y+XB842Llb;s)M84Sl_)mf;kQ;0TGSy^0Mbf4{T>FBY_epAjMU^fv_ zjaVq8MK9u*>E-8VeSWCqQQT6zN=8MDT%=pAZK*H2?orjus)2d3=6G&az13vaF7Y|x z$D#JTuWzO&G!$Vqs}uR)b;mj;+qazyxqCa<2L=W{{IXf-NuPl&HZDCjFl~ldMP(&s zY?or(6|2Su1VSwae$%%DGgkgVEuu+dZ^FeqpBo{x54~2a==&lqkk36iDXfs9BX!pM za<08q#(S^HxuzD~N%A;Ktqn4wOE=D)JICK98vgR7J1)#We4x+yYhS1feMX0KcAN9Y zg5wC6+RH9~|Mi>oLg}0nXYn(H$u{ey`Dt#J-OJ0%AMS0~Y3R58lT94^*aE_Eppp4p zsHdkVQol*}(AD25Uz6@Iu|KKZ8{N5{Y>`n_PJzcRUiiDF@8smupbPJw)V}t_cn85DB)OrytV5I5+7M64S%5GGZdNAIrN%qJ%=AN;{DB*a(< zNbr7Al1a!?VPTHxfUBKufAaE2 zSQ*52s4ISYc)IHBW^r$Tz<9D5j9XT*;NcM4KfLL-ZhiUFFKc6@0{IAT#`a8Dvth^0 z(N?OU*_XeJvjz?swPs$P9%`tp9|O{|#1cFrEUe~ThiKMj)u0x8HTflm2q}ptK|S{x z?kxLD-zn42r=>!Kl)a~V{54{AD??~obhNbt4lI=T9g71JD%7#l(TTF38P@Yqc=X^w z)oaayI%F{t7N-a6pJQbq2l3cV+)&H0y^LDmRht#Lcw4s3NiqKVL^TzgcDv}r>`}or zYu64oCTm^Gcg}U5>)Dwd{u2rcR=};^^1!&WuC%dANg)`W@X5Z)p{f^W@c@1^!%g8! z3$t;!tLAj0K!N$DEixmaqgTf;K?Dg|4SkZrU?>9q^dLuLC+Q>p&d;=%_=bceHfXr0 zAz|{&YUz&VuQ-#t!VG*8o^1{9@4p!=>Y{$idiZ$D4Q*{UUS37TScS_O#%-3UIpX9Z zB__tlt1&qV2toQ7;F6CYtHM1+Vvi`sJ_{AU*NKFt-js?1ppe zR;&W#m**c1dsO7xj=#*Z8j8XoN_xjCyn3lwaR1eryKBkHA?3(OMh8KpvPg|;9%Ip= z&NV>-7IgIVv7H63aoVMRC1quGCBE#uPV<(ygZ`9C5&40dNM5rq=c1jc^SNA@^70@7 zd2c#8x`=mo*TvzcCZ?zBqoe|PaMK459vo;f$yY>S0*GM3o&C0F>$Yw6NgASz!uATd zC#8-&Co=pPDvjq{+HSPWBJoZOGq122$uvp$bBs*Puo=onT{>aX0Witr{@zN=gP|H< zbx=IvTADYDQG1T^)59`X08%}eMH8{vLE6n_+VdYNuHJ>@98#c)yA@h}P=*q6IpIy@O z6pjMY75D!6=bw3Ozy67ZAT1&y()?QUZq%iKm%1T#AB_AjQ7D=BS8JfAk9tpPO=som}O4zM> ziCXojYYebF$j$MYv>Q6Fy_#?}`EQ+Elz&xNP}{d}*RJL0TW$pm&2A~%RwTSIGg6Do z7*;%Kh`D?Kf%hVYx&b&l960;J9zMg@0u^E>`rdsxN;!CohTm=9fdd6P=i1VZS{P=G zOy~+wTIg}ObUi-Fka(D#z4R*8JhHY?ZTCU16hU?GExUK?Edqc`Gd~Tj_VacJ9@Rp2 zpf-;*r4;F;R^u5$;~t;7pmL2WcR8f?!{a@hL`6l;o)ic=oW+o zub(*9>B?M@=D7H6ymtcC#BMQ9rt*3n#Fh&Hhvn9jz2&9#y%>-SK(Xa~NXe;Q$ZIE~ zFgsfB?yRAXc2F&%YrQT91O)Jng+6<>D|_T0ymHEA+0nJ7>iEV3A|maroB#g%Z+9%& z79)R)y4x&W4-bc!3hMJ=yq<}=kS+dWSAxq$^&*qEX#lV}Yj!YvlM@a@|!Or&YnCe=Ch00E1rrgQ{F z>y{5iMS@*mR%{4r{%)AbGrihj7sWzv6Xo8B~^uY{7PAj3oD`_WPqv%*{6SB z1;{ZZfA6-OGto7tpA(ZV*defK?4w?$#acEt2fz2&FKkNL+YU=NI)0D|;;%M#k@P!i zCb*{^g>ZSmV=#9Nc^3#NKutt=$0lQ>U1&|!dy4vE?5mxcvtf>Y{Sk{#sbOJzvnv~p zZ`48p)nV=`@~li-tQKSDW!a7V#0R*%QT&3Tp<$x+r``37OG}TtySr6Og{@k(g7-;! zpRnw&QaypBwHtu&&A`*$5pE}GA}DmR>;e}n1Ng7y*d`a`jW9-tPF0=lok-EG+`%&C zxU@cV@v})qkbs@Yq5R_a2qul18d!HHTiR^99#j1sxB_kl-w{4~w2a@p`$26wXYQ1D z(=1Z3pG95;#(QhkDw=)V_9*-D)Wievbw;aG6a&-*6T$!^@*)z=y zsfxRUzyW=B;y)=ljTi3J$&b5|P-4+G)9w4LCC4s>>R1#tv< zPpGr$${TAeDtW^{=GobI1=-j8goh9IzMd(Mnjdjs7%v~sfd37p+^l}y_ zO8LZd!5(O>EH7~gi`pF8HaVZYC?NBkr&!e*2u)B`?81fnX2_+1Mk9#o4;(h9@8%IoQ#Yqy`SEuTuiDp672P~<)J6n0+3dV1yfB$uO}`DO4;J=a2fd~WQBS@-L&{KC=}@2iu?G2CU6Ecabq_po1yH*FNV zcI~+Ue*~f<)lX%^hY;kGl>V`a%82;*qm$IeQy!MS*`5yR#%=oP38lMKo0urrEcz@X)P9gHT0RyFIK`Uz zu(n3~yZNOWETsCF%LSUEoQ#l^wx8EmkRt4>b;NiMpXs1W}M?C+td zKx3|jtQ<7}zlh<1cHojBORUJg)#a?O9}B?ASv-?_lQ0gJfK1!>UZ=eE-5Wj5Nmet9 zIeo~rAGN60Hi1pE-&eLtrQWysdBI#q71T84l~rIIZ=wKOb4>`UpcyZpjQ2(YHq++YCS+L-yaZJu?#*Z1+CTG` z&Y5`Sln;)KPEJnKh$I3=p_6a*W=GrjO5@jg0fgEy?X?2PrBwG`3BRz3AXChD4N$bz z;y1HEgBy}Hn9Fj5%^hGtsq>A1tajNDTkH~3a8KZjLnDTN;U5~Zof%f1oHlBF{Q}gb z;_JiHE60@IoI@#62UKMR`Vi?|;F1mlP=8OUzfGfm^o)#6!fbc(Q&RR3ZWi~`nM2ZydPGD-oe=Ao;pN3inc!14(g?^$zGiK>jxX_!c<^9-;8Oh-gP%{J=mNz2 z`f;;@K@gT4K%PZ&I>0SJgAa#l=93|@YE&d;Ep|?c3Xr2v*JeK*KJOF;twLV#~a*cL2$Cx-r3fy@&J;Xry}uqN^mVSB}!#*z_1PgPE5 z7kuBI&kbmj6 z&;r-Z4JaU=#;M)B8ABu&Jd-3IpNJQvT1C~iL*G%rX`=WrskRGxK<9vv$u9uomZVeu zTwvr>pGc6>{sQ017H}j-Kw1|U7hlVBq#}d~SOoBjH)6FBh9CL!TIx+NKTZwfkoc04 zk{VD1EZT>=?%uu2m{=OzkzkiQ1|2LhHI+wP4p5_KXlUqlg3nTbov?KkkA zWY^G`UlGxNIk6Ogw-OSg}Y#-bx1x z>eog+7PRaSQ!&Y{AGmFLrib_C>%H*@zN#8oKtjnVlKma(TJMOb$oxfSrzx*ePpOL+ zU(e-F3m^dW3cUH-ccdp4Y6IB}q=JS<5RQ_vM#rw=^0S!>f+#CQ!EihG_ZU8%Oh(e< zHPH~CICSE~Wl$WtiPiBHSD!|Ds1`5VVe1$*y(#SMB5H*i$_rzSBvAh;fMlE5pr2Oe zMdW1M>z>zri`5vX8|w6_XUn6+`0Ty$W3>PpZO#iLbN0hMW3?}@JDHpVsSRMJr-H0Q z_>`6mlLknGzJ7i>hhA8WZcALOiUT!PTKxXbaHxm(L%R|A(fJ^%DgwmhrCqjz+0kI4 z2U{|d+O(nIMf8>hgfA~G41vKQZ=ymB?o0kFeA%s}XJO**W|98aMi62Mh;f=MKj|F- zmUM?6S2Dr}hiAJGYQ%t;7?8jrBR~K#z%@|E1lvRpHi(#59Qv&xURld|Z4kfN-$G3{ zgafLumbn9NrfC(?PNq~;i@W~)&A5Ds+T)ntGdsDKB8yeINQz{_F{&kPCx2e)<}p7I zeK~cNulFi%v7jIQMyrASLFU|F_d#k#xy23&HT?4|p*^5-vWOE>=kI$PCXfD_Qlea! zYS_e;a=Ii-loa%M60Vh8)6P5CYdX#@JwXG*6T77ngv?r>hB~FGQqA2SJa|3F_~6S! zV5dSkQaSlMK*XA8D}p-wibE}{KiH=IM6_vZwoMWdGJzLK?-)qQ9OJs^^)M7mIr?!*yT-y$%RRIwp$M^a3Sa5$ehXQPS%XVAL z_v|srsUG!=z{{}^^afkDXSv;R{}d_bO}mbtMtR}ovs92~GuEzO6Y&tFUp#0G3&CpO zC8CjfcUwL;94MKo1m`0VEawbz6w!hKpj+Sb&J2`Q*_FSp7)B#dz&{a{rP8D z)2HV*N)1#lCOWNk1f1bpkV8RFxN{;8VWnNN4i>VL+blfwq;!a5y-l9XBe&_?&OwKz ziCIOcVJyWu=RC4SRa)n$!3`okiIoARY6%^}ax*<^*3vS>Krtl0&g`<79OANUC*#_q zaPHh+b}iSFV=#EqEhkOdTQ40N?a0qRel~mmT$^a*uoDlotbt`Vs&kpP)Yd4lG{~6x z9eKF{w#kR~B#cB1|61Fg>!1jgEb#ry@Eg}P9W#2TTg~%2+~ynCQ=XhOr%zA()FnZJ zT|8pj{%AAp)jz!U2t59B7ew{bT3bA6u=>2oME7d6J)#g3hO(Wfayf`vQ|X`UFbn!o zJqy@~g~&jI2r`3pufkEFKv4Q@*u`Z< zXko*qO_8SukDP`d;Q9cFjg#z?XV0FkheGZ+tC@V2=!ct_MN|n90|Fz>dgL1T{!7@3 zykeDv&ID;>N!{AKjX`QlfP4jWp_2LW?h1pzK6ip{OrP_ZM<%_0|K8E$5&QNnTV7xY zhrpUOU@xRP%o?ZAyI=|R(#`*}jy84STTr;J>O=OvcjgsE$f3=~fs2HfnbV2}!MR#g zo?Ajas5+X#IxAz7c=FJp53L=dOH=X$A==G!a*&b{uwn?*0%4J@Ta;z~qKm=~8~_$^ zb91YCan_Z_$!YZW9Xp~d`YNc4&HVNnJ&P{}M=SkaI%;GNS@fER*qdZadNK(^&Sw|L zmzpKZ9QqWeRGGs_Pk%&Iv`@4?US3X8$yzS#b&t)8&vKjCB)!x~Y*2V3(qtltlEB}d zL;DB_T~i)-QVD7?6tYwZUiGi!i!KSSUAwk9H~x&KQ%3K>zJ-Js&&L#s-2svjzegaA z?*zabuiB4~gDzC>Uj?0Z?CwDqAh`x{WT+nzfSJWjS2Y_e2IO+vQ4jmhdpoD{SC%R} z-EBPbIu6bW{u#T}9iV)A~$w_&D-ZazBKN3)Gyn6LY;^M_B z;CQy;^wUX*t2(!Re4L64b94kJ>&!f-(@H;nl(o0F4>>Q_-H}Ku5<*L%49}q!EN?Mt zPA#6iCCG`otp+P*3YC*VWr#3@@e~2K&awQJPvF_?&M)mu2gvM+EALr%R3I*TlAf_A zzCc%+(uOtl!UTiFpGj_F2+dID1Pdy>X$1rQ*IO|Dpv?&4fflpr2n9Dm`&; zpqS1n*ZbMq49Pks9I&>G%LuF2)m=mBMe`!U&|Cr-I7j2h;>eB6b-_Af!Ez0Orpzne zdql+S^V{1(4MP%p;@Fp3D>mCWRj3-HMCIC8prXf6_v9 z)B5$tbp=bCMzIE}QFUo^*D6=sIn6G^G~2DKA#BmRF-1)6AnTK#Wrqp=m>*85EL+*k zEYdnMbn{bu03lD7T>!tR%~v-|Ow8|7q=Q{2xX*JjXx$bBi&SMl3%8aWTAfcx8e^7J zD+xumF{9cSXV+Pr>g}WdG?S`dFCjN`Q+DX%Ix%c}`}0Ftt;aH2uP?JMVUcO?x=+i% z9|oC4+?yysli|FF4O;Q}(4209fsl3-5vmW0pRyU{d3t%y?92P#CXr~;7}F&sB~zM0 zC(I)kbkC?2+k>dlT9&m*KgOa5Fe%ygYhcHP&U zJlh@uZ20`QqM$K^&2_$t>I;joR*{m)UrROJFrc=b9Rn% ze8s?4msQdY&6<@ob28d`$^y81N8&msu7lSn=_F!!EDJn@pi#cvq|T%@@cjnfban*V zGN&#CccNX?&Lob8&UtZ*XTm%{>{Jdj!(J>_12u(F3B2OlcJ7SE1)e|LBzB;YW+ttjbB3BE~!*L&*~9MOE=Xs_onU6endaLwYu{r^tMN_|Hj2 zAwl76kjB$U8GYH`BkHC98?g;48lD=?bA@_bU*PmOE_Q|#0MIE?m5mvv1AKiWcdFa& z>!>`?mzEr?*0Kb(t)wBQg}qI5!=S{cp~3Wv5)$TF`P=`B8$$Jw&ucZPkdfFDf=wTg zfy?$ds;RWfLkclz58*d^S-TCuYW#j%5lTMpp@+8wEPDAFH7DM!W9}B1KNgLuR8VTT z@WY4a^6NURTa0>Hv+q3c@VEl7#G)-DK5Uo^Mu;naoqk=+3(yv)MiwBsS!WIt!dX}I z;>8DZb{eYPi)RdT-7lKiz?tzZgXBWj5$PcQ7x=D=B_hQQ02P0Dj!1MnjtK;I?&s!~ z2XmadkjbtswKgg$YA~%q{pk%CH#>uD#Jv^<8GeaFJ3iSMsQ>)SFIJ_;GA@?1Gn-c^ zm{Z}T$u?ryM|mq?PCr9L_zKee0CWR&mz^zp{W4I8&744Y57dNun8fz6IrNGsxkbeL z+xZx72<7Nlfv8meB@y-Uvo#f1`{@A!pGuytk{?97b}byaubJu8#(Hiog>JuiZzwlp z^bZj7^&zV_7;0o$>S;j|wIhaopeLw)h@E3(;|{)OHlP&q`gsb-JpSKK1r? zXA-v0Ock{nZmf>16f8LI4u1=ztGXvQ$W!4SD(ic{jdxMoJz=@vsLE2Ca~uTLZt8^u z&jlHm@k1z-sd88A# z$2jvHT(YW(d@!vY$09XS-|P_DLi|8_@(I^?wIB*po8csDXN5UzU-8%R@#O5yeKsl; z&v{D?HK^v=T41oK&2R*Mj8V%@Z>nfFxVbC6qs`kL1v#ryAjV@C_$1yIA;b{754Mr! z&L)EM&TpP`>w0v80Z(5+6dFmlnC3s2Ps=29HEiq!o_&%Ryy^#(>QCdOKroc2=R?|K zobhiE31lYzS5u9?%FvTXuEgIIajuCFr#9cXabwqJ(Z$zIQOI7_3nfF}ioHNB1$w58 z?m>usv&VIszo?mmpTDuw#tC>JYa&l0P2aaE50=Xh@UK~d~ZUxaXIEqZWE^zs_~^2;w6{uos!%aLY9@TRhu9Q)bP1Y$RU z^4>c&2x^uqE@eXt;Kob%ISAoJ9EYGMIW;sih?9jF2%@i%QnA46H?0ruC!%M@@R$3W z_C$fR0{cvC6ZdesL?B-C{sj0=HmW4MQ#(g97dt`K9N z{pZ{3e4acBgKsYuUrg#f6x3`kE-u79O#IYHj8(3o&Im9Sn9$_auLKKj^g;9BwKy;|wkT{RP-cEe+ zs5y9GFv%!n{QdVZS0?+5CizS0HMO<%K$qlUfIfZl^Ye)soX6ajYb=)c?aKzRcVf_J zvO$GL!k92vacWEfutNbV5b+OTHPvH}k^yJKYu67s-d!ta52 zs>0$Uej5u*eM~%YQfE88FDi<|*z~S!I;r*r0qPiE$0=_gn3;&TY5Ys!bI2IfL^7=z z?e=jhpN2l7(!V*m^8Wp~$r)}Stbemb-P^9es+Q|KiG6>EqRC+s}r`eX3d%bm@89Tq~8Sp)=+&(*SpUH zrrzCk#6*n|9KxVAkg>Q~wO0}C7Nj2r3_}7IEx6q)f<#*JKk%2U9kTCf@vzbhhTYRU zw`|cAnrYO6zxGb4Q%l^>qZky)A$E|ZiCgrpK$JLZ@O*J4SroMP$IMqBVAoumw=!n zyG~I~8_z9y9R5coW!M+p88YqtoxHfOSg{lwbl+fN57eNdN))5Z?X%|tP(f^zBc@z% zA>EcPH{nb-gVEM?3F4N8^lT(H7d=7Jvue6#(UFn6SLP~Rf}C|!RyagwN)6Vs#X)ij z7$K%dS8j1Ai0^cqkd%^JI8b3cc<|tO3);Z;fZd{P7%T2#!ZV>`r?Je>eog~UT6b(M z>XiMv$sqrSzntDTj@}5YEn;~lQV}BEC*;vMf+YxIyCB=$019y$I(a`hE+vwQcJC&s z8mpF-i%Slnq#ky7u;l8l4Vj78(qfTwUXsv-;fAp(vdqgb#1v@P0Gcp8CKV)BBs@62 z$5gK1rmKmqjufV&7x%Nnj*7Bj5X{+SzzC9-U|T&RC`in**ztip*Wa)AzmEHQ2i<_DrdAo~+5DY60{DHgKXAYAwD zsNtM2G*l)B$aea`D#XmCm~CxD23JUk3}GF2rBFC=RyC|tc>Yt58z9)hFFXjx^A*&0 zgkHvl5vrs}ULx5YnXZ^YU|1UInCO^9eoZrOyN1Sz-XR-=OMUP_^PoW{TX(jSmXhZ% znUQ%TPG{gQqBQ*DPax7Bk*eCA7mP+a@->ojCM3KHdoLn>kU7sqfF>?b1iL?-^U?Hi z4bxl;JX0SowmEv~s^}Q-fe`?#1f;AhsNFFV(EvmS;9-d{3|G_vHV2+7us}|_6Bk1@ z%tKc2FtHywQnHwnjeI$c#~g%<)?&zD7@mnrk%de6@>*m=t8(53R*e0-ckg1s;t-$l zF|{m(4yW1R(giCd{#3+?Toh3HP}yPjBMmyFB`7I(<~g$H3pi5+936p6>{_b(va-L; z&Al>gdP5pT2<@|qZ=)_1Tby$=tAjg_KuppUb1mKQ6?*WB!O}vll>_Tb=n#CmejuIG zV}5Et%h{Ru{CX#e2u;id=q?$6_YHfjoJM;|Kj9oXcutH5GMf9Qzm>#;{fB$T{6e2X5 z{2>F%2(d+F%`_WqXJ9ZiZuxE7p_|zKkT8z;SQRWCz26xnu_`2Rm6|6hejX z-`n!-_1OjP1MwLoQ8clU8{nI%&F?>a(8nElN8nw3n8waCXpc_c>JBuRY8H z!VFtTBeN|C9a|cK$9MKc|3tYptGSj9Z|>cUS>sA#9vimrf1P%tc&oUaf`ZTTWax@5 zKUxX#k^neba~8d&l(Dn7>Bt+Y*g|Xq0!^J02)zAXVob0wEsdcyMONX3TK$_hZ-^`+ zUW2^rs3^T*Gr7oVvkTJUrlgzp?Ws9CLGsqyrzD&?f&3b*4tF)q>|9HBIFh3vIw-(a zz63W_YOgp*^mkN-Odj6`U!#vTSis@}kV^w91R+KU7I>+HN;TIZ> z#YbUH1g8L~76G#;>L+_a2HPhHDu0|kyM4!wL!zRzFMbUE6Mp|YF571<*me{g-zc;0 zr`eRNN4;6p)v>L5K{^RKetV5YD#S?R2Jg(mTk8R;?9wdrF z^C{eM;chBT<@E(}@08Yu7Hl71?d_ApFuqe}#N|s_4jv3xH$_4$T!{Q(oShpBWP!E` z$qOSR+)=N9A46z2kY6V+&(Eqf97?T~k>6~|{<-z#kbrVdXH_6vmQPDrbpfNeT0=uZDL7k{J1+mvqT7eBeTxiFnP_K5Kz)6L z2lt_ei~!bwK4e%-uP^2{4QVn^8?HT|Sq0?5g6YkmWRsKeOi0{pK;^)Fq-> zLQSOzvIZQ-)xeV`d}CuMRlzL2f>3Y`MP7#83Mf3}&BLUn$j#GJ3C5iu|bFQl0s>*(mcrS83d|Ng0o zQqCb%iO)f-ZZG`z2o?#Biby&WXS6i(0>9j(wmhJ^?;h_nyXVz4?0Pl07Y1b ziCrsY5r71Dm-W(?G!7FONt}{N>MI^!pJBq`^|yj+;Q6d_m*k~OJV`~twEzt)Yz;Rz z{yW1;W*m{W;j1asj+mDm@-iNXO8RJEoW(u86crva*b002%;6 ziIhD(+NOe*DT~A+!jQvss{k^JoDV?EpD2*wMm`%dw1@QI18AYWw|y~RnAoxjoJU)h z0ZKel-_fD;B7SE)Q!|>5z@CyGb@e9TrGH+gcP!XK-mL-y(?CMh;+YgdZIV7vC)lD$ zsS0}lv8OgUl|_g@wIoy_Ag?|CaT=4X2j+w$Aql60kiLIH@xInP>pB#3CE+=#f?*IM zo;YF-JtPwn555I`A`+pF6A!U16+XjC@z3is4W{CL;qi?HwP)TgAxuXM>osD;mdPm* zFwv4D9w3DPGrDadJD(rOAEJ(tJek(p&eHofHVPX|v=rR38|&I3VwM!d{X@h;j#hFc*|V(xmp!H0vsR zui%WLB9mvoDP%Qc=PRqfO<^?##XmYVhCp_*1zne|cfGzN0Nc$vPrS*J><0mGe1#eD z4X14oDFcFKJ=XX%4v8Rrti;rO`%@=sEND1_s-t~HlHU=AY>iX^*LiOUrKiv>R#{N} z{Tpr9#hSB_+8Jjm0C5KsL+k1>B`|1*F*|=d16c;=5=GY`co0SzhTb`KF;Y_P+}YRu z_WQLz^Fri^TWw~&;J&M~^XB(!=d4>J=Z)QbV|E$dDD*a*LGh#j`Szq8RUPB zEn@4xf7(PhSojDC2}1X}hJU~CS0=iP)X&)UJSNN#zwUaiepHqVz(w-K)E?11r595h zx*HI~NXx8*H%nyZqVlG7%cB>ydM-#ztQA(>%I{5F~eAG69d02 zq&G9k(AsF3h6h^?OkuqlV0w%PQX;}n4{{;3qj{y=$^{Gxam%wGKmPjkQV6{< z!ZCy;0H$yjxjlMR3x+KTOV9xQYe#_Eq1TO==U)o+Sp2iulJb#RDeB&_QP-EF&nrIR z9-=>{F^Tp#8e)d$#_IV-jLm{~?4U^e?A-7+P+8X#__YQc-q=5be)@7O7F+aL)-5!; zF;$93fg=gsk40(;0|MmQ%NSO9h%#l8BRxe|KAuRQ&%N*yYu**FH8Be6K8>pdpm$736=qu*AI7EH7eqIC|xL&#qrO;S^hq1 zM~!o>(^=Z!5$=s&4_Wy2K<)f4afwTp>NarZgkgKK9zA*qg|-4Za)v7|`f__9M!f>s zCAb?1t3J&@3D7(7D<4AK+wBsPlC`i-vxyVe0kD(Mp5IsZnyE4XCXrahp9AS2_RsCa zu{k)Y2t^2#y6Hupi0U2CxRjVCaZq-V_BqPw)UizA#@C zZVz*+KzfqE#1qmb3nyEFjb$DHuHs_u>+^TIm`&(JFp`MlX^5pSpym?Ume4aq55usi zjq|Z3*8ctTY4NvX^tN&M#(t~aTBD*lu7^a7I-jOh~YPcsc}s4HhBBuI!B z$DVKfgraLH(#MC6)Ji}Oyl;PidBoXM zDW~Yam;biwfV9*fay~&$@^IRS=!7?kI{Or zk};IG^`5|)N?$faprO=+klX|U!aw?_eBBgqvH|iYIpF~u3h?A~_fwf>A(2CeE})3S z|InGkS4v94K;tZNZk@6!Xmn)@wY9Zr$QI;;F{t?dEaVt8tIPucJ#@OGIdqq7Z8KR0 zV0W_iUqQs?8^O&bUP0GHH3t8nIHK8CQ=ZjTA9mb~qi}Lji1D=9Yo`Tka)Ow5#I0Ma zMs29N3UH@1MF9Y##Iw$eo!?kcg?f*CwYZ$A>1*s_-)GNarlu^#y>U)Xh|`R+FuYvG zju&w8(axbEc5!1y-p}!sBrM?&FVdoe9cI4$2`98*Z*pErMj_U~=idN=DlKNrR_{r6 z(F{vZ_KWTlc*EUbvN$C?@$rCu^~-xW(IyG1Fgu1Bd5Y*X#n|7Sqb?evA9tVIfE3KFayYj6%`@% zpb22~4TK0iaO+CH6Us~%WSPXtR@1vcRgnv$*)k*_fdT|Ulm7E;6I%6wzxs*U5G^ca zy9G%DB{-MI%%bX8{WUOlCPBW61+*s7lN^di?hx5i2^NXAX}W*!ipJf7P04ku#_9p6 zeXv2lAg6zj^VHx4J5X?(ctc2YYD_+Sem4;!?NQ?6q&5TfPv77lRl>?42i)NH27q81 zplF9dbX0+kLl_PS9c)%(?QM@Mzth(kh|eV;SHAsAuHbw=mhp){ykCYcMas@UX6}fS zonJ$d`JY^itsn#4f#E%r&jO2y)OoMCuSe5G&A)JO$K~Go?$@9dQ*Ap5wy{&5#o*ru zTOISwVXGDX>d)l=ZZ&*o&HrD2V`9>B^*i7E4@mvvHcWZ{xzAgFoEtyz&%3re{x_?y zTW7K9u`KDiZak12(bCc)hmr2tqY`8G?_kia3zDP?ey^==L37r7EW~(6K7aFaTkXCd zH*sq#mIABvYK3>Jr4=TF)iW$%(35ffULE0+vcse*|K~G*xw;ZtXszatOHk+}|3g3A zio+SKIplmOUx=~5x@1Sf1gu;8q1q}XJUkrk=>+&~gke*Rh1d+f1xee9Y@Yj?be-J) z-n^+O_#urlvO;eOL6o=%W>$+;~WeCt^1v=em-~ zf+nip{n0Cg_FwMoZ8*QbobBq`X1&$1Ze?`fceiuE&KnC%Gh6M;4dQ^MZUWQMwz#^w z4uK%3TF318Icoing?ifTJ=MoBNRo|-^~oQ}JHQU#t1(&prhc!6^WQ&aKwLjj2*m(?{z5o6CU%kOV7^8h*&oo1SeEPFScexq$ zOP0*4q+q%mhEZ`jv`HlkKR89O(ry z=jU}ZPc(PInI;R_D8gm-yOxGul-OSp`Psa1N!9HiZOa()Jz*xW1_xj_A)+G)Tf{6O zMtY&Qg3a&jPmh7~$9YAY#eQ(axB5&u)(EI=`XTVRgRFh|@+CQ%6PIB-aG*da>}+u_ zehn)7+??62A6ywL6?ZJ=HbnLLvK{Z8KCuPK9SE46jZGX4cMKn0mWW)TDvLl}xd8D$VuC=d`4ka@c6*#(L@IsKjP@80i^@7ML~PTN$~ zuD##k8P>C&^KDzy3>89xKwU^?mZ&PJy|f7&sw~ zeg8+1Gr@KL55sZ)b5V@Ie`!qRo`rijuGl%NYd`VnX!1=0Nh*aTMSOnG(&l@Qa z$+ST|_MuJjxEIeZhLqj0=MSfl1PSguaJGPq5d7`)FHUd?oIo_IjS`YJ(mg5DjBv^d zU;?OY1bPku1wHLG`Y4JHw92CSjmTHvGu^g|d!UCA)&a)?8Vx-Nc&X@0wLSFsXl_*> zk7~3305q_IVWlIAkrNSEAEi8{A1;TWf^!6HSc;{Z|&%4fL4^SQ}aY!iEngr|bQMU^{VlLQ+^qV?rqa*jSR_hHRI8gzcH^ms-^LHGj2}v;TC9gqfjcy^On6vIE6qcF3JO zB&32a-b^&ruV&wGx@JY=;y(m^kT92YbV1FoG4RK2MtMCE#1Fyuc=`DWY4Tu{lXd_S zIc^BN3dIgcsX#RXqX|Cmt!lJjfa?CSAVlg*R0>cRH111pb+nJo>pnLG7=WyGB1(YWPFTU8%C8hWhph^V z5wzt2AF{a$Xeaa+MIBtYJ0f)-BPoMuLPHax;Z*zk=ie1 z6Giq=abOIhVs;6(C(}8A@*+2cDt$Sp-yzY*3mNRAm$yKHP|dGj@s=2VSZ(Z*AE$l# zI)x3PQY;_Vh-@}PBoh+K&<8Vv#93e(4?pypLLxtufK%lj?F_!GbBks5V5xh*ACC#^ zZvH-B6(pkkb+f{3(3%_xCaNkI4xrJ#9(WOG#9S7wZRC(?)zp!9A2J_kR&)gSGOc!t z;HhHwq6AK~3{eVZ*@#2f+Hr|Gb{31aAFK2&?a6v&qdRp~IN11#uxX z#4dmEu0)L+f}mRiaD!yVDuGY*I`9ZP1Q^w0)N7Anhrn31fMG!uZSu?Nrv<*?x$eIO zTcw}{tGoO-a1G?DVdxS(Tp4RZ=9t1(9A^<47N(Y;nN$bfLqq1O_n|qBlhPN{v?d@Z zG)Wu*?hSuQvP3IXa4`` zqQo?9T!C{osPPUWrNm9<9f)_HKtr6r?$QuUQe9rdMpdFNVaJ{A<_B^Yc|v0Nc%_F9i&RL1Y;)5d;u-fQeLrvL61q z{03U<&@tQlZMfiD_q;ETgM6#6%9$>8eD_L|t%az_=1maVhEZlT^&D{|Mjy2gqN#%Q zXk8MtSJeM8ZiAz~auLLfUfjHfX<8^YYN39O%%)%nSWU8B!_EOv^3mJTG=eK!MEVYg z+x&~h_~Q$I^!WL(=Sa>s_YrR&g=DG_QUTBpymxVF>EzWr1ZLN3{?f^#1`v@$ZKZFX zr56s7G&)-gMJ1c+g{N*J+$RR4^1^X&HP+*!;R1nO^VD~Tfxa`!^(X*>8NUzSF}2qI z%$^#GUV(&=X+M2SFij83z2yW^wNhSV=O+>9gFy9O1DSyi5WsRAs)Fj^F2sv^sDoNp zOYOU>#HDcQUkwv^3!Z^(X zaJQEAu_@zlO!WJ^+70!d=b%zl;(#EctP$Y`^$$%vG6;OvM+G-8G_ZZ*Va>bxXK9fJ z7$Qq(nV`+tQvLM!W!EZy^eHqep-{DsO<*uRuGoj6&iOQi>h6!$$t@m_Lo-A^TJ$9J zLlg-_n6BFdi$lnH{YMv|eq|~xK|iAm-J9~`tbU=4xgV|sEAzJ*GpzEsE2y~|wAba7 zhEPcxLQJJO$Q{FfUN*h~wtTb!b`jmi5x@W?@7Q%^e><|<`oyXU~+hy4+=FpiunBAtTqn$6a@1y12d+9R4&~!9)iiJRoq7}qX zgs6T@WQctPe&Ks2x9OdG{F|_pIylMYG!L!pXVxto|27|UjRJT6fq2O87@*ZUP36E= zK!L=k!~oLy1Bh}jAi1G0Cm(+D4aGxXADIScF8Exqo;Bp=o|ev5~%b*s74 zs>;NnP#R`MAB#TcKrX;e6xbN3pZekfdk}*Q_r~de3k^mzaJy_`hCp8H! zpn9 z9sFahWow{(V~lK+Dzhlog`#L^2H-w$wgA3(7=?O~7b4aOMBCK~7H5#khasnm0AHM{ z?t}M`rDZA4y$OS3UUz6FRY*~xiPE;??VtsS86?UNa1+P3&$lm@bJiecx2k5M-@ zwKah@LHQNsF@Z{g>pa)6%iBRGkf7sH7xWJ>ZRDxz5Q;G9|#s6cm>8ivGMwi zhdEQ{#q+fp*c9{YtMFg- z7j99mzfflfC!B-A{grm!!j(Kqy+BS3BG7T}A+n+?7Jw7L8 zIZoj0G-+63t@2Q|v(AbBz`UX-$Gs<;*Z*{RF`WoN`jG4rR0?{a&@s3A{=`Q0x{mIY zI8f`kGZNQO%Y*%C1^ecOA53!Vx7JCS@0vzCA+Q(*RGo--&3Z|O|TRt zh5R$iBm{59?fgM!HeoIJwG|fgRSYg^p8VzW9|Ui%$blgwA5NmQ+rEG9Ww4H?Na8dN_h#0jYy<} zOFe=pdH|b&U`4t0Mv%2o3$4-~FKamq6E&5NrpnethUb*&eH(`u0*uA=KoYe~#EKEz zXfEK5U9`TB;eR7>9tg3GiC6{mMHZryv> zRf$_40I?)&TO>tvm$_3`chBmx}s z?vmrv)|==4FY2y&vteStjejVR|IY`z&J9=)f=HVrR*1+=ykUf&i&K)(vw~>&^PS@v zvF(T00Gk}l9vTz$Y#tp|Ts!Yf=-o$xo$cj6;R=|rycGw|lrmDOl(y3mYd~Lei;Ej> z3JlQ0GZ#Jt(SajZw>d^4awd8iV2m|zr{|E`u`s^%a5^49R5{@@Wf`(Z1%PlGr=la8r`V9U42J|K*E)Cf-m zWrX>5|3G1AEdF@&3%&B5(TDXz-zy#ZR?;q4$>CJlw}rF5lwK7*Y;b7re%Hc;jF-PB zOUPewDO?v2y{RzG`f=2uHS>Pl;pgznA$9&wc~ zb_pyrwDPR9%=)gwq3Cq6DVz`I2L%e+l-!SeIaO<{TXF0Xpy~$)H*z{NwIa%{@oCgF zmcP46kJhSQJ|kqb^So!{+edSXafaUp^V|!w7HnXxy8it)@KwHGpsquUMJgZ$r zaoEjmv@#nqBp#B8eeTz3t&vgoFf)3I=5#03U%Riy|M40u(`f$$fv&$+#N{}To+xqo zqX@K$amwozz<2BFhwAza!*I?np7YfBb4Ct&m4Rhg^7QEyv>e~JFpAL0$D{bk^oHN| z*#$RjkAyn)&@c3*GNJLG(djw=6l6SYz-;%*QjqvrV&T$>h6@*s#r+O4`W?i2F1=Nj zM(Mlhi`6pB7b9aj7G$j*y7pzUs8+A^%?h27nOe~0`mPQN-J7;-NwsbR^UV8JF2wr# zLVZXfdn1zgf*{4LWF8s7;2WjPMh{ol6}?b>;EZI#_b+}MMkBFxRSKFBLRWjA1s)r5 zFl)3YZWTMF)}E&qL(rR7uevCuNx=%b?n)f=uJ6*En}*J1qZIefGiJM?S?e*kI1krs7rF<<+F^*K862k>2?>g5w-ys!Neg`ukACbT8z!P!+MwErYJ zyg#$=K?fhKz#>H~iQxv(vBOozpMH0m5APYhGlH&=ba0RXDtaPXr)t5~d&J&*>1~nX z$rm@vLyyjl`qAE07bxD8qTKcxY_2LWg`+f9&{x}UOq#BI>*R91fx*fw)AkMYX1JZ; zy2(*Ds|7BB{;K0WWjve0gC9z6Re62Gp-6_!h2vd|1^Sy-jb5dJ6Qv_meyqkzJ$b|HoXVp~m?KaarkCj7__{ zFKV`GI~ag#e0NouU)tj*=-%PqVixzq12Nl{f)095x0*@dIs{8k=!~>ZI{h^B^4hh2 z>HB*xi`EDB| zfniJG$AK*$ZpZ|O2jPa+cptdx9;$W3w_>&THwS4Mc=?tv%V>YR-r@^rHF!01kp}9N z4sb+)@Yj}*;wnH>#&y=!O-HnFpw|nC$ZSC;MA0h6fxHTd1#}4~Rc1wbWgB7*^BmCE zNLyf_RX^+6*kIh)E);wHs}&uZ3fzWj>We&a?y4p9`P0y*X?W9+p14|^Sx!RpXo&d( zRmC2DW+ScoM>w4PEt#F?_)AJrn7)pqXAG=jl1s|UE+cPkRvc|mG)&WCzQ1u|IbzF8 z;TRE||74C_sKfa+ip8amA79`}rAK$2WDJ(}xWUpUg9Ry+J5c0md2%~Pc1aC><=yM5 zjbp>jibgk?<;3~r!&=jEaA1Z7naqQRc%(Bt6nsl+YinovG8a~xbYX@5U+3Ie?R_1q z3^%bQYoHK&+7jL^MuUb-DEZqDo3K^QY)}h3eU4}UBR5C^6l1BU7OaOTSnm29kxQb( z&u3ujWM2H^%Sa7o6}MjSKM$Oz2F`FsZ!gM)0}#@^Awv@=wM{H zjp$KvNkL>u&v13KjUhKH7u&nWv{xS6Dmdzf4F194Fr63VHbC7 zxOl7pT(=--lrD$*xW?#6hvzDG9c=C)2;L0z=P1(Z(ofq(gHbIzK7A#JriXCNW0EWK zS1EM<;`Z{(pl~PViObfFf8q%yn7+ci#6tI&Y;Of=oCnTAHzak@?PQYC_=Tb=YrzI{RnRRm3hW?XQ;2aJz+^Y4W{ubgi_ty~ zmY-y*??Z~RyE1E38sz9BB*Fp9EZI^XRcU{IPYd11Qq%ljDS38-Q&-9MxYe z?J__SC0PB)$dCB?nBg$7g^=wwCUUD#^3D%Sm4ki-9;){jfgfWJ&=N&gXkF(QujrF+ zohJQy`s)NWZX>B3$u1dsBs#i|y#XW*?+Wr=O6^ZR_^xeDR}N~;RmMKIZG;Ss=?go5 zV|#Yy*hoj1ju`(_v;J?y_5My@l6*Y=rE}#{`@w-5py7hc{2F*&0eFvr$d6aDrBI=h zgPvO&Y%RJp9%dIER|P7DENjE%?n;a==TfZ0c|)&M6tA84cz3&8in+yw zVS-y!To`GBpBB87Lcbze0l|x-7cltq1FVy^XuaJ>%T;3E-Tk3CfsPdhF*+a3nn%(v zQz8%#NKza|TUa;q{2vSy7o{ec*HImJCDQIIAd4+s9@S&uK%tp^{dwij*1 zHpQys3^0FjL#T(-3UYvYFAuOuF-T@Q7=uqGmn)QMtIPYMaX>GfKw7VBx;;}O5R`2L zoxBi%hC%W|G{s#LkG;1{Wr>S{n2qkjt@%DkBv4NoD?ml3r--(D= zGj;kbRX__&K#^3sHw~t2qvz=G203zrLV9;3-`%W>O-qxZ02BFV;cF-{mP?UA3|@&) zsr8K6@BtjQbOC&v6pl2=Hmm59kYc?rH_&+L$kC&Fk?n0n8%{00EYtt|Cx?eqW|+ew zStX64p&@9F$J?J4$k>}h45)Li@CWgl1Lbx{vd{IC26cXJ-kYXx2?>XEc2hp3WP|!t& z>kG-7M(CCA;nqUQidJmp;dXVH)~j3X4s3q9|EFQZ>;({?L@ociF;)^>p$f9YQ$+Pd*FX(1H&JFF~>9a%KC!|=(qK4-cIKE?paRXO}}Hc zb~zSk4;a8BUGK|imKT;`a%~%PrB+MDAp?p^SGcq_yaEq*fWJgv%old2YLPZ?k&j^s z+(>|$R2qxjg`bM)^(uSmAh860Z&v6_qsP9w?z?Z`TRBsih(lPXU7?KYhyY+EjwL=0 zMA5Lfhwh|b5#cz=IT$J2kA?VN6}ssYfop}u$G3(i0}gfIwZlxrbd7-0eF?6jn9T)? z*fT8$dxZksfjo5&F5G6rGhteo4`(L(Do|2l58baYv=2$z=985OwvSXedFh zk()J)V9T&&MPZZ$x*R1_XUzKQZhqehfJCLWwbxlK^;u&x&l_dXi5cX|D-i2Si<1@b z66T1zhoezT0eqTVxU6<-AnXZMZZh~Cnz9se{BGuoB zw$c*8q4h)7XtR>xU)GM)qES$K5HaYbgMCN4+WPLV)61nS78Dj1%Cv^x?!uA`fwdTT za^54!urEl);SjGK=~(RBdn@`RT}1J9q5QCF_i#i_(&)WYM}F6p7&HH+xs# zT8~5%aQgF|iYj`4Z~N@u839ObN@2$y$8mvBX;I(HDQK1c3Rx;-XJxB7L4bQU6zsJv6K#`l)zj3>kpMCX_uw>))t+&od zO!f@g8H#^Wp`!&1kQqV9<-u2zSrnW4=|3*G088*NF z{JCxfIh0^OaufK3dC|7EXeRnYQ6LDOP%k~?>{siTLGk@Nt~p@x1qyEc3u6)?XOL1G zxamj!3XcWE8>WK)d-l= zQ@;)MW@+FIMrbz=q<$2hNgOikCGy+YWRf$Y#(&8sg7Mi(ZCH`*5YkR63rAF?KiLygeXdF8KxXmj8&RyPGC>&=l-vH7Ma z+I^dvnn*iN2Ihpz!tZ*5S{K(Px0FV}@5wwtxR#W8iZfrs2Ath*GP`edu-FS1F6?Tz z^^^tPVDJLR0^ioN8u{aoKbSxjpai5eE(3*>1){2k%5oRdwapOd-|2)5-x$up*V&n_ zpY-LPJ$u%8FH_dkNoP#WHLV_TRb3+ohiRj!}{ztIgBWr4+$2EcbC z9L+Ht@`b3y1|ja<>&`EKX(@{+<;1=3%!oFAHb)g~**=p8;vIk$i@*mM@yWqi%*&yl zQ2^#uWxp}q1ic=%Mhu<*AR@u>pNs`8m=sm{{abHPEO0z)=ozQJ8R2D|5`#&UZ5WSy z$T>fnEh_<&J|-qQ7P{7b4SBk0;!M0VZ#;?&`-3xo9UQKR^Yb6l(7uyM7V_OwCT&{b zZ_K5gt>s8JUpA92>?Qp%7-6fAp;V32ndLRH97_4d+H@U)4 z90S;qP`J6K5nxZ@cc&No-CAwmbk(xy$4{kH324g})^BtO8Zt)sj5`QkJf1-*QtTfO zEhavmx>!Y_>tZe1tOg#RR=AwAk(h;4pN51aTn(Le`l^Qz&coKyl?G^ zrZHBs5j^42Bm5~5*tQjNi|Nyjy9D+!`(Nh*AFWzxy>zc^c%Q523zMeg0NiXp2<1k< zr5hrbD;&jy4|)Q$s>~nxj3XFAO(B=|<|CIgOc7R&FR;TvDr+8QaPhl4->1GaLQuxK zfk)h|tVynw32tEM?@(|^L~A)@*7aPoP@GHAI!RwE`(%laVUv)zQt}*wd4$iMbP1GL_0q{b2(?aN0-SIs8gjwQ_rQlH z({O|7ArpW1x<=u--!ZIKKJ7HMwGS?%4-rW*6aU(S;FW4W$OS|0Q3h)3)^VbqN0$I-*9c!)2ohTm z3lNDuVOxmt7rWrL$zWMRhXGQrYJ?q&=`%%y6Yq-%9@8QSJ16L@BcrIU{4th(MHhtKjX*ptd`ys2QFKc3-Z2Kt=xL>uKCDX4 zqtCV!0DbYRPQi#-LP*yIp@v^|=Kf!Q%z!hzy8hr!D?bMR_-xXTxx#h@0jJ%FquR81 zDL0&|*Rbtjwxt|jLmcEP42X5olbtb!%jx&pgC;F=JPwxyRb8tpr!q|ylnDq|9l*lA z9yIdWav)znlN;y87pGx+75r1AHdB^v_nU+`l3kCz-x^8D59QumRC>N1sBKp82WFr^ z#R4NAN$+K_K#mkEa(u~cE6F0Nxvc#sAfzmj);!`XWLLj!sENbwn6*T1Yt~?STo+1V z1kF~V&}`ot5k~*k4QB$0UfR{pt9-xm*wp`bAq{sL{EIMe4L-6`z^maiXk!GQ8n-ej z_zeNvUyMFY7ZiT<`5YRTaNUVjhZ+RphTW6_l*f4_SBKKu8js!^vl$@>Iv)_~0lrf7 zpG*(cnW*HlJ4ituRHm3ey-k*h5-g-818Rl@c+hB* zUcG`Djv3>_aY+y+wf;D!AVxpb3AUh=`F$PC z9I(<*{7*zgi%HBgbO9w)bW&`dPgC0D3cv{ds?y9eh7{AoStsFm9krUsM?}}odALP z=j-ot3ebzQS5 zOnD%_@p(nyMg#3}B;$ycSf_2mk}-MLqyDD_5Km55=Yuv4Pe7ur&JfjDrX#;Xb{8I=b@!1@KdRKOM?c>!b%{dNVyic z-Nk4|>q?GumxTPt_nAF0wxM8}%fQ&`yF7Ds*57}iHPM*hEJAqBOE6%U0RM@1GdE!V ztTh|AE{&7R{A!5Rhyp`pqfz~!o>QGmvm0vi#jsY1`Bsv#CzR*)aj_E;cP6JyQ_nEH z+UJ_myu0GE;HoOs*LDpDMCW*(pFq+8F1fkIt(h~n(!fMF>&G&3H>^B>&-crEqhjxi zfrNE{Ai?3+mWn)NMgiD~TcLNK4-rD+b6SBEy1vXUuri#Zfb0Nnx&iDc0D@mF)=aTexFg-l%j5Yr zFal~eS%`Hg8N;*k8Mrzcu>ezK!})m(qHN`F=tSMj1Qb&adlijwRQQAlg0t@%qd6Tj zckg!h8^;lIx-MD>Ftv|HTqXae{JWdXfWEseS_ti$8;FQE%~;~RKk5dvGW2w@GN72p zhD|O73E0mDemf0qps;aWSBu^5SPbQW)PMqWBlvxhJ0pbS(FmGS>w%~(av$S&ZLHS1 zho9SAjY3MC=SWm}F$Ja?XW)fFA-+`r(cB0d)aG!Erb7Y%!#h|fk~NE9O;A?FVQ%RY z{0_QIw(YGp0P;~m%+1Xs$_D&tF7a!ClK9!6X1A9MBPVLm)Q_5+1<8Sb1QCYNj|a28kju$3j;K!_@%1QB!6W z5_@|B_+ej+B9FgANY_LmPMAVZ1ojv0a47xQR~0cMRE!ZK#B>E4XOUcs#`P#x^9OTZ z0bOjoA7(7MR~C@O4m~1cdAFA59Am2AkS-bDOPxm ztP|J4H~_Sj>5}5wRnD^o=6y5H0L>Z0H_N?qbRM_ zrsANWn3QTUwAmd5$l|}P7j|zi7xC`B*N(66KAMed*OhklV31a_35b(O9uYSi$QO$< zn5#?+ojCCB9Vg5n@^y`>8)V(+q2y3&I&%1MZe?a~GLYC?ug}vm^cvH~l;c>4EA_)o zTbKaosL+q=^`QMad-mkZ2@A=79oBg-gY5O&cwYsK3PPcm;A$Mtz!$i{8B~ ze1_Ny*+pZ$=W~p7VQjz+F?^GW;tH9V)@$>p(e-u@kTxZP z0AoT`p8xo1$>Y0S_Qd}pM+MBwJ1^^#)+LNJ(OZ#mI~V`%MNjkb)v=-Am+;tcG}K9_>dzt3Rw# z@;6LSQuf89OatSvoK3i2MngO$V@SQp($3UprZ#4RF@QYiIbl_l%UuV%y9#(fA|TYY z4g~-muW9deV{*0xv9Q&qWQnvhq89$-{T zKM5h&fN1zND)tMue!ri%AuJh$>50he__%$~ScDU-JfbtuwkiLJ_yj5)oCbRXj+G*> zMMVVA7SAVHk0F`#1($$@DAkFoj}{0%%f*;A?zC3`0*q3Do+SfxzyZYVCOXF@2n3`I z&_oQ94r{{AUqXl|Nl7;owQZnfeYkl%7gGyZ0J;rfVi2iZu-SUwpK{;xfcEM6nU}^I zWXfO}?_)SZsDGDB(b#$|yfcyOjDCKCC=pNwu4b_T%Tu}5FYdD~wRiK&>3226C#A$b zs8Ui50>Ee%c_SQP^)=NNF)sZ{0v8b2#+{ zvQE}i9wB>QEZ4o8vKKzMy?eOUFj|rwt+KFlQ3jjoQQFUbSdV1~P*T+N+145^;#j81 z=2F3%=nP0SuA*}Uo5wWolDfiGc$y7hIH(g(QcRyn1xhEzlRPt^s}H^3wNZ5#b8=cM3gZLBsDn2f{q=%#t;y@Z#sjvU!Zu?)jh{1ecF+8GD# zDGi~5;X+MBll@v9MKwUK$z&XY+V286=kGDq+I(+jde2W9<_erjK*$49kfIvow=^fO z;ZW&!y0NI@@`cIiYJ58BI|P&h5CN_G5NL2z^?x8sNt9Q@cvh_b8Q=uKsOp+?DQqi3 zXi0!k6*vM8FMnCpgsooy?`qoq1>H~hl=q3Q{sk1_!nz>oVTr>I(x{}5vyX@Y;ez^9 zH>jRl0d*2*%}B}-sYE)tzj7%GU%Z?qF^R<3n^Bn(vHSgVqBn3^kb5bBQU$8eS7=Q! z1{b?co-%GKC=kMh9Ul~BER9PEwVxBr#7G1Ty`(x-kih#=fR(hN@)Xg|$o~Ro(ki0K+~4Ii#E}R3I+C9HI{QXN(vN6<&8<-Vt8*r<8yJq4NaeQeD=I@ ziI2im@&5=+GK++bV0R&OY1$Yx zup4q7nJk0IDLn}5nnA73g{`Ut*DiMS7yhI6E?gud~KRUC@MB0ysfTpIX?IP6)WP3{kv?iq=%dN{J|F2V}A{8{#3vf0zpLFfj>#;Rg$mR)QJ0=U{`ZXSEr` z5h;QLHk9%gUJwboB2h6wmw)S%gVSl&V-$Thvq6!-B?@4`BydjVG1GNp{4NwJX*@vr+HASE56_A!D*Q9M zl;TidaY9dn=e@6TZCslC6oKcan>3Z-TgT&$$14>MkX>Y6!C%b!06j`a>4NA9z*T~w zRwGqd!($v+*%Icvg8Ci&i?78E;HZ8~4JCY!z;sAdL{8PR403m}VtOvUwh6Z8!vg|1 zGhAlkV;`|uv%Lu2W?mYJEpY{7La3sUGO7rSqnsRc)cK;SOCQ=0zzx} zQA>s(D50#B;7Bn1j8dM#l0zpL88%NO3xNX^vmYc22v|EOMY`^08DEYs z{4$2LM-*?$KM4FK95xYSZ2>x`N$V9f5r?7TLRr`Btxa5i#S-51f7c+6%Aw%B0P#tCk2iTI z2sWjXI8hy_&_~2X$|uQJMT1H}e#7DM_8EN7(Pqy`6S@JV!XwxPHosEJhZC0gVE7a* zKQG5UE$XqAXfE7FC~xl7)-Fv~$f2Zx;5?WxE8@pej}b9X&|>CCmF`fj7Ly=KZM;%q z!0n~R9$paCRm{t$G8`0C-3+M*HTV&o_=Pkm;h7VhPOw{U;P8MgL>cFr zzx@3d8E}alPy}tp4Dg6&k51S-`Q@~@%BG9P_hnHb5<$E;^rT#gu7P$A1q*}YfDvTg zKAcK-K-jwrnr`VPfI7(njz3R6mbgt|!>#2F&^;VZf0NN&R132EK#6~5*#;3`7J>H` z{w97@(_(6lO@Ilb^{&##rL(atpf+a#X=?@(VSv)1+U5l0W4;#csG?YagGe=7_+~|O z^a8Mn?84`njoPEIa+O7^s2kKPfBtp4VA38`XiZy2F%v2L-W_w~bz1D{@2x)~X@U-K z_|s}?df+l#5gY@YS}+Psdp)|Ki?Ji@ajd&;G{p=;)tNxBQ)^9;QVEMQG40~%A!4t9 z;pM~CUSJW_=zn52%?fY|R^H;{wNE(7^ufoyXjhBn_Y!QOFDS&)>Pmi%akou-mgPAn zyrdHUC4TJyGS(*xEts!O#{uT@VF~5U3?Wpbfo$EjoQ%7&dv#dX(&TZEM0vu2Z zrbLMiiMDPqd%~+%&@+_HnyMwd2@Axp1{T=lg`L`)031Rf_8GHRHDvd0YYC|ZR?p^w zDZiD1e*LylL?Xw=UT&%BX(_cVoM}BZ(D}j5Wo?rHYUbW_mZ{13sXwq-GjTf;0QK~c z>WII7kmf91^Jifd&%mM2tjRh)wqzVcA4hssCBaAsd+#gi9ps~+5@D}QGoKu~;h8d* zh>eHSZE@C4J&VF(5T&z6UOwb(w>V&?jMBNivq*g>y?UZne zJ*ZlS_o>!LHB2@aUZNRo18i#EV6h1hG(RJm68`!pCi>!#J$1uCvAKko!@@aH)#A6f zY|&o{f_j)<3O7{kz1ZpL*ANaE-cXZ;)H`WYEY}=p1~GzAJ0Y$YitmL?l@8!9!834l z7m54;r9RcJ3Y9iO$vOtt(<420lJwY7_p%0h&aJ!7dk!t?p}+3iI6cJhR{$1QsKP|O zg$^i_k>T1+Sr2wEg-XQdU%1d%P^|O=y>JV07n$~h`v;X;JtaFDI`x#bwYAZ=TGx4x zLUjsLyHQR(ey)fX6!VH!07v{`$u(Eq{e=toX5)%E`-S_B$Cd05m>&9wcR0aL^xAdg zLlCskTxq6Pb#UJJRUg{-kGBEh4}bdpzr|YwH4ItmhbRk|5mSkSLG#+2&-mMzs4vD` zS{3)w&KKG@MKqq~hne2N|L*(o|G|AT zmy!O+uP=-;5qenj**%Vua>ZgrYd1pV@Cs+|c_5bb=4^g6YR2OG-`D&PyPvZAL64z1 zc{vs93ZjL5pA#Klqb=i8Dlp^A2bl=lYtZ87xxgFpeUD8M(lgnSEO?js#9TZIZdf>W zgJ8nix$j4B-{QL6Gako&G|p*1&Q(fwIZb2oa3jfKe5?y|38ILNm~TTMSj8C<@8HQ@NYt#%VQVl*1ft)Q04LJ zjriA1vCKWR`keokzKoF+6bR=>OdtQDF56ZroS;dE^xE*@o|SGQG9I*YO~NxA2S~QR@CB z5#^&d&V3jauM~^P^A&w|V#x&`)jc5e&~tREQq9Zdy%WFYb?u`IO-ux)@#8lpej&JG z*2lZZ%dlxCMWD<}sc^&6@n^((l6XHGh?4(1ZzTWYKkuiKHcUjb6{6-zoy;sbC@kjZYWkZS^Ih92{Tc#1LbpBQO_9!f*s`0K?CtaJQaKtUVM=*HV(d z1i~gnouuj|!rKQs+m554(FU%}%)Tv~2czg`!X4+xy;Y)wMZmk!&^O*LmPZiaFY0c~ zC>5N($oc{`rC-J2Fw}uVUK61~E;yRkkRsThEAG5_-i?LM7t4BC8*A%Uqv$+yw} zRW3!_Eci{?h!@qjx{%P}M3}MBp|yR4|DvsbE!!THxFkax`VPGwrfc_2)q)s^IpV%c zEJ#b3y|6pe2bmX_tGwViZ*q|MqdEwTrBvxet2?z1bP+?^;Dz8CxQZ;{4CTGRfm3Vi z4&Oef=UmGjf=D!s?5V z^~w}OOlWUCL_ac-Tt>rCbXYsTZ{Gl+{>; zi15N<=-!h0>X~+s;DxOoN|7bhYi*H&RBH{&jOv8d5x0hNu2KY642}YI*~oEG0|U*u z;?J`B3!M;T9XV$&_VSm?MPdm@gNi((a__6hq_dSPBy;L-O7QpX?bxg)lZhz*iC$0* z?xvSUC&$X1_zv&K#=1DGL4OBzNuW2H-;2e;%#dG3arzTLk5oBhy0D2f03?w574=&( z17Xe4Iv+ipPK$xGc2B4cVUue#KB8kiRQf$d3^z->(K+L%fX%km-vW9r&IYoAP%S zm1o`{F}LU)Yub`^WT2mfmSel_Gdaqi*$Y;_m4tXQuez&KA~K|xO7DN@1%MUkXepT; zZU_K@ zF848glaE)n6Y&7$aQ~{&8y#+|>IaIB8;;7M?r#iCHfG-f+<}tKhNV4OI36#6S}dUR z!!Y$wW($GH8B;L=97)n1yCJ!$XH__5>U8P@1L`o!MH?rPw67|-yAtSalTt;akUt-( z!mO{>1%Gq?y^C!Pm=dvq`hdq9c=QJp{#@qgT#Y3A7(5TYAznazEA#!h!1*tI<^R0# zPvOJ=49Wit3APix{yXvQe`opltpk{}Nm|+hN+>PxvtTs%Qe~I;?BGX@#a7V43l1nR z9ir+GsGbSnWtk&LBC03)XRa<`@{0EDD~1^PAc>g&TF_W+_3xvwMW+eHP+Fvu$Dr9l zJbP5H(8yIl;v^KFF0#J6eH2nAhtP9QJQw1KkOe!hFnHJpNPM~JiT?uNIyRLtZhIok z8DfYRf8$c$Mr!2j?`*ilqT?2b?8Rb(r#l-0o5mp4vAK$9hKFY=;Kay-RJ1!#PR8W^ zu{w$36fSE>Y`!S=0?f9ZChG z#qMt$s9M5pM5Wg>CV9SSR^vjoZJob=NkJje5#ZUMwfYgZswAs5F+%zXEiuD+RJdj3 zF`c;wQCOu8MFvuh&$m*f{(~1M=j+{f8V6g;Nox$rX((4vS<{PtGb1F>tt$`CZ54a4 zuOo{hgaA1gdtz7bvI2XefmiEkVKoM}!M;#OS`mjm#i|54M%nQGtdJpOf zqBbUO6ogt7z1qm+q=dhTo0P7_-p~4xj?p$eT8ujco|tBvHxyk|4Yfy!nrehGC3YFI zXX=Yp=s7Kd zNI(2uv|)|Mjp{O$Z>kcT#oRL(`OS9q_v42k(*_KkBZr;EPvwZ3by_M&)m0J$n=*kU zCrH%#r;X)T)S+ghcZ7)otAi$1Ui^L03#|}!1$Lgw$|DkZ?Lkz45wl)mA!;rbi+hkg z&j>PFn+;m3!M3OD5q<2mQyypPBrs*dhu-BQC_4~>$KHVYAXO`Pug9oo0(EXF_69V^ z=$fYLh$V}lnqNub6Z&0I-?zkaXu%016sSvw(_?w{-qIRhgauwYb)4OhJzk3H9LNL5 zh=R$2%tw)}M|S{8$}qwWiqsIH)n~)u>;h{J=>8j_GLrNF8I&S9k_IBxELI+MoDo-{ zzWDf@AoL@(i8(56%jIV2&TY@+3x zBVccNmfok;X9a~UW@|%bD4 zn#3li%D@N@Osp1)^e0w{)?TaX)3@$t%)b2BwDs|Qu$P2?0=7u?HBZd=$3ELGD!(JKG?3(#9Y} z5(CG)I3KmZeuS(}04<0bKZacY(>L_0Aijz_x5tUi{uU3QieXDyWN6p$`)u>mrI1z6 zCv`3A^Tydoss=@x{{bjEm)f@(tra|XDHbaZQoU+mt@hJ^`ky!66^TA54} z&ACka#lY*AQRmZZlKWPiVnjkA9!QWk6)BLddWQD-BkwZwY434) zAO~sMV?@2iRS3HX&B?dR)?1|lbNHEuho_oRn{Anu&o4`I4%o!xbUN7^Z*1>Ldm!hk z%)hChx>MsK;`F7oXuHs@NJ^1q(}{na-XOQ7B9aaN*$N6Sni;;SU2iBU8IfjBmlV)%b411h`vJtFuSj9S`&7| z$&d&u*c6GJo;r@sy>MUFsS^DFJ`~9J#|eDlW$bu#n9L)-!g>%q z51<;ehCBD%xpUdDkQ#p7Ny-Tl=k1&OofIMY<~@7ywz+6hkmAn8f(XxbiQMDE*L))a zYiLTOFJ0d9A)MCBltn+*smy`a964&v&%zaF83aGjZt?GY=P6dJ~JA~FfD(Bjca#9dV_ zV(E#?oBHrl@5_85ekTi~CLt>CLw>7F&GU9i>lzistgO(Oo-M7?Ikz4+?2C@++3Ls2 zk*+iis<92z*IDAK998T!SGxGqL!|05tU(Pg7&K?-Y|LeQ(}Li=UIb$5#gMO{kYrF) zUQxEgj|emu&C|??Q@S2@eHq&hJty<~o*g%f2T?(oeBk}#9j1f@WTOk@o?&whq?gXH zvQZ~a`a>EN=Jc&l0y~3RZu~1mn@_+PI1m++B1YVp(1oYZY;?y3&!GN$_bEi!&cHB6 zJq4EKO>={wY9HT$h7gm{EXh!DIbteD zW$sIK(HXz-kcqvQ3?I#f)xQx9Z!jdG?@sW|P<-YYm<*{Ut{H=zuGX zqtTLPXrkytHZB5~>r`*jSMp4o5P@;lk!%xldORmfm&kwzTe^017PU1AgI*~CL1 za4wk^J8i*M0To7vcXEPse)@Sd&BNeOD`N;=)#11!xOjU16*!7RrE z+R^1y02lk$J0o>u^pCDGj*s%qoF_|MMeL(Hno~?KnrbF5xNz%Ea_anol-+*(Cv_g- z)=`*5-`vjAio@Ss4A}}4++U`i9Sg zz70daB3UqGts#$L(F?kXmD)%``aOe0$Y0HeKBiTb0-7vdLi`U`y*Jnzg?R&m(ClJJ ze~Y^3{of8Y_7_F^Vo?2?dzhs_>TO6d)aNR9dM?Na;N9Wbl{@^+Pu-K2w}Pt(*TeH`i4p>YKQ5_h#BHX)*dC`Z_i!m$|gJOiPTpWZ{w&CMs65 z0XT7*!*w_wLik}WUxV3_4I-!Ub>te^kiS0z>dw6S!HQI|kqeP`OcRWKbJA{A4rOP# z`H+fG^A#|Vc(T8Hq8Bet^9Cx+oF_OZi)5$NOulhZo;S8EA;WAlPL!{@1$F;dHK`r4 zu?`?v&kxm#cnX%1|KPL}F6L?WQP(VRBS;wIA^S3q2z~28iS=vm=pFi%C@w792Ru7n zk0pl2HC+ogoDr==n^vb!o5ayu=u+%elMhV2)R3<$p2I(~wz!S@?Uf^R#4@qNQmw@; zOVxa?nZ7;x>LbHYQXih=n*XG@LtH9z;gy;O*0?J3U?to?o5E=S z!P!9)7fo8~43U)TI7ApUXc-BEoCqLP$Sk7!9C7u{1Ia0laDe4`c5iCaNOp@_+L{pj zaoV+NHp%$qv^G8k-b@64qM>(FgD~k)DbF{#4BG5!YPX~&TfNv!n+Si28J{h@eCm{i zXF)9(KU@|v=~pv5WKZu%f<^kW*gjIxCB)sq==Sq2kW92h4rgS z0mGGPlv=s>`V}-F#l6iWpFxoW_0E%Jqx#Xx(6!2b{5*R{dl8U8@OWv)j^(;7PvaZX zw{>tSP+iIHMTa22B4HVi*GZ8ni0F4qvc_19N)gB+Bm#)bJxceGv$q}*sgHdi7YA1# zNP-`vf4LV5|I(42H*!A@-!yU#{&L`z9og&Krh|)o_0(l59u;@+va#lZBw-_q;&phAPz%q^Wt>A;nJM0eK$k^vtA=Q7%7($=CB(q zcC;+!QS^n!SVccg{e2Z;4+=4+bl^DuXa^8>t~rtz-ZMpvCrNXIc4oU5Vrry_VcM{8 z-q>$b2GW&0!g2Sy6}}sD z#;L&660YZ*MO@+_aKE~!5tOPduvGUAn@!3MzmSQIb#*ok0$Z73^@02AJ))8#^24su zdWzJ}FHvDi4WU_@M|v(Flph`*bt->tn{h*=GGmFnPkG#miw&)jr(s5)F}rqKZ=pm+DJBp2X3nRtly=2UR%|9wa zH9z8v*r%!kL}62(w?!|`B-}`-jBhU&LFkd>H>oR=>k*|At&j2r zq&4PL<`4mm z@f&w!m-V#5^FaCzLf9vc)-XQjhJz!A-FWYQ>kF3=^cI<8w^ebXV*Cb&M3E$N8*ztf zVCWLka%j$eX^sI`K0xpEiNdLn|AbUk{?oo9Oj8qhqG1L>Nd z8c2EQ{{||ao^t^jsRKYUvZ$LT12qa(9%*}(^#Z*zhgIW|Q=zbksEzfGH>i699rpIv zxt4&}qTFO^XeC$UX77p=C<@CrtJHII6@MzUvq)lSfs zKLlAl-tfpO+mSD*I6}QLs4cw~%Q~~iN6qR}8L5gTL6zh_p?i6+V8jf?6Go;In1>%V zwgrINWna8JC;3vwkjM|mRX*})Cr5W0X^6`V4!GlflTVKRaiHhQ!CT1KY;NX!t-@G? zQgM#*w6c4^NpB(`0RtBCGHP3hfGFaH@0Ls)BhEhp=q*QUwQ4BgaHIHuJJ&MMf?w}5N6DQD2bZII%rBzu7ivR=loj4 z^dP5Q%biP-SpIqL?_&@eOk5osY#yrf)>WD4kba41*Yv16=27bYu@*ttVtS4+OiDt-;wn$}8Th zZ{k?4Wktm;WdNy9(7zv=U?yxotZ1_)r*}BRG%$OCAl=#PBc2-huY%u@Jf;S zmBBfkeko-Q`#zVcD0VpE^3MC6w;|Zfn%%@0KxR(Z_Q@Q%?t+b^%Xpo{btG$uFUGXOQu zojaG@XTLV5Yjt0@D}m(jy`*z!Y?m!8d1(-upn;$@YcMZ)E)H=Qg?uyz2iC9boXCilmzM;lS3u-69<6Z;DkJ@qv zAkz|vWX&!@t(;)ufwC|KUqZboNMJ?`i=9_?HzOzUibQ?a(L{;kKEB6bjSo?871i(= z?gb(TR0^mikF=r&5I|qebRUv(3?lhpO4&%Yi)xo?h*P_<$`3)mNbLd723d{_X?y@{ zBh3hJ@z{M$!_>NYYK~}TZFOV6Uy9$s6Yi#2M)rRb+w;*D_8;9DTCPn&>Bp#xCL8|i zg!@r^CB!;XHfUN*1BJtyV!qy>gk&7%aKs1(na6Z7+*#9y8MuXJ=jh&=-iyf9MaUi&PP>ur^Lthbccwc650T+1h7C}l5Fx*`!# zbYI>AjUT%ZVWuSIy)O}8OORYLE`ch%Ftu9xE;+)7|D%7}q1Ebtw|C`HP2OwzPTOnM zE-DoiMbH{SMWleBEQXfRa>Fj00s>NyEg*vkvg_62!WKltE#yH&PPsm_XX4|*aMn~R%)D3CSF?`xetIlnmCdB*)({~~`pJ~_Gnh4uBkUGe1^ zQuM?hq%wG!atT*@p2t0|+7tC5rft-u({C`0Ir!-^{~928nCsZ~E!>iT+L5>ZejKs4#z`;Pa>?aS z&yQ>z-^`!+kC?5ped}qVC1v&Pq8<9-;kBhB8^t%AU)pF~q{{SF6n>j0sjd+3S9RL! zik`!pJx&8!oal=PZ2Y7(d0T&%+3VBg1=C5J`n|IwGAb$;d#1U_$MGS+K;CoHqnR^} zOj~hEf46StUcs45=q^_F-oI9DYh`g@jY?3y{JJa7Iy!?DZ~b*n?$H0s(e6uMX!3Ah zJXv7h+*C~Qm94tv{0N!eUQNS1->2tgdaoLGcl2HI`uX(Y;mzkEU&9M;yG=v5d3kxl za?B$o*ehpPV}&5*S+75^%7(YKMkblDN=$dw>8Z)bmL#%RJsVOKT?h!(%&${1fmfVE zi+O8Sos9mxWpB?9u{d1P|6=|()Avcd=HZezu5XIX5bEUw;0zOdXspOx-0BgpJ_w z8U_I$!FLCF_z{8|BNLLrh5)3sO%NPIkoaiuWt{XEHXOvsS+{T3^$8fjmE=|A2iF+V zF<~MT@_DfHfqx^a8X1@sLdOG zhMY!clM$xX8uYc&b=-^;o{X>tLxs+PVR51V^jm0_wop@1pfC4twK}zHa>ku%?xd)> z#$?h;;#B-x0-2LHl*7Jc)97Q-!VkX%3C_xp1YpARp?kp@mUgsobW9aJ7#L?@Fm#-hCi_&~Lk5rj#Fu*8X^ zhu{Ne2ob)tlGkysKcSl*lSo%I!W?&M%5V>>eT-iU&YZDh!y`$XKr z!a{igI(S^@UI=(R*kXuTE{pYa<;s;_FmM_n(_YEsum`|oJs}YoK{q~4j;2E!{P@LO z%hWO$zxg-#l})N}$w}gx8OzfRe%21N%L503VvohF@-vybp+eOOt4fbf-1_IL?wMe! zFu!CoQYO-AzfGyg<^QmfAh^Mu@b!BhKYkoil7FuSZ3P{qwqoC2+v6Op$=GY?#CzEJ zES)l+Zt_M|mrE25nufG`TlJ&Y*Y`{y-_wqCb#8HSt0$G(n4%j#U#z}o&w;XwEnN8L zUU2c`3BsPz4F1NA8@;jI@`9|atfc+0SK7I7F*TeXe@1j!6hu8iRnM5dP2zGKA6Z$` zheK&mMs!G;mS`fVdY@*1R*>Fu=46u3$=8$PKd+v3AK2U5*DCB=vE@_)wM>xT$p1!f z=vHEPG{af<8qT|xN&X}yh`E*S>=vWDB~G)6$`j-l@D?M;JiehEV{kGdKHem_daq`{ zY@puqussvWz9(Ow68*edOQ_zB3-_+rGWmX)c4)ShLLtfv07f-W(G8xw4YtYZJZxPe zj4|f6H(|6DTEQcgHg)WOwBS*dY^aey=*4AXX!_H63pXE-}j01D8tV0c}Y zR#xf5!=9t9`CGGngDR;Y9j3DgK911@B-^vo()6Gun=tXCF~yW55sXM~3gVCr<#Khf z9MRi0N@cvVY0x}WxNXVFbPSfd4Suo0p@P85eV0u#R;Dw$M8;{fJ9@%feagKgt;C&i zCc%|^FO%^Rjb0wjXisBwaYU{9zpDB89gv`gW?dPu^pRG`=cRrxXXm(kT170IA`&%I ze8lqge9K=Z4~gP^C74E%1`4J9$p_>tA{zLi^pdvkV=D?{o69Z&Izggf>U}!kc)tK!p|St4r6mi*4?;`=^r$c9_7^%mSw=C(!t;93yp}mT z8{E{=l8s(HW10_mr!g_*CW?+4otPA%uzwL?$vlEE5gzizuJAc;qU{9=F!j2UUB-nA z%am+52fzT~5JiqEie?PCWSMCRmAXl@2|MG+2Jci&(im7uR#7Np$T`bwcDuN8-yJ2K zp-2n+OlLwvpOnwbTaqrEyFONG5I63A_s$+&!aW6h_;z z;3pmijB-$B>41ZH9GsT&0yLL2n)T z@?EPXEW%U9=*cZi6H4UJ5UZO|qbtKEI|${Z?~#jNpIg9Kb8FTdI?@|fd&fa7_VI?) zNbTqm~O8c(8wBou5=+U*^ zNU+Nb$TP1#Ra5J@>vZ0KsO}Llfoi&aKhi%(rf<05ON8_b7;s}YiTd|#x)Or(7|gW)XB_D!suWLEb94-@`vYO zv6eQF&)V698bi72c~_YT6I>5U`Nv;qLN3fr;ksA9;>*ZG7o|IT;o9?^FJeERZgB3I z4@tu=jm7j57Kfvnd$hzHu;$9)ux&z2CA~^8+-~k4C;TrM)}2a9#@ZQo@0y{wTaV#C zc-JjVU!Js*R(w1cfGK547Moo=GY|H|ZOoau9LPx{=DJzh+vmbRisqAQtt>2Vq0;V& zd(0vysSbiuf=ISB9Kc-+GX6W#6UhU*?#mohWAzjogGO2L_&)ik9Okx*1^ZSqjRp%D z4kGLkd~~Z%p?7;`7Q=xfdI?pm>~~;o^USG~K}%(6eWw`HgL0<3eK1-~jDwOsuEqOaX^w(uOip^|ovz|XUAcn5&+_8gTz%>mvvbSfV{*j+k!|>qP z-E+T+Mu8kK%eXgXSq;OsT8|~(bFJP$G0=X8AUc)2Fln(c(@O*AuHJJ$K))XbjCJLr z=qE&=e`36Z+vktDibELYQWq(yxCeQnxIqn{t{kY|Yw_`WOZh|^49eU!lnrL#5unZ+ z;UVP!N9mDVTr zP(Xv>(jH=m#m&3O^jrEj7+R;y!IwK8dkI2QGjU^hTkrOq@jFAkf`~eOfvJOows2eb z_DL0H3zqGl*FRz19~zfGDypiwf1=Y{<{HYyPx1ncO^p-11l?^8L?QsQ`!eQZ8elsfqE*D$oYx^-VE%W0UGjdx5VcP@^Z+(^6qIpi8{F0EMobgGOcvaWfe>m}!Y8CWMJo!Q!KlJG))~g?( zznTE5u?Q#wIkyls#}@#X69gO4MFomVN=-o6C%s$5N;Dy5&qfJ<1iiUtaP{wnF9hcv z$=8E1NYe1OZ8vEK9g`yrX&UrH8ZtdpaOM!Ge9Oa!GeLdr00l%0IpWBcU*v*S)(T=n zFik^1vT|}P@f=gA8V;dzs2NN}=}3kR>BeLu-X;WxxFqt8LL@vI0RGQHPX9K%ELgV& zBA4tgqrO0&cS`jPK69K+&<*?2!EUVX6V*c( zI&W5c0PNo{?7^QT5yYN5K^K3^s4H7-u>*q=-g z*AmyqI@uuOK*T+mdiRr+l~n-j4x_BR1z}DC3@oC#0>UO{Fq{IQVJB6p5av`D!x>S} zCaog85iR|@yo=hI!PN=sDdYijUe(o)&rC46&<35_!2fP8f!3awNMsN`{Q1PPZfV-@ zha+d3A20i}NptD&qo2DRl>Wwzd%wT;oO7M?-}lFNU59IjE|+UP&;87M&N0Ur^L}tiPHMwihP8BbbQ>tr zXBFt^)=1OQExWpUC0-fN+`JtBJ7IH9%|_8&-^Tu$l^)%NYc>`p<~AmVI(zN(tgH>q z&G@)@j&TWd?7d-QV_|)Qo7?n1@8B}GGT=TaS*MOqSz{rsZcRtWm_+{lLyN(OiH`2q z2Flq}N;g9WY4$g_E6*~(#KYaWc z2_5UH4yNiCd(aOLZ=`#1>(gAAMaG`L z{`%{PVw6Wqy7B9a67xzwfBtN4?28uH@a_q7Ytt9Q{mpa*Xpi^DCB?eTP9C|O7JdG9 z$|1Rs&r&Qd5j#0AN6QBBj!sRfKG=G|ri9r(+HG;+nW#g`R;tGszPcO=4aEdmu{sfsHp|c zOb*3$7Tr5~{CItW3zN8W7Fx&L)J1FMBO2W#s15H)~h9o;SG&i;nD6RvYvUMv!c z=N|3s%b#ki`(FCQc8X@-W0<=*KjJnqm|ERU$xkX=$lU#BXc@uzBpUE3`fPIWjaAmHy4LPz)geaE{y z4LP>PaZOvdZXKU$cQY>YV?SZhajPmsI2@OB<6E(3ta6-6hOK^qT=Hv$NEvml^A|3p z&xjm5c1bx_sb+GBnwn7ke3!PfJd4|0G8;P^+gVzUtzx*Or&8Jt`78O3%BwaqeNNFU zPcv$W)y%YHEtu&)`t-2;`)fh?G+IO(xs}azZzS;mZnf@1!iHZvpbF`eNuL5cOLVk zm(J^X?zEkeF%Cg%z%FR=YVEz?4y4DAQC>a(jbQnGy zw;00*Ufgy_PHB3yYsao#J;TFv7OYz%=)SMRmYiy}_I}2193>Ze!muG$Ime>&w&Ubr z46}&cr}FZ0+7*nPq_hqRS=lB`?a$0TdIX_?OVcPGAiOB~_gko8S7ZsN*i zWy97ayE8m|sL_7zmu3D$qlQsyR`SE`?EAH~weQ}$7bo-dh*4W!w)=YeS5J?sw!{k- zjdqn+Ht|aMb86`Ot7n)5O*_iUdaxNz4bvPKrh9CurkODoo$piBI8^GYr;u+nOPW0< zBo^lsy(Jd1jwr`QynTCe2e+1{DK(>Elf+zyMgEsBYN-i>QExL<1-Q86c0T%OH2&$s zgKef}j7}px6+wIqJUV}41rqPyUp+lN9dq;NS8_?>b3e{pDR6F%8M5g8yl=xEQ8iqk z%GIkcWc&`l?5+$Rh)XI^kdb+)Xe;FF>-)<0!1*$r{Hl>B&X$m zN!EFuX7zavWA#be63Q_O=l1Q}w|4DX?5n5bmCOq+L()1ruf*ra71NCxWVUSC@=s2z zk5}i*UtE|aA*nc_nQM0;%w;l&f!`?N#}5l^3h7LmT`)=6%;L^!3X>(K8YR{E+JVk{ zEGo$NfByOB_AOgplKe2-n!R!MN${AAJ1(;CX9{}7D#$A#T->l{GL@1{L*ovzuVei>aV%0tyz|21=#u! z&vv_|^OVVM9LXE=Vh}Wo#}+NGuD*!dRG?;7K? z@p*q7DVR-Ew?h&W7zpOkb#LIWs0o)c^`Av0s^mrGsZMF#(bLE{-G;}nfv-?Gd-iW~ zcZdY3Jq~A)PHev%qPNM-PHUA849IlZhDO0ruP!Raii=38NbJ2VPU6FpH@mYQYQK1s7m}N zdAG^I=I{V*SM?1H$1nZ-`mxgZHddpr?{!!sZr9o+c%<(2#kf;tajnRm9?9?cpUak|#`9f!zuY>sz-9JX5{u}~W9Ju> zlB{bZDDH)Yg|r2m{`%4^l>SOn9=t8sU{tj&OiU)3Uiy{K=t=b-W@&bqY&NJo;kr0KWl~-^UVC1D8r9N3-LN_`5UfjFbbF)Lo)Ik>esyO!N=PbLSmJR7onGFWibL|r3b8HOn&2s{i@VG5F-y32UbF7b6PmQ9T zcwbcH`SH;%1EgRbq?)>;%fIe|S+gUKcmMOyie^ba09 zu*;O{{?>zJ=gF&A_9cP9sp7wutzt6MRxQn&PZbG3HL2jxKqTdM@fPkkl^3dklhe8;)3+sAdtP# z+0~`zj4~#ivvWD!+|kUYwFF9iIU{=~(ETX&K-5oevT=B?i@R>E`^2onkg=sMD_xg? ze(1V*Sl8L^k#p?@E}qk4J&Z43zOo-r&!@k3vBezJ($jJClREG%bs&VYJo`1OG zXp4yfih@49MibX?&M=xKwN!mi`FJ&+T+_yQ^}73y9`SkeXVa=JtZ!<~aM5}Hz!D4; zZIbX~OHfahEG;b^otUt7oYyV!>U+0}<+#*{ZGSyws3l#ouQu|jzkihCYdJ&Yqsv(q z*Zk#oW4i{h(NH&hBLD*WQ8+4Vi-$7X%~FSuN{v6<-!eKm+4%OaWesVDiYt~czk&Mt z@cw;^8L{gb*NRpGEIRo{{IP6hG@v7Sr~3xRMcGKt&(s#5`SUf<-I)$D1Tg9dC^1E%wWCR+FCz9zlhq%^Kq!%Ip4fQlv1u!92^|_ zKKZa-MsOi}Mf&+|7aObKJ8~r_%|W(oT~Af0BI;3PO(^;gl}t05Qnr;I{)%CV7c(F~ zxf1f2BO@ZNi&%*ynKq}KOVG#^C|rm7+JIn;|MW>ZRLJtQ#KNy|l_af%RD z+bh@*Y4tqmk*Z|jxTQ>lOfOBwqJ>&aPs6W(ghv7@az+n zs=+8|R{!hgH(O+4vVsIvbXSEM^XI;u;L^-`VMfd8Ln$-h zEaLG%2n2?ar-bhs9G@&T{#N2G7?x>DZS)kgY(v4e{n2rXqEO_%;Q%FAz_d1yM>ox+ znU}1>-F3TEJQ?`wBhT*=oBsX;;K#}!!1_BH8C85m%Z0A_4bkcvS~ojaRECPEK0TtWK!9SxW&Q&P4?fG>d*jQSm$z@<_6rP* zjyxaG9~q!krci?CojOO$wKtDzT3>W*zy!tg_MJNk{yiVF8=6PvQ>(?}9&9@#<2yGu zSD$IFO^QD9Jo&C-Z9>1kdT8bpeI3!6QufUZ&8#L}4wrY~> zdA+-j#}h!W@2^#RKBH_jq#9hb?frZTKP?3=xdqeRjHEjybnyO@(UqC^4G3in?cuUGWqe?dfSnR*M+!7byjcP=Gc$`l83Vd5r(+Xe&bygR_5+?^ zP|2q z0R({UD($yO`R`I6nPsI}o9wI@g?(fYcT&t;tAo_-| zS1&VZ%WFQ2oTF}rtfXERBH`9{7{Elmo4_sKCr?Z#+4t>x7bs}1<=NIxSL_?ru%nV1z$gumYo+E~0iy)UHFIoAZ(u{JH??M4TQ~UgrNkWq zY%;R>Ze1;AU_F~RKS|XO0@`S1a+pPD9jG^)Gr;u?I99!^B3O{Sy!XqXq=pPHa*QcJ z;-T}|NN3N&tMl{nJb-CMT31$usG{>V^5+Dqt4skv?!F;Tv6C9rH*3<$J|DoP?rG!L z9b_;#S*e`DXd3kR@l~=;H&He!5lxo>2Ri_<nitur;A(RGN&WH))lph$dAfG`ozb#;aZP->_~SpMY6w zU}QzR+oI|7^C9{Fjczo(v0Xj$ZUh8Mo>)RRxeJ+mJf0z}QP(^J-3>k**A%|v=O=#t zs|P(_ik}FoYy*#LeYCv5fJWL4pQ7d;sKL5bCf@;_rub$$?rf^az>e`mWuYcmqeMt* zYKG9JZ0FN|qGna9*P$;lDbOxD=7DV&Wb45s%v(Nc@Z6#J@%8$|E7a9%*YdK_CnqPX zm+4hRcMl90v$3&VS07Fsc)urfzANy{{yW6YaFqb}Y+T3Y%T?#g_jvUr*Ya^JQhL@K(@xUaYT z5{69?q_`>*lt_uG@X=gHR6WCq<`Zkn?9i8R%uY0EYjEZ|PQ=QCvKo_Wn~>PdhqM~l zP|zcLtVZReL=pCJrPU_!UoVPGJ1QI^QPK3(r7}OOWtxT3i?X9ns5JEE%QTKNr0AkaA}|Yl7+7G9w=`j@pa8JA!0pxpDrBZ858%J zW{Ad?C`PyD0Tjo8@|3x@|E{|`!|c?EegAwmZ(tzp+su@x+JcU@GtusW;oKt<-W5RcQnE!eV%zGZ?c++0 zYliNfC_9?;g>x2J``Mb?D5B{O@m9qkBPu7`9C|`I%DKU09*mxyo)*rNOD$^7BaUnpU5>jiY>GL^{u`t2LPK@}6F_RJ=+2sBkf9E&I`&#HHEs9jCx z_$Kg&?{ReMxwgw!tRUTY)dhRg$wRua-JlYx^n>D5llsx6`}m3TJA*cg_fJA=aOmQ5 zKQ`OJFY+iBDO4?jHvsA=eRZFogFG^t##7*?jE;_`I6PXeT6U8K549Dc#;YZtrcfvjwXX-~ zW(JbF#{#q*OFRAglk!F%R!&5WQ$?Me)ov*}yla5FdkpNi^#Hd6}?{j)46e+wX28g>4*;D6EFYV z1aG_h`^Ne)U!_7%M6E`daxJ zwEa+PwxVwuy2XH1$E|h2gLn7z4h<#XAqa7VO}{q;ldMwC%e5a>IVJxjfZ+u$$6#f! z%wQq;5;U_F&@nXvtHuxea8iH-c}K+NKCyaVkI@Ad>mCESZ$Q=0wHpHeEl%m%WkDbI zE!(=MqIznQYOeb6;STGmsD(nLXG379762-O=aI*nH|R5o*vaAzsTu*flNVLfo%-w9 zFFZR=0TcVVDpX_#3ri9}X7B{zMq*rrkAf8jkyMW)Mns4Kz>FMl{E5^mPf)%xzF;ik zCsmzz|Gba*C-Q`b>kIwyIyLLw{_J8W!;g&4R@M+Q9Z!)032;CJo z>n-9-vpmdtmTdOB28P&~i2;%LI5b>@ z++~Y&E7=2X!-Q&o_uZXU!GEN$$FkOuj`2p#e$q7n9A}f*M?Q+P{8rrD7&N^qi4**a zV13V^Cd)4-A&VD#9D84KbfHb)*DKqu9Iqx1ZinnB(x-wo>^5BForRjE^y=)I+;YA! zJ~S59omfCc^vg_*p$;rnT#q{|erA9GT)T0j%IVX$!NbL>YP&RO`fp)Fo_q>+m@o;P zYHv>B5%4dj9^vG?fDjGv+vBlm{rWVj|IwxNkG?7jSfAp1Im3kA+}u1@FLGSBcA`lI zGQv68Z`!2w*j)E17bReG-D9Nx)3QYjJf3b_>fQ{gcpILTk_sMa@Ihaq=agW)+r6lZ z2 zv>-HaXOsA`C)P)y9jsWlQ^=Oq7^{4gQ#0$*nl)=kTm?n@C5O%Y`UM(If9-}1PuU1B z4PEKjKvl@Gr@}lx90g}{4q01i*!4)NUMXT7@!7j#Z+n?&{|nSy;mes|0UUpI-Xm*I zSZIKPn#cRjC`HQb;7917NBj)BW#GqYq0AIU`9MZ`)*m69){sjAruGSb{ra_{2OmJ# z+)yn+aq+hP5sxYG-#$a1)*564q&iLwCm>3X@bfE6%>B4Sddax1AgOCD41GV)fCW~E zPQMJumhHCnW){&GFZAiCMAur6Lc$TQ&Rn|Mzr~t$wg-A*ItYl|0p%pEE8vIh9dhXb zV4?uWPVeH<{FK4PK5M#9*lqnUG;iamz(Qg=H9;BHtlyj%u$e$uNJtH6EsEu<^A<1y z_OD&D#!{`Zy!<>`+FEE0HppYxE0ya{8?+q?nSQAXE#&o`2ieUMLo2qWQeSJ@hGc7j zP*2&gJ8Pg-F-c3z7X3nZ$}X*&r=UG4p_?YD5HBhtACygAey`;sSUxd33P!;!(*)1I z0X_IU7&G4pbVBX*S9QR(h3?6BKn(C*lE+7zlm~#Q0>MZ4fueautKpl$!qQpT=VQAq zM!!?WB!YPt_+BE7>L24NLhh;54b5M)`Si%z*lvV#Y>M-B2_xOx%2WS|Q47;t`lVPs zDa(3Phe2=`kEhLt1S8dOp1KOZ06>S|CC_;RT=H)={u0NrR1 zyxM=CwL%>fii+hi9`niAUPqLoA5V`Ul@PsmZ0@N;RO*1i@yZUbf#)HEaY%MVifI@* z`TZh9%1;3bii!zx!?@*YOXhZ$snE-rW(la{^&#&q!6dZg+9w^93ptE^hU|QvNTgt( zyHW~7%HCHeRMZVnp9sITn;z98krtAFDfxiz@>A{}Q+{=(mf=J!M8-MF&Q2kptbFVz zI<4;Uau6OV)2tvJ&kOgfzC3-`Br$kehs!!Bq`xNo9>i}p_W66p$Z^V$j`8?qKp1q} zyaGFk{7}bM+IhUW)^jNAyg!GfP~_a3Yen0)ZTmKA7dhTn51sg$_1`4t)MuD#5d8*^ z3!0H&x&HFz6zWc2>yld|g^O)Q+C-m20*Q%^K79A@zXRVT;^X@s9C~h%Gt_F)H8rkZ z`pZmc_DbP`lQJ+hC56G~Tfgq3qu$i!A;rWKRhlCzVpGz6nY(&vxtVYufJtR{neGa3 z97=C$rE%Uv!msFovVk0It5)DTKcj!la&ckap*fw=w{Bihrd^6#Gpoa5(J!R1yKME= zg9i^9fuAN9MSrt&wSd0gliFpnPE*6mxc~S+A5GqV;9zN1Ih?PzP(T|rW~DnhdEZbi zS2^h@Nphb~E@e2zGfV%Azi;@R<->8a(7<9a9s@rb-E@mRYOvVKc{eRR83nnhGfdev z-^$zaof={k4_(Q&I_Il|UZ3DUteCvd{&U+mZ@%j9GWWBDX!*g^R^=vDdIbfAiXO;< zr8U_i)?e%Jc@t=>rDp_B9noDZ52WDQ9WLkC#H`%F5P>c&0a#CAhCe1@zT`@i7=2i% zx3@Pkd};!>kWk?A7!5j@c19y1y4F=xY(hbRx^Jy{@1sr)xEFqlj+4wn7ANCW5^6y` zry-7qXt_;HVwqrWfpOmT$=%lA007$p3OqgP0zLah z8d43?Q8GqrgG;WPXsej7dpcD7;X~l;EP9;rFS_1(8u#Mzrn`fEYq{eMK$8;om?PjP z%RYw~H0dYY7V^+Gg6?`LYF@nqVwnxR6|W^2o!a9nB$TeWm}CdFuTy0gf$sdI z`VEZRD&jZYtG=>8{HOXwTGHhrC|J6x(1+mlctHEKji`4_@O2d zAp#KSp|^JvC<)He6Q{UOoM@4K>|1M|9Y~WkfUZ&cYY?&K7ZMU;>bw1CrpBFsToF3D z3=rKPo!cTPPfWO@Qx%c@C;IB7v|Xno2?8PPUHEPaNWMaN%@NwCg=Waq^zPTDu9-)4d1g4l za6?e3uMOCn6LT*+`{}3>Gkj_))e;Vv| z7v7q1qvln&v`VRCm=JbA#?T4HgGHWuGOgb$mjn$AwHIo;UlWQiS035NecdDW>1eg- zWu_SDa?p|vM6SBb>)n!az?zQ7`YKN$yJprK;gl+LqmURYr@CWZ`m01njxS@Ne-|(y z^@g(OrBklvI=?!jg7p{g_l9hKk)X^Aa^8Y~P-vDJe#^v~RIVl11cLX!%2S9nx@fcU z-9gE;^PpNmT2Rfbz+VTD3Y}}}$Enfi{G2)vH{eOnCs{2%8_yO*QK;H;hj+4s^q#O- z9Gg;mtIzSV*DvIW<}}>G{En&rKHc!N79p@p8$e%p z#}7K00~CmpX)`s?)Flfe$7{r!OEnLU<-ouqj|#>{5wROQc1K>=l9YZ2;DbSbAd6=W z3x1)DGzseg$rdno@zZB$H5rd1P^;Tda;I8NT7P|nsE~k6r9Vn^t8+rrQd^o-U2OcwzCyEgf18TR_SyUzLsr!W| z8J`i#>E*>uKcEtQ>0N)zxFa5nm5_*0vAvuxl%B3P62d&7iNx?kd~2UX!1@y^gAyqzt&G6aBtLXz$*)-AEdF!=-a#Rl}`;?URaMq`(R~cOe17y%ylmX-Din zkc=E^iSuK+weh~YNBrE}u>)U8C&+GVs{ZEcU&5xgAS`!)ZB7>*^H!2N^b_iSKVsYZ zOF%{4U;3G*L{=aIf`Vg5kK~^Z>8CdMM-Z4~mQAF;!7s_#W+na=;q2eOeuYu&NSu7Y zWYnvGTB&sg&U>u8D=c?={6gBSl)r?ustBq#-rvAuM)v&qI;_a^Z&N$dEUPnC5ma~B=Jhd+!@Io z5x4fcVs`f2xfo~yzO}E}?COU>DOZl2M1QRe-r}CE+i2+_y8v_?EuuSDu2>ONGCQ|_ zY|~c{>izHbo5aQ+Pdh{Nv?~u4bx5?jJa*-!Yei(F?O4BxiY*3rL;IbfsE2VGtOm~n>q98bt08h5%bg-=y36NUF zro)hE60lUBHrHAF9as#HCv>M&vxz`Ss0|Qdb?v9onpcikWLtHmuu1EgOf6fr@r&3h zT*R}mbTRB@C14x2$V2ZIfe7RZ?+rRbN+6c71CiCOJ+i(^#in}>419P!FeL%J%a?5o z*OUHjLS-lDdsTK+4Lq-C`azmy+IPF6W14LOqmTcfbCC9(wqWDrto(&0k$4TCbpbh; zAR<_)#Do&0*ZiZi$P;bhZfMDNO^awTDo?Cjzx%P?G_!r{HECWn)t(>|?#+YyicxO@ zpk?x}{0!mEI+S=!;`wx);18W{%5`aH4%J(`Ek$A>{(`qj@vBP)>bJh2CZ^|V~4^;m_4oArai7!-pTNzj^wbEMK1 z1oe&7c0$@f7>Lj4#Jf}dtS1)wI8-w5;HwCZ0!Jm6?Afqkbt*3`i3BIHfMa?u}emlXq+3{Fbm3JCr0fpQ|M zYAVc{>xjn36X^1pp8G~-qp_P3Iv-Z2$|7_;u%WNBkNrP2qv{`1Q<*cOCd`?KWCO1% zM-dqZ-AV+FqrSR98rT8)O8!7o)Wh8p@Wewz%99tg_@l1aj_f8K*RHeKQ(kr2!W2)Cp-|+@}83ZE% zF(HpFWaC*jz-kj8)x@Qd5%2n|a}}W)RZXD+o`ry~2!GuPo8A|oY6*i15}EiqP`32i zQXAXbwZY7kDo8=(Pji{I163D|{-d9onS_tjmrH5`@Zb{>Nuoicgv;h7j2^LwpjcGt z|8mG!$95s5Q<;N)fW3RBIBO4;n>bHUC$H~z$Mbjv$5}6S65&=|XZjA^n1?DZ;anb`L!GW26OS&DVvC3mT-~ zLIpTbS|L~60UGw_w~{!p8N^V6q?BgSd0Gq`>WKWSzp)=9;723$E$GfdeD(9-3#O{b zl*$kePy;Tan<3s9Hd1;8Qd{^KPuN7e&JGEpF_3|OO$lX?|LR*hFhv*fOfI9#0?iPO z3PGvW^%f(RW4)L|$B!!^b14E=5r?V^7_cSRJnTHZ8C0_;a3KQ6zl5@K15E(zrx^{a zL%<|yBT3c}e$#c{u_0Ri49b8Ks7PCMVDm^lGY}BAGcmsH2 ztv`(5q(DX@&F0u_Z-2(nGI zWoPWW%DyTnElY+ZxKnx|T+ATz61*ce`ib=jf>Q3eXbgy$n!JYlczAeI#I0MmIBfGL zTa3x_5aTo)H&t_5sJv&8bje4%&i#}k?J(5IZs~JUbu8x<&?X+Y?v@1yAVn7F&dLpb z&z{A?l5hjQrtTY0z?{81P?t3Y+J~64P+=yJJmDE|1Oy9%jg89&%t4_@e_9 zJol!>`SwNO_*6VU(gHts>}RhGis{*#_E9C;y}F-&WRut1X8TiNWLoRO_9UKD+SvFsRigIXGYu>gIA4wN=Ivc z9F|V#Y%VevE;L|gzDCa!hs1J3D=&)(0;Nw5k#gqUlLMKtN-Xb`tj0(S7?) zfzC|N*}01DI}?v*E9*Z}PyfHlCGpV+p;6Z*pjs1M@@0~mVAy_*`-la74X;ow?sNjT z)CBx0&rCU>850qe&7;H)58NJuHxj$taN}Ge(BRNtsVW$#6g)KS5Vg5Cc2&slmHOX6Gd7CL)+ z44mg78HC>*)cb}s`EfM_291D(DL&&y4x;GGxp09g<(CT+H%%RajjVwwMt`6t&%v4% zK3Xcr&(`}79+cL$VKG84PXW0UFY)g#8XP-ykwi91tzmS(2L6miC0uJzjCca|Xz1P; zIy*Zz7T!TWzyroc-&(u6qkq0mJ<&uE`0?g zY%@rmqE3!KgQlh+vZ{G%7Az19ql@}%H<06M)8J`gvoUpQH&of8Ek6_cHRNXXMrK1g zvDxi!ZrER%1q%+d+3dhYz*h+lM*Un(}))7^@=`w(fmqLU&xZ!OTU)h&znW|ZpB~~eVy-V+^psB;d$36jtzOamOWa*CYl>KlgXYAS? zN65ZoAY~D`nN$~S$SU63`(G2&B|=WQT5Ku=05ysXmCVePVPF8d2rOrS0r zBU>==>C0w>XAyrF;Vl3{z~zT3!Nm@lx)h-7A6vx)itqqbys8v4Y=|^ds=(5t@r35300mc$1E8^kEKduBq zz(bv#C_NN*1N+PoW(V`j?SqCGc_KL*ApR`&q7s>}7(0(UhDssp9CV&2{Y0`E}K;|;)c>7nZW_AjC)g8NcMJ*$geX!^P`58qGne%Cp~SGjz#HP3_L}yP&{@cw30%9MNzuh^-I)9J zv)Su@wKr;iG=^D-=Uf>vOa}3=17eYvi5L5zY~TyLTc~w%S{@*#;#xs-O7nkKjMeX=!zKk`O6D*X9Im(TiY{C1rS6^Oz?m&0bn4& z*iXQ;MTXrR`yvBIBTVTIe}bHX0u5mfY7cch0`xFZ;eFn2VuOTo;gW9!gavRB?B5seRuY9=U3336{v`F$~Km#?R zcTs{y^LpkhMy<$rC0y;n&z**aaXsgeXV4?4K)VC{Nz(NL7aP?@%kCS*^31>)AGpDDL&fdROE{?}4XGM@OfXx^Me#v_5eAe+E{v z#enJ!%nhiQeZUZy+v0+61R9iR*xCfM1E#B?@kbsCb5`&jM^N|P-KAw=?M>#D--_Kg ze0v3}i)kb}I?+vh(tqI9=0Lz?hsB#xQ)lj_qXbiY$wP5loN_Z8#@q0-ViVxfKFpEc z>e+XSjKi;JLBe?o|E2vjC_gf?1_#i>6kI&H>Q{1M%8E3E*614niMrjl?)@_IA&Ev`2b8X0gqd)h3z>q8*-AUKe@v8+jT)lUw!x+;_Ta)$tqmB>T z&!6PpZ}q_R`H*xPAnWYySFCu9J^@jWMBDp@DS7Rb&MbLmC*PflUp?%nt$U75+TrkR zE%LdSj)B@Cg+l*rc}a;B6s+!%(|fshyT8Qj3q~-cWi~mZq?C5{p48bf{Bd#>`MTR! z!q!fe`#ZV{g%il9(A}y7>b$-(PsX2vI1~TW%3{X^9&k2kHE~ho27wNShkwWD3WntU zS?6&x|B0r(Qy9Nyi}YT})^RtuR44hcsAwzGn!sf4-_LexpUQIk(vjr-|2&=-MgT_LFp>lnP{XqooE`^VxBOen(+ zL!;4_{*>;d_{sRgZa=N;$z#Be=Dww2;&bq-DT0oLN$V7_k}MET&RCTQKXJE?Vy7>i z34j_weB+o3RYdV8pvw??wu+vvt}ezT*(k&)1xbg6hlfYKrP>0%q;j2Pl9dH!MNM0d zqgTuS_4ScTQW(Zi$*`bV?vTKWl`D^cVliz=tHJb*nZ?oHcd>o_hlh{p|2b6SZ8-3& zG$#6du}Py9;Gh9j-{Xe0Y{BI?`rt4{<*yX9t1)O18HBAO2$CdqgETp$ zQW5n9u-Mj2@OMr-`PZ5F)xOsOw=L`sbvV)oJ#@|^tJ={_*eMX|1pfd?3Y4GC=j}N- zb;RJSq@C3ToV`H!1~zsKC#+YX)PhGN2O5w;D*}BEM0&54m6KCR*7+FUk14fiVk*Vh zGcgYs1tE^W)qscQ%^ku}pX!ok4_WgFxIjF%6L|j;Ym3{tgyB9|jZCq+rOsk#-{CCy z?8mN1(AKl0FdS}x9=ip`Q~0wm-Ba3XdU%($K8xRd(imn%I zFb;(S6Dk#u&@jQeWCi(>e1_033_&+BEla7z*QbNt%(c}5Q|&(c&uwqUZT~L9baaPb zv^kGmKto{&)SUz6CElM9fR_uLv&&oh>t1`e-TbzJ^a6xuCyptKiel9HKE>?wwOAH% zPDW0=%kP^Kl8pb9`k&NEH~a?XRKgOSe|r>;N06{7*hkCt=(=FmN~c+Pdx& zKvfMstWs93DqvoS$U6_Bf}m!w9SQeXT-8uURJw-q#^m_T+ceR%#TCp?Y6RO75)-_) zlGp6-`=Fb>zr#VmEctgfTN^>FcI55}<_EW0-qfT@q5{2QL$EqNJ!9RKuP7W^; z)PvDQWk9QXjP{vO2f&kqNR4RHa;lu8`m-xWEx~!n{$#>3cQ}V&V6w$B%-a(I%5Gp! zf~5N3Y5LFg1d{8~{|C;jsNHr^^d$kJP+^`S+zJ*KoM3a5BODVbag#(oqB4(vK0^lE z+FYit+iGJ^%BLj)Tc+9C8w96uX9pf0yf<%m0IK=nfwSDCo0rz*69u z9io5}cvsnro&Z0$N?8A?hICmJF!7xh)1(t2XQ}uex9E5qTyx;}*Ljz4PX7MYEoU7x zdA<<<(4;s07t`M|__waoeOd*%krY;nFBpgK-8f%`{M_8QHG&N;2X%~=G!YgQP(Kb$ z6^S4;K}692pL`mw!E$99iXcw+Xck91)brK;_inzWK`9=ThaHfpCc#d>7q>$gxb&~N z4s@&yiwjc{V<7Dp!BhdNiUf z%O`X+VhNDZt$l_{Jb-2JV&*t&EL}$S<|ktsAKR69c4;b9 zCbQnNz4)XhC#mF&V^1Borh);Egu)z6(m(wuSIo97EkprIHJLyE=T>f&9HmQsg%gLs zo}Gfmu54m5=s89C@W9NLO}g0!1JZ5_(^9b2G-{aVAD|E#=qOrA`IFC(xoAuc z0X&?JcEr>cJp8_>e*MMrCyvd9;~bYUZG##0&FGogVFw2fJttRj7hEZEzWZ9aIsMKv9hY0LRE&ME ztn4EPvO1H@>_F#xH)ez`Q zZp*KKA-MvD7%?WNFk}7;GH{?RfJ~#T8vXGbkHO_tNEM&(wSg-tA5SP24momG2?i4Y zLrc4*i+~Nt=_}@i^BIH}L+cX_6Bumy%0#yXmdK+@e+u9=OHzfFNjwLYzms=IoGtLyltdL$KJu`Cx=F86el94Xu zo(aoW;6#)`>L`LR1R`m8d1e)`^5OFmd{eq8Pvk?OmXQ+=fMtxJ964louHCQyI7hCV zc?t=D90exnvHK-Ngnm!Kw$Dgkq2V)o9KQQ-Bw=M@5X4}H?p{v2LB!F+2M_-QU^s4N=j0 zs>7Oq$Otqi69Sqlz_R41F<(6S%j8TUaP~oj0EY$RQ+0K)A|kv*ez?5uez8`o0lh zLbKC|yDYeR6O&L!ZO&5KyOnOlZyZ!AFW3*!Wo9^cL8h)ori-CrZ_eV2^un)(7k8rqc6R|HYthue%N61p|Zum zQ)Lu*0({trJFQY|=^d>yue5@m9GWP``6Dj-u3I{VI&#GsBU{NJVZwEQld60;%AvU? zKp9pAwOyJr2-&`3^ci7oL6rRZbu(aM%XAa4tIe+;*C@46&|3c~j%%Oe)^p;yzMh?C zLDeb?q~Xx650MiKK-%47NFiBiW=pl4vw-(dS{s_e3SHJ9|ZU5A^WpPzcQ9%Ou_VL9!5onH!=mFuor64Nipv9300OcQ7y zX-h+b2Et+9!_wgWRV^&u{BwE?PPIEv zN-8XCU82tSmcH<2!TXoj047gZ4&Io&8%m@IN-db=RE{yY4OqQJUtw6S4;p;h z8^B`uid_l`Irc_lGwF`DKXi|1C#L|U?4T_SYc@4!S?Yow>BCVUb-;U-e<{Lgr~+>v zIm`@AQa^0hO$gz72>Rr^$YCw0lM$e>HL;!+4I2Mo!)=J=2-2FtG*?en1COr^hz%Tr zcH#h=oQ?6(A8Qn#IgrsFjHvp=k$@p-!&)7%9cPMtX6$8NTIo*@wuHIFc(=J;hzsCs zYz->>bZq@o4&}>{<>$I%3qVWGLv1AV1d1-${=F0e;Sk45=dqv_bi|PRa8= z^vU~ib8oG8_^E9fO`2mkI-0}&n{)jg&S;_-aSrCczajsBtBc{k|I=UShydLHSVVLq z&LOFD=h%-NDeD`B*#~~^D=`S6O7QMS= zT}$4c<~> z3jUJpTt>Mu^qHX1XtdvGmCpMt-r3oOBVU((vfihxC>%&$l^cCwzCR~bfs&r!7QAx2 zH^OjRVGY_6;ni!9BANH2p{P@q}hcyfU(|{DO!Eg#u<)R@Ku3f#FP?i(#*Kzlvk_h+X$X5KE zbS0LUbi+F0CicY#l0%t*O8a3PI;MM&(9UdeLK~9zS9| zC$%l%&5!*j>?b`Y2WQUnxrMy)y5iJdx@#d^UE)H`&v2>n@9P$Q*q(AQkz0(*Atj$k zqUgcNkxS$NCuA8R6E`GC(!Aps8yn;oa+*~wY{nR`7a-O^$NretAQzc|cx8e8RIxJ? z{*(_|X5Nfb=~Ms}i5G#m9^v452E(V8L)Q*+yergl>zqWA6Tr}3f<2EM#zguQa-^h6 zqGk->B|8QU`f`>Qgl_Z^&arYk7dOj(b!vhp@VQ4%iaJE7}j*B4*w4!Z-Rz5M8k-7Q<6mVMwWN?!>P*_~XeL#Cw zYEJyt+}?(g!*2-*imFTeVGSuj$%oOH{xA041FFh%>l!5|Cz@lTrdVPD+eVRU1EgaN z>INy&rARC&QbeiJIWbX#5pYY9UIb|Z(nTqzC>@k4MNnyiVgW(v|6K0|#hje`op0Pb z#()2Rd<=&o?EUWdeab3x&b1!E3W(-|@W34#)V5zO-h4o*d9XcgA@q_n?`mc7SIf%U{G$qNwi(N?bP{zNNyUf(y8uCy0y) zC9fHjMTkegfD?n^Ur_>QO%;DCqn|VLa(pwzsk=WHME@`Er({%$ye>__Q!V|^dz!RX z6b_tzZjCk|j?b@*oDWV^J9Se##;o#Vl9W*Ivry4V;5NHBq&#fTPB~%`^F2tTt5+Q6 zBQ@+S_0RBwlM7|7M+J4ykmV;m_@sD>BanxZFGk2#GssvXo{rf*vi)bFD^CPI@&vv1 zw80t>M1`)hvZMZ~+yP?kT4nNqy`$$i?j!G`kozSDCVli+-+`0GObK2xJgB`4g{Nu}h#5n4E*Js?` z@Qbj|lP6D>r8I=2eOZ(1I4VD&LRrZLj9R=X9z%e!9ZKQ4P$^p=hl*4^3U-6k1z{;? z^e-Iw!)qGX=8HXU79X5*EOq}1?|L`674<(2ylTojcAiyI)@-O*xueoF4(;AJG0=@l z>$HNEitTGViz_SF2wjGpKRAZ)#>82_+`gBf*>eXVe zbw%$!Hz#k$U78WI*2v3x3#9emN&c3zGfm=xzDAZ+?&A#o?v>X5Zw zPHheQ%={G&dz=68>Rr9#@l6E9Z-$>e< z$x7H?v&dkB%jwgyt%enbSz5!KM}lba1NJWLY4$nzG=kP4a3v)uNo{1FU*As165vvb z`*~$g-W6^`%Agu1O>Jv+ftGhbkpgUd={D!sgZJ(>Y^yUrcTNbUMK3MrXaVeOMC_>0Q_m@$yWzR8m^4l z0YBOD)$0++NW|Sn?A#y|Mc+s*1gnEBwpBgQwCO4;-co(RHEy0l=wqcx`Yy!agHX;2 zL25@V=j;`(k#XV9rN1nMz-bqzz8WrK5P;ZMLj4c*FWkN+9{0^LCv0qvrP>llp`T~O zJs~D(ydUvg7e8J(OIB;2N6!2Y=uFhHxnCQ9GAx_$#yfq#lcu#hqA3OHj5m8epSHGk z?cBP~XYxnOR<`hCQQgD#&B{7$KORo9Up(P>K#=%_3r=9fdnUcNFDLX0*!x1 z_#fL3%mZoH zZ(4}@bA0{Q4f09bty1qDv|Dodqs@Oqcf@_vu);Q#xuW8>pBJw3$X$27ee1o zTx>h2G@QDVx%SE;U<=f76ls;CBfq+Y-O)NVVB#`zIL85NisULo* zMWsq%lY5VubB2bF-*&doHiE zHX-o<41n856OsZkJrFKY=XKm?D6S&R03|Y1@I@MQI%9;sG*)gnWmF(Thp!>1#p$94}O zy{-mjNh5_05XMx3#zIEHG>QitH`rv63bpeI0@roW7CSI+Y8pDkkiQav#_B;Fw^a7z zHDm`Y)M25_JtQp-D3%4}>L^PJg``Z&UdE6LjB}qW$nKm$jPm%JY&`XzwKVgtuec;J zPR+0{Uf#3}*CaR#EEqL?lB}Ov#2toMqg?|80|VMa#Y==8zPb~UuWvu zyd{!l6U_E0P6g`i4Myko?b|(o^Yl>q!_Nk`r;CXTOM%>pw`%~z(Su{TznG#TuiVs{ zo4ok8QA3=nB<+L){>%M;MVbXYI}S2{b{u5@H5I*l%#C^$9hmi(jO7bEV|3Y-j0jw> zb%(A^ktHG`kgHM4Uw^}W$ebU_UqykzZx+#`e*VKyMc6eMJ9ZxB(hNMZy#LT__!;tI zaOA{xqmq)_6L2Gq$?iBC~<(=`B~7;WRl&=HcO^$*P@HPJ`9G3#pnWvS+;<4jEi z79Shz<7ip#?P$C3RQ_|61TjsK}SH`yg9W?4D~_(@Uu~&m~MC&1IvK>9ZnJ4$CTyW zyh}<0%p+MDw7sJO$wA~)@i=&Z%GtsU6hI5OhC>j7G`Cqe5aM`S-kqC#Seg6>?e-&M z(;Dp5^SIrOUWUm9#x`DkpYjx2gjyJ#>ihKM+JfekUmbpodjgV;a0ifnMaTmbc0)E)mC84*@F55C#Nq9xNo_>r+j56^`zWIO7!m6 z(qNEo4!dHhkz&d+pgHZ}8CC+yS0wLl=Z;Xz9E6YX3hOE7l5>9T3xo;%su!$SVI8mp zYI8K{etKv6=I5P`l%mbg2`&62oJFcGECo&s?y^z%qOWfx7Pm47&DT9n<#VS}(S0%M zJjrKKb_bymm3;z8NkYaBXOo;1hwj*;%nV!Xk}0T@NaSxyAoZmZWuOhk{S$PSY1^�L@VmR^G#w z^bZXcUO!&%jE=VnPg_4cvW?B!m4FpRW|EqmqY_mO=78FTl5z^o-|Ns@3C`yp^@6{3 z>y}X#q`8!3nNSPiEfBrZ!A1ySsh+70T@cW!Uvb~QPuw?gquHtFxz#zx^#>|A0;v#% za{)Tc$ALIwG7lN1piYDea|zk(NaT_W)1x3&l&LiRJ@fDTplIsH|MP?Y3j^N&zs+p` zIKUap@Y6r zhFrOA=6DZ9@yMAi2c4naz2W#G1p;Ewf>5&p>`Vf-Llkz1GilzCvKlTtA;A6ahmyX3 zXre*j1r|x0dk3lFT3&u10W&1^MS~v1{3;Zg;4FrgdpgX!cv^ltyPbP03w9q<_az*d zMY>|$R7kM2fYlH-ju_}Nsq)a^1?U4ZCC8n5Q}_Sb2lwBLqU<@JeMVghh7BN}l(4+N z5;bq=LfPn=4@pBms_W4F;op5angCFWUuQv8b1oerg6Ht;%m!6J7uUr&f9+1rP;Xdy5!7tBnXHDQzOZdvVbW;@! zEJ88pkF@AG3sR1N!y{Zy*A!hSl2KIRFKXPRn$Ml$TghHqxxv5UuN^~wPrzGCkw{`p z^e}ANFzaDxR!X`-#79?4D?=pJIkjkk;b=Eihmo`4u?F{EKDpTZbqrDeNP-e@_z#8_ z;|*4~26;?|Ul__%SwzH0PP^*sx8L79)fTZ){}e{$Z;*vk6Fgx0XpRE$0Ww8S3`9dM z&qTQod+XybrXTxcztRudr;#l+8>8WT5*BFdA5(hMTUzg~zqQ5vPx{(op<(k`3CI)B zmQarpST!{V{cOTU6ZE*x^K#1{QQYt5c)MO$Nd@gpe+IZYI9s$5MWr}Upq3xQxohXv z#rHRbxu70gR2hppR?hx!)RNmRrvw@+KYl;iWe<2yqTQq&vREn`p*9Slf#92|j_AyM z?#HiOH)BmlY@>ofFbBcX_K94@FZWfqw!eGYOSEm_<57LIOM_eM!sGPeJR?cMCUkV< z#xXp6-XE+2DiA>neX0$8$E%+FaH5GBK~zMVg$88v52g!oP71tV53rD5AV19>nk#6Z9T1tt5WS<#l3ymkKT_8Ha-VZlsHfg5WDWdKlNyF#2b*KFZ}l_ z+?V?v99}hVA&yFfDDQ~jF$6#)Sy5@`3SiS@^$7u#27=FNd2Zu@RucE8Rk58z!Z8T%dr+-MWzGc?BNm|OG!YX{mrwD??sh-7 zouuf=d>UIew%@!2wke*y*TLR)Q^WT2_2T2{M6*LUMG*pA+5cu?K zqXyn(%Y@F|=JXxJ)QY<9S@y|v%dr-R=w)Mns&*>Pa6z3^-cx{!!^u{?p z04I=$m>V6BsC9Tt{TaXxW)RCTOfNb>4h(`XcEgzshY!>uJCDQh8rPGiLiy_-VUGP9 zK_F2pGom1HBr8n_pWEv_q)K=&HJ2|1juqZ+r894EGSibOS{E>aYt86gs9(;Mp&4L2 zDf~H^C)#)5S)U)ahhrZ+dBMq z>*50FN!0vhPCWkOyxkJM=))%s!fPo}_p~@sIg4j>>yFiNSanp*O)9|UN9P&!ocj!t9f;~NH)=n6K_ zQLG?LKGU2tFq;b{YkYyx6+{ix@<)fiIXy*lo8D}#*zH>?o9u+`W}~z7j$hAB#}~Sg z4^x>kc|9b8UTC|ax|3_RzSndAd3zQP7lp_s$GHO7Gn%B+VS&(&NmjsV(mJ3c?MR;0 zEIpU-igvf1-+k9n(EjopkJ~ONX?SMB!*hL6wamTG-H>u?) zTNC4pbT7-OuDx`94fmscZHMLN@v|w{5;ymwW1wL5sK|uQF}!*66P{a>2VYeFo}5@V z@v896E6?(?dmn#uNo&qLDY?Ub@d;;jeM~$5?iFMj6mKRGZzd-{Z|jyzQdOp>Os^y? z{7v#q=_-{iHJV>2ZTu+cLeJH<6%$*1{#f}?{zzEHo2T~1qc2|z|Xf6w2K+=d*YYC@EY0p)=NzTG6O{3!*A4q0OcYNs?(B-IV zU%aMvZG!{;8SKdI4ljc)ox=jyAR@ICH|9B1HAl1TMAOl@&ggTZ%A8-8iVGq(vre-W z*}F;ZXMSe3%v+k>dXfdaLa}q%tz$M1(4*@9()~1%wOi(DPClyDQm;T|_$5~E!yi7g zDP14ux13QNWIe`Di*W#Huu#hN)c{FdZGYF+($boE9j9Q5wK&`<<`Kk_dVak^#sl9Cl*#t5dad?XHjNz>2_nglhC)3A60R>? z+RA9C#|Be9DlAg-15ocEM|dX=iW>NJzWWA@ZW#_l+~W|BH}Q!7E;7Ukb!a+TQLzV% z`zv$toyIs|B0*aaf0m%CQgc+8x%@3dYF2JC(I04ACX&;U5Eour2)seq==uiOg;KO! z2(w7Z-C;}7m!f~&_#90H#^p4E9ThsEm9XHUs96wNj;3~Jf5A}rcjp|EraPc7zqsei zVz5R7oF^cWOtBjSvZv>COl8X0ZG@62SHcAu(6*ZG0=6_Z=lWnX;<^gRG| zRVl|UV>L?+3; zGq>R~lS)KJw%Divyi9l{zMJ_sQr^FHW#l_b7#qsLofzxD3|q4?#8%^HyG6ztfE*~4wqt zir&T99lCWR){D{e`~2+-n3zc4czQTq&q_dpgBZ;OKVBvr5?6bFclJPO0(qGz+Fo($ z$KrnP0TxA-^=_Z6$23ONyfDAuv@TTgx>6T6Rp5zq%b1eY(T!y^mCw;cXjTagQV0UZ zm16m?HM`1#E=CIk$Yg6KkfE{Y{FD|){3lbl7OT2t*LZ(mVL4Fuw(M`R4w=Mi9<6wZ zm+-`YIrW-2+&*m-sb|-I-{dhd_Bo&IR(Rt|1ey`3o9TjyDn&Fn0KP^ZU|9v!$bl$@ zr1K<9&(P9Ql({^;bgX@EbZ}JYD$7s&@g*(Di!A7aT&ea40UkfQU}7+}8yr*Rsx!+u zSbwWfsjL{ZiY)EjdDCm;bo$5N$T(PH#tERTL zHZK|R)_Ski9;3}|0ZT_g$Q^+?2OV;^{mUDzeVc5`L^4ovS6^^Jf@?}3tG9!gv- zaeDLcqxl|}lw&y&eSb&1mu)2vyCNbzydz z6BBO=yV2dvWuRdDe7P;_LXUsDB#IX8UGM2h!Nw)?O{SsDauRDQ6c%6|DkQ1~&~ot# zSh7ZlE;CSw>i-2l^Vwn31D(Ik+%&t@`KjwbNSm+>BRxa!C}tM=%mv&?&*j@zO08q3Oo+IC`)9^-Y6At^eqrd+dS}U6kj0C%PdRWSyWm zY|CJal!Q_v`b8}tV&8d zsL(z18`;(zxGlYep9k;jV*FGUOV=f7|h+quHcE*uep z>78FzD9BTgLV=IAp#O4@u~+=xkzZ1DxiV!VmX^(Xh&uf$NTqDy0lLu{w;WYxmsBMh zS<9?4_izpYsdEA)aghb*jnnz0(3-aFbGlhihh6X8hd&D&%F5yX{xfvHK@wI5;Hocy z3l0>D73{g=%)=yDgZ;DaqUe)Q|Pn(Ti6R9AlV88UK*W$J=7v4YGztHh;sApBY0 zK;`h0oeIIIUiDx8!JFNVZpND_)a#BsHLrBrY+h{#1|qE8Fy-R!X*kfJa=%1kRg+f^ z;t1jnoJd>onOL4LRD@k%q^(+Wa3XUau0tF1`%`n3f1wg+&!H*r=$`rWtOT4|#e^M$AlZ%l<+qm|9!5Wv;&5B_xA1Eu_!$< zHSL%;%$e~9kcFBF+UYn(uAuKeXlyc*LDKf-Lf_FARTP;KpP>h&Bp7dX01J9Y$AGxj z+`oVSw>`|Zu$ZweD9=IW76kIXutLTEa9o?=#&4PT z{B!C(5b?|xeDfF-LiO?uKp<(sQ4 zW8>-R;>D+es>=wx9-zut3TKmU27;j@zhggH2+uku%;nMWP)4kTb-@JUK$+DzmN?vZ zFEX2&`})piJSsi5BgP_|Ui>&PE2o%xvix|FIzZFRvJnS6vv(yg|e&Yu+Fpp#%Lk zM47Sh4AyZzMaKtLqhn&;3%3_SGUTjz%=HbgZb#jmDHe=V`+3Q99P1m%OR1frs$l-M zUdq`a1`4G3BJ&%xtO&;96WzJ|#CO?fWp0OqK9KG?NTBP7>*KE@ip6j@j;`K7uT9~- zd)f*;fW`zM)FLQ7Q&QD``ZH;0>l zqNpex7N-T```Fue0m6K~OhP!P#YQC@4WGA&i^rl-PSBR4n(l*1CY|6-D?ggOoQze< zy)m)8m){aHKrxLID&r7pA-|1)$=|YO$1lYm;|(62nD5m7BnpZiCO3V{+nbDoog=JW z+m|^t+-&s+ED$qwGrG7Lq|1lUPL_kJw{Dcci}0TYreq4aM#X&hm#0XUAB3ZSsqH&< zaQCHy-fT__Ef-j#db8)J=1B{Dy>@r;tYwzIZR5`!K?JLz1APc;>BV6qs-P+gU*o+s zx=D;#);VUM*~jNdqgOR53qp-tU)^?kbFJ1Uy*5r1FCtCygyxj?DzGp#ZUZ|9_g z-coDzj7}(VI^fHU`(`EZck5R#AJmN)e|BWxytV5Zp!rpBvhhq18TpC7F}nMKH0lIW zq6(c!h=m&6NMiD2xfH#db^#%gB-&bl!fhygA%9{YnQX`;V$=n5_0n!1e5>uWk(S zmGr`tP_O15csPms-=9j~zIO-UD{rK7dHtnAWcmvb$E4Q30<2^1-U@%e(A(u{K@J~f zk&1)35$W;QHJdS$uCLCSlw?Dc}|OiTjBsXfxLaV>x7_Du(eT zh=-e6=4hgf{s0QUD_JF`fyM#Q;iPn&olyS|hXIG&yK*`(bXAc56jZ0FJeEP%BU?1b z%(gS5i4adVO+1(M2u`FN3vJ-kim}LN8UdStJ9RC*ESlTFf#I^u>ON@^WrcH>g_ggm zce6@nQ_LHwn_0(=%0-eenLcjM4oV>*vIU^gf!i%Z=djXw>*K;43`B1{iDs2Rg0%?~ zA|8AM39F^M{L$)|KabT?WZGWQ`j6VLW>lK<-8>XhcbV?7HMiP2EeF9TKbr)EWaozK za|-;eV9H~K*6RrRV;kR4dMq_qtF;;Rw5*a(7906$sY#ih?{0l*cnMS6K7o>V3McW8><9h+>Ezb&ngYgenfVSe2JB?vurFvp6V#iP(N2vkl5U$k-fG)5}0TsHg-7;+@Fp~YqnKntb z9N#{K@R$Y!Zl(<6TnLMzSNdUZ-b#E7PryCwT%pZ29mU9{(w}XhpP8#|Z~Vy53deNh z!X92^&7C_}3Z>^Y*h~xMAqmSe7?6~Z=<=3%z6-W}6J{4aT)qc1A7O(@D*+l-02$))6Y-b5OZV*L9n5Zr^dKB}BLn5-(9(!0`UieM&feoa zH5}=1^nEK$^MF}Q($3uB;s59tx(6FS8XIXStVEiUX;8G>vE@WAqE`cf)pQYa2L8Ae z{9!CLnecjH%sg}&8bjaWeVwmN#vU)Wh3U%qo&ktzSI>x{5mMUXd^4O`AMVtgW-?r= zXLEa@UJT6BUu2f{7-&=Y4+264a9ror<1JY9p}pd`_HQ#jgCYVZ zAfn_6t6BlAk@XLk1HcX@eCGW5^SS7`EH-Lexm_>ev{Dq&)4Q$1mNr9pZujpc+ui+-+y?T2g$}OEo(5C0om}%IeblyW4Cz=HccLivw70=DcQmN$* z*yOTBhL(H&gbHXQK^|>=Q&1kEpn!x%bSh`K^ghw-J-f!c7%7u2ZZnX{W_V@-`Qq?Y zl+jSmG(Zg8pJ@k-CGDYZC`Ze&A%jp?CLb|U8<&pAJr=Z-klZmKDai{~;jVeIrqOm# z1opLicE)NZu-JHfk(?He;jvuP*dlGgg;OBmx%c;Ij*QewlmX%tRc1W6>ZJ`)&%WUC zl&=2}@6)%PTAoCTy!$fn`gfL$3|3wpKvus$L-;9Ntng^(6S_?N!s%sC5I$T~`TjG! z0U(mZRIJ1pS}nKvPdo$x`B2&(MmvsDR6(R8rtMy{VtAMPNKM2Q$mWMT)+q#`2&Efi zYl~)b{s6_rfSZWQ9HLKbT1DF?USXP&3NKvD({vG7{B=~uZzkUY*X0BR+6EBBS~lvP zgV4BqW8rw8^#FpMZsH@sb5S!mEftyvm!wQ6UQM-NsNlP|-e@nA@v9SKN^829l0J)KH*1<7%*8BV>7ql|m^@96Ylr zV@Va~PI)MGfcV#dFA^(5*jr;14VA|4@7C5_O#(?tfB-!9FYMAFg>cB5Y*2-eM@%#h zkWg-1MYpJNYWgbJ*3Ur9u{r3(z))uVfkqtCCA@H|av=vQ@`k7xOh(S(OLPO7?+gdF zEwVh(!56syZrHUFS}wm)`1{#T#sDYJutQ=WWyss;lwX7ca8;_#l z6smo*>DiojXc*56zkg@rJ_zH)19)$Fxxxfhf-ftZM_y^hhgzPhbu1VN!^h$)a^94; z%0}+tSSi)r0cHsS8bKgHzq>R4B0z(PgVGNjJ`cc$1YsTf^(IP>N#IsuAMO4Bnm=Zfbo zewMB!61GhA=GEagBD;YuP$Dji?v+l!2W^n+QB+R8c3+(>J#%dmA(CQ1f69r_ff4I?@WS1Zcon@+xbn0V(k7>sVukLShrhECFYFzs?G0?qUX85Hu~K%RJh2Gk$y5p&&D=xns@Xv}gQ%`+^5JZSOV}B@R`R%Y z9^I7W`Y7G5!ilGa!4ZYI0yXI^skZHXly())?7ZqrcGYD6x|f?a_) zDgtpv!EhTG9~GKsBzN`mVjB%Cl69JLX+fAR!FzEAYIjwYP--3rMk2%Mb|UTl*}Qfd zB^T&5wWoEheMi*6NKJ2EGYk883fa&Gpvv4?p_IB*Hw+ek0NkxGM| z$atnve_u;o&L6$LJ5hG14$MuW$?UQ#B438A$H|y!&tBrgcy>LlkD|!{5W5Bp8hXIM zxRUv%93AkuT~-&;hNLaW`a4RJc#jySS)~RXJ#JAYwO4Bz&oQj|O5x(*+kq!dDf67J za{ur_di8r1D=7&EhKNUU|M1FHn?`YFSpu;D6jP=HdO8e5$q5IAs+{c)3xTyH1eUr) z&=?^nWMFxjzD%ux;R@>jUMh}LtN<1ekDjB4a8E&UE)M&EKawztR(=LB1Bo_s1~YI> zLH33~;zlSeA)yZ$XnrSY9RWCX-d6M!#lpZ+X7>Z&J&eq=1T{Ofzx@{Wn>QXfCt~yK zCyP5kzMn*uR0w6Z2tA-8vSfq_@Kj}!V^xzeM6GrDY3Vw(I=|KZKKf~fK}VW2^T)X( zo%_v)|Gi*tR%RdnCa+tL-nR+f`x($w1~I`wE8>Xt9t!dnvYs#8tEtla;N}|{?<`^U z{MJovgO>mCsN=cafvl`!6VfW2HBK-rItpEaDUhU~;1~!5iaiM976o;(>09G9XsmNm zO>VH(5)2FFpqv<>ER7I;4&XpQMgzkk>K1X0z-DbsgJ*C(U#|w?iC{egqG%xhjpb!r41%a7di zISi(1iJY??Svno*Y`%hpm!6DVUG#2w-h#_X;FhTbXk_PwgBNraW%ZGx?8VSMY94v@=8?HDq{GSL}j1F$vFAL!>GY9 zIl#gdc(Ri7Yoh{c&jP4u?d>K)E?x;jDYU%UiggOH0pOcWD0;R8#nl4?H84aQvA0eL zXOSK4+rVu7?$VU)c*9#Oy-!x)$$YnyN62nQ>9l4M4E=1XD*+kR?J;Aq*@|FpRSVK|q^Ofx{{xT$u=&^XFC0I09Z-%Z6H8 ze{d_kGmq#C%R>B3PFiBp<;mTsTs)uN(aLCnO<1zTfv_o6B32B$kJ4f;z6a>q79eB( z0Mnl1)*LJY9N=_N9x}I1t%worx-MEC5Bo5@{&|gduY1%zq|rhC5FseU)v0X%z51M? zi2k*eMBUmxODj5zbH(_TZa9NtAtuH-KPtq}eg4e`<}EdlLU2Kz z_|LKoxEhk%P2Zi;?(gbVCPsG78%MM*!0~QJ(*`EH`(EkRQ4j{;vh%qMQDFoESDd0@ zkL<@g8z6{A39FTezWnBqUf z=WO=#hq*VkL^mO@CyR88TBb ztcNm`>!1)eBf_CXX86`@d0&i5EPDHt{Q}__z?6d^ok$Y!SJDBn)K17a(*e9Nd9<4R za0fh_Q|>{uk_lxE!?oNG{t)TX0!~v1-5cU6@;)(9`#T?5X4UDEJ%*{s)%7C4$@?geNG~-%Kb0nu763rsMv|#i;yuqYYt*nc-2#>jKq%XVlF~}p-7rjq3~4l#&k~dD zIRyPC0=EyLl*lZA(zO5?LVzVol4ch=kiamCYk-q;)XG(G#Fs=wz<3-6#{cKM6)Kk(NZ?BeL101or4IXYf9yVlCo10m zvSJwa7w1Kii!6Qns(rKj*js{O*h~`Y(wC)&cVD7OrG!-3n@%P298#Xv6u?#k*Jd$# zGO+q$IH{ooWqxF?J77nSVEKA<0@B{i$%C1y=?c7g#-oi!r(1WC7kff(%da zPcV-FJR~x(VUNidlj4h`Num;45S9ZAXiCT#2=L7~hv7BUZGXBk+Q%=P8*ZjUd4W7b z#v1chiIOLn`mNiaeCfJ=}u!zl?

__DN6Bmz;bcAp}r|wFuop;Q5n{01nIAcvmn%PY$&I&g1l$6KspIu;jaWg ze2_j1C~n5%;Ep;FGh4Wm6EHnPyHauPl;X<`yV4w5XNuHqSD|TTxOuDI0ws=Qpu&o+ z$G?ZhEcNr{Td!T!U{+Gm)Jh_NoI+VX2=w6N`G*U?t!2b-03es(ZL}i*2d9@$#d1?` zqj~m`A2H)l8ZcbEQ@(nvFAtNE-SMwh5}v~p$2>LyoW)RoAZ6!aEV(rJ_cG?h0#E74 z_b5A!8}k(zn;A4K0TRP=CS$A5EVtQ3c?mEil2`ZO1W5Vu9`$ZSZt8m$n0}~3Hi0MG z$H^P%O&YF^llOsWh~sE~=zV3ehVWR4+`uc9zQ_U*${#8*;$i~KjLqnH_J^%)}u<$x1asbSHq zApycX*_XFW@3=lSY%8^f9E7Ylfv){va5m7Y{>MEAG2Aze@d0RlOSg(B54ehDu2KR& z<#R?3ji$)w$leR>pa?=yWE%EDtS*OWiQFhLN0?!AqnUhI771t3x0ti(+WB9jqNoE}+#3IgY@_1N3}N zYo?zMB^qqH+fKr$bPcWiu{VJOle~fnZty+U1F?Y=w~(YG{db1-3wx))OkJ7v2P%>- zeLP=w13I*np|Mghc+NI9hmzn%m~4t8IcsPeM=qMe^#UboR%U&N62FB}bVFt*rVL<| z>U85%dNse<3`$rxsr7(0>~dnMEFusLR?1&!sR<-9MUJEC&wx4cs@&@e-%FgUc_hqU z*CL$pm(pDA-L0>t08^>ymwljT{ea6uEM@>DKZ$O^oQP#uQwobRDH4o4Lx@NMTnb#j zie?@(e;RWJD|#H1a`%s4s}+-rN9l@amNf!#C$PzIW1J4~dyIquhL$Qvz7(Yq>GPXL z-(lMsO@j@jzam-VG*KjY!0tc~5ZQ>)mhNhlj<vV#2k}Y7i(cVK9VJbFWrT#ok8>iO?;u7RK0pMuJ1gcoAfEVC& z*hY*R4P>JjY!kabr*J%Z@NdjBuYE@`E{maZl+&kUf4{V0|F=+uXIZ9~tR*WpFa(oE z`B(|MAsNW*WatO4ngEf7=1-EP}2fd$!z#9(>r2Z!qFm`0k+;1tt22qoe#i} z%|T7qT6T49tsi}ZB?6es$ov`IK1IS*fzP*&KCX|;=PJ5EUStjv2BQKe>5M9OKQeq! z_O7Tw;I`bRc$La|I#A&vi-`NOzZlt|D%Z7C;Y9%lz&jQtQ)Hc)ugnEA^{dq@X%{nJ zwZhzA+*SUMbeiegX|Cg?u&$Z64@P5T_ku(pD3r19^&=n+g7#^3B~xYyaYZ;*kl#1- z6pDR!SNb2K3HP{_7`Hvq{of!VQUtLIHd* z;Y>($cq2aMiFX^fe$uvjzg2@SqYk!XT zf{}<|%Wy;BNC`7v<^Cz0+bnH(h0C+FXXwC0L=h(jxHgqK0=)YtyxGUY^hjvlu?7kl z%QP~}JOWTZJ_rPDDVmWFePu4ZF?8FPsPdAWAZ0|f2l&4uo?XLmY3aPkxpE|@83U1mu4u!m8>6C+PKehfr9PJ?sRlU-lVeFixBKns1{NovbR!pzhif(fe>r$u16s zi&j}oUSVcmrXxfxrl@4>XTY$p)87W;;BDz z{5I=->a?tL{fc^hnLn>?UXdQ#oql7>uB^zJ5=oUI6l`yvDr9VRcp@s5E`0 zouWyX^E*!@-3i#M^f2sBb8G*7-I9a$#RmtAP4PGCgR`EK3=m0f5Gxnh3I*%_OdT)n zn7SW8)7OpVp>B~;0U{?hh~=_w=D*`yyCV6u0*!PpMQ zjG14Ym9QxD(dlIp3K2NwbWV{-V*`3va2u?}e@tp@h#EP0jmzzk1_VqERW*hR-7p)* z#vr&ddUTzCln|{&K8(7>29?_f5}JFDdQ-${ws&ca!QBAEV?hL9vTYz+5_ZG}u(g36 za2tYh9p8)njwO-ZyE}�DLgYJz4Bypv`x~(*Y;5q{eSoisye_|@7~E}BPd{LKkaQXRUR3ce~$@6G)#k6&;p8t zp@YS~`sL)n3T(Q&Uq#5c5yl-a2H^%>gZa9viAU79e6@Evw3#_Jmzc5fJgm#qdI6lA z9xE)BVAD_5$FFY7B;1s~%nWfM?zdQU!qc@rcgH54=Pndmt)%EXp zH%-3}*H|#urIv$)n7JbTWroHI)Bo`dj-6R0JM$NH2r!v1#j;+H?URdw99!<55TPc3 zisH0MroQ4~0;j+Kf4JIz{V^B<>kp!^$O8SDAMZ z)jLz85-jA)sOQdWL5wUXY!Hrm|fd({2rRt=>wb;^)h&JS$k&p|avWfjk%yLh50?1~I z_`U;Wv`i5cdJCjI^QDJ1-v=F0=0Yk*8UnEn@3M*wdM1!?dV-|Ti02+jxrs%^`7>EN z1{|N880R%9ZB0dHWj;O z8!#9rZkFekjMkgEnglmP;KgB~my;od2WU?n3Jpl|kP}H4UO#ORqu$QwMnOGMrpKV* z@IT*urZpL~(jf>PB^BDtE}z%k`aF4*$|b4zCdodW$TYOM0|{qOl%gyOoJatHO4Ei8 zOulmgWRj`wOcLuWsi$bN5Yuw((Jq5ZZfQ_JX^HN6LwK&bvwEVPX~C^W^ZMVeb-L=E z?QB(ca4<8@sPy3ANgpD}-~TAPyYN++1|ILvg)jpSwM~JbZyuBSaX;-I6mji zE~l&E*}*DKu39dWGE5p$O1B`=ha}46uwjumZw6+s3ozetd6d*JX$-c(KyHPp^`M?v^&wRZm^NrcRq zXzIt86eriI$(K~*a)^TGcRxy(n>=rF`i8Y{FGAme0i-x7$6E~F_rtxn^Hx+0H0R$(IXgZ8z}vlXTv?jjT8?#$2MiE+=YGb(KlX&P-C zrd!uI9grZJy@r~n3=E*WEP48~%#nduyW^<2Dm^PEccND-p3Il|17yung>Q_l4>I_8jCnzEUI+=P! zHIzq?nzPPr4|gbGAyHEZ1nMHnzgAPiOjt_Sxne2w4v_|0%j%TJi&4$p$1i* zCb^dX3yU7Be2>P3`C>#@)BoZ5_xkb=w3q*duVXIvS5EMMela7v_@6<NBLVF_e zKPyV3y-hVgWDm)10!Ab~DwwQ5rXK6s_rJ;nkD~2K#rn8VPI7jWAWn>`BnTWQFLyvT zp|)p%jx&>_WX=}Z%a!091@wAX7wP(ERAjh$|8GCC=tvvtwIn<8jDArxWbcq8Y zK)@T3bkA{YGZ^r^5wXwlA)XSk{7*IL>E?wGBP&>Eg zTe5Iib%zgMzJaX|RH2BT)PO0W25f9C1h=+~u5q|Ju)vSBDojyG7i1{O^(vH>Mb0%Z z;Q}CKaoYr~7hT~TBGQ(i+r~+FF$rj3R5>jt+s`M7zu~;H4r~82n3;YfibBD=-kFCO z$%JPoRXxI2cKvICIKo`?f?2T`c? zPwfHELZE&|*SBnCTLmJImlvkzlA=1bnt?Sk6{c6jZFHRLg}CpSdjXU^(_x5tr5xhV za)?v>yC~<4bUQlr>_WQhp`MUAy9pex0@Fc(H4L#_5cDOkx$W*~8DI)DTa`m9b{d;D zhvy{3;YtH>1m5n02it9mjHwY2Bo+}J710f7Q1CL_R!`7BhnE?i`dZudiI`S%C491? zORYgx^bM048KE(Vl6Y_Q9e@xM+IKw)qFbHId&?hy*-NZ2u4>D+R8=W(Ti^ELmuUzR zC?ilvE*v@Id(F8RhyIMl{=E6xGk-thCGRlIYz6zmA8MhOy+7s(nu5Vnt>vx++r9x= zjh$f_%N25)7^E{4{VlILHx#qAO-^jtk{5KE$XuG3r|7*vdP|+)n^2Q86GO5QonVp) zdJe@+t!HLdadtaDxP+aY4pM_YM-w3e@7FT;qE5IAc4|4aJvXXzyRk#i_!P}9|2$iW zs@a>swzC)|4ra*I31``bInG2y2#ZZpUUIc7sk~G=x+wW?3I4X>@zYZ zAWI}9rD%K*LlUUSVIi11h`k-QFW*_jz6GCZi_~cyn~IN7SS-Bx4Wi5d!XpvQt8%C> zn|0o+9Vnwj>&DLN7vN#3gjM8quf7%l(~^*~=8Uds!8UZ-Gd0`O-(N5{sF7aK*)Ojl zHzj%J_BMFe(cs;G8z=O)^-eZG)o8Wu)=0F1-> zQ3%%=b5X4nHIf(5xsj9{!*|cf`%xh*l-D_30<>JD?{OGfkS72D6%7}Caj{J9_vnE3 z{|=`47yL%w6xA0b*+zl3j{h*}dJd!W3W@5{p$BH9;Ye+5P{GoL<}4~YmG{1)L8btl zWRcZ)ZjyIUsth{EmNd42K&)lX7!~w{b)`2sjj1c`=|CNW8MF`hLo$Tb-A-{t3sVId zwFV*YK%!UbV}gRJ&~u_U1aQKje$T7^w^y}!Q zfH%#GXltvj?_D++T=d=fgFk2Mw!c;HzkaysX;!|hxl&z}Zl?2=U3&)d_A^FfzCU`# z*4_Kc3HbhDMxF>kCFSVUSnM_8cc$qPi0Z_!xa13E;sZa4xvqkG(Sf?Z5cWd=Hi(5l z68b_0Zb(`zKiT zq;(a~FGGFeg_v)o?aXw#fX_VeV%-K#uTAaCZe!i*l8Qu$6P=cpb{#%A(Iz~57w9XR zR4Twa2XZ5c4*0teqksMQ{&S&mmgogiSJ{q~=NzS5yEI8teH$k>okQ1~#Vg+4nvG9D z<1?e(OcxY$i(dY~I`=TAdrSe>>TU%O~^D4AV1ZS(i5j+m~R6_(;*3Js`m%BrStK1sbEcZFE*>nePO zQl^kLNoteuTYKj#RVxvnzQG$4>NLIgpcr{f>d^}^TOoVXlitS{Xl+06>D5akJBXc& zKcNjjvNOL{(!fBj^IrEG$*ioA9r;~p0<~KM%q+ycb2vrf4f|62CS!E9dCirQ)K4uC zuUdAPXuY=@OiL7u461E3SnIvzN1m7t;WVSnU7D2X;VBKZ=I6>nC*Z{@b0d-FZAdLK zR#=!i zP)vI+Ij8joXxOg<1ec4J^uT@VQN(#08Xp}pk+-2M)TS5=A9zPFVgSL5FlPY+ihYx7 zByJs87Es;}HPSg_hsN0Z=!tQTdO4XF&WA|Az!jA^V$_Z;8a_Y?T5B(Aj6%bIL;KYW zsx-b3gAhv6i~0sA!E$>+p|A+mWqXOxsT=vl4qk5o+Y6GRv8ZjNJ}21rTKa+WvZ8IhU0>=;4W1_gc29de;a<-1)|scBOD{GFTMn`Obg(b}%zsJcX1{!%oIl2|b9cvL%I}@rbs_;Q@n>TDv6=6yMwa$G!Z)@aR zDn2EJNJU%vI1sLDrQOYkL#!}$#GZ?LIRXJ>uE1KK-DIBhSjc2pc_aMh{rYc>p z0F?+XUA8PFwaR=<{v;IQA>Gsn(hC(%x!=gUo;(3ec>ySWVff89SqFAn*TEH|`f0H2 z3rKKx0)4F(qY3b6@MmK1y!em^^bb|ypcU8KKWyHV9c->@@6SB*)ERr~Hi{dJneB|l zpqHS(#s}8-+JW9>WKJ*N*+V^l1~uEvjt=_mjA(0icRT20F7Bfgb>|+R`SO1KwuWqj z3kG`{VTo&+nwK3mTpL(-uT@?1J zA7on@ES_`;V!eUFz_C?Slaxeiq=6w+*Di*X`#82dij8%9@>IM6pa2?CE-LA4HuN{j z{*DhP6+R;S_ZA9?#=PXEh4ZPq)oS($RK1I4ynu+^u-E4S|843y5q$}r90rCUA6z&& zXfmn8T7L|+S31%!a3rn?YrQ-5qd5MHY(ai{|8?2E&Knb@dIg%y9z+eF&Vt*Moe#r= zLqvU&!gqc@Tq{XtoUz(z(!B`<_c*;GKJtZy`RTpO%Cw$33C|Jt)``-sEf}C{HR%dz z3JHeHT5rD3#rb#3rzlBlsyS1dy_=8BbtpX0EA-L6hV_G-@-};2m^M`i-G;X{LaW5s zw-J-W5drS(*n~i%l+ve0_eRG#3i4RvH>+)The>A*FMCYkOoKorj{D6rGcyyQ3Z~cq zjeQsRJ2#g*+c!sou$d|IPS_Flmu}<-dk4qXIT__n4;>oI znIPbL3~1H5RktEKP*%|SSDB~+T8DDx%V|;W5gR-opsVimdf2VAG|^gEU9N+Sb*s0* zp3zFI>p%;`WV5~IO3A6K6eZeR2gvKGSd9`q?&I3AW*7w}{uGAC7O96wh@iNI&7n4r zjL=26K3h#R>7af8Rw(^Q+ujJvNYPKWL%db6tZq?nM;bLTgr&AYPTh&-bd^;8O|9aI zwILlVHFz=We9|1?T+)durRuxao)wjo9G~5f&bFy9@8cu|rY>;;BBL%8QIw&?AbrP1 zN~8(cJfkr72w{`Gzt1QX$?4_U)VngoGMy1?qc{LpAO$cD9`*i#U`hl>P#avl8@L+g zqqDc;6tZw39Tq7BK|s4Rzn!D8>uH@K)d)2+`eFk3+yE&C! zwWdi8UV6r0k4r_nhq)N6#J$q2SlN2%xthM&uTJk3qLuvHnNow5K6Po*PN4u3u=xu~ z4ND!INhX0)==dq$A-l{pqyU*_DD+w)Zl4VmScy5hAi2EALa?R(d;(*7Xi9Tw1MAm~ z^<&chO~BZ%*C=7GKVh+Slxzvwy9uDR5`pgN;L=j+ohA#1p37}b?4Q^E_TrDMV)3u$ zB$e^I;kT8XIdi@d{YT-%e#vXk510JLPA|-8tJ;0I<5{@e3wc?W%Knk( zYoEOG?QA?B8y)>E1XT5Kl951zLI+N$FFxdr15vebXd<*6CUv=OfH;znJF+p08ct5R z|Es<0j%qSZ_v)(S%;3z}MnDjkA|v2PS1^JFg(#r(CP5|?5$RE+y1!TVuFg4g?_c+?d#=ZGgyhR--sgLt=Y8t$M=~+( zeq+18j*?;}e8C-q8|Q#N;z%oN>&_Ez<ee`9Oi>hcuj!zu~>V!OXKT6#EUoty1>S5@;TCa&V= z>8i|7$3FNKdAYgw5I0;w0c4xLz7yC;L5s^wRI$5ESf(l}D$IBp=UNpS;T!+<=nYaa zR3_NYHwE;dYjp+@g3*Qz8^8(JiIjSvDZQ?-ktGkhPDRe*UAOVE&cZf$KMC>imuzUE z2T{5*@h)B(_p?e%gbHC-P2sTF1`-kyX8PbrV#>R^yDQ7d9RSf(ggJecI)8NZ6qcT6 zlBVr97YZ#6#5{a1#$NH2kCDxx>!wbRuT!%VPFD67JAa>-m)DNnoO|mnpp6eR(e;`a z9WAz6YS-j==H9h*9ci6qMk`-zy1j+}Se3s<%wOzKWs{HQ^}q%->E2seMV#TCriQV@ z+M9=VF=c)+h^;^Tvp_rGU{sEIth1ni^T`tS+0s4EG!y~}y9j8}Q*vMHmx%GYt2rv$ zwnbGP9a!g$MHyFpE~~RrE=BMXI=X{e=EF1H*iFhbEA|-dymQ`lz~WO3dIj|kBU{ss zR%E)~3OgF@TWTc=CE4YXuLqMPHQb^*xlUYU8a?4m4^!^SEIj_Cp{c1IMUhJY5@;8V zomN&-BFw3|ULa1mhtn?)QC8=hKPi5P9X1abse*?Oj~CKv?2~9=9yNTU#89&v8D5F| zhF%d9jM0ubkvPFQ4}3N04EdViAvCnKa0Rbq0hlIBwXX>)LnwFbId`E5`vcCXWZ4A1 zuqFJBroLM<&H877(VK^kLJ!bXy?nENlg^HU`LNFx57dd~s&jkHog?IAr2{S`_rEH)Ozz)e zNb70ZpujtBCa1n-08@JFSS|z%w8;zcxE8fD-)2WXhZA*^f$^}d^$m@U+2t~nkb79i zcg)i$f34P@wLFk`8~cfn-lG$7^W0UCX>F}XYR(OkxPksd_U$YOx-9?1^2N&70ni`K zCX0M}%A4iwurJRt)^3w^ETIIBoOg9`=|WgU_x^=)lsbz#so|qpAXTKwkKtKat|WXf zSbCRyH{ko0Nsl3FG-fsfzk2H&wthE|c0J!xKxfNxftHoS_kJ*etq_db{BP2IU@DzJ z6i~Xb5Z(itu-1OLMvJ=&jTZw7P!V%9W`ZtAmNS*{j1;b^p6*xydSkdR>K=5{3rO_@rf`7L}4^KFJL+_4Z@2?DPA>2t%_ zsE(b6OM>VT#GXhE?IOm-DCu>~Pl={F({C0pSf=(F38ducCt~|IaWwg68~u&51i_T1 z4Qe*+U9VPd`=J*A23)8a17*Bu%khUhjB!%^+t4mbUCl$4a(OWlvI-lSa+EJDU8 z5T7d> z5T@{2M%yaLwJ#B9jR3XO3g#M#uEO+mDF*09iN07D+Qa5w&SemSFRYYM_z`L?Yc8!2 zDkg*%5jCGT4?n^Pb7qCHoA=})cs&I|H2a(Wu1(@k_hhXqS>WAtlL(AmEyQG$xtUo! zPREGtPPNAm9z3Y7;q|D}3Qe%*w0(LDpzenz+Q1NsX33-Av#<;fE+<4r;367J^(=~- zztDs7Flf{?$vR*(EIm9tnxe8cN`;+%738DS)3ni(G=i9&0R*S^`7|TPl_MFbg!{+Rf{{mb*wqZ^54b zD*u^_%5`|EOPB2?l0o8OIlRL4Y1$CY(_m6sb2<_nA zJo%?~ z3H(EQo(Z!KQ)dn&#;{z^>vSsQj_q@G&DpSVql2^arNN;gd#u9?kQ%{gQSHGgb?W)X z*P1;jZF|8`b(VSBG}hN&!z#LoEOBz0#W~p9hw;P+$FuM0f#fiqFovT*A?0XkJpIut z%Pn7t{mzX3+aJJ#vjBWbMu3=ZW2Gv8Uo25q%OuOYYLBk%{mkQb+ou{G@vf?r8ji6m zyoied4?#`52o#87%t|_#n#PiEii!OR4676)y|;6^)bqG7b5PjQmR6^4pa)pDE1S=y z#Je7&42!1}P*)6yi?!%XSAKCwX>iqFU&mZLPxnpkU$qjC@6qtiv-@MKBftDiXC*%% zvB_`w_fp@-ALe@b8>ruzTjcJQLMur-+>ni>HPzMCv$u9sjjJQS;prY;fLhwhUFwl_ z>f-x`Y?A`(l3)e`W0B==!;F$-`{eOIA3=ikAD?_Ig7@;k%bXm)h$+Q?Ct%LgZnMy_$MYSzgVe5YFiPLZ}xmJ!jH@F{~D;P7yP;B zeLv!VEyd)UQurdj`?LDT6(7)?$UFOg`GXgXsN|5Zt2>4R0lS$tyoj=<-I+cpdW!{hMArRu&u`&ItD0QYf3+>YFq1AA~gaB-81;%rpw%3DP zux=`;sI()cSQh#A%@6dK=}Gboxbu~WBReh~_=K>%!x^xqQloHOjRuVqpy#p{Pbh=m z^B16}@g&sUZCB&$ub;exRMRN11KUWY!1wf;OpPn7Cu8j4a3THG{5P-fCY%1mWo5p| ziF-1ZJ+Ql<_l!MUATPsMPI(a?`&d%uZb05}^i5tH-?AY&z~z;$P{(OryjJgU?Z>xC zP82iR$3zj;ZEHRH@;A)sIh3D{s&s;dyhsnh5o62*Z|B{jTW`Wh1KT(%Ha9n{U!N>r zg&u93xJu8uppf4wC@84u$LYTbc}NEprU11VALeNcBn9ID=|PFq6I3=OSk4173`nFR zK!&&w@ZpIW8D6M+mOp*^9f5Q6`Wl!3CLs^ul%&XecH|{cOK^Vi9uC7km4lYk9uEwo z7oBi9y&$Mgj)fx2g+%nQ?CMYaDDK9J{}fPLUbpfmU*2!}3olSTJYyz(2SdzBra9dG zVHbW$NYY!D8LDUK8}sF`PDcRmc)yUY z2L`%8aQdCMe&{sY%Em@9T#Orve;qKrV!|v$f*|;eiAfCM1KV1UN`>vP8k+0_0Pvg* zPQCg=(c)&qzd3_-77kuR!{u@rz3?soBg?W}5Pf`_l_iLlqhDoR{oxXe_97-aLoajf zPkO1Zn|G-P>xLc{*A;5@)pZH`w>ghwr}W`cpC;3=<^~Iyr7hK&ab)|Uk*SkjhPIz&#T=aWaL}x zu`?1UbwZlEqD^`IHCVV-cHC6RbF#wb5f@7DZlRWw#=Lmk9FasbHze!T&Sv3`kU{eZozg2xj#Xd5E=J6f z4~+*9+?(}f3{VNrBN`9FoYv0zijvY&1u7ztU(Sf&?GiGPljxtduMW9}fW~QwKG&VHMz}G%i z&s0F9ax`QC14C3ME#%3Mxf7V?Ekpe(8#a6dwj*BDYlAnP^V5@`mX3?sUp$Vxqx3%=_lkakR)nQ5^KVzH@ z;dJrq=phY_4vIB{oA$|M|CkfUycI1s;&QzhpQ~Lnqh@bApKSVc=O&{y(AiE*qOeO! z)F*mt9Kf7$NnQx!W&?)$KxEU&khb64_8mKHLPJB%`Ds8+sd+XB5gDC}G-t8lB6=VV zXV!zJ+>0nwn=gvy;vT#~uFr_s8zJ&~M<|JnGU3(T-JPo89!rMV!eSa7XflkMczI57 zvFfu3WQtm`FRr^be*F^eL$Cu<@@cLKOqA^{T}MMp;!(IZUi z*$}7_D@VsdJX_(}&%e$6UlQWNp~{z#bkdFg=~1W^YQD6wT{<=L)+x!|tvq5ejO7CS za1D)8I03}|5$-chjrC{qtfxjjIm&IFoN zW+AyQG;NqLLD31cre5iwx8Sr$lgCCXA+WjSbp98+LCp&H`k`*%$T2Q=*-Way%Er-^ zIga#9Ox5#u@f+E)Wj&xJI*s@;7_ct}4963fxf?Y#dTP$xKwP78Y5`A#z(MgAAKdnN zk$O!c!+7hC9T_vJTnPsGiJ|!-?T=ry>;Frur*-B$w%nsPph`p z?mpjl_I?132Kxm~PBx~|=1&goq-*G=;Kz?%d~bvT=j(wQhy%DY=g3Ax#MgjT_7C}P z(4aOIJYzW?u&oWh2~%RXZ{Od%`fB%Z7+johEEix6MnA}8vFOaxs8p$m3hc`oTDTjW;^Bx-~q847wdVu=Ec4g&8w5+=V zK*#vi@oKm9RT(oqRPO7o`(+kq?>G7IqV7>|K`cJ9GxvRDnit&CjSd8TSbzK~K}g@Q zC09c6j+UMzgYp+nm!Zb3_eOZ#rML(Y7T<3$!tNvXcg_6BjX2@oSC|`7`}DdOfr+6? zorNKCKwe@Tp%$m6@S16`vUNf6`1olNAZb0*cKVoxy(KvS`EGPZTrM}Xu0sJXTTNtX zVXl3KLb^*_{d{f>rFt4OYmk(2bDKrLXXou*go>G$+Faq-Q{Sh*eu|L) zDhU$G5eFnM0({bgS-@t?KY1~g>L(kho=7YyzNL8Mk#{Ryl@UFd10!wngC1|X73-2v zY&`~9BwQsY^BzNatU2*&K%O9a-4jWf7`@Ci%L$nB>gvE&&)TSaeA{==9?N%8Av+5q zR7*_KyOQm;OsWDI)gY8NJm4I_lE$G(AHZXrU0__sY-TjDBQKvlyR6Wz^boAW;ZZ*Y zA?l^E@x_12dFYP+)0gm4Gat1D76JpN|DH*P&5>7gE1#jjlwar((@O+A2o$U|Ae9D|+1}*0t*3)+6s1D*EPi{jPZ4C_AZuGPh>3 zsZmFEP0`dfN|r^9OZqCotCKTRO?U1Y)*q$Ob_*5y3`XcGT*ie3yaySVv1rrpWc+|1 z_>X1VD$XY*T?Nb5D8>mb8xyx{g-#D>JhPzDiV%k?1f_>4R-w!H6sr07%jzt0%tr{M z_%eCTn1GwaPF^&Ok<3BJC`1dvJwP#E_Qni+7V<7Q_LN~)YJ2(9OaYixE)i7UFeLA^ z04bK+15%1hkXSBKH)U}ku(vS*V&*eCf%KTiA@(kj2}zGWD9e{8jeATs+?oo}MBgnZ z<^>juHF+~4E$NTM@&~f>C@>AHZwMgHOuF1!Y@Ws!Ivq!NQNA&8hh$n*#2}i1kHiAG zK@aO~Ka!pC=O&9AtoKspI|YL&DF~h$tqW|OeQIC3wa@;vP+5%=t+A=;I*Ch>#A^w8 zi?7}dx|@FQmX;PIqfm>=?a^DDvGBZq`Ns?}tC`)(zFt}R0CSbrto3taGp55@TqJWz zEKMx|i*%>=yVT9-ZFJRij&%hwY91p7#?ja!-;I%#%|SiLpA=vk5zZh+^Czcj+~Ayc z!<4iVlL$)^sGbfjfRF1=wCmyek2DEB?sJ_E&ksKC$5(!I8#U>v)#kq0Nb!s!+Rm2t z!NOCuy;4mxDI9EC-lA?*bu|UYZs}|b@!(&gu;qaq)I4CTt|Gfib~+GyN^mp;!u#PD zxRA&D5}4*AXfbaCo{*OFo#hB5Oa})C!w~RW+1loko(XDP-vOGPot@J5?Hy$6V0)f| zbB%9NN<6_Dj8sbM2IfYfN}ttM;U5rCcJjU~u|1Wjz7fDQE70d&`$B5hMMMl9=-&b9 zvr$IA|GKI3Y8l2^reSq33`b=YWF*?q#?Jm3q0>B^op`^zz}5rSy~NjfA{DPRY*BAW z>uup|RyR$+XU^$uvo1_t#TGXc z71&<8gO;!pAoeRptAhiQQ3#m2D*XB{TmQ@}4W6*yv?uI3?M}= zKXKy6mwt2~x^9E2#=75y+CLm~vv>!u{oz%~&p!Xp5Z=32fB5?UU-%%NB}@KK$)oe_ Yl0kaO+N7$PPO?VoK~saG{U^@;3x3sT-~a#s diff --git a/benchmarks/results/pr328_rocm_mi300x/report.md b/benchmarks/results/pr328_rocm_mi300x/report.md index 1ad0b72a..c9d28b34 100644 --- a/benchmarks/results/pr328_rocm_mi300x/report.md +++ b/benchmarks/results/pr328_rocm_mi300x/report.md @@ -8,12 +8,11 @@ |---|---| | architecture | gfx942:sramecc+:xnack- | | extension_symbols | hip_deterministic_logp_tile_stats, hip_deterministic_logp_backward | -| git_commit | 1c73a427881252871b46936f6d14be36a22e307f | +| git_commit | e9f1d2a5c67a283bd987978614b4444a9416c1f0 | | gpu | AMD Instinct MI300X | | gpu_count | 8 | | hip | 7.14.60850 | | native_collective | torch.distributed ProcessGroupNCCL (RCCL on ROCm) | -| native_single_gpu | torch.logsumexp + gather on FP32 logits | | python | 3.12.3 | | torch | 2.12.0+rocm7.14.0a20260608 | @@ -22,8 +21,8 @@ - Qwen3 vocabulary `V=151936` split into 64 tiles of 2374 columns; seeded logits (`randn * 2.0`), random targets, every seventh token inactive. - Measured paths: - `native`: `pytorch-vocab-parallel-logp-ws2`, the WS2 vocab-parallel reference operator: a PyTorch tile loop for the per-tile FP32 `(max, sumexp)` partials, all-gather of the partials, fixed global tile-order merge, and a PyTorch autograd backward. - - `triton`: `triton-batch-invariant-logp-ws1`, the single-shard Triton online-softmax op; it has no vocab-parallel (TP) path, so it appears only in the single-GPU results. - - `strict-hip`: `rocm-vocab-parallel-logp-ws2`, the same contract, transport, and merge with two HIP kernels: `hip_deterministic_logp_tile_stats` reads the stored BF16/FP16/FP32 shard directly (8-element vector loads, no FP32 copy) and `hip_deterministic_logp_backward` produces the gradient in one fused pass. + - `triton`: `triton-vocab-parallel-logp-ws2`, the same contract, transport, and merge with two Triton kernels (tile statistics read from the stored shard, fused backward); one source for CUDA and ROCm. + - `hip`: `rocm-vocab-parallel-logp-ws2`, the same contract, transport, and merge with two HIP kernels: `hip_deterministic_logp_tile_stats` reads the stored BF16/FP16/FP32 shard directly (8-element vector loads, no FP32 copy) and `hip_deterministic_logp_backward` produces the gradient in one fused pass. - Distributed: one process per GPU; the WS2 operators all-gather per-tile `(max, sumexp)` partials over RCCL and merge them in fixed global tile order; CP ranks shard tokens and never enter the merge. - Forward returns the selected-token logprob and the vocabulary LSE; forward+backward computes `grad_logits` for `sum(active * logp)`. The WS2 operators run with `validate=False`; the `validate=True` production entry point is measured separately. - Single-GPU timing: GPU events, median and p95. Distributed timing: synchronized wall clock, slowest rank per sample. Peak memory is the per-call increase in `torch.cuda.max_memory_allocated` (distributed: max over ranks). @@ -38,8 +37,8 @@ python benchmarks/benchmark_rocm_logp.py \ --warmup 5 \ --samples 20 \ --training-samples 10 \ - --report-paths ws2-reference,ws1-triton,ws2-rocm \ - --rename ws2-reference=native,ws1-triton=triton,ws2-rocm=strict-hip \ + --report-paths ws2-reference,ws2-triton,ws2-rocm \ + --rename ws2-reference=native,ws2-triton=triton,ws2-rocm=hip \ --report-baseline ws2-reference \ --table-tokens 2048 \ --output-dir benchmarks/results/pr328_rocm_mi300x @@ -47,12 +46,13 @@ python benchmarks/benchmark_rocm_logp.py \ ## Key findings -- Single GPU: `triton` is 8.90-10.79x faster than `native` in forward and 6.73-11.28x in forward+backward, with 0.15-0.33x its peak memory. -- Single GPU: `strict-hip` is 5.97-7.23x faster than `native` in forward and 4.94-6.83x in forward+backward, with 0.15-0.33x its peak memory. -- Distributed: `strict-hip` is 1.69-4.60x faster than `native` in forward and 1.97-4.64x in forward+backward across 6 TP/CP topologies, at 0.15x the per-rank peak memory (absolute forward 0.773-1.041 ms). -- `strict-hip` runs at 1.49x the latency of `triton` in forward and 1.36-1.65x in forward+backward with the same peak memory, while carrying the vocab-parallel contract (tile partials, all-gather, fixed tile-order merge, vocab-domain LSE export) that the single-shard Triton op does not provide; the gap is the operator's fixed Python/launch floor, not the HIP kernels. -- The `hip_deterministic_logp_tile_stats` kernel alone is 10.2-10.6x faster than the PyTorch tile loop and allocates 57x less transient memory (it writes only the `[tokens, 64]` FP32 partials). -- `strict-hip` vs `native`: tile maxima are bitwise equal; sumexp partials differ only by FP32 summation order, so final outputs differ in 0-65 elements per case with relative-L2 0.0e+00-1.3e-08. Both paths are equally close to FP64. +- Single GPU: `triton` is 5.26-6.45x faster than `native` in forward and 4.29-5.71x in forward+backward, with 0.15-0.33x its peak memory. +- Distributed: `triton` is 1.62-4.28x faster than `native` in forward and 1.82-4.05x in forward+backward across 6 TP/CP topologies, at 0.15x the per-rank peak memory (absolute forward 0.824-1.115 ms). +- Single GPU: `hip` is 5.59-6.88x faster than `native` in forward and 4.47-6.07x in forward+backward, with 0.15-0.33x its peak memory. +- Distributed: `hip` is 1.70-4.67x faster than `native` in forward and 2.05-4.49x in forward+backward across 6 TP/CP topologies, at 0.15x the per-rank peak memory (absolute forward 0.755-1.063 ms). +- The `hip_deterministic_logp_tile_stats` kernel alone is 9.6-9.9x faster than the PyTorch tile loop and allocates 57x less transient memory (it writes only the `[tokens, 64]` FP32 partials). +- `triton` vs `native`: tile maxima are bitwise equal; sumexp partials differ only by FP32 summation order, so final outputs differ in 0-169 elements per case with relative-L2 0.0e+00-6.9e-08. Both paths are equally close to FP64. +- `hip` vs `native`: tile maxima are bitwise equal; sumexp partials differ only by FP32 summation order, so final outputs differ in 0-65 elements per case with relative-L2 0.0e+00-1.3e-08. Both paths are equally close to FP64. - Repeat bitwise: yes; batch-invariant: yes; all gradients finite: yes. - Distributed: TP-replicated and repeat bitwise on every topology: yes. @@ -62,23 +62,24 @@ python benchmarks/benchmark_rocm_logp.py \ | Tokens | Path | Median (ms) | p95 (ms) | Speedup vs native | Peak MiB | logp max-abs vs FP64 | LSE max-abs vs FP64 | Repeat | Batch-inv | |---:|---|---:|---:|---:|---:|---:|---:|:---:|:---:| -| 2048 | native | 6.4801 | 6.7318 | 1.00× | 1245.0 | 1.557e-06 | 1.382e-06 | yes | yes | -| 2048 | triton | 0.6005 | 0.6081 | 10.79× | 0.0 | 1.589e-06 | 1.516e-06 | yes | yes | -| 2048 | strict-hip | 0.8964 | 0.9366 | 7.23× | 2.0 | 1.621e-06 | 1.318e-06 | yes | yes | +| 2048 | native | 6.1387 | 6.3072 | 1.00× | 1245.0 | 1.557e-06 | 1.382e-06 | yes | yes | +| 2048 | triton | 0.9514 | 0.9801 | 6.45× | 2.0 | 1.557e-06 | 1.318e-06 | yes | yes | +| 2048 | hip | 0.8923 | 0.9162 | 6.88× | 2.0 | 1.621e-06 | 1.318e-06 | yes | yes | ### Forward+backward | Tokens | Path | Median (ms) | p95 (ms) | Speedup vs native | Peak MiB | Memory vs native | Grad finite | |---:|---|---:|---:|---:|---:|---:|:---:| -| 2048 | native | 12.8275 | 12.9236 | 1.00× | 7715.7 | 1.00× | yes | -| 2048 | triton | 1.1376 | 1.1707 | 11.28× | 1187.0 | 0.15× | yes | -| 2048 | strict-hip | 1.8781 | 1.9318 | 6.83× | 1187.1 | 0.15× | yes | +| 2048 | native | 12.3477 | 12.4680 | 1.00× | 7715.7 | 1.00× | yes | +| 2048 | triton | 2.1619 | 2.1945 | 5.71× | 1187.1 | 0.15× | yes | +| 2048 | hip | 2.0359 | 2.0513 | 6.07× | 1187.1 | 0.15× | yes | -### `strict-hip` versus `native` numerics +### Numerics versus `native` -| Tokens | Mismatched elements (logp+LSE) | Relative L2 | -|---:|---:|---:| -| 2048 | 65 | 1.098e-08 | +| Tokens | Path | Mismatched elements (logp+LSE) | Relative L2 | +|---:|---|---:|---:| +| 2048 | triton | 169 | 1.713e-08 | +| 2048 | hip | 65 | 1.098e-08 | ## Single-GPU logprob (FP32 logits, V=151936) @@ -86,29 +87,30 @@ python benchmarks/benchmark_rocm_logp.py \ | Tokens | Path | Median (ms) | p95 (ms) | Speedup vs native | Peak MiB | logp max-abs vs FP64 | LSE max-abs vs FP64 | Repeat | Batch-inv | |---:|---|---:|---:|---:|---:|---:|---:|:---:|:---:| -| 2048 | native | 6.0009 | 6.2081 | 1.00× | 58.0 | 1.747e-06 | 1.271e-06 | yes | yes | -| 2048 | triton | 0.6741 | 0.6921 | 8.90× | 0.0 | 1.909e-06 | 1.550e-06 | yes | yes | -| 2048 | strict-hip | 1.0045 | 1.0516 | 5.97× | 2.0 | 1.747e-06 | 1.271e-06 | yes | yes | +| 2048 | native | 5.5791 | 5.6527 | 1.00× | 58.0 | 1.747e-06 | 1.271e-06 | yes | yes | +| 2048 | triton | 1.0602 | 1.0851 | 5.26× | 2.0 | 1.711e-06 | 1.197e-06 | yes | yes | +| 2048 | hip | 0.9972 | 1.0351 | 5.59× | 2.0 | 1.747e-06 | 1.271e-06 | yes | yes | ### Forward+backward | Tokens | Path | Median (ms) | p95 (ms) | Speedup vs native | Peak MiB | Memory vs native | Grad finite | |---:|---|---:|---:|---:|---:|---:|:---:| -| 2048 | native | 12.5812 | 12.7740 | 1.00× | 7122.2 | 1.00× | yes | -| 2048 | triton | 1.8702 | 1.9688 | 6.73× | 2374.0 | 0.33× | yes | -| 2048 | strict-hip | 2.5475 | 2.6220 | 4.94× | 2374.1 | 0.33× | yes | +| 2048 | native | 12.1054 | 12.1808 | 1.00× | 7122.2 | 1.00× | yes | +| 2048 | triton | 2.8249 | 2.8799 | 4.29× | 2374.1 | 0.33× | yes | +| 2048 | hip | 2.7086 | 3.3877 | 4.47× | 2374.1 | 0.33× | yes | -### `strict-hip` versus `native` numerics +### Numerics versus `native` -| Tokens | Mismatched elements (logp+LSE) | Relative L2 | -|---:|---:|---:| -| 2048 | 37 | 8.004e-09 | +| Tokens | Path | Mismatched elements (logp+LSE) | Relative L2 | +|---:|---|---:|---:| +| 2048 | triton | 144 | 1.466e-08 | +| 2048 | hip | 37 | 8.004e-09 | -### `validate=True` production entry point (strict-hip, BF16) +### `validate=True` production entry point (hip, BF16) | Tokens | validate=False (ms) | validate=True (ms) | Overhead | |---:|---:|---:|---:| -| 2048 | 0.8911 | 1.0743 | 1.21× | +| 2048 | 0.9116 | 1.0558 | 1.16× | `validate=True` adds host-side target-range checks and a non-finite LSE check that synchronizes the stream; the cost is a fixed per-call overhead. @@ -118,8 +120,8 @@ python benchmarks/benchmark_rocm_logp.py \ | Logits dtype | Tokens | PyTorch tile loop (ms) | HIP kernel on FP32 (ms) | HIP kernel on stored dtype (ms) | Speedup | Loop peak MiB | HIP peak MiB | Max bitwise | sumexp max rel | Repeat | |---|---:|---:|---:|---:|---:|---:|---:|:---:|---:|:---:| -| bf16 | 2048 | 5.5575 | 0.5239 | 0.4659 | 10.61× | 56.6 | 1.0 | yes | 3.64e-07 | yes | -| fp32 | 2048 | 5.5582 | 0.5459 | 0.5364 | 10.18× | 56.6 | 1.0 | yes | 4.48e-07 | yes | +| bf16 | 2048 | 5.0990 | 0.5171 | 0.4669 | 9.86× | 56.6 | 1.0 | yes | 3.64e-07 | yes | +| fp32 | 2048 | 5.1025 | 0.5328 | 0.5331 | 9.58× | 56.6 | 1.0 | yes | 4.48e-07 | yes | ## Distributed vocab-parallel logprob (BF16, RCCL) @@ -127,46 +129,64 @@ python benchmarks/benchmark_rocm_logp.py \ | Topology | Tokens | Path | Median (ms) | p95 (ms) | Speedup vs native | Peak MiB/rank | logp max-abs vs FP64 | TP-replicated | Repeat | |---|---:|---|---:|---:|---:|---:|---:|:---:|:---:| -| tp2 | 2048 | native | 3.5356 | 3.5972 | 1.00× | 650.3 | 1.452e-06 | yes | yes | -| tp2 | 2048 | strict-hip | 0.8436 | 0.8830 | 4.19× | 3.0 | 1.452e-06 | yes | yes | -| tp4 | 2048 | native | 2.3684 | 2.4618 | 1.00× | 353.0 | 1.452e-06 | yes | yes | -| tp4 | 2048 | strict-hip | 0.9375 | 1.0703 | 2.53× | 2.5 | 1.452e-06 | yes | yes | -| tp8 | 2048 | native | 1.7633 | 1.9199 | 1.00× | 204.3 | 1.452e-06 | yes | yes | -| tp8 | 2048 | strict-hip | 1.0407 | 1.1309 | 1.69× | 2.3 | 1.452e-06 | yes | yes | -| tp2_cp2 | 2048 | native | 3.3620 | 3.7359 | 1.00× | 325.5 | 1.452e-06 | yes | yes | -| tp2_cp2 | 2048 | strict-hip | 0.7731 | 0.8305 | 4.35× | 1.5 | 1.452e-06 | yes | yes | -| tp4_cp2 | 2048 | native | 2.2469 | 2.5062 | 1.00× | 176.7 | 1.452e-06 | yes | yes | -| tp4_cp2 | 2048 | strict-hip | 0.8380 | 0.9284 | 2.68× | 1.3 | 1.452e-06 | yes | yes | -| tp2_cp4 | 2048 | native | 3.5902 | 3.9379 | 1.00× | 163.1 | 1.452e-06 | yes | yes | -| tp2_cp4 | 2048 | strict-hip | 0.7803 | 0.8550 | 4.60× | 0.8 | 1.452e-06 | yes | yes | +| tp2 | 2048 | native | 3.5877 | 3.6322 | 1.00× | 650.3 | 1.452e-06 | yes | yes | +| tp2 | 2048 | triton | 0.8894 | 0.9138 | 4.03× | 3.0 | 1.452e-06 | yes | yes | +| tp2 | 2048 | hip | 0.8487 | 0.8818 | 4.23× | 3.0 | 1.452e-06 | yes | yes | +| tp4 | 2048 | native | 2.2691 | 2.3080 | 1.00× | 353.0 | 1.452e-06 | yes | yes | +| tp4 | 2048 | triton | 0.9221 | 0.9652 | 2.46× | 2.5 | 1.452e-06 | yes | yes | +| tp4 | 2048 | hip | 0.8714 | 0.9162 | 2.60× | 2.5 | 1.452e-06 | yes | yes | +| tp8 | 2048 | native | 1.8066 | 1.8356 | 1.00× | 204.3 | 1.452e-06 | yes | yes | +| tp8 | 2048 | triton | 1.1149 | 1.1961 | 1.62× | 2.3 | 1.452e-06 | yes | yes | +| tp8 | 2048 | hip | 1.0631 | 1.1280 | 1.70× | 2.3 | 1.452e-06 | yes | yes | +| tp2_cp2 | 2048 | native | 3.3882 | 3.5075 | 1.00× | 325.5 | 1.452e-06 | yes | yes | +| tp2_cp2 | 2048 | triton | 0.8308 | 0.9605 | 4.08× | 1.5 | 1.452e-06 | yes | yes | +| tp2_cp2 | 2048 | hip | 0.7809 | 0.8174 | 4.34× | 1.5 | 1.452e-06 | yes | yes | +| tp4_cp2 | 2048 | native | 2.2382 | 2.2630 | 1.00× | 176.7 | 1.452e-06 | yes | yes | +| tp4_cp2 | 2048 | triton | 0.8831 | 0.9622 | 2.53× | 1.3 | 1.452e-06 | yes | yes | +| tp4_cp2 | 2048 | hip | 0.8386 | 0.8978 | 2.67× | 1.3 | 1.452e-06 | yes | yes | +| tp2_cp4 | 2048 | native | 3.5261 | 3.6071 | 1.00× | 163.1 | 1.452e-06 | yes | yes | +| tp2_cp4 | 2048 | triton | 0.8240 | 1.3511 | 4.28× | 0.8 | 1.452e-06 | yes | yes | +| tp2_cp4 | 2048 | hip | 0.7547 | 0.7834 | 4.67× | 0.8 | 1.452e-06 | yes | yes | ### Forward+backward | Topology | Tokens | Path | Median (ms) | p95 (ms) | Speedup vs native | Peak MiB/rank | Memory vs native | Grad finite | |---|---:|---|---:|---:|---:|---:|---:|:---:| -| tp2 | 2048 | native | 7.1279 | 7.1917 | 1.00× | 3857.9 | 1.00× | yes | -| tp2 | 2048 | strict-hip | 1.5376 | 1.5811 | 4.64× | 593.6 | 0.15× | yes | -| tp4 | 2048 | native | 4.4086 | 4.5749 | 1.00× | 1929.0 | 1.00× | yes | -| tp4 | 2048 | strict-hip | 1.4017 | 1.5074 | 3.15× | 296.8 | 0.15× | yes | -| tp8 | 2048 | native | 3.3382 | 3.7182 | 1.00× | 964.5 | 1.00× | yes | -| tp8 | 2048 | strict-hip | 1.6933 | 2.2111 | 1.97× | 148.5 | 0.15× | yes | -| tp2_cp2 | 2048 | native | 5.4500 | 5.4998 | 1.00× | 1929.0 | 1.00× | yes | -| tp2_cp2 | 2048 | strict-hip | 1.2361 | 1.6392 | 4.41× | 296.8 | 0.15× | yes | -| tp4_cp2 | 2048 | native | 3.7984 | 3.9900 | 1.00× | 964.5 | 1.00× | yes | -| tp4_cp2 | 2048 | strict-hip | 1.3963 | 1.5687 | 2.72× | 148.4 | 0.15× | yes | -| tp2_cp4 | 2048 | native | 5.0433 | 5.2447 | 1.00× | 964.5 | 1.00× | yes | -| tp2_cp4 | 2048 | strict-hip | 1.3849 | 1.9775 | 3.64× | 148.4 | 0.15× | yes | - -### `strict-hip` versus `native` numerics (distributed) - -| Topology | Tokens | Mismatched elements (logp+LSE) | Relative L2 | -|---|---:|---:|---:| -| tp2 | 2048 | 54 | 9.958e-09 | -| tp4 | 2048 | 54 | 9.958e-09 | -| tp8 | 2048 | 54 | 9.958e-09 | -| tp2_cp2 | 2048 | 54 | 1.069e-08 | -| tp4_cp2 | 2048 | 54 | 1.069e-08 | -| tp2_cp4 | 2048 | 54 | 1.162e-08 | +| tp2 | 2048 | native | 7.1482 | 7.5344 | 1.00× | 3857.9 | 1.00× | yes | +| tp2 | 2048 | triton | 1.7667 | 1.8881 | 4.05× | 593.6 | 0.15× | yes | +| tp2 | 2048 | hip | 1.5938 | 1.7476 | 4.49× | 593.6 | 0.15× | yes | +| tp4 | 2048 | native | 4.4688 | 4.5386 | 1.00× | 1929.0 | 1.00× | yes | +| tp4 | 2048 | triton | 1.7425 | 2.1374 | 2.56× | 296.8 | 0.15× | yes | +| tp4 | 2048 | hip | 1.5001 | 1.6030 | 2.98× | 296.8 | 0.15× | yes | +| tp8 | 2048 | native | 3.3579 | 3.9040 | 1.00× | 964.5 | 1.00× | yes | +| tp8 | 2048 | triton | 1.8465 | 1.9464 | 1.82× | 148.5 | 0.15× | yes | +| tp8 | 2048 | hip | 1.6417 | 1.8066 | 2.05× | 148.5 | 0.15× | yes | +| tp2_cp2 | 2048 | native | 5.6092 | 5.7342 | 1.00× | 1929.0 | 1.00× | yes | +| tp2_cp2 | 2048 | triton | 1.4580 | 1.7000 | 3.85× | 296.8 | 0.15× | yes | +| tp2_cp2 | 2048 | hip | 1.3925 | 1.6368 | 4.03× | 296.8 | 0.15× | yes | +| tp4_cp2 | 2048 | native | 3.7864 | 3.8899 | 1.00× | 964.5 | 1.00× | yes | +| tp4_cp2 | 2048 | triton | 1.6803 | 1.7273 | 2.25× | 148.4 | 0.15× | yes | +| tp4_cp2 | 2048 | hip | 1.4029 | 1.4394 | 2.70× | 148.4 | 0.15× | yes | +| tp2_cp4 | 2048 | native | 4.9700 | 5.0383 | 1.00× | 964.5 | 1.00× | yes | +| tp2_cp4 | 2048 | triton | 1.5611 | 1.6497 | 3.18× | 148.4 | 0.15× | yes | +| tp2_cp4 | 2048 | hip | 1.3959 | 1.5368 | 3.56× | 148.4 | 0.15× | yes | + +### Numerics versus `native` (distributed) + +| Topology | Tokens | Path | Mismatched elements (logp+LSE) | Relative L2 | +|---|---:|---|---:|---:| +| tp2 | 2048 | triton | 137 | 1.454e-08 | +| tp2 | 2048 | hip | 54 | 9.958e-09 | +| tp4 | 2048 | triton | 137 | 1.454e-08 | +| tp4 | 2048 | hip | 54 | 9.958e-09 | +| tp8 | 2048 | triton | 137 | 1.454e-08 | +| tp8 | 2048 | hip | 54 | 9.958e-09 | +| tp2_cp2 | 2048 | triton | 137 | 1.477e-08 | +| tp2_cp2 | 2048 | hip | 54 | 1.069e-08 | +| tp4_cp2 | 2048 | triton | 137 | 1.477e-08 | +| tp4_cp2 | 2048 | hip | 54 | 1.069e-08 | +| tp2_cp4 | 2048 | triton | 137 | 1.735e-08 | +| tp2_cp4 | 2048 | hip | 54 | 1.162e-08 | ## Figures diff --git a/benchmarks/results/pr328_rocm_mi300x/results.json b/benchmarks/results/pr328_rocm_mi300x/results.json index e9c27234..65b06afb 100644 --- a/benchmarks/results/pr328_rocm_mi300x/results.json +++ b/benchmarks/results/pr328_rocm_mi300x/results.json @@ -8,10 +8,10 @@ { "cp": 1, "forward": { - "max_ms": 0.5072727799415588, - "median_ms": 0.42468123137950897, - "min_ms": 0.4007662646472454, - "p95_ms": 0.5017554387450218 + "max_ms": 0.48768380656838417, + "median_ms": 0.4190076142549515, + "min_ms": 0.4007359966635704, + "p95_ms": 0.45819974038749933 }, "forward_peak_mib": 222.5634765625, "grad_finite": true, @@ -19,6 +19,7 @@ "logp_vs_fp64_max_abs": 1.2246337153243303e-06, "logp_vs_fp64_rel_l2": 3.358224717653215e-08, "lse_vs_fp64_max_abs": 1.2246337153243303e-06, + "mismatch_vs_reference": 14, "path": "native", "rel_l2_vs_reference": 1.4190225998557055e-08, "repeat_bitwise": true, @@ -28,20 +29,20 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 0.7819016464054585, - "median_ms": 0.703359255567193, - "min_ms": 0.6873710080981255, - "p95_ms": 0.769756268709898 + "max_ms": 0.7625329308211803, + "median_ms": 0.6916869897395372, + "min_ms": 0.6808820180594921, + "p95_ms": 0.7598829921334982 }, "train_peak_mib": 296.75732421875 }, { "cp": 1, "forward": { - "max_ms": 3.5062278620898724, - "median_ms": 3.322325646877289, - "min_ms": 3.264498896896839, - "p95_ms": 3.463823697529733 + "max_ms": 3.3959737047553062, + "median_ms": 3.283221973106265, + "min_ms": 3.2631871290504932, + "p95_ms": 3.336320770904422 }, "forward_peak_mib": 81.8603515625, "grad_finite": true, @@ -57,20 +58,51 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 4.906749352812767, - "median_ms": 4.854129860177636, - "min_ms": 4.76876413449645, - "p95_ms": 4.896933306008577 + "max_ms": 4.6745226718485355, + "median_ms": 4.359946120530367, + "min_ms": 4.28913114592433, + "p95_ms": 4.57268045283854 }, "train_peak_mib": 482.30419921875 }, { "cp": 1, "forward": { - "max_ms": 0.8099130354821682, - "median_ms": 0.7396885193884373, - "min_ms": 0.7042759098112583, - "p95_ms": 0.7940537761896849 + "max_ms": 0.7916060276329517, + "median_ms": 0.7639192044734955, + "min_ms": 0.7495642639696598, + "p95_ms": 0.7835091790184379 + }, + "forward_peak_mib": 0.37548828125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 9.623255152746424e-07, + "logp_vs_fp64_rel_l2": 3.1934274077238606e-08, + "lse_vs_fp64_max_abs": 9.623255152746424e-07, + "mismatch_vs_reference": 23, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.7002553523193623e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 256, + "topology": "tp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 2.098106313496828, + "median_ms": 1.546927960589528, + "min_ms": 1.3789492659270763, + "p95_ms": 1.8794237170368429 + }, + "train_peak_mib": 74.19873046875 + }, + { + "cp": 1, + "forward": { + "max_ms": 0.7618209347128868, + "median_ms": 0.7162289693951607, + "min_ms": 0.707410741597414, + "p95_ms": 0.7614219095557928 }, "forward_peak_mib": 0.37548828125, "grad_finite": true, @@ -88,20 +120,20 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.3461587950587273, - "median_ms": 1.1561571154743433, - "min_ms": 1.121405977755785, - "p95_ms": 1.2802621582522988 + "max_ms": 1.3348530046641827, + "median_ms": 1.1684747878462076, + "min_ms": 1.128236297518015, + "p95_ms": 1.3319233199581504 }, "train_peak_mib": 74.19873046875 }, { "cp": 1, "forward": { - "max_ms": 2.130434848368168, - "median_ms": 1.3161206152290106, - "min_ms": 1.290236134082079, - "p95_ms": 1.9846681272611024 + "max_ms": 1.3412716798484325, + "median_ms": 1.3117636553943157, + "min_ms": 1.2905574403703213, + "p95_ms": 1.3381803408265114 }, "forward_peak_mib": 1780.5078125, "grad_finite": true, @@ -109,6 +141,7 @@ "logp_vs_fp64_max_abs": 1.4523163969215602e-06, "logp_vs_fp64_rel_l2": 3.3102978989626796e-08, "lse_vs_fp64_max_abs": 1.3112566623618704e-06, + "mismatch_vs_reference": 88, "path": "native", "rel_l2_vs_reference": 1.261659473422803e-08, "repeat_bitwise": true, @@ -118,20 +151,20 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 2.777086105197668, - "median_ms": 2.7411116752773523, - "min_ms": 2.6900162920355797, - "p95_ms": 2.7740074321627617 + "max_ms": 3.034427296370268, + "median_ms": 2.72187776863575, + "min_ms": 2.6905969716608524, + "p95_ms": 2.901718509383499 }, "train_peak_mib": 2374.0498046875 }, { "cp": 1, "forward": { - "max_ms": 3.6533367820084095, - "median_ms": 3.5356120206415653, - "min_ms": 3.4904242493212223, - "p95_ms": 3.5971936769783497 + "max_ms": 3.6768210120499134, + "median_ms": 3.5876790061593056, + "min_ms": 3.5461867228150368, + "p95_ms": 3.63216248806566 }, "forward_peak_mib": 650.3125, "grad_finite": true, @@ -147,20 +180,51 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 7.222007028758526, - "median_ms": 7.127888035029173, - "min_ms": 7.024975959211588, - "p95_ms": 7.191663910634816 + "max_ms": 7.70503468811512, + "median_ms": 7.148197619244456, + "min_ms": 7.095188833773136, + "p95_ms": 7.534388988278806 }, "train_peak_mib": 3857.9091796875 }, { "cp": 1, "forward": { - "max_ms": 0.9156097657978535, - "median_ms": 0.843642745167017, - "min_ms": 0.8321460336446762, - "p95_ms": 0.8830241858959198 + "max_ms": 0.9566112421452999, + "median_ms": 0.8894458878785372, + "min_ms": 0.8791857399046421, + "p95_ms": 0.913844769820571 + }, + "forward_peak_mib": 3.001953125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.214240525949182e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 137, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.4538504826335258e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 2048, + "topology": "tp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.9067618995904922, + "median_ms": 1.7666991334408522, + "min_ms": 1.579316332936287, + "p95_ms": 1.8881176132708788 + }, + "train_peak_mib": 593.5771484375 + }, + { + "cp": 1, + "forward": { + "max_ms": 0.8942978456616402, + "median_ms": 0.8487105369567871, + "min_ms": 0.8369931019842625, + "p95_ms": 0.8818245492875576 }, "forward_peak_mib": 3.001953125, "grad_finite": true, @@ -178,20 +242,20 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.5859352424740791, - "median_ms": 1.5376345254480839, - "min_ms": 1.4071101322770119, - "p95_ms": 1.5810548793524504 + "max_ms": 1.7551067285239697, + "median_ms": 1.593762543052435, + "min_ms": 1.4162138104438782, + "p95_ms": 1.747589628212154 }, "train_peak_mib": 593.5771484375 }, { "cp": 1, "forward": { - "max_ms": 0.4723607562482357, - "median_ms": 0.4185016732662916, - "min_ms": 0.40345918387174606, - "p95_ms": 0.4652172327041626 + "max_ms": 0.4662228748202324, + "median_ms": 0.4294831305742264, + "min_ms": 0.41098007932305336, + "p95_ms": 0.46536598820239305 }, "forward_peak_mib": 111.2822265625, "grad_finite": true, @@ -199,6 +263,7 @@ "logp_vs_fp64_max_abs": 1.2246337153243303e-06, "logp_vs_fp64_rel_l2": 3.358954781712721e-08, "lse_vs_fp64_max_abs": 1.2246337153243303e-06, + "mismatch_vs_reference": 12, "path": "native", "rel_l2_vs_reference": 1.3632370710873585e-08, "repeat_bitwise": true, @@ -208,20 +273,20 @@ "tp": 4, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 0.8185659535229206, - "median_ms": 0.7887370884418488, - "min_ms": 0.7499642670154572, - "p95_ms": 0.8185255108401179 + "max_ms": 0.7736789993941784, + "median_ms": 0.7229440379887819, + "min_ms": 0.6758440285921097, + "p95_ms": 0.7698032073676586 }, "train_peak_mib": 148.38232421875 }, { "cp": 1, "forward": { - "max_ms": 2.1284017711877823, - "median_ms": 2.066188957542181, - "min_ms": 2.0460491068661213, - "p95_ms": 2.0982032641768456 + "max_ms": 2.1146913059055805, + "median_ms": 2.0956681109964848, + "min_ms": 2.0710560493171215, + "p95_ms": 2.1139870397746563 }, "forward_peak_mib": 44.40966796875, "grad_finite": true, @@ -237,20 +302,51 @@ "tp": 4, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 3.6968719214200974, - "median_ms": 3.3774820622056723, - "min_ms": 3.315543755888939, - "p95_ms": 3.6180405411869287 + "max_ms": 3.994573373347521, + "median_ms": 3.3612672705203295, + "min_ms": 3.3069918863475323, + "p95_ms": 3.754704259335994 }, "train_peak_mib": 241.158203125 }, { "cp": 1, "forward": { - "max_ms": 0.848931260406971, - "median_ms": 0.7611459586769342, - "min_ms": 0.7448559626936913, - "p95_ms": 0.7933490909636021 + "max_ms": 0.9129159152507782, + "median_ms": 0.8004489354789257, + "min_ms": 0.7914560846984386, + "p95_ms": 0.8500181371346117 + }, + "forward_peak_mib": 0.31298828125, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 9.623255152746424e-07, + "logp_vs_fp64_rel_l2": 3.1934274077238606e-08, + "lse_vs_fp64_max_abs": 9.623255152746424e-07, + "mismatch_vs_reference": 23, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.7002553523193623e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 256, + "topology": "tp4", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.6431300900876522, + "median_ms": 1.5929813962429762, + "min_ms": 1.4386768452823162, + "p95_ms": 1.631475891917944 + }, + "train_peak_mib": 37.10498046875 + }, + { + "cp": 1, + "forward": { + "max_ms": 1.531684771180153, + "median_ms": 0.7509302813559771, + "min_ms": 0.737345777451992, + "p95_ms": 0.8267639204859739 }, "forward_peak_mib": 0.31298828125, "grad_finite": true, @@ -268,20 +364,20 @@ "tp": 4, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.4106249436736107, - "median_ms": 1.3137920759618282, - "min_ms": 1.13757885992527, - "p95_ms": 1.3786639086902142 + "max_ms": 1.3830838724970818, + "median_ms": 1.2959898449480534, + "min_ms": 1.1800932697951794, + "p95_ms": 1.3811147538945079 }, "train_peak_mib": 37.10498046875 }, { "cp": 1, "forward": { - "max_ms": 0.9429808706045151, - "median_ms": 0.7553219329565763, - "min_ms": 0.7309648208320141, - "p95_ms": 0.926045305095613 + "max_ms": 0.769542995840311, + "median_ms": 0.7375199347734451, + "min_ms": 0.7233750075101852, + "p95_ms": 0.7685909979045391 }, "forward_peak_mib": 890.2578125, "grad_finite": true, @@ -289,6 +385,7 @@ "logp_vs_fp64_max_abs": 1.7274325472271812e-06, "logp_vs_fp64_rel_l2": 3.3315658627415676e-08, "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 105, "path": "native", "rel_l2_vs_reference": 1.322253230879159e-08, "repeat_bitwise": true, @@ -298,20 +395,20 @@ "tp": 4, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.5383749268949032, - "median_ms": 1.5194814186543226, - "min_ms": 1.4623119495809078, - "p95_ms": 1.537352125160396 + "max_ms": 2.801039721816778, + "median_ms": 1.5266572590917349, + "min_ms": 1.4295442961156368, + "p95_ms": 2.2766862995922557 }, "train_peak_mib": 1187.0498046875 }, { "cp": 1, "forward": { - "max_ms": 2.864705864340067, - "median_ms": 2.3683831095695496, - "min_ms": 2.306436188519001, - "p95_ms": 2.4618386523798113 + "max_ms": 2.3512630723416805, + "median_ms": 2.269135322421789, + "min_ms": 2.2396366111934185, + "p95_ms": 2.3080206010490656 }, "forward_peak_mib": 352.98681640625, "grad_finite": true, @@ -327,20 +424,51 @@ "tp": 4, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 4.653112031519413, - "median_ms": 4.408588400110602, - "min_ms": 4.365826025605202, - "p95_ms": 4.574884986504912 + "max_ms": 4.546563141047955, + "median_ms": 4.468842409551144, + "min_ms": 4.372625146061182, + "p95_ms": 4.538554861210287 }, "train_peak_mib": 1928.99462890625 }, { "cp": 1, "forward": { - "max_ms": 2.37527908757329, - "median_ms": 0.9374623186886311, - "min_ms": 0.9183650836348534, - "p95_ms": 1.0703487088903796 + "max_ms": 1.1006738059222698, + "median_ms": 0.9221145883202553, + "min_ms": 0.9061959572136402, + "p95_ms": 0.9652029955759646 + }, + "forward_peak_mib": 2.501953125, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.214240525949182e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 137, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.4538504826335258e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 2048, + "topology": "tp4", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 2.352474257349968, + "median_ms": 1.742522930726409, + "min_ms": 1.550281886011362, + "p95_ms": 2.1373696858063336 + }, + "train_peak_mib": 296.8271484375 + }, + { + "cp": 1, + "forward": { + "max_ms": 0.9811879135668278, + "median_ms": 0.8714443538337946, + "min_ms": 0.854909885674715, + "p95_ms": 0.9162158239632845 }, "forward_peak_mib": 2.501953125, "grad_finite": true, @@ -358,20 +486,20 @@ "tp": 4, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.5082103200256824, - "median_ms": 1.4016875065863132, - "min_ms": 1.3047782704234123, - "p95_ms": 1.507394341751933 + "max_ms": 1.6562901437282562, + "median_ms": 1.5001380816102028, + "min_ms": 1.3153338804841042, + "p95_ms": 1.6030208440497518 }, "train_peak_mib": 296.8271484375 }, { "cp": 1, "forward": { - "max_ms": 0.45378413051366806, - "median_ms": 0.39185211062431335, - "min_ms": 0.3761090338230133, - "p95_ms": 0.4420726094394922 + "max_ms": 0.4549059085547924, + "median_ms": 0.40683988481760025, + "min_ms": 0.3881859593093395, + "p95_ms": 0.4503392381593585 }, "forward_peak_mib": 55.6416015625, "grad_finite": true, @@ -379,6 +507,7 @@ "logp_vs_fp64_max_abs": 1.192615400213981e-06, "logp_vs_fp64_rel_l2": 3.3019515775307595e-08, "lse_vs_fp64_max_abs": 9.63235819995134e-07, + "mismatch_vs_reference": 16, "path": "native", "rel_l2_vs_reference": 1.2835542362061829e-08, "repeat_bitwise": true, @@ -388,20 +517,20 @@ "tp": 8, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 0.8295630104839802, - "median_ms": 0.7557619828730822, - "min_ms": 0.7264092564582825, - "p95_ms": 0.81349175889045 + "max_ms": 0.8490509353578091, + "median_ms": 0.797174172475934, + "min_ms": 0.7719169370830059, + "p95_ms": 0.8395010139793158 }, "train_peak_mib": 74.19482421875 }, { "cp": 1, "forward": { - "max_ms": 1.6495096497237682, - "median_ms": 1.6037765890359879, - "min_ms": 1.5587848611176014, - "p95_ms": 1.6448863549157977 + "max_ms": 1.7498787492513657, + "median_ms": 1.6366967465728521, + "min_ms": 1.585954800248146, + "p95_ms": 1.7218226799741387 }, "forward_peak_mib": 25.68408203125, "grad_finite": true, @@ -417,20 +546,51 @@ "tp": 8, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 3.228455316275358, - "median_ms": 2.7676159515976906, - "min_ms": 2.665730193257332, - "p95_ms": 3.08224034961313 + "max_ms": 2.874980214983225, + "median_ms": 2.7750616427510977, + "min_ms": 2.6837559416890144, + "p95_ms": 2.871970064006746 }, "train_peak_mib": 120.58544921875 }, { "cp": 1, "forward": { - "max_ms": 1.0527628473937511, - "median_ms": 0.8622754830867052, - "min_ms": 0.835892278701067, - "p95_ms": 0.9595537558197976 + "max_ms": 2.6385802775621414, + "median_ms": 0.9444185998290777, + "min_ms": 0.8997973054647446, + "p95_ms": 2.2007678402587776 + }, + "forward_peak_mib": 0.28173828125, + "grad_finite": true, + "local_vocab": 18992, + "logp_vs_fp64_max_abs": 9.623255152746424e-07, + "logp_vs_fp64_rel_l2": 3.1934274077238606e-08, + "lse_vs_fp64_max_abs": 9.623255152746424e-07, + "mismatch_vs_reference": 23, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.7002553523193623e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 256, + "topology": "tp8", + "tp": 8, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.8622861243784428, + "median_ms": 1.6909562982618809, + "min_ms": 1.5980438329279423, + "p95_ms": 1.8021621042862535 + }, + "train_peak_mib": 18.55810546875 + }, + { + "cp": 1, + "forward": { + "max_ms": 0.9400867857038975, + "median_ms": 0.8753803558647633, + "min_ms": 0.857312697917223, + "p95_ms": 0.9393630549311638 }, "forward_peak_mib": 0.28173828125, "grad_finite": true, @@ -448,20 +608,20 @@ "tp": 8, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.5623611398041248, - "median_ms": 1.4463732950389385, - "min_ms": 1.2542130425572395, - "p95_ms": 1.5456543304026127 + "max_ms": 1.6698800027370453, + "median_ms": 1.4559680130332708, + "min_ms": 1.2819040566682816, + "p95_ms": 1.6370807774364948 }, "train_peak_mib": 18.55810546875 }, { "cp": 1, "forward": { - "max_ms": 0.9086602367460728, - "median_ms": 0.504503957927227, - "min_ms": 0.47396402806043625, - "p95_ms": 0.5674065789207818 + "max_ms": 0.5170977674424648, + "median_ms": 0.4875045269727707, + "min_ms": 0.4633176140487194, + "p95_ms": 0.5150433629751205 }, "forward_peak_mib": 445.1328125, "grad_finite": true, @@ -469,6 +629,7 @@ "logp_vs_fp64_max_abs": 1.4523163969215602e-06, "logp_vs_fp64_rel_l2": 3.363677481837659e-08, "lse_vs_fp64_max_abs": 1.3207212923305178e-06, + "mismatch_vs_reference": 151, "path": "native", "rel_l2_vs_reference": 1.6072921840575048e-08, "repeat_bitwise": true, @@ -478,20 +639,20 @@ "tp": 8, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 3.279380965977907, - "median_ms": 1.0833695996552706, - "min_ms": 0.8646342903375626, - "p95_ms": 2.7472038753330694 + "max_ms": 1.050389837473631, + "median_ms": 0.9598308242857456, + "min_ms": 0.9048338979482651, + "p95_ms": 1.0330617194995284 }, "train_peak_mib": 593.5498046875 }, { "cp": 1, "forward": { - "max_ms": 1.9366671331226826, - "median_ms": 1.7633400857448578, - "min_ms": 1.7227185890078545, - "p95_ms": 1.9199213478714228 + "max_ms": 1.8442091532051563, + "median_ms": 1.8065536860376596, + "min_ms": 1.7527742311358452, + "p95_ms": 1.8355898559093475 }, "forward_peak_mib": 204.32373046875, "grad_finite": true, @@ -507,20 +668,51 @@ "tp": 8, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 3.7340661510825157, - "median_ms": 3.3381988760083914, - "min_ms": 3.2053212635219097, - "p95_ms": 3.7182031897827983 + "max_ms": 4.241500049829483, + "median_ms": 3.357867943122983, + "min_ms": 3.2875332981348038, + "p95_ms": 3.904002718627452 }, "train_peak_mib": 964.537109375 }, { "cp": 1, "forward": { - "max_ms": 1.1442299000918865, - "median_ms": 1.0406605433672667, - "min_ms": 0.988238025456667, - "p95_ms": 1.130919624119997 + "max_ms": 1.1981800198554993, + "median_ms": 1.1148559860885143, + "min_ms": 1.0922616347670555, + "p95_ms": 1.1961441952735186 + }, + "forward_peak_mib": 2.251953125, + "grad_finite": true, + "local_vocab": 18992, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.214240525949182e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 137, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.4538504826335258e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 2048, + "topology": "tp8", + "tp": 8, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.9532912410795689, + "median_ms": 1.8464724998921156, + "min_ms": 1.7132037319242954, + "p95_ms": 1.9464143086224794 + }, + "train_peak_mib": 148.4521484375 + }, + { + "cp": 1, + "forward": { + "max_ms": 1.148466020822525, + "median_ms": 1.0631137993186712, + "min_ms": 1.042358111590147, + "p95_ms": 1.128039206378162 }, "forward_peak_mib": 2.251953125, "grad_finite": true, @@ -538,20 +730,20 @@ "tp": 8, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 2.4306410923600197, - "median_ms": 1.6933211591094732, - "min_ms": 1.5776939690113068, - "p95_ms": 2.211051783524453 + "max_ms": 1.8442701548337936, + "median_ms": 1.6416581347584724, + "min_ms": 1.5701819211244583, + "p95_ms": 1.806598319672048 }, "train_peak_mib": 148.4521484375 }, { "cp": 2, "forward": { - "max_ms": 0.9187641553580761, - "median_ms": 0.4205044824630022, - "min_ms": 0.40137581527233124, - "p95_ms": 0.5561909638345245 + "max_ms": 0.9105419740080833, + "median_ms": 0.42372941970825195, + "min_ms": 0.4016370512545109, + "p95_ms": 0.5494242068380121 }, "forward_peak_mib": 111.28173828125, "grad_finite": true, @@ -559,6 +751,7 @@ "logp_vs_fp64_max_abs": 1.2246337153243303e-06, "logp_vs_fp64_rel_l2": 3.5790710063228845e-08, "lse_vs_fp64_max_abs": 1.2246337153243303e-06, + "mismatch_vs_reference": 14, "path": "native", "rel_l2_vs_reference": 1.8103588949760993e-08, "repeat_bitwise": true, @@ -568,20 +761,20 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 0.7929271087050438, - "median_ms": 0.7523719687014818, - "min_ms": 0.7147910073399544, - "p95_ms": 0.7845988497138023 + "max_ms": 0.7477812469005585, + "median_ms": 0.6717976648360491, + "min_ms": 0.6506461650133133, + "p95_ms": 0.7439276669174433 }, "train_peak_mib": 148.37939453125 }, { "cp": 2, "forward": { - "max_ms": 3.39422095566988, - "median_ms": 3.2698858994990587, - "min_ms": 3.200352657586336, - "p95_ms": 3.3097355626523495 + "max_ms": 3.271800000220537, + "median_ms": 3.2116554211825132, + "min_ms": 3.1774090602993965, + "p95_ms": 3.26847021933645 }, "forward_peak_mib": 41.2568359375, "grad_finite": true, @@ -597,20 +790,51 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 4.5003630220890045, - "median_ms": 4.432037007063627, - "min_ms": 4.371511749923229, - "p95_ms": 4.489488760009408 + "max_ms": 4.6104080975055695, + "median_ms": 4.5303895603865385, + "min_ms": 4.495627712458372, + "p95_ms": 4.583457973785698 }, "train_peak_mib": 241.18994140625 }, { "cp": 2, "forward": { - "max_ms": 1.3963528908789158, - "median_ms": 0.7118615321815014, - "min_ms": 0.6896951235830784, - "p95_ms": 1.39130181632936 + "max_ms": 0.7943799719214439, + "median_ms": 0.7632041815668344, + "min_ms": 0.7462790235877037, + "p95_ms": 0.7903264602646232 + }, + "forward_peak_mib": 0.18798828125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 9.623255152746424e-07, + "logp_vs_fp64_rel_l2": 3.4143571581866415e-08, + "lse_vs_fp64_max_abs": 9.623255152746424e-07, + "mismatch_vs_reference": 23, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.920175576528477e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 128, + "topology": "tp2_cp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.5599783509969711, + "median_ms": 1.5244900714606047, + "min_ms": 1.417235005646944, + "p95_ms": 1.5520641580224037 + }, + "train_peak_mib": 37.1005859375 + }, + { + "cp": 2, + "forward": { + "max_ms": 0.7526082918047905, + "median_ms": 0.7347017526626587, + "min_ms": 0.6997189484536648, + "p95_ms": 0.748821534216404 }, "forward_peak_mib": 0.18798828125, "grad_finite": true, @@ -628,20 +852,20 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.4072302728891373, - "median_ms": 1.2146991211920977, - "min_ms": 1.0791430249810219, - "p95_ms": 1.3531540054827926 + "max_ms": 2.083924598991871, + "median_ms": 1.2248244602233171, + "min_ms": 1.148295123130083, + "p95_ms": 2.082325122319162 }, "train_peak_mib": 37.1005859375 }, { "cp": 2, "forward": { - "max_ms": 1.6678068786859512, - "median_ms": 0.7275603711605072, - "min_ms": 0.6844867020845413, - "p95_ms": 1.4014774933457377 + "max_ms": 0.7630637846887112, + "median_ms": 0.7253275252878666, + "min_ms": 0.695972703397274, + "p95_ms": 0.7621122291311622 }, "forward_peak_mib": 890.25390625, "grad_finite": true, @@ -649,6 +873,7 @@ "logp_vs_fp64_max_abs": 1.4523163969215602e-06, "logp_vs_fp64_rel_l2": 3.322330169050237e-08, "lse_vs_fp64_max_abs": 1.3112566623618704e-06, + "mismatch_vs_reference": 88, "path": "native", "rel_l2_vs_reference": 1.32910346411907e-08, "repeat_bitwise": true, @@ -658,20 +883,20 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.4916150830686092, - "median_ms": 1.4505994040519, - "min_ms": 1.4055166393518448, - "p95_ms": 1.4872347004711628 + "max_ms": 1.48736871778965, + "median_ms": 1.447330228984356, + "min_ms": 1.4351606369018555, + "p95_ms": 1.4782787533476949 }, "train_peak_mib": 1187.025390625 }, { "cp": 2, "forward": { - "max_ms": 5.079873837530613, - "median_ms": 3.362047951668501, - "min_ms": 3.308674320578575, - "p95_ms": 3.7359460489824423 + "max_ms": 3.6023613065481186, + "median_ms": 3.3882129937410355, + "min_ms": 3.35833802819252, + "p95_ms": 3.5074760438874364 }, "forward_peak_mib": 325.482421875, "grad_finite": true, @@ -687,20 +912,51 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 5.5114347487688065, - "median_ms": 5.449994234368205, - "min_ms": 5.38027985021472, - "p95_ms": 5.4998258128762245 + "max_ms": 5.740798078477383, + "median_ms": 5.609216867014766, + "min_ms": 5.572086665779352, + "p95_ms": 5.734217865392566 }, "train_peak_mib": 1928.99169921875 }, { "cp": 2, "forward": { - "max_ms": 0.8522761054337025, - "median_ms": 0.7730736397206783, - "min_ms": 0.7586758583784103, - "p95_ms": 0.830516847781837 + "max_ms": 1.2611830607056618, + "median_ms": 0.8307832758873701, + "min_ms": 0.809862744063139, + "p95_ms": 0.960499467328191 + }, + "forward_peak_mib": 1.5009765625, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.238843099724084e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 137, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.4772159104804024e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 1024, + "topology": "tp2_cp2", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.7547458410263062, + "median_ms": 1.4580159913748503, + "min_ms": 1.441081054508686, + "p95_ms": 1.6999759711325169 + }, + "train_peak_mib": 296.7890625 + }, + { + "cp": 2, + "forward": { + "max_ms": 0.8257459849119186, + "median_ms": 0.7809000089764595, + "min_ms": 0.7658181712031364, + "p95_ms": 0.817440333776176 }, "forward_peak_mib": 1.5009765625, "grad_finite": true, @@ -718,20 +974,20 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.788655761629343, - "median_ms": 1.2361111585050821, - "min_ms": 1.2153349816799164, - "p95_ms": 1.6391555080190297 + "max_ms": 1.6649230383336544, + "median_ms": 1.3925188686698675, + "min_ms": 1.26316724345088, + "p95_ms": 1.6368331853300333 }, "train_peak_mib": 296.7890625 }, { "cp": 2, "forward": { - "max_ms": 0.9403661824762821, - "median_ms": 0.4830381367355585, - "min_ms": 0.45273173600435257, - "p95_ms": 0.5635019624605778 + "max_ms": 0.9470968507230282, + "median_ms": 0.4659618716686964, + "min_ms": 0.4259929992258549, + "p95_ms": 0.6475532660260799 }, "forward_peak_mib": 55.64111328125, "grad_finite": true, @@ -739,6 +995,7 @@ "logp_vs_fp64_max_abs": 1.2246337153243303e-06, "logp_vs_fp64_rel_l2": 3.573556089056118e-08, "lse_vs_fp64_max_abs": 1.2246337153243303e-06, + "mismatch_vs_reference": 11, "path": "native", "rel_l2_vs_reference": 1.693435683024808e-08, "repeat_bitwise": true, @@ -748,20 +1005,20 @@ "tp": 4, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.6028108075261116, - "median_ms": 0.9448430500924587, - "min_ms": 0.8031027391552925, - "p95_ms": 1.380775030702352 + "max_ms": 0.8712639100849628, + "median_ms": 0.7841994520276785, + "min_ms": 0.7658470422029495, + "p95_ms": 0.8438089862465858 }, "train_peak_mib": 74.19189453125 }, { "cp": 2, "forward": { - "max_ms": 2.4561178870499134, - "median_ms": 2.218499779701233, - "min_ms": 2.147639635950327, - "p95_ms": 2.4312858702614903 + "max_ms": 2.360045909881592, + "median_ms": 2.1834224462509155, + "min_ms": 2.1604690700769424, + "p95_ms": 2.347819460555911 }, "forward_peak_mib": 22.36865234375, "grad_finite": true, @@ -777,20 +1034,51 @@ "tp": 4, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 3.5232328809797764, - "median_ms": 3.4358128905296326, - "min_ms": 3.351978026330471, - "p95_ms": 3.516418603248894 + "max_ms": 3.557675052434206, + "median_ms": 3.459763713181019, + "min_ms": 3.397907130420208, + "p95_ms": 3.5327076679095626 }, "train_peak_mib": 120.59912109375 }, { "cp": 2, "forward": { - "max_ms": 1.11051881685853, - "median_ms": 0.7829479873180389, - "min_ms": 0.7545412518084049, - "p95_ms": 0.8756248280405999 + "max_ms": 0.9298510849475861, + "median_ms": 0.8165778126567602, + "min_ms": 0.7963827811181545, + "p95_ms": 0.8834703825414181 + }, + "forward_peak_mib": 0.15673828125, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 9.623255152746424e-07, + "logp_vs_fp64_rel_l2": 3.4143571581866415e-08, + "lse_vs_fp64_max_abs": 9.623255152746424e-07, + "mismatch_vs_reference": 23, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.920175576528477e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 128, + "topology": "tp4_cp2", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.890428364276886, + "median_ms": 1.6159303486347198, + "min_ms": 1.582070253789425, + "p95_ms": 1.8853715620934963 + }, + "train_peak_mib": 18.5537109375 + }, + { + "cp": 2, + "forward": { + "max_ms": 1.038712915033102, + "median_ms": 0.7646705489605665, + "min_ms": 0.7514357566833496, + "p95_ms": 0.8226943202316763 }, "forward_peak_mib": 0.15673828125, "grad_finite": true, @@ -808,20 +1096,20 @@ "tp": 4, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.538364216685295, - "median_ms": 1.3768249191343784, - "min_ms": 1.2114690616726875, - "p95_ms": 1.510734111070633 + "max_ms": 1.5644128434360027, + "median_ms": 1.407951582223177, + "min_ms": 1.3164947740733624, + "p95_ms": 1.5445253113284707 }, "train_peak_mib": 18.5537109375 }, { "cp": 2, "forward": { - "max_ms": 0.5847080610692501, - "median_ms": 0.4740641452372074, - "min_ms": 0.4566791467368603, - "p95_ms": 0.5416566971689464 + "max_ms": 1.428872812539339, + "median_ms": 0.49603707157075405, + "min_ms": 0.45394524931907654, + "p95_ms": 0.6215794477611786 }, "forward_peak_mib": 445.12890625, "grad_finite": true, @@ -829,6 +1117,7 @@ "logp_vs_fp64_max_abs": 1.7274325472271812e-06, "logp_vs_fp64_rel_l2": 3.337932457263568e-08, "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 101, "path": "native", "rel_l2_vs_reference": 1.386501020113035e-08, "repeat_bitwise": true, @@ -838,20 +1127,20 @@ "tp": 4, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.1372589506208897, - "median_ms": 0.9921537712216377, - "min_ms": 0.900637824088335, - "p95_ms": 1.1346454732120037 + "max_ms": 1.417775172740221, + "median_ms": 1.0220431722700596, + "min_ms": 0.8883103728294373, + "p95_ms": 1.257734047248959 }, "train_peak_mib": 593.525390625 }, { "cp": 2, "forward": { - "max_ms": 2.624067012220621, - "median_ms": 2.2469221148639917, - "min_ms": 2.2091111168265343, - "p95_ms": 2.5062156841158867 + "max_ms": 2.2686789743602276, + "median_ms": 2.2382440511137247, + "min_ms": 2.197904046624899, + "p95_ms": 2.263028477318585 }, "forward_peak_mib": 176.65673828125, "grad_finite": true, @@ -867,20 +1156,51 @@ "tp": 4, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 4.030606243759394, - "median_ms": 3.79836093634367, - "min_ms": 3.7597129121422768, - "p95_ms": 3.9899602998048067 + "max_ms": 3.9137532003223896, + "median_ms": 3.7863540928810835, + "min_ms": 3.763199783861637, + "p95_ms": 3.889885521493852 }, "train_peak_mib": 964.51611328125 }, { "cp": 2, "forward": { - "max_ms": 0.9333761408925056, - "median_ms": 0.8380450308322906, - "min_ms": 0.8166632615029812, - "p95_ms": 0.9284091182053089 + "max_ms": 1.006274949759245, + "median_ms": 0.8831163868308067, + "min_ms": 0.8671479299664497, + "p95_ms": 0.9622140787541866 + }, + "forward_peak_mib": 1.2509765625, + "grad_finite": true, + "local_vocab": 37984, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.238843099724084e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 137, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.4772159104804024e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 1024, + "topology": "tp4_cp2", + "tp": 4, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.7355279996991158, + "median_ms": 1.6802609898149967, + "min_ms": 1.5333471819758415, + "p95_ms": 1.7273348988965154 + }, + "train_peak_mib": 148.4140625 + }, + { + "cp": 2, + "forward": { + "max_ms": 0.9049340151250362, + "median_ms": 0.838590320199728, + "min_ms": 0.8162329904735088, + "p95_ms": 0.8977984543889761 }, "forward_peak_mib": 1.2509765625, "grad_finite": true, @@ -898,20 +1218,20 @@ "tp": 4, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.585574820637703, - "median_ms": 1.3963484670966864, - "min_ms": 1.2736623175442219, - "p95_ms": 1.5687244711443782 + "max_ms": 1.4411099255084991, + "median_ms": 1.402924070134759, + "min_ms": 1.2808619067072868, + "p95_ms": 1.4394335448741913 }, "train_peak_mib": 148.4140625 }, { "cp": 4, "forward": { - "max_ms": 0.9446828626096249, - "median_ms": 0.46018860302865505, - "min_ms": 0.4228181205689907, - "p95_ms": 0.5898057250306012 + "max_ms": 0.9297211654484272, + "median_ms": 0.4601990804076195, + "min_ms": 0.4170001484453678, + "p95_ms": 0.5840539000928404 }, "forward_peak_mib": 55.64111328125, "grad_finite": true, @@ -919,6 +1239,7 @@ "logp_vs_fp64_max_abs": 1.2246337153243303e-06, "logp_vs_fp64_rel_l2": 3.730282272543457e-08, "lse_vs_fp64_max_abs": 1.2246337153243303e-06, + "mismatch_vs_reference": 14, "path": "native", "rel_l2_vs_reference": 2.3950332323559945e-08, "repeat_bitwise": true, @@ -928,20 +1249,20 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 0.84735918790102, - "median_ms": 0.8005395065993071, - "min_ms": 0.7707341574132442, - "p95_ms": 0.8399948477745056 + "max_ms": 1.0157087817788124, + "median_ms": 0.8255115244537592, + "min_ms": 0.7808287627995014, + "p95_ms": 0.986437709070742 }, "train_peak_mib": 74.19140625 }, { "cp": 4, "forward": { - "max_ms": 3.747745882719755, - "median_ms": 3.4810248762369156, - "min_ms": 3.437324892729521, - "p95_ms": 3.573427628725767 + "max_ms": 3.438376821577549, + "median_ms": 3.382489550858736, + "min_ms": 3.308103885501623, + "p95_ms": 3.4154762281104922 }, "forward_peak_mib": 20.970703125, "grad_finite": true, @@ -957,20 +1278,51 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 4.93944576010108, - "median_ms": 4.78887720964849, - "min_ms": 4.680632147938013, - "p95_ms": 4.929828573949635 + "max_ms": 5.002471152693033, + "median_ms": 4.629456205293536, + "min_ms": 4.59907203912735, + "p95_ms": 4.858391894958913 }, "train_peak_mib": 120.63427734375 }, { "cp": 4, "forward": { - "max_ms": 1.1552250944077969, - "median_ms": 0.7531433366239071, - "min_ms": 0.723003875464201, - "p95_ms": 1.0587618686258795 + "max_ms": 1.042709220200777, + "median_ms": 0.768306665122509, + "min_ms": 0.7598591037094593, + "p95_ms": 0.9768236195668579 + }, + "forward_peak_mib": 0.09423828125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 9.623255152746424e-07, + "logp_vs_fp64_rel_l2": 3.534562677873104e-08, + "lse_vs_fp64_max_abs": 9.623255152746424e-07, + "mismatch_vs_reference": 23, + "path": "ws2-triton", + "rel_l2_vs_reference": 2.2173699061384028e-08, + "repeat_bitwise": true, + "tokens": 256, + "tokens_per_cp_rank": 64, + "topology": "tp2_cp4", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.9474220462143421, + "median_ms": 1.5305932611227036, + "min_ms": 1.3936497271060944, + "p95_ms": 1.927389926277101 + }, + "train_peak_mib": 18.552734375 + }, + { + "cp": 4, + "forward": { + "max_ms": 1.0852618142962456, + "median_ms": 0.7238299585878849, + "min_ms": 0.7138201035559177, + "p95_ms": 0.788106629624963 }, "forward_peak_mib": 0.09423828125, "grad_finite": true, @@ -988,20 +1340,20 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.505616120994091, - "median_ms": 1.3656336814165115, - "min_ms": 1.260161865502596, - "p95_ms": 1.4661509776487947 + "max_ms": 1.569060143083334, + "median_ms": 1.2379284016788006, + "min_ms": 1.1319010518491268, + "p95_ms": 1.5026589157059789 }, "train_peak_mib": 18.552734375 }, { "cp": 4, "forward": { - "max_ms": 0.5427957512438297, - "median_ms": 0.4715598188340664, - "min_ms": 0.4449011757969856, - "p95_ms": 0.5398092558607459 + "max_ms": 0.5426062270998955, + "median_ms": 0.4625171422958374, + "min_ms": 0.43542683124542236, + "p95_ms": 0.5310371518135071 }, "forward_peak_mib": 445.126953125, "grad_finite": true, @@ -1009,6 +1361,7 @@ "logp_vs_fp64_max_abs": 1.4523163969215602e-06, "logp_vs_fp64_rel_l2": 3.339807416706557e-08, "lse_vs_fp64_max_abs": 1.3112566623618704e-06, + "mismatch_vs_reference": 88, "path": "native", "rel_l2_vs_reference": 1.368420826566803e-08, "repeat_bitwise": true, @@ -1018,20 +1371,20 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 1.897196751087904, - "median_ms": 1.007936429232359, - "min_ms": 0.9225797839462757, - "p95_ms": 1.5796012012287965 + "max_ms": 1.1636787094175816, + "median_ms": 0.9963898919522762, + "min_ms": 0.9112842381000519, + "p95_ms": 1.1164123192429543 }, "train_peak_mib": 593.51318359375 }, { "cp": 4, "forward": { - "max_ms": 4.713290371000767, - "median_ms": 3.5901865921914577, - "min_ms": 3.531333990395069, - "p95_ms": 3.9379455614835033 + "max_ms": 3.6994353868067265, + "median_ms": 3.526117419824004, + "min_ms": 3.4705549478530884, + "p95_ms": 3.6070818547159433 }, "forward_peak_mib": 163.0673828125, "grad_finite": true, @@ -1047,20 +1400,51 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 5.251959897577763, - "median_ms": 5.043254466727376, - "min_ms": 4.988599102944136, - "p95_ms": 5.244744545780122 + "max_ms": 5.049319937825203, + "median_ms": 4.9699717201292515, + "min_ms": 4.881730768829584, + "p95_ms": 5.038278875872493 }, "train_peak_mib": 964.53271484375 }, { "cp": 4, "forward": { - "max_ms": 0.8831419982016087, - "median_ms": 0.7803190965205431, - "min_ms": 0.7526376284658909, - "p95_ms": 0.8549518883228303 + "max_ms": 1.389303244650364, + "median_ms": 0.8240439929068089, + "min_ms": 0.7940488867461681, + "p95_ms": 1.3511136174201965 + }, + "forward_peak_mib": 0.75048828125, + "grad_finite": true, + "local_vocab": 75968, + "logp_vs_fp64_max_abs": 1.4523163969215602e-06, + "logp_vs_fp64_rel_l2": 3.277486808560087e-08, + "lse_vs_fp64_max_abs": 1.2623544449752444e-06, + "mismatch_vs_reference": 137, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.7349402400487287e-08, + "repeat_bitwise": true, + "tokens": 2048, + "tokens_per_cp_rank": 512, + "topology": "tp2_cp4", + "tp": 2, + "tp_replicated": true, + "train_fwd_bwd": { + "max_ms": 1.657932996749878, + "median_ms": 1.5610686969012022, + "min_ms": 1.4730580151081085, + "p95_ms": 1.649744505994022 + }, + "train_peak_mib": 148.39501953125 + }, + { + "cp": 4, + "forward": { + "max_ms": 0.8596866391599178, + "median_ms": 0.754661625251174, + "min_ms": 0.7468885742127895, + "p95_ms": 0.7833839161321521 }, "forward_peak_mib": 0.75048828125, "grad_finite": true, @@ -1078,10 +1462,10 @@ "tp": 2, "tp_replicated": true, "train_fwd_bwd": { - "max_ms": 2.1002087742090225, - "median_ms": 1.384851522743702, - "min_ms": 1.292981207370758, - "p95_ms": 1.9774960353970525 + "max_ms": 1.5739467926323414, + "median_ms": 1.3958783820271492, + "min_ms": 1.319149974733591, + "p95_ms": 1.5368028078228235 }, "train_peak_mib": 148.39501953125 } @@ -1089,12 +1473,11 @@ "environment": { "architecture": "gfx942:sramecc+:xnack-", "extension_symbols": "hip_deterministic_logp_tile_stats, hip_deterministic_logp_backward", - "git_commit": "1c73a427881252871b46936f6d14be36a22e307f", + "git_commit": "e9f1d2a5c67a283bd987978614b4444a9416c1f0", "gpu": "AMD Instinct MI300X", "gpu_count": 8, "hip": "7.14.60850", "native_collective": "torch.distributed ProcessGroupNCCL (RCCL on ROCm)", - "native_single_gpu": "torch.logsumexp + gather on FP32 logits", "python": "3.12.3", "torch": "2.12.0+rocm7.14.0a20260608" }, @@ -1104,10 +1487,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.19532999396324158, - "median_ms": 0.11733400076627731, - "min_ms": 0.11172600090503693, - "p95_ms": 0.154381200671196 + "max_ms": 0.20935100317001343, + "median_ms": 0.12019799649715424, + "min_ms": 0.11268699914216995, + "p95_ms": 0.16653735563158992 }, "forward_peak_mib": 1.1611328125, "grad_finite": true, @@ -1123,10 +1506,10 @@ "repeat_bitwise": true, "tokens": 1, "train_fwd_bwd": { - "max_ms": 0.46933698654174805, - "median_ms": 0.39827150106430054, - "min_ms": 0.3791230022907257, - "p95_ms": 0.45484338849782946 + "max_ms": 0.4332039952278137, + "median_ms": 0.3826484978199005, + "min_ms": 0.3718720078468323, + "p95_ms": 0.42406404614448545 }, "train_peak_mib": 3.1904296875 }, @@ -1134,10 +1517,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.2908720076084137, - "median_ms": 0.19056300073862076, - "min_ms": 0.1811089962720871, - "p95_ms": 0.28451651185750965 + "max_ms": 0.28041699528694153, + "median_ms": 0.18904100358486176, + "min_ms": 0.18267099559307098, + "p95_ms": 0.2714727446436882 }, "forward_peak_mib": 1.744140625, "grad_finite": true, @@ -1153,10 +1536,10 @@ "repeat_bitwise": true, "tokens": 1, "train_fwd_bwd": { - "max_ms": 1.1102089881896973, - "median_ms": 1.0265445113182068, - "min_ms": 0.9868659973144531, - "p95_ms": 1.0943811655044555 + "max_ms": 0.693149983882904, + "median_ms": 0.5962255001068115, + "min_ms": 0.5884339809417725, + "p95_ms": 0.6668128371238708 }, "train_peak_mib": 2.03466796875 }, @@ -1164,10 +1547,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.5845879912376404, - "median_ms": 0.5019449889659882, - "min_ms": 0.4906879961490631, - "p95_ms": 0.5795643866062165 + "max_ms": 0.3535250127315521, + "median_ms": 0.30254900455474854, + "min_ms": 0.29940399527549744, + "p95_ms": 0.3455716043710709 }, "forward_peak_mib": 0.00341796875, "grad_finite": true, @@ -1183,10 +1566,10 @@ "repeat_bitwise": true, "tokens": 1, "train_fwd_bwd": { - "max_ms": 0.630295991897583, - "median_ms": 0.5488154888153076, - "min_ms": 0.5224959850311279, - "p95_ms": 0.6256272315979003 + "max_ms": 0.3893379867076874, + "median_ms": 0.3491179943084717, + "min_ms": 0.34563401341438293, + "p95_ms": 0.3789542436599731 }, "train_peak_mib": 0.5830078125 }, @@ -1194,10 +1577,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 5.795327186584473, - "median_ms": 5.484604835510254, - "min_ms": 5.387600898742676, - "p95_ms": 5.649418306350708 + "max_ms": 5.050661087036133, + "median_ms": 4.9217095375061035, + "min_ms": 4.765758037567139, + "p95_ms": 5.02729377746582 }, "forward_peak_mib": 1.96630859375, "grad_finite": true, @@ -1213,10 +1596,10 @@ "repeat_bitwise": true, "tokens": 1, "train_fwd_bwd": { - "max_ms": 6.590147972106934, - "median_ms": 6.508185386657715, - "min_ms": 6.449540138244629, - "p95_ms": 6.570390820503235 + "max_ms": 5.651031970977783, + "median_ms": 5.608389139175415, + "min_ms": 5.570312976837158, + "p95_ms": 5.641712045669555 }, "train_peak_mib": 3.91943359375 }, @@ -1224,10 +1607,42 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.4892059862613678, - "median_ms": 0.3573305010795593, - "min_ms": 0.339944988489151, - "p95_ms": 0.4387818902730942 + "max_ms": 0.453794002532959, + "median_ms": 0.39360450208187103, + "min_ms": 0.38184699416160583, + "p95_ms": 0.4045488551259041 + }, + "forward_peak_mib": 0.005859375, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 5.384272654396227e-07, + "relative_l2": 3.365204219985256e-08 + }, + "lse_vs_fp64": { + "max_abs": 5.384272654396227e-07, + "relative_l2": 3.8718812743187373e-08 + }, + "mismatch_vs_reference": 0, + "path": "ws2-triton", + "rel_l2_vs_reference": 0.0, + "repeat_bitwise": true, + "tokens": 1, + "train_fwd_bwd": { + "max_ms": 0.8825510144233704, + "median_ms": 0.7837440073490143, + "min_ms": 0.7605699896812439, + "p95_ms": 0.8616399496793746 + }, + "train_peak_mib": 0.5859375 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.3889780044555664, + "median_ms": 0.34681499004364014, + "min_ms": 0.33045101165771484, + "p95_ms": 0.3638219863176346 }, "forward_peak_mib": 0.005859375, "grad_finite": true, @@ -1245,10 +1660,10 @@ "repeat_bitwise": true, "tokens": 1, "train_fwd_bwd": { - "max_ms": 0.8597570061683655, - "median_ms": 0.7872090041637421, - "min_ms": 0.7644150257110596, - "p95_ms": 0.8305177897214888 + "max_ms": 0.6693540215492249, + "median_ms": 0.6335409879684448, + "min_ms": 0.627170979976654, + "p95_ms": 0.6575284570455551 }, "train_peak_mib": 0.5859375 }, @@ -1256,10 +1671,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.18343199789524078, - "median_ms": 0.13664299249649048, - "min_ms": 0.13035400211811066, - "p95_ms": 0.14256014302372938 + "max_ms": 0.15094399452209473, + "median_ms": 0.12752900272607803, + "min_ms": 0.12198100239038467, + "p95_ms": 0.14850819632411003 }, "forward_peak_mib": 9.27685546875, "grad_finite": true, @@ -1275,10 +1690,10 @@ "repeat_bitwise": true, "tokens": 8, "train_fwd_bwd": { - "max_ms": 0.5676829814910889, - "median_ms": 0.5142639875411987, - "min_ms": 0.5031470060348511, - "p95_ms": 0.5522160500288009 + "max_ms": 0.4181010127067566, + "median_ms": 0.38661450147628784, + "min_ms": 0.3778409957885742, + "p95_ms": 0.41184600740671157 }, "train_peak_mib": 25.50439453125 }, @@ -1286,10 +1701,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.24724699556827545, - "median_ms": 0.1849144995212555, - "min_ms": 0.18062900006771088, - "p95_ms": 0.21668834313750268 + "max_ms": 0.22136899828910828, + "median_ms": 0.186537005007267, + "min_ms": 0.1813099980354309, + "p95_ms": 0.20447135344147685 }, "forward_peak_mib": 13.91552734375, "grad_finite": true, @@ -1305,10 +1720,10 @@ "repeat_bitwise": true, "tokens": 8, "train_fwd_bwd": { - "max_ms": 0.9568219780921936, - "median_ms": 0.8782845139503479, - "min_ms": 0.836562991142273, - "p95_ms": 0.941102597117424 + "max_ms": 0.7002000212669373, + "median_ms": 0.6067410111427307, + "min_ms": 0.5904369950294495, + "p95_ms": 0.6935661137104034 }, "train_peak_mib": 16.234375 }, @@ -1316,10 +1731,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.3518419861793518, - "median_ms": 0.3047724962234497, - "min_ms": 0.30160799622535706, - "p95_ms": 0.31199708580970764 + "max_ms": 0.3513219952583313, + "median_ms": 0.3033300042152405, + "min_ms": 0.29960501194000244, + "p95_ms": 0.30980129539966583 }, "forward_peak_mib": 0.00341796875, "grad_finite": true, @@ -1335,10 +1750,10 @@ "repeat_bitwise": true, "tokens": 8, "train_fwd_bwd": { - "max_ms": 0.5473729968070984, - "median_ms": 0.4635085016489029, - "min_ms": 0.45010900497436523, - "p95_ms": 0.5311666876077652 + "max_ms": 0.37147098779678345, + "median_ms": 0.3569300025701523, + "min_ms": 0.3475959897041321, + "p95_ms": 0.3688393920660019 }, "train_peak_mib": 4.6396484375 }, @@ -1346,10 +1761,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 6.244194030761719, - "median_ms": 6.082473516464233, - "min_ms": 5.9791998863220215, - "p95_ms": 6.183037853240966 + "max_ms": 5.572154998779297, + "median_ms": 5.4453675746917725, + "min_ms": 5.37157678604126, + "p95_ms": 5.541255283355713 }, "forward_peak_mib": 6.22314453125, "grad_finite": true, @@ -1365,10 +1780,10 @@ "repeat_bitwise": true, "tokens": 8, "train_fwd_bwd": { - "max_ms": 7.186433792114258, - "median_ms": 7.1299498081207275, - "min_ms": 7.041217803955078, - "p95_ms": 7.173869466781616 + "max_ms": 6.1761322021484375, + "median_ms": 6.16493558883667, + "min_ms": 6.118648052215576, + "p95_ms": 6.175754117965699 }, "train_peak_mib": 30.29052734375 }, @@ -1376,10 +1791,42 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.43849000334739685, - "median_ms": 0.35905349254608154, - "min_ms": 0.3502410054206848, - "p95_ms": 0.39031168818473816 + "max_ms": 0.43841099739074707, + "median_ms": 0.3986324965953827, + "min_ms": 0.3909800052642822, + "p95_ms": 0.42318820357322695 + }, + "forward_peak_mib": 0.0107421875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.1808696349646652e-06, + "relative_l2": 4.670865489897585e-08 + }, + "lse_vs_fp64": { + "max_abs": 8.398243291907193e-07, + "relative_l2": 3.2524573103729204e-08 + }, + "mismatch_vs_reference": 2, + "path": "ws2-triton", + "rel_l2_vs_reference": 5.157416402269682e-08, + "repeat_bitwise": true, + "tokens": 8, + "train_fwd_bwd": { + "max_ms": 1.199422001838684, + "median_ms": 1.0126639902591705, + "min_ms": 0.978613018989563, + "p95_ms": 1.185198837518692 + }, + "train_peak_mib": 4.642578125 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.39446601271629333, + "median_ms": 0.35200299322605133, + "min_ms": 0.3450320065021515, + "p95_ms": 0.36558124870061875 }, "forward_peak_mib": 0.0107421875, "grad_finite": true, @@ -1397,10 +1844,10 @@ "repeat_bitwise": true, "tokens": 8, "train_fwd_bwd": { - "max_ms": 0.933987021446228, - "median_ms": 0.8225814998149872, - "min_ms": 0.8035140037536621, - "p95_ms": 0.9159964531660079 + "max_ms": 0.9747679829597473, + "median_ms": 0.7770539820194244, + "min_ms": 0.7595679759979248, + "p95_ms": 0.9017950803041456 }, "train_peak_mib": 4.642578125 }, @@ -1408,10 +1855,10 @@ "batch_invariant": false, "dtype": "bf16", "forward": { - "max_ms": 0.19561000168323517, - "median_ms": 0.16456449776887894, - "min_ms": 0.13676300644874573, - "p95_ms": 0.16843809783458713 + "max_ms": 0.1689710021018982, + "median_ms": 0.1626409962773323, + "min_ms": 0.13307799398899078, + "p95_ms": 0.16706815287470816 }, "forward_peak_mib": 37.10302734375, "grad_finite": true, @@ -1427,10 +1874,10 @@ "repeat_bitwise": true, "tokens": 32, "train_fwd_bwd": { - "max_ms": 0.6035760045051575, - "median_ms": 0.5357359945774078, - "min_ms": 0.5176489949226379, - "p95_ms": 0.5917504400014877 + "max_ms": 0.5343539714813232, + "median_ms": 0.5017854869365692, + "min_ms": 0.4919700026512146, + "p95_ms": 0.5303696841001511 }, "train_peak_mib": 102.01025390625 }, @@ -1438,10 +1885,10 @@ "batch_invariant": false, "dtype": "bf16", "forward": { - "max_ms": 0.25301599502563477, - "median_ms": 0.21045250445604324, - "min_ms": 0.1858759969472885, - "p95_ms": 0.24114195480942727 + "max_ms": 0.23274600505828857, + "median_ms": 0.19759349524974823, + "min_ms": 0.17562100291252136, + "p95_ms": 0.2175992012023926 }, "forward_peak_mib": 55.6513671875, "grad_finite": true, @@ -1457,10 +1904,10 @@ "repeat_bitwise": true, "tokens": 32, "train_fwd_bwd": { - "max_ms": 1.0021679401397705, - "median_ms": 0.9008379876613617, - "min_ms": 0.8616790175437927, - "p95_ms": 0.9823387175798415 + "max_ms": 0.88919997215271, + "median_ms": 0.8541084825992584, + "min_ms": 0.8356819748878479, + "p95_ms": 0.8848016858100891 }, "train_peak_mib": 64.92529296875 }, @@ -1468,10 +1915,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.3513619899749756, - "median_ms": 0.3042919933795929, - "min_ms": 0.2994439899921417, - "p95_ms": 0.31357196122407915 + "max_ms": 0.35424599051475525, + "median_ms": 0.3021489977836609, + "min_ms": 0.2995249927043915, + "p95_ms": 0.3195387080311775 }, "forward_peak_mib": 0.00341796875, "grad_finite": true, @@ -1487,10 +1934,10 @@ "repeat_bitwise": true, "tokens": 32, "train_fwd_bwd": { - "max_ms": 0.5292260050773621, - "median_ms": 0.5104179978370667, - "min_ms": 0.47462499141693115, - "p95_ms": 0.5285591006278991 + "max_ms": 0.49056899547576904, + "median_ms": 0.4564579874277115, + "min_ms": 0.44145599007606506, + "p95_ms": 0.48285329937934873 }, "train_peak_mib": 18.5498046875 }, @@ -1498,10 +1945,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 6.187510013580322, - "median_ms": 6.039510488510132, - "min_ms": 5.880495071411133, - "p95_ms": 6.155314278602599 + "max_ms": 5.4874701499938965, + "median_ms": 5.397536516189575, + "min_ms": 5.278359889984131, + "p95_ms": 5.447852993011475 }, "forward_peak_mib": 20.78515625, "grad_finite": true, @@ -1517,10 +1964,10 @@ "repeat_bitwise": true, "tokens": 32, "train_fwd_bwd": { - "max_ms": 7.214517116546631, - "median_ms": 7.1387834548950195, - "min_ms": 7.0237531661987305, - "p95_ms": 7.204205203056335 + "max_ms": 6.701952934265137, + "median_ms": 6.617066860198975, + "min_ms": 6.586542129516602, + "p95_ms": 6.6741920709609985 }, "train_peak_mib": 120.70654296875 }, @@ -1528,10 +1975,42 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.43793100118637085, - "median_ms": 0.35977399349212646, - "min_ms": 0.35096099972724915, - "p95_ms": 0.39481238871812824 + "max_ms": 0.44017401337623596, + "median_ms": 0.40299899876117706, + "min_ms": 0.3907009959220886, + "p95_ms": 0.4346934497356415 + }, + "forward_peak_mib": 0.0341796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 9.292139733219074e-07, + "relative_l2": 3.208024712173244e-08 + }, + "lse_vs_fp64": { + "max_abs": 9.292139733219074e-07, + "relative_l2": 3.1245170134439244e-08 + }, + "mismatch_vs_reference": 4, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.8415464321385415e-08, + "repeat_bitwise": true, + "tokens": 32, + "train_fwd_bwd": { + "max_ms": 1.0324130058288574, + "median_ms": 0.984281986951828, + "min_ms": 0.9665150046348572, + "p95_ms": 1.0247517585754395 + }, + "train_peak_mib": 18.552734375 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.4256319999694824, + "median_ms": 0.3501400053501129, + "min_ms": 0.34258899092674255, + "p95_ms": 0.37573989033699035 }, "forward_peak_mib": 0.0341796875, "grad_finite": true, @@ -1549,10 +2028,10 @@ "repeat_bitwise": true, "tokens": 32, "train_fwd_bwd": { - "max_ms": 0.8513439893722534, - "median_ms": 0.79117551445961, - "min_ms": 0.7693020105361938, - "p95_ms": 0.8505870968103408 + "max_ms": 0.8069980144500732, + "median_ms": 0.7662184834480286, + "min_ms": 0.7476300001144409, + "p95_ms": 0.8005449950695038 }, "train_peak_mib": 18.552734375 }, @@ -1560,10 +2039,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.3250420093536377, - "median_ms": 0.29465749859809875, - "min_ms": 0.2627910077571869, - "p95_ms": 0.32371008992195127 + "max_ms": 0.33245301246643066, + "median_ms": 0.29698099195957184, + "min_ms": 0.26515400409698486, + "p95_ms": 0.32309265732765197 }, "forward_peak_mib": 148.40771484375, "grad_finite": true, @@ -1579,10 +2058,10 @@ "repeat_bitwise": true, "tokens": 128, "train_fwd_bwd": { - "max_ms": 0.6414729952812195, - "median_ms": 0.5895554721355438, - "min_ms": 0.5013440251350403, - "p95_ms": 0.6334868520498276 + "max_ms": 0.626570999622345, + "median_ms": 0.5764960050582886, + "min_ms": 0.5228559970855713, + "p95_ms": 0.6142585605382919 }, "train_peak_mib": 408.03369140625 }, @@ -1590,10 +2069,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.3740359842777252, - "median_ms": 0.30306999385356903, - "min_ms": 0.26948100328445435, - "p95_ms": 0.33719690591096885 + "max_ms": 0.3854529857635498, + "median_ms": 0.303850993514061, + "min_ms": 0.2683979868888855, + "p95_ms": 0.33457099646329885 }, "forward_peak_mib": 222.59765625, "grad_finite": true, @@ -1609,10 +2088,10 @@ "repeat_bitwise": true, "tokens": 128, "train_fwd_bwd": { - "max_ms": 1.0661829710006714, - "median_ms": 0.8456965088844299, - "min_ms": 0.8178949952125549, - "p95_ms": 1.0009622424840925 + "max_ms": 1.1429779529571533, + "median_ms": 0.8844340145587921, + "min_ms": 0.7590069770812988, + "p95_ms": 1.0753051757812497 }, "train_peak_mib": 259.6923828125 }, @@ -1620,10 +2099,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.3670249879360199, - "median_ms": 0.3225190043449402, - "min_ms": 0.31927400827407837, - "p95_ms": 0.32923495918512347 + "max_ms": 0.36718499660491943, + "median_ms": 0.31983450055122375, + "min_ms": 0.3176719844341278, + "p95_ms": 0.3301939100027085 }, "forward_peak_mib": 0.00439453125, "grad_finite": true, @@ -1639,10 +2118,10 @@ "repeat_bitwise": true, "tokens": 128, "train_fwd_bwd": { - "max_ms": 0.5265020132064819, - "median_ms": 0.46553100645542145, - "min_ms": 0.45211100578308105, - "p95_ms": 0.5206794470548629 + "max_ms": 0.517408013343811, + "median_ms": 0.46639250218868256, + "min_ms": 0.45699799060821533, + "p95_ms": 0.503797760605812 }, "train_peak_mib": 74.19091796875 }, @@ -1650,10 +2129,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 6.214951038360596, - "median_ms": 6.065549373626709, - "min_ms": 5.840994834899902, - "p95_ms": 6.191926646232605 + "max_ms": 5.568389892578125, + "median_ms": 5.465717315673828, + "min_ms": 5.350906848907471, + "p95_ms": 5.539391398429871 }, "forward_peak_mib": 79.03369140625, "grad_finite": true, @@ -1669,10 +2148,10 @@ "repeat_bitwise": true, "tokens": 128, "train_fwd_bwd": { - "max_ms": 7.209747791290283, - "median_ms": 7.14607310295105, - "min_ms": 7.076991081237793, - "p95_ms": 7.198733782768249 + "max_ms": 6.825016975402832, + "median_ms": 6.794530868530273, + "min_ms": 6.7457780838012695, + "p95_ms": 6.824007177352906 }, "train_peak_mib": 482.3720703125 }, @@ -1680,10 +2159,42 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.45359399914741516, - "median_ms": 0.37147149443626404, - "min_ms": 0.35677099227905273, - "p95_ms": 0.38836511075496677 + "max_ms": 0.46625199913978577, + "median_ms": 0.4062635004520416, + "min_ms": 0.3971889913082123, + "p95_ms": 0.42511413693428046 + }, + "forward_peak_mib": 0.12841796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.4593867163625873e-06, + "relative_l2": 3.478970586475828e-08 + }, + "lse_vs_fp64": { + "max_abs": 8.965857389853227e-07, + "relative_l2": 2.903246856398248e-08 + }, + "mismatch_vs_reference": 7, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.5511401243457844e-08, + "repeat_bitwise": true, + "tokens": 128, + "train_fwd_bwd": { + "max_ms": 1.1938129663467407, + "median_ms": 1.0162895321846008, + "min_ms": 0.9801759719848633, + "p95_ms": 1.1880444467067717 + }, + "train_peak_mib": 74.1943359375 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.4069640040397644, + "median_ms": 0.35552799701690674, + "min_ms": 0.3464739918708801, + "p95_ms": 0.37290458977222446 }, "forward_peak_mib": 0.12841796875, "grad_finite": true, @@ -1701,10 +2212,10 @@ "repeat_bitwise": true, "tokens": 128, "train_fwd_bwd": { - "max_ms": 0.9247739911079407, - "median_ms": 0.821900486946106, - "min_ms": 0.7889320254325867, - "p95_ms": 0.906566995382309 + "max_ms": 0.9827389717102051, + "median_ms": 0.791395515203476, + "min_ms": 0.7667790055274963, + "p95_ms": 0.9700485289096832 }, "train_peak_mib": 74.1943359375 }, @@ -1712,10 +2223,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.8934069871902466, - "median_ms": 0.8458364903926849, - "min_ms": 0.8043149709701538, - "p95_ms": 0.8792500913143159 + "max_ms": 0.8830310106277466, + "median_ms": 0.8542289733886719, + "min_ms": 0.8202589750289917, + "p95_ms": 0.8800632119178772 }, "forward_peak_mib": 593.630859375, "grad_finite": true, @@ -1731,10 +2242,10 @@ "repeat_bitwise": true, "tokens": 512, "train_fwd_bwd": { - "max_ms": 1.9239779710769653, - "median_ms": 1.8965969681739807, - "min_ms": 1.8372089862823486, - "p95_ms": 1.919651246070862 + "max_ms": 1.9153649806976318, + "median_ms": 1.8550350069999695, + "min_ms": 1.806203007698059, + "p95_ms": 1.909361982345581 }, "train_peak_mib": 1632.1318359375 }, @@ -1742,10 +2253,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.9607880115509033, - "median_ms": 0.9033215045928955, - "min_ms": 0.8575940132141113, - "p95_ms": 0.9529467195272446 + "max_ms": 0.9521740078926086, + "median_ms": 0.9142979979515076, + "min_ms": 0.8730159997940063, + "p95_ms": 0.9433076441287995 }, "forward_peak_mib": 890.38916015625, "grad_finite": true, @@ -1761,10 +2272,10 @@ "repeat_bitwise": true, "tokens": 512, "train_fwd_bwd": { - "max_ms": 2.2761409282684326, - "median_ms": 2.2405279874801636, - "min_ms": 2.1743900775909424, - "p95_ms": 2.2733644127845762 + "max_ms": 2.2674479484558105, + "median_ms": 2.255669951438904, + "min_ms": 2.2023909091949463, + "p95_ms": 2.267303538322449 }, "train_peak_mib": 1038.76806640625 }, @@ -1772,10 +2283,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.38609299063682556, - "median_ms": 0.33629900217056274, - "min_ms": 0.33293500542640686, - "p95_ms": 0.3548104375600815 + "max_ms": 0.3829689919948578, + "median_ms": 0.33712050318717957, + "min_ms": 0.33297398686408997, + "p95_ms": 0.3422111362218857 }, "forward_peak_mib": 0.01123046875, "grad_finite": true, @@ -1791,10 +2302,10 @@ "repeat_bitwise": true, "tokens": 512, "train_fwd_bwd": { - "max_ms": 0.599049985408783, - "median_ms": 0.5090160071849823, - "min_ms": 0.47718900442123413, - "p95_ms": 0.5664758443832397 + "max_ms": 0.5339930057525635, + "median_ms": 0.4645095020532608, + "min_ms": 0.4525119960308075, + "p95_ms": 0.5160924524068832 }, "train_peak_mib": 296.7607421875 }, @@ -1802,10 +2313,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 6.331644058227539, - "median_ms": 6.183424472808838, - "min_ms": 5.945109844207764, - "p95_ms": 6.294121098518372 + "max_ms": 5.67739200592041, + "median_ms": 5.543973922729492, + "min_ms": 5.352829933166504, + "p95_ms": 5.667003011703491 }, "forward_peak_mib": 312.21923828125, "grad_finite": true, @@ -1821,10 +2332,10 @@ "repeat_bitwise": true, "tokens": 512, "train_fwd_bwd": { - "max_ms": 8.401599884033203, - "median_ms": 8.282042026519775, - "min_ms": 8.15567398071289, - "p95_ms": 8.375352001190185 + "max_ms": 7.908987045288086, + "median_ms": 7.684312582015991, + "min_ms": 7.606616973876953, + "p95_ms": 7.843495869636535 }, "train_peak_mib": 1929.0439453125 }, @@ -1832,10 +2343,42 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.485960990190506, - "median_ms": 0.40726499259471893, - "min_ms": 0.39743098616600037, - "p95_ms": 0.4436423107981682 + "max_ms": 0.4894070029258728, + "median_ms": 0.4432384967803955, + "min_ms": 0.43492600321769714, + "p95_ms": 0.4798547476530075 + }, + "forward_peak_mib": 0.5107421875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.39654445874271e-06, + "relative_l2": 3.263898958223032e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.265310253018015e-06, + "relative_l2": 2.91707358884744e-08 + }, + "mismatch_vs_reference": 43, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.757610490869294e-08, + "repeat_bitwise": true, + "tokens": 512, + "train_fwd_bwd": { + "max_ms": 1.2311480045318604, + "median_ms": 1.1473445296287537, + "min_ms": 1.0201549530029297, + "p95_ms": 1.223937636613846 + }, + "train_peak_mib": 296.77001953125 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.44249799847602844, + "median_ms": 0.40269799530506134, + "min_ms": 0.3962689936161041, + "p95_ms": 0.4225175932049751 }, "forward_peak_mib": 0.5107421875, "grad_finite": true, @@ -1853,10 +2396,10 @@ "repeat_bitwise": true, "tokens": 512, "train_fwd_bwd": { - "max_ms": 0.9189249873161316, - "median_ms": 0.8466775119304657, - "min_ms": 0.8301529884338379, - "p95_ms": 0.8978874832391739 + "max_ms": 1.4356119632720947, + "median_ms": 0.8617799878120422, + "min_ms": 0.8286309838294983, + "p95_ms": 1.252316182851791 }, "train_peak_mib": 296.77001953125 }, @@ -1864,10 +2407,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 2.8302440643310547, - "median_ms": 2.3364105224609375, - "min_ms": 2.2821099758148193, - "p95_ms": 2.690348017215729 + "max_ms": 2.7969539165496826, + "median_ms": 2.302619457244873, + "min_ms": 2.263361930847168, + "p95_ms": 2.788733637332916 }, "forward_peak_mib": 2374.015625, "grad_finite": true, @@ -1883,10 +2426,10 @@ "repeat_bitwise": true, "tokens": 2048, "train_fwd_bwd": { - "max_ms": 6.796535015106201, - "median_ms": 6.494866847991943, - "min_ms": 6.18959379196167, - "p95_ms": 6.789486598968505 + "max_ms": 6.601564884185791, + "median_ms": 6.448056936264038, + "min_ms": 6.240708827972412, + "p95_ms": 6.571694302558899 }, "train_peak_mib": 6528.5244140625 }, @@ -1894,10 +2437,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 2.8184258937835693, - "median_ms": 2.304742932319641, - "min_ms": 2.2484591007232666, - "p95_ms": 2.7830338239669796 + "max_ms": 2.7635040283203125, + "median_ms": 2.2864960432052612, + "min_ms": 2.2438929080963135, + "p95_ms": 2.6859070658683777 }, "forward_peak_mib": 3561.103515625, "grad_finite": true, @@ -1913,10 +2456,10 @@ "repeat_bitwise": true, "tokens": 2048, "train_fwd_bwd": { - "max_ms": 7.514641761779785, - "median_ms": 7.046125411987305, - "min_ms": 6.996592044830322, - "p95_ms": 7.323990345001221 + "max_ms": 7.106554985046387, + "median_ms": 7.031663656234741, + "min_ms": 6.984973907470703, + "p95_ms": 7.0932692527771 }, "train_peak_mib": 4154.619140625 }, @@ -1924,10 +2467,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.6434760093688965, - "median_ms": 0.6005325019359589, - "min_ms": 0.5902770161628723, - "p95_ms": 0.6080828070640565 + "max_ms": 0.6292549967765808, + "median_ms": 0.5970470011234283, + "min_ms": 0.594202995300293, + "p95_ms": 0.6152880758047103 }, "forward_peak_mib": 0.044921875, "grad_finite": true, @@ -1943,10 +2486,10 @@ "repeat_bitwise": true, "tokens": 2048, "train_fwd_bwd": { - "max_ms": 1.178589940071106, - "median_ms": 1.1376099586486816, - "min_ms": 1.098431944847107, - "p95_ms": 1.1706946969032286 + "max_ms": 1.160202980041504, + "median_ms": 1.1344245076179504, + "min_ms": 1.1143759489059448, + "p95_ms": 1.1601490139961244 }, "train_peak_mib": 1187.0400390625 }, @@ -1954,10 +2497,10 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 6.943714141845703, - "median_ms": 6.480145215988159, - "min_ms": 6.294229030609131, - "p95_ms": 6.7318147182464605 + "max_ms": 6.33885383605957, + "median_ms": 6.138717412948608, + "min_ms": 5.910459041595459, + "p95_ms": 6.307230234146118 }, "forward_peak_mib": 1244.96435546875, "grad_finite": true, @@ -1973,10 +2516,10 @@ "repeat_bitwise": true, "tokens": 2048, "train_fwd_bwd": { - "max_ms": 12.976513862609863, - "median_ms": 12.827512741088867, - "min_ms": 12.742406845092773, - "p95_ms": 12.92358765602112 + "max_ms": 12.480816841125488, + "median_ms": 12.347699165344238, + "min_ms": 12.259528160095215, + "p95_ms": 12.468018198013306 }, "train_peak_mib": 7715.73779296875 }, @@ -1984,10 +2527,42 @@ "batch_invariant": true, "dtype": "bf16", "forward": { - "max_ms": 0.9866660237312317, - "median_ms": 0.8963714838027954, - "min_ms": 0.8853949904441833, - "p95_ms": 0.9365820139646531 + "max_ms": 1.002089023590088, + "median_ms": 0.9513734877109528, + "min_ms": 0.9448429942131042, + "p95_ms": 0.9800537467002869 + }, + "forward_peak_mib": 2.04296875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.5566917461740104e-06, + "relative_l2": 3.227948726455519e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.3182267863953712e-06, + "relative_l2": 2.972527831148482e-08 + }, + "mismatch_vs_reference": 169, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.7134613775374536e-08, + "repeat_bitwise": true, + "tokens": 2048, + "train_fwd_bwd": { + "max_ms": 2.2043540477752686, + "median_ms": 2.1618911027908325, + "min_ms": 2.0272109508514404, + "p95_ms": 2.19452965259552 + }, + "train_peak_mib": 1187.0771484375 + }, + { + "batch_invariant": true, + "dtype": "bf16", + "forward": { + "max_ms": 0.9396349787712097, + "median_ms": 0.892345517873764, + "min_ms": 0.8854749798774719, + "p95_ms": 0.9161548167467117 }, "forward_peak_mib": 2.04296875, "grad_finite": true, @@ -2005,10 +2580,10 @@ "repeat_bitwise": true, "tokens": 2048, "train_fwd_bwd": { - "max_ms": 1.932750940322876, - "median_ms": 1.878069519996643, - "min_ms": 1.8332430124282837, - "p95_ms": 1.9317951619625093 + "max_ms": 2.052248001098633, + "median_ms": 2.0358840227127075, + "min_ms": 2.0259690284729004, + "p95_ms": 2.051292598247528 }, "train_peak_mib": 1187.0771484375 }, @@ -2016,10 +2591,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.17421899735927582, - "median_ms": 0.12049899995326996, - "min_ms": 0.11060500144958496, - "p95_ms": 0.14784510135650636 + "max_ms": 0.1543090045452118, + "median_ms": 0.11416950076818466, + "min_ms": 0.10904199630022049, + "p95_ms": 0.12074360288679602 }, "forward_peak_mib": 0.58154296875, "grad_finite": true, @@ -2035,10 +2610,10 @@ "repeat_bitwise": true, "tokens": 1, "train_fwd_bwd": { - "max_ms": 0.6362249851226807, - "median_ms": 0.5021460205316544, - "min_ms": 0.46448999643325806, - "p95_ms": 0.5911399483680724 + "max_ms": 0.5347949862480164, + "median_ms": 0.49237099289894104, + "min_ms": 0.47746899724006653, + "p95_ms": 0.5201029449701309 }, "train_peak_mib": 2.900390625 }, @@ -2046,10 +2621,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.24876900017261505, - "median_ms": 0.1910640001296997, - "min_ms": 0.18271200358867645, - "p95_ms": 0.20439544916152957 + "max_ms": 0.2101919949054718, + "median_ms": 0.1837330013513565, + "min_ms": 0.17950700223445892, + "p95_ms": 0.2077571451663971 }, "forward_peak_mib": 1.16455078125, "grad_finite": true, @@ -2065,10 +2640,10 @@ "repeat_bitwise": true, "tokens": 1, "train_fwd_bwd": { - "max_ms": 1.1947740316390991, - "median_ms": 0.8466980159282684, - "min_ms": 0.8064379692077637, - "p95_ms": 1.0594113111495969 + "max_ms": 1.0200350284576416, + "median_ms": 0.8074195086956024, + "min_ms": 0.8034729957580566, + "p95_ms": 1.0148069202899932 }, "train_peak_mib": 2.322265625 }, @@ -2076,10 +2651,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.3544459939002991, - "median_ms": 0.3016285002231598, - "min_ms": 0.29928499460220337, - "p95_ms": 0.310148461163044 + "max_ms": 0.36177799105644226, + "median_ms": 0.2994849979877472, + "min_ms": 0.29600000381469727, + "p95_ms": 0.34925700277090077 }, "forward_peak_mib": 0.00341796875, "grad_finite": true, @@ -2095,10 +2670,10 @@ "repeat_bitwise": true, "tokens": 1, "train_fwd_bwd": { - "max_ms": 0.5243780016899109, - "median_ms": 0.46805500984191895, - "min_ms": 0.446382999420166, - "p95_ms": 0.5172576487064362 + "max_ms": 0.49481499195098877, + "median_ms": 0.45453500747680664, + "min_ms": 0.4421769976615906, + "p95_ms": 0.4892268911004066 }, "train_peak_mib": 1.162109375 }, @@ -2106,10 +2681,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 5.4307861328125, - "median_ms": 5.160525560379028, - "min_ms": 5.13234281539917, - "p95_ms": 5.209868025779724 + "max_ms": 4.817515850067139, + "median_ms": 4.744045972824097, + "min_ms": 4.700301170349121, + "p95_ms": 4.814090299606324 }, "forward_peak_mib": 1.38671875, "grad_finite": true, @@ -2125,10 +2700,10 @@ "repeat_bitwise": true, "tokens": 1, "train_fwd_bwd": { - "max_ms": 6.593995094299316, - "median_ms": 6.322952032089233, - "min_ms": 6.285376071929932, - "p95_ms": 6.512369465827941 + "max_ms": 6.1407999992370605, + "median_ms": 6.0981974601745605, + "min_ms": 6.018377780914307, + "p95_ms": 6.135987257957458 }, "train_peak_mib": 3.62939453125 }, @@ -2136,10 +2711,42 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.6988779902458191, - "median_ms": 0.3520229905843735, - "min_ms": 0.3469559848308563, - "p95_ms": 0.44055113494396225 + "max_ms": 0.4266740083694458, + "median_ms": 0.38156700134277344, + "min_ms": 0.37199199199676514, + "p95_ms": 0.39862620979547503 + }, + "forward_peak_mib": 0.005859375, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3678638666192455e-07, + "relative_l2": 8.545758893188667e-09 + }, + "lse_vs_fp64": { + "max_abs": 1.3678638666192455e-07, + "relative_l2": 9.836276637086277e-09 + }, + "mismatch_vs_reference": 1, + "path": "ws2-triton", + "rel_l2_vs_reference": 6.857849803488313e-08, + "repeat_bitwise": true, + "tokens": 1, + "train_fwd_bwd": { + "max_ms": 0.8591560125350952, + "median_ms": 0.7713660001754761, + "min_ms": 0.7473899722099304, + "p95_ms": 0.8526125490665436 + }, + "train_peak_mib": 1.1650390625 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.3813669979572296, + "median_ms": 0.34324949979782104, + "min_ms": 0.3323740065097809, + "p95_ms": 0.3569719552993775 }, "forward_peak_mib": 0.005859375, "grad_finite": true, @@ -2157,10 +2764,10 @@ "repeat_bitwise": true, "tokens": 1, "train_fwd_bwd": { - "max_ms": 0.8288310170173645, - "median_ms": 0.7904340028762817, - "min_ms": 0.7575650215148926, - "p95_ms": 0.8223415464162827 + "max_ms": 0.7179059982299805, + "median_ms": 0.6274919807910919, + "min_ms": 0.615032970905304, + "p95_ms": 0.6916952908039092 }, "train_peak_mib": 1.1650390625 }, @@ -2168,10 +2775,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.1482599973678589, - "median_ms": 0.12208149954676628, - "min_ms": 0.11745499819517136, - "p95_ms": 0.13984964415431023 + "max_ms": 0.14429500699043274, + "median_ms": 0.11721399798989296, + "min_ms": 0.11284700036048889, + "p95_ms": 0.13398180454969408 }, "forward_peak_mib": 4.64013671875, "grad_finite": true, @@ -2187,10 +2794,10 @@ "repeat_bitwise": true, "tokens": 8, "train_fwd_bwd": { - "max_ms": 0.5449299812316895, - "median_ms": 0.488986000418663, - "min_ms": 0.4679349958896637, - "p95_ms": 0.5280747920274734 + "max_ms": 0.3843710124492645, + "median_ms": 0.3623580038547516, + "min_ms": 0.3551679849624634, + "p95_ms": 0.38348765373229976 }, "train_peak_mib": 23.18603515625 }, @@ -2198,10 +2805,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.2609480023384094, - "median_ms": 0.19470950216054916, - "min_ms": 0.18827900290489197, - "p95_ms": 0.20846810042858127 + "max_ms": 0.22168999910354614, + "median_ms": 0.18493500351905823, + "min_ms": 0.17918600142002106, + "p95_ms": 0.2052492991089821 }, "forward_peak_mib": 9.27880859375, "grad_finite": true, @@ -2217,10 +2824,10 @@ "repeat_bitwise": true, "tokens": 8, "train_fwd_bwd": { - "max_ms": 0.8992949724197388, - "median_ms": 0.8240644931793213, - "min_ms": 0.7896119952201843, - "p95_ms": 0.8870369821786881 + "max_ms": 0.6065809726715088, + "median_ms": 0.572409987449646, + "min_ms": 0.5615940093994141, + "p95_ms": 0.5971350371837616 }, "train_peak_mib": 18.55224609375 }, @@ -2228,10 +2835,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.3505209982395172, - "median_ms": 0.3025895059108734, - "min_ms": 0.2998049855232239, - "p95_ms": 0.317638635635376 + "max_ms": 0.3464750051498413, + "median_ms": 0.3008265048265457, + "min_ms": 0.2979219853878021, + "p95_ms": 0.3128326579928399 }, "forward_peak_mib": 0.00341796875, "grad_finite": true, @@ -2247,10 +2854,10 @@ "repeat_bitwise": true, "tokens": 8, "train_fwd_bwd": { - "max_ms": 0.5011450052261353, - "median_ms": 0.4633080065250397, - "min_ms": 0.4420959949493408, - "p95_ms": 0.49667425900697704 + "max_ms": 0.37323498725891113, + "median_ms": 0.35190249979496, + "min_ms": 0.3442310094833374, + "p95_ms": 0.3687642410397529 }, "train_peak_mib": 9.2763671875 }, @@ -2258,10 +2865,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 5.932651996612549, - "median_ms": 5.814716815948486, - "min_ms": 5.754907131195068, - "p95_ms": 5.861942601203919 + "max_ms": 5.581049919128418, + "median_ms": 5.453480005264282, + "min_ms": 5.361602783203125, + "p95_ms": 5.529634046554565 }, "forward_peak_mib": 1.58642578125, "grad_finite": true, @@ -2277,10 +2884,10 @@ "repeat_bitwise": true, "tokens": 8, "train_fwd_bwd": { - "max_ms": 7.035409927368164, - "median_ms": 6.977062940597534, - "min_ms": 6.87232780456543, - "p95_ms": 7.014931797981262 + "max_ms": 6.8699631690979, + "median_ms": 6.778928995132446, + "min_ms": 6.7133307456970215, + "p95_ms": 6.835946750640869 }, "train_peak_mib": 27.97216796875 }, @@ -2288,10 +2895,42 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.4312010109424591, - "median_ms": 0.3580314964056015, - "min_ms": 0.34767699241638184, - "p95_ms": 0.36920684874057774 + "max_ms": 0.42835599184036255, + "median_ms": 0.390780508518219, + "min_ms": 0.38400998711586, + "p95_ms": 0.40350588709115986 + }, + "forward_peak_mib": 0.0107421875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 7.819555989385663e-07, + "relative_l2": 3.607215816086566e-08 + }, + "lse_vs_fp64": { + "max_abs": 3.7962424670467954e-07, + "relative_l2": 1.7082649106274754e-08 + }, + "mismatch_vs_reference": 0, + "path": "ws2-triton", + "rel_l2_vs_reference": 0.0, + "repeat_bitwise": true, + "tokens": 8, + "train_fwd_bwd": { + "max_ms": 1.200382947921753, + "median_ms": 1.0127645134925842, + "min_ms": 0.9833800196647644, + "p95_ms": 1.1908830463886262 + }, + "train_peak_mib": 9.279296875 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.39678898453712463, + "median_ms": 0.34459200501441956, + "min_ms": 0.3358590006828308, + "p95_ms": 0.3572490349411964 }, "forward_peak_mib": 0.0107421875, "grad_finite": true, @@ -2309,10 +2948,10 @@ "repeat_bitwise": true, "tokens": 8, "train_fwd_bwd": { - "max_ms": 0.8861969709396362, - "median_ms": 0.7877904772758484, - "min_ms": 0.7638139724731445, - "p95_ms": 0.847366926074028 + "max_ms": 0.9615880250930786, + "median_ms": 0.7646960020065308, + "min_ms": 0.7506340146064758, + "p95_ms": 0.8864344060420988 }, "train_peak_mib": 9.279296875 }, @@ -2320,10 +2959,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.18960200250148773, - "median_ms": 0.1570734977722168, - "min_ms": 0.1322769969701767, - "p95_ms": 0.18153364658355714 + "max_ms": 0.16805000603199005, + "median_ms": 0.15777450054883957, + "min_ms": 0.13067400455474854, + "p95_ms": 0.16165650114417077 }, "forward_peak_mib": 18.55615234375, "grad_finite": true, @@ -2339,10 +2978,10 @@ "repeat_bitwise": true, "tokens": 32, "train_fwd_bwd": { - "max_ms": 0.5316699743270874, - "median_ms": 0.4779900014400482, - "min_ms": 0.466933012008667, - "p95_ms": 0.5286594897508621 + "max_ms": 0.542165994644165, + "median_ms": 0.47486498951911926, + "min_ms": 0.46753400564193726, + "p95_ms": 0.5188208922743797 }, "train_peak_mib": 92.73681640625 }, @@ -2350,10 +2989,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.2554590106010437, - "median_ms": 0.20001699775457382, - "min_ms": 0.18471400439739227, - "p95_ms": 0.21884885057806971 + "max_ms": 0.23334699869155884, + "median_ms": 0.19080349802970886, + "min_ms": 0.16196100413799286, + "p95_ms": 0.21979905590415003 }, "forward_peak_mib": 37.1044921875, "grad_finite": true, @@ -2369,10 +3008,10 @@ "repeat_bitwise": true, "tokens": 32, "train_fwd_bwd": { - "max_ms": 0.9258949756622314, - "median_ms": 0.852606475353241, - "min_ms": 0.821461021900177, - "p95_ms": 0.9069850802421568 + "max_ms": 0.8767020106315613, + "median_ms": 0.8396669924259186, + "min_ms": 0.823743999004364, + "p95_ms": 0.8703750014305114 }, "train_peak_mib": 74.19873046875 }, @@ -2380,10 +3019,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.3556880056858063, - "median_ms": 0.3040114939212799, - "min_ms": 0.30184799432754517, - "p95_ms": 0.3192687973380089 + "max_ms": 0.35901200771331787, + "median_ms": 0.3029704988002777, + "min_ms": 0.29780200123786926, + "p95_ms": 0.32449566274881364 }, "forward_peak_mib": 0.00341796875, "grad_finite": true, @@ -2399,10 +3038,10 @@ "repeat_bitwise": true, "tokens": 32, "train_fwd_bwd": { - "max_ms": 0.5708879828453064, - "median_ms": 0.5329720079898834, - "min_ms": 0.48099398612976074, - "p95_ms": 0.5704015374183655 + "max_ms": 0.5286250114440918, + "median_ms": 0.485180988907814, + "min_ms": 0.4702180027961731, + "p95_ms": 0.5186201572418212 }, "train_peak_mib": 37.0966796875 }, @@ -2410,10 +3049,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 6.055755138397217, - "median_ms": 5.980923891067505, - "min_ms": 5.832943916320801, - "p95_ms": 6.0501996040344235 + "max_ms": 5.558335781097412, + "median_ms": 5.474370956420898, + "min_ms": 5.384798049926758, + "p95_ms": 5.557803058624267 }, "forward_peak_mib": 2.23828125, "grad_finite": true, @@ -2429,10 +3068,10 @@ "repeat_bitwise": true, "tokens": 32, "train_fwd_bwd": { - "max_ms": 7.136520862579346, - "median_ms": 7.03747296333313, - "min_ms": 6.882863998413086, - "p95_ms": 7.119196367263794 + "max_ms": 6.803424835205078, + "median_ms": 6.748803615570068, + "min_ms": 6.687372207641602, + "p95_ms": 6.789111709594726 }, "train_peak_mib": 111.43310546875 }, @@ -2440,10 +3079,42 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.4266740083694458, - "median_ms": 0.3478565067052841, - "min_ms": 0.34162798523902893, - "p95_ms": 0.39109080284833914 + "max_ms": 0.46192601323127747, + "median_ms": 0.39490650594234467, + "min_ms": 0.38837599754333496, + "p95_ms": 0.4057923927903176 + }, + "forward_peak_mib": 0.0341796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3984818885148798e-06, + "relative_l2": 3.32171929410509e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.1600633094133173e-06, + "relative_l2": 3.373302046539746e-08 + }, + "mismatch_vs_reference": 0, + "path": "ws2-triton", + "rel_l2_vs_reference": 0.0, + "repeat_bitwise": true, + "tokens": 32, + "train_fwd_bwd": { + "max_ms": 1.2516590356826782, + "median_ms": 1.0681869983673096, + "min_ms": 1.0127040147781372, + "p95_ms": 1.2389685928821563 + }, + "train_peak_mib": 37.099609375 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.3988319933414459, + "median_ms": 0.3438510000705719, + "min_ms": 0.33401599526405334, + "p95_ms": 0.35868310481309895 }, "forward_peak_mib": 0.0341796875, "grad_finite": true, @@ -2461,10 +3132,10 @@ "repeat_bitwise": true, "tokens": 32, "train_fwd_bwd": { - "max_ms": 0.842290997505188, - "median_ms": 0.7901535034179688, - "min_ms": 0.770704984664917, - "p95_ms": 0.8351881861686706 + "max_ms": 0.8316360116004944, + "median_ms": 0.7883504927158356, + "min_ms": 0.7761529684066772, + "p95_ms": 0.8204049050807952 }, "train_peak_mib": 37.099609375 }, @@ -2472,10 +3143,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.3078170120716095, - "median_ms": 0.27308548986911774, - "min_ms": 0.24384300410747528, - "p95_ms": 0.30477319508790973 + "max_ms": 0.3082979917526245, + "median_ms": 0.2700809985399246, + "min_ms": 0.21864500641822815, + "p95_ms": 0.30304638892412183 }, "forward_peak_mib": 74.22021484375, "grad_finite": true, @@ -2491,10 +3162,10 @@ "repeat_bitwise": true, "tokens": 128, "train_fwd_bwd": { - "max_ms": 0.6033759713172913, - "median_ms": 0.5621750056743622, - "min_ms": 0.5227360129356384, - "p95_ms": 0.6017537444829941 + "max_ms": 0.6854979991912842, + "median_ms": 0.5381385087966919, + "min_ms": 0.5144850015640259, + "p95_ms": 0.6242975533008573 }, "train_peak_mib": 370.93994140625 }, @@ -2502,10 +3173,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.37095001339912415, - "median_ms": 0.278654009103775, - "min_ms": 0.24876999855041504, - "p95_ms": 0.3066359490156174 + "max_ms": 0.35492798686027527, + "median_ms": 0.27833350002765656, + "min_ms": 0.2521750032901764, + "p95_ms": 0.3129512906074524 }, "forward_peak_mib": 148.41015625, "grad_finite": true, @@ -2521,10 +3192,10 @@ "repeat_bitwise": true, "tokens": 128, "train_fwd_bwd": { - "max_ms": 1.0451929569244385, - "median_ms": 0.838605523109436, - "min_ms": 0.7780759930610657, - "p95_ms": 0.9953847378492354 + "max_ms": 1.0999130010604858, + "median_ms": 0.8765020072460175, + "min_ms": 0.794219970703125, + "p95_ms": 1.0784614980220795 }, "train_peak_mib": 296.78515625 }, @@ -2532,10 +3203,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.369828999042511, - "median_ms": 0.3232399970293045, - "min_ms": 0.32083699107170105, - "p95_ms": 0.3327618941664696 + "max_ms": 0.3689480125904083, + "median_ms": 0.3239609897136688, + "min_ms": 0.3197549879550934, + "p95_ms": 0.3447429448366165 }, "forward_peak_mib": 0.00439453125, "grad_finite": true, @@ -2551,10 +3222,10 @@ "repeat_bitwise": true, "tokens": 128, "train_fwd_bwd": { - "max_ms": 0.5017049908638, - "median_ms": 0.4668534994125366, - "min_ms": 0.4521920084953308, - "p95_ms": 0.49999229311943055 + "max_ms": 0.5482540130615234, + "median_ms": 0.48772451281547546, + "min_ms": 0.479312002658844, + "p95_ms": 0.5361944526433945 }, "train_peak_mib": 148.37841796875 }, @@ -2562,10 +3233,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 6.2147908210754395, - "median_ms": 5.9756762981414795, - "min_ms": 5.825572967529297, - "p95_ms": 6.2053526639938354 + "max_ms": 5.440080165863037, + "median_ms": 5.354552984237671, + "min_ms": 5.318378925323486, + "p95_ms": 5.409367537498474 }, "forward_peak_mib": 4.84619140625, "grad_finite": true, @@ -2581,10 +3252,10 @@ "repeat_bitwise": true, "tokens": 128, "train_fwd_bwd": { - "max_ms": 7.188317775726318, - "median_ms": 7.1406660079956055, - "min_ms": 7.0864057540893555, - "p95_ms": 7.186334228515625 + "max_ms": 6.947038173675537, + "median_ms": 6.898946523666382, + "min_ms": 6.871885776519775, + "p95_ms": 6.9410529851913445 }, "train_peak_mib": 445.2783203125 }, @@ -2592,10 +3263,42 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.43035900592803955, - "median_ms": 0.3568100035190582, - "min_ms": 0.34992000460624695, - "p95_ms": 0.3875843018293381 + "max_ms": 0.4412960112094879, + "median_ms": 0.40109600126743317, + "min_ms": 0.39162200689315796, + "p95_ms": 0.41389514654874804 + }, + "forward_peak_mib": 0.12841796875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3963434604136182e-06, + "relative_l2": 3.8277114241343485e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.125072301988439e-06, + "relative_l2": 2.914173983910747e-08 + }, + "mismatch_vs_reference": 5, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.4822693151362585e-08, + "repeat_bitwise": true, + "tokens": 128, + "train_fwd_bwd": { + "max_ms": 1.2369179725646973, + "median_ms": 1.0605154633522034, + "min_ms": 1.0286879539489746, + "p95_ms": 1.230464094877243 + }, + "train_peak_mib": 148.3818359375 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.39907199144363403, + "median_ms": 0.3528845012187958, + "min_ms": 0.34455201029777527, + "p95_ms": 0.37250998914241795 }, "forward_peak_mib": 0.12841796875, "grad_finite": true, @@ -2613,10 +3316,10 @@ "repeat_bitwise": true, "tokens": 128, "train_fwd_bwd": { - "max_ms": 0.9075480103492737, - "median_ms": 0.8009094893932343, - "min_ms": 0.7760729789733887, - "p95_ms": 0.8808863967657088 + "max_ms": 1.038902997970581, + "median_ms": 0.8048155009746552, + "min_ms": 0.787289023399353, + "p95_ms": 1.0180824041366576 }, "train_peak_mib": 148.3818359375 }, @@ -2624,10 +3327,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.8196970224380493, - "median_ms": 0.7682015001773834, - "min_ms": 0.7305650115013123, - "p95_ms": 0.8118575990200043 + "max_ms": 0.8246650099754333, + "median_ms": 0.7760325074195862, + "min_ms": 0.7508350014686584, + "p95_ms": 0.8231051206588745 }, "forward_peak_mib": 296.880859375, "grad_finite": true, @@ -2643,10 +3346,10 @@ "repeat_bitwise": true, "tokens": 512, "train_fwd_bwd": { - "max_ms": 1.847985029220581, - "median_ms": 1.784870982170105, - "min_ms": 1.7256829738616943, - "p95_ms": 1.8451365232467651 + "max_ms": 1.8172589540481567, + "median_ms": 1.7723724842071533, + "min_ms": 1.719514012336731, + "p95_ms": 1.8108774960041045 }, "train_peak_mib": 1483.7568359375 }, @@ -2654,10 +3357,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.9186049699783325, - "median_ms": 0.8245844841003418, - "min_ms": 0.7842050194740295, - "p95_ms": 0.8577517896890641 + "max_ms": 0.9324650168418884, + "median_ms": 0.8163724839687347, + "min_ms": 0.788811981678009, + "p95_ms": 0.844249901175499 }, "forward_peak_mib": 593.63916015625, "grad_finite": true, @@ -2673,10 +3376,10 @@ "repeat_bitwise": true, "tokens": 512, "train_fwd_bwd": { - "max_ms": 2.225145101547241, - "median_ms": 2.170043468475342, - "min_ms": 2.116302967071533, - "p95_ms": 2.2047388553619385 + "max_ms": 2.1884100437164307, + "median_ms": 2.1150814294815063, + "min_ms": 2.0706350803375244, + "p95_ms": 2.188031530380249 }, "train_peak_mib": 1187.1376953125 }, @@ -2684,10 +3387,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.4085260033607483, - "median_ms": 0.36115649342536926, - "min_ms": 0.3568499982357025, - "p95_ms": 0.3712318629026413 + "max_ms": 0.40275800228118896, + "median_ms": 0.36109650135040283, + "min_ms": 0.35961398482322693, + "p95_ms": 0.3949186071753502 }, "forward_peak_mib": 0.01123046875, "grad_finite": true, @@ -2703,10 +3406,10 @@ "repeat_bitwise": true, "tokens": 512, "train_fwd_bwd": { - "max_ms": 0.6068210005760193, - "median_ms": 0.5617550015449524, - "min_ms": 0.5477340221405029, - "p95_ms": 0.6044418543577195 + "max_ms": 0.5963649749755859, + "median_ms": 0.554082989692688, + "min_ms": 0.5410839915275574, + "p95_ms": 0.5855132400989532 }, "train_peak_mib": 593.5107421875 }, @@ -2714,10 +3417,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 6.162874221801758, - "median_ms": 6.032660961151123, - "min_ms": 5.892111778259277, - "p95_ms": 6.1107371807098385 + "max_ms": 5.619105815887451, + "median_ms": 5.537344932556152, + "min_ms": 5.441641807556152, + "p95_ms": 5.605291271209717 }, "forward_peak_mib": 15.46923828125, "grad_finite": true, @@ -2733,10 +3436,10 @@ "repeat_bitwise": true, "tokens": 512, "train_fwd_bwd": { - "max_ms": 8.313629150390625, - "median_ms": 8.26219129562378, - "min_ms": 8.17958927154541, - "p95_ms": 8.310960674285887 + "max_ms": 7.834836959838867, + "median_ms": 7.765453577041626, + "min_ms": 7.677241802215576, + "p95_ms": 7.826418471336365 }, "train_peak_mib": 1780.6689453125 }, @@ -2744,10 +3447,42 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.5036280155181885, - "median_ms": 0.44291800260543823, - "min_ms": 0.4307999908924103, - "p95_ms": 0.4818976894021035 + "max_ms": 0.5381199717521667, + "median_ms": 0.47991301119327545, + "min_ms": 0.46781501173973083, + "p95_ms": 0.5134969592094422 + }, + "forward_peak_mib": 0.5107421875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.3502658759989572e-06, + "relative_l2": 3.740717508776414e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.1845403324883819e-06, + "relative_l2": 2.8888100679186516e-08 + }, + "mismatch_vs_reference": 46, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.6010327429749704e-08, + "repeat_bitwise": true, + "tokens": 512, + "train_fwd_bwd": { + "max_ms": 1.4293630123138428, + "median_ms": 1.2251800298690796, + "min_ms": 1.1932519674301147, + "p95_ms": 1.355453187227249 + }, + "train_peak_mib": 593.52001953125 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 0.4631679952144623, + "median_ms": 0.43050000071525574, + "min_ms": 0.42539098858833313, + "p95_ms": 0.45030499547719954 }, "forward_peak_mib": 0.5107421875, "grad_finite": true, @@ -2765,10 +3500,10 @@ "repeat_bitwise": true, "tokens": 512, "train_fwd_bwd": { - "max_ms": 1.139691948890686, - "median_ms": 1.0942450165748596, - "min_ms": 1.0628589391708374, - "p95_ms": 1.135095661878586 + "max_ms": 1.1435790061950684, + "median_ms": 1.0652225017547607, + "min_ms": 1.0553679466247559, + "p95_ms": 1.1227039635181426 }, "train_peak_mib": 593.52001953125 }, @@ -2776,10 +3511,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 2.281989097595215, - "median_ms": 2.052648901939392, - "min_ms": 1.8154170513153076, - "p95_ms": 2.1600565791130064 + "max_ms": 2.212686061859131, + "median_ms": 1.968884527683258, + "min_ms": 1.815858006477356, + "p95_ms": 2.167170453071594 }, "forward_peak_mib": 1187.015625, "grad_finite": true, @@ -2795,10 +3530,10 @@ "repeat_bitwise": true, "tokens": 2048, "train_fwd_bwd": { - "max_ms": 6.306046009063721, - "median_ms": 6.086260557174683, - "min_ms": 5.54748010635376, - "p95_ms": 6.2793127536773685 + "max_ms": 6.234379768371582, + "median_ms": 6.0170769691467285, + "min_ms": 5.560098171234131, + "p95_ms": 6.218876385688782 }, "train_peak_mib": 5935.0244140625 }, @@ -2806,10 +3541,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 2.301619052886963, - "median_ms": 2.1072499752044678, - "min_ms": 1.874743938446045, - "p95_ms": 2.2768060088157656 + "max_ms": 2.2421700954437256, + "median_ms": 2.091405987739563, + "min_ms": 1.8722610473632812, + "p95_ms": 2.2312479257583617 }, "forward_peak_mib": 2374.103515625, "grad_finite": true, @@ -2825,10 +3560,10 @@ "repeat_bitwise": true, "tokens": 2048, "train_fwd_bwd": { - "max_ms": 7.16956901550293, - "median_ms": 6.578951835632324, - "min_ms": 6.511311054229736, - "p95_ms": 7.140474343299866 + "max_ms": 7.097422122955322, + "median_ms": 6.734924077987671, + "min_ms": 6.4440507888793945, + "p95_ms": 7.07494285106659 }, "train_peak_mib": 4748.0400390625 }, @@ -2836,10 +3571,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 0.7294430136680603, - "median_ms": 0.674140989780426, - "min_ms": 0.667352020740509, - "p95_ms": 0.692109915614128 + "max_ms": 0.7211099863052368, + "median_ms": 0.6786679923534393, + "min_ms": 0.6683520078659058, + "p95_ms": 0.7013974994421005 }, "forward_peak_mib": 0.044921875, "grad_finite": true, @@ -2855,10 +3590,10 @@ "repeat_bitwise": true, "tokens": 2048, "train_fwd_bwd": { - "max_ms": 2.0440359115600586, - "median_ms": 1.8701984882354736, - "min_ms": 1.8029580116271973, - "p95_ms": 1.9687563359737394 + "max_ms": 2.07828688621521, + "median_ms": 1.84471994638443, + "min_ms": 1.7994730472564697, + "p95_ms": 2.0619906425476073 }, "train_peak_mib": 2374.0400390625 }, @@ -2866,10 +3601,10 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 7.918561935424805, - "median_ms": 6.000913619995117, - "min_ms": 5.853054046630859, - "p95_ms": 6.208111476898194 + "max_ms": 6.019101142883301, + "median_ms": 5.579066514968872, + "min_ms": 5.509542942047119, + "p95_ms": 5.652730774879456 }, "forward_peak_mib": 57.96435546875, "grad_finite": true, @@ -2885,10 +3620,10 @@ "repeat_bitwise": true, "tokens": 2048, "train_fwd_bwd": { - "max_ms": 12.87468433380127, - "median_ms": 12.581206798553467, - "min_ms": 12.432106018066406, - "p95_ms": 12.77396845817566 + "max_ms": 12.192548751831055, + "median_ms": 12.105359554290771, + "min_ms": 12.014043807983398, + "p95_ms": 12.180759477615357 }, "train_peak_mib": 7122.23779296875 }, @@ -2896,10 +3631,42 @@ "batch_invariant": true, "dtype": "fp32", "forward": { - "max_ms": 1.0888580083847046, - "median_ms": 1.0044524669647217, - "min_ms": 0.9957990050315857, - "p95_ms": 1.0515999376773835 + "max_ms": 1.120864987373352, + "median_ms": 1.0601550340652466, + "min_ms": 1.0489180088043213, + "p95_ms": 1.0850918352603913 + }, + "forward_peak_mib": 2.04296875, + "grad_finite": true, + "logp_vs_fp64": { + "max_abs": 1.7105002712014539e-06, + "relative_l2": 3.6994881489459246e-08 + }, + "lse_vs_fp64": { + "max_abs": 1.1972776015056752e-06, + "relative_l2": 2.985702593475806e-08 + }, + "mismatch_vs_reference": 144, + "path": "ws2-triton", + "rel_l2_vs_reference": 1.4661270414586789e-08, + "repeat_bitwise": true, + "tokens": 2048, + "train_fwd_bwd": { + "max_ms": 2.9037530422210693, + "median_ms": 2.82489550113678, + "min_ms": 2.676975965499878, + "p95_ms": 2.8798858880996705 + }, + "train_peak_mib": 2374.0771484375 + }, + { + "batch_invariant": true, + "dtype": "fp32", + "forward": { + "max_ms": 1.0352979898452759, + "median_ms": 0.99720099568367, + "min_ms": 0.990149974822998, + "p95_ms": 1.035107958316803 }, "forward_peak_mib": 2.04296875, "grad_finite": true, @@ -2917,10 +3684,10 @@ "repeat_bitwise": true, "tokens": 2048, "train_fwd_bwd": { - "max_ms": 2.634474039077759, - "median_ms": 2.5475040674209595, - "min_ms": 2.4890170097351074, - "p95_ms": 2.6219990849494934 + "max_ms": 3.8982300758361816, + "median_ms": 2.7085630893707275, + "min_ms": 2.5650100708007812, + "p95_ms": 3.3876757740974415 }, "train_peak_mib": 2374.0771484375 } @@ -2930,97 +3697,98 @@ "ws1-pytorch", "ws1-triton", "ws2-reference", + "ws2-triton", "ws2-rocm" ], "validate_overhead": [ { "tokens": 1, "validate_false": { - "max_ms": 0.46549201011657715, - "median_ms": 0.36105599999427795, - "min_ms": 0.34206798672676086, - "p95_ms": 0.38701819032430657 + "max_ms": 0.38557198643684387, + "median_ms": 0.3474554866552353, + "min_ms": 0.34082698822021484, + "p95_ms": 0.36266274005174637 }, "validate_true": { - "max_ms": 0.5614339709281921, - "median_ms": 0.536176472902298, - "min_ms": 0.5264220237731934, - "p95_ms": 0.5588081836700439 + "max_ms": 0.5197709798812866, + "median_ms": 0.502346009016037, + "min_ms": 0.4909690022468567, + "p95_ms": 0.5150913119316101 } }, { "tokens": 8, "validate_false": { - "max_ms": 0.4257519841194153, - "median_ms": 0.363740012049675, - "min_ms": 0.3520819842815399, - "p95_ms": 0.38267235755920415 + "max_ms": 0.3962689936161041, + "median_ms": 0.35082100331783295, + "min_ms": 0.34082600474357605, + "p95_ms": 0.375033649802208 }, "validate_true": { - "max_ms": 0.5631170272827148, - "median_ms": 0.541905015707016, - "min_ms": 0.5269420146942139, - "p95_ms": 0.5520039200782776 + "max_ms": 0.5299869775772095, + "median_ms": 0.5066129863262177, + "min_ms": 0.4946550130844116, + "p95_ms": 0.524735403060913 } }, { "tokens": 32, "validate_false": { - "max_ms": 0.4319620132446289, - "median_ms": 0.3632794916629791, - "min_ms": 0.34891799092292786, - "p95_ms": 0.378378190100193 + "max_ms": 0.3971090018749237, + "median_ms": 0.34933900833129883, + "min_ms": 0.34283000230789185, + "p95_ms": 0.36491445600986483 }, "validate_true": { - "max_ms": 0.5548239946365356, - "median_ms": 0.5365969836711884, - "min_ms": 0.5251399874687195, - "p95_ms": 0.5512092620134353 + "max_ms": 0.5520200133323669, + "median_ms": 0.5064724981784821, + "min_ms": 0.4932529926300049, + "p95_ms": 0.5254570484161377 } }, { "tokens": 128, "validate_false": { - "max_ms": 0.4474650025367737, - "median_ms": 0.36904749274253845, - "min_ms": 0.35268399119377136, - "p95_ms": 0.38824863582849506 + "max_ms": 0.38757601380348206, + "median_ms": 0.35624949634075165, + "min_ms": 0.3489989936351776, + "p95_ms": 0.37349510788917545 }, "validate_true": { - "max_ms": 0.5527009963989258, - "median_ms": 0.5418049991130829, - "min_ms": 0.5232570171356201, - "p95_ms": 0.5510261595249176 + "max_ms": 0.5195310115814209, + "median_ms": 0.5095964968204498, + "min_ms": 0.4995020031929016, + "p95_ms": 0.518656051158905 } }, { "tokens": 512, "validate_false": { - "max_ms": 0.45796000957489014, - "median_ms": 0.3971495032310486, - "min_ms": 0.3919830024242401, - "p95_ms": 0.42089290469884877 + "max_ms": 0.42298799753189087, + "median_ms": 0.4019169956445694, + "min_ms": 0.3943859934806824, + "p95_ms": 0.4149196416139603 }, "validate_true": { - "max_ms": 0.5918390154838562, - "median_ms": 0.5795609951019287, - "min_ms": 0.5639979839324951, - "p95_ms": 0.5884522527456283 + "max_ms": 0.5864710211753845, + "median_ms": 0.5539029836654663, + "min_ms": 0.5408830046653748, + "p95_ms": 0.573798930644989 } }, { "tokens": 2048, "validate_false": { - "max_ms": 0.9521740078926086, - "median_ms": 0.8911034762859344, - "min_ms": 0.8809080123901367, - "p95_ms": 0.9250400990247727 + "max_ms": 1.5755000114440918, + "median_ms": 0.9115940034389496, + "min_ms": 0.8969720005989075, + "p95_ms": 1.4987020194530487 }, "validate_true": { - "max_ms": 3.0691590309143066, - "median_ms": 1.0742555260658264, - "min_ms": 1.0614559650421143, - "p95_ms": 1.2030188441276564 + "max_ms": 1.080083966255188, + "median_ms": 1.0558279752731323, + "min_ms": 1.0382219552993774, + "p95_ms": 1.0768872916698455 } } ] @@ -3029,24 +3797,24 @@ { "dtype": "bf16", "hip_fp32_input": { - "max_ms": 0.06269299983978271, - "median_ms": 0.020930999889969826, - "min_ms": 0.01818699948489666, - "p95_ms": 0.036663001589477086 + "max_ms": 0.020750999450683594, + "median_ms": 0.014341499656438828, + "min_ms": 0.011858000420033932, + "p95_ms": 0.019952050223946572 }, "hip_native_dtype_input": { - "max_ms": 0.032607998698949814, - "median_ms": 0.013680499978363514, - "min_ms": 0.011978000402450562, - "p95_ms": 0.01689119935035707 + "max_ms": 0.014059999957680702, + "median_ms": 0.012438500300049782, + "min_ms": 0.011857000179588795, + "p95_ms": 0.013946950202807784 }, "hip_peak_mib": 0.0009765625, "max_bitwise": true, "pytorch_loop": { - "max_ms": 5.524566173553467, - "median_ms": 4.910352945327759, - "min_ms": 4.4162797927856445, - "p95_ms": 5.317995333671569 + "max_ms": 5.015689849853516, + "median_ms": 4.467295408248901, + "min_ms": 4.213459014892578, + "p95_ms": 4.686615490913391 }, "pytorch_loop_peak_mib": 0.08154296875, "repeat_bitwise": true, @@ -3057,24 +3825,24 @@ { "dtype": "bf16", "hip_fp32_input": { - "max_ms": 0.025117000564932823, - "median_ms": 0.019989999942481518, - "min_ms": 0.016665000468492508, - "p95_ms": 0.02466100063174963 + "max_ms": 0.058646999299526215, + "median_ms": 0.01962899975478649, + "min_ms": 0.015502999536693096, + "p95_ms": 0.03893354982137681 }, "hip_native_dtype_input": { - "max_ms": 0.02619899995625019, - "median_ms": 0.022133000195026398, - "min_ms": 0.013178999535739422, - "p95_ms": 0.025628050696104767 + "max_ms": 0.015863999724388123, + "median_ms": 0.012879000045359135, + "min_ms": 0.011978000402450562, + "p95_ms": 0.01392315020784736 }, "hip_peak_mib": 0.00390625, "max_bitwise": true, "pytorch_loop": { - "max_ms": 6.1993279457092285, - "median_ms": 5.494100332260132, - "min_ms": 5.012845993041992, - "p95_ms": 5.790409994125366 + "max_ms": 5.4044270515441895, + "median_ms": 5.02221941947937, + "min_ms": 4.736033916473389, + "p95_ms": 5.30616557598114 }, "pytorch_loop_peak_mib": 0.28125, "repeat_bitwise": true, @@ -3085,24 +3853,24 @@ { "dtype": "bf16", "hip_fp32_input": { - "max_ms": 0.07238700240850449, - "median_ms": 0.020330000668764114, - "min_ms": 0.015623999759554863, - "p95_ms": 0.06005695164203645 + "max_ms": 0.02275400049984455, + "median_ms": 0.01882850006222725, + "min_ms": 0.01510199997574091, + "p95_ms": 0.021879050694406033 }, "hip_native_dtype_input": { - "max_ms": 0.0392180010676384, - "median_ms": 0.022373000159859657, - "min_ms": 0.019509000703692436, - "p95_ms": 0.02867680042982102 + "max_ms": 0.017464999109506607, + "median_ms": 0.015683500096201897, + "min_ms": 0.015583000145852566, + "p95_ms": 0.01731394995003939 }, "hip_peak_mib": 0.015625, "max_bitwise": true, "pytorch_loop": { - "max_ms": 6.249682903289795, - "median_ms": 5.507360935211182, - "min_ms": 4.994658946990967, - "p95_ms": 5.8171706914901735 + "max_ms": 5.307322978973389, + "median_ms": 5.033536434173584, + "min_ms": 4.758028030395508, + "p95_ms": 5.305229234695434 }, "pytorch_loop_peak_mib": 0.93310546875, "repeat_bitwise": true, @@ -3113,24 +3881,24 @@ { "dtype": "bf16", "hip_fp32_input": { - "max_ms": 0.06762000173330307, - "median_ms": 0.0408409982919693, - "min_ms": 0.03881800174713135, - "p95_ms": 0.04695559833198787 + "max_ms": 0.043744999915361404, + "median_ms": 0.04050050117075443, + "min_ms": 0.03941800072789192, + "p95_ms": 0.043555000238120554 }, "hip_native_dtype_input": { - "max_ms": 0.03449099883437157, - "median_ms": 0.0323684997856617, - "min_ms": 0.03172700107097626, - "p95_ms": 0.034224050119519234 + "max_ms": 0.03417100012302399, + "median_ms": 0.03240850009024143, + "min_ms": 0.03132700175046921, + "p95_ms": 0.03356109857559204 }, "hip_peak_mib": 0.0625, "max_bitwise": true, "pytorch_loop": { - "max_ms": 6.158267021179199, - "median_ms": 5.456785440444946, - "min_ms": 4.993456840515137, - "p95_ms": 5.7379707336425785 + "max_ms": 5.240784168243408, + "median_ms": 4.972766637802124, + "min_ms": 4.74508810043335, + "p95_ms": 5.230013751983643 }, "pytorch_loop_peak_mib": 3.54052734375, "repeat_bitwise": true, @@ -3141,24 +3909,24 @@ { "dtype": "bf16", "hip_fp32_input": { - "max_ms": 0.16484500467777252, - "median_ms": 0.12496599927544594, - "min_ms": 0.12105999886989594, - "p95_ms": 0.14391364902257922 + "max_ms": 0.14028899371623993, + "median_ms": 0.12786950170993805, + "min_ms": 0.12033899873495102, + "p95_ms": 0.13621635362505916 }, "hip_native_dtype_input": { - "max_ms": 0.12794999778270721, - "median_ms": 0.11775499954819679, - "min_ms": 0.10916200280189514, - "p95_ms": 0.1253622055053711 + "max_ms": 0.12490599602460861, + "median_ms": 0.1133279986679554, + "min_ms": 0.10651800036430359, + "p95_ms": 0.12429705001413822 }, "hip_peak_mib": 0.25, "max_bitwise": true, "pytorch_loop": { - "max_ms": 6.2457170486450195, - "median_ms": 5.5467188358306885, - "min_ms": 5.0723748207092285, - "p95_ms": 5.867206406593323 + "max_ms": 5.42746114730835, + "median_ms": 5.089759588241577, + "min_ms": 4.867790222167969, + "p95_ms": 5.367902684211731 }, "pytorch_loop_peak_mib": 14.16064453125, "repeat_bitwise": true, @@ -3169,24 +3937,24 @@ { "dtype": "bf16", "hip_fp32_input": { - "max_ms": 0.5361970067024231, - "median_ms": 0.523857980966568, - "min_ms": 0.46677300333976746, - "p95_ms": 0.5341791957616806 + "max_ms": 0.5528209805488586, + "median_ms": 0.5171479880809784, + "min_ms": 0.456277996301651, + "p95_ms": 0.5297968149185182 }, "hip_native_dtype_input": { - "max_ms": 0.5962859988212585, - "median_ms": 0.46587200462818146, - "min_ms": 0.45467498898506165, - "p95_ms": 0.4989755868911744 + "max_ms": 0.5951640009880066, + "median_ms": 0.46687400341033936, + "min_ms": 0.45643800497055054, + "p95_ms": 0.4840007096529008 }, "hip_peak_mib": 1.0, "max_bitwise": true, "pytorch_loop": { - "max_ms": 6.220038890838623, - "median_ms": 5.557474136352539, - "min_ms": 5.107827186584473, - "p95_ms": 6.041820120811463 + "max_ms": 5.4985270500183105, + "median_ms": 5.0989930629730225, + "min_ms": 4.875161170959473, + "p95_ms": 5.356805205345155 }, "pytorch_loop_peak_mib": 56.642578125, "repeat_bitwise": true, @@ -3197,24 +3965,24 @@ { "dtype": "fp32", "hip_fp32_input": { - "max_ms": 0.03917799890041351, - "median_ms": 0.021391499787569046, - "min_ms": 0.01742500066757202, - "p95_ms": 0.02791384961456061 + "max_ms": 0.024315999820828438, + "median_ms": 0.014480999670922756, + "min_ms": 0.012218000367283821, + "p95_ms": 0.0229460995644331 }, "hip_native_dtype_input": { - "max_ms": 0.025918999686837196, - "median_ms": 0.013819999992847443, - "min_ms": 0.012338999658823013, - "p95_ms": 0.022494250815361744 + "max_ms": 0.013380000367760658, + "median_ms": 0.012277999892830849, + "min_ms": 0.011216999962925911, + "p95_ms": 0.01326600038446486 }, "hip_peak_mib": 0.0009765625, "max_bitwise": true, "pytorch_loop": { - "max_ms": 5.71785306930542, - "median_ms": 4.998143911361694, - "min_ms": 4.4156389236450195, - "p95_ms": 5.3919374465942385 + "max_ms": 5.198602199554443, + "median_ms": 4.477289915084839, + "min_ms": 4.219387054443359, + "p95_ms": 4.824810409545899 }, "pytorch_loop_peak_mib": 0.08154296875, "repeat_bitwise": true, @@ -3225,24 +3993,24 @@ { "dtype": "fp32", "hip_fp32_input": { - "max_ms": 0.05628399923443794, - "median_ms": 0.020610500127077103, - "min_ms": 0.012218999676406384, - "p95_ms": 0.033030851557850856 + "max_ms": 0.023475000634789467, + "median_ms": 0.018106999807059765, + "min_ms": 0.012178000062704086, + "p95_ms": 0.021115199476480488 }, "hip_native_dtype_input": { - "max_ms": 0.03176699951291084, - "median_ms": 0.020770499482750893, - "min_ms": 0.013019000180065632, - "p95_ms": 0.02826625006273389 + "max_ms": 0.01734600029885769, + "median_ms": 0.012338500004261732, + "min_ms": 0.011617000214755535, + "p95_ms": 0.015100200008600953 }, "hip_peak_mib": 0.00390625, "max_bitwise": true, "pytorch_loop": { - "max_ms": 6.11968994140625, - "median_ms": 5.53972864151001, - "min_ms": 5.055549144744873, - "p95_ms": 5.798912191390992 + "max_ms": 5.431427001953125, + "median_ms": 5.071452856063843, + "min_ms": 4.788833141326904, + "p95_ms": 5.370270824432373 }, "pytorch_loop_peak_mib": 0.28125, "repeat_bitwise": true, @@ -3253,24 +4021,24 @@ { "dtype": "fp32", "hip_fp32_input": { - "max_ms": 0.07563299685716629, - "median_ms": 0.02119150012731552, - "min_ms": 0.015943000093102455, - "p95_ms": 0.05310280136764051 + "max_ms": 0.0389380007982254, + "median_ms": 0.019168000668287277, + "min_ms": 0.015022000297904015, + "p95_ms": 0.023030249774456037 }, "hip_native_dtype_input": { - "max_ms": 0.02711999975144863, - "median_ms": 0.02153200004249811, - "min_ms": 0.01886799931526184, - "p95_ms": 0.025712099205702544 + "max_ms": 0.01642500050365925, + "median_ms": 0.01522199995815754, + "min_ms": 0.014941999688744545, + "p95_ms": 0.016310999635607004 }, "hip_peak_mib": 0.015625, "max_bitwise": true, "pytorch_loop": { - "max_ms": 6.234379768371582, - "median_ms": 5.507981538772583, - "min_ms": 5.032916069030762, - "p95_ms": 5.821695160865784 + "max_ms": 5.403584957122803, + "median_ms": 5.046475410461426, + "min_ms": 4.80033016204834, + "p95_ms": 5.3525518655776985 }, "pytorch_loop_peak_mib": 0.93310546875, "repeat_bitwise": true, @@ -3281,24 +4049,24 @@ { "dtype": "fp32", "hip_fp32_input": { - "max_ms": 0.06401500105857849, - "median_ms": 0.042402999475598335, - "min_ms": 0.04058000072836876, - "p95_ms": 0.049896100163459786 + "max_ms": 0.047189999371767044, + "median_ms": 0.041881999000906944, + "min_ms": 0.04086099937558174, + "p95_ms": 0.0458960996940732 }, "hip_native_dtype_input": { - "max_ms": 0.04562800005078316, - "median_ms": 0.04182199947535992, - "min_ms": 0.04033999890089035, - "p95_ms": 0.043649150244891646 + "max_ms": 0.07026399672031403, + "median_ms": 0.04184200055897236, + "min_ms": 0.04010000079870224, + "p95_ms": 0.04438599962741138 }, "hip_peak_mib": 0.0625, "max_bitwise": true, "pytorch_loop": { - "max_ms": 6.129784107208252, - "median_ms": 5.4718873500823975, - "min_ms": 5.026266098022461, - "p95_ms": 5.79066231250763 + "max_ms": 5.275114059448242, + "median_ms": 5.003592491149902, + "min_ms": 4.78242301940918, + "p95_ms": 5.232073163986207 }, "pytorch_loop_peak_mib": 3.54052734375, "repeat_bitwise": true, @@ -3309,24 +4077,24 @@ { "dtype": "fp32", "hip_fp32_input": { - "max_ms": 0.1773039996623993, - "median_ms": 0.12897150218486786, - "min_ms": 0.1261879950761795, - "p95_ms": 0.14754240661859516 + "max_ms": 0.16576600074768066, + "median_ms": 0.12873099744319916, + "min_ms": 0.12374399602413177, + "p95_ms": 0.1486413061618805 }, "hip_native_dtype_input": { - "max_ms": 0.179066002368927, - "median_ms": 0.13676300644874573, - "min_ms": 0.13492000102996826, - "p95_ms": 0.14382479414343835 + "max_ms": 0.13616199791431427, + "median_ms": 0.1302734985947609, + "min_ms": 0.12626700103282928, + "p95_ms": 0.1339931555092335 }, "hip_peak_mib": 0.25, "max_bitwise": true, "pytorch_loop": { - "max_ms": 6.199489116668701, - "median_ms": 5.575060844421387, - "min_ms": 5.104261875152588, - "p95_ms": 5.851804351806641 + "max_ms": 5.358519077301025, + "median_ms": 5.106103420257568, + "min_ms": 4.881009101867676, + "p95_ms": 5.346721720695496 }, "pytorch_loop_peak_mib": 14.16064453125, "repeat_bitwise": true, @@ -3337,24 +4105,24 @@ { "dtype": "fp32", "hip_fp32_input": { - "max_ms": 0.6945109963417053, - "median_ms": 0.5459109842777252, - "min_ms": 0.47338300943374634, - "p95_ms": 0.5795429408550263 + "max_ms": 0.686178982257843, + "median_ms": 0.5328314900398254, + "min_ms": 0.4858019948005676, + "p95_ms": 0.5776804357767106 }, "hip_native_dtype_input": { - "max_ms": 0.5639979839324951, - "median_ms": 0.5364370048046112, - "min_ms": 0.5297470092773438, - "p95_ms": 0.544399482011795 + "max_ms": 0.5658810138702393, + "median_ms": 0.53313148021698, + "min_ms": 0.5255410075187683, + "p95_ms": 0.5386321574449539 }, "hip_peak_mib": 1.0, "max_bitwise": true, "pytorch_loop": { - "max_ms": 6.186348915100098, - "median_ms": 5.5582358837127686, - "min_ms": 5.091322898864746, - "p95_ms": 5.904921054840088 + "max_ms": 5.419569969177246, + "median_ms": 5.102478504180908, + "min_ms": 4.886377811431885, + "p95_ms": 5.320242595672608 }, "pytorch_loop_peak_mib": 56.642578125, "repeat_bitwise": true, diff --git a/benchmarks/results/pr328_rocm_mi300x/single_gpu_latency.png b/benchmarks/results/pr328_rocm_mi300x/single_gpu_latency.png index 08bd5892c5f0be4dd5cd7d0d8121e38cbd05e86a..af279da3a237fae691dddc232858dce7d9ca9534 100644 GIT binary patch literal 91786 zcmeFZWmuGL*EWn@h?fDs#9xTVT0ZIoMeR zSpGJ$wY9VnWMeb`pJ%XISR1n4km%5WS2<<*K*NTJh$;*HJ)}+PPfc`)h)C+*9Tmrj z*#QSv<<-5~MThosqcg`&2b2e2e&&@!ee(0L-QQlP=d7$|R+*TF?X>3YMrpb_Yk$4Cnj1h;B5CdJktaZJ9+t^He*X z@Af~BuNLWhB>(djd=is=?ZuJ*`o`Cm2gCpMCD8)c1^)m3ddKJJ|Ml}?|NoEwXLfB)W@C`)ygPE@OK&^b+XYx0`@vu7JZoyT?i@{Gs* z)V-@ehg(z>E6wb#SMJ+s%r|ni*y1zA-;%)fPPfLs!jIt#%sV0kY;3DZ)ssE;_e?qx zKim};C#9fB$;6aSdogFW=IA#t?K=8W3w);FGDwz>7WMM-QhNB%=QuHWOf8*Y!zo#`7-&s0pQeVDLQPR0J7i{FfF2>Tr}@B4=i9nvhdH-Ce7+S{CH znpnuLrA@PSRaM;Lv9CCK{5TUUt6Wl~eHg9NpYt>#&Ut-qffYqAP2OH!D|m~_sV;S$ zuKXVDv`=z^QZHV-;GukO{_D$~!Xfuk=k>Xa$w{O8_wFStC&>v;MA)o!D$6qp2o!7X zss-tN#e|-DWeq#NL8XTsdfG^!)tgIQuMP?4;DbH5afC?0Jlb{X{dp=X(+8;w)^*o| ziwITOuH%iN6=y5F)Y2d6)|5Q0K3!SOVK=Q3Jdv-FYansw&Yd?it5Y4d`sG~B{8i?q zYXm$M&Idl0Vnp@p1Ek9bFKb!8@c;B_lp|3!O^pIJ>}H`~bEFUzzR+U*m!zVizZEQb z0RCKVKqN&MC0@G+v~ zpg?Bz=@D->HMqUQhY#Csjs?&3+qCes+z_yx{MeoGIEk%tM`@-zgSn8dirtmmu1Yui z(Isl?biG>NDY*Ua*QYMln{RK=SDG%5H{s%2TNTE}##CKOVZ#N-0+em1Q<7340!_M7 z14l+iY*!`~&HM7C;6-!n=La+V#P%MkYda;7v1&~gM90U|%SP}k*80*Y!%^AWZ1FIe z{gutc!qO>8=N2EV?JVW7x0%!|vaXf!xKOUGtqr?fygul%+AK8x0C)T>WxUH=u_TS4 zZEJx!eq}Pw&t#%G(s?$!7KQlkDBb?eT=%WUI-=xNe0$XTT zb(ZfYGqa3ZnyN+FqUYfwru&9J+7o2D;AfZZ@7#HT!{JuojV3?Ii?(I!)pA>n+#~$T z7QgklAYDl(;n;~2!No0`yA+g^sqVX5QIC^U^p8KiUjN z<-}yemZ;Nb&vt!#bt2z+b=rD8xG%@J=zdr@_Uw=K>*`4>` zqo=s6b!6)9XVX4*0p8239V- z>(*3~evsJqZ0`8d30lXm=5&y5xLQ~QU@v{j%7lZ*Qc_YRV2931N!Z%vS%BE=OV2^C7(EVZC#0l5`o0I%66+ z&)zTAGiQ-k?ro(xt$nhI4qRPoU<)@iEMIAtp~b6>&lZGPic+1lJ41?Uqui550-aB> z{1X)Ff(M~aVIx$7?@CcoP!}CC^*T+hDt?TFh(|XAJ92`E+u+1G+Rm-jnd?#$(H<3Y zV!O6}1edkh6sb=)Hm~MMj6t6KNbd$|P0)R(2+zwKE<0bo9woCKcJk!Og(6{YZWWZD zownc7x!>#AuXaD~Ds`}sYKL5`qM)FV*p%BM${gNw zBgHYQ|KrEMg2%G6vzP*6CJe&Fk`ndL8KxOwxz=H_Mz?1@(MODQR-HX4Wi zD^eyy73Gjd{g$iNhwLY&y$c^UY_^nXHLN=!5cfO~2g{J#0-siBBHpu*sl9hOIpQin!-ZK|xJ_e}78$C(onY)-*`T$kd85#u|c6M}E9F z>CH(%wJYCLeZZ=gM$mHjIeln?QIc%9T3qeQR8sUo5-GJOT!I4-5<-5KnqMK;utiN% zQ}e@?J+J$YV|7gp6C0a6o!g2(o_EM?J(7e*F#hY;+p}QR%9oB_=NS7yC)FmW5)78@8 zV2XOY9VLDL{(Z@8PDA6mqnTOy@pH84kaP0-kCZC)w&d;XxQgtqbWjRAehK3>XM!4S z811HJ+MUjU&(JDvH`j67TH(O6Xy*Mi5C2{Dw&7{E5hO1CAe8F%_tq=zx-|?_WW#w+ zlaQ#Xs05Ta%x6OGtoW`}*pVpvsrsqnTM`<2sS_mBts?9FaZw^JOq`rbP=BGyxP+Q( zyKj#DTjjed+*~kjt6hC1c4@WdPTm999w_#<6Oj(n0m^cUP+2Zrx)h%%8*VcE^@a2B zixUA65wrG~+EH0)X&il1IA4qF9!;=foJ28Zec^Xi7>AzcaT4le_yHz*`g`X1g06U} zPkCm&Io6{C0D5}YAxy4I>3Hns*L&p=2`D!0KLSKHM*D)fZ06&gGL!V*_ zo9)AT*nQnbj96`d;Mfl4A{A8{l+?47+z)KV8#zAJ^rzH?ROCVGDc>0J7KFrfkdarX zda|Mu<)bs9RIAQpqnweWnE@DyQp~-~+QvpzMI|0y27p{sln9>S441&SUicVWTx}kH zg3eU|u1Zl+k%@`PTEl*_^-B4|_cJB%x2xF35blE+t{X12$9&QojWF8T?N0VnaqlU5TQZ?zxjJ$jIZfB}0 zGsbx;VLZ=_bQ4l2I_zocS=wH{zG_exp>D-3j@Hh=%UW*Aazj>!yIdvU3%|P;S`1-D zPx#)G{W`JpnCK|cbkgcfPx$r58*&kk;wq?j8zTgA8#yPeH^TWWcqKki>JPHIQSf0S zYL()e_3_d=nt4XL>5mJW^^AAw1GZ&kWqBZRu2>h2{_39YN=w@`D-N>ZA+9R=?MX!} zRRc9{prCTZiy1yIo`^Cf_f#CU&J45~3sevDTZih$mb(KfoiCV9)U~MJT`f!7yufAQ z+X+k%tZjpqJ)}fupNZMr7T)OGot+(PJRk=O5$6?ldI}B>jvA}p zzp@CxH8VRD?n%r+Pf_l?y7UX^q6&%L94WT&TSzIn-dvdO0cyTTZWFxU-)(z3@VZP8xnY!( zL2*k19Q+TBjSmZVJD)wApAFV==jbaif2^yMpt`anvNahuUJ|Tj{l3BWzFv*b;s&NY zNuJJTpu%mt#GK1@!xDgmK_hwaHlTdsv@XrxqoKeW@OtMw6*t#?t00s3K$-|D&45$( zr8`L>rEa@%b5Tv7Zew?SsA+z2iS^a-vrOxyJ(~`#-`|}1@Z}48q1$UuPv;2HEC9?Q zqHZM(69J{(%Z8iGS!GWL_Z!$M9bM<3IBSbD?P-g_HrE#1Hjh1gf=ETYB}#;pl9DN9 zqme|ixV6uAz^-QsplAJrb(wq!6 zjA{cCxKeda%m~;+eSLj<^<|y(h=>T2+TiHuEUp$|<%VE3_1m|fC(1>pLRA^+Z@%Uy z{26K}uoVp;F1}7=4zd`?(71Sj(kAb_!KK;yEGI_Op zcMXq=#}1Sfi=5E~ScQv+LYWWAw%^?aaIpc6Lt!M|X}Z=RcM5Rv%|gtB2k*392t07S z6Oxi zH`6`cUz{7pZTwlc%5!SePi)O5k92!NY(J$(+eKDsac>vW&@{wA8}PEUnaol-$c%5# z(RIVpXpHYZ>Vw#fgUU5EotD+zqvMeQuQ(OwC)y5&hws?&<0ikp=;FLugqaAi?hcFs zvzbk>ppBeAuSxpaYCy4z|M~L)lz5G-^0hFvOsy2CC8|1ZtAVs`D~kC>ZFlWUF^CaX z?5;ib^ScHB{4&PY|Mcn8E3M+^4)(Gy-*gq;Jk_`(5f`6t+G8@^nZk7Qrh=nm5v}v& zl?=^%X-JOUh-X7~oq>eY1DQhvo0bTy7a@PyXi*H5`yP3*-FS5Mun65^dyD7ZyncQ2 z)-74Y`hY5r{!ZLpI^O=GMh!0P|R9SRQ@3t*&MDCMP!+RR|GC8RP3c zh?H23)=1$7U6*Bdm!tPr$hS?i$`cu)U6l+YtUSqB9>qZZHLb&4A9wM9SS^777|0Bd zjLd)vjBYNN{w$543gR1(95Zy>H}9Xh%-Si-HxLg2R-Eq8#MASDDYND`|8{#MB=~Su zzRZ$oCO#~*(}5B@bOP+^ z8Iv9`MUkbJO(mOfCOZ7e(f5Y@p8()n1~wd>cO*q1S_KLJ6S^Uc!673}cV|Wf(Omq-52V2%IXcWf1|j1exH2hEDj}FjSXc|!$dok)EU0|BSqK*o zcLJ}aHdb)VBv33@k|;*BbaOl$!zl=G+k9t6XP;AadzNdoHC8-uP7=#`mqyyK${p&W zeqVe-0>$Fu;+VnBiScntD=RBc`Z)cDpqjyQmxerzQu~ba7cM--CJ7oI8SvQO^;B_9 z%&|FMmRs(!P6_9Llb!;hFFY?PqZ{wX0W2TvhtUJhD94dg!397TGq2Pcwa32)i9?Uv z)1l=!0R7v=6crFye7qKev-TpAp7znAZcP22qdxCq#ow0n?;}QrhmznYytY`mzuRI~ z3TWqa;S_&W{pJ!v*F{l<{L$vQKri%Q#m2`VMevAj{<%2%*_+k6ZaPa&CxMMOJfnV5 z_1XMjxdGl|XOyl6?p(ycbI7(sc5F?R%J7JjvuFW!U^-{e^DI?k;~FsRUIN9H{dJFh z!!@_I`>@Gcncwf+||P8|k`K_YZ9)^nQF? zh~@bl7uU3Bl6JTjKzO|f{s%j;9t5@ofc7hOA%K?gyiv`RcJ`32mpL#VyQ^c^p2Y#( ze9aHAiox#-Kd--qbqB0$=*~thU>*N39FHw+njOt7onzt~PfusL#>3NzpvBw zp#^0836a15{<^U->eoO3MzP0a6J2_mnm4?u2>4}V(LA8l!d(cBdNbvcyFWquGkd@Z zhsMB`(AV`*geAniK5Y+#bG;i2;HaYUFjHbFiF@~6`$t7-C(5c03L%lIx4D?< zt+;_fVzIp1T5e;c7n7yqvYT4yt}_$>PZce~BqRdt!L=Fi1VC-cXtbV~rh!_|=}RZ3 zo!A6f7h9l_XQUW9hP1l|mtG8=r(&5!P-f?J@^BaI#x*v!to~Ys-g%E0UjMI$h`ItI zBh7<6Y#gRKQ$AUNJVOpCXYrT3{<#~)R3QTW4nuARrR`#O5!hn2S+A=54Xd2G_AMT; zjm_UH5V|g|Wo+duxGQxhPoGY;rI0a9ICq1>Q&)K5>k;g^M1lGcZ2;ZlZuxHPbtd_g zT!ziuK(OmEY{J6*ADihS*sL*@Tj{-t<&=`09@)^Dy zTSI3%cK~q4bPfvyFH0Q=gRRnZ-1Z?NE(OJ6rQ4v4D*A8kiGbvj+6e;ff)B^~Da7nt zE+R3)698#yS>V?;U^QI7GxCm{APMRC`Y?n&wjR$x+@iMyKrO^-2g+x<^-#h^eyIIE zGOWcvW|#dXX&D^`fP34Rp<6Fyv5BxnNItyoZ=}u0goCb2f!lr^SI-ZH2V_*x#ejM~ zGH}u#5^{2Kk^mxqrC1KH0&ICkYjD0xQSq>(zRH@LTATYWpwaN%069@+^Y(ZtPkKEF zWBnij2Frlp0K1(Ht7SLKZ9-wN#@vHW0J^D5QryhyncRhyPre;4G^gIS1+br*pLJ;P z`z0nK1=-)fe{aF2@p$aH!r90L>O6zQZ^&~z#_ja_1_s?mE|8iviW}raH=iwUL2}Uw zx}{U0g}X#fo?Q2dAs8>|G`7j4H&!25|MTZZdW>z`eOkOPImca`%A-fgfFjgg!kekH zEkID9;==~e(}=o0f;w-bF=vj%zv8AJKkkXF{n9fTD9Jw%I8kz~7>8BBsiauHzH;RX zF228CtI$m)kxr6sZp+FLh>*0543|Yumy#7Q+t#$K3KKbRh0FSehT%QSOG}*qnlm0{ z-vutD_UKW_)2B~uQMKdXPz0ugi|@@fEWMQ>Y~IIBQC(MvIw@JSFY6ZFr-7FvwF+B` zz@(B@-8X>6LHN0L8Mf7;aySRfEG!RkT3b@^-VOW!yxLhJr}`6PV!lSSf3`K2bpY}n z7!VK-P{af>Z(o^HF6h#YV^q9m3q=)4wIhy$^u_=I#|pv)?NV9DWZq;uD}$1Q)MU_N zL;OlX;lp{!NC?&cR(q>g?|sriPB(|<|NCnQ(Oyk`t zo=#1BmamDCHWJ*1s@B9=_AnaKu>;4r=nIsNJwSYX>{HyJu2b7T_b`mR&UH`*pjCde z=(d?uJIGSc3h^Rqs$CqrAh_ykR}u>FCQIxA;o)iTFEUI6y9fXk>kM1j6S>$CIiT>H z7;WcDyM9&+k5A9sNg}Q58A;u5;LUhx_$_Zkw}CG@_|=JXVI7Ix&paMix|eTm5{x~l zK$a7*^UmsAaT|X>N-huOv@6$8-p`z-QXJ^VWl~boVso2zDTYwx&OkC!QhhlOr7b=r ze{J@10de%}h!4CBC2oMeI9lSmhzKUyZ7r1ExITbUHA?#A!VTy^#vBJdjJ0%@6SOXQ zLF+&75GEXeTtf+^tl{zw@HHOyoegV`JQbAk8ZhG_%Zf^&N638zj)L@;nw*?$y*=5H zACnX)PQler8P`X4MrVO`cJ*l+10wB+x$}x%3Lz5uDL9A0O$N%K%L5)A z3nT<|1HgtpjU+IcF6e^TfmCRWu;u2TPpgA2^Tu#cL9%rWU8S}WP#=$u^DW+!J~u~= z0()Zfa03;+;JO_lzo&%w!bDSe5-GEq3c0EIFX){(5mmDj_nPruXJD8CUA5!uizE34 zO_U!fxFkT7R1Dl^asv%;(Rrsi>l3}m&O+6=t9Nq=+*UZYL<$!W7B+27^qCj1N_$5~ zlRap-z)XVUM_vg>L8Yzh=P zV3$D}2|pp%AQTc18k*v`m%^S1Bg<~yE9Wlu_Z$R}abv)`Z1r` zURlg*_UrE|c6!x$azA>csN&)g?SM^VZ4liUKEn0l(b(QhGRuB{?syDIv*9H;!@>H#>BB`u{C>M9jvl?q%UhtCE#OY1 z9$3mno7!Mi(*m@Qt#kkz#VDd*e<|AFm!za*_^W(4C^OJqW$vi4QfS$H=)c6@cTYm1 zgb>HM%g@F(Aa4wcKoNjsCMlF^izBHl%G$!j;bm5p!;|sG8(>6yffLu-C$orRRXb4uATARNvUsl&$ed`s4 z58DE`h1wzIz|;CFJv`nJm~GZ#grHf}wKefRY5w)J`-#N(_>i777lTpI6t1OgJgkfM z*y}AD+HCXzVx|QR+OBE3e~4|W-oFja?Gt))o*ckyy`EZXYj3S)OrfyKH13K!(R6qJfj#8vEti%3U$NM7I2&{1SH z$_;gE7cd1YX+VF<%PmTN`XB&ya?5vx7a6HTnU#ETr5rF^<8`zDX0qVOe}MT zpu5h>OpiRsN@;5Q&kH{SO3kO|oW<#+U*vVyEmi)Y415I`_YOtTAR&jFTZ|$R4Uc-OJGzKmiDA| zx^jtM{mN)rnXOne9Bu>J9Tfg(CzjWsExih?%(>q;3*_n9_LHbh=s%cR2Q{(8_~mOA~zSN&Cku z;{8!$j!3^XY|?Tei+z)73GbI3zMtjmSD2cAjlNgom&*&-?+z*qe>C5` z;^G3p$X3(uQ+&OqKmR>+J=w=Iw?-F&(3AUHIwWbG?kO#DGgNH#+l@DHxZMZqfVul| zxRmAb6-rX8K^^LCg{fUxrBGuPY7ge7*G z>2t#FPE6$yqUWA-VWycJCw}`{zpMDeUsWq0-bEAMG>%);A%!BCFfu!`v9V#Ys(cY{ zum@7=3gE!0D3>{Z`dxyB2eX4{c^}#3%Sn+!4$$#)(+^sm?Ny%X$x?TAE&(j0H0SCD zSj&Y z7O5%I=IiU!HhJCG0ji;<0!qoxJ_L`K3A2B}?%uNzRbGJ7N{bg{b71Ac(15!{6|}~s z<>kPTkYu0-;n&3?TS~~G>aWfA#sRd9gav`laCCj1)vpS-Qum!jNvU?ofh+SK`*=b@ z+P2f9uw}z91K4#(2?e_rqhudH(XF2me7mSy4!w`x1(AHT4pTh?I?+{-XTCtc%UZzb zvBtM=-%6l?umX5v=G8gTR*;M0pm)j@s`vHzA<`fQ=zHhC zl1eXyd>QFqzI+}seUqhsMSMKe0u-XWo)s-+b%-V&YVe@$|_MQpzlX}X~P zsX$d(E{ckK+5kds7TgsQQS_NA_%ka(m2<`sr_4%-S^1 zcm4VE%nmY1e?h*JPi@aUSMB`A+1Xh^JD;SI%#WG*z$g+RJ_kyT$LOlMn=XyWK=1b@*TB;VgfBx5v6LsQclBH zsQKkEKZx7ej{LjWUCe1KJ^h^?WgzfjH;Ao>&_hJGX}*Wx9iV4X9fIE{w^4^ZA4;3F z5@L#lVweiBXnc2&yhEr$2GG8_yQi8uXta5*o7Q%dt#{BZ55GLK0u3{@qA0!SO$b+8 zuqSZo{wzNHc61B0r2rgmghTfx505I8@Sp>dHAb5Y!o}O1oE)a@h5E6bsI*yA4XUzL z(KNGay4s3dn+dr}HFiX5C~RbX7eRm#7ZgV1f9skV}XaO=$0WCXy?pzO)gXv&? zWVQkQQflr*2b6R6b1Z_vqYeh-o(L*QRWG~w(u(xy7sH{4x1}fDH)5MVu?e)?#9(qC z%pMMS1L!@JK-AKc2Hzoq$(|>`+K=3?h#+Fz11_(R0myrGkdt+ux5})_S77 z4+pQoe0{N&F1$xxh>+JuoT5o>b7~(XfW9xf;>CnP$I-LUI*Bf((ozqM^F#5AXlHSj zZ8*f`0f&emp?|3?Cy?;a!`I0zVQFF{4^J2$1=EK>2+h2)16)2qb9dHOc&@Ch%zEO! z2=I!{oxobF0-5{!u9mK%=7VJ#0P-=aRjXmpP9E5C592bFdt>uGC<*u)rJ!wc{78#= zt)T;?C`3-Oet@4e6|%xSWCd@I!hOI86rh3UTmHVwbt*FtvRU!NV7;X;JpW`y7TWhms86>K;sCtHIx=9Sm65nYVWi`t}EkJ zupuKOOGHH&6ztyKUc@-mw}*;tlE4)AJG%*xdn4U*+%^{uPoSEl_eG(g zBCbt81uai27cyks!8SKIg2WoMnH?DvWYBzKAYSSs22cT357hkdz|c@eFhYRT*F-2l zsKjorUjt_g{ehtEBTgW!Lo*)!kdpM7Oas+|es*vpo;1*8B^X5V7GY)!Xt<3(|Jn<% zwdsWpu-O6HhhahvnVbF1VLWMSdiE3GZ{f`ChsKfSvJaIZ;H83!t!ZYd>Jx0`i?p`1 z{rf}vyV3MLTwL@iu;-}G&Dxi0yDr{`fUSJ<=1t&KOL2VmvQ}_2y1cB&v)s>q{smHs zypq-77p1u=_Oriqo0A?truGwp4lkj<3)GSlz^{Sf;WOEEwG^kD4YP`E@aBb~hT>qk zQ-zk@@~=I>k)0s$iWH>y!Y&{U7#GiD+C3S)zhhWzo|~I15_H*4<@W~C5!|dG##pbb z0FiQFAGfa#**=P(j*MO-hRdheZ3b*b`naO_5k_}x(eLBSHsb)~Y>}ZZ!-e_0^Od z!96E@LzpmBS^2&8Dk*~;D`-ZbjHqF;naBp`H#CqGVeJKVkUsLkRmSp4DTXx}$?%w% zzY9-^0|HS8i`D_fRtIz)$}1DZvZ&d92hI`T(iu`tccA-YCH2n65cK^mh(Ne;)>^Ab5Lq$cyc|z@YvrQKPRjMb*=-@LrPj^iL|6IbsJ~~wnz&A{tvc|cIfHEp++0D zOX}*Jfo-RwJ}ew{XIMN`98kz^;fMAT7Zxm`j~Z_H6!xqKSdB&T3BKu6 zIF8eh1=S&Y+CiQ$1>k|qPN0O4&VxhWv;k9qLP7VA`y9@8uD@6KVp1OJT|pG|K-(b!7`rLd--9QE=hPHp$r*&&e3x~s zjmzA)1R^6%be((ywx6I-joUv4n(qui+!mZej3Cw&4%mm+<~Ee+Lp^2{`*F5P!NepL z!7!Ac#$9oM15#u(bS>)Y>zTON?W5F7S9-OOH&?Ch*&t{tJq7Zk;x!2$1#RspWZ4^VTOXk0GVlQfa0P^-8K_gifJuT` zF(&4NYasMZL7{@!egW~Ec$SKH65P@%&z{A*@9#RGCN1(cyzz5^3)IYh`Ua_FyAaK7 zKmaeDKYt}FIb~c~VmtL2 zS^7GY6$M>4zBCI=P`iP>$beMF5JYTBZljOxfW=O)7eZ4WZ^mWR`giVyz+9jp82es* zXK)AS>sB2{?se1@5_FhL@9b1@1Lqy`H+F*a6U^+6`dfaXT!u*Y?1X}23kh@zS|MHF z;<*IoE~IooD=`ZN#Gxa{i&Rc6Bf*AxND8a~e1nd0*y1s=>gwvyhDb3BXmp62r4@bz zIzq|hXK!xZ??+|BxTU~>cCfskKCL37Ip}FUAdA@cU>u57+!liXT_KkM+ErvZ077HB zwK9e5$gu6PfaNnZawVWpMX`c!%=+RI68WHhnt=K~ZczuellF)GM|YY*i|CyjC`H9q zTq%JSweSi zuqKPBdW``JVJgf5{QC9XyLUGo1|WCpwSB%nmN~!UG(5NbIS0D!#%s!|=o$b3%`XbP z^Yh~a2&?{EFS&lk3j78kGms+N%){Z{tFHTkx)y{Vf<}z6yJxh0pK%SUp%t0FZVdy5 zZ-h{BKY!`cAA1+bdo^GHZR#=j^V2_YIycWNY6xMBFPa%+qI1wq&G;?~l7s~)IGu#8 zxe@I$N2_8&Bpdj#9)Y)@p`>|UOn&}W)xr+Ogxu5W&(HItFj|no?z>bmM4ef&Ws++S zq`?QeseS|?9IH^4E6<{$Okq>0iq%2?_Kb(%7yH$CX$_mB&7o#;H1m)8SVxD0gNCBk zkIn(_$5*og_A4OWJb8rdB{$=&$wv`0;t0GK>xpqawL-FGKMu-iwc|u9>^s~@oiPtaZ7;x?hZfK!XPKIs)8IEh@~Ju3M;1MCdf&ERuF>r zB&PTb=Nkh8Nzcyaa?{ii;nhMg%7aPH4b_yZSFd&gCS9E$(gC?486@2pIIk|V*P!5` zh(b1t572!EjDVcC&||nE8y1f`Wsr@rJ>|$}os$kWQ$;uwOs*uzRFx~d0HTUkV*CVT z6nZIi=(pAgk99p%^hV2YTYD2y`UJX{r3v_y@IjLwa0vZ?Kc3K)dqF`k_flN-&ZsQqB zLb^V3)PMw5tkx`jlTb2~sSIwk8xKQbK%xf9jLc49?7A-z4L}r#bgd1>laPO>1dITk zFwb;)$OQ~@QAh|vcnj)C3D`tq0b@d}p{|4=AdpoA72tmL$7$9Rk+Tk3%dIbuov}sC z3o;S9QD`EfM2oauKzk{k=ji6Lqv5OmvgS|m{Rc^Tg+C+Qim%3Zi{p+8eu6VMH^WI>)!1rmDt z)G0-g$1VKKjrB+kmdGC0<1<0gDm?xu8VKv6K5A|wi`|I zr_otMmZ3_J0KI&CRvyJBLmzGn01awrO?^MZ#_bV~#1YjrHA-lI&^itu0;NU!oXE<@ zLb{wfMOnmL5B3AZngg44%S4n*rnb`5r3Ql1GUKS7Gc28Rmmkw6J|n zgJ`Hyp4=ub5ly@ei-4G|Q0cx)FNJ)3;Ik0iKYo-SnJ{nTR$vLkdj^^mPpbSuo;H;T z(v}zWL}%OfcZc?+n4(<5|C}WvQXusyR#ZlV2@6q>I+Ss!6r(exHy^?YiJZG|VOw%-RLT0ozC z54!<2aw4L#YuB&ex0z@Tq(=toz_4-Rh3kn5hn`EsBqu+B(t_OKh&Z7xTWoBsAj~)g zv0_pIo?u`c2I%S*j012xuNZl}B_?`4bp85uMqtuVFxp{CBNn9lG+>%r;AMg6nw(!C z0>$cE2o(%?VrAsyy1~$s@3HSL0(4z7-y{%FD+&$B2m?dE53wHHB)SebyR`VD$Bv7|?%~3%X3YkWK)cVv~ajR<7q@{wEoaKrUqC;{J6s1A-bt(N_=G z*Ba{)wVs4&ylK?bggeOqt}z4hE^_0ck&R4@P^482MG0t}T`9^89&b+*-Hz;lyoqqW z^JMH>#4$F7sNtPz|9K}mWI~5f8vqkQ$y8$a4h_jL*})VJ1jRp(oyO_;|9pP%FQw4` zwQ-pe#swo$@fSOiGI&U)5S!$F%xN{msExi{_o!p3tUW=+?H_? zAFN&Wp=!EPopikchEE)z(3yz`t*zNv--3S)4a>}0ERe48Y$&<)4Z*gJ&Hpvbpn@?D zA)ymxkph$dx(&S%F+7iX)^y?e8sEl+Z7SCUQG#hjCZ!Y%9R1frEbg%6kzg3}v&Q>h zQK8ETRp2VCIe2aKGw*lLR(!8GuJhF6pHblC(q!l#{qJMg5CozBdTda@5b1wC+$G?+ z{qN&>G%`v3?_=2T#s3T)C#M6x0ow_akkDf=gAM(TISj2aTs(Mbveu~n`YYz$yOV(2 z7}-CHN1#`rJT3vo9rQ@P4E~FJiy?5}*?YWwDQ!*P%YQ+jjFF2=xvs7bX)F*@WrJeq zxf_)KxxnOQs6JXaJ}_BA`;i`YY2xSb`_a`L&2qj^xw_;)qEx^n1o8}U-y~>WDZfW+ zaJ#%V8BK!2!%iDuZYVK~uFAlm5Uxz|+$AzHX&fjUWqn!TqLz{A0E7-U3y_^s0HtJM z4;In2$tC6AIJ|wUuDLl#Ma${cpR>^AhF1ww<)hv{ShCLlS=yN9++H)>k32?pa8>8# zyw%%B@BV!2e+I5Qspu`fQf#AFf9^@&I`p3+}C?$#V${dFg`jEUEVItJ{J8kHM2<`{*Mp?Z+5KaeSJd)@!<} zN6}Ak+y1-ft+xf3PR(vGYY4ot&o>K_xqc|-QWg8kN}R2ZizBviTMAIb zZo;db`_GjS@nq01G!_b#Q=eBTY|{CpGWF`+7zgoGdA`)(spQjZa|6tvOM+e87ytb4D5P9PkBV#Hx8*gV5S- zT&!hGeaonVJ|{|^C{vXho@Zd--YPaw+;7}-^)8?l)=sdSwHL443!Doi><{_5tZrA> zYGeem&EwPTE;b2gIeaLL1XC2)r_z8iqLvx>QBtAns&J7(t`mScs##L)(C+01*6;y* zW6-U88ENTuct#$KxFhNhm`~N-zMx`bl+D8cY6tL| zRb(`NI6T>#n}ifNz^TEj>(GRTn@7TlrPS~wv}4eO1K4OoRQZ^eU{W3fyemx1qXLeNwQ?+G4U(kxIpKxp4Q?-6*h%2?ZB1K^nzl+_m?1d(C|l^HE^Lk_fP~hb*BQ8w=N48GpAG)48Mj3OL)DYlJqD9}`tN3~1p7ZuCdGf^ z$+Te3S~(G8k5Dhr4gEO_0Db^u3D!e^7m@)r%E7RN8tA*A*l7jPf!+nsGEGq!2wDfk zNG9M5us-~aq- zi|2x=tp+{cRH_U*F8F{SJX`wuivT_I&m+8rvB%$b2O2OBmist_K_n;km&%A zoWMwTgRbLrdTz54FeOqD=F+-SfusSvR|1ie$JY?_yXo$Xa56USbZ8brW}gOissjir zq8FtqhXerW&`$wV?1Jw<#zw;k7?1D4lFd`EELpv&v5$E9?=RHY0Y^7Y1M5wx)|(al z45rleKLg7>ZEDn1?cMz?_)c~zWOd7?mdTmezu0@Qrv3DG51GFS<$0~CwXXNYH)vfK z?f_0ln&>n{XG0DZw&)J{LRb0HI= z28=ITo_w!?^znrD8m~@5y_wZ7D*?4x$9$IjwgbbLV)l3~gu;)YXhK2QS87UqLGka=g!9PwC0Cy~;CA`|_( zjFqVlKoEc0u`t(131fqzdmDN{BX*-fx|!y5Lj$#-v#5Z9DV6%5wC|rhmFJva`CnVi z^wD2A+0FNdPabtJqdZCcL~85!sFR$D6jWe|^8>}(m!`0jhqbo7c#gYd?oCD9| z1`Y)|_w#oGD#K{}2DlxY6iU`RH>4hp87(!A5V*d_v7pvs=fyy;wK(#k1%^a6g^qGx_%As6 z#w`QOcr$0l(%=Qjvp+MpB`Qd0*ecxFuj!oNIvt=YnJAF`z=W0}zPs&;$pFmxety*@ zD0ok7uC6sF5@dp!b+gnJJxNpAn+oS_eY{&&+A!RPo0%r-Rt=M^(|=w5U7c)F3nrcz zux22U012xA1G!-uupC$guwH-sQD3x$FiQlbFcS6Y06;=o&s91_jV7%27X3UXVOn;2 zptKONx&jQSG2K}@SpqiWMv&rA?l$8MhqeH)7dB+fMGYDdZy*kz_d!>+0w0wAw(?i1 zTZ(`-p;S~wc!k$nxATwikxn}>jYIBaltOHd?(+*a{=M^m{Xatdp&TktU;WF|!;3F* z@jk9JKX0~v8?El~GayaPB=)`bGQ;E|&1bYm@Q<48%dW&f&aNt7UJe9Y9S9Cek{LHo zY18}iK$AVAu&-vLklPq667-~WM0ZD@WIv);-3em=y2!vF2|~Kb(4dei8(4PAy?IT( zmF%i&JG6g2ijbDUjwQ@FD&XoL#*W;|&(?|AvyR9E%hCM*%}A0{2PAV>7A`*yEUMaO zc=d>--K$h$Q^dF;M^DWn_J3$b_N&Ej_=QEvD)nDI2Sf%Dl+)W`>(9n6Pi3j}Ov!pB z3D)tSpA7kPv{W^3dY~-qVXQuJ5 z_eal4;v{k{tNBi;KJVJzUnyTs#|l~tXY~aho#Rgn@6kIveFOkqX^m~qF$RW7fO(o< zu$7$FXBzUnNzSy#zc>X4*ey=@-ARim%e0A zxoma%UzTe35=cc8t7S@<)n_jWUy{U$I)f~#4wKi%jzv6ZG@`VJ@8()w|KZ6f{A<0+ zoaq!1PX`CDER!Jm3iLshwN9LSO)(Uy`gMssG40m#tezVb*x`D zVWV(eSM$9{xRQ7T{L>Y+6yr4RSw;$fybZiy{O+s6+)(H}-X3rhySox{vGB@o2Q@y$ zdo`ySU%Z%jqJ@x<>r%+=7j7Bx+2r3U5g0`$2Qb zmVJF6Z>M1?yEOQGXJ^+OkJK_U`TcWi3$gu&!!J0{o7U>o)idU$>E~+YXBr;q^6CE$ z?;fHYaY^3E`tMun$9PLC#RKS%ozM*YYlgf0eMbO;dAMhql+njZ@pL<1>BGC>m!nrN zpwsd0-~w)A*6|hGO-p>2ZaF8f|Uos>-e4@nq40>pyHLl#Gt z$%bKCiisal@<(MVAePY8){=1bjMuE-$Ee4tUJ_QHYbx{v1f$gd^lU^x2=ZLyY-FNr z^2#cFciO*8b3IYu9|D-{?Q1YT^Zq)1wQXycw1Chrd^h*TO7}u$liOb!qj&y+yPa46 zi?RtfUJL_N#-~PT3m2PfgPdwRp)t@{eAo0^Vw}dNoGQ20Cms@-mSyrZw)>t(Q_VXp z^GzwoRqn*`-??^$H+@j$*1vC^cG}#&yj)PK#l2KHbTdl6|NTjlR&zu9t~^~DQtC$@ zNzoZ;XpdE-#BXks5X~)YdiFgb_ngDzDW2(8b?yHC{{P~7U2-Eoo7tE$!>ne1tlZd+ zNHzXL+Wo;;?r;WK{`<4a_+J-0!t+civ2rdKF4C%)zGpdDd7=*k@})}LD6IV>RCV!Z zwhG%SYgQG!Z^VkeOSifTAW(sEHDy+aNaa|xMY62CM`^Bf9aS@M&8&?Zu_ZPS9SR$~ z&FtaW2>FNj2O%(FK>9(i_ZZKqfwYmUuP}-HRu7%pAtQ~1zh+{8p3C#k`?^lucFv^T zwN9((vn~2wP_D_-R&~+E^vT;q^uFj9qYW!!Shjs@UU{8NmUXR&Eg(qMb)R9j?@IOq zhMQuTc7rL0sxLS!ZWkRdfsawl;pNyyn+ufBc$f|NFK|(NTzpz}>HPQBiwAEQ!}^~Z zT>S$>s%G@z)VcNny^@*sZ-h%v#zKhS9ECj%yJLC(?gO} zV)1s_Wsg;9VIJN52N|EMyqFDtL_KJ=q&NO?|M+s$KR1+I1$80Eyl!j#^5c9qo?5-f zY4=Oa9taKCsX0_j9NoTo_W=?Ggp-?c=AR3x+cG$YfY#s<&4pKihu8+JM22UGB>t8@BQg z!>C@MA&a;7O{I{nYV@Kcgn6dfJq&qyBy{{hkbq&;J~wWea~|2M?}dETe%U|Se$*7! z`q-`S-uB;|0Z4-SdMhiyO($E%Lrmbs=~KyvR?*CiigxK$+U%SxmrQV3G)sHK~INRUY;(=tV``$Gy{9jXF|N4Tfr_g1<@kl3%yeB!c zc&mZ1bbb%I)K)AEb_ zam6bo`w_H)l{!?u`P=7F5-2QFS{wK{_&7YCV${Wn;U>2cT1m&5WrCQFMaLTkh9Y{A zq8pM$S%Xh<@gnq+MK>@3a?b3ZSf#Iv1Yivr>SEhWdg*_phxseuV%m6l9}>der0J_9 zr#GseCdwqSz~R(3g}j)0;kTrK(3%Y7(qyi`SbNF_K9;~Z(F)K?W9SoUPoc1SzE2Ln zwuXvidNHeDW?@m-&-_RR8OBXha=_a8^8Gi~Fa0W!baeAtR%_y$nY4AkmG;4Hr)asE zb}ij+hfaTyR)b&0rslobGgeZ4)%!knzEE?l;FdJ#;O~Hui@;+Z7CSA_=vtDyg_VJ;?C9iXM8LJ*7sxz zsTZYnZtLGadLr3e>ZwF&;_vZQfCmA8QDwHq5P7^kNt8Sa-Af^lwF`EgMnMMFVd5qD#xAixt%5hlz)_1)G&vu(Lb}Y;~lDSKEENQoEbC&Y$chZuY= zqU-tba_Pe52xjg8IW6{{TXuNNOYzXxrzdK^1zfsaZrNl^v+CD|xkN>KZ|N#gj0AWY zX8)0vJ74-l+!fg+kd)`%Ncn!c{8YBC^FqHiWk=fe+OTPIG$-|<4q!SIPXLn)dejX} z{Qh9R-{HH?tiz~rOuB*bV@6a@Zy^!{KG)M|LX0rF*qsS<$NZDJGL}q1Vgx~8r z2lct{`}_U<{`rkRe&_MHeQrAEeXjTQzQ*hIx}GCLZQHhkZnw8>)1WZ6E2g``W?@;t zE|etFv)9Op)A!EA)iz#+PP64rAo&8UNE4{+;Nwv)4_V*WTsZ%dMzUy-z4O_8l|f+c zFd%QD11`v?C%W>GXqp0=Wfbz&3|Od`Hv=k<8LO^i7lZ@{^5km;@PVjm24Jv7AR8ls z9td|JWCnsCzCS-L6T%O*oCf|HKl&^L{(!tr32-^+WD_Y4R(g`AB^V)PGvmuQ>_vCC zFDI;W_QkZ0Z}u}6-JDTuqGHs?pL;s!@)#@XUth>O6NV3tXxQITIyD{iIWW}vgrJ_} zX?Z5YBlUIa_IDckkHu}u=_2hKRlNDQ!NPdC%of5SUIDd7&}XVo+F~p4@tzg1gsH!;`WtHV0&+{53*m3+;0x;RcRNqEbaa6B7s(m^hrKSws{wsgkSk+hPk$D@&V zo~RqQJgF_4#)+qmWFLhZZ*0vn)I-amT74}@!U(BiqJ)Hbk_n90oxjCBf?oE0E9|sThY|?X|`Ny zMw_&GZvDx+HoaZZXPld|0$kuJo zu`1u6%vgP1$;Iw_|8^(WLlTPsI^3*kZW*J=D_6mN@K-FQTEE;26}|i0w~c(>w9Dyg zTO`ddK9eh(jl?h_*O$-eEiRYV6M2OJEkD}ZL-LIB@8XrkvDrF~tk7M) zP&Ug#npJ?oRz;OJ&5X(cgvkU(R183OVANg2#{9}voOy}RFX!aJZDiltAh5oR8F@=O#plUJ;r3GhZ zVYta?9=7~0zNw^ACyOrnx_8_^JI|bAQYeaOQdNFnr(hM`CP@~X2=Su(%`XQJvN7k0 zu*tOqgiBTN^Y~zKYGq0`B&IVTo)*7haJ>HcvDXt!>8)>XNp@zxF+6+i?4GXiYAm;> z{u@Iou3tS92~($W-J?~@%;s|WswstXzd}sLdb4_~uDr2O?^=3%bMfYZdwo{EKg9Gz z#KSpGK4)Un;3Tn1O;-@G9uR95?W)=~>`lTldBI=f(|Km`r{)5xI;;T-0|({1>^NRT z(zBnv<*fB6rBXnFdehJ-6=lq~xDw;{KXuwB2ZNh*T90wPex$ewLvs%*!_FC5fOKPp z>>JG}5NXpgLK)Ls$p4?rT1)k@^+bd57hABsvE7_FJ zop-RLPB|odsbTB=Xm_PuyO^q{^~pgZvfGhuA}rqnFE}7pmpDhG^O9v4c~->v??XNQ@PupbZ7K(I)Pq(KOiy zFs0H|4+x#lDcWw&`pm^=@eZVev%&gQb!^!Y$=?1!SrG2y-kw+(<3&U5PBV>0(4I%) z-lY4uuvM(s8ZRrRtNi4Aa=pGg_a--Ac)+$a>8h3{uBKVF>a62f!tmOtMSd`9^t;!G zv-5Z$LL8usj1QK*gierxx~EPAf37?15Z>lf#f7;YgvZAMMV$bt0Qf3`x41n}$qBVr zT4I+`zIp5FkAtjyWX)pU7gBDTU+r$Yp&j(4Vag9gX|c@%m6Q7p(F@qVNuY&0#7zB_ z57cqwJKs2bfG)nI*0v+zUJp-@jS!r5xCc#95*n-CpshS=dc+RaMTX~OyO2OGCSaFWF?2`D_d73E8?|lDW=je$q zfIUc-CKOH%yYt*OB_Dt@>I8&D09XyM8I=K6Q!bjSvC38*Qs-p$GB7{Yu;=jwY;m2# zTh2|Js>>ZpHDA3}jsi^rb&yO#7^#_=nNzBKzz8^oMW-XBhxWl;HilAr#$4~g&rWd{ zn(CSQ))uNgvh8|eUp$sR>Fu*~PVs)SETkpLs-F6+B^7E)6qWw&1IfMv0@u-^0kPiU z%R!G52VQ-hVe>Nz+G^SE(A7Z2~9-}xX}+DALczT=8vN#-}@=ymXbpi!vp>GTC-EH-QTX+V1f-3WH#jG&^gD*AeJ+5 zu|SJCuJG@tmIN`f&sYx*iWO6mfHk}AyAz`j1&k$Az_>3$y&VxHVUo$hi40(^E)^`D zM^IEcx=f@Z2?Wg3<XKiMl#&_R~UdAL6LD87fH-o>pD(@5~SBhf+-L zn=#ZpWRu2ONNyKYeQSJ~J|fWdL6!p9rm~vh!;lQV0c<_Qf(5sioe1(xphlp{tPkA{ z=r}bG*K3A`1l%)3Bl(~)ylv}N1Z_fxuIPrg?NBx>Dgz3RgKaIXgE8n}Ijt`Lx=!T@ zAS(n7Hl3veKrrYbqIeZ?KR{oL84D=!7a+rfBw?U1u7z+e=oA1!yAUS;I1_ddff>IG z<dg}GPE$`T=SQ1a1^N(+O8GlzI@6DBJ&m9~UI8Os{ z)+OI=$n)oSM_FRri^l={-45bf$`i8zcsws?g-`UAsN>PY1aJkwz~vxXX#ud}EPxc* zK<`hQtdbIeu7QWxJ8>%RE`77FYfL^I!y;+pp-jpH2jW;FZ(Wi<|3l2<2Z)Z>JqF$A z8+UIw^1!WOeBS=YT4I8D{62J8lsAmRhdp@YSF_da>hzrUxj< zKFPb3K@O1$2nz(<|2hN*u8p8*%5gahfJ+1%;A9`{2wLBOAA~gG*h=~`J$6%$0YjuZ+X4sQJd<|C?l`_VrzQW<$KpuyD-$BiF2VESRL3mzYMNX6E*fP4F{d@qm;)e^a^sTU`Gd5_JNlW?;e|xxFK6RGK_cy~cr962VRlE`&vaXWW z3A{J8M?n2} z`JR?k-lv)Hh16(w`-HknfI{E7MCYLP(|bf~Z64Y3a`eko@vGsz{67bJSwmd`$_qhV zB5BvEUO)&DS@)%i_qetHvB2wSeROw1sFd8EM@a+YAOQPf;z&J!x)5c7576V+7tm@t zGh++NDSkd@Eqaqtu!vtj%scZMoiDmOWx%QS%c!~07uVfbPxnu+(lCl{w(FtR;j`1& z0flSVgU3eM$@yuXBT?PbRS-ETBdzjrHF0#(U&y4HCNDg2HoYOr@vXg=y2f>#_S=bxWvjvt<4w9-Mde9MXd|Y&R)YP^35A@2 zJEk;rD!pcw@&SS`iO=&JVQ8#1EdzL zwH$3Mzd!i@R@R1J;g29*HCpw9Yo4;-;V)2IgJAoz-DUZVd1+^*X&7#qM#{^B`CF^S zx3cjuckO6uBONyzf->#x>9>h-so$G?51&dCVz={Al&uclR zYy80;{MQH134ez@f0+VhGnk4GARW>$HZU;s;o11hcaF3V;H2A$j|=^qaq?bz?#Ag0 zzY~Vs;?+9~wWm+^@fAm`vO^DDP~nr>zP6D0^pzTdA-o{uV1?7B(; z@B|>A`L~&>LSE7F`JQEKyP08u76l;efG{CEc*Q`^J?qZ}R0E)}FxEfS{^(&zp!&!H zeF)FM#p2=d$*}Xu5t~it%PTzHT{BL=45j2KWZ_^ex*Z}2ds*v=X8*@zvCqA(g!D8p z+@qizkZRqjCUwQVT$C{Jc*y}}|8+JzJp5+QIc^(W!nydTr}|V{P$FDiF-V(B2);0h zXZqJNbE_niRR#B6LzVNx0T+0!Ro;5prs4;bXa~g%Bm%N4a32N{h63Zx4N@B2FK>UU z*n|+H7Cpd{s+bTU{L{|~;jo0RD9g6G2uY{}X1k`Ngj(1<7A-Y^2>nC}CxgOWatiJV z{F@k4`N=B%LkCb2r5I&l!8kw*Q=v?d|e2;Rk(jXpE7OeeTn-yjH{?(Z!)_P4>$B5s6}+*Jp2P z*-I&S!f}eR%j?QYJCr^nh&rBG+7^O|KodA&1|Tpy7O~B;MAIp6lGq%iJ2*LY618w1 z_2tWfXu2g+lu!NYJkw~}#Fu2(%4uY)8XvOC;_c}twEcFIfIw^7vMgPz+~8v2Ptz#$ zQB1250DRg2sTB=UM8*KNp*eNUtNthKnWj*-!N()E5&f}t6Gw5!;)RlDvo)b-4;l~c zr-6?)YWz2KR$ehRw;k>^CJ--sL**Q&*Q9@urZ3JzeUWff7o7GMw;DjHrLTM^XKnoo#|IHa5n5TO_Vup%ND&~XrP6A7=molh6Yc%vKtLi;NY zulsvhe28Lb;p|ap8AkYO-eITJVON^vYq1}7BbZ7QZ}*_1%L{pF!F}hHjDGooNYxLW zXyYeRpWORpsJnIfmiOdo#6!mA%tE*Vh9(pmUmf!Rqcb@zR89wpdW}hpNrr(F*&6GZE_cw=x{X=SgoeF z>#uzVk=aPjMCZpJTK2o_qMhc>6c`<7%NV$*9SbsfTA;JBoac zI&LZOBsuaOAv{^Q&SN>B-5>R!C{cMjxcQK@^^lL?`_Y>7U^4};idF|*y`o>1VZ|x0 z?E-A@b~Ue0*XgWG7CP9O_(`{&?_Ivr%~@M${F5@EzPCTmM4pr1@}fsimM>uNQDHkm zz^)nCbuozGI6v?pjNdvI0hNGgtllXJrEOI720F1BL}DOE+=%M2z;rZ$bPh>vnK9d= zHuT@_4T{x#R5arEC7q;4*XrisWwRUm3XI7|@4aaZg(IJGa3U%;lX1|ox_K8JR&5)b zgLQvATnVOg81Ko8%AWZ~-3JLb^hx=J_zNey!8E5UBU@JXRtJJs3##=a9=$6Pey#DP zO+&cWAlxT@fWiP#Fdmjs4iNUoBQPP7)JHU6t`}d(5n8F84xtr3neAw{nH>qyo^4C4 zgoEwA;)isRo!VQv&(2=FhS_x_SyZ4#+_+|57qv2@GcsM2+Oh6Ex00At^jf-3LQ3-8 zA(YU>Xn(DI8~@C4yM%5&li?`^CM?k0I=1Y4empQ72;m4mej6ZrxL?yy+<4=GAAMCnOcnW*u!9L0lJLqT|UYWr)TQB^9;AnfCsl3{Ah;Swv8P=Iq zaN1u^<_+@zF|$ZU83LjJ6UhvkZ-YFSgm?B0rLyhw(;>m^sm#?M;)Y! zkit4YyJ`C7UQJOPx7;&Dlf+!)*XTMt!{oTdj>MUe=K$2$kN}Kxkm*v4gdGlO?68x8 zZHW_Sv3gW(>`*R&cz)DDJN_v2;;H7TZ2A`G;BKKLP?=omW#FPoUn&a4So5d@_b*+fV8yqw8Ktg zrZwnQ!eVjx^)D3PX4`hm{-}w|Rey4|FBO|@LEogtC?HvH@Bs>onq|AlxkkszzV80> znCm#|T9yPqX3Z^fTijA)?0&2eU9duDup2Z4Av~D@+Zj`+{@}1!WQ>GLA0zau^AE}^ z(d@0eo_BcBMcP|;<%j#JpEu}wJhy7t4n^ORrKZYi6gi$E(x>Vs83J#FSF!{7fde`g z+9wWSm0&>D&LX|1fB;57+XCV~t15TogE+1}Yvrk)v+qo3l%-Wt8wJUxkB*_2CWoTu{4@r1#^vBK2lTz9?9rm zpDP>ZnSGRZp6$)O5cpK@OOE+9sb~wqxU|H*W6@UT|aez=Toa~3QS{uxTMR>Y>sPb6!3QSke3S)_l>V=@`5ro`~o>UmYqODQn+wK=+@=DLkNwayoN@A9OQii zBjg~T-2Su*bbOTn_zt-&a>fG^0~;%=7n85Rdy67zJji`?EZVAkv|_RZVRGZPhPdJpPydL6 zG*!Hg#x7J{golNR*4+uD5&*^{;uGi;oj!-LQ->s-i49b$)0Ve`#v}v7B}f37EKz&) z|MMhMo?5ZrI(lG(LG2ale2C@6571&JHeun9AN=7W6(RW3(=$~ag83sI>sU9I znI;pRL6`m13PtO)l+eE9oJ-Girf+wr4F@mBymR@WOef*Fb+qtwvFT{`UXs1}i!fd- z5J+p|rNDS$64#WzapK1R!e@oA@PQWZE(kb8me-(k#J5Ll{quCG#Ey8nly3>~llvkv zxY~G$1pUuv&dNXL8coW-qZEDjwvEtn9;7*_kO&h1CC~tc5_e1Z>NMW-_q?()7up*A zYK{6pYy-{HjY{T}x`ahc3;rHH9?oub&L4#^u28-9`A%%}WSnN$@&X*GIp)7!L1z); z4@=uh>}<{N@8c!FzRw4>OL|iHd1@hzvjWd^8gs8A+QsxHokzczQlk2;!_*U{B4@Emc86J-l2m&C0hsZ!_tdI zjcuY@Yz)uGTRAZ78?;21+#R4HIn-boeDfs^+oo4>TXA-dJfl3R^oq?Ka+<6 zorjn&lW|=LK<7r+z+M)K&j5r=mc^KYM5OxC7KL?29^uN#EJ}I1FO&1GoDa+BW7F(* zI3X!+U&+FAZJ2W5nxM$ljU5QRyY<*MakxAu_PteRvOhj!=E#(chtdJEW51tKCzHO+ z`_tBnT0j@(jBhSdjvSY6@NbZ8<4uiIwM{M;#n3RVZ;t5g(ZOFRyYiUXJqkFwe0-7MK-p=&O0*^rSV#@xDN{ zc&hUBPV;#V9S-{=8P??i+mEyIvR3RS+swL#1t+#L`TomG&+xuO&t)8c8sV&8OC-{# znrNI*8XGxjXG#|X3pc__3-EEvb$VHngV~xvVIMsBs)&cDOrOXYT6;%iM7lqARPVG@^agq|D18YPMO zK!nW&y*u_kuNd(tfgDy|Hzo%swUC9X4N`%{+24Ifir1%tV7VO7z0BHf#|81cxwc39 zF0}E+0)jl{TIzveHhBTwG#5UfQJ@_vcx-f1SkzlF{*G1sT3`|Dhpc08~42(WvT09<6_Qi{+POLmNRNHtM`Dn}Eb z*T?P#9GLmo%>)hBqkiFWy<7ajHWL5`06-