diff --git a/.claude/ci/appsec-gradle-integration.md b/.claude/ci/appsec-gradle-integration.md index d72151e408..cdab521934 100644 --- a/.claude/ci/appsec-gradle-integration.md +++ b/.claude/ci/appsec-gradle-integration.md @@ -273,7 +273,8 @@ If you need to inspect sidecar/helper or PHP issues: - The `test` task itself is disabled (`tasks['test'].enabled = false`). Use versioned tasks like `test8.3-debug`. - Docker images are pulled from `docker.io/datadog/dd-appsec-php-ci`. Without `-PfloatingImageTags`, images are resolved by SHA256 digest from `gradle/tag_mappings.gradle`. If a digest is not locally available, Docker will pull it. - `buildPortableLibdatadogPhp` uses the - `nginx-fpm-php-8.5-release-musl` image with nightly Rust. The image must + `nginx-fpm-php-8.5-release-musl` image with stable Rust (`compile_rust.sh` + sets `RUSTC_BOOTSTRAP=1` for `-Zbuild-std`). The image must be available locally or pullable. - On first run, Gradle downloads its wrapper, dependencies, and Docker images. Expect 5-10 minutes. Subsequent runs with warm caches take ~20-50 seconds for a single test. - **c-ares DNS failure in Alpine containers.** Alpine's `curl` and `git` use diff --git a/.claude/ci/building-locally.md b/.claude/ci/building-locally.md index f246dba38a..affb639718 100644 --- a/.claude/ci/building-locally.md +++ b/.claude/ci/building-locally.md @@ -73,10 +73,10 @@ cargo +nightly fmt --all --quiet && \ cargo clippy --workspace --all-targets --all-features -- -D warnings ``` -A rustc ≥1.87 toolchain override is active in the repo, so plain `cargo clippy` -picks up the correct toolchain — do NOT force `+stable` (the default stable is -older) or a pinned `+1.87.0`, and do not lint per-crate with `-p`. Only `fmt` -needs `+nightly`. +A rustc ≥1.91 toolchain override is active in the repo, so plain `cargo clippy` +picks up the correct toolchain. The `+nightly` toolchain is likely not in the +development images. You can install it temporarily in a running image, or run +it on the host. ## Tracer Extension (ddtrace.so) @@ -212,18 +212,17 @@ helper artifact. ### For correctness tests (bookworm) -`CARGO_TARGET_DIR` **must** be set explicitly (see -[github-actions-profiler.md](github-actions-profiler.md) for why): - ```bash dockerh --cache profiler-8.3-nts --php nts \ datadog/dd-trace-ci:php-8.3_bookworm-6 -- bash -c ' -export CARGO_TARGET_DIR=/project/dd-trace-php/target -cd profiling && cargo rustc --features=trigger_time_sample \ - --profile profiler-release --crate-type=cdylib +cd /project/dd-trace-php && make compile_profiler PROFILER_FEATURES=trigger_time_sample ' ``` +Output: `tmp/build_profiler/modules/datadog-profiling.so`. See +[github-actions-profiler.md](github-actions-profiler.md) for +`PROFILER_FEATURES` and the ASAN variant (`make compile_profiler_asan`). + ### For release / packaging / system tests (centos-7) Bookworm is too recent for binary compatibility purposes. diff --git a/.claude/ci/github-actions-profiler.md b/.claude/ci/github-actions-profiler.md index ac225c74f3..4795809c71 100644 --- a/.claude/ci/github-actions-profiler.md +++ b/.claude/ci/github-actions-profiler.md @@ -13,15 +13,17 @@ |--------|--------|-------------| | `Profiling correctness / prof-correctness ({ver}, nts)` | `ubuntu-24.04` | Builds profiler + runs NTS correctness test cases | | `Profiling correctness / prof-correctness ({ver}, zts)` | `ubuntu-24.04` | Same + `exceptions_zts` (requires `parallel` PECL extension) | -| `Profiling ASAN Tests / prof-asan ({ver}, {arch})` | `arm-8core-linux` / `ubuntu-8-core-latest` | Builds profiler with ASAN + runs `.phpt` profiling tests | +| `Profiling ASAN/UBSAN Tests / PHP 8.5 {nts,zts} UBSAN ({arch})` | `arm-8core-linux` / `ubuntu-8-core-latest` | Builds profiler with UBSAN + runs `.phpt` profiling tests | Correctness matrix: PHP 8.0+ × {nts, zts}. -ASAN matrix: PHP 8.3+ × {arm64, amd64}. +ASAN matrix: PHP 8.3+ × {nts-asan, debug-zts-asan} × {arm64, amd64}. +UBSAN matrix: PHP 8.5 × {nts, zts} × {arm64, amd64}. ## What It Tests -Each job builds the profiler extension with `--features=trigger_time_sample`, then runs -PHP scripts that exercise profiling (allocations, wall/cpu time, exceptions, IO, timeline, +Each job builds the profiler with +`make compile_profiler PROFILER_FEATURES=trigger_time_sample`, then runs PHP +scripts that exercise profiling (allocations, wall/cpu time, exceptions, IO, timeline, strange frames). The scripts output pprof files (zstd-compressed protobuf). The `Datadog/prof-correctness/analyze` GitHub Action then checks each pprof against a JSON expectations file. @@ -36,7 +38,8 @@ ZTS adds: `exceptions_zts`. Use `.claude/ci/dockerh` with the `datadog/dd-trace-ci:php-_bookworm-{N}` image matching the PHP version under test (see `index.md` for image version and contents). The CI -uses clang-19 on ubuntu-24.04; clang-17 in the image works fine. +uses clang-20 (`LLVM_VERSION` in `prof_correctness.yml`) on ubuntu-24.04; clang-21 in the +image works fine. Actions jobs use `shivammathur/setup-php` instead, but the same `dd-trace-ci` image is a suitable local substitute. @@ -50,33 +53,31 @@ PHP version being tested. ### Build the profiler extension -`cargo rustc` must be run from the `profiling/` subdirectory (the workspace `profiler-release` -profile is defined in the repo root `Cargo.toml`, but the crate itself lives in `profiling/`). - -**`CARGO_TARGET_DIR` must be set explicitly** to `/project/dd-trace-php/target`. Without -it the `cbindgen` build script inside `libdatadog` calls `cargo locate-project --workspace`, -which resolves to the `libdatadog/` submodule's own workspace (not the repo root), and -tries to create `libdatadog/target/include/datadog/library-config.h`. That path is inside -the read-only source mount with no writable overlay, causing a -`ReadOnlyFilesystem (os error 30)` panic. Pointing `CARGO_TARGET_DIR` at the already- -overlaid `/project/dd-trace-php/target` fixes it. +Build through the top-level `Makefile`, the same way CI does. The build runs +out-of-tree in `tmp/build_profiler/` (writable under `dockerh`), and cargo's +target dir defaults to `tmp/build_profiler/target-profiling/`. ```bash # NTS example (PHP 8.3) -dockerh --cache profiler-8.3-nts --php nts datadog/dd-trace-ci:php-8.3_bookworm-6 -- bash -c ' -export CARGO_TARGET_DIR=/project/dd-trace-php/target -cd profiling && cargo rustc --features=trigger_time_sample --profile profiler-release --crate-type=cdylib +dockerh --cache profiler-8.3-nts --php nts datadog/dd-trace-ci:php-8.3_bookworm-11 -- bash -c ' +cd /project/dd-trace-php && make compile_profiler PROFILER_FEATURES=trigger_time_sample ' # ZTS example (PHP 8.1) — note --php zts, matching image version, and separate cache name -dockerh --cache profiler-8.1-zts --php zts datadog/dd-trace-ci:php-8.1_bookworm-6 -- bash -c ' -export CARGO_TARGET_DIR=/project/dd-trace-php/target -cd profiling && cargo rustc --features=trigger_time_sample --profile profiler-release --crate-type=cdylib +dockerh --cache profiler-8.1-zts --php zts datadog/dd-trace-ci:php-8.1_bookworm-11 -- bash -c ' +cd /project/dd-trace-php && make compile_profiler PROFILER_FEATURES=trigger_time_sample ' ``` -Output: `/project/dd-trace-php/target/profiler-release/libdatadog_php_profiling.so` -(persisted in the host cache at `~/.cache/dd-ci//target/`). +Output: `/project/dd-trace-php/tmp/build_profiler/modules/datadog-profiling.so`. + +`PROFILER_FEATURES` adds Cargo features on top of `profiling` (default: none). +The correctness tests need `trigger_time_sample` (`strange_frames.php`); add +more comma-separated, e.g. `PROFILER_FEATURES=trigger_time_sample,debug_stats`. +Features are fixed at configure time, so remove `tmp/build_profiler/` after +changing them. +Release packages don't use these targets (`.gitlab/build-profiler.sh` runs +`phpize`/`configure` with no extra features). The second run reuses the build cache and completes in seconds. Never run `--clean-cache` between iterations — the Rust build takes 5–15 minutes from scratch. @@ -88,8 +89,7 @@ write pprof output there — no extra mounts needed: ```bash dockerh --cache profiler-8.3-nts --php nts \ - datadog/dd-trace-ci:php-8.3_bookworm-6 -- bash -c ' -export CARGO_TARGET_DIR=/project/dd-trace-php/target + datadog/dd-trace-ci:php-8.3_bookworm-11 -- bash -c ' export DD_PROFILING_LOG_LEVEL=warn # use "trace" only when debugging — trace is verbose and slows execution export DD_PROFILING_EXPERIMENTAL_FEATURES_ENABLED=1 export DD_PROFILING_EXPERIMENTAL_EXCEPTION_SAMPLING_DISTANCE=1 @@ -100,7 +100,7 @@ TEST_CASE=allocations OUT=/project/dd-trace-php/tmp/correctness/$TEST_CASE mkdir -p $OUT DD_PROFILING_OUTPUT_PPROF=$OUT/test.pprof \ - php -d extension=/project/dd-trace-php/target/profiler-release/libdatadog_php_profiling.so \ + php -d extension=/project/dd-trace-php/tmp/build_profiler/modules/datadog-profiling.so \ /project/dd-trace-php/profiling/tests/correctness/$TEST_CASE.php ls -la $OUT/ ' @@ -134,7 +134,7 @@ The pprof files are zstd-compressed protobuf. Use `go tool pprof` (available in dd-trace-ci image) to inspect them. Pass `--user root` so `apt-get install` works: ```bash -dockerh --cache profiler-8.3-nts --php nts datadog/dd-trace-ci:php-7.3_bookworm-6 --user root -- bash -c ' +dockerh --cache profiler-8.3-nts --php nts datadog/dd-trace-ci:php-7.3_bookworm-11 --user root -- bash -c ' apt-get update -qq > /dev/null 2>&1 && apt-get install -y -qq zstd > /dev/null 2>&1 PPROF_DIR=/project/dd-trace-php/tmp/correctness/allocations @@ -202,14 +202,15 @@ frame name formatting. The implementation is in `profiling/src/capi.rs` and ## Debug Build -For a debug (unoptimized) build: +For a debug (unoptimized) Rust build, add `RUST_DEBUG_BUILD=1` (use a separate +build dir, or remove `tmp/build_profiler/` first): ```bash -cargo rustc --features=trigger_time_sample --profile dev --crate-type=cdylib +make compile_profiler RUST_DEBUG_BUILD=1 PROFILER_BUILD_SUFFIX=profiler_debug \ + PROFILER_FEATURES=trigger_time_sample ``` -Output: `target/debug/libdatadog_php_profiling.so` (~144 MB vs ~20 MB for profiler-release). -Use the same `php -d extension=...` command, just point to the debug path. +Output: `tmp/build_profiler_debug/modules/datadog-profiling.so`. ## ZTS tests -- parallel PECL extension @@ -222,28 +223,44 @@ CI, on the other hand, runs on a bare `ubuntu-24.04` runner and installs PHP via installs version `v1.2.7` from GitHub via the `extensions` matrix parameter (`parallel-krakjoe/parallel@v1.2.7`). -## ASAN Build +## ASAN / UBSAN Builds + +Both jobs live in `.github/workflows/prof_asan.yml` and build through the +top-level `Makefile` (out-of-tree, in `tmp/build_profiler*/`), not by running +`phpize`/`./configure` in the source root. + +- **ASAN** (`prof-asan`): `make compile_profiler_asan`. Uses the pinned + **stable** toolchain from `rust-toolchain.toml`; the target sets + `RUSTC_BOOTSTRAP=1` so stable accepts `-Zsanitizer=address` and + `-Zbuild-std=std,panic_abort` (std is rebuilt instrumented, which needs the + `rust-src` component and an explicit `--target`). Output: + `tmp/build_profiler_asan/modules/datadog-profiling.so`. +- **UBSAN** (`prof-ubsan`): `make compile_profiler` with UBSAN `CFLAGS`/`LDFLAGS` + and `-C link-arg=-fsanitize=...` in `RUSTFLAGS`. Output: + `tmp/build_profiler/modules/datadog-profiling.so`. -Builds the profiler with AddressSanitizer using a pinned nightly Rust toolchain -and clang-17, then runs the `.phpt` test suite with `--asan`. +`CC`/`CFLAGS`/`LDFLAGS` from the environment still apply to C code built by +cargo build scripts. The standalone profiler has no C of its own; the `.so` is +the Rust cdylib. -### Local reproduction +The workflow copies the `.so` into the extension dir rather than using +`make install_profiler`, because the install target also writes a +`datadog-profiling.ini` and the test step loads the extension with `-d +extension=...` (it would be loaded twice). + +### Local reproduction (ASAN) ```bash -dockerh --cache profiler-asan-8.3-nts --php nts-asan \ - datadog/dd-trace-ci:php-8.3_bookworm-6 --user root --privileged -- bash -c ' -export CARGO_TARGET_DIR=/project/dd-trace-php/target -export CC=clang-17 -export CFLAGS="-fsanitize=address -fno-omit-frame-pointer" +dockerh --cache profiler-asan-8.5-nts --php nts-asan \ + datadog/dd-trace-ci:php-8.5_bookworm-11 --user root --privileged -- bash -c ' +export CARGO_TARGET_DIR=/project/dd-trace-php/tmp/build-cargo +export CC=clang-21 +export CFLAGS="-fsanitize=address -fsanitize-address-use-after-scope -fno-omit-frame-pointer" export LDFLAGS="-fsanitize=address -shared-libasan" -export RUSTC_LINKER=lld-17 -RUST_TOOLCHAIN=nightly-2025-10-31 - -cd profiling -triplet=$(uname -m)-unknown-linux-gnu -RUSTFLAGS="-Zsanitizer=address" cargo +${RUST_TOOLCHAIN} build -Zbuild-std=std,panic_abort \ - --target $triplet --profile profiler-release -cp -v "$CARGO_TARGET_DIR/$triplet/profiler-release/libdatadog_php_profiling.so" \ + +cd /project/dd-trace-php +make compile_profiler_asan +cp -v tmp/build_profiler_asan/modules/datadog-profiling.so \ "$(php-config --extension-dir)/datadog-profiling.so" # run-tests.php writes temp files next to .phpt files, so both must be in a writable dir. @@ -258,18 +275,19 @@ DD_PROFILING_OUTPUT_PPROF=/tmp/pprof \ ' ``` -Requires `--user root --privileged` — ASAN needs both. - -The nightly toolchain version (`nightly-2025-10-31`) is pinned in -`.github/workflows/prof_asan.yml`, not in `profiling/rust-toolchain.toml`. Check -the workflow file for the current pinned version. +Requires `--user root --privileged` — ASAN needs both. For UBSAN, use +`--php nts` (or `zts`), the UBSAN flags from the workflow, `make +compile_profiler`, and `LD_PRELOAD` the clang UBSAN runtime when running tests +(see the workflow). ## Gotchas -- **Expected ASAN test counts:** 39 total, ~27 pass, ~12 skip (30%), 0 fail. The skips are normal +- **Expected test counts (PHP 8.5):** ASAN 47 total, 32 pass, 15 skip, 0 fail; UBSAN nts + 47 total, 35 pass, 12 skip, 0 fail. The skips are normal (platform/env conditions). A non-zero fail count indicates a real problem. -- The `profiler-release` profile is defined in the workspace root `Cargo.toml`, not in - `profiling/Cargo.toml`. It inherits from `release` with `panic = "abort"`. +- The profiler is built from the root `datadog-php` crate (`Cargo.toml`, `--features profiling`); + there is no `profiling/Cargo.toml`. The `profiler-release` profile is defined there too and + inherits from `release` with `panic = "abort"`. - `dockerh` runs the container as your host UID so cache dirs are writable without any permission tricks. Pass `--user root` after the image name if you need to install packages with `apt-get`. diff --git a/.claude/ci/index.md b/.claude/ci/index.md index 5cb396d0c1..4029671fcf 100644 --- a/.claude/ci/index.md +++ b/.claude/ci/index.md @@ -111,8 +111,8 @@ CI images are tagged `datadog/dd-trace-ci:php-{version}_bookworm-{N}` where `N` is an iteration number shared across all GitLab appsec jobs. Find the current value by searching for `bookworm-` in `.gitlab/generate-appsec.php` . -The `php-8.3_bookworm-{N}` image contains: Rust (see -`profiling/rust-toolchain.toml` for the pinned version), clang-17, Go, and +The `php-8.3_bookworm-{N}` image contains: Rust (see `rust-toolchain.toml` +for the pinned version), clang-21, Go, and multiple PHP builds under `/opt/php/` (nts, zts, debug, etc.). Use `--php nts` (or another variant) with `dockerh` to select the right build — see the `--php` section above. @@ -122,7 +122,7 @@ in CI scripts are mirrors of `datadog/dd-trace-ci:TAG` on Docker Hub. Pull them directly without authentication — no registry login or image export/import needed: ```bash -docker pull datadog/dd-trace-ci:php-8.3_bookworm-6 +docker pull datadog/dd-trace-ci:php-8.3_bookworm-11 ``` (The exception is registry.ddbuild.io/images/mirror/b1o7r7e0/nginx_musl_toolchain, @@ -255,7 +255,7 @@ this file instead of duplicating build commands. ### Group A — Native Linux unit and extension tests Runner: `arch:amd64` + `arch:arm64` -Image: `datadog/dd-trace-ci:php-{version}_bookworm-6` +Image: `datadog/dd-trace-ci:php-{version}_bookworm-11` No Docker daemon — tests run directly in the container. → **[appsec-native-tests.md](appsec-native-tests.md)** @@ -276,7 +276,7 @@ Covers: `Unit tests`, `PHP Language Tests`, `test_c`, `ASAN test_c`, `Opcache te ### Group B — Native Linux web framework tests Runner: `arch:amd64` -Image: `datadog/dd-trace-ci:php-{version}_bookworm-6` +Image: `datadog/dd-trace-ci:php-{version}_bookworm-11` GitLab service containers: test-agent, httpbin, request-replayer → **[tracer-web-tests.md](tracer-web-tests.md)** @@ -291,7 +291,7 @@ Covers: `test_web_laravel_*`, `test_web_symfony_*`, `test_web_wordpress_*`, ### Group C — Native Linux service integration tests Runner: `arch:amd64` -Image: `datadog/dd-trace-ci:php-{version}_bookworm-6` +Image: `datadog/dd-trace-ci:php-{version}_bookworm-11` GitLab service containers: MySQL, Redis, Kafka, Elasticsearch, MongoDB, etc. → **[tracer-integration-tests.md](tracer-integration-tests.md)** @@ -310,7 +310,7 @@ Covers: `test_integrations_amqp*`, `test_integrations_curl`, `test_integrations_ ### Group D — Native Linux compile / artifact build Runner: `arch:amd64` + `arch:arm64` -Image: `datadog/dd-trace-ci:php-{version}_bookworm-6` +Image: `datadog/dd-trace-ci:php-{version}_bookworm-11` Produces `.so` artifacts consumed by Groups B, C, H. → **[compile-artifacts.md](compile-artifacts.md)** diff --git a/.claude/project/index.md b/.claude/project/index.md index a9047aa23b..4e34c14496 100644 --- a/.claude/project/index.md +++ b/.claude/project/index.md @@ -85,7 +85,7 @@ request crashes and is shared across language tracers. See `compile_rust.sh` (invoked from the Makefile) drives it. Minimize FFI surface; headers are generated with cbindgen (see [components.md](components.md)). Toolchain is pinned — see `Cargo.toml` (`rust-version`) and -`profiling/rust-toolchain.toml`, not a hardcoded version. +`rust-toolchain.toml`, not a hardcoded version. ## Configuration & INI diff --git a/.claude/project/profiling.md b/.claude/project/profiling.md index dea4aa2361..a1761b6b4f 100644 --- a/.claude/project/profiling.md +++ b/.claude/project/profiling.md @@ -8,10 +8,11 @@ agent. ## Key files & dirs -- `profiling/Cargo.toml` — `cdylib`; depends on `libdd-profiling`/`alloc`/ - `common` from libdatadog. -- `profiling/rust-toolchain.toml` — pins Rust (distinct from the workspace - toolchain). +- There is no separate profiler crate: it is built from the root + `datadog-php` crate (`Cargo.toml`) with `--no-default-features --features + profiling`, which pulls in `libdd-profiling`/`libdd-alloc` from libdatadog. + `components-rs/lib.rs` includes `profiling/src/lib.rs` as the `profiling` + module, and `components-rs/build.rs` includes `profiling/build.rs`. - `profiling/src/lib.rs` — module entry: `minit`/`rinit`/`prshutdown`, Zend interrupt registration. - `profiling/src/profiling/` — `mod.rs` (`Profiler` + `SampleValues`), @@ -33,12 +34,11 @@ pprof via a background thread; `prshutdown` flushes. Builds independently from the tracer; depends on libdatadog for pprof. The main tracer looks up `ddog_php_prof_interrupt_function` by symbol to call on interrupt (see [tracer.md](tracer.md)). Not sidecar-dependent — it uploads -directly (contrast with [sidecar.md](sidecar.md)). `build.rs` reads -`../VERSION`. +directly (contrast with [sidecar.md](sidecar.md)). `profiling/build.rs` reads +the repo-root `VERSION`. ## Gotchas -- Pinned `rust-toolchain.toml`, not the workspace default. - `CARGO_TARGET_DIR` must be set (the Makefile uses `tmp/build_profiler`). - `cdylib` is release-only — phpt tests fail on debug builds. - Allocation hook differs: PHP 8.4+ vs ≤8.3 use different modules. diff --git a/.github/workflows/prof_asan.yml b/.github/workflows/prof_asan.yml index 83e1993c31..d028d43559 100644 --- a/.github/workflows/prof_asan.yml +++ b/.github/workflows/prof_asan.yml @@ -19,7 +19,6 @@ jobs: CARGO_HOME: /rust/cargo RUSTUP_HOME: /rust/rustup CARGO_TARGET_DIR: /tmp/build-cargo - RUST_TOOLCHAIN: nightly-2025-10-31 container: image: datadog/dd-trace-ci:php-${{matrix.php-version}}_bookworm-11 # https://docs.github.com/en/actions/creating-actions/dockerfile-support-for-github-actions#user @@ -47,7 +46,7 @@ jobs: uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: /tmp/build-cargo/ - key: ${{ runner.os }}-${{ runner.arch }}-cargo-target-asan-${{ matrix.php-version }}-${{ matrix.php-build }}-${{ env.RUST_TOOLCHAIN }}-${{ github.sha }}-${{ hashFiles('.github/workflows/prof_asan.yml') }} + key: ${{ runner.os }}-${{ runner.arch }}-cargo-target-asan-${{ matrix.php-version }}-${{ matrix.php-build }}-${{ hashFiles('rust-toolchain.toml') }}-${{ github.sha }}-${{ hashFiles('.github/workflows/prof_asan.yml') }} - name: Build and install profiler run: | @@ -56,15 +55,10 @@ jobs: export CC=clang-21 export CFLAGS='-fsanitize=address -fsanitize-address-use-after-scope -fno-omit-frame-pointer' export LDFLAGS='-fsanitize=address -shared-libasan' - export RUSTC_LINKER=lld-21 - rustup override set "${RUST_TOOLCHAIN}" - export RUSTFLAGS='-Zsanitizer=address -C force-frame-pointers=yes -C link-arg=-fsanitize=address -C link-arg=-shared-libasan' - export DDTRACE_PROFILING_TARGET="$(uname -m)-unknown-linux-gnu" - export DDTRACE_PROFILING_CARGO_BUILD_FLAGS='-Zbuild-std=std,panic_abort' - phpize - ./configure --disable-ddtrace-tracer --enable-ddtrace-profiling --disable-ddtrace-rust-debug - make -j"$(nproc)" - cp -v modules/datadog-profiling.so "$(php-config --extension-dir)/datadog-profiling.so" + # Uses the stable toolchain from rust-toolchain.toml; see the + # compile_profiler_asan target for the Rust sanitizer setup. + make compile_profiler_asan + cp -v tmp/build_profiler_asan/modules/datadog-profiling.so "$(php-config --extension-dir)/datadog-profiling.so" - name: Run phpt tests run: | @@ -125,12 +119,9 @@ jobs: export CXX=clang++-21 export CFLAGS='-fsanitize=undefined,local-bounds -fno-sanitize-recover=all -fno-omit-frame-pointer' export LDFLAGS='-fsanitize=undefined,local-bounds -fno-sanitize-recover=all' - export RUSTC_LINKER=clang-21 export RUSTFLAGS='-C link-arg=-fsanitize=undefined,local-bounds -C link-arg=-fno-sanitize-recover=all' - phpize - ./configure --disable-ddtrace-tracer --enable-ddtrace-profiling - make -j"$(nproc)" - cp -v modules/datadog-profiling.so "$(php-config --extension-dir)/datadog-profiling.so" + make compile_profiler + cp -v tmp/build_profiler/modules/datadog-profiling.so "$(php-config --extension-dir)/datadog-profiling.so" - name: Run phpt tests run: | diff --git a/.github/workflows/prof_correctness.yml b/.github/workflows/prof_correctness.yml index 9e9c226233..f44d1bb519 100644 --- a/.github/workflows/prof_correctness.yml +++ b/.github/workflows/prof_correctness.yml @@ -13,6 +13,7 @@ jobs: runs-on: ubuntu-24.04 env: LLVM_VERSION: "20" + PROFILER_SO: ${{ github.workspace }}/tmp/build_profiler/modules/datadog-profiling.so strategy: fail-fast: false matrix: @@ -47,7 +48,7 @@ jobs: ~/.cargo/registry/index/ ~/.cargo/registry/cache/ ~/.cargo/git/db/ - target-profiling/ + tmp/build_profiler/target-profiling/ key: ${{ runner.os }}-cargo-llvm-${{ env.LLVM_VERSION }}-${{ hashFiles('**/Cargo.lock', 'rust-toolchain.toml') }}-${{ matrix.php-version }}-${{ matrix.phpts }} - name: Build profiler @@ -78,9 +79,8 @@ jobs: "$LLVM_CONFIG_PATH" --version version_number=$(awk -F' = ' '$1 == "channel" { gsub(/"/, "", $2); print $2 }' rust-toolchain.toml) curl https://sh.rustup.rs -sSf | sh -s -- --profile minimal -y --default-toolchain "$version_number" - phpize - DDTRACE_PROFILING_FEATURES=trigger_time_sample ./configure --disable-ddtrace-tracer --enable-ddtrace-profiling - make -j"$(nproc)" + # trigger_time_sample is used by strange_frames.php. + make compile_profiler PROFILER_FEATURES=trigger_time_sample - name: Cache build dependencies uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 @@ -90,7 +90,7 @@ jobs: ~/.cargo/registry/index/ ~/.cargo/registry/cache/ ~/.cargo/git/db/ - target-profiling/ + tmp/build_profiler/target-profiling/ key: ${{ runner.os }}-cargo-llvm-${{ env.LLVM_VERSION }}-${{ hashFiles('**/Cargo.lock', 'rust-toolchain.toml') }}-${{ matrix.php-version }}-${{ matrix.phpts }} - name: Run no profile test @@ -99,11 +99,11 @@ jobs: export DD_PROFILING_EXPERIMENTAL_FEATURES_ENABLED=1 export DD_PROFILING_EXCEPTION_MESSAGE_ENABLED=1 php -v - php -d extension=modules/datadog-profiling.so --ri datadog-profiling + php -d extension="$PROFILER_SO" --ri datadog-profiling for test_case in "allocations" "time" "strange_frames" "timeline" "exceptions" "io" "socket_io" "io_upscaling" "allocation_time_combined" "generators"; do mkdir -p profiling/tests/correctness/"$test_case"/ export DD_PROFILING_OUTPUT_PPROF=$PWD/profiling/tests/correctness/"$test_case"/test.pprof - php -d extension="${PWD}/modules/datadog-profiling.so" "profiling/tests/correctness/${test_case}.php" + php -d extension="$PROFILER_SO" "profiling/tests/correctness/${test_case}.php" if compgen -G "$DD_PROFILING_OUTPUT_PPROF.*" > /dev/null; then echo "Profile output should not exist:" ls -l "$DD_PROFILING_OUTPUT_PPROF".* @@ -118,20 +118,20 @@ jobs: export DD_PROFILING_EXPERIMENTAL_EXCEPTION_SAMPLING_DISTANCE=1 export DD_PROFILING_EXCEPTION_MESSAGE_ENABLED=1 php -v - php -d extension=modules/datadog-profiling.so --ri datadog-profiling + php -d extension="$PROFILER_SO" --ri datadog-profiling for test_case in "allocations" "time" "strange_frames" "timeline" "exceptions" "io" "socket_io" "io_upscaling" "allocation_time_combined" "generators"; do mkdir -p profiling/tests/correctness/"$test_case"/ export DD_PROFILING_OUTPUT_PPROF=$PWD/profiling/tests/correctness/"$test_case"/test.pprof - php -d extension=$PWD/modules/datadog-profiling.so profiling/tests/correctness/"$test_case".php + php -d extension="$PROFILER_SO" profiling/tests/correctness/"$test_case".php done mkdir -p profiling/tests/correctness/allocations_1byte/ export DD_PROFILING_OUTPUT_PPROF=$PWD/profiling/tests/correctness/allocations_1byte/test.pprof export DD_PROFILING_ALLOCATION_SAMPLING_DISTANCE=1 - php -d extension=$PWD/modules/datadog-profiling.so profiling/tests/correctness/allocations.php + php -d extension="$PROFILER_SO" profiling/tests/correctness/allocations.php mkdir -p profiling/tests/correctness/allocations_1byte_no_zend_alloc/ export DD_PROFILING_OUTPUT_PPROF=$PWD/profiling/tests/correctness/allocations_1byte_no_zend_alloc/test.pprof export DD_PROFILING_ALLOCATION_SAMPLING_DISTANCE=1 - USE_ZEND_ALLOC=0 php -d extension=$PWD/modules/datadog-profiling.so profiling/tests/correctness/allocations.php + USE_ZEND_ALLOC=0 php -d extension="$PROFILER_SO" profiling/tests/correctness/allocations.php unset DD_PROFILING_ALLOCATION_SAMPLING_DISTANCE - name: Run ZTS tests @@ -142,11 +142,11 @@ jobs: export DD_PROFILING_EXPERIMENTAL_EXCEPTION_SAMPLING_DISTANCE=1 export DD_PROFILING_EXCEPTION_MESSAGE_ENABLED=1 php -v - php -d extension=modules/datadog-profiling.so --ri datadog-profiling + php -d extension="$PROFILER_SO" --ri datadog-profiling for test_case in "exceptions_zts"; do mkdir -p profiling/tests/correctness/"$test_case"/ export DD_PROFILING_OUTPUT_PPROF=$PWD/profiling/tests/correctness/"$test_case"/test.pprof - php -d extension=$PWD/modules/datadog-profiling.so profiling/tests/correctness/"$test_case".php + php -d extension="$PROFILER_SO" profiling/tests/correctness/"$test_case".php done - name: Check profiler correctness for allocations diff --git a/Makefile b/Makefile index b8fb7ddec1..ad3e24d8d8 100644 --- a/Makefile +++ b/Makefile @@ -419,13 +419,35 @@ prod: strict: $(eval CFLAGS=-Wall -Werror -Wextra) +PROFILER_BUILD_SUFFIX ?= profiler +# Extra Cargo features for the profiler, comma-separated (`profiling` is always +# enabled), e.g. PROFILER_FEATURES=trigger_time_sample for correctness tests and +# benchmarks. Features are baked in at configure time: remove +# tmp/build_$(PROFILER_BUILD_SUFFIX) when changing them. +PROFILER_FEATURES ?= + compile_profiler: - DDTRACE_PROFILING_FEATURES=trigger_time_sample $(MAKE) BUILD_SUFFIX=profiler PROFILING=1 EXTRA_CONFIGURE_OPTIONS="--disable-ddtrace-tracer --enable-ddtrace-profiling" all + DDTRACE_PROFILING_FEATURES="$(PROFILER_FEATURES)" $(MAKE) BUILD_SUFFIX=$(PROFILER_BUILD_SUFFIX) PROFILING=1 EXTRA_CONFIGURE_OPTIONS="--disable-ddtrace-tracer --enable-ddtrace-profiling" all install_profiler: compile_profiler - cp $(PROJECT_ROOT)/tmp/build_profiler/modules/datadog-profiling.so $(PHP_EXTENSION_DIR)/datadog-profiling.so + cp $(PROJECT_ROOT)/tmp/build_$(PROFILER_BUILD_SUFFIX)/modules/datadog-profiling.so $(PHP_EXTENSION_DIR)/datadog-profiling.so $(Q) echo "extension=datadog-profiling.so" | $(SUDO) tee $(INI_DIR)/datadog-profiling.ini +# Profiler with AddressSanitizer on the Rust side. Uses the pinned stable +# toolchain; RUSTC_BOOTSTRAP=1 unlocks -Zsanitizer and -Zbuild-std. std is +# rebuilt so it is instrumented too, which requires an explicit --target so the +# sanitizer flags don't apply to build scripts and proc-macros. +# C code pulled in by build scripts (cc crate) still honors CC/CFLAGS/LDFLAGS +# from the environment. +PROFILER_ASAN_RUSTFLAGS = -Zsanitizer=address -C force-frame-pointers=yes -C link-arg=-fsanitize=address -C link-arg=-shared-libasan +PROFILER_ASAN_ENV = RUSTC_BOOTSTRAP=1 \ + RUSTFLAGS="$(PROFILER_ASAN_RUSTFLAGS) $${RUSTFLAGS:-}" \ + DDTRACE_PROFILING_TARGET=$(ARCHITECTURE)-unknown-linux-gnu \ + DDTRACE_PROFILING_CARGO_BUILD_FLAGS=-Zbuild-std=std,panic_abort + +compile_profiler_asan: + $(PROFILER_ASAN_ENV) $(MAKE) PROFILER_BUILD_SUFFIX=profiler_asan compile_profiler + clang_find_files_to_lint: @find . \( \ -path ./.git -prune -o \ @@ -460,26 +482,26 @@ remove_cbindgen: generate_cbindgen: cbindgen_binary # Regenerate components-rs/datadog.h components-rs/live-debugger.h components-rs/telemetry.h components-rs/sidecar.h components-rs/common.h components-rs/crashtracker.h components-rs/library-config.h ( \ - $(command rustup && echo run nightly --) cbindgen --crate datadog-php \ + cbindgen --crate datadog-php \ --config cbindgen.toml \ --output $(PROJECT_ROOT)/components-rs/datadog.h; \ cd libdatadog; \ - $(command rustup && echo run nightly --) cbindgen --crate libdd-common-ffi \ + cbindgen --crate libdd-common-ffi \ --config libdd-common-ffi/cbindgen.toml \ --output $(PROJECT_ROOT)/components-rs/common.h; \ - $(command rustup && echo run nightly --) cbindgen --crate libdd-live-debugger-ffi \ + cbindgen --crate libdd-live-debugger-ffi \ --config libdd-live-debugger-ffi/cbindgen.toml \ --output $(PROJECT_ROOT)/components-rs/live-debugger.h; \ - $(command rustup && echo run nightly --) cbindgen --crate libdd-telemetry-ffi \ + cbindgen --crate libdd-telemetry-ffi \ --config libdd-telemetry-ffi/cbindgen.toml \ --output $(PROJECT_ROOT)/components-rs/telemetry.h; \ - $(command rustup && echo run nightly --) cbindgen --crate datadog-sidecar-ffi \ + cbindgen --crate datadog-sidecar-ffi \ --config datadog-sidecar-ffi/cbindgen.toml \ --output $(PROJECT_ROOT)/components-rs/sidecar.h; \ - $(command rustup && echo run nightly --) cbindgen --crate libdd-crashtracker-ffi \ + cbindgen --crate libdd-crashtracker-ffi \ --config libdd-crashtracker-ffi/cbindgen.toml \ --output $(PROJECT_ROOT)/components-rs/crashtracker.h; \ - $(command rustup && echo run nightly --) cbindgen --crate libdd-library-config-ffi \ + cbindgen --crate libdd-library-config-ffi \ --config libdd-library-config-ffi/cbindgen.toml \ --output $(PROJECT_ROOT)/components-rs/library-config.h; \ if test -d $(PROJECT_ROOT)/tmp; then \ @@ -1685,4 +1707,5 @@ composer.lock: composer.json $(call run_composer_with_retry,,) .PHONY: dev dist_clean clean cores all clang_format_check clang_format_fix install sudo_install test_c test_c_mem test_extension_ci test_extension_ci_normal test_extension_ci_valgrind test_zai test_zai_asan test install_ini install_all \ - .apk .rpm .deb .tar.gz sudo debug prod strict run-tests.php verify_pecl_file_definitions verify_package_xml cbindgen cbindgen_binary + .apk .rpm .deb .tar.gz sudo debug prod strict run-tests.php verify_pecl_file_definitions verify_package_xml cbindgen cbindgen_binary \ + compile_profiler install_profiler compile_profiler_asan diff --git a/benchmark/otel-profiler-context/run.sh b/benchmark/otel-profiler-context/run.sh index 3479b3236f..f28be3c1a3 100755 --- a/benchmark/otel-profiler-context/run.sh +++ b/benchmark/otel-profiler-context/run.sh @@ -53,7 +53,7 @@ build_extensions() { sh -c ' make -C /work -j"$(nproc)" all \ "CFLAGS=-O2 -g0 -DNDEBUG -Wall -Wextra" - make -C /work compile_profiler + make -C /work compile_profiler PROFILER_FEATURES=trigger_time_sample ' } diff --git a/benchmark/runall.sh b/benchmark/runall.sh index 4e60be6c96..842dd7700b 100755 --- a/benchmark/runall.sh +++ b/benchmark/runall.sh @@ -6,7 +6,7 @@ if [ "$SCENARIO" = "profiler" ]; then # Run Profiling Benchmarks cd ../profiling/ - make -C .. compile_profiler + make -C .. compile_profiler PROFILER_FEATURES=trigger_time_sample sirun benches/timeline.json > "$ARTIFACTS_DIR/sirun_timeline.ndjson" diff --git a/profiling/benches/fakeapp/bench.sh b/profiling/benches/fakeapp/bench.sh index 3a478589fc..cb53b20413 100755 --- a/profiling/benches/fakeapp/bench.sh +++ b/profiling/benches/fakeapp/bench.sh @@ -10,7 +10,7 @@ rm -v trigger-{0..2}.txt set -eu -make -C ../../.. compile_profiler +make -C ../../.. compile_profiler PROFILER_FEATURES=trigger_time_sample RUST_LOG=trace php -c . -dextension="$PWD/../../../tmp/build_profiler/modules/datadog-profiling.so" -S 0.0.0.0:8080 -t public &> output.txt & pid=$!