[pylaterreur]

Description

The crewai tests have failed at random in CI since 2026-10-26: 13 of the 36 crewai jobs in 9 runs (#285 and other PRs), each on one or two tests, with a real 401 from DashScope for the fake key (litellm.AuthenticationError), with OpenAIException - Connection error. (litellm.InternalServerError, e.g. test_agent_with_multiple_tools), or, in an error test, with no ERROR span. The 80 crewai jobs of the 20 runs before, since 2026-09-22, all passed.

crewAI exports anonymous telemetry to telemetry.crewai.com from a background thread, and the conftest lists that host in ignore_hosts, so vcrpy sends the exports to the real host. It sends such http.client requests inside vcr.patch.force_reset(), which puts back the original methods of every HTTP client it patches, httpx included, for the whole process while it connects and sends (0.3 to 0.5 s here). An LLM call that crewAI makes on its worker thread in that window bypasses the cassette and reaches the provider. When the window spans the end of a test, the patches go back out of order and the next test is replayed from the previous test's cassette. The exports only go out since vcrpy 8.5.0 (2026-10-25): before, filter_post_data_parameters failed on their binary body with UnicodeDecodeError, which the exporter swallowed.

Changes, in the crewai tests' conftest.py:

• set CREWAI_DISABLE_TELEMETRY and CREWAI_DISABLE_VERSION_CHECK (crewAI's PyPI lookup when a verbose crew starts);
• drop ignore_hosts, so VCR passes nothing through to the network;
• mark the crewai tests with pytest-recording's block_network, so a request that gets past VCR fails with "Network is disabled" instead of reaching a real host. Recording with a --record-mode other than none still has network access.

Type of change

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

tox -c tox-loongsuite.ini -e py312-test-loongsuite-instrumentation-crewai -- -ra as CI runs it (tox-uv, GitHub's CPython 3.12.13 build), with the package versions of the failing CI jobs (crewai 1.15.30, vcrpy 9.2.0, litellm 1.95.0, openai 2.54.0), on main and on this branch:

• Network on, as in CI, 20 runs each, interleaved: on main, 1 run failed (test_sync_call_with_error), and 19 of the 20 sent LLM requests to DashScope (401 for the fake key; the error tests accept any error). On this branch, 20 passed and no request reached the network.
• Network off (bwrap --unshare-net), this branch: 20 of 20 passed in a row.
• Network off, with telemetry.crewai.com mapped to a local endpoint that holds each TLS handshake for 0.4 s: main failed 4 of 13 runs, e.g. test_agent_with_multiple_tools with litellm.InternalServerError: ... OpenAIException - Connection error.; this branch passed 13 of 13.
• This branch on Python 3.10 (vcrpy 8.5.0), 3.11 and 3.13: 5 runs offline and 3 online each, all passed.
• tox -c tox-loongsuite.ini -e lint-loongsuite-instrumentation-crewai (Python 3.13) and tox -e precommit (Python 3.11) pass.

Does This PR Require a Core Repo Change?

• No.

Checklist:

• Followed the style guidelines of this project
• Changelogs have been updated
• Unit tests have been added
• Documentation has been updated

🤖 Generated with Claude Code

https://claude.ai/code/session_01GwUvhmzgfkiZ6vo4uGyesN

[Copilot]

🟢 Approval recommended

No unresolved blocking issues were identified, and the changes address the reported test flakiness.

No open issues 🎉

What changed in this PR

This pull request makes CrewAI instrumentation tests deterministic and network-isolated by disabling background network activity and blocking unintended external requests.

Changes:

• Disable CrewAI telemetry and version checks during tests.
• Remove VCR pass-through hosts and apply block_network to collected tests.
• Document the reliability fix in the CrewAI changelog.

File	Description
instrumentation-loongsuite/​loongsuite-instrumentation-crewai/​tests/​conftest.py	Configures CrewAI and VCR for offline, network-blocked tests.
instrumentation-loongsuite/​loongsuite-instrumentation-crewai/​CHANGELOG.md	Records the test hermeticity and flakiness fix.

_{Review effort: Lite}