feat: modernize langchain integration connector

[daveomri]

Summary

Umbrella PR for the langchain-apify modernisation. Four feature sub-branches were squash-merged into this branch (each independently reviewed, with per-tool behaviour validated in the original sub-PRs by the QA tester), plus umbrella-level cleanup, a few bug fixes caught during review, schema-level clamp documentation, and refreshed user-facing docs.

What lands

Four squash-merged sub-branches:

• Core generic tools (feat: modernize langchain integration core tools #28):
  • 6 tools (ApifyRunActorTool, ApifyGetDatasetItemsTool, ApifyRunActorAndGetDatasetTool, ApifyScrapeUrlTool, ApifyRunTaskTool, ApifyRunTaskAndGetDatasetTool); APIFY_TOKEN env-var rename with APIFY_API_TOKEN kept as a deprecated alias; apify_token kwarg.
• LangChain-native components (feat: modernize langchain integration native components #29):
  • ApifySearchRetriever (BaseRetriever over apify/rag-web-browser) and ApifyCrawlLoader (BaseLoader over apify/website-content-crawler).
• Search & crawling tools (feat: modernize langchain integration crawl tools #31):
  • 6 tools (ApifyGoogleSearchTool, ApifyWebCrawlerTool, ApifyRAGWebBrowserTool, ApifyGoogleMapsTool, ApifyYouTubeScraperTool, ApifyEcommerceScraperTool).
• Social-media tools (feat: modernize langchain integration social tools #30):
  • 7 tools (Instagram; LinkedIn posts / search / detail; Twitter; TikTok; Facebook).

Plus README improvements (#27) and convenience lists APIFY_CORE_TOOLS / APIFY_SEARCH_TOOLS / APIFY_SOCIAL_TOOLS.

Umbrella-level changes on top of the merges

• Package split:
  • collapsed _actor_tools.py + tools.py into a tools/ package (base / core / actors / search / social); public API unchanged.
• Single-source constants:
  • shared defaults in _constants.py, shared Literal aliases in _types.py, centralised error strings in _error_messages.py.
• Schema-level clamp documentation:
  • lifted the 4 clamp ceilings (max_timeout_secs=600, max_memory_mbytes=32768, max_items=1000, max_crawl_depth=5) from hardcoded Field defaults on _ApifyGenericTool into named constants in _constants.py (_MAX_TIMEOUT_SECS_CAP, etc.).
  • every clamp-relevant Field description across the 8 input schemas now carries (clamped to N max), interpolated from the same constant the base class uses. Addresses the tester's "Fetch 999999 items" feedback: the LLM now sees the cap in the schema and won't promise above-cap results in its narration.
  • new tests/unit_tests/test_clamp_descriptions.py (30 parametrised tests) pins the description text and asserts no drift between the advertised cap and the live _ApifyGenericTool default.
• Bug fixes:
  • Integration tests now assert against the real {run: {...}, items: [...]} envelope shape (the old asserts would KeyError against the live API).
  • ApifyWrapper.(a)call_actor(_task) forwards apify_token to the ApifyDatasetLoader it constructs; an explicit wrapper token no longer requires APIFY_TOKEN to also be in the environment.
  • _prune_actor_input_schema preserves Actor-schema entries with default: 0, default: false, or default: "" (22 such fields in apify/website-content-crawler alone were being silently dropped).
• Minor perf:
  • ApifyActorsTool.__init__ now fetches the Actor build once and reuses it across _create_description and _build_tool_args_schema_model (was 2 fetches per init).
• Cleanup:
  • ApifyDatasetLoader.apify_client is now PrivateAttr instead of Field(default=None) # type: ignore, matching the pattern used by every other internal-client field in the package.

Docs

• README rewrite:
  • Tools section restructured into three families with a minimal .invoke() example per family.
  • New "Using tools with an agent" section showing how to bind a convenience list to create_react_agent.
  • New sections for ApifySearchRetriever and ApifyCrawlLoader.
  • Existing ApifyActorsTool / ApifyDatasetLoader / ApifyWrapper / "Note for Apify Actor developers" sections preserved.
• DEVELOPMENT.md: light edits; canonicalised APIFY_TOKEN references with a footer note about the APIFY_API_TOKEN deprecation alias; added a pointer to the new CONTRIBUTING.md.
• CONTRIBUTING.md (new): issue / PR scope conventions, commit message prefixes that drive git-cliff version bumps, review expectations, and the auto-release workflow.

Test plan

• make test: 352 unit tests pass (322 from sub-PRs + 30 new clamp-description tests).
• make lint: clean (35 source files, including the new test module).
• Manual smoke run via dist/.playground.py against the real Apify API: offline checks (merge surface, social input mapping, Instagram URL building, retriever source order, Google-search page count, the new clamp-offline block: 20/20 checks pass), plus live core + selected social / search / crawling probes all green. Tester-confirmed in the sub-PRs; the umbrella was re-validated against the live API after each fix.

Follow-ups (separate PRs)

• Use ActorClient.default_build() from the Apify SDK in place of the raw requests.GET in _get_actor_latest_build. Deferred because it requires bumping apify-client ^2.3.0 → ^2.5.0 (the sync method was incorrectly async def until 2.5.0); the raw call works fine for both public and private Actors today.
• Migrate the LangGraph examples (docs/examples/tools_example.py) to the LangGraph 1.0 API. The connector itself does not depend on LangGraph, so this is docs/examples / dev-deps only.

[jirispilka]

I pushed follow-up changes on top of this PR as #36 (base: feat/modernize-langchain-integration). No change to public tool behaviour.

Fixes

• APIFY_API_TOKEN env var now emits the DeprecationWarning that DEVELOPMENT.md promised (with tests).
• Added the CHANGELOG breaking-change entry (apify_client → private _apify_client; APIFY_API_TOKEN deprecated alias).
• Dropped 8 internal _-prefixed symbols from langchain_apify.tools.__all__.

Quality (behaviour-equivalent)

• Hardcoded max_results defaults now use _constants.py.
• Removed duplicate _list_items_or_raise (use get_dataset_items).
• _scrape_url → public scrape_url_with_metadata (drops a # noqa: SLF001).
• Core tools catch _TOOL_RUN_ERRORS, matching search/social tools.
• ApifySearchRetriever guards a missing token, matching the base tool.

---

Generated by Claude Code

[jirispilka]

Thanks. Approving. Decide whether makes sense for you or not #36

[drobnikj]

👍

[vojtechj-apify]

👋 Testing note on this PR — looks solid overall (tools, retriever/loader, and LangChain agent-routing all validated live). One connector item to consider:

max_results cap doesn't match the Actor's real limit

ApifyRAGWebBrowserTool / ApifySearchRetriever advertise max_results as "clamped to 1000" (the generic _MAX_ITEMS_CAP), but the apify/rag-web-browser Actor's input schema enforces maxResults <= 100. _clamp_items clamps only to 1000, and ApifySearchRetriever (retrievers.py:97 / :112) passes max_results through with no clamp at all — so any value in (100, 1000] reaches the Actor and is rejected, and the call fails instead of clamping down.

Repro:

ApifyRAGWebBrowserTool().invoke({"query": "x", "max_results": 150})
# -> 'Apify Actor call failed ... Field input.maxResults must be <= 100'
# max_results=100 succeeds

ApifyScrapeUrlTool is unaffected (single URL, no max_results).

Two directions (leaving the choice to you)

1. Minimal, matches this PR's pattern — a dedicated _RAG_MAX_RESULTS_CAP = 100 in _constants.py, referenced by the tool (max_items), the retriever, and the interpolated field description (keeps test_clamp_descriptions.py truthful). One line to change if the Actor's limit ever moves.
2. Self-correcting — derive the cap from the Actor's input schema at init (properties.maxResults.maximum) and clamp to it. The connector already reads Actor input schemas via _get_actor_latest_build() / _prune_actor_input_schema() for ApifyActorsTool, so the machinery exists. This drops the need to hardcode/monitor per-Actor limits and generalizes to the other tools — at the cost of a fetch at construction and making the advertised-cap description dynamic (currently static + drift-tested). Could ride on the planned ActorClient.default_build() follow-up.

Either way test_clamp_descriptions.py will need a matching update. Happy to open a follow-up issue if that's easier to track. (Found via live testing; details on request.)

cc: @daveomri

---

✅ Option A implementation posted as inline suggestions in the review just below (apply as a batch — Add to batch → Commit suggestions). Option B remains the more robust alternative.

[vojtechj-apify]

Option A — inline suggestions (named constant). Draft of implementation of the constant-based fix from my comment above. ⚠️ Apply these as a batch (GitHub: Add to batch → Commit suggestions) — they introduce _RAG_MAX_RESULTS_CAP plus its imports across 3 files, so applying them one at a time can transiently break until all land.

Option B (deriving the cap from the Actor's input schema at init) is still the more robust route if you'd rather not track per-Actor caps by hand.

After applying, run make test — test_clamp_descriptions.py may need a small tweak since ApifyRAGWebBrowserTool's advertised cap changes 1000 → 100.

[jirispilka]

@daveomri I think we can merge this irrespective of other PRs, right?