diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..de8eac4 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,65 @@ +# Contributing + +Thanks for considering a contribution to `langchain-apify`. This file covers the contribution flow: what to file, how to scope a PR, and what review looks for. + +> Setting up locally for the first time? Start with [DEVELOPMENT.md](DEVELOPMENT.md). Install, lint, and test commands live there. + +## Filing issues + +Open a GitHub issue at . A few notes: + +- **Bug reports**: include the package version, a short reproducer, the expected behaviour, and what you saw instead. If you have an Apify run ID from the failing call, paste it; it's the fastest way for us to inspect the run server-side. +- **Feature requests**: describe the use case before the proposed API. We'd rather discuss the shape of the solution before code lands. +- Search existing issues first; the maintainers may already be tracking the same thing. + +## Working on a pull request + +- Branch from `main`. +- Keep one logical change per PR. A PR that fixes a bug *and* adds a feature is harder to review and harder to revert if needed. +- Before opening: run `make lint` and `make test` locally. CI will also run integration tests; you don't need an `APIFY_TOKEN` to open the PR (CI has its own). +- If your change affects the public API, update the README and any in-repo examples. +- If your change adds a new tool family or generic primitive, add or extend the corresponding test file under `tests/unit_tests/`. + +## Commit message conventions + +The release workflow uses `git-cliff` to read commit-message prefixes and auto-generate both the version bump and the changelog. **Don't manually edit `version =` in `pyproject.toml` or write `CHANGELOG.md` entries by hand.** + +Use these prefixes: + +- `feat:` for a new feature → minor version bump +- `fix:` for a bug fix → patch bump +- `ref:` for a refactor → patch +- `test:` for a test-only change → patch +- `chore:` for housekeeping → patch +- `docs:` for a docs-only change → no bump on a stable release (skipped from pre-releases) +- `ci:` for a CI / workflow change (skipped from pre-releases) + +A `BREAKING CHANGE:` footer in the commit body triggers a major bump. Use it sparingly and only when the public API genuinely breaks. + +A good message is short on the subject line and explains *why* (not what) in the body: + +``` +fix: forward apify_token from ApifyWrapper to ApifyDatasetLoader + +Without this, an explicit wrapper token still required APIFY_TOKEN to +also be set in the environment. The loader fell back to env-var +resolution and raised ValueError if neither was present. +``` + +## What review looks for + +- **Correctness on the public API surface.** Any new tool must follow the `_ApifyGenericTool` envelope contract (a JSON string of `{"run": {...}, "items": [...]}`) and route Actor calls through `ApifyToolsClient` (`_client.py`), not the SDK directly. +- **`make lint` and `make test` pass locally.** Integration tests pass under CI's token; you don't need to run them yourself unless you're touching `_client.py`. +- **No new `apify_api_token` field declarations.** The canonical token kwarg is `apify_token`; the legacy `apify_api_token` is honoured only via the existing deprecation plumbing in `_utils.py` and per-tool model validators. New code should not introduce fresh `apify_api_token` fields. +- **No manually bumped `version =` in `pyproject.toml`.** Versions come from commit messages via `git-cliff`. +- **Shared defaults stay in `_constants.py`.** Don't reintroduce magic literals (`300`, `100`, `120`, etc.); import the named constant instead. + +## Releases + +Releases are automated. After a PR merges to `main`, the release workflow: + +1. Reads commit messages since the last tag. +2. Computes the new version with `git-cliff`. +3. Bumps `pyproject.toml`, writes the changelog, tags the release, and pushes to PyPI. + +You don't need to do any of those steps manually. diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 2c94ae5..885d8d3 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -1,8 +1,8 @@ # Development -## Contributing +This file covers everything you need to run the code locally: install, format, lint, and test. -If you want to contribute, please ensure that you have your environment properly set up. Run the following commands to make sure that the code is properly formatted and is not breaking before submitting a pull request. +> Planning a pull request? Read [CONTRIBUTING.md](CONTRIBUTING.md) for issue and PR conventions before opening one. ## Installation @@ -42,12 +42,14 @@ make test To run integration tests, use the following command: ```bash -APIFY_API_TOKEN="YOUR_TOKEN" make integration_test +APIFY_TOKEN="YOUR_TOKEN" make integration_test ``` To run single test file, use `TEST_FILE` argument: ```bash make test TEST_FILE=path_to/test_file.py -APIFY_API_TOKEN="YOUR_TOKEN" make integration_test TEST_FILE=path_to/test_file.py +APIFY_TOKEN="YOUR_TOKEN" make integration_test TEST_FILE=path_to/test_file.py ``` + +> `APIFY_API_TOKEN` is also accepted as a deprecated alias for `APIFY_TOKEN` (emits a `DeprecationWarning`). New code and examples should use `APIFY_TOKEN`. diff --git a/README.md b/README.md index e95b86a..2d743f4 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ LangChain Apify: A full-stack scraping platform built on Apify's infrastructure --- -Build web scraping and automation workflows in Python by connecting Apify Actors with LangChain. This package gives you programmatic access to Apify's infrastructure - run scraping tasks, handle datasets, and use the API directly through LangChain's tools. +Build web scraping and automation workflows in Python by connecting Apify Actors with LangChain. This package gives you programmatic access to Apify's infrastructure: run scraping tasks, handle datasets, and use the API directly through LangChain's tools. ## Agentic LLMs @@ -39,97 +39,202 @@ pip install langchain-apify ## Prerequisites -You should configure credentials by setting the following environment variables: -- `APIFY_API_TOKEN` - Apify API token +You should configure credentials by setting the following environment variable: +- `APIFY_TOKEN`: Apify API token. (`APIFY_API_TOKEN` is also honoured as a deprecated alias for backwards compatibility.) Register your free Apify account [here](https://console.apify.com/sign-up) and learn how to get your API token in the [Apify documentation](https://docs.apify.com/platform/integrations/api). ## Tools -`ApifyActorsTool` class provides access to [Apify Actors](https://apify.com/store), which are cloud-based web-scraping and automation programs that you can run without managing any infrastructure. For more detailed information, see the [Apify Actors documentation](https://docs.apify.com/platform/actors). +The package ships dedicated tools across three families plus a generic "wrap any Actor by ID" tool for everything else. All return a uniform `{"run": {...}, "items": [...]}` JSON envelope (parse with `json.loads`). -`ApifyActorsTool` is useful when you need to run an Apify Actor as a tool in LangChain. You can use the tool to interact with the Actor manually or as part of an agent workflow. +### Core tools + +Generic platform primitives: run any Actor or task and fetch dataset items. Available as the convenience list `APIFY_CORE_TOOLS`: + +- `ApifyRunActorTool`: start any Actor, return run metadata +- `ApifyGetDatasetItemsTool`: fetch items from a dataset by ID +- `ApifyRunActorAndGetDatasetTool`: run + fetch in one call +- `ApifyScrapeUrlTool`: single URL to markdown +- `ApifyRunTaskTool`: run a saved Actor task +- `ApifyRunTaskAndGetDatasetTool`: task run + fetch in one call -Example usage of `ApifyActorsTool` with the [RAG Web Browser](https://apify.com/apify/rag-web-browser) Actor, which searches for information on the web: ```python -import os -import json -from langchain_apify import ApifyActorsTool +import os, json +from langchain_apify import ApifyRunActorAndGetDatasetTool -os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY" -os.environ["APIFY_API_TOKEN"] = "YOUR_APIFY_API_TOKEN" +os.environ["APIFY_TOKEN"] = "YOUR_APIFY_TOKEN" -browser = ApifyActorsTool('apify/rag-web-browser') -search_results = browser.invoke(input={ - "run_input": {"query": "what is Apify Actor?", "maxResults": 3} +result = ApifyRunActorAndGetDatasetTool().invoke({ + "actor_id": "apify/python-example", + "run_input": {"first_number": 2, "second_number": 3}, }) +print(json.loads(result)) +``` + +### Search & crawling tools + +Web search, maps, video, e-commerce, and content crawling. Available as `APIFY_SEARCH_TOOLS`: + +- `ApifyGoogleSearchTool`: Google search results +- `ApifyWebCrawlerTool`: multi-page website crawler +- `ApifyRAGWebBrowserTool`: search + fetch top results in one call +- `ApifyGoogleMapsTool`: places, reviews, business details +- `ApifyYouTubeScraperTool`: videos, channels, metadata +- `ApifyEcommerceScraperTool`: product pages and category listings + +```python +import os, json +from langchain_apify import ApifyGoogleSearchTool + +os.environ["APIFY_TOKEN"] = "YOUR_APIFY_TOKEN" + +result = ApifyGoogleSearchTool().invoke({ + "query": "langchain apify integration", + "max_results": 5, +}) +print(json.loads(result)) +``` + +### Social media tools + +Instagram, LinkedIn, Twitter/X, TikTok, and Facebook. Available as `APIFY_SOCIAL_TOOLS`: -# use the tool with an agent +- `ApifyInstagramScraperTool`: profiles, hashtags, posts, comments +- `ApifyLinkedInProfilePostsTool`: posts from a LinkedIn profile +- `ApifyLinkedInProfileSearchTool`: keyword search for profiles +- `ApifyLinkedInProfileDetailTool`: full profile detail +- `ApifyTwitterScraperTool`: tweets and users +- `ApifyTikTokScraperTool`: videos, users, hashtags +- `ApifyFacebookPostsScraperTool`: public page posts + +```python +import os, json +from langchain_apify import ApifyInstagramScraperTool + +os.environ["APIFY_TOKEN"] = "YOUR_APIFY_TOKEN" + +result = ApifyInstagramScraperTool().invoke({ + "search_type": "user", + "search_query": "apify", + "max_results": 3, +}) +print(json.loads(result)) +``` + +### Using tools with an agent + +Each convenience list lets you bind a whole tool family to an agent in one line. Don't bind all tools at once. Most LLMs lose routing accuracy past ~8 tools, so pick the family the agent actually needs. + +```python +import os +from langchain_apify import APIFY_SEARCH_TOOLS from langchain_openai import ChatOpenAI from langgraph.prebuilt import create_react_agent -model = ChatOpenAI(model="gpt-4o-mini") -tools = [browser] +os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY" +os.environ["APIFY_TOKEN"] = "YOUR_APIFY_TOKEN" + +model = ChatOpenAI(model="gpt-5.4-mini") +tools = [tool_cls() for tool_cls in APIFY_SEARCH_TOOLS] agent = create_react_agent(model, tools) for chunk in agent.stream( - {"messages": [("human", "search for what is Apify?")]}, - stream_mode="values" + {"messages": [("human", "search the web for what Apify Actors are")]}, + stream_mode="values", ): chunk["messages"][-1].pretty_print() ``` +### `ApifyActorsTool`: wrap any Actor by ID + +For Actors without a dedicated wrapper above, `ApifyActorsTool` builds an input schema from the Actor's build at construction time and exposes it as a generic LangChain tool: + +```python +import os +from langchain_apify import ApifyActorsTool + +os.environ["APIFY_TOKEN"] = "YOUR_APIFY_TOKEN" + +tool = ApifyActorsTool("apify/rag-web-browser") +result = tool.invoke(input={ + "run_input": {"query": "what is an Apify Actor?", "maxResults": 3}, +}) +``` + +## Retriever + +`ApifySearchRetriever` is a `BaseRetriever` over `apify/rag-web-browser` for RAG pipelines. Each result becomes a LangChain `Document` with `metadata['source']`, `metadata['title']`, and any additional fields the Actor returns. + +```python +import os +from langchain_apify import ApifySearchRetriever + +os.environ["APIFY_TOKEN"] = "YOUR_APIFY_TOKEN" + +retriever = ApifySearchRetriever(max_results=3) +docs = retriever.invoke("what is web scraping") +for doc in docs: + print(doc.metadata["source"], "-", doc.metadata.get("title")) +``` + ## Document loaders -> **⚠️ Note for Actor Developers**: If you're building an Apify Actor, use `Actor.open_dataset()` from the Apify SDK instead of this loader. See the [Note for Apify Actor developers](#note-for-apify-actor-developers) section for details. +> **⚠️ Note for Actor Developers**: If you're building an Apify Actor, use `Actor.open_dataset()` from the Apify SDK instead of these loaders. See the [Note for Apify Actor developers](#note-for-apify-actor-developers) section for details. -`ApifyDatasetLoader` class provides access to [Apify datasets](https://docs.apify.com/platform/storage/dataset) as document loaders. Datasets are storage solutions that store results from web scraping, crawling, or data processing. +### `ApifyCrawlLoader` -`ApifyDatasetLoader` is useful when you need to process data from an Apify Actor run **from outside the Actor runtime** (e.g., in an external script, notebook, or application). If you are extracting webpage content, you would typically use this loader after running an Apify Actor manually from the [Apify console](https://console.apify.com), where you can access the results stored in the dataset. +Active crawler that wraps `apify/website-content-crawler`. Crawls a seed URL and returns each page as a `Document` with `metadata = {"source", "title", "crawl_depth"}`. Implements `lazy_load()` for streaming and `load()` for the eager collection. -Example usage for `ApifyDatasetLoader` with a custom dataset mapping function for loading webpage content and source URLs as a list of `Document` objects containing the page content and source URL. ```python import os -from langchain_apify import ApifyDatasetLoader +from langchain_apify import ApifyCrawlLoader -os.environ["APIFY_API_TOKEN"] = "YOUR_APIFY_API_TOKEN" +os.environ["APIFY_TOKEN"] = "YOUR_APIFY_TOKEN" -# Example dataset structure -# [ -# { -# "text": "Example text from the website.", -# "url": "http://example.com" -# }, -# ... -# ] +loader = ApifyCrawlLoader( + url="https://docs.apify.com", + max_crawl_pages=5, + max_crawl_depth=1, +) +documents = loader.load() +``` + +### `ApifyDatasetLoader` + +Loads an existing Apify dataset by ID and maps items to `Document` objects via a user-supplied function. Useful when you have a dataset from a previous run and want to reshape it for downstream LangChain steps. + +```python +import os +from langchain_apify import ApifyDatasetLoader +from langchain_core.documents import Document + +os.environ["APIFY_TOKEN"] = "YOUR_APIFY_TOKEN" loader = ApifyDatasetLoader( dataset_id="your-dataset-id", - dataset_mapping_function=lambda dataset_item: Document( - page_content=dataset_item["text"], - metadata={"source": dataset_item["url"]} + dataset_mapping_function=lambda item: Document( + page_content=item["text"], + metadata={"source": item["url"]}, ), ) ``` ## Wrappers -`ApifyWrapper` class wraps the Apify API to easily convert Apify datasets into documents. It is useful when you need to run an Apify Actor programmatically and process the results in LangChain. Available methods include: +`ApifyWrapper` is a higher-level facade that runs an Actor (or task) and returns an `ApifyDatasetLoader` over the result dataset. Useful when you want to run an Actor programmatically and process the results in LangChain in a single chain. -- **call_actor**: Runs an Apify Actor and returns an `ApifyDatasetLoader` for the results. -- **acall_actor**: Asynchronous version of `call_actor`. -- **call_actor_task**: Runs a saved Actor task and returns an `ApifyDatasetLoader` for the results. Actor tasks allow you to create and reuse multiple configurations of a single Actor for different use cases. -- **acall_actor_task**: Asynchronous version of `call_actor_task`. +Methods: -For more information, see the [Apify LangChain integration documentation](https://docs.apify.com/platform/integrations/langchain). +- `call_actor` / `acall_actor`: run an Actor and return a loader for the results. +- `call_actor_task` / `acall_actor_task`: run a saved Actor task and return a loader for the results. -Example usage for `call_actor` involves running the [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor, which extracts content from webpages. The wrapper then returns the results as a list of `Document` objects containing the page content and source URL: ```python import os from langchain_apify import ApifyWrapper from langchain_core.documents import Document -os.environ["APIFY_API_TOKEN"] = "YOUR_APIFY_API_TOKEN" +os.environ["APIFY_TOKEN"] = "YOUR_APIFY_TOKEN" apify = ApifyWrapper() @@ -138,16 +243,18 @@ loader = apify.call_actor( run_input={ "startUrls": [{"url": "https://python.langchain.com/docs/get_started/introduction"}], "maxCrawlPages": 10, - "crawlerType": "cheerio" + "crawlerType": "cheerio", }, dataset_mapping_function=lambda item: Document( page_content=item["text"] or "", - metadata={"source": item["url"]} + metadata={"source": item["url"]}, ), ) documents = loader.load() ``` +For more information, see the [Apify LangChain integration documentation](https://docs.apify.com/platform/integrations/langchain). + ## Note for Apify Actor developers **If you are building an Apify Actor that will run on the Apify platform**, you should **NOT** use this package for dataset loading. Instead: @@ -191,3 +298,8 @@ This package is designed for: It is **NOT** designed for: - Code running inside an Apify Actor (use Actor SDK instead) + +## Contributing + +For local setup (Poetry install, running tests and linting), see [DEVELOPMENT.md](DEVELOPMENT.md). +For PR scope, commit message conventions, and review expectations, see [CONTRIBUTING.md](CONTRIBUTING.md). diff --git a/docs/examples/tools_example.py b/docs/examples/tools_example.py index 08815a7..fa38590 100644 --- a/docs/examples/tools_example.py +++ b/docs/examples/tools_example.py @@ -7,9 +7,9 @@ from langchain_apify import ApifyActorsTool os.environ['OPENAI_API_KEY'] = 'YOUR_OPENAI_API_KEY' -os.environ['APIFY_API_TOKEN'] = 'YOUR_APIFY_API_TOKEN' +os.environ['APIFY_TOKEN'] = 'YOUR_APIFY_TOKEN' -model = ChatOpenAI(model='gpt-4o-mini') +model = ChatOpenAI(model='gpt-5.4-mini') tool = ApifyActorsTool(actor_id='apify/rag-web-browser') diff --git a/langchain_apify/__init__.py b/langchain_apify/__init__.py index 66142be..00c0a13 100644 --- a/langchain_apify/__init__.py +++ b/langchain_apify/__init__.py @@ -1,19 +1,79 @@ +from __future__ import annotations + from importlib import metadata +from typing import TYPE_CHECKING -from langchain_apify.document_loaders import ApifyDatasetLoader -from langchain_apify.tools import ApifyActorsTool +from langchain_apify.document_loaders import ApifyCrawlLoader, ApifyDatasetLoader +from langchain_apify.retrievers import ApifySearchRetriever +from langchain_apify.tools import ( + APIFY_CORE_TOOLS, + APIFY_SEARCH_TOOLS, + APIFY_SOCIAL_TOOLS, + ApifyActorsTool, + ApifyEcommerceScraperTool, + ApifyFacebookPostsScraperTool, + ApifyGetDatasetItemsTool, + ApifyGoogleMapsTool, + ApifyGoogleSearchTool, + ApifyInstagramScraperTool, + ApifyLinkedInProfileDetailTool, + ApifyLinkedInProfilePostsTool, + ApifyLinkedInProfileSearchTool, + ApifyRAGWebBrowserTool, + ApifyRunActorAndGetDatasetTool, + ApifyRunActorTool, + ApifyRunTaskAndGetDatasetTool, + ApifyRunTaskTool, + ApifyScrapeUrlTool, + ApifyTikTokScraperTool, + ApifyTwitterScraperTool, + ApifyWebCrawlerTool, + ApifyYouTubeScraperTool, +) from langchain_apify.wrappers import ApifyWrapper +if TYPE_CHECKING: + from langchain_core.tools import BaseTool + try: - __version__ = metadata.version(__package__) + __version__ = metadata.version(__package__) if __package__ is not None else '' except metadata.PackageNotFoundError: - # Case where package metadata is not available. __version__ = '' del metadata # optional, avoids polluting the results of dir(__package__) __all__ = [ + # Existing components (backward-compatible) 'ApifyActorsTool', + 'ApifyCrawlLoader', 'ApifyDatasetLoader', + 'ApifySearchRetriever', 'ApifyWrapper', + # Core generic tools + 'ApifyGetDatasetItemsTool', + 'ApifyRunActorAndGetDatasetTool', + 'ApifyRunActorTool', + 'ApifyRunTaskAndGetDatasetTool', + 'ApifyRunTaskTool', + 'ApifyScrapeUrlTool', + # Social media Actor tools + 'ApifyFacebookPostsScraperTool', + 'ApifyInstagramScraperTool', + 'ApifyLinkedInProfileDetailTool', + 'ApifyLinkedInProfilePostsTool', + 'ApifyLinkedInProfileSearchTool', + 'ApifyTikTokScraperTool', + 'ApifyTwitterScraperTool', + # Search & crawling tools + 'ApifyGoogleSearchTool', + 'ApifyWebCrawlerTool', + 'ApifyRAGWebBrowserTool', + 'ApifyGoogleMapsTool', + 'ApifyYouTubeScraperTool', + 'ApifyEcommerceScraperTool', + # Tool group lists + 'APIFY_CORE_TOOLS', + 'APIFY_SOCIAL_TOOLS', + 'APIFY_SEARCH_TOOLS', + # Meta '__version__', ] diff --git a/langchain_apify/_client.py b/langchain_apify/_client.py new file mode 100644 index 0000000..49b864c --- /dev/null +++ b/langchain_apify/_client.py @@ -0,0 +1,862 @@ +from __future__ import annotations + +import httpx +from apify_client import ApifyClient +from apify_client.errors import ApifyClientError +from pydantic import SecretStr + +from langchain_apify._constants import ( + _DEFAULT_CRAWLER_TYPE, + _DEFAULT_DATASET_ITEMS_LIMIT, + _DEFAULT_ECOMMERCE_MAX_RESULTS, + _DEFAULT_GOOGLE_MAPS_MAX_RESULTS, + _DEFAULT_GOOGLE_MAX_RESULTS, + _DEFAULT_LINKEDIN_SEARCH_MAX_RESULTS, + _DEFAULT_MAX_CRAWL_DEPTH, + _DEFAULT_MAX_CRAWL_PAGES, + _DEFAULT_RAG_MAX_RESULTS, + _DEFAULT_RUN_TIMEOUT_SECS, + _DEFAULT_SCRAPE_TIMEOUT_SECS, + _DEFAULT_SOCIAL_RESULTS_LIMIT, + _DEFAULT_SOCIAL_TIMEOUT_SECS, + _DEFAULT_YOUTUBE_MAX_RESULTS, +) +from langchain_apify._error_messages import ( + _ERROR_ACTOR_RUN_FAILED, + _ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET, + _ERROR_SCRAPE_EMPTY, +) +from langchain_apify._types import CrawlerType # noqa: TCH001 # runtime-needed: pydantic-free annotation +from langchain_apify._utils import ( + _create_apify_client, + _extract_content, + _resolve_apify_token, + _resolve_deprecated_token, +) + +# Transport errors caught when calling Apify; anything else propagates. +_TRANSPORT_EXCEPTIONS = (ApifyClientError, httpx.HTTPError) + +# Apify run status indicating a successful finish. +_RUN_STATUS_SUCCEEDED = 'SUCCEEDED' + +# Actor IDs - search & crawling. +_WEBSITE_CONTENT_CRAWLER_ACTOR_ID = 'apify/website-content-crawler' +_GOOGLE_SEARCH_ACTOR_ID = 'apify/google-search-scraper' +_RAG_WEB_BROWSER_ACTOR_ID = 'apify/rag-web-browser' +_GOOGLE_MAPS_ACTOR_ID = 'compass/crawler-google-places' +_YOUTUBE_SCRAPER_ACTOR_ID = 'streamers/youtube-scraper' +_ECOMMERCE_SCRAPER_ACTOR_ID = 'apify/e-commerce-scraping-tool' + +# Actor IDs - social media. +_INSTAGRAM_ACTOR_ID = 'apify/instagram-scraper' +_LINKEDIN_POSTS_ACTOR_ID = 'apimaestro/linkedin-profile-posts' +_LINKEDIN_SEARCH_ACTOR_ID = 'harvestapi/linkedin-profile-search' +_LINKEDIN_DETAIL_ACTOR_ID = 'apimaestro/linkedin-profile-detail' +_TWITTER_ACTOR_ID = 'apidojo/twitter-scraper-lite' +_TIKTOK_ACTOR_ID = 'clockworks/tiktok-scraper' +_FACEBOOK_ACTOR_ID = 'apify/facebook-posts-scraper' + +# Accepted parameter values validated client-side before a run. +_YOUTUBE_SEARCH_TYPES = ('search', 'video', 'channel') +_ECOMMERCE_URL_TYPES = ('product', 'category') + +# Instagram search_type -> Actor resultsType mapping. +_INSTAGRAM_RESULTS_TYPE_MAP = { + 'user': 'posts', + 'hashtag': 'posts', + 'post': 'posts', + 'comments': 'comments', +} + + +class ApifyToolsClient: + """Internal helper that wraps ``ApifyClient`` for the tools layer. + + One convenience method per tool operation. All methods are synchronous and + block until the Actor run finishes. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable (or ``APIFY_API_TOKEN`` for backwards + compatibility) when *None*. + + Raises: + ValueError: If no token is provided and the env var is not set. + """ + + def __init__( + self, + apify_token: SecretStr | str | None = None, + *, + apify_api_token: SecretStr | str | None = None, + ) -> None: + apify_token = _resolve_deprecated_token(apify_token, apify_api_token) + + if isinstance(apify_token, SecretStr): + _token: str | None = apify_token.get_secret_value() + else: + _token = apify_token or _resolve_apify_token() + + if not _token: + msg = _ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET + raise ValueError(msg) + self._client = _create_apify_client(ApifyClient, _token) + + def run_actor( + self, + actor_id: str, + run_input: dict | None = None, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + memory_mbytes: int | None = None, + ) -> dict: + """Start an Actor and block until it finishes. + + Args: + actor_id: Actor ID or name (e.g. ``"apify/python-example"``). + run_input: JSON-serialisable input for the Actor. + timeout_secs: Maximum time to wait for the run to finish. + memory_mbytes: Memory limit for the run, or *None* for Actor default. + + Returns: + Full run-details dict returned by the Apify API. + + Raises: + RuntimeError: If the run does not finish with status ``SUCCEEDED``. + """ + call_kwargs: dict = {'run_input': run_input, 'timeout_secs': timeout_secs, 'logger': None} + if memory_mbytes is not None: + call_kwargs['memory_mbytes'] = memory_mbytes + + try: + run = self._client.actor(actor_id).call(**call_kwargs) + except _TRANSPORT_EXCEPTIONS as exc: + msg = f'Apify Actor call failed for {actor_id}: {exc}' + raise RuntimeError(msg) from exc + if run is None: + msg = f'Actor {actor_id} call returned no run details.' + raise RuntimeError(msg) + self._check_run_status(run) + return run + + def get_dataset_items( + self, dataset_id: str, limit: int = _DEFAULT_DATASET_ITEMS_LIMIT, offset: int = 0 + ) -> list[dict]: + """Fetch items from an existing dataset. + + Args: + dataset_id: Apify dataset ID. + limit: Maximum number of items to return. + offset: Number of items to skip from the start. + + Returns: + List of dataset item dicts (may be empty). + """ + try: + return self._client.dataset(dataset_id).list_items(limit=limit, offset=offset, clean=True).items + except _TRANSPORT_EXCEPTIONS as exc: + msg = f'Apify dataset fetch failed for {dataset_id}: {exc}' + raise RuntimeError(msg) from exc + + def run_actor_and_get_items( + self, + actor_id: str, + run_input: dict | None = None, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + memory_mbytes: int | None = None, + dataset_items_limit: int = _DEFAULT_DATASET_ITEMS_LIMIT, + ) -> tuple[dict, list[dict]]: + """Run an Actor, then fetch items from its default dataset. + + Args: + actor_id: Actor ID or name. + run_input: JSON-serialisable input for the Actor. + timeout_secs: Maximum time to wait for the run to finish. + memory_mbytes: Memory limit for the run, or *None* for Actor default. + dataset_items_limit: Maximum number of dataset items to return. + + Returns: + A ``(run_details, items)`` tuple. + + Raises: + RuntimeError: If the run does not finish with status ``SUCCEEDED``. + """ + run = self.run_actor(actor_id, run_input, timeout_secs, memory_mbytes) + dataset_id = run.get('defaultDatasetId') + if not dataset_id: + msg = f'Actor {actor_id} run succeeded but returned no default dataset ID.' + raise RuntimeError(msg) + items = self.get_dataset_items(dataset_id, dataset_items_limit) + return run, items + + def run_task( + self, + task_id: str, + task_input: dict | None = None, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + memory_mbytes: int | None = None, + ) -> dict: + """Start a saved Actor task and block until it finishes. + + Args: + task_id: Task ID or name (e.g. ``"user/my-task"``). + task_input: JSON-serialisable input that overrides the task's + pre-saved input. + timeout_secs: Maximum time to wait for the run to finish. + memory_mbytes: Memory limit for the run, or *None* for task default. + + Returns: + Full run-details dict returned by the Apify API. + + Raises: + RuntimeError: If the run does not finish with status ``SUCCEEDED``. + """ + call_kwargs: dict = {'task_input': task_input, 'timeout_secs': timeout_secs} + if memory_mbytes is not None: + call_kwargs['memory_mbytes'] = memory_mbytes + + try: + run = self._client.task(task_id).call(**call_kwargs) + except _TRANSPORT_EXCEPTIONS as exc: + msg = f'Apify task call failed for {task_id}: {exc}' + raise RuntimeError(msg) from exc + if run is None: + msg = f'Task {task_id} call returned no run details.' + raise RuntimeError(msg) + self._check_run_status(run) + return run + + def run_task_and_get_items( + self, + task_id: str, + task_input: dict | None = None, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + memory_mbytes: int | None = None, + dataset_items_limit: int = _DEFAULT_DATASET_ITEMS_LIMIT, + ) -> tuple[dict, list[dict]]: + """Run a saved Actor task, then fetch items from its default dataset. + + Args: + task_id: Task ID or name. + task_input: JSON-serialisable input that overrides the task's + pre-saved input. + timeout_secs: Maximum time to wait for the run to finish. + memory_mbytes: Memory limit for the run, or *None* for task default. + dataset_items_limit: Maximum number of dataset items to return. + + Returns: + A ``(run_details, items)`` tuple. + + Raises: + RuntimeError: If the run does not finish with status ``SUCCEEDED``. + """ + run = self.run_task(task_id, task_input, timeout_secs, memory_mbytes) + dataset_id = run.get('defaultDatasetId') + if not dataset_id: + msg = f'Task {task_id} run succeeded but returned no default dataset ID.' + raise RuntimeError(msg) + items = self.get_dataset_items(dataset_id, dataset_items_limit) + return run, items + + def scrape_url_with_metadata( + self, url: str, timeout_secs: int = _DEFAULT_SCRAPE_TIMEOUT_SECS + ) -> tuple[dict, list[dict], str, str]: + """Scrape a single URL and return run/items/content metadata. + + Uses ``apify/website-content-crawler`` with ``maxCrawlPages=1``. + + Args: + url: The URL to scrape. + timeout_secs: Maximum time to wait for the crawl to finish. + + Returns: + Tuple: ``(run, items, content, content_source)`` where + ``content_source`` is ``"markdown"`` or ``"text"``. + + Raises: + RuntimeError: If the Actor run fails or no content is extracted. + """ + run_input = { + 'startUrls': [{'url': url}], + 'maxCrawlPages': 1, + } + run, items = self.run_actor_and_get_items( + _WEBSITE_CONTENT_CRAWLER_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=1, + ) + if not items: + msg = _ERROR_SCRAPE_EMPTY.format(url=url) + raise RuntimeError(msg) + + markdown = items[0].get('markdown') or '' + content = _extract_content(items[0]) + if not content: + msg = _ERROR_SCRAPE_EMPTY.format(url=url) + raise RuntimeError(msg) + return run, items, content, 'markdown' if markdown else 'text' + + def scrape_url(self, url: str, timeout_secs: int = _DEFAULT_SCRAPE_TIMEOUT_SECS) -> str: + """Scrape a single URL and return only the page content. + + Thin public wrapper over :meth:`scrape_url_with_metadata` for callers + that don't need the run/items metadata. + """ + _, _, content, _ = self.scrape_url_with_metadata(url=url, timeout_secs=timeout_secs) + return content + + def instagram_scrape( + self, + search_type: str, + search_query: str, + max_results: int = _DEFAULT_SOCIAL_RESULTS_LIMIT, + only_posts_newer_than: str | None = None, + timeout_secs: int = _DEFAULT_SOCIAL_TIMEOUT_SECS, + ) -> tuple[dict, list[dict]]: + """Scrape Instagram via ``apify/instagram-scraper``. + + Args: + search_type: One of ``"user"``, ``"hashtag"``, ``"post"``, ``"comments"``. + search_query: Username, hashtag, or Instagram URL depending on + ``search_type``. + max_results: Maximum number of items to return. + only_posts_newer_than: Optional date filter. Accepts ``YYYY-MM-DD``, + ISO-8601, or relative (e.g. ``"1 day"``, ``"2 months"``). + timeout_secs: Maximum time to wait for the run to finish. + + Returns: + A ``(run_details, items)`` tuple. + + Raises: + ValueError: If ``search_type`` is not recognised. + RuntimeError: If the Actor run does not succeed. + """ + results_type = _INSTAGRAM_RESULTS_TYPE_MAP.get(search_type) + if results_type is None: + msg = ( + f'Unsupported Instagram search_type {search_type!r}. ' + f'Expected one of: {sorted(_INSTAGRAM_RESULTS_TYPE_MAP)}.' + ) + raise ValueError(msg) + + direct_url = self._build_instagram_url(search_type, search_query) + run_input: dict = { + 'directUrls': [direct_url], + 'resultsType': results_type, + 'resultsLimit': max_results, + } + if only_posts_newer_than is not None: + run_input['onlyPostsNewerThan'] = only_posts_newer_than + return self.run_actor_and_get_items( + _INSTAGRAM_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=max_results, + ) + + def google_search( + self, + query: str, + max_results: int = _DEFAULT_GOOGLE_MAX_RESULTS, + country_code: str | None = None, + language_code: str | None = None, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + ) -> tuple[dict, list[dict]]: + """Run a Google search and return structured results. + + Uses ``apify/google-search-scraper`` with a single query. + + Args: + query: Search query string. + max_results: Maximum number of results to return. + country_code: Two-letter country code for localised results. + language_code: Two-letter language code. + timeout_secs: Maximum time to wait for the run to finish. + + Returns: + A ``(run_details, results)`` tuple. Each result dict has + ``title``, ``url``, and ``description`` keys. + + Raises: + RuntimeError: If the Actor run fails. + """ + # apify/google-search-scraper has no resultsPerPage input; result count + # is driven by maxPagesPerQuery (~10 results/page). Request enough pages + # to cover max_results, then slice the flattened results below. + run_input: dict = { + 'queries': query, + 'maxPagesPerQuery': max(1, (max_results + 9) // 10), + } + if country_code is not None: + run_input['countryCode'] = country_code + if language_code is not None: + run_input['languageCode'] = language_code + + run, items = self.run_actor_and_get_items( + _GOOGLE_SEARCH_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=max_results, + ) + results: list[dict] = [ + { + 'title': organic.get('title', ''), + 'url': organic.get('url', ''), + 'description': organic.get('description', ''), + } + for item in items + for organic in item.get('organicResults', []) + ] + return run, results[:max_results] + + def rag_web_search( + self, + query: str, + max_results: int = _DEFAULT_RAG_MAX_RESULTS, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + ) -> tuple[dict, list[dict]]: + """Search the web and return crawled page content for RAG. + + Uses ``apify/rag-web-browser``. + + Args: + query: Search query string. + max_results: Maximum number of results to return. + timeout_secs: Maximum time to wait for the run to finish. + + Returns: + A ``(run_details, items)`` tuple. Each item dict has at least + ``crawledUrl``, ``text``, and a nested ``metadata`` block (among + other keys returned by the Actor). + + Raises: + RuntimeError: If the Actor run fails. + """ + run_input: dict = { + 'query': query, + 'maxResults': max_results, + } + return self.run_actor_and_get_items( + _RAG_WEB_BROWSER_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=max_results, + ) + + def linkedin_profile_posts( + self, + profile_url: str, + max_results: int = _DEFAULT_SOCIAL_RESULTS_LIMIT, + timeout_secs: int = _DEFAULT_SOCIAL_TIMEOUT_SECS, + ) -> tuple[dict, list[dict]]: + """Scrape LinkedIn profile posts via ``apimaestro/linkedin-profile-posts``. + + Args: + profile_url: LinkedIn profile URL or username. + max_results: Maximum number of posts to return. + timeout_secs: Maximum time to wait for the run to finish. + + Returns: + A ``(run_details, items)`` tuple. + + Raises: + RuntimeError: If the Actor run does not succeed. + """ + run_input: dict = { + 'username': profile_url, + 'total_posts': max_results, + } + return self.run_actor_and_get_items( + _LINKEDIN_POSTS_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=max_results, + ) + + def linkedin_profile_search( + self, + query: str, + max_results: int = _DEFAULT_LINKEDIN_SEARCH_MAX_RESULTS, + timeout_secs: int = _DEFAULT_SOCIAL_TIMEOUT_SECS, + ) -> tuple[dict, list[dict]]: + """Search LinkedIn profiles via ``harvestapi/linkedin-profile-search``. + + Args: + query: Search keywords (e.g., name, title, company). + max_results: Maximum number of profiles to return. + timeout_secs: Maximum time to wait for the run to finish. + + Returns: + A ``(run_details, items)`` tuple. + + Raises: + RuntimeError: If the Actor run does not succeed. + """ + run_input: dict = { + 'searchQuery': query, + 'maxItems': max_results, + } + return self.run_actor_and_get_items( + _LINKEDIN_SEARCH_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=max_results, + ) + + def google_maps_search( + self, + query: str, + max_results: int = _DEFAULT_GOOGLE_MAPS_MAX_RESULTS, + language: str | None = None, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + ) -> tuple[dict, list[dict]]: + """Search Google Maps places, reviews, and business details. + + Uses ``compass/crawler-google-places``. + + Args: + query: Search query string (e.g. ``"coffee shops in Berlin"``). + max_results: Maximum number of places to return. + language: Optional ISO language code for results (e.g. ``"en"``). + timeout_secs: Maximum time to wait for the run to finish. + + Returns: + A ``(run_details, items)`` tuple where each item is a place dict. + + Raises: + RuntimeError: If the Actor run fails. + """ + run_input: dict = { + 'searchStringsArray': [query], + 'maxCrawledPlacesPerSearch': max_results, + } + if language is not None: + run_input['language'] = language + + return self.run_actor_and_get_items( + _GOOGLE_MAPS_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=max_results, + ) + + def linkedin_profile_detail( + self, + profile_url: str, + *, + include_email: bool = False, + timeout_secs: int = _DEFAULT_SOCIAL_TIMEOUT_SECS, + ) -> tuple[dict, list[dict]]: + """Fetch a LinkedIn profile via ``apimaestro/linkedin-profile-detail``. + + Args: + profile_url: LinkedIn profile URL or username. + include_email: If True, attempt to include the profile email when + available. + timeout_secs: Maximum time to wait for the run to finish. + + Returns: + A ``(run_details, items)`` tuple. ``items`` typically contains a + single profile dict. + + Raises: + RuntimeError: If the Actor run does not succeed. + """ + run_input: dict = { + 'username': profile_url, + 'includeEmail': include_email, + } + return self.run_actor_and_get_items( + _LINKEDIN_DETAIL_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=1, + ) + + def twitter_scrape( # noqa: PLR0913 + self, + search_query: str, + search_mode: str = 'search', + max_results: int = _DEFAULT_SOCIAL_RESULTS_LIMIT, + start: str | None = None, + end: str | None = None, + sort: str | None = None, + timeout_secs: int = _DEFAULT_SOCIAL_TIMEOUT_SECS, + ) -> tuple[dict, list[dict]]: + """Scrape Twitter/X via ``apidojo/twitter-scraper-lite``. + + Args: + search_query: Search term, username, or tweet URL. + search_mode: One of ``"search"``, ``"user"``, ``"replies"``. + max_results: Maximum number of tweets to return. + start: Optional ISO-8601 start date; only return tweets newer + than this date. + end: Optional ISO-8601 end date; only return tweets older than + this date. + sort: Optional sort order. One of ``"Latest"`` or ``"Top"``. + timeout_secs: Maximum time to wait for the run to finish. + + Returns: + A ``(run_details, items)`` tuple. + + Raises: + ValueError: If ``search_mode`` is not recognised. + RuntimeError: If the Actor run does not succeed. + """ + run_input: dict = {'maxItems': max_results} + if search_mode == 'search': + run_input['searchTerms'] = [search_query] + elif search_mode == 'user': + run_input['twitterHandles'] = [search_query.removeprefix('@')] + elif search_mode == 'replies': + run_input['startUrls'] = [search_query] + else: + msg = f"Unsupported Twitter search_mode {search_mode!r}. Expected one of: ['search', 'user', 'replies']." + raise ValueError(msg) + if start is not None: + run_input['start'] = start + if end is not None: + run_input['end'] = end + if sort is not None: + run_input['sort'] = sort + return self.run_actor_and_get_items( + _TWITTER_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=max_results, + ) + + def tiktok_scrape( + self, + search_query: str, + search_type: str = 'search', + max_results: int = _DEFAULT_SOCIAL_RESULTS_LIMIT, + timeout_secs: int = _DEFAULT_SOCIAL_TIMEOUT_SECS, + ) -> tuple[dict, list[dict]]: + """Scrape TikTok via ``clockworks/tiktok-scraper``. + + Args: + search_query: Username, hashtag, search keyword, or TikTok post URL. + search_type: One of ``"search"``, ``"user"``, ``"hashtag"``, ``"post"``. + max_results: Maximum number of items to return. + timeout_secs: Maximum time to wait for the run to finish. + + Returns: + A ``(run_details, items)`` tuple. + + Raises: + ValueError: If ``search_type`` is not recognised. + RuntimeError: If the Actor run does not succeed. + """ + run_input: dict = {'resultsPerPage': max_results} + if search_type == 'search': + run_input['searchQueries'] = [search_query] + elif search_type == 'user': + run_input['profiles'] = [search_query.removeprefix('@')] + elif search_type == 'hashtag': + run_input['hashtags'] = [search_query.removeprefix('#')] + elif search_type == 'post': + run_input['postURLs'] = [search_query] + else: + msg = ( + f'Unsupported TikTok search_type {search_type!r}. ' + "Expected one of: ['search', 'user', 'hashtag', 'post']." + ) + raise ValueError(msg) + return self.run_actor_and_get_items( + _TIKTOK_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=max_results, + ) + + def facebook_posts_scrape( + self, + page_url: str, + max_results: int = _DEFAULT_SOCIAL_RESULTS_LIMIT, + only_posts_newer_than: str | None = None, + only_posts_older_than: str | None = None, + timeout_secs: int = _DEFAULT_SOCIAL_TIMEOUT_SECS, + ) -> tuple[dict, list[dict]]: + """Scrape Facebook page posts via ``apify/facebook-posts-scraper``. + + Args: + page_url: Facebook page URL. + max_results: Maximum number of posts to return. + only_posts_newer_than: Optional date filter. Accepts ``YYYY-MM-DD``, + ISO-8601, or relative (e.g. ``"1 day"``, ``"2 months"``). + only_posts_older_than: Optional date filter. Accepts ``YYYY-MM-DD``, + ISO-8601, or relative (e.g. ``"1 day"``, ``"2 months"``). + timeout_secs: Maximum time to wait for the run to finish. + + Returns: + A ``(run_details, items)`` tuple. + + Raises: + RuntimeError: If the Actor run does not succeed. + """ + run_input: dict = { + 'startUrls': [{'url': page_url}], + 'resultsLimit': max_results, + } + if only_posts_newer_than is not None: + run_input['onlyPostsNewerThan'] = only_posts_newer_than + if only_posts_older_than is not None: + run_input['onlyPostsOlderThan'] = only_posts_older_than + return self.run_actor_and_get_items( + _FACEBOOK_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=max_results, + ) + + @staticmethod + def _build_instagram_url(search_type: str, search_query: str) -> str: + """Build an Instagram URL from a username/hashtag/URL based on search type.""" + if search_query.startswith(('http://', 'https://')): + return search_query + if search_type == 'hashtag': + tag = search_query.removeprefix('#') + return f'https://www.instagram.com/explore/tags/{tag}/' + if search_type == 'user': + handle = search_query.removeprefix('@') + return f'https://www.instagram.com/{handle}/' + # post/comments expect a URL; if a bare ID is given, build a /p/ URL + return f'https://www.instagram.com/p/{search_query}/' + + def youtube_scrape( + self, + search_query: str, + search_type: str = 'search', + max_results: int = _DEFAULT_YOUTUBE_MAX_RESULTS, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + ) -> tuple[dict, list[dict]]: + """Scrape YouTube videos, channels, or search results. + + Uses ``streamers/youtube-scraper``. + + Args: + search_query: Keyword for ``search`` mode, or a video/channel URL + for ``video``/``channel`` modes. + search_type: One of ``"search"``, ``"video"``, ``"channel"``. + max_results: Maximum number of items to return. + timeout_secs: Maximum time to wait for the run to finish. + + Returns: + A ``(run_details, items)`` tuple. + + Raises: + ValueError: If ``search_type`` is not a supported value. + RuntimeError: If the Actor run fails. + """ + if search_type not in _YOUTUBE_SEARCH_TYPES: + msg = f'Invalid search_type {search_type!r}; expected one of {_YOUTUBE_SEARCH_TYPES}.' + raise ValueError(msg) + + run_input: dict = {'maxResults': max_results} + if search_type == 'search': + run_input['searchQueries'] = [search_query] + else: + run_input['startUrls'] = [{'url': search_query}] + + return self.run_actor_and_get_items( + _YOUTUBE_SCRAPER_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=max_results, + ) + + def ecommerce_scrape( + self, + url: str, + url_type: str = 'product', + max_results: int = _DEFAULT_ECOMMERCE_MAX_RESULTS, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + ) -> tuple[dict, list[dict]]: + """Extract product data from an e-commerce URL. + + Uses ``apify/e-commerce-scraping-tool``. ``url_type`` selects which + Actor input field the URL is sent as: ``"product"`` -> ``detailsUrls`` + (a single product-detail page), ``"category"`` -> ``listingUrls`` + (a category / listing page that the Actor will expand into product + results). + + Args: + url: Product-detail or category / listing URL to scrape. + url_type: One of ``"product"`` or ``"category"``. + max_results: Maximum number of products to return. + timeout_secs: Maximum time to wait for the run to finish. + + Returns: + A ``(run_details, items)`` tuple. + + Raises: + ValueError: If ``url_type`` is not a supported value. + RuntimeError: If the Actor run fails. + """ + if url_type not in _ECOMMERCE_URL_TYPES: + msg = f'Invalid url_type {url_type!r}; expected one of {_ECOMMERCE_URL_TYPES}.' + raise ValueError(msg) + + input_key = 'detailsUrls' if url_type == 'product' else 'listingUrls' + run_input: dict = { + input_key: [{'url': url}], + 'maxProductResults': max_results, + } + return self.run_actor_and_get_items( + _ECOMMERCE_SCRAPER_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=max_results, + ) + + def crawl_website( + self, + url: str, + max_crawl_pages: int = _DEFAULT_MAX_CRAWL_PAGES, + max_crawl_depth: int = _DEFAULT_MAX_CRAWL_DEPTH, + crawler_type: CrawlerType = _DEFAULT_CRAWLER_TYPE, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + ) -> tuple[dict, list[dict]]: + """Crawl a website and return page content. + + Uses ``apify/website-content-crawler``. + + Args: + url: Seed URL to start crawling from. + max_crawl_pages: Maximum number of pages to crawl. + max_crawl_depth: Maximum link-follow depth from the seed URL. + crawler_type: Crawler engine (e.g. ``"cheerio"``, ``"playwright:firefox"``). + timeout_secs: Maximum time to wait for the run to finish. + + Returns: + A ``(run_details, items)`` tuple. Each page dict has at least + ``url``, ``title``, and ``markdown`` (or ``text``) keys. + + Raises: + RuntimeError: If the Actor run fails. + """ + run_input: dict = { + 'startUrls': [{'url': url}], + 'maxCrawlPages': max_crawl_pages, + 'maxCrawlDepth': max_crawl_depth, + 'crawlerType': crawler_type, + } + return self.run_actor_and_get_items( + _WEBSITE_CONTENT_CRAWLER_ACTOR_ID, + run_input=run_input, + timeout_secs=timeout_secs, + dataset_items_limit=max_crawl_pages, + ) + + @staticmethod + def _check_run_status(run: dict) -> None: + """Raise if the run did not succeed.""" + status = run.get('status') + if status != _RUN_STATUS_SUCCEEDED: + run_id = run.get('id', 'unknown') + msg = _ERROR_ACTOR_RUN_FAILED.format(run_id=run_id, status=status) + if status_message := run.get('statusMessage'): + msg = f'{msg} {status_message}' + raise RuntimeError(msg) diff --git a/langchain_apify/_constants.py b/langchain_apify/_constants.py new file mode 100644 index 0000000..9615854 --- /dev/null +++ b/langchain_apify/_constants.py @@ -0,0 +1,48 @@ +"""Shared default values for Apify tool and client parameters. + +Centralized so every tunable default lives in one place and is imported by the +client, tools, retriever, and loaders rather than reaching into one another. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from langchain_apify._types import CrawlerType + +# Default timeouts (seconds), by operation class. +_DEFAULT_RUN_TIMEOUT_SECS = 300 +_DEFAULT_SCRAPE_TIMEOUT_SECS = 120 +_DEFAULT_SOCIAL_TIMEOUT_SECS = 600 + +# Default result / page limits, by operation. +_DEFAULT_DATASET_ITEMS_LIMIT = 100 +_DEFAULT_MAX_CRAWL_PAGES = 10 +_DEFAULT_MAX_CRAWL_DEPTH = 1 +_DEFAULT_GOOGLE_MAX_RESULTS = 10 +_DEFAULT_RAG_MAX_RESULTS = 5 +_DEFAULT_SOCIAL_RESULTS_LIMIT = 20 +_DEFAULT_LINKEDIN_SEARCH_MAX_RESULTS = 10 +_DEFAULT_GOOGLE_MAPS_MAX_RESULTS = 10 +_DEFAULT_YOUTUBE_MAX_RESULTS = 10 +_DEFAULT_ECOMMERCE_MAX_RESULTS = 20 + +# Upper-bound clamp ceilings applied by _ApifyGenericTool to LLM-supplied +# values. The Pydantic Field defaults on the base class reference these so +# the schema descriptions can interpolate the same numbers, keeping the +# clamp documentation truthful even if the cap moves. +_MAX_TIMEOUT_SECS_CAP = 600 +_MAX_MEMORY_MBYTES_CAP = 32768 +_MAX_ITEMS_CAP = 1000 +_MAX_CRAWL_DEPTH_CAP = 5 + +# The apify/rag-web-browser Actor rejects maxResults > 100 at runtime. This +# limit is enforced by the Actor, not declared in its input schema (which only +# carries a default), so it cannot be derived by schema introspection and must +# be tracked here by hand. Applied by ApifyRAGWebBrowserTool and +# ApifySearchRetriever, both of which wrap that Actor. +_RAG_MAX_RESULTS_CAP = 100 + +# Default crawler engine for the website-content-crawler Actor. +_DEFAULT_CRAWLER_TYPE: CrawlerType = 'cheerio' diff --git a/langchain_apify/_error_messages.py b/langchain_apify/_error_messages.py new file mode 100644 index 0000000..95c1aa0 --- /dev/null +++ b/langchain_apify/_error_messages.py @@ -0,0 +1,17 @@ +_ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET = ( + 'APIFY_TOKEN environment variable is not set.' + ' Please set it to your Apify API token by using `os.environ["APIFY_TOKEN"] = "YOUR_APIFY_TOKEN"`' + ' in your code or pass it as environment variable.' + ' To pass it as environment variable, you can use the following command:' + ' `APIFY_TOKEN="YOUR_APIFY_TOKEN" python your_script.py`' + ' (`APIFY_API_TOKEN` is also accepted for backwards compatibility).' +) + +_ERROR_ACTOR_RUN_FAILED = 'Actor run {run_id} ended with status {status}.' + +_ERROR_SCRAPE_EMPTY = 'No content extracted from {url}.' + +_NOTICE_TWITTER_DEMO = ( + 'The Twitter/X Actor returned demo placeholder data instead of real tweets.' + ' This happens on the Apify free plan; a paid Apify plan is required to scrape real tweets.' +) diff --git a/langchain_apify/_types.py b/langchain_apify/_types.py new file mode 100644 index 0000000..d54e9d3 --- /dev/null +++ b/langchain_apify/_types.py @@ -0,0 +1,17 @@ +from __future__ import annotations + +from typing import Literal + +# Shared Literal aliases for tool/client parameters, centralized so every +# accepted-value set is declared once and reused across schemas and signatures. + +# Search & crawling +CrawlerType = Literal['cheerio', 'playwright:adaptive', 'playwright:firefox'] +YouTubeSearchType = Literal['search', 'video', 'channel'] +EcommerceUrlType = Literal['product', 'category'] + +# Social media +InstagramSearchType = Literal['user', 'hashtag', 'post', 'comments'] +TwitterSearchMode = Literal['search', 'user', 'replies'] +TwitterSort = Literal['Latest', 'Top'] +TikTokSearchType = Literal['search', 'user', 'hashtag', 'post'] diff --git a/langchain_apify/_utils.py b/langchain_apify/_utils.py new file mode 100644 index 0000000..3debed4 --- /dev/null +++ b/langchain_apify/_utils.py @@ -0,0 +1,243 @@ +from __future__ import annotations + +import os +import string +import warnings +from typing import TypeVar + +import requests +from apify_client import ApifyClientAsync +from apify_client.client import ApifyClient +from pydantic import SecretStr + +_MAX_DESCRIPTION_LEN: int = 350 + +_DEPRECATED_APIFY_API_TOKEN_MSG = "The 'apify_api_token' parameter is deprecated, use 'apify_token' instead." +_DEPRECATED_APIFY_API_TOKEN_ENV_MSG = ( + "The 'APIFY_API_TOKEN' environment variable is deprecated, use 'APIFY_TOKEN' instead." +) +_BOTH_TOKENS_MSG = ( + "Both 'apify_token' and 'apify_api_token' were specified; using 'apify_token' " + "and ignoring the deprecated 'apify_api_token'." +) +_REQUESTS_TIMEOUT_SECS: float = 10.0 +_APIFY_API_ENDPOINT_GET_DEFAULT_BUILD: str = 'https://api.apify.com/v2/acts/{actor_id}/builds/default' + + +def _resolve_deprecated_token( + apify_token: SecretStr | str | None, + apify_api_token: SecretStr | str | None, +) -> SecretStr | str | None: + """Apply the ``apify_api_token`` → ``apify_token`` deprecation policy. + + For classes with an explicit ``__init__`` (``ApifyToolsClient``, + ``ApifyDatasetLoader``, ``ApifyCrawlLoader``, ``ApifyActorsTool``). Emits a + ``DeprecationWarning`` when the legacy ``apify_api_token`` is supplied and + prefers ``apify_token`` when both are given. Returns the token to use. + """ + if apify_api_token is None: + return apify_token + if apify_token is not None: + warnings.warn(_BOTH_TOKENS_MSG, DeprecationWarning, stacklevel=3) + return apify_token + warnings.warn(_DEPRECATED_APIFY_API_TOKEN_MSG, DeprecationWarning, stacklevel=3) + return apify_api_token + + +def _resolve_deprecated_token_values(values: dict) -> dict: + """Same deprecation policy as :func:`_resolve_deprecated_token`, for dicts. + + For pydantic ``model_validator(mode='before')`` hooks (``_ApifyGenericTool``, + ``ApifySearchRetriever``), which receive the raw input ``values`` dict. + """ + if isinstance(values, dict) and 'apify_api_token' in values: + if 'apify_token' in values: + warnings.warn(_BOTH_TOKENS_MSG, DeprecationWarning, stacklevel=3) + del values['apify_api_token'] + else: + warnings.warn(_DEPRECATED_APIFY_API_TOKEN_MSG, DeprecationWarning, stacklevel=3) + values['apify_token'] = values.pop('apify_api_token') + return values + + +def _resolve_apify_token() -> str | None: + """Resolve the Apify API token from environment variables. + + ``APIFY_TOKEN`` (SDK-standard) takes precedence; ``APIFY_API_TOKEN`` is + kept as a deprecated fallback for backwards compatibility with this + package's historical naming, and emits a ``DeprecationWarning`` when used. + """ + if token := os.getenv('APIFY_TOKEN'): + return token + if token := os.getenv('APIFY_API_TOKEN'): + warnings.warn(_DEPRECATED_APIFY_API_TOKEN_ENV_MSG, DeprecationWarning, stacklevel=2) + return token + return None + + +def _apify_token_secret_factory() -> SecretStr | None: + """Pydantic ``default_factory`` returning the resolved token as ``SecretStr``.""" + token = _resolve_apify_token() + return SecretStr(token) if token else None + + +def _extract_content(item: dict) -> str: + """Return an Actor item's content, preferring markdown over plain text. + + Both ``apify/website-content-crawler`` and ``apify/rag-web-browser`` emit + ``markdown`` (the richer field) and ``text`` (the plain-text fallback). The + trailing ``or ''`` guarantees a string even when a key is present but null. + """ + return item.get('markdown') or item.get('text') or '' + + +def _item_metadata(item: dict) -> dict: + """Return an item's ``metadata`` block, or ``{}`` if missing/non-dict. + + Some Actors surface a ``null`` (or otherwise non-dict) ``metadata`` value, + so a plain ``item.get('metadata', {})`` would raise ``AttributeError`` on + the chained ``.get(...)``. + """ + meta = item.get('metadata') + return meta if isinstance(meta, dict) else {} + + +def _safe_title(item: dict) -> str: + """Return an Actor item's title from its nested ``metadata`` object. + + Both ``apify/website-content-crawler`` and ``apify/rag-web-browser`` nest + the page title under ``metadata.title``. The guard tolerates Actor + responses where ``metadata`` is missing or not a dict. + """ + return _item_metadata(item).get('title', '') + + +def _extract_source(item: dict) -> str: + """Return an Actor item's source URL via one canonical fallback order. + + ``apify/rag-web-browser`` items expose the page URL in several places. To + keep every consumer (RAG tool, retriever, loaders) in agreement, the order + is fixed here: nested ``metadata.url`` first, then ``crawledUrl``, then the + top-level ``url``. + """ + return _item_metadata(item).get('url') or item.get('crawledUrl') or item.get('url', '') + + +def _prune_actor_input_schema( + input_schema: dict, + max_description_len: int = _MAX_DESCRIPTION_LEN, +) -> tuple[dict, list[str]]: + """Get the input schema from the Actor build. + + Trim descriptions to ``_MAX_DESCRIPTION_LEN`` characters. + + Args: + input_schema (dict): The input schema from the Actor build. + max_description_len (int): The maximum length of the description. + + Returns: + tuple[dict, list[str]]: A tuple containing the pruned properties + and required fields. + """ + properties = input_schema.get('properties', {}) + required = input_schema.get('required', []) + + properties_out: dict = {} + for item, meta in properties.items(): + properties_out[item] = {} + if desc := meta.get('description'): + properties_out[item]['description'] = ( + desc[:max_description_len] + '...' if len(desc) > max_description_len else desc + ) + for key_name in ('type', 'default', 'prefill', 'enum'): + if (value := meta.get(key_name)) is not None: + properties_out[item][key_name] = value + + return properties_out, required + + +T = TypeVar('T', ApifyClient, ApifyClientAsync) + + +def _create_apify_client(client_cls: type[T], token: str) -> T: + """Create an Apify client instance with a custom user-agent. + + Args: + client_cls (ApifyClient | ApifyClientAsync): ApifyClient or ApifyClientAsync class. + token (str): API token. + + Returns: + T: ApifyClient or ApifyClientAsync instance. + + Raises: + ValueError: If the API token is not provided. + """ + if not token: + msg = 'API token is required to create an Apify client.' + raise ValueError(msg) + client = client_cls(token) + + # Check for new attribute names first (without 'x'), then fall back to old names (with 'x') + if isinstance(client, ApifyClientAsync): + http_client_attr = ( + 'httpx_async_client' if hasattr(client.http_client, 'httpx_async_client') else 'http_async_client' + ) + else: + http_client_attr = 'httpx_client' if hasattr(client.http_client, 'httpx_client') else 'http_client' + + if http_client := getattr(client.http_client, http_client_attr, None): + http_client.headers['user-agent'] += '; Origin/langchain' + return client + + +def _actor_id_to_tool_name(actor_id: str) -> str: + """Turn actor_id into a valid tool name. + + Tool name must only contain letters, numbers, underscores, dashes, + and cannot contain spaces. + + Args: + actor_id (str): Actor ID from Apify store. + + Returns: + str: A valid tool name. + """ + valid_chars = string.ascii_letters + string.digits + '_-' + return 'apify_actor_' + ''.join(char if char in valid_chars else '_' for char in actor_id) + + +def _get_actor_latest_build(apify_client: ApifyClient, actor_id: str) -> dict: + """Get the latest build of an Actor from the default build tag. + + Args: + apify_client (ApifyClient): An instance of the ApifyClient class. + actor_id (str): Actor name from Apify store to run. + + Returns: + dict: The latest build of the Actor. + + Raises: + ValueError: If the Actor is not found or the build data is not found. + TypeError: If the build is not a dictionary. + """ + if not (actor := apify_client.actor(actor_id).get()): + msg = f'Actor {actor_id} not found.' + raise ValueError(msg) + + if not (actor_obj_id := actor.get('id')): + msg = f'Failed to get the Actor object ID for {actor_id}.' + raise ValueError(msg) + + url = _APIFY_API_ENDPOINT_GET_DEFAULT_BUILD.format(actor_id=actor_obj_id) + response = requests.request('GET', url, timeout=_REQUESTS_TIMEOUT_SECS) + + build = response.json() + if not isinstance(build, dict): + msg = f'Failed to get the latest build of the Actor {actor_id}.' + raise TypeError(msg) + + if (data := build.get('data')) is None: + msg = f'Failed to get the latest build data of the Actor {actor_id}.' + raise ValueError(msg) + + return data diff --git a/langchain_apify/const.py b/langchain_apify/const.py deleted file mode 100644 index 87e0d0e..0000000 --- a/langchain_apify/const.py +++ /dev/null @@ -1,2 +0,0 @@ -REQUESTS_TIMEOUT_SECS: float = 10.0 -MAX_DESCRIPTION_LEN: int = 350 diff --git a/langchain_apify/document_loaders.py b/langchain_apify/document_loaders.py index 49befb6..49c232c 100644 --- a/langchain_apify/document_loaders.py +++ b/langchain_apify/document_loaders.py @@ -1,27 +1,42 @@ from __future__ import annotations -import os from collections.abc import Callable from typing import TYPE_CHECKING, Any from apify_client import ApifyClient from langchain_core.document_loaders.base import BaseLoader -from langchain_core.documents import Document # noqa: TCH002 -from langchain_core.utils import get_from_dict_or_env -from pydantic import BaseModel, ConfigDict, model_validator - -from langchain_apify.utils import create_apify_client +from langchain_core.documents import Document +from pydantic import BaseModel, ConfigDict, Field, PrivateAttr, SecretStr, model_validator + +from langchain_apify._client import ApifyToolsClient +from langchain_apify._constants import ( + _DEFAULT_CRAWLER_TYPE, + _DEFAULT_MAX_CRAWL_DEPTH, + _DEFAULT_MAX_CRAWL_PAGES, + _DEFAULT_RUN_TIMEOUT_SECS, +) +from langchain_apify._error_messages import _ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET +from langchain_apify._utils import ( + _apify_token_secret_factory, + _create_apify_client, + _extract_content, + _resolve_deprecated_token, + _safe_title, +) if TYPE_CHECKING: from collections.abc import Iterator + from langchain_apify._types import CrawlerType + class ApifyDatasetLoader(BaseLoader, BaseModel): """Load datasets from Apify web scraping, crawling, and data extraction platform. - To use, you should have the environment variable `APIFY_API_TOKEN` set - with your API key, or pass `apify_api_token` - as a named parameter to the constructor. + To use, you should have the environment variable ``APIFY_TOKEN`` set + with your API key, or pass ``apify_token`` as a named parameter to the + constructor. ``APIFY_API_TOKEN`` is still accepted for backwards + compatibility. For details, see https://docs.apify.com/platform/integrations/langchain @@ -40,10 +55,15 @@ class ApifyDatasetLoader(BaseLoader, BaseModel): documents = loader.load() """ - model_config = ConfigDict(arbitrary_types_allowed=True) + model_config = ConfigDict(arbitrary_types_allowed=True, populate_by_name=True) - apify_client: ApifyClient - """An instance of the ApifyClient class from the apify-client Python package.""" + apify_token: SecretStr | None = Field( + default_factory=_apify_token_secret_factory, + description='Apify API token. Falls back to the APIFY_TOKEN environment variable when None.', + exclude=True, + repr=False, + ) + _apify_client: ApifyClient = PrivateAttr() dataset_id: str """The ID of the dataset on the Apify platform.""" dataset_mapping_function: Callable[[dict], Document] @@ -54,7 +74,9 @@ def __init__( self, dataset_id: str, dataset_mapping_function: Callable[[dict], Document], - apify_api_token: str | None = None, + apify_token: str | SecretStr | None = None, + *, + apify_api_token: str | SecretStr | None = None, ) -> None: """Initialize the loader with an Apify dataset ID and a mapping function. @@ -63,34 +85,38 @@ def __init__( dataset_mapping_function (Callable): A function that takes a single dictionary (an Apify dataset item) and converts it to an instance of the Document class. - apify_api_token (str): Apify API token. + apify_token (str | SecretStr): Apify API token. Falls back to the + ``APIFY_TOKEN`` environment variable when *None*. + apify_api_token: Deprecated alias for ``apify_token``. """ - super().__init__( - dataset_id=dataset_id, - dataset_mapping_function=dataset_mapping_function, - apify_api_token=apify_api_token, - ) + apify_token = _resolve_deprecated_token(apify_token, apify_api_token) - @model_validator(mode='before') - @classmethod - def validate_environment(cls, values: dict) -> Any: # noqa: ANN401 - """Validate environment. + init_kwargs: dict[str, Any] = { + 'dataset_id': dataset_id, + 'dataset_mapping_function': dataset_mapping_function, + } + if apify_token is not None: + init_kwargs['apify_token'] = apify_token + super().__init__(**init_kwargs) - Args: - values (dict): The values to validate. + @model_validator(mode='after') + def _init_client(self) -> ApifyDatasetLoader: + """Validate the resolved Apify token and initialise the client. - Returns: - Any: The validated values. - """ - apify_api_token = get_from_dict_or_env(values, 'apify_api_token', 'APIFY_API_TOKEN') - # when running at Apify platform, use APIFY_TOKEN environment variable - apify_api_token = apify_api_token or os.getenv('APIFY_TOKEN', '') - - client = create_apify_client(ApifyClient, apify_api_token) + The token default factory resolves ``APIFY_TOKEN`` first and + ``APIFY_API_TOKEN`` as a legacy fallback. - values['apify_client'] = client + Returns: + ApifyDatasetLoader: The validated loader instance. - return values + Raises: + ValueError: If no token is available from any source. + """ + if self.apify_token is None: + msg = _ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET + raise ValueError(msg) + self._apify_client = _create_apify_client(ApifyClient, self.apify_token.get_secret_value()) + return self def load(self) -> list[Document]: """Load documents. @@ -98,7 +124,7 @@ def load(self) -> list[Document]: Returns: list[Document]: A list of mapped Document objects. """ - dataset_items = self.apify_client.dataset(self.dataset_id).list_items(clean=True).items + dataset_items = self._apify_client.dataset(self.dataset_id).list_items(clean=True).items return list(map(self.dataset_mapping_function, dataset_items)) def lazy_load(self) -> Iterator[Document]: @@ -107,8 +133,95 @@ def lazy_load(self) -> Iterator[Document]: Yields: Document: A mapped Document object. """ - dataset_items = self.apify_client.dataset(self.dataset_id).iterate_items( + dataset_items = self._apify_client.dataset(self.dataset_id).iterate_items( clean=True, ) for item in dataset_items: yield self.dataset_mapping_function(item) + + +class ApifyCrawlLoader(BaseLoader): + """Crawl a website and load pages as LangChain Documents. + + Wraps the ``apify/website-content-crawler`` Actor. Runs a crawl starting + from the seed URL and converts each crawled page into a ``Document`` with + markdown content and metadata (source URL, title, crawl depth). + + Args: + url: Seed URL to start crawling from. + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + apify_api_token: Deprecated alias for ``apify_token``. + max_crawl_pages: Maximum number of pages to crawl. + max_crawl_depth: Maximum link-follow depth from the seed URL. + crawler_type: Crawler engine (e.g. ``"cheerio"``, ``"playwright:firefox"``). + timeout_secs: Maximum time in seconds to wait for the crawl. + + Returns: + Iterator (or list) of ``Document`` objects. ``page_content`` contains + the page markdown; ``metadata`` includes ``source``, ``title``, and + ``crawl_depth``. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyCrawlLoader + + loader = ApifyCrawlLoader( + url="https://docs.apify.com", + max_crawl_pages=5, + ) + documents = loader.load() + """ + + def __init__( # noqa: PLR0913 + self, + url: str, + apify_token: str | SecretStr | None = None, + *, + apify_api_token: str | SecretStr | None = None, + max_crawl_pages: int = _DEFAULT_MAX_CRAWL_PAGES, + max_crawl_depth: int = _DEFAULT_MAX_CRAWL_DEPTH, + crawler_type: CrawlerType = _DEFAULT_CRAWLER_TYPE, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + ) -> None: + apify_token = _resolve_deprecated_token(apify_token, apify_api_token) + + self.url = url + self.max_crawl_pages = max_crawl_pages + self.max_crawl_depth = max_crawl_depth + self.crawler_type: CrawlerType = crawler_type + self.timeout_secs = timeout_secs + self._client = ApifyToolsClient(apify_token=apify_token) + + def lazy_load(self) -> Iterator[Document]: + """Crawl the website and yield Documents. + + Yields: + Document: One document per crawled page. + """ + _, items = self._client.crawl_website( + self.url, + max_crawl_pages=self.max_crawl_pages, + max_crawl_depth=self.max_crawl_depth, + crawler_type=self.crawler_type, + timeout_secs=self.timeout_secs, + ) + for item in items: + # Some Actor responses surface list-typed entries (e.g. nested + # arrays for sitemap-style outputs). Skip anything non-dict. + if not isinstance(item, dict): + continue + page_content = _extract_content(item) + # website-content-crawler nests depth under crawl.depth; there is no + # top-level crawlDepth field. + crawl_meta = item.get('crawl') + metadata: dict[str, Any] = { + 'source': item.get('url', ''), + 'title': _safe_title(item), + 'crawl_depth': crawl_meta.get('depth', 0) if isinstance(crawl_meta, dict) else 0, + } + yield Document(page_content=page_content, metadata=metadata) diff --git a/langchain_apify/error_messages.py b/langchain_apify/error_messages.py deleted file mode 100644 index 87462b8..0000000 --- a/langchain_apify/error_messages.py +++ /dev/null @@ -1,7 +0,0 @@ -ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET = ( - 'APIFY_API_TOKEN environment variable is not set.' - ' Please set it to your Apify API token by using `os.environ["APIFY_API_TOKEN"] = "YOUR_APIFY_API_TOKEN"' - ' in your code or pass it as environment variable.' - ' To pass it as environment variable, you can use the following command:' - ' `APIFY_API_TOKEN="YOUR_APIFY_API_TOKEN" python your_script.py`' -) diff --git a/langchain_apify/retrievers.py b/langchain_apify/retrievers.py new file mode 100644 index 0000000..cc39aa6 --- /dev/null +++ b/langchain_apify/retrievers.py @@ -0,0 +1,140 @@ +"""LangChain retrievers backed by Apify Actors.""" + +from __future__ import annotations + +import asyncio +from typing import TYPE_CHECKING, Any + +from langchain_core.documents import Document +from langchain_core.retrievers import BaseRetriever +from pydantic import Field, PrivateAttr, SecretStr, model_validator + +from langchain_apify._client import ApifyToolsClient +from langchain_apify._constants import ( + _DEFAULT_RAG_MAX_RESULTS, + _DEFAULT_RUN_TIMEOUT_SECS, + _RAG_MAX_RESULTS_CAP, +) +from langchain_apify._error_messages import _ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET +from langchain_apify._utils import ( + _apify_token_secret_factory, + _extract_content, + _extract_source, + _resolve_deprecated_token_values, + _safe_title, +) + +if TYPE_CHECKING: + from langchain_core.callbacks import ( + AsyncCallbackManagerForRetrieverRun, + CallbackManagerForRetrieverRun, + ) + + +class ApifySearchRetriever(BaseRetriever): + """Retrieve documents from the web for RAG using Apify. + + Wraps the ``apify/rag-web-browser`` Actor. Each invocation runs a web + search, crawls the top results, and returns their content as LangChain + ``Document`` objects ready for a RAG pipeline. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + apify_api_token: Deprecated alias for ``apify_token``. + max_results: Maximum number of ``Document`` objects to return per query. + timeout_secs: Maximum time in seconds to wait for the Actor run. + + Returns: + List of ``Document`` objects. ``page_content`` contains the crawled + text; ``metadata`` includes ``source`` (URL) and ``title``. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifySearchRetriever + + retriever = ApifySearchRetriever(max_results=3) + docs = retriever.invoke("What is LangChain?") + """ + + apify_token: SecretStr | None = Field( + default_factory=_apify_token_secret_factory, + description='Apify API token. Falls back to the APIFY_TOKEN environment variable when None.', + exclude=True, + repr=False, + ) + max_results: int = Field(default=_DEFAULT_RAG_MAX_RESULTS, description='Maximum number of documents to return.') + timeout_secs: int = Field(default=_DEFAULT_RUN_TIMEOUT_SECS, description='Maximum Actor run time in seconds.') + + _client: ApifyToolsClient = PrivateAttr() + + @model_validator(mode='before') + @classmethod + def _handle_deprecated_apify_api_token(cls, values: dict) -> dict: + return _resolve_deprecated_token_values(values) + + def model_post_init(self, context: Any) -> None: # noqa: ANN401 + """Construct the underlying ``ApifyToolsClient``. + + Mirrors ``_ApifyGenericTool``: guard against a missing token locally + before constructing the client, so the failure mode is consistent + across tools and the retriever. + """ + if self.apify_token is None: + msg = _ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET + raise ValueError(msg) + self._client = ApifyToolsClient(apify_token=self.apify_token.get_secret_value()) + super().model_post_init(context) + + def _clamped_max_results(self) -> int: + """Clamp ``max_results`` to the rag-web-browser Actor's ceiling. + + The Actor rejects ``maxResults`` above :data:`_RAG_MAX_RESULTS_CAP` at + runtime, so clamp here rather than let the request fail outright. + """ + return max(1, min(self.max_results, _RAG_MAX_RESULTS_CAP)) + + def _get_relevant_documents( + self, + query: str, + *, + run_manager: CallbackManagerForRetrieverRun | None = None, # noqa: ARG002 + ) -> list[Document]: + _, items = self._client.rag_web_search( + query, + max_results=self._clamped_max_results(), + timeout_secs=self.timeout_secs, + ) + return self._items_to_documents(items) + + async def _aget_relevant_documents( + self, + query: str, + *, + run_manager: AsyncCallbackManagerForRetrieverRun | None = None, # noqa: ARG002 + ) -> list[Document]: + # ApifyToolsClient is sync-only. + _, items = await asyncio.to_thread( + self._client.rag_web_search, + query, + max_results=self._clamped_max_results(), + timeout_secs=self.timeout_secs, + ) + return self._items_to_documents(items) + + @staticmethod + def _items_to_documents(items: list[dict]) -> list[Document]: + """Convert Actor dataset items to LangChain Documents.""" + docs: list[Document] = [] + for item in items: + page_content = _extract_content(item) + metadata: dict[str, Any] = { + 'source': _extract_source(item), + 'title': _safe_title(item), + } + docs.append(Document(page_content=page_content, metadata=metadata)) + return docs diff --git a/langchain_apify/tools/__init__.py b/langchain_apify/tools/__init__.py new file mode 100644 index 0000000..c0ff4c6 --- /dev/null +++ b/langchain_apify/tools/__init__.py @@ -0,0 +1,109 @@ +"""Apify tools package. + +Re-exports every public tool class, input schema, helper, and convenience +tool-class list so they remain importable from ``langchain_apify.tools``. +""" + +from __future__ import annotations + +from langchain_apify.tools.actors import ApifyActorsTool +from langchain_apify.tools.core import ( + APIFY_CORE_TOOLS, + ApifyGetDatasetItemsInput, + ApifyGetDatasetItemsTool, + ApifyRunActorAndGetDatasetInput, + ApifyRunActorAndGetDatasetTool, + ApifyRunActorInput, + ApifyRunActorTool, + ApifyRunTaskAndGetDatasetInput, + ApifyRunTaskAndGetDatasetTool, + ApifyRunTaskInput, + ApifyRunTaskTool, + ApifyScrapeUrlInput, + ApifyScrapeUrlTool, +) +from langchain_apify.tools.search import ( + APIFY_SEARCH_TOOLS, + ApifyEcommerceScraperInput, + ApifyEcommerceScraperTool, + ApifyGoogleMapsInput, + ApifyGoogleMapsTool, + ApifyGoogleSearchInput, + ApifyGoogleSearchTool, + ApifyRAGWebBrowserInput, + ApifyRAGWebBrowserTool, + ApifyWebCrawlerInput, + ApifyWebCrawlerTool, + ApifyYouTubeScraperInput, + ApifyYouTubeScraperTool, +) +from langchain_apify.tools.social import ( + APIFY_SOCIAL_TOOLS, + ApifyFacebookPostsScraperInput, + ApifyFacebookPostsScraperTool, + ApifyInstagramScraperInput, + ApifyInstagramScraperTool, + ApifyLinkedInProfileDetailInput, + ApifyLinkedInProfileDetailTool, + ApifyLinkedInProfilePostsInput, + ApifyLinkedInProfilePostsTool, + ApifyLinkedInProfileSearchInput, + ApifyLinkedInProfileSearchTool, + ApifyTikTokScraperInput, + ApifyTikTokScraperTool, + ApifyTwitterScraperInput, + ApifyTwitterScraperTool, + InstagramSearchType, + TikTokSearchType, + TwitterSearchMode, + TwitterSort, +) + +__all__ = [ + 'APIFY_CORE_TOOLS', + 'APIFY_SEARCH_TOOLS', + 'APIFY_SOCIAL_TOOLS', + 'ApifyActorsTool', + 'ApifyEcommerceScraperInput', + 'ApifyEcommerceScraperTool', + 'ApifyFacebookPostsScraperInput', + 'ApifyFacebookPostsScraperTool', + 'ApifyGetDatasetItemsInput', + 'ApifyGetDatasetItemsTool', + 'ApifyGoogleMapsInput', + 'ApifyGoogleMapsTool', + 'ApifyGoogleSearchInput', + 'ApifyGoogleSearchTool', + 'ApifyInstagramScraperInput', + 'ApifyInstagramScraperTool', + 'ApifyLinkedInProfileDetailInput', + 'ApifyLinkedInProfileDetailTool', + 'ApifyLinkedInProfilePostsInput', + 'ApifyLinkedInProfilePostsTool', + 'ApifyLinkedInProfileSearchInput', + 'ApifyLinkedInProfileSearchTool', + 'ApifyRAGWebBrowserInput', + 'ApifyRAGWebBrowserTool', + 'ApifyRunActorAndGetDatasetInput', + 'ApifyRunActorAndGetDatasetTool', + 'ApifyRunActorInput', + 'ApifyRunActorTool', + 'ApifyRunTaskAndGetDatasetInput', + 'ApifyRunTaskAndGetDatasetTool', + 'ApifyRunTaskInput', + 'ApifyRunTaskTool', + 'ApifyScrapeUrlInput', + 'ApifyScrapeUrlTool', + 'ApifyTikTokScraperInput', + 'ApifyTikTokScraperTool', + 'ApifyTwitterScraperInput', + 'ApifyTwitterScraperTool', + 'ApifyWebCrawlerInput', + 'ApifyWebCrawlerTool', + 'ApifyYouTubeScraperInput', + 'ApifyYouTubeScraperTool', + 'InstagramSearchType', + 'TikTokSearchType', + 'TwitterSearchMode', + 'TwitterSort', +] diff --git a/langchain_apify/tools.py b/langchain_apify/tools/actors.py similarity index 66% rename from langchain_apify/tools.py rename to langchain_apify/tools/actors.py index 135314a..378004e 100644 --- a/langchain_apify/tools.py +++ b/langchain_apify/tools/actors.py @@ -1,23 +1,29 @@ +"""Legacy dynamic-actor tool. + +:class:`ApifyActorsTool` builds its argument schema and description at +construction time from a single Apify Actor's build, then runs that Actor. +""" + from __future__ import annotations import json -import os from typing import TYPE_CHECKING, Any from apify_client import ApifyClient from langchain_core.tools import BaseTool -from pydantic import BaseModel, Field, create_model - -from langchain_apify.error_messages import ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET -from langchain_apify.utils import ( - actor_id_to_tool_name, - create_apify_client, - get_actor_latest_build, - prune_actor_input_schema, +from pydantic import BaseModel, Field, PrivateAttr, SecretStr, create_model + +from langchain_apify._error_messages import _ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET +from langchain_apify._utils import ( + _MAX_DESCRIPTION_LEN, + _actor_id_to_tool_name, + _create_apify_client, + _get_actor_latest_build, + _prune_actor_input_schema, + _resolve_apify_token, + _resolve_deprecated_token, ) -from .const import MAX_DESCRIPTION_LEN - if TYPE_CHECKING: from langchain_core.callbacks import ( CallbackManagerForToolRun, @@ -27,9 +33,9 @@ class ApifyActorsTool(BaseTool): # type: ignore[override, override] """Tool that runs Apify Actors. - To use, you should have the environment variable `APIFY_API_TOKEN` set - with your API key, or pass `apify_api_token` - as a named parameter to the constructor. + To use, you should have the environment variable ``APIFY_TOKEN`` set + with your API key, or pass ``apify_token`` as a named parameter to the + constructor. For details, see https://docs.apify.com/platform/integrations/langchain @@ -56,10 +62,13 @@ class ApifyActorsTool(BaseTool): # type: ignore[override, override] chunk["messages"][-1].pretty_print() """ + _apify_client: ApifyClient = PrivateAttr() + _actor_id: str = PrivateAttr() + def __init__( self, actor_id: str, - apify_api_token: str | None = None, + apify_token: str | SecretStr | None = None, *args: Any, # noqa: ANN401 **kwargs: Any, # noqa: ANN401 ) -> None: @@ -67,28 +76,34 @@ def __init__( Args: actor_id (str): Actor name from Apify store to run. - apify_api_token (Optional[str]): Apify API token. + apify_token (Optional[str]): Apify API token. + apify_api_token: Deprecated alias for ``apify_token``. *args: Additional arguments. **kwargs: Additional keyword arguments. Raises: - ValueError: If the `APIFY_API_TOKEN` environment variable is not set + ValueError: If the ``APIFY_TOKEN`` environment variable is not set """ - apify_api_token = apify_api_token or os.getenv('APIFY_API_TOKEN') - if not apify_api_token: - msg = ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET + if 'apify_api_token' in kwargs: + apify_token = _resolve_deprecated_token(apify_token, kwargs.pop('apify_api_token')) + + _raw_token: str | None = ( + apify_token.get_secret_value() + if isinstance(apify_token, SecretStr) + else apify_token or _resolve_apify_token() + ) + if not _raw_token: + msg = _ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET raise ValueError(msg) - apify_client = create_apify_client(ApifyClient, apify_api_token) + apify_client = _create_apify_client(ApifyClient, _raw_token) + build = _get_actor_latest_build(apify_client, actor_id) kwargs.update( { - 'name': actor_id_to_tool_name(actor_id), - 'description': self._create_description(apify_client, actor_id), - 'args_schema': self._build_tool_args_schema_model( - apify_client, - actor_id, - ), + 'name': _actor_id_to_tool_name(actor_id), + 'description': self._create_description(build), + 'args_schema': self._build_tool_args_schema_model(build, actor_id), }, ) @@ -116,32 +131,27 @@ def _run( return self._run_actor(input_dict) @staticmethod - def _create_description(apify_client: ApifyClient, actor_id: str) -> str: - """Create a description for the tool. + def _create_description(build: dict) -> str: + """Create a description for the tool from an Actor build. Args: - apify_client (ApifyClient): Apify client instance. - actor_id (str): Actor name from Apify store to run. + build (dict): The Actor build, as returned by ``_get_actor_latest_build``. Returns: str: The description. """ - build = get_actor_latest_build(apify_client, actor_id) actor_description = build.get('actorDefinition', {}).get('description', '') - if len(actor_description) > MAX_DESCRIPTION_LEN: - actor_description = actor_description[:MAX_DESCRIPTION_LEN] + '...(TRUNCATED, TOO LONG)' + if len(actor_description) > _MAX_DESCRIPTION_LEN: + actor_description = actor_description[:_MAX_DESCRIPTION_LEN] + '...(TRUNCATED, TOO LONG)' return actor_description @staticmethod - def _build_tool_args_schema_model( - apify_client: ApifyClient, - actor_id: str, - ) -> type[BaseModel]: + def _build_tool_args_schema_model(build: dict, actor_id: str) -> type[BaseModel]: """Build a tool class for an agent that runs the Apify Actor. Args: - apify_client (ApifyClient): Apify client instance. - actor_id (str): Actor name from Apify store to run. + build (dict): The Actor build, as returned by ``_get_actor_latest_build``. + actor_id (str): Actor name from Apify store to run (used for error messages). Returns: type[BaseModel]: The tool input model class for the Apify Actor. @@ -149,12 +159,11 @@ def _build_tool_args_schema_model( Raises: ValueError: If the input schema is not found in the Actor build. """ - build = get_actor_latest_build(apify_client, actor_id) if not (actor_input := build.get('actorDefinition', {}).get('input')): msg = f'Input schema not found in the Actor build for Actor: {actor_id}' raise ValueError(msg) - properties, required = prune_actor_input_schema(actor_input) + properties, required = _prune_actor_input_schema(actor_input) properties = {'run_input': properties} description = ( diff --git a/langchain_apify/tools/base.py b/langchain_apify/tools/base.py new file mode 100644 index 0000000..cc84147 --- /dev/null +++ b/langchain_apify/tools/base.py @@ -0,0 +1,145 @@ +"""Shared base for generic Apify tools. + +Hosts the :class:`_ApifyGenericTool` base class, the JSON envelope helper, +run-metadata helpers, the developer-controlled clamp methods, and the shared +``_TOOL_RUN_ERRORS`` contract used by every tool ``_run``. +""" + +from __future__ import annotations + +import bisect +import json +from collections.abc import Callable +from datetime import datetime +from typing import Any + +from langchain_core.tools import BaseTool, ToolException +from pydantic import Field, PrivateAttr, SecretStr, model_validator + +from langchain_apify._client import ApifyToolsClient +from langchain_apify._constants import ( + _MAX_CRAWL_DEPTH_CAP, + _MAX_ITEMS_CAP, + _MAX_MEMORY_MBYTES_CAP, + _MAX_TIMEOUT_SECS_CAP, +) +from langchain_apify._error_messages import _ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET +from langchain_apify._utils import ( + _apify_token_secret_factory, + _resolve_deprecated_token_values, +) + +# Apify accepts memory_mbytes only as one of these power-of-2 values. +# https://docs.apify.com/api/v2/act-runs-post +_VALID_MEMORY_MBYTES: tuple[int, ...] = (128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768) + +# Errors a tool ``_run`` converts into a ``ToolException``: ``RuntimeError`` from +# a failed/empty Actor run, ``ValueError`` from client-side input validation. +_TOOL_RUN_ERRORS: tuple[type[Exception], ...] = (RuntimeError, ValueError) + + +def _iso(value: str | datetime | None) -> str | None: + """Coerce a possible ``datetime`` to an ISO-8601 string.""" + if isinstance(value, datetime): + return value.isoformat() + return value + + +def _run_meta(run: dict) -> dict: + """Extract a compact metadata dict from an Apify run-details dict.""" + return { + 'run_id': run.get('id'), + 'status': run.get('status'), + 'dataset_id': run.get('defaultDatasetId'), + 'started_at': _iso(run.get('startedAt')), + 'finished_at': _iso(run.get('finishedAt')), + } + + +# --------------------------------------------------------------------------- +# Shared base for generic tools +# --------------------------------------------------------------------------- + + +class _ApifyGenericTool(BaseTool): # type: ignore[override] + """Shared base for all generic Apify tools. + + Handles ``ApifyToolsClient`` creation, sets ``handle_tool_error``, + and defines developer-controlled safety limits that clamp values the + LLM may provide at invocation time. + + Subclasses only need to declare ``name``, ``description``, + ``args_schema``, and ``_run()``. + """ + + handle_tool_error: bool | str | Callable[[ToolException], str] | None = True + + apify_token: SecretStr | None = Field( + default_factory=_apify_token_secret_factory, + description='Apify API token. Falls back to the APIFY_TOKEN environment variable when None.', + exclude=True, + repr=False, + ) + max_timeout_secs: int = Field( + default=_MAX_TIMEOUT_SECS_CAP, description='Upper bound for timeout_secs the LLM may request.' + ) + max_memory_mbytes: int = Field( + default=_MAX_MEMORY_MBYTES_CAP, description='Upper bound for memory_mbytes the LLM may request.' + ) + max_items: int = Field( + default=_MAX_ITEMS_CAP, description='Upper bound for limit / dataset_items_limit the LLM may request.' + ) + max_crawl_depth: int = Field( + default=_MAX_CRAWL_DEPTH_CAP, description='Upper bound for max_crawl_depth the LLM may request.' + ) + + _client: ApifyToolsClient = PrivateAttr() + + @model_validator(mode='before') + @classmethod + def _handle_deprecated_apify_api_token(cls, values: dict) -> dict: + return _resolve_deprecated_token_values(values) + + def model_post_init(self, context: Any) -> None: # noqa: ANN401 + if self.apify_token is None: + msg = _ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET + raise ValueError(msg) + self._client = ApifyToolsClient(apify_token=self.apify_token.get_secret_value()) + super().model_post_init(context) + + def _clamp_timeout(self, value: int) -> int: + return max(1, min(value, self.max_timeout_secs)) + + def _clamp_memory(self, value: int | None) -> int | None: + if value is None or value <= 0: + return None + clamped = max(128, min(value, self.max_memory_mbytes)) + idx = bisect.bisect_left(_VALID_MEMORY_MBYTES, clamped) + # If snap-up exceeds cap, use largest valid at-or-below cap + if idx >= len(_VALID_MEMORY_MBYTES) or _VALID_MEMORY_MBYTES[idx] > self.max_memory_mbytes: + idx = bisect.bisect_right(_VALID_MEMORY_MBYTES, self.max_memory_mbytes) - 1 + # Misconfigured cap below the platform minimum, return the minimum. + return _VALID_MEMORY_MBYTES[max(idx, 0)] + + def _clamp_items(self, value: int) -> int: + return max(1, min(value, self.max_items)) + + def _clamp_depth(self, value: int) -> int: + # Floor at 0 (a depth of 0 means "only crawl the seed URL"). + return max(0, min(value, self.max_crawl_depth)) + + @staticmethod + def _envelope(run: dict | None, items: list, notice: str | None = None) -> str: + """Serialise the standard ``{"run": ..., "items": ...}`` tool envelope. + + ``run`` is a raw Apify run-details dict (passed through :func:`_run_meta`) + or ``None`` for dataset-only tools. When ``notice`` is provided it is added + under a ``notice`` key to surface an out-of-band hint to the caller (e.g. + the Actor returned demo placeholder data). ``default=str`` coerces + non-JSON-native values (e.g. ``datetime`` objects from the ``clean=True`` + deserialiser) so serialisation never raises ``TypeError``. + """ + payload: dict = {'run': _run_meta(run) if run is not None else None, 'items': items} + if notice is not None: + payload['notice'] = notice + return json.dumps(payload, default=str) diff --git a/langchain_apify/tools/core.py b/langchain_apify/tools/core.py new file mode 100644 index 0000000..9f15ccf --- /dev/null +++ b/langchain_apify/tools/core.py @@ -0,0 +1,469 @@ +"""Core generic Apify tools and their input schemas. + +These tools wrap general Apify platform primitives (running Actors and tasks, +fetching dataset items, scraping a single URL) behind LLM-friendly interfaces. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from langchain_core.tools import ArgsSchema, ToolException +from pydantic import BaseModel, Field + +from langchain_apify._constants import ( + _DEFAULT_DATASET_ITEMS_LIMIT, + _DEFAULT_RUN_TIMEOUT_SECS, + _DEFAULT_SCRAPE_TIMEOUT_SECS, + _MAX_ITEMS_CAP, + _MAX_MEMORY_MBYTES_CAP, + _MAX_TIMEOUT_SECS_CAP, +) +from langchain_apify.tools.base import _TOOL_RUN_ERRORS, _ApifyGenericTool + +if TYPE_CHECKING: + from langchain_core.callbacks import CallbackManagerForToolRun + from langchain_core.tools import BaseTool + + +# --------------------------------------------------------------------------- +# Input schemas for the generic tools +# --------------------------------------------------------------------------- + +_DESC_RUN_TIMEOUT_SECS = ( + f'Maximum time in seconds to wait for the run to finish (clamped to {_MAX_TIMEOUT_SECS_CAP} max).' +) +_DESC_MEMORY_MBYTES = ( + f'Memory per run in MB. Power of 2 from 128, or null for default (clamped to {_MAX_MEMORY_MBYTES_CAP} max).' +) +_DESC_DATASET_ITEMS_LIMIT = f'Maximum number of dataset items to return (clamped to {_MAX_ITEMS_CAP} max).' + + +class ApifyRunActorInput(BaseModel): + """Input schema for :class:`ApifyRunActorTool`.""" + + actor_id: str = Field(description='Actor ID or name (e.g. "apify/python-example").') + run_input: dict | None = Field(default=None, description='JSON-serialisable input for the Actor.') + timeout_secs: int = Field(default=_DEFAULT_RUN_TIMEOUT_SECS, description=_DESC_RUN_TIMEOUT_SECS) + memory_mbytes: int | None = Field(default=None, description=_DESC_MEMORY_MBYTES) + + +class ApifyGetDatasetItemsInput(BaseModel): + """Input schema for :class:`ApifyGetDatasetItemsTool`.""" + + dataset_id: str = Field(description='Apify dataset ID.') + limit: int = Field( + default=_DEFAULT_DATASET_ITEMS_LIMIT, + description=f'Maximum number of items to return (clamped to {_MAX_ITEMS_CAP} max).', + ) + offset: int = Field(default=0, description='Number of items to skip from the start.') + + +class ApifyRunActorAndGetDatasetInput(BaseModel): + """Input schema for :class:`ApifyRunActorAndGetDatasetTool`.""" + + actor_id: str = Field(description='Actor ID or name (e.g. "apify/python-example").') + run_input: dict | None = Field(default=None, description='JSON-serialisable input for the Actor.') + timeout_secs: int = Field(default=_DEFAULT_RUN_TIMEOUT_SECS, description=_DESC_RUN_TIMEOUT_SECS) + memory_mbytes: int | None = Field(default=None, description=_DESC_MEMORY_MBYTES) + dataset_items_limit: int = Field(default=_DEFAULT_DATASET_ITEMS_LIMIT, description=_DESC_DATASET_ITEMS_LIMIT) + + +class ApifyScrapeUrlInput(BaseModel): + """Input schema for :class:`ApifyScrapeUrlTool`.""" + + url: str = Field(description='The URL to scrape.') + timeout_secs: int = Field( + default=_DEFAULT_SCRAPE_TIMEOUT_SECS, + description=( + f'Maximum time in seconds to wait for the crawl to finish (clamped to {_MAX_TIMEOUT_SECS_CAP} max).' + ), + ) + + +class ApifyRunTaskInput(BaseModel): + """Input schema for :class:`ApifyRunTaskTool`.""" + + task_id: str = Field(description='Task ID or name (e.g. "user/my-task").') + task_input: dict | None = Field( + default=None, description="JSON-serialisable input that overrides the task's pre-saved input." + ) + timeout_secs: int = Field(default=_DEFAULT_RUN_TIMEOUT_SECS, description=_DESC_RUN_TIMEOUT_SECS) + memory_mbytes: int | None = Field(default=None, description=_DESC_MEMORY_MBYTES) + + +class ApifyRunTaskAndGetDatasetInput(BaseModel): + """Input schema for :class:`ApifyRunTaskAndGetDatasetTool`.""" + + task_id: str = Field(description='Task ID or name (e.g. "user/my-task").') + task_input: dict | None = Field( + default=None, description="JSON-serialisable input that overrides the task's pre-saved input." + ) + timeout_secs: int = Field(default=_DEFAULT_RUN_TIMEOUT_SECS, description=_DESC_RUN_TIMEOUT_SECS) + memory_mbytes: int | None = Field(default=None, description=_DESC_MEMORY_MBYTES) + dataset_items_limit: int = Field(default=_DEFAULT_DATASET_ITEMS_LIMIT, description=_DESC_DATASET_ITEMS_LIMIT) + + +# --------------------------------------------------------------------------- +# Generic tools +# --------------------------------------------------------------------------- + + +class ApifyRunActorTool(_ApifyGenericTool): # type: ignore[override] + """Run any Apify Actor by ID with an arbitrary JSON input. + + Returns run metadata in a JSON envelope. Use + :class:`ApifyGetDatasetItemsTool` afterwards to retrieve the results from + the dataset. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON object ``{"run": {...}, "items": []}`` where ``run`` holds + ``run_id``, ``status``, ``dataset_id``, ``started_at``, ``finished_at``. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyRunActorTool + + tool = ApifyRunActorTool() + result = tool.invoke({ + "actor_id": "apify/python-example", + "run_input": {"first_number": 2, "second_number": 3}, + }) + """ + + name: str = 'apify_run_actor' + description: str = ( + 'Run an Apify Actor synchronously and return a JSON envelope.' + ' Required: actor_id (str); Actor ID or name (e.g. "apify/python-example").' + f' Optional: run_input (dict), timeout_secs (int, default {_DEFAULT_RUN_TIMEOUT_SECS}),' + ' memory_mbytes (int|null).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at), items.' + ' Use apify_get_dataset_items with run.dataset_id to fetch results.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyRunActorInput + + def _run( + self, + actor_id: str, + run_input: dict | None = None, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + memory_mbytes: int | None = None, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run = self._client.run_actor( + actor_id, run_input, self._clamp_timeout(timeout_secs), self._clamp_memory(memory_mbytes) + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, []) + + +class ApifyGetDatasetItemsTool(_ApifyGenericTool): # type: ignore[override] + """Fetch items from an existing Apify dataset by ID. + + Returns a JSON object with ``"run"`` (always ``null`` here, since no Actor + is run) and ``"items"`` (the list of item dicts, empty when the dataset + has no items). + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON object ``{"run": null, "items": [...]}``. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyGetDatasetItemsTool + + tool = ApifyGetDatasetItemsTool() + result = tool.invoke({"dataset_id": "abc123", "limit": 10}) + """ + + name: str = 'apify_get_dataset_items' + description: str = ( + 'Fetch items from an Apify dataset by ID and return a JSON envelope.' + ' Required: dataset_id (str); Apify dataset ID.' + f' Optional: limit (int, default {_DEFAULT_DATASET_ITEMS_LIMIT}), offset (int, default 0).' + ' Returns JSON with keys: run (null), items (empty array when the dataset has no items).' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyGetDatasetItemsInput + + def _run( + self, + dataset_id: str, + limit: int = _DEFAULT_DATASET_ITEMS_LIMIT, + offset: int = 0, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + items = self._client.get_dataset_items(dataset_id, self._clamp_items(limit), max(0, offset)) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(None, items) + + +class ApifyRunActorAndGetDatasetTool(_ApifyGenericTool): # type: ignore[override] + """Run any Apify Actor and return both run metadata and dataset items. + + Combines :class:`ApifyRunActorTool` and :class:`ApifyGetDatasetItemsTool` + into a single call. Returns a JSON envelope. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON object ``{"run": {...}, "items": [...]}`` where ``run`` holds + ``run_id``, ``status``, ``dataset_id``, ``started_at``, ``finished_at`` + and ``items`` are the dataset item dicts. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyRunActorAndGetDatasetTool + + tool = ApifyRunActorAndGetDatasetTool() + result = tool.invoke({ + "actor_id": "apify/python-example", + "run_input": {"first_number": 2, "second_number": 3}, + }) + """ + + name: str = 'apify_run_actor_and_get_dataset' + description: str = ( + 'Run an Apify Actor synchronously and return a JSON envelope.' + ' Required: actor_id (str); Actor ID or name (e.g. "apify/python-example").' + f' Optional: run_input (dict), timeout_secs (int, default {_DEFAULT_RUN_TIMEOUT_SECS}),' + f' memory_mbytes (int|null), dataset_items_limit (int, default {_DEFAULT_DATASET_ITEMS_LIMIT}).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at)' + ' and items (list of dataset item dicts).' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyRunActorAndGetDatasetInput + + def _run( + self, + actor_id: str, + run_input: dict | None = None, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + memory_mbytes: int | None = None, + dataset_items_limit: int = _DEFAULT_DATASET_ITEMS_LIMIT, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.run_actor_and_get_items( + actor_id, + run_input, + self._clamp_timeout(timeout_secs), + self._clamp_memory(memory_mbytes), + self._clamp_items(dataset_items_limit), + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, items) + + +class ApifyScrapeUrlTool(_ApifyGenericTool): # type: ignore[override] + """Scrape a single URL and return its content in a JSON envelope. + + Uses the ``apify/website-content-crawler`` Actor under the hood with + ``maxCrawlPages=1``. The scraped content (markdown, or plain text when + markdown is unavailable) is the ``content`` field of the single item. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON object ``{"run": {...}, "items": [{"url": ..., "content": ...}]}``. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyScrapeUrlTool + + tool = ApifyScrapeUrlTool() + markdown = tool.invoke({"url": "https://apify.com"}) + """ + + name: str = 'apify_scrape_url' + description: str = ( + 'Scrape a single URL using Apify and return a JSON envelope.' + ' Required: url (str); the URL to scrape.' + f' Optional: timeout_secs (int, default {_DEFAULT_SCRAPE_TIMEOUT_SECS}).' + ' Returns JSON with keys: run, items ([{url, content}];' + ' content is markdown, or plain text when markdown is unavailable).' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyScrapeUrlInput + + def _run( + self, + url: str, + timeout_secs: int = _DEFAULT_SCRAPE_TIMEOUT_SECS, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + # scrape_url_with_metadata is the rich primitive; the plain + # scrape_url() drops the run metadata + content source this tool needs. + run, _, content, _ = self._client.scrape_url_with_metadata(url, self._clamp_timeout(timeout_secs)) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, [{'url': url, 'content': content}]) + + +class ApifyRunTaskTool(_ApifyGenericTool): # type: ignore[override] + """Run a saved Apify Actor task by ID and return run metadata. + + Actor tasks are pre-configured Actor runs saved in the Apify Console. + This tool starts a task with optional input overrides and returns run + metadata in a JSON envelope. Use :class:`ApifyGetDatasetItemsTool` + afterwards to retrieve results. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON object ``{"run": {...}, "items": []}`` where ``run`` holds + ``run_id``, ``status``, ``dataset_id``, ``started_at``, ``finished_at``. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyRunTaskTool + + tool = ApifyRunTaskTool() + result = tool.invoke({ + "task_id": "user/my-task", + "task_input": {"key": "value"}, + }) + """ + + name: str = 'apify_run_task' + description: str = ( + 'Run a saved Apify Actor task synchronously and return a JSON envelope.' + ' Required: task_id (str); task ID or name (e.g. "user/my-task").' + f' Optional: task_input (dict), timeout_secs (int, default {_DEFAULT_RUN_TIMEOUT_SECS}),' + ' memory_mbytes (int|null).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at), items.' + ' Use apify_get_dataset_items with run.dataset_id to fetch results.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyRunTaskInput + + def _run( + self, + task_id: str, + task_input: dict | None = None, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + memory_mbytes: int | None = None, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run = self._client.run_task( + task_id, task_input, self._clamp_timeout(timeout_secs), self._clamp_memory(memory_mbytes) + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, []) + + +class ApifyRunTaskAndGetDatasetTool(_ApifyGenericTool): # type: ignore[override] + """Run a saved Apify Actor task and return both run metadata and dataset items. + + Combines :class:`ApifyRunTaskTool` and :class:`ApifyGetDatasetItemsTool` + into a single call. Returns a JSON envelope. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON object ``{"run": {...}, "items": [...]}`` where ``run`` holds + ``run_id``, ``status``, ``dataset_id``, ``started_at``, ``finished_at`` + and ``items`` are the dataset item dicts. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyRunTaskAndGetDatasetTool + + tool = ApifyRunTaskAndGetDatasetTool() + result = tool.invoke({ + "task_id": "user/my-task", + "task_input": {"key": "value"}, + }) + """ + + name: str = 'apify_run_task_and_get_dataset' + description: str = ( + 'Run a saved Apify Actor task synchronously and return a JSON envelope.' + ' Required: task_id (str); task ID or name (e.g. "user/my-task").' + f' Optional: task_input (dict), timeout_secs (int, default {_DEFAULT_RUN_TIMEOUT_SECS}),' + f' memory_mbytes (int|null), dataset_items_limit (int, default {_DEFAULT_DATASET_ITEMS_LIMIT}).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at)' + ' and items (list of dataset item dicts).' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyRunTaskAndGetDatasetInput + + def _run( + self, + task_id: str, + task_input: dict | None = None, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + memory_mbytes: int | None = None, + dataset_items_limit: int = _DEFAULT_DATASET_ITEMS_LIMIT, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.run_task_and_get_items( + task_id, + task_input, + self._clamp_timeout(timeout_secs), + self._clamp_memory(memory_mbytes), + self._clamp_items(dataset_items_limit), + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, items) + + +# Convenience tool-class list for selective agent binding. +APIFY_CORE_TOOLS: list[type[BaseTool]] = [ + ApifyRunActorTool, + ApifyGetDatasetItemsTool, + ApifyRunActorAndGetDatasetTool, + ApifyScrapeUrlTool, + ApifyRunTaskTool, + ApifyRunTaskAndGetDatasetTool, +] diff --git a/langchain_apify/tools/search.py b/langchain_apify/tools/search.py new file mode 100644 index 0000000..878dcae --- /dev/null +++ b/langchain_apify/tools/search.py @@ -0,0 +1,546 @@ +"""Search & crawling Actor tools and their input schemas. + +Each tool wraps a single Apify search/crawl Actor behind a simplified, +LLM-friendly interface returning the standard JSON envelope. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from langchain_core.tools import ArgsSchema, ToolException +from pydantic import BaseModel, Field, field_validator + +from langchain_apify._constants import ( + _DEFAULT_CRAWLER_TYPE, + _DEFAULT_ECOMMERCE_MAX_RESULTS, + _DEFAULT_GOOGLE_MAPS_MAX_RESULTS, + _DEFAULT_GOOGLE_MAX_RESULTS, + _DEFAULT_MAX_CRAWL_DEPTH, + _DEFAULT_MAX_CRAWL_PAGES, + _DEFAULT_RAG_MAX_RESULTS, + _DEFAULT_RUN_TIMEOUT_SECS, + _DEFAULT_YOUTUBE_MAX_RESULTS, + _MAX_CRAWL_DEPTH_CAP, + _MAX_ITEMS_CAP, + _RAG_MAX_RESULTS_CAP, +) +from langchain_apify._types import ( # noqa: TCH001 # runtime-needed: pydantic Field annotations + CrawlerType, + EcommerceUrlType, + YouTubeSearchType, +) +from langchain_apify._utils import _extract_content, _extract_source, _safe_title +from langchain_apify.tools.base import _TOOL_RUN_ERRORS, _ApifyGenericTool +from langchain_apify.tools.core import _DESC_RUN_TIMEOUT_SECS + +if TYPE_CHECKING: + from langchain_core.callbacks import CallbackManagerForToolRun + from langchain_core.tools import BaseTool + + +# --------------------------------------------------------------------------- +# Input schemas +# --------------------------------------------------------------------------- + + +class ApifyGoogleSearchInput(BaseModel): + """Input schema for :class:`ApifyGoogleSearchTool`.""" + + query: str = Field(description='Search query string.') + max_results: int = Field( + default=_DEFAULT_GOOGLE_MAX_RESULTS, + description=f'Maximum number of search results to return (clamped to {_MAX_ITEMS_CAP} max).', + ) + country_code: str | None = Field( + default=None, + description='Two-letter country code (case-insensitive; normalised to lowercase, e.g. "us", "gb").', + pattern=r'^[a-zA-Z]{2}$', + ) + language_code: str | None = Field( + default=None, + description='Two-letter language code (case-insensitive; normalised to lowercase, e.g. "en", "fr").', + pattern=r'^[a-zA-Z]{2}$', + ) + timeout_secs: int = Field(default=_DEFAULT_RUN_TIMEOUT_SECS, description=_DESC_RUN_TIMEOUT_SECS) + + @field_validator('country_code', 'language_code') + @classmethod + def _normalise_locale_code(cls, value: str | None) -> str | None: + return value.lower() if value else value + + +class ApifyWebCrawlerInput(BaseModel): + """Input schema for :class:`ApifyWebCrawlerTool`.""" + + url: str = Field(description='Seed URL to start crawling from.') + max_crawl_pages: int = Field( + default=_DEFAULT_MAX_CRAWL_PAGES, + description=f'Maximum number of pages to crawl (clamped to {_MAX_ITEMS_CAP} max).', + ) + max_crawl_depth: int = Field( + default=_DEFAULT_MAX_CRAWL_DEPTH, + description=f'Maximum link-follow depth from the seed URL (clamped to {_MAX_CRAWL_DEPTH_CAP} max).', + ) + crawler_type: CrawlerType = Field( + default=_DEFAULT_CRAWLER_TYPE, + description='Crawler engine: "cheerio" (fast, static HTML), "playwright:adaptive" or "playwright:firefox".', + ) + timeout_secs: int = Field(default=_DEFAULT_RUN_TIMEOUT_SECS, description=_DESC_RUN_TIMEOUT_SECS) + + +class ApifyRAGWebBrowserInput(BaseModel): + """Input schema for :class:`ApifyRAGWebBrowserTool`.""" + + query: str = Field(description='Search query string.') + max_results: int = Field( + default=_DEFAULT_RAG_MAX_RESULTS, + description=f'Maximum number of results to return (clamped to {_RAG_MAX_RESULTS_CAP} max).', + ) + + +class ApifyGoogleMapsInput(BaseModel): + """Input schema for :class:`ApifyGoogleMapsTool`.""" + + query: str = Field(description='Search query (e.g. "coffee shops in Berlin").') + max_results: int = Field( + default=_DEFAULT_GOOGLE_MAPS_MAX_RESULTS, + description=f'Maximum number of places to return (clamped to {_MAX_ITEMS_CAP} max).', + ) + language: str | None = Field( + default=None, + description='Optional ISO language code for results (e.g. "en", "de").', + ) + + +class ApifyYouTubeScraperInput(BaseModel): + """Input schema for :class:`ApifyYouTubeScraperTool`.""" + + search_query: str = Field( + description=('Keyword for "search" mode, or a video/channel URL for "video"/"channel" modes.'), + ) + search_type: YouTubeSearchType = Field( + default='search', + description='Scrape mode: search keyword, single video URL, or channel URL.', + ) + max_results: int = Field( + default=_DEFAULT_YOUTUBE_MAX_RESULTS, + description=f'Maximum number of items to return (clamped to {_MAX_ITEMS_CAP} max).', + ) + + +class ApifyEcommerceScraperInput(BaseModel): + """Input schema for :class:`ApifyEcommerceScraperTool`.""" + + url: str = Field(description='Product-detail URL or category / listing page URL to scrape.') + url_type: EcommerceUrlType = Field( + default='product', + description=( + 'Type of page the URL points to: "product" for a product-detail page, ' + '"category" for a category / listing page.' + ), + ) + max_results: int = Field( + default=_DEFAULT_ECOMMERCE_MAX_RESULTS, + description=f'Maximum number of products to return (clamped to {_MAX_ITEMS_CAP} max).', + ) + + +# --------------------------------------------------------------------------- +# Tools +# --------------------------------------------------------------------------- + + +class ApifyGoogleSearchTool(_ApifyGenericTool): # type: ignore[override] + """Search Google and return structured results via Apify. + + Wraps the ``apify/google-search-scraper`` Actor behind a simplified, + LLM-friendly interface. Returns a JSON envelope whose ``items`` are + result objects, each with ``title``, ``url``, and ``description`` keys. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON object ``{"run": null, "items": [{"title", "url", "description"}]}``. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyGoogleSearchTool + + tool = ApifyGoogleSearchTool() + results = tool.invoke({"query": "LangChain framework"}) + """ + + name: str = 'apify_google_search' + description: str = ( + 'Search Google using Apify and return a JSON envelope.' + ' Each item has keys: title, url, description.' + ' Required: query (str); the search query.' + f' Optional: max_results (int, default {_DEFAULT_GOOGLE_MAX_RESULTS}),' + ' country_code (str|null), language_code (str|null),' + f' timeout_secs (int, default {_DEFAULT_RUN_TIMEOUT_SECS}).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at), items.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyGoogleSearchInput + + def _run( + self, + query: str, + max_results: int = _DEFAULT_GOOGLE_MAX_RESULTS, + country_code: str | None = None, + language_code: str | None = None, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, results = self._client.google_search( + query, + max_results=self._clamp_items(max_results), + country_code=country_code, + language_code=language_code, + timeout_secs=self._clamp_timeout(timeout_secs), + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + # default=str coerces any non-JSON-native types (e.g. datetime from + # the Apify client's clean=True deserialiser) to their string repr + # so the LLM never sees a serialisation failure. + return self._envelope(run, results) + + +class ApifyWebCrawlerTool(_ApifyGenericTool): # type: ignore[override] + """Crawl a website and return page content as JSON via Apify. + + Wraps the ``apify/website-content-crawler`` Actor. Returns a JSON envelope + whose ``items`` are page objects, each with ``url``, ``title``, and + ``content`` (markdown) keys. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON object ``{"run": null, "items": [{"url", "title", "content"}]}``. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyWebCrawlerTool + + tool = ApifyWebCrawlerTool() + pages = tool.invoke({ + "url": "https://docs.apify.com", + "max_crawl_pages": 5, + }) + """ + + name: str = 'apify_web_crawler' + description: str = ( + 'Crawl a website using Apify and return a JSON envelope.' + ' Each item has keys: url, title, content (markdown).' + ' Required: url (str); seed URL to crawl.' + f' Optional: max_crawl_pages (int, default {_DEFAULT_MAX_CRAWL_PAGES}),' + f' max_crawl_depth (int, default {_DEFAULT_MAX_CRAWL_DEPTH}),' + f' crawler_type (str, default "{_DEFAULT_CRAWLER_TYPE}"),' + f' timeout_secs (int, default {_DEFAULT_RUN_TIMEOUT_SECS}).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at), items.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyWebCrawlerInput + + def _run( + self, + url: str, + max_crawl_pages: int = _DEFAULT_MAX_CRAWL_PAGES, + max_crawl_depth: int = _DEFAULT_MAX_CRAWL_DEPTH, + crawler_type: CrawlerType = _DEFAULT_CRAWLER_TYPE, + timeout_secs: int = _DEFAULT_RUN_TIMEOUT_SECS, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.crawl_website( + url, + max_crawl_pages=self._clamp_items(max_crawl_pages), + max_crawl_depth=self._clamp_depth(max_crawl_depth), + crawler_type=crawler_type, + timeout_secs=self._clamp_timeout(timeout_secs), + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + # Defensive filter: some Actor responses occasionally surface list-typed + # entries (e.g. nested arrays for sitemap-style outputs). Skip anything + # that isn't a dict so .get() never blows up. + pages = [ + { + 'url': item.get('url', ''), + 'title': _safe_title(item), + 'content': _extract_content(item), + } + for item in items + if isinstance(item, dict) + ] + return self._envelope(run, pages) + + +class ApifyRAGWebBrowserTool(_ApifyGenericTool): # type: ignore[override] + """Search the web and return content from top results. + + Wraps the ``apify/rag-web-browser`` Actor. Unlike + :class:`ApifySearchRetriever` (which returns LangChain ``Document`` + objects for RAG pipelines), this tool returns a JSON envelope + suitable for agent tool-calling. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON object ``{"run": {...}, "items": [{"url", "title", "content"}]}``. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyRAGWebBrowserTool + + tool = ApifyRAGWebBrowserTool() + result = tool.invoke({"query": "what is LangChain?", "max_results": 3}) + """ + + name: str = 'apify_rag_web_browser' + description: str = ( + 'Search the web and return a JSON envelope with crawled results.' + ' Each item has keys: url, title, content.' + ' Required: query (str) - the search query.' + f' Optional: max_results (int, default {_DEFAULT_RAG_MAX_RESULTS}).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at), items.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyRAGWebBrowserInput + # The rag-web-browser Actor caps maxResults at 100; override the generic + # 1000 ceiling so _clamp_items clamps to the value the Actor accepts. + max_items: int = _RAG_MAX_RESULTS_CAP + + def _run( + self, + query: str, + max_results: int = _DEFAULT_RAG_MAX_RESULTS, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.rag_web_search( + query, + max_results=self._clamp_items(max_results), + timeout_secs=self.max_timeout_secs, + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + results = [ + { + 'url': _extract_source(item), + 'title': _safe_title(item), + 'content': _extract_content(item), + } + for item in items + if isinstance(item, dict) + ] + return self._envelope(run, results) + + +class ApifyGoogleMapsTool(_ApifyGenericTool): # type: ignore[override] + """Search Google Maps for places, reviews, and business details. + + Wraps the ``compass/crawler-google-places`` Actor. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON object ``{"run": {...}, "items": [...]}`` where ``run`` holds + ``run_id``, ``status``, ``dataset_id``, ``started_at``, ``finished_at`` + and ``items`` are place dicts. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyGoogleMapsTool + + tool = ApifyGoogleMapsTool() + result = tool.invoke({"query": "coffee shops in Berlin", "max_results": 5}) + """ + + name: str = 'apify_google_maps' + description: str = ( + 'Search Google Maps places, reviews, and business details and return a JSON envelope.' + ' Required: query (str) - the search query.' + f' Optional: max_results (int, default {_DEFAULT_GOOGLE_MAPS_MAX_RESULTS}),' + ' language (str|null - ISO code, e.g. "en").' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at), items.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyGoogleMapsInput + + def _run( + self, + query: str, + max_results: int = _DEFAULT_GOOGLE_MAPS_MAX_RESULTS, + language: str | None = None, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.google_maps_search( + query, + max_results=self._clamp_items(max_results), + language=language, + timeout_secs=self.max_timeout_secs, + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, items) + + +class ApifyYouTubeScraperTool(_ApifyGenericTool): # type: ignore[override] + """Scrape YouTube videos, channels, or search results. + + Wraps the ``streamers/youtube-scraper`` Actor. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON object ``{"run": {...}, "items": [...]}`` where ``run`` holds + ``run_id``, ``status``, ``dataset_id``, ``started_at``, ``finished_at`` + and ``items`` are video / channel dicts. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyYouTubeScraperTool + + tool = ApifyYouTubeScraperTool() + result = tool.invoke({ + "search_query": "langchain tutorial", + "search_type": "search", + "max_results": 5, + }) + """ + + name: str = 'apify_youtube_scraper' + description: str = ( + 'Scrape YouTube by keyword, video URL, or channel URL and return a JSON envelope.' + ' Required: search_query (str - keyword for "search" mode, or a video/channel URL).' + ' Optional: search_type (one of "search", "video", "channel"; default "search"),' + f' max_results (int, default {_DEFAULT_YOUTUBE_MAX_RESULTS}).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at), items.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyYouTubeScraperInput + + def _run( + self, + search_query: str, + search_type: YouTubeSearchType = 'search', + max_results: int = _DEFAULT_YOUTUBE_MAX_RESULTS, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.youtube_scrape( + search_query=search_query, + search_type=search_type, + max_results=self._clamp_items(max_results), + timeout_secs=self.max_timeout_secs, + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, items) + + +class ApifyEcommerceScraperTool(_ApifyGenericTool): # type: ignore[override] + """Extract product or listing data from an e-commerce URL. + + Wraps the ``apify/e-commerce-scraping-tool`` Actor. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON object ``{"run": {...}, "items": [...]}`` where ``run`` holds + ``run_id``, ``status``, ``dataset_id``, ``started_at``, ``finished_at`` + and ``items`` are product / listing dicts. + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyEcommerceScraperTool + + tool = ApifyEcommerceScraperTool() + result = tool.invoke({ + "url": "https://shop.example.com/category/123", + "url_type": "category", + "max_results": 20, + }) + """ + + name: str = 'apify_ecommerce_scraper' + description: str = ( + 'Extract product data from an e-commerce URL and return a JSON envelope.' + ' Required: url (str) - product-detail or category / listing URL.' + ' Optional: url_type (one of "product", "category"; default "product"),' + f' max_results (int, default {_DEFAULT_ECOMMERCE_MAX_RESULTS}).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at), items.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyEcommerceScraperInput + + def _run( + self, + url: str, + url_type: EcommerceUrlType = 'product', + max_results: int = _DEFAULT_ECOMMERCE_MAX_RESULTS, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.ecommerce_scrape( + url, + url_type=url_type, + max_results=self._clamp_items(max_results), + timeout_secs=self.max_timeout_secs, + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, items) + + +# Convenience tool-class list for selective agent binding. +APIFY_SEARCH_TOOLS: list[type[BaseTool]] = [ + ApifyGoogleSearchTool, + ApifyWebCrawlerTool, + ApifyRAGWebBrowserTool, + ApifyGoogleMapsTool, + ApifyYouTubeScraperTool, + ApifyEcommerceScraperTool, +] diff --git a/langchain_apify/tools/social.py b/langchain_apify/tools/social.py new file mode 100644 index 0000000..cab0bea --- /dev/null +++ b/langchain_apify/tools/social.py @@ -0,0 +1,623 @@ +"""Social-media Actor tools and their input schemas. + +Each tool wraps a single Apify social-media Actor behind a simplified, +LLM-friendly interface returning the standard JSON envelope. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from langchain_core.tools import ArgsSchema, ToolException +from pydantic import BaseModel, Field + +from langchain_apify._constants import ( + _DEFAULT_LINKEDIN_SEARCH_MAX_RESULTS, + _DEFAULT_SOCIAL_RESULTS_LIMIT, + _MAX_ITEMS_CAP, +) +from langchain_apify._error_messages import _NOTICE_TWITTER_DEMO +from langchain_apify._types import ( # noqa: TCH001 # runtime-needed: pydantic Field annotations + InstagramSearchType, + TikTokSearchType, + TwitterSearchMode, + TwitterSort, +) +from langchain_apify.tools.base import _TOOL_RUN_ERRORS, _ApifyGenericTool + +if TYPE_CHECKING: + from langchain_core.callbacks import CallbackManagerForToolRun + from langchain_core.tools import BaseTool + + +def _has_demo_items(items: list[dict]) -> bool: + """Return True if the Twitter Actor returned demo placeholder items. + + The ``apidojo/twitter-scraper-lite`` Actor emits ``{"demo": true}`` items + instead of real tweets when run on the Apify free plan. Real tweets never + carry a truthy ``demo`` field. + """ + return any(isinstance(item, dict) and item.get('demo') is True for item in items) + + +# --------------------------------------------------------------------------- +# Input schemas +# --------------------------------------------------------------------------- + + +class ApifyInstagramScraperInput(BaseModel): + """Input schema for :class:`ApifyInstagramScraperTool`.""" + + search_type: InstagramSearchType = Field( + description=( + 'Type of data to scrape: "user" for a profile\'s posts, "hashtag" ' + 'for posts under a tag, "post" for a single post, "comments" for ' + 'comments on a post.' + ), + ) + search_query: str = Field( + description=( + 'Username, hashtag, or a full Instagram URL including the scheme ' + '(e.g. https://www.instagram.com/p/ABC123/). For "comments" you must ' + 'pass a full post URL.' + ), + ) + max_results: int = Field( + default=_DEFAULT_SOCIAL_RESULTS_LIMIT, + description=f'Maximum number of items to return (clamped to {_MAX_ITEMS_CAP} max).', + ) + only_posts_newer_than: str | None = Field( + default=None, + description=( + 'Optional date filter. Accepts YYYY-MM-DD, ISO-8601, or relative ' + 'values like "1 day", "2 months", "3 years".' + ), + ) + + +class ApifyLinkedInProfilePostsInput(BaseModel): + """Input schema for :class:`ApifyLinkedInProfilePostsTool`.""" + + profile_url: str = Field( + description='LinkedIn profile URL or username (e.g. "satyanadella" or "linkedin.com/in/satyanadella").', + ) + max_results: int = Field( + default=_DEFAULT_SOCIAL_RESULTS_LIMIT, + description=f'Maximum number of posts to return (clamped to {_MAX_ITEMS_CAP} max).', + ) + + +class ApifyLinkedInProfileSearchInput(BaseModel): + """Input schema for :class:`ApifyLinkedInProfileSearchTool`.""" + + query: str = Field(description='Search keywords (e.g. name, title, company).') + max_results: int = Field( + default=_DEFAULT_LINKEDIN_SEARCH_MAX_RESULTS, + description=f'Maximum number of profiles to return (clamped to {_MAX_ITEMS_CAP} max).', + ) + + +class ApifyLinkedInProfileDetailInput(BaseModel): + """Input schema for :class:`ApifyLinkedInProfileDetailTool`.""" + + profile_url: str = Field( + description='LinkedIn profile URL, username, or URN (e.g. "neal-mohan").', + ) + include_email: bool = Field( + default=False, + description='If True, attempt to include the profile email when available.', + ) + + +class ApifyTwitterScraperInput(BaseModel): + """Input schema for :class:`ApifyTwitterScraperTool`.""" + + search_query: str = Field(description='Search term, Twitter handle, or tweet URL.') + search_mode: TwitterSearchMode = Field( + default='search', + description=( + 'Scraping mode: "search" for keyword search, "user" for a handle\'s ' + 'tweets, "replies" for a tweet URL\'s replies.' + ), + ) + max_results: int = Field( + default=_DEFAULT_SOCIAL_RESULTS_LIMIT, + description=f'Maximum number of tweets to return (clamped to {_MAX_ITEMS_CAP} max).', + ) + start: str | None = Field( + default=None, + description='Optional start date - only return tweets newer than this date.', + ) + end: str | None = Field( + default=None, + description='Optional end date - only return tweets older than this date.', + ) + sort: TwitterSort | None = Field( + default=None, + description='Optional sort order: "Latest" for most recent first, "Top" for most popular.', + ) + + +class ApifyTikTokScraperInput(BaseModel): + """Input schema for :class:`ApifyTikTokScraperTool`.""" + + search_query: str = Field(description='Username, hashtag, search keyword, or TikTok post URL.') + search_type: TikTokSearchType = Field( + default='search', + description=( + 'Type of content to scrape: "search" for keyword search, "user" for ' + 'a profile\'s videos, "hashtag" for videos under a tag, "post" for a ' + 'specific TikTok post URL.' + ), + ) + max_results: int = Field( + default=_DEFAULT_SOCIAL_RESULTS_LIMIT, + description=f'Maximum number of items to return (clamped to {_MAX_ITEMS_CAP} max).', + ) + + +class ApifyFacebookPostsScraperInput(BaseModel): + """Input schema for :class:`ApifyFacebookPostsScraperTool`.""" + + page_url: str = Field(description='Facebook page URL to scrape (public pages only).') + max_results: int = Field( + default=_DEFAULT_SOCIAL_RESULTS_LIMIT, + description=f'Maximum number of posts to return (clamped to {_MAX_ITEMS_CAP} max).', + ) + only_posts_newer_than: str | None = Field( + default=None, + description=( + 'Optional date filter. Accepts YYYY-MM-DD, ISO-8601, or relative ' + 'values like "1 day", "2 months", "3 years".' + ), + ) + only_posts_older_than: str | None = Field( + default=None, + description=( + 'Optional date filter. Accepts YYYY-MM-DD, ISO-8601, or relative ' + 'values like "1 day", "2 months", "3 years".' + ), + ) + + +# --------------------------------------------------------------------------- +# Tools +# --------------------------------------------------------------------------- + + +class ApifyInstagramScraperTool(_ApifyGenericTool): # type: ignore[override] + """Scrape Instagram profiles, hashtags, posts, or comments. + + Uses the ``apify/instagram-scraper`` Actor under the hood. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON string with two keys: ``run`` (dict with ``run_id``, ``status``, + ``dataset_id``, ``started_at``, ``finished_at``) and ``items`` (list + of scraped item dicts). + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyInstagramScraperTool + + tool = ApifyInstagramScraperTool() + result = tool.invoke({ + "search_type": "user", + "search_query": "apify", + "max_results": 10, + }) + """ + + name: str = 'apify_instagram_scraper' + description: str = ( + 'Scrape Instagram profiles, hashtags, posts, or comments and return the results as JSON.' + ' Required: search_type (one of "user", "hashtag", "post", "comments"),' + ' search_query (str - username, hashtag, or a full Instagram URL including the scheme,' + ' e.g. https://www.instagram.com/p/ABC123/; "comments" requires a full post URL).' + f' Optional: max_results (int, default {_DEFAULT_SOCIAL_RESULTS_LIMIT}),' + ' only_posts_newer_than (str - date filter, e.g. "2025-01-01" or "1 week").' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at) and items.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyInstagramScraperInput + + def _run( + self, + search_type: InstagramSearchType, + search_query: str, + max_results: int = _DEFAULT_SOCIAL_RESULTS_LIMIT, + only_posts_newer_than: str | None = None, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.instagram_scrape( + search_type=search_type, + search_query=search_query, + max_results=self._clamp_items(max_results), + only_posts_newer_than=only_posts_newer_than, + timeout_secs=self.max_timeout_secs, + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, items) + + +class ApifyLinkedInProfilePostsTool(_ApifyGenericTool): # type: ignore[override] + """Extract posts from a LinkedIn profile. + + Uses the ``apimaestro/linkedin-profile-posts`` Actor under the hood. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON string with two keys: ``run`` (dict with ``run_id``, ``status``, + ``dataset_id``, ``started_at``, ``finished_at``) and ``items`` (list + of post dicts). + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyLinkedInProfilePostsTool + + tool = ApifyLinkedInProfilePostsTool() + result = tool.invoke({ + "profile_url": "https://www.linkedin.com/in/satyanadella", + "max_results": 10, + }) + """ + + name: str = 'apify_linkedin_profile_posts' + description: str = ( + 'Extract posts from a LinkedIn profile and return them as JSON.' + ' Required: profile_url (str - LinkedIn profile URL or username, e.g. "satyanadella").' + f' Optional: max_results (int, default {_DEFAULT_SOCIAL_RESULTS_LIMIT}).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at) and items.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyLinkedInProfilePostsInput + + def _run( + self, + profile_url: str, + max_results: int = _DEFAULT_SOCIAL_RESULTS_LIMIT, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.linkedin_profile_posts( + profile_url=profile_url, + max_results=self._clamp_items(max_results), + timeout_secs=self.max_timeout_secs, + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, items) + + +class ApifyLinkedInProfileSearchTool(_ApifyGenericTool): # type: ignore[override] + """Search for LinkedIn profiles by keyword or criteria. + + Uses the ``harvestapi/linkedin-profile-search`` Actor under the hood. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON string with two keys: ``run`` (dict with ``run_id``, ``status``, + ``dataset_id``, ``started_at``, ``finished_at``) and ``items`` (list + of profile dicts). + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyLinkedInProfileSearchTool + + tool = ApifyLinkedInProfileSearchTool() + result = tool.invoke({ + "query": "Founder", + "max_results": 10, + }) + """ + + name: str = 'apify_linkedin_profile_search' + description: str = ( + 'Search for LinkedIn profiles by keyword (name, title, company) and return matching profiles as JSON.' + ' Required: query (str - search keywords).' + f' Optional: max_results (int, default {_DEFAULT_LINKEDIN_SEARCH_MAX_RESULTS}).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at) and items.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyLinkedInProfileSearchInput + + def _run( + self, + query: str, + max_results: int = _DEFAULT_LINKEDIN_SEARCH_MAX_RESULTS, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.linkedin_profile_search( + query=query, + max_results=self._clamp_items(max_results), + timeout_secs=self.max_timeout_secs, + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, items) + + +class ApifyLinkedInProfileDetailTool(_ApifyGenericTool): # type: ignore[override] + """Retrieve detailed information from a specific LinkedIn profile. + + Uses the ``apimaestro/linkedin-profile-detail`` Actor under the hood. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON string with two keys: ``run`` (dict with ``run_id``, ``status``, + ``dataset_id``, ``started_at``, ``finished_at``) and ``items`` (typically + a single-element list with the profile dict). + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyLinkedInProfileDetailTool + + tool = ApifyLinkedInProfileDetailTool() + result = tool.invoke({ + "profile_url": "https://www.linkedin.com/in/neal-mohan", + }) + """ + + name: str = 'apify_linkedin_profile_detail' + description: str = ( + 'Retrieve detailed information from a specific LinkedIn profile and return it as JSON.' + ' Required: profile_url (str - LinkedIn profile URL, username, or URN, e.g. "neal-mohan").' + ' Optional: include_email (bool, default False - include profile email if available).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at) and items.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyLinkedInProfileDetailInput + + def _run( + self, + profile_url: str, + *, + include_email: bool = False, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.linkedin_profile_detail( + profile_url=profile_url, + include_email=include_email, + timeout_secs=self.max_timeout_secs, + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, items) + + +class ApifyTwitterScraperTool(_ApifyGenericTool): # type: ignore[override] + """Scrape tweets, profiles, or replies from Twitter/X. + + Uses the ``apidojo/twitter-scraper-lite`` Actor under the hood. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON string with two keys: ``run`` (dict with ``run_id``, ``status``, + ``dataset_id``, ``started_at``, ``finished_at``) and ``items`` (list + of tweet dicts). + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyTwitterScraperTool + + tool = ApifyTwitterScraperTool() + result = tool.invoke({ + "search_query": "apify", + "search_mode": "search", + "max_results": 20, + }) + """ + + name: str = 'apify_twitter_scraper' + description: str = ( + 'Scrape tweets from Twitter/X by search term, user handle, or tweet URL and return them as JSON.' + ' Required: search_query (str - search term, handle, or tweet URL).' + ' Optional: search_mode (one of "search", "user", "replies"; default "search"),' + f' max_results (int, default {_DEFAULT_SOCIAL_RESULTS_LIMIT}),' + ' start (str - ISO date, only return tweets newer than this date),' + ' end (str - ISO date, only return tweets older than this date),' + ' sort (one of "Latest", "Top" - sort order for results).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at) and items.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyTwitterScraperInput + + def _run( # noqa: PLR0913 + self, + search_query: str, + search_mode: TwitterSearchMode = 'search', + max_results: int = _DEFAULT_SOCIAL_RESULTS_LIMIT, + start: str | None = None, + end: str | None = None, + sort: TwitterSort | None = None, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.twitter_scrape( + search_query=search_query, + search_mode=search_mode, + max_results=self._clamp_items(max_results), + start=start, + end=end, + sort=sort, + timeout_secs=self.max_timeout_secs, + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + notice = _NOTICE_TWITTER_DEMO if _has_demo_items(items) else None + return self._envelope(run, items, notice=notice) + + +class ApifyTikTokScraperTool(_ApifyGenericTool): # type: ignore[override] + """Scrape TikTok videos, profiles, or hashtag content. + + Uses the ``clockworks/tiktok-scraper`` Actor under the hood. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON string with two keys: ``run`` (dict with ``run_id``, ``status``, + ``dataset_id``, ``started_at``, ``finished_at``) and ``items`` (list + of TikTok item dicts). + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyTikTokScraperTool + + tool = ApifyTikTokScraperTool() + result = tool.invoke({ + "search_query": "cooking", + "search_type": "search", + "max_results": 20, + }) + """ + + name: str = 'apify_tiktok_scraper' + description: str = ( + 'Scrape TikTok by search keyword, profile, hashtag, or post URL and return the results as JSON.' + ' Required: search_query (str - keyword, username, hashtag, or TikTok post URL).' + ' Optional: search_type (one of "search", "user", "hashtag", "post"; default "search"),' + f' max_results (int, default {_DEFAULT_SOCIAL_RESULTS_LIMIT}).' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at) and items.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyTikTokScraperInput + + def _run( + self, + search_query: str, + search_type: TikTokSearchType = 'search', + max_results: int = _DEFAULT_SOCIAL_RESULTS_LIMIT, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.tiktok_scrape( + search_query=search_query, + search_type=search_type, + max_results=self._clamp_items(max_results), + timeout_secs=self.max_timeout_secs, + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, items) + + +class ApifyFacebookPostsScraperTool(_ApifyGenericTool): # type: ignore[override] + """Scrape public Facebook page posts. + + Uses the ``apify/facebook-posts-scraper`` Actor under the hood. + Only public Facebook pages are supported - personal profiles cannot + be scraped. + + Args: + apify_token: Apify API token. Falls back to the ``APIFY_TOKEN`` + environment variable when *None*. + + Returns: + JSON string with two keys: ``run`` (dict with ``run_id``, ``status``, + ``dataset_id``, ``started_at``, ``finished_at``) and ``items`` (list + of post dicts). + + Example: + .. code-block:: python + + import os + os.environ["APIFY_TOKEN"] = "your-apify-token" + + from langchain_apify import ApifyFacebookPostsScraperTool + + tool = ApifyFacebookPostsScraperTool() + result = tool.invoke({ + "page_url": "https://www.facebook.com/humansofnewyork/", + "max_results": 20, + }) + """ + + name: str = 'apify_facebook_posts_scraper' + description: str = ( + 'Scrape posts from a public Facebook page and return them as JSON.' + ' Required: page_url (str - Facebook page URL; personal profiles are not supported).' + f' Optional: max_results (int, default {_DEFAULT_SOCIAL_RESULTS_LIMIT}),' + ' only_posts_newer_than (str - date filter, e.g. "2025-01-01" or "1 week"),' + ' only_posts_older_than (str - date filter, e.g. "2025-01-01" or "1 week").' + ' Returns JSON with keys: run (run_id, status, dataset_id, started_at, finished_at) and items.' + ' Use only the data returned; do not hallucinate missing fields.' + ) + args_schema: ArgsSchema | None = ApifyFacebookPostsScraperInput + + def _run( + self, + page_url: str, + max_results: int = _DEFAULT_SOCIAL_RESULTS_LIMIT, + only_posts_newer_than: str | None = None, + only_posts_older_than: str | None = None, + _run_manager: CallbackManagerForToolRun | None = None, + ) -> str: + try: + run, items = self._client.facebook_posts_scrape( + page_url=page_url, + max_results=self._clamp_items(max_results), + only_posts_newer_than=only_posts_newer_than, + only_posts_older_than=only_posts_older_than, + timeout_secs=self.max_timeout_secs, + ) + except _TOOL_RUN_ERRORS as exc: + raise ToolException(str(exc)) from exc + return self._envelope(run, items) + + +# Convenience tool-class list for selective agent binding. +APIFY_SOCIAL_TOOLS: list[type[BaseTool]] = [ + ApifyInstagramScraperTool, + ApifyLinkedInProfilePostsTool, + ApifyLinkedInProfileSearchTool, + ApifyLinkedInProfileDetailTool, + ApifyTwitterScraperTool, + ApifyTikTokScraperTool, + ApifyFacebookPostsScraperTool, +] diff --git a/langchain_apify/utils.py b/langchain_apify/utils.py deleted file mode 100644 index 8cdc835..0000000 --- a/langchain_apify/utils.py +++ /dev/null @@ -1,132 +0,0 @@ -from __future__ import annotations - -import string -from typing import TypeVar - -import requests -from apify_client import ApifyClientAsync -from apify_client.client import ApifyClient - -from langchain_apify.const import MAX_DESCRIPTION_LEN, REQUESTS_TIMEOUT_SECS - -APIFY_API_ENDPOINT_GET_DEFAULT_BUILD = 'https://api.apify.com/v2/acts/{actor_id}/builds/default' - - -def prune_actor_input_schema( - input_schema: dict, - max_description_len: int = MAX_DESCRIPTION_LEN, -) -> tuple[dict, list[str]]: - """Get the input schema from the Actor build. - - Trim the description to 250 characters. - - Args: - input_schema (dict): The input schema from the Actor build. - max_description_len (int): The maximum length of the description. - - Returns: - tuple[dict, list[str]]: A tuple containing the pruned properties - and required fields. - """ - properties = input_schema.get('properties', {}) - required = input_schema.get('required', []) - - properties_out: dict = {} - for item, meta in properties.items(): - properties_out[item] = {} - if desc := meta.get('description'): - properties_out[item]['description'] = ( - desc[:max_description_len] + '...' if len(desc) > max_description_len else desc - ) - for key_name in ('type', 'default', 'prefill', 'enum'): - if value := meta.get(key_name): - properties_out[item][key_name] = value - - return properties_out, required - - -T = TypeVar('T', ApifyClient, ApifyClientAsync) - - -def create_apify_client(client_cls: type[T], token: str) -> T: - """Create an Apify client instance with a custom user-agent. - - Args: - client_cls (ApifyClient | ApifyClientAsync): ApifyClient or ApifyClientAsync class. - token (str): API token. - - Returns: - T: ApifyClient or ApifyClientAsync instance. - - Raises: - ValueError: If the API token is not provided. - """ - if not token: - msg = 'API token is required to create an Apify client.' - raise ValueError(msg) - client = client_cls(token) - - # Check for new attribute names first (without 'x'), then fall back to old names (with 'x') - if isinstance(client, ApifyClientAsync): - http_client_attr = ( - 'httpx_async_client' if hasattr(client.http_client, 'httpx_async_client') else 'http_async_client' - ) - else: - http_client_attr = 'httpx_client' if hasattr(client.http_client, 'httpx_client') else 'http_client' - - if http_client := getattr(client.http_client, http_client_attr, None): - http_client.headers['user-agent'] += '; Origin/langchain' - return client - - -def actor_id_to_tool_name(actor_id: str) -> str: - """Turn actor_id into a valid tool name. - - Tool name must only contain letters, numbers, underscores, dashes, - and cannot contain spaces. - - Args: - actor_id (str): Actor ID from Apify store. - - Returns: - str: A valid tool name. - """ - valid_chars = string.ascii_letters + string.digits + '_-' - return 'apify_actor_' + ''.join(char if char in valid_chars else '_' for char in actor_id) - - -def get_actor_latest_build(apify_client: ApifyClient, actor_id: str) -> dict: - """Get the latest build of an Actor from the default build tag. - - Args: - apify_client (ApifyClient): An instance of the ApifyClient class. - actor_id (str): Actor name from Apify store to run. - - Returns: - dict: The latest build of the Actor. - - Raises: - ValueError: If the Actor is not found or the build data is not found. - TypeError: If the build is not a dictionary. - """ - if not (actor := apify_client.actor(actor_id).get()): - msg = f'Actor {actor_id} not found.' - raise ValueError(msg) - - if not (actor_obj_id := actor.get('id')): - msg = f'Failed to get the Actor object ID for {actor_id}.' - raise ValueError(msg) - - url = APIFY_API_ENDPOINT_GET_DEFAULT_BUILD.format(actor_id=actor_obj_id) - response = requests.request('GET', url, timeout=REQUESTS_TIMEOUT_SECS) - - build = response.json() - if not isinstance(build, dict): - msg = f'Failed to get the latest build of the Actor {actor_id}.' - raise TypeError(msg) - - if (data := build.get('data')) is None: - msg = f'Failed to get the latest build data of the Actor {actor_id}.' - raise ValueError(msg) - - return data diff --git a/langchain_apify/wrappers.py b/langchain_apify/wrappers.py index ef17873..f9fd42d 100644 --- a/langchain_apify/wrappers.py +++ b/langchain_apify/wrappers.py @@ -4,11 +4,15 @@ from typing import TYPE_CHECKING, Any from apify_client import ApifyClient, ApifyClientAsync -from langchain_core.utils import get_from_dict_or_env -from pydantic import BaseModel, ConfigDict, model_validator - +from pydantic import BaseModel, ConfigDict, Field, SecretStr, model_validator + +from langchain_apify._error_messages import _ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET +from langchain_apify._utils import ( + _apify_token_secret_factory, + _create_apify_client, + _resolve_deprecated_token, +) from langchain_apify.document_loaders import ApifyDatasetLoader -from langchain_apify.utils import create_apify_client if TYPE_CHECKING: from collections.abc import Callable @@ -19,9 +23,9 @@ class ApifyWrapper(BaseModel): """Wrapper around Apify client for LangChain. - To use, you should have the environment variable `APIFY_API_TOKEN` set - with your API key, or pass `apify_api_token` - as a named parameter to the constructor. + To use, you should have the environment variable ``APIFY_TOKEN`` set + with your API key, or pass ``apify_token`` as a named parameter to the + constructor. For details, see https://docs.apify.com/platform/integrations/langchain @@ -51,49 +55,56 @@ class ApifyWrapper(BaseModel): """ # allow arbitrary types in the model config for the apify client fields - model_config = ConfigDict(arbitrary_types_allowed=True) + model_config = ConfigDict(arbitrary_types_allowed=True, populate_by_name=True) - apify_client: ApifyClient - apify_client_async: ApifyClientAsync - apify_api_token: str | None = None + apify_token: SecretStr | None = Field( + default_factory=_apify_token_secret_factory, + description='Apify API token. Falls back to the APIFY_TOKEN environment variable when None.', + exclude=True, + repr=False, + ) + apify_client: ApifyClient = Field(default=None, exclude=True) # type: ignore[assignment] + apify_client_async: ApifyClientAsync = Field(default=None, exclude=True) # type: ignore[assignment] def __init__( self, - apify_api_token: str | None = None, + apify_token: str | SecretStr | None = None, *args: Any, # noqa: ANN401 **kwargs: Any, # noqa: ANN401 ) -> None: - """Initialize the loader with an Apify dataset ID and a mapping function. + """Initialise the wrapper. Args: - dataset_id (str): The ID of the dataset on the Apify platform. - dataset_mapping_function (Callable): A function that takes a single - dictionary (an Apify dataset item) and converts it to an instance - of the Document class. - apify_api_token (Optional[str]): Apify API token. - *args: Any: Additional positional arguments. - **kwargs: Any: Additional keyword arguments. + apify_token (Optional[str | SecretStr]): Apify API token. Falls + back to the ``APIFY_TOKEN`` environment variable when *None*. + apify_api_token: Deprecated alias for ``apify_token``. + *args: Any: Additional positional arguments forwarded to Pydantic. + **kwargs: Any: Additional keyword arguments forwarded to Pydantic. """ - kwargs.update({'apify_api_token': apify_api_token}) - super().__init__(*args, **kwargs) + if 'apify_api_token' in kwargs: + apify_token = _resolve_deprecated_token(apify_token, kwargs.pop('apify_api_token')) - @model_validator(mode='before') - @classmethod - def validate_environment(cls, values: dict) -> Any: # noqa: ANN401 - """Validate environment. + if apify_token is not None: + kwargs['apify_token'] = apify_token + super().__init__(*args, **kwargs) - Validate that an Apify API token is set and the apify-client - Python package exists in the current environment. + @model_validator(mode='after') + def _init_clients(self) -> ApifyWrapper: + """Validate the token and initialise both sync and async Apify clients. Returns: - Any: The validated values. - """ - apify_api_token = get_from_dict_or_env(values, 'apify_api_token', 'APIFY_API_TOKEN') - - values['apify_client'] = create_apify_client(ApifyClient, apify_api_token) - values['apify_client_async'] = create_apify_client(ApifyClientAsync, apify_api_token) + ApifyWrapper: The validated wrapper instance. - return values + Raises: + ValueError: If no token is provided and APIFY_TOKEN is not set. + """ + if self.apify_token is None: + msg = _ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET + raise ValueError(msg) + token = self.apify_token.get_secret_value() + self.apify_client = _create_apify_client(ApifyClient, token) + self.apify_client_async = _create_apify_client(ApifyClientAsync, token) + return self def call_actor( # noqa: PLR0913 self, @@ -140,6 +151,7 @@ def call_actor( # noqa: PLR0913 return ApifyDatasetLoader( dataset_id=actor_call['defaultDatasetId'], dataset_mapping_function=dataset_mapping_function, + apify_token=self.apify_token, ) async def acall_actor( # noqa: PLR0913 @@ -187,6 +199,7 @@ async def acall_actor( # noqa: PLR0913 return ApifyDatasetLoader( dataset_id=actor_call['defaultDatasetId'], dataset_mapping_function=dataset_mapping_function, + apify_token=self.apify_token, ) def call_actor_task( # noqa: PLR0913 @@ -235,6 +248,7 @@ def call_actor_task( # noqa: PLR0913 return ApifyDatasetLoader( dataset_id=task_call['defaultDatasetId'], dataset_mapping_function=dataset_mapping_function, + apify_token=self.apify_token, ) async def acall_actor_task( # noqa: PLR0913 @@ -283,4 +297,5 @@ async def acall_actor_task( # noqa: PLR0913 return ApifyDatasetLoader( dataset_id=task_call['defaultDatasetId'], dataset_mapping_function=dataset_mapping_function, + apify_token=self.apify_token, ) diff --git a/llms.txt b/llms.txt index a011fa6..a39a99a 100644 --- a/llms.txt +++ b/llms.txt @@ -10,11 +10,11 @@ To install the package, use pip: pip install langchain-apify ``` -Ensure you have set the `APIFY_API_TOKEN` environment variable with your Apify API token. +Ensure you have set the `APIFY_TOKEN` environment variable with your Apify API token. ```python import os -os.environ["APIFY_API_TOKEN"] = "YOUR_APIFY_API_TOKEN" +os.environ["APIFY_TOKEN"] = "YOUR_APIFY_TOKEN" ``` ## Key Imports @@ -110,5 +110,5 @@ documents = loader.load() **Note:** - This document assumes you're familiar with Python and LangChain basics. -- Adjust the `YOUR_APIFY_API_TOKEN` placeholder with your actual token or follow the instructions for setting environment variables. +- Adjust the `YOUR_APIFY_TOKEN` placeholder with your actual token or follow the instructions for setting environment variables. - The `dataset_mapping_function` is crucial for shaping the data into `Document` objects that LangChain can understand. Adjust based on the structure of your data. diff --git a/poetry.lock b/poetry.lock index 04123e8..bc99d20 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1,4 +1,4 @@ -# This file is automatically @generated by Poetry 2.2.1 and should not be changed by hand. +# This file is automatically @generated by Poetry 2.3.4 and should not be changed by hand. [[package]] name = "annotated-types" @@ -2840,4 +2840,4 @@ cffi = ["cffi (>=1.11)"] [metadata] lock-version = "2.1" python-versions = ">=3.10,<4.0" -content-hash = "2ae02e1a71f608d9e6c6429a4f87baa818c0582de958ce9227f09f048d63fba0" +content-hash = "f536d920e814b92fcc10e8a335ac79e0d20b3c877afbc3b5a125cafe50c74e72" diff --git a/pyproject.toml b/pyproject.toml index cc4f76d..237c4da 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -22,9 +22,9 @@ disallow_untyped_defs = true [tool.poetry.dependencies] python = ">=3.10,<4.0" -langchain-core = "^0.3.15" +langchain-core = ">=0.3.15,<2.0.0" apify-client = "^2.3.0" -eval-type-backport = "^0.2.2" +eval-type-backport = ">=0.2.2" [tool.ruff] line-length = 120 diff --git a/tests/integration_tests/test_document_loaders.py b/tests/integration_tests/test_document_loaders.py index 2fa2b2c..96eb4d1 100644 --- a/tests/integration_tests/test_document_loaders.py +++ b/tests/integration_tests/test_document_loaders.py @@ -1,10 +1,16 @@ -import os from collections.abc import Iterator +import pytest from apify_client import ApifyClient from langchain_core.documents import Document from langchain_apify import ApifyDatasetLoader +from langchain_apify._utils import _resolve_apify_token + +pytestmark = pytest.mark.skipif( + not _resolve_apify_token(), + reason='APIFY_TOKEN not set', +) def test_apify_dataset_loader_load() -> None: @@ -13,7 +19,7 @@ def test_apify_dataset_loader_load() -> None: Creates a new dataset, pushes items to it, and then loads the items using the loader. """ - token = os.getenv('APIFY_API_TOKEN') + token = _resolve_apify_token() client = ApifyClient(token=token) dataset_name = 'langchain-test-apify-dataset-loader-load' @@ -53,7 +59,7 @@ def test_apify_dataset_loader_lazy_load() -> None: Creates a new dataset, pushes items to it, and then loads the items using the loader. """ - token = os.getenv('APIFY_API_TOKEN') + token = _resolve_apify_token() client = ApifyClient(token=token) dataset_name = 'langchain-test-apify-dataset-loader-lazy-load' diff --git a/tests/integration_tests/test_generic_tools.py b/tests/integration_tests/test_generic_tools.py new file mode 100644 index 0000000..d2b3aba --- /dev/null +++ b/tests/integration_tests/test_generic_tools.py @@ -0,0 +1,97 @@ +"""Integration smoke tests for the generic Apify tools. + +These tests hit the real Apify API and require the ``APIFY_TOKEN`` +environment variable to be set (``APIFY_API_TOKEN`` is also accepted for +backwards compatibility). They use ``apify/python-example`` (a trivial +Actor that adds two numbers) to keep execution fast and cheap. +""" + +from __future__ import annotations + +import json +import os + +import pytest + +from langchain_apify import ( + ApifyGetDatasetItemsTool, + ApifyRunActorAndGetDatasetTool, + ApifyRunActorTool, + ApifyRunTaskAndGetDatasetTool, + ApifyRunTaskTool, + ApifyScrapeUrlTool, +) +from langchain_apify._utils import _resolve_apify_token + +_ACTOR_ID = 'apify/python-example' +_RUN_INPUT = {'first_number': 2, 'second_number': 3} + +pytestmark = pytest.mark.skipif( + not _resolve_apify_token(), + reason='APIFY_TOKEN not set', +) + + +def test_run_actor_tool_smoke() -> None: + tool = ApifyRunActorTool() + result = tool.invoke({'actor_id': _ACTOR_ID, 'run_input': _RUN_INPUT}) + + parsed = json.loads(result) + assert parsed['run']['status'] == 'SUCCEEDED' + assert parsed['run']['run_id'] + assert parsed['run']['dataset_id'] + + +def test_get_dataset_items_tool_smoke() -> None: + run_tool = ApifyRunActorTool() + run_result = json.loads(run_tool.invoke({'actor_id': _ACTOR_ID, 'run_input': _RUN_INPUT})) + dataset_id = run_result['run']['dataset_id'] + + items_tool = ApifyGetDatasetItemsTool() + result = items_tool.invoke({'dataset_id': dataset_id, 'limit': 10}) + + parsed = json.loads(result) + assert 'items' in parsed + assert isinstance(parsed['items'], list) + + +def test_run_actor_and_get_items_tool_smoke() -> None: + tool = ApifyRunActorAndGetDatasetTool() + result = tool.invoke({'actor_id': _ACTOR_ID, 'run_input': _RUN_INPUT}) + + parsed = json.loads(result) + assert parsed['run']['status'] == 'SUCCEEDED' + assert isinstance(parsed['items'], list) + + +def test_scrape_url_tool_smoke() -> None: + tool = ApifyScrapeUrlTool() + result = tool.invoke({'url': 'https://crawlee.dev'}) + + parsed = json.loads(result) + assert parsed['run']['status'] == 'SUCCEEDED' + assert parsed['items'][0]['content'] + + +_TASK_ID = os.getenv('APIFY_TASK_ID', '') + + +@pytest.mark.skipif(not _TASK_ID, reason='APIFY_TASK_ID not set') +def test_run_task_tool_smoke() -> None: + tool = ApifyRunTaskTool() + result = tool.invoke({'task_id': _TASK_ID}) + + parsed = json.loads(result) + assert parsed['run']['status'] == 'SUCCEEDED' + assert parsed['run']['run_id'] + assert parsed['run']['dataset_id'] + + +@pytest.mark.skipif(not _TASK_ID, reason='APIFY_TASK_ID not set') +def test_run_task_and_get_items_tool_smoke() -> None: + tool = ApifyRunTaskAndGetDatasetTool() + result = tool.invoke({'task_id': _TASK_ID}) + + parsed = json.loads(result) + assert parsed['run']['status'] == 'SUCCEEDED' + assert isinstance(parsed['items'], list) diff --git a/tests/integration_tests/test_tools.py b/tests/integration_tests/test_tools.py index 084cbb6..6b4369e 100644 --- a/tests/integration_tests/test_tools.py +++ b/tests/integration_tests/test_tools.py @@ -3,13 +3,20 @@ import json from typing import TYPE_CHECKING +import pytest from langchain_tests.integration_tests import ToolsIntegrationTests +from langchain_apify._utils import _resolve_apify_token from langchain_apify.tools import ApifyActorsTool if TYPE_CHECKING: from langchain_core.tools import BaseTool +pytestmark = pytest.mark.skipif( + not _resolve_apify_token(), + reason='APIFY_TOKEN not set', +) + class TestApifyActorsToolIntegration(ToolsIntegrationTests): """Integration tests for the ApifyActorsTool. diff --git a/tests/integration_tests/test_utils.py b/tests/integration_tests/test_utils.py index 1107c7a..9d6bc9c 100644 --- a/tests/integration_tests/test_utils.py +++ b/tests/integration_tests/test_utils.py @@ -1,24 +1,22 @@ -import os - +import pytest from apify_client.client import ApifyClient -from langchain_apify.error_messages import ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET -from langchain_apify.utils import create_apify_client, get_actor_latest_build +from langchain_apify._error_messages import _ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET +from langchain_apify._utils import _create_apify_client, _get_actor_latest_build, _resolve_apify_token def test_get_actor_latest_build() -> None: """Tests the get_actor_latest_build function. Raises: - ValueError: If the APIFY_API_TOKEN environment variable is not set. + ValueError: If the APIFY_TOKEN environment variable is not set. """ - if (token := os.getenv('APIFY_API_TOKEN')) is None: - msg = ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET - raise ValueError(msg) + if (token := _resolve_apify_token()) is None: + pytest.skip(_ERROR_APIFY_TOKEN_ENV_VAR_NOT_SET) - apify_client = create_apify_client(ApifyClient, token) + apify_client = _create_apify_client(ApifyClient, token) - build = get_actor_latest_build(apify_client, 'apify/rag-web-browser') + build = _get_actor_latest_build(apify_client, 'apify/rag-web-browser') assert isinstance(build, dict) assert 'id' in build diff --git a/tests/integration_tests/test_wrappers.py b/tests/integration_tests/test_wrappers.py index 73a8f40..3c2ee19 100644 --- a/tests/integration_tests/test_wrappers.py +++ b/tests/integration_tests/test_wrappers.py @@ -1,6 +1,13 @@ +import pytest from langchain_core.documents import Document from langchain_apify import ApifyWrapper +from langchain_apify._utils import _resolve_apify_token + +pytestmark = pytest.mark.skipif( + not _resolve_apify_token(), + reason='APIFY_TOKEN not set', +) def test_apify_wrapper_call_actor() -> None: diff --git a/tests/unit_tests/conftest.py b/tests/unit_tests/conftest.py new file mode 100644 index 0000000..3bea881 --- /dev/null +++ b/tests/unit_tests/conftest.py @@ -0,0 +1,51 @@ +from __future__ import annotations + +from typing import Any +from unittest.mock import MagicMock, patch + +import pytest + +from langchain_apify._client import ApifyToolsClient + +SUCCEEDED_RUN: dict = { + 'id': 'run-abc', + 'status': 'SUCCEEDED', + 'defaultDatasetId': 'dataset-xyz', + 'startedAt': '2025-01-01T00:00:00.000Z', + 'finishedAt': '2025-01-01T00:01:00.000Z', +} + +FAILED_RUN: dict = { + 'id': 'run-fail', + 'status': 'FAILED', + 'defaultDatasetId': 'dataset-xyz', +} + +SAMPLE_ITEMS: list[dict] = [ + {'text': 'item-1', 'url': 'https://example.com/1'}, + {'text': 'item-2', 'url': 'https://example.com/2'}, +] + + +@pytest.fixture +def mock_tools_client() -> MagicMock: + return MagicMock(spec=ApifyToolsClient) + + +@pytest.fixture +def mock_apify_client() -> MagicMock: + return MagicMock() + + +@pytest.fixture +def client(mock_apify_client: MagicMock) -> ApifyToolsClient: + with patch('langchain_apify._client._create_apify_client', return_value=mock_apify_client): + return ApifyToolsClient(apify_token='dummy-token') + + +def make_tool(tool_cls: type, mock_client: MagicMock, **kwargs: Any) -> Any: # noqa: ANN401 + """Instantiate a generic tool with a mocked ApifyToolsClient.""" + with patch.object(ApifyToolsClient, '__init__', return_value=None): + tool = tool_cls(apify_token='dummy-token', **kwargs) + tool._client = mock_client + return tool diff --git a/tests/unit_tests/test_actor_tools.py b/tests/unit_tests/test_actor_tools.py new file mode 100644 index 0000000..7d89dd7 --- /dev/null +++ b/tests/unit_tests/test_actor_tools.py @@ -0,0 +1,1023 @@ +from __future__ import annotations + +import json +from unittest.mock import MagicMock, patch + +import pytest +from langchain_core.tools import ToolException +from pydantic import SecretStr + +from langchain_apify import ( + APIFY_SEARCH_TOOLS, + ApifyEcommerceScraperTool, + ApifyFacebookPostsScraperTool, + ApifyGoogleMapsTool, + ApifyGoogleSearchTool, + ApifyInstagramScraperTool, + ApifyLinkedInProfileDetailTool, + ApifyLinkedInProfilePostsTool, + ApifyLinkedInProfileSearchTool, + ApifyRAGWebBrowserTool, + ApifyTikTokScraperTool, + ApifyTwitterScraperTool, + ApifyWebCrawlerTool, + ApifyYouTubeScraperTool, +) +from langchain_apify._client import ApifyToolsClient +from langchain_apify._constants import _RAG_MAX_RESULTS_CAP +from langchain_apify._error_messages import _NOTICE_TWITTER_DEMO +from langchain_apify.tools.base import _ApifyGenericTool +from langchain_apify.tools.social import _has_demo_items +from tests.unit_tests.conftest import SAMPLE_ITEMS, SUCCEEDED_RUN, make_tool + +# --------------------------------------------------------------------------- +# ApifyGoogleSearchTool +# --------------------------------------------------------------------------- + + +def test_google_search_tool_returns_json(mock_tools_client: MagicMock) -> None: + mock_tools_client.google_search.return_value = ( + SUCCEEDED_RUN, + [ + {'title': 'Result 1', 'url': 'https://example.com/1', 'description': 'Desc 1'}, + {'title': 'Result 2', 'url': 'https://example.com/2', 'description': 'Desc 2'}, + ], + ) + tool = make_tool(ApifyGoogleSearchTool, mock_tools_client) + + result = tool._run(query='test query') + + parsed = json.loads(result) + assert parsed['run']['status'] == 'SUCCEEDED' + assert len(parsed['items']) == 2 + assert parsed['items'][0]['title'] == 'Result 1' + assert parsed['items'][1]['url'] == 'https://example.com/2' + + +def test_google_search_tool_passes_params(mock_tools_client: MagicMock) -> None: + mock_tools_client.google_search.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyGoogleSearchTool, mock_tools_client) + + tool._run(query='test', max_results=5, country_code='us', language_code='en', timeout_secs=120) + + mock_tools_client.google_search.assert_called_once_with( + 'test', + max_results=5, + country_code='us', + language_code='en', + timeout_secs=120, + ) + + +def test_google_search_tool_clamps_timeout(mock_tools_client: MagicMock) -> None: + mock_tools_client.google_search.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyGoogleSearchTool, mock_tools_client, max_timeout_secs=60) + + tool._run(query='test', timeout_secs=9999) + + assert mock_tools_client.google_search.call_args.kwargs['timeout_secs'] == 60 + + +def test_google_search_tool_clamps_max_results(mock_tools_client: MagicMock) -> None: + mock_tools_client.google_search.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyGoogleSearchTool, mock_tools_client, max_items=3) + + tool._run(query='test', max_results=100) + + call_kwargs = mock_tools_client.google_search.call_args + assert call_kwargs.kwargs['max_results'] == 3 + + +def test_google_search_tool_empty_results(mock_tools_client: MagicMock) -> None: + mock_tools_client.google_search.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyGoogleSearchTool, mock_tools_client) + + result = tool._run(query='nothing') + + parsed = json.loads(result) + assert parsed['items'] == [] + assert parsed['run']['status'] == 'SUCCEEDED' + + +def test_google_search_tool_failure_raises_tool_exception(mock_tools_client: MagicMock) -> None: + mock_tools_client.google_search.side_effect = RuntimeError('Actor run run-bad ended with status FAILED.') + tool = make_tool(ApifyGoogleSearchTool, mock_tools_client) + + with pytest.raises(ToolException, match='FAILED'): + tool._run(query='test') + + +def test_google_search_tool_missing_token(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + ApifyGoogleSearchTool() + + +@pytest.mark.parametrize('bad_code', ['USA', 'english', 'u', 'us1', '']) +def test_google_search_tool_rejects_malformed_locale(mock_tools_client: MagicMock, bad_code: str) -> None: + """country_code and language_code must be exactly two letters.""" + tool = make_tool(ApifyGoogleSearchTool, mock_tools_client) + + with pytest.raises(ValueError, match='string_pattern_mismatch|String should match pattern'): + tool.invoke({'query': 'test', 'country_code': bad_code}) + + with pytest.raises(ValueError, match='string_pattern_mismatch|String should match pattern'): + tool.invoke({'query': 'test', 'language_code': bad_code}) + + +@pytest.mark.parametrize('raw_country', ['us', 'US', 'Us', 'uS']) +def test_google_search_tool_normalises_country_code_to_lower(mock_tools_client: MagicMock, raw_country: str) -> None: + mock_tools_client.google_search.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyGoogleSearchTool, mock_tools_client) + + tool.invoke({'query': 'test', 'country_code': raw_country}) + + assert mock_tools_client.google_search.call_args.kwargs['country_code'] == 'us' + + +@pytest.mark.parametrize('raw_language', ['en', 'EN', 'En', 'eN']) +def test_google_search_tool_normalises_language_code_to_lower(mock_tools_client: MagicMock, raw_language: str) -> None: + mock_tools_client.google_search.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyGoogleSearchTool, mock_tools_client) + + tool.invoke({'query': 'test', 'language_code': raw_language}) + + assert mock_tools_client.google_search.call_args.kwargs['language_code'] == 'en' + + +# --------------------------------------------------------------------------- +# ApifyWebCrawlerTool +# --------------------------------------------------------------------------- + + +def test_web_crawler_tool_returns_json(mock_tools_client: MagicMock) -> None: + mock_tools_client.crawl_website.return_value = ( + SUCCEEDED_RUN, + [ + {'url': 'https://example.com/', 'markdown': '# Home', 'text': 'Home', 'metadata': {'title': 'Home'}}, + {'url': 'https://example.com/about', 'markdown': '', 'text': 'About us', 'metadata': {'title': 'About'}}, + ], + ) + tool = make_tool(ApifyWebCrawlerTool, mock_tools_client) + + result = tool._run(url='https://example.com') + + parsed = json.loads(result) + assert parsed['run']['status'] == 'SUCCEEDED' + assert len(parsed['items']) == 2 + assert parsed['items'][0] == {'url': 'https://example.com/', 'title': 'Home', 'content': '# Home'} + assert parsed['items'][1] == {'url': 'https://example.com/about', 'title': 'About', 'content': 'About us'} + + +def test_web_crawler_tool_passes_params(mock_tools_client: MagicMock) -> None: + mock_tools_client.crawl_website.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyWebCrawlerTool, mock_tools_client) + + tool._run( + url='https://example.com', + max_crawl_pages=5, + max_crawl_depth=2, + crawler_type='playwright:firefox', + timeout_secs=120, + ) + + mock_tools_client.crawl_website.assert_called_once_with( + 'https://example.com', + max_crawl_pages=5, + max_crawl_depth=2, + crawler_type='playwright:firefox', + timeout_secs=120, + ) + + +def test_web_crawler_tool_clamps_pages_and_timeout(mock_tools_client: MagicMock) -> None: + mock_tools_client.crawl_website.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyWebCrawlerTool, mock_tools_client, max_items=3, max_timeout_secs=60) + + tool._run(url='https://example.com', max_crawl_pages=100, timeout_secs=9999) + + call_kwargs = mock_tools_client.crawl_website.call_args + assert call_kwargs.kwargs['max_crawl_pages'] == 3 + assert call_kwargs.kwargs['timeout_secs'] == 60 + + +def test_web_crawler_tool_clamps_depth(mock_tools_client: MagicMock) -> None: + mock_tools_client.crawl_website.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyWebCrawlerTool, mock_tools_client, max_crawl_depth=2) + + tool._run(url='https://example.com', max_crawl_depth=999) + assert mock_tools_client.crawl_website.call_args.kwargs['max_crawl_depth'] == 2 + + mock_tools_client.crawl_website.reset_mock() + tool._run(url='https://example.com', max_crawl_depth=-1) + assert mock_tools_client.crawl_website.call_args.kwargs['max_crawl_depth'] == 0 + + +def test_web_crawler_tool_empty_results(mock_tools_client: MagicMock) -> None: + mock_tools_client.crawl_website.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyWebCrawlerTool, mock_tools_client) + + result = tool._run(url='https://example.com') + + parsed = json.loads(result) + assert parsed['items'] == [] + assert parsed['run']['status'] == 'SUCCEEDED' + + +def test_web_crawler_tool_failure_raises_tool_exception(mock_tools_client: MagicMock) -> None: + mock_tools_client.crawl_website.side_effect = RuntimeError('Actor run run-bad ended with status TIMED-OUT.') + tool = make_tool(ApifyWebCrawlerTool, mock_tools_client) + + with pytest.raises(ToolException, match='TIMED-OUT'): + tool._run(url='https://example.com') + + +def test_web_crawler_tool_missing_token(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + ApifyWebCrawlerTool() + + +# --------------------------------------------------------------------------- +# Search & Crawling tools - happy paths +# --------------------------------------------------------------------------- + + +def test_rag_web_browser_tool_returns_json(mock_tools_client: MagicMock) -> None: + items = [ + { + 'crawledUrl': 'https://example.com/1', + 'metadata': {'url': 'https://example.com/1', 'title': 'Page 1'}, + 'markdown': '# Page 1', + 'text': 'Page 1 plain', + }, + { + 'crawledUrl': 'https://example.com/2', + 'metadata': {'title': 'Page 2'}, + 'text': 'Page 2 plain', + }, + ] + mock_tools_client.rag_web_search.return_value = (SUCCEEDED_RUN, items) + tool = make_tool(ApifyRAGWebBrowserTool, mock_tools_client) + + parsed = json.loads(tool._run(query='what is langchain', max_results=3)) + + assert parsed['items'] == [ + {'url': 'https://example.com/1', 'title': 'Page 1', 'content': '# Page 1'}, + {'url': 'https://example.com/2', 'title': 'Page 2', 'content': 'Page 2 plain'}, + ] + assert parsed['run']['status'] == 'SUCCEEDED' + mock_tools_client.rag_web_search.assert_called_once_with( + 'what is langchain', + max_results=3, + timeout_secs=tool.max_timeout_secs, + ) + + +def test_rag_web_browser_tool_clamps_max_results(mock_tools_client: MagicMock) -> None: + # The rag-web-browser Actor rejects maxResults > 100, so the tool's max_items + # ceiling defaults to that cap and _clamp_items must clamp larger requests. + mock_tools_client.rag_web_search.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyRAGWebBrowserTool, mock_tools_client) + + tool._run(query='q', max_results=_RAG_MAX_RESULTS_CAP + 50) + + assert mock_tools_client.rag_web_search.call_args.kwargs['max_results'] == _RAG_MAX_RESULTS_CAP + + +def test_google_maps_tool_returns_json(mock_tools_client: MagicMock) -> None: + items = [{'name': 'Cafe A', 'address': 'Berlin'}] + mock_tools_client.google_maps_search.return_value = (SUCCEEDED_RUN, items) + tool = make_tool(ApifyGoogleMapsTool, mock_tools_client) + + parsed = json.loads(tool._run(query='cafe in Berlin', max_results=2, language='en')) + + assert parsed['run']['dataset_id'] == SUCCEEDED_RUN['defaultDatasetId'] + assert parsed['items'] == items + mock_tools_client.google_maps_search.assert_called_once_with( + 'cafe in Berlin', + max_results=2, + language='en', + timeout_secs=tool.max_timeout_secs, + ) + + +def test_youtube_tool_returns_json(mock_tools_client: MagicMock) -> None: + items = [{'title': 'Vid 1'}] + mock_tools_client.youtube_scrape.return_value = (SUCCEEDED_RUN, items) + tool = make_tool(ApifyYouTubeScraperTool, mock_tools_client) + + parsed = json.loads(tool._run(search_query='langchain', search_type='search', max_results=4)) + + assert parsed['items'] == items + mock_tools_client.youtube_scrape.assert_called_once_with( + search_query='langchain', + search_type='search', + max_results=4, + timeout_secs=tool.max_timeout_secs, + ) + + +def test_youtube_tool_invalid_search_type_raises_tool_exception(mock_tools_client: MagicMock) -> None: + mock_tools_client.youtube_scrape.side_effect = ValueError('Invalid search_type playlist') + tool = make_tool(ApifyYouTubeScraperTool, mock_tools_client) + + with pytest.raises(ToolException, match='Invalid search_type'): + tool._run(search_query='x', search_type='search') + + +def test_ecommerce_tool_returns_json(mock_tools_client: MagicMock) -> None: + items = [{'sku': 'A1', 'price': 9.99}] + mock_tools_client.ecommerce_scrape.return_value = (SUCCEEDED_RUN, items) + tool = make_tool(ApifyEcommerceScraperTool, mock_tools_client) + + parsed = json.loads(tool._run(url='https://shop.example.com/p/123', max_results=5)) + + assert parsed['items'] == items + mock_tools_client.ecommerce_scrape.assert_called_once_with( + 'https://shop.example.com/p/123', + url_type='product', + max_results=5, + timeout_secs=tool.max_timeout_secs, + ) + + +def test_ecommerce_tool_category_mode_passes_url_type(mock_tools_client: MagicMock) -> None: + items = [{'sku': 'B2', 'price': 19.99}] + mock_tools_client.ecommerce_scrape.return_value = (SUCCEEDED_RUN, items) + tool = make_tool(ApifyEcommerceScraperTool, mock_tools_client) + + parsed = json.loads(tool._run(url='https://shop.example.com/cat/42', url_type='category', max_results=8)) + + assert parsed['items'] == items + mock_tools_client.ecommerce_scrape.assert_called_once_with( + 'https://shop.example.com/cat/42', + url_type='category', + max_results=8, + timeout_secs=tool.max_timeout_secs, + ) + + +def test_ecommerce_tool_invalid_url_type_raises_tool_exception(mock_tools_client: MagicMock) -> None: + mock_tools_client.ecommerce_scrape.side_effect = ValueError('Invalid url_type listing') + tool = make_tool(ApifyEcommerceScraperTool, mock_tools_client) + + with pytest.raises(ToolException, match='Invalid url_type'): + tool._run(url='https://shop.example.com', url_type='product') + + +# --------------------------------------------------------------------------- +# US-4 Search & Crawling tools - parametrized error / empty / handle_tool_error +# --------------------------------------------------------------------------- + +# Each entry: (tool_class, helper_attribute_name, kwargs_for_run) +_TOOL_INVOCATIONS: list[tuple[type[_ApifyGenericTool], str, dict]] = [ + (ApifyGoogleSearchTool, 'google_search', {'query': 'q'}), + (ApifyWebCrawlerTool, 'crawl_website', {'url': 'https://example.com'}), + (ApifyRAGWebBrowserTool, 'rag_web_search', {'query': 'q'}), + (ApifyGoogleMapsTool, 'google_maps_search', {'query': 'q'}), + (ApifyYouTubeScraperTool, 'youtube_scrape', {'search_query': 'q'}), + (ApifyEcommerceScraperTool, 'ecommerce_scrape', {'url': 'https://example.com'}), +] + +# Tools that return the {run, items} envelope on success. +_ENVELOPE_TOOL_INVOCATIONS: list[tuple[type[_ApifyGenericTool], str, dict]] = [ + (ApifyGoogleSearchTool, 'google_search', {'query': 'q'}), + (ApifyWebCrawlerTool, 'crawl_website', {'url': 'https://example.com'}), + (ApifyGoogleMapsTool, 'google_maps_search', {'query': 'q'}), + (ApifyYouTubeScraperTool, 'youtube_scrape', {'search_query': 'q'}), + (ApifyEcommerceScraperTool, 'ecommerce_scrape', {'url': 'https://example.com'}), +] + + +@pytest.mark.parametrize(('tool_cls', 'helper_attr', 'run_kwargs'), _TOOL_INVOCATIONS) +def test_search_tool_runtime_error_raises_tool_exception( + mock_tools_client: MagicMock, + tool_cls: type, + helper_attr: str, + run_kwargs: dict, +) -> None: + getattr(mock_tools_client, helper_attr).side_effect = RuntimeError('Actor run run-bad ended with status FAILED.') + tool = make_tool(tool_cls, mock_tools_client) + + with pytest.raises(ToolException, match='FAILED'): + tool._run(**run_kwargs) + + +@pytest.mark.parametrize(('tool_cls', 'helper_attr', 'run_kwargs'), _ENVELOPE_TOOL_INVOCATIONS) +def test_search_tool_empty_dataset_returns_empty_items( + mock_tools_client: MagicMock, + tool_cls: type, + helper_attr: str, + run_kwargs: dict, +) -> None: + getattr(mock_tools_client, helper_attr).return_value = (SUCCEEDED_RUN, []) + tool = make_tool(tool_cls, mock_tools_client) + + parsed = json.loads(tool._run(**run_kwargs)) + assert parsed['items'] == [] + assert parsed['run']['status'] == 'SUCCEEDED' + + +def test_rag_web_browser_tool_empty_dataset_returns_empty_array(mock_tools_client: MagicMock) -> None: + mock_tools_client.rag_web_search.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyRAGWebBrowserTool, mock_tools_client) + + parsed = json.loads(tool._run(query='q')) + assert parsed['items'] == [] + assert parsed['run']['status'] == 'SUCCEEDED' + + +@pytest.mark.parametrize(('tool_cls', 'helper_attr', 'run_kwargs'), _TOOL_INVOCATIONS) +def test_search_tool_handle_tool_error_swallows( + mock_tools_client: MagicMock, + tool_cls: type, + helper_attr: str, + run_kwargs: dict, +) -> None: + """``handle_tool_error=True`` (inherited) means ``invoke`` returns the error string.""" + getattr(mock_tools_client, helper_attr).side_effect = RuntimeError('Actor run run-bad ended with status FAILED.') + tool = make_tool(tool_cls, mock_tools_client) + + result = tool.invoke(run_kwargs) + assert 'FAILED' in result + + +@pytest.mark.parametrize(('tool_cls', 'helper_attr', 'run_kwargs'), _TOOL_INVOCATIONS) +def test_search_tool_missing_token( + monkeypatch: pytest.MonkeyPatch, + tool_cls: type, + helper_attr: str, # noqa: ARG001 + run_kwargs: dict, # noqa: ARG001 +) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + tool_cls() + + +def test_search_tools_inherit_from_generic_base() -> None: + for tool_cls, _, _ in _TOOL_INVOCATIONS: + assert issubclass(tool_cls, _ApifyGenericTool), f'{tool_cls.__name__} must extend _ApifyGenericTool' + + +def test_search_tools_have_correct_metadata() -> None: + cases: list[tuple[type, str]] = [ + (ApifyGoogleSearchTool, 'apify_google_search'), + (ApifyWebCrawlerTool, 'apify_web_crawler'), + (ApifyRAGWebBrowserTool, 'apify_rag_web_browser'), + (ApifyGoogleMapsTool, 'apify_google_maps'), + (ApifyYouTubeScraperTool, 'apify_youtube_scraper'), + (ApifyEcommerceScraperTool, 'apify_ecommerce_scraper'), + ] + with patch.object(ApifyToolsClient, '__init__', return_value=None): + for tool_cls, expected_name in cases: + tool = tool_cls(apify_token=SecretStr('dummy')) + assert tool.name == expected_name + assert tool.description + assert tool.args_schema is not None + assert tool.handle_tool_error is True + + +def test_apify_search_tools_list() -> None: + assert set(APIFY_SEARCH_TOOLS) == { + ApifyGoogleSearchTool, + ApifyWebCrawlerTool, + ApifyRAGWebBrowserTool, + ApifyGoogleMapsTool, + ApifyYouTubeScraperTool, + ApifyEcommerceScraperTool, + } + assert len(APIFY_SEARCH_TOOLS) == 6 + + +# --------------------------------------------------------------------------- +# Regression: dataset items containing datetime values must not break JSON +# serialisation. The Apify client's clean=True deserialiser returns datetime +# objects for certain timestamp fields (notably Google Maps reviews and +# YouTube publishedAt), which previously raised +# ``TypeError: Object of type datetime is not JSON serializable`` inside +# ``_serialize_tool_response``. +# --------------------------------------------------------------------------- + + +# Tools that hand the client's items list straight to _serialize_tool_response, +# i.e. those most exposed to raw datetime values from the Actor's dataset. +_RETURN_LIST = 'list' +_RETURN_ENVELOPE = 'envelope' + +# Each entry: (tool_cls, client_helper_attr, run_kwargs, client_return_shape). +# Listed tools hand the client's items straight to _serialize_tool_response, +# i.e. they are most exposed to raw datetime values from the Actor's dataset. +_PASSTHROUGH_TOOL_INVOCATIONS: list[tuple[type[_ApifyGenericTool], str, dict, str]] = [ + (ApifyGoogleSearchTool, 'google_search', {'query': 'q'}, _RETURN_ENVELOPE), + (ApifyGoogleMapsTool, 'google_maps_search', {'query': 'q'}, _RETURN_ENVELOPE), + (ApifyYouTubeScraperTool, 'youtube_scrape', {'search_query': 'q'}, _RETURN_ENVELOPE), + (ApifyEcommerceScraperTool, 'ecommerce_scrape', {'url': 'https://example.com'}, _RETURN_ENVELOPE), +] + + +@pytest.mark.parametrize( + ('tool_cls', 'helper_attr', 'run_kwargs', 'client_return_shape'), + _PASSTHROUGH_TOOL_INVOCATIONS, +) +def test_search_tool_serialises_datetime_in_items( + mock_tools_client: MagicMock, + tool_cls: type, + helper_attr: str, + run_kwargs: dict, + client_return_shape: str, +) -> None: + from datetime import datetime, timezone + + timestamp = datetime(2026, 1, 2, 3, 4, 5, tzinfo=timezone.utc) + item_with_datetime = {'id': 'item-1', 'published_at': timestamp, 'text': 'hi'} + items = [item_with_datetime] + + if client_return_shape == _RETURN_ENVELOPE: + getattr(mock_tools_client, helper_attr).return_value = (SUCCEEDED_RUN, items) + else: + getattr(mock_tools_client, helper_attr).return_value = items + tool = make_tool(tool_cls, mock_tools_client) + + result = tool._run(**run_kwargs) + parsed = json.loads(result) + + assert isinstance(parsed['items'], list) + assert len(parsed['items']) == 1 + assert parsed['items'][0]['id'] == 'item-1' + assert isinstance(parsed['items'][0]['published_at'], str) + assert '2026-01-02' in parsed['items'][0]['published_at'] + + +# --------------------------------------------------------------------------- +# Social media Actor tools +# --------------------------------------------------------------------------- + + +EXPECTED_RUN_META: dict = { + 'run_id': 'run-abc', + 'status': 'SUCCEEDED', + 'dataset_id': 'dataset-xyz', + 'started_at': '2025-01-01T00:00:00.000Z', + 'finished_at': '2025-01-01T00:01:00.000Z', +} + + +# --------------------------------------------------------------------------- +# Missing token (shared base behavior) +# --------------------------------------------------------------------------- + + +def test_missing_token_raises(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + ApifyInstagramScraperTool() + + +# --------------------------------------------------------------------------- +# ApifyInstagramScraperTool +# --------------------------------------------------------------------------- + + +def test_instagram_tool_happy_path(mock_tools_client: MagicMock) -> None: + mock_tools_client.instagram_scrape.return_value = (SUCCEEDED_RUN, SAMPLE_ITEMS) + tool = make_tool(ApifyInstagramScraperTool, mock_tools_client) + + result = tool._run(search_type='user', search_query='apify', max_results=10) + + parsed = json.loads(result) + assert parsed['run'] == EXPECTED_RUN_META + assert parsed['items'] == SAMPLE_ITEMS + + +def test_instagram_tool_passes_params(mock_tools_client: MagicMock) -> None: + mock_tools_client.instagram_scrape.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyInstagramScraperTool, mock_tools_client) + + tool._run( + search_type='hashtag', + search_query='#travel', + max_results=5, + only_posts_newer_than='1 week', + ) + + mock_tools_client.instagram_scrape.assert_called_once_with( + search_type='hashtag', + search_query='#travel', + max_results=5, + only_posts_newer_than='1 week', + timeout_secs=600, + ) + + +def test_instagram_tool_clamps_max_results(mock_tools_client: MagicMock) -> None: + mock_tools_client.instagram_scrape.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyInstagramScraperTool, mock_tools_client, max_items=3) + + tool._run(search_type='user', search_query='apify', max_results=100) + + assert mock_tools_client.instagram_scrape.call_args.kwargs['max_results'] == 3 + + +def test_instagram_tool_runtime_error_raises_tool_exception(mock_tools_client: MagicMock) -> None: + mock_tools_client.instagram_scrape.side_effect = RuntimeError('Actor run run-X ended with status FAILED.') + tool = make_tool(ApifyInstagramScraperTool, mock_tools_client) + + with pytest.raises(ToolException, match='run-X'): + tool._run(search_type='user', search_query='apify') + + +# --------------------------------------------------------------------------- +# ApifyLinkedInProfilePostsTool +# --------------------------------------------------------------------------- + + +def test_linkedin_posts_tool_happy_path(mock_tools_client: MagicMock) -> None: + mock_tools_client.linkedin_profile_posts.return_value = (SUCCEEDED_RUN, SAMPLE_ITEMS) + tool = make_tool(ApifyLinkedInProfilePostsTool, mock_tools_client) + + result = tool._run(profile_url='satyanadella', max_results=10) + parsed = json.loads(result) + + assert parsed['run'] == EXPECTED_RUN_META + assert parsed['items'] == SAMPLE_ITEMS + mock_tools_client.linkedin_profile_posts.assert_called_once_with( + profile_url='satyanadella', + max_results=10, + timeout_secs=600, + ) + + +def test_linkedin_posts_tool_clamps_max_results(mock_tools_client: MagicMock) -> None: + mock_tools_client.linkedin_profile_posts.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyLinkedInProfilePostsTool, mock_tools_client, max_items=5) + + tool._run(profile_url='satyanadella', max_results=999) + + assert mock_tools_client.linkedin_profile_posts.call_args.kwargs['max_results'] == 5 + + +# --------------------------------------------------------------------------- +# ApifyLinkedInProfileSearchTool +# --------------------------------------------------------------------------- + + +def test_linkedin_search_tool_happy_path(mock_tools_client: MagicMock) -> None: + mock_tools_client.linkedin_profile_search.return_value = (SUCCEEDED_RUN, SAMPLE_ITEMS) + tool = make_tool(ApifyLinkedInProfileSearchTool, mock_tools_client) + + result = tool._run(query='Founder', max_results=10) + parsed = json.loads(result) + + assert parsed['items'] == SAMPLE_ITEMS + mock_tools_client.linkedin_profile_search.assert_called_once_with( + query='Founder', + max_results=10, + timeout_secs=600, + ) + + +def test_linkedin_search_tool_default_max_results(mock_tools_client: MagicMock) -> None: + mock_tools_client.linkedin_profile_search.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyLinkedInProfileSearchTool, mock_tools_client) + + tool._run(query='CTO') + + assert mock_tools_client.linkedin_profile_search.call_args.kwargs['max_results'] == 10 + + +# --------------------------------------------------------------------------- +# ApifyLinkedInProfileDetailTool +# --------------------------------------------------------------------------- + + +def test_linkedin_detail_tool_happy_path(mock_tools_client: MagicMock) -> None: + profile_item = [{'firstName': 'Neal', 'lastName': 'Mohan'}] + mock_tools_client.linkedin_profile_detail.return_value = (SUCCEEDED_RUN, profile_item) + tool = make_tool(ApifyLinkedInProfileDetailTool, mock_tools_client) + + result = tool._run(profile_url='neal-mohan', include_email=True) + parsed = json.loads(result) + + assert parsed['run'] == EXPECTED_RUN_META + assert parsed['items'] == profile_item + mock_tools_client.linkedin_profile_detail.assert_called_once_with( + profile_url='neal-mohan', + include_email=True, + timeout_secs=600, + ) + + +def test_linkedin_detail_tool_default_include_email_false(mock_tools_client: MagicMock) -> None: + mock_tools_client.linkedin_profile_detail.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyLinkedInProfileDetailTool, mock_tools_client) + + tool._run(profile_url='neal-mohan') + + assert mock_tools_client.linkedin_profile_detail.call_args.kwargs['include_email'] is False + + +# --------------------------------------------------------------------------- +# ApifyTwitterScraperTool +# --------------------------------------------------------------------------- + + +def test_twitter_tool_happy_path(mock_tools_client: MagicMock) -> None: + mock_tools_client.twitter_scrape.return_value = (SUCCEEDED_RUN, SAMPLE_ITEMS) + tool = make_tool(ApifyTwitterScraperTool, mock_tools_client) + + result = tool._run(search_query='apify', max_results=20) + parsed = json.loads(result) + + assert parsed['items'] == SAMPLE_ITEMS + mock_tools_client.twitter_scrape.assert_called_once_with( + search_query='apify', + search_mode='search', + max_results=20, + start=None, + end=None, + sort=None, + timeout_secs=600, + ) + + +def test_twitter_tool_passes_sort(mock_tools_client: MagicMock) -> None: + mock_tools_client.twitter_scrape.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyTwitterScraperTool, mock_tools_client) + + tool._run(search_query='apify', sort='Top') + + kwargs = mock_tools_client.twitter_scrape.call_args.kwargs + assert kwargs['sort'] == 'Top' + + +def test_twitter_tool_passes_date_range(mock_tools_client: MagicMock) -> None: + mock_tools_client.twitter_scrape.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyTwitterScraperTool, mock_tools_client) + + tool._run(search_query='apify', search_mode='user', start='2025-01-01', end='2025-02-01') + + kwargs = mock_tools_client.twitter_scrape.call_args.kwargs + assert kwargs['search_mode'] == 'user' + assert kwargs['start'] == '2025-01-01' + assert kwargs['end'] == '2025-02-01' + + +def test_twitter_tool_value_error_raises_tool_exception(mock_tools_client: MagicMock) -> None: + mock_tools_client.twitter_scrape.side_effect = ValueError('Unsupported Twitter search_mode') + tool = make_tool(ApifyTwitterScraperTool, mock_tools_client) + + with pytest.raises(ToolException, match='Unsupported Twitter search_mode'): + tool._run(search_query='apify', search_mode='replies') # type: ignore[arg-type] + + +def test_twitter_tool_demo_items_add_notice(mock_tools_client: MagicMock) -> None: + # The Actor returns {"demo": true} placeholders on the Apify free plan. + mock_tools_client.twitter_scrape.return_value = (SUCCEEDED_RUN, [{'demo': True}]) + tool = make_tool(ApifyTwitterScraperTool, mock_tools_client) + + parsed = json.loads(tool._run(search_query='apify')) + + assert parsed['notice'] == _NOTICE_TWITTER_DEMO + assert parsed['items'] == [{'demo': True}] + + +def test_twitter_tool_real_items_have_no_notice(mock_tools_client: MagicMock) -> None: + mock_tools_client.twitter_scrape.return_value = (SUCCEEDED_RUN, SAMPLE_ITEMS) + tool = make_tool(ApifyTwitterScraperTool, mock_tools_client) + + parsed = json.loads(tool._run(search_query='apify')) + + assert 'notice' not in parsed + + +@pytest.mark.parametrize( + ('items', 'expected'), + [ + ([{'demo': True}], True), + ([{'text': 'real'}, {'demo': True}], True), + ([{'text': 'real'}], False), + ([{'demo': False}], False), + ([], False), + ], +) +def test_has_demo_items(items: list[dict], expected: bool) -> None: # noqa: FBT001 + assert _has_demo_items(items) is expected + + +# --------------------------------------------------------------------------- +# ApifyTikTokScraperTool +# --------------------------------------------------------------------------- + + +def test_tiktok_tool_happy_path(mock_tools_client: MagicMock) -> None: + mock_tools_client.tiktok_scrape.return_value = (SUCCEEDED_RUN, SAMPLE_ITEMS) + tool = make_tool(ApifyTikTokScraperTool, mock_tools_client) + + result = tool._run(search_query='cooking', search_type='search', max_results=12) + parsed = json.loads(result) + + assert parsed['items'] == SAMPLE_ITEMS + mock_tools_client.tiktok_scrape.assert_called_once_with( + search_query='cooking', + search_type='search', + max_results=12, + timeout_secs=600, + ) + + +def test_tiktok_tool_clamps_max_results(mock_tools_client: MagicMock) -> None: + mock_tools_client.tiktok_scrape.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyTikTokScraperTool, mock_tools_client, max_items=4) + + tool._run(search_query='cooking', max_results=500) + + assert mock_tools_client.tiktok_scrape.call_args.kwargs['max_results'] == 4 + + +def test_tiktok_tool_passes_post_search_type(mock_tools_client: MagicMock) -> None: + mock_tools_client.tiktok_scrape.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyTikTokScraperTool, mock_tools_client) + + tool._run(search_query='https://www.tiktok.com/@charlidamelio/video/123', search_type='post') + + assert mock_tools_client.tiktok_scrape.call_args.kwargs['search_type'] == 'post' + + +# --------------------------------------------------------------------------- +# ApifyFacebookPostsScraperTool +# --------------------------------------------------------------------------- + + +def test_facebook_tool_happy_path(mock_tools_client: MagicMock) -> None: + mock_tools_client.facebook_posts_scrape.return_value = (SUCCEEDED_RUN, SAMPLE_ITEMS) + tool = make_tool(ApifyFacebookPostsScraperTool, mock_tools_client) + + result = tool._run(page_url='https://www.facebook.com/humansofnewyork/', max_results=15) + parsed = json.loads(result) + + assert parsed['run'] == EXPECTED_RUN_META + assert parsed['items'] == SAMPLE_ITEMS + mock_tools_client.facebook_posts_scrape.assert_called_once_with( + page_url='https://www.facebook.com/humansofnewyork/', + max_results=15, + only_posts_newer_than=None, + only_posts_older_than=None, + timeout_secs=600, + ) + + +def test_facebook_tool_passes_only_posts_newer_than(mock_tools_client: MagicMock) -> None: + mock_tools_client.facebook_posts_scrape.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyFacebookPostsScraperTool, mock_tools_client) + + tool._run(page_url='https://www.facebook.com/humansofnewyork/', only_posts_newer_than='2025-01-01') + + assert mock_tools_client.facebook_posts_scrape.call_args.kwargs['only_posts_newer_than'] == '2025-01-01' + + +def test_facebook_tool_passes_only_posts_older_than(mock_tools_client: MagicMock) -> None: + mock_tools_client.facebook_posts_scrape.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyFacebookPostsScraperTool, mock_tools_client) + + tool._run(page_url='https://www.facebook.com/humansofnewyork/', only_posts_older_than='2025-12-31') + + assert mock_tools_client.facebook_posts_scrape.call_args.kwargs['only_posts_older_than'] == '2025-12-31' + + +def test_facebook_tool_runtime_error_raises_tool_exception(mock_tools_client: MagicMock) -> None: + mock_tools_client.facebook_posts_scrape.side_effect = RuntimeError('Network error') + tool = make_tool(ApifyFacebookPostsScraperTool, mock_tools_client) + + with pytest.raises(ToolException, match='Network error'): + tool._run(page_url='https://www.facebook.com/humansofnewyork/') + + +# --------------------------------------------------------------------------- +# Empty results - tools should still return valid JSON +# --------------------------------------------------------------------------- + + +def test_tool_returns_valid_json_for_empty_items(mock_tools_client: MagicMock) -> None: + mock_tools_client.linkedin_profile_search.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyLinkedInProfileSearchTool, mock_tools_client) + + result = tool._run(query='nonexistent') + parsed = json.loads(result) + + assert parsed['items'] == [] + assert parsed['run']['status'] == 'SUCCEEDED' + + +# --------------------------------------------------------------------------- +# handle_tool_error is True on every social tool (existing base behavior) +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + 'tool_cls', + [ + ApifyInstagramScraperTool, + ApifyLinkedInProfilePostsTool, + ApifyLinkedInProfileSearchTool, + ApifyLinkedInProfileDetailTool, + ApifyTwitterScraperTool, + ApifyTikTokScraperTool, + ApifyFacebookPostsScraperTool, + ], +) +def test_social_tool_handle_tool_error_enabled(tool_cls: type, mock_tools_client: MagicMock) -> None: + tool = make_tool(tool_cls, mock_tools_client) + assert tool.handle_tool_error is True + + +# --------------------------------------------------------------------------- +# Per-tool RuntimeError -> ToolException coverage +# --------------------------------------------------------------------------- + +# (tool_cls, client_method_name, _run kwargs) +_SOCIAL_TOOL_INVOCATIONS: list[tuple[type, str, dict]] = [ + (ApifyInstagramScraperTool, 'instagram_scrape', {'search_type': 'user', 'search_query': 'apify'}), + (ApifyLinkedInProfilePostsTool, 'linkedin_profile_posts', {'profile_url': 'satyanadella'}), + (ApifyLinkedInProfileSearchTool, 'linkedin_profile_search', {'query': 'Founder'}), + (ApifyLinkedInProfileDetailTool, 'linkedin_profile_detail', {'profile_url': 'neal-mohan'}), + (ApifyTwitterScraperTool, 'twitter_scrape', {'search_query': 'apify'}), + (ApifyTikTokScraperTool, 'tiktok_scrape', {'search_query': 'cooking'}), + (ApifyFacebookPostsScraperTool, 'facebook_posts_scrape', {'page_url': 'https://www.facebook.com/x/'}), +] + + +@pytest.mark.parametrize(('tool_cls', 'method_name', 'run_kwargs'), _SOCIAL_TOOL_INVOCATIONS) +def test_social_tool_runtime_error_raises_tool_exception( + tool_cls: type, + method_name: str, + run_kwargs: dict, + mock_tools_client: MagicMock, +) -> None: + getattr(mock_tools_client, method_name).side_effect = RuntimeError( + 'Actor run run-XYZ ended with status FAILED.', + ) + tool = make_tool(tool_cls, mock_tools_client) + + with pytest.raises(ToolException, match='run-XYZ'): + tool._run(**run_kwargs) + + +# --------------------------------------------------------------------------- +# Per-tool empty-dataset coverage +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize(('tool_cls', 'method_name', 'run_kwargs'), _SOCIAL_TOOL_INVOCATIONS) +def test_social_tool_returns_valid_json_for_empty_items( + tool_cls: type, + method_name: str, + run_kwargs: dict, + mock_tools_client: MagicMock, +) -> None: + getattr(mock_tools_client, method_name).return_value = (SUCCEEDED_RUN, []) + tool = make_tool(tool_cls, mock_tools_client) + + result = tool._run(**run_kwargs) + parsed = json.loads(result) + + assert parsed['items'] == [] + assert parsed['run'] == EXPECTED_RUN_META + + +# --------------------------------------------------------------------------- +# Regression: dataset items containing datetime values must not break JSON +# serialisation. The Apify client's clean=True deserialiser returns datetime +# objects for certain timestamp fields (notably on the Instagram, LinkedIn +# search and Facebook posts Actors), which previously caused +# `TypeError: Object of type datetime is not JSON serializable`. +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize(('tool_cls', 'method_name', 'run_kwargs'), _SOCIAL_TOOL_INVOCATIONS) +def test_social_tool_serialises_datetime_in_items( + tool_cls: type, + method_name: str, + run_kwargs: dict, + mock_tools_client: MagicMock, +) -> None: + from datetime import datetime, timezone + + timestamp = datetime(2026, 1, 2, 3, 4, 5, tzinfo=timezone.utc) + items_with_datetime = [{'id': 'post-1', 'timestamp': timestamp, 'text': 'hello'}] + getattr(mock_tools_client, method_name).return_value = (SUCCEEDED_RUN, items_with_datetime) + tool = make_tool(tool_cls, mock_tools_client) + + result = tool._run(**run_kwargs) + parsed = json.loads(result) + + assert isinstance(parsed['items'], list) + assert len(parsed['items']) == 1 + assert parsed['items'][0]['id'] == 'post-1' + assert isinstance(parsed['items'][0]['timestamp'], str) + assert '2026-01-02' in parsed['items'][0]['timestamp'] + assert parsed['run'] == EXPECTED_RUN_META diff --git a/tests/unit_tests/test_clamp_descriptions.py b/tests/unit_tests/test_clamp_descriptions.py new file mode 100644 index 0000000..cf56331 --- /dev/null +++ b/tests/unit_tests/test_clamp_descriptions.py @@ -0,0 +1,128 @@ +"""Tests for the clamp-ceiling text in tool input-schema descriptions. + +``_ApifyGenericTool`` silently clamps ``timeout_secs`` / ``memory_mbytes`` +/ ``limit`` / ``dataset_items_limit`` / ``max_crawl_depth`` / ``max_results`` +/ ``max_crawl_pages`` to the per-tool ``max_*`` ceilings. The Pydantic input +schemas advertise those ceilings in their ``Field(description=...)`` strings +so an LLM agent doesn't promise results above the cap. These tests pin +every clamp-relevant field across the core, search, and social tools: + + 1. every clamp-relevant Field carries a ``"clamped to N max"`` phrase; + 2. ``N`` matches the live default cap on ``_ApifyGenericTool`` (no + drift between the description text and the actual clamp). +""" + +from __future__ import annotations + +import re +from typing import TYPE_CHECKING + +import pytest + +from langchain_apify.tools.base import _ApifyGenericTool + +if TYPE_CHECKING: + from pydantic import BaseModel +from langchain_apify.tools.core import ( + ApifyGetDatasetItemsInput, + ApifyRunActorAndGetDatasetInput, + ApifyRunActorInput, + ApifyRunTaskAndGetDatasetInput, + ApifyRunTaskInput, + ApifyScrapeUrlInput, +) +from langchain_apify.tools.search import ( + ApifyEcommerceScraperInput, + ApifyGoogleMapsInput, + ApifyGoogleSearchInput, + ApifyRAGWebBrowserInput, + ApifyRAGWebBrowserTool, + ApifyWebCrawlerInput, + ApifyYouTubeScraperInput, +) +from langchain_apify.tools.social import ( + ApifyFacebookPostsScraperInput, + ApifyInstagramScraperInput, + ApifyLinkedInProfilePostsInput, + ApifyLinkedInProfileSearchInput, + ApifyTikTokScraperInput, + ApifyTwitterScraperInput, +) + +# (schema, field_name, base-class cap-field name) +_CLAMP_FIELDS: list[tuple[type[BaseModel], str, str]] = [ + (ApifyRunActorInput, 'timeout_secs', 'max_timeout_secs'), + (ApifyRunActorInput, 'memory_mbytes', 'max_memory_mbytes'), + (ApifyGetDatasetItemsInput, 'limit', 'max_items'), + (ApifyRunActorAndGetDatasetInput, 'timeout_secs', 'max_timeout_secs'), + (ApifyRunActorAndGetDatasetInput, 'memory_mbytes', 'max_memory_mbytes'), + (ApifyRunActorAndGetDatasetInput, 'dataset_items_limit', 'max_items'), + (ApifyScrapeUrlInput, 'timeout_secs', 'max_timeout_secs'), + (ApifyGoogleSearchInput, 'timeout_secs', 'max_timeout_secs'), + (ApifyWebCrawlerInput, 'timeout_secs', 'max_timeout_secs'), + (ApifyWebCrawlerInput, 'max_crawl_depth', 'max_crawl_depth'), + (ApifyRunTaskInput, 'timeout_secs', 'max_timeout_secs'), + (ApifyRunTaskInput, 'memory_mbytes', 'max_memory_mbytes'), + (ApifyRunTaskAndGetDatasetInput, 'timeout_secs', 'max_timeout_secs'), + (ApifyRunTaskAndGetDatasetInput, 'memory_mbytes', 'max_memory_mbytes'), + (ApifyRunTaskAndGetDatasetInput, 'dataset_items_limit', 'max_items'), + # max_results / max_crawl_pages are clamped via _clamp_items (max_items cap). + (ApifyGoogleSearchInput, 'max_results', 'max_items'), + (ApifyWebCrawlerInput, 'max_crawl_pages', 'max_items'), + (ApifyRAGWebBrowserInput, 'max_results', 'max_items'), + (ApifyGoogleMapsInput, 'max_results', 'max_items'), + (ApifyYouTubeScraperInput, 'max_results', 'max_items'), + (ApifyEcommerceScraperInput, 'max_results', 'max_items'), + (ApifyInstagramScraperInput, 'max_results', 'max_items'), + (ApifyLinkedInProfilePostsInput, 'max_results', 'max_items'), + (ApifyLinkedInProfileSearchInput, 'max_results', 'max_items'), + (ApifyTwitterScraperInput, 'max_results', 'max_items'), + (ApifyTikTokScraperInput, 'max_results', 'max_items'), + (ApifyFacebookPostsScraperInput, 'max_results', 'max_items'), +] + +# Most caps live on the _ApifyGenericTool base class. Tools that override a +# cap for their Actor advertise their own ceiling, so their description must be +# validated against the override rather than the generic default. +_CAP_SOURCE: dict[type[BaseModel], type[BaseModel]] = { + ApifyRAGWebBrowserInput: ApifyRAGWebBrowserTool, +} + + +def _cap_default(schema: type[BaseModel], cap_field: str) -> int: + """Return the live clamp ceiling that ``schema``'s description must match.""" + source = _CAP_SOURCE.get(schema, _ApifyGenericTool) + return source.model_fields[cap_field].default + + +_CAP_PATTERN = re.compile(r'clamped to (\d+) max') + + +@pytest.mark.parametrize(('schema', 'field', 'cap_field'), _CLAMP_FIELDS) +def test_field_description_carries_cap_text(schema: type[BaseModel], field: str, cap_field: str) -> None: + """Every clamp-relevant Field description ends with ``(clamped to N max)``.""" + description = schema.model_fields[field].description or '' + expected_cap = _cap_default(schema, cap_field) + assert f'clamped to {expected_cap} max' in description, ( + f'{schema.__name__}.{field} description does not mention the cap ' + f'(expected "clamped to {expected_cap} max"): {description!r}' + ) + + +@pytest.mark.parametrize(('schema', 'field', 'cap_field'), _CLAMP_FIELDS) +def test_field_description_cap_matches_base_class_default(schema: type[BaseModel], field: str, cap_field: str) -> None: + """The number in the description text equals the live ``_ApifyGenericTool`` cap. + + Catches drift where a cap default moves but the description string is + forgotten, or vice versa. + """ + description = schema.model_fields[field].description or '' + match = _CAP_PATTERN.search(description) + assert match is not None, f'no "clamped to N max" phrase in {schema.__name__}.{field}: {description!r}' + + advertised_cap = int(match.group(1)) + actual_cap = _cap_default(schema, cap_field) + assert advertised_cap == actual_cap, ( + f'{schema.__name__}.{field} advertises cap={advertised_cap} but ' + f'_ApifyGenericTool.{cap_field} default is {actual_cap}' + ) diff --git a/tests/unit_tests/test_client.py b/tests/unit_tests/test_client.py new file mode 100644 index 0000000..6cb2097 --- /dev/null +++ b/tests/unit_tests/test_client.py @@ -0,0 +1,1055 @@ +from __future__ import annotations + +from unittest.mock import MagicMock, patch + +import httpx +import pytest +from apify_client import ApifyClient + +from langchain_apify._client import ApifyToolsClient +from tests.unit_tests.conftest import FAILED_RUN, SAMPLE_ITEMS, SUCCEEDED_RUN + +# --------------------------------------------------------------------------- +# __init__ +# --------------------------------------------------------------------------- + + +def test_init_with_explicit_token(mock_apify_client: MagicMock) -> None: + with patch('langchain_apify._client._create_apify_client', return_value=mock_apify_client) as mock_create: + c = ApifyToolsClient(apify_token='my-token') + mock_create.assert_called_once() + assert c._client is mock_apify_client + + +def test_init_with_apify_token_env(monkeypatch: pytest.MonkeyPatch, mock_apify_client: MagicMock) -> None: + """``APIFY_TOKEN`` (SDK-standard) should be picked up when set.""" + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.setenv('APIFY_TOKEN', 'sdk-token') + with patch('langchain_apify._client._create_apify_client', return_value=mock_apify_client): + c = ApifyToolsClient() + assert c._client is mock_apify_client + + +def test_init_with_legacy_apify_api_token_env(monkeypatch: pytest.MonkeyPatch, mock_apify_client: MagicMock) -> None: + """``APIFY_API_TOKEN`` is still honoured for backwards compatibility.""" + monkeypatch.delenv('APIFY_TOKEN', raising=False) + monkeypatch.setenv('APIFY_API_TOKEN', 'legacy-token') + with patch('langchain_apify._client._create_apify_client', return_value=mock_apify_client): + c = ApifyToolsClient() + assert c._client is mock_apify_client + + +def test_init_apify_token_takes_precedence(monkeypatch: pytest.MonkeyPatch, mock_apify_client: MagicMock) -> None: + """When both env vars are set, ``APIFY_TOKEN`` wins over ``APIFY_API_TOKEN``.""" + monkeypatch.setenv('APIFY_API_TOKEN', 'legacy-token') + monkeypatch.setenv('APIFY_TOKEN', 'sdk-token') + with patch('langchain_apify._client._create_apify_client', return_value=mock_apify_client) as mock_create: + ApifyToolsClient() + mock_create.assert_called_once_with(ApifyClient, 'sdk-token') + + +def test_init_missing_token_raises(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + ApifyToolsClient() + + +# --------------------------------------------------------------------------- +# run_actor +# --------------------------------------------------------------------------- + + +def test_run_actor_success(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + + result = client.run_actor('apify/test-actor', run_input={'key': 'val'}) + + mock_apify_client.actor.assert_called_once_with('apify/test-actor') + mock_apify_client.actor.return_value.call.assert_called_once_with( + run_input={'key': 'val'}, timeout_secs=300, logger=None + ) + assert result == SUCCEEDED_RUN + + +def test_run_actor_with_memory(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + + client.run_actor('apify/test-actor', memory_mbytes=512) + + mock_apify_client.actor.return_value.call.assert_called_once_with( + run_input=None, timeout_secs=300, logger=None, memory_mbytes=512 + ) + + +def test_run_actor_failed_status_raises(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = FAILED_RUN + + with pytest.raises(RuntimeError, match='run-fail'): + client.run_actor('apify/test-actor') + + +# --------------------------------------------------------------------------- +# get_dataset_items +# --------------------------------------------------------------------------- + + +def test_get_dataset_items_success(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.dataset.return_value.list_items.return_value.items = SAMPLE_ITEMS + + items = client.get_dataset_items('dataset-xyz', limit=50, offset=10) + + mock_apify_client.dataset.assert_called_once_with('dataset-xyz') + mock_apify_client.dataset.return_value.list_items.assert_called_once_with(limit=50, offset=10, clean=True) + assert items == SAMPLE_ITEMS + + +def test_get_dataset_items_empty(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.dataset.return_value.list_items.return_value.items = [] + + items = client.get_dataset_items('dataset-empty') + assert items == [] + + +# --------------------------------------------------------------------------- +# run_actor_and_get_items +# --------------------------------------------------------------------------- + + +def test_run_actor_and_get_items_success(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = SAMPLE_ITEMS + + run, items = client.run_actor_and_get_items('apify/test-actor', run_input={'q': '1'}) + + assert run == SUCCEEDED_RUN + assert items == SAMPLE_ITEMS + mock_apify_client.dataset.assert_called_once_with('dataset-xyz') + + +def test_run_actor_and_get_items_missing_dataset_id_raises( + client: ApifyToolsClient, mock_apify_client: MagicMock +) -> None: + run_no_dataset = {**SUCCEEDED_RUN, 'defaultDatasetId': None} + mock_apify_client.actor.return_value.call.return_value = run_no_dataset + + with pytest.raises(RuntimeError, match='no default dataset ID'): + client.run_actor_and_get_items('apify/test-actor') + + +# --------------------------------------------------------------------------- +# run_task +# --------------------------------------------------------------------------- + + +def test_run_task_success(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.task.return_value.call.return_value = SUCCEEDED_RUN + + result = client.run_task('user/my-task', task_input={'key': 'val'}) + + mock_apify_client.task.assert_called_once_with('user/my-task') + mock_apify_client.task.return_value.call.assert_called_once_with(task_input={'key': 'val'}, timeout_secs=300) + assert result == SUCCEEDED_RUN + + +def test_run_task_failed_status_raises(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.task.return_value.call.return_value = FAILED_RUN + + with pytest.raises(RuntimeError, match='run-fail'): + client.run_task('user/my-task') + + +# --------------------------------------------------------------------------- +# run_task_and_get_items +# --------------------------------------------------------------------------- + + +def test_run_task_and_get_items_success(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.task.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = SAMPLE_ITEMS + + run, items = client.run_task_and_get_items('user/my-task') + + assert run == SUCCEEDED_RUN + assert items == SAMPLE_ITEMS + + +def test_run_task_and_get_items_missing_dataset_id_raises( + client: ApifyToolsClient, mock_apify_client: MagicMock +) -> None: + run_no_dataset = {**SUCCEEDED_RUN, 'defaultDatasetId': None} + mock_apify_client.task.return_value.call.return_value = run_no_dataset + + with pytest.raises(RuntimeError, match='no default dataset ID'): + client.run_task_and_get_items('user/my-task') + + +# --------------------------------------------------------------------------- +# scrape_url +# --------------------------------------------------------------------------- + + +def test_scrape_url_returns_markdown(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [ + {'markdown': '# Hello', 'text': 'Hello', 'url': 'https://example.com'}, + ] + + content = client.scrape_url('https://example.com') + assert content == '# Hello' + + +def test_scrape_url_falls_back_to_text(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [ + {'text': 'Plain text content', 'url': 'https://example.com'}, + ] + + content = client.scrape_url('https://example.com') + assert content == 'Plain text content' + + +def test_scrape_url_empty_items_raises(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [] + + with pytest.raises(RuntimeError, match='No content extracted'): + client.scrape_url('https://example.com') + + +def test_scrape_url_empty_content_raises(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [ + {'markdown': '', 'text': '', 'url': 'https://example.com'}, + ] + + with pytest.raises(RuntimeError, match='No content extracted'): + client.scrape_url('https://example.com') + + +# --------------------------------------------------------------------------- +# _check_run_status +# --------------------------------------------------------------------------- + + +def test_check_run_status_succeeded() -> None: + ApifyToolsClient._check_run_status({'id': 'run-ok', 'status': 'SUCCEEDED'}) + + +def test_check_run_status_failed() -> None: + with pytest.raises(RuntimeError, match='run-bad'): + ApifyToolsClient._check_run_status({'id': 'run-bad', 'status': 'FAILED'}) + + +def test_check_run_status_failed_includes_status_message() -> None: + with pytest.raises(RuntimeError, match='Actor exited out of memory'): + ApifyToolsClient._check_run_status( + {'id': 'run-oom', 'status': 'FAILED', 'statusMessage': 'Actor exited out of memory'}, + ) + + +# --------------------------------------------------------------------------- +# None returns from actor/task .call() +# --------------------------------------------------------------------------- + + +def test_run_actor_none_return_raises(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = None + + with pytest.raises(RuntimeError, match='returned no run details'): + client.run_actor('apify/broken-actor') + + +def test_run_task_none_return_raises(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.task.return_value.call.return_value = None + + with pytest.raises(RuntimeError, match='returned no run details'): + client.run_task('user/broken-task') + + +# --------------------------------------------------------------------------- +# Transport-error wrapping (httpx / ApifyClientError -> RuntimeError) +# --------------------------------------------------------------------------- + + +def test_run_actor_network_error_wraps(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.side_effect = httpx.ConnectError('conn refused') + + with pytest.raises(RuntimeError, match='Apify Actor call failed'): + client.run_actor('apify/test-actor') + + +def test_get_dataset_items_network_error_wraps(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.dataset.return_value.list_items.side_effect = httpx.ConnectError('timeout') + + with pytest.raises(RuntimeError, match='Apify dataset fetch failed'): + client.get_dataset_items('dataset-xyz') + + +def test_run_actor_and_get_items_dataset_fetch_network_error( + client: ApifyToolsClient, mock_apify_client: MagicMock +) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.side_effect = httpx.ConnectError('reset') + + with pytest.raises(RuntimeError, match='Apify dataset fetch failed'): + client.run_actor_and_get_items('apify/test-actor') + + +def test_run_task_network_error_wraps(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.task.return_value.call.side_effect = httpx.ConnectError('conn refused') + + with pytest.raises(RuntimeError, match='Apify task call failed'): + client.run_task('user/my-task') + + +def test_run_task_and_get_items_dataset_fetch_network_error( + client: ApifyToolsClient, mock_apify_client: MagicMock +) -> None: + mock_apify_client.task.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.side_effect = httpx.ConnectError('reset') + + with pytest.raises(RuntimeError, match='Apify dataset fetch failed'): + client.run_task_and_get_items('user/my-task') + + +def test_run_actor_programming_error_propagates(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + """Non-transport exceptions (programming errors) must NOT be wrapped as RuntimeError.""" + mock_apify_client.actor.return_value.call.side_effect = AttributeError('bug in SDK') + + with pytest.raises(AttributeError, match='bug in SDK'): + client.run_actor('apify/test-actor') + + +# --------------------------------------------------------------------------- +# instagram_scrape +# --------------------------------------------------------------------------- + + +def _setup_run_and_items(mock_apify_client: MagicMock, items: list[dict] | None = None) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = items or SAMPLE_ITEMS + + +def test_instagram_scrape_user_builds_profile_url(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + run, items = client.instagram_scrape('user', 'apify', max_results=5) + + mock_apify_client.actor.assert_called_once_with('apify/instagram-scraper') + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input'] == { + 'directUrls': ['https://www.instagram.com/apify/'], + 'resultsType': 'posts', + 'resultsLimit': 5, + } + assert run == SUCCEEDED_RUN + assert items == SAMPLE_ITEMS + + +def test_instagram_scrape_hashtag_builds_tag_url(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + client.instagram_scrape('hashtag', '#travel', max_results=10) + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input']['directUrls'] == ['https://www.instagram.com/explore/tags/travel/'] + assert call_kwargs['run_input']['resultsType'] == 'posts' + + +def test_instagram_scrape_comments_uses_comments_results_type( + client: ApifyToolsClient, mock_apify_client: MagicMock +) -> None: + _setup_run_and_items(mock_apify_client) + + client.instagram_scrape('comments', 'https://www.instagram.com/p/ABC123/', max_results=15) + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input']['resultsType'] == 'comments' + assert call_kwargs['run_input']['directUrls'] == ['https://www.instagram.com/p/ABC123/'] + + +def test_instagram_scrape_passes_only_posts_newer_than(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + client.instagram_scrape('user', 'apify', only_posts_newer_than='1 week') + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input']['onlyPostsNewerThan'] == '1 week' + + +def test_instagram_scrape_invalid_search_type_raises(client: ApifyToolsClient) -> None: + with pytest.raises(ValueError, match='Unsupported Instagram search_type'): + client.instagram_scrape('reels', 'apify') + + +# --------------------------------------------------------------------------- +# linkedin_profile_posts +# --------------------------------------------------------------------------- + + +def test_linkedin_profile_posts_maps_input(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + run, items = client.linkedin_profile_posts('https://www.linkedin.com/in/satyanadella', max_results=30) + + mock_apify_client.actor.assert_called_once_with('apimaestro/linkedin-profile-posts') + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input'] == { + 'username': 'https://www.linkedin.com/in/satyanadella', + 'total_posts': 30, + } + assert run == SUCCEEDED_RUN + assert items == SAMPLE_ITEMS + + +# --------------------------------------------------------------------------- +# linkedin_profile_search +# --------------------------------------------------------------------------- + + +def test_linkedin_profile_search_maps_input(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + client.linkedin_profile_search('Founder', max_results=25) + + mock_apify_client.actor.assert_called_once_with('harvestapi/linkedin-profile-search') + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input'] == {'searchQuery': 'Founder', 'maxItems': 25} + + +def test_linkedin_profile_search_default_max_results(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + client.linkedin_profile_search('CTO') + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input']['maxItems'] == 10 + + +# --------------------------------------------------------------------------- +# linkedin_profile_detail +# --------------------------------------------------------------------------- + + +def test_linkedin_profile_detail_maps_input(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client, items=[{'firstName': 'Neal'}]) + + run, items = client.linkedin_profile_detail('neal-mohan', include_email=True) + + mock_apify_client.actor.assert_called_once_with('apimaestro/linkedin-profile-detail') + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input'] == {'username': 'neal-mohan', 'includeEmail': True} + assert run == SUCCEEDED_RUN + assert items == [{'firstName': 'Neal'}] + + +def test_linkedin_profile_detail_default_include_email_false( + client: ApifyToolsClient, mock_apify_client: MagicMock +) -> None: + _setup_run_and_items(mock_apify_client) + + client.linkedin_profile_detail('neal-mohan') + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input']['includeEmail'] is False + + +# --------------------------------------------------------------------------- +# twitter_scrape +# --------------------------------------------------------------------------- + + +def test_twitter_scrape_search_mode(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + client.twitter_scrape('apify', max_results=50) + + mock_apify_client.actor.assert_called_once_with('apidojo/twitter-scraper-lite') + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input'] == {'maxItems': 50, 'searchTerms': ['apify']} + + +def test_twitter_scrape_user_mode_strips_at(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + client.twitter_scrape('@apify', search_mode='user', max_results=10) + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input'] == {'maxItems': 10, 'twitterHandles': ['apify']} + + +def test_twitter_scrape_replies_mode_uses_start_urls(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + client.twitter_scrape('https://x.com/apify/status/123', search_mode='replies') + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input']['startUrls'] == ['https://x.com/apify/status/123'] + + +def test_twitter_scrape_passes_date_range(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + client.twitter_scrape('apify', start='2025-01-01', end='2025-02-01') + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input']['start'] == '2025-01-01' + assert call_kwargs['run_input']['end'] == '2025-02-01' + + +def test_twitter_scrape_passes_sort(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + client.twitter_scrape('apify', sort='Top') + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input']['sort'] == 'Top' + + +def test_twitter_scrape_invalid_mode_raises(client: ApifyToolsClient) -> None: + with pytest.raises(ValueError, match='Unsupported Twitter search_mode'): + client.twitter_scrape('apify', search_mode='followers') + + +# --------------------------------------------------------------------------- +# tiktok_scrape +# --------------------------------------------------------------------------- + + +def test_tiktok_scrape_search_mode(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + client.tiktok_scrape('cooking', max_results=12) + + mock_apify_client.actor.assert_called_once_with('clockworks/tiktok-scraper') + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input'] == {'resultsPerPage': 12, 'searchQueries': ['cooking']} + + +def test_tiktok_scrape_user_mode_strips_at(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + client.tiktok_scrape('@charlidamelio', search_type='user') + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input']['profiles'] == ['charlidamelio'] + + +def test_tiktok_scrape_hashtag_mode_strips_hash(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + client.tiktok_scrape('#fyp', search_type='hashtag') + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input']['hashtags'] == ['fyp'] + + +def test_tiktok_scrape_post_mode_uses_post_urls(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + client.tiktok_scrape('https://www.tiktok.com/@charlidamelio/video/123', search_type='post') + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input']['postURLs'] == ['https://www.tiktok.com/@charlidamelio/video/123'] + + +def test_tiktok_scrape_invalid_type_raises(client: ApifyToolsClient) -> None: + with pytest.raises(ValueError, match='Unsupported TikTok search_type'): + client.tiktok_scrape('cooking', search_type='trending') + + +# --------------------------------------------------------------------------- +# facebook_posts_scrape +# --------------------------------------------------------------------------- + + +def test_facebook_posts_scrape_maps_input(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + _setup_run_and_items(mock_apify_client) + + run, items = client.facebook_posts_scrape('https://www.facebook.com/humansofnewyork/', max_results=15) + + mock_apify_client.actor.assert_called_once_with('apify/facebook-posts-scraper') + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input'] == { + 'startUrls': [{'url': 'https://www.facebook.com/humansofnewyork/'}], + 'resultsLimit': 15, + } + assert run == SUCCEEDED_RUN + assert items == SAMPLE_ITEMS + + +def test_facebook_posts_scrape_passes_only_posts_newer_than( + client: ApifyToolsClient, mock_apify_client: MagicMock +) -> None: + _setup_run_and_items(mock_apify_client) + + client.facebook_posts_scrape('https://www.facebook.com/humansofnewyork/', only_posts_newer_than='2025-01-01') + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input']['onlyPostsNewerThan'] == '2025-01-01' + + +def test_facebook_posts_scrape_passes_only_posts_older_than( + client: ApifyToolsClient, mock_apify_client: MagicMock +) -> None: + _setup_run_and_items(mock_apify_client) + + client.facebook_posts_scrape('https://www.facebook.com/humansofnewyork/', only_posts_older_than='2025-12-31') + + call_kwargs = mock_apify_client.actor.return_value.call.call_args.kwargs + assert call_kwargs['run_input']['onlyPostsOlderThan'] == '2025-12-31' + + +# --------------------------------------------------------------------------- +# Failed run propagates from social helpers +# --------------------------------------------------------------------------- + + +def test_social_helper_propagates_failed_run(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = FAILED_RUN + + with pytest.raises(RuntimeError, match='run-fail'): + client.instagram_scrape('user', 'apify') + + +# --------------------------------------------------------------------------- +# _build_instagram_url +# --------------------------------------------------------------------------- + + +def test_build_instagram_url_passthrough_for_full_url() -> None: + assert ( + ApifyToolsClient._build_instagram_url('post', 'https://www.instagram.com/p/abc/') + == 'https://www.instagram.com/p/abc/' + ) + + +def test_build_instagram_url_user() -> None: + assert ApifyToolsClient._build_instagram_url('user', '@apify') == 'https://www.instagram.com/apify/' + + +def test_build_instagram_url_hashtag() -> None: + assert ( + ApifyToolsClient._build_instagram_url('hashtag', '#travel') == 'https://www.instagram.com/explore/tags/travel/' + ) + + +def test_build_instagram_url_post_from_id() -> None: + assert ApifyToolsClient._build_instagram_url('post', 'ABC123') == 'https://www.instagram.com/p/ABC123/' + + +def test_build_instagram_url_bare_user_handle() -> None: + assert ApifyToolsClient._build_instagram_url('user', 'apify') == 'https://www.instagram.com/apify/' + + +def test_build_instagram_url_strips_only_one_leading_prefix() -> None: + # removeprefix drops a single leading '@'; lstrip would wrongly strip both. + assert ApifyToolsClient._build_instagram_url('user', '@@apify') == 'https://www.instagram.com/@apify/' + + +# --------------------------------------------------------------------------- +# scrape_url_with_metadata +# --------------------------------------------------------------------------- + + +def test_scrape_url_with_metadata_returns_markdown(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [ + {'markdown': '# Hello', 'text': 'Hello', 'url': 'https://example.com'}, + ] + + run, items, content, source = client.scrape_url_with_metadata('https://example.com') + assert run == SUCCEEDED_RUN + assert items + assert content == '# Hello' + assert source == 'markdown' + + +def test_scrape_url_with_metadata_falls_back_to_text(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [ + {'text': 'Plain text content', 'url': 'https://example.com'}, + ] + + _, _, content, source = client.scrape_url_with_metadata('https://example.com') + assert content == 'Plain text content' + assert source == 'text' + + +# --------------------------------------------------------------------------- +# google_search +# --------------------------------------------------------------------------- + + +def test_google_search_input_mapping(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [ + { + 'organicResults': [ + {'title': 'A', 'url': 'https://a.com', 'description': 'da'}, + {'title': 'B', 'url': 'https://b.com', 'description': 'db'}, + ] + } + ] + + run, results = client.google_search('langchain', max_results=5, country_code='us', language_code='en') + + assert run == SUCCEEDED_RUN + mock_apify_client.actor.assert_called_once_with('apify/google-search-scraper') + run_input = mock_apify_client.actor.return_value.call.call_args.kwargs['run_input'] + assert run_input == { + 'queries': 'langchain', + 'maxPagesPerQuery': 1, + 'countryCode': 'us', + 'languageCode': 'en', + } + assert len(results) == 2 + assert results[0]['title'] == 'A' + + +def test_google_search_scales_pages_to_max_results(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [] + + # ~10 results/page, so 25 results needs ceil(25 / 10) == 3 pages. + client.google_search('langchain', max_results=25) + + run_input = mock_apify_client.actor.return_value.call.call_args.kwargs['run_input'] + assert run_input['maxPagesPerQuery'] == 3 + assert 'resultsPerPage' not in run_input + + +@pytest.mark.parametrize( + ('max_results', 'expected_pages'), + [(5, 1), (10, 1), (11, 2), (25, 3), (100, 10)], +) +def test_google_search_page_count_ceil_boundaries( + client: ApifyToolsClient, + mock_apify_client: MagicMock, + max_results: int, + expected_pages: int, +) -> None: + # maxPagesPerQuery == ceil(max_results / 10); 10->1 and 11->2 pin the boundary. + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [] + + client.google_search('langchain', max_results=max_results) + + run_input = mock_apify_client.actor.return_value.call.call_args.kwargs['run_input'] + assert run_input['maxPagesPerQuery'] == expected_pages + assert 'resultsPerPage' not in run_input + + +def test_google_search_omits_optional_locale_params(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [] + + client.google_search('langchain') + + run_input = mock_apify_client.actor.return_value.call.call_args.kwargs['run_input'] + assert 'countryCode' not in run_input + assert 'languageCode' not in run_input + + +# google_search +# --------------------------------------------------------------------------- + +GOOGLE_SEARCH_ITEMS: list[dict] = [ + { + 'organicResults': [ + {'title': 'Result 1', 'url': 'https://example.com/1', 'description': 'Desc 1'}, + {'title': 'Result 2', 'url': 'https://example.com/2', 'description': 'Desc 2'}, + ], + }, +] + + +def test_google_search_success(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = GOOGLE_SEARCH_ITEMS + + run, results = client.google_search('test query', max_results=5) + + assert run == SUCCEEDED_RUN + assert len(results) == 2 + assert results[0] == {'title': 'Result 1', 'url': 'https://example.com/1', 'description': 'Desc 1'} + assert results[1] == {'title': 'Result 2', 'url': 'https://example.com/2', 'description': 'Desc 2'} + + +def test_google_search_with_locale(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = GOOGLE_SEARCH_ITEMS + + client.google_search('test', country_code='us', language_code='en') + + call_args = mock_apify_client.actor.return_value.call.call_args + run_input = call_args.kwargs['run_input'] + assert run_input['countryCode'] == 'us' + assert run_input['languageCode'] == 'en' + + +def test_google_search_caps_results(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + many_results = [{'title': f'R{i}', 'url': f'https://example.com/{i}', 'description': f'D{i}'} for i in range(20)] + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [{'organicResults': many_results}] + + _, results = client.google_search('test', max_results=3) + + assert len(results) == 3 + + +def test_google_search_empty_results(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [{'organicResults': []}] + + _, results = client.google_search('test') + + assert results == [] + + +def test_google_search_failed_run_raises(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = FAILED_RUN + + with pytest.raises(RuntimeError, match='run-fail'): + client.google_search('test') + + +# --------------------------------------------------------------------------- +# rag_web_search +# --------------------------------------------------------------------------- + +RAG_SEARCH_ITEMS: list[dict] = [ + {'crawledUrl': 'https://example.com/1', 'text': 'Page 1 content', 'metadata': {'title': 'Page 1'}}, + {'crawledUrl': 'https://example.com/2', 'text': 'Page 2 content', 'metadata': {'title': 'Page 2'}}, +] + + +def test_rag_web_search_success(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = RAG_SEARCH_ITEMS + + run, items = client.rag_web_search('test query', max_results=5) + + assert run == SUCCEEDED_RUN + assert len(items) == 2 + assert items[0]['crawledUrl'] == 'https://example.com/1' + assert items[1]['text'] == 'Page 2 content' + + +def test_rag_web_search_empty(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [] + + run, items = client.rag_web_search('test') + + assert run == SUCCEEDED_RUN + assert items == [] + + +def test_rag_web_search_failed_run_raises(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = FAILED_RUN + + with pytest.raises(RuntimeError, match='run-fail'): + client.rag_web_search('test') + + +# --------------------------------------------------------------------------- +# crawl_website +# --------------------------------------------------------------------------- + +CRAWL_ITEMS: list[dict] = [ + {'url': 'https://example.com/', 'markdown': '# Home', 'text': 'Home', 'metadata': {'title': 'Home'}}, + {'url': 'https://example.com/about', 'markdown': '# About', 'text': 'About', 'metadata': {'title': 'About'}}, +] + + +def test_crawl_website_success(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = CRAWL_ITEMS + + run, items = client.crawl_website('https://example.com') + + assert run == SUCCEEDED_RUN + assert len(items) == 2 + assert items[0]['url'] == 'https://example.com/' + assert items[1]['markdown'] == '# About' + + +def test_crawl_website_passes_params(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [] + + client.crawl_website('https://example.com', max_crawl_pages=5, max_crawl_depth=2, crawler_type='playwright:firefox') + + call_args = mock_apify_client.actor.return_value.call.call_args + run_input = call_args.kwargs['run_input'] + assert run_input['startUrls'] == [{'url': 'https://example.com'}] + assert run_input['maxCrawlPages'] == 5 + assert run_input['maxCrawlDepth'] == 2 + assert run_input['crawlerType'] == 'playwright:firefox' + + +def test_crawl_website_empty(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [] + + _, items = client.crawl_website('https://example.com') + + assert items == [] + + +def test_crawl_website_failed_run_raises(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = FAILED_RUN + + with pytest.raises(RuntimeError, match='run-fail'): + client.crawl_website('https://example.com') + + +# --------------------------------------------------------------------------- +# google_maps_search +# --------------------------------------------------------------------------- + + +def test_google_maps_search_input_mapping(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = SAMPLE_ITEMS + + run, items = client.google_maps_search('coffee in Berlin', max_results=5, language='en') + + mock_apify_client.actor.assert_called_once_with('compass/crawler-google-places') + run_input = mock_apify_client.actor.return_value.call.call_args.kwargs['run_input'] + assert run_input == { + 'searchStringsArray': ['coffee in Berlin'], + 'maxCrawledPlacesPerSearch': 5, + 'language': 'en', + } + assert run == SUCCEEDED_RUN + assert items == SAMPLE_ITEMS + + +def test_google_maps_search_omits_language_when_none(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [] + + client.google_maps_search('parks') + + run_input = mock_apify_client.actor.return_value.call.call_args.kwargs['run_input'] + assert 'language' not in run_input + + +def test_google_maps_search_failed_run_raises(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = FAILED_RUN + + with pytest.raises(RuntimeError, match='run-fail'): + client.google_maps_search('parks') + + +# --------------------------------------------------------------------------- +# youtube_scrape +# --------------------------------------------------------------------------- + + +def test_youtube_scrape_search_mode_input_mapping(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = SAMPLE_ITEMS + + run, items = client.youtube_scrape('langchain', search_type='search', max_results=7) + + mock_apify_client.actor.assert_called_once_with('streamers/youtube-scraper') + run_input = mock_apify_client.actor.return_value.call.call_args.kwargs['run_input'] + assert run_input == {'maxResults': 7, 'searchQueries': ['langchain']} + assert run == SUCCEEDED_RUN + assert items == SAMPLE_ITEMS + + +@pytest.mark.parametrize('search_type', ['video', 'channel']) +def test_youtube_scrape_url_modes_use_start_urls( + client: ApifyToolsClient, mock_apify_client: MagicMock, search_type: str +) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [] + + client.youtube_scrape('https://www.youtube.com/@apify', search_type=search_type, max_results=4) + + run_input = mock_apify_client.actor.return_value.call.call_args.kwargs['run_input'] + assert run_input == { + 'maxResults': 4, + 'startUrls': [{'url': 'https://www.youtube.com/@apify'}], + } + + +def test_youtube_scrape_invalid_search_type_raises(client: ApifyToolsClient) -> None: + with pytest.raises(ValueError, match='Invalid search_type'): + client.youtube_scrape('langchain', search_type='playlist') + + +def test_youtube_scrape_failed_run_raises(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = FAILED_RUN + + with pytest.raises(RuntimeError, match='run-fail'): + client.youtube_scrape('langchain') + + +# --------------------------------------------------------------------------- +# ecommerce_scrape +# --------------------------------------------------------------------------- + + +def test_ecommerce_scrape_input_mapping(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = SAMPLE_ITEMS + + run, items = client.ecommerce_scrape('https://shop.example.com/cat/123', max_results=15) + + mock_apify_client.actor.assert_called_once_with('apify/e-commerce-scraping-tool') + run_input = mock_apify_client.actor.return_value.call.call_args.kwargs['run_input'] + assert run_input == { + 'detailsUrls': [{'url': 'https://shop.example.com/cat/123'}], + 'maxProductResults': 15, + } + assert run == SUCCEEDED_RUN + assert items == SAMPLE_ITEMS + + +def test_ecommerce_scrape_category_mode_uses_listing_urls( + client: ApifyToolsClient, mock_apify_client: MagicMock +) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = SAMPLE_ITEMS + + client.ecommerce_scrape('https://shop.example.com/category/123', url_type='category', max_results=5) + + run_input = mock_apify_client.actor.return_value.call.call_args.kwargs['run_input'] + assert run_input == { + 'listingUrls': [{'url': 'https://shop.example.com/category/123'}], + 'maxProductResults': 5, + } + + +def test_ecommerce_scrape_invalid_url_type_raises(client: ApifyToolsClient) -> None: + with pytest.raises(ValueError, match='Invalid url_type'): + client.ecommerce_scrape('https://shop.example.com', url_type='listing') + + +def test_ecommerce_scrape_failed_run_raises(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = FAILED_RUN + + with pytest.raises(RuntimeError, match='run-fail'): + client.ecommerce_scrape('https://shop.example.com') + + +# --------------------------------------------------------------------------- +# rag_web_search input mapping +# --------------------------------------------------------------------------- + + +def test_rag_web_search_input_mapping(client: ApifyToolsClient, mock_apify_client: MagicMock) -> None: + mock_apify_client.actor.return_value.call.return_value = SUCCEEDED_RUN + mock_apify_client.dataset.return_value.list_items.return_value.items = [] + + client.rag_web_search('what is langchain', max_results=4) + + mock_apify_client.actor.assert_called_once_with('apify/rag-web-browser') + run_input = mock_apify_client.actor.return_value.call.call_args.kwargs['run_input'] + assert run_input == {'query': 'what is langchain', 'maxResults': 4} diff --git a/tests/unit_tests/test_deprecated_token_alias.py b/tests/unit_tests/test_deprecated_token_alias.py new file mode 100644 index 0000000..d79ba4f --- /dev/null +++ b/tests/unit_tests/test_deprecated_token_alias.py @@ -0,0 +1,322 @@ +"""Tests for the ``apify_api_token`` → ``apify_token`` deprecation alias. + +Every public class that accepts an Apify token is covered: + - ApifyToolsClient + - ApifyDatasetLoader + - ApifyWrapper + - ApifyActorsTool + - _ApifyGenericTool (via ApifyRunActorTool) + +For each class, the matrix is: + 1. ``apify_token`` → works, NO warning + 2. ``apify_api_token`` → works, emits DeprecationWarning ("deprecated, use apify_token") + 3. both specified → ``apify_token`` wins, emits DeprecationWarning + ("ignoring the deprecated 'apify_api_token'") +""" + +from __future__ import annotations + +import warnings +from contextlib import ExitStack +from typing import TYPE_CHECKING +from unittest.mock import MagicMock, patch + +from apify_client._types import ListPage +from apify_client.clients import DatasetClient +from langchain_core.documents import Document +from pydantic import BaseModel + +from langchain_apify import ApifyDatasetLoader, ApifyWrapper +from langchain_apify._client import ApifyToolsClient +from langchain_apify._utils import _resolve_apify_token +from langchain_apify.tools import ApifyActorsTool, ApifyRunActorTool + +if TYPE_CHECKING: + import pytest + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +_EMPTY_LIST_PAGE = ListPage(data={'items': []}) + + +def _noop_mapping(item: dict) -> Document: + return Document(page_content=item.get('text', '')) + + +# --------------------------------------------------------------------------- +# ApifyToolsClient +# --------------------------------------------------------------------------- + + +class TestApifyToolsClientTokenAlias: + """Token-alias tests for :class:`ApifyToolsClient`.""" + + def test_apify_token_no_warning(self, mock_apify_client: MagicMock) -> None: + with patch('langchain_apify._client._create_apify_client', return_value=mock_apify_client): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + c = ApifyToolsClient(apify_token='new-style') + assert len(w) == 0 + assert c._client is mock_apify_client + + def test_apify_api_token_emits_warning(self, mock_apify_client: MagicMock) -> None: + with patch('langchain_apify._client._create_apify_client', return_value=mock_apify_client): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + c = ApifyToolsClient(apify_api_token='legacy-style') + assert len(w) == 1 + assert issubclass(w[0].category, DeprecationWarning) + assert 'apify_api_token' in str(w[0].message) + assert c._client is mock_apify_client + + def test_both_specified_uses_apify_token(self, mock_apify_client: MagicMock) -> None: + """When both are given, ``apify_token`` wins and the user is warned.""" + with patch('langchain_apify._client._create_apify_client', return_value=mock_apify_client) as mock_create: + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + ApifyToolsClient(apify_token='primary', apify_api_token='ignored') + assert len(w) == 1 + assert issubclass(w[0].category, DeprecationWarning) + assert 'ignoring' in str(w[0].message) + # apify_token was passed to _create_apify_client, not apify_api_token + assert mock_create.call_args.args[1] == 'primary' + + +# --------------------------------------------------------------------------- +# ApifyDatasetLoader +# --------------------------------------------------------------------------- + + +class TestApifyDatasetLoaderTokenAlias: + """Token-alias tests for :class:`ApifyDatasetLoader`.""" + + def test_apify_token_no_warning(self) -> None: + with patch.object(DatasetClient, 'list_items', return_value=_EMPTY_LIST_PAGE): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + loader = ApifyDatasetLoader( + dataset_id='d', + dataset_mapping_function=_noop_mapping, + apify_token='new-style', + ) + assert len(w) == 0 + assert loader.apify_token is not None + + def test_apify_api_token_emits_warning(self) -> None: + with patch.object(DatasetClient, 'list_items', return_value=_EMPTY_LIST_PAGE): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + loader = ApifyDatasetLoader( + dataset_id='d', + dataset_mapping_function=_noop_mapping, + apify_api_token='legacy-style', + ) + assert len(w) == 1 + assert issubclass(w[0].category, DeprecationWarning) + assert 'apify_api_token' in str(w[0].message) + assert loader.apify_token is not None + + def test_both_specified_uses_apify_token(self) -> None: + """When both are given, ``apify_token`` wins and the user is warned.""" + with patch.object(DatasetClient, 'list_items', return_value=_EMPTY_LIST_PAGE): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + loader = ApifyDatasetLoader( + dataset_id='d', + dataset_mapping_function=_noop_mapping, + apify_token='primary', + apify_api_token='ignored', + ) + assert len(w) == 1 + assert issubclass(w[0].category, DeprecationWarning) + assert 'ignoring' in str(w[0].message) + assert loader.apify_token is not None + assert loader.apify_token.get_secret_value() == 'primary' + + +# --------------------------------------------------------------------------- +# ApifyWrapper +# --------------------------------------------------------------------------- + + +class TestApifyWrapperTokenAlias: + """Token-alias tests for :class:`ApifyWrapper`.""" + + def test_apify_token_no_warning(self) -> None: + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + wrapper = ApifyWrapper(apify_token='new-style') + assert len(w) == 0 + assert wrapper.apify_token is not None + + def test_apify_api_token_emits_warning(self) -> None: + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + wrapper = ApifyWrapper(apify_api_token='legacy-style') + assert len(w) == 1 + assert issubclass(w[0].category, DeprecationWarning) + assert 'apify_api_token' in str(w[0].message) + assert wrapper.apify_token is not None + + def test_both_specified_uses_apify_token(self) -> None: + """When both are given, ``apify_token`` wins and the user is warned.""" + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + wrapper = ApifyWrapper(apify_token='primary', apify_api_token='ignored') + assert len(w) == 1 + assert issubclass(w[0].category, DeprecationWarning) + assert 'ignoring' in str(w[0].message) + assert wrapper.apify_token is not None + assert wrapper.apify_token.get_secret_value() == 'primary' + + +# --------------------------------------------------------------------------- +# ApifyActorsTool +# --------------------------------------------------------------------------- + + +class _DummySchema(BaseModel): + run_input: str + + +class TestApifyActorsToolTokenAlias: + """Token-alias tests for :class:`ApifyActorsTool`.""" + + @staticmethod + def _patches() -> tuple: + """Return patch context managers that stub the network calls ApifyActorsTool makes at init.""" + return ( + patch('langchain_apify.tools.actors._get_actor_latest_build', return_value={}), + patch.object(ApifyActorsTool, '_create_description', return_value='stub'), + patch.object(ApifyActorsTool, '_build_tool_args_schema_model', return_value=_DummySchema), + ) + + def test_apify_token_no_warning(self) -> None: + with ExitStack() as stack: + for p in self._patches(): + stack.enter_context(p) + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + ApifyActorsTool(actor_id='apify/test', apify_token='new-style') + assert len(w) == 0 + + def test_apify_api_token_emits_warning(self) -> None: + with ExitStack() as stack: + for p in self._patches(): + stack.enter_context(p) + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + tool = ApifyActorsTool(actor_id='apify/test', apify_api_token='legacy-style') + assert len(w) == 1 + assert issubclass(w[0].category, DeprecationWarning) + assert 'apify_api_token' in str(w[0].message) + assert isinstance(tool, ApifyActorsTool) + + def test_both_specified_uses_apify_token(self) -> None: + """When both are given, ``apify_token`` wins and the user is warned.""" + with ExitStack() as stack: + for p in self._patches(): + stack.enter_context(p) + with patch('langchain_apify.tools.actors._create_apify_client') as mock_create: + mock_create.return_value = MagicMock() + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + ApifyActorsTool(actor_id='apify/test', apify_token='primary', apify_api_token='ignored') + assert len(w) == 1 + assert issubclass(w[0].category, DeprecationWarning) + assert 'ignoring' in str(w[0].message) + # ``apify_token`` was passed through to the client constructor. + assert mock_create.call_args.args[1] == 'primary' + + +# --------------------------------------------------------------------------- +# _ApifyGenericTool (tested via ApifyRunActorTool) +# --------------------------------------------------------------------------- + + +class TestGenericToolTokenAlias: + """Token-alias tests for :class:`_ApifyGenericTool` (via :class:`ApifyRunActorTool`).""" + + def test_apify_token_no_warning(self) -> None: + with patch.object(ApifyToolsClient, '__init__', return_value=None): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + tool = ApifyRunActorTool(apify_token='new-style') # type: ignore[call-arg,arg-type] + assert len(w) == 0 + assert tool.apify_token is not None + + def test_apify_api_token_emits_warning(self) -> None: + with patch.object(ApifyToolsClient, '__init__', return_value=None): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + tool = ApifyRunActorTool(apify_api_token='legacy-style') # type: ignore[call-arg,arg-type] + assert len(w) == 1 + assert issubclass(w[0].category, DeprecationWarning) + assert 'apify_api_token' in str(w[0].message) + assert tool.apify_token is not None + + def test_both_specified_uses_apify_token(self) -> None: + """When both are given, ``apify_token`` wins and the user is warned.""" + with patch.object(ApifyToolsClient, '__init__', return_value=None): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + tool = ApifyRunActorTool(apify_token='primary', apify_api_token='ignored') # type: ignore[call-arg,arg-type] + assert len(w) == 1 + assert issubclass(w[0].category, DeprecationWarning) + assert 'ignoring' in str(w[0].message) + assert tool.apify_token is not None + assert tool.apify_token.get_secret_value() == 'primary' + + +# --------------------------------------------------------------------------- +# APIFY_API_TOKEN environment variable (deprecated alias for APIFY_TOKEN) +# --------------------------------------------------------------------------- + + +class TestApifyApiTokenEnvVarAlias: + """The ``APIFY_API_TOKEN`` env var is honoured but deprecated. + + Mirrors the kwarg matrix: ``APIFY_TOKEN`` is preferred and silent, + ``APIFY_API_TOKEN`` works but emits a ``DeprecationWarning``. + """ + + def test_apify_token_env_no_warning(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv('APIFY_TOKEN', 'new-style') + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + token = _resolve_apify_token() + assert token == 'new-style' + assert len(w) == 0 + + def test_apify_api_token_env_emits_warning(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_TOKEN', raising=False) + monkeypatch.setenv('APIFY_API_TOKEN', 'legacy-style') + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + token = _resolve_apify_token() + assert token == 'legacy-style' + assert len(w) == 1 + assert issubclass(w[0].category, DeprecationWarning) + assert 'APIFY_API_TOKEN' in str(w[0].message) + + def test_apify_token_env_takes_precedence_silently(self, monkeypatch: pytest.MonkeyPatch) -> None: + """When both env vars are set, ``APIFY_TOKEN`` wins with no warning.""" + monkeypatch.setenv('APIFY_TOKEN', 'primary') + monkeypatch.setenv('APIFY_API_TOKEN', 'ignored') + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + token = _resolve_apify_token() + assert token == 'primary' + assert len(w) == 0 + + def test_no_token_env_returns_none_without_warning(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_TOKEN', raising=False) + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always') + token = _resolve_apify_token() + assert token is None + assert len(w) == 0 diff --git a/tests/unit_tests/test_document_loaders.py b/tests/unit_tests/test_document_loaders.py index a6c7a61..a09ec09 100644 --- a/tests/unit_tests/test_document_loaders.py +++ b/tests/unit_tests/test_document_loaders.py @@ -1,10 +1,17 @@ -from unittest.mock import patch +from __future__ import annotations +from typing import Any +from unittest.mock import MagicMock, patch + +import pytest from apify_client._types import ListPage from apify_client.clients import DatasetClient from langchain_core.documents import Document +from pydantic import SecretStr -from langchain_apify import ApifyDatasetLoader +from langchain_apify import ApifyCrawlLoader, ApifyDatasetLoader +from langchain_apify._client import ApifyToolsClient +from tests.unit_tests.conftest import SUCCEEDED_RUN def test_apify_dataset_loader_load() -> None: @@ -18,7 +25,7 @@ def test_apify_dataset_loader_load() -> None: ) loader = ApifyDatasetLoader( - apify_api_token='dummy-token', + apify_token='dummy-token', dataset_id='dummy-dataset-id', dataset_mapping_function=lambda item: Document( page_content=item['text'], @@ -43,7 +50,7 @@ def test_apify_dataset_loader_lazy_load() -> None: ) loader = ApifyDatasetLoader( - apify_api_token='dummy-token', + apify_token='dummy-token', dataset_id='dummy-dataset-id', dataset_mapping_function=lambda item: Document( page_content=item['text'], @@ -55,3 +62,168 @@ def test_apify_dataset_loader_lazy_load() -> None: mock_list_items.assert_called_once() assert documents[0].page_content == 'Apify is great!' assert documents[0].metadata['source'] == 'https://apify.com' + + +# --------------------------------------------------------------------------- +# ApifyCrawlLoader +# --------------------------------------------------------------------------- + +CRAWL_ITEMS: list[dict] = [ + { + 'url': 'https://example.com/', + 'markdown': '# Home', + 'text': 'Home', + 'metadata': {'title': 'Home Page'}, + 'crawl': {'depth': 0}, + }, + { + 'url': 'https://example.com/about', + 'markdown': '# About', + 'text': 'About', + 'metadata': {'title': 'About Page'}, + 'crawl': {'depth': 1}, + }, +] + + +def _make_crawl_loader( + mock_client: MagicMock, + **kwargs: Any, # noqa: ANN401 +) -> ApifyCrawlLoader: + with patch.object(ApifyToolsClient, '__init__', return_value=None): + loader = ApifyCrawlLoader(url='https://example.com', apify_token='dummy', **kwargs) + loader._client = mock_client + return loader + + +def test_crawl_loader_lazy_load() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.crawl_website.return_value = (SUCCEEDED_RUN, CRAWL_ITEMS) + loader = _make_crawl_loader(mock_client) + + docs = list(loader.lazy_load()) + + assert len(docs) == 2 + assert all(isinstance(d, Document) for d in docs) + assert docs[0].page_content == '# Home' + assert docs[0].metadata['source'] == 'https://example.com/' + assert docs[0].metadata['title'] == 'Home Page' + assert docs[0].metadata['crawl_depth'] == 0 + assert docs[1].page_content == '# About' + assert docs[1].metadata['crawl_depth'] == 1 + + +def test_crawl_loader_load_delegates_to_lazy_load() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.crawl_website.return_value = (SUCCEEDED_RUN, CRAWL_ITEMS) + loader = _make_crawl_loader(mock_client) + + docs = loader.load() + + assert len(docs) == 2 + assert docs[0].page_content == '# Home' + + +def test_crawl_loader_passes_params() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.crawl_website.return_value = (SUCCEEDED_RUN, []) + loader = _make_crawl_loader( + mock_client, + max_crawl_pages=5, + max_crawl_depth=2, + crawler_type='playwright:firefox', + timeout_secs=120, + ) + + list(loader.lazy_load()) + + mock_client.crawl_website.assert_called_once_with( + 'https://example.com', + max_crawl_pages=5, + max_crawl_depth=2, + crawler_type='playwright:firefox', + timeout_secs=120, + ) + + +def test_crawl_loader_empty_results() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.crawl_website.return_value = (SUCCEEDED_RUN, []) + loader = _make_crawl_loader(mock_client) + + docs = loader.load() + + assert docs == [] + + +def test_crawl_loader_text_fallback() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.crawl_website.return_value = ( + SUCCEEDED_RUN, + [{'url': 'https://example.com/', 'text': 'Plain text', 'metadata': {'title': 'T'}}], + ) + loader = _make_crawl_loader(mock_client) + + docs = list(loader.lazy_load()) + + assert docs[0].page_content == 'Plain text' + + +def test_crawl_loader_missing_metadata() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.crawl_website.return_value = ( + SUCCEEDED_RUN, + [{'url': 'https://example.com/', 'markdown': '# Content'}], + ) + loader = _make_crawl_loader(mock_client) + + docs = list(loader.lazy_load()) + + assert docs[0].metadata['title'] == '' + assert docs[0].metadata['crawl_depth'] == 0 + + +def test_crawl_loader_missing_token(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + ApifyCrawlLoader(url='https://example.com') + + +def test_crawl_loader_accepts_secretstr_token() -> None: + with patch('langchain_apify._client._create_apify_client'): + loader = ApifyCrawlLoader(url='https://example.com', apify_token=SecretStr('s')) + assert loader.url == 'https://example.com' + + +def test_crawl_loader_failure_raises() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.crawl_website.side_effect = RuntimeError('Actor run run-bad ended with status FAILED.') + loader = _make_crawl_loader(mock_client) + + with pytest.raises(RuntimeError, match='FAILED'): + loader.load() + + +def test_apify_dataset_loader_apify_token_fallback(monkeypatch: pytest.MonkeyPatch) -> None: + """Loader should accept APIFY_TOKEN as a secondary env-var fallback.""" + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.setenv('APIFY_TOKEN', 'platform-token') + + with patch.object(DatasetClient, 'list_items') as mock_list_items: + mock_list_items.return_value = ListPage(data={'items': []}) + loader = ApifyDatasetLoader( + dataset_id='d', + dataset_mapping_function=lambda _item: Document(page_content='x'), + ) + assert loader.load() == [] + + +def test_apify_dataset_loader_missing_token(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + ApifyDatasetLoader( + dataset_id='d', + dataset_mapping_function=lambda _item: Document(page_content='x'), + ) diff --git a/tests/unit_tests/test_merge_surface.py b/tests/unit_tests/test_merge_surface.py new file mode 100644 index 0000000..c9c6b52 --- /dev/null +++ b/tests/unit_tests/test_merge_surface.py @@ -0,0 +1,51 @@ +"""Package-surface invariants for the merged tool families. + +Ported from the offline ``test_merge_surface`` playground check: the core, +social, and search tool groups must keep their expected sizes, stay disjoint, +expose globally-unique tool ``name`` values (a name collision would surface +here), and keep ``_run_meta`` in a single canonical home. +""" + +from __future__ import annotations + +import pytest + +from langchain_apify import ( + APIFY_CORE_TOOLS, + APIFY_SEARCH_TOOLS, + APIFY_SOCIAL_TOOLS, +) + + +@pytest.mark.parametrize( + ('group', 'expected'), + [ + (APIFY_CORE_TOOLS, 6), + (APIFY_SOCIAL_TOOLS, 7), + (APIFY_SEARCH_TOOLS, 6), + ], +) +def test_tool_group_sizes(group: list[type], expected: int) -> None: + assert len(group) == expected + + +def test_tool_groups_are_disjoint() -> None: + assert not set(APIFY_SOCIAL_TOOLS) & set(APIFY_SEARCH_TOOLS) + assert not set(APIFY_CORE_TOOLS) & set(APIFY_SOCIAL_TOOLS) + assert not set(APIFY_CORE_TOOLS) & set(APIFY_SEARCH_TOOLS) + + +def test_tool_names_are_globally_unique() -> None: + all_tools = [*APIFY_CORE_TOOLS, *APIFY_SOCIAL_TOOLS, *APIFY_SEARCH_TOOLS] + names = [tool_cls.model_fields['name'].default for tool_cls in all_tools] + dupes = sorted({n for n in names if names.count(n) > 1}) + assert not dupes, f'duplicate tool names: {dupes}' + + +def test_run_meta_has_single_canonical_home() -> None: + # _run_meta lives in tools.base; a stale duplicate in _utils would mean two + # diverging copies after the tools/ package split. + from langchain_apify.tools.base import _run_meta # noqa: F401 + + with pytest.raises(ImportError): + from langchain_apify._utils import _run_meta as _stale # type: ignore[attr-defined] # noqa: F401 diff --git a/tests/unit_tests/test_retrievers.py b/tests/unit_tests/test_retrievers.py new file mode 100644 index 0000000..dfca4e3 --- /dev/null +++ b/tests/unit_tests/test_retrievers.py @@ -0,0 +1,277 @@ +from __future__ import annotations + +from typing import Any +from unittest.mock import MagicMock, patch + +import pytest +from langchain_core.documents import Document +from pydantic import SecretStr + +from langchain_apify._client import ApifyToolsClient +from langchain_apify._constants import _RAG_MAX_RESULTS_CAP +from langchain_apify.retrievers import ApifySearchRetriever + +RAG_ITEMS: list[dict] = [ + { + 'crawledUrl': 'https://example.com/1', + 'text': 'Page 1 content', + 'metadata': {'title': 'Page 1'}, + }, + { + 'crawledUrl': 'https://example.com/2', + 'text': 'Page 2 content', + 'metadata': {'title': 'Page 2'}, + }, +] + + +def _make_retriever(mock_client: MagicMock, **kwargs: Any) -> ApifySearchRetriever: # noqa: ANN401 + """Instantiate a retriever with a mocked ApifyToolsClient.""" + with patch.object(ApifyToolsClient, '__init__', return_value=None): + retriever = ApifySearchRetriever(apify_token=SecretStr('dummy-token'), **kwargs) + retriever._client = mock_client + return retriever + + +# --------------------------------------------------------------------------- +# __init__ +# --------------------------------------------------------------------------- + + +def test_missing_token_raises(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + ApifySearchRetriever() + + +def test_init_with_explicit_token() -> None: + with patch.object(ApifyToolsClient, '__init__', return_value=None): + retriever = ApifySearchRetriever(apify_token=SecretStr('my-token')) + assert retriever.max_results == 5 + assert retriever.timeout_secs == 300 + + +def test_init_custom_params() -> None: + with patch.object(ApifyToolsClient, '__init__', return_value=None): + retriever = ApifySearchRetriever(apify_token=SecretStr('t'), max_results=3, timeout_secs=60) + assert retriever.max_results == 3 + assert retriever.timeout_secs == 60 + + +def test_deprecated_apify_api_token_alias_warns() -> None: + # ``apify_api_token`` is a runtime alias handled by a model validator, not a + # declared field, hence the call-arg ignore. + with patch.object(ApifyToolsClient, '__init__', return_value=None): + with pytest.warns(DeprecationWarning, match='apify_api_token'): + retriever = ApifySearchRetriever(apify_api_token=SecretStr('legacy-token')) # type: ignore[call-arg] + assert retriever.apify_token == SecretStr('legacy-token') + + +# --------------------------------------------------------------------------- +# Sync retrieval +# --------------------------------------------------------------------------- + + +def test_sync_returns_documents() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.rag_web_search.return_value = ({}, RAG_ITEMS) + retriever = _make_retriever(mock_client, max_results=5) + + docs = retriever._get_relevant_documents('test query') + + assert len(docs) == 2 + assert all(isinstance(d, Document) for d in docs) + assert docs[0].page_content == 'Page 1 content' + assert docs[0].metadata['source'] == 'https://example.com/1' + assert docs[0].metadata['title'] == 'Page 1' + assert docs[1].page_content == 'Page 2 content' + assert docs[1].metadata['source'] == 'https://example.com/2' + + +def test_sync_calls_helper_with_correct_args() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.rag_web_search.return_value = ({}, []) + retriever = _make_retriever(mock_client, max_results=3, timeout_secs=60) + + retriever._get_relevant_documents('my search') + + mock_client.rag_web_search.assert_called_once_with( + 'my search', + max_results=3, + timeout_secs=60, + ) + + +def test_max_results_clamped_to_actor_cap() -> None: + # The rag-web-browser Actor rejects maxResults > 100; a larger value must be + # clamped down rather than passed through and rejected at runtime. + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.rag_web_search.return_value = ({}, []) + retriever = _make_retriever(mock_client, max_results=_RAG_MAX_RESULTS_CAP + 50) + + retriever._get_relevant_documents('big query') + + _, kwargs = mock_client.rag_web_search.call_args + assert kwargs['max_results'] == _RAG_MAX_RESULTS_CAP + + +def test_sync_empty_results() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.rag_web_search.return_value = ({}, []) + retriever = _make_retriever(mock_client) + + docs = retriever._get_relevant_documents('test') + + assert docs == [] + + +def test_sync_helper_failure_propagates() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.rag_web_search.side_effect = RuntimeError( + 'Actor run run-bad ended with status FAILED.', + ) + retriever = _make_retriever(mock_client) + + with pytest.raises(RuntimeError, match='FAILED'): + retriever._get_relevant_documents('test') + + +# --------------------------------------------------------------------------- +# Async retrieval +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_async_returns_documents() -> None: + """Async path wraps the sync helper via asyncio.to_thread.""" + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.rag_web_search.return_value = ({}, RAG_ITEMS) + retriever = _make_retriever(mock_client, max_results=5) + + docs = await retriever._aget_relevant_documents('test query') + + assert len(docs) == 2 + assert all(isinstance(d, Document) for d in docs) + assert docs[0].page_content == 'Page 1 content' + assert docs[0].metadata['source'] == 'https://example.com/1' + + +@pytest.mark.asyncio +async def test_async_calls_helper_with_correct_args() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.rag_web_search.return_value = ({}, []) + retriever = _make_retriever(mock_client, max_results=3, timeout_secs=60) + + await retriever._aget_relevant_documents('my search') + + mock_client.rag_web_search.assert_called_once_with( + 'my search', + max_results=3, + timeout_secs=60, + ) + + +@pytest.mark.asyncio +async def test_async_empty_results() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.rag_web_search.return_value = ({}, []) + retriever = _make_retriever(mock_client) + + docs = await retriever._aget_relevant_documents('test') + + assert docs == [] + + +@pytest.mark.asyncio +async def test_async_helper_failure_propagates() -> None: + mock_client = MagicMock(spec=ApifyToolsClient) + mock_client.rag_web_search.side_effect = RuntimeError( + 'Actor run run-bad ended with status FAILED.', + ) + retriever = _make_retriever(mock_client) + + with pytest.raises(RuntimeError, match='FAILED'): + await retriever._aget_relevant_documents('test') + + +# --------------------------------------------------------------------------- +# _items_to_documents edge cases +# --------------------------------------------------------------------------- + + +def test_items_to_documents_uses_url_fallback() -> None: + items = [{'url': 'https://fallback.com', 'text': 'content', 'metadata': {'title': 'T'}}] + + docs = ApifySearchRetriever._items_to_documents(items) + + assert docs[0].metadata['source'] == 'https://fallback.com' + + +def test_items_to_documents_uses_metadata_url_fallback() -> None: + """apify/rag-web-browser nests the page URL under metadata.url.""" + items = [ + { + 'metadata': {'url': 'https://nested.example.com', 'title': 'Nested'}, + 'text': 'content', + }, + ] + + docs = ApifySearchRetriever._items_to_documents(items) + + assert docs[0].metadata['source'] == 'https://nested.example.com' + assert docs[0].metadata['title'] == 'Nested' + + +def test_items_to_documents_metadata_url_wins_over_crawled_url() -> None: + """Regression: when both are present, metadata.url must win over crawledUrl.""" + items = [ + { + 'metadata': {'url': 'https://meta.example.com', 'title': 'T'}, + 'crawledUrl': 'https://crawled.example.com', + 'text': 'content', + }, + ] + + docs = ApifySearchRetriever._items_to_documents(items) + + assert docs[0].metadata['source'] == 'https://meta.example.com' + + +def test_items_to_documents_metadata_url_wins_over_top_level_url() -> None: + items = [ + { + 'metadata': {'url': 'https://meta.example.com'}, + 'url': 'https://top.example.com', + 'text': 'content', + }, + ] + + docs = ApifySearchRetriever._items_to_documents(items) + + assert docs[0].metadata['source'] == 'https://meta.example.com' + + +def test_items_to_documents_uses_markdown_fallback() -> None: + items = [{'crawledUrl': 'https://example.com', 'markdown': '# MD content', 'metadata': {'title': 'T'}}] + + docs = ApifySearchRetriever._items_to_documents(items) + + assert docs[0].page_content == '# MD content' + + +def test_items_to_documents_missing_metadata() -> None: + items = [{'crawledUrl': 'https://example.com', 'text': 'content'}] + + docs = ApifySearchRetriever._items_to_documents(items) + + assert docs[0].metadata['title'] == '' + assert docs[0].metadata['source'] == 'https://example.com' + + +def test_items_to_documents_non_dict_metadata() -> None: + items = [{'crawledUrl': 'https://example.com', 'text': 'content', 'metadata': 'not-a-dict'}] + + docs = ApifySearchRetriever._items_to_documents(items) + + assert docs[0].metadata['title'] == '' diff --git a/tests/unit_tests/test_tool_response_schema.py b/tests/unit_tests/test_tool_response_schema.py new file mode 100644 index 0000000..504b31c --- /dev/null +++ b/tests/unit_tests/test_tool_response_schema.py @@ -0,0 +1,116 @@ +from __future__ import annotations + +import json +from unittest.mock import MagicMock + +import pytest + +from langchain_apify import ( + APIFY_CORE_TOOLS, + APIFY_SEARCH_TOOLS, + ApifyGetDatasetItemsTool, + ApifyRunActorAndGetDatasetTool, + ApifyRunActorTool, + ApifyRunTaskAndGetDatasetTool, + ApifyRunTaskTool, + ApifyScrapeUrlTool, +) +from langchain_apify.tools import ( + ApifyEcommerceScraperTool, + ApifyGoogleMapsTool, + ApifyGoogleSearchTool, + ApifyRAGWebBrowserTool, + ApifyWebCrawlerTool, + ApifyYouTubeScraperTool, +) +from tests.unit_tests.conftest import SAMPLE_ITEMS, SUCCEEDED_RUN, make_tool + + +def _assert_envelope_shape(payload: dict) -> None: + assert set(payload) == {'run', 'items'} + assert isinstance(payload['items'], list) + assert payload['run'] is None or isinstance(payload['run'], dict) + + +@pytest.mark.parametrize( + ('tool_cls', 'setup_method', 'run_kwargs'), + [ + (ApifyRunActorTool, 'run_actor', {'actor_id': 'apify/test'}), + (ApifyGetDatasetItemsTool, 'get_dataset_items', {'dataset_id': 'dataset-xyz'}), + (ApifyRunActorAndGetDatasetTool, 'run_actor_and_get_items', {'actor_id': 'apify/test'}), + (ApifyScrapeUrlTool, 'scrape_url_with_metadata', {'url': 'https://example.com'}), + (ApifyRunTaskTool, 'run_task', {'task_id': 'user/my-task'}), + (ApifyRunTaskAndGetDatasetTool, 'run_task_and_get_items', {'task_id': 'user/my-task'}), + (ApifyGoogleSearchTool, 'google_search', {'query': 'apify'}), + (ApifyWebCrawlerTool, 'crawl_website', {'url': 'https://example.com'}), + (ApifyRAGWebBrowserTool, 'rag_web_search', {'query': 'langchain'}), + (ApifyGoogleMapsTool, 'google_maps_search', {'query': 'coffee'}), + (ApifyYouTubeScraperTool, 'youtube_scrape', {'search_query': 'langchain'}), + (ApifyEcommerceScraperTool, 'ecommerce_scrape', {'url': 'https://shop.example.com/p/1'}), + ], +) +def test_all_tools_return_normalized_envelope( + mock_tools_client: MagicMock, tool_cls: type, setup_method: str, run_kwargs: dict +) -> None: + if setup_method in {'run_actor', 'run_task'}: + getattr(mock_tools_client, setup_method).return_value = SUCCEEDED_RUN + elif setup_method == 'get_dataset_items': + getattr(mock_tools_client, setup_method).return_value = SAMPLE_ITEMS + elif setup_method == 'scrape_url_with_metadata': + mock_tools_client.scrape_url_with_metadata.return_value = ( + SUCCEEDED_RUN, + [{'url': 'https://example.com', 'markdown': '# content'}], + '# content', + 'markdown', + ) + elif setup_method in {'run_actor_and_get_items', 'run_task_and_get_items'}: + getattr(mock_tools_client, setup_method).return_value = (SUCCEEDED_RUN, SAMPLE_ITEMS) + elif setup_method == 'google_search': + mock_tools_client.google_search.return_value = ( + SUCCEEDED_RUN, + [{'title': 'A', 'url': 'https://a', 'description': 'd'}], + ) + elif setup_method == 'crawl_website': + mock_tools_client.crawl_website.return_value = ( + SUCCEEDED_RUN, + [{'url': 'https://example.com', 'markdown': '# Home', 'metadata': {'title': 'Home'}}], + ) + elif setup_method == 'rag_web_search': + mock_tools_client.rag_web_search.return_value = ( + SUCCEEDED_RUN, + [{'crawledUrl': 'https://example.com', 'metadata': {'title': 'Home'}, 'text': 'Home'}], + ) + elif setup_method in {'google_maps_search', 'youtube_scrape', 'ecommerce_scrape'}: + getattr(mock_tools_client, setup_method).return_value = (SUCCEEDED_RUN, SAMPLE_ITEMS) + + tool = make_tool(tool_cls, mock_tools_client) + payload = json.loads(tool._run(**run_kwargs)) + _assert_envelope_shape(payload) + + +def test_empty_result_is_normalized(mock_tools_client: MagicMock) -> None: + mock_tools_client.google_maps_search.return_value = (SUCCEEDED_RUN, []) + tool = make_tool(ApifyGoogleMapsTool, mock_tools_client) + + payload = json.loads(tool._run(query='empty')) + _assert_envelope_shape(payload) + assert payload['items'] == [] + + +def test_tool_group_lists_cover_all_normalized_tools() -> None: + assert set(APIFY_CORE_TOOLS) == { + ApifyRunActorTool, + ApifyGetDatasetItemsTool, + ApifyRunActorAndGetDatasetTool, + ApifyScrapeUrlTool, + ApifyRunTaskTool, + ApifyRunTaskAndGetDatasetTool, + } + assert set(APIFY_SEARCH_TOOLS) == { + ApifyGoogleSearchTool, + ApifyWebCrawlerTool, + ApifyRAGWebBrowserTool, + ApifyGoogleMapsTool, + ApifyYouTubeScraperTool, + ApifyEcommerceScraperTool, + } diff --git a/tests/unit_tests/test_tools.py b/tests/unit_tests/test_tools.py index b10df2f..4593ffd 100644 --- a/tests/unit_tests/test_tools.py +++ b/tests/unit_tests/test_tools.py @@ -1,13 +1,28 @@ from __future__ import annotations +import json +from datetime import datetime, timezone from typing import TYPE_CHECKING -from unittest.mock import patch +from unittest.mock import MagicMock, patch import pytest +from langchain_core.tools import ToolException from pydantic import BaseModel -from langchain_apify.tools import ApifyActorsTool -from langchain_apify.utils import actor_id_to_tool_name +from langchain_apify import APIFY_CORE_TOOLS +from langchain_apify._client import ApifyToolsClient +from langchain_apify._utils import _actor_id_to_tool_name +from langchain_apify.tools import ( + ApifyActorsTool, + ApifyGetDatasetItemsTool, + ApifyRunActorAndGetDatasetTool, + ApifyRunActorTool, + ApifyRunTaskAndGetDatasetTool, + ApifyRunTaskTool, + ApifyScrapeUrlTool, +) +from langchain_apify.tools.base import _ApifyGenericTool, _iso, _run_meta +from tests.unit_tests.conftest import SAMPLE_ITEMS, SUCCEEDED_RUN, make_tool if TYPE_CHECKING: from collections.abc import Generator @@ -20,6 +35,7 @@ def test_apify_actors_tool_instance() -> None: checks if the instance is created correctly. """ with ( + patch('langchain_apify.tools.actors._get_actor_latest_build', return_value={}), patch.object( ApifyActorsTool, '_create_description', @@ -37,10 +53,10 @@ class DummyModel(BaseModel): mock_build_tool_args_schema_model.return_value = DummyModel actor_id = 'apify/python-example' - tool = ApifyActorsTool(actor_id=actor_id, apify_api_token='dummy-token') + tool = ApifyActorsTool(actor_id=actor_id, apify_token='dummy-token') assert isinstance(tool, ApifyActorsTool) assert tool.description == 'Mocked description' - assert tool.name == actor_id_to_tool_name(actor_id) + assert tool.name == _actor_id_to_tool_name(actor_id) assert tool.args_schema == DummyModel @@ -52,8 +68,8 @@ def test_run_actor_method(apify_actors_tool_fixture: ApifyActorsTool) -> None: with patch.object(ApifyActorsTool, '_run_actor') as mock_run_actor: mock_run_actor.return_value = [{'text': 'Apify is great!'}] - result = apify_actors_tool_fixture.invoke( - input={'run_input': {'query': 'what is Apify?', 'maxResults': 3}}, + result = apify_actors_tool_fixture._run( + run_input={'query': 'what is Apify?', 'maxResults': 3}, ) mock_run_actor.assert_called_once() assert result[0]['text'] == 'Apify is great!' @@ -67,6 +83,7 @@ def apify_actors_tool_fixture() -> Generator[ApifyActorsTool, None, None]: ApifyActorsTool: An instance of the ApifyActorsTool. """ with ( + patch('langchain_apify.tools.actors._get_actor_latest_build', return_value={}), patch.object( ApifyActorsTool, '_create_description', @@ -83,5 +100,631 @@ class DummyModel(BaseModel): mock_build_tool_args_schema_model.return_value = DummyModel - tool = ApifyActorsTool(actor_id='apify/python-example', apify_api_token='dummy-token') + tool = ApifyActorsTool(actor_id='apify/python-example', apify_token='dummy-token') yield tool + + +# --------------------------------------------------------------------------- +# _iso / _run_meta helpers +# --------------------------------------------------------------------------- + + +def test_iso_converts_datetime_to_string() -> None: + dt = datetime(2025, 6, 15, 12, 30, 45, tzinfo=timezone.utc) + assert _iso(dt) == '2025-06-15T12:30:45+00:00' + + +def test_iso_passes_through_string() -> None: + assert _iso('2025-01-01T00:00:00.000Z') == '2025-01-01T00:00:00.000Z' + + +def test_iso_passes_through_none() -> None: + assert _iso(None) is None + + +def test_run_meta_with_datetime_values_is_json_serializable() -> None: + run = { + 'id': 'run-dt', + 'status': 'SUCCEEDED', + 'defaultDatasetId': 'ds-dt', + 'startedAt': datetime(2025, 3, 1, 10, 0, 0, tzinfo=timezone.utc), + 'finishedAt': datetime(2025, 3, 1, 10, 1, 0, tzinfo=timezone.utc), + } + meta = _run_meta(run) + serialized = json.dumps(meta) + parsed = json.loads(serialized) + assert parsed['run_id'] == 'run-dt' + assert parsed['started_at'] == '2025-03-01T10:00:00+00:00' + assert parsed['finished_at'] == '2025-03-01T10:01:00+00:00' + + +def test_run_meta_with_string_values_is_json_serializable() -> None: + meta = _run_meta(SUCCEEDED_RUN) + serialized = json.dumps(meta) + parsed = json.loads(serialized) + assert parsed['started_at'] == '2025-01-01T00:00:00.000Z' + assert parsed['finished_at'] == '2025-01-01T00:01:00.000Z' + + +def test_run_meta_with_missing_timestamps() -> None: + run = {'id': 'run-none', 'status': 'RUNNING', 'defaultDatasetId': 'ds-none'} + meta = _run_meta(run) + serialized = json.dumps(meta) + parsed = json.loads(serialized) + assert parsed['started_at'] is None + assert parsed['finished_at'] is None + + +def test_run_actor_tool_with_datetime_run(mock_tools_client: MagicMock) -> None: + """End-to-end: ApifyRunActorTool returns valid JSON when the client returns datetime objects.""" + mock_tools_client.run_actor.return_value = { + 'id': 'run-real', + 'status': 'SUCCEEDED', + 'defaultDatasetId': 'ds-real', + 'startedAt': datetime(2025, 6, 1, 8, 0, 0, tzinfo=timezone.utc), + 'finishedAt': datetime(2025, 6, 1, 8, 5, 0, tzinfo=timezone.utc), + } + tool = make_tool(ApifyRunActorTool, mock_tools_client) + + result = tool._run(actor_id='apify/test') + + parsed = json.loads(result) + assert parsed['run']['run_id'] == 'run-real' + assert parsed['run']['started_at'] == '2025-06-01T08:00:00+00:00' + assert parsed['run']['finished_at'] == '2025-06-01T08:05:00+00:00' + assert parsed['items'] == [] + + +def test_tool_response_handles_datetime_in_items(mock_tools_client: MagicMock) -> None: + """Regression: datetime values inside ``items`` must not break serialization. + + The Apify client's ``clean=True`` deserialiser returns ``datetime`` + objects for certain timestamp fields (Google Maps reviews, YouTube + publishedAt, etc.). Without ``default=str`` in ``json.dumps``, this + raised ``TypeError: Object of type datetime is not JSON serializable`` + and the LLM saw an empty / error tool result instead of data. + """ + timestamp = datetime(2026, 1, 2, 3, 4, 5, tzinfo=timezone.utc) + items = [{'id': 'item-1', 'published_at': timestamp, 'text': 'hi'}] + mock_tools_client.run_actor_and_get_items.return_value = (SUCCEEDED_RUN, items) + tool = make_tool(ApifyRunActorAndGetDatasetTool, mock_tools_client) + + parsed = json.loads(tool._run(actor_id='apify/test')) + + assert parsed['items'][0]['id'] == 'item-1' + assert isinstance(parsed['items'][0]['published_at'], str) + assert '2026-01-02' in parsed['items'][0]['published_at'] + + +# --------------------------------------------------------------------------- +# ApifyRunActorTool +# --------------------------------------------------------------------------- + + +def test_run_actor_tool_returns_json(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_actor.return_value = SUCCEEDED_RUN + tool = make_tool(ApifyRunActorTool, mock_tools_client) + + result = tool._run(actor_id='apify/test', run_input={'key': 'val'}) + + parsed = json.loads(result) + assert parsed['run']['run_id'] == 'run-abc' + assert parsed['run']['status'] == 'SUCCEEDED' + assert parsed['run']['dataset_id'] == 'dataset-xyz' + assert parsed['run']['started_at'] == '2025-01-01T00:00:00.000Z' + assert parsed['run']['finished_at'] == '2025-01-01T00:01:00.000Z' + assert parsed['items'] == [] + mock_tools_client.run_actor.assert_called_once_with('apify/test', {'key': 'val'}, 300, None) + + +def test_run_actor_tool_failure_raises_tool_exception(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_actor.side_effect = RuntimeError('Actor run run-bad ended with status FAILED.') + tool = make_tool(ApifyRunActorTool, mock_tools_client) + + with pytest.raises(ToolException, match='FAILED'): + tool._run(actor_id='apify/test') + + +def test_run_actor_tool_missing_token(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + ApifyRunActorTool() + + +# --------------------------------------------------------------------------- +# ApifyGetDatasetItemsTool +# --------------------------------------------------------------------------- + + +def test_get_dataset_items_tool_returns_json_object(mock_tools_client: MagicMock) -> None: + mock_tools_client.get_dataset_items.return_value = SAMPLE_ITEMS + tool = make_tool(ApifyGetDatasetItemsTool, mock_tools_client) + + result = tool._run(dataset_id='dataset-xyz', limit=50, offset=5) + + parsed = json.loads(result) + assert len(parsed['items']) == 2 + assert parsed['items'][0]['text'] == 'item-1' + mock_tools_client.get_dataset_items.assert_called_once_with('dataset-xyz', 50, 5) + + +def test_get_dataset_items_tool_empty_returns_empty_items(mock_tools_client: MagicMock) -> None: + mock_tools_client.get_dataset_items.return_value = [] + tool = make_tool(ApifyGetDatasetItemsTool, mock_tools_client) + + result = tool._run(dataset_id='dataset-empty') + + parsed = json.loads(result) + assert parsed == {'run': None, 'items': []} + + +def test_get_dataset_items_tool_network_error_raises_tool_exception(mock_tools_client: MagicMock) -> None: + mock_tools_client.get_dataset_items.side_effect = RuntimeError( + 'Apify dataset fetch failed for ds-bad: connection reset' + ) + tool = make_tool(ApifyGetDatasetItemsTool, mock_tools_client) + + with pytest.raises(ToolException, match='Apify dataset fetch failed'): + tool._run(dataset_id='ds-bad') + + +def test_get_dataset_items_tool_missing_token(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + ApifyGetDatasetItemsTool() + + +# --------------------------------------------------------------------------- +# ApifyRunActorAndGetDatasetTool +# --------------------------------------------------------------------------- + + +def test_run_actor_and_get_items_tool_returns_json(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_actor_and_get_items.return_value = (SUCCEEDED_RUN, SAMPLE_ITEMS) + tool = make_tool(ApifyRunActorAndGetDatasetTool, mock_tools_client) + + result = tool._run(actor_id='apify/test', run_input={'q': '1'}, dataset_items_limit=50) + + parsed = json.loads(result) + assert parsed['run']['run_id'] == 'run-abc' + assert parsed['run']['status'] == 'SUCCEEDED' + assert len(parsed['items']) == 2 + mock_tools_client.run_actor_and_get_items.assert_called_once_with('apify/test', {'q': '1'}, 300, None, 50) + + +def test_run_actor_and_get_items_tool_failure_raises_tool_exception(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_actor_and_get_items.side_effect = RuntimeError( + 'Actor run run-bad ended with status TIMED-OUT.' + ) + tool = make_tool(ApifyRunActorAndGetDatasetTool, mock_tools_client) + + with pytest.raises(ToolException, match='TIMED-OUT'): + tool._run(actor_id='apify/test') + + +def test_run_actor_and_get_items_tool_missing_token(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + ApifyRunActorAndGetDatasetTool() + + +# --------------------------------------------------------------------------- +# ApifyScrapeUrlTool +# --------------------------------------------------------------------------- + + +def test_scrape_url_tool_returns_markdown(mock_tools_client: MagicMock) -> None: + mock_tools_client.scrape_url_with_metadata.return_value = ( + SUCCEEDED_RUN, + [{'url': 'https://example.com', 'markdown': '# Hello World'}], + '# Hello World', + 'markdown', + ) + tool = make_tool(ApifyScrapeUrlTool, mock_tools_client) + + result = tool._run(url='https://example.com') + + parsed = json.loads(result) + assert parsed['run']['status'] == 'SUCCEEDED' + assert parsed['items'] == [{'url': 'https://example.com', 'content': '# Hello World'}] + mock_tools_client.scrape_url_with_metadata.assert_called_once_with('https://example.com', 120) + + +def test_scrape_url_tool_empty_raises_tool_exception(mock_tools_client: MagicMock) -> None: + mock_tools_client.scrape_url_with_metadata.side_effect = RuntimeError( + 'No content extracted from https://example.com.' + ) + tool = make_tool(ApifyScrapeUrlTool, mock_tools_client) + + with pytest.raises(ToolException, match='No content extracted'): + tool._run(url='https://example.com') + + +def test_scrape_url_tool_missing_token(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + ApifyScrapeUrlTool() + + +# --------------------------------------------------------------------------- +# ApifyRunTaskTool +# --------------------------------------------------------------------------- + + +def test_run_task_tool_returns_json(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_task.return_value = SUCCEEDED_RUN + tool = make_tool(ApifyRunTaskTool, mock_tools_client) + + result = tool._run(task_id='user/my-task', task_input={'key': 'val'}) + + parsed = json.loads(result) + assert parsed['run']['run_id'] == 'run-abc' + assert parsed['run']['status'] == 'SUCCEEDED' + assert parsed['run']['dataset_id'] == 'dataset-xyz' + assert parsed['run']['started_at'] == '2025-01-01T00:00:00.000Z' + assert parsed['run']['finished_at'] == '2025-01-01T00:01:00.000Z' + assert parsed['items'] == [] + mock_tools_client.run_task.assert_called_once_with('user/my-task', {'key': 'val'}, 300, None) + + +def test_run_task_tool_failure_raises_tool_exception(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_task.side_effect = RuntimeError('Actor run run-bad ended with status FAILED.') + tool = make_tool(ApifyRunTaskTool, mock_tools_client) + + with pytest.raises(ToolException, match='FAILED'): + tool._run(task_id='user/my-task') + + +def test_run_task_tool_missing_token(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + ApifyRunTaskTool() + + +# --------------------------------------------------------------------------- +# ApifyRunTaskAndGetDatasetTool +# --------------------------------------------------------------------------- + + +def test_run_task_and_get_items_tool_returns_json(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_task_and_get_items.return_value = (SUCCEEDED_RUN, SAMPLE_ITEMS) + tool = make_tool(ApifyRunTaskAndGetDatasetTool, mock_tools_client) + + result = tool._run(task_id='user/my-task', task_input={'q': '1'}, dataset_items_limit=50) + + parsed = json.loads(result) + assert parsed['run']['run_id'] == 'run-abc' + assert parsed['run']['status'] == 'SUCCEEDED' + assert len(parsed['items']) == 2 + mock_tools_client.run_task_and_get_items.assert_called_once_with('user/my-task', {'q': '1'}, 300, None, 50) + + +def test_run_task_and_get_items_tool_failure_raises_tool_exception(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_task_and_get_items.side_effect = RuntimeError( + 'Actor run run-bad ended with status TIMED-OUT.' + ) + tool = make_tool(ApifyRunTaskAndGetDatasetTool, mock_tools_client) + + with pytest.raises(ToolException, match='TIMED-OUT'): + tool._run(task_id='user/my-task') + + +def test_run_task_and_get_items_tool_missing_token(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv('APIFY_API_TOKEN', raising=False) + monkeypatch.delenv('APIFY_TOKEN', raising=False) + with pytest.raises(ValueError, match='APIFY_TOKEN'): + ApifyRunTaskAndGetDatasetTool() + + +# --------------------------------------------------------------------------- +# Value clamping (developer safety limits) +# --------------------------------------------------------------------------- + + +def test_run_actor_tool_clamps_timeout(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_actor.return_value = SUCCEEDED_RUN + tool = make_tool(ApifyRunActorTool, mock_tools_client, max_timeout_secs=60) + + tool._run(actor_id='apify/test', timeout_secs=9999) + + mock_tools_client.run_actor.assert_called_once_with('apify/test', None, 60, None) + + +def test_run_actor_tool_clamps_memory(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_actor.return_value = SUCCEEDED_RUN + tool = make_tool(ApifyRunActorTool, mock_tools_client, max_memory_mbytes=512) + + tool._run(actor_id='apify/test', memory_mbytes=8192) + + mock_tools_client.run_actor.assert_called_once_with('apify/test', None, 300, 512) + + +def test_run_actor_tool_passes_none_memory_through(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_actor.return_value = SUCCEEDED_RUN + tool = make_tool(ApifyRunActorTool, mock_tools_client, max_memory_mbytes=512) + + tool._run(actor_id='apify/test', memory_mbytes=None) + + mock_tools_client.run_actor.assert_called_once_with('apify/test', None, 300, None) + + +def test_get_dataset_items_tool_clamps_limit(mock_tools_client: MagicMock) -> None: + mock_tools_client.get_dataset_items.return_value = SAMPLE_ITEMS + tool = make_tool(ApifyGetDatasetItemsTool, mock_tools_client, max_items=10) + + tool._run(dataset_id='ds-1', limit=50000) + + mock_tools_client.get_dataset_items.assert_called_once_with('ds-1', 10, 0) + + +def test_run_actor_and_get_items_tool_clamps_all(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_actor_and_get_items.return_value = (SUCCEEDED_RUN, SAMPLE_ITEMS) + tool = make_tool( + ApifyRunActorAndGetDatasetTool, + mock_tools_client, + max_timeout_secs=30, + max_memory_mbytes=256, + max_items=5, + ) + + tool._run(actor_id='a', timeout_secs=9999, memory_mbytes=9999, dataset_items_limit=9999) + + mock_tools_client.run_actor_and_get_items.assert_called_once_with('a', None, 30, 256, 5) + + +def test_scrape_url_tool_clamps_timeout(mock_tools_client: MagicMock) -> None: + mock_tools_client.scrape_url_with_metadata.return_value = ( + SUCCEEDED_RUN, + [{'url': 'https://example.com', 'text': '# content'}], + '# content', + 'text', + ) + tool = make_tool(ApifyScrapeUrlTool, mock_tools_client, max_timeout_secs=30) + + tool._run(url='https://example.com', timeout_secs=9999) + + mock_tools_client.scrape_url_with_metadata.assert_called_once_with('https://example.com', 30) + + +def test_run_task_tool_clamps_timeout_and_memory(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_task.return_value = SUCCEEDED_RUN + tool = make_tool(ApifyRunTaskTool, mock_tools_client, max_timeout_secs=60, max_memory_mbytes=512) + + tool._run(task_id='t/1', timeout_secs=9999, memory_mbytes=9999) + + mock_tools_client.run_task.assert_called_once_with('t/1', None, 60, 512) + + +def test_run_task_and_get_items_tool_clamps_all(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_task_and_get_items.return_value = (SUCCEEDED_RUN, SAMPLE_ITEMS) + tool = make_tool( + ApifyRunTaskAndGetDatasetTool, + mock_tools_client, + max_timeout_secs=30, + max_memory_mbytes=256, + max_items=5, + ) + + tool._run(task_id='t/1', timeout_secs=9999, memory_mbytes=9999, dataset_items_limit=9999) + + mock_tools_client.run_task_and_get_items.assert_called_once_with('t/1', None, 30, 256, 5) + + +def test_clamp_timeout_floor_is_one(mock_tools_client: MagicMock) -> None: + mock_tools_client.run_actor.return_value = SUCCEEDED_RUN + tool = make_tool(ApifyRunActorTool, mock_tools_client, max_timeout_secs=600) + + tool._run(actor_id='apify/test', timeout_secs=-1) + mock_tools_client.run_actor.assert_called_once_with('apify/test', None, 1, None) + + mock_tools_client.run_actor.reset_mock() + tool._run(actor_id='apify/test', timeout_secs=0) + mock_tools_client.run_actor.assert_called_once_with('apify/test', None, 1, None) + + +def test_clamp_memory_non_positive_is_treated_as_none(mock_tools_client: MagicMock) -> None: + """memory_mbytes <= 0 maps to None so the Apify platform default is used.""" + mock_tools_client.run_actor.return_value = SUCCEEDED_RUN + tool = make_tool(ApifyRunActorTool, mock_tools_client, max_memory_mbytes=4096) + + tool._run(actor_id='apify/test', memory_mbytes=-1) + mock_tools_client.run_actor.assert_called_once_with('apify/test', None, 300, None) + + mock_tools_client.run_actor.reset_mock() + tool._run(actor_id='apify/test', memory_mbytes=0) + mock_tools_client.run_actor.assert_called_once_with('apify/test', None, 300, None) + + +def test_clamp_memory_floors_positive_below_platform_minimum(mock_tools_client: MagicMock) -> None: + """A positive memory_mbytes below the Apify platform minimum (128 MB) is floored to 128.""" + mock_tools_client.run_actor.return_value = SUCCEEDED_RUN + tool = make_tool(ApifyRunActorTool, mock_tools_client, max_memory_mbytes=4096) + + tool._run(actor_id='apify/test', memory_mbytes=64) + mock_tools_client.run_actor.assert_called_once_with('apify/test', None, 300, 128) + + mock_tools_client.run_actor.reset_mock() + tool._run(actor_id='apify/test', memory_mbytes=1) + mock_tools_client.run_actor.assert_called_once_with('apify/test', None, 300, 128) + + +@pytest.mark.parametrize( + ('input_mb', 'expected_mb'), + [ + (128, 128), # already valid + (200, 256), # snap up + (500, 512), # snap up + (1024, 1024), # already valid + (1500, 2048), # snap up + (2048, 2048), # already valid + (3000, 4096), # snap up + (16384, 16384), # already valid + (32768, 32768), # already valid (top of range) + ], +) +def test_clamp_memory_snaps_up_to_power_of_two(mock_tools_client: MagicMock, input_mb: int, expected_mb: int) -> None: + """``memory_mbytes`` is snapped UP to the next valid Apify power-of-2 value.""" + mock_tools_client.run_actor.return_value = SUCCEEDED_RUN + tool = make_tool(ApifyRunActorTool, mock_tools_client, max_memory_mbytes=32768) + + tool._run(actor_id='apify/test', memory_mbytes=input_mb) + mock_tools_client.run_actor.assert_called_once_with('apify/test', None, 300, expected_mb) + + +def test_clamp_memory_snap_up_capped_to_max(mock_tools_client: MagicMock) -> None: + """When snap-up would exceed ``max_memory_mbytes``, the largest valid value at-or-below the cap is used.""" + mock_tools_client.run_actor.return_value = SUCCEEDED_RUN + # cap is not itself a power of 2; clamped value (500) snaps up to 512 which exceeds cap → fall back to 256. + tool = make_tool(ApifyRunActorTool, mock_tools_client, max_memory_mbytes=500) + + tool._run(actor_id='apify/test', memory_mbytes=500) + mock_tools_client.run_actor.assert_called_once_with('apify/test', None, 300, 256) + + +def test_clamp_memory_misconfigured_cap_below_platform_minimum(mock_tools_client: MagicMock) -> None: + """If the developer-set cap is below 128 (the Apify minimum), fall back to 128 rather than overshooting.""" + mock_tools_client.run_actor.return_value = SUCCEEDED_RUN + tool = make_tool(ApifyRunActorTool, mock_tools_client, max_memory_mbytes=100) + + tool._run(actor_id='apify/test', memory_mbytes=100) + mock_tools_client.run_actor.assert_called_once_with('apify/test', None, 300, 128) + + +def test_clamp_items_floor_is_one(mock_tools_client: MagicMock) -> None: + mock_tools_client.get_dataset_items.return_value = SAMPLE_ITEMS + tool = make_tool(ApifyGetDatasetItemsTool, mock_tools_client, max_items=100) + + tool._run(dataset_id='ds-1', limit=-1) + mock_tools_client.get_dataset_items.assert_called_once_with('ds-1', 1, 0) + + mock_tools_client.get_dataset_items.reset_mock() + tool._run(dataset_id='ds-1', limit=0) + mock_tools_client.get_dataset_items.assert_called_once_with('ds-1', 1, 0) + + +@pytest.mark.parametrize( + ('depth', 'expected'), + [ + (-999, 0), # floored to 0 + (-1, 0), + (0, 0), # 0 means "only the seed URL" + (3, 3), # within range, passes through + (5, 5), # at the cap + (100, 5), # above the cap, clamped down + ], +) +def test_clamp_depth_floors_at_zero_and_caps(mock_tools_client: MagicMock, depth: int, expected: int) -> None: + """_clamp_depth floors negatives at 0 and clamps above-cap values to max_crawl_depth.""" + tool = make_tool(ApifyRunActorTool, mock_tools_client, max_crawl_depth=5) + assert tool._clamp_depth(depth) == expected + + +def test_negative_offset_clamped_to_zero(mock_tools_client: MagicMock) -> None: + """Negative offset values should be clamped to 0.""" + mock_tools_client.get_dataset_items.return_value = SAMPLE_ITEMS + tool = make_tool(ApifyGetDatasetItemsTool, mock_tools_client) + + tool._run(dataset_id='ds-1', offset=-5) + mock_tools_client.get_dataset_items.assert_called_once_with('ds-1', 100, 0) + + mock_tools_client.get_dataset_items.reset_mock() + tool._run(dataset_id='ds-1', offset=-1) + mock_tools_client.get_dataset_items.assert_called_once_with('ds-1', 100, 0) + + +def test_values_below_max_pass_through(mock_tools_client: MagicMock) -> None: + """When LLM values are within limits they should pass through unchanged.""" + mock_tools_client.run_actor.return_value = SUCCEEDED_RUN + tool = make_tool(ApifyRunActorTool, mock_tools_client, max_timeout_secs=600, max_memory_mbytes=4096) + + tool._run(actor_id='apify/test', timeout_secs=120, memory_mbytes=1024) + + mock_tools_client.run_actor.assert_called_once_with('apify/test', None, 120, 1024) + + +# --------------------------------------------------------------------------- +# Tool metadata assertions +# --------------------------------------------------------------------------- + + +def test_generic_tools_have_correct_metadata() -> None: + """Verify name, description, and args_schema are set on all generic tools.""" + with patch.object(ApifyToolsClient, '__init__', return_value=None): + tools = [ + ApifyRunActorTool(apify_token='dummy'), # type: ignore[call-arg,arg-type] + ApifyGetDatasetItemsTool(apify_token='dummy'), # type: ignore[call-arg,arg-type] + ApifyRunActorAndGetDatasetTool(apify_token='dummy'), # type: ignore[call-arg,arg-type] + ApifyScrapeUrlTool(apify_token='dummy'), # type: ignore[call-arg,arg-type] + ApifyRunTaskTool(apify_token='dummy'), # type: ignore[call-arg,arg-type] + ApifyRunTaskAndGetDatasetTool(apify_token='dummy'), # type: ignore[call-arg,arg-type] + ] + + expected_names = [ + 'apify_run_actor', + 'apify_get_dataset_items', + 'apify_run_actor_and_get_dataset', + 'apify_scrape_url', + 'apify_run_task', + 'apify_run_task_and_get_dataset', + ] + + for tool, expected_name in zip(tools, expected_names): + assert tool.name == expected_name + assert tool.description + assert tool.args_schema is not None + assert tool.handle_tool_error is True + + +def test_apify_token_excluded_from_model_dump() -> None: + """The apify_token field must not appear in model_dump() output.""" + with patch.object(ApifyToolsClient, '__init__', return_value=None): + tool = ApifyRunActorTool(apify_token='x') # type: ignore[call-arg,arg-type] + dumped = tool.model_dump() + assert 'apify_token' not in dumped + assert 'apify_api_token' not in dumped + + +# --------------------------------------------------------------------------- +# _ApifyGenericTool inheritance +# --------------------------------------------------------------------------- + + +def test_all_generic_tools_inherit_from_base() -> None: + """Every generic tool must be a subclass of _ApifyGenericTool.""" + for tool_cls in ( + ApifyRunActorTool, + ApifyGetDatasetItemsTool, + ApifyRunActorAndGetDatasetTool, + ApifyScrapeUrlTool, + ApifyRunTaskTool, + ApifyRunTaskAndGetDatasetTool, + ): + assert issubclass(tool_cls, _ApifyGenericTool), f'{tool_cls.__name__} must extend _ApifyGenericTool' + + +def test_legacy_tool_does_not_inherit_from_generic_base() -> None: + """ApifyActorsTool is legacy and must NOT inherit from _ApifyGenericTool.""" + assert not issubclass(ApifyActorsTool, _ApifyGenericTool) + + +# --------------------------------------------------------------------------- +# APIFY_CORE_TOOLS list +# --------------------------------------------------------------------------- + + +def test_apify_core_tools_contains_all_generic_classes() -> None: + """APIFY_CORE_TOOLS must list exactly the 6 generic tool classes.""" + assert set(APIFY_CORE_TOOLS) == { + ApifyRunActorTool, + ApifyGetDatasetItemsTool, + ApifyRunActorAndGetDatasetTool, + ApifyScrapeUrlTool, + ApifyRunTaskTool, + ApifyRunTaskAndGetDatasetTool, + } + assert len(APIFY_CORE_TOOLS) == 6