Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,20 +98,20 @@ Use `uv` for all environment and dependency operations in this monorepo. Do not

#### PR and commit titles

Follow Conventional Commits. See `.github/workflows/pr_lint.yml` for allowed types and scopes. All titles must include a scope with no exceptions — even for the main `langchain` package.
Follow Conventional Commits. See `.github/workflows/pr_lint.yml` for allowed types. Leave out the scope: this fork only releases `langchain-decodo`, so `(decodo)` adds nothing.

- Start the text after `type(scope):` with a lowercase letter, unless the first word is a proper noun (e.g. `Azure`, `GitHub`, `OpenAI`) or a named entity (class, function, method, parameter, or variable name).
- Start the text after `type:` with a lowercase letter, unless the first word is a proper noun (e.g. `Azure`, `GitHub`, `OpenAI`) or a named entity (class, function, method, parameter, or variable name).
- Wrap named entities in backticks so they render as code. Proper nouns are left unadorned.
- Keep titles short and descriptive — save detail for the body.

Examples:

```txt
feat(langchain): add new chat completion feature
fix(core): resolve type hinting issue in vector store
chore(anthropic): update infrastructure dependencies
feat(langchain): `ls_agent_type` tag on `create_agent` calls
fix(openai): infer Azure chat profiles from model name
feat: add web search, map and crawl tools
fix: return collected pages when a crawl is cancelled
docs: update model names in README examples
fix: retry `DecodoSearchTool` on status 613
ci: set up semantic release
```

#### Branch naming
Expand Down
14 changes: 7 additions & 7 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,20 +98,20 @@ Use `uv` for all environment and dependency operations in this monorepo. Do not

#### PR and commit titles

Follow Conventional Commits. See `.github/workflows/pr_lint.yml` for allowed types and scopes. All titles must include a scope with no exceptions — even for the main `langchain` package.
Follow Conventional Commits. See `.github/workflows/pr_lint.yml` for allowed types. Leave out the scope: this fork only releases `langchain-decodo`, so `(decodo)` adds nothing.

- Start the text after `type(scope):` with a lowercase letter, unless the first word is a proper noun (e.g. `Azure`, `GitHub`, `OpenAI`) or a named entity (class, function, method, parameter, or variable name).
- Start the text after `type:` with a lowercase letter, unless the first word is a proper noun (e.g. `Azure`, `GitHub`, `OpenAI`) or a named entity (class, function, method, parameter, or variable name).
- Wrap named entities in backticks so they render as code. Proper nouns are left unadorned.
- Keep titles short and descriptive — save detail for the body.

Examples:

```txt
feat(langchain): add new chat completion feature
fix(core): resolve type hinting issue in vector store
chore(anthropic): update infrastructure dependencies
feat(langchain): `ls_agent_type` tag on `create_agent` calls
fix(openai): infer Azure chat profiles from model name
feat: add web search, map and crawl tools
fix: return collected pages when a crawl is cancelled
docs: update model names in README examples
fix: retry `DecodoSearchTool` on status 613
ci: set up semantic release
```

#### Branch naming
Expand Down
109 changes: 98 additions & 11 deletions libs/partners/decodo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

LangChain partner package for the [Decodo](https://decodo.com) web scraping API.

Provides two **LangChain tools** and a **document loader** that let LLM agents
and RAG pipelines fetch live web content without managing proxies, JavaScript
rendering, or anti-bot protection.
Provides **LangChain tools** for scraping, web search, site mapping and crawling,
plus a **document loader**, that let LLM agents and RAG pipelines fetch live web
content without managing proxies, JavaScript rendering, or anti-bot protection.

## Installation

Expand All @@ -15,20 +15,26 @@ pip install langchain-decodo
## Authentication

Use the Web Data API key from your Web Data API subscription on the
[Decodo Dashboard](https://dashboard.decodo.com/web-data/playground). All classes read it from the `DECODO_API_TOKEN` environment variable, or you can pass it explicitly. Pass `auth_mode="token"` when using the key:
[Decodo Dashboard](https://dashboard.decodo.com/web-data/playground). All classes read it from the `DECODO_API_TOKEN` environment variable, or you can pass it explicitly. `DecodoWebScrapeTool`, `DecodoSearchTool` and `DecodoLoader` also need `auth_mode="token"` when using the key:

```bash
export DECODO_API_TOKEN="your-decodo-api-key"
```

```python
from langchain_decodo import DecodoWebScrapeTool

tool = DecodoWebScrapeTool(auth_mode="token")
```

Older plans only have a basic authentication token (base64-encoded
`username:password`). That is the default `auth_mode="basic"`, so omit
`auth_mode` and export the basic token as `DECODO_API_TOKEN`.

`DecodoWebSearchTool`, `DecodoMapTool`, `DecodoCrawlTool` and
`DecodoCrawlCancelTool` call endpoints that only accept an API key, so they need
no `auth_mode` and do not work with a basic authentication token.

## Components

### `DecodoWebScrapeTool`
Expand Down Expand Up @@ -86,6 +92,86 @@ Supported engines:
| `amazon` | `amazon_search` | Amazon product search |
| `reddit` | `google_search` + `site:reddit.com` | Reddit via Google |

### `DecodoWebSearchTool`

Search the web and return ranked results with title, link and description.
Unlike `DecodoSearchTool`, which scrapes a specific engine's results page, this
returns one normalized result document. Requires an API key.

```python
from langchain_decodo import DecodoWebSearchTool

tool = DecodoWebSearchTool()
results = tool.invoke({"query": "coffee shops", "geo": "United States", "page": 1})
```

Returns a JSON string; the results are under `organic`, alongside
`top_stories`, `people_also_ask` and `related_searches` when the search has them.

### `DecodoMapTool`

List the URLs on a website without fetching their content. Requires an API key.
Raises an error when the site could not be mapped, for example when it blocks
the request.

```python
from langchain_decodo import DecodoMapTool

tool = DecodoMapTool()
result = tool.invoke(
{
"url": "https://www.python.org",
"select_paths": ["^/downloads/"],
"limit": 50,
}
)
```

Optional arguments: `sitemap` (`include`, `only` or `skip`), `domain_filter`
(`domain`, `subdomain` or `all`), `select_paths` and `exclude_paths` (regular
expressions matched against URL paths), `geo`, `locale`, `limit` (up to 500,
default 100) and `js_render`.

### `DecodoCrawlTool` and `DecodoCrawlCancelTool`

Crawl a website from a start URL and return the content of every page found.
The tool runs the crawl through the Decodo SDK, waits for it to finish and
returns `{"status": {...}, "pages": [...]}`. Requires an API key.

```python
from langchain_decodo import DecodoCrawlTool

tool = DecodoCrawlTool(timeout=300, poll_interval=5)
result = tool.invoke({"url": "https://example.com", "limit": 10, "max_depth": 2})
```

Takes the same optional arguments as `DecodoMapTool` except `js_render`, plus
`limit` (up to 100, default 10), `max_depth` and `output` (`markdown` or
`html`).

`status` is the crawl's status response and each entry in `pages` is a page as
the API returns it, including its own `status`. A `status.status` other than
`done` means the crawl stopped early and `pages` holds only what was collected.
`status` is `null` when it could not be read, for example when the crawl
collected no pages.

`timeout` (seconds, default 600) is also sent as the crawl's `max_duration`,
capped at the API's 4-hour limit. A crawl still running at `timeout` comes back
as `pending` with the pages collected so far; Decodo stops it at the same limit
shortly after. To stop it sooner, pass `status.crawl_id` to
`DecodoCrawlCancelTool`:

```python
import json

from langchain_decodo import DecodoCrawlCancelTool, DecodoCrawlTool

result = json.loads(DecodoCrawlTool(timeout=60).invoke({"url": "https://example.com"}))
status = result["status"]
if status and status["status"] == "pending":
DecodoCrawlCancelTool().invoke({"crawl_id": status["crawl_id"]})
```

### `DecodoLoader`

Load one or more URLs as LangChain `Document` objects for use in RAG pipelines.
Expand Down Expand Up @@ -121,13 +207,14 @@ pip install langchain langchain-openai langchain-decodo

```python
from langchain.agents import create_agent
from langchain_decodo import DecodoSearchTool, DecodoWebScrapeTool
from langchain_decodo import DecodoMapTool, DecodoWebScrapeTool, DecodoWebSearchTool

agent = create_agent(
model="openai:gpt-4o-mini",
model="openai:gpt-6.1-sol",
tools=[
DecodoWebSearchTool(),
DecodoMapTool(),
DecodoWebScrapeTool(auth_mode="token"),
DecodoSearchTool(auth_mode="token"),
],
)

Expand All @@ -141,7 +228,7 @@ result = agent.invoke(
]
}
)
print(result["messages"][-1].content)
print(result["messages"][-1].text)
```

## RAG pipeline example
Expand All @@ -166,16 +253,16 @@ context = "\n\n".join(
doc.page_content for doc in store.similarity_search("What is Python used for?", k=4)
)

llm = ChatOpenAI(model="gpt-4o-mini")
llm = ChatOpenAI(model="gpt-6.1-sol")
answer = llm.invoke(
f"Answer using only this context:\n\n{context}\n\nQuestion: What is Python used for?"
)
print(answer.content)
print(answer.text)
```

## Links

- [Decodo website](https://decodo.com)
- [Decodo API documentation](https://help.decodo.com)
- [Decodo Dashboard](https://dashboard.decodo.com/)
- [LangChain documentation](https://python.langchain.com)
- [LangChain documentation](https://docs.langchain.com/oss/python/langchain/overview)
10 changes: 10 additions & 0 deletions libs/partners/decodo/langchain_decodo/__init__.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,20 @@
from langchain_decodo._version import __version__
from langchain_decodo.data_api_tools import (
DecodoCrawlCancelTool,
DecodoCrawlTool,
DecodoMapTool,
DecodoWebSearchTool,
)
from langchain_decodo.document_loaders import DecodoLoader
from langchain_decodo.tools import DecodoSearchTool, DecodoWebScrapeTool

__all__ = [
"DecodoCrawlCancelTool",
"DecodoCrawlTool",
"DecodoLoader",
"DecodoMapTool",
"DecodoSearchTool",
"DecodoWebScrapeTool",
"DecodoWebSearchTool",
"__version__",
]
32 changes: 32 additions & 0 deletions libs/partners/decodo/langchain_decodo/_common.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
"""Helpers shared by the scrape tools, the document loader and the Web Data API tools."""

from __future__ import annotations

import os
from typing import Any

from pydantic import SecretStr

INTEGRATION_HEADER = "langchain-python"

TOKEN_ENV_VAR = "DECODO_API_TOKEN"


def token_from_env(values: dict[str, Any]) -> dict[str, Any]:
"""Fill `decodo_api_token` from `DECODO_API_TOKEN` when it was not given.

Args:
values: Raw field values before model construction.

Returns:
The same values, with `decodo_api_token` populated from env if needed.
"""
if not values.get("decodo_api_token"):
values["decodo_api_token"] = SecretStr(os.environ.get(TOKEN_ENV_VAR, ""))
return values


def is_failed(entry: dict[str, Any]) -> bool:
"""Return whether a result entry carries a failed (4xx/5xx or 6xx) status."""
status = entry.get("status_code")
return isinstance(status, int) and status >= 400
Loading
Loading