Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion libs/partners/decodo/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -43,4 +43,4 @@ help:
@echo 'lint - run linters'
@echo 'format - run formatters'
@echo 'tests - run unit tests'
@echo 'integration_tests - run integration tests (requires DECODO_API_TOKEN)'
@echo 'integration_tests - run integration tests (requires an API key in DECODO_API_TOKEN)'
35 changes: 20 additions & 15 deletions libs/partners/decodo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,21 @@ pip install langchain-decodo

## Authentication

All classes read your Decodo API token from the `DECODO_API_TOKEN` environment
variable, or you can pass it explicitly.
Use the Web Data API key from your Web Data API subscription on the
[Decodo Dashboard](https://dashboard.decodo.com/web-data/playground). All classes read it from the `DECODO_API_TOKEN` environment variable, or you can pass it explicitly. Pass `auth_mode="token"` when using the key:

```bash
export DECODO_API_TOKEN="your-decodo-api-token"
export DECODO_API_TOKEN="your-decodo-api-key"
```

Get a token from the [Decodo Dashboard](https://app.decodo.com).

By default every class expects basic credentials (base64-encoded
`username:password`). If your token is a plain API token, pass
`auth_mode="token"`.:

```python
tool = DecodoWebScrapeTool(auth_mode="token")
```

Older plans only have a basic authentication token (base64-encoded
`username:password`). That is the default `auth_mode="basic"`, so omit
`auth_mode` and export the basic token as `DECODO_API_TOKEN`.

## Components

### `DecodoWebScrapeTool`
Expand All @@ -41,7 +39,7 @@ Handles JavaScript rendering, CAPTCHAs, and geo-blocking automatically.
```python
from langchain_decodo import DecodoWebScrapeTool

tool = DecodoWebScrapeTool() # reads DECODO_API_TOKEN from env
tool = DecodoWebScrapeTool(auth_mode="token") # reads DECODO_API_TOKEN from env
content = tool.run("https://example.com")
print(content)
```
Expand All @@ -52,7 +50,10 @@ Pass explicitly:
from pydantic import SecretStr
from langchain_decodo import DecodoWebScrapeTool

tool = DecodoWebScrapeTool(decodo_api_token=SecretStr("YOUR_TOKEN"))
tool = DecodoWebScrapeTool(
decodo_api_token=SecretStr("YOUR_API_KEY"),
auth_mode="token",
)
```

### `DecodoSearchTool`
Expand All @@ -62,7 +63,7 @@ Search Google, Amazon, or Reddit and return structured JSON results.
```python
from langchain_decodo import DecodoSearchTool

tool = DecodoSearchTool()
tool = DecodoSearchTool(auth_mode="token")

# Google search (default)
results = tool.run({"query": "LangChain latest release", "engine": "google"})
Expand Down Expand Up @@ -97,6 +98,7 @@ loader = DecodoLoader(
"https://python.org/about/",
"https://docs.python.org/3/whatsnew/3.12.html",
],
auth_mode="token",
)
docs = loader.load()

Expand All @@ -123,7 +125,10 @@ from langchain_decodo import DecodoSearchTool, DecodoWebScrapeTool

agent = create_agent(
model="openai:gpt-4o-mini",
tools=[DecodoWebScrapeTool(), DecodoSearchTool()],
tools=[
DecodoWebScrapeTool(auth_mode="token"),
DecodoSearchTool(auth_mode="token"),
],
)

result = agent.invoke(
Expand Down Expand Up @@ -151,7 +156,7 @@ from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_decodo import DecodoLoader

docs = DecodoLoader(urls=["https://python.org/about/"]).load()
docs = DecodoLoader(urls=["https://python.org/about/"], auth_mode="token").load()

splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter.split_documents(docs)
Expand All @@ -172,5 +177,5 @@ print(answer.content)

- [Decodo website](https://decodo.com)
- [Decodo API documentation](https://help.decodo.com)
- [Decodo Dashboard](https://app.decodo.com)
- [Decodo Dashboard](https://dashboard.decodo.com/)
- [LangChain documentation](https://python.langchain.com)
2 changes: 1 addition & 1 deletion libs/partners/decodo/langchain_decodo/_version.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "0.1.1"
__version__ = "0.1.2"
18 changes: 9 additions & 9 deletions libs/partners/decodo/langchain_decodo/document_loaders.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

* ``"basic"`` *(default)* — username:password credentials encoded as a Basic
auth token. Uses the ``/v2/scrape`` endpoint.
* ``"token"`` — a plain API token. Uses the ``/unified/v1/scrape`` endpoint.
* ``"token"`` — an API key. Uses the ``/unified/v1/scrape`` endpoint.

Typical usage — basic auth (default)::

Expand All @@ -27,7 +27,7 @@

loader = DecodoLoader(
urls=["https://example.com"],
api_token="your-api-token",
api_token="your-api-key",
auth_mode="token",
)
docs = loader.load()
Expand Down Expand Up @@ -69,7 +69,7 @@ def _build_headers(token: str) -> dict[str, str]:
Args:
token: Raw token string. For ``"basic"`` auth this is the
base64-encoded ``username:password`` value; for ``"token"`` auth
this is the plain API token.
this is the API key.

Returns:
Dictionary of HTTP headers including ``Authorization``.
Expand All @@ -95,7 +95,7 @@ def _scrape_url(
url: Full URL to scrape (must include scheme).
timeout: Request timeout in seconds.
auth_mode: ``"basic"`` (username:password, ``/v2/scrape``) or
``"token"`` (API token, ``/unified/v1/scrape``).
``"token"`` (API key, ``/unified/v1/scrape``).

Returns:
Parsed JSON response from the Decodo API.
Expand Down Expand Up @@ -170,13 +170,13 @@ class DecodoLoader(BaseLoader):
urls: A single URL string or a list of URL strings to scrape.
api_token: Credential value. For ``"basic"`` auth (default) this is
the base64-encoded ``username:password`` string; for ``"token"``
auth this is the plain API token. Falls back to the
auth this is the API key. Falls back to the
``DECODO_API_TOKEN`` environment variable when omitted.
auth_mode: Authentication mode.

* ``"basic"`` *(default)* — username:password credentials,
uses ``/v2/scrape``.
* ``"token"`` — plain API token, uses ``/unified/v1/scrape``.
* ``"token"`` — API key, uses ``/unified/v1/scrape``.

timeout: HTTP request timeout in seconds (default 180).
continue_on_error: When ``True`` (default), skip URLs that fail and
Expand All @@ -191,11 +191,11 @@ class DecodoLoader(BaseLoader):
)
docs = loader.load()

Example — token auth::
Example — API key auth::

loader = DecodoLoader(
urls=["https://example.com", "https://example.org"],
api_token="your-api-token",
api_token="your-api-key",
auth_mode="token",
)
docs = loader.load()
Expand All @@ -217,7 +217,7 @@ def __init__(
raw_token = api_token or os.environ.get("DECODO_API_TOKEN", "")
if not raw_token:
raise ValueError(
"Decodo API token is required. Provide it via the ``api_token`` "
"Decodo API key or token is required. Provide it via the ``api_token`` "
"argument or the ``DECODO_API_TOKEN`` environment variable."
)
self._token: SecretStr = SecretStr(raw_token)
Expand Down
36 changes: 18 additions & 18 deletions libs/partners/decodo/langchain_decodo/tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

* ``"basic"`` *(default)* — username:password credentials encoded as a Basic
auth token. Uses the ``/v2/scrape`` endpoint.
* ``"token"`` — a plain API token. Uses the ``/unified/v1/scrape`` endpoint.
* ``"token"`` — an API key. Uses the ``/unified/v1/scrape`` endpoint.

In both modes the ``Authorization: Basic <value>`` header is used; only the
value and the target endpoint differ.
Expand Down Expand Up @@ -63,7 +63,7 @@ def _build_headers(token: str) -> dict[str, str]:
Args:
token: Raw token string. For ``"basic"`` auth this is the
base64-encoded ``username:password`` value; for ``"token"`` auth
this is the plain API token.
this is the API key.

Returns:
Dictionary of HTTP headers including ``Authorization``.
Expand All @@ -90,7 +90,7 @@ def _do_scrape(
base_url: Base URL of the Decodo API.
payload: Request body to send as JSON.
auth_mode: ``"basic"`` (username:password, ``/v2/scrape``) or
``"token"`` (API token, ``/unified/v1/scrape``).
``"token"`` (API key, ``/unified/v1/scrape``).
timeout: Request timeout in seconds.

Returns:
Expand Down Expand Up @@ -191,7 +191,7 @@ class DecodoWebScrapeTool(BaseTool):
* ``"basic"`` *(default)* — supply the base64-encoded
``username:password`` string as ``decodo_api_token`` (or set
``DECODO_API_TOKEN``). Requests go to ``/v2/scrape``.
* ``"token"`` — supply a plain API token as ``decodo_api_token`` (or set
* ``"token"`` — supply an API key as ``decodo_api_token`` (or set
``DECODO_API_TOKEN``). Requests go to ``/unified/v1/scrape``.

Both modes send ``Authorization: Basic <value>``; the value and endpoint
Expand All @@ -200,7 +200,7 @@ class DecodoWebScrapeTool(BaseTool):
Attributes:
decodo_api_token: Credential value. For ``"basic"`` mode this is the
base64-encoded ``username:password``; for ``"token"`` mode this is
the plain API token. Falls back to the ``DECODO_API_TOKEN``
the API key. Falls back to the ``DECODO_API_TOKEN``
environment variable when not provided explicitly.
auth_mode: Authentication mode — ``"basic"`` (default) or ``"token"``.
base_url: Base URL of the Decodo Scraper API.
Expand All @@ -212,10 +212,10 @@ class DecodoWebScrapeTool(BaseTool):
tool = DecodoWebScrapeTool(decodo_api_token="base64(user:pass)")
result = tool.run("https://example.com")

Example — token auth::
Example — API key auth::

tool = DecodoWebScrapeTool(
decodo_api_token="your-api-token",
decodo_api_token="your-api-key",
auth_mode="token",
)
result = tool.run("https://example.com")
Expand All @@ -235,7 +235,7 @@ class DecodoWebScrapeTool(BaseTool):
default=SecretStr(""),
description=(
"Credential value. For 'basic' auth: base64-encoded username:password. "
"For 'token' auth: plain API token. "
"For 'token' auth: API key. "
"Falls back to the DECODO_API_TOKEN environment variable."
),
)
Expand All @@ -244,7 +244,7 @@ class DecodoWebScrapeTool(BaseTool):
description=(
"Authentication mode. "
"'basic' (default): username:password credentials, uses /v2/scrape. "
"'token': plain API token, uses /unified/v1/scrape."
"'token': API key, uses /unified/v1/scrape."
),
)
base_url: str = Field(
Expand Down Expand Up @@ -283,13 +283,13 @@ def _run(
Page content as a string (markdown or plain text).

Raises:
ValueError: If no API token is configured.
ValueError: If no API key or token is configured.
RuntimeError: On API errors or network failures.
"""
token = self.decodo_api_token.get_secret_value()
if not token:
raise ValueError(
"Decodo API token is required. Set it via the ``decodo_api_token`` "
"Decodo API key or token is required. Set it via the ``decodo_api_token`` "
"field or the ``DECODO_API_TOKEN`` environment variable."
)

Expand Down Expand Up @@ -317,7 +317,7 @@ class DecodoSearchTool(BaseTool):
* ``"basic"`` *(default)* — supply the base64-encoded
``username:password`` string as ``decodo_api_token`` (or set
``DECODO_API_TOKEN``). Requests go to ``/v2/scrape``.
* ``"token"`` — supply a plain API token as ``decodo_api_token`` (or set
* ``"token"`` — supply an API key as ``decodo_api_token`` (or set
``DECODO_API_TOKEN``). Requests go to ``/unified/v1/scrape``.

Attributes:
Expand All @@ -332,10 +332,10 @@ class DecodoSearchTool(BaseTool):
tool = DecodoSearchTool(decodo_api_token="base64(user:pass)")
results = tool.run({"query": "best Python web scraping libraries"})

Example — token auth::
Example — API key auth::

tool = DecodoSearchTool(
decodo_api_token="your-api-token",
decodo_api_token="your-api-key",
auth_mode="token",
)
results = tool.run({"query": "best Python web scraping libraries"})
Expand All @@ -358,7 +358,7 @@ class DecodoSearchTool(BaseTool):
default=SecretStr(""),
description=(
"Credential value. For 'basic' auth: base64-encoded username:password. "
"For 'token' auth: plain API token. "
"For 'token' auth: API key. "
"Falls back to the DECODO_API_TOKEN environment variable."
),
)
Expand All @@ -367,7 +367,7 @@ class DecodoSearchTool(BaseTool):
description=(
"Authentication mode. "
"'basic' (default): username:password credentials, uses /v2/scrape. "
"'token': plain API token, uses /unified/v1/scrape."
"'token': API key, uses /unified/v1/scrape."
),
)
engine: str = Field(
Expand Down Expand Up @@ -411,13 +411,13 @@ def _run(
``status_code``, and ``url`` fields.

Raises:
ValueError: If no API token is configured.
ValueError: If no API key or token is configured.
RuntimeError: On API errors or network failures.
"""
token = self.decodo_api_token.get_secret_value()
if not token:
raise ValueError(
"Decodo API token is required. Set it via the ``decodo_api_token`` "
"Decodo API key or token is required. Set it via the ``decodo_api_token`` "
"field or the ``DECODO_API_TOKEN`` environment variable."
)

Expand Down
2 changes: 1 addition & 1 deletion libs/partners/decodo/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "langchain-decodo"
version = "0.1.1"
version = "0.1.2"
description = "LangChain integration for the Decodo web scraping API"
readme = "README.md"
license = { text = "MIT" }
Expand Down
Loading
Loading