Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 7 additions & 10 deletions docs/reference/client.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,9 @@ let api = blocking::create_api_service();
```

The blocking client covers most of the async one, not all of it. It has no `subscriptions`, and
it lacks `time_series.filter`, `resources.filter`, `resources.get_by_id`,
`resources.fetch_nearest`, `events.get`, `events.update` and `events.count`. `api.async_api()`
it lacks `time_series.filter`, `time_series.listen_datapoints`, `resources.filter`,
`resources.get_by_id`, `resources.fetch_nearest`, `events.get`, `events.update` and
`events.count`. `api.async_api()`
hands you the async service for those.

</TabItem>
Expand All @@ -96,26 +97,22 @@ hands you the async service for those.
| Subscriptions | `client.subscriptions()` | `client.subscriptions` | `api.subscriptions` |
| Edges | `client.edges()` | `client.edges` | `api.edges` |
| [Labels](./labels) | `client.labels()` | `client.labels` | `api.labels` |
| [Tenant](./tenant) | `client.tenant()` | `client.tenant` | `api.tenant` |

Assets and functions are the `ASSET`- and `FUNCTION`-labelled resources, typed. A service being
in a client does not mean every call is: each reference page has a "What each client covers"
table.

The Java client adds three more services:
The Java client adds two more services:

| Service | Java | Covers |
| --- | --- | --- |
| [Policies](./policies) | `client.policies()` | `/policies`, including the naming dry-run |
| [Governance templates](./policies#governance-templates) | `client.governance()` | `/governance/templates` |
| [Tenant](./tenant) | `client.tenant()` | `/tenant/features` and `/tenant/settings` |

:::note Two endpoints no client wraps, on purpose
:::note One endpoint no client wraps, on purpose
`GET /stats` is internal to the console and marked hidden: it is not part of the published
contract, so nothing should call it. The browser datapoint tail
`/timeseries/datapoints/listen` authenticates with a `?token=` query parameter, which is for a
page that cannot set a header. Server-side code wants the durable equivalent instead,
[`subscriptions().listen`](./subscriptions#live-delivery), which authenticates the upgrade with
the usual `Authorization` header and survives a reconnect.
contract, so nothing should call it.
:::

## Authentication
Expand Down
7 changes: 7 additions & 0 deletions docs/reference/datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,16 @@ api.datasets.create(&vec![dataset]).await?;

## Look up & delete

`by_ids` takes ids and external ids together and leaves out what it cannot find rather than
failing. `GET /datasets/{id}` reads one data set by numeric id and answers `404` for a miss.

<Tabs groupId="lang">
<TabItem value="java" label="Java">

```java
DataWrapper<DataSetModel> some = client.datasets()
.byIds(List.of(IdCollection.createFromExternalId("plant_a")));
DataWrapper<DataSetModel> one = client.datasets().getById(5677892L);

client.datasets().delete(List.of(IdCollection.createFromExternalId("plant_a")));
```
Expand All @@ -86,6 +90,7 @@ client.datasets().delete(List.of(IdCollection.createFromExternalId("plant_a")));

```python
some = client.datasets.by_ids(["plant_a"])
one = client.datasets.get_by_id(5677892)
client.datasets.delete(["plant_a"])
```

Expand All @@ -96,6 +101,7 @@ client.datasets.delete(["plant_a"])
use intellistream_datahub_sdk::generic::IdAndExtId;

let some = api.datasets.by_ids(&vec![IdAndExtId::from_external_id("plant_a")]).await?;
let one = api.datasets.get_by_id(5677892).await?;
api.datasets.delete(&vec![IdAndExtId::from_external_id("plant_a")]).await?;
```

Expand Down Expand Up @@ -241,6 +247,7 @@ A changed grant therefore takes effect within about a minute, without a new toke
| Operation | Java | Python | Rust |
| --- | --- | --- | --- |
| Create | `datasets().create` | `datasets.create` | `datasets.create` |
| Get by numeric id | `datasets().getById` | `datasets.get_by_id` | `datasets.get_by_id` |
| Look up by id / external id | `datasets().byIds` | `datasets.by_ids` | `datasets.by_ids` |
| List (`GET /datasets`) | `datasets().list(limit)` | `datasets.list(limit=None)` | `datasets.list(limit)` |
| Filter | `datasets().filter` | `datasets.filter` | `datasets.filter` |
Expand Down
82 changes: 66 additions & 16 deletions docs/reference/resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -896,9 +896,13 @@ The nodes and relationships it creates count against their own daily quotas thro
ordinary create path. A reverse proxy in front of the API has to pass the upload through
unbuffered and uncapped, as the shipped nginx examples do for `/resources/import`.

The Java client wraps the pair. `export` hands back the file as bytes and `importGraph` takes
the same bytes, so moving a sub-graph between two tenants is two calls and no temporary file
unless you want one:
Every client wraps the pair. Export hands back the file as bytes and import takes the same
bytes, so moving a sub-graph between two tenants is two calls and no temporary file unless you
want one. Python and Rust also stream either end to or from a path, for a file too large to hold
in memory.

<Tabs groupId="lang">
<TabItem value="java" label="Java">

```java
import ai.intellistream.datahub.api.graphtransfer.GraphImportResult;
Expand All @@ -914,7 +918,45 @@ System.out.println(result.nodesCreated() + " created, "
`GraphImportResult` is a record, so its fields read as `result.nodesCreated()` and so on, one
per key of the JSON above.

Python and Rust call the endpoints over HTTP with the bearer token the client already holds:
</TabItem>
<TabItem value="python" label="Python">

```python
graph = source.resources.export_graph(5677892) # bytes

result = target.resources.import_graph(graph)
print(result.nodes_created, "created,",
result.nodes_skipped_existing, "already there,",
result.nodes_skipped_timeseries, "to create as series first")

# Or through a file, never held whole in memory:
source.resources.export_graph_to_path(5677892, "plant_oslo.dhgraph")
target.resources.import_graph_from_path("plant_oslo.dhgraph")
```

</TabItem>
<TabItem value="rust" label="Rust">

```rust
let graph: Vec<u8> = source.resources.export_graph(5677892).await?;

let result = target.resources.import_graph(graph).await?;
println!("{} created, {} already there, {:?} to create as series first",
result.nodes_created, result.nodes_skipped_existing, result.nodes_skipped_timeseries);

// Or through a file, never held whole in memory:
source.resources.export_graph_to_path(5677892, "plant_oslo.dhgraph").await?;
target.resources.import_graph_from_path("plant_oslo.dhgraph").await?;
```

</TabItem>
</Tabs>

The export walks the graph, which settles a moment after a write. A component created and
exported straight away can come back short; read it with `fetch_related` first when that
matters.

Over plain HTTP, send the bearer token the client already holds:

```bash
curl -fsS -H "Authorization: Bearer $TOKEN" \
Expand Down Expand Up @@ -1056,33 +1098,41 @@ DataWrapper<Function> page = client.functions().filter(criteria);
```python
# entity objects, external-id strings, or numeric ids
some = client.functions.by_ids(["fn_rolling_average"])

page = client.functions.filter(labels=["AGGREGATION"]) # a Page; page.next_cursor pages on
hits = client.functions.search("rolling average")
```

`filter` takes the criteria as keywords or a prepared `FunctionFilter` as `filter=`, never both,
plus `limit`, `sort_by`, `sort_order` and `cursor`. `search` takes `query`, an optional
`FunctionFilter` and `limit`.

</TabItem>
<TabItem value="rust" label="Rust">

```rust
use intellistream_datahub_sdk::filters::NodeFilter;
use intellistream_datahub_sdk::functions::{FunctionFilter, FunctionFilterForm};
use intellistream_datahub_sdk::generic::IdAndExtId;

let some = api.functions.by_ids(&vec![
IdAndExtId::from_external_id("fn_rolling_average"),
]).await?;

let criteria = FunctionFilter {
node: NodeFilter { labels: Some(vec!["AGGREGATION".into()]), ..Default::default() },
data_set_id: None,
};
let page = api.functions.filter(&FunctionFilterForm::new(criteria)).await?;
let hits = api.functions.search_by_query("rolling average").await?;
```

`FunctionFilterForm` carries `limit` and a `PageRequest` for `sort` and `cursor`; `search` takes a
`SearchAndFilterForm<FunctionFilter>`, and `search_by_query` is the shorthand for a bare phrase.

</TabItem>
</Tabs>

:::note Python and Rust are catching up
`filter` and `search` on functions are in the Java client today. Python and Rust will get them
under the names their `assets` service already uses: `functions.filter(...)` taking keyword
criteria in Python and a filter form in Rust, and `functions.search(query, ...)`. Until then, ask
[`resources.filter`](#filter) narrowed to the `function` node type.

In those two clients `functions.by_ids` still reads the listing and matches on the client, so it
only sees the newest 10 000 functions. It moves to `POST /functions/byids` in the same change;
the call itself stays as it is.
:::

## What each client covers {#client-coverage}

| Operation | Java | Python | Rust |
Expand All @@ -1097,7 +1147,7 @@ the call itself stays as it is.
| Filter | `resources().filter` | `resources.filter` | `resources.filter` |
| Traverse (`fetch-related`) | `resources().fetchRelated` | `resources.fetch_related` | `resources.fetch_related` |
| Nearest N (`fetch-nearest`) | `resources().fetchNearest` | `resources.fetch_nearest` | `resources.fetch_nearest` |
| [Export / import a graph](#graph-transfer) | `resources().export` / `importGraph` | HTTP only | HTTP only |
| [Export / import a graph](#graph-transfer) | `resources().export` / `importGraph` | `resources.export_graph` / `import_graph`, plus `_to_path` / `_from_path` | `resources.export_graph` / `import_graph`, plus `_to_path` / `_from_path` |


Relations have their own client surface in all three clients, `edges()` in Java, `edges` in
Expand Down
55 changes: 55 additions & 0 deletions docs/reference/subscriptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,60 @@ Ack a message only after you've durably handled it. If your process dies before
the server redelivers it, so make your handler idempotent.
:::

## A live tail without a subscription {#live-tail}

`/timeseries/datapoints/listen` streams the datapoints written to a set of series with no
subscription behind it: nothing to create first, nothing to ack, nothing left behind when the
socket closes. It is the lighter tool and the weaker one. It starts at the latest point, keeps no
cursor, and does not replay what was written while the socket was down, so a reader that must
see every point uses [live delivery](#live-delivery) instead.

The token travels as a WebSocket subprotocol, not a header, because a browser cannot set one on
the handshake: offer `datahub.bearer.<jwt>` and `datahub.v1`, and the server echoes `datahub.v1`.
An `externalIds=a,b,c` query parameter seeds the set, and
`{"action": "set" | "subscribe" | "unsubscribe", "externalIds": [...]}` changes it. Every change
is narrowed to the series whose data set you may read, silently: an id you cannot read, or that
does not exist, is dropped rather than refused. A frame is
`{"datapoints": [{"externalId", "valueType", "timestamp", "value"}, ...]}`, `value` a string.

A bad token or a missing role is refused after the upgrade, with a **1008** close; a connection
over the [WebSocket cap](./limits#websockets) gets the error frame described there first.

<Tabs groupId="lang">
<TabItem value="java" label="Java">

The Java client does not wrap the tail. Use [live delivery](#live-delivery).

</TabItem>
<TabItem value="python" label="Python">

```python
with client.timeseries.listen_datapoints(["engine_temperature"]) as tail:
for point in tail:
print(point.external_id, point.timestamp, point.value)
```

`subscribe`, `unsubscribe` and `set_timeseries` change the set while it runs. On an
`AsyncDataHubClient` it returns a `DatapointListenerAsync`, driven with `async for`.

</TabItem>
<TabItem value="rust" label="Rust">

```rust
let mut tail = api.time_series.listen_datapoints(&["engine_temperature"]).await?;
while let Some(point) = tail.next().await {
let point = point?;
println!("{} {} {}", point.external_id, point.timestamp, point.value);
}
```

`subscribe`, `unsubscribe` and `set_timeseries` change the set while it runs. A dropped
connection is reopened with a fresh token and the current set; a refusal is returned as an
error instead of retried.

</TabItem>
</Tabs>

## What each client covers {#client-coverage}

| Operation | Java | Python | Rust |
Expand All @@ -263,5 +317,6 @@ the server redelivers it, so make your handler idempotent.
| Filter | `subscriptions().filter` | `subscriptions.filter(form=None, *, timeseries=, limit=, sort=)` | `subscriptions.filter` |
| Delete | `subscriptions().delete` | `subscriptions.delete` | `subscriptions.delete` |
| Live delivery | `subscriptions().listen` | `subscriptions.listen` (`SubscriptionListenerAsync` with `async for` on the async client) | `subscriptions.listen` |
| [Live tail](#live-tail), no subscription | — | `timeseries.listen_datapoints` | `time_series.listen_datapoints` |


Loading
Loading