Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions docs/advanced/data-cleaning-lineage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -157,10 +157,25 @@ it poison?** Walk *forward* from the raw sensor and every downstream series it t
lights up, while the vibration-only features stay clean. That's the precise list to
recompute or quarantine.

`fetch_related` walks edges in both directions, so from the raw sensor it reaches the
whole connected pipeline, the vibration branch included. Each returned edge keeps its
direction, though, `start` → `end`, so follow those from the sensor to keep only what lies
downstream of it.

```python
fwd = client.resources.fetch_related(external_id="engine_temperature_raw", depth=12)
affected = [n.external_id for n in fwd.nodes
if n.external_id.startswith("engine_") and n.external_id != "engine_temperature_raw"]
net = client.resources.fetch_related(external_id="engine_temperature_raw", depth=12)
ext_id = {n.id: n.external_id for n in net.nodes}
start = next(i for i, x in ext_id.items() if x == "engine_temperature_raw")

downstream, frontier = set(), [start]
while frontier:
node = frontier.pop()
for e in net.edges:
if e.start == node and e.end not in downstream:
downstream.add(e.end)
frontier.append(e.end)

affected = sorted(ext_id[i] for i in downstream if ext_id[i].startswith("engine_"))
print("affected by the bad temperature sensor:", affected)
# engine_temperature_clean, engine_temp_roll_mean, engine_temp_roc,
# engine_feature_vector, engine_health_score — but NOT the vibration features
Expand Down
7 changes: 4 additions & 3 deletions docs/advanced/sustained-alarm-window.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -399,9 +399,10 @@ assert_eq!(found.get_items().len(), 1, "expected exactly one finding");
</TabItem>
</Tabs>

Re-running the replay is safe to try, and instructive: the detector raises the same finding
again with the same external id, and the platform collapses it rather than storing a
duplicate. Events are keyed by id, datapoints by series and timestamp.
Re-running the replay is instructive: the detector raises the same finding again with the
same external id, and the platform stores it as a second event rather than collapsing it, so
the check above then finds two. An event's external id is not unique; each create without an
`id` gets a new one. Only the datapoints collapse, keyed by series and timestamp.

## 5. Run the same detector live

Expand Down
12 changes: 7 additions & 5 deletions docs/guides/correlate-alarms.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,10 @@ let nb = api.resources.fetch_related(

## 3. Find the shared subsystem

Intersect the two node sets. Whatever remains (besides the sensors themselves) is a
subsystem both alarms belong to, the likely common cause.
Intersect the two node sets. The walk follows edges in both directions, so once it reaches
the shared skid it also comes back down the other branch: besides the sensors themselves, the
intersection holds both pumps as well as `skid_1` and `cooling_system`, the subsystems both
alarms belong to and the likely common cause.

<Tabs groupId="lang">
<TabItem value="java" label="Java">
Expand All @@ -105,7 +107,7 @@ Set<String> shared = nb.nodes().stream()
shared.remove("sensor_a");
shared.remove("sensor_b");

System.out.println("Both alarms are part of: " + shared); // [cooling_system, skid_1]
System.out.println("Both alarms are part of: " + shared); // [cooling_system, skid_1, pump_a, pump_b]
if (!shared.contains("cooling_system")) {
throw new AssertionError("expected cooling_system, found " + shared);
}
Expand All @@ -119,7 +121,7 @@ a_nodes = {n.external_id for n in na.nodes}
shared = {n.external_id for n in nb.nodes} & a_nodes
shared -= {"sensor_a", "sensor_b"}

print("Both alarms are part of:", shared) # {'cooling_system', 'skid_1'}
print("Both alarms are part of:", shared) # {'cooling_system', 'skid_1', 'pump_a', 'pump_b'}
assert "cooling_system" in shared, f"expected cooling_system, found {shared}"
```

Expand All @@ -137,7 +139,7 @@ let mut shared: HashSet<&str> = nb.nodes().iter()
shared.remove("sensor_a");
shared.remove("sensor_b");

println!("Both alarms are part of: {:?}", shared); // {"cooling_system", "skid_1"}
println!("Both alarms are part of: {:?}", shared); // {"cooling_system", "skid_1", "pump_a", "pump_b"}
assert!(shared.contains("cooling_system"), "expected cooling_system, found {:?}", shared);
```

Expand Down
3 changes: 2 additions & 1 deletion docs/guides/ingest-timeseries.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ Datapoints are split into batches (the store is optimised for large batches) and
concurrently up to a bounded in-flight limit. The caps below are the API's and bind every client.
Java sends **10,000** datapoints per request by default, retries transient failures (HTTP
429/5xx, network) in process and returns an `IngestResult` summarising what landed and what
didn't. Python and Rust send up to **100,000** per request, four at once, and retry through the
didn't. Python and Rust send up to **100,000** per request, four at once (one at a time with
buffering on), and retry through the
[durable spool](/reference/client#durable-ingest-buffering): with buffering on, a transient
failure (429, 5xx, network, or a `401`/`403`) writes the input to disk, and the next ingest call
sends the spool again, oldest first, before its own data. With buffering off, the first request
Expand Down
6 changes: 3 additions & 3 deletions docs/industries/agriculture-food/precision-farming.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ import intellistream_datahub_sdk, numpy as np, pandas as pd
client = intellistream_datahub_sdk.DataHubClient.from_env()

for s, u in [("field_north_40_soil_moisture", "pct"), ("field_north_40_soil_temp_c", "deg_c")]:
client.timeseries.create([intellistream_datahub_sdk.TimeSeries(external_id=s, name=s, unit=u)])
client.timeseries.create([intellistream_datahub_sdk.TimeSeries(value_type="float", external_id=s, name=s, unit=u)])
idx = pd.date_range(end=pd.Timestamp.now(tz="UTC"), periods=48, freq="1h")
moisture = 35 - np.linspace(0, 15, 48) + np.random.normal(0, 1, 48) # drying out
client.timeseries.insert_from_lists(timestamps=idx, values=moisture, ts="field_north_40_soil_moisture")
Expand Down Expand Up @@ -63,9 +63,9 @@ client.timeseries().ingest(Map.of(
import intellistream_datahub_sdk

client.timeseries.create([
intellistream_datahub_sdk.TimeSeries(external_id="field_north_40_soil_moisture",
intellistream_datahub_sdk.TimeSeries(value_type="float", external_id="field_north_40_soil_moisture",
name="North 40 soil moisture", unit="pct"),
intellistream_datahub_sdk.TimeSeries(external_id="field_north_40_soil_temp_c",
intellistream_datahub_sdk.TimeSeries(value_type="float", external_id="field_north_40_soil_temp_c",
name="North 40 soil temperature", unit="deg_c")])
client.timeseries.insert_from_lists(
timestamps=timestamps, values=moisture, ts="field_north_40_soil_moisture")
Expand Down
4 changes: 2 additions & 2 deletions docs/industries/agriculture-food/salmon-farming.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -139,8 +139,8 @@ client.timeseries().ingest(Map.of(
import intellistream_datahub_sdk

client.timeseries.create([
intellistream_datahub_sdk.TimeSeries(external_id="pen_h_07_feed_kg", name="Pen H-07 feed", unit="kg"),
intellistream_datahub_sdk.TimeSeries(external_id="pen_h_07_biomass_kg", name="Pen H-07 biomass", unit="kg")])
intellistream_datahub_sdk.TimeSeries(value_type="float", external_id="pen_h_07_feed_kg", name="Pen H-07 feed", unit="kg"),
intellistream_datahub_sdk.TimeSeries(value_type="float", external_id="pen_h_07_biomass_kg", name="Pen H-07 biomass", unit="kg")])
client.timeseries.insert_from_lists(
timestamps=[pd.Timestamp.now(tz="UTC")], values=[420.0], ts="pen_h_07_feed_kg")
```
Expand Down
9 changes: 2 additions & 7 deletions docs/industries/built-environment/construction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,14 @@ This scenario pairs **document management** with safety events on a modelled sit

## Set up demo data

New workspace? Run this once (Python) to create the site graph and a crane-utilisation
series. (The drawing upload in step 1 needs a local file — any PDF works.) Safe to re-run.
New workspace? Run this once (Python) to create a crane-utilisation series; step 1
builds the site graph itself. (The drawing upload in step 1 needs a local file — any PDF works.)

```python
import intellistream_datahub_sdk, numpy as np, pandas as pd

client = intellistream_datahub_sdk.DataHubClient.from_env()

client.resources.create(
[intellistream_datahub_sdk.Resource(external_id=x, name=x, labels=[lbl]) for x, lbl in
[("site_harbour_tower", "Site"), ("zone_level_12", "Zone")]],
[intellistream_datahub_sdk.RelForm.by_external_ids("site_harbour_tower", "zone_level_12", "contains")])

client.timeseries.create([intellistream_datahub_sdk.TimeSeries(external_id="crane_02_hours", name="Crane 02 hours", unit="h", value_type="float")])
idx = pd.date_range(end=pd.Timestamp.now(tz="UTC"), periods=14, freq="1d")
client.timeseries.insert_from_lists(timestamps=idx, values=np.random.uniform(4, 9, 14), ts="crane_02_hours")
Expand Down
8 changes: 4 additions & 4 deletions docs/industries/energy-utilities/air-quality.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ client = intellistream_datahub_sdk.DataHubClient.from_env()

for s, u in [("station_kirkeveien_pm25", "ug_m3"), ("station_kirkeveien_no2", "ppb"),
("station_kirkeveien_co", "ppm")]:
client.timeseries.create([intellistream_datahub_sdk.TimeSeries(external_id=s, name=s, unit=u)])
client.timeseries.create([intellistream_datahub_sdk.TimeSeries(value_type="float", external_id=s, name=s, unit=u)])
idx = pd.date_range(end=pd.Timestamp.now(tz="UTC"), periods=24, freq="1h")
client.timeseries.insert_from_lists(timestamps=idx, values=np.random.uniform(5, 45, 24), ts="station_kirkeveien_pm25")
```
Expand Down Expand Up @@ -102,9 +102,9 @@ client.timeseries().create(pm25, no2, co);
import intellistream_datahub_sdk

client.timeseries.create([
intellistream_datahub_sdk.TimeSeries(external_id="station_kirkeveien_pm25", name="Kirkeveien PM2.5", unit="ug_m3"),
intellistream_datahub_sdk.TimeSeries(external_id="station_kirkeveien_no2", name="Kirkeveien NO2", unit="ppb"),
intellistream_datahub_sdk.TimeSeries(external_id="station_kirkeveien_co", name="Kirkeveien CO", unit="ppm")])
intellistream_datahub_sdk.TimeSeries(value_type="float", external_id="station_kirkeveien_pm25", name="Kirkeveien PM2.5", unit="ug_m3"),
intellistream_datahub_sdk.TimeSeries(value_type="float", external_id="station_kirkeveien_no2", name="Kirkeveien NO2", unit="ppb"),
intellistream_datahub_sdk.TimeSeries(value_type="float", external_id="station_kirkeveien_co", name="Kirkeveien CO", unit="ppm")])
```

</TabItem>
Expand Down
2 changes: 1 addition & 1 deletion docs/industries/energy-utilities/grid.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ import intellistream_datahub_sdk, numpy as np, pandas as pd

client = intellistream_datahub_sdk.DataHubClient.from_env()

client.timeseries.create([intellistream_datahub_sdk.TimeSeries(external_id="feeder_f12_load_mw", name="Feeder F12 load", unit="mw")])
client.timeseries.create([intellistream_datahub_sdk.TimeSeries(value_type="float", external_id="feeder_f12_load_mw", name="Feeder F12 load", unit="mw")])
idx = pd.date_range(end=pd.Timestamp.now(tz="UTC"), periods=24, freq="1h")
load = 8 + 4 * np.sin(np.arange(24) / 24 * 2 * np.pi) + np.random.normal(0, 0.5, 24)
load[18] = 14.8 # an evening overload past the 12 MW rating
Expand Down
6 changes: 3 additions & 3 deletions docs/industries/energy-utilities/wind.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ import intellistream_datahub_sdk, numpy as np, pandas as pd
client = intellistream_datahub_sdk.DataHubClient.from_env()

for s, u in [("turbine_t14_power_kw", "kw"), ("turbine_t14_wind_ms", "m_s")]:
client.timeseries.create([intellistream_datahub_sdk.TimeSeries(external_id=s, name=s, unit=u)])
client.timeseries.create([intellistream_datahub_sdk.TimeSeries(value_type="float", external_id=s, name=s, unit=u)])
idx = pd.date_range(end=pd.Timestamp.now(tz="UTC"), periods=168, freq="1h")
wind = np.clip(np.random.weibull(2.0, 168) * 6.0, 0, 25)
power = np.clip(wind**3 * 2.0, 0, 3500) # rated ~3500 kW
Expand Down Expand Up @@ -65,9 +65,9 @@ client.timeseries().ingest(Map.of(
import intellistream_datahub_sdk

client.timeseries.create([
intellistream_datahub_sdk.TimeSeries(external_id="turbine_t14_power_kw",
intellistream_datahub_sdk.TimeSeries(value_type="float", external_id="turbine_t14_power_kw",
name="Turbine T14 power", unit="kw"),
intellistream_datahub_sdk.TimeSeries(external_id="turbine_t14_wind_ms",
intellistream_datahub_sdk.TimeSeries(value_type="float", external_id="turbine_t14_wind_ms",
name="Turbine T14 wind speed", unit="m_s")])
client.timeseries.insert_from_lists(
timestamps=timestamps, values=power_kw, ts="turbine_t14_power_kw")
Expand Down
4 changes: 2 additions & 2 deletions docs/industries/financial-services/portfolio-risk.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ client = intellistream_datahub_sdk.DataHubClient.from_env()

desk_id = client.datasets.create([intellistream_datahub_sdk.Dataset(external_id="desk_equities", name="Equities desk")])[0].id
client.timeseries.create([intellistream_datahub_sdk.TimeSeries(external_id="portfolio_growth_mtm_usd",
name="Growth portfolio — MtM (USD)", unit="usd", value_type="numeric", data_set_id=desk_id)])
name="Growth portfolio — MtM (USD)", unit="usd", value_type="float", data_set_id=desk_id)])
idx = pd.date_range(end=pd.Timestamp.now(tz="UTC"), periods=30, freq="1d")
client.timeseries.insert_from_lists(timestamps=idx,
values=1e7 + np.cumsum(np.random.normal(0, 5e4, 30)), ts="portfolio_growth_mtm_usd")
Expand Down Expand Up @@ -105,7 +105,7 @@ client.timeseries().ingest(Map.of(
client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
external_id="portfolio_growth_mtm_usd",
name="Growth portfolio — mark-to-market (USD)",
unit="usd", value_type="numeric", data_set_id=desk_id)])
unit="usd", value_type="float", data_set_id=desk_id)])

client.timeseries.insert_from_lists(
timestamps=[pd.Timestamp.now(tz="UTC")], values=[10428155.42],
Expand Down
6 changes: 3 additions & 3 deletions docs/industries/healthcare/medical-devices.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ What we solve here is device availability and fast, complete recall tracing.

## Set up demo data

New workspace? Run this once (Python) to create three pumps of the same model in a ward —
so the recall walk in step 2 returns the full pull list. Safe to re-run.
New workspace? Run this once (Python) to create the model, the ward and two more pumps of
the same model in it. Step 1 registers a third, so the recall walk in step 2 returns all three.

```python
import intellistream_datahub_sdk
Expand All @@ -28,7 +28,7 @@ client = intellistream_datahub_sdk.DataHubClient.from_env()
nodes = [intellistream_datahub_sdk.Resource(external_id="model_acme_x200", name="Acme X200 infusion pump", labels=["PumpModel"]),
intellistream_datahub_sdk.Resource(external_id="ward_icu", name="Intensive care", labels=["Ward"])]
edges = []
for u in ["3391", "3392", "3401"]:
for u in ["3392", "3401"]:
nodes.append(intellistream_datahub_sdk.Resource(external_id=f"pump_icu_{u}", name=f"Infusion pump {u}", labels=["Pump"]))
edges += [intellistream_datahub_sdk.RelForm.by_external_ids(f"pump_icu_{u}", "ward_icu", "located_in"),
intellistream_datahub_sdk.RelForm.by_external_ids(f"pump_icu_{u}", "model_acme_x200", "is_model")]
Expand Down
2 changes: 1 addition & 1 deletion docs/industries/manufacturing-process/discrete.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ client.timeseries().ingest(Map.of(
import intellistream_datahub_sdk

client.timeseries.create([
intellistream_datahub_sdk.TimeSeries(external_id="station_07_cycle_time_s",
intellistream_datahub_sdk.TimeSeries(value_type="float", external_id="station_07_cycle_time_s",
name="Station 07 cycle time", unit="s"),
intellistream_datahub_sdk.TimeSeries(external_id="station_07_scrap", name="Station 07 scrap count", unit="count")])

Expand Down
2 changes: 1 addition & 1 deletion docs/industries/manufacturing-process/pharma.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ client.timeseries().ingest(Map.of(
import intellistream_datahub_sdk, pandas as pd

client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
external_id="batch_22f_ph", name="Batch 22F — pH", unit="ph", value_type="numeric")])
external_id="batch_22f_ph", name="Batch 22F — pH", unit="ph", value_type="float")])

client.timeseries.insert_from_lists(
timestamps=[pd.Timestamp.now(tz="UTC")], values=[7.0421], ts="batch_22f_ph")
Expand Down
2 changes: 1 addition & 1 deletion docs/industries/manufacturing-process/semiconductor.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ import intellistream_datahub_sdk, pandas as pd

client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
external_id="tool_etch_07_chamber_pressure_mtorr",
name="Etch-07 chamber pressure", unit="mtorr", value_type="numeric")])
name="Etch-07 chamber pressure", unit="mtorr", value_type="float")])
client.timeseries.insert_from_lists(
timestamps=[pd.Timestamp.now(tz="UTC")], values=[12.0436],
ts="tool_etch_07_chamber_pressure_mtorr")
Expand Down
7 changes: 4 additions & 3 deletions docs/industries/transport-logistics/aerospace.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,9 @@ and when one component looks suspect, walk the graph to its whole blast radius.

## Set up demo data

New workspace? Run this once (Python) to create three aircraft whose pumps are all the same
part type — so the blast-radius walk in step 2 returns the whole affected fleet. Safe to re-run.
New workspace? Run this once (Python) to create the part type and two aircraft whose pumps
are of that type. Step 1 adds a third, so the blast-radius walk in step 2 returns the whole
affected fleet.

```python
import intellistream_datahub_sdk
Expand All @@ -29,7 +30,7 @@ client = intellistream_datahub_sdk.DataHubClient.from_env()

nodes = [intellistream_datahub_sdk.Resource(external_id="part_type_hp_47", name="Hydraulic pump type HP-47", labels=["part_type_hp_47"])]
edges = []
for ac in ["ln_312", "ln_318", "ln_401"]:
for ac in ["ln_318", "ln_401"]:
nodes += [intellistream_datahub_sdk.Resource(external_id=f"aircraft_{ac}", name=f"Aircraft {ac}", labels=[f"aircraft_{ac}"]),
intellistream_datahub_sdk.Resource(external_id=f"hyd_pump_{ac}_1", name=f"Pump {ac}", labels=[f"hyd_pump_{ac}_1"])]
edges += [intellistream_datahub_sdk.RelForm.by_external_ids(f"aircraft_{ac}", f"hyd_pump_{ac}_1", "contains"),
Expand Down
10 changes: 6 additions & 4 deletions docs/mcp-server.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ neighbour ("the time series measuring this pump").
| `timeseries_get` | Look up by externalId or numeric id |
| `timeseries_create` | Create one time series, returns the assigned id. Needs a `unit` or a `unitExternalId` (see `unit_list`) |
| `timeseries_update` | Update common fields by id |
| `timeseries_delete` | Delete the definition. Existing datapoints are not removed |
| `timeseries_delete` | Delete the definition. Its datapoints are purged afterwards, asynchronously |
| `timeseries_get_latest` | Most recent datapoint, for "what is it reading now?" |
| `timeseries_fetch_datapoints` | History between two timestamps, ISO-8601 or epoch millis |
| `timeseries_send_datapoint` | Append a single datapoint |
Expand Down Expand Up @@ -314,9 +314,11 @@ and daily ingest quota as your own jobs, and hit the same field, batch and body-
agent that lists in a loop can rate-limit the pipeline it shares a tenant with. The refusal
comes back as a `429` carrying `Retry-After`, which is worth teaching the agent to honour.

**Deletes do not cascade.** Deleting a data set leaves its resources and time series in place;
deleting a time series leaves its datapoints. This is deliberate, but it means an agent
tidying up will need more than one call, and a half-finished tidy leaves orphans.
**Deletes do not cascade.** Deleting a data set that still has resources or time series in it
fails: remove or reassign the members first. (Deleting a time series is the exception: its
datapoints are purged after it, asynchronously.) This is deliberate, but it means an agent
tidying up will need more than one call, in the right order, and a half-finished tidy leaves
orphans.

**Name coercion is server-side, and not uniform.** `edge_create_type` and `label_create`
normalise a name to `SCREAMING_SNAKE_CASE`, so `derived from` becomes `DERIVED_FROM`, and the
Expand Down
Loading
Loading