Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ jobs:
# without failing a release. `blocking` did exactly that.
- run: cargo build --all-features
- run: cargo test --no-run --all-features
# Only --doc: every other test needs a live backend.
- run: cargo test --doc --all-features
- run: cargo publish --dry-run

wheels:
Expand Down
12 changes: 6 additions & 6 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ rather than run unoptimised, and the same applies to the PyO3 module, which need
Most tests are integration tests that call a live backend via `create_api_service()`. They read configuration from a local `.env` file (gitignored). Required:

- `BASE_URL` — backend root, e.g. `http://localhost:8081`
- Either `TOKEN` (bearer token used as-is, no expiry) **or** the OAuth2 client-credentials set: `CLIENT_ID`, `CLIENT_SECRET`, `TOKEN_URI` (optional: `PROJECT_NAME`)
- Either `TOKEN` (bearer token used as-is, no expiry) **or** the OAuth2 client-credentials set: `CLIENT_ID`, `CLIENT_SECRET`, `TOKEN_URI`

Tests that mutate backend state (create/delete) often `sleep` a few seconds between operations and are sensitive to race conditions — prefer running them serially or isolating by unique external IDs.

Expand Down Expand Up @@ -185,7 +185,7 @@ The API wraps collections in `{ "items": [...] }`. `DataWrapper<T>` mirrors that

### Entity → request-body conversion

`DataHubEntity` is a marker trait (`ext_id()` + `Clone + Serialize`) that unlocks generic `From` impls so a `T`, `&T`, `Vec<T>`, or `&Vec<T>` can be passed to service methods and auto-wrapped into `DataWrapper<T>`. Service methods accept `&I where for<'a> &'a I: Into<DataWrapper<Event>>` (see `EventsService::create`) — implement `DataHubEntity` on new entity types to get this ergonomics for free. `IdAndExtId` has its own parallel set of `From` impls for delete/byids endpoints.
`DataHubEntity` is a marker trait (`Clone + Serialize`, no methods) that unlocks generic `From` impls so a `T`, `&T`, `Vec<T>`, or `&Vec<T>` can be passed to service methods and auto-wrapped into `DataWrapper<T>`. Service methods accept `&I where for<'a> &'a I: Into<DataWrapper<Event>>` (see `EventsService::create`) — implement `DataHubEntity` on new entity types to get this ergonomics for free. `IdAndExtId` has its own parallel set of `From` impls for delete/byids endpoints.

### Auth (`src/datahub.rs`)

Expand Down Expand Up @@ -245,10 +245,10 @@ refusal, so a by-id 404 carries `type: …/errors/not-found` and **`slug()` matc
stays as the regression guard.

401s carry a problem document too, with `type: …/errors/unauthorized` and a `detail` naming the
check that failed — a missing, empty, malformed or ambiguous `organization` claim. That is the
reason `src/auth_diagnostics.rs` reconstructs from the token it just sent, so the SDK now appends
a near-duplicate of what the server already said. Note every entry-point 401 shares the
`unauthorized` slug, so the cause is in the prose and cannot be branched on.
check that failed — a missing, empty, malformed or ambiguous `organization` claim. That retired
the SDK's own `auth_diagnostics` module, which existed only to reconstruct the reason from the
token it had just sent. Note every entry-point 401 shares the `unauthorized` slug, so the cause is
in the prose and cannot be branched on.

### Filters (`src/filters.rs`)

Expand Down
8 changes: 5 additions & 3 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ exclude = [
# Synchronous client (intellistream_datahub_sdk::blocking), same split as reqwest::blocking.
blocking = []

# docs.rs builds with default features only, which would leave `blocking` undocumented even
# though the README points callers at it.
[package.metadata.docs.rs]
all-features = true

[dependencies]
maplit = "1"
tokio = { version = "1", features = ["full"] }
Expand All @@ -39,9 +44,6 @@ infer = { version = "0.19", default-features = false, features = ["std"] }
thiserror = "2.0.17"
zstd = "0.13"
geojson = "1"
# Reading (never verifying) the payload of a JWT the SDK already holds, to explain an
# otherwise-unexplained 401 — see `auth_diagnostics`.
base64 = "0.22"
arrow-array = "59.3"
arrow-schema = "59.3"
arrow-ipc = "59.3"
Expand Down
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,6 @@ environment:
- `BASE_URL` — backend root, e.g. `http://localhost:8081` (required)
- Either `TOKEN` (bearer token used as-is, never considered expired), **or** the OAuth2
client-credentials set: `CLIENT_ID`, `CLIENT_SECRET`, `TOKEN_URI`
- `PROJECT_NAME` — optional
- `SCOPE` — against DataHub, needed when the realm uses Keycloak Organizations: that claim comes
from a dynamic client scope, so the request must name it (`organization:*`, or
`organization:<alias>` to pin one tenant). Without it the token carries no tenant and every call
Expand Down
3 changes: 1 addition & 2 deletions datahub_python_bindings/src/datasets/async_service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,9 @@ use crate::datasets::{
DatasetIdentifiable, PyDataset, PyDatasetUpdate,
};
use crate::resources::PyResource;
use crate::{DatahubIdentity, Identifiable, PyIdCollection};
use intellistream_datahub_sdk::ApiService;
use intellistream_datahub_sdk::datasets::{Dataset, DatasetUpdate};
use intellistream_datahub_sdk::generic::{DataWrapper, IdAndExtId};
use intellistream_datahub_sdk::generic::IdAndExtId;
use pyo3::{Bound, PyAny, PyResult, Python, pyclass, pymethods};
use pyo3_async_runtimes::tokio::future_into_py;
use std::sync::Arc;
Expand Down
4 changes: 1 addition & 3 deletions datahub_python_bindings/src/datasets/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@ pub(crate) mod sync_service;
use crate::PyIdCollection;
use crate::events::{PyEvent, PyTimeFilter};
use crate::resources::PyResourceNetwork;
use crate::{PyFieldBool, PyFieldStr, PyListFieldStr, PyMapField};
use crate::{PyFieldStr, PyListFieldStr, PyMapField};
use intellistream_datahub_sdk::filters::{EventFilter, EventFilterForm};
use intellistream_datahub_sdk::datahub::to_snake_lower_cased_allow_start_with_digits;
use intellistream_datahub_sdk::datasets::{
DatasetFilter, Dataset, DatasetFilterForm, DatasetUpdate, DatasetUpdateFields,
};
Expand All @@ -19,7 +18,6 @@ use pyo3::{Bound, PyResult, pyclass, pymethods};
use pyo3_async_runtimes::tokio::future_into_py;
use std::collections::HashMap;
use std::sync::Arc;
use uuid::Uuid;

#[pyclass(module = "intellistream_datahub_sdk", name = "Dataset", from_py_object)]
#[derive(Clone)]
Expand Down
5 changes: 2 additions & 3 deletions datahub_python_bindings/src/datasets/sync_service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,10 @@ use crate::datasets::{
DatasetIdentifiable, PyDataset, PyDatasetUpdate,
};
use crate::resources::PyResource;
use crate::{PyIdCollection};
use intellistream_datahub_sdk::datasets::{Dataset, DatasetUpdate};
use intellistream_datahub_sdk::generic::IdAndExtId;
use intellistream_datahub_sdk::{ApiService, Resource};
use pyo3::{Bound, PyAny, PyResult, Python, pyclass, pymethods};
use intellistream_datahub_sdk::ApiService;
use pyo3::{PyResult, Python, pyclass, pymethods};
use std::sync::Arc;

#[pyclass(module = "intellistream_datahub_sdk", name = "DatasetsServiceSync")]
Expand Down
6 changes: 1 addition & 5 deletions datahub_python_bindings/src/events/async_service.rs
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
use crate::events::{
EventIdentifyable, PyEventFilter, PyEvent, PyEventDimension, PyEventUpdate,
};
use crate::timeseries::async_service::PyTimeSeriesServiceAsync;
use crate::timeseries::{PyTimeSeries, PyTimeSeriesUpdate};
use crate::{PyIdCollection};
use intellistream_datahub_sdk::events::{EventDimension, EventIdCollection, EventUpdate};
use intellistream_datahub_sdk::generic::DataWrapper;
use intellistream_datahub_sdk::{
ApiService, Event, TimeSeries, TimeSeriesUpdate, TimeSeriesUpdateCollection,
ApiService, Event,
};
use pyo3::{Bound, PyAny, PyResult, Python, pyclass, pymethods};
use pyo3_async_runtimes::tokio::future_into_py;
Expand Down
1 change: 0 additions & 1 deletion datahub_python_bindings/src/events/general.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
use crate::PyEvent;
use crate::PyIdCollection;
use crate::datetime::py_datetime_to_utc;
use crate::resources::PyResource;
use chrono::{DateTime, Utc};
use intellistream_datahub_sdk::generic::IdAndExtId;
use pyo3::prelude::*;
Expand Down
7 changes: 1 addition & 6 deletions datahub_python_bindings/src/events/mod.rs
Original file line number Diff line number Diff line change
@@ -1,17 +1,12 @@
use crate::{DataSetRef, PyIdCollection, StringOrList, opt_data_set_refs, opt_patterns};
use crate::datetime::opt_py_datetime_to_utc;
use crate::timeseries::datapoints::{
PyDatapointString, PyDatapointsCollectionDatapoints, PyDatapointsCollectionString,
PyRetrieveFilter,
};
use crate::timeseries::{PyDeleteFilter, PyTimeSeries, PyTimeSeriesUpdate};
use crate::{PyFieldStr, PyFieldU64, PyListFieldIdCollection, PyMapField};
use intellistream_datahub_sdk::filters::{EventFilter, DataSort, EventFilterForm, TimeFilter};
use intellistream_datahub_sdk::events::{
EventDimension, EventIdCollection, EventUpdate, EventUpdateFields,
};
use intellistream_datahub_sdk::generic::IdAndExtId;
use intellistream_datahub_sdk::{ApiService, Event, TimeSeries};
use intellistream_datahub_sdk::{ApiService, Event};
use std::sync::Arc;
use pyo3::exceptions::PyValueError;
use pyo3::prelude::*;
Expand Down
12 changes: 3 additions & 9 deletions datahub_python_bindings/src/events/sync_service.rs
Original file line number Diff line number Diff line change
@@ -1,17 +1,12 @@
use crate::events::{
EventIdentifyable, PyEventFilter, PyEvent, PyEventDimension, PyEventUpdate,
};
use crate::{PyIdCollection};
use intellistream_datahub_sdk::events::{EventDimension, EventIdCollection, EventUpdate};
use intellistream_datahub_sdk::filters::EventFilterForm;
use intellistream_datahub_sdk::generic::DataWrapper;
use intellistream_datahub_sdk::{
ApiService, Event, TimeSeries, TimeSeriesUpdate, TimeSeriesUpdateCollection,
ApiService, Event,
};
use pyo3::{Bound, PyAny, PyResult, Python, pyclass, pymethods};
use pyo3_async_runtimes::tokio::future_into_py;
use pyo3::{PyResult, Python, pyclass, pymethods};
use std::sync::Arc;
use tokio::runtime;
use uuid::Uuid;

#[pyclass(module = "intellistream_datahub_sdk", name = "EventsServiceSync")]
Expand Down Expand Up @@ -96,8 +91,7 @@ impl PyEventsServiceSync {
input.iter().map(|u| EventIdCollection::from(u.clone())).collect();

py.detach(|| {
let result = self
.runtime
self.runtime
.block_on(service.events.delete(&input_ids))
.map_err(|e| crate::datahub_err(e))?;

Expand Down
2 changes: 0 additions & 2 deletions datahub_python_bindings/src/files/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,13 @@ pub mod async_service;
pub mod sync_service;

use crate::datetime::opt_py_datetime_to_utc;
use crate::resources::PyResource;
use chrono::{DateTime, Utc};
use intellistream_datahub_sdk::files::{FileDownload, FileUpdate};
use intellistream_datahub_sdk::generic::{INode, IdAndExtId};
use intellistream_datahub_sdk::{ApiService, FileUpload};
use pyo3::prelude::*;
use pyo3::types::{PyBytes, PyType};
use pyo3_async_runtimes::tokio::future_into_py;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::sync::Arc;

Expand Down
3 changes: 1 addition & 2 deletions datahub_python_bindings/src/files/sync_service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,7 @@ impl PyFilesServiceSync {
let input_ids: Vec<IdAndExtId> = input.into_iter().map(|u| IdAndExtId::from(u)).collect();

py.detach(|| {
let result = self
.runtime
self.runtime
.block_on(service.files.delete(&DataWrapper::from_vec(input_ids)))
.map_err(|e| crate::datahub_err(e))?;

Expand Down
21 changes: 5 additions & 16 deletions datahub_python_bindings/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ pub mod timeseries;
pub mod units;
mod functions;

use crate::datasets::PyDataset;
use crate::datasets::async_service::PyDatasetsServiceAsync;
use crate::datasets::sync_service::PyDatasetsServiceSync;
use crate::files::async_service::PyFilesServiceAsync;
Expand All @@ -33,7 +32,7 @@ use crate::subscriptions::sync_service::PySubscriptionsServiceSync;
use crate::timeseries::async_service::PyTimeSeriesServiceAsync;
use crate::timeseries::datapoints::PyRetrieveFilter;
use crate::timeseries::sync_service::PyTimeSeriesServiceSync;
use crate::timeseries::{PyDeleteFilter, PyTimeSeries};
use crate::timeseries::PyTimeSeries;
use crate::units::PyUnit;
use crate::assets::async_service::PyAssetsServiceAsync;
use crate::assets::sync_service::PyAssetsServiceSync;
Expand All @@ -52,11 +51,9 @@ use pyo3::create_exception;
use pyo3::exceptions::{PyException, PyValueError};
use pyo3::prelude::*;
use pyo3::types::PyType;
use pyo3_async_runtimes::tokio::future_into_py;
use std::collections::HashMap;
use std::sync::Arc;
use std::sync::OnceLock;
use units::*;

create_exception!(
intellistream_datahub_sdk,
Expand Down Expand Up @@ -153,7 +150,6 @@ fn build_buffered_config(
token_url: Option<String>,
client_id: Option<String>,
client_secret: Option<String>,
project_name: Option<String>,
enable_buffering: bool,
buffer_retention_secs: Option<i64>,
buffer_max_bytes: Option<u64>,
Expand All @@ -174,7 +170,6 @@ fn build_buffered_config(
token_url,
client_id,
client_secret,
project_name,
);
if let Some(secs) = buffer_retention_secs {
config.set_buffer_retention_secs(secs);
Expand Down Expand Up @@ -229,7 +224,6 @@ impl PySyncClient {
token_url=None,
client_id=None,
client_secret=None,
project_name=None,
enable_buffering=false,
buffer_retention_secs=None,
buffer_max_bytes=None,
Expand All @@ -250,7 +244,6 @@ impl PySyncClient {
token_url: Option<String>,
client_id: Option<String>,
client_secret: Option<String>,
project_name: Option<String>,
enable_buffering: bool,
buffer_retention_secs: Option<i64>,
buffer_max_bytes: Option<u64>,
Expand All @@ -272,7 +265,6 @@ impl PySyncClient {
token_url,
client_id,
client_secret,
project_name,
enable_buffering,
buffer_retention_secs,
buffer_max_bytes,
Expand All @@ -291,14 +283,14 @@ impl PySyncClient {
}
}
#[classmethod]
fn from_env(py: Py<PyType>) -> PyResult<Self> {
fn from_env(_py: Py<PyType>) -> PyResult<Self> {
Ok(Self {
inner: ApiService::new(DataHubConfig::from_env().unwrap()),
runtime: Arc::new(tokio::runtime::Runtime::new().unwrap()),
})
}
#[classmethod]
fn from_envfile(py: Py<PyType>, path: Option<&str>) -> PyResult<Self> {
fn from_envfile(_py: Py<PyType>, path: Option<&str>) -> PyResult<Self> {
Ok(Self {
inner: ApiService::new(DataHubConfig::from_envfile(path).unwrap()),
runtime: Arc::new(tokio::runtime::Runtime::new().unwrap()),
Expand Down Expand Up @@ -407,7 +399,6 @@ impl PyAsyncClient {
token_url=None,
client_id=None,
client_secret=None,
project_name=None,
enable_buffering=false,
buffer_retention_secs=None,
buffer_max_bytes=None,
Expand All @@ -428,7 +419,6 @@ impl PyAsyncClient {
token_url: Option<String>,
client_id: Option<String>,
client_secret: Option<String>,
project_name: Option<String>,
enable_buffering: bool,
buffer_retention_secs: Option<i64>,
buffer_max_bytes: Option<u64>,
Expand All @@ -450,7 +440,6 @@ impl PyAsyncClient {
token_url,
client_id,
client_secret,
project_name,
enable_buffering,
buffer_retention_secs,
buffer_max_bytes,
Expand All @@ -468,13 +457,13 @@ impl PyAsyncClient {
}
}
#[classmethod]
fn from_env(py: Py<PyType>) -> PyResult<Self> {
fn from_env(_py: Py<PyType>) -> PyResult<Self> {
Ok(Self {
inner: ApiService::new(DataHubConfig::from_env().unwrap()),
})
}
#[classmethod]
fn from_envfile(py: Py<PyType>, path: Option<&str>) -> PyResult<Self> {
fn from_envfile(_py: Py<PyType>, path: Option<&str>) -> PyResult<Self> {
Ok(Self {
inner: ApiService::new(DataHubConfig::from_envfile(path).unwrap()),
})
Expand Down
1 change: 0 additions & 1 deletion datahub_python_bindings/src/resources/sync_service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ use crate::relations::{PyGraphResult, PyRelForm};
use crate::resources::{PyResourceFilter, ResourceIdentifiable};
use crate::resources::{PyResourceNetwork, PyResourceUpdate};
use intellistream_datahub_sdk::resources::ResourceUpdate;
use crate::resources::async_service::PyResourcesServiceAsync;
use crate::{DataSetRef, StringOrList, opt_data_set_refs, opt_patterns};
use intellistream_datahub_sdk::filters::NodeFilter;
use intellistream_datahub_sdk::generic::IdAndExtId;
Expand Down
9 changes: 3 additions & 6 deletions datahub_python_bindings/src/timeseries/async_service.rs
Original file line number Diff line number Diff line change
@@ -1,18 +1,17 @@
use crate::timeseries::datapoints::{
PyDatapoint, PyDatapointsCollectionDatapoints, PyDatapointsCollectionString,
PyDatapointsCollectionDatapoints, PyDatapointsCollectionString,
};
use crate::timeseries::{
PyDeleteFilter, PyTimeSeries, PyTimeSeriesUpdate, PyTimeseriesIdentifiable,
};
use crate::{
DatahubIdentity, Identifiable, PyIdCollection, PyRetrieveFilter,
DatahubIdentity, Identifiable, PyRetrieveFilter,
};
use crate::datetime::py_datetime_to_utc;
use intellistream_datahub_sdk::generic::{
DataWrapper, DatapointString, DatapointsCollection, DeleteFilter, IdAndExtId, RetrieveFilter,
};
use intellistream_datahub_sdk::{ApiService, TimeSeries, TimeSeriesUpdate, TimeSeriesUpdateCollection};
use pyo3::prelude::*;
use pyo3::{Bound, PyAny, PyResult, Python, pyclass, pymethods};
use pyo3_async_runtimes::tokio::future_into_py;
use std::sync::Arc;
Expand Down Expand Up @@ -236,7 +235,6 @@ impl PyTimeSeriesServiceAsync {
.insert_datapoints(&mut wrapper)
.await
.map_err(|e| crate::datahub_err(e))?;
let val = result.get_items().clone();
Ok(result.get_items().clone())
})
}
Expand Down Expand Up @@ -274,7 +272,6 @@ impl PyTimeSeriesServiceAsync {
.insert_datapoints(&mut wrapper)
.await
.map_err(|e| crate::datahub_err(e))?;
let val = result.get_items().clone();
Ok(result.get_items().clone())
})
}
Expand Down Expand Up @@ -309,7 +306,7 @@ impl PyTimeSeriesServiceAsync {
let wrapper =
DataWrapper::<DeleteFilter>::from_vec(input.into_iter().map(|f| f.into()).collect());
future_into_py(py, async move {
let result = service
service
.time_series
.delete_datapoints(&wrapper)
.await
Expand Down
Loading
Loading