Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,13 +47,15 @@ these packages. All five distributions share one version and one entry.
row, output and column, with each row's `hop` from the entity.
- `result.explanations.columns` is a column analysis the client computes from
the labelled context rows: each column split into cohorts, each cohort's
share and mean outcome, and a score per column, normalized to sum to 1.
share and mean outcome.
`result.explanations.columns.to_pandas()` gives one line per table, hop,
column, cohort and class. `predict()` computes it on each call that asks for
explanations; a session computes it once at `fit()`, after `holdout` is
taken out, and attaches it to every answer when `fit()` asked for
`'explanation'`, otherwise only to a prediction whose query begins
`EXPLAIN`. It is `None` when explanations were not asked for.
`EXPLAIN`. It is `None` when explanations were not asked for. It describes
the labelled rows the request sends, not the rows gathered before the leak
check; the request no longer keeps a copy of the gathered context.
- The `EXPLAIN` query prefix: `EXPLAIN PREDICT …` adds `'explanation'` to the
call's outputs. A ranking refuses it; a forecast explains each step.

Expand Down
23 changes: 11 additions & 12 deletions docs/reference/prediction-output.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,8 +289,9 @@ user_5 events 1 None 1 probability_yes kind_login -0.02

`result.explanations.columns` shows, for each column, how the outcome differs
across groups of the labelled context rows. The client computes it from the
labelled rows it sends; the endpoint is not asked for it. It is `None` when
explanations were not asked for.
labelled rows the request sends; rows the leak check removed are not in it.
The endpoint is not asked for it. It is `None` when explanations were not
asked for.

When it is computed:

Expand All @@ -307,28 +308,26 @@ bins, categories into one cohort each (the tenth onward merged into
`OTHER(*)`, missing values under `N/A`), timestamps into ranges of time before
the anchor. Each cohort has its `share` of the entities and its `mean`
outcome: the rate of each class for classification (classes by frequency, at
most 30), the plain mean for regression. A column's `score` is how much its
cohort means spread, weighted by share and averaged over classes; scores sum
to 1 across columns.
most 30), the plain mean for regression.

| Model | Columns analysed |
|---|---|
| kumo-tabular | every column of the one table except the target, the primary key and timestamps (there is no anchor), reported under table `instance_table` |
| kumo-relational | every column of each related table except keys, per hop the table is reached at, plus `COUNT(*)`, the number of that table's rows linked to the entity. The entity's own table is not analysed. |
| kumo-relational | every column of each related table except keys, per hop the table is reached at, plus `COUNT(*)`, the number of that table's rows linked to the entity. The entity's own table is included, at hop 0; only the question rows themselves (the anchor and target) are not analysed. |

On `kumo-relational` an entity's related rows at one hop are averaged first,
then entities are grouped into cohorts.

`result.explanations.columns.to_pandas()` gives one line per (table, hop,
column, cohort, class), highest `score` first. Values are illustrative, and
column, cohort, class), columns in context order. Values are illustrative, and
the `no` lines are left out:

```
table hop column stype cohort class share mean score
orders 1 COUNT(*) None [0 - 0] yes 0.30 0.12 0.41
orders 1 COUNT(*) None (0 - 3] yes 0.45 0.38 0.41
orders 1 COUNT(*) None (3 - 18+] yes 0.25 0.71 0.41
orders 1 amount numerical [3 - 12] yes 0.50 0.22 0.27
table hop column stype cohort class share mean
orders 1 COUNT(*) None [0 - 0] yes 0.30 0.12
orders 1 COUNT(*) None (0 - 3] yes 0.45 0.38
orders 1 COUNT(*) None (3 - 18+] yes 0.25 0.71
orders 1 amount numerical [3 - 12] yes 0.50 0.22
```

`class` is `None` for regression. `stype` is `None` on a `COUNT(*)` line.
114 changes: 95 additions & 19 deletions packages/structured-data-models-client/src/sdm_client/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -595,6 +595,9 @@ def _answer_in_parts(
# of a forecast is an ordinary regression, so the caller's fields are
# what each part is asked for.
wanted = _RANKING_PARTS if ranking else outputs
if request.parts:
# Refused before the analysis below, not after it.
self._prepare(request.parts[0], model, wanted)
# A forecast's analysis reads every step's labelled rows at once.
columns = (
_analysed_or_warned(
Expand Down Expand Up @@ -1051,23 +1054,26 @@ class _LabelledRows:
r"""The labelled rows a request's column analysis reads."""

_INSTANCE_ID = 'instance_id'
_ENTITY_REF = '__kumo_entity_ref'

def __init__(self) -> None:
# Each step's instance ids, renumbered so no two steps share one.
self._ids: dict[tuple[int, Any], int] = {}

@staticmethod
def of(request: Request, planned: Mapping[str, Any]) -> dict[str, Any]:
r"""``planned``'s task and schema over the request's labelled context.
@classmethod
def of(cls, request: Request, planned: Mapping[str, Any]) -> dict[str, Any]:
r"""``planned``'s task over the labelled rows the request sends.

A gathered request carries its context as it was before the rows
answering the scored ones were dropped; any other request's planned
context is that context.
A request built from frames has no payloads; its planned context is
what it sends.
"""
labelled = request._labelled_context
payloads = request.payloads
if not payloads:
return {**planned}
return {
**planned,
'context': planned['context'] if labelled is None else labelled,
'schema': cls._schema([payload['schema'] for payload in payloads]),
'context': cls._sent(payloads),
}

def across(self, steps: Sequence[Request]) -> dict[str, Any]:
Expand Down Expand Up @@ -1105,25 +1111,84 @@ def across(self, steps: Sequence[Request]) -> dict[str, Any]:
)
for name in names
}
# A step's schema describes only the tables that step reached.
schemas = [payload['schema'] for payload in each]
return {
**each[0],
'schema': self._schema([payload['schema'] for payload in each]),
'context': context,
}

@classmethod
def _sent(cls, payloads: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
r"""The payloads' labelled contexts as one, each instance once.

Every payload carries the same labelled draw, less its own scored
entities, under the same instance ids. An instance is keyed by entity
and anchor; the first payload sending it gives its row and its
related rows.
"""
seen: set[tuple[Any, Any]] = set()
instances: list[Any] = []
related: dict[str, dict[str, Any]] = {}
for payload in payloads:
context = payload['context']
table = context['instance_table']
columns = list(table['columns'])
at = columns.index(cls._INSTANCE_ID)
entity = next(
index
for index, name in enumerate(columns)
if str(name).startswith(cls._ENTITY_REF)
Comment thread
JGagacheva marked this conversation as resolved.
Resolved
)
anchor_column = payload['task'].get('anchor_time_column')
anchor = (
columns.index(anchor_column)
if anchor_column in columns
else None
)
keys: set[tuple[Any, Any]] = set()
kept: set[Any] = set()
for row in table['rows']:
key = (row[entity], None if anchor is None else row[anchor])
keys.add(key)
if key not in seen:
kept.add(row[at])
instances.append(row)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
seen |= keys
for name, rows in (context.get('related_tables') or {}).items():
held = related.setdefault(name, {**rows, 'rows': []})
where = list(rows['columns']).index(cls._INSTANCE_ID)
held['rows'].extend(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Batches can send the same related table with different columns (a later batch reaching another relationship adds one), and this appends later rows under the first batch's columns, so the analysis fails and is dropped; please align rows by column name.

row for row in rows['rows'] if row[where] in kept
)
first = payloads[0]['context']
sent: dict[str, Any] = {
'instance_table': {**first['instance_table'], 'rows': instances}
}
if any('related_tables' in payload['context'] for payload in payloads):
sent['related_tables'] = related
return sent

@staticmethod
def _schema(schemas: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
r"""One schema describing every table any of ``schemas`` reached.

A payload's schema describes only the tables that payload reached.
"""
schema = dict(schemas[0])
if any('related_tables' in step_schema for step_schema in schemas):
if any('related_tables' in each for each in schemas):
described: dict[str, Any] = {}
for step_schema in schemas:
for name, table in (
step_schema.get('related_tables') or {}
).items():
for each in schemas:
for name, table in (each.get('related_tables') or {}).items():
described.setdefault(name, table)
schema['related_tables'] = described
if any('relationships' in step_schema for step_schema in schemas):
if any('relationships' in each for each in schemas):
links: list[Any] = []
for step_schema in schemas:
for link in step_schema.get('relationships') or []:
for each in schemas:
for link in each.get('relationships') or []:
if link not in links:
links.append(link)
schema['relationships'] = links
return {**each[0], 'schema': schema, 'context': context}
return schema

def _stacked(
self, tables: Sequence[tuple[int, Mapping[str, Any]]]
Expand Down Expand Up @@ -1531,10 +1596,21 @@ def _by_step(
),
columns,
)
# Each step is its own call, and only the first step's id would otherwise
# survive. That is the one you do not have when a later step is the one
# that misbehaved. A step sent in several calls keeps every one of them.
metadata = {
'steps': [step for _, step in rewritten.parts],
'step_request_ids': [
list(answer.metadata.get('batch_request_ids', [answer.request_id]))
for answer in answers
],
}
built = PredictionResult(
model=answers[0].model if answers else '',
request_id=answers[0].request_id if answers else '',
items=items,
metadata=metadata,
outputs=tuple(answers[0].outputs) if answers else (),
explanations=explanations,
)
Expand Down
Loading
Loading