diff --git a/examples/databricks/kumo_relational_quickstart.ipynb b/examples/databricks/kumo_relational_quickstart.ipynb index 8a6f1b9c8..247b32ed8 100644 --- a/examples/databricks/kumo_relational_quickstart.ipynb +++ b/examples/databricks/kumo_relational_quickstart.ipynb @@ -464,6 +464,10 @@ "\n", "A comparison after the target turns it into a yes-or-no question: `COUNT(sales_order_headers.*, 0, 90, days) = 0` asks whether a customer places no order in the next 90 days.\n", "A `WHERE` inside the aggregation counts only some rows: `COUNT(sales_order_headers.* WHERE sales_order_headers.TotalDue > 1000, 0, 90, days)` counts only orders over $1,000.\n", + "\n", + "**Explanations**\n", + "\n", + "`EXPLAIN PREDICT …` returns the same predictions, and also explanations of them; see Example 5.\n", "" ] }, @@ -884,6 +888,177 @@ "This lets a team ask questions about the measures it already defines in metric views." ] }, + { + "cell_type": "markdown", + "id": "620b2a07b418435c90d8e7dca1fd869f", + "metadata": {}, + "source": [ + "### Example 5: Explain the predictions\n", + "\n", + "We predicted which customers are likely to place no orders in the next 90 days. Now we'll explore patterns in the historical examples and inspect how one customer's prediction responds to small input changes.\n", + "\n", + "Explanations are available with NIM version 1.1+ and client version 1.1.0+.\n", + "\n", + "The result carries explanations at two levels:\n", + "\n", + "- `result.explanations.columns`, computed by the client from the historical examples the model reads. It groups customers by ranges or categories in each column. Each group reports its share of the examples and the fraction who placed no orders in the following 90 days.\n", + "- `result.explanations.gradients`, returned by the model for each explained customer. A gradient describes how the predicted probability responds to a small change in a preprocessed numerical input. It does not measure how much that input contributed to the prediction.\n", + "\n", + "Add `EXPLAIN` to request both column analysis and gradients." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b2ee7bc6cf2e43d8b7e7e3207117ea3d", + "metadata": {}, + "outputs": [], + "source": [ + "# Explain the 25 customers Example 1 rated riskiest.\n", + "riskiest_25 = churn.loc[riskiest[:25], 'id'].tolist()\n", + "request = PredictiveQuery(\n", + " 'EXPLAIN PREDICT COUNT(sales_order_headers.*, 0, 90, days) = 0 '\n", + " 'FOR EACH customers.CustomerID',\n", + " schema,\n", + ").materialize(indices=riskiest_25, anchor_time=PAST, num_context_samples=5000)\n", + "result = client.predict(request, model=model)" + ] + }, + { + "cell_type": "markdown", + "id": "dce952ea80cc428fb83782dd23aa2b3f", + "metadata": {}, + "source": [ + "#### What patterns do we see in the historical data?\n", + "\n", + "The column analysis groups customers by ranges or categories in each column. For each group, the table shows what percentage of the historical examples it represents and what percentage placed no orders in the following 90 days.\n", + "\n", + "Columns of linked tables, such as orders, are grouped per order, not per customer. For `OrderDate`, each order falls into a window of time before the customer's anchor date, for example `[0 days, 30 days)`. A customer whose orders fall into several windows is split across them by their share of orders: with 3 of 4 orders in the last 30 days, they count as three quarters in that window and one quarter in the window holding the fourth order." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2b5bb9ac990d40baab07c0a0cefab7e9", + "metadata": {}, + "outputs": [], + "source": [ + "columns = result.explanations.columns.to_pandas()\n", + "\n", + "# True means the customer placed no orders in the following 90 days.\n", + "groups = columns.loc[\n", + " columns['class'] == True, # noqa: E712\n", + " ['table', 'hop', 'column', 'cohort', 'share', 'mean'],\n", + "].copy()\n", + "\n", + "display(columns)" + ] + }, + { + "cell_type": "markdown", + "id": "7e07694e4f5c42a191903997b324abfc", + "metadata": {}, + "source": [ + "Each row answers: **How common is this group, and how often did customers in it stop ordering?**\n", + "\n", + "For example, a row with `30` under `share` and `45` under `mean` would mean that the group contains 30% of the historical examples, and 45% of that group placed no orders.\n", + "\n", + "#### Do customers with older ordering activity stop ordering more often?\n", + "\n", + "Zoom in on `OrderDate`. Each group is a window of time before the anchor date, and its percentages are weighted by how much of each customer's orders fall into it. Customers with no orders before the anchor date do not appear in these groups." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c2becede8cda4ba09b497c1d48b39369", + "metadata": {}, + "outputs": [], + "source": [ + "order_dates = groups[\n", + " (groups['table'] == 'sales_order_headers')\n", + " & (groups['column'] == 'OrderDate')\n", + "]\n", + "display(order_dates)" + ] + }, + { + "cell_type": "markdown", + "id": "27b22710d2de41ff9da8cc017d6dfb61", + "metadata": {}, + "source": [ + "Compare the percentages across time windows within the same hop. If older groups have higher percentages of customers placing no orders, that is a pattern in these historical examples. The group sizes show how much of the data the pattern covers.\n", + "\n", + "This helps you understand the data behind the predictions. It does not establish why a particular customer received their risk score.\n", + "\n", + "#### How would small input changes affect one customer's predicted probability?\n", + "\n", + "Gradients describe how the model output responds to small changes near the current input values. They use the model's numerical inputs after preprocessing, which may differ from the original columns.\n", + "\n", + "For the riskiest customer, inspect ten entries with the largest absolute gradients:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7b914676ad1e4ecbac19bf90c24bb735", + "metadata": {}, + "outputs": [], + "source": [ + "gradients = result.explanations.gradients.to_pandas()\n", + "customer_gradients = gradients[\n", + " (gradients['id'] == riskiest_25[0])\n", + " & (gradients['output'] == 'probability_True')\n", + "]\n", + "display(\n", + " customer_gradients.loc[\n", + " customer_gradients['gradient'].abs().nlargest(10).index\n", + " ]\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "a7b667632bfa4a48be8e94665ef115ca", + "metadata": {}, + "source": [ + "A positive gradient means a small increase in that preprocessed value would locally increase the predicted probability of placing no orders; a negative gradient means it would decrease it.\n", + "\n", + "The sizes depend on the input scales, so these ten entries are not necessarily the inputs that contributed most to the prediction.\n", + "\n", + "Use this table to investigate sensitivity—for example, when checking a possibly incorrect source value. Correct the value using the source record, rerun the prediction, and check whether the customer's risk score changes enough to affect your outreach decision. Gradients alone do not tell you which business action would prevent churn." + ] + }, + { + "cell_type": "markdown", + "id": "be17fd43", + "metadata": {}, + "source": [ + "#### 5.2 Ask with outputs\n", + "\n", + "The `EXPLAIN` prefix and the `outputs` argument ask for the same thing.\n", + "To keep the query text fixed and decide per call, pass `outputs=['prediction', 'probabilities', 'explanation']`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1575e783", + "metadata": {}, + "outputs": [], + "source": [ + "# The same request and explanations, without the EXPLAIN prefix.\n", + "request = PredictiveQuery(\n", + " 'PREDICT COUNT(sales_order_headers.*, 0, 90, days) = 0 '\n", + " 'FOR EACH customers.CustomerID',\n", + " schema,\n", + ").materialize(indices=riskiest_25, anchor_time=PAST, num_context_samples=5000)\n", + "result = client.predict(\n", + " request, model=model, outputs=['prediction', 'probabilities', 'explanation']\n", + ")\n", + "print(f'{len(result.explanations.gradients)} predictions explained.')" + ] + }, { "cell_type": "markdown", "id": "064697a1bad1b5348afae26f2d9acec4",