Skip to content

Commit 5bdb68d

Browse files
feat(app): add an option to show MATERIALIZED and ALIAS columns in the row details panel (#3231)
## Summary The row details panel fetches the row with `SELECT *`. ClickHouse leaves MATERIALIZED and ALIAS columns out of `*` unless `asterisk_include_materialized_columns` and `asterisk_include_alias_columns` are on, so the panel never showed them: computed columns that users add, and the stock `otel_logs` `*AttributeItems` ALIAS columns. As discussed in #3206, showing them is opt-in. The properties view options menu (the gear in Column Values) gets a "Show materialized and alias columns" item. It is off by default and saved in local storage like the other panel preferences (`hooks/useMaterializedAliasColumnsOption.ts`). - `useRowData` adds the two settings to the bounded and the fallback row query only when the option is on and the query selects `*`. Sources with a Known Columns List are unchanged. - `useQueriedChartConfig` gets an `additionalQuerySettings` option. It is merged after the source's `querySettings`, goes through the existing `SETTINGS` path, and is added to the query key, also when the caller passes its own `queryKey`. Since #3234 the row query carries the source, so a value that the source sets for either setting wins: a source with both settings on shows the columns without the option (the workaround from #3206), and a source with `asterisk_include_alias_columns = 0` keeps ALIAS columns out even with the option on. - If the connection's user cannot change these settings (for example, a `readonly = 1` user, or a settings profile that makes them `CONST`), ClickHouse rejects the row query. The query fails like any other row error: the error state shows the error and offers to turn the option off. An earlier version fetched the row again without the settings; as discussed in review, a visible error is better for an opt-in option. Checking `system.settings` first (the setting exists and `readonly = 0`) and showing a warning when the settings are not applied is a possible follow-up. Things to be aware of: - With the option on, the `__hdx_materialized_*` columns are fetched too, but `DBRowJsonViewer` still hides keys that start with `__hdx_`. I left a toggle for those out of this PR. Happy to add it to this option or a separate one if you'd like. - With the option on, the row query fails if the user cannot change the settings, or if an ALIAS column cannot be evaluated. A failed row shows the error state instead of Column Values, so the option's menu is out of reach there. The error state therefore offers "Hide materialized and alias columns" whenever the failed query carried the settings. Setting `asterisk_include_alias_columns = 0` on the source also avoids the ALIAS case. - `SpanDetailPanel`, `DBRowOverviewPanel` and `DBTracePanel` read the row data but do not show `isError` (as on `main`), so they stay empty when the row query fails. ### Screenshots Local mode against a local ClickHouse 26.8, stock `otel_logs` schema plus `HttpStatusCode UInt16 MATERIALIZED toUInt16OrZero(LogAttributes['http.response.status_code'])` and `StatusClass String ALIAS multiIf(...)`. Column Values is filtered to "Status". | Option off (the default) | The new menu item | | :----------------------- | :---------------- | | ![Option off: only LogAttributes http.response.status_code](https://raw.githubusercontent.com/BilalAtique/hyperdx/pr-3231-screenshots/opt-in-01-option-off.png) | ![Properties view options menu with "Show materialized and alias columns"](https://raw.githubusercontent.com/BilalAtique/hyperdx/pr-3231-screenshots/opt-in-02-menu.png) | | Option on: MATERIALIZED `HttpStatusCode`, ALIAS `LogAttributeItems` and `StatusClass` | User whose profile makes both settings `CONST` (452), option on: the row query fails and the error state offers to turn the option off | | :--- | :--- | | ![HttpStatusCode 503, LogAttributeItems, StatusClass 5xx](https://raw.githubusercontent.com/BilalAtique/hyperdx/pr-3231-screenshots/opt-in-03-option-on.png) | ![Restricted user: error 452 with the "Hide materialized and alias columns" action](https://raw.githubusercontent.com/BilalAtique/hyperdx/pr-3231-screenshots/opt-in-04-restricted-user-error.png) | Option on, and an ALIAS column that throws for this row (`throwIf(HttpStatusCode >= 500, ...)`): the error state offers to turn the option off. After the click, the row loads without the settings. ![Error state with the "Hide materialized and alias columns" action](https://raw.githubusercontent.com/BilalAtique/hyperdx/pr-3231-screenshots/opt-in-05-error-hint.png) ### Tests - `DBRowDataPanel.test.ts`: no settings while the option is off; both settings on the bounded and the fallback query when it is on; only the settings a source doesn't set itself, and none when it sets both; a Known Columns List passes none; a `READONLY` rejection is reported as the error, and every query keeps the settings. - `DBRowJsonViewer.test.tsx`: the menu item turns the option on and off and saves it. - `DBRowSidePanelErrorState.test.tsx`: the "Hide materialized and alias columns" action appears when the failed query carried the settings (an ALIAS column error and a `READONLY` rejection) and turns the option off; it does not appear while the option is off, when the source sets both settings itself, or when a Known Columns List replaces `SELECT *`. - `useChartConfig.test.tsx`: `additionalQuerySettings` reaches `queryChartConfig` as `querySettings` and is added to a caller-supplied `queryKey`; `mergeQuerySettings` cases, including a setting that the source already defines. - The tests from earlier rounds were each run against the code without their change and failed there. The new `READONLY` test in `DBRowDataPanel.test.ts` fails against the previous commit, which fetched the row again without the settings. All 145 tests in those four files and `__tests__/DBRowJsonViewer.test.ts` pass. - `queryChartConfig.int.test.ts`: `SELECT *` through `queryChartConfig` on a table with a MATERIALIZED and an ALIAS column returns only the plain columns by default, and all four with the settings. - End to end, in the app in local mode against ClickHouse 26.8, checked in `system.query_log`: - Option off: the row queries ran without either setting, and Column Values lists only the plain columns. - Option on: the row queries ran with `asterisk_include_materialized_columns = 1` and `asterisk_include_alias_columns = 1`, and `HttpStatusCode`, `StatusClass` and the `*AttributeItems` columns are listed. - Constrained user, option on: the row query failed with 452 (`Setting asterisk_include_materialized_columns should not be changed`), and its one retry too. No query ran without the settings. The error state showed the error and the action, and after the click the plain query loaded the row. - Option on, ALIAS column that throws for the row: the row query failed with 395 (and its one retry), the error state showed the action, and after the click the plain query loaded the row. The option was saved as off. - ESLint (no new warnings) and Prettier pass on the changed files. I did not run `tsc` locally; CI covers it. ### How to test on Vercel preview **Preview routes:** /search **Steps:** 1. Open /search and select a Logs source. 2. Click the first row in the results table to open the row side panel, and open the "Column Values" tab. 3. Verify that only the plain columns are listed. 4. Open the gear menu next to the search box and click "Show materialized and alias columns". 5. Verify that the row loads without an error and, if the logs table has ALIAS columns (the stock `otel_logs` schema has `LogAttributeItems` and `ResourceAttributeItems`), that those columns are listed. ### References - Closes #3206 - Related: #3234 (source query settings on the row lookup), #1400 (the same row query runs twice when the panel opens) - Introduction and vouch request: #3232 Co-authored-by: Drew Davis <6097246+pulpdrew@users.noreply.github.com>
1 parent 8a9fcd2 commit 5bdb68d

11 files changed

Lines changed: 558 additions & 8 deletions
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
---
2+
'@hyperdx/app': patch
3+
---
4+
5+
feat: add an option to show MATERIALIZED and ALIAS columns in the row details panel
6+
7+
ClickHouse leaves MATERIALIZED and ALIAS columns out of `SELECT *`, so the row
8+
details panel never showed them. A new "Show materialized and alias columns"
9+
item in the properties view options menu, off by default, adds
10+
`asterisk_include_materialized_columns` and `asterisk_include_alias_columns` to
11+
the row query. It has no effect on a source with a Known Columns List, and a
12+
value that the source's query settings give for either setting wins. If the row
13+
fails to load while the option is on (for example, because the connection's user
14+
is `readonly = 1`, or an ALIAS column cannot be evaluated), the error state
15+
offers to turn the option off.

‎packages/app/src/components/DBRowDataPanel.tsx‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,11 @@ import {
1010
import { Box } from '@mantine/core';
1111

1212
import { useQueriedChartConfig } from '@/hooks/useChartConfig';
13+
import {
14+
getKnownColumnsList,
15+
getSelectAllColumnsQuerySettings,
16+
useSelectAllColumnsQuerySettings,
17+
} from '@/hooks/useMaterializedAliasColumnsOption';
1318
import { WithClause } from '@/hooks/useRowWhere';
1419
import {
1520
getDisplayedTimestampValueExpression,
@@ -90,10 +95,7 @@ export function useRowData({
9095
// target tables declare different column sets. When the source declares a
9196
// "known columns" list (columns known to exist across all target tables) we
9297
// select that instead of `*` when fetching full row data.
93-
const knownColumns =
94-
isLogSource(source) || isTraceSource(source)
95-
? source.knownColumnsListExpression?.trim()
96-
: undefined;
98+
const knownColumns = getKnownColumnsList(source);
9799

98100
const baseConfig = {
99101
source: source.id,
@@ -210,6 +212,8 @@ export function useRowData({
210212
...(aliasWith && aliasWith.length > 0 ? { with: aliasWith } : {}),
211213
};
212214

215+
const additionalQuerySettings = useSelectAllColumnsQuerySettings(source);
216+
213217
const baseQueryKey = ['row_side_panel', rowId, aliasWith, source];
214218
// Both halves of the filter are needed for `renderChartConfig` to emit one, so
215219
// a source with no usable timestamp expression can't be bounded at all.
@@ -225,6 +229,7 @@ export function useRowData({
225229
{
226230
queryKey: [...baseQueryKey, dateRange],
227231
enabled: rowId != null && hasWindow,
232+
additionalQuerySettings,
228233
},
229234
);
230235

@@ -245,6 +250,7 @@ export function useRowData({
245250
const fallbackResult = useQueriedChartConfig(baseConfig, {
246251
queryKey: [...baseQueryKey, undefined],
247252
enabled: rowId != null && isFallbackActive,
253+
additionalQuerySettings,
248254
});
249255

250256
const queryResult = isFallbackActive ? fallbackResult : boundedResult;
@@ -379,6 +385,9 @@ export function RowDataPanel({
379385
data={firstRow}
380386
jsonColumns={jsonColumns}
381387
mapColumns={mapColumns}
388+
showMaterializedAliasColumnsOption={
389+
getSelectAllColumnsQuerySettings(source) != null
390+
}
382391
/>
383392
</Box>
384393
</div>

‎packages/app/src/components/DBRowJsonViewer.test.tsx‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
import React from 'react';
22
import { fireEvent, screen, within } from '@testing-library/react';
33

4+
import { useMaterializedAliasColumnsOption } from '@/hooks/useMaterializedAliasColumnsOption';
5+
46
import { buildJSONExtractQuery, DBRowJsonViewer } from './DBRowJsonViewer';
57
import { RowSidePanelContext } from './DBRowSidePanel';
68

@@ -186,6 +188,46 @@ describe('DBRowJsonViewer', () => {
186188
expect(mockToggleColumn).toHaveBeenCalledWith("LogAttributes['field1']");
187189
});
188190

191+
it('turns the materialized and alias columns option on and off from the menu', async () => {
192+
const OptionValue = () => (
193+
<span data-testid="materialized-alias-option">
194+
{String(useMaterializedAliasColumnsOption()[0])}
195+
</span>
196+
);
197+
const { container } = renderWithMantine(
198+
<RowSidePanelContext value={defaultContext}>
199+
<DBRowJsonViewer data={logData} showMaterializedAliasColumnsOption />
200+
<OptionValue />
201+
</RowSidePanelContext>,
202+
);
203+
const option = screen.getByTestId('materialized-alias-option');
204+
const toggle = async () => {
205+
fireEvent.click(container.querySelector('[aria-haspopup="menu"]')!);
206+
fireEvent.click(
207+
await screen.findByTestId('json-viewer-materialized-alias-toggle'),
208+
);
209+
};
210+
211+
expect(option).toHaveTextContent('false');
212+
await toggle();
213+
expect(option).toHaveTextContent('true');
214+
expect(
215+
localStorage.getItem('hdx-row-show-materialized-alias-columns'),
216+
).toBe('true');
217+
await toggle();
218+
expect(option).toHaveTextContent('false');
219+
});
220+
221+
it('hides the materialized and alias columns option unless asked to show it', async () => {
222+
const { container } = renderComponent(logData);
223+
fireEvent.click(container.querySelector('[aria-haspopup="menu"]')!);
224+
225+
expect(await screen.findByText('Hide blank values')).toBeInTheDocument();
226+
expect(
227+
screen.queryByTestId('json-viewer-materialized-alias-toggle'),
228+
).not.toBeInTheDocument();
229+
});
230+
189231
describe('timestamp fields', () => {
190232
it('displays Timestamp using the same formatter as the results table', () => {
191233
renderComponent({

‎packages/app/src/components/DBRowJsonViewer.tsx‎

Lines changed: 37 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ import HyperJson, {
3232
GetLineActions,
3333
LineAction,
3434
} from '@/components/HyperJson';
35+
import { useMaterializedAliasColumnsOption } from '@/hooks/useMaterializedAliasColumnsOption';
3536
import { useFormatTime } from '@/useFormatTime';
3637
import { isColumnInSelect, mergePath } from '@/utils';
3738
import {
@@ -227,8 +228,16 @@ const viewerOptionsAtom = atomWithStorage<ViewerOptions>(
227228
viewerOptionsStorage,
228229
);
229230

230-
function HyperJsonMenu({ rowData }: { rowData: any }) {
231+
function HyperJsonMenu({
232+
rowData,
233+
showMaterializedAliasColumnsOption,
234+
}: {
235+
rowData: any;
236+
showMaterializedAliasColumnsOption: boolean;
237+
}) {
231238
const [jsonOptions, setJsonOptions] = useAtom(viewerOptionsAtom);
239+
const [showMaterializedAliasColumns, setShowMaterializedAliasColumns] =
240+
useMaterializedAliasColumnsOption();
232241
const effectiveWhiteSpace = jsonOptions.whiteSpace ?? 'pre-wrap';
233242

234243
return (
@@ -335,6 +344,23 @@ function HyperJsonMenu({ rowData }: { rowData: any }) {
335344
>
336345
Hide blank values
337346
</Menu.Item>
347+
{showMaterializedAliasColumnsOption && (
348+
<Menu.Item
349+
lh="1"
350+
py={8}
351+
data-testid="json-viewer-materialized-alias-toggle"
352+
rightSection={
353+
showMaterializedAliasColumns ? (
354+
<IconCheck size={14} className="ps-2" />
355+
) : null
356+
}
357+
onClick={() =>
358+
setShowMaterializedAliasColumns(!showMaterializedAliasColumns)
359+
}
360+
>
361+
Show materialized and alias columns
362+
</Menu.Item>
363+
)}
338364
</Menu.Dropdown>
339365
</Menu>
340366
</Group>
@@ -345,13 +371,17 @@ export function DBRowJsonViewer({
345371
data,
346372
jsonColumns,
347373
mapColumns,
374+
showMaterializedAliasColumnsOption = false,
348375
}: {
349376
data: any;
350377
jsonColumns?: string[];
351378
// Map column names from the result-set metadata. Threaded into
352379
// `mergePath` so numeric-looking sub-keys on a Map render as
353380
// `Map['key']` instead of the array `Map[N+1]`. HDX-4369.
354381
mapColumns?: string[];
382+
// Only a viewer of the whole row from `useRowData` is affected by the option,
383+
// and only when the source's row query would change.
384+
showMaterializedAliasColumnsOption?: boolean;
355385
}) {
356386
const formatTime = useFormatTime();
357387
const {
@@ -699,7 +729,12 @@ export function DBRowJsonViewer({
699729
</Button>
700730
)}
701731
<div className="flex-grow-1" />
702-
<HyperJsonMenu rowData={rowData} />
732+
<HyperJsonMenu
733+
rowData={rowData}
734+
showMaterializedAliasColumnsOption={
735+
showMaterializedAliasColumnsOption
736+
}
737+
/>
703738
</Group>
704739
</Box>
705740
<Paper bg="transparent" mt="sm">

‎packages/app/src/components/DBRowSidePanelErrorState.tsx‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,10 @@ import { useDisclosure } from '@mantine/hooks';
1414
import { IconAlertTriangle } from '@tabler/icons-react';
1515

1616
import { IS_LOCAL_MODE } from '@/config';
17+
import {
18+
useMaterializedAliasColumnsOption,
19+
useSelectAllColumnsQuerySettings,
20+
} from '@/hooks/useMaterializedAliasColumnsOption';
1721
import { useTableMetadata } from '@/hooks/useMetadata';
1822
import { useBrandDisplayName } from '@/theme/ThemeProvider';
1923

@@ -82,6 +86,30 @@ function KnownColumnsListHint({
8286
);
8387
}
8488

89+
// The option lives in the Column Values menu, which a failed row does not show.
90+
function MaterializedAliasColumnsHint({ onHide }: { onHide: () => void }) {
91+
return (
92+
<Alert
93+
variant="warning"
94+
icon={<IconAlertTriangle size={16} />}
95+
title="Materialized and alias columns are on"
96+
data-testid="materialized-alias-columns-hint"
97+
>
98+
<Stack gap="xs" align="start">
99+
<Text size="sm">
100+
The row query includes MATERIALIZED and ALIAS columns. The whole row
101+
fails to load if the connection&apos;s user cannot change query
102+
settings (for example, a readonly user) or if one of these columns
103+
cannot be evaluated.
104+
</Text>
105+
<Button size="xs" variant="subtle" onClick={onHide}>
106+
Hide materialized and alias columns
107+
</Button>
108+
</Stack>
109+
</Alert>
110+
);
111+
}
112+
85113
export function DBRowSidePanelErrorState({
86114
error,
87115
source,
@@ -94,6 +122,9 @@ export function DBRowSidePanelErrorState({
94122

95123
const showHint =
96124
isMissingColumnError(error) && !!tableMetadata?.isPointerTable;
125+
const selectAllColumnsSettings = useSelectAllColumnsQuerySettings(source);
126+
const [, setShowMaterializedAliasColumns] =
127+
useMaterializedAliasColumnsOption();
97128

98129
return (
99130
<Stack gap="sm" data-testid="row-error-state">
@@ -103,6 +134,12 @@ export function DBRowSidePanelErrorState({
103134
<KnownColumnsListHint onEditClick={editModal.open} source={source} />
104135
)}
105136

137+
{selectAllColumnsSettings != null && (
138+
<MaterializedAliasColumnsHint
139+
onHide={() => setShowMaterializedAliasColumns(false)}
140+
/>
141+
)}
142+
106143
<Stack align="start">
107144
<Text size="sm" mt={10}>
108145
Error Message:

0 commit comments

Comments
 (0)