Skip to content

ydb workload kv run: add workload type upsert-seq - #2084

Open
ijon wants to merge 4 commits into
ydb-platform:mainfrom
ijon:ydb_workload_kv_run-add-opsert_seq-workload-type
Open

ijon wants to merge 4 commits into
ydb-platform:mainfrom
ijon:ydb_workload_kv_run-add-opsert_seq-workload-type

Conversation

@ijon

@ijon ijon commented Feb 19, 2024

Copy link
Copy Markdown
Contributor

Changelog category

  • Not for changelog (changelog entry is not required)

Additional information

Add upsert-seq workload type for ydb workload kv run command.
upsert-seq makes upserts on the sequentially increasing keys -- special mode needed to induce (together with specific auto partitioning settings) table datashard splitting (by size).

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@ijon
ijon requested a review from kungasc February 20, 2024 15:07
Comment thread ydb/library/workload/kv_workload.h Resolved

@kungasc kungasc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation is required

@kungasc
kungasc self-requested a review February 20, 2024 17:34
Comment thread ydb/library/workload/kv_workload.cpp Resolved
Comment thread ydb/library/workload/kv_workload.cpp Resolved

@kungasc kungasc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixes are needed

@ijon
ijon force-pushed the ydb_workload_kv_run-add-opsert_seq-workload-type branch from c344298 to d2e86b9 Compare February 29, 2024 17:19
@ijon
ijon requested a review from a team as a code owner February 29, 2024 17:19
@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

Comment thread ydb/library/workload/kv_workload.cpp Outdated
TVector<IWorkloadQueryGenerator::TWorkloadType> TKvWorkloadGenerator::GetSupportedWorkloadTypes() const {
TVector<TWorkloadType> result;
result.emplace_back(static_cast<int>(EType::UpsertRandom), "upsert", "Upsert random rows into table");
result.emplace_back(static_cast<int>(EType::UpsertSequential), "upsert-seq", "Upsert sequential rows into table");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

назвал бы тогда уж полно флаг upsert-sequential

kungasc
kungasc previously approved these changes Feb 29, 2024
@github-actions

This comment was marked as outdated.

Comment thread ydb/library/workload/kv_workload.cpp Resolved
Comment thread ydb/library/workload/kv_workload.cpp Outdated
result[row].Ints[col] = val;
} else {
TString val;
if (col < Params.KeyColumnsCnt) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Тут бы сделать режим с аналогичным GenerateRandomRows(bool randomValues), чтобы буквы случайные генерились

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Я бы на самом деле и вообще бы выкинул без букв, с ними удобнее смотреть чем со странной строчкой, которая ещё и сжимается lz4 зачем-то

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Было бы странно сделать это только для GenerateSequentialRows().
Если избавляться от пререндеренной BigString в пользу случайных текстовых строк, то наверно унифицированно для всего TKvWorkloadGenerator'а.
Но в этом pr это не совсем по теме.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Documentation build

Revision built successfully
Revision preview link

@ijon
ijon force-pushed the ydb_workload_kv_run-add-opsert_seq-workload-type branch from 2441546 to c4f0f93 Compare June 24, 2024 12:51
@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@ijon
ijon force-pushed the ydb_workload_kv_run-add-opsert_seq-workload-type branch from c4f0f93 to d6743c9 Compare June 24, 2024 13:06
@github-actions

github-actions Bot commented Jun 24, 2024 •

Copy link
Copy Markdown
Contributor

⚪ 2024-06-24 13:09:42 UTC Pre-commit check for f25e2cf has started.
⚪ 2024-06-24 13:12:18 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-06-24 13:13:27 UTC Build successful.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Documentation build

Revision built successfully
Revision preview link

@github-actions

github-actions Bot commented Jun 24, 2024 •

Copy link
Copy Markdown
Contributor

⚪ 2024-06-24 13:12:02 UTC Pre-commit check for f25e2cf has started.
⚪ 2024-06-24 13:14:38 UTC Build linux-x86_64-release-asan is running...
🟢 2024-06-24 13:15:53 UTC Build successful.
⚪ 2024-06-24 13:16:06 UTC Tests are running...
🔴 2024-06-24 15:20:40 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9499 8918 0 87 321 173

🟢 2024-06-24 15:21:38 UTC ydbd size 5.5 GiB changed* by +90.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 3d06832 merge: f25e2cf diff diff %
ydbd size 5 856 967 440 Bytes 5 857 060 344 Bytes +90.7 KiB +0.002%
ydbd stripped size 1 255 871 784 Bytes 1 255 895 272 Bytes +22.9 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions

github-actions Bot commented Jun 24, 2024 •

Copy link
Copy Markdown
Contributor

⚪ 2024-06-24 13:12:50 UTC Pre-commit check for f25e2cf has started.
⚪ 2024-06-24 13:15:15 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-06-24 13:16:19 UTC Build successful.
⚪ 2024-06-24 13:16:28 UTC Tests are running...
🔴 2024-06-24 14:46:37 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13601 12248 0 7 1338 8

🟢 2024-06-24 14:47:10 UTC ydbd size 8.3 GiB changed* by +64.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 3d06832 merge: f25e2cf diff diff %
ydbd size 8 964 770 096 Bytes 8 964 836 592 Bytes +64.9 KiB +0.001%
ydbd stripped size 488 532 744 Bytes 488 537 736 Bytes +4.9 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions

Copy link
Copy Markdown
Contributor

✅ Documentation build

Revision built successfully
Revision preview link

@github-actions

github-actions Bot commented Jun 27, 2024 •

Copy link
Copy Markdown
Contributor

⚪ 2024-06-27 14:51:46 UTC Pre-commit check for baf1156 has started.
⚪ 2024-06-27 14:54:14 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-06-27 15:31:36 UTC Build successful.
⚪ 2024-06-27 15:31:57 UTC Tests are running...
🔴 2024-06-27 16:58:38 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12857 11545 0 4 1291 17

🟢 2024-06-27 16:59:13 UTC ydbd size 8.4 GiB changed* by +64.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: bb4992d merge: baf1156 diff diff %
ydbd size 9 021 404 864 Bytes 9 021 471 360 Bytes +64.9 KiB +0.001%
ydbd stripped size 492 787 400 Bytes 492 792 392 Bytes +4.9 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions

github-actions Bot commented Jun 27, 2024 •

Copy link
Copy Markdown
Contributor

⚪ 2024-06-27 14:58:35 UTC Pre-commit check for baf1156 has started.
⚪ 2024-06-27 15:01:02 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-06-27 15:37:38 UTC Build successful.

@github-actions

github-actions Bot commented Jun 27, 2024 •

Copy link
Copy Markdown
Contributor

⚪ 2024-06-27 15:05:59 UTC Pre-commit check for baf1156 has started.
⚪ 2024-06-27 15:08:26 UTC Build linux-x86_64-release-asan is running...
🟢 2024-06-27 15:31:10 UTC Build successful.
⚪ 2024-06-27 15:31:27 UTC Tests are running...
🔴 2024-06-27 17:36:25 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9515 8926 0 84 327 178

🟢 2024-06-27 17:37:15 UTC ydbd size 5.5 GiB changed* by +86.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: f400427 merge: baf1156 diff diff %
ydbd size 5 916 265 904 Bytes 5 916 354 712 Bytes +86.7 KiB +0.002%
ydbd stripped size 1 271 071 528 Bytes 1 271 090 920 Bytes +18.9 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@ijon
ijon requested review from blinkov and kungasc June 28, 2024 09:54

This load test runs several types of load:
* [upsert](#upsert-kv): Using the UPSERT operation, inserts rows that are tuples (key1, key2, ... keyK, value1, value2, ... valueN) into the table created previously with the init command, the K and N numbers are specified in the settings.
* [upsert-seq](#upsert-seq-kv): variant of upsert load, but inserts rows by sequential keys.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* [upsert-seq](#upsert-seq-kv): variant of upsert load, but inserts rows by sequential keys.
* [upsert-seq](#upsert-seq-kv): a variant of the upsert load that inserts rows with sequential keys.

Comment on lines +166 to +170
## Upsert-seq load {#upsert-seq-kv}

Upsert load variant for inserting rows by sequential keys instead of random.

Sequential upserts generate a concentrated load on a small number of table partitions (at each point in time) instead of creating a uniform load on the entire table.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## Upsert-seq load {#upsert-seq-kv}
Upsert load variant for inserting rows by sequential keys instead of random.
Sequential upserts generate a concentrated load on a small number of table partitions (at each point in time) instead of creating a uniform load on the entire table.
## Upsert-seq load {#upsert-seq-kv}
An upsert load variant that inserts rows using sequential keys instead of random keys.
Sequential upserts concentrate the load on a small number of table partitions at any given time rather than distributing it uniformly across the entire table.

Comment on lines +178 to +179
* `global workload options`: [The global options for all load types](#global-workload-options).
* `specific workload options`: [Options of a specific load type](#upsert-seq-options).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* `global workload options`: [The global options for all load types](#global-workload-options).
* `specific workload options`: [Options of a specific load type](#upsert-seq-options).
* `global workload options`: [global options for all load types](#global-workload-options).
* `specific workload options`: [options for this specific load type](#upsert-seq-options).

Comment on lines +181 to +186
### Parameters for upsert {#upsert-seq-options}

Parameter name | Parameter description
---|---
`--len` | The size of the rows in bytes inserted into the table as values. Default: 8.
`--start-first-key` | The starting value for the primary key's first column. Default: 0.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Parameters for upsert {#upsert-seq-options}
Parameter name | Parameter description
---|---
`--len` | The size of the rows in bytes inserted into the table as values. Default: 8.
`--start-first-key` | The starting value for the primary key's first column. Default: 0.
### Parameters for upsert-seq {#upsert-seq-options}
Parameter name | Parameter description
---|---
`--len` | The size of the rows, in bytes, inserted into the table as values. Default: 8.
`--start-first-key` | The starting value for the primary key's first column. Default: 0.

Имя параметра | Описание параметра
---|---
`--len` | Размер строк в байтах, которые вставляются в таблицу, как values. Значение по умолчанию: 8.
`--start-first-key` | Начальное значение первого компонента primary ключа. Значение по умолчанию: 0.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`--start-first-key` | Начальное значение первого компонента primary ключа. Значение по умолчанию: 0.
`--start-first-key` | Начальное значение первого компонента первичного ключа. Значение по умолчанию: 0.

@sintjuri sintjuri added the documentation Improvements or additions to documentation label Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation not-for-changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants