Translations from-spreadsheet: default locale and version bump

README.md

@@ -258,6 +258,8 @@ Update objects from spreadsheet. Update any type of DHIS2 metadata object using
 $ yarn start translations from-spreadsheet \
   --url='http://USER:PASSWORD@HOST:PORT' \
   --save-payload=payload.json \
+  --default-locale=en \
+  --bump-versions \
   --post \
   translations.xlsx
 ```
@@ -268,6 +270,12 @@ Expected format of `xlsx` file:
 -   A Column named `type`/`kind` specifies the DHIS2 entity type (singular). Example: `dataElement`, `dataSet`.
 -   Columns named `id`/`name`/`code` will be used to match the existing object in the database. No need to specify all of them.
 -   Translation columns should have the format: `field:localeName`. A DHIS2 Locale with that name should exist in the database. Example: `formName:French`.
+-   Field columns (a translatable field with no locale, e.g. `formName`) update the object field itself (`name` is written only when the row also has an `id`/`code`; otherwise it is just the lookup key).
+
+Notes:
+
+-   `--default-locale`: locale code of the default (DB) language, `en` for example. Its columns update the object field itself (`formName: English` writes `formName`) on top of creating the translation, so both stay in sync. The match uses only the language part, so `en` also matches a locale `en_GB`.
+-   `--bump-versions`: increment the `version` of every data set and program using an updated data element. Apps typically cache the metadata, so without this the new translations keep showing as the old ones.
 
 ### To spreadsheet
 
@@ -927,14 +935,16 @@ A config file with the access info of the server and the message webhook details
 ```
 
 This reports stores data into the `d2-tools.user-roles-authorities-monitoring` datastore.
-If some change is detected for a UserRole Authorities a message is generated with three categories: 
-- New user roles detected with its authorities
-- Deleted user roles detected with its authorities
-- Updated user roles detected with a list of added and removed authorities
+If some change is detected for a UserRole Authorities a message is generated with three categories:
+
+-   New user roles detected with its authorities
+-   Deleted user roles detected with its authorities
+-   Updated user roles detected with a list of added and removed authorities
 
 If a authority assigned to a UserRole is deprecated (legacy authority, missing or removed app authority, etc) its name will be set to "DEPRECATED_AUTHORITY".
 
 Example of the message:
+
 ```
 New user roles detected:
 - NEW USER ROLE (Id: XXXXXXXXXXX) with authorities:

openspec/specs/import-translations-from-spreadsheet/spec.md

@@ -0,0 +1,153 @@
+# import-translations-from-spreadsheet Specification
+
+## Purpose
+
+Bulk-update the translations of any DHIS2 metadata model from an xlsx spreadsheet, so translators
+work in a familiar tool and their output is applied without manual editing in the DHIS2 UI. The
+counterpart command `translations to-spreadsheet` generates a re-importable file of this format.
+
+## Requirements
+
+### Requirement: Read translations from an xlsx spreadsheet
+
+The `translations from-spreadsheet` command SHALL take an input xlsx path and parse every sheet in
+it, except sheets whose name starts with `!`. Each row describes one metadata object.
+
+A row SHALL declare its model in a `type`/`kind` column, holding the singular model name (e.g.
+`dataElement`), which the command pluralizes to address the DHIS2 API. Translation columns SHALL
+have the format `<field>: <LocaleName>` (e.g. `formName: French`); one column MAY list several
+comma-separated fields sharing a value. Column matching is case-insensitive for the metadata
+columns, and locale names are matched ignoring any ` (...)` suffix, so `Spanish` matches a
+`Spanish (Spain)` DB locale.
+
+Rows without a model, without any identifier, or with an empty cell SHALL be skipped with a
+warning rather than aborting the run: a partially filled translation sheet is the normal case.
+
+#### Scenario: Locale column not defined in the instance
+
+- **WHEN** a column names a locale that does not exist in `/api/locales/dbLocales`
+- **THEN** the column is skipped and a warning naming the locale is logged once for the sheet
+
+### Requirement: Match objects by id, code or name
+
+The command SHALL locate the object each row refers to using the `id`/`uid`, `code` and `name`
+columns, in that order of precedence, within the row's model. A row need only provide one of them.
+Name matching SHALL be case-insensitive. A row matching no object SHALL be reported and skipped.
+
+#### Scenario: Unmatched row does not abort the run
+
+- **WHEN** a row's identifier matches no object of its model
+- **THEN** a warning naming the model and identifier is logged, the row is skipped, and the
+  remaining rows are still imported
+
+### Requirement: Merge translations, never drop existing ones
+
+The command SHALL merge the spreadsheet translations into the object's existing ones, keyed by
+(locale, property): a spreadsheet value overwrites the same locale/property, and any translation
+absent from the spreadsheet SHALL be preserved. Only objects whose resulting payload differs from
+the current one SHALL be posted.
+
+#### Scenario: Untouched locale survives the import
+
+- **WHEN** an object has a French `FORM_NAME` translation and the sheet only carries a Spanish one
+- **THEN** the posted object keeps the French translation and gains the Spanish one
+
+### Requirement: Bare field columns update the object field
+
+A column headed exactly as a translatable field of the row's model, with no locale (`formName`,
+`shortName`), SHALL write that field on the object. This is the layout `to-spreadsheet` generates
+(source column followed by its locale columns), so an unedited export round-trips as a no-op and
+edited source cells are posted without any option. Columns matching no translatable field of the
+model SHALL be ignored with a warning once per sheet. Empty cells SHALL not blank the field.
+
+`name` doubles as a lookup key: it SHALL be written only when the row also carries an `id` or
+`code`, otherwise a case-insensitive name match would rename the object to the sheet's spelling.
+The run SHALL log, per model, how many rows update which fields, so field changes are visible in
+the dry run alongside the translations.
+
+#### Scenario: Edited source column is posted
+
+- **WHEN** a row has `id`, `name` and `name: French` columns and the `name` cell differs from the
+  object's current name
+- **THEN** the posted object has the new `name` and the French `NAME` translation
+
+#### Scenario: Name-only rows are not renamed
+
+- **WHEN** a row identifies the object only by `name`
+- **THEN** the name is used to find the object and is not written back as a field
+
+### Requirement: Default-locale columns also update the object field
+
+The command SHALL provide a `--default-locale` option taking the locale code of the instance's
+default (DB) language. Columns of that locale SHALL update the object's own field (`formName:
+English` writes `formName`) **in addition to** writing the translation, so field and translation
+stay in sync and the sheet round-trips through `to-spreadsheet`, which reads the locale columns
+from the translations.
+
+The field name SHALL be derived from the column prefix (`Left side description` →
+`leftSideDescription`). Locale comparison SHALL use only the language part, so `en` matches a
+`en_GB` DB locale. Without the option, no object field is written.
+
+#### Scenario: Default-locale column writes both
+
+- **WHEN** the sheet has a `formName: English` column, English is the `en_GB` DB locale and the
+  command runs with `--default-locale=en`
+- **THEN** the posted object has both `formName` set and an `en_GB` `FORM_NAME` translation
+
+### Requirement: Warn about fields that would silently do nothing or break the import
+
+The command SHALL check every column's field against the translatable properties the instance
+declares in `/api/schemas`, and SHALL warn once per model/field (not per row) when the field is
+not translatable there — DHIS2 ignores unknown properties, so such a column would post
+successfully while changing nothing.
+
+It SHALL also warn when a column writes a unique-constrained field (`name`, `shortName`),
+because a duplicated value makes the whole metadata payload fail to validate.
+
+Both cases are warnings, not errors: the operator reviews them in the dry run before posting.
+
+#### Scenario: Typo in a column field name
+
+- **WHEN** a column is headed `fromName: English` and the instance declares `formName` (not
+  `fromName`) as translatable for that model
+- **THEN** a warning naming the model and field is logged once, and the run continues
+
+### Requirement: Refresh the Capture apps cache with --bump-versions
+
+The Capture apps cache the metadata of each data set and program, and refresh it only when that
+object's `version` changes, so an imported translation stays invisible in the app until the owning
+objects are bumped.
+
+The command SHALL provide a `--bump-versions` option that increments the `version` of every data
+set and program referencing a data element whose translations changed. The scope is deliberately
+data elements only: translating an option set, a tracked entity attribute or a program stage does
+not bump anything, even though those are cached too.
+
+The bump SHALL be opt-in, because it writes objects the operator never listed in the spreadsheet.
+It SHALL be skipped entirely, with no lookup request, when no object changed.
+
+#### Scenario: Only the owners of a changed data element are bumped
+
+- **WHEN** a data element's translations changed and the command runs with `--bump-versions --post`
+- **THEN** the data sets and programs referencing that data element are saved with their version
+  incremented by one (starting at 1 when unset), and the ones referencing only other data elements
+  are left untouched
+
+#### Scenario: Dry run reports the bumps without writing
+
+- **WHEN** the command runs with `--bump-versions` but without `--post`
+- **THEN** the intended bumps are logged and no data set or program is written
+
+### Requirement: Dry-run by default, apply with --post
+
+The command SHALL only persist changes when `--post` is given; otherwise it SHALL validate the
+payload against the instance (`importMode=VALIDATE`) and report what would change. The
+`--save-payload` option SHALL write the computed metadata payload to a JSON file in either mode,
+so it can be reviewed or replayed. The version bumps of `--bump-versions` are posted separately and
+are therefore not part of that payload.
+
+#### Scenario: Dry run reports without writing
+
+- **WHEN** the command runs without `--post`
+- **THEN** no metadata is modified, the import stats are reported, and the operator is told to add
+  `--post` to persist

src/data/DataSetsD2Repository.ts

@@ -109,6 +109,7 @@ const fields = {
     id: true,
     name: true,
     code: true,
+    version: true,
     categoryCombo: { id: true, categoryOptionCombos: { id: true, name: true } },
     dataSetElements: {
         dataElement: {

src/data/ExportTranslationsSpreadsheetRepository.ts

@@ -77,7 +77,9 @@ export class ExportTranslationsSpreadsheetRepository implements ExportTranslatio
 
         _.forEach(files, (content, name) => {
             if (!/^xl\/worksheets\/sheet\d+\.xml$/.test(name)) return;
-            const xml = decoder.decode(content).replace(/<sheetView ([^>]*?)\/>/, `<sheetView $1>${pane}</sheetView>`);
+            const xml = decoder
+                .decode(content)
+                .replace(/<sheetView ([^>]*?)\/>/, `<sheetView $1>${pane}</sheetView>`);
             files[name] = encoder.encode(xml);
         });
 
@@ -150,11 +152,13 @@ export class ExportTranslationsSpreadsheetRepository implements ExportTranslatio
             const palette = fieldPalette[index % fieldPalette.length];
             if (!palette) return [];
             const base: ColumnStyle = { kind: "base", headerColor: palette.header, bodyColor: palette.base };
-            const locales = sheet.locales.map((): ColumnStyle => ({
-                kind: "locale",
-                headerColor: palette.header,
-                bodyColor: palette.locale,
-            }));
+            const locales = sheet.locales.map(
+                (): ColumnStyle => ({
+                    kind: "locale",
+                    headerColor: palette.header,
+                    bodyColor: palette.locale,
+                })
+            );
             return [base, ...locales];
         });
 

src/data/ImportTranslationsRepositorySpreadsheetRepository.ts

@@ -1,6 +1,6 @@
 import _ from "lodash";
 import { Async } from "domain/entities/Async";
-import { FieldTranslation, FieldTranslations } from "domain/entities/FieldTranslations";
+import { FieldTranslation, FieldTranslations, FieldValues } from "domain/entities/FieldTranslations";
 import {
     ImportTranslationsRepository,
     GetFieldTranslationsOptions,
@@ -9,7 +9,8 @@ import { SpreadsheetXlsxDataSource } from "domain/repositories/SpreadsheetXlsxRe
 import log from "utils/log";
 import { Maybe } from "utils/ts-utils";
 import { getPluralModel } from "./dhis2-utils";
-import { translationFieldToProperty } from "domain/entities/Translation";
+import { Translation, translationFieldToProperty } from "domain/entities/Translation";
+import { getLocaleInfo, LocaleCode } from "domain/entities/Locale";
 
 const columnsMapping = {
     id: ["id", "uid"],
@@ -24,11 +25,15 @@ const columnsMapping = {
     - id/uid
     - code
     - name
-    - FIELD1: LANGUAGE_NAME1
+    - FIELD1 -> updates the object field itself (example: formName)
+    - FIELD1: LANGUAGE_NAME1 -> updates the translation
     - FIELD2: LANGUAGE_NAME2
     - ...
 
     Spreadsheets starting with '!' will be skipped.
+
+    Columns of the default locale (option --default-locale) also update the object field itself,
+    on top of its translation.
 */
 export class ImportTranslationsRepositorySpreadsheetRepository implements ImportTranslationsRepository {
     async get(options: GetFieldTranslationsOptions): Async<FieldTranslations> {
@@ -38,11 +43,14 @@ export class ImportTranslationsRepositorySpreadsheetRepository implements Import
         return _(spreadsheet.sheets)
             .reject(sheet => sheet.name.startsWith("!"))
             .flatMap((sheet): FieldTranslations => {
-                return _(sheet.rows)
+                const fieldTranslations = _(sheet.rows)
                     .map((row, rowIndex) => this.fromRow(row, rowIndex, `${sheet.name}:${rowIndex}`, options))
                     .compact()
                     .value();
+                log.info(`Sheet '${sheet.name}': ${fieldTranslations.length} rows parsed`);
+                return fieldTranslations;
             })
+
             .value();
     }
 
@@ -79,14 +87,14 @@ export class ImportTranslationsRepositorySpreadsheetRepository implements Import
             .filter(column => column.includes(":"))
             .value();
 
-        const translations = _(translationColumns)
+        const entries = _(translationColumns)
             .flatMap(translationColumn => {
                 const [fields = "", localeName] = translationColumn.split(":").map(s => s.trim());
 
                 return fields
                     .split(",")
                     .map(s => s.trim())
-                    .map(field => {
+                    .map((field): Maybe<ColumnEntry> => {
                         const locale = localeName ? localesByName[localeName] : undefined;
                         const text = row[translationColumn];
 
@@ -98,14 +106,54 @@ export class ImportTranslationsRepositorySpreadsheetRepository implements Import
                             return undefined;
                         } else {
                             const property = translationFieldToProperty(field);
-                            return { property: property, locale: locale.locale, value: text };
+                            const translation = { property: property, locale: locale.locale, value: text };
+                            // A default-locale column also updates the object field itself, so both
+                            // the field and its translation stay in sync.
+                            const isDefault = isDefaultLocale(locale.locale, options.defaultLocale);
+                            const fieldValue = isDefault ? { [_.camelCase(field)]: text } : undefined;
+                            return { translation, fieldValue };
                         }
                     });
             })
             .compact()
             .value();
 
-        return { model: getPluralModel(model), identifier: identifier, translations };
+        const translations = entries.map(entry => entry.translation);
+        const pluralModel = getPluralModel(model);
+        const bareFieldValues = this.getBareFieldValues(row, pluralModel, identifier, options);
+
+        // An explicit default-locale column wins over the bare column of the same field.
+        const fieldValues: FieldValues = Object.assign(
+            {},
+            bareFieldValues,
+            ...entries.map(entry => entry.fieldValue)
+        );
+
+        return { model: pluralModel, identifier: identifier, translations, fields: fieldValues };
+    }
+
+    /* A bare column (no locale) named exactly as a translatable field of the model writes that
+       field. The name column doubles as the lookup key: it's written only when the row also has
+       an id or code, otherwise a case-insensitive lookup match would rename the object. */
+    private getBareFieldValues(
+        row: Row,
+        model: string,
+        identifier: Identifier,
+        options: GetFieldTranslationsOptions
+    ): FieldValues {
+        const translatableFields = options.translatableFields[model] ?? [];
+        const identifierColumns = _.flatMap(columnsMapping);
+        const hasIdOrCode = Boolean(identifier.id || identifier.code);
+
+        return _(row)
+            .toPairs()
+            .reject(([column]) => column.includes(":"))
+            .reject(([column]) =>
+                column === "name" ? !hasIdOrCode : identifierColumns.includes(column.toLowerCase())
+            )
+            .filter(([column, text]) => (translatableFields.includes(column) ? Boolean(text) : false))
+            .fromPairs()
+            .value();
     }
 
     private getHeaderValue(row: Row, column: ColumnsMappingKey): Maybe<string> {
@@ -120,4 +168,18 @@ export class ImportTranslationsRepositorySpreadsheetRepository implements Import
 
 type Row = Record<string, string>;
 
+type Identifier = FieldTranslation["identifier"];
+
 type ColumnsMappingKey = keyof typeof columnsMapping;
+
+interface ColumnEntry {
+    translation: Translation;
+    fieldValue: Maybe<FieldValues>;
+}
+
+/* Locales may be LANGUAGE or LANGUAGE_COUNTRY: compare only the language part, so a
+   --default-locale=en also matches a column mapped to en_GB. */
+function isDefaultLocale(locale: LocaleCode, defaultLocale: Maybe<LocaleCode>): boolean {
+    if (!defaultLocale) return false;
+    return getLocaleInfo(locale).language === getLocaleInfo(defaultLocale).language;
+}