Ease of use and Base64 update

.claude/settings.local.json

@@ -0,0 +1,10 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(bundle info *)",
+      "Bash(gem which *)",
+      "Read(//c/Users/travis.wiese/.gem/**)",
+      "Bash(ls /c/Ruby*/lib/ruby/gems/*/gems)"
+    ]
+  }
+}

.gitignore

@@ -3,3 +3,10 @@ config/*.yml
 !config/servername_environment_import_config.yml
 !config/servername_environment_export_config.yml
 !config/servername_environment_export_specific_config.yml
+exports/*
+Local_Gems/*
+Tools/*
+GrabNGoBundle/*
+*.log
+bundle/*
+.bundle/*

Gemfile

@@ -1,3 +1,11 @@
 source 'https://rubygems.org'
 
-gem 'kinetic_sdk', '5.0.22'
+gem 'kinetic_sdk', '5.0.31'
+gem 'find', '0.1.1'
+gem 'logger', '1.4.2'
+gem 'json', '2.3.0'
+gem 'optparse', '0.6.0'
+gem 'rexml', '3.2.3'
+gem 'io-console', '0.5.6'
+gem 'base64', '0.2.0'
+gem 'concurrent-ruby', '1.3.6'

Gemfile.lock

@@ -1,27 +1,40 @@
 GEM
   remote: https://rubygems.org/
   specs:
-    kinetic_sdk (5.0.19)
+    base64 (0.2.0)
+    concurrent-ruby (1.3.6)
+    find (0.1.1)
+    io-console (0.5.6)
+    json (2.3.0)
+    kinetic_sdk (5.0.31)
       mime-types (>= 3.3.1)
       multipart-post (= 2.0.0)
-      parallel (= 1.12.1)
-      ruby-progressbar (= 1.9.0)
       slugify (= 1.0.7)
-    mime-types (3.3.1)
-      mime-types-data (~> 3.2015)
-    mime-types-data (3.2021.0704)
+    logger (1.4.2)
+    mime-types (3.7.0)
+      logger
+      mime-types-data (~> 3.2025, >= 3.2025.0507)
+    mime-types-data (3.2026.0203)
     multipart-post (2.0.0)
-    parallel (1.12.1)
-    ruby-progressbar (1.9.0)
+    optparse (0.6.0)
+    rexml (3.2.3)
     slugify (1.0.7)
 
 PLATFORMS
-  java
-  ruby
+  x64-mingw-ucrt
   x64-mingw32
+  x86_64-linux
 
 DEPENDENCIES
-  kinetic_sdk (= 5.0.19)
+  base64 (= 0.2.0)
+  concurrent-ruby (= 1.3.6)
+  find (= 0.1.1)
+  io-console (= 0.5.6)
+  json (= 2.3.0)
+  kinetic_sdk (= 5.0.31)
+  logger (= 1.4.2)
+  optparse (= 0.6.0)
+  rexml (= 3.2.3)
 
 BUNDLED WITH
-   2.2.5
+   2.2.33

IMPORT-RB-NOTES.md

@@ -0,0 +1,124 @@
+# import.rb — field notes & enhancement candidates
+
+Observations from a real cross-space deployment (hanford-dev 6.1.7 → servicecatalog-dev 6.0.7).
+Intended to help whoever next enhances `import.rb`. **Line numbers are from the version in this
+repo as of mid-2026 and will drift — confirm against the actual file.** Several items below are
+marked ⚠ UNCONFIRMED: observed behavior that may be a version-specific quirk rather than intended
+design. Verify before relying on them.
+
+---
+
+## Status update (2026-06-24)
+
+- **Item 2 (definition-level methods never called) — ADDRESSED.** `import_space()` now calls
+  `update_space_attributes`, `update_user_attributes`, `update_user_profile_attributes`,
+  `update_team_attributes`, `update_security_policy`, `import_space_teams`,
+  `update_datastore_attributes`, and (per-kapp) `import_kapp_category_definitions`, plus
+  `import_datastore_data` for datastore submissions. These methods also previously referenced the
+  local `vars` without taking it as a parameter (a latent `NameError`); they now take `vars`.
+  Users and routine response templates still have no import path (manual).
+- **Skip-if-unchanged for forms — ADDRESSED.** The old `updatedAt` comparison was dead (exports
+  carry no `updatedAt`); replaced with a content deep-equality comparison, on by default.
+- **Item 1 (routine layout) — STILL OPEN.** Routines under `task/sources/*/trees/` vs
+  `task/routines/` remains version/export-format dependent; a layout-agnostic routine import
+  (route by the XML `<type>` element) is still a candidate enhancement.
+
+---
+
+## 1. ⚠ UNCONFIRMED — Global Routines import only from `task/routines/`
+
+`import.rb` ingests Global Routines via `import_routines_threaded` (≈ line 384) and identifies them
+for delete-reconciliation from `Dir["#{task_path}/routines/*.xml"]` (≈ line 462). Source-group
+**Trees** are read separately from `Dir["#{task_path}/sources/*/trees/*.xml"]` (≈ line 473).
+
+**Observed problem:** a bundle that stored Global Routines under `task/sources/<source>/trees/`
+(the layout one space's `export.rb` produced) had those routines **silently not import** — forms
+landed, but new routines never appeared and existing ones were never overwritten. Moving them to
+`task/routines/` (or importing them another way) fixed it.
+
+**Why this is marked UNCONFIRMED / possible version issue:**
+- The standard Kinetic `export.rb` appears to write Global Routines to `task/routines/` with
+  lowercase-hyphenated filenames. One source space instead exported them under
+  `task/sources/kinetic-task/trees/` with Title-Case filenames. It's unclear whether that
+  alternate layout is an older/newer export-format variant, a custom task source, or an anomaly.
+- So the "bug" may be on the **export** side (non-standard placement) rather than import. Either
+  way, the failure mode is real: routine files outside `task/routines/` don't import.
+
+**Enhancement candidate:** make routine import layout-agnostic — scan all task `*.xml`, read the
+`<type>` element, and route any `<type>Global Routine</type>` through the routine importer
+regardless of folder. That would make import.rb tolerant of either export layout.
+
+**Reliable workaround (no import.rb change):** push trees + routines via the Task component API,
+multipart `POST /app/components/task/app/api/v2/trees?force=true`, form field name `content`. The
+endpoint reads `<type>`/`<definitionId>` from the XML, so it imports a routine or a tree correctly
+regardless of source folder. (A working Ruby/stdlib importer doing exactly this was built for the
+delegation deployment — see the HMIS_IMPORT bundle's `servicecatalog_import.rb`.)
+
+---
+
+## 2. Definition-level methods are defined but NOT called from `import_space()`
+
+These exist in import.rb but are never invoked by the main `import_space()` path, so the
+corresponding artifacts do **not** import on a normal run:
+- space attribute definitions
+- team attribute definitions
+- space-level security policy definitions
+- teams (`import_space_teams`)
+
+And these have **no import path at all**:
+- users (service accounts etc.)
+- source-group routine **response templates** (the `.response.erb` — console-paste only; no API)
+
+**Enhancement candidate:** call the existing `update_space_attributes` / `update_team_attributes`
+/ `update_security_policy` / `import_space_teams` methods from `import_space()` (a known ~5-line
+patch), guarded by a config flag. Users + response templates still need a separate path or remain
+documented manual steps.
+
+Until then, a deployment needs a manual/scripted companion for those (we used a direct-API script;
+the API endpoints are `/spaceAttributeDefinitions`, `/teamAttributeDefinitions`,
+`/securityPolicyDefinitions`, `/teams`, `/users`).
+
+---
+
+## 3. Datastore forms import via the normal kapp loop
+
+The modern `datastore` kapp is a normal kapp; its forms import through the same `import_forms`
+path as `services` forms. The legacy `import_datastore_forms` / `add_datastore_form` methods are
+for the deprecated datastore type and don't apply. Don't treat datastore forms as a special case.
+
+---
+
+## 4. `options.delete` reconciliation
+
+Import reconciles destination vs source and can delete trees/routines/forms not present in the
+source data. For an additive cross-space deployment set `options.delete: false` (or equivalent) so
+destination-only artifacts aren't culled. The delete-identification for routines reads
+`task/routines/*.xml` — see item 1; if routines are mislaid the reconciler also can't see them.
+
+---
+
+## 5. Folder the script actually reads
+
+`import.rb` reads `core/` and `task/` from `exports/<folderName>/`, where `<folderName>` derives
+from config `old_space_slug` → `space_slug` → `space_name`. A bundle in a differently-named folder
+is ignored — symptom is an import that finishes in seconds with no `Adding new form` lines. Confirm
+the config folder name matches where the files actually are before running.
+
+---
+
+## 6. Cross-version form keys (6.1.x → 6.0.x)
+
+6.1.x adds form-element keys `defaultDataSource`, `choicesDataSource`, and
+`renderAttributes.width` that 6.0.x **rejects on import**. API `export=true` omits them;
+`export.rb` includes them. Strip them before importing a 6.1 export into 6.0. Task-tree XML is
+otherwise version-identical between 6.0 and 6.1.
+
+---
+
+## Suggested verification before enhancing import.rb elsewhere
+
+1. Export a space that has Global Routines; note whether they land in `task/routines/` or under
+   `task/sources/*/trees/`. That tells you if item 1 affects your export-format version.
+2. Grep `import.rb` for which `import_*` / `update_*` methods are actually called inside
+   `import_space()` vs merely defined — item 2 may differ by version.
+3. Dry-run against a sandbox space with `delete:false` and diff before/after.

README.md

@@ -88,10 +88,14 @@ There is now a repository to help track changes and maintian the Kinetic Core an
 There is now an inital export of whatever was determined to be the baseline export. There are a couple of optons on how to promote the changes to another server.
 
 1. Use import.rb
-   This script migrates everthing from an export to another environment. Everything will be migrated even if the source and destination are the same.  In the case of a Form it will be updated even if it is the same and the "Updated At" date and time will be modified for all forms.  
-
+   This script migrates everything from an export to another environment. Most artifacts are migrated even if the source and destination are the same.
+
+   **Forms are compared by content and skipped when unchanged.** Each form in the export is compared (deep-equality, ignoring server-managed keys such as `updatedAt`/`createdAt`) against the destination form fetched in the same export shape; if they are identical the form is left untouched (no needless `Updated At` churn). Any real difference triggers an update — when in doubt, the form is updated. (Cross-version migrations, e.g. 6.1→6.0, will differ and therefore always update, which is safe.)
+
    This is the sure way to update an environment to get it into sync with another.
    This script may be used at any point in time to migrate the current state contained in the export to another server.
+
+   By default the script runs every phase. You can run a subset interactively, or non-interactively via config — see **Selective Import/Export** below.
 
 2. Use import_git_diff.rb
    This script will import only the newest changes. Only the changes since the last git commit to the repository will be migrated.  This script works best when it is part of a process that is used consistently. Any one off changes made to the destination server outside of this script may get differences out of sync.  The differences are also determined from one export to another and **not** between the export definitions and the destination server.
@@ -141,7 +145,7 @@ options:
   - datastore: # true or false: true for datastore forms false for regular form data exports
     formSlug: # Slug of the datastore or form to have submissions exported
 
-  REMOVE_DATA_PROPERTIES: # The listed properties will be removed the form definition
+  REMOVE_DATA_PROPERTIES: # The listed properties will be removed from each submission
   - createdAt
   - createdBy
   - updatedAt
@@ -160,10 +164,24 @@ Below is a listing of the config elements in the **Import** script and how they
 
 options:
   delete: true
+  # Optional. Restrict which phases run. Omit (or use [0]) for ALL.
+  # Values are the 1-based numbers shown in the interactive menu, or category keys.
+  # When present, the interactive prompt is skipped (useful for unattended/CI runs).
+  categories: [10, 16]   # e.g. forms + task trees only
+
+## Selective Import/Export (Category Selection)
+Both `export.rb` and `import.rb` let you process a subset of artifact categories.
+
+- **Interactive:** when run without an `options.categories` value in the config, each script prints a numbered menu of categories. Enter a comma-separated list (e.g. `1,2,5,8`). Enter `0` (or just press Enter) to process **all** categories.
+- **Unattended:** set `options.categories` in the config (an array of the menu numbers and/or category keys). When present, the prompt is skipped.
+
+`0`/empty/omitted = all, and is the safe default. Subset selections are a power-user feature: you are responsible for prerequisites (for example, importing `forms` assumes the target kapp already exists on the destination). For finer-than-Core export granularity (specific forms, teams, or workflows only), use `export-specific.rb`, which is fully config-driven per artifact.
 
 ## Migrated Components
 Below is a list of components and what is migrated as part of the export and import process.  Not included in a migration is Space slug and name, Bundle configuration, Agent URL, Task URL, Oauth, Security, Bridges, Sources (some), and individual Handler configuration.
 
+> **Note:** `import.rb` now also imports space / user / user-profile / team **attribute definitions**, space **security policy definitions**, **teams** (with their attributes), kapp **category attribute definitions**, and **datastore submission data**. These were previously defined in the script but never invoked, so they did not migrate on a normal run. Users and source-group routine response templates still have no import path and remain manual steps. Destructive deletes for teams, forms, and datastore forms remain disabled by design.
+
 ### Space
 
 #### BUILD (Space)

config/servername_environment_export_config.yml

@@ -7,6 +7,9 @@ core:
   service_user_username: <USER_NAME>
   service_user_password: <PASSWORD>
 options:
+  # Optional. Restrict which categories are exported (skips the interactive menu when present).
+  # Values are the 1-based menu numbers and/or category keys. Omit or use [0] for ALL.
+  # categories: [1, 3]   # e.g. Core space + task trees only
   SUBMISSIONS_TO_EXPORT:
   - datastore: #{true:false}
     formSlug: #FORM_SLUG

config/servername_environment_import_config.yml

@@ -4,10 +4,14 @@ core:
   server_url: https://web-server.com
   space_slug: <SPACE_SLUG>
   space_name: <SPACE_NAME>
+  old_space_slug: 
   service_user_username: <USER_NAME>
   service_user_password: <PASSWORD>
 options:
   delete: false
+  # Optional. Restrict which phases run (skips the interactive menu when present).
+  # Values are the 1-based menu numbers and/or category keys. Omit or use [0] for ALL.
+  # categories: [10, 16]   # e.g. forms + task trees only
 task:
   # server_url: https://<SPACE>.kinops.io/app/components/task   OR https://<SERVER_NAME>.com/kinetic-task
   server_url: https://web-server.com