diff --git a/.claude/settings.local.json b/.claude/settings.local.json new file mode 100644 index 0000000..b54a861 --- /dev/null +++ b/.claude/settings.local.json @@ -0,0 +1,10 @@ +{ + "permissions": { + "allow": [ + "Bash(bundle info *)", + "Bash(gem which *)", + "Read(//c/Users/travis.wiese/.gem/**)", + "Bash(ls /c/Ruby*/lib/ruby/gems/*/gems)" + ] + } +} diff --git a/.gitignore b/.gitignore index 31286c8..5903b58 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,10 @@ config/*.yml !config/servername_environment_import_config.yml !config/servername_environment_export_config.yml !config/servername_environment_export_specific_config.yml +exports/* +Local_Gems/* +Tools/* +GrabNGoBundle/* +*.log +bundle/* +.bundle/* \ No newline at end of file diff --git a/Gemfile b/Gemfile index 6734f78..c0bf440 100644 --- a/Gemfile +++ b/Gemfile @@ -1,3 +1,11 @@ source 'https://rubygems.org' -gem 'kinetic_sdk', '5.0.22' +gem 'kinetic_sdk', '5.0.31' +gem 'find', '0.1.1' +gem 'logger', '1.4.2' +gem 'json', '2.3.0' +gem 'optparse', '0.6.0' +gem 'rexml', '3.2.3' +gem 'io-console', '0.5.6' +gem 'base64', '0.2.0' +gem 'concurrent-ruby', '1.3.6' \ No newline at end of file diff --git a/Gemfile.lock b/Gemfile.lock index 564e43b..4dfb522 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,27 +1,40 @@ GEM remote: https://rubygems.org/ specs: - kinetic_sdk (5.0.19) + base64 (0.2.0) + concurrent-ruby (1.3.6) + find (0.1.1) + io-console (0.5.6) + json (2.3.0) + kinetic_sdk (5.0.31) mime-types (>= 3.3.1) multipart-post (= 2.0.0) - parallel (= 1.12.1) - ruby-progressbar (= 1.9.0) slugify (= 1.0.7) - mime-types (3.3.1) - mime-types-data (~> 3.2015) - mime-types-data (3.2021.0704) + logger (1.4.2) + mime-types (3.7.0) + logger + mime-types-data (~> 3.2025, >= 3.2025.0507) + mime-types-data (3.2026.0203) multipart-post (2.0.0) - parallel (1.12.1) - ruby-progressbar (1.9.0) + optparse (0.6.0) + rexml (3.2.3) slugify (1.0.7) PLATFORMS - java - ruby + x64-mingw-ucrt x64-mingw32 + x86_64-linux DEPENDENCIES - kinetic_sdk (= 5.0.19) + base64 (= 0.2.0) + concurrent-ruby (= 1.3.6) + find (= 0.1.1) + io-console (= 0.5.6) + json (= 2.3.0) + kinetic_sdk (= 5.0.31) + logger (= 1.4.2) + optparse (= 0.6.0) + rexml (= 3.2.3) BUNDLED WITH - 2.2.5 + 2.2.33 diff --git a/IMPORT-RB-NOTES.md b/IMPORT-RB-NOTES.md new file mode 100644 index 0000000..3e44a87 --- /dev/null +++ b/IMPORT-RB-NOTES.md @@ -0,0 +1,124 @@ +# import.rb — field notes & enhancement candidates + +Observations from a real cross-space deployment (hanford-dev 6.1.7 → servicecatalog-dev 6.0.7). +Intended to help whoever next enhances `import.rb`. **Line numbers are from the version in this +repo as of mid-2026 and will drift — confirm against the actual file.** Several items below are +marked ⚠ UNCONFIRMED: observed behavior that may be a version-specific quirk rather than intended +design. Verify before relying on them. + +--- + +## Status update (2026-06-24) + +- **Item 2 (definition-level methods never called) — ADDRESSED.** `import_space()` now calls + `update_space_attributes`, `update_user_attributes`, `update_user_profile_attributes`, + `update_team_attributes`, `update_security_policy`, `import_space_teams`, + `update_datastore_attributes`, and (per-kapp) `import_kapp_category_definitions`, plus + `import_datastore_data` for datastore submissions. These methods also previously referenced the + local `vars` without taking it as a parameter (a latent `NameError`); they now take `vars`. + Users and routine response templates still have no import path (manual). +- **Skip-if-unchanged for forms — ADDRESSED.** The old `updatedAt` comparison was dead (exports + carry no `updatedAt`); replaced with a content deep-equality comparison, on by default. +- **Item 1 (routine layout) — STILL OPEN.** Routines under `task/sources/*/trees/` vs + `task/routines/` remains version/export-format dependent; a layout-agnostic routine import + (route by the XML `` element) is still a candidate enhancement. + +--- + +## 1. ⚠ UNCONFIRMED — Global Routines import only from `task/routines/` + +`import.rb` ingests Global Routines via `import_routines_threaded` (≈ line 384) and identifies them +for delete-reconciliation from `Dir["#{task_path}/routines/*.xml"]` (≈ line 462). Source-group +**Trees** are read separately from `Dir["#{task_path}/sources/*/trees/*.xml"]` (≈ line 473). + +**Observed problem:** a bundle that stored Global Routines under `task/sources//trees/` +(the layout one space's `export.rb` produced) had those routines **silently not import** — forms +landed, but new routines never appeared and existing ones were never overwritten. Moving them to +`task/routines/` (or importing them another way) fixed it. + +**Why this is marked UNCONFIRMED / possible version issue:** +- The standard Kinetic `export.rb` appears to write Global Routines to `task/routines/` with + lowercase-hyphenated filenames. One source space instead exported them under + `task/sources/kinetic-task/trees/` with Title-Case filenames. It's unclear whether that + alternate layout is an older/newer export-format variant, a custom task source, or an anomaly. +- So the "bug" may be on the **export** side (non-standard placement) rather than import. Either + way, the failure mode is real: routine files outside `task/routines/` don't import. + +**Enhancement candidate:** make routine import layout-agnostic — scan all task `*.xml`, read the +`` element, and route any `Global Routine` through the routine importer +regardless of folder. That would make import.rb tolerant of either export layout. + +**Reliable workaround (no import.rb change):** push trees + routines via the Task component API, +multipart `POST /app/components/task/app/api/v2/trees?force=true`, form field name `content`. The +endpoint reads ``/`` from the XML, so it imports a routine or a tree correctly +regardless of source folder. (A working Ruby/stdlib importer doing exactly this was built for the +delegation deployment — see the HMIS_IMPORT bundle's `servicecatalog_import.rb`.) + +--- + +## 2. Definition-level methods are defined but NOT called from `import_space()` + +These exist in import.rb but are never invoked by the main `import_space()` path, so the +corresponding artifacts do **not** import on a normal run: +- space attribute definitions +- team attribute definitions +- space-level security policy definitions +- teams (`import_space_teams`) + +And these have **no import path at all**: +- users (service accounts etc.) +- source-group routine **response templates** (the `.response.erb` — console-paste only; no API) + +**Enhancement candidate:** call the existing `update_space_attributes` / `update_team_attributes` +/ `update_security_policy` / `import_space_teams` methods from `import_space()` (a known ~5-line +patch), guarded by a config flag. Users + response templates still need a separate path or remain +documented manual steps. + +Until then, a deployment needs a manual/scripted companion for those (we used a direct-API script; +the API endpoints are `/spaceAttributeDefinitions`, `/teamAttributeDefinitions`, +`/securityPolicyDefinitions`, `/teams`, `/users`). + +--- + +## 3. Datastore forms import via the normal kapp loop + +The modern `datastore` kapp is a normal kapp; its forms import through the same `import_forms` +path as `services` forms. The legacy `import_datastore_forms` / `add_datastore_form` methods are +for the deprecated datastore type and don't apply. Don't treat datastore forms as a special case. + +--- + +## 4. `options.delete` reconciliation + +Import reconciles destination vs source and can delete trees/routines/forms not present in the +source data. For an additive cross-space deployment set `options.delete: false` (or equivalent) so +destination-only artifacts aren't culled. The delete-identification for routines reads +`task/routines/*.xml` — see item 1; if routines are mislaid the reconciler also can't see them. + +--- + +## 5. Folder the script actually reads + +`import.rb` reads `core/` and `task/` from `exports//`, where `` derives +from config `old_space_slug` → `space_slug` → `space_name`. A bundle in a differently-named folder +is ignored — symptom is an import that finishes in seconds with no `Adding new form` lines. Confirm +the config folder name matches where the files actually are before running. + +--- + +## 6. Cross-version form keys (6.1.x → 6.0.x) + +6.1.x adds form-element keys `defaultDataSource`, `choicesDataSource`, and +`renderAttributes.width` that 6.0.x **rejects on import**. API `export=true` omits them; +`export.rb` includes them. Strip them before importing a 6.1 export into 6.0. Task-tree XML is +otherwise version-identical between 6.0 and 6.1. + +--- + +## Suggested verification before enhancing import.rb elsewhere + +1. Export a space that has Global Routines; note whether they land in `task/routines/` or under + `task/sources/*/trees/`. That tells you if item 1 affects your export-format version. +2. Grep `import.rb` for which `import_*` / `update_*` methods are actually called inside + `import_space()` vs merely defined — item 2 may differ by version. +3. Dry-run against a sandbox space with `delete:false` and diff before/after. diff --git a/README.md b/README.md index 10c5e6b..82f2565 100644 --- a/README.md +++ b/README.md @@ -88,10 +88,14 @@ There is now a repository to help track changes and maintian the Kinetic Core an There is now an inital export of whatever was determined to be the baseline export. There are a couple of optons on how to promote the changes to another server. 1. Use import.rb - This script migrates everthing from an export to another environment. Everything will be migrated even if the source and destination are the same. In the case of a Form it will be updated even if it is the same and the "Updated At" date and time will be modified for all forms. - + This script migrates everything from an export to another environment. Most artifacts are migrated even if the source and destination are the same. + + **Forms are compared by content and skipped when unchanged.** Each form in the export is compared (deep-equality, ignoring server-managed keys such as `updatedAt`/`createdAt`) against the destination form fetched in the same export shape; if they are identical the form is left untouched (no needless `Updated At` churn). Any real difference triggers an update — when in doubt, the form is updated. (Cross-version migrations, e.g. 6.1→6.0, will differ and therefore always update, which is safe.) + This is the sure way to update an environment to get it into sync with another. This script may be used at any point in time to migrate the current state contained in the export to another server. + + By default the script runs every phase. You can run a subset interactively, or non-interactively via config — see **Selective Import/Export** below. 2. Use import_git_diff.rb This script will import only the newest changes. Only the changes since the last git commit to the repository will be migrated. This script works best when it is part of a process that is used consistently. Any one off changes made to the destination server outside of this script may get differences out of sync. The differences are also determined from one export to another and **not** between the export definitions and the destination server. @@ -141,7 +145,7 @@ options: - datastore: # true or false: true for datastore forms false for regular form data exports formSlug: # Slug of the datastore or form to have submissions exported - REMOVE_DATA_PROPERTIES: # The listed properties will be removed the form definition + REMOVE_DATA_PROPERTIES: # The listed properties will be removed from each submission - createdAt - createdBy - updatedAt @@ -160,10 +164,24 @@ Below is a listing of the config elements in the **Import** script and how they options: delete: true + # Optional. Restrict which phases run. Omit (or use [0]) for ALL. + # Values are the 1-based numbers shown in the interactive menu, or category keys. + # When present, the interactive prompt is skipped (useful for unattended/CI runs). + categories: [10, 16] # e.g. forms + task trees only + +## Selective Import/Export (Category Selection) +Both `export.rb` and `import.rb` let you process a subset of artifact categories. + +- **Interactive:** when run without an `options.categories` value in the config, each script prints a numbered menu of categories. Enter a comma-separated list (e.g. `1,2,5,8`). Enter `0` (or just press Enter) to process **all** categories. +- **Unattended:** set `options.categories` in the config (an array of the menu numbers and/or category keys). When present, the prompt is skipped. + +`0`/empty/omitted = all, and is the safe default. Subset selections are a power-user feature: you are responsible for prerequisites (for example, importing `forms` assumes the target kapp already exists on the destination). For finer-than-Core export granularity (specific forms, teams, or workflows only), use `export-specific.rb`, which is fully config-driven per artifact. ## Migrated Components Below is a list of components and what is migrated as part of the export and import process. Not included in a migration is Space slug and name, Bundle configuration, Agent URL, Task URL, Oauth, Security, Bridges, Sources (some), and individual Handler configuration. +> **Note:** `import.rb` now also imports space / user / user-profile / team **attribute definitions**, space **security policy definitions**, **teams** (with their attributes), kapp **category attribute definitions**, and **datastore submission data**. These were previously defined in the script but never invoked, so they did not migrate on a normal run. Users and source-group routine response templates still have no import path and remain manual steps. Destructive deletes for teams, forms, and datastore forms remain disabled by design. + ### Space #### BUILD (Space) diff --git a/config/servername_environment_export_config.yml b/config/servername_environment_export_config.yml index 410eb07..8f3978a 100644 --- a/config/servername_environment_export_config.yml +++ b/config/servername_environment_export_config.yml @@ -7,6 +7,9 @@ core: service_user_username: service_user_password: options: + # Optional. Restrict which categories are exported (skips the interactive menu when present). + # Values are the 1-based menu numbers and/or category keys. Omit or use [0] for ALL. + # categories: [1, 3] # e.g. Core space + task trees only SUBMISSIONS_TO_EXPORT: - datastore: #{true:false} formSlug: #FORM_SLUG diff --git a/config/servername_environment_import_config.yml b/config/servername_environment_import_config.yml index 056d78d..2903a51 100644 --- a/config/servername_environment_import_config.yml +++ b/config/servername_environment_import_config.yml @@ -4,10 +4,14 @@ core: server_url: https://web-server.com space_slug: space_name: + old_space_slug: service_user_username: service_user_password: options: delete: false + # Optional. Restrict which phases run (skips the interactive menu when present). + # Values are the 1-based menu numbers and/or category keys. Omit or use [0] for ALL. + # categories: [10, 16] # e.g. forms + task trees only task: # server_url: https://.kinops.io/app/components/task OR https://.com/kinetic-task server_url: https://web-server.com diff --git a/export.rb b/export.rb index 285cc78..853b075 100644 --- a/export.rb +++ b/export.rb @@ -1,3 +1,6 @@ +#TODO work +#Bluestone exported a workflow as "inactive" but it imported as "active" + # RUNNING THE SCRIPT: # ruby export.rb -c "<>" # ruby export.rb -c "config/foo-web-server.rb" @@ -19,6 +22,9 @@ SUBMISSIONS_TO_EXPORT: - datastore: true formSlug: + - datastore: false + kappSlug: + formSlug: REMOVE_DATA_PROPERTIES: - createdAt @@ -44,31 +50,42 @@ =end -require 'logger' +#require_relative './bundle/bundler/setup' +require 'logger' #For System Logging require 'json' -require 'optparse' +require 'optparse' #For argument parsing + +#require 'kinetic_sdk' # Note you may need to run "Gem install Kinetic_sdk" +require 'find' #For config list building +require 'io/console' #For password request +require 'base64' #For pwd encoding +require 'concurrent-ruby' +require 'set' #For category selection require 'kinetic_sdk' +# $LOAD_PATH.unshift('C:\Users\travis.wiese\Source\repos\kinetic-sdk-rb\lib') + -template_name = "platform-template" -logger = Logger.new(STDERR) -logger.level = Logger::INFO -logger.formatter = proc do |severity, datetime, progname, msg| +template_name = "platform-template" +$pwdFields = ["core","task"] +PWD = File.expand_path(File.dirname(__FILE__)) +$logger = Logger.new(STDERR) # "#{PWD}/output.log" +$logger.level = Logger::INFO +$logger.formatter = proc do |severity, datetime, progname, msg| date_format = datetime.utc.strftime("%Y-%m-%dT%H:%M:%S.%LZ") "[#{date_format}] #{severity}: #{msg}\n" end + # Determine the Present Working Directory pwd = File.expand_path(File.dirname(__FILE__)) -ARGV << '-h' if ARGV.empty? - +starting = Process.clock_gettime(Process::CLOCK_MONOTONIC) # The options specified on the command line will be collected in *options*. options = {} OptionParser.new do |opts| opts.banner = "Usage: example.rb [options]" - opts.on("-c", "--c CONFIG_FILE", "The Configuration file to use") do |config| options["CONFIG_FILE"] = config end @@ -81,10 +98,125 @@ end end.parse! + +#Configuration Selection +def config_selection(config_folder_path) + + #Ensure config folder exists + if !File.directory?(config_folder_path) + $logger.info "Config folder not found at #{config_folder_path}" + puts "Cannot find config folder!" + puts "Exiting..." + gets + exit + end + + # #Determine Config file to use + config_exts = ['.yaml','.yml'] + configArray = [] + $logger.info "Checking #{config_folder_path} for config files" + #Check config folder for yaml/yml files containing the word 'export' + begin + Find.find("#{config_folder_path}/") do |file| + configArray.append(File.basename(file)) if config_exts.include?(File.extname(file)) && (File.basename(file).include?('export')) + end + rescue error + #No config files found in config folder + $logger.error "Error finding default config file path!" + $logger.error "Error reported: #{error}" + puts "Cannot find config files in default path! (#{pwd})" + puts "Exiting script..." + gets + exit + end + $logger.info "Found config files" + + #Print config file options with number indicators to select + puts "Select your config file" + configArray.each_with_index do |cFile, index| + puts "#{index+1}) #{cFile}" + end + $logger.info "Select section" + begin + print "Selection (0 to repeat options): " + sel = gets.chomp.to_i + begin + if sel === 0 + configArray.each_with_index do |cFile, index| + puts "#{index+1}) #{cFile}" + end + next + end + configFile = configArray[sel-1] + $logger.info "Option #{sel} - #{configFile}" + break + rescue + $logger.info "Error selecting config file! Exiting..." + puts "Error selecting config file!" + puts "Exiting..." + gets + exit + end + end while true + return configFile +end + +# ------------------------------------------------------------------------------ +# Category selection (selective export) +# +# `categories` is an ordered Array of [key(Symbol), label(String)] pairs. +# Returns a Set of selected keys. Resolution order: +# 1. If config provides `options.categories` (Array of 1-based numbers and/or +# string keys), use it WITHOUT prompting - keeps unattended/CI runs non-blocking. +# 2. Otherwise print a numbered menu and read a comma-separated line. +# `0` (or empty input) selects ALL categories. +# ------------------------------------------------------------------------------ +def normalize_category_selection(raw, categories) + keys = categories.map { |(key, _label)| key } + tokens = (raw.is_a?(Array) ? raw : raw.to_s.split(",")).map { |t| t.to_s.strip }.reject(&:empty?) + return keys.to_set if tokens.empty? || tokens.include?("0") + selected = Set.new + tokens.each do |token| + if token =~ /\A\d+\z/ + idx = token.to_i - 1 + selected << keys[idx] if idx >= 0 && idx < keys.length + elsif keys.include?(token.to_sym) + selected << token.to_sym + else + $logger.warn "Ignoring unknown category selection: #{token}" + end + end + selected +end + +def select_categories(categories, vars) + configured = ((vars || {})["options"] || {})["categories"] + unless configured.nil? || (configured.is_a?(Array) && configured.empty?) + selected = normalize_category_selection(configured, categories) + $logger.info "Categories from config (options.categories): #{selected.to_a.join(', ')}" + return selected + end + + puts "" + puts "Select categories to export (comma-separated numbers, 0 = all):" + categories.each_with_index { |(_key, label), i| puts " #{i + 1}) #{label}" } + print "Selection [0 = all]: " + STDOUT.flush + input = (gets || "").strip + selected = normalize_category_selection(input, categories) + $logger.info "Selected categories: #{selected.to_a.join(', ')}" + selected +end + +#End method + # determine the directory paths platform_template_path = File.dirname(File.expand_path(__FILE__)) -core_path = File.join(platform_template_path, "core") -task_path = File.join(platform_template_path, "task") +config_folder_path = File.join(platform_template_path,'config') + +if options["CONFIG_FILE"].nil? + options["CONFIG_FILE"] = config_selection(config_folder_path) +end # ------------------------------------------------------------------------------ # methods @@ -112,20 +244,24 @@ def remove_discussion_id_attribute(model) # setup # ------------------------------------------------------------------------------ -logger.info "Installing gems for the \"#{template_name}\" template." +$logger.info "Installing gems for the \"#{template_name}\" template." Dir.chdir(platform_template_path) { system("bundle", "install") } vars = {} file = "#{platform_template_path}/#{options['CONFIG_FILE']}" # Check if configuration file exists -logger.info "Validating configuration file." +$logger.info "Validating configuration file." begin if File.exist?(file) != true - raise "The file \"#{options['CONFIG_FILE']}\" does not exist." + file = "#{config_folder_path}/#{options['CONFIG_FILE']}" + if File.exist?(file) != true + raise "The file \"#{options['CONFIG_FILE']}\" does not exist in the base or config directories." + end end rescue => error - logger.info error + $logger.info error + $logger.info "Exiting..." exit end @@ -133,11 +269,75 @@ def remove_discussion_id_attribute(model) begin vars.merge!( YAML.load(File.read(file)) ) rescue => error - logger.info "Error loading YAML configuration" - logger.info error + $logger.info "Error loading YAML configuration" + $logger.info error + $logger.info "Exiting..." + gets + exit +end +$logger.info "Configuration file passed validation." + + +#Check if nil/unencoded and update accordingly +def SecurePWD(file,vars,pwdAttribute) + #If no pwd, then ask for one, otherwise take current string that was not found to be B64 and convert + if vars[pwdAttribute]["service_user_password"].nil? + password = IO::console.getpass "Enter Password(#{pwdAttribute}): " + else + password = vars[pwdAttribute]["service_user_password"] + end + enc = Base64.strict_encode64(password) + vars[pwdAttribute]["service_user_password"] = enc.to_s + begin + fileObj = File.open(file, 'w') + puts "Updated pwd in #{pwdAttribute} to #{enc}" + fileObj.write vars.to_yaml + #{ |f| f.write vars.to_yaml } + rescue ArgumentError + $logger.error("There was an error while updating variables file:") + $logger.error(ArgumentError) + ensure + fileObj.close + end + #TODO - If you cannot properly write an encoded pwd, exit +end + +#Decode password to utilize +def DecodePWD(file, vars, pwdLoc) + pwdAttribute = vars[pwdLoc]["service_user_password"] + return Base64.decode64(pwdAttribute) +end + +#Confirm passwords exist and are in a proper format, call SecurePWD for any exceptions +def ValidatePWD(file, vars) + $pwdFields.each do |field| + t = vars[field]["service_user_password"] + #See if not a string, not encoded, or default + if !t.is_a?(String) || Base64.strict_encode64(Base64.decode64(t)) != t || t === "" + puts "Updating password #{t}" + SecurePWD(file, vars, field) + end + end +end + +ValidatePWD(file, vars) +vars["core"]["service_user_password"] = DecodePWD(file, vars, "core") +vars["task"]["service_user_password"] = DecodePWD(file, vars, "task") + +if vars["core"]["service_user_password"].empty? || vars["core"]["service_user_password"].nil? + #TODO - Ask for password + puts "Core password is blank! Password required. Exiting..." + gets exit end -logger.info "Configuration file passed validation." +if vars["task"]["service_user_password"].empty? || vars["task"]["service_user_password"].nil? + #TODO - Ask for password + puts "Task password is blank! Password required. Exiting..." + gets + exit +end + + # Set http_options based on values provided in the config file. http_options = (vars["http_options"] || {}).each_with_object({}) do |(k,v),result| @@ -148,12 +348,40 @@ def remove_discussion_id_attribute(model) SUBMISSIONS_TO_EXPORT = vars["options"]["SUBMISSIONS_TO_EXPORT"] REMOVE_DATA_PROPERTIES = vars["options"]["REMOVE_DATA_PROPERTIES"] +#Config exports folder exists, if not then create +if !File.directory?(File.join(platform_template_path,"exports")) + Dir.mkdir(File.join(platform_template_path, "exports")) +end + +#Setting core paths utilzing variables +if !vars['core']['space_slug'].nil? + folderName = vars['core']['space_slug'] +elsif !vars['core']['space_name'].nil? + folderName = vars['core']['space_name'] +else + puts "No space slug or name provided! Please provide one in order to export..." + gets + exit +end +core_path = File.join(platform_template_path, "exports", folderName, "core") +task_path = File.join(platform_template_path, "exports", folderName, "task") + + +#Confirmation of space +puts "Exporting #{vars['core']['space_name']} to #{folderName} in 5 seconds..." +4.downto(1) do |n| + puts n + sleep(1) +end + + + # Output the yml file config -logger.info "Output of Configuration File: \r #{JSON.pretty_generate(vars)}" +$logger.info "Output of Configuration File: \r #{JSON.pretty_generate(vars)}" -logger.info "Setting up the SDK" +$logger.info "Setting up the SDK" -space_sdk = KineticSdk::Core.new({ +$space_sdk = KineticSdk::Core.new({ space_server_url: vars["core"]["server_url"], space_slug: vars["core"]["space_slug"], username: vars["core"]["service_user_username"], @@ -161,7 +389,7 @@ def remove_discussion_id_attribute(model) options: http_options.merge({ export_directory: "#{core_path}" }) }) -task_sdk = KineticSdk::Task.new({ +$task_sdk = KineticSdk::Task.new({ app_server_url: "#{vars["task"]["server_url"]}", username: vars["task"]["service_user_username"], password: vars["task"]["service_user_password"], @@ -174,185 +402,325 @@ def remove_discussion_id_attribute(model) # Validate Core Connection begin - logger.info "Validating connection to Core \"#{space_sdk.api_url}\"" - response = space_sdk.me() + $logger.info "Validating connection to Core \"#{$space_sdk.api_url}\"" + response = $space_sdk.me() if response.status == 0 raise response.message elsif response.status.to_s.match(/4\d{2}/) raise response.content['error'] end rescue => error - logger.info error + $logger.info error exit end # Validate Task Connection begin - logger.info "Validating connection to Task \"#{task_sdk.api_url}\"" - response = task_sdk.environment() + $logger.info "Validating connection to Task \"#{$task_sdk.api_url}\"" + response = $task_sdk.environment() if response.status == 0 raise response.message elsif response.status.to_s.match(/4\d{2}/) raise response.content['error'] end rescue => error - logger.info error + $logger.info error exit end -logger.info "Validating connection to Cors and Task was Successful" +$logger.info "Validating connection to Cors and Task was Successful" + +# ---------------------------------------------------------------------------- +# Category selection - controls which phases run. `0`/empty input, or no +# `options.categories` in config, selects ALL. `0 = all` is the safe default. +# For finer-than-Core granularity (specific forms/teams only) use export-specific.rb, +# which is config-driven per artifact type. +# ---------------------------------------------------------------------------- +export_categories = [ + [:core, "Core space (kapps, forms, web APIs, models, webhooks, attribute & security definitions, teams) - via export_space"], + [:submissions, "Submission data (per options.SUBMISSIONS_TO_EXPORT)"], + [:task_trees, "Task trees"], + [:task_sources, "Task sources"], + [:task_groups, "Task groups"], + [:task_policy_rules, "Task policy rules"], + [:task_categories, "Task categories"], + [:task_access_keys, "Task access keys"], + [:task_handlers, "Task handlers"], + [:task_routines, "Task routines"], +] +selected = select_categories(export_categories, vars) # ------------------------------------------------------------------------------ # core # ------------------------------------------------------------------------------ -logger.info "Removing files and folders from the existing \"#{template_name}\" template." -FileUtils.rm_rf Dir.glob("#{core_path}/*") +##Clear old folder/files (only when re-exporting Core) +if selected.include?(:core) + $logger.info "Removing files and folders from the existing \"#{template_name}\" template." + FileUtils.rm_rf Dir.glob("#{core_path}/*") +end -logger.info "Setting up the Core SDK" +$logger.info "Setting up the Core SDK" # fetch export from core service and write to export directory -logger.info "Exporting the core components for the \"#{template_name}\" template." -logger.info " exporting with api: #{space_sdk.api_url}" -logger.info " - exporting configuration data (Kapps,forms, etc)" -space_sdk.export_space - -# cleanup properties that should not be committed with export -# bridge keys -Dir["#{core_path}/space/bridges/*.json"].each do |filename| - bridge = JSON.parse(File.read(filename)) - if bridge.has_key?("key") - bridge.delete("key") - File.open(filename, 'w') { |file| file.write(JSON.pretty_generate(bridge)) } - end -end +$logger.info "Exporting the core components for the \"#{template_name}\" template." +$logger.info " exporting with api: #{$space_sdk.api_url}" +$logger.info " - exporting configuration data (Kapps,forms, etc)" +core_thread = Thread.new do + max_threads=30 + pool = Concurrent::FixedThreadPool.new(max_threads) + promises = [] + + if selected.include?(:core) + $space_sdk.export_space + # cleanup properties that should not be committed with export + # bridge keys + + Dir["#{core_path}/space/bridges/*.json"].each do |filename| + promises << Concurrent::Promise.execute(executor: pool) do + + begin + bridge = JSON.parse(File.read(filename)) + if bridge.has_key?("key") + bridge.delete("key") + File.open(filename, 'w') { |file| file.write(JSON.pretty_generate(bridge)) } + end -# cleanup space -filename = "#{core_path}/space.json" -space = JSON.parse(File.read(filename)) -# filestore key -if space.has_key?("filestore") && space["filestore"].has_key?("key") - space["filestore"].delete("key") -end -# platform components -if space.has_key?("platformComponents") - if space["platformComponents"].has_key?("task") - space["platformComponents"].delete("task") - end - (space["platformComponents"]["agents"] || []).each_with_index do |agent,idx| - space["platformComponents"]["agents"][idx]["url"] = "" + end + end end -end -# rewrite the space file -File.open(filename, 'w') { |file| file.write(JSON.pretty_generate(space)) } + promises.each(&:wait!) -# cleanup discussion ids -Dir["#{core_path}/**/*.json"].each do |filename| - model = remove_discussion_id_attribute(JSON.parse(File.read(filename))) - File.open(filename, 'w') { |file| file.write(JSON.pretty_generate(model)) } -end -# export submissions -logger.info "Exporting and writing submission data" -(SUBMISSIONS_TO_EXPORT || []).delete_if{ |item| item["kappSlug"].nil?}.each do |item| - is_datastore = item["datastore"] || false - logger.info "Exporting - #{is_datastore ? 'datastore' : 'kapp'} form #{item['formSlug']}" - # build directory to write files to - submission_path = is_datastore ? - "#{core_path}/space/datastore/forms/#{item['formSlug']}" : - "#{core_path}/space/kapps/#{item['kappSlug']}/forms/#{item['formSlug']}" - - # get attachment fields from form definition - attachment_form = is_datastore ? - space_sdk.find_datastore_form(item['formSlug'], {"include" => "fields.details"}) : - space_sdk.find_form(item['kappSlug'], item['formSlug'], {"include" => "fields.details"}) - - # get attachment fields from form definition - attachement_files = attachment_form.status == 200 ? attachment_form.content['form']['fields'].select{ | file | file['dataType'] == "file" }.map { | field | field['name'] } : {} - - # set base url for attachments - attachment_base_url = is_datastore ? - "#{space_sdk.api_url.gsub("/app/api/v1", "")}/app/datastore" : - "#{space_sdk.api_url.gsub("/app/api/v1", "")}" - - # create folder to write submission data to - FileUtils.mkdir_p(submission_path, :mode => 0700) - - # build params to pass to the retrieve_form_submissions method - params = {"include" => "details,children,origin,parent,values", "limit" => 1000, "direction" => "ASC"} - - # open the submissions file in write mode - file = File.open("#{submission_path}/submissions.ndjson", 'w'); - - # ensure the file is empty - file.truncate(0) - response = nil - begin - # get submissions from datastore form or form - response = is_datastore ? - space_sdk.find_all_form_datastore_submissions(item['formSlug'], params).content : - space_sdk.find_form_submissions(item['kappSlug'], item['formSlug'], params).content - if response.has_key?("submissions") - # iterate over each submission - (response["submissions"] || []).each do |submission| - # write each attachment to a a dir - submission['values'].select{ |field, value| attachement_files.include?(field)}.each{ |field,value| - submission_id = submission['id'] - # define the dir to contain the attahment - download_dir = "#{submission_path}/#{submission_id}/#{field}" - # evaluate fields with multiple attachments - value.map.with_index{ | attachment, index | - # create folder to write attachment - FileUtils.mkdir_p(download_dir, :mode => 0700) - # dir and file name to write attachment - download_path = "#{download_dir}/#{File.join(".", attachment['name'])}" - # url to retrieve the attachment - url = "#{attachment_base_url}/submissions/#{submission_id}/files/#{ERB::Util.url_encode(field)}/#{index}/#{ERB::Util.url_encode(attachment['name'])}" - # retrieve and write attachment - space_sdk.stream_download_to_file(download_path, url, {}, space_sdk.default_headers) - # add the "path" key to indicate the attachment's location - attachment['path'] = "/#{submission_id}/#{field}/#{attachment['name']}" - } - } - # append each submission (removing the submission unwanted attributes) - file.puts(JSON.generate(submission.delete_if { |key, value| REMOVE_DATA_PROPERTIES.member?(key)})) + # cleanup space + filename = "#{core_path}/space.json" + space = JSON.parse(File.read(filename)) + # filestore key + if space.has_key?("filestore") && space["filestore"].has_key?("key") + space["filestore"].delete("key") + end + # platform components + if space.has_key?("platformComponents") + if space["platformComponents"].has_key?("task") + space["platformComponents"].delete("task") + end + (space["platformComponents"]["agents"] || []).each_with_index do |agent,idx| + space["platformComponents"]["agents"][idx]["url"] = "" + end + end + # rewrite the space file + File.open(filename, 'w') { |file| file.write(JSON.pretty_generate(space)) } + + # cleanup discussion ids + dir_promises = [] + Dir["#{core_path}/**/*.json"].each do |filename| + dir_promises << Concurrent::Promise.execute(executor: pool) do + model = remove_discussion_id_attribute(JSON.parse(File.read(filename))) + File.open(filename, 'w') { |file| file.write(JSON.pretty_generate(model)) } + end + end + dir_promises.each(&:wait!) + end # end :core + + + #################################################################### + ## Look to method and thread here + #################################################################### + + if selected.include?(:submissions) + #TODO - Flag for submissions to export + # export submissions + $logger.info "Exporting and writing submission data" + # Keep both datastore and kapp submission entries. Every entry needs a formSlug; + # kapp (non-datastore) entries also need a kappSlug. The previous filter dropped + # every item without a kappSlug, which silently excluded ALL datastore submissions. + (SUBMISSIONS_TO_EXPORT || []).reject { |item| + item["formSlug"].nil? || (!item["datastore"] && item["kappSlug"].nil?) + }.each do |item| + promises << Concurrent::Promise.execute(executor: pool) do + is_datastore = item["datastore"] || false + $logger.info "Exporting - #{is_datastore ? 'datastore' : 'kapp'} form #{item['formSlug']}" + # build directory to write files to + submission_path = is_datastore ? + "#{core_path}/space/datastore/forms/#{item['formSlug']}" : + "#{core_path}/space/kapps/#{item['kappSlug']}/forms/#{item['formSlug']}" + + # get attachment fields from form definition + attachment_form = is_datastore ? + $space_sdk.find_datastore_form(item['formSlug'], {"include" => "fields.details"}) : + $space_sdk.find_form(item['kappSlug'], item['formSlug'], {"include" => "fields.details"}) + + # get attachment fields from form definition + attachement_files = attachment_form.status == 200 ? attachment_form.content['form']['fields'].select{ | file | file['dataType'] == "file" }.map { | field | field['name'] } : {} + + # set base url for attachments + attachment_base_url = is_datastore ? + "#{$space_sdk.api_url.gsub("/app/api/v1", "")}/app/datastore" : + "#{$space_sdk.api_url.gsub("/app/api/v1", "")}" + + # create folder to write submission data to + FileUtils.mkdir_p(submission_path, :mode => 0700) + + + # open the submissions file in write mode + file = File.open("#{submission_path}/submissions.ndjson", 'w'); + # ensure the file is empty + file.truncate(0) + file.close() + file = File.open("#{submission_path}/submissions.ndjson", 'a'); + processed_submissions = false + createdAt = Time.now + previous = nil + # dataBlock = {} + # --------------------------------------------------- + # Iterate submissions in case over 1000 exist + # --------------------------------------------------- + while !processed_submissions && !createdAt.nil? do + # build params to pass to the retrieve_form_submissions method + params = {"include" => "details,children,origin,parent,values", "limit" => 1000, "direction" => "ASC"} + if !createdAt.nil? + params["q"] = "createdAt>=\"#{createdAt}\"" + end + + response = nil + begin + # get submissions from datastore form or form + response = is_datastore ? + $space_sdk.find_all_form_datastore_submissions(item['formSlug'], params).content : + $space_sdk.find_form_submissions(item['kappSlug'], item['formSlug'], params).content + if response.has_key?("submissions") + # File.write("outputtest.txt","#{response}") + # exit + # iterate over each submission + (response["submissions"] || []).each do |submission| + # write each attachment to a a dir + submission['values'].select{ |field, value| attachement_files.include?(field)}.each{ |field,value| + submission_id = submission['id'] + # define the dir to contain the attahment + download_dir = "#{submission_path}/#{submission_id}/#{field}" + # evaluate fields with multiple attachments + value.map.with_index{ | attachment, index | + # create folder to write attachment + FileUtils.mkdir_p(download_dir, :mode => 0700) + # dir and file name to write attachment + download_path = "#{download_dir}/#{File.join(".", attachment['name'])}" + # url to retrieve the attachment + url = "#{attachment_base_url}/submissions/#{submission_id}/files/#{ERB::Util.url_encode(field)}/#{index}/#{ERB::Util.url_encode(attachment['name'])}" + # retrieve and write attachment + $space_sdk.stream_download_to_file(download_path, url, {}, $space_sdk.default_headers) + # add the "path" key to indicate the attachment's location + attachment['path'] = "/#{submission_id}/#{field}/#{attachment['name']}" + } + } + # append each submission (removing the submission unwanted attributes) + # dataBlock = dataBlock.merge(JSON.generate(submission.delete_if { |key, value| REMOVE_DATA_PROPERTIES.member?(key)})) + json_string = JSON.generate(submission.delete_if { |key, value| REMOVE_DATA_PROPERTIES.member?(key)}) + unless json_string == previous + file.puts(json_string) + previous = json_string + end + # file.puts(JSON.generate(submission.delete_if { |key, value| REMOVE_DATA_PROPERTIES.member?(key)})) + end + end + params['pageToken'] = response['nextPageToken'] + # get next page of submissions if there are more + end while !response.nil? && !response['nextPageToken'].nil? + # close the submissions file + # file.close() + # $logger.info "Subs" + + if response["submissions"].count == 1000 + #Check if another batch exists + createdAt = (response["submissions"].last)["createdAt"] + # $logger.info "LastSub: #{response["submissions"].last}" + $logger.debug "New created at #{createdAt}" + else + #If not, exit loop + $logger.debug "Exiting submission loop" + processed_submissions = true + createdAt = nil + end + end + file.close() + #Write to file + # file.puts(dataBlock) end - params['pageToken'] = response['nextPageToken'] - # get next page of submissions if there are more - end while !response.nil? && !response['nextPageToken'].nil? - # close the submissions file - file.close() + end + promises.each(&:wait!) + $logger.info " - submission data export complete" + end # end :submissions + + pool.shutdown + pool.wait_for_termination end -logger.info " - submission data export complete" + + # ------------------------------------------------------------------------------ # task # ------------------------------------------------------------------------------ -logger.info "Removing files and folders from the existing \"#{template_name}\" template." -FileUtils.rm_rf Dir.glob("#{task_path}/*") - -logger.info "Exporting the task components for the \"#{template_name}\" template." -logger.info " exporting with api: #{task_sdk.api_url}" - -# export all sources, trees, routines, handlers, -# groups, policy rules, categories, and access keys -task_sdk.export_sources() -task_sdk.find_sources().content['sourceRoots'].each do |source| - task_sdk.find_trees({ "source" => source['name'] }).content['trees'].each do |tree| - task_sdk.export_tree(tree['title']) +task_thread = Thread.new do + any_task = [:task_trees, :task_sources, :task_groups, :task_policy_rules, :task_categories, :task_access_keys, :task_handlers, :task_routines].any? { |k| selected.include?(k) } + + max_threads=30 + pool = Concurrent::FixedThreadPool.new(max_threads) + if any_task + $logger.info "Removing files and folders from the existing \"#{template_name}\" template." + FileUtils.rm_rf Dir.glob("#{task_path}/*") end -end - -task_sdk.export_routines() -task_sdk.export_handlers() -task_sdk.export_groups() -task_sdk.export_policy_rules() -task_sdk.export_categories() -task_sdk.export_access_keys() + $logger.info "Exporting the task components for the \"#{template_name}\" template." + $logger.info " exporting with api: #{$task_sdk.api_url}" + + # export all sources, trees, routines, handlers, + # groups, policy rules, categories, and access keys + if selected.include?(:task_trees) + tree_promises = [] + $task_sdk.find_sources().content['sourceRoots'].each do |source| + $task_sdk.find_trees({ "source" => source['name'], "limit" => 1000 }).content['trees'].each do |tree| + tree_promises << Concurrent::Promise.execute(executor: pool) do + $task_sdk.export_tree(tree['title']) + end + end + end + tree_promises.each(&:wait!) + end # end :task_trees + + #Is above tied to below? + # Each task export is selectable independently (0 = all). + task_promises = [] + task_promises << Concurrent::Promise.execute(executor: pool) { $task_sdk.export_sources() } if selected.include?(:task_sources) + task_promises << Concurrent::Promise.execute(executor: pool) { $task_sdk.export_groups() } if selected.include?(:task_groups) + task_promises << Concurrent::Promise.execute(executor: pool) { $task_sdk.export_policy_rules() } if selected.include?(:task_policy_rules) + task_promises << Concurrent::Promise.execute(executor: pool) { $task_sdk.export_categories() } if selected.include?(:task_categories) + task_promises << Concurrent::Promise.execute(executor: pool) { $task_sdk.export_access_keys() } if selected.include?(:task_access_keys) + #NOTE - HARD LIMIT AT 1000 - would need date iteration for more + if selected.include?(:task_handlers) + $task_sdk.find_handlers({ "limit" => 1000 }).content["handlers"].each do |handler| + task_promises << Concurrent::Promise.execute(executor: pool) do + $task_sdk.export_handler(handler['definitionId']) + end + end + end + #NOTE - HARD LIMIT AT 1000 - would need date iteration for more + if selected.include?(:task_routines) + $task_sdk.find_routines({ "limit" => 1000 }).content["trees"].each do |routine| + task_promises << Concurrent::Promise.execute(executor: pool) do + $task_sdk.export_tree(routine['title']) + end + end + end + task_promises.each(&:wait!) + pool.shutdown + pool.wait_for_termination +end # ------------------------------------------------------------------------------ # complete # ------------------------------------------------------------------------------ - -logger.info "Finished exporting the \"#{template_name}\" template." +core_thread.join +task_thread.join +$logger.info "Finished exporting the \"#{template_name}\" template." +ending = Process.clock_gettime(Process::CLOCK_MONOTONIC) +elapsed = ending - starting +puts "Time: #{elapsed}" \ No newline at end of file diff --git a/import.rb b/import.rb index 8a09c6c..f8c3870 100644 --- a/import.rb +++ b/import.rb @@ -8,6 +8,7 @@ # Teams are not deleted from destination. It could be too dangerous to delete them. # TODO +#Have better validation/notification if you cannot connect (Certificate issue) # RUNNING THE SCRIPT: # ruby import_script.rb -c "<>" @@ -35,869 +36,1431 @@ log_level: info log_output: stderr =end - -require 'logger' +#require_relative './bundle/bundler/setup' +require 'logger' #For System Logging require 'json' require 'rexml/document' -require 'optparse' +require 'optparse' #For argument parsing +# require 'kinetic_sdk' +require 'find' #For config list building +require 'io/console' #For password request +require 'base64' #For pwd encoding +require 'concurrent-ruby' +require 'set' #For category selection + + require 'kinetic_sdk' -include REXML -template_name = "platform-template" -logger = Logger.new(STDERR) -logger.level = Logger::INFO -logger.formatter = proc do |severity, datetime, progname, msg| - date_format = datetime.utc.strftime("%Y-%m-%dT%H:%M:%S.%LZ") - "[#{date_format}] #{severity}: #{msg}\n" +# ------------------------------------------------------------------------------ +# Category selection (selective import/export) +# +# `categories` is an ordered Array of [key(Symbol), label(String)] pairs. +# Returns a Set of selected keys. Resolution order: +# 1. If the config provides `options.categories` (Array of 1-based numbers and/or +# string keys), use it WITHOUT prompting - keeps unattended/CI runs non-blocking. +# 2. Otherwise print a numbered menu and read a comma-separated line. +# `0` (or empty input) selects ALL categories. +# ------------------------------------------------------------------------------ +def normalize_category_selection(raw, categories) + keys = categories.map { |(key, _label)| key } + tokens = (raw.is_a?(Array) ? raw : raw.to_s.split(",")).map { |t| t.to_s.strip }.reject(&:empty?) + return keys.to_set if tokens.empty? || tokens.include?("0") + selected = Set.new + tokens.each do |token| + if token =~ /\A\d+\z/ + idx = token.to_i - 1 + selected << keys[idx] if idx >= 0 && idx < keys.length + elsif keys.include?(token.to_sym) + selected << token.to_sym + else + $logger.warn "Ignoring unknown category selection: #{token}" + end + end + selected end -######################################### +def select_categories(categories, vars) + configured = ((vars || {})["options"] || {})["categories"] + unless configured.nil? || (configured.is_a?(Array) && configured.empty?) + selected = normalize_category_selection(configured, categories) + $logger.info "Categories from config (options.categories): #{selected.to_a.join(', ')}" + return selected + end -# Determine the Present Working Directory -pwd = File.expand_path(File.dirname(__FILE__)) + puts "" + puts "Select categories to process (comma-separated numbers, 0 = all):" + categories.each_with_index { |(_key, label), i| puts " #{i + 1}) #{label}" } + print "Selection [0 = all]: " + STDOUT.flush + input = (gets || "").strip + selected = normalize_category_selection(input, categories) + $logger.info "Selected categories: #{selected.to_a.join(', ')}" + selected +end -ARGV << '-h' if ARGV.empty? -# The options specified on the command line will be collected in *options*. -options = {} -OptionParser.new do |opts| - opts.banner = "Usage: example.rb [options]" +def import_space() + template_name = "platform-template" + $pwdFields = ["core","task"] - opts.on("-c", "--c CONFIG_FILE", "The Configuration file to use") do |config| - options["CONFIG_FILE"] = config - end - - # No argument, shows at tail. This will print an options summary. - # Try it and see! - opts.on_tail("-h", "--help", "Show this message") do - puts opts - exit - end -end.parse! -#Now raise an exception if we have not found a CONFIG_FILE option -raise OptionParser::MissingArgument if options["CONFIG_FILE"].nil? + $logger = Logger.new(STDERR) + $logger.level = Logger::INFO + $logger.formatter = proc do |severity, datetime, progname, msg| + date_format = datetime.utc.strftime("%Y-%m-%dT%H:%M:%S.%LZ") + "[#{date_format}] #{severity}: #{msg}\n" + end + ######################################### + # Determine the Present Working Directory + pwd = File.expand_path(File.dirname(__FILE__)) -# determine the directory paths -platform_template_path = File.dirname(File.expand_path(__FILE__)) -core_path = File.join(platform_template_path, "core") -task_path = File.join(platform_template_path, "task") + # ARGV << '-h' if ARGV.empty? -# ------------------------------------------------------------------------------ -# methods -# ------------------------------------------------------------------------------ + # The options specified on the command line will be collected in *options*. + options = {} + OptionParser.new do |opts| + opts.banner = "Usage: example.rb [options]" + opts.on("-c", "--c CONFIG_FILE", "The Configuration file to use") do |config| + options["CONFIG_FILE"] = config + end + + # No argument, shows at tail. This will print an options summary. + # Try it and see! + opts.on_tail("-h", "--help", "Show this message") do + puts opts + exit + end + end.parse! + max_threads = 10 + $pool = Concurrent::FixedThreadPool.new(max_threads) + $mutex = Mutex.new + kapps_array = [] -# ------------------------------------------------------------------------------ -# constants -# ------------------------------------------------------------------------------ -# ------------------------------------------------------------------------------ -# setup -# ------------------------------------------------------------------------------ + #End method -logger.info "Installing gems for the \"#{template_name}\" template." -Dir.chdir(platform_template_path) { system("bundle", "install") } + # determine the directory paths + platform_template_path = File.dirname(File.expand_path(__FILE__)) + config_folder_path = File.join(platform_template_path,'config') + if options["CONFIG_FILE"].nil? + options["CONFIG_FILE"] = config_selection(config_folder_path) + end + $logger.info "Installing gems for the \"#{template_name}\" template." + Dir.chdir(platform_template_path) { system("bundle", "install") } -# ------------------------------------------------------------------------------ -# core -# ------------------------------------------------------------------------------ -vars = {} -# Read the config file specified in the command line into the variable "vars" -if File.file?(file = "#{platform_template_path}/#{options['CONFIG_FILE']}") - vars.merge!( YAML.load(File.read("#{platform_template_path}/#{options['CONFIG_FILE']}")) ) -elsif - raise "Config file not found: #{file}" -end + vars = {} + file = "#{platform_template_path}/#{options['CONFIG_FILE']}" -# Set http_options based on values provided in the config file. -http_options = (vars["http_options"] || {}).each_with_object({}) do |(k,v),result| - result[k.to_sym] = v -end + # Check if configuration file exists + $logger.info "Validating configuration file." + begin + if File.exist?(file) != true + file = "#{config_folder_path}/#{options['CONFIG_FILE']}" + if File.exist?(file) != true + raise "The file \"#{options['CONFIG_FILE']}\" does not exist in the base or config directories." + end + end + rescue => error + $logger.info error + $logger.info "Exiting..." + exit + end -# Set option values to default values if not included -vars["options"] = !vars["options"].nil? ? vars["options"] : {} -vars["options"]["delete"] = !vars["options"]["delete"].nil? ? vars["options"]["delete"] : false + # Read the config file specified in the command line into the variable "" + begin + vars.merge!( YAML.load(File.read(file)) ) + rescue => error + $logger.info "Error loading YAML configuration" + $logger.info error + $logger.info "Exiting..." + gets + exit + end + $logger.info "Configuration file passed validation." -logger.info "Importing using the config: #{JSON.pretty_generate(vars)}" + vars["options"] ||= {} + ValidatePWD(file, vars) + #Will confirm there is a valid, encoded password and decode. Otherwise it will prompt/encode pwd and return decoded variant + vars["core"]["service_user_password"] = DecodePWD(file, vars,"core") + vars["task"]["service_user_password"] = DecodePWD(file, vars, "task") -space_sdk = KineticSdk::Core.new({ - space_server_url: vars["core"]["server_url"], - space_slug: vars["core"]["space_slug"], - username: vars["core"]["service_user_username"], - password: vars["core"]["service_user_password"], - options: http_options.merge({ export_directory: "#{core_path}" }) -}) -puts "Are you sure you want to perform an import of data to #{vars["core"]["server_url"]}? [Y/N]" -STDOUT.flush -case (gets.downcase.chomp) -when 'y' - puts "Continuing Import" - STDOUT.flush -else - abort "Exiting Import" -end + if vars["core"]["service_user_password"].empty? || vars["core"]["service_user_password"].nil? + puts "Core password is blank! Password required. Exiting..." + gets + exit + end + if vars["task"]["service_user_password"].empty? || vars["task"]["service_user_password"].nil? + puts "Task password is blank! Password required. Exiting..." + gets + exit + end -################################################################### -# ------------------------------------------------------------------------------ -# Update Space Attributes -# ------------------------------------------------------------------------------ -sourceSpaceAttributeArray = [] -destinationSpaceAttributeArray = (space_sdk.find_space_attribute_definitions().content['spaceAttributeDefinitions']|| {}).map { |definition| definition['name']} -if File.file?(file = "#{core_path}/space/spaceAttributeDefinitions.json") - spaceAttributeDefinitions = JSON.parse(File.read(file)) + # Set http_options based on values provided in the config file. + http_options = (vars["http_options"] || {}).each_with_object({}) do |(k,v),result| + result[k.to_sym] = v + end - spaceAttributeDefinitions.each { |attribute| - if destinationSpaceAttributeArray.include?(attribute['name']) - space_sdk.update_space_attribute_definition(attribute['name'], attribute) - else - space_sdk.add_space_attribute_definition(attribute['name'], attribute['description'], attribute['allowsMultiple']) - end - sourceSpaceAttributeArray.push(attribute['name']) - } -end + #Config exports folder exists, if not then create + if !File.directory?(File.join(platform_template_path,"exports")) + Dir.mkdir(File.join(platform_template_path, "exports")) + end -destinationSpaceAttributeArray.each { | attribute | - if vars["options"]["delete"] && !sourceSpaceAttributeArray.include?(attribute) - space_sdk.delete_space_attribute_definition(attribute) + #Setting core paths utilzing variables - Check old_space_slug -> space_slug -> space_name + if !vars['core']['old_space_slug'].nil? + folderName = vars['core']['old_space_slug'] + elsif !vars['core']['space_slug'].nil? + folderName = vars['core']['space_slug'] + elsif !vars['core']['space_name'].nil? + folderName = vars['core']['space_name'] + else + puts "No space slug or name provided! Please provide one in order to export..." + gets + exit end -} + core_path = File.join(platform_template_path, "exports", folderName, "core") + task_path = File.join(platform_template_path, "exports", folderName, "task") -# ------------------------------------------------------------------------------ -# Update User Attributes -# ------------------------------------------------------------------------------ -sourceUserAttributeArray = [] -destinationUserAttributeArray = (space_sdk.find_user_attribute_definitions().content['userAttributeDefinitions'] || {}).map { |definition| definition['name']} - -if File.file?(file = "#{core_path}/space/userAttributeDefinitions.json") - userAttributeDefinitions = JSON.parse(File.read(file)) - userAttributeDefinitions.each { |attribute| - if destinationUserAttributeArray.include?(attribute['name']) - space_sdk.update_user_attribute_definition(attribute['name'], attribute) - else - space_sdk.add_user_attribute_definition(attribute['name'], attribute['description'], attribute['allowsMultiple']) - end - sourceUserAttributeArray.push(attribute['name']) - } -end + # Output the yml file config + $logger.info "Output of Configuration File: \r #{JSON.pretty_generate(vars)}" -destinationUserAttributeArray.each { | attribute | - if vars["options"]["delete"] && !sourceUserAttributeArray.include?(attribute) - space_sdk.delete_user_attribute_definition(attribute) + $logger.info "Setting up the SDK" + + $space_sdk = KineticSdk::Core.new({ + space_server_url: vars["core"]["server_url"], + space_slug: vars["core"]["space_slug"], + username: vars["core"]["service_user_username"], + password: vars["core"]["service_user_password"], + options: http_options.merge({ export_directory: "#{core_path}" }) + }) + + puts "Are you sure you want to perform an import of data from #{folderName} to #{vars["core"]["server_url"]}? [Y/N]" + STDOUT.flush + case (gets.downcase.chomp) + when 'y' + puts "Continuing Import" + STDOUT.flush + else + abort "Exiting Import" end -} -# ------------------------------------------------------------------------------ -# Update User Profile Attributes -# ------------------------------------------------------------------------------ -sourceUserProfileAttributeArray = [] -destinationUserProfileAttributeArray = (space_sdk.find_user_profile_attribute_definitions().content['userProfileAttributeDefinitions'] || {}).map { |definition| definition['name']} + # ---------------------------------------------------------------------------- + # Category selection - controls which phases run. `0`/empty input, or no + # `options.categories` in config, selects ALL. Subset selections are a power-user + # feature: the caller is responsible for prerequisites (e.g. forms need the kapp + # to already exist on the destination). `0 = all` is the safe default. + # ---------------------------------------------------------------------------- + import_categories = [ + [:bridge_models, "Bridge models"], + [:space_web_apis, "Space web APIs"], + [:space_attributes, "Space attribute definitions"], + [:user_attributes, "User & user-profile attribute definitions"], + [:team_attributes, "Team attribute definitions"], + [:teams, "Teams"], + [:space_security, "Space security policies"], + [:datastore_attributes, "Datastore form attribute definitions"], + [:kapps, "Kapps + per-kapp config (attrs, form types, security, categories, webhooks)"], + [:forms, "Forms"], + [:form_data, "Kapp form submission data"], + [:kapp_web_apis, "Kapp web APIs"], + [:datastore_data, "Datastore submission data"], + [:task_handlers, "Task handlers"], + [:task_routines, "Task routines"], + [:task_trees, "Task trees"], + [:task_categories, "Task categories"], + [:task_policy_rules, "Task policy rules"], + [:workflows, "Workflows (v6)"], + ] + selected = select_categories(import_categories, vars) + + import_bridge_models(core_path, vars) if selected.include?(:bridge_models) -if File.file?(file = "#{core_path}/space/userProfileAttributeDefinitions.json") - userProfileAttributeDefinitions = JSON.parse(File.read(file)) + # ------------------------------------------------------------------------------ + # delete bridge models + # Delete any Bridges from the destination which are missing from the import data + # ------------------------------------------------------------------------------ + import_space_web_apis(core_path, vars) if selected.include?(:space_web_apis) - userProfileAttributeDefinitions.each { |attribute| - if destinationUserProfileAttributeArray.include?(attribute['name']) - space_sdk.update_user_profile_attribute_definition(attribute['name'], attribute) - else - space_sdk.add_user_profile_attribute_definition(attribute['name'], attribute['description'], attribute['allowsMultiple']) - end - sourceUserProfileAttributeArray.push(attribute['name']) - } -end + # ------------------------------------------------------------------------------ + # Space-level definitions and teams + # Imported BEFORE kapps/forms that may reference them. These methods existed but + # were never called (and referenced `vars` without taking it as a parameter), so + # these artifacts silently did not migrate on a normal run. + # ------------------------------------------------------------------------------ + update_space_attributes(core_path, vars) if selected.include?(:space_attributes) + update_user_attributes(core_path, vars) if selected.include?(:user_attributes) + update_user_profile_attributes(core_path, vars) if selected.include?(:user_attributes) + update_team_attributes(core_path, vars) if selected.include?(:team_attributes) + import_space_teams(core_path, vars) if selected.include?(:teams) + update_security_policy(core_path, vars) if selected.include?(:space_security) + update_datastore_attributes(core_path, vars) if selected.include?(:datastore_attributes) -destinationUserProfileAttributeArray.each { | attribute | - if vars["options"]["delete"] && !sourceUserProfileAttributeArray.include?(attribute) - space_sdk.delete_user_profile_attribute_definition(attribute) - end -} + # ------------------------------------------------------------------------------ + # delete space teams + # TODO: A method doesn't exist for deleting the team + # ------------------------------------------------------------------------------ + # ------------------------------------------------------------------------------ + # import kapp data + # ------------------------------------------------------------------------------ -# ------------------------------------------------------------------------------ -# Update Team Attributes -# ------------------------------------------------------------------------------ + # Kapps are iterated sequentially (there are only a handful). The expensive + # per-kapp work - forms, form data, web APIs - is still parallelized on $pool + # inside the called methods. Running kapps sequentially avoids a thread-pool + # deadlock: if each kapp ran as a $pool promise that then waited on inner $pool + # promises, the pool could starve once #kapps approached the pool size. + # The loop only iterates when at least one kapp-scoped category is selected. + if [:kapps, :forms, :form_data, :kapp_web_apis].any? { |k| selected.include?(k) } + Dir["#{core_path}/space/kapps/*"].each { |file| + begin + kapp_slug = file.split(File::SEPARATOR).map {|x| x=="" ? File::SEPARATOR : x}.last.gsub('.json','') + already_processed = $mutex.synchronize do + if kapps_array.include?(kapp_slug) + true + else + kapps_array.push(kapp_slug) + false + end + end + next if already_processed + kapp = {} + kapp['slug'] = kapp_slug # set kapp_slug + + # Kapp create/update + per-kapp config, guarded by the :kapps category. + # kapp['slug'] is set above regardless, so the independently-selectable + # :forms / :form_data / :kapp_web_apis phases below still work when :kapps + # itself is not selected (assuming the kapp already exists on the destination). + if selected.include?(:kapps) + if File.file?(file) or ( File.directory?(file) and File.file?(file = "#{file}.json") ) # If the file is a file or a dir with a corresponding json file + kapp = JSON.parse( File.read(file) ) + kappExists = $space_sdk.find_kapp(kapp['slug']).code.to_i == 200 + if kappExists + $space_sdk.update_kapp(kapp['slug'], kapp) + else + $space_sdk.add_kapp(kapp['name'], kapp['slug'], kapp) + end + end -sourceTeamAttributeArray = [] -destinationTeamAttributeArray = (space_sdk.find_team_attribute_definitions().content['teamAttributeDefinitions']|| {}).map { |definition| definition['name']} -if File.file?(file = "#{core_path}/space/teamAttributeDefinitions.json") - teamAttributeDefinitions = JSON.parse(File.read(file)) - teamAttributeDefinitions.each { |attribute| - if destinationTeamAttributeArray.include?(attribute['name']) - space_sdk.update_team_attribute_definition(attribute['name'], attribute) - else - space_sdk.add_team_attribute_definition(attribute['name'], attribute['description'], attribute['allowsMultiple']) - end - sourceTeamAttributeArray.push(attribute['name']) - } -end + import_kapp_attribute_definitions(core_path, kapp, vars) + import_kapp_form_attribute_definitions(core_path, kapp, vars) + import_kapp_form_type_definitions(core_path, kapp, vars) + import_kapp_category_definitions(core_path, kapp, vars) -destinationTeamAttributeArray.each { | attribute | - if vars["options"]["delete"] && !sourceTeamAttributeArray.include?(attribute) - space_sdk.delete_team_attribute_definition(attribute) - end -} + import_kapp_security_policy_definitions(core_path, kapp, vars) + # ------------------------------------------------------------------------------ + # Migrate Kapp Categories + # ------------------------------------------------------------------------------ + import_kapp_categories(core_path, kapp, vars) -# ------------------------------------------------------------------------------ -# Update Datastore Attributes -# ------------------------------------------------------------------------------ + -sourceDatastoreAttributeArray = [] -destinationDatastoreAttributeArray =(space_sdk.find_datastore_form_attribute_definitions().content['datastoreFormAttributeDefinitions'] || {}).map { |definition| definition['name']} + # ------------------------------------------------------------------------------ + # import space webhooks + # ------------------------------------------------------------------------------ + sourceSpaceWebhooksArray = [] + destinationSpaceWebhooksNames = ($space_sdk.find_webhooks_on_space({"include"=>"details"}).content['webhooks'] || {}).map { |webhook| webhook['name'] } -if File.file?(file = "#{core_path}/space/datastoreFormAttributeDefinitions.json") - datastoreFormAttributeDefinitions = JSON.parse(File.read(file)) - datastoreFormAttributeDefinitions.each { |attribute| - if destinationDatastoreAttributeArray.include?(attribute['name']) - space_sdk.update_datastore_form_attribute_definition(attribute['name'], attribute) - else - space_sdk.add_datastore_form_attribute_definition(attribute['name'], attribute['description'], attribute['allowsMultiple']) - end - sourceDatastoreAttributeArray.push(attribute['name']) - } -end + Dir["#{core_path}/space/webhooks/*.json"].each { |file| + webhook = JSON.parse(File.read(file)) + if destinationSpaceWebhooksNames.include?(webhook['name']) + $space_sdk.update_webhook_on_space(webhook['name'], webhook) + else + $space_sdk.add_webhook_on_space(webhook) + end + sourceSpaceWebhooksArray.push(webhook['name']) + } -destinationDatastoreAttributeArray.each { | attribute | - if vars["options"]["delete"] && !sourceDatastoreAttributeArray.include?(attribute) - #Delete form is disabled - #space_sdk.delete_datastore_form_attribute_definition(attribute) - end -} + # ------------------------------------------------------------------------------ + # delete space webhooks + # TODO: A method doesn't exist for deleting the webhook + # ------------------------------------------------------------------------------ + destinationSpaceWebhooksNames.each do |webhook| + if vars["options"]["delete"] && !sourceSpaceWebhooksArray.include?(webhook) + $space_sdk.delete_webhook_on_space(webhook) + end + end -# ------------------------------------------------------------------------------ -# Update Security Policy -# ------------------------------------------------------------------------------ + # ------------------------------------------------------------------------------ + # Migrate Kapp Webhooks + # ------------------------------------------------------------------------------ + sourceWebhookArray = [] + webhooks_on_kapp = $space_sdk.find_webhooks_on_kapp(kapp['slug']) + + if webhooks_on_kapp.code=="200" + destinationWebhookArray = (webhooks_on_kapp.content['webhooks'] || {}).map { |definition| definition['name']} + Dir["#{core_path}/space/kapps/#{kapp['slug']}/webhooks/*.json"].each{ |webhookFile| + webhookDef = JSON.parse(File.read(webhookFile)) + if destinationWebhookArray.include?(webhookDef['name']) + $space_sdk.update_webhook_on_kapp(kapp['slug'], webhookDef['name'], webhookDef) + else + $space_sdk.add_webhook_on_kapp(kapp['slug'], webhookDef) + end + sourceWebhookArray.push(webhookDef['name']) + } + + # ------------------------------------------------------------------------------ + # Delete Kapp Webhooks + # ------------------------------------------------------------------------------ + destinationWebhookArray.each { | attribute | + if vars["options"]["delete"] && !sourceWebhookArray.include?(attribute) + $space_sdk.delete_webhook_on_kapp(kapp['slug'],attribute) + end + } + end -sourceSecurityPolicyArray = [] -destinationSecurityPolicyArray = (space_sdk.find_space_security_policy_definitions().content['securityPolicyDefinitions'] || {}).map { |definition| definition['name']} -if File.file?(file = "#{core_path}/space/securityPolicyDefinitions.json") - securityPolicyDefinitions = JSON.parse(File.read(file)) - securityPolicyDefinitions.each { |attribute| - if destinationSecurityPolicyArray.include?(attribute['name']) - space_sdk.update_space_security_policy_definition(attribute['name'], attribute) - else - space_sdk.add_space_security_policy_definition(attribute) - end - sourceSecurityPolicyArray.push(attribute['name']) - } -end + + end + # End :kapps-guarded block. The phases below are independently selectable. -destinationSecurityPolicyArray.each { | attribute | - if vars["options"]["delete"] && !sourceSecurityPolicyArray.include?(attribute) - space_sdk.delete_space_security_policy_definition(attribute) - end -} + import_forms(core_path,kapp,vars) if selected.include?(:forms) -# ------------------------------------------------------------------------------ -# import bridge models -# *NOTE* - This if the bridge doesn't exist the model will be imported w/ an empty "Bridge Slug" value. -# ------------------------------------------------------------------------------ + ##TODO - Convert to csv upload + ## PATCH https://playground-travis-wiese.kinopsdev.io/app/api/v1/kapps/kapp1/forms/f1/submissions?import + ## -destinationModels = space_sdk.find_bridge_models() -destinationModels_Array = (destinationModels.content['models'] || {}).map{ |model| model['name']} + # ------------------------------------------------------------------------------ + # Import Kapp Form Data + # ------------------------------------------------------------------------------ -Dir["#{core_path}/space/models/*.json"].each{ |model| - body = JSON.parse(File.read(model)) - if destinationModels_Array.include?(body['name']) - space_sdk.update_bridge_model(body['name'], body) - elsif - space_sdk.add_bridge_model(body) - end -} + import_kapp_form_data(core_path, kapp) if selected.include?(:form_data) + import_kapp_web_apis(core_path, kapp, vars) if selected.include?(:kapp_web_apis) + rescue => e + slug_for_log = (defined?(kapp) && kapp.is_a?(Hash) ? kapp['slug'] : nil) || (defined?(kapp_slug) ? kapp_slug : 'unknown') + $mutex.synchronize do + $logger.error "Error processing kapp '#{slug_for_log}': #{e.class}: #{e.message}" + $logger.error e.backtrace.join("\n") + end + end + } + end # end kapp-scoped category guard -# ------------------------------------------------------------------------------ -# delete bridge models -# Delete any Bridges from the destination which are missing from the import data -# ------------------------------------------------------------------------------ -SourceModelsArray = Dir["#{core_path}/space/models/*.json"].map{ |model| JSON.parse(File.read(model))['name'] } -destinationModels_Array.each do |model| - if vars["options"]["delete"] && !SourceModelsArray.include?(model) - space_sdk.delete_bridge_model(model) - end -end + #End Kapp loop -# ------------------------------------------------------------------------------ -# Import Space Web APIs -# ------------------------------------------------------------------------------ + # ------------------------------------------------------------------------------ + # Import Datastore Submission Data + # Datastore form definitions import via the kapp loop above (datastore is a normal + # kapp). Datastore SUBMISSIONS use a distinct API (add_datastore_submission) and are + # imported here. NOTE: this fully replaces existing datastore submissions for each + # form found on disk; it is a no-op when no datastore submission files were exported. + # ------------------------------------------------------------------------------ + import_datastore_data(core_path) if selected.include?(:datastore_data) -sourceSpaceWebApisArray = [] -destinationSpaceWebApisArray = (space_sdk.find_space_webapis().content['webApis'] || {}).map { |definition| definition['slug']} + # ------------------------------------------------------------------------------ + # task + # ------------------------------------------------------------------------------ - -Dir["#{core_path}/space/webApis/*"].each{ |file| - body = JSON.parse(File.read(file)) - if destinationSpaceWebApisArray.include?(body['slug']) - space_sdk.update_space_webapi(body['slug'], body) - else - space_sdk.add_space_webapi(body) - end - sourceSpaceWebApisArray.push(body['slug']) -} + task_selected = [:task_handlers, :task_routines, :task_trees, :task_categories, :task_policy_rules].any? { |k| selected.include?(k) } + if task_selected + $task_sdk = KineticSdk::Task.new({ + app_server_url: "#{vars["task"]["server_url"]}", + username: vars["task"]["service_user_username"], + password: vars["task"]["service_user_password"], + options: http_options.merge({ export_directory: "#{task_path}" }) + }) -# ------------------------------------------------------------------------------ -# Delete Space Web APIs -# Delete any Web APIs from the destination which are missing from the import data -# ------------------------------------------------------------------------------ -destinationSpaceWebApisArray.each { | webApi | - if vars["options"]["delete"] && !sourceSpaceWebApisArray.include?(webApi) - space_sdk.delete_space_webapi(webApi) - end -} + # ------------------------------------------------------------------------------ + # task import + # ------------------------------------------------------------------------------ -# ------------------------------------------------------------------------------ -# import datastore forms -# ------------------------------------------------------------------------------ -destinationDatastoreForms = [] #From destination server -sourceDatastoreForms = [] #From import data + $logger.info "Importing the task components for the \"#{template_name}\" template." + $logger.info " importing with api: #{$task_sdk.api_url}" -logger.info "Importing datastore forms for #{vars["core"]["space_slug"]}" + # ------------------------------------------------------------------------------ + # task handlers + # ------------------------------------------------------------------------------ - destinationDatastoreForms = (space_sdk.find_datastore_forms().content['forms'] || {}).map{ |datastore| datastore['slug']} - Dir["#{core_path}/space/datastore/forms/*.json"].each { |datastore| - body = JSON.parse(File.read(datastore)) - sourceDatastoreForms.push(body['slug']) - if destinationDatastoreForms.include?(body['slug']) - space_sdk.update_datastore_form(body['slug'], body) - else - space_sdk.add_datastore_form(body) - end - } + # import handlers forcing overwrite + $task_sdk.import_handlers_threaded(true) if selected.include?(:task_handlers) -# ------------------------------------------------------------------------------ -# delete datastore forms -# Delete any form from the destination which are missing from the import data -# ------------------------------------------------------------------------------ + # ------------------------------------------------------------------------------ + # Import Task Trees and Routines + # ------------------------------------------------------------------------------ + # import routines and force overwrite + $task_sdk.import_routines_threaded(true) if selected.include?(:task_routines) + # import trees and force overwrite + $task_sdk.import_trees_threaded(true) if selected.include?(:task_trees) -destinationDatastoreForms.each { |datastore_slug| - if vars["options"]["delete"] && !sourceDatastoreForms.include?(datastore_slug) - space_sdk.delete_datastore_form(datastore_slug) - end -} -# ------------------------------------------------------------------------------ -# Import Datastore Data -# ------------------------------------------------------------------------------ -Dir["#{core_path}/space/datastore/forms/**/submissions*.ndjson"].sort.each { |filename| - dir = File.dirname(filename) - form_slug = filename.match(/forms\/(.+)\/submissions\.ndjson/)[1] - (space_sdk.find_all_form_datastore_submissions(form_slug).content['submissions'] || []).each { |submission| - space_sdk.delete_datastore_submission(submission['id']) - } - File.readlines(filename).each { |line| - submission = JSON.parse(line) - submission["values"].map { |field, value| - # if the value contains an array of files - if value.is_a?(Array) && !value.empty? && value.first.is_a?(Hash) && value.first.has_key?('path') - value.map.with_index { |file, index| - # add 'path' key to the attribute value indicating the location of the attachment - file['path'] = "#{dir}#{file['path']}" - } - end - } - body = { - "values" => submission["values"], - "coreState" => submission["coreState"] - } - space_sdk.add_datastore_submission(form_slug, body).content - } -} -# ------------------------------------------------------------------------------ -# import space teams -# ------------------------------------------------------------------------------ -if (teams = Dir["#{core_path}/space/teams/*.json"]).length > 0 - SourceTeamArray = [] - destinationTeamsArray = (space_sdk.find_teams().content['teams'] || {}).map{ |team| {"slug" => team['slug'], "name"=>team['name']} } - teams.each{ |team| - body = JSON.parse(File.read(team)) - if !destinationTeamsArray.find {|destination_team| destination_team['slug'] == body['slug'] }.nil? - space_sdk.update_team(body['slug'], body) + # ------------------------------------------------------------------------------ + # import task categories + # ------------------------------------------------------------------------------ + + if selected.include?(:task_categories) + sourceCategories = [] #From import data + destinationCategoryNames = ($task_sdk.find_categories().content['categories'] || {}).map{ |category| category['name'] } + + Dir["#{task_path}/categories/*.json"].each { |file| + category = JSON.parse(File.read(file)) + + sourceCategories.push(category['name']) + + if destinationCategoryNames.include?(category['name']) + $task_sdk.update_category(category['name'], category) else - space_sdk.add_team(body) + $task_sdk.add_category(category) end - #Add Attributes to the Team - (body['attributes'] || []).each{ | attribute | - space_sdk.add_team_attribute(body['name'], attribute['name'], attribute['values']) - } - SourceTeamArray.push({'name' => body['name'], 'slug'=>body['slug']} ) } # ------------------------------------------------------------------------------ - # delete space teams - # TODO: A method doesn't exist for deleting the team + # delete task categories # ------------------------------------------------------------------------------ - destinationTeamsArray.each { |team| - #if !SourceTeamArray.include?(team) - if SourceTeamArray.find {|source_team| source_team['slug'] == team['slug'] }.nil? - #Delete has been disabled. It is potentially too dangerous to include w/o advanced knowledge. - #space_sdk.delete_team(team['slug']) + destinationCategoryNames.each { |category| + if vars["options"]["delete"] && !sourceCategories.include?(category) + $task_sdk.delete_category(category) end } -end + end # end :task_categories -# ------------------------------------------------------------------------------ -# import kapp data -# ------------------------------------------------------------------------------ + # ------------------------------------------------------------------------------ + # import task policy rules + # ------------------------------------------------------------------------------ -kapps_array = [] -Dir["#{core_path}/space/kapps/*"].each { |file| - kapp_slug = file.split(File::SEPARATOR).map {|x| x=="" ? File::SEPARATOR : x}.last.gsub('.json','') - next if kapps_array.include?(kapp_slug) # If the loop has already iterated over the kapp from the kapp file or the kapp dir skip the iteration - kapps_array.push(kapp_slug) # Append the kapp_slug to an array so a duplicate iteration doesn't occur - kapp = {} - kapp['slug'] = kapp_slug # set kapp_slug - - if File.file?(file) or ( File.directory?(file) and File.file?(file = "#{file}.json") ) # If the file is a file or a dir with a corresponding json file - kapp = JSON.parse( File.read(file) ) - kappExists = space_sdk.find_kapp(kapp['slug']).code.to_i == 200 - if kappExists - space_sdk.update_kapp(kapp['slug'], kapp) + if selected.include?(:task_policy_rules) + destinationPolicyRuleArray = $task_sdk.find_policy_rules().content['policyRules'] + sourcePolicyRuleArray = Dir["#{task_path}/policyRules/*.json"].map{ |file| + rule = JSON.parse(File.read(file)) + {"name" => rule['name'], "type" => rule['type']} + } + + Dir["#{task_path}/policyRules/*.json"].each { |file| + rule = JSON.parse(File.read(file)) + if !destinationPolicyRuleArray.find {|dest_rule| dest_rule['name']==rule['name'] && dest_rule['type']==rule['type'] }.nil? + $task_sdk.update_policy_rule(rule.slice('type', 'name'), rule) else - space_sdk.add_kapp(kapp['name'], kapp['slug'], kapp) + $task_sdk.add_policy_rule(rule) end - end + } # ------------------------------------------------------------------------------ - # Migrate Kapp Attribute Definitions + # delete task policy rules # ------------------------------------------------------------------------------ - if File.file?(file = "#{core_path}/space/kapps/#{kapp['slug']}/kappAttributeDefinitions.json") - sourceKappAttributeArray = [] - destinationKappAttributeArray = (space_sdk.find_kapp_attribute_definitions(kapp['slug']).content['kappAttributeDefinitions'] || {}).map { |definition| definition['name']} - kappAttributeDefinitions = JSON.parse(File.read(file)) - (kappAttributeDefinitions || []).each { |attribute| - if destinationKappAttributeArray.include?(attribute['name']) - space_sdk.update_kapp_attribute_definition(kapp['slug'], attribute['name'], attribute) - else - space_sdk.add_kapp_attribute_definition(kapp['slug'], attribute['name'], attribute['description'], attribute['allowsMultiple']) - end - sourceKappAttributeArray.push(attribute['name']) - } - # ------------------------------------------------------------------------------ - # Delete Kapp Attribute Definitions - # ------------------------------------------------------------------------------ - destinationKappAttributeArray.each { | attribute | - if vars["options"]["delete"] && !sourceKappAttributeArray.include?(attribute) - space_sdk.delete_kapp_attribute_definition(kapp['slug'],attribute) + destinationPolicyRuleArray.each { |rule| + if vars["options"]["delete"] && sourcePolicyRuleArray.find {|source_rule| source_rule['name']==rule['name'] && source_rule['type']==rule['type'] }.nil? + $task_sdk.delete_policy_rule(rule) + end + } + end # end :task_policy_rules + + # ------------------------------------------------------------------------------ + # Delete Trees and Routines not in the Source Data + # (runs when trees or routines were imported, to reconcile destination-only items) + # ------------------------------------------------------------------------------ + + if selected.include?(:task_trees) || selected.include?(:task_routines) + # identify Trees and Routines on destination + destinationtrees = [] + trees = $task_sdk.find_trees().content + (trees['trees'] || []).each { |tree| + destinationtrees.push( tree['title'] ) + } + + # identify Routines in source data + begin + sourceTrees = [] + Dir["#{task_path}/routines/*.xml"].each {|routine| + doc = REXML::Document.new(File.read(routine)) + root = doc.root + sourceTrees.push("#{root.elements["taskTree/name"].text}") + } + rescue + $logger.error "Error while identifying routines" + end + + begin + # identify trees in source data + Dir["#{task_path}/sources/*"].each {|source| + if File.directory? source + Dir["#{source}/trees/*.xml"].each { |tree| + doc = REXML::Document.new(File.read(tree)) + root = doc.root + tree = "#{root.elements["sourceName"].text} :: #{root.elements["sourceGroup"].text} :: #{root.elements["taskTree/name"].text}" + sourceTrees.push(tree) + } end } + rescue + $logger.error "Error identifying trees" end - # ------------------------------------------------------------------------------ - # Migrate Kapp Category Definitions - # ------------------------------------------------------------------------------ - if File.file?(file = "#{core_path}/space/kapps/#{kapp['slug']}/categoryAttributeDefinitions.json") - sourceKappCategoryArray = [] - destinationKappAttributeArray = (space_sdk.find_category_attribute_definitions(kapp['slug']).content['categoryAttributeDefinitions'] || {}).map { |definition| definition['name']} - kappCategoryDefinitions = JSON.parse(File.read(file)) - (kappCategoryDefinitions || []).each { |attribute| - if destinationKappAttributeArray.include?(attribute['name']) - space_sdk.update_category_attribute_definition(kapp['slug'], attribute['name'], attribute) - else - space_sdk.add_category_attribute_definition(kapp['slug'], attribute['name'], attribute['description'], attribute['allowsMultiple']) - end - sourceKappCategoryArray.push(attribute['name']) - } - # ------------------------------------------------------------------------------ - # Delete Kapp Category Definitions - # ------------------------------------------------------------------------------ - destinationKappAttributeArray.each { | attribute | - if vars["options"]["delete"] && !sourceKappCategoryArray.include?(attribute) - space_sdk.delete_category_attribute_definition(kapp['slug'],attribute) + begin + # Delete the extra tress and routines on the source + destinationtrees.each { | tree | + if vars["options"]["delete"] && !sourceTrees.include?(tree) + treeDef = tree.split(' :: ') + $task_sdk.delete_tree( tree ) end } + rescue + $logger.error "Error deleting extra trees/routines on source" + end + end # end trees/routines reconciliation guard + end # end task_selected guard + + + # Import v6 workflows as these are not the same as Trees and Routines + if selected.include?(:workflows) + $logger.info "Importing workflows" + $space_sdk.import_workflows(vars["core"]["space_slug"]) end + + # ------------------------------------------------------------------------------ + # complete + # ------------------------------------------------------------------------------ + + $logger.info "Finished importing the \"#{template_name}\" forms." + + $pool.shutdown + $pool.wait_for_termination +end + + + + + + + ################################################################################ + # Import Methods + ################################################################################ + # ------------------------------------------------------------------------------ - # Migrate Kapp Form Attribute Definitions + # Update Space Attributes # ------------------------------------------------------------------------------ - if File.file?(file = "#{core_path}/space/kapps/#{kapp['slug']}/formAttributeDefinitions.json") - sourceFormAttributeArray = [] - destinationFormAttributeArray = (space_sdk.find_form_attribute_definitions(kapp['slug']).content['formAttributeDefinitions'] || {}).map { |definition| definition['name']} - formAttributeDefinitions = JSON.parse(File.read(file)) - (formAttributeDefinitions || []).each { |attribute| - if destinationFormAttributeArray.include?(attribute['name']) - space_sdk.update_form_attribute_definition(kapp['slug'], attribute['name'], attribute) - else - space_sdk.add_form_attribute_definition(kapp['slug'], attribute['name'], attribute['description'], attribute['allowsMultiple']) - end - sourceFormAttributeArray.push(attribute['name']) - } - # ------------------------------------------------------------------------------ - # Delete Kapp Form Attribute Definitions - # ------------------------------------------------------------------------------ - destinationFormAttributeArray.each { | attribute | - if vars["options"]["delete"] && !sourceFormAttributeArray.include?(attribute) - space_sdk.delete_form_attribute_definition(kapp['slug'],attribute) + + def update_space_attributes(core_path, vars) + sourceSpaceAttributeArray = [] + destinationSpaceAttributeArray = ($space_sdk.find_space_attribute_definitions().content['spaceAttributeDefinitions']|| {}).map { |definition| definition['name']} + + if File.file?(file = "#{core_path}/space/spaceAttributeDefinitions.json") + spaceAttributeDefinitions = JSON.parse(File.read(file)) + + spaceAttributeDefinitions.each { |attribute| + if destinationSpaceAttributeArray.include?(attribute['name']) + $space_sdk.update_space_attribute_definition(attribute['name'], attribute) + else + $space_sdk.add_space_attribute_definition(attribute['name'], attribute['description'], attribute['allowsMultiple']) + end + sourceSpaceAttributeArray.push(attribute['name']) + } + end + destinationSpaceAttributeArray.each { | attribute | + if vars["options"]["delete"] && !sourceSpaceAttributeArray.include?(attribute) + $space_sdk.delete_space_attribute_definition(attribute) end } end - + + + + # ------------------------------------------------------------------------------ - # Migrate Kapp Form Type Definitions + # Update User Attributes # ------------------------------------------------------------------------------ - if File.file?(file = "#{core_path}/space/kapps/#{kapp['slug']}/formTypes.json") - sourceFormTypesArray = [] - destinationFormTypesArray = (space_sdk.find_formtypes(kapp['slug']).content['formTypes'] || {}).map { |formTypes| formTypes['name']} - formTypes = JSON.parse(File.read(file)) - (formTypes || []).each { |body| - if destinationFormTypesArray.include?(body['name']) - space_sdk.update_formtype(kapp['slug'], body['name'], body) - else - space_sdk.add_formtype(kapp['slug'], body) - end - sourceFormTypesArray.push(body['name']) - } - # ------------------------------------------------------------------------------ - # Delete Kapp Form Type Definitions - # ------------------------------------------------------------------------------ - destinationFormTypesArray.each { | name | - if vars["options"]["delete"] && !sourceFormTypesArray.include?(name) - space_sdk.delete_formtype(kapp['slug'],name) + def update_user_attributes( core_path, vars) + sourceUserAttributeArray = [] + destinationUserAttributeArray = ($space_sdk.find_user_attribute_definitions().content['userAttributeDefinitions'] || {}).map { |definition| definition['name']} + + if File.file?(file = "#{core_path}/space/userAttributeDefinitions.json") + userAttributeDefinitions = JSON.parse(File.read(file)) + userAttributeDefinitions.each { |attribute| + if destinationUserAttributeArray.include?(attribute['name']) + $space_sdk.update_user_attribute_definition(attribute['name'], attribute) + else + $space_sdk.add_user_attribute_definition(attribute['name'], attribute['description'], attribute['allowsMultiple']) + end + sourceUserAttributeArray.push(attribute['name']) + } + end + + destinationUserAttributeArray.each { | attribute | + if vars["options"]["delete"] && !sourceUserAttributeArray.include?(attribute) + $space_sdk.delete_user_attribute_definition(attribute) end } end # ------------------------------------------------------------------------------ - # Migrate Kapp Security Policy Definitions + # Update User Profile Attributes # ------------------------------------------------------------------------------ - if File.file?(file = "#{core_path}/space/kapps/#{kapp['slug']}/securityPolicyDefinitions.json") - sourceSecurtyPolicyArray = [] - destinationSecurtyPolicyArray = (space_sdk.find_security_policy_definitions(kapp['slug']).content['securityPolicyDefinitions'] || {}).map { |definition| definition['name']} - securityPolicyDefinitions = JSON.parse(File.read(file)) - (securityPolicyDefinitions || []).each { |attribute| - if destinationSecurtyPolicyArray.include?(attribute['name']) - space_sdk.update_security_policy_definition(kapp['slug'], attribute['name'], attribute) - else - space_sdk.add_security_policy_definition(kapp['slug'], attribute) - end - sourceSecurtyPolicyArray.push(attribute['name']) - } + def update_user_profile_attributes(core_path, vars) + sourceUserProfileAttributeArray = [] + destinationUserProfileAttributeArray = ($space_sdk.find_user_profile_attribute_definitions().content['userProfileAttributeDefinitions'] || {}).map { |definition| definition['name']} + + if File.file?(file = "#{core_path}/space/userProfileAttributeDefinitions.json") + userProfileAttributeDefinitions = JSON.parse(File.read(file)) + + userProfileAttributeDefinitions.each { |attribute| + if destinationUserProfileAttributeArray.include?(attribute['name']) + $space_sdk.update_user_profile_attribute_definition(attribute['name'], attribute) + else + $space_sdk.add_user_profile_attribute_definition(attribute['name'], attribute['description'], attribute['allowsMultiple']) + end + sourceUserProfileAttributeArray.push(attribute['name']) + } + end - destinationSecurtyPolicyArray.each { | attribute | - if vars["options"]["delete"] && !sourceSecurtyPolicyArray.include?(attribute) - space_sdk.delete_security_policy_definition(kapp['slug'],attribute) + destinationUserProfileAttributeArray.each { | attribute | + if vars["options"]["delete"] && !sourceUserProfileAttributeArray.include?(attribute) + $space_sdk.delete_user_profile_attribute_definition(attribute) end } end - + + + # ------------------------------------------------------------------------------ - # Migrate Kapp Categories + # Update Team Attributes # ------------------------------------------------------------------------------ - if File.file?(file = "#{core_path}/space/kapps/#{kapp['slug']}/categories.json") - sourceCategoryArray = [] - destinationCategoryArray = (space_sdk.find_categories(kapp['slug']).content['categories'] || {}).map { |definition| definition['slug']} - categories = JSON.parse(File.read(file)) - (categories || []).each { |attribute| - if destinationCategoryArray.include?(attribute['slug']) - space_sdk.update_category_on_kapp(kapp['slug'], attribute['slug'], attribute) - else - space_sdk.add_category_on_kapp(kapp['slug'], attribute) - end - sourceCategoryArray.push(attribute['slug']) - } - # ------------------------------------------------------------------------------ - # Delete Kapp Categories - # ------------------------------------------------------------------------------ - - destinationCategoryArray.each { | attribute | - if vars["options"]["delete"] && !sourceCategoryArray.include?(attribute) - space_sdk.delete_category_on_kapp(kapp['slug'],attribute) + def update_team_attributes( core_path, vars) + sourceTeamAttributeArray = [] + destinationTeamAttributeArray = ($space_sdk.find_team_attribute_definitions().content['teamAttributeDefinitions']|| {}).map { |definition| definition['name']} + + if File.file?(file = "#{core_path}/space/teamAttributeDefinitions.json") + teamAttributeDefinitions = JSON.parse(File.read(file)) + teamAttributeDefinitions.each { |attribute| + if destinationTeamAttributeArray.include?(attribute['name']) + $space_sdk.update_team_attribute_definition(attribute['name'], attribute) + else + $space_sdk.add_team_attribute_definition(attribute['name'], attribute['description'], attribute['allowsMultiple']) + end + sourceTeamAttributeArray.push(attribute['name']) + } + end + + destinationTeamAttributeArray.each { | attribute | + if vars["options"]["delete"] && !sourceTeamAttributeArray.include?(attribute) + $space_sdk.delete_team_attribute_definition(attribute) end } end + # ------------------------------------------------------------------------------ - # import space webhooks + # Update Datastore Attributes # ------------------------------------------------------------------------------ - sourceSpaceWebhooksArray = [] - destinationSpaceWebhooksArray = (space_sdk.find_webhooks_on_space().content['webhooks'] || {}).map{ |webhook| webhook['name']} - - Dir["#{core_path}/space/webhooks/*.json"].each{ |file| - webhook = JSON.parse(File.read(file)) - if destinationSpaceWebhooksArray.include?(webhook['name']) - space_sdk.update_webhook_on_space(webhook['name'], webhook) - elsif - space_sdk.add_webhook_on_space(webhook) + def update_datastore_attributes( core_path, vars) + sourceDatastoreAttributeArray = [] + destinationDatastoreAttributeArray =($space_sdk.find_datastore_form_attribute_definitions().content['datastoreFormAttributeDefinitions'] || {}).map { |definition| definition['name']} + + if File.file?(file = "#{core_path}/space/datastoreFormAttributeDefinitions.json") + datastoreFormAttributeDefinitions = JSON.parse(File.read(file)) + datastoreFormAttributeDefinitions.each { |attribute| + if destinationDatastoreAttributeArray.include?(attribute['name']) + $space_sdk.update_datastore_form_attribute_definition(attribute['name'], attribute) + else + $space_sdk.add_datastore_form_attribute_definition(attribute['name'], attribute['description'], attribute['allowsMultiple']) + end + sourceDatastoreAttributeArray.push(attribute['name']) + } end - sourceSpaceWebhooksArray.push(webhook['name']) - } + + destinationDatastoreAttributeArray.each { | attribute | + if vars["options"]["delete"] && !sourceDatastoreAttributeArray.include?(attribute) + #Delete form is disabled + #$space_sdk.delete_datastore_form_attribute_definition(attribute) + end + } + end + # ------------------------------------------------------------------------------ - # delete space webhooks - # TODO: A method doesn't exist for deleting the webhook + # Update Security Policy # ------------------------------------------------------------------------------ - - destinationSpaceWebhooksArray.each do |webhook| - if vars["options"]["delete"] && !sourceSpaceWebhooksArray.include?(webhook) - space_sdk.delete_webhook_on_space(webhook) + def update_security_policy( core_path, vars) + sourceSecurityPolicyArray = [] + destinationSecurityPolicyArray = ($space_sdk.find_space_security_policy_definitions().content['securityPolicyDefinitions'] || {}).map { |definition| definition['name']} + + if File.file?(file = "#{core_path}/space/securityPolicyDefinitions.json") + securityPolicyDefinitions = JSON.parse(File.read(file)) + securityPolicyDefinitions.each { |attribute| + if destinationSecurityPolicyArray.include?(attribute['name']) + $space_sdk.update_space_security_policy_definition(attribute['name'], attribute) + else + $space_sdk.add_space_security_policy_definition(attribute) + end + sourceSecurityPolicyArray.push(attribute['name']) + } end - end + + destinationSecurityPolicyArray.each { | attribute | + if vars["options"]["delete"] && !sourceSecurityPolicyArray.include?(attribute) + $space_sdk.delete_space_security_policy_definition(attribute) + end + } + end # ------------------------------------------------------------------------------ - # Migrate Kapp Webhooks + # import datastore forms # ------------------------------------------------------------------------------ - sourceWebhookArray = [] - webhooks_on_kapp = space_sdk.find_webhooks_on_kapp(kapp['slug']) - - if webhooks_on_kapp.code=="200" - destinationWebhookArray = (webhooks_on_kapp.content['webhooks'] || {}).map { |definition| definition['name']} - Dir["#{core_path}/space/kapps/#{kapp['slug']}/webhooks/*.json"].each{ |webhookFile| - webhookDef = JSON.parse(File.read(webhookFile)) - if destinationWebhookArray.include?(webhookDef['name']) - space_sdk.update_webhook_on_kapp(kapp['slug'], webhookDef['name'], webhookDef) - else - space_sdk.add_webhook_on_kapp(kapp['slug'], webhookDef) - end - sourceWebhookArray.push(webhookDef['name']) - } - - # ------------------------------------------------------------------------------ - # Delete Kapp Webhooks - # ------------------------------------------------------------------------------ - destinationWebhookArray.each { | attribute | - if vars["options"]["delete"] && !sourceWebhookArray.include?(attribute) - space_sdk.delete_webhook_on_kapp(kapp['slug'],attribute) + + def import_datastore_forms( core_path) + $logger.info "Importing datastore forms for #{vars["core"]["space_slug"]}" + destinationDatastoreForms = [] #From destination server + sourceDatastoreForms = [] #From import data + destinationDatastoreForms = fetch_all_datastore_forms.map { |datastore| datastore['slug'] } + Dir["#{core_path}/space/datastore/forms/*.json"].each { |datastore| + body = JSON.parse(File.read(datastore)) + sourceDatastoreForms.push(body['slug']) + if destinationDatastoreForms.include?(body['slug']) + $space_sdk.update_datastore_form(body['slug'], body) + else + $space_sdk.add_datastore_form(body) end } - end + end # ------------------------------------------------------------------------------ - # Add Kapp Forms + # delete datastore forms + # Delete any form from the destination which are missing from the import data # ------------------------------------------------------------------------------ - - if (forms = Dir["#{core_path}/space/kapps/#{kapp['slug']}/forms/*.json"]).length > 0 - sourceForms = [] #From import data - destinationForms = (space_sdk.find_forms(kapp['slug']).content['forms'] || {}).map{ |form| form['slug']} - forms.each { |form| - properties = File.read(form) - form = JSON.parse(properties) - sourceForms.push(form['slug']) - if destinationForms.include?(form['slug']) - space_sdk.update_form(kapp['slug'] ,form['slug'], form) - else - space_sdk.add_form(kapp['slug'], form) + def delete_datastore_forms(core_path) + destinationDatastoreForms.each { |datastore_slug| + if vars["options"]["delete"] && !sourceDatastoreForms.include?(datastore_slug) + $space_sdk.delete_datastore_form(datastore_slug) end } - # ------------------------------------------------------------------------------ - # delete forms - # ------------------------------------------------------------------------------ - destinationForms.each { |slug| - if vars["options"]["delete"] && !sourceForms.include?(slug) - #Delete form is disabled - #space_sdk.delete_form(kapp['slug'], slug) - end - } end - + + + # ------------------------------------------------------------------------------ - # Import Kapp Form Data + # Import Datastore Data # ------------------------------------------------------------------------------ - Dir["#{core_path}/space/kapps/#{kapp['slug']}/forms/**/submissions*.ndjson"].sort.each { |filename| - dir = File.dirname(filename) - form_slug = filename.match(/forms\/(.+)\/submissions\.ndjson/)[1] - - # This code could delete all submissions form the form before importing new data - # It is commented out because it could be dangerous to have in place and the delete_submission method doesn't exist currently. - #(space_sdk.find_all_form_submissions(kapp['slug'], form_slug).content['submissions'] || []).each { |submission| - # space_sdk.delete_submission(submission['id']) - #} - - File.readlines(filename).each { |line| - submission = JSON.parse(line) - submission["values"].map { |field, value| - # if the value contains an array of files - if value.is_a?(Array) && !value.empty? && value.first.is_a?(Hash) && value.first.has_key?('path') - value.map.with_index { |file, index| - # add 'path' key to the attribute value indicating the location of the attachment - file['path'] = "#{dir}#{file['path']}" - } - end + + def import_datastore_data( core_path) + Dir["#{core_path}/space/datastore/forms/**/submissions*.ndjson"].sort.each { |filename| + dir = File.dirname(filename) + form_slug = filename.match(/forms\/(.+)\/submissions\.ndjson/)[1] + ($space_sdk.find_all_form_datastore_submissions(form_slug).content['submissions'] || []).each { |submission| + $space_sdk.delete_datastore_submission(submission['id']) } - body = { - "values" => submission["values"], - "coreState" => submission["coreState"] + File.foreach(filename) { |line| + submission = JSON.parse(line) + submission["values"].map { |field, value| + # if the value contains an array of files + if value.is_a?(Array) && !value.empty? && value.first.is_a?(Hash) && value.first.has_key?('path') + value.map.with_index { |file, index| + # add 'path' key to the attribute value indicating the location of the attachment + file['path'] = "#{dir}#{file['path']}" + } + end + } + body = { + "values" => submission["values"], + "coreState" => submission["coreState"] + } + $space_sdk.add_datastore_submission(form_slug, body).content } - space_sdk.add_submission(kapp['slug'], form_slug, body).content } - } - # ------------------------------------------------------------------------------ - # Add Kapp Web APIs - # ------------------------------------------------------------------------------ - sourceWebApisArray = [] - destinationWebApisArray = (space_sdk.find_kapp_webapis(kapp['slug']).content['webApis'] || {}).map { |definition| definition['slug']} - Dir["#{core_path}/space/kapps/#{kapp['slug']}/webApis/*"].each { |webApi| - body = JSON.parse(File.read(webApi)) - if destinationWebApisArray.include?(body['slug']) - space_sdk.update_kapp_webapi(kapp['slug'], body['slug'], body) - else - space_sdk.add_kapp_webapi(kapp['slug'], body) - end - sourceWebApisArray.push(body['slug']) - } + end + # ------------------------------------------------------------------------------ - # Delete Kapp Web APIs + # import space teams # ------------------------------------------------------------------------------ - destinationWebApisArray.each { | webApi | - if vars["options"]["delete"] && !sourceWebApisArray.include?(webApi) - space_sdk.delete_kapp_webapi(kapp['slug'], webApi) - end - } -} - -# ------------------------------------------------------------------------------ -# task -# ------------------------------------------------------------------------------ -task_sdk = KineticSdk::Task.new({ - app_server_url: "#{vars["task"]["server_url"]}", - username: vars["task"]["service_user_username"], - password: vars["task"]["service_user_password"], - options: http_options.merge({ export_directory: "#{task_path}" }) -}) + def import_space_teams( core_path, vars) -# ------------------------------------------------------------------------------ -# task import -# ------------------------------------------------------------------------------ + if (teams = Dir["#{core_path}/space/teams/*.json"]).length > 0 + sourceTeamArray = [] + destinationTeamsArray = ($space_sdk.find_teams({"include"=>"details"}).content['teams'] || {}).map{ |team| {"slug" => team['slug'], "name"=>team['name'], "updatedAt"=>team['updatedAt']} } + teams.each{ |team| + body = JSON.parse(File.read(team)) + destinationTeam = destinationTeamsArray.find {|destination_team| destination_team['slug'] == body['slug']} + if !destinationTeam.nil? + $space_sdk.update_team(body['slug'], body) + else + $space_sdk.add_team(body) + end + #Add Attributes to the Team + (body['attributes'] || []).each{ | attribute | + $space_sdk.add_team_attribute(body['name'], attribute['name'], attribute['values']) + } + sourceTeamArray.push({'name' => body['name'], 'slug'=>body['slug']} ) + } + destinationTeamsArray.each { |team| + if vars["options"]["delete"] && sourceTeamArray.find {|source_team| source_team['slug'] == team['slug'] }.nil? + #Delete has been disabled. It is potentially too dangerous to include w/o advanced knowledge. + #$space_sdk.delete_team(team['slug']) + end + } + end + end -logger.info "Importing the task components for the \"#{template_name}\" template." -logger.info " importing with api: #{task_sdk.api_url}" + # ------------------------------------------------------------------------------ + # Import Kapp Categories + # ------------------------------------------------------------------------------ + def import_kapp_categories(core_path, kapp, vars) + if File.file?(file = "#{core_path}/space/kapps/#{kapp['slug']}/categories.json") + sourceCategoryArray = [] + destinationCategoryArray = ($space_sdk.find_categories(kapp['slug']).content['categories'] || {}).map { |definition| definition['slug']} + categories = JSON.parse(File.read(file)) + (categories || []).each { |attribute| + if destinationCategoryArray.include?(attribute['slug']) + $space_sdk.update_category_on_kapp(kapp['slug'], attribute['slug'], attribute) + else + $space_sdk.add_category_on_kapp(kapp['slug'], attribute) + end + sourceCategoryArray.push(attribute['slug']) + } + # ------------------------------------------------------------------------------ + # Delete Kapp Categories + # ------------------------------------------------------------------------------ + + destinationCategoryArray.each { | attribute | + if vars["options"]["delete"] && !sourceCategoryArray.include?(attribute) + $space_sdk.delete_category_on_kapp(kapp['slug'],attribute) + end + } + end + end + ################################################################################ + # Helpers + ################################################################################ + + #Configuration Selection + def config_selection(config_folder_path) + + #Ensure config folder exists + if !File.directory?(config_folder_path) + $logger.info "Config folder not found at #{config_folder_path}" + puts "Cannot find config folder!" + puts "Exiting..." + gets + exit + end -# ------------------------------------------------------------------------------ -# task handlers -# ------------------------------------------------------------------------------ + # #Determine Config file to use + config_exts = ['.yaml','.yml'] + configArray = [] + $logger.info "Checking #{config_folder_path} for config files" + #Check config folder for yaml/yml files containing the word 'import' + begin + Find.find("#{config_folder_path}/") do |file| + configArray.append(File.basename(file)) if config_exts.include?(File.extname(file)) && (File.basename(file).include?('import')) + end + rescue => error + #No config files found in config folder + $logger.error "Error finding default config file path!" + $logger.error "Error reported: #{error}" + puts "Cannot find config files in default path! (#{pwd})" + puts "Exiting script..." + gets + exit + end + $logger.info "Found config files" -# import handlers forcing overwrite -task_sdk.import_handlers(true) + #Print config file options with number indicators to select + puts "Select your config file" + configArray.each_with_index do |cFile, index| + puts "#{index+1}) #{cFile}" + end + $logger.info "Select section" + begin + print "Selection (0 to repeat options): " + sel = gets.chomp.to_i + begin + if sel === 0 + configArray.each_with_index do |cFile, index| + puts "#{index+1}) #{cFile}" + end + next + end + configFile = configArray[sel-1] + $logger.info "Option #{sel} - #{configFile}" + break + rescue + $logger.info "Error selecting config file! Exiting..." + puts "Error selecting config file!" + puts "Exiting..." + gets + exit + end + end while true + return configFile + end + + #Check if nil/unencoded and update accordingly +def SecurePWD(file,vars,pwdAttribute) + #If no pwd, then ask for one, otherwise take current string that was not found to be B64 and convert + if vars[pwdAttribute]["service_user_password"].nil? + password = IO::console.getpass "Enter Password(#{pwdAttribute}): " + else + password = vars[pwdAttribute]["service_user_password"] + end + enc = Base64.strict_encode64(password) + vars[pwdAttribute]["service_user_password"] = enc.to_s + begin + fileObj = File.open(file, 'w') + puts "Updated pwd in #{pwdAttribute} to #{enc}" + fileObj.write vars.to_yaml + #{ |f| f.write vars.to_yaml } + rescue ArgumentError + $logger.error("There was an error while updating variables file:") + $logger.error(ArgumentError) + ensure + fileObj.close + end +end -# ------------------------------------------------------------------------------ -# Import Task Trees and Routines -# ------------------------------------------------------------------------------ +#Decode password to utilize +def DecodePWD(file, vars, pwdLoc) + pwdAttribute = vars[pwdLoc]["service_user_password"] + return Base64.decode64(pwdAttribute) +end -# import routines and force overwrite -task_sdk.import_routines(true) -# import trees and force overwrite -task_sdk.import_trees(true) +#Confirm passwords exist and are in a proper format, call SecurePWD for any exceptions +def ValidatePWD(file, vars) + $pwdFields.each do |field| + t = vars[field]["service_user_password"] + #See if not a string, not encoded, or default + if !t.is_a?(String) || Base64.strict_encode64(Base64.decode64(t)) != t || t === "" + puts "Updating password #{t}" + SecurePWD(file, vars, field) + end + end +end +def convert_json_to_csv(json_file) + csv_file = json_file.gsub("ndjson","csv") + CSV.open(csv_file, 'w') do |csv| + File.foreach(json_file).with_index do |line, index| + record = JSON.parse(line) + + # Write header on first row + csv << record.keys if index == 0 + + # Write values + csv << record.values + end + end + end + # ------------------------------------------------------------------------------ + # Form content comparison (skip-if-unchanged) + # + # Both sides are compared in the same "export" shape: local files are export_space + # output, and the destination is fetched via fetch_all_forms(kapp, {'export'=>'true'}). + # We deep-compare the full definition after recursively stripping a MINIMAL set of + # volatile, server-managed keys. + # + # PRINCIPLE: when in doubt, UPDATE. Never add a meaningful key to the strip list - a + # false "unchanged" would silently drop a real change. (This comparison is on by + # default, so correctness here matters more than maximizing skips.) + # + # CROSS-VERSION NOTE: a 6.1 -> 6.0 migration (or any export-method shape delta such as + # defaultDataSource / choicesDataSource / renderAttributes.width) will simply differ and + # trigger an update - which is safe and matches the prior "always update" behavior. + # Same-version, same-shape forms compare equal and are skipped. + # ------------------------------------------------------------------------------ + FORM_COMPARE_STRIP_KEYS = %w[updatedAt createdAt updatedBy createdBy version].freeze + + # Recursively remove volatile/server-managed keys so two export payloads can be + # compared on content alone. + def strip_volatile_keys(value) + case value + when Hash + value.each_with_object({}) do |(k, v), acc| + next if FORM_COMPARE_STRIP_KEYS.include?(k) + acc[k] = strip_volatile_keys(v) + end + when Array + value.map { |v| strip_volatile_keys(v) } + else + value + end + end + # True when the local export form and the destination form are equivalent in content. + # Ruby Hash#== is recursive and order-independent, so normalized hashes compare cleanly. + def forms_equivalent?(local_form, remote_form) + return false if local_form.nil? || remote_form.nil? + strip_volatile_keys(local_form) == strip_volatile_keys(remote_form) + end -# ------------------------------------------------------------------------------ -# import task categories -# ------------------------------------------------------------------------------ + # ------------------------------------------------------------------------------ + # Migrate Kapp Form Attribute Definitions + # ------------------------------------------------------------------------------ + def import_kapp_form_attribute_definitions(core_path, kapp, vars) + if File.file?(file = "#{core_path}/space/kapps/#{kapp['slug']}/formAttributeDefinitions.json") + sourceFormAttributeArray = [] + destinationFormAttributeArray = ($space_sdk.find_form_attribute_definitions(kapp['slug']).content['formAttributeDefinitions'] || {}).map { |definition| definition['name']} + formAttributeDefinitions = JSON.parse(File.read(file)) + (formAttributeDefinitions || []).each { |attribute| + if destinationFormAttributeArray.include?(attribute['name']) + $space_sdk.update_form_attribute_definition(kapp['slug'], attribute['name'], attribute) + else + $space_sdk.add_form_attribute_definition(kapp['slug'], attribute['name'], attribute['description'], attribute['allowsMultiple']) + end + sourceFormAttributeArray.push(attribute['name']) + } + # ------------------------------------------------------------------------------ + # Delete Kapp Form Attribute Definitions + # ------------------------------------------------------------------------------ + destinationFormAttributeArray.each { | attribute | + if vars["options"]["delete"] && !sourceFormAttributeArray.include?(attribute) + $space_sdk.delete_form_attribute_definition(kapp['slug'],attribute) + end + } + end + end -sourceCategories = [] #From import data -destinationCategories = (task_sdk.find_categories().content['categories'] || {}).map{ |category| category['name']} + # ------------------------------------------------------------------------------ + # import bridge models + # *NOTE* - This if the bridge doesn't exist the model will be imported w/ an empty "Bridge Slug" value. + # ------------------------------------------------------------------------------ + def import_bridge_models(core_path,vars) + destinationModels = $space_sdk.find_bridge_models() + destinationModels_Array = (destinationModels.content['models'] || {}).map{ |model| model['name']} + + Dir["#{core_path}/space/models/*.json"].each{ |model| + body = JSON.parse(File.read(model)) + if destinationModels_Array.include?(body['name']) + $space_sdk.update_bridge_model(body['name'], body) + else + $space_sdk.add_bridge_model(body) + end + } + sourceModelsArray = Dir["#{core_path}/space/models/*.json"].map{ |model| JSON.parse(File.read(model))['name'] } -Dir["#{task_path}/categories/*.json"].each { |file| - category = JSON.parse(File.read(file)) - sourceCategories.push(category['name']) - if destinationCategories.include?(category['name']) - task_sdk.update_category(category['name'], category) - else - task_sdk.add_category(category) + destinationModels_Array.each do |model| + if vars["options"]["delete"] && !sourceModelsArray.include?(model) + $space_sdk.delete_bridge_model(model) + end + end end -} -# ------------------------------------------------------------------------------ -# delete task categories -# ------------------------------------------------------------------------------ + def import_kapp_form_data(core_path,kapp) + + promises = [] + Dir["#{core_path}/space/kapps/#{kapp['slug']}/forms/**/submissions*.ndjson"].sort.each { |filename| + promises << Concurrent::Promise.execute(executor: $pool) do + begin + dir = File.dirname(filename) + form_slug = filename.match(/forms\/(.+)\/submissions\.ndjson/)[1] + + #TODO - Convert to CSV upload path. Disabled until import_submissions_csv signature/body are wired up. + # convert_json_to_csv(filename) + # $space_sdk.import_submissions_csv(kapp['slug'], form_slug, body).content + + ## This code could delete all submissions from the form before importing new data + ## It is commented out because it could be dangerous to have in place and the delete_submission method doesn't exist currently. + #($space_sdk.find_all_form_submissions(kapp['slug'], form_slug).content['submissions'] || []).each { |submission| + # $space_sdk.delete_submission(submission['id']) + #} + + File.foreach(filename) { |line| + submission = JSON.parse(line) + submission["values"].map { |field, value| + # if the value contains an array of files + if value.is_a?(Array) && !value.empty? && value.first.is_a?(Hash) && value.first.has_key?('path') + value.map.with_index { |file, index| + # add 'path' key to the attribute value indicating the location of the attachment + file['path'] = "#{dir}#{file['path']}" + } + end + } + body = { + "values" => submission["values"], + "coreState" => submission["coreState"] + } + $space_sdk.add_submission(kapp['slug'], form_slug, body).content + } + rescue => e + $mutex.synchronize do + $logger.error("Failed to import form data from : #{e.message}") + $logger.error(e.backtrace.join("\n")) + end + raise + end + end + } + promises.each(&:wait!) + + $mutex.synchronize { $logger.info("Finished importing form data for kapp #{kapp['slug']}") } -destinationCategories.each { |category| - if vars["options"]["delete"] && !sourceCategories.include?(category) - task_sdk.delete_category(category) end -} -# ------------------------------------------------------------------------------ -# import task policy rules -# ------------------------------------------------------------------------------ + # ------------------------------------------------------------------------------ + # Import Space Web APIs + # ------------------------------------------------------------------------------ -destinationPolicyRuleArray = task_sdk.find_policy_rules().content['policyRules'] -sourcePolicyRuleArray = Dir["#{task_path}/policyRules/*.json"].map{ |file| - rule = JSON.parse(File.read(file)) - {"name" => rule['name'], "type" => rule['type']} - } + def import_space_web_apis(core_path, vars) + sourceSpaceWebApisArray = [] + destinationSpaceWebApisArray = ($space_sdk.find_space_webapis().content['webApis'] || {}).map { |definition| definition['slug']} + promises = [] + Dir["#{core_path}/space/webApis/*"].each{ |file| + promises << Concurrent::Promise.execute(executor: $pool) do + begin + body = JSON.parse(File.read(file)) + if destinationSpaceWebApisArray.include?(body['slug']) + $space_sdk.update_space_webapi(body['slug'], body) + else + $space_sdk.add_space_webapi(body) + end + $mutex.synchronize { sourceSpaceWebApisArray.push(body['slug']) } + rescue => e + $mutex.synchronize { $logger.error("Failed to import space web api from #{file}: #{e.class}: #{e.message}") } + end + end + } + promises.each(&:wait!) -Dir["#{task_path}/policyRules/*.json"].each { |file| - rule = JSON.parse(File.read(file)) - if !destinationPolicyRuleArray.find {|dest_rule| dest_rule['name']==rule['name'] && dest_rule['type']==rule['type'] }.nil? - task_sdk.update_policy_rule(rule.slice('type', 'name'), rule) - else - task_sdk.add_policy_rule(rule) + # ------------------------------------------------------------------------------ + # Delete Space Web APIs not present in source + # ------------------------------------------------------------------------------ + destinationSpaceWebApisArray.each { |webApi| + if vars["options"]["delete"] && !sourceSpaceWebApisArray.include?(webApi) + $space_sdk.delete_space_webapi(webApi) + end + } + end + # ------------------------------------------------------------------------------ + # Migrate Kapp Attribute Definitions + # ------------------------------------------------------------------------------ + def import_kapp_attribute_definitions(core_path,kapp, vars) + if File.file?(file = "#{core_path}/space/kapps/#{kapp['slug']}/kappAttributeDefinitions.json") + sourceKappAttributeArray = [] + destinationKappAttributeArray = ($space_sdk.find_kapp_attribute_definitions(kapp['slug']).content['kappAttributeDefinitions'] || {}).map { |definition| definition['name']} + kappAttributeDefinitions = JSON.parse(File.read(file)) + (kappAttributeDefinitions || []).each { |attribute| + if destinationKappAttributeArray.include?(attribute['name']) + $space_sdk.update_kapp_attribute_definition(kapp['slug'], attribute['name'], attribute) + else + $space_sdk.add_kapp_attribute_definition(kapp['slug'], attribute['name'], attribute['description'], attribute['allowsMultiple']) + end + sourceKappAttributeArray.push(attribute['name']) + } + # ------------------------------------------------------------------------------ + # Delete Kapp Attribute Definitions + # ------------------------------------------------------------------------------ + destinationKappAttributeArray.each { | attribute | + if vars["options"]["delete"] && !sourceKappAttributeArray.include?(attribute) + $space_sdk.delete_kapp_attribute_definition(kapp['slug'],attribute) + end + } + end + end + # ------------------------------------------------------------------------------ + # Page through find_forms; returns the merged array of form hashes. + # ------------------------------------------------------------------------------ + def fetch_all_forms(kapp_slug, params={}) + results = [] + params = params.merge('limit' => 1000) + loop do + response = $space_sdk.find_forms(kapp_slug, params) + break unless response.code.to_i == 200 + results.concat(response.content['forms'] || []) + token = response.content['nextPageToken'] + break if token.nil? + params['pageToken'] = token + end + results end -} -# ------------------------------------------------------------------------------ -# delete task policy rules -# ------------------------------------------------------------------------------ -destinationPolicyRuleArray.each { |rule| - if vars["options"]["delete"] && sourcePolicyRuleArray.find {|source_rule| source_rule['name']==rule['name'] && source_rule['type']==rule['type'] }.nil? - task_sdk.delete_policy_rule(rule) + # ------------------------------------------------------------------------------ + # Page through find_datastore_forms; returns the merged array of form hashes. + # ------------------------------------------------------------------------------ + def fetch_all_datastore_forms(params={}) + results = [] + params = params.merge('limit' => 1000) + loop do + response = $space_sdk.find_datastore_forms(params) + break unless response.code.to_i == 200 + results.concat(response.content['forms'] || []) + token = response.content['nextPageToken'] + break if token.nil? + params['pageToken'] = token + end + results end -} -# ------------------------------------------------------------------------------ -# Delete Trees and Routines not in the Source Data -# ------------------------------------------------------------------------------ + # ------------------------------------------------------------------------------ + # Import Kapp Forms + # ------------------------------------------------------------------------------ + def import_forms(core_path,kapp, vars) + if (forms = Dir["#{core_path}/space/kapps/#{kapp['slug']}/forms/*.json"]).length > 0 + sourceForms = [] #From import data + #destinationForms = ($space_sdk.find_forms(kapp['slug']).content['forms'] || {}).map{ |form| form['slug']} + destinationForms = fetch_all_forms(kapp['slug'], {'export'=>'true'}) + $logger.info ("Iterating kapp forms") + promises = [] + + + forms.each do |form_file| + promises << Concurrent::Promise.execute(executor: $pool) do + begin + properties = File.read(form_file) + form = JSON.parse(properties) + $mutex.synchronize do + $logger.info "Currently #{form['slug']}" + sourceForms.push(form['slug']) + end + + prev_form = (destinationForms.find { |f| f["slug"] == form['slug'] }) + if !prev_form.nil? + # Content comparison: skip the update when the destination form is already + # identical to the source export (ignoring volatile keys). When in doubt we + # UPDATE - see forms_equivalent? for the rationale. + if forms_equivalent?(form, prev_form) + $mutex.synchronize { $logger.info("Form #{form['slug']} unchanged, skipping...") } + else + $mutex.synchronize { $logger.info("Updating form #{form['slug']}") } + $space_sdk.update_form(kapp['slug'] ,form['slug'], form) + end + else + $mutex.synchronize { $logger.info("Adding new form #{form['slug']}") } + $space_sdk.add_form(kapp['slug'], form) + end + rescue => e + $mutex.synchronize do + $logger.error("Failed to import form from #{form_file}: #{e.class}: #{e.message}") + $logger.error(e.backtrace.join("\n")) + end + raise + end + end + end -# identify Trees and Routines on destination -destinationtrees = [] -trees = task_sdk.find_trees().content -(trees['trees'] || []).each { |tree| - destinationtrees.push( tree['title'] ) -} - -# identify Routines in source data -sourceTrees = [] -Dir["#{task_path}/routines/*.xml"].each {|routine| - doc = Document.new(File.new(routine)) - root = doc.root - sourceTrees.push("#{root.elements["taskTree/name"].text}") -} -# identify trees in source data -Dir["#{task_path}/sources/*"].each {|source| - if File.directory? source - Dir["#{source}/trees/*.xml"].each { |tree| - doc = Document.new(File.new(tree)) - root = doc.root - tree = "#{root.elements["sourceName"].text} :: #{root.elements["sourceGroup"].text} :: #{root.elements["taskTree/name"].text}" - sourceTrees.push(tree) - } + promises.each(&:wait!) + + $mutex.synchronize { $logger.info("Finished importing #{sourceForms.size} forms for kapp #{kapp['slug']}") } + + # ------------------------------------------------------------------------------ + # delete forms + # ------------------------------------------------------------------------------ + destinationForms.each { |dest_form| + if vars["options"]["delete"] && !sourceForms.include?(dest_form["slug"]) + #Delete form is disabled + #$space_sdk.delete_form(kapp['slug'], dest_form["slug"]) + end + } + end + end + # ------------------------------------------------------------------------------ + # Migrate Kapp Category Definitions + # ------------------------------------------------------------------------------ + def import_kapp_category_definitions(core_path,kapp,vars) + if File.file?(file = "#{core_path}/space/kapps/#{kapp['slug']}/categoryAttributeDefinitions.json") + sourceKappCategoryArray = [] + destinationKappAttributeArray = ($space_sdk.find_category_attribute_definitions(kapp['slug']).content['categoryAttributeDefinitions'] || {}).map { |definition| definition['name']} + kappCategoryDefinitions = JSON.parse(File.read(file)) + (kappCategoryDefinitions || []).each { |attribute| + if destinationKappAttributeArray.include?(attribute['name']) + $space_sdk.update_category_attribute_definition(kapp['slug'], attribute['name'], attribute) + else + $space_sdk.add_category_attribute_definition(kapp['slug'], attribute['name'], attribute['description'], attribute['allowsMultiple']) + end + sourceKappCategoryArray.push(attribute['name']) + } + # ------------------------------------------------------------------------------ + # Delete Kapp Category Definitions + # ------------------------------------------------------------------------------ + destinationKappAttributeArray.each { | attribute | + if vars["options"]["delete"] && !sourceKappCategoryArray.include?(attribute) + $space_sdk.delete_category_attribute_definition(kapp['slug'],attribute) + end + } + end end -} -# Delete the extra tress and routines on the source -destinationtrees.each { | tree | - if vars["options"]["delete"] && !sourceTrees.include?(tree) - treeDef = tree.split(' :: ') - task_sdk.delete_tree( tree ) + def import_kapp_form_type_definitions(core_path, kapp, vars) + # ------------------------------------------------------------------------------ + # Migrate Kapp Form Type Definitions + # ------------------------------------------------------------------------------ + if File.file?(file = "#{core_path}/space/kapps/#{kapp['slug']}/formTypes.json") + sourceFormTypesArray = [] + destinationFormTypesArray = ($space_sdk.find_formtypes(kapp['slug']).content['formTypes'] || {}).map { |formTypes| formTypes['name']} + formTypes = JSON.parse(File.read(file)) + (formTypes || []).each { |body| + if destinationFormTypesArray.include?(body['name']) + $space_sdk.update_formtype(kapp['slug'], body['name'], body) + else + $space_sdk.add_formtype(kapp['slug'], body) + end + sourceFormTypesArray.push(body['name']) + } + # ------------------------------------------------------------------------------ + # Delete Kapp Form Type Definitions + # ------------------------------------------------------------------------------ + destinationFormTypesArray.each { | name | + if vars["options"]["delete"] && !sourceFormTypesArray.include?(name) + $space_sdk.delete_formtype(kapp['slug'],name) + end + } + end + end + def import_kapp_web_apis(core_path, kapp, vars) + # ------------------------------------------------------------------------------ + # Add Kapp Web APIs + # ------------------------------------------------------------------------------ + sourceWebApisArray = [] + destinationWebApisArray = ($space_sdk.find_kapp_webapis(kapp['slug']).content['webApis'] || {}).map { |definition| definition['slug']} + Dir["#{core_path}/space/kapps/#{kapp['slug']}/webApis/*"].each { |webApi| + body = JSON.parse(File.read(webApi)) + if destinationWebApisArray.include?(body['slug']) + $space_sdk.update_kapp_webapi(kapp['slug'], body['slug'], body) + else + $space_sdk.add_kapp_webapi(kapp['slug'], body) + end + sourceWebApisArray.push(body['slug']) + } + # ------------------------------------------------------------------------------ + # Delete Kapp Web APIs not present in source + # ------------------------------------------------------------------------------ + destinationWebApisArray.each { |webApi| + if vars["options"]["delete"] && !sourceWebApisArray.include?(webApi) + $space_sdk.delete_kapp_webapi(kapp['slug'], webApi) + end + } end -} -# Import v6 workflows as these are not not the same as Trees and Routines -logger.info "Importing workflows" -space_sdk.import_workflows(vars["core"]["space_slug"]) + def import_kapp_security_policy_definitions(core_path,kapp,vars) + # ------------------------------------------------------------------------------ + # Migrate Kapp Security Policy Definitions + # ------------------------------------------------------------------------------ + if File.file?(file = "#{core_path}/space/kapps/#{kapp['slug']}/securityPolicyDefinitions.json") + sourceSecurtyPolicyArray = [] + destinationSecurtyPolicyArray = ($space_sdk.find_security_policy_definitions(kapp['slug']).content['securityPolicyDefinitions'] || {}).map { |definition| definition['name']} + securityPolicyDefinitions = JSON.parse(File.read(file)) + (securityPolicyDefinitions || []).each { |attribute| + if destinationSecurtyPolicyArray.include?(attribute['name']) + $space_sdk.update_security_policy_definition(kapp['slug'], attribute['name'], attribute) + else + $space_sdk.add_security_policy_definition(kapp['slug'], attribute) + end + sourceSecurtyPolicyArray.push(attribute['name']) + } -# ------------------------------------------------------------------------------ -# complete -# ------------------------------------------------------------------------------ + destinationSecurtyPolicyArray.each { | attribute | + if vars["options"]["delete"] && !sourceSecurtyPolicyArray.include?(attribute) + $space_sdk.delete_security_policy_definition(kapp['slug'],attribute) + end + } + end + end -logger.info "Finished importing the \"#{template_name}\" forms." +starting = Process.clock_gettime(Process::CLOCK_MONOTONIC) +import_space() +ending = Process.clock_gettime(Process::CLOCK_MONOTONIC) +elapsed = ending - starting +puts "Time: #{elapsed}" \ No newline at end of file diff --git a/kinetic_sdk-5.0.31.gem b/kinetic_sdk-5.0.31.gem new file mode 100644 index 0000000..095f5c8 Binary files /dev/null and b/kinetic_sdk-5.0.31.gem differ diff --git a/vendor/cache/base64-0.2.0.gem b/vendor/cache/base64-0.2.0.gem new file mode 100644 index 0000000..a45f09b Binary files /dev/null and b/vendor/cache/base64-0.2.0.gem differ diff --git a/vendor/cache/concurrent-ruby-1.3.6.gem b/vendor/cache/concurrent-ruby-1.3.6.gem new file mode 100644 index 0000000..5689df5 Binary files /dev/null and b/vendor/cache/concurrent-ruby-1.3.6.gem differ diff --git a/vendor/cache/find-0.1.1.gem b/vendor/cache/find-0.1.1.gem new file mode 100644 index 0000000..ee28fd9 Binary files /dev/null and b/vendor/cache/find-0.1.1.gem differ diff --git a/vendor/cache/io-console-0.5.6.gem b/vendor/cache/io-console-0.5.6.gem new file mode 100644 index 0000000..bf0046a Binary files /dev/null and b/vendor/cache/io-console-0.5.6.gem differ diff --git a/vendor/cache/json-2.3.0.gem b/vendor/cache/json-2.3.0.gem new file mode 100644 index 0000000..df1b6fa Binary files /dev/null and b/vendor/cache/json-2.3.0.gem differ diff --git a/vendor/cache/kinetic_sdk-5.0.31.gem b/vendor/cache/kinetic_sdk-5.0.31.gem new file mode 100644 index 0000000..095f5c8 Binary files /dev/null and b/vendor/cache/kinetic_sdk-5.0.31.gem differ diff --git a/vendor/cache/logger-1.4.2.gem b/vendor/cache/logger-1.4.2.gem new file mode 100644 index 0000000..49d2765 Binary files /dev/null and b/vendor/cache/logger-1.4.2.gem differ diff --git a/vendor/cache/mime-types-3.7.0.gem b/vendor/cache/mime-types-3.7.0.gem new file mode 100644 index 0000000..4430455 Binary files /dev/null and b/vendor/cache/mime-types-3.7.0.gem differ diff --git a/vendor/cache/mime-types-data-3.2026.0203.gem b/vendor/cache/mime-types-data-3.2026.0203.gem new file mode 100644 index 0000000..cd3a904 Binary files /dev/null and b/vendor/cache/mime-types-data-3.2026.0203.gem differ diff --git a/vendor/cache/multipart-post-2.0.0.gem b/vendor/cache/multipart-post-2.0.0.gem new file mode 100644 index 0000000..abfff3d Binary files /dev/null and b/vendor/cache/multipart-post-2.0.0.gem differ diff --git a/vendor/cache/optparse-0.6.0.gem b/vendor/cache/optparse-0.6.0.gem new file mode 100644 index 0000000..d2b93e0 Binary files /dev/null and b/vendor/cache/optparse-0.6.0.gem differ diff --git a/vendor/cache/rexml-3.2.3.gem b/vendor/cache/rexml-3.2.3.gem new file mode 100644 index 0000000..47b76c5 Binary files /dev/null and b/vendor/cache/rexml-3.2.3.gem differ diff --git a/vendor/cache/slugify-1.0.7.gem b/vendor/cache/slugify-1.0.7.gem new file mode 100644 index 0000000..c610f3b Binary files /dev/null and b/vendor/cache/slugify-1.0.7.gem differ