Create function to build ARTIS `.duckdb` directly from KNB

[theamarks]

Running To-do List

• add artis-duckdb-metadata/blob/develop/process_knb_to_duckdb.R to exploreARTIS/develop-build-knb-duckdb branch
  • function .R scripts need to live in ./R/ directory when developing an R package
• Move all roxygen2 documentation decorators to the very top of the function script. It may pass checks and build when it is separated across the script, but lets stick to explected formatting conventions.
• explicitly #' @export process_knb_to_duckdb in roxygen2 header to signify the main function to export the the package NAMESPACE.
• Reorganize script so helper functions are listed first. Think about maintaining an "executable timeline" from top to bottom
• call package dependencies in roxygen2 header instead of in script
• add directory argument to process_knb_to_duckdb() to replace download_dir <- "~/Downloads/artis_downloads"
• We need the ability to generate a duckdb of "custom artis timeseires" HS_version/year pairings. Very few end users will need all HS_versions and years. Perhaps have two functions? Or an argument along the lines of "artis_custome_timeseries = TRUE"

Problem

Storing the ARTIS database on KNB requires the use of many .csv to ensure the data is preserved in a accessible and usable format. For our end users, a duckdb will provide many benefits. However, we want to make it as easy as possible to set up a duckdb for our end users without introducing even more technologies. They will already have a learning curve to querying and running analyses with duckdb (trying to minimize!). We want to make the uptake of our new duckdb distribution as smooth and reproducible as possible.

Solution

Call a single function and BAM, you have the power of the new ARTIS duckdb on your local computer!

Add an exploreARTIS function that standardizes and streamlines building the ARTIS duckdb pulling data directly from the ARTIS KNB record. Use @Anurag19101996's script as foundation to insert data in a standardized way into duckdb. Pulling data directly from KNB would also register every ARTIS data download through the built in KNB metric service!

Function arguments:

• version: "latest" or "DOI" or "1.0.0"
• model_run?: "FAO" of "SAU" (not sure how we will separate these in KNB. might not need if new DOI assigned to different data versions)
• user_orcid: "https://orcid.org/0000-0002-9370-9128" (for signing into KNB)
• other KNB credentials needed?
• path: file path for duckdb file

Questions

• Use DataOne API to request data OR rdataone package?
• Is rdataone just a R client for the DataONE API?
• How do we specify the KNB repository through the DataONE API and/or rdataone package?
• Is it possible to download/pull KNB data directly into duckdb without saving it locally first? Insert R function into SQL query passed to duckdb?
• Find the persistent DOI for the ARTIS data and the versioned DOIs

Relevant info and resources

The Knowledge Network for Biocomplexity (KNB) data repository is a member of the DataOne network of data repositories.

• rdataone package https://github.com/DataONEorg/rdataone
• DataONE API documentation https://releases.dataone.org/online/api-documentation-v2.0.1/apis/CN_APIs.html#CNRead.resolve
• Searching DataONE Data Holdings
• ARTIS data on KNB - https://knb.ecoinformatics.org/view/doi%3A10.5063%2FF1CZ35N7
• DOI - doi:10.5063/F1CZ35N7
• Guide to R package development

Ideas for function name

• exploreARTIS::build_artis_with_ducks() 🦆
• build_artis_duckdb()
• make_artis_duckdb()
• setup_artis_duckdb()

[theamarks]

Notes from Meeting (Jessica, Anurag, Althea) 2025-03-18

• Alternative way / function to build .duckdb from local files

  • Future proof & minimal dependency in case KNB resources are not maintained
  • Also useful for core-team development uses

• Documentation for build_artis_duckdb() workflow will live in in ARTIS manual

• Need to update ARTIS manual infrastructure schematic to include duckdb and remove API

[theamarks]

Notes from Meeting (Anurag & Althea) 2025-04-28

• How to document new function?
  • R package uses Roxygen2 package decorators at the top of the function itself
  • Guide to R package dev with documentation specifics - https://r-pkgs.org/man.html#roxygen2-basics
  • Populates console call ?exploreARTIS::plot_sankey() in help window (Rstudio IDE)
• ./R/ directory is where all R functions live for the package
• created new dev branch develop-build-knb-duckdb to push new function to
  • follow "gitflow" dev model: merge/rebase with develop branch for testing and staging for next release. When ready for next release will merge with main
• New IDE for python & R - https://positron.posit.co/

[theamarks]

Notes from Meeting (Anurag & Althea) 2025-05-05

• Anurag's 1st draft of function and helper functions rebased and merged through PR to artis-duckdb-metadata/devleop branch commit 164151f
• Need testing on a different machine (Althea's)

# test duckdb and KNB function
# 2025-05-05

source(process_knb_to_duckdb.R)
process_knb_to_duckdb(resource_map_id = "resource_map_urn:uuid:a7bb0df6-7ce3-4319-8db7-a2a92c8aeca8")

• Need to add unit testing (Althea found this blog "Why Good Developers Write Bad Unit Tests"
• R package testing chapter

Althea local test results

> process_knb_to_duckdb(resource_map_id = "resource_map_urn:uuid:a7bb0df6-7ce3-4319-8db7-a2a92c8aeca8")
❌ Failed to download (HTTP): trade.zip Error: download from 'https://knb.ecoinformatics.org/knb/d1/mn/v2/object/urn:uuid:34e544cf-bc32-4919-b143-74d5c111c44e' failed 
✅ DuckDB created at ARTIS_SAU.duckdb
Warning messages:
1: In download.file(url, destfile = filepath, mode = "wb", quiet = TRUE) :
  URL 'https://knb.ecoinformatics.org/knb/d1/mn/v2/object/urn:uuid:34e544cf-bc32-4919-b143-74d5c111c44e': Timeout of 600 seconds was reached
2: In unzip(zip_file, exdir = local_path) :
  error 1 in extracting from zip file