diff --git a/docs/_toc.yml b/docs/_toc.yml index 5686cb83f..f88fc8252 100644 --- a/docs/_toc.yml +++ b/docs/_toc.yml @@ -14,8 +14,8 @@ chapters: - file: fundamentals/regularization/rotated_gradients - file: fundamentals/mesh_design - file: fundamentals/joint_inversion - - file: fundamentals/depth_of_investigation - file: fundamentals/optimization + - file: fundamentals/parallelization - file: tutorials/introduction sections: - file: tutorials/background @@ -34,10 +34,10 @@ chapters: - file: case_studies/Forrestania/python_code/unconstrained_gravity_inv_training - file: case_studies/Forrestania/python_code/unconstrained_magnetics_inv_training - file: case_studies/Forrestania/python_code/joint_grav_mag -- file: plate-simulation/index +- file: fundamentals/depth_of_investigation +- file: plate-simulation/simulation sections: - - file: plate-simulation/usage - title: Basic Usage - - file: plate-simulation/methodology - title: Methodology + - file: plate-simulation/standalone + - file: plate-simulation/sweep + - file: plate-simulation/match - file: THIRD_PARTY_SOFTWARE diff --git a/docs/conf.py b/docs/conf.py index 2f18464f2..ed45e1180 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -42,6 +42,7 @@ exclude_patterns = [] todo_include_todos = True + # -- Options for auto-doc ---------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#module-sphinx.ext.autodoc diff --git a/docs/fundamentals/depth_of_investigation.ipynb b/docs/fundamentals/depth_of_investigation.ipynb index 7f32087fc..4fe1abaa5 100644 --- a/docs/fundamentals/depth_of_investigation.ipynb +++ b/docs/fundamentals/depth_of_investigation.ipynb @@ -7,7 +7,58 @@ "source": [ "# Depth of Investigation\n", "\n", - "In geophysics, \"depth of investigation\" refers to the maximum depth below the surface from which a geophysical survey can reliably measure. It depends on factors like the survey design and physical properties of the subsurface material. Several strategies have been proposed to assess uncertainties in models recovered from inversion. {cite:t}`nabighian_1989` used a skin depth approach for electromagnetic surveys, assuming a background halfspace resistivity. {cite:t}`li_1999` implemented a cut-off value based on two inverted models obtained with slightly different assumptions. {cite:t}`christiansen_2012` proposed a mask based on the sum-square of sensitivities to estimate a volume of low confidence. In the following, we discuss the algorithm and implementation of the sensitivity cutoff strategy." + "This application allows users to compute a depth of investigation from model values. In geophysics, \"depth of investigation\" refers to the maximum depth below the surface from which a geophysical survey can reliably measure. It depends on factors like the survey design and physical properties of the subsurface material. \n", + "\n", + "![masked_model](./images/masked_model.png)\n", + "\n", + "Several strategies have been proposed to assess uncertainties in models recovered from inversion. {cite:t}`nabighian_1989` used a skin depth approach for electromagnetic surveys, assuming a background halfspace resistivity. {cite:t}`li_1999` implemented a cut-off value based on two inverted models obtained with slightly different assumptions. \n", + "\n", + "This application uses the method proposed by {cite:t}`christiansen_2012`, based on the sum-square of sensitivities to estimate a volume of low confidence." + ] + }, + { + "cell_type": "markdown", + "id": "83b4ae8f-5509-40b1-98c2-39acefba438c", + "metadata": { + "jp-MarkdownHeadingCollapsed": true + }, + "source": [ + "## Interface\n", + "\n", + "The depth of investigation methods based on sensitivity cutoffs relies on the ui.json interface. \n", + "\n", + "![interface](./images/uijson.png)\n", + "\n", + "### Inputs\n", + "\n", + "- **Mesh**: The mesh used for inversion\n", + "- **Sensitivity**: Model selector for the sensitivities at a given iteration (see [Pre-requisite](pre-requisite))\n", + "- **Cut-off**: Percentage value to threshold the sentivities, relative to the **Method** used.\n", + "- **Method**: One of *percentile*, *percent* or *log percent*\n", + "\n", + "![cutoff_options](./images/cutoff_options.png)\n", + "\n", + "### Output\n", + "\n", + "After running, the application will create a masking attribute saved on the mesh.\n", + "\n", + "![mask](./images/sensitivity_mask.png)\n", + "\n", + "The mask can then be applied to any of the iterations to show only the cells that exceeded the sensitivity threshold.\n", + "\n", + "![apply_mask](./images/apply_mask.png)\n", + "\n", + "\n", + "(pre-requisite)=\n", + "### Pre-requisite\n", + "\n", + "In order to save the sensitivities during a SimPEG inversion, the 'Save sensitivities' option must be selected from the 'Optional parameters' tab of an inversion.\n", + "\n", + "![save_sensitivities](./images/save_sensitivities.png)\n", + "\n", + "This will result in a new model generated and saved into the computational mesh at each iteration.\n", + "\n", + "![sensitivity_models](./images/sensitivity_models.png)" ] }, { @@ -15,7 +66,7 @@ "id": "c13ffcec-8d95-4ee9-876f-9aa92e2c534a", "metadata": {}, "source": [ - "## Sensitivities\n", + "## Methodology\n", "\n", "The sensitivity matrix is calculated as part of the optimization problem solved by SimPEG while inverting geophysical data. The sensitivity matrix represents the degree to which each predicted datum changes with respect to a perturbation in each model cell. It is given in matrix form by\n", "\n", @@ -28,12 +79,12 @@ "The depth of investigation mask is a property of the cells of the mesh only so the rows of the sensitivity matrix (data) are sum-square normalized as follows.\n", "\n", "$$\n", - "\\mathbf{J}_{norm} = \\left[\\sum_{n=1}^{N}\\left(\\frac{\\mathbf{J}_{n:}}{w_n}\\right)^{2}\\right]^{(1/2)}\n", + "\\|\\mathbf{j}\\| = \\left[\\sum_{n=1}^{N}\\left(\\frac{\\mathbf{J}_{n:}}{w_n}\\right)^{2}\\right]^{(1/2)}\n", "$$\n", "\n", "where $w_n$ are the data uncertainties associated with the $n^{th}$ datum.\n", "\n", - "The resulting vector $J_{norm}$ can then be thought of as the degree to which the aggregate data changes due to a small perturbation in each model cell. The depth of investigation mask is then computed by thresholding those sensitivities" + "The resulting vector $\\|\\mathbf{j}\\|$ can then be thought of as the degree to which the aggregate data changes due to a small perturbation in each model cell. The depth of investigation mask is then computed by thresholding those sensitivities" ] }, { @@ -41,7 +92,7 @@ "id": "d8bb85c5-5e63-4617-ab6b-a7b6536c1d2c", "metadata": {}, "source": [ - "## Thresholding\n", + "### Thresholding\n", "\n", "The depth of investigation can be estimated by assigning a threshold on the sum-squared sensitivity vector. This can be done as a percentile, percentage, or log-percentage. In the percentile method, the mask is formed by eliminating all cells in which the sensitivity falls below the lowest $n$% of the number of data where $n$ is the chosen cutoff. In the percent method the data are transformed into a percentage of the largest value\n", "\n", @@ -51,56 +102,6 @@ "\n", "and the mask is formed by eliminating all cells in which the sensitivity falls below the lowest $n$% of the data values where $n$ is the chosen cutoff. Finally, the log-percent mask transforms the data into log-space before carrying out the percentage thresholding described above." ] - }, - { - "cell_type": "markdown", - "id": "83b4ae8f-5509-40b1-98c2-39acefba438c", - "metadata": {}, - "source": [ - "## Usage\n", - "\n", - "The depth of investigation methods based on sensitivity cutoffs described above are exposed to Geoscience ANALYST Pro Geophysics users through a ui.json interface. In order to save the sensitivities during a SimPEG inversion, the 'Save sensitivities' option must be selected from the 'Optional parameters' tab of the SimPEG inversion ui.json window.\n", - "\n", - "![save_sensitivities](./images/save_sensitivities.png)\n", - "\n", - "This will result in a new model generated and saved into the computational mesh at each iteration.\n", - "\n", - "![sensitivity_models](./images/sensitivity_models.png)\n", - "\n", - "The ui.json interface allows the user to select a mesh from the **simpeg-drivers** result and any of the generated sensitivity models, a cutoff threshold, method, and optional name for the output.\n", - "\n", - "![interface](./images/uijson.png)\n", - "\n", - "The cutoff methods are selectable in the dropdown\n", - "\n", - "![cutoff_options](./images/cutoff_options.png)\n", - "\n", - "Whatever the options, the application will create a sensitivity cutoff mask saved on the mesh\n", - "\n", - "![mask](./images/sensitivity_mask.png)\n", - "\n", - "which can then be applied to any of the iterations to show only the cells that exceeded the sensitivity threshold.\n", - "\n", - "![apply_mask](./images/apply_mask.png)\n", - "\n", - "![masked_model](./images/masked_model.png)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1079da0d-4ea2-4f00-b957-92199dd39cdb", - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "920f0637-4028-40c7-b2c6-5976f5e90afd", - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { @@ -119,7 +120,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.16" + "version": "3.10.19" } }, "nbformat": 4, diff --git a/docs/fundamentals/images/distributed_parallelization.svg b/docs/fundamentals/images/distributed_parallelization.svg new file mode 100644 index 000000000..3292e58b1 --- /dev/null +++ b/docs/fundamentals/images/distributed_parallelization.svg @@ -0,0 +1 @@ + diff --git a/docs/fundamentals/parallelization.rst b/docs/fundamentals/parallelization.rst new file mode 100644 index 000000000..fd7dd779d --- /dev/null +++ b/docs/fundamentals/parallelization.rst @@ -0,0 +1,59 @@ +.. _parallelization: + +Parallelization +=============== + +This section describes the different levels of parallelization used by the inversion routines and how to optimize resources. +For a given inversion routine, the application decomposes the problem into a series of sub-problems, or tiles, each assigned a mesh and a survey. During the inversion process, the application continuously requests predicted data and derivatives from the sub-problems. The application parallelizes these operations within each sub-problem, as well as externally so that sub-problems can be computed concurrently. + + +.. figure:: ./images/distributed_parallelization.svg + :align: center + :width: 80% + + Schematic representation of the computing elements of a tiled inversion. Each tile receives an assigned mesh and a survey (single-frequency if applicable), with array operations parallelized by dask. The dask operations always bookend the direct solver when employed. Optionally, the workload can be distributed across multiple workers to optimize performance. Each worker is responsible for a sub-set of tiles and with a limited number of threads. Only 1-dimensional arrays are passed between the main process and the workers. + + +Dask +---- + +The `dask `_ library handles most operations related to generating arrays. A mixture of ``dask.arrays`` and ``dask.delayed`` calls parallelizes the computations across multiple threads. If a direct solver is involved, the dask operations bookend the solver to avoid thread-safety issues. The application converts dask.arrays to numpy arrays before passing them to the direct solvers and before returning them to the main process. Only 1-dimensional arrays are returned to the main process, while higher-dimensional arrays and solvers remain in the distributed memory of the workers. + +Sensitivity matrices can optionally be stored on disk using the `zarr `_ library, which is optimized for parallel read/write access. In this case, the workers never hold the entire sensitivity matrix in memory, but rather read and write small chunks of the matrix as needed. This allows for efficient handling of large sensitivity matrices that may not fit in memory, at the cost of increased disk I/O. The use of zarr is optional and can be enabled by setting the ``store_sensitivity`` parameter to true in the ui.json file. + + +Direct Solvers +-------------- + +A direct solver is used for any method evaluated by partial differential equation (PDE), such as electromagnetics and electric surveys. The `Pardiso `_ and `Mumps `_ solvers parallelize operations during the factorization and backward substitution calls. Note that the current implementation of the solvers is not thread-safe and therefore cannot be shared across parallel processes. Any other levels of parallelization need to occur outside the direct solver calls or be encapsulated within a distributed process. + +The number of threads used by the solvers can be set by running the command + +.. code-block:: + + set OMP_NUM_THREADS=X + +before launching the python program. Alternatively, setting ``OMP_NUM_THREADS`` as a local environment variable will set it permanently. The default value is the number of threads available on the machine. + + +.. _parallelization_distributed: + +Dask.distributed +---------------- + +It has been found that parallel processes, both for dask.delayed and the direct solver, tend to saturate on large numbers of threads. Too many small tasks tend to overwhelm the scheduler at the cost of performance. This can be alleviated by resorting to the ``dask.distributed`` library to split the computation of tiles across multiple ``workers``. Each worker can be responsible for a subset of the tiles, and can be configured to use a limited number of threads to optimize performance. This allows for better resource utilization and improved performance, especially when dealing with large problems that require significant computational resources. The number of workers and threads (per worker) can be set with the following parameters added to the ui.json file: + +.. code-block:: + + { + ... + "n_workers": i, + "n_threads": n, + "performance_report": true + } + +Where ``n_workers`` is the number of processes to spawn, and ``n_threads`` is the number of threads to use for each process. Setting ``performance_report`` to true will generate an ``html`` performance report at the end of the inversion, which can be used to identify bottlenecks and optimize the parallelization settings. + +It is good practice to set an even number of threads per worker to optimize the load. Setting too many workers with too few threads can lead to increased overhead from inter-process communication, while setting too few workers with too many threads can lead to saturation of the direct solvers and reduced performance. For example, if the machine has 32 threads available, setting 4 workers with 8 threads each will fully use the resources. + +It is also recommended to set the number of workers as a multiple of the number of tiles, to ensure that all workers are utilized. For example, if there are 8 tiles, setting 4 workers will allow each worker to process 2 tiles concurrently. If fewer tiles than workers are available, the program will automatically split surveys into smaller chunks, while preserving the mesh, to ensure even load across the workers. This is less efficient than having a dedicated optimized mesh per tile, but will still provide performance benefits. diff --git a/docs/images/plate-simulation/basic_usage/analyst_geophysics_menu.png b/docs/images/analyst_geophysics_menu.png similarity index 100% rename from docs/images/plate-simulation/basic_usage/analyst_geophysics_menu.png rename to docs/images/analyst_geophysics_menu.png diff --git a/docs/intro.md b/docs/intro.md index 83659657f..2407d34ea 100644 --- a/docs/intro.md +++ b/docs/intro.md @@ -1,16 +1,45 @@ # About -This document contains training material for geophysical inversions using [SimPEG](https://simpeg.xyz/) and [Geoscience ANALYST](https://www.mirageoscience.com/mining-industry-software/geoscience-analyst/). +This document contains training material for geophysical forward modeling and inversions using [SimPEG](https://simpeg.xyz/) and [Geoscience ANALYST](https://www.mirageoscience.com/mining-industry-software/geoscience-analyst/). ```{image} ./images/ore_body.png :width: 500px ``` -# Table of content + +# Table of contents ```{tableofcontents} ``` +# Running the applications + +The main entry point to the various modules are the [*.ui.json](https://github.com/MiraGeoscience/simpeg-drivers/blob/develop/simpeg_drivers-assets/uijson) +files. The ``ui.json`` serves a dual purpose: + +(1) rendering a user-interface in Geoscience ANALYST + +(2) storing the input parameters chosen by the user for the program to run. See the [UIJson documentation](https://mirageoscience-geoh5py.readthedocs-hosted.com/en/latest/content/uijson_format/usage.html) +for more information about the ui.json interface. + +The various user-interfaces can be accessed from the Geoscience ANALYST Pro Geophysics menu. + +```{image} ./images/analyst_geophysics_menu.png +:width: 500px +``` + +The application can also be run from command line if all required fields in +the ui.json are provided. This approach is useful for advanced users who want to +automate the mesh creation process or re-run an existing mesh with different parameters. + +To run any of the applications, assuming a valid Conda environment, use the following command: + +``conda activate simpeg-drivers`` + +``python -m simpeg-drivers.driver [YOUR.ui.json]`` + +where ``[YOUR.ui.json]`` is the path to the input ui.json file on disk. + # References diff --git a/docs/images/plate-simulation/index.png b/docs/plate-simulation/images/index.png similarity index 100% rename from docs/images/plate-simulation/index.png rename to docs/plate-simulation/images/index.png diff --git a/docs/plate-simulation/images/match_landing.png b/docs/plate-simulation/images/match_landing.png new file mode 100644 index 000000000..1a52955ba Binary files /dev/null and b/docs/plate-simulation/images/match_landing.png differ diff --git a/docs/plate-simulation/images/match_template.png b/docs/plate-simulation/images/match_template.png new file mode 100644 index 000000000..a31318705 Binary files /dev/null and b/docs/plate-simulation/images/match_template.png differ diff --git a/docs/plate-simulation/images/match_uijson.png b/docs/plate-simulation/images/match_uijson.png new file mode 100644 index 000000000..24085356f Binary files /dev/null and b/docs/plate-simulation/images/match_uijson.png differ diff --git a/docs/images/plate-simulation/methodology/copy_options.png b/docs/plate-simulation/images/methodology/copy_options.png similarity index 100% rename from docs/images/plate-simulation/methodology/copy_options.png rename to docs/plate-simulation/images/methodology/copy_options.png diff --git a/docs/images/plate-simulation/methodology/data/simpeg_group_creation.png b/docs/plate-simulation/images/methodology/data/simpeg_group_creation.png similarity index 100% rename from docs/images/plate-simulation/methodology/data/simpeg_group_creation.png rename to docs/plate-simulation/images/methodology/data/simpeg_group_creation.png diff --git a/docs/images/plate-simulation/methodology/data/simpeg_group_edit_options.png b/docs/plate-simulation/images/methodology/data/simpeg_group_edit_options.png similarity index 100% rename from docs/images/plate-simulation/methodology/data/simpeg_group_edit_options.png rename to docs/plate-simulation/images/methodology/data/simpeg_group_edit_options.png diff --git a/docs/images/plate-simulation/methodology/data/simpeg_group_options.png b/docs/plate-simulation/images/methodology/data/simpeg_group_options.png similarity index 100% rename from docs/images/plate-simulation/methodology/data/simpeg_group_options.png rename to docs/plate-simulation/images/methodology/data/simpeg_group_options.png diff --git a/docs/images/plate-simulation/methodology/data/simulation_options.png b/docs/plate-simulation/images/methodology/data/simulation_options.png similarity index 100% rename from docs/images/plate-simulation/methodology/data/simulation_options.png rename to docs/plate-simulation/images/methodology/data/simulation_options.png diff --git a/docs/images/plate-simulation/methodology/make_plate_images.pub b/docs/plate-simulation/images/methodology/make_plate_images.pub similarity index 100% rename from docs/images/plate-simulation/methodology/make_plate_images.pub rename to docs/plate-simulation/images/methodology/make_plate_images.pub diff --git a/docs/images/plate-simulation/methodology/mesh/mesh_options.png b/docs/plate-simulation/images/methodology/mesh/mesh_options.png similarity index 100% rename from docs/images/plate-simulation/methodology/mesh/mesh_options.png rename to docs/plate-simulation/images/methodology/mesh/mesh_options.png diff --git a/docs/images/plate-simulation/methodology/mesh/refinement.png b/docs/plate-simulation/images/methodology/mesh/refinement.png similarity index 100% rename from docs/images/plate-simulation/methodology/mesh/refinement.png rename to docs/plate-simulation/images/methodology/mesh/refinement.png diff --git a/docs/images/plate-simulation/methodology/model/basement_options.png b/docs/plate-simulation/images/methodology/model/basement_options.png similarity index 100% rename from docs/images/plate-simulation/methodology/model/basement_options.png rename to docs/plate-simulation/images/methodology/model/basement_options.png diff --git a/docs/images/plate-simulation/methodology/model/n_plates_options.png b/docs/plate-simulation/images/methodology/model/n_plates_options.png similarity index 100% rename from docs/images/plate-simulation/methodology/model/n_plates_options.png rename to docs/plate-simulation/images/methodology/model/n_plates_options.png diff --git a/docs/images/plate-simulation/methodology/model/overburden_and_basement.png b/docs/plate-simulation/images/methodology/model/overburden_and_basement.png similarity index 100% rename from docs/images/plate-simulation/methodology/model/overburden_and_basement.png rename to docs/plate-simulation/images/methodology/model/overburden_and_basement.png diff --git a/docs/images/plate-simulation/methodology/model/overburden_options.png b/docs/plate-simulation/images/methodology/model/overburden_options.png similarity index 100% rename from docs/images/plate-simulation/methodology/model/overburden_options.png rename to docs/plate-simulation/images/methodology/model/overburden_options.png diff --git a/docs/images/plate-simulation/methodology/model/plate_location.png b/docs/plate-simulation/images/methodology/model/plate_location.png similarity index 100% rename from docs/images/plate-simulation/methodology/model/plate_location.png rename to docs/plate-simulation/images/methodology/model/plate_location.png diff --git a/docs/images/plate-simulation/methodology/model/plate_location_options.png b/docs/plate-simulation/images/methodology/model/plate_location_options.png similarity index 100% rename from docs/images/plate-simulation/methodology/model/plate_location_options.png rename to docs/plate-simulation/images/methodology/model/plate_location_options.png diff --git a/docs/images/plate-simulation/methodology/model/plate_options.png b/docs/plate-simulation/images/methodology/model/plate_options.png similarity index 100% rename from docs/images/plate-simulation/methodology/model/plate_options.png rename to docs/plate-simulation/images/methodology/model/plate_options.png diff --git a/docs/images/plate-simulation/methodology/model/plate_orientation.png b/docs/plate-simulation/images/methodology/model/plate_orientation.png similarity index 100% rename from docs/images/plate-simulation/methodology/model/plate_orientation.png rename to docs/plate-simulation/images/methodology/model/plate_orientation.png diff --git a/docs/images/plate-simulation/methodology/model/plate_orientation_options.png b/docs/plate-simulation/images/methodology/model/plate_orientation_options.png similarity index 100% rename from docs/images/plate-simulation/methodology/model/plate_orientation_options.png rename to docs/plate-simulation/images/methodology/model/plate_orientation_options.png diff --git a/docs/images/plate-simulation/methodology/model/plate_resistivity_option.png b/docs/plate-simulation/images/methodology/model/plate_resistivity_option.png similarity index 100% rename from docs/images/plate-simulation/methodology/model/plate_resistivity_option.png rename to docs/plate-simulation/images/methodology/model/plate_resistivity_option.png diff --git a/docs/images/plate-simulation/methodology/model/plate_size.png b/docs/plate-simulation/images/methodology/model/plate_size.png similarity index 100% rename from docs/images/plate-simulation/methodology/model/plate_size.png rename to docs/plate-simulation/images/methodology/model/plate_size.png diff --git a/docs/images/plate-simulation/methodology/model/plate_size_options.png b/docs/plate-simulation/images/methodology/model/plate_size_options.png similarity index 100% rename from docs/images/plate-simulation/methodology/model/plate_size_options.png rename to docs/plate-simulation/images/methodology/model/plate_size_options.png diff --git a/docs/images/plate-simulation/methodology/model/three_plates.png b/docs/plate-simulation/images/methodology/model/three_plates.png similarity index 100% rename from docs/images/plate-simulation/methodology/model/three_plates.png rename to docs/plate-simulation/images/methodology/model/three_plates.png diff --git a/docs/images/plate-simulation/methodology/results.png b/docs/plate-simulation/images/methodology/results.png similarity index 100% rename from docs/images/plate-simulation/methodology/results.png rename to docs/plate-simulation/images/methodology/results.png diff --git a/docs/images/plate-simulation/methodology/sweep_option.png b/docs/plate-simulation/images/methodology/sweep_option.png similarity index 100% rename from docs/images/plate-simulation/methodology/sweep_option.png rename to docs/plate-simulation/images/methodology/sweep_option.png diff --git a/docs/images/plate-simulation/methodology/uijson.png b/docs/plate-simulation/images/methodology/uijson.png similarity index 100% rename from docs/images/plate-simulation/methodology/uijson.png rename to docs/plate-simulation/images/methodology/uijson.png diff --git a/docs/plate-simulation/images/summary.png b/docs/plate-simulation/images/summary.png new file mode 100644 index 000000000..8517c2bdb Binary files /dev/null and b/docs/plate-simulation/images/summary.png differ diff --git a/docs/plate-simulation/images/sweep_landing.png b/docs/plate-simulation/images/sweep_landing.png new file mode 100644 index 000000000..59874636c Binary files /dev/null and b/docs/plate-simulation/images/sweep_landing.png differ diff --git a/docs/plate-simulation/images/sweep_uijson.png b/docs/plate-simulation/images/sweep_uijson.png new file mode 100644 index 000000000..6a3bff526 Binary files /dev/null and b/docs/plate-simulation/images/sweep_uijson.png differ diff --git a/docs/plate-simulation/match.rst b/docs/plate-simulation/match.rst new file mode 100644 index 000000000..c44dad3a9 --- /dev/null +++ b/docs/plate-simulation/match.rst @@ -0,0 +1,133 @@ +.. _plate_simulation_match: + +Plate Matching +============== + +The plate matching algorithm searches through a library of pre-computed plate simulations to identify the best-fit conductor geometry for observed electromagnetic data at query locations. The matching process involves spatial and temporal interpolation, data normalization, and correlation-based scoring to rank candidate simulations. + +.. figure:: /plate-simulation/images/match_landing.png + :align: center + :width: 500pt + + + +Interface +--------- + +This section describes the parameters controlling the application. + +.. figure:: /plate-simulation/images/match_uijson.png + :align: center + :width: 300pt + + *Rendered user-interface in Geoscience ANALYST.* + + +Inputs +^^^^^^ + +The algorithm operates on the following inputs: + +- **Survey**: A geophysical survey object containing the airborne TEM measurements and associated metadata +- **EM data**: Airborne time-domain electromagnetic (TEM) measurements from the survey +- **Query Points**: User-defined positions where plate parameters are to be estimated +- **Query Max Distance**: Maximum distance from the query points along the survey line segments for consideration in the matching process +- **Strike angle**: Optional angle to correct for the strike of the conductor relative to the survey line direction +- **Topography**: Object representing the earth-air interface +- **Elevation channel**: Optional channel containing elevation data for the topography locations. Mainly used for digital elevation models provided on Grid2D objects. +- **Simulation directory**: A directory of pre-computed forward simulations representing various plate geometries and conductivities + + +Methodology +----------- + +For each query location, the application identifies the closest survey line segment, extracts relevant observed data, and compares it against all simulations in the library using a multi-step matching process. To accommodate for variable position, time sampling, it is expected that the simulated library contains sufficient variability in plate parameters and survey geometries to capture the range of possible responses. + +.. figure:: /plate-simulation/images/match_template.png + :align: center + :width: 500pt + + Template for the simulated data (black) and observed data (red) used in the matching process. The radial pattern centered on the plate allows to tighten sampling around the peak response, which is critical for accurate matching. + + +The following sections describe the key steps of the matching algorithm. + + +Spatial Interpolation +^^^^^^^^^^^^^^^^^^^^^ + +The application performs spatial interpolation to align simulation data with observation points: + +1. **Local coordinate transformation**: The application identifies the survey line segment nearest to each query location. It transforms both the observed and simulated receiver locations into a local polar coordinate system (distance, azimuth, height) centered on the line segment. + +2. **Strike angle correction**: If strike angles are provided, the application rotates the coordinate system to align the plate orientation with the survey geometry. This accounts for variations in conductor strike relative to the flight line direction. + +3. **Inverse distance weighting**: The application constructs a sparse interpolation matrix using the 8 nearest simulation receivers for each observation point. The weights are computed using inverse distance weighting with a power of 2.0 and a minimum distance threshold of 0.1 to prevent singularities. + +Temporal Interpolation +^^^^^^^^^^^^^^^^^^^^^^ + +Time-domain electromagnetic data requires careful handling of the temporal sampling: + +1. **Time channel alignment**: The application identifies the overlap between simulated and observed time channels. Only time gates within the simulated range are considered for matching. + +2. **Interpolation matrix**: For each observation time within the valid range, the application constructs a sparse interpolation matrix using inverse distance weighting based on the temporal separation between channels. This allows for comparison even when simulated and observed time samplings differ. + +Data Normalization +^^^^^^^^^^^^^^^^^^ + +To enable robust comparison across different conductivity models and survey geometries, the application normalizes both observed and simulated data: + +1. **Symlog transformation**: The application applies a symmetric logarithmic transformation to handle the large dynamic range of electromagnetic decay curves. The threshold is set at the 5th percentile of absolute data values. + +2. **Median centering**: The application subtracts the median value from each time channel to remove DC offsets. + +3. **Amplitude scaling**: The application scales each data profile to unit maximum absolute value, ensuring that shape matching dominates over amplitude matching. + +Orientation Detection +^^^^^^^^^^^^^^^^^^^^^ + +Electromagnetic anomalies are strongly dependent on the direction of approach relative to the conductor dip. Since simulations are only computed in the down-dip direction, the application automatically detects and corrects for orientation reversals: + +1. **Spatial integration**: The application computes the integral of positive anomaly values on each half of the survey segment. + +2. **Asymmetry test**: If more than 50% of the time channels show stronger anomalies on the left side of the segment, the application infers an up-dip approach and reverses the spatial ordering of the data. + +Scoring and Ranking +^^^^^^^^^^^^^^^^^^^ + +For each simulation file in the library, the application computes a match score: + +1. **Cross-correlation**: For each time channel, the application computes the normalized cross-correlation between observed and simulated data. The peak correlation indicates the optimal spatial alignment. + +2. **Alignment consistency**: The application records the index of maximum correlation for each time channel and computes the median alignment position across all channels. + +3. **Residual norm**: The application computes the L2 norm of the difference between observed and amplitude-scaled simulated data across all time channels and spatial locations. This serves as the primary score metric, with lower values indicating better matches. + +4. **Ranking**: The application sorts all simulations by ascending score, with the lowest-scoring simulation representing the best match. + +Plate Parameter Extraction +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Once the best-matching simulation is identified, the application extracts the plate parameters from the simulation metadata: + +1. **Parameter retrieval**: The application reads the conductor geometry (strike length, dip length, width, dip angle, conductivity) from the ui.json metadata stored in the simulation file. + +2. **Spatial positioning**: The application positions the plate at the centroid of the matched survey segment, adjusted vertically by the overburden thickness. + +3. **Orientation correction**: The application applies strike angle corrections based on the local line azimuth and any user-specified strike angle offsets. + +4. **Plate creation**: The application creates a new ``MaxwellPlate`` object in the output workspace with the extracted geometry and stores the full model parameters in the plate metadata. + +Parallel Processing +^^^^^^^^^^^^^^^^^^^ + +The matching algorithm leverages parallel processing to accelerate the comparison of large simulation libraries: + +1. **File batching**: The application divides the simulation library into batches, with approximately 10 batches per worker process. + +2. **Distributed scoring**: Each batch is submitted to a Dask worker for independent scoring. Workers compute spatial and temporal interpolations, normalize data, and calculate correlation scores without inter-process communication. + +3. **Result aggregation**: The main process collects scores from all workers and performs the final ranking to identify the best match for each query location. + +This parallel architecture allows efficient processing of libraries containing hundreds to thousands of pre-computed simulations. diff --git a/docs/plate-simulation/methodology.rst b/docs/plate-simulation/methodology.rst deleted file mode 100644 index 87bbeeea5..000000000 --- a/docs/plate-simulation/methodology.rst +++ /dev/null @@ -1,274 +0,0 @@ -.. _plate_simulation_methodology: - -Plate Simulation: Methodology -============================= - -In order to simulate geophysical data from a physical property model, we -need three things: a computational mesh, a discretization of the model -within that mesh and a means to simulate the data. Plate simulation -relies on `discretize `_ for -octree mesh creation, and `SimPEG `_ for finite -volume based forward modeling. Plate simulation includes a module for -generating a simple two-layer model with embedded plate anomalies within -octree meshes. In this section, we will discuss all three of these -components, their interface exposed by the ui.json file, and the storage -of results. - -.. figure:: /images/plate-simulation/methodology/uijson.png - :align: center - :width: 80% - - *Merged images of both tabs of the ui.json rendered interface.* - -.. contents:: - -.. toctree:: - :maxdepth: 3 - -Octree Mesh ------------ - -In order to accurately simulate our earth model, we need a mesh -that is refined in key areas, while being coarse enough elsewhere to -efficiently simulate data. The plate simulation package includes -refinements at the earth-air interface, the transmitter and receiver -sites and on the surface of plates. - -.. figure:: /images/plate-simulation/methodology/mesh/refinement.png - :align: center - :width: 100% - - *Octree mesh refinement for earth-air interface, receiver sites, - and within the mesh.* - -The meshing can be controlled by options exposed in the ui.json. -Those options are significantly reduced compared with the octree creation from -`grid-app `_ -since we have tailored many of the parameters to suit the needs of plate simulation. - -.. figure:: /images/plate-simulation/methodology/mesh/mesh_options.png - :align: center - - *Octree mesh parameters exposed in the ui.json.* - -Geological Model ----------------- - -The plate simulation package includes a module for generating -plate(s) embedded in a two-layer Earth model within octree meshes. -There are many permutations of this simple geological scenario -leading to a complex interface. To simplify things, we have -broken the discussion into two sub-sections: background -(basement and overburden) and plates. - -Background -~~~~~~~~~~ - -All model values within plate-simulation are to be provided in -ohm-metres. The basement resistivity is actually closer to a -halfspace in the sense that it fills the model anywhere outside -of the overburden and plate. So the basement resistivity should -be chosen as an effective resistivity for the whole geological -section. This should be quite reasonable for most applications -where the differences in resistivity between layers is much smaller -than the difference between overburden and any anomalous bodies -(plates). - -.. figure:: /images/plate-simulation/methodology/model/basement_options.png - :align: center - - *Basement resistivity option.* - -The overburden is discretized by the resistivity and thickness -of the layer. The thickness is referenced to the earth-air -interface and extends into the earth by the amount specified -in the thickness parameter. - -.. figure:: /images/plate-simulation/methodology/model/overburden_options.png - :align: center - - *Overburden resistivity and thickness options.* - -.. figure:: /images/plate-simulation/methodology/model/overburden_and_basement.png - :align: center - :width: 100% - - *Model section highlighting the overburden and basement boundary.* - -Plates -~~~~~~ - -In this section we will discuss the various plate options available -through the ui.json and their impact on the resulting discretized -model. - -.. figure:: /images/plate-simulation/methodology/model/plate_options.png - :align: center - - *Plate options available in the ui.json.* - -The first set of options allows the user to specify the number of -plates and their spacing. - -.. figure:: /images/plate-simulation/methodology/model/n_plates_options.png - :align: center - - *Number of plates and spacing options.* - -For all choices of ``n>1``, the plates will be evenly spaced at the requested -spacing and will share the same resistivity, size and orientation. - -.. figure:: /images/plate-simulation/methodology/model/three_plates.png - :align: center - :width: 100% - - *Model created by choosing three plates spaced at 200m.* - -The plate resistivity is expected to be entered in ohm-metres. - -.. figure:: /images/plate-simulation/methodology/model/plate_resistivity_option.png - :align: center - - *Plate resistivity option.* - -The size of the plate is given as a 'thickness', 'strike length', and -'dip length'. - -.. figure:: /images/plate-simulation/methodology/model/plate_size_options.png - :align: center - - *Plate size options.* - -The image below shows a dipping plate with annotations showing the size -parameters for that particular plate. - -.. figure:: /images/plate-simulation/methodology/model/plate_size.png - :align: center - :width: 100% - - *A dipping plate striking northeast with annotations for its thickness, - strike length and dip length.* - -The orientation of the plate is provided in terms of a dip and dip direction. -The dip is defined as the angle between the horizontal projection of the plate -normal and the plate tangent sharing the same origin. The dip direction is -measured between the horizontal projection of the plate normal and the North -arrow. See the image below for a visual representation of these angles. - -.. figure:: /images/plate-simulation/methodology/model/plate_orientation.png - :align: center - :width: 100% - - *Plate orientation options. Plate orientation is given as a dip and dip direction. - The dip (b) is defined as the angle between the horizontal the projection of the - plate normal (n\') and the plate tangent sharing the same origin (t). The dip - direction (a) is the angle measured between the horizontal projection of the plate - normal (n\') and due north (N).* - -The location of the plate can be provided in both relative and absolute terms. -The position parameters are given as an easting, northing, and elevation. If the -relative locations checkbox is chosen, then the easting and northing will be -relative to the center of the survey and the elevation will be relative to one of -the available references. The elevation may either be referenced to the earth-air -interface or the overburden provided by the ``Depth reference`` dropdown. Either of -these choice can be relative to the minimum, maximum, or mean of the points making -up the reference surface as given by the ``Reference type`` dropdown. In all of these -cases the distance provided will act as a depth below the reference to the *top of -plate* in the *z negative down* convention. If the relative locations checkbox is not -chosen, then the easting, northing, and elevation is simply the location of the -center of the plate. - -.. figure:: /images/plate-simulation/methodology/model/plate_location_options.png - :align: center - - *Plate location options in relative mode. Notice the* ``Elevation`` *is given as - negative to ensure the top of the plate is below the selected min of the - overburden.* - -.. figure:: /images/plate-simulation/methodology/model/plate_location.png - :align: center - :width: 100% - - *Example of a relative elevation referenced 100m below the minimum of the - overburden layer.* - -Data Simulation ---------------- - -.. _simpeg_group_options: - -The simulation parameters control the forward modeling of the plate model -discretized within the octree mesh. Rather than exposing the parameters within -the plate simulation interface all over again, we simply allow the user to -select an existing forward modelling SimPEG group. It is expected that the -user will have already edited those options and provided at least a topography -and survey object as well as selected one or more components to simulate. The -user may also provide a name to the new SimPEG group that will be used to store -the results. - - -.. figure:: /images/plate-simulation/methodology/data/simpeg_group_options.png - :align: center - - *Selecting the initialized forward modelling SimPEG group and naming the - group that will store the plate simulation results.* - -The required SimPEG group can be created within Geoscience ANALYST through the -``Geophysics`` menu under ``SimPEG Python Interface`` entry. - -.. figure:: /images/plate-simulation/methodology/data/simpeg_group_creation.png - :align: center - :width: 100% - - *Creating a SimPEG group to be selected within the plate simulation interface.* - -Once created, the options can be edited by right-clicking the group and choosing -the 'Edit Options' entry. - -.. figure:: /images/plate-simulation/methodology/data/simpeg_group_edit_options.png - :align: center - - *Editing the SimPEG group options.* - -Since plate-simulation will create its own mesh and model, the mesh and -conductivity selections can be ignored. Selecting a value will not conflict -with the plate-simulation objects and will simply be ignored. In contrast, the -survey, topography and at least one component must be selected to run the simulation. - -.. figure:: /images/plate-simulation/methodology/data/simulation_options.png - :align: center - :width: 80% - - *Simulation options with annotations for required and not required components.* - -Results -------- - -The results of the simulation are stored in the SimPEG group named in the -:ref:`simpeg group option ` section. - -.. figure:: /images/plate-simulation/methodology/results.png - :align: center - - *Results group containing a survey object with all the simulated data channels - stored in property groups, and an octree mesh containing the model parameterized - in the interface.* - -To iterate on the design of experiment, simply copy the options, edit, and -run again. - -.. figure:: /images/plate-simulation/methodology/copy_options.png - :align: center - - *Copying the options to run a new simulation.* - -If the user wishes to sweep one or more of the input parameters to run a large number of -simulations, they can use the ``generate sweep file`` option to write a file used -by the `param-sweeps `_ package to do just -that. It is beyond the scope of this document to discuss the use of that package; -refer to its README for further details. - -.. figure:: /images/plate-simulation/methodology/sweep_option.png - :align: center - - *Generating a sweep file to run multiple simulations.* diff --git a/docs/plate-simulation/index.rst b/docs/plate-simulation/simulation.rst similarity index 75% rename from docs/plate-simulation/index.rst rename to docs/plate-simulation/simulation.rst index eb8b29512..9bbd6608d 100644 --- a/docs/plate-simulation/index.rst +++ b/docs/plate-simulation/simulation.rst @@ -1,12 +1,8 @@ -.. _plate_simulation_index: +.. _plate_simulation: Plate Simulation ================ -.. figure:: /images/plate-simulation/index.png - :align: center - :width: 100% - The plate-simulation application is a tool for simulating geophysical data over a simple two-layer earth model with plate(s). It relies on the `discretize `_ @@ -15,7 +11,14 @@ simulate data over the parameterized model. The mesh, model and simulation details are parameterized in a ui.json file that can be rendered in `Geoscience ANALYST Pro `_. -See: +.. figure:: /plate-simulation/images/index.png + :align: center + :width: 500pt + + +.. toctree:: + :maxdepth: 1 -- :ref:`Basic Usage ` -- :ref:`Methodology ` + standalone + sweep + match diff --git a/docs/plate-simulation/standalone.rst b/docs/plate-simulation/standalone.rst new file mode 100644 index 000000000..d2122498b --- /dev/null +++ b/docs/plate-simulation/standalone.rst @@ -0,0 +1,246 @@ +.. _plate_simulation_standalone: + +Interface +========= + +Simulating geophysical data from a physical property model requires three things: +a computational mesh, a discretization of the model within that mesh, and a means +to simulate the data. Plate simulation includes a module for generating a simple +two-layer model with embedded plate anomalies within octree meshes. This section +discusses all three of these components, their interface exposed by the ui.json file, +and the storage of results. + +.. figure:: /plate-simulation/images/methodology/uijson.png + :align: center + :width: 300pt + + *Merged images of both tabs of the ui.json rendered interface.* + + +Geological Model +---------------- + +Plate simulation includes a module for generating plates embedded in a two-layer +Earth model within octree meshes. Many permutations of this simple geological +scenario result in a complex interface. To simplify this, the discussion is +organized into two sub-sections: background (basement and overburden) and plates. +All model values within plate-simulation must be provided in SI units that varies depending on the chosen forward simulation (g/cc, SI or Ohm.m) + +Background +~~~~~~~~~~ + +The basement resistivity is actually closer to a halfspace in the sense that it +fills the model anywhere outside of the overburden and plate. Therefore, the +basement resistivity should be chosen as an effective resistivity for the whole +geological section. This approach is quite reasonable for most applications +where the differences in resistivity between layers are much smaller than the +difference between overburden and any anomalous bodies (plates). + +.. figure:: /plate-simulation/images/methodology/model/basement_options.png + :align: center + + *Basement resistivity option.* + +The overburden is discretized by the resistivity and thickness of the layer. +The thickness is referenced to the earth-air interface and extends into the +earth by the amount specified in the thickness parameter. + +.. figure:: /plate-simulation/images/methodology/model/overburden_options.png + :align: center + + *Overburden resistivity and thickness options.* + +.. figure:: /plate-simulation/images/methodology/model/overburden_and_basement.png + :align: center + :width: 500pt + + *Model section highlighting the overburden and basement boundary.* + +Plates +~~~~~~ + +This section discusses the various plate options available through the ui.json +and their impact on the resulting discretized model. + +.. figure:: /plate-simulation/images/methodology/model/plate_options.png + :align: center + + *Plate options available in the ui.json.* + +The first set of options allows the user to specify the number of plates and +their spacing. + +.. figure:: /plate-simulation/images/methodology/model/n_plates_options.png + :align: center + + *Number of plates and spacing options.* + +For all choices of ``n>1``, the plates are evenly spaced at the requested spacing +and share the same resistivity, size, and orientation. + +.. figure:: /plate-simulation/images/methodology/model/three_plates.png + :align: center + :width: 500pt + + *Model created by choosing three plates spaced at 200m.* + +The plate resistivity must be entered in SI units (g/cc, SI or Ohm.m). + +.. figure:: /plate-simulation/images/methodology/model/plate_resistivity_option.png + :align: center + + *Plate resistivity option.* + +The size of the plate is defined by three parameters: thickness, strike length, +and dip length. + +.. figure:: /plate-simulation/images/methodology/model/plate_size_options.png + :align: center + + *Plate size options.* + +The image below shows a dipping plate with annotations indicating the size +parameters for that particular plate. + +.. figure:: /plate-simulation/images/methodology/model/plate_size.png + :align: center + :width: 500pt + + *A dipping plate striking northeast with annotations for its thickness, + strike length and dip length.* + +The plate orientation is defined in terms of dip and dip direction. The dip +is the angle between the horizontal projection of the plate normal and the +plate tangent sharing the same origin. The dip direction is measured between +the horizontal projection of the plate normal and the North arrow. The image +below provides a visual representation of these angles. + +.. figure:: /plate-simulation/images/methodology/model/plate_orientation.png + :align: center + :width: 500pt + + *Plate orientation options. Plate orientation is given as a dip and dip direction. + The dip (b) is defined as the angle between the horizontal the projection of the + plate normal (n\') and the plate tangent sharing the same origin (t). The dip + direction (a) is the angle measured between the horizontal projection of the plate + normal (n\') and due north (N).* + +The plate location can be specified in both relative and absolute terms. The position +parameters are given as easting, northing, and elevation. If the relative locations +checkbox is selected, the easting and northing are relative to the center of the +survey and the elevation is relative to one of the available references. The elevation +may be referenced to either the earth-air interface or the overburden via the +``Depth reference`` dropdown. Either choice can be relative to the minimum, maximum, +or mean of the points making up the reference surface as given by the ``Reference type`` +dropdown. In all these cases, the distance provided acts as a depth below the reference +to the *top of plate* in the *z negative down* convention. If the relative locations +checkbox is not selected, the easting, northing, and elevation specify the center +location of the plate. + +.. figure:: /plate-simulation/images/methodology/model/plate_location_options.png + :align: center + + *Plate location options in relative mode. Notice the* ``Elevation`` *is given as + negative to ensure the top of the plate is below the selected min of the + overburden.* + +.. figure:: /plate-simulation/images/methodology/model/plate_location.png + :align: center + :width: 500pt + + *Example of a relative elevation referenced 100m below the minimum of the + overburden layer.* + +Data Simulation +--------------- + +.. _simpeg_group_options: + +The simulation parameters control the forward modeling of the plate model +discretized within the octree mesh. Rather than exposing parameters within +the plate simulation interface, the application allows the user to select an +existing forward modelling SimPEG group. The user must ensure that the SimPEG +group has been previously edited with appropriate options, includes at least a +topography and survey object, and has selected one or more components to simulate. +The user may also provide a name for the new SimPEG group to store the results. + +.. figure:: /plate-simulation/images/methodology/data/simpeg_group_options.png + :align: center + + *Selecting the initialized forward modelling SimPEG group and naming the + group that will store the plate simulation results.* + +Create the required SimPEG group within Geoscience ANALYST through the +``Geophysics`` menu under ``SimPEG Python Interface`` entry. + +.. figure:: /plate-simulation/images/methodology/data/simpeg_group_creation.png + :align: center + :width: 300pt + + *Creating a SimPEG group to be selected within the plate simulation interface.* + +Edit the options by right-clicking the group and selecting 'Edit Options'. + +.. figure:: /plate-simulation/images/methodology/data/simpeg_group_edit_options.png + :align: center + :witdht: 300pt + + *Editing the SimPEG group options.* + +Since plate-simulation creates its own mesh and model, the mesh and conductivity +selections can be ignored. Selecting a value does not conflict with the +plate-simulation objects and is simply ignored. In contrast, the survey, topography, +and at least one component must be selected to run the simulation. + +.. figure:: /plate-simulation/images/methodology/data/simulation_options.png + :align: center + :width: 300pt + + *Simulation options with annotations for required and not required components.* + +Octree Mesh +----------- + +To accurately simulate the earth model, the mesh must be refined in key areas +while remaining coarse enough elsewhere to efficiently simulate data. Plate +simulation includes refinements at the earth-air interface, the transmitter and +receiver sites, and on the surface of plates. + +.. figure:: /plate-simulation/images/methodology/mesh/refinement.png + :align: center + :width: 500pt + + *Octree mesh refinement for earth-air interface, receiver sites, + and within the mesh.* + +The meshing is controlled by options exposed in the ui.json. These options are +significantly reduced compared with octree creation from `grid-app `_, +as many parameters have been tailored to suit the needs of plate simulation. + +.. figure:: /plate-simulation/images/methodology/mesh/mesh_options.png + :align: center + :width: 300pt + + *Octree mesh parameters exposed in the ui.json.* + +Results +------- + +The results of the simulation are stored in the SimPEG group named in the +:ref:`simpeg group option ` section. + +.. figure:: /plate-simulation/images/methodology/results.png + :align: center + :width: 300pt + + *Results group containing a survey object with all the simulated data channels + stored in property groups, and an octree mesh containing the model parameterized + in the interface.* + +To iterate on the design of experiment, copy the options, edit them, and run again. + +.. figure:: /plate-simulation/images/methodology/copy_options.png + :align: center + :width: 300pt + + *Copying the options to run a new simulation.* diff --git a/docs/plate-simulation/sweep.rst b/docs/plate-simulation/sweep.rst new file mode 100644 index 000000000..857cc859d --- /dev/null +++ b/docs/plate-simulation/sweep.rst @@ -0,0 +1,48 @@ +.. _plate_simulation_batch: + +Batch Simulations +================= + +The Plate Sweep module provides a user interface for generating and running a batch of simulations by sweeping one or more of the input parameters. The user can select which parameters to sweep and the range of values for each parameter. + +.. figure:: /plate-simulation/images/sweep_landing.png + :align: center + :width: 500pt + +The following sections describe the user interface, inputs, and methodology of the Plate Sweep module. + + +Interface +--------- + +This section describes the parameters controlling the application. + +.. figure:: /plate-simulation/images/sweep_uijson.png + :align: center + :width: 300pt + + *Rendered user-interface in Geoscience ANALYST.* + +Inputs +^^^^^^ + +- **Plate simulation**: A Plate Simulation group that contains the input parameters for a single plate simulation, as well as the connection to a SimPEG Forward group. Parameters that are not included in the sweep will be taken from this group and used for all simulations. +- **Output directory**: A directory where the results of each simulation will be stored. Each simulation will be saved in a separate ``*.geoh5`` file named with a unique identifier. The directory is created if it does not exist, otherwise simulations are appended to it. +- **Generate summary file**: A boolean option to generate a summary file in the output directory. The summary file is a ``*.xls`` file that contains the input parameters and results of each simulation, allowing users to easily sort over the range of simulation parameters. +- **[Optional] Parameter blocks**: For every parameter of `Plate Simulation `_, users can choose a **starting**, **ending** and **number of samples** values to sweep over. The application will generate a simulation for each value in the range, while keeping all other parameters constant. If a parameter is not included in the sweep, the value set in the input Plate Simulation group will be used for all simulations. + + +Methodology +----------- + +After loading the input parameters from the Plate Simulation group, the application generates a list of parameter combinations based on the specified sweep ranges. For each combination of parameters, a unique identifier is generated using a hash system. If this unique identifier already exists in the ``Output directory``, the simulation is skipped. Otherwise a copy of the original input ``geoh5`` is created. + +The target file is then opened and the Plate Simulation group is modified with the corresponding parameters before running the simulation. The results are saved in the target ``geoh5`` file, and the process is repeated for the next combination of parameters until all combinations have been processed. + +If the `dask.distributed `_ library is enabled, the simulations are run in parallel using a local cluster. Otherwise, the simulations are run sequentially. + +Finally, if the option to generate a summary file is enabled, a routine extracts parameters from all ``*.geoh5`` files present in the ``Output director`` and tabulates them in a ``summary.xls`` file for easy reference and analysis. + +.. figure:: /plate-simulation/images/summary.png + :align: center + :width: 500pt diff --git a/docs/plate-simulation/usage.rst b/docs/plate-simulation/usage.rst deleted file mode 100644 index f940b6ca8..000000000 --- a/docs/plate-simulation/usage.rst +++ /dev/null @@ -1,35 +0,0 @@ -.. _plate_simulation_usage: - -Plate Simulation: Basic usage -============================= - -The main entry points to the various modules is the `plate_simulation.ui.json `_ -file. The ``ui.json`` has the dual purpose of (1) rendering a user-interface from -Geoscience ANALYST and (2) storing the input parameters chosen by the user for the -program to run. To learn more about the ui.json interface visit the -`UIJson documentation `_ page. - - -User-interface --------------- - -The user-interface is accessible from Geoscience ANALYST Pro Geophysics menu. - -.. figure:: /images/plate-simulation/basic_usage/analyst_geophysics_menu.png - :align: center - :width: 800 - - -From command line ------------------ - -The application can also be run from the command line if all required fields in the ui.json are provided. -This is useful for more advanced users wanting to automate the mesh creation process or re-run an existing mesh with different parameters. - -To run the application from the command line, use the following command in a Conda Prompt: - -``conda activate plate-simulation`` - -``python -m plate-simulation.driver input_file.json`` - -where ``input_file.json`` is the path to the input file on disk. diff --git a/tests/plate_simulation/runtest/sweep_test.py b/tests/plate_simulation/runtest/sweep_test.py index 2a765c781..9977c378c 100644 --- a/tests/plate_simulation/runtest/sweep_test.py +++ b/tests/plate_simulation/runtest/sweep_test.py @@ -7,7 +7,6 @@ # (see LICENSE file at the root of this source code package). ' # ' # ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' -from io import BytesIO from geoh5py import Workspace from geoh5py.groups import SimPEGGroup @@ -105,10 +104,6 @@ def test_sweep(tmp_path): n = len(list(workdir.glob("*.geoh5"))) assert n == 6 - # Check that the summary file was created and has the expected number of rows - with Workspace(tmp_path / "test.geoh5") as ws: - file_data = ws.get_entity("summary.xlsx")[0] - blob = BytesIO(file_data.file_bytes) - xls = read_excel(blob) + xls = read_excel(tmp_path / "summary.xlsx") assert len(xls) == 6 diff --git a/tests/run_tests/driver_airborne_tem_1d_test.py b/tests/run_tests/driver_airborne_tem_1d_test.py index b36be0d96..a4c1878b4 100644 --- a/tests/run_tests/driver_airborne_tem_1d_test.py +++ b/tests/run_tests/driver_airborne_tem_1d_test.py @@ -38,7 +38,7 @@ # To test the full run and validate the inversion. # Move this file out of the test directory and run. -target_run = {"data_norm": 6.413150962712146e-10, "phi_d": 24, "phi_m": 205000} +target_run = {"data_norm": 6.413150955633971e-10, "phi_d": 42.3, "phi_m": 116000} def test_airborne_tem_1d_fwr_run(