diff --git a/docs/source/effects/custom_effects.md b/docs/source/effects/custom_effects.md new file mode 100644 index 00000000..07cbfb61 --- /dev/null +++ b/docs/source/effects/custom_effects.md @@ -0,0 +1,334 @@ +--- +jupytext: + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 (ipykernel) + language: python + name: python3 +--- + +# Creating Custom Effects + +ScopeSim's built-in effects cover the most common physical phenomena in optical +systems, but you may need to model instrument-specific behaviour that isn't +provided out of the box. Creating a custom `Effect` subclass lets you inject +arbitrary transformations into the simulation pipeline. + +For a worked example that creates a `PointSourceJitter` effect and adds it to +a full MICADO simulation, see the +[Custom Effects Example Notebook](../examples/3_custom_effects.ipynb). +This page focuses on a complementary example – a non-symmetric vignetting flat +field applied at the image plane level. + +## Anatomy of an Effect Subclass + +Every custom effect needs three things: + +1. **`z_order`** – a class variable (tuple of ints) that tells ScopeSim *when* + in the pipeline to apply the effect. +2. **`__init__`** – calls `super().__init__()` and sets default parameters in + `self.meta`. +3. **`apply_to(self, obj)`** – the method that does the work. It receives an + object, optionally modifies it, and **must return it**. + +The `apply_to` method should use `isinstance` checks to determine whether to +act on the given object. During a simulation run, ScopeSim passes different +object types at different stages – your effect will only modify the types it +knows how to handle, and pass everything else through unchanged. + +## Choosing the Right Z-Order + +The z_order determines which pipeline stage your effect participates in, and +therefore what type of object it receives: + +| Z-Order Range | Object Type | Use When... | +|:---:|---|---| +| 500–599 | `Source` | Modifying the original light distribution (e.g., spectral shifts, flux scaling) | +| 600–699 | `FieldOfView` | Modifying per-wavelength spatial cutouts (e.g., PSF convolution, dispersion) | +| 700–799 | `ImagePlane` | Modifying the wavelength-integrated focal plane image (e.g., vignetting, flat fields) | +| 800–899 | `Detector` | Modifying the detector readout (e.g., noise, dark current, gain variations) | + +An effect can have multiple z_order values to participate in both a setup stage +and an application stage. For a simple custom effect, a single value is +usually sufficient. + +## Example: Non-Symmetric Vignetting Flat Field + +This example creates an effect that applies a spatially-varying throughput +pattern to the image plane, simulating optical vignetting that is not radially +symmetric – for instance, caused by an off-axis obstruction or asymmetric optics. + +The vignetting is modelled as an elliptical Gaussian decay with configurable +center offset, semi-axes, rotation angle, and throughput range. + +### Defining the effect class + +```{code-cell} ipython3 +import numpy as np +from scopesim.effects import Effect +from scopesim.optics.image_plane import ImagePlane + + +class NonSymmetricVignetting(Effect): + """Apply a non-symmetric vignetting pattern to the image plane.""" + + z_order = (710,) + + def __init__(self, **kwargs): + super().__init__(**kwargs) + params = { + "x_center_offset": 0.1, # fractional offset from image center + "y_center_offset": -0.05, + "sigma_x": 0.8, # fractional semi-axis (1.0 = full frame) + "sigma_y": 0.6, + "rotation_deg": 15.0, # rotation angle of the vignetting ellipse + "max_throughput": 1.0, + "min_throughput": 0.3, + } + for key, val in params.items(): + self.meta.setdefault(key, val) + self.meta.update(kwargs) + + def _make_vignetting_map(self, shape): + """Generate a 2D vignetting map for a given image shape.""" + ny, nx = shape + y, x = np.mgrid[:ny, :nx] + + # Normalise pixel coordinates to [-1, 1] and apply center offset + x_norm = 2.0 * x / nx - 1.0 - self.meta["x_center_offset"] + y_norm = 2.0 * y / ny - 1.0 - self.meta["y_center_offset"] + + # Rotate coordinate frame + angle = np.deg2rad(self.meta["rotation_deg"]) + cos_a, sin_a = np.cos(angle), np.sin(angle) + x_rot = x_norm * cos_a + y_norm * sin_a + y_rot = -x_norm * sin_a + y_norm * cos_a + + # Elliptical Gaussian falloff + r2 = (x_rot / self.meta["sigma_x"]) ** 2 + \ + (y_rot / self.meta["sigma_y"]) ** 2 + t_min = self.meta["min_throughput"] + t_max = self.meta["max_throughput"] + vmap = t_min + (t_max - t_min) * np.exp(-0.5 * r2) + + return np.clip(vmap, t_min, t_max) + + def apply_to(self, obj, **kwargs): + if isinstance(obj, ImagePlane): + vignetting = self._make_vignetting_map(obj.hdu.data.shape) + obj.hdu.data *= vignetting + return obj +``` + +### Setting up the simulation + +```{code-cell} ipython3 +import scopesim as sim +from scopesim.source.source_templates import star_field + +# Load the example optical train and create a star field source +opt = sim.load_example_optical_train() +src = star_field(n=50, mmin=15, mmax=20, width=200) + +# Create and add the vignetting effect +vig = NonSymmetricVignetting(name="asymmetric_vignetting") +opt.optics_manager.add_effect(vig) + +opt.effects +``` + +### Observing and visualising + +```{code-cell} ipython3 +import matplotlib.pyplot as plt + +opt.observe(src, update=True) + +fig, axes = plt.subplots(1, 2, figsize=(14, 5)) + +# Show the vignetted image +axes[0].imshow(opt.image_planes[0].data, origin="lower") +axes[0].set_title("Image plane with vignetting") + +# Show the vignetting map itself +vmap = vig._make_vignetting_map(opt.image_planes[0].data.shape) +im = axes[1].imshow(vmap, origin="lower", cmap="RdYlGn", vmin=0, vmax=1) +axes[1].set_title("Vignetting map (throughput)") +fig.colorbar(im, ax=axes[1]) + +plt.tight_layout() +plt.show() +``` + +### Comparing with and without vignetting + +```{code-cell} ipython3 +# Observe without vignetting +vig.include = False +opt.observe(src, update=True) +no_vig_data = opt.image_planes[0].data.copy() + +# Observe with vignetting +vig.include = True +opt.observe(src, update=True) +vig_data = opt.image_planes[0].data.copy() + +fig, axes = plt.subplots(1, 2, figsize=(14, 5)) +axes[0].imshow(no_vig_data, origin="lower") +axes[0].set_title("Without vignetting") +axes[1].imshow(vig_data, origin="lower") +axes[1].set_title("With vignetting") +plt.tight_layout() +plt.show() +``` + +## Modifying Parameters at Runtime + +Effect parameters live in the `.meta` dictionary and can be changed between +observations: + +```{code-cell} ipython3 +# Make the vignetting more extreme +opt["asymmetric_vignetting"].meta["sigma_x"] = 0.4 +opt["asymmetric_vignetting"].meta["sigma_y"] = 0.3 +opt["asymmetric_vignetting"].meta["min_throughput"] = 0.1 + +opt.observe(src, update=True) + +fig, axes = plt.subplots(1, 2, figsize=(14, 5)) +axes[0].imshow(opt.image_planes[0].data, origin="lower") +axes[0].set_title("Tighter vignetting") + +vmap = vig._make_vignetting_map(opt.image_planes[0].data.shape) +im = axes[1].imshow(vmap, origin="lower", cmap="RdYlGn", vmin=0, vmax=1) +axes[1].set_title("Updated vignetting map") +fig.colorbar(im, ax=axes[1]) +plt.tight_layout() +plt.show() +``` + +## Tips for Writing Robust Effects + +- **Always return `obj`** from `apply_to`, even when your `isinstance` check + doesn't match. ScopeSim passes many object types through the same list of + effects – returning `None` will break the pipeline. + +- **Use `isinstance` guards** to decide whether to act. Your `apply_to` will + be called with `Source`, `FieldOfView`, `ImagePlane`, and `Detector` objects + at different stages. + +- **Choose the right pipeline stage** carefully: + - `FieldOfView` (z=600–699): your effect is applied per wavelength bin and + per spatial chunk – appropriate for wavelength-dependent effects. + - `ImagePlane` (z=700–799): your effect sees the wavelength-integrated focal + plane image – appropriate for achromatic spatial effects like vignetting. + - `Detector` (z=800–899): your effect sees the detector readout after + extraction – appropriate for electronic effects like noise. + +- **Look at built-in effects for patterns.** For example, + `PixelResponseNonUniformity` in `scopesim/effects/electronic/noise.py` is a + simple multiplicative detector-level effect. `SeeingPSF` in + `scopesim/effects/psfs/analytical.py` shows how to build a convolution kernel. + +- **Use `from_currsys`** for parameters that should be resolvable as bang + strings (`!OBS.some_param`): + ```python + from scopesim.utils import from_currsys + value = from_currsys(self.meta["my_param"], self.cmds) + ``` + +## Adding Custom Effects to the Optical Train + +Custom effects are added programmatically using `optics_manager.add_effect()`: + +```python +my_effect = MyCustomEffect(name="my_effect", some_param=42) +opt.optics_manager.add_effect(my_effect) +``` + +After adding an effect, pass `update=True` to `opt.observe()` so the optical +train rebuilds its internal structures to include the new effect. + +Note that YAML-based instrument packages resolve effect class names from the +`scopesim.effects` namespace. Custom effect classes from third-party packages +currently need to be added programmatically as shown above. + +## Sharing Your Custom Effect + +Once you've written and tested a custom effect, there are several ways to make +it available for use – either for yourself or for the wider community. + +### Option 1: Add it directly to the ScopeSim effects module (local) + +If you want your effect to be available via YAML instrument packages (i.e., +referenced by class name in a YAML file), the simplest approach is to place +your Python file inside the `scopesim/effects/` directory of your local +ScopeSim installation and import it in `scopesim/effects/__init__.py`. + +For example, if you save your effect class in +`scopesim/effects/my_vignetting.py`: + +```python +# scopesim/effects/my_vignetting.py +from .effects import Effect + +class NonSymmetricVignetting(Effect): + ... +``` + +Then add the import to `scopesim/effects/__init__.py`: + +```python +from .my_vignetting import * +``` + +After this, the class name `NonSymmetricVignetting` can be used directly in +YAML configuration files: + +```yaml +effects: + - name: vignetting + class: NonSymmetricVignetting + kwargs: + sigma_x: 0.8 + sigma_y: 0.6 +``` + +Note that this modifies your local ScopeSim installation and will be +overwritten when you upgrade the package. For a more permanent solution, +consider contributing it upstream (Option 3). + +### Option 2: Keep it in your own script or package + +For effects that are specific to your analysis, you can keep the effect class +in your own Python script or package and add it programmatically at runtime as +shown [above](#adding-custom-effects-to-the-optical-train). This is the +simplest approach and doesn't require modifying ScopeSim itself. + +### Option 3: Contribute it to ScopeSim + +If your effect is general-purpose and would be useful to other users, we +welcome contributions! You can: + +- **Open an issue** on the + [ScopeSim GitHub repository](https://github.com/AstarVienna/ScopeSim/issues) + describing your effect and sharing the code. The ScopeSim team can help + integrate it into the package. + +- **Submit a pull request** with your effect class added to the + `scopesim/effects/` module, including the import in `__init__.py` and + ideally a test in `scopesim/tests/`. See the + [existing effects](https://github.com/AstarVienna/ScopeSim/tree/main/scopesim/effects) + for examples of the expected code style and structure. + +## See Also + +- [Effects Overview](overview.md) – reference for all built-in effect types + and the simulation pipeline +- [Custom Effects Example Notebook](../examples/3_custom_effects.ipynb) – a + worked example with `PointSourceJitter` and MICADO +- The auto-generated [API Reference for scopesim.effects.Effect](../_autosummary/scopesim.effects.html) diff --git a/docs/source/effects/index.rst b/docs/source/effects/index.rst new file mode 100644 index 00000000..cc9e4dea --- /dev/null +++ b/docs/source/effects/index.rst @@ -0,0 +1,9 @@ +Effects +======= + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + overview + custom_effects diff --git a/docs/source/effects/overview.md b/docs/source/effects/overview.md new file mode 100644 index 00000000..d695377c --- /dev/null +++ b/docs/source/effects/overview.md @@ -0,0 +1,310 @@ +--- +jupytext: + text_representation: + extension: .md + format_name: myst + format_version: 0.13 + jupytext_version: 1.16.1 +kernelspec: + display_name: Python 3 (ipykernel) + language: python + name: python3 +--- + +# Effects Overview + +In ScopeSim, **Effects** are the building blocks of an optical system simulation. +Each `Effect` object represents a single physical phenomenon – atmospheric +seeing, mirror reflectivity, filter transmission, detector noise, and so on. +An `OpticalTrain` collects all the effects from an instrument package and +applies them in sequence to transform a `Source` (the on-sky light distribution) +into a realistic detector readout. + +Effects are typically defined in YAML instrument configuration files, but can +also be created and added programmatically. Each effect implements an +`apply_to(obj)` method that receives an object at a specific stage of the +simulation pipeline and returns the modified object. + +## Listing Effects in an Optical Train + +To see all effects loaded in an optical train, use the `.effects` attribute: + +```{code-cell} ipython3 +import scopesim as sim + +opt = sim.load_example_optical_train() +opt.effects +``` + +## The Simulation Pipeline + +ScopeSim processes effects in a strict order determined by each effect's +**z_order** — a numerical priority that maps to a specific pipeline stage. +Effects with lower z_order values run first. Each effect's z_order can contain +multiple values, allowing it to participate in more than one stage (e.g., setup +and application). + +The pipeline has three main phases: + +```{mermaid} +%%{init: {"theme": "dark"} }%% +flowchart LR + subgraph Setup ["Setup Phase"] + direction TB + S1["FOV Setup
z = 200..299"] + S2["Image Plane Setup
z = 300..399"] + S3["Detector Setup
z = 400..499"] + S1 --> S2 --> S3 + end + subgraph Observe ["observe() Phase"] + direction TB + O1["Source Effects
z = 500..599
TER curves, filters"] + O2["FOV Effects
z = 600..699
PSFs, spectral traces, shifts"] + O3["Image Plane Effects
z = 700..799
Vibration, flat fields"] + O1 --> O2 --> O3 + end + subgraph Readout ["readout() Phase"] + direction TB + R1["Detector Effects
z = 800..899
Noise, dark current, QE"] + R2["Detector Array Effects
z = 900..999
Exposure integration"] + R3["FITS Header Effects
z = 1000+"] + R1 --> R2 --> R3 + end + S3 --> O1 + O3 --> R1 +``` + +### Z-Order Reference + +| Z-Order Range | Pipeline Stage | Applied To | OpticsManager Property | +|:---:|---|---|---| +| 200–299 | FOV setup | `FovVolumeList` | `fov_setup_effects` | +| 300–399 | Image plane setup | `FovVolumeList` | `image_plane_setup_effects` | +| 400–499 | Detector setup | `FovVolumeList` | `detector_setup_effects` | +| 500–599 | Source effects | `Source` | `source_effects` | +| 600–699 | FOV effects | `FieldOfView` | `fov_effects` | +| 700–799 | Image plane effects | `ImagePlane` | `image_plane_effects` | +| 800–899 | Detector effects | `Detector` | `detector_effects` | +| 900–999 | Detector array effects | `Detector` | `detector_array_effects` | +| 1000+ | FITS header effects | `HDUList` | `fits_header_effects` | + +## YAML Configuration + +Effects are typically defined in YAML instrument packages. Each effect entry +specifies the class name and configuration parameters: + +```yaml +effects: + - name: detector_qe_curve + description: Quantum efficiency of the battery of detectors + class: QuantumEfficiencyCurve + kwargs: + filename: QE_detector_H2RG.dat + + - name: dark_current + description: Detector dark current + class: DarkCurrent + kwargs: + value: 0.1 # electrons/s/pixel + + - name: filter_wheel + class: FilterWheel + kwargs: + current_filter: "!OBS.filter_name" + filter_names: [J, H, Ks] + filename_format: "filters/TC_filter_{}.dat" +``` + +Parameters prefixed with `!` (called **bang strings**) are resolved dynamically +from the simulation configuration at runtime. For example, `!OBS.filter_name` +reads the current filter selection from the observation commands. + +```{note} +For real-world examples of YAML effect configurations, browse the instrument +packages in the [Instrument Reference Database (IRDB)](https://github.com/AstarVienna/irdb). +If you have instrument packages installed locally, you can also look inside the +`inst_pkgs/` folder in your ScopeSim data directory (see `scopesim.rc.__config__["!SIM.file.local_packages_path"]`). +``` + +## Interacting with Effects at Runtime + +Effects can be accessed, toggled, and modified after the optical train is loaded. + +### Enabling and disabling effects + +```{code-cell} ipython3 +# Turn off an effect +opt["detector_linearity"].include = False +print("detector_linearity included:", opt["detector_linearity"].include) + +# Turn it back on +opt["detector_linearity"].include = True +``` + +### Inspecting effect metadata + +```{code-cell} ipython3 +opt["dark_current"].meta +``` + +### Modifying parameters + +```{code-cell} ipython3 +# Change the dark current value +opt["dark_current"].meta["value"] = 0.5 +print("New dark current:", opt["dark_current"].meta["value"]) +``` + +For more tips on interacting with effects, see: +- [Turning Effects on or off](../5_liners/effects_include.md) +- [Using bang strings and hash strings](../5_liners/bang_strings.md) + +## Effect Categories + +### Transmission, Emission, and Reflection (TER) Curves + +TER curves describe how optical surfaces transmit, emit, and reflect light as +a function of wavelength. They are the most common type of effect and model +everything from mirror coatings to atmospheric transmission to detector quantum +efficiency. + +| Class | Description | +|---|---| +| `TERCurve` | Base wrapper for spectral transmission/emission/reflection curves | +| `SurfaceList` | Combines multiple optical surfaces into a system-level TER curve | +| `AtmosphericTERCurve` | Atmospheric transmission and emission | +| `SkycalcTERCurve` | Atmospheric TER from ESO's SkyCalc service | +| `QuantumEfficiencyCurve` | Detector quantum efficiency vs wavelength | +| `FilterCurve` | Bandpass filter transmission from file | +| `TopHatFilterCurve` | Rectangular (top-hat) filter transmission | +| `FilterWheel` | A wheel of selectable filters | +| `TopHatFilterWheel` | A wheel of selectable top-hat filters | +| `SpanishVOFilterCurve` | Filters from the Spanish Virtual Observatory | +| `DownloadableFilterCurve` | Filters downloadable from remote servers | +| `PupilTransmission` | Pupil plane transmission curves | +| `PupilMaskWheel` | Wheel of pupil masks | +| `ADCWheel` | Atmospheric Dispersion Corrector wheel | + +### Apertures and Field Masks + +Aperture effects define the on-sky field geometry — imaging windows, spectrograph +slits, and IFU fields. + +| Class | Description | +|---|---| +| `ApertureMask` | Defines on-sky window coordinates (imaging, slit, IFU, MOS) | +| `RectangularApertureMask` | Rectangular aperture variant | +| `ApertureList` | Container for multiple apertures | +| `SlitWheel` | A wheel of selectable slits | + +### Point Spread Functions (PSFs) + +PSF effects model the spatial blurring of point sources due to diffraction, +atmospheric seeing, and optical aberrations. + +| Class | Description | +|---|---| +| `Vibration` | Wavelength-independent Gaussian vibration kernel | +| `SeeingPSF` | Atmospheric seeing as a Gaussian kernel | +| `GaussianDiffractionPSF` | Diffraction-limited PSF (Gaussian approximation) | +| `NonCommonPathAberration` | NCPA PSF from wavefront error maps | +| `AnisocadoConstPSF` | SCAO PSF from AnisoCADO at a given Strehl ratio | +| `FieldConstantPSF` | Field-constant PSF loaded from a FITS file | +| `FieldVaryingPSF` | Field-varying PSF loaded from a FITS file | + +### Shifts and Atmospheric Dispersion + +Shift effects apply wavelength-dependent positional offsets to the light +distribution. + +| Class | Description | +|---|---| +| `Shift3D` | Base class for wavelength-dependent positional shifts | +| `AtmosphericDispersion` | Wavelength-dependent atmospheric refraction | +| `AtmosphericDispersionCorrection` | ADC correction for atmospheric dispersion | + +### Spectral Traces + +Spectral trace effects map 3D spectral data cubes onto the 2D detector plane +for spectrographic modes. + +| Class | Description | +|---|---| +| `SpectralTraceList` | Maps spectral cubes to detector plane via trace geometries | +| `SpectralTraceListWheel` | A wheel of selectable spectral trace configurations | +| `SpectralEfficiency` | Grating/dispersion efficiency (blaze function) | + +### Detector Geometry + +Detector geometry effects define the physical layout of detector chips on +the focal plane. + +| Class | Description | +|---|---| +| `DetectorList` | Detector chip positions, sizes, pixel scale, and rotation | +| `DetectorWindow` | A sub-region readout window on a detector | +| `DetectorList3D` | 3D detector array definition for spectroscopic modes | + +### Electronic and Noise Effects + +These effects model the detector electronics and noise sources that affect the +final readout. + +| Class | Description | +|---|---| +| `ShotNoise` | Poissonian photon noise | +| `DarkCurrent` | Thermal dark current | +| `Bias` | Constant bias level | +| `BasicReadoutNoise` | Generic readout noise | +| `PoorMansHxRGReadoutNoise` | Simplified HAWAII detector readout noise model | +| `LinearityCurve` | Detector linearity and saturation | +| `ADConversion` | Analog-to-digital conversion (electrons to ADU) | +| `PixelResponseNonUniformity` | Per-pixel gain variations (flat field) | +| `InterPixelCapacitance` | Inter-pixel capacitance crosstalk kernel | + +### Exposure and Readout + +Exposure effects handle integration time, auto-exposure, and readout formatting. + +| Class | Description | +|---|---| +| `AutoExposure` | Auto-determines DIT/NDIT from saturation limits | +| `ExposureIntegration` | Integrates signal over the exposure time | +| `ExposureOutput` | Formats the readout output | +| `DetectorModePropertiesSetter` | Sets mode-specific detector parameters (MINDIT, FULL_WELL, RON) | + +### Detector Pixel Effects + +| Class | Description | +|---|---| +| `BinnedImage` | Equal pixel binning | +| `UnequalBinnedImage` | Non-uniform pixel binning | +| `ReferencePixelBorder` | Masks reference pixels at detector edges | +| `Rotate90CCD` | Rotates CCD by integer multiples of 90 degrees | + +### Observing Strategies + +| Class | Description | +|---|---| +| `ChopNodCombiner` | Combines 4 chop/nod positions (AA, AB, BA, BB) | + +### FITS Header Effects + +| Class | Description | +|---|---| +| `ExtraFitsKeywords` | Adds custom FITS keywords to output headers | +| `EffectsMetaKeywords` | Adds effect metadata to FITS headers | +| `SourceDescriptionFitsKeywords` | Adds source description keywords | +| `SimulationConfigFitsKeywords` | Adds simulation configuration keywords | + +### Other + +| Class | Description | +|---|---| +| `Shutter` | Simulates a closed shutter (zeros all pixels) | + +## See Also + +- [Creating Custom Effects](custom_effects.md) — how to write your own Effect subclasses +- [Custom Effects Example Notebook](../examples/3_custom_effects.ipynb) — a worked example using MICADO and a PointSourceJitter effect +- The auto-generated [API Reference for scopesim.effects](../_autosummary/scopesim.effects.html) diff --git a/docs/source/index.rst b/docs/source/index.rst index 60250db7..c04cf645 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -30,6 +30,7 @@ ScopeSim_ is on pip:: getting_started examples/index + effects/index 5_liners/index faqs/index