Swapi proton moments

[hafarooki]

Change Summary

New response function for SWAPI + new proton/alpha fitting algorithm to use the new response function. See new Markdown document file included in the PR for extensive details about the changes.

[jtniehof]

Can you rebase or merge main into the PR and resolve conflicts?

[hafarooki]

Should be all good now

[pleasant-menlo]

We manage the Python dependencies in the Docker containers using uv. Since this PR adds a dependency on numba, can you install uv and run uv add numba, then commit the updated pyproject.toml and uv.lock files?

After making this change locally, we were able to run the integration test in Docker. It returned a failure:

AssertionError:
Not equal to tolerance rtol=0.01, atol=0
proton_sw_speed regression
Mismatched elements: 1 / 3 (33.3%)
Max absolute difference among violations: 30.87983496
Max relative difference among violations: 0.06362372
 ACTUAL: array([474.574554, 475.23233 , 516.230835])
 DESIRED: array([474.576, 475.232, 485.351])

Is this expected and the test just needs to be updated, or does it indicate an error in the code?

[hafarooki]

will do!

That is expected, I made some algorithm changes over the past few days and need to update the assertions

[hafarooki]

Just to double check, I plotted the values around that time and compared with another spacecraft and it does look like the correct speed should be 516, not 485. So I'll go ahead and update those tests

[hafarooki]

Fixed the integration test, please try again when you get a chance @pleasant-menlo

[pleasant-menlo]

We have created two pull requests to configure the SDS triggering system.

• Update SWAPI l3a dependencies for new algorithm sds-data-manager#1319
  • Adds the new ancillary, MAG, and SPICE requirements
• SWAPI remove unused ancillary dependencies sds-data-manager#1320
  • Removes the no-longer-used lookup tables; to be merged only after the new processing code is deployed.

Before getting the first one merged, we would like to upload the new ancillary files to production so that the current processor continues to be triggered. These are:

• azimuthal-transmission
• central-effective-area
• passband-fit-coefficients

Are the versions in this PR suitable, or would you like us to wait to upload until you produce finalized versions?

[hafarooki]

The current versions are thoroughly validated and not likely to change much if at all for the upcoming data release. I think you can go ahead and upload this version of them for now. Will it be possible to upload a new version with minor changes later on?

[pleasant-menlo]

We renamed the files to 20250924, following the conventions, and uploaded them to production.

If you have an update in the future that should apply over the whole mission, you can upload v002 with the same 20250924 start date (mission launch date). If an update should only be applied for data after a specific time, you can upload v002 with a later start date and it will only affect data from the specified time onward.

We ran the integration test in Docker with your latest changes, and it passed.

[pleasant-menlo]

Does tests.swapi.l3a.science.solar_wind.test_optimizer.TestOptimizeSolarWindParamsRecoversTruth pass for you? We are seeing the perturbed parameters returned directly without converging to the "true" values.

[hafarooki]

I just reran the full unit test suite and all 336 tests pass on my end.

[hafarooki]

Do you have the latest version pulled to your local clone?

[pleasant-menlo]

Running on commit 59647a1, which looks to be the latest.

I set up a little logging and it evaluates at a few points with oddly large velocity, then decides it can't get any better:

state=array([   1.70474809,   11.69524702,    0.        , -463.5       ,
          0.        ])
np.linalg.norm(residues)=np.float64(9424.992804857537)
==========
state=array([-2.35404529e+00,  1.21973802e+01,  1.40608869e+16, -5.28397482e+02,
        6.37225581e+02])
np.linalg.norm(residues)=np.float64(28438.78358766678)
==========
state=array([ 1.28223838e+00,  1.17907393e+01, -2.63240577e+16, -4.57281866e+02,
        6.17704532e+01])
np.linalg.norm(residues)=np.float64(28438.78358766678)
==========
state=array([ 1.68878202e+00,  1.17077522e+01, -8.18615819e+16, -4.60957050e+02,
        9.76050853e-01])
np.linalg.norm(residues)=np.float64(28438.78358766678)
==========
state=array([ 1.70296392e+00,  1.16967668e+01, -9.47344666e+15, -4.63253020e+02,
        6.15139279e-02])
np.linalg.norm(residues)=np.float64(28438.78358766678)
==========
state=array([   1.70474809,   11.69524702,    0.        , -463.5       ,
          0.        ])
np.linalg.norm(residues)=np.float64(9424.992804857537)
==========
`xtol` termination condition is satisfied.
Function evaluations 5, initial cost 4.4415e+07, final cost 4.4415e+07, first-order optimality 9.40e+07.

It works as expected if I use method="trf" in the optimize call, though.

[hafarooki]

That velocity looks weird, I'll take a closer look at the test to see if it's just some ridiculous misspecified initial condition.

method="trf" is slower so I'd rather avoid that route if possible.

[pleasant-menlo]

Another finding: if we pass x_scale=1 into optimize, it works. We saw the following in the scipy documentation:

Changed in version 1.16.0: The default keyword value is changed from 1 to None to indicate that a default approach to scaling is used. For the ‘lm’ method the default scaling is changed from 1 to ‘jac’. This has been found to give better performance, and is the same scaling as performed by leastsq.

Maybe you are using an older version of scipy?

[hafarooki]

Good catch! It looks like I have scipy 1.13 based on pyproject.toml. uv.lock has spicy 1.15 on python <3.11 and 1.17 for python >= 3.11. What version of Python and spicy should we actually be using?

[pleasant-menlo]

I believe we use 3.12 in the pipeline. If you use uv to set up the environment you can get the exact versions: either uv sync and activate the environment, or use uv run instead of python in command line invocations.

[hafarooki]

I was using conda before. I've switched to the uv environment for now with python 3.12. And now I can indeed reproduce the failure. Working to fix it now

[hafarooki]

(Perhaps we should delete the environment.yaml file in the repo? It looks like that's where I got my outdated dependencies from)

[hafarooki]

Should be all good now @pleasant-menlo

[pleasant-menlo]

Thanks, that made it work!

There are a few test failures in tests/test_utils.py where we construct a SwapiL3AlphaSolarWindData with the old parameters.

We have some Windows development machines, where "fork" does not work. It's not worth doing anything complicated, but could you apply something like @skipIf(platform.system() == "Windows", "fork is not available on Windows") to the TestParallelChunkRunnerOrchestration class?

Otherwise, this is looking pretty good.

[hafarooki]

should be fixed

[hafarooki]

Also wanted to raise the question about whether any of the large CSV files/test data added in this PR should be moved to Git LFS. Or kept locally cached instead of on Git. Here is a breakdown of the lines added:

• SVG figures (~43K)
• reference_integrals.csv (10,001)
• wind_solar_wind_samples_2025.csv (10,001)
• azimuthal-transmission CSV (1,802)
• passband-fit-coefficients CSV (1,147)
• central-effective-area CSV (63)

Any thoughts on this?

[pleasant-menlo]

One more thing I noticed when trying to plot a file: the variable attributes for the RTN velocities point to rtn_label as the DEPEND_1.

1. This rtn_label variable is missing from the CDF.
2. If it is a label, it should be specified in the LABL_PTR_1 attribute instead. Since this dimension 1 is vector components without meaningful associated numeric values, we don't need a DEPEND_1.

Example from SWE:

imap_L3_processing/imap_l3_processing/cdf/config/imap_swe_l3_variable_attrs.yaml

Lines 675 to 692 in 18fdd17

	core_velocity_vector_rtn_fit:
	NAME: core_velocity_vector_rtn_fit
	DATA_TYPE: CDF_REAL4
	CATDESC: Core velocity vector RTN fit
	DEPEND_0: epoch
	VAR_TYPE: data
	RECORD_VARYING: RV
	DISPLAY_TYPE: time_series
	VARIABLE_PURPOSE: PRIMARY_VAR,SUMMARY
	FIELDNAM: Core Velocity Vector RTN Fit
	FORMAT: F15.3
	UNITS: km/s
	VALIDMIN: -5000
	VALIDMAX: 5000
	FILLVAL: -1.00E+31
	LABL_PTR_1: rtn_label
	VAR_NOTES: Analytically calculated from Bi-Maxellian fit
	LABLAXIS: Core v vec rtn fit

[pleasant-menlo]

Also wanted to raise the question about whether any of the large CSV files/test data added in this PR should be moved to Git LFS. Or kept locally cached instead of on Git

I'm not particularly concerned about those files, they seem relatively small compared to what is already in the repository.

[hafarooki]

One more thing I noticed when trying to plot a file: the variable attributes for the RTN velocities point to rtn_label as the DEPEND_1.

1. This rtn_label variable is missing from the CDF.
2. If it is a label, it should be specified in the LABL_PTR_1 attribute instead. Since this dimension 1 is vector components without meaningful associated numeric values, we don't need a DEPEND_1.

Example from SWE:

imap_L3_processing/imap_l3_processing/cdf/config/imap_swe_l3_variable_attrs.yaml

Lines 675 to 692 in 18fdd17

	core_velocity_vector_rtn_fit:
	NAME: core_velocity_vector_rtn_fit
	DATA_TYPE: CDF_REAL4
	CATDESC: Core velocity vector RTN fit
	DEPEND_0: epoch
	VAR_TYPE: data
	RECORD_VARYING: RV
	DISPLAY_TYPE: time_series
	VARIABLE_PURPOSE: PRIMARY_VAR,SUMMARY
	FIELDNAM: Core Velocity Vector RTN Fit
	FORMAT: F15.3
	UNITS: km/s
	VALIDMIN: -5000
	VALIDMAX: 5000
	FILLVAL: -1.00E+31
	LABL_PTR_1: rtn_label
	VAR_NOTES: Analytically calculated from Bi-Maxellian fit
	LABLAXIS: Core v vec rtn fit

@jtniehof is this something that needs to be addressed at this stage or can that wait for the metadata validation stage?

[pleasant-menlo]

Good point, that's more of a metadata issue.

We went ahead and merged the PR. Thanks for all your work!

[hafarooki]

Thanks!

[pleasant-menlo]

@hafarooki FYI, we have deployed the new version of the code and reprocessed all proton-sw files.

[hafarooki]

Thanks! Took a look at a few days of data, seems like it worked OK. (We had to change the CDF file version to v001 to load it in CAVA, however)