Profile: Improve step-size defaults and step-size robustness

[Doresic]

Absolute profiling step sizes can be too small for parameters with very wide bounds. This PR updates profiling step-size handling so that pyPESTO can choose between absolute and relative step sizes per profiled parameter.

Relative step sizes are computed as fractions of the parameter span ub - lb on the optimization scale. pyPESTO then uses either the full absolute or the full relative step-size family, based on the resolved default step size.

Also:

• rename the absolute step-size options to make them explicit
• keep the old option names with deprecation warnings
• allow disabling a step-size family by setting its default step size to 0
• require finite profiling bounds
• resolve profile step sizes once before profiling and pass them downstream
• add a small precheck warning for setups that may require many profile steps
• adjust the related tests

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 92.36641% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.31%. Comparing base (541d5d7) to head (2bd0854).

Files with missing lines	Patch %	Lines
pypesto/profile/options.py	86.66%	6 Missing ⚠️
pypesto/profile/util.py	93.54%	4 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1711      +/-   ##
===========================================
+ Coverage    84.28%   84.31%   +0.03%     
===========================================
  Files          164      164              
  Lines        14645    14752     +107     
===========================================
+ Hits         12343    12438      +95     
- Misses        2302     2314      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Doresic]

Thanks @dweindl for the comments, I updated the step-size handling quite a bit so here's a short summary:

• relative step sizes are now applied on the optimization scale for all parameter scales
• step-size options are split into explicit absolute/relative names
• old absolute option names are kept with deprecation warnings
• a step-size family can be disabled by setting its default step size to 0
• pyPESTO now resolves one full step-size family per parameter instead of mixing absolute/relative min/default/max values
• step sizes are resolved once before profile walking and passed downstream
• the precheck docs/message were shortened and made more explicit
• the np.isclose bound checks in the line search were replaced by directional <= / >= checks
• tests were trimmed to cover the new behavior

[dweindl]

As the goal here is improving robustness, I am wondering if the absolute step sizes shouldn't be much smaller. Or disabled by default?! For a parameter in [1e-4, 1e-3], the current defaults wouldn't be too helpful. Do you see a case where just relying on relative tolerances would be problematic?

[Doresic]

The only downside I see of using only relative stepping is that it would take unnecessary many steps for very tightly constrained parameters, e.g., if there is a parameter with range 0-2 (some exponent).

The absolute step sizes I chose here were chosen based on the profiling benchmark -- as a tradeoff between robustness and speed. So I would keep them as such. But you do raise a good point -- actually the relative defaults should be different, better thought through. Generally, the median range of parameters the benchmark spanned 8 orders of magnitude.

So making the relative method kick in when the span exceeds 8 would be appropriate. And then from then on it would make as many steps as it would make for parameters with span 8. That would make the relative defaults be:

min_step_size_relative = 0.00125
default_step_size_relative = 0.0025
max_step_size_relative = 0.025

So for very small ranges the absolute method would cause the step size to be appropriate and finish the profiling quickly. For ones exceeding 8 orders of magnitude, the relative stepping would kick in which would cause all profiling to have per default step size 400 steps, and per min step size (worse case scenario) 800 steps.

[PaulJonasJost]

I am not fully clear on the usefulness of the Resolved Step size, i.e. adding data class, multiple checks at multiple points etc, for it to check two simple calculations of "how many steps are expected" and "How many in a worst case?". Have I missed a crucial functionality for them?

[Doresic]

I am not fully clear on the usefulness of the Resolved Step size, i.e. adding data class, multiple checks at multiple points etc, for it to check two simple calculations of "how many steps are expected" and "How many in a worst case?". Have I missed a crucial functionality for them?

It's more just an implementational thing. It causes for very simple downstream coding as we can call resolved_steps.min_step_size and we can pass only one object for all step sizes, instead of many defaults and flags and if statements.