Running into issues with SE++: getting stuck and WCS error

[mpiecka]

Hello everyone!

We are currently working on extracing sources from VIRCAM images for VISIONS survey in order to derive proper motions of stars. We already have some preliminary results based on SourceXtractor + SCAMP. Our idea was to get dimmer sources and hopefully also improve proper motion precision if we merge jitter images together, detect the sources there, and then measure on the original images. For this, we wanted to make use of SourceXtractor++, which seems to have the ability to do precisely this. However, we are running into two big problems.

1. For some of the images, there appears to be a crash during measurement. The error states something about WCS. The issue is fixed when we remove SIP coefficients from the header. However, we require SIP coefficients in order to obtain precise positions of the stars.

2. In general, SE++ appears to get stuck during measurements. This seems to occur randomly - sometimes for the same region the code gets stuck (and then crashes after several hours of not doing anything obvious), sometimes we get the full output catalogue. Removing SIP coefficient results in lower chances of this happening, but we need SE++ to run stably for a an extended period of time with SIP (we have hundreds of images to analyse). For example, the smallest region has 5 paw prints (each contains 16 images) and 6 epochs, so running into an issue while analysing the 3rd/4th image is a big issue, which is the best we were able to achieve during our tests.

Any suggestions about how to solve these problems would be appreciated! If it helps, when SE++ is most likely to get stuck, we are seeing the following error: LAPACK error: dgesdd (dbdsdc)/dgesvd (dbdsqr) failed to converge in dlevmar_pseudoinverse() [info=2]. This did not appear when analysing the control field (where we did not run into any issues). Except for including SIP coefficient in the header, the only difference between control field and the regions of interest is the number of sources (for our current paw prints slightly above 1e5, for the control field 3e4).

Furthermore, we noticed that during detection the code seems to slow down with time. For example, first 10% of detections are done in a minute or so, going from 10% to 20% takes about twice as long, and so on. This does not improve at higher %. Is this an expected behaviour of SE++?

We have prepared a script that makes our job a bit more easy. It is started by running "script_launch.sh", which rewrites some of the contents of the config files, but this only applies to file paths, everything else is the same as in the config files located in "DefaultFiles".
VISIONS_testing.zip

[mkuemmel]

On the slowing down, a reason could be the RAM given to the image data. The default is 512MB which may not be enough. Maybe increase this ("--tile-memory-limit") to one or two GB. This may also help with problem 2.

For the other problems, are you willing to share some example data? Then we could make some example runs.

[mkuemmel]

Also make sure to work with v0.19.2. Previous versions have a cast problem which limits the accuracy of world coordinates, which isn't good for your science case.

[mpiecka]

@mkuemmel Thanks for your suggestions and the quick reply! I tried increasing tile memory limit to 2048 but it did not help with the problems. I also checked the version - it is 0.19.2.

You can find the images for the first epoch here:

https://drive.google.com/file/d/1djVH_zRdT2mBsVD9qvkMwZnynpd25o07/view?usp=share_link

[mkuemmel]

Have the data, thanks!

Could you give me:

• the PSF file(s)?
• the SE++ call specific for this dataset?

I do have you scripts linked above, but every minute I don't have to re-engineer what's in them I can spend on your problem.

[mpiecka]

Please, find the required files in the attached zipfile.
SE++configs.zip

[mkuemmel]

I had a look into your data and can reproduce your problems.

Note that SIP transformation are non-linear, hence the world_to_sky projection, which is done in wcslib, is not guaranteed to work. Exactly this situation is indicated with:
2023-05-19T10:48:01CEST WCS ERROR : wcs.c:3368 wcss2p
2023-05-19T10:48:01CEST WCS ERROR : One or more of the world coordinates were invalid
2023-05-19T10:48:01CEST WCS ERROR : lin.c:956 linx2p
2023-05-19T10:48:01CEST WCS ERROR : De-distort error
2023-05-19T10:48:01CEST WCS ERROR : dis.c:1298 disx2p
2023-05-19T10:48:01CEST WCS ERROR : Convergence not achieved after 30 iterations, residual 1.8e-12

The error comes from wcslib, and then SE++ can not continue and stops.

What worries me a bit is that the error is not reproducible. I experienced one image where the run mostly failed but sometimes succeeded and also once I ended in an infinite loop.

This all can be explained by projection problems/errors and we will certainly discuss this case in our next SE++ telecon.

What you could do is try to avoid world_to_pixel projections from far outside of an extension of a pawprint. This means:

• splitting up the measurement image into 2x2 or even 4x4 images (with some overlap between adjacent images to avoid border problems);
• if you only modify the CRPIX1/2 values the split images have equivalent wcs compared to the original;
• use only the corresponding extensions of the pawprint as measurement image;
• as a start you could use the lower left corner (--> no need to modify the WCS)

That's a bit of work on your side, but certainly a solution if the problem is in the projections and wcslib.

Two remarks:

1. You can push parameters from the command line to the python configuration (see below). Then you do not need to 'sed' it.
2. Looks like the PSF fit is very quick, and there is no need to use more than maybe 2 or 3 cores;

Let me know if there are news from your side!

vision.zip

[mpiecka]

@mkuemmel Thanks very much for the information and the recommendations! We will try to do something about the projection and test what you suggested, I will contact you once we have some results.

[mkuemmel]

I just realized, a cheaper solution would be to to work with associations.

• run over the coadd without fitting, which gives you the reference catalog;
• split the reference catalog up in 2x2 or 4x4 sub-areas;
• run with measurements/fitting on parts of the pawprint, using the corresponding sub-area as association table;

With identical detection params there should be a 1:1 relation for the measurement runs. Then you do not have to break up the detection image.

[mkuemmel]

@mpiecka we discussed internally, and it looks like the endless loop is indeed a problem in SE++.

Occasionally some us us experienced this as well, but it was never frequently (I can not remember that in at least the last year). Your dataset seems to be very prone to this behaviour. Are we allowed to use it in the development team for debugging?

Needless to say that we won't pass it outside of the SE++ group or use it for science.

[mpiecka]

@mkuemmel Yes, you can use the data for debugging, and many thanks for working on this!

[marcschefer]

I ran the data a few times, it always get stuck in a function from lapack library called dbdsqr() which is used by the levmar minimization library. It seems to be used as part of its matrix inversion and sometimes fails and runs forver.

[mkuemmel]

Hi @mpiecka, any news on your project? Could you do some serious processing? On of the developers is going to make tests using a version that circumvents the WCS issue.
Cheers,
Martin

[mkuemmel]

Hi @mpiecka, any news from your side? We hope you could use SE++ for your processing.
Also we would recommend using the newest version v1.0.3 for processing.