Incorrect interpretation of DJI gimbal angles for triangulation sfm_algorithm

[thadwald]

With sfm_algorithm set to tirangulation, oblique image sets from a DJI Mavic 3E produce a 3d point cloud with half the points on the ground and the other half at the same distance but above the camera positions. Turning on the camera icons in 3D view reveals that both, nadir and backwards-facing gimbal poses are interpreted as facing upwards.

setting sfm_algorithm to incremental does not produce this effect.

[thadwald]

Fix for this issue:

#1970

[Springreens]

The link to the fix appears to be dead... I also have seen this problem... I wonder if it is when the drone is moving fast at the time the picture is triggered and the operator decelerates quickly and the breaking action causes the drone to tip over an angle?

I think drones need to increase the gimbal range to account for rapid speed and breaking and also when fighting against the wind

[thadwald]

I fixed the link.

The way DJI records gimbal pitch is: nadir = 0, any angle away from nadir is negative. If the gimbal swings up towards the rear of aircraft, recorded pitch remains negative, recorded gimbal roll flips to 180, and recorded yaw turns by 180. It also records the image upside-down -- at least on the Mavic 3E.

Note that this is not the way the gimbal physically moves; it's just recorded that way.

When using the smart oblique mapping flight, this results in about half the images being recorded like this.

I never understood why triangulation algorithm results in two separate point clouds for these images until I wrote an image clustering app that needs to account for gimbal attitude.

https://github.com/thadwald/ODM-Oblique-Clustering-App

[thadwald]

See #1974

[originlake]

The following calculation of rotation matrix for earth center coordinate system is probably also wrong, I submitted the fix for it, please advise #1977

[thadwald]

@originlake

I created a stage for testing gimbal priors against reconstruction values for this very reason; gimbal angles appear to be off. I agree that znp is incorrect; z axis is aligned with the ECEF Z instead of local down. This error is subtle; subsequent transformations correct for it except when heading is north-south AND gimbal angles are not nadir. When angles are away from nadir, error is approximately half of gimbal angle at my location (44 latitude)

I just ran your fix and it actually made the outcome worse. Did it work for you? I tried fixing just the znp z component myself earlier and got the same result; outcomes are worse. If you can, have a look at the subsequent transformation and see where it goes wrong

So I rewrote the whole thing. I also added in correction for magnetic declination. I believe I now have a working and correct compute_opk function with one caveat; I used AI extensively to create it.

from my analysis stage:
gimbal prior vs reconstruction errors:

INFO] OMEGA (roll): mean= -0.32 std= 1.64 min= -4.51 max= +2.80
[INFO] PHI (pitch): mean= -0.27 std= 1.02 min= -1.84 max= +1.67
[INFO] KAPPA (yaw): mean= -0.41 std= 1.18 min= -3.72 max= +0.74

Correct gimbal priors only ever matter if images have ypr information and if used with the triangulation SfM algorithm. I am finding that triangulation is a great tool to use for ambiguous features that are easy to mismatch, such s repeating patters on buildings and row crops.

[originlake]

Sorry, I forgot to mention it after I closed it, you can ignore my PR. It's supposed to calculate a topocentric based rotation matrix instead of ecef(my fix make it correct in ecef but still won't give right result). There is an ongoing OpenSfM optimization and bugfix(YanNoun/OpenSfM@dd18fe7) that contains the fix so I closed mine, it's in plan to be merged in ODM. .

[thadwald]

If it helps to explain what DJI did with the M3E gimbal here's that:

• the physical gimbal yaw range is only enough for stabilization
• physical gimbal pitch axis can rotate backwards to 45º
• encoding avoids a sign flip in pitch angle

orientation is encoded as a virtual 4-axis gimbal, starting from innermost axis:

1. image z-axis tied 1:1 with roll
2. pitch
3. yaw
4. roll (recorded as 180 whenever gimbal swings pas nadir in backward direction)

other than the roll eccentricities, normalizations are handled correctly in openSfM. ODM reads heading from aircraft, but I see that OpenSfM correctly prefers gimbal heading.

Edit: I would like to add that roll on this gimbal is only ever 0 or 180. Is this the case for all gimbals? How does the Matrice 4E smart oblique mode work, for instance? I think that this model will break just like the original one did with actual non-zero gimbal roll values.

[thadwald]

I have the working code on my branch:
see commit e51e8c8 for working compute_opk code
see commit acb350f for rotation analysis