Is Dlib Internally Tied to RGB, or Can It Support Arbitrary Image Channels?

[alexandreofbh]

Hello!

I am evaluating the possibility of using Dlib with multispectral/hyperspectral images instead of traditional RGB images.

Currently, dlib::input_rgb_image works with 3-channel RGB images. My question is the following:

Would it be possible to create an equivalent class, for example dlib::input_45_channels_image, capable of processing images with 45 spectral bands, and then use the rest of the Dlib infrastructure with little or no additional modifications?

More specifically, if I:

• Define a new input type for 45-channel images;
• Load multispectral images containing these 45 bands;
• Annotate the images normally; (Of course, with a change in the image tools)
• Train object detectors or neural networks using these images;
• Run inference using the same input format;

Would Dlib continue to work transparently, or are there parts of the library that explicitly assume RGB images and therefore require additional modifications?

In other words, does support for an arbitrary number of channels depend only on the network input layer, or are there other components such as training pipelines or inference code that would also need to be adapted?

If anyone has experience using Dlib with multispectral or hyperspectral imagery, I would also be interested in learning about any limitations encountered and what modifications were required.

Thanks for all

[arrufat]

You can use dlib::input<std::array<dlib::matrix<float,256,256>,45>>. See https://dlib.net/dlib/dnn/input_abstract.h.html#input

[alexandreofbh]

Hello Adrià,

Thank you for the clear and detailed explanation.

This completely clarified my doubt.

I would also like to thank you for your continued support and for always being so helpful and cordial in these discussions. This makes a huge difference for everyone who works with dlib.

Sincerely, Alexandre.

[arrufat]

I remember a long time ago I used that input layer paired with the loss_mean_squared_per_channel_and_pixel_ to implement this paper: https://arxiv.org/abs/1804.06208.

This is the first time I got it to work. I got too excited that I recorded it. After some tuning, it worked much better: https://x.com/arrufatio/status/1176146507436716033

I think that loss layer was my first serious contribution to dlib.

[alexandreofbh]

I saw the video!
Wow! That's incredible, Arrufat. You're amazing!
Tank you!

[davisking]

Yeah Adria does great work :D

[arrufat]

I learned a lot from you :)

[alexandreofbh]

Those who use DLIB see programming as an art. One day I'll be like you!