[Owen Lewis (orlewis)]

For FCI, some of the channels are required to be processed but not all are required to be written out into the varobs file.

This change enables the var_channels option to be used to reduce the number of channels which are written out into the varobs file.
For example if we had channels 5,6,7,9,10,11 and we only wanted channels 5 and 10 in the varobs file then var_channels would be set to 5, 10.
Previously the code would have put channels 5 and 10 into the 5 and 6 spaces. This change will now put the channels into the correct location of 5 and 10.

This however does hijack how var_channels is intended to be used (it was always assumed that the length of channels selected would be the same as the var_channels). It does mean if the length of var_channels is less then all_channels then this cannot be used for mapping the channels to new identities at the same time.

This has been run through here https://cylchub/services/cylc-review/view/owen.lewis?&suite=sith_fci_opsinputtest%2Frun1&no_fuzzy_time=0&path=log/job/20210701T1200Z/glu_var_jopa_anal_low/22/job.stats#957

ctests have also been created.

Running through all the ctests there are failures but that looks to be due to a missing file.
https://cylchub/services/cylc-review/taskjobs?per_page=15&user=owen.lewis&no_fuzzy_time=0&suite=mobbm-opsinputs%2Frun13&task_status=expired&task_status=failed&task_status=preparing&task_status=running&task_status=submitted&task_status=submit-failed&task_status=succeeded&task_status=waiting&page=1

Github copilot was used.

[PJLevensMO]

This looks to work for both SEVIRI and FCI in my testing so far - thanks for adding it.

• SEVIRI test, only keeping channels 5 and 10: https://cylchub/services/cylc-review/view/peter.levens?&suite=seviri_reducechan%2Frun3&no_fuzzy_time=0&path=log/job/20250812T1200Z/glu_var_jopa_anal_low/01/job.stats#889
• FCI test, only keeping channels 10 and 15: https://cylchub/services/cylc-review/view/peter.levens?&suite=fci_reducechan%2Frun9&no_fuzzy_time=0&path=log/job/20250812T1200Z/glu_var_jopa_anal_low/01/job.stats#901

To note, though, I had failures for FCI when I had the increase_chan_array varobs writer option set to true. Not sure why that was the case, but the output/error can be seen in this run: https://cylchub/services/cylc-review/view/peter.levens?&suite=fci_reducechan%2Frun9&no_fuzzy_time=0&path=log/job/20250812T1200Z/glu_jopa_process_background_fciclr/01/job.out - I can't see any obvious reason for the failure in the output. But removing that option appears to work as expected.

[PJLevensMO]

This however does hijack how var_channels is intended to be used (it was always assumed that the length of channels selected would be the same as the var_channels). It does mean if the length of var_channels is less then all_channels then this cannot be used for mapping the channels to new identities at the same time.

I think it may be worth at least opening an issue on this point to be looked at soon-ish - just in case we ever need to pull a channel from ABI or AHI which do use the varChannels option for mapping.

[Michael Cooke (mikecooke77)]

I've just been discussing this with Owen and I can't figure out why this should be needed. The mapping issue looks exactly the same issue as with SEVIRI but with different numbers. If I'm understanding the varobs writer correctly you should need something like this for FCI:

filter: VarObs Writer
    reject_obs_with_all_variables_failing_qc: true
    variables_for_quality_control:
    - name: brightnessTemperature
      channels: 10, 11, 12, 14, 15, 16
    varChannels: 10, 11, 12, 14, 15, 16
    compress_var_channels: false
    size_of_varobs_array: 16

PJLevensMO could you explain why this does not work?

[PJLevensMO]

I've just been discussing this with Owen and I can't figure out why this should be needed. The mapping issue looks exactly the same issue as with SEVIRI but with different numbers. If I'm understanding the varobs writer correctly you should need something like this for FCI:

filter: VarObs Writer
    reject_obs_with_all_variables_failing_qc: true
    variables_for_quality_control:
    - name: brightnessTemperature
      channels: 10, 11, 12, 14, 15, 16
    varChannels: 10, 11, 12, 14, 15, 16
    compress_var_channels: false
    size_of_varobs_array: 16

PJLevensMO could you explain why this does not work?

With everything the same as SEVIRI, this does work. The problem is when we want to remove channels in the QC - then the remaining channels get compressed.

This channel removal is something I'm investigating for the FCI implementation. One solution that partly works is to remove the unwanted channels from the all_channels list, but in doing that means those channels cannot be used in the QC, which we do want - e.g. channel 14 is used in undetected cloud identification but I'm looking to test removing it from assimilation. There appears to be no way currently that allows me to use a channel in QC but then not assimilate it without compressing the other channels into the wrong positions. Owen's solution here works in the way I'm looking to test.

[PJLevensMO]

Can we open this up properly for review Owen Lewis (@orlewis) ? It is looking likely that this will be required for FCI implementation. I will be running a trial with these settings shortly (I'll create a JEDI build using this branch)

[Chris Thomas (ctgh)]

This looks fine from my perspective. Do you need it to be merged asap?

[PJLevensMO]

This looks fine from my perspective. Do you need it to be merged asap?

It is not absolutely urgent. I can continue testing using settings from this branch, especially as it is up to date with develop. The upcoming tests I am considering for FCI actually may negate the requirement for this change (in the global at least).

[PJLevensMO]

Tests are still ongoing as to the best setup for FCI in operations, and it is not yet clear whether this PR is needed. I'll keep this updated as testing continues.

[Copilot]

🟡 Changes recommended

Unresolved channel/value mismatches, incompatible array shapes, incomplete subset handling, and invalid test parameter types block approval.

5 open issues

• Apply channel filtering when output size is zero · New
• Keep ChanNum allocation width aligned with channel subsets · New
• Map uncompressed values using channel identities · New
• Count selected passing channels independently of field 55 · New
• Use scalar string syntax for varChannels configuration · New

What changed in this PR

This PR extends the VarObs writer’s varChannels handling so FCI can process more channels than it writes, while preserving selected channel identities.

Changes:

• Adds subset-aware channel-number allocation, population, and counting.
• Adds smoke and checker configurations for channel subsets.
• Registers both tests with CMake.

File	Description
test/​testinput/​VarObsWriterNamelists_varobswriter_varChanneltest_checker/​SEVIRIClr.nl	Selects channel-count and channel-number fields for checking.
test/​testinput/​varobswriter_varChanneltest.yaml	Adds a channel-subset smoke-test configuration.
test/​testinput/​varobswriter_varChanneltest_checker.yaml	Adds expected channel metadata for subset output.
test/​testinput/​varobswriter_varChanneltest	Adds an extensionless copy of the smoke-test input.
test/​CMakeLists.txt	Registers the new writer tests.
src/​opsinputs/​opsinputs_varobswriter_mod.F90	Implements subset handling for ChanNum and NumChans.

_{Review effort: Balanced}