useWaterDemandAveYear

[StavrosChlorokostas]

Dear devs,

I have extracted a single year (12 months) time series from the EFAS datasets for domestic, livestock, energy and manufacturing, with datetimes (2022-02-01 to 2023-01-01), intending to use them for the Water Use module with "useWaterDemandAveYear" active. However, Lisflood run crashes raising a pandas error (see below).
Dom.nc structure:

dom.time.encoding:

Traceback error:

Traceback (most recent call last):
  File "src/lisf1.py", line 34, in <module>
    sys.exit(main())
  File "/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/src/lisflood/main.py", line 226, in main
    lisfloodexe(lissettings)
  File "/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/src/lisflood/main.py", line 90, in lisfloodexe
    Lisflood = LisfloodModel()
  File "/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/src/lisflood/Lisflood_initial.py", line 222, in __init__
    self.waterabstraction_module.initial()
  File "/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/src/lisflood/hydrological_modules/waterabstraction.py", line 131, in initial
    self.forcings[data] = xarray_reader(data, indexer='ffill', climatology=option['useWaterDemandAveYear'])
  File "/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/src/lisflood/global_modules/netcdf.py", line 338, in xarray_reader
    data = XarrayChunked(data_path, None, dates, indexer, climatology)
  File "/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/src/lisflood/global_modules/netcdf.py", line 209, in __init__
    self.index_map = map_dates_index(dates, da.time, indexer, climatology)
  File "/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/src/lisflood/global_modules/netcdf.py", line 164, in map_dates_index
    run_dates = time.sel(time=dates_run, method=indexer).values
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/core/dataarray.py", line 1337, in sel
    **indexers_kwargs,
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/core/dataset.py", line 2505, in sel
    self, indexers=indexers, method=method, tolerance=tolerance
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/core/coordinates.py", line 422, in remap_label_indexers
    obj, v_indexers, method=method, tolerance=tolerance
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/core/indexing.py", line 120, in remap_label_indexers
    idxr, new_idx = index.query(labels, method=method, tolerance=tolerance)
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/core/indexes.py", line 240, in query
    indexer = get_indexer_nd(self.index, label, method, tolerance)
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/core/indexes.py", line 142, in get_indexer_nd
    flat_indexer = index.get_indexer(flat_labels, method=method, tolerance=tolerance)
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 3486, in get_indexer
    return self._get_indexer(target, method, limit, tolerance)
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 3508, in _get_indexer
    indexer = self._get_fill_indexer(target, method, limit, tolerance)
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 3584, in _get_fill_indexer
    indexer = self._get_fill_indexer_searchsorted(target, method, limit)
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 3608, in _get_fill_indexer_searchsorted
    indexer[nonexact] = self._searchsorted_monotonic(target[nonexact], side)
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 5765, in _searchsorted_monotonic
    raise ValueError("index must be monotonic increasing or decreasing")
ValueError: index must be monotonic increasing or decreasing

Is there any example of water demand ncdfs in the form LISFLOOD expects them when using the "useWaterDemandAveYear" option?

[StefaniaGrimaldi]

Dear @StavrosChlorokostas ,

you can find examples of daily water demand files used in combination with the option "useWaterDemandAveYear" here
https://github.com/ec-jrc/lisflood-code/tree/master/tests/data/LF_ETRS89_UseCase/maps/waterdemand .

Furthermore, please find attached an example with values for each month:
example_waterdemavgyear_dom_2018_1arcmin.zip

I had a look at the screenshot that you kindly provided and I believe that your input maps is missing the first month (the water demand module looks for input on the same time step or at a previous time step).

I hope that this answer helps,
on behalf of OS LISFLOOD team,
kind regards,
Stefania

[StavrosChlorokostas]

@StefaniaGrimaldi
I managed to solve the issue by limiting the "time" coordinate values from 2022-01-01 to 2022-12-01 and changing the attributes for "lat", "lon", value layer and "time" to match the attributes in the EFAS ncdfs. I might be wrong on this, but apparently the internal LISFLOOD code parses attributes of each variable/coordinate so any problem with the attributes will cause issues down the line. It doesn't seem to be a problem for 2d maps but when the time dimension is introduced it breaks it for some reason. Through some hack-ish error traceback I managed to display the issue in the following snippet:

/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/axis_1km/mapsIn/water_use/dom.nc
VARIABLE: <xarray.Variable (lon: 1101)>
array([19.  , 19.01, 19.02, ..., 29.98, 29.99, 30.  ])
ATTRS: {}
TIME_UNITS : frozenset({'seconds', 'microseconds', 'nanoseconds', 'milliseconds', 'minutes', 'hours', 'days'})
VARIABLE: <xarray.Variable (lat: 901)>
array([43.  , 42.99, 42.98, ..., 34.02, 34.01, 34.  ])
ATTRS: {}
TIME_UNITS : frozenset({'seconds', 'microseconds', 'nanoseconds', 'milliseconds', 'minutes', 'hours', 'days'})
VARIABLE: <xarray.Variable (time: 12, lat: 901, lon: 1101)>
[11904012 values with dtype=float32]
Attributes:
    AREA_OR_POINT:    Area
    cell_methods:     - time: mean
    esri_pe_string:   GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378...
    long_name:        domestic demand
    NETCDF_DIM_time:  11719
    standard_name:    domestic demand
    units:            ['mm/day', 'mm/day', 'mm/day', 'mm/day', 'mm/day', 'mm/...
    coordinates:      wgs_1984
ATTRS: {'AREA_OR_POINT': 'Area', 'cell_methods': '- time: mean', 'esri_pe_string': 'GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433],AUTHORITY["EPSG","4326"]]', 'long_name': 'domestic demand', 'NETCDF_DIM_time': 11719, 'standard_name': 'domestic demand', 'units': ['mm/day', 'mm/day', 'mm/day', 'mm/day', 'mm/day', 'mm/day', 'mm/day', 'mm/day', 'mm/day', 'mm/day', 'mm/day', 'mm/day'], 'coordinates': 'wgs_1984'}
TIME_UNITS : frozenset({'seconds', 'microseconds', 'nanoseconds', 'milliseconds', 'minutes', 'hours', 'days'})
Traceback (most recent call last):
  File "src/lisf1.py", line 34, in <module>
    sys.exit(main())
  File "/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/src/lisflood/main.py", line 226, in main
    lisfloodexe(lissettings)
  File "/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/src/lisflood/main.py", line 90, in lisfloodexe
    Lisflood = LisfloodModel()
  File "/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/src/lisflood/Lisflood_initial.py", line 222, in __init__
    self.waterabstraction_module.initial()
  File "/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/src/lisflood/hydrological_modules/waterabstraction.py", line 131, in initial
    self.forcings[data] = xarray_reader(data, indexer='ffill', climatology=option['useWaterDemandAveYear'])
  File "/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/src/lisflood/global_modules/netcdf.py", line 336, in xarray_reader
    data = XarrayChunked(data_path, None, dates, indexer, climatology)
  File "/mnt/c/users/chlorokostas/downloads/Flood_Risk/lisflood-code/src/lisflood/global_modules/netcdf.py", line 183, in __init__
    mask_and_scale=True)
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/backends/api.py", line 908, in open_mfdataset
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/backends/api.py", line 908, in <listcomp>
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/backends/api.py", line 499, in open_dataset
    **kwargs,
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/backends/netCDF4_.py", line 572, in open_dataset
    decode_timedelta=decode_timedelta,
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/backends/store.py", line 36, in open_dataset
    decode_timedelta=decode_timedelta,
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/conventions.py", line 520, in decode_cf_variables
    decode_timedelta=decode_timedelta,
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/conventions.py", line 358, in decode_cf_variable
    var = times.CFTimedeltaCoder().decode(var, name=name)
  File "/home/chlorokostas/miniconda3/envs/lisflood/lib/python3.7/site-packages/xarray/coding/times.py", line 553, in decode
    if "units" in attrs and attrs["units"] in TIME_UNITS:
TypeError: unhashable type: 'list'

Here the value corresponding to the "units" key caused the issue. This caused me quite a bit of pain since when inspecting an ncdf as an xarray, the specific attribute of each variable/coordinate of the array are not visible.