diff --git a/documentation/api/change_log.rst b/documentation/api/change_log.rst index 39414fd269..a1e9833455 100644 --- a/documentation/api/change_log.rst +++ b/documentation/api/change_log.rst @@ -11,6 +11,8 @@ v3.0-31 | 2026-05-20 - Added a unified job status endpoint ``GET /api/v3_0/jobs/`` that returns the current execution status and a human-readable result message for any background job (scheduling, forecasting, etc.) identified by its UUID. - Switched from ``force_new_job_creation`` to ``force-new-job-creation`` (maintaining backwards compatibility) and added the field to `/assets/(id)/schedules/trigger` (POST) endpoint, too. - Support both snake_case and kebab-case fields in `/sensors//data` (GET), while only documenting the kebab-case ones. +- The ``consumption`` and ``production`` flex-model fields for the ``StorageScheduler`` now act as output sensors: the scheduler writes the resulting power schedule to those sensors, with unit conversion and resampling applied. When only one of the two is defined, the full power profile is written (sign-inverted for the production sensor). When both are defined, the schedule is split into its non-negative (consumption) and non-positive (production) parts. +- Added a ``sign-convention`` query parameter to ``GET /sensors//schedules/`` with three modes: ``consumption-positive`` (default, unchanged behaviour), ``production-positive``, and ``wysiwyg`` (sign of database values and as seen in UI charts). v3.0-30 | 2026-04-15 """""""""""""""""""" diff --git a/documentation/api/notation.rst b/documentation/api/notation.rst index 75a9e0eb26..68b98e9d3a 100644 --- a/documentation/api/notation.rst +++ b/documentation/api/notation.rst @@ -302,3 +302,10 @@ We'd recommend to use positive power values to indicate consumption and negative Read more at :ref:`signs_of_power_beliefs` about our treatment of data, which includes data you send in, or you get from forecasts and schedules (hint: you are free to define the sign for your data, but it might affect how you receive your schedules). + +The ``GET /api/v3_0/sensors//schedules/`` endpoint supports three sign conventions via the ``sign-convention`` query parameter: + +- ``consumption-positive`` (**default**): schedules are returned with consumption as positive values and production as negative values, regardless of how they are stored in the database. +- ``production-positive``: schedules are returned with production as positive values and consumption as negative values. +- ``wysiwyg`` (*what-you-see-is-what-you-get*): schedules are returned with the same sign as database values and as seen in the UI charts. + The values indicate exactly what was stored, which was itself governed by the sensor's ``consumption_is_positive`` attribute (if present) or the scheduler's default convention (which stored production as positive values in the database). diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 8ab2780d6b..91f5763a66 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -23,6 +23,7 @@ New features * Improve UX after deleting a child asset through the UI [see `PR #2119 `_] * Improve source filtering in the sensor data GET endpoint by exposing the documented query parameters in Swagger and allowing filtering by the account linked to data sources [see `PR #2083 `_ and `PR #2151 `_] * Support sensor references for efficiency fields in storage flex-models [see `PR #2142 `_] +* Introduce the ``consumption`` and ``production`` flex-model fields for the ``StorageScheduler`` to save schedules to [see `PR #2190 `_] * Added a unified job status endpoint ``GET /api/v3_0/jobs/`` to retrieve the current execution status and result message for any background job [see `PR #2141 `_] * Add ``flexmeasures jobs inspect-job`` CLI command to show job status and metadata information (similar to the job status endpoint in the API) [see `PR #2202 `_] * New ``GET /api/v3_0/sources`` endpoint to list accessible data sources and defined types, with ``only_latest=true`` by default to return only the most recent version per source [see `PR #2126 `_] @@ -141,6 +142,7 @@ Bugfixes v0.31.3 | April 11, 2026 +============================ Bugfixes ----------- diff --git a/documentation/concepts/data-model.rst b/documentation/concepts/data-model.rst index 6c2267768a..1aedfeb563 100644 --- a/documentation/concepts/data-model.rst +++ b/documentation/concepts/data-model.rst @@ -79,6 +79,7 @@ A data source can be a FlexMeasures user, but also simply a named source from ou In FlexMeasures, data sources have a type. It is just a string which you can freely choose (we do not model them explicitly im the data model like Asset types). We do support some types out of the box: "scheduler", "forecaster" "reporter", "demo script" and "user". +.. _beliefs: Beliefs --------- @@ -156,7 +157,8 @@ More information, including code examples, is available in :ref:`annotations`. About signs of power & energy values ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In short: You can use any sign you want for power data. -But the scheduler in FlexMeasures needs to know how to apply the signs. Positive (+) means consumption, negative (-) means production. +What is recorded in the database is exactly as seen in UI charts. +But the scheduler in FlexMeasures needs to know how to apply the signs. Let us explain. When beliefs are about power or energy, the sign of the value is important. It indicates whether the asset is consuming or producing. @@ -168,12 +170,25 @@ For example, users can create PV power data with positive values indicating prod We allow this because we want the UI to match what is in the database, and users often desire both of these datasets to be shown as positive values. We assume that this is what users send in. -Note that, if forecasts are created, they will have the same sign as original data. +Note that, if forecasts are created, they will have the same sign as the original data. -For schedules, the sign of resulting power data (beliefs) is being switched when data is stored (assuming consumption , and you can prevent that by setting ``sensor.attributes["consumption_is_positive"] = True``. +For schedules, the sign of the power schedule (as :ref:`beliefs `) recorded in the database, and as seen in UI charts, is determined as follows: +- If the flex-model contains the ``sensor`` field, and that sensor has power units (e.g. kW), the ``"consumption_is_positive"`` attribute of the sensor is used to decide the sign of the recorded data. + If `True`, consumption will be saved as positive, otherwise not. To clarify: + - If the attribute is not defined, **by default, scheduled power is recorded with production as positive values** (and consumption as negative values). + - To record scheduled power data with consumption as positive values, set ``sensor.attributes["consumption_is_positive"] = True``. + - To record scheduled power data with production as positive values (already the default, but this makes it explicit), set ``sensor.attributes["consumption_is_positive"] = False``. +- If the flex-model contains the ``consumption`` field, scheduled power is recorded with consumption as positive values. + The ``"consumption_is_positive"`` attribute of the referenced sensor is set automatically to ``True``. +- If the flex-model contains the ``production`` field, scheduled power is recorded with production as positive values. + The ``"consumption_is_positive"`` attribute of the referenced sensor is set automatically to ``False``. -.. note:: We will soon document better what the scheduler does in detail, and how the attribute works. +The ``GET /api/v3_0/sensors//schedules/`` endpoint supports three sign conventions via the ``sign-convention`` query parameter: + +- ``consumption-positive`` (default): schedules are always returned with consumption as positive values and production as negative values. +- ``production-positive``: schedules are returned with production as positive values and consumption as negative values. +- ``wysiwyg`` (*what-you-see-is-what-you-get*): schedules are returned with the same sign as database values and as seen in the UI charts, indicating exactly what the scheduler stored. Accounts & Users diff --git a/documentation/features/scheduling.rst b/documentation/features/scheduling.rst index 3801f2bda0..1643330a6d 100644 --- a/documentation/features/scheduling.rst +++ b/documentation/features/scheduling.rst @@ -182,7 +182,13 @@ For more details on the possible formats for field values, see :ref:`variable_qu * - Field - Example value - - Description + - Description + * - ``consumption`` + - |CONSUMPTION.example| + - .. include:: ../_autodoc/CONSUMPTION.rst + * - ``production`` + - |PRODUCTION.example| + - .. include:: ../_autodoc/PRODUCTION.rst * - ``state-of-charge`` - |STATE_OF_CHARGE.example| - .. include:: ../_autodoc/STATE_OF_CHARGE.rst diff --git a/flexmeasures/api/v3_0/sensors.py b/flexmeasures/api/v3_0/sensors.py index dc76b4059b..3215446c3b 100644 --- a/flexmeasures/api/v3_0/sensors.py +++ b/flexmeasures/api/v3_0/sensors.py @@ -65,7 +65,10 @@ ) from flexmeasures.data.schemas import AssetIdField, SourceIdField from flexmeasures.api.common.schemas.search import SearchFilterField -from flexmeasures.data.schemas.scheduling import GetScheduleSchema +from flexmeasures.data.schemas.scheduling import ( + GetScheduleSchema, + ScheduleSignConvention, +) from flexmeasures.data.schemas.units import UnitField from flexmeasures.data.services.sensors import get_sensor_stats from flexmeasures.data.services.sensors import delete_sensor as delete_sensor_and_data @@ -1040,6 +1043,7 @@ def get_schedule( # noqa: C901 job_id: str, duration: timedelta, unit: str | None = None, + sign_convention: str = ScheduleSignConvention.CONSUMPTION_POSITIVE, **kwargs, ): """ @@ -1054,6 +1058,21 @@ def get_schedule( # noqa: C901 - "duration" (6 hours by default; can be increased to plan further into the future) - "unit" (by default, the unit of the schedule is the sensor's unit; a compatible unit can be requested) + - "sign-convention" (controls how power values are signed in the response; see below) + + **Sign convention** + + By default (``sign-convention: consumption-positive``), the endpoint always returns schedules where + consumption is positive and production is negative, regardless of how the values are stored in the database. + This is the most common convention and matches the perspective of a consumer. + + Set ``sign-convention: production-positive`` to flip the sign so that production is returned as + positive and consumption as negative. This matches the perspective of a producer. + + Set ``sign-convention: wysiwyg`` (*what-you-see-is-what-you-get*) to return the values with the same sign + as database values and what is seen in UI charts. The values will indicate exactly what is stored, + which is itself determined by the sensor's ``consumption_is_positive`` attribute (if set) + or by the scheduler's default storage convention (production positive in the database). security: - ApiKeyAuth: [] parameters: @@ -1095,6 +1114,21 @@ def get_schedule( # noqa: C901 example: kW schema: type: string + - in: query + name: sign-convention + required: false + description: | + Sign convention applied to power values in the response. + - ``consumption-positive`` (default): consumption is positive, production is negative. + - ``production-positive``: production is positive, consumption is negative. + - ``wysiwyg`` (*what-you-see-is-what-you-get*): sign of database values and as seen in UI charts. + example: consumption-positive + schema: + type: string + enum: + - consumption-positive + - production-positive + - wysiwyg responses: 200: description: PROCESSED @@ -1221,12 +1255,25 @@ def get_schedule( # noqa: C901 ) sign = 1 - if sensor.measures_power and not sensor.get_attribute( - "consumption_is_positive", False - ): - sign = -1 - - # For consumption schedules, positive values denote consumption. For the db, consumption is negative unless specified explicitly + if sign_convention == ScheduleSignConvention.WYSIWYG: + # Return values without adjusting the sign of database values. + # No sign inversion: what's in the DB and what is seen in UI charts is what the caller receives. + pass + elif sensor.measures_power: + # Determine whether the database stores consumption as positive or negative. + db_consumption_is_positive = sensor.get_attribute( + "consumption_is_positive", False + ) + if sign_convention == ScheduleSignConvention.CONSUMPTION_POSITIVE: + # Caller wants consumption positive. Invert if DB stores it as negative. + if not db_consumption_is_positive: + sign = -1 + elif sign_convention == ScheduleSignConvention.PRODUCTION_POSITIVE: + # Caller wants production positive. Invert if DB already stores consumption positive. + if db_consumption_is_positive: + sign = -1 + + # Apply sign to get the values in the requested convention consumption_schedule = sign * simplify_index(power_values)["event_value"] if consumption_schedule.empty: # for not in-built schedulers, we are not sure if they would store time series in the db diff --git a/flexmeasures/api/v3_0/tests/test_sensor_schedules_fresh_db.py b/flexmeasures/api/v3_0/tests/test_sensor_schedules_fresh_db.py index 200243b1fe..4ee841a502 100644 --- a/flexmeasures/api/v3_0/tests/test_sensor_schedules_fresh_db.py +++ b/flexmeasures/api/v3_0/tests/test_sensor_schedules_fresh_db.py @@ -16,6 +16,7 @@ handle_scheduling_exception, get_data_source_for_job, ) +from flexmeasures.data.schemas.scheduling import ScheduleSignConvention from flexmeasures.utils.calculations import integrate_time_series @@ -215,7 +216,8 @@ def test_trigger_schedule_uses_state_of_charge_sensor_for_soc_at_start( keep_scheduling_queue_empty, requesting_user, ): - message = message_for_trigger_schedule(resolution="PT1H") + scheduling_resolution = "PT1H" + message = message_for_trigger_schedule(resolution=scheduling_resolution) message["flex-context"] = { "consumption-price": {"sensor": add_market_prices_fresh_db["epex_da"].id}, "production-price": {"sensor": add_market_prices_fresh_db["epex_da"].id}, @@ -235,14 +237,26 @@ def test_trigger_schedule_uses_state_of_charge_sensor_for_soc_at_start( event_value=50, ) ) - fresh_db.session.commit() + # Also add a production output sensor to verify the production schedule is stored. + # (Using only a production sensor means the full power profile is stored with sign inverted: + # production positive, consumption negative.) sensor = ( Sensor.query.filter(Sensor.name == "power") .join(GenericAsset, GenericAsset.id == Sensor.generic_asset_id) .filter(GenericAsset.name == "Test battery") .one_or_none() ) + production_output_sensor = Sensor( + name="production output", + generic_asset=sensor.generic_asset, + unit="MW", + event_resolution=sensor.event_resolution, + ) + fresh_db.session.add(production_output_sensor) + fresh_db.session.commit() + + message["flex-model"]["production"] = {"sensor": production_output_sensor.id} with app.test_client() as client: trigger_schedule_response = client.post( @@ -263,11 +277,119 @@ def test_trigger_schedule_uses_state_of_charge_sensor_for_soc_at_start( ) assert get_soc_schedule_response.status_code == 200 assert get_soc_schedule_response.json["unit"] == "%" - assert get_soc_schedule_response.json["values"][0] == 50 + soc_values = get_soc_schedule_response.json["values"] + assert soc_values[0] == 50 sensor = fresh_db.session.get(Sensor, sensor.id) assert sensor.generic_asset.get_attribute("soc_in_mwh") == pytest.approx(0.02) + # Verify the production output sensor received schedule data. + # Only the production sensor is defined (no consumption sensor), so the full power profile + # is stored with inverted sign: production as positive, consumption as negative. + production_output_sensor = fresh_db.session.get(Sensor, production_output_sensor.id) + production_beliefs = production_output_sensor.search_beliefs( + event_starts_after=parse_datetime(message["start"]) + ) + assert len(production_beliefs) == 96 # 24h at 15-min resolution + assert ( + soc_values[1] - soc_values[0] == -50 + ), "expected SoC to go down in the first hour" + assert ( + production_beliefs.resample_events(scheduling_resolution).iloc[0].event_value + > 0 + ), "expected first hour to show positive production" + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_conflicting_consumption_is_positive_attribute_prevents_job_creation( + app, + fresh_db, + add_market_prices_fresh_db, + add_battery_assets_fresh_db, + battery_soc_sensor_fresh_db, + setup_sources_fresh_db, + keep_scheduling_queue_empty, + requesting_user, +): + """Scheduling must be rejected immediately when consumption_is_positive is already set + to a value that conflicts with the flex-model field used for the sensor. + + Here a sensor that already has ``consumption_is_positive=True`` (consumption sensor) is + mistakenly referenced under the ``production`` flex-model field. The trigger API should + raise a ``ValueError`` and return a 422 response, so the job is never created and the + sensor's belief table remains empty. + + ``force-new-job-creation`` is set to bypass the job-cache, which would otherwise serve the + cached result of a prior test that ran against the same battery sensor and time window + (``fresh_db`` resets sensor IDs to the same values each test, producing the same cache key). + """ + message = message_for_trigger_schedule(resolution="PT1H") + message["force-new-job-creation"] = True + message["flex-context"] = { + "consumption-price": {"sensor": add_market_prices_fresh_db["epex_da"].id}, + "production-price": {"sensor": add_market_prices_fresh_db["epex_da"].id}, + "site-power-capacity": "1 TW", + } + message["flex-model"].pop("soc-at-start") + message["flex-model"]["state-of-charge"] = { + "sensor": battery_soc_sensor_fresh_db.id + } + + # Find the main power sensor for the battery. + power_sensor = ( + Sensor.query.filter(Sensor.name == "power") + .join(GenericAsset, GenericAsset.id == Sensor.generic_asset_id) + .filter(GenericAsset.name == "Test battery") + .one_or_none() + ) + + # Create an output sensor that is pre-marked as a *consumption* sensor. + # We will then (wrongly) pass it as the `production` field in the flex-model. + mismatched_sensor = Sensor( + name="consumption output (misused as production)", + generic_asset=power_sensor.generic_asset, + unit="MW", + event_resolution=power_sensor.event_resolution, + attributes={"consumption_is_positive": True}, # consumption sensor + ) + fresh_db.session.add(mismatched_sensor) + fresh_db.session.add( + TimedBelief( + sensor=battery_soc_sensor_fresh_db, + source=setup_sources_fresh_db["Seita"], + event_start=parse_datetime(message["start"]), + belief_horizon=timedelta(0), + event_value=50, + ) + ) + fresh_db.session.commit() + + # Reference the consumption-marked sensor under the `production` field — a mismatch. + message["flex-model"]["production"] = {"sensor": mismatched_sensor.id} + + with app.test_client() as client: + trigger_response = client.post( + url_for("SensorAPI:trigger_schedule", id=power_sensor.id), + json=message, + ) + # The conflict is detected at trigger time: the API returns 422 immediately, + # without creating a job. + assert trigger_response.status_code == 422 + assert "consumption_is_positive" in str( + trigger_response.json.get("message", "") + ) + + # No beliefs should have been written for the mismatched sensor. + mismatched_sensor = fresh_db.session.get(Sensor, mismatched_sensor.id) + saved_beliefs = mismatched_sensor.search_beliefs( + event_starts_after=parse_datetime(message["start"]) + ) + assert ( + len(saved_beliefs) == 0 + ), "No data should be saved when the attribute conflict is detected at job-creation time" + @pytest.mark.parametrize( "context_sensor, asset_sensor, parent_sensor, expect_sensor", @@ -610,6 +732,280 @@ def test_multiple_contracts( ) +def _assert_schedule_sign_convention( + output_type: str, + sign_convention: str, + main_values: list, + output_values: list, +) -> None: + """Assert that *output_values* satisfy the expected sign convention relative to *main_values*. + + *main_values* are always fetched with the default ``consumption-positive`` convention so that + positive values represent charging and negative values represent discharging. + + The expected sign of *output_values* depends on both ``output_type`` and ``sign_convention``: + + - ``consumption`` output sensor stores consumption as positive (``consumption_is_positive=True``): + - ``consumption-positive``: same as DB → same sign as main. + - ``production-positive``: inverts DB → opposite sign from main. + - ``wysiwyg``: raw DB (consumption positive) → same sign as main. + - ``production`` output sensor stores production as positive (``consumption_is_positive=False``): + - ``consumption-positive``: inverts DB → same sign as main. + - ``production-positive``: raw DB (production positive) → opposite sign from main. + - ``wysiwyg``: raw DB (production positive) → opposite sign from main. + """ + same_sign_as_main = ( + sign_convention == ScheduleSignConvention.CONSUMPTION_POSITIVE + or ( + output_type == "consumption" + and sign_convention == ScheduleSignConvention.WYSIWYG + ) + ) + + for main_val, out_val in zip(main_values, output_values): + if main_val > 0: + if same_sign_as_main: + assert out_val >= 0, ( + f"[output_type={output_type}, sign_convention={sign_convention}] " + f"Expected output >= 0 for charging (main={main_val}), got {out_val}" + ) + else: + assert out_val <= 0, ( + f"[output_type={output_type}, sign_convention={sign_convention}] " + f"Expected output <= 0 for charging (main={main_val}), got {out_val}" + ) + elif main_val < 0: + if same_sign_as_main: + assert out_val <= 0, ( + f"[output_type={output_type}, sign_convention={sign_convention}] " + f"Expected output <= 0 for discharging (main={main_val}), got {out_val}" + ) + else: + assert out_val >= 0, ( + f"[output_type={output_type}, sign_convention={sign_convention}] " + f"Expected output >= 0 for discharging (main={main_val}), got {out_val}" + ) + + +@pytest.mark.parametrize( + "sign_convention", + [ + ScheduleSignConvention.CONSUMPTION_POSITIVE, + ScheduleSignConvention.PRODUCTION_POSITIVE, + ScheduleSignConvention.WYSIWYG, + ], +) +@pytest.mark.parametrize( + "output_type", + ["consumption", "production"], +) +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_get_schedule_sign_convention_json_flex_model( + app, + fresh_db, + add_market_prices_fresh_db, + add_battery_assets_fresh_db, + battery_soc_sensor_fresh_db, + keep_scheduling_queue_empty, + output_type, + sign_convention, + requesting_user, +): + """Test that get_schedule returns correct values for all three sign-convention modes + when the schedule was triggered with a consumption/production sensor in the JSON flex-model. + + The main power sensor is used as a reference (always fetched with the default + ``consumption-positive`` convention). The expected sign of the output sensor values + relative to that reference depends on ``output_type`` × ``sign_convention``; + see :func:`_assert_schedule_sign_convention` for the full decision table. + """ + battery_asset = add_battery_assets_fresh_db["Test battery"] + + # Create a dedicated output sensor on the battery asset + output_sensor = Sensor( + name=f"{output_type} output", + generic_asset=battery_asset, + event_resolution=timedelta(minutes=15), + unit="MW", + ) + fresh_db.session.add(output_sensor) + fresh_db.session.flush() + + # Build the trigger message with the output sensor in the flex-model. + # force-new-job-creation bypasses the Redis job cache, which would otherwise return a cached + # job_id from the previous parametrized run (fresh_db resets sensor IDs to the same values + # each function, producing the same trigger hash). + message = message_for_trigger_schedule(resolution="PT1H") + message["force-new-job-creation"] = True + price_sensor_id = add_market_prices_fresh_db["epex_da"].id + message["flex-context"] = { + "consumption-price": {"sensor": price_sensor_id}, + "production-price": {"sensor": price_sensor_id}, + "site-power-capacity": "1 TW", + } + message["flex-model"][output_type] = {"sensor": output_sensor.id} + + # Find the main power sensor to trigger the schedule on + sensor = ( + Sensor.query.filter(Sensor.name == "power") + .join(GenericAsset, GenericAsset.id == Sensor.generic_asset_id) + .filter(GenericAsset.name == "Test battery") + .one_or_none() + ) + + with app.test_client() as client: + trigger_response = client.post( + url_for("SensorAPI:trigger_schedule", id=sensor.id), + json=message, + ) + assert trigger_response.status_code == 200 + job_id = trigger_response.json["schedule"] + + work_on_rq(app.queues["scheduling"], exc_handler=handle_scheduling_exception) + + job = Job.fetch(job_id, connection=app.queues["scheduling"].connection) + assert job.is_finished + + # Retrieve the schedule from the output sensor using the requested sign convention + with app.test_client() as client: + get_response = client.get( + url_for("SensorAPI:get_schedule", id=output_sensor.id, uuid=job_id), + query_string={"duration": "PT24H", "sign-convention": sign_convention}, + ) + assert get_response.status_code == 200 + values = get_response.json["values"] + + assert len(values) > 0 + assert any(v != 0 for v in values), "Schedule should have non-zero values" + + # Get the main power sensor schedule using the default convention as reference + with app.test_client() as client: + main_response = client.get( + url_for("SensorAPI:get_schedule", id=sensor.id, uuid=job_id), + query_string={"duration": "PT24H"}, + ) + assert main_response.status_code == 200 + main_values = main_response.json["values"] + + _assert_schedule_sign_convention(output_type, sign_convention, main_values, values) + + +@pytest.mark.parametrize( + "sign_convention", + [ + ScheduleSignConvention.CONSUMPTION_POSITIVE, + ScheduleSignConvention.PRODUCTION_POSITIVE, + ScheduleSignConvention.WYSIWYG, + ], +) +@pytest.mark.parametrize( + "output_type", + ["consumption", "production"], +) +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_get_schedule_sign_convention_db_flex_model( + app, + fresh_db, + add_market_prices_fresh_db, + add_battery_assets_fresh_db, + battery_soc_sensor_fresh_db, + keep_scheduling_queue_empty, + output_type, + sign_convention, + requesting_user, +): + """Test that get_schedule returns correct values for all three sign-convention modes + when the consumption/production sensor is stored in the DB flex-model (asset attributes). + + See :func:`_assert_schedule_sign_convention` for the full decision table. + """ + battery_asset = add_battery_assets_fresh_db["Test battery"] + + # Create a dedicated output sensor on the battery asset + output_sensor = Sensor( + name=f"{output_type} output db", + generic_asset=battery_asset, + event_resolution=timedelta(minutes=15), + unit="MW", + ) + fresh_db.session.add(output_sensor) + fresh_db.session.flush() + + # Store the output sensor reference in the asset's flex_model (DB flex-model) + battery_asset.flex_model = battery_asset.flex_model or {} + battery_asset.flex_model[output_type] = {"sensor": output_sensor.id} + fresh_db.session.add(battery_asset) + fresh_db.session.flush() + + # Build the trigger message without consumption/production in JSON flex-model + # (it will be read from the asset's DB flex-model) + message = message_for_trigger_schedule(resolution="PT1H") + price_sensor_id = add_market_prices_fresh_db["epex_da"].id + message["flex-context"] = { + "consumption-price": {"sensor": price_sensor_id}, + "production-price": {"sensor": price_sensor_id}, + "site-power-capacity": "1 TW", + } + + # Find the main power sensor to trigger the schedule on + sensor = ( + Sensor.query.filter(Sensor.name == "power") + .join(GenericAsset, GenericAsset.id == Sensor.generic_asset_id) + .filter(GenericAsset.name == "Test battery") + .one_or_none() + ) + + # Flush Redis to clear any job dedup cache from prior parametrized runs + app.redis_connection.flushdb() + + with app.test_client() as client: + trigger_response = client.post( + url_for("SensorAPI:trigger_schedule", id=sensor.id), + json=message, + ) + assert trigger_response.status_code == 200 + job_id = trigger_response.json["schedule"] + + work_on_rq(app.queues["scheduling"], exc_handler=handle_scheduling_exception) + + job = Job.fetch(job_id, connection=app.queues["scheduling"].connection) + assert job.is_finished + + # Retrieve the schedule from the output sensor using the requested sign convention + with app.test_client() as client: + get_response = client.get( + url_for("SensorAPI:get_schedule", id=output_sensor.id, uuid=job_id), + query_string={"duration": "PT24H", "sign-convention": sign_convention}, + ) + assert ( + get_response.status_code == 200 + ), f"GET schedule failed: {get_response.json}" + values = get_response.json["values"] + + assert len(values) > 0 + assert any(v != 0 for v in values), "Schedule should have non-zero values" + + # Get the main power sensor schedule using the default convention as reference + with app.test_client() as client: + main_response = client.get( + url_for("SensorAPI:get_schedule", id=sensor.id, uuid=job_id), + query_string={"duration": "PT24H"}, + ) + assert main_response.status_code == 200 + main_values = main_response.json["values"] + + _assert_schedule_sign_convention(output_type, sign_convention, main_values, values) + + # Clean up: remove the flex_model entry so it doesn't affect other tests + del battery_asset.flex_model[output_type] + fresh_db.session.add(battery_asset) + fresh_db.session.flush() + + def setup_inflexible_device_sensors(fresh_db, asset, sensor_name, sensor_num): """Test helper function to add sensor_num sensors to an asset""" sensors = list() diff --git a/flexmeasures/cli/tests/test_data_add_fresh_db.py b/flexmeasures/cli/tests/test_data_add_fresh_db.py index 6abc036585..cfd50741b9 100644 --- a/flexmeasures/cli/tests/test_data_add_fresh_db.py +++ b/flexmeasures/cli/tests/test_data_add_fresh_db.py @@ -472,12 +472,41 @@ def test_add_storage_schedule_uses_state_of_charge_sensor_for_soc_at_start( add_charging_station_assets_fresh_db, setup_sources_fresh_db, ): + """Test that the StorageScheduler reads soc-at-start from a sensor and stores + schedules on dedicated consumption and production output sensors with the correct + sign convention and clipping behaviour. + + Setup: + - Bidirectional charging station (can both charge and discharge). + - SOC at start: 2.5 MWh (read from a sensor belief). + - Schedule window: 2015-01-03 00:00–12:00 CET. + On this date the market data has consumption prices of -10 EUR/MWh for hours 0–7 + (incentivises charging) and production prices of +60 EUR/MWh for hours 8–23 + (incentivises discharging), so the optimizer is guaranteed to do both. + + Sign-convention and clipping assertions (both consumption and production sensors defined): + - Consumption sensor (consumption_is_positive=True): all stored values ≥ 0 + (charging intervals are positive; discharging intervals are clipped to 0). + - Production sensor (consumption_is_positive=False): all stored values ≥ 0 + (discharging intervals are stored as positive; charging intervals are clipped to 0). + - No single timestep may carry both a positive consumption and a positive production + value (the two sensors together partition the schedule without overlap). + - At least one charging and one discharging event must actually occur. + """ from flexmeasures.cli.data_add import add_schedule - charging_station = add_charging_station_assets_fresh_db["Test charging station"] - power_sensor = next(s for s in charging_station.sensors if s.name == "power") - soc_sensor = add_charging_station_assets_fresh_db["uni-soc"] - start = "2015-01-01T00:00:00+01:00" + # Use the bidirectional station so both charging and discharging can occur. + bidirectional_charging_station = add_charging_station_assets_fresh_db[ + "Test charging station (bidirectional)" + ] + power_sensor = next( + s for s in bidirectional_charging_station.sensors if s.name == "power" + ) + soc_sensor = add_charging_station_assets_fresh_db["bi-soc"] + + # 2015-01-03: consumption prices are -10 EUR/MWh in hours 0-7 (charging rewarded) + # and production prices are +60 EUR/MWh in hours 8-23 (discharging rewarded). + start = "2015-01-03T00:00:00+01:00" fresh_db.session.add( TimedBelief( @@ -488,15 +517,38 @@ def test_add_storage_schedule_uses_state_of_charge_sensor_for_soc_at_start( belief_time=datetime.fromisoformat(start), ) ) + + # Add dedicated output sensors for the consumption (charging) and production + # (discharging) parts of the schedule. + consumption_output_sensor = Sensor( + name="consumption output", + generic_asset=bidirectional_charging_station, + unit="MW", + event_resolution=power_sensor.event_resolution, + ) + production_output_sensor = Sensor( + name="production output", + generic_asset=bidirectional_charging_station, + unit="MW", + event_resolution=power_sensor.event_resolution, + ) + fresh_db.session.add(consumption_output_sensor) + fresh_db.session.add(production_output_sensor) fresh_db.session.commit() + epex_da = add_market_prices_fresh_db["epex_da"] + epex_da_production = add_market_prices_fresh_db["epex_da_production"] + cli_input_params = { "sensor": power_sensor.id, "start": start, "duration": "PT12H", "scheduler": "StorageScheduler", "flex-context": json.dumps( - {"consumption-price": {"sensor": add_market_prices_fresh_db["epex_da"].id}} + { + "consumption-price": {"sensor": epex_da.id}, + "production-price": {"sensor": epex_da_production.id}, + } ), "flex-model": json.dumps( { @@ -504,6 +556,8 @@ def test_add_storage_schedule_uses_state_of_charge_sensor_for_soc_at_start( "soc-min": "0 MWh", "soc-max": "5 MWh", "power-capacity": "2 MW", + "consumption": {"sensor": consumption_output_sensor.id}, + "production": {"sensor": production_output_sensor.id}, } ), } @@ -513,3 +567,42 @@ def test_add_storage_schedule_uses_state_of_charge_sensor_for_soc_at_start( check_command_ran_without_error(result) assert len(power_sensor.search_beliefs()) == 48 assert power_sensor.generic_asset.get_attribute("soc_in_mwh") == 2.5 + + # Reload sensors from the DB after the schedule has been committed. + consumption_output_sensor = fresh_db.session.get( + Sensor, consumption_output_sensor.id + ) + production_output_sensor = fresh_db.session.get(Sensor, production_output_sensor.id) + consumption_beliefs = consumption_output_sensor.search_beliefs() + production_beliefs = production_output_sensor.search_beliefs() + + assert len(consumption_beliefs) == 48 + assert len(production_beliefs) == 48 + + consumption_values = consumption_beliefs.values.flatten() + production_values = production_beliefs.values.flatten() + + # Sign convention: consumption sensor (consumption_is_positive=True) stores + # charging as positive values; discharging intervals are clipped to 0. + assert ( + consumption_values >= 0 + ).all(), "Consumption output sensor must only hold non-negative values" + # Sign convention: production sensor (consumption_is_positive=False) stores + # discharging as positive values; charging intervals are clipped to 0. + assert ( + production_values >= 0 + ).all(), "Production output sensor must only hold non-negative values" + # Clipping: the two sensors partition the schedule without overlap — no + # single timestep should carry both a positive consumption and a positive + # production value. + assert not ( + (consumption_values > 0) & (production_values > 0) + ).any(), "No timestep may have both positive consumption and positive production" + # With negative consumption prices in hours 0-7 and positive production prices + # in hours 8+, both charging and discharging must occur. + assert ( + consumption_values > 0 + ).any(), "Some charging must occur given the negative consumption prices" + assert ( + production_values > 0 + ).any(), "Some discharging must occur given the positive production prices" diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 7317003cf3..4af89ae816 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -144,6 +144,8 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ] soc_gain = [flex_model_d.get("soc_gain") for flex_model_d in flex_model] soc_usage = [flex_model_d.get("soc_usage") for flex_model_d in flex_model] + consumption = [flex_model_d.get("consumption") for flex_model_d in flex_model] + production = [flex_model_d.get("production") for flex_model_d in flex_model] consumption_capacity = [ flex_model_d.get("consumption_capacity") for flex_model_d in flex_model ] @@ -906,10 +908,30 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 device_constraints[d]["efficiency"] = storage_efficiency[d] # Convert efficiency from sensor resolution to scheduling resolution - if sensor_d is not None and sensor_d.event_resolution != timedelta(0): + if isinstance(storage_efficiency[d], Sensor): + # Resample from the resolution of the storage-efficiency sensor + device_constraints[d]["efficiency"] **= ( + resolution / storage_efficiency[d].event_resolution + ) + elif sensor_d is not None and sensor_d.event_resolution != timedelta(0): + # Resample from the resolution of the power sensor device_constraints[d]["efficiency"] **= ( resolution / sensor_d.event_resolution ) + elif isinstance(consumption[d], Sensor) and consumption[ + d + ].event_resolution != timedelta(0): + # Resample from the resolution of the consumption sensor + device_constraints[d]["efficiency"] **= ( + resolution / consumption[d].event_resolution + ) + elif isinstance(production[d], Sensor) and production[ + d + ].event_resolution != timedelta(0): + # Resample from the resolution of the production sensor + device_constraints[d]["efficiency"] **= ( + resolution / production[d].event_resolution + ) # check that storage constraints are fulfilled if not skip_validation: @@ -1567,6 +1589,90 @@ def _build_soc_schedule( ) return soc_schedule + @staticmethod + def _build_consumption_production_schedules( + flex_model: list[dict], + ems_schedule: pd.DataFrame, + ) -> dict: + """Build consumption and/or production power schedules for devices that define output sensors. + + Each device's flex model may define a ``consumption`` sensor, a ``production`` sensor, or both. + The schedule stored on each sensor depends on which sensors are defined: + + - **Only** ``consumption`` **sensor defined**: the full power schedule is written to that + sensor using the scheduler's native sign convention (consumption positive, production + negative). ``make_schedule`` applies no further sign change because the sensor already + has ``consumption_is_positive=True``. + - **Only** ``production`` **sensor defined**: the full power schedule is written to that + sensor in the scheduler's native sign convention (consumption positive, production + negative). ``make_schedule`` inverts the sign based on the sensor's + ``consumption_is_positive=False`` attribute so that production is stored as positive values. + - **Both** ``consumption`` **and** ``production`` **sensors defined**: only the non-negative + part of the schedule (charging / consuming) is written to the consumption sensor, and only + the non-positive part (discharging / producing, still as negative values) is written to + the production sensor. ``make_schedule`` inverts the sign for the production sensor. + + The ``consumption_is_positive`` attribute is set on each output sensor when the scheduling + job is created (see ``create_scheduling_job``), not here. This method only clips the + series; sign handling is left entirely to ``make_schedule``. + + Unit conversion from MW to each sensor's unit is applied. + + :param flex_model: List of per-device flex models (after deserialization). + :param ems_schedule: DataFrame of per-device power schedules in MW (consumption positive). + :returns: Dict mapping each output sensor to its power schedule. + """ + schedules: dict = {} + for d, flex_model_d in enumerate(flex_model): + consumption_field = flex_model_d.get("consumption") + production_field = flex_model_d.get("production") + consumption_sensor = ( + consumption_field["sensor"] + if isinstance(consumption_field, dict) and "sensor" in consumption_field + else None + ) + production_sensor = ( + production_field["sensor"] + if isinstance(production_field, dict) and "sensor" in production_field + else None + ) + if consumption_sensor is None and production_sensor is None: + continue + power_series = ems_schedule[d] # in MW; consumption is positive + if consumption_sensor is not None and production_sensor is None: + # Full power profile on the consumption sensor (consumption positive, production negative). + schedules[consumption_sensor] = convert_units( + power_series, + "MW", + consumption_sensor.unit, + event_resolution=consumption_sensor.event_resolution, + ) + elif production_sensor is not None and consumption_sensor is None: + # Full power profile on the production sensor in native scheduler convention. + # make_schedule inverts the sign via consumption_is_positive=False on the sensor. + schedules[production_sensor] = convert_units( + power_series, + "MW", + production_sensor.unit, + event_resolution=production_sensor.event_resolution, + ) + else: + # Both sensors defined: clip to non-negative (consumption) and non-positive (production) parts. + # make_schedule inverts the sign for the production sensor via consumption_is_positive=False. + schedules[consumption_sensor] = convert_units( + power_series.clip(lower=0), + "MW", + consumption_sensor.unit, + event_resolution=consumption_sensor.event_resolution, + ) + schedules[production_sensor] = convert_units( + power_series.clip(upper=0), + "MW", + production_sensor.unit, + event_resolution=production_sensor.event_resolution, + ) + return schedules + def compute(self, skip_validation: bool = False) -> SchedulerOutputType: """Schedule a battery or Charge Point based directly on the latest beliefs regarding market prices within the specified time window. For the resulting consumption schedule, consumption is defined as positive values. @@ -1640,6 +1746,10 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: flex_model, ems_schedule, soc_at_start, device_constraints, resolution ) + consumption_production_schedule = self._build_consumption_production_schedules( + flex_model, ems_schedule + ) + # Resample each device schedule to the resolution of the device's power sensor if self.resolution is None: storage_schedule = { @@ -1649,6 +1759,12 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: for sensor in storage_schedule.keys() if sensor is not None } + consumption_production_schedule = { + sensor: consumption_production_schedule[sensor] + .resample(sensor.event_resolution) + .mean() + for sensor in consumption_production_schedule.keys() + } # Round schedule if self.round_to_decimals: @@ -1661,6 +1777,12 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: sensor: soc_schedule[sensor].round(self.round_to_decimals) for sensor in soc_schedule.keys() } + consumption_production_schedule = { + sensor: consumption_production_schedule[sensor].round( + self.round_to_decimals + ) + for sensor in consumption_production_schedule.keys() + } if self.return_multiple: storage_schedules = [ @@ -1694,7 +1816,32 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: } for sensor, soc in soc_schedule.items() ] - return storage_schedules + commitment_costs + soc_schedules + # Determine which sensors are consumption vs. production output sensors + consumption_output_sensors = { + flex_model_d["consumption"]["sensor"] + for flex_model_d in flex_model + if isinstance(flex_model_d.get("consumption"), dict) + and "sensor" in flex_model_d["consumption"] + } + consumption_production_schedules = [ + { + "name": ( + "consumption_schedule" + if sensor in consumption_output_sensors + else "production_schedule" + ), + "data": data, + "sensor": sensor, + "unit": sensor.unit, + } + for sensor, data in consumption_production_schedule.items() + ] + return ( + storage_schedules + + commitment_costs + + soc_schedules + + consumption_production_schedules + ) else: return storage_schedule[sensors[0]] diff --git a/flexmeasures/data/models/planning/tests/test_storage.py b/flexmeasures/data/models/planning/tests/test_storage.py index dbb5d792f7..16e2608643 100644 --- a/flexmeasures/data/models/planning/tests/test_storage.py +++ b/flexmeasures/data/models/planning/tests/test_storage.py @@ -29,6 +29,24 @@ def test_battery_solver_multi_commitment(add_battery_assets, db): index = initialize_index(start=start, end=end, resolution=resolution) production_prices = pd.Series(90, index=index) consumption_prices = pd.Series(100, index=index) + + # Add consumption and production output sensors to the battery asset + consumption_output_sensor = Sensor( + name="consumption output", + generic_asset=battery.generic_asset, + unit="kW", + event_resolution=resolution, + ) + production_output_sensor = Sensor( + name="production output", + generic_asset=battery.generic_asset, + unit="kW", + event_resolution=resolution, + ) + db.session.add(consumption_output_sensor) + db.session.add(production_output_sensor) + db.session.flush() + scheduler: Scheduler = StorageScheduler( battery, start, @@ -37,6 +55,8 @@ def test_battery_solver_multi_commitment(add_battery_assets, db): flex_model={ "soc-at-start": f"{soc_at_start} MWh", "soc-min": "0 MWh", + "consumption": {"sensor": consumption_output_sensor.id}, + "production": {"sensor": production_output_sensor.id}, "soc-max": "1 MWh", "power-capacity": "1 MVA", "soc-minima": [ @@ -131,6 +151,24 @@ def test_battery_solver_multi_commitment(add_battery_assets, db): costs["a sample commitment penalizing demand/supply"], 1 * (1 - 0.4) ) + # Check consumption/production output sensor schedules. + # The battery charges at a constant rate (all positive values), so the consumption schedule + # should match the power schedule in kW, and the production schedule should be all zeros. + consumption_result = next( + r for r in results if r.get("name") == "consumption_schedule" + ) + production_result = next( + r for r in results if r.get("name") == "production_schedule" + ) + assert consumption_result["sensor"] is consumption_output_sensor + assert consumption_result["unit"] == "kW" + assert production_result["sensor"] is production_output_sensor + assert production_result["unit"] == "kW" + # Both sensors have the same resolution as the power sensor, so no resampling occurs. + expected_kw = (1 - 0.4) / 24 * 1000 # MW -> kW + np.testing.assert_allclose(consumption_result["data"], expected_kw) + np.testing.assert_allclose(production_result["data"], 0) + def test_battery_relaxation(add_battery_assets, db): """Check that resolving SoC breaches is more important than resolving device power breaches. diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 917a6ba990..38ab39a340 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -582,6 +582,24 @@ def _to_currency_per_mwh(price_unit: str) -> str: } UI_FLEX_MODEL_SCHEMA: Dict[str, Dict[str, Any]] = { + "consumption": { + "default": None, + "description": rst_to_openapi(metadata.CONSUMPTION.description), + "types": { + "backend": "typeTwo", + "ui": "A sensor which records the scheduled consumption.", + }, + "example-units": EXAMPLE_UNIT_TYPES["power"], + }, + "production": { + "default": None, + "description": rst_to_openapi(metadata.PRODUCTION.description), + "types": { + "backend": "typeTwo", + "ui": "A sensor which records the scheduled production.", + }, + "example-units": EXAMPLE_UNIT_TYPES["power"], + }, "soc-min": { "default": None, "description": rst_to_openapi(metadata.SOC_MIN.description), @@ -1002,11 +1020,49 @@ def check_flex_model_sensors(self, data, **kwargs): return data +class ScheduleSignConvention: + """Named constants for the three sign-convention modes of the get_schedule endpoint. + + :cvar CONSUMPTION_POSITIVE: Always return schedules with consumption as positive values + and production as negative values. This is the default and + matches the view a *consumer* has of their device. + :cvar PRODUCTION_POSITIVE: Always return schedules with production as positive values + and consumption as negative values. This matches the view a + *producer* (or generator) has of their device. + :cvar WYSIWYG: Return the raw values from the database without any sign inversion, + regardless of the sensor's ``consumption_is_positive`` attribute. + Useful when you want to see exactly what was stored. + """ + + CONSUMPTION_POSITIVE = "consumption-positive" + PRODUCTION_POSITIVE = "production-positive" + WYSIWYG = "wysiwyg" + + ALL = (CONSUMPTION_POSITIVE, PRODUCTION_POSITIVE, WYSIWYG) + + class GetScheduleSchema(Schema): sensor = SensorIdField(required=True, data_key="id") job_id = fields.Str(required=True, data_key="uuid") duration = DurationField(load_default=timedelta(hours=6)) unit = UnitField(load_default=None) + sign_convention = fields.Str( + data_key="sign-convention", + load_default=ScheduleSignConvention.CONSUMPTION_POSITIVE, + validate=validate.OneOf(ScheduleSignConvention.ALL), + metadata=dict( + description=( + "Controls the sign convention applied to schedule values in the response. " + f"``{ScheduleSignConvention.CONSUMPTION_POSITIVE}`` (default): consumption is always returned as positive values " + f"and production as negative values. " + f"``{ScheduleSignConvention.PRODUCTION_POSITIVE}``: production is always returned as positive values " + f"and consumption as negative values. " + f"``{ScheduleSignConvention.WYSIWYG}``: returns values with the same sign as database values and as seen in the UI charts, " + "without adjusting their sign for the sensor's ``consumption_is_positive`` attribute." + ), + example=ScheduleSignConvention.CONSUMPTION_POSITIVE, + ), + ) @post_load def finalize_unit_and_duration(self, data, **kwargs): diff --git a/flexmeasures/data/schemas/scheduling/metadata.py b/flexmeasures/data/schemas/scheduling/metadata.py index cf03767d2b..2aaa23b9a1 100644 --- a/flexmeasures/data/schemas/scheduling/metadata.py +++ b/flexmeasures/data/schemas/scheduling/metadata.py @@ -185,6 +185,32 @@ def to_dict(self): # FLEX-MODEL +CONSUMPTION = MetaData( + description="""Sensor used to record the scheduled power as seen from a consumption perspective. + +The sign convention is determined by the key name, and is stored on the sensor itself using the ``consumption_is_positive`` attribute. + +Depending on which output sensors are defined: + +- **Only** ``consumption`` **defined**: the full power schedule is stored on this sensor using the + consumption-positive sign convention (consumption positive, production negative). +- **Only** ``production`` **defined**: the full power schedule is stored on the production sensor + with the production-positive convention (production positive, consumption negative). +- **Both defined**: only the non-negative part of the schedule is stored on this sensor (zero for + time steps with net production), and only the non-positive part (sign-flipped) is stored on the + production sensor. +""", + example={"sensor": 14}, +) +PRODUCTION = MetaData( + description="""Sensor used to record the scheduled power as seen from a production perspective. + +The sign convention is determined by the key name, and is stored on the sensor itself using the ``consumption_is_positive`` attribute. + +See ``consumption`` for the full description of the split logic when both sensors are defined. +""", + example={"sensor": 15}, +) STATE_OF_CHARGE = MetaData( description="Sensor used to record the scheduled state of charge. If ``soc-at-start`` is omitted, FlexMeasures will also use this field to infer the starting state of charge. For this use case, the field may also contain a time series specification instead. When a sensor is used, its unit may be an energy unit (e.g. MWh or kWh) or a percentage (%). For sensors with a % unit, the ``soc-max`` flex-model field must be set to a non-zero value to allow converting between the energy-based schedule and a percentage. Also, the state-of-charge sensor's resolution should be instantaneous (i.e. `PT0M`).", example={"sensor": 12}, @@ -289,8 +315,10 @@ def to_dict(self): As a result, each time step the energy is held longer leads to higher losses. This setting is crucial to some sorts of energy storage, e.g. thermal buffers. To give an example, when this setting is at 95% (or 0.95), this means a loss of 5% per time step. Defaults to 100% (no storage loss over time). -Note that the storage efficiency used by the scheduler is applied over each time step equal to the sensor resolution. +Note that the storage efficiency used by the scheduler is applied over each time step equal to the scheduling resolution. For example, a storage efficiency of 95 percent per (absolute) day, for scheduling a 1-hour resolution sensor, should be passed as a storage efficiency of :math:`0.95^{1/24} = 0.997865`. +Alternatively, to let FlexMeasures handle the conversion for you, record the storage-efficiency on a dedicated sensor (in this example, with a 24-hour event resolution). +Then reference that sensor in the storage-efficiency field. """, example="99.9%", ) diff --git a/flexmeasures/data/schemas/scheduling/storage.py b/flexmeasures/data/schemas/scheduling/storage.py index e1c60c7143..183565c4b5 100644 --- a/flexmeasures/data/schemas/scheduling/storage.py +++ b/flexmeasures/data/schemas/scheduling/storage.py @@ -18,7 +18,10 @@ from flexmeasures.data.schemas.generic_assets import GenericAssetIdField from flexmeasures.data.schemas.units import QuantityField from flexmeasures.data.schemas.scheduling import metadata -from flexmeasures.data.schemas.sensors import VariableQuantityField +from flexmeasures.data.schemas.sensors import ( + SensorReferenceSchema, + VariableQuantityField, +) from flexmeasures.utils.unit_utils import ( ur, is_power_unit, @@ -82,6 +85,15 @@ class StorageFlexModelSchema(Schema): metadata=dict(description="ID of the asset that is requested to be scheduled."), ) + consumption = fields.Nested( + SensorReferenceSchema, + metadata=metadata.CONSUMPTION.to_dict(), + ) + production = fields.Nested( + SensorReferenceSchema, + metadata=metadata.PRODUCTION.to_dict(), + ) + soc_at_start = QuantityField( required=False, to_unit="MWh", @@ -313,19 +325,6 @@ def validate_asset(self, asset: Asset, **kwargs): if self.sensor is not None and self.sensor.asset != asset: raise ValidationError("Sensor does not belong to asset.") - @validates("storage_efficiency") - def validate_storage_efficiency_resolution( - self, unit: Sensor | ur.Quantity, **kwargs - ): - if ( - self.sensor is not None - and isinstance(unit, Sensor) - and unit.event_resolution != self.sensor.event_resolution - ): - raise ValidationError( - "Event resolution of the storage efficiency and the power sensor don't match. Resampling the storage efficiency is not supported." - ) - @validates_schema def check_redundant_efficiencies(self, data: dict, **kwargs): """ @@ -375,6 +374,9 @@ class DBStorageFlexModelSchema(Schema): Schema for flex-models stored in the db. Supports fixed quantities and sensor references, while disallowing time series specs. """ + consumption = fields.Nested(SensorReferenceSchema) + production = fields.Nested(SensorReferenceSchema) + soc_min = VariableQuantityField( to_unit="MWh", data_key="soc-min", @@ -507,6 +509,22 @@ def __init__(self, *args, **kwargs): for field in self.declared_fields } + @validates("storage_efficiency") + def validate_storage_efficiency_resolution( + self, unit: Sensor | ur.Quantity, **kwargs + ): + if ( + isinstance(unit, ur.Quantity) + and not isinstance(self.consumption, Sensor) + and not isinstance(self.production, Sensor) + ): + raise ValidationError( + "The storage-efficiency cannot be interpreted without a resolution. " + "Record the storage-efficiency on a sensor instead (with a non-zero resolution) and then reference that sensor in the flex-model. " + "Alternatively, set the consumption or production field in the flex-model to reference a sensor, " + "and the scheduler will assume their resolution is the one to use." + ) + @validates_schema def forbid_time_series_specs(self, data: dict, **kwargs): """Do not allow time series specs for the flex-model fields saved in the db.""" diff --git a/flexmeasures/data/services/scheduling.py b/flexmeasures/data/services/scheduling.py index 8c32c01c72..5c7c7886fd 100644 --- a/flexmeasures/data/services/scheduling.py +++ b/flexmeasures/data/services/scheduling.py @@ -254,6 +254,10 @@ def create_scheduling_job( scheduler_kwargs["flex_model"] = scheduler.flex_model scheduler.deserialize_config() + # Set consumption_is_positive on output sensors now (at trigger time) so that any + # attribute conflict raises an error immediately, before the job is enqueued. + _set_flex_model_output_sensors_consumption_is_positive(scheduler.flex_model) + asset_or_sensor = get_asset_or_sensor_ref(asset_or_sensor) job = Job.create( make_schedule, @@ -522,6 +526,168 @@ def create_simultaneous_scheduling_job( return job +def _is_consumption_production_output( + result: dict, asset_or_sensor: Asset | Sensor +) -> bool: + """Return True when *result* is a dedicated consumption or production output schedule. + + A dedicated output schedule is one whose sensor is different from the main asset_or_sensor being scheduled, + and whose name is ``"consumption_schedule"`` or ``"production_schedule"``. + The main power schedule (including the backwards-compat wrapper for custom schedulers that return a plain Series) + uses the same sensor as *asset_or_sensor* and is therefore not considered an output schedule. + + :param result: Schedule output result dict with keys 'name', 'sensor', 'data'. + :param asset_or_sensor: The Asset or Sensor being scheduled (main power sensor). + :return: True when the result targets a dedicated output sensor. + """ + result_name = result.get("name", "") + if result_name not in ("consumption_schedule", "production_schedule"): + return False + + result_sensor = result["sensor"] + # The main power schedule uses the same object as asset_or_sensor, + # or – when asset_or_sensor is an Asset – one of its sensors. + is_main = result_sensor == asset_or_sensor or ( + hasattr(asset_or_sensor, "generic_asset") + and result_sensor.generic_asset == asset_or_sensor + ) + return not is_main + + +def _set_flex_model_output_sensors_consumption_is_positive( + flex_model: dict | list, +) -> None: + """Set the ``consumption_is_positive`` attribute on consumption and production output sensors. + + Iterates over every device in *flex_model* and assigns:: + + consumption sensor → consumption_is_positive = True + production sensor → consumption_is_positive = False + + A ``ValueError`` is raised immediately when the attribute is already present on a sensor + but has the wrong value for the flex-model field that references it. Calling this function + at job-creation time lets the API surface conflicts before any work is queued. + + :param flex_model: Deserialized flex model — either a single-device ``dict`` or a + ``list`` of per-device dicts. Consumption/production fields are + expected to be dicts with a ``"sensor"`` key. + :raises ValueError: When ``consumption_is_positive`` is already set to the wrong value + for the given flex-model field. + """ + models = flex_model if isinstance(flex_model, list) else [flex_model] + for flex_model_d in models: + consumption_field = flex_model_d.get("consumption") + production_field = flex_model_d.get("production") + consumption_sensor = ( + consumption_field.get("sensor") + if isinstance(consumption_field, dict) + else None + ) + production_sensor = ( + production_field.get("sensor") + if isinstance(production_field, dict) + else None + ) + for sensor, intended in [ + (consumption_sensor, True), + (production_sensor, False), + ]: + if sensor is None: + continue + field_name = "consumption_schedule" if intended else "production_schedule" + existing = sensor.attributes.get("consumption_is_positive") + if existing is not None and existing != intended: + raise ValueError( + f"Sensor {sensor} already has `consumption_is_positive={existing}`, " + f"which conflicts with the '{field_name}' output schedule " + f"(expected `consumption_is_positive={intended}`). " + f"Remove or correct the attribute before running the scheduler." + ) + sensor.attributes["consumption_is_positive"] = intended + + +def _set_output_sensor_consumption_is_positive( + result: dict, asset_or_sensor: Asset | Sensor +) -> None: + """Set the ``consumption_is_positive`` attribute on a dedicated output sensor. + + For consumption output sensors the attribute is set to ``True`` (consumption is stored as + positive values). For production output sensors it is set to ``False`` (production is stored + as positive values, consumption as negative). + + The function is a no-op when *result* is not a dedicated consumption/production output + schedule (as determined by :func:`_is_consumption_production_output`). + + A ``ValueError`` is raised when the attribute is already present on the sensor but points + in the wrong direction for the flex-model field being used. This check runs *before* any + data are written so that the error surfaces as early as possible. + + :param result: Schedule output result dict with keys ``'name'``, ``'sensor'``, + ``'data'``. + :param asset_or_sensor: The Asset or Sensor being scheduled (main power sensor). + :raises ValueError: When ``consumption_is_positive`` is already set to the wrong value + for the given flex-model field. + """ + if not _is_consumption_production_output(result, asset_or_sensor): + return + + result_sensor = result["sensor"] + result_name = result.get("name", "") + # consumption_schedule → True (consumption positive) + # production_schedule → False (production positive, i.e. consumption negative) + intended = result_name == "consumption_schedule" + existing = result_sensor.attributes.get("consumption_is_positive") + if existing is not None and existing != intended: + raise ValueError( + f"Sensor {result_sensor} already has `consumption_is_positive={existing}`, " + f"which conflicts with the '{result_name}' output schedule " + f"(expected `consumption_is_positive={intended}`). " + f"Remove or correct the attribute before re-running the scheduler." + ) + # Direct attribute assignment works for both new and existing attributes. + # set_attribute() is intentionally not used here because it silently + # no-ops when the attribute does not yet exist. + result_sensor.attributes["consumption_is_positive"] = intended + + +def _resolve_schedule_output_sign( + result: dict, + asset_or_sensor: Asset | Sensor, +) -> int: + """Determine the sign multiplier for a schedule output result. + + Returns 1 (no sign change) or -1 (invert sign) depending on whether the result + is a power schedule that needs sign conversion so that production is stored as positive + values in the database. + + The scheduler always produces consumption-positive values. For sensors that carry + ``consumption_is_positive=True`` (including consumption output sensors) no conversion + is needed. For sensors with ``consumption_is_positive=False`` (production output sensors + and the default convention for main power sensors) the sign is inverted. + + .. note:: + For consumption/production output sensors the ``consumption_is_positive`` attribute + must be set before this function is called. It is set eagerly at job-creation time + by :func:`_set_flex_model_output_sensors_consumption_is_positive`, and again (as a + safety-net for direct :func:`make_schedule` calls) by + :func:`_set_output_sensor_consumption_is_positive` earlier in the same loop iteration. + + :param result: Schedule output result dict with keys 'name', 'sensor', 'data'. + :param asset_or_sensor: The Asset or Sensor being scheduled (main power sensor). + :return: Sign multiplier: 1 (keep sign) or -1 (invert sign). + """ + result_sensor = result["sensor"] + + # Apply sign inversion for power sensors that record production as positive values + # (i.e. those that do not carry consumption_is_positive=True). + if result_sensor.measures_power and not result_sensor.get_attribute( + "consumption_is_positive", False + ): + return -1 + + return 1 + + def make_schedule( # noqa: C901 sensor_id: int | None = None, start: datetime | None = None, @@ -637,12 +803,13 @@ def make_schedule( # noqa: C901 if "sensor" not in result: continue - sign = 1 + # Ensure consumption_is_positive is set before resolving the sign. + # At job-creation time this is already done eagerly; calling it here again + # acts as a safety net for direct make_schedule invocations and raises a + # ValueError on attribute conflicts before any data are written. + _set_output_sensor_consumption_is_positive(result, asset_or_sensor) - if result["sensor"].measures_power and not result["sensor"].get_attribute( - "consumption_is_positive", False - ): - sign = -1 + sign = _resolve_schedule_output_sign(result, asset_or_sensor) ts_value_schedule = [ TimedBelief( diff --git a/flexmeasures/ui/static/openapi-specs.json b/flexmeasures/ui/static/openapi-specs.json index 757c1b02bb..277e3c5ed4 100644 --- a/flexmeasures/ui/static/openapi-specs.json +++ b/flexmeasures/ui/static/openapi-specs.json @@ -690,7 +690,7 @@ "/api/v3_0/sensors/{id}/schedules/{uuid}": { "get": { "summary": "Get schedule for one device", - "description": "Get a schedule from FlexMeasures.\n\nOptional fields:\n\n- \"duration\" (6 hours by default; can be increased to plan further into the future)\n- \"unit\" (by default, the unit of the schedule is the sensor's unit; a compatible unit can be requested)\n", + "description": "Get a schedule from FlexMeasures.\n\nOptional fields:\n\n- \"duration\" (6 hours by default; can be increased to plan further into the future)\n- \"unit\" (by default, the unit of the schedule is the sensor's unit; a compatible unit can be requested)\n- \"sign-convention\" (controls how power values are signed in the response; see below)\n\nSign convention\n\nBy default (sign-convention: consumption-positive), the endpoint always returns schedules where\nconsumption is positive and production is negative, regardless of how the values are stored in the database.\nThis is the most common convention and matches the perspective of a consumer.\n\nSet sign-convention: production-positive to flip the sign so that production is returned as\npositive and consumption as negative. This matches the perspective of a producer.\n\nSet sign-convention: wysiwyg (what-you-see-is-what-you-get) to return the values with the same sign\nas database values and what is seen in UI charts. The values will indicate exactly what is stored,\nwhich is itself determined by the sensor's consumption_is_positive attribute (if set)\nor by the scheduler's default storage convention (production positive in the database).\n", "security": [ { "ApiKeyAuth": [] @@ -736,6 +736,21 @@ "schema": { "type": "string" } + }, + { + "in": "query", + "name": "sign-convention", + "required": false, + "description": "Sign convention applied to power values in the response.\n- consumption-positive (default): consumption is positive, production is negative.\n- production-positive: production is positive, consumption is negative.\n- wysiwyg (what-you-see-is-what-you-get): sign of database values and as seen in UI charts.\n", + "example": "consumption-positive", + "schema": { + "type": "string", + "enum": [ + "consumption-positive", + "production-positive", + "wysiwyg" + ] + } } ], "responses": { @@ -5999,6 +6014,20 @@ "StorageFlexModelSchemaOpenAPI": { "type": "object", "properties": { + "consumption": { + "description": "Sensor used to record the scheduled power as seen from a consumption perspective.\n\nThe sign convention is determined by the key name, and is stored on the sensor itself using the consumption_is_positive attribute.\n\nDepending on which output sensors are defined:\n\n- Only consumption defined: the full power schedule is stored on this sensor using the\n consumption-positive sign convention (consumption positive, production negative).\n- Only production defined: the full power schedule is stored on the production sensor\n with the production-positive convention (production positive, consumption negative).\n- Both defined: only the non-negative part of the schedule is stored on this sensor (zero for\n time steps with net production), and only the non-positive part (sign-flipped) is stored on the\n production sensor.\n", + "example": { + "sensor": 14 + }, + "$ref": "#/components/schemas/SensorReference" + }, + "production": { + "description": "Sensor used to record the scheduled power as seen from a production perspective.\n\nThe sign convention is determined by the key name, and is stored on the sensor itself using the consumption_is_positive attribute.\n\nSee consumption for the full description of the split logic when both sensors are defined.\n", + "example": { + "sensor": 15 + }, + "$ref": "#/components/schemas/SensorReference" + }, "soc-at-start": { "type": "string", "description": "The (estimated) state of charge at the beginning of the schedule (for storage devices, this defaults to 0).\nUsually added to each scheduling request.\n", @@ -6114,7 +6143,7 @@ "example": "90%" }, "storage-efficiency": { - "description": "The efficiency of keeping the storage's state of charge at its present level, used to encode losses over time.\nAs a result, each time step the energy is held longer leads to higher losses.\nThis setting is crucial to some sorts of energy storage, e.g. thermal buffers.\nTo give an example, when this setting is at 95% (or 0.95), this means a loss of 5% per time step. Defaults to 100% (no storage loss over time).\nNote that the storage efficiency used by the scheduler is applied over each time step equal to the sensor resolution.\nFor example, a storage efficiency of 95 percent per (absolute) day, for scheduling a 1-hour resolution sensor, should be passed as a storage efficiency of 0.951/24 = 0.997865.\n", + "description": "The efficiency of keeping the storage's state of charge at its present level, used to encode losses over time.\nAs a result, each time step the energy is held longer leads to higher losses.\nThis setting is crucial to some sorts of energy storage, e.g. thermal buffers.\nTo give an example, when this setting is at 95% (or 0.95), this means a loss of 5% per time step. Defaults to 100% (no storage loss over time).\nNote that the storage efficiency used by the scheduler is applied over each time step equal to the scheduling resolution.\nFor example, a storage efficiency of 95 percent per (absolute) day, for scheduling a 1-hour resolution sensor, should be passed as a storage efficiency of 0.951/24 = 0.997865.\nAlternatively, to let FlexMeasures handle the conversion for you, record the storage-efficiency on a dedicated sensor (in this example, with a 24-hour event resolution).\nThen reference that sensor in the storage-efficiency field.\n", "example": "99.9%", "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, diff --git a/flexmeasures/ui/tests/test_utils.py b/flexmeasures/ui/tests/test_utils.py index 57f42bb166..09a68ff005 100644 --- a/flexmeasures/ui/tests/test_utils.py +++ b/flexmeasures/ui/tests/test_utils.py @@ -106,7 +106,7 @@ def test_ui_flexmodel_schema(): schema_keys = [] for value in DBStorageFlexModelSchema().fields.values(): - schema_keys.append(value.data_key) + schema_keys.append(value.data_key if value.data_key else value.name) schema_keys = set(schema_keys) ui_flexmodel_schema_fields = set(ui_flexmodel_schema_fields)