Detect shadows from fixed objects in GHI data by wfvining · Pull Request #101 · pvlib/pvanalytics

wfvining · 2021-02-11T21:13:18Z

Description

This is a port of the pvl_detect_shadows function from MATLAB_PVLIB.

Checklist

Closes Add shadow detection #24
Added new API functions to docs/api.rst
Clearly documented all new API functions with PEP257 and numpydoc compliant docstrings
Adds description and name entries in the appropriate "what's new" file
in docs/whatsnew
for all changes. Includes link to the GitHub Issue with :issue:`num`
or this Pull Request with :pull:`num`. Includes contributor name
and/or GitHub username (link with :ghuser:`user`).
Non-API functions clearly documented with docstrings or comments as necessary
Added tests to cover all new or modified code
Pull request is nearly complete and ready for detailed review

This is a rought implementation that returns almost the correct shadow image. There is still some work to be done. The output needs to be returned as a series instead of an image and steps should be taken to minimize memory use, especially in the _clean_wires() function, by passing ndarrays rather than allocating a new array for every step. Currently this implementation appears to be more sensitive than the matlab version, and includes more remnants of clouds in the output.

Initially adds basic tests for data without any fixed shadows.

In the remove_pillars step the masks were passed in the wrong order.

This is consistent with the comments in the original matlab code, although it is not 100% clear to me that it matches the implementation. Since it produces results that are closer to the output (in addition to matching the comments) from the MATLAB code, I will assume that my reading of the original implementation was not correct.

Changes to better match the original implementation. Still not completely sure that this correct though.

Rather than applying it on successively more cleaned up images.

This should be slightly more efficient, plus it matched the intent of the code better. Also updates name of structuring element to match its shape

Converts the image back to a series and re-indexes it like the input series.

wfvining · 2021-03-24T17:08:45Z

@cwhanse I've pretty much got the Matlab version ported over. This version does not seem to do quite as good a job removing cloud shadows that are "connected" to wires in the demo data from PVLIB_MATLAB. If you have a little time, I would appreciate a second pair of eyes on the clean_wires function (or all of it really). Any other feedback on what parameters should be exposed to the user is also very welcome.

Still some work to do on tests and performance. As it stands it is pretty slow, but there are some obvious places for performance improvement (notably in the clean_wires function which can be made to operate in place rather than repeatedly copying the image).

cwhanse

It looks well structured and readable. Do we want to add a test that produces an image similar to the output of the matlab code?

Co-authored-by: Cliff Hansen <cwhanse@sandia.gov>

Adds a better docstring for the returned series.

Following NEP 29, we can now require at least 0.24.0 which was released 24 months ago.

Without specifying an output array the image processing and numpy functions will allocate a new array for every operation. This can be a very expensive operation; fortunately we can avoid it by using a temporary array for the morphological operations and specifying an output array for the resulting image with horizontal bars removed.

Improves performance by reducing the number of new arrays that must be allocated.

wfvining · 2021-04-22T18:40:48Z

@cwhanse I think this is ready for a closer look. The test failure is in the features.clearsky.reno(), not features.shading, they will need to be resolved as part of #105. As far as I can tell, with the exception of the comment above, I have faithfully re-implemented the matlab code, but this version is not doing quite as good a job of eliminating the cloud shadows that are 'connected' to wire shadows.

…hadows

wfvining · 2021-05-28T17:04:40Z

I verified that the structuring elements used in _filter_bars() include a vertical bar (spanning multiple days) and exclude a horizontal bar (within a single day).

Add image to return values

cwhanse

Nice piece of code @wfvining

Co-authored-by: Cliff Hansen <cwhanse@sandia.gov>

wfvining added 15 commits February 11, 2021 14:04

Add test module for features.shading

2fcfb9e

Initially adds basic tests for data without any fixed shadows.

Correct order of masks passed to binary hit/miss filter

5facb77

In the remove_pillars step the masks were passed in the wrong order.

Clean up variable naming and comments in shading.fixed()

a532ea6

Tweak shading._filter_bars()

370fc06

Changes to better match the original implementation. Still not completely sure that this correct though.

Apply _filter_bars() to the raw shadow image

25144fb

Rather than applying it on successively more cleaned up images.

Make min_gradient a parameter to shading.fixed()

e94b2f8

Use morphology.remove_small_objects in initial shadow detection

54ac08e

This should be slightly more efficient, plus it matched the intent of the code better. Also updates name of structuring element to match its shape

Make the minimum duration of cloudy times a parameter

36f3f60

Return shadows as a series and an image

f482827

Converts the image back to a series and re-indexes it like the input series.

Fix errors in test_shading.test_fixed_same_index

7ea77fb

Test that a fake shadow at the same time every day is detected

574afa0

Fix grammar in comment

1040f85

Add scikit-image to dependencies

a130e78

cwhanse reviewed Mar 24, 2021

View reviewed changes

wfvining and others added 5 commits March 30, 2021 09:28

Use a 3x3 mask in shading._smooth()

d88e19c

Remove extra calls to _to_image() in shading._prepare_images()

ef32bec

Docstring and comment updates

1dd87da

Raise a ValueError for data with invalid intervals

b35d37b

Apply docstring edits from code review

8522144

Co-authored-by: Cliff Hansen <cwhanse@sandia.gov>

wfvining commented Mar 30, 2021

View reviewed changes

Comment thread pvanalytics/features/shading.py Outdated

wfvining added 5 commits March 30, 2021 10:59

Rename window to duration in shading._detect_clouds()

15c448a

Adds a better docstring for the returned series.

Bump minimum pandas version to 0.24.0

f1d1bc4

Following NEP 29, we can now require at least 0.24.0 which was released 24 months ago.

Add parameter descriptions to shading._prepare_images()

10e6954

Remove unused shading._width() function

6485079

wholmgren mentioned this pull request Apr 21, 2021

implement shading flag SolarArbiter/solarforecastarbiter-core#672

Open

Pass output array to _fill_gaps() and _remove_spikes()

4b0d2e3

Improves performance by reducing the number of new arrays that must be allocated.

wfvining commented Apr 22, 2021

View reviewed changes

Comment thread pvanalytics/features/shading.py

wfvining and others added 3 commits May 13, 2021 11:41

Merge branch 'master' into shadows

847c8e1

debugging

469a4a1

Merge branch 'shadows' of https://github.com/pvlib/pvanalytics into s…

3ab7488

…hadows

wfvining added 2 commits May 28, 2021 11:25

Tidy up doc string

1788e43

Add image to return values

Add shading to api.rst and whatsnew

ac18e80

wfvining marked this pull request as ready for review May 28, 2021 17:28

cwhanse approved these changes Jun 3, 2021

View reviewed changes

Comment thread pvanalytics/features/shading.py Outdated

Comment thread pvanalytics/features/shading.py Outdated

Comment thread pvanalytics/features/shading.py Outdated

Comment thread pvanalytics/features/shading.py Outdated

Comment thread pvanalytics/features/shading.py Outdated

wfvining and others added 2 commits June 3, 2021 13:14

Docstring edits

e97a3bf

Co-authored-by: Cliff Hansen <cwhanse@sandia.gov>

Add features.shading to package overview in index.rst

8b2b17d

wfvining merged commit a3d3860 into master Jun 3, 2021

kandersolar added this to the v0.1.1 milestone Feb 18, 2022

kandersolar deleted the shadows branch February 12, 2024 18:46

Conversation

wfvining commented Feb 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

wfvining commented Mar 24, 2021

Uh oh!

cwhanse left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wfvining commented Apr 22, 2021

Uh oh!

wfvining commented May 28, 2021

Uh oh!

cwhanse left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wfvining commented Feb 11, 2021 •

edited

Loading