-
Notifications
You must be signed in to change notification settings - Fork 1
dask doesn't support rolling over multiple partitions #48
Copy link
Copy link
Open
Description
Getting this error from error_detect which uses dask method.
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "d:\ProgramData\miniconda3\envs\dms_datastore\Lib\site-packages\vtools\functions\error_detect.py", line 279, in median_test_twoside
filt = dds.rolling(filt_len,center=True).apply(lambda x: np.nanmedian(x[medseq]),raw=True,engine='numba').compute()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "d:\ProgramData\miniconda3\envs\dms_datastore\Lib\site-packages\dask_expr\_collection.py", line 481, in compute
return DaskMethodsMixin.compute(out, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "d:\ProgramData\miniconda3\envs\dms_datastore\Lib\site-packages\dask\base.py", line 372, in compute
(result,) = compute(self, traverse=False, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "d:\ProgramData\miniconda3\envs\dms_datastore\Lib\site-packages\dask\base.py", line 660, in compute
results = schedule(dsk, keys, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "d:\ProgramData\miniconda3\envs\dms_datastore\Lib\site-packages\dask_expr\_expr.py", line 1043, in _combined_parts
raise NotImplementedError(msg)
NotImplementedError: Partition size is less than overlapping window size. Try using ``df.repartition`` to increase the partition size.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels