Inherit from Dask Dataframe and respond to cudf update

[mrocklin]

This is a large refactor with a variety of changes:

1. We now depend on the mainline dask dataframe codebase,
   This allows us to reuse existing code and should make it easier to bring
   over existing functionality in the future, but adds the expectation that
   cudf and pandas will look more alike in the future.

2. We change the default scheduler to single-threaded.

   This is because some parts of cudf seem to not be threadsafe
   (we'll raise an issue for this and handle it in followup work)

3. We xfail many of the tests in groupby and categoricals,
   which seem to be failing regardless

Supercedes #43

[mrocklin]

To be clear, we're losing functionality here relative to previous releases (assuming that they're pinned to previous cudf releases). I think that this is a win generally though. It will actually work with the recent cudf release, and puts us on track to move forward more quickly.

However, we are making a bet here that cudf will start to adhere more strongly to pandas semantics, which may not actually occur.

[randerzander]

@mrocklin thanks for this PR. When trying it out with cudf/branch-0.5, I notice the following:

from dask.distributed import Client
from dask.delayed import delayed
import dask.dataframe as dd
import dask_cudf

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-3c55a73a0651> in <module>
      3 from dask.delayed import delayed
      4 import dask.dataframe as dd
----> 5 import dask_cudf

/conda/envs/cudf/lib/python3.5/site-packages/dask_cudf-0.0.1+218.gc3729ba-py3.5.egg/dask_cudf/__init__.py in <module>
----> 1 from .core import DataFrame, Series, from_cudf, from_dask_dataframe, concat, \
      2     from_delayed
      3 
      4 from cudf._version import get_versions
      5 __version__ = get_versions()['version']

/conda/envs/cudf/lib/python3.5/site-packages/dask_cudf-0.0.1+218.gc3729ba-py3.5.egg/dask_cudf/core.py in <module>
    918 
    919 
--> 920 @dd.core.get_parallel_type.register(cudf.DataFrame)
    921 def get_parallel_type_dataframe(_):
    922     return DataFrame

AttributeError: module 'dask.dataframe.core' has no attribute 'get_parallel_type'

I'm running Dask '0.19.2'. Is there a particular version I should use instead?

[mrocklin]

Yes. I recommend that you bump up to latest release at 1.0.0, but I suspect that anything newer than 0.20 would work.

…

On Sat, Dec 15, 2018 at 5:22 PM Randy Gelhausen ***@***.***> wrote: @mrocklin <https://github.com/mrocklin> thanks for this PR. When trying it out with cudf/branch-0.5, I notice the following: import dask from dask.distributed import Client from dask.delayed import delayed import dask.dataframe as dd import dask_cudf import cudf --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-2-3c55a73a0651> in <module> 3 from dask.delayed import delayed 4 import dask.dataframe as dd ----> 5 import dask_cudf /conda/envs/cudf/lib/python3.5/site-packages/dask_cudf-0.0.1+218.gc3729ba-py3.5.egg/dask_cudf/__init__.py in <module> ----> 1 from .core import DataFrame, Series, from_cudf, from_dask_dataframe, concat, \ 2 from_delayed 3 4 from cudf._version import get_versions 5 __version__ = get_versions()['version'] /conda/envs/cudf/lib/python3.5/site-packages/dask_cudf-0.0.1+218.gc3729ba-py3.5.egg/dask_cudf/core.py in <module> 918 919 --> 920 @dd.core.get_parallel_type.register(cudf.DataFrame) 921 def get_parallel_type_dataframe(_): 922 return DataFrame AttributeError: module 'dask.dataframe.core' has no attribute 'get_parallel_type' I'm running Dask '0.19.2'. Is there a particular version I should use instead? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#48 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszHaWWRqM26EbKEWCLCsz3FwwZwrtks5u5XYMgaJpZM4ZUFpO> .

[mrocklin]

@randerzander I'm curious if things ended up working out for you here, or if there were other issues that you encountered.

[mrocklin]

@kkraus14 can I have commit access to this repository?

[kkraus14]

@kkraus14 can I have commit access to this repository?

Should be done, let me know if you see otherwise.