import: remote file can be downloaded via `dvc get` but can't be downloaded with import

[hv10]

Bug Report

Description

When using a dvc repo as a data registry which uses an sshfs as remote we ran into issues importing data into depended repositories, specifically importing a directory which includes several files, while the directory itself is the object tracked added to the remote as a stage output.

Reproduce

1. have a repo that uses sshfs as default remote (using ssh-key auth)
2. init new repo
3. dvc import a directory --> leads to PermissionError
4. dvc get the same directory --> works as expected

Expected

dvc import and dvc get should both be able to pull the directory and files in question.

Environment information

• there exists a ssh config containing the config with two keyfiles, one for the repo host (gitlab) and one for the ssh remote (our lab cluster)

Output of dvc doctor:

DVC version: 2.29.0 (pip)
---------------------------------
Platform: Python 3.10.6 on macOS-12.6-arm64-arm-64bit
Subprojects:
	dvc_data = 0.14.0
	dvc_objects = 0.5.0
	dvc_render = 0.0.11
	dvc_task = 0.1.3
	dvclive = 0.11.0
	scmrepo = 0.1.1
Supports:
	http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
	ssh (sshfs = 2022.6.0)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s3s1
Caches: local
Remotes: ssh, local
Workspace directory: apfs on /dev/disk3s3s1
Repo: dvc, git

Additional Information (if any):

Additional Info will be added when I tried the Debugging method from the Wiki.
Sadly I can not publish the output for dvc import -v <directory> for now as I would have to clean it from references to our cluster.
I know its fairly vague, but I do not have a clue where the issue could come from, therefore I would be happy for any pointers on where to look.

[hv10]

A note: the issue exists both on linux and macos for me, I guess it's some configuration error I am unaware of, will investigate.

[daavoo]

Hi @hv10 ! could you share some more details on the permission error you are getting?

The traceback with the detailed exception and point in code should not contain any sensitive information

[hv10]

dvc import -v git@gitlab.com:<repo_name> <file_path>
2022-10-06 12:28:17,754 ERROR: failed to transfer 'md5: ba9dbba77caf30f81ecba107250ee945' - Permission denied: Permission denied
------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/sshfs/utils.py", line 27, in wrapper
    return await func(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/sshfs/spec.py", line 91, in _connect
    client = await self._stack.enter_async_context(_raw_client)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/contextlib.py", line 619, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/asyncssh/misc.py", line 274, in __aenter__
    self._coro_result = await self._coro
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/asyncssh/connection.py", line 7834, in connect
    return await asyncio.wait_for(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/asyncssh/connection.py", line 447, in _connect
    await options.waiter
asyncssh.misc.PermissionDenied: Permission denied

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 31, in wrapper
    func(path, *args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 170, in func
    return dest.add(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_data/hashfile/db/__init__.py", line 105, in add
    super().add(path, fs, oid, hardlink=hardlink, callback=callback)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/db.py", line 136, in add
    generic.transfer(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 101, in transfer
    _try_links(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 64, in _try_links
    return copy(from_fs, from_path, to_fs, to_path, callback=callback)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 44, in copy
    return from_fs.get_file(from_path, tmp_file, callback=callback)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 387, in get_file
    self.fs.get_file(from_info, to_info, callback=callback, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/fs/dvc.py", line 317, in get_file
    dvc_fs.get_file(dvc_path, lpath, callback=callback, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 387, in get_file
    self.fs.get_file(from_info, to_info, callback=callback, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_data/fs.py", line 117, in get_file
    fs.get_file(path, lpath, callback=callback, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 387, in get_file
    self.fs.get_file(from_info, to_info, callback=callback, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/funcy/objects.py", line 50, in __get__
    return prop.__get__(instance, type)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/funcy/objects.py", line 28, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_ssh/__init__.py", line 114, in fs
    return _SSHFileSystem(**self.fs_args)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/fsspec/spec.py", line 76, in __call__
    obj = super().__call__(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/sshfs/spec.py", line 76, in __init__
    self._client, self._pool = self.connect(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/fsspec/asyn.py", line 111, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/fsspec/asyn.py", line 96, in sync
    raise return_result
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/sshfs/utils.py", line 29, in wrapper
    raise PermissionError(exc.reason) from exc
PermissionError: Permission denied
------------------------------------------------------------
2022-10-06 12:28:17,831 DEBUG: Removing '/Users/hv10/Software_Projects/my-import-repo/.dvc/cache/7d/.Mx8DLLzEwpNXvtbr3fnVZm.tmp'
2022-10-06 12:28:17,831 ERROR: failed to transfer 'md5: 7d5bb9babeffeb0ce9016e905756208b' - Permission denied: Permission denied
------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/sshfs/utils.py", line 27, in wrapper
    return await func(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/sshfs/spec.py", line 91, in _connect
    client = await self._stack.enter_async_context(_raw_client)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/contextlib.py", line 619, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/asyncssh/misc.py", line 274, in __aenter__
    self._coro_result = await self._coro
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/asyncssh/connection.py", line 7834, in connect
    return await asyncio.wait_for(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/asyncssh/connection.py", line 447, in _connect
    await options.waiter
asyncssh.misc.PermissionDenied: Permission denied

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 31, in wrapper
    func(path, *args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 170, in func
    return dest.add(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_data/hashfile/db/__init__.py", line 105, in add
    super().add(path, fs, oid, hardlink=hardlink, callback=callback)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/db.py", line 136, in add
    generic.transfer(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 101, in transfer
    _try_links(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 64, in _try_links
    return copy(from_fs, from_path, to_fs, to_path, callback=callback)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 44, in copy
    return from_fs.get_file(from_path, tmp_file, callback=callback)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 387, in get_file
    self.fs.get_file(from_info, to_info, callback=callback, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/fs/dvc.py", line 317, in get_file
    dvc_fs.get_file(dvc_path, lpath, callback=callback, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 387, in get_file
    self.fs.get_file(from_info, to_info, callback=callback, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_data/fs.py", line 117, in get_file
    fs.get_file(path, lpath, callback=callback, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 387, in get_file
    self.fs.get_file(from_info, to_info, callback=callback, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/funcy/objects.py", line 50, in __get__
    return prop.__get__(instance, type)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/funcy/objects.py", line 28, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_ssh/__init__.py", line 114, in fs
    return _SSHFileSystem(**self.fs_args)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/fsspec/spec.py", line 76, in __call__
    obj = super().__call__(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/sshfs/spec.py", line 76, in __init__
    self._client, self._pool = self.connect(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/fsspec/asyn.py", line 111, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/fsspec/asyn.py", line 96, in sync
    raise return_result
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/sshfs/utils.py", line 29, in wrapper
    raise PermissionError(exc.reason) from exc
PermissionError: Permission denied
------------------------------------------------------------
2022-10-06 12:28:17,832 DEBUG: failed to upload full contents of 'md5: ba73d37455dff7222084843515b58115.dir', aborting .dir file upload
2022-10-06 12:28:17,833 DEBUG: failed to upload 'memory://.jnXQjWdbRH6yReZSCxKzAv.tmp' to '/Users/hv10/Software_Projects/my-import-repo/.dvc/cache/ba/73d37455dff7222084843515b58115.dir'
2022-10-06 12:28:17,834 ERROR: failed to import '/data/artifacts/cases_rki' from '<my-data-repo (git@host adress)>'. - 3 files failed to transfer: 3 transfer failed
------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/data_cloud.py", line 114, in transfer
    return transfer(src_odb, dest_odb, objs, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 184, in transfer
    _do_transfer(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 119, in _do_transfer
    raise TransferError(total_fails)
dvc_data.hashfile.transfer.TransferError: 3 transfer failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/commands/imp.py", line 15, in run
    self.repo.imp(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/repo/imp.py", line 6, in imp
    return self.imp_url(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 156, in run
    return method(repo, *args, **kw)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/repo/imp_url.py", line 88, in imp_url
    stage.run(jobs=jobs, no_download=no_download)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/funcy/decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/stage/decorators.py", line 43, in rwlocked
    return call()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/funcy/decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/stage/__init__.py", line 554, in run
    self._sync_import(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/funcy/decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/stage/decorators.py", line 43, in rwlocked
    return call()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/funcy/decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/stage/__init__.py", line 580, in _sync_import
    sync_import(self, dry, force, jobs, no_download)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/stage/imports.py", line 60, in sync_import
    stage.deps[0].download(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/dependency/repo.py", line 69, in download
    self.repo.cloud.pull(objs, jobs=jobs, odb=odb)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/data_cloud.py", line 162, in pull
    return self.transfer(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-data-repo/lib/python3.10/site-packages/dvc/data_cloud.py", line 116, in transfer
    raise FileTransferError(exc.fails) from exc
dvc.exceptions.FileTransferError: 3 files failed to transfer
------------------------------------------------------------
2022-10-06 12:28:17,840 DEBUG: Analytics is enabled.
2022-10-06 12:28:17,895 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/jl/qpj5k6q55896ggyklz87ysb80000gn/T/tmp_0cal_wi']'
2022-10-06 12:28:17,897 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/jl/qpj5k6q55896ggyklz87ysb80000gn/T/tmp_0cal_wi']'

I had to remove some of the output - specifically the mentions of the repository. I hope that it will help nonetheless.

[hv10]

When running the dvc getcommand, I now get a PermissionDenied error as well, but the contained files of the folder got downloaded anyways... yesterday they got downloaded without the error.

[hv10]

The folder contains a .gitignore file which does not get downloaded when using dvc get, the two stage outputs contained in the folder do get downloaded.

[edwardwbarber]

@hv10 your traceback includes the same asyncssh.misc.PermissionDenied: Permission denied error folks reported in treeverse/dvc#7702 for window users using ssh-agent (granted, supposedly that is a windows-only issue). Have you tried downgrading to 2.9.5? Not a long-term solution, but better than nothing.

[hv10]

Referencing the issue treeverse/dvc#7702 in question, it seems that the issue is similar indeed. When running the snippet provided by a comment in the issue (dvc/issues/7702) it seems that within my environment the keys do not get picked up either.

I'll try downgrading next. Any major downsides to that? Any features I would dearly miss?

[edwardwbarber]

Interesting - you might want to comment in that thread if the ssh-agent issue is affecting you on linux/macOS too.

2.9.5 was before dvc data status existed. It's also not compatible with the dvc VSCode extension, if you use that. I haven't done an exhaustive check of the release notes, but those are the main two that have stood out to me.

[daavoo]

I am not really sure this is the same issue as treeverse/dvc#7702 because of:

A note: the issue exists both on linux and macos for me, I guess it's some configuration error I am unaware of, will investigate.

And:

dvc get the same directory --> works as expected