From aceafa5147bd73b9f4d3b39b30522da74d60b215 Mon Sep 17 00:00:00 2001 From: Volodymyr Savchenko Date: Mon, 8 May 2023 16:14:22 +0200 Subject: [PATCH 1/9] first draft started --- .../012-facilitated-storage-compute-access.md | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md diff --git a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md new file mode 100644 index 0000000..ea25109 --- /dev/null +++ b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md @@ -0,0 +1,74 @@ +- Start Date: 2023-05-08 +- Status: Proposed + +# Facilitated Access to Storage and Compute Resources + +## Summary + +> One paragraph explanation of the change. + +In the context of large astrophysical projects, an important benefit of an interactive data analysis and exporation platform like renkulab is to provide direct access to data and computing resources close to the platform, saving on transfer costs. Since access to these resources is typically restricted, it would be useful to use link the resource access account to the renkulab session and allow users to access the resources without any additional authentication. + +## Problem Statement + +> Why are we doing this? What use cases does it support? What is the expected +outcome? + +This will enable users of large astronomical infrastractures to easily access their data archives and computing clusters from renkulab, making renkulab much more attractive for them + +> Can we distinguish between essential requirements and 'nice to have' requirements + +## Key Assumptions + +> Are there any key assumptions underpinning this work which should be highlighted? +These could be assumptions on user behaviour, assumptions on the deployment +context, assumptions on external services etc + +## Possible Solutions + +> In most cases in which an RFC is required, there is not a single obvious solution +and it makes sense to itemize a few different alternatives/approaches. If that is +the case, these should be included here. + +***It can make sense to organize a meeting at which the Problem Statement, Key +Assumptions and Possible Solutions are discussed to explore the problem space +as a lead in to defining the Proposed Solution.*** + +## Proposed solution + +> This is the bulk of the RFC. + +> Explain the design in enough detail for somebody familiar with the +infrastructure to understand. This should get into specifics and corner-cases, +and include examples of how the service is used. Any new terminology should be +defined here. + +***It can make sense to organize a meeting to discuss the Proposed Solution such +that the proposal is clear, the solution is convincing and the team believes it +can be implemented in a reasonable amount of time.*** + +## Drawbacks + +> Why should we *not* do this? Please consider the impact on users, +on the integration of this change with other existing and planned features etc. + +> There are tradeoffs to choosing any path, please attempt to identify them here. + +## Rationale and Alternatives + +> Why is this design the best in the space of possible designs? + +> What other designs have been considered and what is the rationale for not choosing them? + +> What is the impact of not doing this? + +## Unresolved questions + +> What parts of the design do you expect to resolve through the RFC process +before this gets merged? + +> What parts of the design do you expect to resolve through the implementation +of this feature before stabilisation? + +> What related issues do you consider out of scope for this RFC that could be +addressed in the future independently of the solution that comes out of this RFC? From 7ef1ca875cda41c74a363a1d376b11fddcb94419 Mon Sep 17 00:00:00 2001 From: Volodymyr Savchenko Date: Mon, 15 May 2023 15:28:30 +0200 Subject: [PATCH 2/9] first draft of obvious parts --- .../012-facilitated-storage-compute-access.md | 37 +++++++++++++++++-- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md index ea25109..3cc004c 100644 --- a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md +++ b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md @@ -1,29 +1,48 @@ - Start Date: 2023-05-08 - Status: Proposed -# Facilitated Access to Storage and Compute Resources +# Facilitated Access to Project Storage and Compute Resources ## Summary > One paragraph explanation of the change. -In the context of large astrophysical projects, an important benefit of an interactive data analysis and exporation platform like renkulab is to provide direct access to data and computing resources close to the platform, saving on transfer costs. Since access to these resources is typically restricted, it would be useful to use link the resource access account to the renkulab session and allow users to access the resources without any additional authentication. +In the context of large astrophysical projects, it is often necessary to access astronomical data archives and computing resources (e.g. grid-like clusters). + +One of the key benefits of interactive data analysis and exporation platforms like jupyterhub (and, hopefully, renkulab) is that it is possible to facilitate access to data and computing resources. + +Often these resources are located close to the platform, saving on data transfer costs. + +Since access to these resources is typically restricted, it is possible to rely the identity of the users logged in the interactive session to enable access to the resources without any additional authentication. ## Problem Statement > Why are we doing this? What use cases does it support? What is the expected outcome? -This will enable users of large astronomical infrastractures to easily access their data archives and computing clusters from renkulab, making renkulab much more attractive for them +This will enable users of large astronomical infrastractures to easily access their data archives and computing clusters from renkulab, making renkulab much more attractive for them. + +A typical data-intensive astronomer life cycle is like so: + +* query a large private data archive to discover and select datasets +* fetch small datasets from the private data archive and explore them interactively +* submit processing of the datasets to private computing cluster (could be dask, ARC, reana, slurm, etc) storing the results in private storage +* fetch some results to interactive session, explore and visualize them > Can we distinguish between essential requirements and 'nice to have' requirements +Nice to have feature would be to add features in the session UI to make it easier to start connection to the storage and compute services. For example, a button "fetch data from ESAC archive" or "run in Pulsar Network". Something similar is done in dask jupyterlab plugin. + +It could be also interesting to present the archives as mounted filesystems (it is done like that in ESA DataLabs and also ESCAPE/ESAP). + ## Key Assumptions > Are there any key assumptions underpinning this work which should be highlighted? These could be assumptions on user behaviour, assumptions on the deployment context, assumptions on external services etc +This requires depenency on external compute and storage services. There might be testing to make sure these services remain functional. + ## Possible Solutions > In most cases in which an RFC is required, there is not a single obvious solution @@ -34,6 +53,11 @@ the case, these should be included here. Assumptions and Possible Solutions are discussed to explore the problem space as a lead in to defining the Proposed Solution.*** +* A possible soluton might be similar to "jupyterhub services" - specialized services deployed along with a jupyterhub instance. Jupyterhub provides a framework for sharing user account with these services. +If similar solution is available in renkulab, it would be possible outsource development of specialized domain-specific compute and storage interfaces to domain-specific teams. + +* Another possible solution could be to allow users to access external compute and storage services directly from renku session without any additional domain-specific services near renku, using renku to store authorization for accessing external services associating it with user account. + ## Proposed solution > This is the bulk of the RFC. @@ -52,8 +76,13 @@ can be implemented in a reasonable amount of time.*** > Why should we *not* do this? Please consider the impact on users, on the integration of this change with other existing and planned features etc. +Interfaces to storage and compute backends proposed in this RFC would be specialized, useful to different individual project. In principle this may lead to numerous project-specific and domain-specific interfaces which will be difficult to support on single renkulab instance. + +Second possible solution, just storing credentials, is more favorable in this respect, but still requires storing + > There are tradeoffs to choosing any path, please attempt to identify them here. + ## Rationale and Alternatives > Why is this design the best in the space of possible designs? @@ -62,6 +91,8 @@ on the integration of this change with other existing and planned features etc. > What is the impact of not doing this? +When we need to access resouure + ## Unresolved questions > What parts of the design do you expect to resolve through the RFC process From cb66f0e0a90dc0c0af58bc1df1c12bf82f5d1b0c Mon Sep 17 00:00:00 2001 From: Volodymyr Savchenko Date: Mon, 15 May 2023 16:05:00 +0200 Subject: [PATCH 3/9] add a drawback --- .../012-facilitated-storage-compute-access.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md index 3cc004c..640b808 100644 --- a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md +++ b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md @@ -78,11 +78,12 @@ on the integration of this change with other existing and planned features etc. Interfaces to storage and compute backends proposed in this RFC would be specialized, useful to different individual project. In principle this may lead to numerous project-specific and domain-specific interfaces which will be difficult to support on single renkulab instance. -Second possible solution, just storing credentials, is more favorable in this respect, but still requires storing +Second possible solution, just storing credentials, is more favorable in this respect, but still requires keeping track of some domain-specific external services. > There are tradeoffs to choosing any path, please attempt to identify them here. + ## Rationale and Alternatives > Why is this design the best in the space of possible designs? @@ -91,7 +92,8 @@ Second possible solution, just storing credentials, is more favorable in this re > What is the impact of not doing this? -When we need to access resouure +Right now, when we need to access secured external resoure from renkulab, we need to manually copy the credentials into the session, and make sure they are not stored in the repository which would compromise them. This is cumbersome and potentially unsafe. +When transferring large datasets to renkulab, we need to take care of transfer costs ourselves. It means that even moderately large data is usually not transferred to renkulab. ## Unresolved questions From 4bed5ae1fc56c7ebf189134db4428e5550506533 Mon Sep 17 00:00:00 2001 From: Volodymyr Savchenko Date: Mon, 15 May 2023 16:24:28 +0200 Subject: [PATCH 4/9] some refinement --- .../012-facilitated-storage-compute-access.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md index 640b808..70a93a1 100644 --- a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md +++ b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md @@ -7,26 +7,27 @@ > One paragraph explanation of the change. -In the context of large astrophysical projects, it is often necessary to access astronomical data archives and computing resources (e.g. grid-like clusters). +In the context of large astrophysical projects, it is often necessary to access astronomical data archives (S3, various block storage, WebDAV, etc) and computing (e.g. dask, ARC, reana, slurm, etc) resources and services. Almost all of these interfaces built upon HTTP and rely on OpenID Connect. -One of the key benefits of interactive data analysis and exporation platforms like jupyterhub (and, hopefully, renkulab) is that it is possible to facilitate access to data and computing resources. +One of the recongised benefits of interactive data analysis and exporation platforms like jupyterhub (and, hopefully, renkulab) is that it is possible to facilitate access to these resources. -Often these resources are located close to the platform, saving on data transfer costs. +Interactive platform may allow users to discover these services, with various complementary UI features. Since access to these resources is typically restricted, it is possible to rely the identity of the users logged in the interactive session to enable access to the resources without any additional authentication. +Often these resources can be located close to the platform, saving on data transfer costs, enabling the "bright code to the data" approach. + ## Problem Statement -> Why are we doing this? What use cases does it support? What is the expected -outcome? +> Why are we doing this? What use cases does it support? What is the expected outcome? -This will enable users of large astronomical infrastractures to easily access their data archives and computing clusters from renkulab, making renkulab much more attractive for them. +This will enable users of astronomical infrastructures to easily access their data archives and computing clusters from renkulab, making renkulab much more attractive for them. -A typical data-intensive astronomer life cycle is like so: +A typical data-intensive astronomer life cycle can be outline like so: * query a large private data archive to discover and select datasets * fetch small datasets from the private data archive and explore them interactively -* submit processing of the datasets to private computing cluster (could be dask, ARC, reana, slurm, etc) storing the results in private storage +* submit processing of the datasets to private computing cluster storing the results in private storage * fetch some results to interactive session, explore and visualize them > Can we distinguish between essential requirements and 'nice to have' requirements From 78579c85a7af7b373d8cad7b2d18008933fd1fc1 Mon Sep 17 00:00:00 2001 From: Volodymyr Savchenko Date: Mon, 15 May 2023 16:55:06 +0200 Subject: [PATCH 5/9] further refinemen --- .../012-facilitated-storage-compute-access.md | 23 +++++++++---------- 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md index 70a93a1..9ef9263 100644 --- a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md +++ b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md @@ -5,17 +5,13 @@ ## Summary -> One paragraph explanation of the change. +> "One paragraph" explanation of the change. -In the context of large astrophysical projects, it is often necessary to access astronomical data archives (S3, various block storage, WebDAV, etc) and computing (e.g. dask, ARC, reana, slurm, etc) resources and services. Almost all of these interfaces built upon HTTP and rely on OpenID Connect. - -One of the recongised benefits of interactive data analysis and exporation platforms like jupyterhub (and, hopefully, renkulab) is that it is possible to facilitate access to these resources. - -Interactive platform may allow users to discover these services, with various complementary UI features. - -Since access to these resources is typically restricted, it is possible to rely the identity of the users logged in the interactive session to enable access to the resources without any additional authentication. - -Often these resources can be located close to the platform, saving on data transfer costs, enabling the "bright code to the data" approach. +In modern astrophysical projects it is often necessary to access astronomical data archives (S3, various block storage, WebDAV, etc) and computing (e.g. dask, ARC, slurm, reana, etc) services. Most of the time, the access is through an interface built upon HTTP and relying on OpenID Connect for access control. +One of the recongised benefits of interactive data analysis and exporation platforms like jupyterhub (and, hopefully, renkulab) is that it is possible to facilitate access to these resources in the following ways: +* Interactive platform may allow users to discover these services, with various complementary UI features. +* Since access to these resources is typically restricted, it is possible to rely the identity of the users logged in the interactive session to enable access to the resources without any additional authentication. +* Often these resources can be located close to the platform, saving on data transfer costs, enabling the "bright code to the data" approach. ## Problem Statement @@ -30,6 +26,8 @@ A typical data-intensive astronomer life cycle can be outline like so: * submit processing of the datasets to private computing cluster storing the results in private storage * fetch some results to interactive session, explore and visualize them +In this process, renkulab can act as an interactive platform. The resources can be accessed through an API. Forumulation of the request and substitution of authorization can be facilitated by renkulab. + > Can we distinguish between essential requirements and 'nice to have' requirements Nice to have feature would be to add features in the session UI to make it easier to start connection to the storage and compute services. For example, a button "fetch data from ESAC archive" or "run in Pulsar Network". Something similar is done in dask jupyterlab plugin. @@ -77,9 +75,9 @@ can be implemented in a reasonable amount of time.*** > Why should we *not* do this? Please consider the impact on users, on the integration of this change with other existing and planned features etc. -Interfaces to storage and compute backends proposed in this RFC would be specialized, useful to different individual project. In principle this may lead to numerous project-specific and domain-specific interfaces which will be difficult to support on single renkulab instance. +Interfaces to storage and compute backends proposed in this RFC would be specialized, useful to different individual projects, communities. In principle this may lead to numerous project-specific and domain-specific interfaces which will be difficult to support on single renkulab instance. -Second possible solution, just storing credentials, is more favorable in this respect, but still requires keeping track of some domain-specific external services. +Second possible solution, just storing the credentials, is more favorable in this respect. It still implies, however, that renkulab is keeping track of multiple domain-specific external services and of the way the can be made available to the user session. So a lightweight "renkulab services" contributed by teams may still be benefitial to separate responsibility for these domain interfaces. > There are tradeoffs to choosing any path, please attempt to identify them here. @@ -94,6 +92,7 @@ Second possible solution, just storing credentials, is more favorable in this re > What is the impact of not doing this? Right now, when we need to access secured external resoure from renkulab, we need to manually copy the credentials into the session, and make sure they are not stored in the repository which would compromise them. This is cumbersome and potentially unsafe. + When transferring large datasets to renkulab, we need to take care of transfer costs ourselves. It means that even moderately large data is usually not transferred to renkulab. ## Unresolved questions From 987bd3728d5962a708f060d14cad8725ffbdf997 Mon Sep 17 00:00:00 2001 From: Volodymyr Savchenko Date: Mon, 15 May 2023 19:09:32 +0200 Subject: [PATCH 6/9] example services --- .../012-facilitated-storage-compute-access.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md index 9ef9263..13eb42f 100644 --- a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md +++ b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md @@ -28,6 +28,14 @@ A typical data-intensive astronomer life cycle can be outline like so: In this process, renkulab can act as an interactive platform. The resources can be accessed through an API. Forumulation of the request and substitution of authorization can be facilitated by renkulab. +Some example services include: + +* WebDAV storage with OIDC or OIDC-to-certificate mapping +* [rucio](https://github.com/rucio/rucio) bulk archive +* EGI/[ARC](http://www.nordugrid.org/arc/) cluster +* custom compute MMODA API https://www.astro.unige.ch/mmoda/ + + > Can we distinguish between essential requirements and 'nice to have' requirements Nice to have feature would be to add features in the session UI to make it easier to start connection to the storage and compute services. For example, a button "fetch data from ESAC archive" or "run in Pulsar Network". Something similar is done in dask jupyterlab plugin. From 0de64da61513921e2bbd56bc11564920baf7b49b Mon Sep 17 00:00:00 2001 From: Volodymyr Savchenko Date: Mon, 15 May 2023 19:11:59 +0200 Subject: [PATCH 7/9] example object storage --- .../012-facilitated-storage-compute-access.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md index 13eb42f..641e0cb 100644 --- a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md +++ b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md @@ -30,10 +30,14 @@ In this process, renkulab can act as an interactive platform. The resources can Some example services include: -* WebDAV storage with OIDC or OIDC-to-certificate mapping -* [rucio](https://github.com/rucio/rucio) bulk archive -* EGI/[ARC](http://www.nordugrid.org/arc/) cluster -* custom compute MMODA API https://www.astro.unige.ch/mmoda/ +* storage: + * WebDAV storage with OIDC or OIDC-to-certificate mapping + * [rucio](https://github.com/rucio/rucio) bulk archive + * CSCS [object storage](https://user.cscs.ch/storage/object_storage/) +* compute + * EGI/[ARC](http://www.nordugrid.org/arc/) cluster + * [DIRAC](https://github.com/DIRACGrid/DIRAC) cluster + * custom compute MMODA API https://www.astro.unige.ch/mmoda/ > Can we distinguish between essential requirements and 'nice to have' requirements From 0aac35791ea8042535f6e9fb901d3d69a95e6a74 Mon Sep 17 00:00:00 2001 From: Volodymyr Savchenko Date: Mon, 15 May 2023 20:51:15 +0200 Subject: [PATCH 8/9] another compute example --- .../012-facilitated-storage-compute-access.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md index 641e0cb..3ba50bf 100644 --- a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md +++ b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md @@ -64,10 +64,16 @@ the case, these should be included here. Assumptions and Possible Solutions are discussed to explore the problem space as a lead in to defining the Proposed Solution.*** -* A possible soluton might be similar to "jupyterhub services" - specialized services deployed along with a jupyterhub instance. Jupyterhub provides a framework for sharing user account with these services. +### A + +A possible soluton might be similar to "jupyterhub services" - specialized services deployed along with a jupyterhub instance. Jupyterhub provides a framework for sharing user account with these services. If similar solution is available in renkulab, it would be possible outsource development of specialized domain-specific compute and storage interfaces to domain-specific teams. -* Another possible solution could be to allow users to access external compute and storage services directly from renku session without any additional domain-specific services near renku, using renku to store authorization for accessing external services associating it with user account. +Renkulab would need to set some environment variables before the session start, specifying the service endpoints and possibly credentials. + +### B + + Another possible solution could be to allow users to access external compute and storage services directly from renku session without any additional domain-specific services near renku, using renku to store authorization for accessing external services associating it with user account. Renkulab would then set some environment variables for the session, and they will be used by the user code. ## Proposed solution From bba9f7f38cc9b50869e44578d57f9eee83181dd1 Mon Sep 17 00:00:00 2001 From: Volodymyr Date: Wed, 17 May 2023 15:53:44 +0200 Subject: [PATCH 9/9] Update 012-facilitated-storage-compute-access.md --- .../012-facilitated-storage-compute-access.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md index 3ba50bf..4a7d7b2 100644 --- a/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md +++ b/rfcs/012-facilitated-storage-compute-access/012-facilitated-storage-compute-access.md @@ -31,12 +31,12 @@ In this process, renkulab can act as an interactive platform. The resources can Some example services include: * storage: - * WebDAV storage with OIDC or OIDC-to-certificate mapping + * WebDAV storage with OIDC * [rucio](https://github.com/rucio/rucio) bulk archive * CSCS [object storage](https://user.cscs.ch/storage/object_storage/) * compute - * EGI/[ARC](http://www.nordugrid.org/arc/) cluster - * [DIRAC](https://github.com/DIRACGrid/DIRAC) cluster + * dask gateway + * EGI/[ARC](http://www.nordugrid.org/arc/) or [DIRAC](https://github.com/DIRACGrid/DIRAC) cluster * custom compute MMODA API https://www.astro.unige.ch/mmoda/