diff --git a/.gitignore b/.gitignore index bac09ac4..c85e3d52 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ node_modules/ public/ +build/ .cache/ _import/ docs/superpowers/ diff --git a/AGENTS.md b/AGENTS.md index 82dfef7d..3caff9ea 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -70,6 +70,20 @@ Node 22 is used in CI. - **DCO sign-off**: Every commit needs a `Signed-off-by` line (`git commit -s`). - **Conventional Commits & Squash Merge**: Use the [Conventional Commits](https://www.conventionalcommits.org/) format. This repository squash-merges, so the PR title becomes the commit message on `main` -- apply Conventional Commits format to PR titles as well. A GitHub Actions workflow enforces this. +## Where the Other Rules Live + +Subdirectory `AGENTS.md` files are loaded on demand when working in that subtree: + +* `content/AGENTS.md` -- AsciiDoc authoring rules (tables, images, lists, heading capitalisation), xref fixes, `doc.owncloud.com` link conversion, external link maintenance, xref link-text checks +* `content/ocis/AGENTS.md` -- oCIS developer service pages, Antora Collector and Swagger UI extensions +* `ui/AGENTS.md` -- `llms.txt` maintenance and supplemental UI JavaScript + +## File Relationship: CLAUDE.md and AGENTS.md + +Every `CLAUDE.md` is a symlink to the `AGENTS.md` beside it (root, `content/`, `content/ocis/`, `ui/`). Both names are provided so that different AI tools find their expected filename — Claude Code reads `CLAUDE.md`, OpenAI Codex-based tools read `AGENTS.md`. **Always edit `AGENTS.md` directly**; `CLAUDE.md` reflects all changes automatically via the symlink. When adding a new subdirectory `AGENTS.md`, add the sibling `CLAUDE.md` symlink and list it under "Where the Other Rules Live". + +Keep these files outside the Antora `start_paths` (never inside a `content///` directory, which is added and dropped over time) and outside `ui/supplemental/`, otherwise they would be published. `extension-tests/static-files.test.js` guards this. + ## Context for AI Agents - Match existing code style - Do not refactor unrelated code in the same PR diff --git a/content/AGENTS.md b/content/AGENTS.md new file mode 100644 index 00000000..d419e36e --- /dev/null +++ b/content/AGENTS.md @@ -0,0 +1,207 @@ +# content + +Agent guidelines for authoring documentation under `content/` (AsciiDoc style, xrefs, links). Loaded on demand; repo-wide rules live in the root `AGENTS.md`. + +## AsciiDoc Table Format + +Always write AsciiDoc tables with one cell per line and a blank line between each complete row: + +```asciidoc +[cols="...",options="header"] +|=== +| Header 1 +| Header 2 +| Header 3 + +| row 1, cell 1 +| row 1, cell 2 +| row 1, cell 3 + +| row 2, cell 1 +| row 2, cell 2 +| row 2, cell 3 +|=== +``` + +- Each `|` starts a new line, regardless of cell count or content length. +- A blank line separates one complete row from the next. +- Span rows (e.g. `6+| _Section heading_`) are also followed by a blank line. +- Empty cells are written as a bare `|` on its own line. + +## Fixing Xrefs After Markdown Migration + +Content migrated from Markdown often carries broken cross-reference patterns. When you encounter them, apply the corrections below. + +### Forbidden prefixes and extensions + +- `xref:./` — relative path prefix is not allowed. Use the Antora page path from the `pages/` root (e.g. `ocis/storage/spaces.adoc`). +- `xref:something.md` or `xref:./something.md#...` — `.md` extension is wrong. Rename to `.adoc`. + +### Broken anchor-only xrefs + +A pattern like `xref:#something.adoc[text]` is not a valid anchor reference — it accidentally includes `.adoc` in what should be a fragment. These were originally links to sections in a different page. Resolve them to the correct page and Asciidoctor auto-generated anchor: + +- Asciidoctor auto-generates anchors from section titles: lowercase, spaces become underscores, prefixed with `_`. Example: `== Storage Spaces` → `#_storage_spaces`. +- When the target page has duplicate section titles, Asciidoctor appends `_2`, `_3`, … to later occurrences. + +### Garbled fragment + extension + +A pattern like `xref:./terminology.md#storage-spaces.adoc[text]` combines all three errors above. Correct form: `xref:ocis/storage/terminology.adoc#_storage_spaces[text]`. + +### Quick checklist when reviewing migrated xrefs + +1. No `./` prefix. +2. No `.md` extension. +3. No `#something.adoc` as a fragment. +4. Fragment anchors use Asciidoctor format (`#_section_title`), not Markdown slugs (`#section-title`). +5. Verify the target file exists under `content///modules//pages/`. + +## Converting External doc.owncloud.com Links to Xrefs + +Links of the form `link:https://doc.owncloud.com//...` that point to pages which exist in this repository must be converted to `xref:` cross-references. External `link:` macros bypass Antora's link validation and break when the site structure changes. + +### Mapping a URL to an xref + +Given a URL `https://doc.owncloud.com///.html#`: + +1. **Find the source file**: `find content/ -path "*//pages/.adoc"`. If the file exists, convert. +2. **Determine the component name**: read `content///antora.yml` → `name:` field. +3. **Build the xref**: `xref:::.adoc#[link text]` + - Omit the version segment to let Antora resolve the latest version automatically. + - Omit `#` when the URL has no fragment. +4. **Verify the fragment**: fragments in HTML are governed by `idprefix` and `idseparator` in `global-attributes.yml`. This repo uses `idprefix: ''` and `idseparator: '-'`, so auto-generated section IDs match the HTML fragment directly (e.g. `==== Function Arguments` → `#function-arguments`). Confirm the section exists in the target file before using a fragment. + +### Example + +``` +# Before +link:https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html#function-arguments[text] + +# After +xref:server:developer_manual:core/apis/ocs-share-api.adoc#function-arguments[text] +``` + +### Links That Cannot Be Converted + +Do **not** convert `doc.owncloud.com` URLs that appear inside any of the following contexts — AsciiDoc does not process macros inside verbatim or code regions: + +- **Literal / source blocks** (`----` or `[source,...]` delimited blocks) — the URL is rendered as plain text and is intentional (e.g. a checklist template or a config comment). +- **`.env` / config file code blocks** — comment lines starting with `#` that reference documentation are part of the embedded file content; they cannot contain xref macros. +- **Diagram blocks** (e.g. Mermaid `@startdiagram`) — participant link metadata is processed by the diagram renderer, not by Asciidoctor. +- **Base URLs with no page path** (e.g. `https://doc.owncloud.com/ocis/`) — there is no target page to map to an xref. + +When presenting a table of convertible links, mark these rows as `NOT CONVERTIBLE` and state the reason (e.g. "in source block", "in Mermaid diagram"). + +### Developer Docs Audit (2026-10-15) + +`content/ocis/9.0/modules/developer/` was fully audited. Every `doc.owncloud.com` occurrence is in a non-convertible context (source/literal blocks, Mermaid diagram participant directives, bare base URLs, plain prose). No xref conversions are outstanding. Re-run the grep if new files are added. + +## External Link Maintenance + +When auditing external `link:` URLs for validity: + +1. **Permanent redirects (301)** — update the link to the canonical target URL. The redirect target is authoritative; leaving the old URL means every visitor follows an unnecessary hop and the link may break again if the redirect is removed. + +2. **Temporary redirects (302/307)** — evaluate case by case. A `master`→`main` branch rename on GitHub (302) should be updated; a login-wall redirect (e.g. transifex requiring sign-in) is expected behaviour and should be left as-is. + +3. **Permanent redirects (308)** — treat the same as 301: update to the redirect target. + +4. **403 bot-blocked** — do **not** treat as broken. Many sites (npmjs.com, medium.com, ClamAV docs) return 403 to headless HTTP clients but serve content normally in a browser. Only flag as broken if a manual browser check also fails. + +5. **Private GitHub repositories** (e.g. `https://github.com/owncloud/enterprise/...`) return 404 to unauthenticated requests. These links are valid — the repository is private, not gone. Do not flag or remove them. + +6. **Links inside code blocks, literal blocks, `.env` blocks, or diagram blocks** — cannot be updated to xref macros and must stay as plain URLs. See "Links That Cannot Be Converted" above for the full list of non-convertible contexts. + +7. **Reporting** — present findings as a numbered table with columns: `#`, `File:Line`, `Status`, `Current URL`, `Suggested URL`. Separate broken links (4xx) from redirects. Apply fixes only after explicit user confirmation. + +## AsciiDoc Image Width + +All images — both block (`image::`) and inline (`image:`) — must include a `width=` attribute. The first positional attribute in the brackets is alt text and is left empty, so the width is always preceded by a comma: + +```asciidoc +image::path/to/image.svg[,width=300] +image:path/to/image.svg[,width=300] +``` + +- If no width is present, add `,width=300` as the default. +- If a width is already set, leave its value unchanged. +- Never write `[width=300]` without the leading comma. + +## AsciiDoc List Formatting + +A list must be preceded by a blank line. If the first list item immediately follows a paragraph or any other block with no blank line in between, AsciiDoc will not render it as a list. + +Use `*` instead of `-` for unordered list items. Both render identically but `*` is easier to read in source. + +```asciidoc +// Wrong — no blank line before list, wrong marker +Some text: +- item one +- item two + +// Correct +Some text: + +* item one +* item two +``` + +This applies to all list types: unordered (`*`), ordered (`.`), and description lists. + +## AsciiDoc Section Heading Capitalisation + +All section headings (lines starting with one or more `=` followed by a space) must use **Associated Press (AP) title case**. Use https://headlinecapitalization.com/ (select "AP" style) as the reference tool when writing or reviewing headings. + +Rules in brief: +- Capitalise the first and last word, all "major" words (nouns, verbs, adjectives, adverbs). +- Do **not** capitalise articles (`a`, `an`, `the`), coordinating conjunctions (`and`, `but`, `or`, `nor`, `for`, `so`, `yet`) or short prepositions unless they are the first or last word. +- **Proper names keep their original casing** regardless of position — e.g. `reva`, `oCIS`, `WebDAV`, `CS3` are never up-cased or down-cased to fit title case rules. + +```asciidoc +// Wrong +== a dedicated shares storage provider +== The Gateway should be responsible for Path transformations + +// Correct +== A Dedicated Shares Storage Provider +== The Gateway Should Be Responsible for Path Transformations +``` + +## Xref Link Text vs. AP-Capitalised Section Heading + +When section headings are updated to AP title case, `xref:` link text across the codebase may lag behind. This check is **never run automatically** — only when explicitly requested, scoped to one of: + +- a **source file** (check all headings in that file, find all xrefs pointing to them anywhere in the module/component) +- a **referencing file** (check all xrefs in that file, compare their link text against the current heading in the target page) +- a **full module** (both of the above across every page in the module) + +### Procedure + +1. For each `xref:` in scope, resolve the target page and anchor, then read the actual section heading. +2. Compare the heading (AP-cased) with the xref link text. +3. **Close match** (same words, different casing, minor article/preposition difference) → flag as a likely stale link text that should be updated to match the heading. +4. **Significant difference** (xref text is a descriptive phrase rather than the heading title) → do **not** auto-correct; this is intentional. Flag it separately so a human can decide. +5. **Cross-component or cross-module references** (xref target includes a component prefix, e.g. `xref:server:…`) → highlight these prominently — they are harder to trace and may span a different release cycle. + +### Output format + +Always report as a **numbered table** so the user can reference items by number. Never silently fix — present findings first and wait for explicit confirmation. + +Use two separate tables: one for close matches (fixable) and one for significant differences (human review). Add a `[CROSS-COMPONENT]` note in the Notes column for xrefs that include a component prefix. + +``` +**Close matches — pending confirmation:** + +| # | File | xref link text | Actual heading/title | Notes | +|---|---|---|---|---| +| 1 | `ocis/storage/namespaces.adoc:42` | `webdav` | `WebDAV` | Casing only | +| 2 | `ocis/storage/index.adoc:10` | `OCS share api` | `OCS Share API` | [CROSS-COMPONENT] | + +**Significant differences — human review required:** + +| # | File | xref link text | Actual heading/title | Notes | +|---|---|---|---|---| +| 3 | `ocis/storage/index.adoc:15` | `_references_` | `References` | Descriptive/intentional | +``` + +Only apply corrections after explicit user confirmation, referencing items by number. diff --git a/content/CLAUDE.md b/content/CLAUDE.md new file mode 120000 index 00000000..47dc3e3d --- /dev/null +++ b/content/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/content/ocis/8.1/modules/ROOT/pages/index.adoc b/content/ocis/8.1/modules/ROOT/pages/index.adoc index 804a89da..5b6abcbe 100644 --- a/content/ocis/8.1/modules/ROOT/pages/index.adoc +++ b/content/ocis/8.1/modules/ROOT/pages/index.adoc @@ -9,7 +9,9 @@ include::admin:partial$multi-location/compose-version.adoc[] {description} -This documentation provides xref:admin:index.adoc[admin] and xref:developer:index.adoc[developer] related information. +// This documentation provides xref:admin:index.adoc[admin] and xref:developer:index.adoc[developer] related information. + +This documentation provides xref:admin:index.adoc[admin] related information. == Release Life Cycle diff --git a/content/ocis/8.1/modules/developer/index.adoc b/content/ocis/8.1/modules/developer/index.adoc deleted file mode 100644 index 34d2a081..00000000 --- a/content/ocis/8.1/modules/developer/index.adoc +++ /dev/null @@ -1,45 +0,0 @@ -= Infinite Scale Developer Documentation -:toc: right -:toclevels: 2 -:description: Welcome to the Infinite Scale developer documentation. Infinite Scale is a data platform providing tools to integrate, organize, share and govern data and metadata. - -## Admin Documentation - -Before you start reading, if you are interested in: - -- learning ocis from an admin perspective, -- different deployment scenarios, -- deployment examples, -- detailed settings and more - -we would recommend to continue with the [ownCloud Admin Documentation for Infinite Scale](https://doc.owncloud.com/ocis/latest/). - -## Developer Documentation - -Welcome to our developer documentation. Here you find documentation with focus for *developers*: - -- [oCIS]({{< ref "./ocis" >}}) server -- oCIS Services -- Clients like: - - [ownCloud Web](https://github.com/owncloud/ocis/tree/master/web) - the new web frontend for oCIS and ownCloud - - [ownCloud Android app](https://github.com/owncloud/android) - - [ownCloud iOS app](https://github.com/owncloud/ios-app) - - [ownCloud Desktop Syncing Client](https://github.com/owncloud/client) -- Integrations - -## We Love Open Source - -The oCIS server is Apache v2 licensed. -The lower storage layer of oCIS is defined by the CS3 APIs and implemented in the REVA project. Our goal is to develop the CS3 APIs to an open standard and collaborate on the open source REVA reference implementation for CS3 APIs. - -You can also find all client sources on https://github.com/owncloud/[GitHub]. - -## Join the oCIS Community - -The https://github.com/owncloud/ocis[ocis repository] is a good entry point to the oCIS project. In addition to that, there are also ownCloud projects for apps such as https://github.com/owncloud/ios-app[iOS], https://github.com/owncloud/android[Android], the https://github.com/owncloud/desktop[Desktop] platforms and https://github.com/owncloud/ocis/tree/master/web[ownCloud Web]. - -To chat about development, join our public chat on https://matrix.to/#/#ocis:matrix.org[matrix: ownCloud Infinite Scale]. - -If you want to help and improve ownCloud or oCIS, start coding or open issues on GitHub in the related repository. - -We are very happy to hear your feedback and ideas! diff --git a/content/ocis/8.1/modules/developer/notes_for_the_navigation_file_nav.txt b/content/ocis/8.1/modules/developer/notes_for_the_navigation_file_nav.txt deleted file mode 100644 index f86388b9..00000000 --- a/content/ocis/8.1/modules/developer/notes_for_the_navigation_file_nav.txt +++ /dev/null @@ -1,13 +0,0 @@ -The navigation files have been relocated and are accessed differently now. - -The main nav.adoc file is in ROOT and includes all nav.adoc files from the modules. To do so, the modules nav.adoc files needed to be relocated from: - -{modulesdir}/nav.adoc -to -{modulesdir}/partials/nav.adoc - -This is due to the fact that "Antora does not classify files located at the root of the module or in the modules folder" which means, there is no access path to a nav.adoc file which is located in the root of a module. - -As a result, the antora.yml file only contains the main ROOT/nav.adoc file. - -This is beneficial, as we now can structure products and their content in a better way. diff --git a/content/ocis/8.1/modules/developer/pages/index.adoc b/content/ocis/8.1/modules/developer/pages/index.adoc deleted file mode 100644 index b4e78c97..00000000 --- a/content/ocis/8.1/modules/developer/pages/index.adoc +++ /dev/null @@ -1,51 +0,0 @@ -= Infinite Scale Developer Documentation -:toc: right -:toclevels: 2 -:description: Welcome to the Infinite Scale developer documentation. Infinite Scale is a data platform providing tools to integrate, organize, share and govern data and metadata. This guide provides information on developer related topics. - -include::admin:partial$multi-location/compose-version.adoc[] - -== Introduction - -{description} - -== Target Audience - -This documentation addresses developers only. + -See the xref:admin:index.adoc[admin] documnetation for prerequisites, configurations and deployment examples. - -== Release Life Cycle - -Starting with version 6 of Infinite Scale, releases are now split into `production` and `rolling`. See the https://owncloud.dev/ocis/release_roadmap/[Developer Docs Release Life Cycle] page for more details and impacts. - -You can find more details on changes made in each release published in the https://doc.owncloud.com/ocis_release_notes.html[Infinite Scale Server Release Notes]. - -IMPORTANT: Rolling releases are published in a cycle of 3 weeks. Due to this relative short cycle, you will find the actual development and the current published rolling release documentation *not* separated. They are both part of the `next` segment in the URL. Compared to production releases where the respective release number is used, the actual rolling release number published can be found e.g. in each service description in section "Environment Variables". - -== We Love Open Source - -The oCIS is Apache v2 licensed. - -The lower storage layer of oCIS is defined by the CS3 APIs and implemented in the REVA project. Our goal is to develop the CS3 APIs to an open standard and collaborate on the open source REVA reference implementation for CS3 APIs. - -You can also find all app sources at:: -+ --- -- https://github.com/owncloud/android[ownCloud Android App] -- https://github.com/owncloud/ios-app[ownCloud iOS App] -- https://github.com/owncloud/client[ownCloud Desktop App] --- - -== Join the oCIS Community - -The https://github.com/owncloud/ocis[oCIS repository] is a good entry point to the oCIS project. The https://github.com/owncloud/ocis/tree/master/web[Web frontend] is part of the oCIS monorepo. - -To chat about development, join our public chat on https://matrix.to/#/#ocis:matrix.org[matrix: ownCloud Infinite Scale]. - -If you want to help and improve, start coding or open issues on GitHub in the related repositories. - -We are very happy to hear your feedback and ideas! - -== End-User License Agreement (EULA) - -ownCloud provides an EULA to clarify, among various topics, who can use this software and which conditions apply to the groups of users defined. See the actual {compose_raw_url}{compose_version}/assets/End-User-License-Agreement-for-ownCloud-Infinite-Scale.pdf[EULA] for details. diff --git a/content/ocis/8.1/modules/developer/partials/nav.adoc b/content/ocis/8.1/modules/developer/partials/nav.adoc deleted file mode 100644 index b0c2e8d7..00000000 --- a/content/ocis/8.1/modules/developer/partials/nav.adoc +++ /dev/null @@ -1,4 +0,0 @@ -// note that the module reference post xref is a mandatory element -* Developer Manual - -** xref:developer:index.adoc[Introduction] diff --git a/content/ocis/8.2/modules/ROOT/pages/index.adoc b/content/ocis/8.2/modules/ROOT/pages/index.adoc index 804a89da..5b6abcbe 100644 --- a/content/ocis/8.2/modules/ROOT/pages/index.adoc +++ b/content/ocis/8.2/modules/ROOT/pages/index.adoc @@ -9,7 +9,9 @@ include::admin:partial$multi-location/compose-version.adoc[] {description} -This documentation provides xref:admin:index.adoc[admin] and xref:developer:index.adoc[developer] related information. +// This documentation provides xref:admin:index.adoc[admin] and xref:developer:index.adoc[developer] related information. + +This documentation provides xref:admin:index.adoc[admin] related information. == Release Life Cycle diff --git a/content/ocis/8.2/modules/developer/index.adoc b/content/ocis/8.2/modules/developer/index.adoc deleted file mode 100644 index 34d2a081..00000000 --- a/content/ocis/8.2/modules/developer/index.adoc +++ /dev/null @@ -1,45 +0,0 @@ -= Infinite Scale Developer Documentation -:toc: right -:toclevels: 2 -:description: Welcome to the Infinite Scale developer documentation. Infinite Scale is a data platform providing tools to integrate, organize, share and govern data and metadata. - -## Admin Documentation - -Before you start reading, if you are interested in: - -- learning ocis from an admin perspective, -- different deployment scenarios, -- deployment examples, -- detailed settings and more - -we would recommend to continue with the [ownCloud Admin Documentation for Infinite Scale](https://doc.owncloud.com/ocis/latest/). - -## Developer Documentation - -Welcome to our developer documentation. Here you find documentation with focus for *developers*: - -- [oCIS]({{< ref "./ocis" >}}) server -- oCIS Services -- Clients like: - - [ownCloud Web](https://github.com/owncloud/ocis/tree/master/web) - the new web frontend for oCIS and ownCloud - - [ownCloud Android app](https://github.com/owncloud/android) - - [ownCloud iOS app](https://github.com/owncloud/ios-app) - - [ownCloud Desktop Syncing Client](https://github.com/owncloud/client) -- Integrations - -## We Love Open Source - -The oCIS server is Apache v2 licensed. -The lower storage layer of oCIS is defined by the CS3 APIs and implemented in the REVA project. Our goal is to develop the CS3 APIs to an open standard and collaborate on the open source REVA reference implementation for CS3 APIs. - -You can also find all client sources on https://github.com/owncloud/[GitHub]. - -## Join the oCIS Community - -The https://github.com/owncloud/ocis[ocis repository] is a good entry point to the oCIS project. In addition to that, there are also ownCloud projects for apps such as https://github.com/owncloud/ios-app[iOS], https://github.com/owncloud/android[Android], the https://github.com/owncloud/desktop[Desktop] platforms and https://github.com/owncloud/ocis/tree/master/web[ownCloud Web]. - -To chat about development, join our public chat on https://matrix.to/#/#ocis:matrix.org[matrix: ownCloud Infinite Scale]. - -If you want to help and improve ownCloud or oCIS, start coding or open issues on GitHub in the related repository. - -We are very happy to hear your feedback and ideas! diff --git a/content/ocis/8.2/modules/developer/notes_for_the_navigation_file_nav.txt b/content/ocis/8.2/modules/developer/notes_for_the_navigation_file_nav.txt deleted file mode 100644 index f86388b9..00000000 --- a/content/ocis/8.2/modules/developer/notes_for_the_navigation_file_nav.txt +++ /dev/null @@ -1,13 +0,0 @@ -The navigation files have been relocated and are accessed differently now. - -The main nav.adoc file is in ROOT and includes all nav.adoc files from the modules. To do so, the modules nav.adoc files needed to be relocated from: - -{modulesdir}/nav.adoc -to -{modulesdir}/partials/nav.adoc - -This is due to the fact that "Antora does not classify files located at the root of the module or in the modules folder" which means, there is no access path to a nav.adoc file which is located in the root of a module. - -As a result, the antora.yml file only contains the main ROOT/nav.adoc file. - -This is beneficial, as we now can structure products and their content in a better way. diff --git a/content/ocis/8.2/modules/developer/pages/index.adoc b/content/ocis/8.2/modules/developer/pages/index.adoc deleted file mode 100644 index b4e78c97..00000000 --- a/content/ocis/8.2/modules/developer/pages/index.adoc +++ /dev/null @@ -1,51 +0,0 @@ -= Infinite Scale Developer Documentation -:toc: right -:toclevels: 2 -:description: Welcome to the Infinite Scale developer documentation. Infinite Scale is a data platform providing tools to integrate, organize, share and govern data and metadata. This guide provides information on developer related topics. - -include::admin:partial$multi-location/compose-version.adoc[] - -== Introduction - -{description} - -== Target Audience - -This documentation addresses developers only. + -See the xref:admin:index.adoc[admin] documnetation for prerequisites, configurations and deployment examples. - -== Release Life Cycle - -Starting with version 6 of Infinite Scale, releases are now split into `production` and `rolling`. See the https://owncloud.dev/ocis/release_roadmap/[Developer Docs Release Life Cycle] page for more details and impacts. - -You can find more details on changes made in each release published in the https://doc.owncloud.com/ocis_release_notes.html[Infinite Scale Server Release Notes]. - -IMPORTANT: Rolling releases are published in a cycle of 3 weeks. Due to this relative short cycle, you will find the actual development and the current published rolling release documentation *not* separated. They are both part of the `next` segment in the URL. Compared to production releases where the respective release number is used, the actual rolling release number published can be found e.g. in each service description in section "Environment Variables". - -== We Love Open Source - -The oCIS is Apache v2 licensed. - -The lower storage layer of oCIS is defined by the CS3 APIs and implemented in the REVA project. Our goal is to develop the CS3 APIs to an open standard and collaborate on the open source REVA reference implementation for CS3 APIs. - -You can also find all app sources at:: -+ --- -- https://github.com/owncloud/android[ownCloud Android App] -- https://github.com/owncloud/ios-app[ownCloud iOS App] -- https://github.com/owncloud/client[ownCloud Desktop App] --- - -== Join the oCIS Community - -The https://github.com/owncloud/ocis[oCIS repository] is a good entry point to the oCIS project. The https://github.com/owncloud/ocis/tree/master/web[Web frontend] is part of the oCIS monorepo. - -To chat about development, join our public chat on https://matrix.to/#/#ocis:matrix.org[matrix: ownCloud Infinite Scale]. - -If you want to help and improve, start coding or open issues on GitHub in the related repositories. - -We are very happy to hear your feedback and ideas! - -== End-User License Agreement (EULA) - -ownCloud provides an EULA to clarify, among various topics, who can use this software and which conditions apply to the groups of users defined. See the actual {compose_raw_url}{compose_version}/assets/End-User-License-Agreement-for-ownCloud-Infinite-Scale.pdf[EULA] for details. diff --git a/content/ocis/8.2/modules/developer/partials/nav.adoc b/content/ocis/8.2/modules/developer/partials/nav.adoc deleted file mode 100644 index b0c2e8d7..00000000 --- a/content/ocis/8.2/modules/developer/partials/nav.adoc +++ /dev/null @@ -1,4 +0,0 @@ -// note that the module reference post xref is a mandatory element -* Developer Manual - -** xref:developer:index.adoc[Introduction] diff --git a/content/ocis/9.0/antora.yml b/content/ocis/9.0/antora.yml index b7ddcdcd..faf3f9a7 100644 --- a/content/ocis/9.0/antora.yml +++ b/content/ocis/9.0/antora.yml @@ -14,6 +14,13 @@ asciidoc: # https://github.com/owncloud/ocis-charts/tree/master/charts/ocis/docs s-path: 'deployment/services/s-list' + # same as s-path but for the developer documentation + dev-s-path: 'services/s-list' + + # only used in the developer docs via partials/services/compute-links.adoc + # path component to the service folder in the ocis repo, used to link the services README + dev_ocis_services_path: '/services/' + # the name of the sub directory that defines the sources folder used in the compose deployment examples ocis_wopi: 'ocis_full' @@ -47,5 +54,75 @@ asciidoc: download-gh-directory-url: 'https://download-directory.github.io' # the url to download ocis - # note that this is inactive but required in the migration section for upgrading old binaries which do not longer exist. + # note that this is inactive but required in the migration section for upgrading old binaries which do not longer exist. ocis-downloadpage-url: 'https://download.owncloud.com/ocis/ocis' + +# The collector fetches the Libre Graph API OpenAPI spec from the source repository and places it +# in the developer module's attachments family so the swagger-ui extension can render it on the +# libre-graph-api page. See https://github.com/owncloud/libre-graph-api for the source repository. +# +# A git ls-remote check avoids downloading when the remote SHA matches the cached SHA. +# The download is only accepted if the file is non-empty. The build only fails if the cache is +# empty AND the fetch fails. +ext: + collector: + - run: + command: | + mkdir -p .cache/libre-graph build/libre-graph + CACHE=".cache/libre-graph/v1.0.yaml" + SHA_FILE=".cache/libre-graph/main.sha" + REMOTE_SHA=$(git ls-remote https://github.com/owncloud/libre-graph-api refs/heads/main 2>/dev/null | awk '{print $1}') + CACHED_SHA=$(cat "$SHA_FILE" 2>/dev/null || echo "") + if [ -z "$REMOTE_SHA" ] || [ "$REMOTE_SHA" != "$CACHED_SHA" ] || [ ! -f "$CACHE" ]; then + if curl -sLf \ + https://raw.githubusercontent.com/owncloud/libre-graph-api/main/api/openapi-spec/v1.0.yaml \ + -o "${CACHE}.new" 2>/dev/null && \ + [ -s "${CACHE}.new" ]; then + mv "${CACHE}.new" "$CACHE" + [ -n "$REMOTE_SHA" ] && echo "$REMOTE_SHA" > "$SHA_FILE" + fi + fi + cp "$CACHE" build/libre-graph/v1.0.yaml + shell: true + scan: + dir: build/libre-graph + into: modules/developer/attachments + + # The collector fetches the phpDocumentor-generated HTML reference from the ocis-php-sdk source + # repository and places it in the developer module's attachments family so it can be embedded + # via iframe on the clients/ocis-php-sdk/ocis-php-sdk-docs page. + # See https://github.com/owncloud/ocis-php-sdk for the source repository. + # + # The whole-repo tarball is downloaded and only the docs/ subtree is extracted. + # The tarball is cached in .cache/ocis-php-sdk/: if the network is unavailable, the last + # successful download is reused silently. The build only fails if the cache is empty AND + # the fetch fails. + - run: + command: | + rm -rf build/ocis-php-sdk/ocis-php-sdk-docs + mkdir -p .cache/ocis-php-sdk build/ocis-php-sdk/ocis-php-sdk-docs + CACHE=".cache/ocis-php-sdk/main.tar.gz" + SHA_FILE=".cache/ocis-php-sdk/main.sha" + REMOTE_SHA=$(git ls-remote https://github.com/owncloud/ocis-php-sdk refs/heads/main 2>/dev/null | awk '{print $1}') + CACHED_SHA=$(cat "$SHA_FILE" 2>/dev/null || echo "") + if ([ -n "$REMOTE_SHA" ] && [ "$REMOTE_SHA" != "$CACHED_SHA" ]) || [ ! -f "$CACHE" ]; then + [ -n "$REMOTE_SHA" ] \ + && URL="https://github.com/owncloud/ocis-php-sdk/archive/${REMOTE_SHA}.tar.gz" \ + || URL="https://github.com/owncloud/ocis-php-sdk/archive/refs/heads/main.tar.gz" + if curl -sLf "$URL" -o "${CACHE}.new" 2>/dev/null; then + TOP=$(tar -tzf "${CACHE}.new" 2>/dev/null | head -1 | cut -d/ -f1) + if tar -tzf "${CACHE}.new" 2>/dev/null | grep -q "^${TOP}/docs/"; then + mv "${CACHE}.new" "$CACHE" + [ -n "$REMOTE_SHA" ] && echo "$REMOTE_SHA" > "$SHA_FILE" + fi + fi + fi + TOP=$(tar -tzf "$CACHE" 2>/dev/null | head -1 | cut -d/ -f1) + tar -xzf "$CACHE" \ + --strip-components=2 \ + -C build/ocis-php-sdk/ocis-php-sdk-docs \ + "${TOP}/docs" + shell: true + scan: + dir: build/ocis-php-sdk + into: modules/developer/attachments diff --git a/content/ocis/9.0/modules/ROOT/partials/nav.adoc b/content/ocis/9.0/modules/ROOT/partials/nav.adoc index 8f0a8fdb..10740084 100644 --- a/content/ocis/9.0/modules/ROOT/partials/nav.adoc +++ b/content/ocis/9.0/modules/ROOT/partials/nav.adoc @@ -1,4 +1,4 @@ * xref:index.adoc[Introduction] include::admin:partial$nav.adoc[] -// include::developer:partial$nav.adoc[] +include::developer:partial$nav.adoc[] diff --git a/content/ocis/9.0/modules/admin/pages/deployment/services/s-list/proxy.adoc b/content/ocis/9.0/modules/admin/pages/deployment/services/s-list/proxy.adoc index cb3e4cf1..e95d3a9c 100644 --- a/content/ocis/9.0/modules/admin/pages/deployment/services/s-list/proxy.adoc +++ b/content/ocis/9.0/modules/admin/pages/deployment/services/s-list/proxy.adoc @@ -12,7 +12,7 @@ include::partial$multi-location/compose-version.adoc[] {description} Every HTTP request goes through this service. Authentication, logging and other preprocessing of requests also happens here. Mechanisms like request rate limiting or intrusion prevention are **not** included in the proxy service and must be set up in front of an external reverse proxy. -See the https://owncloud.dev/services/proxy/[Developer Documentation] for details if you want to write your own extensions and need to change or add routes to endpoints. +See the xref:developer:{dev-s-path}/proxy/proxy.adoc[Developer Documentation] for details if you want to write your own extensions and need to change or add routes to endpoints. == Default Values diff --git a/content/ocis/9.0/modules/admin/pages/deployment/services/s-list/thumbnails.adoc b/content/ocis/9.0/modules/admin/pages/deployment/services/s-list/thumbnails.adoc index 7df0597a..264e7ed6 100644 --- a/content/ocis/9.0/modules/admin/pages/deployment/services/s-list/thumbnails.adoc +++ b/content/ocis/9.0/modules/admin/pages/deployment/services/s-list/thumbnails.adoc @@ -9,7 +9,7 @@ {description} and is also responsible for presenting images, not only thumbnails to the client. It retrieves the sources at the location where the user files are stored and saves the thumbnails where system files are stored. Those locations have defaults but can be manually defined via environment variables. -Note for developers, more information is available with regards to querying thumbnails in the services section of the https://owncloud.dev/services/thumbnails/[Developer Documentation]. +Note for developers, more information is available with regards to querying thumbnails in the services section of the xref:developer:{dev-s-path}/thumbnails/thumbnails.adoc[Developer Documentation]. == Default Values diff --git a/content/ocis/9.0/modules/developer/attachments/.gitkeep b/content/ocis/9.0/modules/developer/attachments/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/content/ocis/9.0/modules/developer/examples/.gitkeep b/content/ocis/9.0/modules/developer/examples/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/content/ocis/9.0/modules/developer/images/ocis/adr/116872534-54fc2200-ac16-11eb-8267-ffe7b03177b3.png b/content/ocis/9.0/modules/developer/images/ocis/adr/116872534-54fc2200-ac16-11eb-8267-ffe7b03177b3.png new file mode 100644 index 00000000..de0b9393 Binary files /dev/null and b/content/ocis/9.0/modules/developer/images/ocis/adr/116872534-54fc2200-ac16-11eb-8267-ffe7b03177b3.png differ diff --git a/content/ocis/9.0/modules/developer/images/ocis/adr/116872568-62b1a780-ac16-11eb-9f29-030a651ee39b.png b/content/ocis/9.0/modules/developer/images/ocis/adr/116872568-62b1a780-ac16-11eb-9f29-030a651ee39b.png new file mode 100644 index 00000000..d876277f Binary files /dev/null and b/content/ocis/9.0/modules/developer/images/ocis/adr/116872568-62b1a780-ac16-11eb-9f29-030a651ee39b.png differ diff --git a/content/ocis/9.0/modules/developer/images/ocis/adr/runtime.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/adr/runtime.drawio.svg new file mode 100644 index 00000000..754593d7 --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/adr/runtime.drawio.svg @@ -0,0 +1,4 @@ + + + +
Runtime
Runtime
ocs
ocs
graph
graph
idm
idm
storage
storage
proxy
proxy
idp
idp
thumbnails
thumbnails
webdav
webdav
suture
suture
$ ocis server
$ ocis kill proxy
$ ocis start proxy
$ ocis list
$ ocis server...
the runtime component starts services
and suture then takes over, keeping
track of every started service, restarting
it when needed if an error occurred.
the runtime component starts services...
each service is implemented as a sub-command. The binary essentially calls itself with the right arguments and the side effect is that a
service is started. The service will then be monitored by Suture.
each service is implemented as a sub-command. The binary essentially calls itself with the right arguments and the side e...
the runtime is a service in itself. Starting an
oCIS instance will start a runtime service.
the runtime is a service in itself. Star...Text is not SVG - cannot display \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/adr/sa1pANQ.png b/content/ocis/9.0/modules/developer/images/ocis/adr/sa1pANQ.png new file mode 100644 index 00000000..5e0772df Binary files /dev/null and b/content/ocis/9.0/modules/developer/images/ocis/adr/sa1pANQ.png differ diff --git a/content/ocis/9.0/modules/developer/images/ocis/apis/cs3org.png b/content/ocis/9.0/modules/developer/images/ocis/apis/cs3org.png new file mode 100644 index 00000000..6f06e216 Binary files /dev/null and b/content/ocis/9.0/modules/developer/images/ocis/apis/cs3org.png differ diff --git a/content/ocis/9.0/modules/developer/images/ocis/apis/grpc-logo.png b/content/ocis/9.0/modules/developer/images/ocis/apis/grpc-logo.png new file mode 100644 index 00000000..11ad9b1e Binary files /dev/null and b/content/ocis/9.0/modules/developer/images/ocis/apis/grpc-logo.png differ diff --git a/content/ocis/9.0/modules/developer/images/ocis/apis/http-logo.png b/content/ocis/9.0/modules/developer/images/ocis/apis/http-logo.png new file mode 100644 index 00000000..b37f7467 Binary files /dev/null and b/content/ocis/9.0/modules/developer/images/ocis/apis/http-logo.png differ diff --git a/content/ocis/9.0/modules/developer/images/ocis/apis/ocis-apis.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/apis/ocis-apis.drawio.svg new file mode 100644 index 00000000..85721884 --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/apis/ocis-apis.drawio.svg @@ -0,0 +1,286 @@ + + + + + + + +
+
+
+ Client 1 +
+
+
+ + + Client 1 + + + + + + + +
+
+
+ HTTP Gateway +
+ (proxy service) +
+
+
+ + + HTTP Gateway... + + + + + + + +
+
+
+ Client 2 +
+
+
+ + + Client 2 + + + + + + + +
+
+
+ Client 3 +
+
+
+ + + Client 3 + + + + + + + + + +
+
+
+ HTTP +
+
+
+ + + HTTP + + + + + + + + + +
+
+
+ HTTP +
+
+
+ + + HTTP + + + + + + + + + +
+
+
+ HTTP +
+
+
+ + + HTTP + + + + + + + +
+
+
+ ocis service +
+
+
+ + + ocis service + + + + + + + +
+
+
+ ocis service +
+
+
+ + + ocis service + + + + + + + +
+
+
+ ocis service +
+
+
+ + + ocis service + + + + + + + + + +
+
+
+ HTTP +
+
+
+ + + HTTP + + + + + + + +
+
+
+ GRPC Gateway +
+ (gateway service) +
+
+
+ + + GRPC Gateway... + + + + + + + + + +
+
+
+ gRPC +
+
+
+ + + gRPC + + + + + + + + + +
+
+
+ gRPC +
+
+
+ + + gRPC + + + + + + + + + +
+
+
+ HTTP +
+
+
+ + + HTTP + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/architecture-overview.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/architecture-overview.drawio.svg new file mode 100644 index 00000000..19102f81 --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/architecture-overview.drawio.svg @@ -0,0 +1,4 @@ + + + +
Storage
Storage
identity management
identity management
reva
reva
proxy
reverse proxy
+ api gateway
proxy...
users
users
idp
konnectd / keycloak
idp...
ldap
AD, openldap, idm
ldap...
web
(aka phoenix)
web...
graph
graph
graph-explorer
graph-explorer
hello
hello
onlyoffice
onlyoffice
settings
settings
store
store
idm
idm
OCIS
OCIS
ocs
ocs
webdav
webdav
registry
registry
authbasic
authbasic
authbearer
authbearer
authprovider
(basic)
authprovider...
authprovider
(oidc)
authprovider...
userprovider
userprovider
gateway
gateway
gateway
gateway
authregistry
authregistry
storageregistry
storageregistry
frontend
frontend
ocs
ocs
ocdav
ocdav
ocm
ocm
datagateway
datagateway
storagehome
storagehome
storageprovider
storageprovider
dataprovider
dataprovider
storageusers
storageusers
storageprovider
storageprovider
dataprovider
dataprovider
storagemeta
storagemeta
storageprovider
storageprovider
dataprovider
dataprovider
storagepubliclink
storagepubliclink
publicstorageprovider
publicstorageprovider
authprovider
(publicshares)
authprovider...
sharing
sharing
usershareprovider
usershareprovider
publicshareprovider
publicshareprovider

single host

- all services run on a single host
- no redundancy
- minimize resource usage
- home setup
single host...Text is not SVG - cannot display \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/architecture/ocis-services-communication.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/architecture/ocis-services-communication.drawio.svg new file mode 100644 index 00000000..06d30268 --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/architecture/ocis-services-communication.drawio.svg @@ -0,0 +1,4 @@ + + + +
public share auth provider
public share auth pr...
machine auth provider
machine auth provider
proxy
proxy
web
web
grpc
grpc
grpc
grpc
ocs
ocs
grpc
grpc
ocdav
ocdav
grpc
grpc
ocs
ocs
grpc
grpc
settings
settings
LDAP
LDAP
idp
idp
nats
nats
grpc
grpc
LDAP
LDAP
graph
graph
http
http
graph-explorer
graph-explorer
grpc
grpc
webdav
webdav
grpc
grpc
thumbnails
thumbnails
gateway
gateway
grpc
grpc
storage registry
storage registry
grpc
grpc
storage provider
storage provider
share storage provider
share storage provid...
public storage provider
public storage provi...
user provider
user provider
group provider
group provider
http
http
http
http
http
http
http
http
http
http
http
http
http
http
http

http
http
http
grpc
grpc
grpc
grpc
grpc
grpc
grpc
grpc
grpc
grpc
grpc
grpc
Storage
Storage
Reva
Reva
oCIS
oCIS
http
http
libregraph/idm
libregraph/idm
oidc authprovider
oidc authprovider
LDAP
LDAP
basic auth provider
basic auth provider
grpc
grpc
grpc
grpcText is not SVG - cannot display \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/deployment/monitoring_tracing_overview.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/deployment/monitoring_tracing_overview.drawio.svg new file mode 100644 index 00000000..79ea6375 --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/deployment/monitoring_tracing_overview.drawio.svg @@ -0,0 +1,280 @@ + + + + + + + + + + + + +
+
+
+ Prometheus +
+
+
+ + + Prometheus + + + + + + + + + + + +
+
+
+ Jaeger Query +
+
+
+ + + Jaeger Query + + + + + + + + + +
+
+
+ Jaeger Collector +
+
+
+ + + Jaeger Collector + + + + + + + +
+
+
+ ElasticSearch +
+
+
+ + + ElasticSearch + + + + + + + + + + + +
+
+
+ Grafana +
+
+
+ + + Grafana + + + + + + + + + +
+
+
+ oCIS +
+
+
+ + + oCIS + + + + + + + +
+
+
+ Telegraf +
+
+
+ + + Telegraf + + + + + + + +
+
+
+ Prometheus output +
+
+
+ + + Prometheus output + + + + + + + + + + + +
+
+
+ Prometheus input +
+
+
+ + + Prometheus input + + + + + + + +
+
+
+ Host & Docker metrics +
+
+
+ + + Host & Docker metrics + + + + + + + + + +
+
+
+ Jaeger Agent +
+
+
+ + + Jaeger Agent + + + + + + + + + + + +
+
+
+ server for monitoring & tracing infrastructure +
+
+
+ + + server for monitoring & tracing infrastru... + + + + + + + +
+
+
+ server(s) for oCIS with monitoring & tracing clients +
+
+
+ + + server(s) for oCIS with monitoring & tracing c... + + + + + + + +
+
+
+ Visualization and querying of +
+ metrics and traces +
+
+
+ + + Visualization and querying o... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/deployment/ocis_and_wopi_drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/deployment/ocis_and_wopi_drawio.svg new file mode 100644 index 00000000..2ceaeddf --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/deployment/ocis_and_wopi_drawio.svg @@ -0,0 +1,4 @@ + + + +
cs3api
cs3api
app registry
app registry
/app/list
/app/open
/app/list...
cs3api
/listapps
/openapp
cs3api...
frontend
frontend
cs3api
/registerapp
cs3api...
GET /hosting/discovery 
GET /hosting/discovery 
/iop/open...
/iop/open...
app provider
(per web office)
app provider...
graph
webdav
ocs
graph...
/space1/foo.docx
/space1/foo.docx
User
User
/space2/bar.docx
/space2/bar.docx
fileid c
fileid c
Web Office
Web Office
Web Office
Web Office
Web Office
Web Office
cs3org/wopiserver
cs3org/wopiserver
cs3org/wopiserver
cs3org/wopiserver
wopi
wopi
fileid a
fileid a
fileid b
fileid b
User
User
Collabora:
for collaborative edit sessions
a sticky route by fileid
(from wopiSrc) is needed

OnlyOffice:
automatic handling
Collabora:...
inter services communication
inter services communication
all outside communication via the proxy service (api gateway)
all outside communication via the proxy service (api gat...
1
1
2
2
8
8
9
9
6
6
4
4
3
3
7
7
19
19
20
20
13
13
11
11
10
10
12
12
17
17
15
15
18
18
14
14
16
16
21
21
5
5
office application
office application
https://office-application.owncloud.test
https://office-application.owncloud....
https://ocis.owncloud.test
https://ocis.owncloud.test
wopi server
wopi server
ocis
ocis
reverse proxy
reverse proxy
User
User
wopi -> general.wopiurl
address an office application can reach the wopi server
wopi -> general.wopiurl...
reverse proxy
reverse proxy
https://wopiserver.owncloud.test
https://wopiserver.owncloud.test
APP_PROVIDER_DRIVER: wopi
  # for display purposes only
  APP_PROVIDER_WOPI_APP_NAME: Office-Application
  # address of the office application server
  APP_PROVIDER_WOPI_APP_URL: https://office-application.owncloud.test
  # where can the app provider reach the wopiserver
  APP_PROVIDER_WOPI_WOPI_SERVER_EXTERNAL_URL: https://wopiserver.owncloud.test
  #  the base URL to access ocis (global OCIS_URL, or overwritten by the below)
  APP_PROVIDER_WOPI_FOLDER_URL_BASE_URL: https://ocis.owncloud.test
APP_PROVIDER_DRIVER: wopi...
wopi -> ocis_wopi/config/wopiserver/wopiserver.conf.dist
wopi -> ocis_wopi/config/wopiserver/wopiserver.conf.dist
a. Open oC web
b. List space
c. Select file
d. Open selected file via office app
e. oC Web opens office url with token in an iframe
a. Open oC web...
a. load:    /app/list => display available office apps
a+b+c                   => obtain file id
e. call:     /app/open
    return: office url to open:
a. load:    /app/list => display available of...
https://office-application.owncloud.test/xyz?
wopiSrc=https://wopiserver.owncloud.test/wopi/xxx/file-id-123

Token management is for security reasons not part of the URL
https://office-application.owncloud.test/xyz?wopiSrc=https:...
reverse proxy
reverse proxy
wopi -> cs3.revagateway
(cs3 api gateway)
wopi -> cs3.revagateway...Text is not SVG - cannot display \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/development/vo0EbcX.png b/content/ocis/9.0/modules/developer/images/ocis/development/vo0EbcX.png new file mode 100644 index 00000000..6d4ba0bc Binary files /dev/null and b/content/ocis/9.0/modules/developer/images/ocis/development/vo0EbcX.png differ diff --git a/content/ocis/9.0/modules/developer/images/ocis/flow-docs/tus-public-upload.svg b/content/ocis/9.0/modules/developer/images/ocis/flow-docs/tus-public-upload.svg new file mode 100644 index 00000000..ad2a5c5f --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/flow-docs/tus-public-upload.svg @@ -0,0 +1,3 @@ + + +

ocis-reva sharing

REVA_SHARING_ADDR = 0.0.0.0:9150

ocis-reva sharing...

ocis-reva frontend

REVA_FRONTEND_ADDR = 0.0.0.0:9140

REVA_GATEWAY_URL = ocis:9142

ocis-reva frontend...

ocis-proxy

PROXY_HTTP_ADDR = 0.0.0.0:9200

ocis-proxy...
 2  POST http://ocis:9140/remote.php/dav/files/einstein/
2  POST http:/...

ocdav

prefix = ""

timeout = 86400

ocdav...

datagateway

prefix = "data"

timeout = 86400

datagateway...

client


client
 22  PATCH https://oc.example.org/data/{token}
Tus-Resumable: 1.0.0
22  PATCH http...

ocis-reva gateway

REVA_GATEWAY_ADDR = 0.0.0.0:9142

ocis-reva gateway...

storage-registry


storage-registry
Expose: true
Expose: true
 24  PATCH http://ocis:9156/data/u-u-i-d
24  PATCH http...
 4  GetStorageProvider
(ShareReference)
4  GetStorageP...
 5  ProviderInfo
 5  ProviderInfo

storageprovider

REVA_STORAGE_HOME_ADDR = 0.0.0.0:9154

REVA_STORAGE_HOME_DRIVER = eoshome

REVA_STORAGE_HOME_EXPOSE_DATA_SERVER = false

REVA_STORAGE_HOME_DATA_SERVER_URL =

    http://ocis:9156/data

storageprovider...
Expose: false
Expose: false
 6  InitiateFileUpload
(ShareReference)
6  InitiateFil...
EOS
EOS
 15  WriteFile(upload info)
15  WriteFile(...
 7  GetPublicShare
7  GetPublicSh...
 19  UploadEndpoint
https://oc.example.org/data/{token}
19  UploadEndp...
 20  201 Created
Location: https://oc.example.org/data/{token}
20  201 Create...
 21  201 Created
Location: https://oc.example.org/data/{token}
21  201 Create...
 1  POST https://oc.example.org/remote.php/dav/files/einstein/
Upload-Length: 100
Tus-Resumable: 1.0.0
Upload-Metadata: filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg==,
dir d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg==
1  POST https:...
 23  PATCH http://ocis:9140/data/{token}
Tus-Resumable: 1.0.0
23  PATCH http...
 3  InitiateFileUpload
3  InitiateFil...
 25  Write(bytes)
25  Write(byte...
 26  204 No Content
26  204 No Con...
 27  204 No Content
27  204 No Con...
 28  204 No Content
28  204 No Con...

publicstorageprovider

expose-data-server = true

publicstorageprovider...

publicshareprovider


publicshareprovider
 8  GetPublicShare
8  GetPublicSh...
 9  PublicShare
 9  PublicShare
 10  PublicShare
 10  PublicShare
 11  InitiateFileUpload(TargetReference)
11  InitiateFi...
 12  GetStorageProvider
(TargetReference)
12  GetStorage...
 13  ProviderInfo
13  ProviderIn...
 14  InitiateFileUpload(TargetReference)
14  InitiateFi...
 16  UploadEndpoint
http://ocis:9156/data/u-u-i-d
Expose: false
16  UploadEndp...
 17  UploadEndpoint
https://oc.example.org/data/
token: sign(http://ocis:9156/data/u-u-i-d)
17  UploadEndp...
 18  UploadEndpoint
https://oc.example.org/data/{token}
Expose: true
18  UploadEndp...

gateway

REVA_TRANSFER_EXPIRES = 86400

REVA_FRONTEND_URL =

    https://oc.example.org

REVA_DATAGATEWAY_URL =

    https://oc.example.org/data

    

gateway...
When a storage provider
sets the Expose flag of an Upload/Download Endpoint to false the gateway will wrap the url in a JWT and return the URL of the datagateway along with a transfer-token.
When a storage provider...

dataprovider

REVA_STORAGE_HOME_DATA_ADDR = 0.0.0.0:9156

REVA_STORAGE_HOME_DATA_DRIVER = eoshome

dataprovider...
GOAL: transfer bytes from the client up here ...
GOAL: tran...
... to the storage system somewhere down here
... to the storage syst...Viewer does not support full SVG 1.1 \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/idea.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/idea.drawio.svg new file mode 100644 index 00000000..068b7318 --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/idea.drawio.svg @@ -0,0 +1,489 @@ + + + + + + + + +
+
+
+ storage provider +
+
+
+ + + storage provider + + + + + + + +
+
+
+ storage provider +
+
+
+ + + storage provider + + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ a sea of storage spaces +
+
+
+ + + a sea of storage spaces + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ einstein +
+
+
+ + + einst... + + + + + + + + +
+
+
+ marie +
+
+
+ + + marie + + + + + + + + +
+
+
+ richard +
+
+
+ + + richa... + + + + + + + + + +
+
+
+ storage space +
+ registry at +
+ + https://cloud.zurich.test + +
+
+
+ + + storage sp... + + + + + + + +
+
+
+ storage space +
+ registry at +
+ + https://cloud.paris.test + +
+
+
+ + + storage sp... + + + + + + + +
+
+
+ storage space +
+ registry at +
+ + https://cloud.princeton.test + +
+
+
+ + + storage sp... + + + + + + + +
+
+
+ oCIS = federated storage +
+
+
+ + + oCIS = federated storage + + + + + + + +
+
+
+ multiple +
+ storage registries +
+
+
+ + + multiple... + + + + + + + +
+
+
+ end user devices +
+
+
+ + + end user devices + + + + + + + +
+
+
+ storage provider +
+
+
+ + + storage provider + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/infinite_scale.png b/content/ocis/9.0/modules/developer/images/ocis/infinite_scale.png new file mode 100644 index 00000000..cde9bb4b Binary files /dev/null and b/content/ocis/9.0/modules/developer/images/ocis/infinite_scale.png differ diff --git a/content/ocis/9.0/modules/developer/images/ocis/ocis-config-redesign.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/ocis-config-redesign.drawio.svg new file mode 100644 index 00000000..e7615dd0 --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/ocis-config-redesign.drawio.svg @@ -0,0 +1,4 @@ + + + +
> ocis server
> ocis server
# global logging config
log:
  pretty: true
  color: true
  level: info
proxy:
  http:
    addr: localhost:9200
  log:
    pretty: false
    color: false
    level: info
accounts:
  log:
    level: debug
    color: false
    pretty: false

# global logging config...
ocis.yaml
ocis.yaml
  http:
    addr: localhost:1234
  log:
    pretty: false
    color: false
    level: info

http:...
proxy.yaml
proxy.yaml
  http:
    addr: localhost:5678


http:...
accounts.yaml
accounts.yaml
resulting proxy config values
resulting proxy config values
resulting accounts config values
resulting accounts config values
  http:
    addr: localhost:1234
  log:
    pretty: false
    color: false
    level: info


http:...
notice how even if ocis supplies
with logging configuration, the values
from the extension overwrite the inherited
values. This can also be achieved using an
environment variable.
notice how even if ocis supplies...
  http:
    addr: localhost:5678
  log:
    pretty: false
    color: false
    level: debug


http:...
on the other hand here we see the extension
inherited the logging information from the parent.
The same holds true from any value. Values from
ocis.yaml are carried down to their respective
extensions.
on the other hand here we see the extension...Viewer does not support full SVG 1.1 diff --git a/content/ocis/9.0/modules/developer/images/ocis/storage/frontend.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/storage/frontend.drawio.svg new file mode 100644 index 00000000..45289f34 --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/storage/frontend.drawio.svg @@ -0,0 +1,303 @@ + + + + + + + + +
+
+
+ + GET /data/<transfer_token> + +
+
+
+ + + GET /da... + + + + + + + + +
+
+
+ + PROPFIND /webdav + +
+
+
+ + + PROPFIN... + + + + + + + + +
+
+
+
+ + POST /ocs/v1/apps/files_sharing/api/v1/shares + +
+
+ + path=/path/to/file + +
+
+ + shareType=0 + +
+
+ + shareWith=<username> + +
+
+
+
+ + + POST /o... + + + + + + + +
+
+
+ frontend +
+
+
+ + + frontend + + + + + + + + + + +
+
+
+ datagateway +
+
+
+ + + datagateway + + + + + + + +
+
+
+ ocdav +
+
+
+ + + ocdav + + + + + + + +
+
+
+ ocs +
+
+
+ + + ocs + + + + + + + +
+
+
+ gateway +
+
+
+ + + gateway + + + + + + + + + + +
+
+
+ storage home | users | ... +
+
+
+ + + storage home | users | ... + + + + + + + + + + +
+
+
+ storageprovider +
+
+
+ + + storageprovider + + + + + + + +
+
+
+ dataprovider +
+
+
+ + + dataprovider + + + + + + + + +
+
+
+ + GET target URL extracted from transfer token (JWT) + +
+
+
+ + + GET tar... + + + + + + + + +
+
+
+ + Stat +
+ ListContainer + +
+
+
+ + + Stat... + + + + + + + + +
+
+
+ + Stat +
+ CreateShare +
+ CreatePublicShare +
+ CreateOCMShare + +
+
+
+ + + Stat... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/storage/namespaces.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/storage/namespaces.drawio.svg new file mode 100644 index 00000000..b3baa589 --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/storage/namespaces.drawio.svg @@ -0,0 +1,341 @@ + + + + + + + + +
+
+
+ /rel/path/to/file.ext +
+
+
+ + + /rel/p... + + + + + + + + +
+
+
+
+ / +
+
+
+ /home +
+
+ /users/<user_layout> +
+
+ /public +
+
+ /shares +
+
+
+
+ + + /... + + + + + + + +
+
+
+ storage home +
+
+
+ + + storage home + + + + + + + + + +
+
+
+ storageprovider +
+
+
+ + + storageprovider + + + + + + + +
+
+
+ dataprovider +
+
+
+ + + dataprovider + + + + + + + +
+
+
+ frontend +
+
+
+ + + frontend + + + + + + + + + +
+
+
+ ocdav +
+
+
+ + + ocdav + + + + + + + +
+
+
+ ocs +
+
+
+ + + ocs + + + + + + + + +
+
+
+ + /webdav +
+ /dav/files/<username> +
+ +
+ + /dav/trash-bin/<username> + +
+
+ + /dav/public-files/<token> + +
+
+ + /dav/meta/<fileid>/v + +
+
+ + /dav/avatars/<username> + +
+
+
+
+ + + /webdav... + + + + + + + + +
+
+
+ + /ocs/v1.php/apps/files_sharing/api/v1/shares + +
+
+
+ + + /ocs/v1.ph... + + + + + + + +
+
+
+ gateway +
+
+
+ + + gateway + + + + + + + + + +
+
+
+ gateway +
+
+
+ + + gateway + + + + + + + +
+
+
+ authregistry +
+
+
+ + + authregistry + + + + + + + +
+
+
+ storageregistry +
+
+
+ + + storageregistry + + + + + + + + + +
+
+
+ oc10 namespace +
+ (all paths aere relative to the users home) +
+
+
+ + + oc10 names... + + + + + + + +
+
+
+ CS3 global namespace +
+
+
+ + + CS3 global... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/storage/overview.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/storage/overview.drawio.svg new file mode 100644 index 00000000..8dc5bf16 --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/storage/overview.drawio.svg @@ -0,0 +1,685 @@ + + + + + + + + + + + + + + + + + +
+
+
+ proxy +
+
+
+ + + proxy + + + + + + + + + + + + +
+
+
+ gateway +
+
+
+ + + gateway + + + + + + + + + + +
+
+
+ gateway +
+
+
+ + + gateway + + + + + + + +
+
+
+ authregistry +
+
+
+ + + authregistry + + + + + + + +
+
+
+ storageregistry +
+
+
+ + + storageregistry + + + + + + + + + + + + + +
+
+
+ storage users +
+
+
+ + + storage users + + + + + + + + + + +
+
+
+ storageprovider +
+
+
+ + + storageprovider + + + + + + + +
+
+
+ dataprovider +
+
+
+ + + dataprovider + + + + + + + +
+
+
+ storage home +
+
+
+ + + storage home + + + + + + + + + + +
+
+
+ storageprovider +
+
+
+ + + storageprovider + + + + + + + +
+
+
+ dataprovider +
+
+
+ + + dataprovider + + + + + + + +
+
+
+ storage public link +
+
+
+ + + storage public link + + + + + + + + + + +
+
+
+ publicstorageprovider +
+
+
+ + + publicstorageprovider + + + + + + + +
+
+
+ authprovider +
+ publicshares +
+
+
+ + + authprovider... + + + + + + + +
+
+
+ storage metadata +
+
+
+ + + storage metadata + + + + + + + + + + +
+
+
+ storageprovider +
+
+
+ + + storageprovider + + + + + + + +
+
+
+ dataprovider +
+
+
+ + + dataprovider + + + + + + + +
+
+
+ sharing +
+
+
+ + + sharing + + + + + + + + + + +
+
+
+ usershareprovider +
+
+
+ + + usershareprovider + + + + + + + +
+
+
+ publicshareprovider +
+
+
+ + + publicshareprovider + + + + + + + +
+
+
+ users +
+
+
+ + + users + + + + + + + + + + +
+
+
+ userprovider +
+
+
+ + + userprovider + + + + + + + +
+
+
+ groups +
+
+
+ + + groups + + + + + + + + + + +
+
+
+ groupprovider +
+
+
+ + + groupprovider + + + + + + + +
+
+
+ authbasic +
+
+
+ + + authbasic + + + + + + + + + + +
+
+
+ authprovider +
+
+
+ + + authprovider + + + + + + + +
+
+
+ authbearer +
+
+
+ + + authbearer + + + + + + + + + + +
+
+
+ authprovider +
+
+
+ + + authprovider + + + + + + + +
+
+
+ accounts +
+
+
+ + + accounts + + + + + + + + + + + +
+
+
+ frontend +
+
+
+ + + frontend + + + + + + + + + + +
+
+
+ datagateway +
+
+
+ + + datagateway + + + + + + + +
+
+
+ ocdav +
+
+
+ + + ocdav + + + + + + + +
+
+
+ ocs +
+
+
+ + + ocs + + + + + + + +
+
+
+ ocis +
+
+
+ + + ocis + + + + + + + +
+
+
+ reva +
+
+
+ + + reva + + + + + + + +
+
+
+ deprecated +
+
+
+ + + deprec... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/storage/semver-schema.png b/content/ocis/9.0/modules/developer/images/ocis/storage/semver-schema.png new file mode 100644 index 00000000..f47823e9 Binary files /dev/null and b/content/ocis/9.0/modules/developer/images/ocis/storage/semver-schema.png differ diff --git a/content/ocis/9.0/modules/developer/images/ocis/storage/spacesprovider.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/storage/spacesprovider.drawio.svg new file mode 100644 index 00000000..d122c58f --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/storage/spacesprovider.drawio.svg @@ -0,0 +1,352 @@ + + + + + + + +
+
+
+ oCIS spaces provider +
+ [Software System] +
+
+
+ + + oCIS spaces provider... + + + + + + + +
+
+
+ + reva storage provider + +
+ [Component: golang] +
+
+
+ hosts multiple storage spaces using a storage driver +
+
+
+
+ + + reva storage provider... + + + + + + + +
+
+
+ + reva gateway + +
+ [Component: golang] +
+
+
+ API facade for internal reva services +
+
+
+
+ + + reva gateway... + + + + + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+ + + Storage System... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + +
+
+
+ + reva frontend + +
+ [Component: golang] +
+
+
+ handles protocol translation +
+
+
+
+ + + reva frontend... + + + + + + + +
+
+
+ + oCIS proxy + +
+ [Component: golang] +
+
+
+ Routes requests to oc10 or ecis +
+
+
+
+ + + oCIS proxy... + + + + + + + + +
+
+
+
+
+ + Mints an internal JWT +
+ and torwards requests to + +
+
+ [WebDAV, OCS, OCM, tus] +
+
+
+
+
+ + + Mints an internal JWT... + + + + + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, +
+ Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+ + + Client... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [WebDAV, libregraph, CS3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [CS3, tus] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Forwards to + +
+
+ [CS3, storage registry] +
+
+
+
+
+ + + Forwards to... + + + + + + + +
+
+
+

+ C4 Component diagram for an oCIS spaces provider +

+

+ An oCIS spaces provider manages resources in storage spaces by persisting them with a specific storage driver in a storage system. +

+

+ Date: 2021-07-22T12:40 +

+
+
+
+ + + C4 Component diagram for an oCIS spaces provider... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/storage/spacesregistry.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/storage/spacesregistry.drawio.svg new file mode 100644 index 00000000..f716b0bb --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/storage/spacesregistry.drawio.svg @@ -0,0 +1,405 @@ + + + + + + + +
+
+
+ oCIS spaces registry +
+ [Software System] +
+
+
+ + + oCIS spaces registry... + + + + + + + +
+
+
+ + reva storage registry + +
+ [Component: golang] +
+
+
+ manages and caches storage space metadata +
+
+
+
+ + + reva storage registry... + + + + + + + +
+
+
+ + reva gateway + +
+ [Component: golang] +
+
+
+ API facade for internal reva services +
+
+
+
+ + + reva gateway... + + + + + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+ + + Storage System... + + + + + + + + +
+
+
+
+
+ + Provisions and manages spaces in + +
+
+ [CS3] +
+
+
+
+
+ + + Provisions and manages spaces... + + + + + + + +
+
+
+ + reva frontend + +
+ [Component: golang] +
+
+
+ handles protocol translation +
+
+
+
+ + + reva frontend... + + + + + + + +
+
+
+ + oCIS proxy + +
+ [Component: golang] +
+
+
+ Routes requests to oc10 or ecis +
+
+
+
+ + + oCIS proxy... + + + + + + + + +
+
+
+
+
+ + Mints an internal JWT +
+ and torwards requests to + +
+
+ [libregraph] +
+
+
+
+
+ + + Mints an internal JWT... + + + + + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, +
+ Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+ + + Client... + + + + + + + + +
+
+
+
+
+ + polls or gets notified about changes in + +
+
+ [libregraph] +
+
+
+
+
+ + + polls or gets notified about c... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [CS3, tus] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Lists spaces using + +
+
+ [CS3] +
+
+
+
+
+ + + Lists spaces using... + + + + + + + +
+
+
+

+ C4 Component diagram for an oCIS spaces registry +

+

+ An oCIS spaces provider manages resources in storage spaces by persisting them with a specific storage driver in a storage system. +

+

+ Date: 2021-07-22T12:40 +

+
+
+
+ + + C4 Component diagram for an oCIS spaces registry... + + + + + + + +
+
+
+ + reva storage provider + +
+ [Component: golang] +
+
+
+ hosts multiple storage spaces using a storage driver +
+
+
+
+ + + reva storage provider... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/storage/storage.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/storage/storage.drawio.svg new file mode 100644 index 00000000..fd6e759c --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/storage/storage.drawio.svg @@ -0,0 +1,434 @@ + + + + + + + +
+
+
+ oCIS System +
+ [Software System] +
+
+
+ + + oCIS System... + + + + + + + + + +
+
+
+ + Einstein + +
+ [Person] +
+
+
+ End user +
+
+
+
+ + + Einstein... + + + + + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+ + + Client... + + + + + + + +
+
+
+ + Storage Space Registry + +
+ [Container: golang, HTTP, libregraph] +
+
+
+ Manages spaces for users +
+
+
+
+ + + Storage Space Registry... + + + + + + + +
+
+
+ + Storage Space Provider + +
+ [Container: golang] +
+
+
+ Persists storage spaces using reva +
+
+
+
+ + + Storage Space Provider... + + + + + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+ + + Storage System... + + + + + + + + + +
+
+
+ + Moss + +
+ [Person] +
+
+
+ Administrator +
+
+
+
+ + + Moss... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [WebDAV, libregraph, CS3, tus] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Manages the users Storage Spaces in + +
+
+ [libregraph] +
+
+
+
+
+ + + Manages the users Storage Spac... + + + + + + + + +
+
+
+
+
+ + Manages resources with + +
+
+ [Web UI or native clients] +
+
+
+
+
+ + + Manages resources with... + + + + + + + + +
+
+
+
+
+ + Registers itself at and +
+ sends space root etag changes to + +
+
+ [CS3, libregraph?, PUSH] +
+
+
+
+
+ + + Registers itself at and... + + + + + + + + +
+
+
+
+
+ + Manages organizational Storage Spaces in + +
+
+ [WebDAV, libregraph, CS3, CLI] +
+
+
+
+
+ + + Manages organizational Storage... + + + + + + + +
+
+
+ + Identity Management System + +
+ [Software System] +
+
+
+ provides users and groups +
+
+
+
+ + + Identity Management System... + + + + + + + + +
+
+
+
+
+ + Authenticates users and searches recipients with + +
+
+ [OpenID Connect, LDAP, REST] +
+
+
+
+
+ + + Authenticates users and search... + + + + + + + +
+
+
+

+ C4 Container diagram for the oCIS System +

+

+ As a platform, the oCIS system may not only includes web, mobile and desktop clients but also the underlying storage system or an identity management system +

+

+ Date: 2021-07-22T16:43 +

+
+
+
+ + + C4 Container diagram for the oCIS System... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/storage/storageprovider-spaces.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/storage/storageprovider-spaces.drawio.svg new file mode 100644 index 00000000..9aed5040 --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/storage/storageprovider-spaces.drawio.svg @@ -0,0 +1,471 @@ + + + + + + + + + +
+
+
+ + CS3 +
+ storage provider +
+ API (GRPC) +
+ +
+
+
+ + + CS3... + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ storage provider +
+
+
+ + + storage provider + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/storage/storageprovider.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/storage/storageprovider.drawio.svg new file mode 100644 index 00000000..e7ba5ea7 --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/storage/storageprovider.drawio.svg @@ -0,0 +1,352 @@ + + + + + + + +
+
+
+ oCIS storage provider +
+ [Software System] +
+
+
+ + + oCIS storage provider... + + + + + + + +
+
+
+ + reva storage provider + +
+ [Component: golang] +
+
+
+ hosts multiple storage spaces using a storage driver +
+
+
+
+ + + reva storage provider... + + + + + + + +
+
+
+ + reva gateway + +
+ [Component: golang] +
+
+
+ API facade for internal reva services +
+
+
+
+ + + reva gateway... + + + + + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+ + + Storage System... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + +
+
+
+ + reva frontend + +
+ [Component: golang] +
+
+
+ handles protocol translation +
+
+
+
+ + + reva frontend... + + + + + + + +
+
+
+ + oCIS proxy + +
+ [Component: golang] +
+
+
+ Routes requests to oc10 or ecis +
+
+
+
+ + + oCIS proxy... + + + + + + + + +
+
+
+
+
+ + Mints an internal JWT +
+ and torwards requests to + +
+
+ [WebDAV, OCS, OCM, tus] +
+
+
+
+
+ + + Mints an internal JWT... + + + + + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, +
+ Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+ + + Client... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [WebDAV, libregraph, CS3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [CS3, tus] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Forwards to + +
+
+ [CS3, storage registry] +
+
+
+
+
+ + + Forwards to... + + + + + + + +
+
+
+

+ C4 Component diagram for an oCIS storage provider +

+

+ An oCIS storage provider manages resources in storage spaces by persisting them with a specific storage driver in a storage system. +

+

+ Date: 2021-07-22T12:40 +

+
+
+
+ + + C4 Component diagram for an oCIS storage provider... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/storage/storageregistry.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/storage/storageregistry.drawio.svg new file mode 100644 index 00000000..2b24b386 --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/storage/storageregistry.drawio.svg @@ -0,0 +1,233 @@ + + + + + + + + +
+
+
+ The storage registry currently maps paths and storageids to the +
+ + address:port + + of the corresponding storage provider +
+
+
+ + + The storage registry currently maps... + + + + + + + + + + +
+
+
+ storage registry +
+
+
+ + + storage registry + + + + + + + + + + +
+
+
+ storage providers +
+
+
+ + + storage providers + + + + + + + + + + + + + + + +
+
+
+ The gateway uses the storage registry to look up the storage provider that is responsible for path and id based references in incoming requests. +
+
+
+ + + The gateway uses the storage regist... + + + + + + + + + + +
+
+
+ gateway +
+
+
+ + + gateway + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + / + +
+
+
+ + + / + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + / + +
+
+
+ + + / + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/ocis/storage/storagespace.drawio.svg b/content/ocis/9.0/modules/developer/images/ocis/storage/storagespace.drawio.svg new file mode 100644 index 00000000..6e60631b --- /dev/null +++ b/content/ocis/9.0/modules/developer/images/ocis/storage/storagespace.drawio.svg @@ -0,0 +1,163 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ the root resource of the storage space +
+
+
+ + + the root resource of... + + + + + + + +
+
+
+ RESOURCE_TYPE_CONTAINER +
+
+
+ + + RESOURCE_TYPE_CONTAINER + + + + + + + +
+
+
+ RESOURCE_TYPE_FILE +
+
+
+ + + RESOURCE_TYPE_FILE + + + + + + + + + +
+
+
+ if the gateway encounters a resource type reference it will look it up and replace the reference with the results of the actual node (which can live in another storage provider) +
+
+
+ + + if the gateway encounters a... + + + + + + + +
+
+
+ every resource can be referenced by path or by id +
+
+
+ + + every resource can be... + + + + + + + +
+
+
+ RESOURCE_TYPE_SYMLINK +
+
+
+ + + RESOURCE_TYPE_SYMLINK + + + + + + + + + + + +
+
+
+ RESOURCE_TYPE_REFERENCE +
+
+
+ + + RESOURCE_TYPE_REFERENCE + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/content/ocis/9.0/modules/developer/images/web/oauth2.png b/content/ocis/9.0/modules/developer/images/web/oauth2.png new file mode 100644 index 00000000..c948efae Binary files /dev/null and b/content/ocis/9.0/modules/developer/images/web/oauth2.png differ diff --git a/content/ocis/9.0/modules/developer/index.adoc b/content/ocis/9.0/modules/developer/index.adoc deleted file mode 100644 index 34d2a081..00000000 --- a/content/ocis/9.0/modules/developer/index.adoc +++ /dev/null @@ -1,45 +0,0 @@ -= Infinite Scale Developer Documentation -:toc: right -:toclevels: 2 -:description: Welcome to the Infinite Scale developer documentation. Infinite Scale is a data platform providing tools to integrate, organize, share and govern data and metadata. - -## Admin Documentation - -Before you start reading, if you are interested in: - -- learning ocis from an admin perspective, -- different deployment scenarios, -- deployment examples, -- detailed settings and more - -we would recommend to continue with the [ownCloud Admin Documentation for Infinite Scale](https://doc.owncloud.com/ocis/latest/). - -## Developer Documentation - -Welcome to our developer documentation. Here you find documentation with focus for *developers*: - -- [oCIS]({{< ref "./ocis" >}}) server -- oCIS Services -- Clients like: - - [ownCloud Web](https://github.com/owncloud/ocis/tree/master/web) - the new web frontend for oCIS and ownCloud - - [ownCloud Android app](https://github.com/owncloud/android) - - [ownCloud iOS app](https://github.com/owncloud/ios-app) - - [ownCloud Desktop Syncing Client](https://github.com/owncloud/client) -- Integrations - -## We Love Open Source - -The oCIS server is Apache v2 licensed. -The lower storage layer of oCIS is defined by the CS3 APIs and implemented in the REVA project. Our goal is to develop the CS3 APIs to an open standard and collaborate on the open source REVA reference implementation for CS3 APIs. - -You can also find all client sources on https://github.com/owncloud/[GitHub]. - -## Join the oCIS Community - -The https://github.com/owncloud/ocis[ocis repository] is a good entry point to the oCIS project. In addition to that, there are also ownCloud projects for apps such as https://github.com/owncloud/ios-app[iOS], https://github.com/owncloud/android[Android], the https://github.com/owncloud/desktop[Desktop] platforms and https://github.com/owncloud/ocis/tree/master/web[ownCloud Web]. - -To chat about development, join our public chat on https://matrix.to/#/#ocis:matrix.org[matrix: ownCloud Infinite Scale]. - -If you want to help and improve ownCloud or oCIS, start coding or open issues on GitHub in the related repository. - -We are very happy to hear your feedback and ideas! diff --git a/content/ocis/9.0/modules/developer/pages/apis/grpc_apis/index.adoc b/content/ocis/9.0/modules/developer/pages/apis/grpc_apis/index.adoc new file mode 100644 index 00000000..cbf0efc2 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/apis/grpc_apis/index.adoc @@ -0,0 +1,48 @@ += gRPC +:toc: right +:toclevels: 3 + +== Remote Procedure Calls + +link:https://grpc.io[gRPC] is a modern open source high performance Remote Procedure Call (RPC) framework that can run in any environment. It can efficiently connect services in and across data centers with pluggable support for load balancing, tracing, health checking and authentication. It is also applicable in last mile of distributed computing to connect devices, mobile applications and browsers to backend services. + +== Advantages of gRPC + +// when available, we can use icons at the bold text + +[width="100%", cols="25%,25%"] +|=== +a| *Performance* + +gRPC uses http/2 by default and is faster than REST. When using protocol buffers for encoding, the information comes on and off the wire much faster than JSON. Latency is an important factor in distributed systems. JSON encoding creates a noticeable factor of latency. For distributed systems and high data loads, gRPC can actually make an important difference. Other than that, gRPC supports multiple calls via the same channel and the connections are bidirectional. A single connection can transmit requests and responses at the same time. gRPC keeps connections open to reuse the same connection again which prevents latency and saves bandwidth. + +a| *Robustness* + +gRPC empowers better relationships between clients and servers. The rules of communication are strictly enforced. That is not the case in REST calls, where the client and the server can send and receive anything they like and hopefully the other end understands what to do with it. In gRPC, to make changes to the communication, both client and server need to change accordingly. This prevents mistakes specially in microservice architectures. + +a| *Debuggability* + +gRPC requests are re-using the same context and can be tracked or traced across multiple service boundaries. This helps to identify slow calls and see what is causing delays. It is possible to cancel requests which cancels them on all involved services. + +a| *Microservices* + +gRPC has been evolving and has become the best option for communication between microservices because of its unmatched performance and its polyglot nature. One of the biggest strengths of microservices is the freedom of programming languages and technologies. By using gRPC we can leverage all the advantages of strictly enforced communication standards combined with freedom of choice between different programming languages - whichever would fit best. +|=== + +[NOTE] +==== +* http/2 +* Protocol buffers +* Reusable connections +* Multi language support +==== + +== CS3 APIs + +image:ocis/apis/cs3org.png[,width=150] + +The link:https://github.com/cs3org/cs3apis[CS3 APIs] connect storages and application providers. + +The CS3 APIs follow Google and Uber API design guidelines, especially on error handling and naming convention. You can read more about these guidelines at https://cloud.google.com/apis/design/ and https://github.com/uber/prototool/blob/dev/style/README.md. + +The CS3 APIs use link:https://github.com/protocolbuffers/protobuf[Protocol Buffers version 3 (proto3)] as their Interface Definition Language (IDL) to define the API interface and the structure of the payload messages. diff --git a/content/ocis/9.0/modules/developer/pages/apis/http/authorization.adoc b/content/ocis/9.0/modules/developer/pages/apis/http/authorization.adoc new file mode 100644 index 00000000..47d3ae41 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/apis/http/authorization.adoc @@ -0,0 +1,169 @@ += Authorization +:toc: right +:toclevels: 3 + +In its default configuration, Infinite Scale supports three authentication methods as outlined on the link:https://openid.net/specs/openid-connect-core-1_0.html#rfc.section.3[OIDC official site]: + +. Authorization Code Flow +. Implicit Flow +. Hybrid Flow + +For detailed information on Infinite Scale's support for OpenID Connect (OIDC), please consult the xref:ocis/identity-provider/oidc.adoc[OIDC section]. To authenticate a client app using OIDC, both `client_id` and `client_secret` are essential. Infinite Scale does not offer dynamic registration. The required data for the default xref:server:admin_manual:configuration/user/oidc/oidc.adoc#client-ids-secrets-and-redirect-uris[ownCloud clients] can be found in the link and are available for the following apps: + +* Desktop +* Android +* iOS + +While selecting an ownCloud client for authentication, take note of specific limitations such as the `Redirect URI`: + +[width="50%", cols="25%,25%", options="header"] +|=== +| Source +| Redirect URI + +| Android +| oc://android.owncloud.com + +| iOS +| oc://ios.owncloud.com +| Desktop +a| http://127.0.0.1 + +http://localhost +|=== + +In this example, the desktop app's `client_id` and `client_secret` are being used. + +[source,plaintext] +---- +client_id=xdXOt13JKxym1B1QcEncf2XDkLAexMBFwiT9j6EfhhHFJhs2KM9jbjTmf8JBXE69 +client_secret=UBntmLjC2yYCeHwsyj73Uwo9TAaecAetRwMw0xYcvNL9yRdLSUi0hUAHfvCHFeFh +---- + +== Authorization Code Flow + +. Requesting authorization + +To initiate the OIDC Code Flow, you can use tools like curl and a web browser. The user should be directed to a URL to authenticate and give their consent (bypassing consent is against the standard): ++ +-- +[source,plaintext] +---- +https://ocis.test/signin/v1/identifier/_/authorize?client_id=client_id&scope=openid+profile+email+offline_access&response_type=code&redirect_uri=http://path-to-redirect-uri +---- + +After a successful authentication, the browser will redirect to a URL that looks like this: + +[source,plaintext] +---- +http://path-to-redirect-uri?code=mfWsjEL0mc8gx0ftF9LFkGb__uFykaBw&scope=openid%20profile%20email%20offline_access&session_state=32b08dd...&state= +---- + +For the next step extract the code from the URL. + +In the above example, the code is `mfWsjEL0mc8gx0ftF9LFkGb__uFykaBw` +-- + +. Requesting an access token + +The next step in the OIDC Code Flow involves an HTTP POST request to the token endpoint of the _Infinite Scale Identity Server_. ++ +-- +[source,bash] +---- +curl -vk -X POST https://ocis.test/konnect/v1/token \ + -d "grant_type=authorization_code" \ + -d "code=3a3PTcO-WWXfN3l1mDN4u7G5PzWFxatU" \ + -d "redirect_uri=http:path-to-redirect-uri" \ + -d "client_id=client_id" \ + -d "client_secret=client_secret" +---- + +Response looks like this: + +[source,json] +---- +{ + "access_token": "eyJhbGciOid...", + "token_type": "Bearer", + "id_token": "eyJhbGciOi...", + "refresh_token": "eyJhbGciOiJ...", + "expires_in": 300 +} +---- +-- + +. Refreshing an access token + +If the access token has expired, you can get a new one using the refresh token. ++ +-- +[source,bash] +---- +curl -vk -X POST https://ocis.test/konnect/v1/token \ + -d "grant_type=refresh_token" \ + -d "refresh_token=eyJhbGciOiJ..." \ + -d "redirect_uri=http://path-to-redirect-uri" \ + -d "client_id=client_id" \ + -d "client_secret=client_secret" +---- + +Response looks like this: + +[source,json] +---- +{ + "access_token": "eyJhbGciOi...", + "token_type": "Bearer", + "expires_in": 300 +} +---- +-- + +== Implicit Flow + +When using the implicit flow, tokens are provided in a URI fragment of the redirect URL. Valid values for the `response_type` request parameter are: + +* token +* id_token token + +[WARNING] +==== +If you are using the implicit flow, `nonce` parameter is required in the initial `/authorize` request. `nonce=pL3UkpAQPZ8bTMGYOmxHY/dQABin8yrqipZ7iN0PY18=` +==== + +Bash command to generate cryptographically random value: + +[source,bash] +---- +openssl rand -base64 32 +---- + +[TIP] +==== +The user should be directed to a URL to authenticate and give their consent (bypassing consent is against the standard): + +[source,bash] +---- +https://ocis.test/signin/v1/identifier/_/authorize?client_id=client_id&scope=openid+profile+email+offline_access&response_type=id_token+token&redirect_uri=http://path-to-redirect-uri&nonce=pL3UkpAQPZ8bTMGYOmxHY/dQABin8yrqipZ7iN0PY18= +---- +==== + +After a successful authentication, the browser will redirect to a URL that looks like this: + +[source,bash] +---- +http://path-to-redirect-uri#access_token=eyJhbGciOiJQUzI...&expires_in=300&id_token=eyJhbGciOiJ...&scope=email%20openid%20profile&session_state=c8a1019f5e054d...&state=&token_type=Bearer +---- + +For the next step, extract the access_token from the URL. + +[source,bash] +---- +access_token = 'eyJhbGciOiJQ...' +---- + +== Hybrid Flow + +The Hybrid Flow in OpenID Connect melds features from both the Implicit and Authorization Code flows. It allows clients to directly retrieve certain tokens from the Authorization Endpoint, yet also offers the option to acquire additional tokens from the Token Endpoint. + +The Authorization Server redirects back to the client with appropriate parameters in the response, based on the value of the response_type request parameter: + +* code token +* code id_token +* code id_token token diff --git a/content/ocis/9.0/modules/developer/pages/apis/http/graph/groups.adoc b/content/ocis/9.0/modules/developer/pages/apis/http/graph/groups.adoc new file mode 100644 index 00000000..863f9184 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/apis/http/graph/groups.adoc @@ -0,0 +1,299 @@ += Groups +:toc: right +:toclevels: 3 + + +== Groups API + +The Groups API is implementing a subset of the functionality of the link:https://learn.microsoft.com/en-us/graph/api/resources/group?view=graph-rest-1.0[MS Graph Group resource] The JSON representation of a Group as handled by the Groups API looks like this: + +[source,json] +---- +{ + "displayName": "group", + "id": "f0d97060-da16-4b0d-9fa4-d1ec43afc5f1" +} +---- + +Our implementation currently supports two Attributes for a Group: + +[width="100%", cols="25%,75%", options="header"] +|=== +| Attribute +| Description + +| displayName +| The groups name + +| id +| An unique, stable readonly identifier for the group that stays the same for the whole lifetime of the Group, usually a UUID +|=== + + +=== Reading Groups + +==== `GET /groups` + +Returns a list of all groups + +Example:: ++ +[source,bash] +---- +curl -k 'https://localhost:9200/graph/v1.0/groups' \ + -u user:password +---- + +Response:: ++ +[source,json] +---- +{ + "value": [ + { + "displayName": "group", + "id": "38580a2e-7018-42ed-aff6-b2af0b4e9790" + }, + { + "displayName": "Example Users", + "id": "7a20f238-8a22-4458-902d-47674c317e5f" + } + ] +} +---- + + +==== `GET /groups?$expand=members` + +Returns a list of all groups including its members + +Example:: ++ +[source,bash] +---- +curl -k 'https://localhost:9200/graph/v1.0/groups?$expand=members' \ + -u user:password +---- + +Response:: ++ +[source,json] +---- +{ + "value": [ + { + "displayName": "group", + "id": "38580a2e-7018-42ed-aff6-b2af0b4e9790", + "members": [ + { + "displayName": "user1", + "id": "2e7b7e23-6c42-4d34-81b0-2bed34e51983", + "mail": "user1@example.org", + "onPremisesSamAccountName": "user1" + }, + { + "displayName": "user2", + "id": "b45c9e35-0d95-4165-96bc-68bff4a316ed", + "mail": "user2@example.org", + "onPremisesSamAccountName": "user2" + } + ] + }, + { + "displayName": "Example Users", + "id": "7a20f238-8a22-4458-902d-47674c317e5f", + "members": [ + { + "displayName": "user3", + "id": "026fbfef-79ef-4f5d-887b-9eaf42777239", + "mail": "user3@example.org", + "onPremisesSamAccountName": "user3" + } + ] + } + ] +} +---- + +==== `GET /groups/\{groupid}` + +Example:: ++ +[source,bash] +---- +curl -k 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f' \ + -u user:password +---- + +Response:: ++ +[source,json] +---- +{ + "displayName": "Example Users", + "id": "7a20f238-8a22-4458-902d-47674c317e5f" +} +---- + +==== `GET /groups/\{groupid}?$expand=members` + +Example:: ++ +[source,bash] +---- +curl -k 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f?$expand=members' \ + -u user:password +---- + +Response:: ++ +[source,json] +---- +{ + "displayName": "Example Users", + "id": "7a20f238-8a22-4458-902d-47674c317e5f", + "members": [ + { + "displayName": "user3", + "id": "026fbfef-79ef-4f5d-887b-9eaf42777239", + "mail": "user3@example.org", + "onPremisesSamAccountName": "user3" + } + ] +} +---- + +=== Getting Group Members + +==== `GET /groups/\{groupid}/members` + +Returns a list of User objects that are members of a group. + +Example:: ++ +[source,bash] +---- +curl -k 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f/members' \ + -u user:password +---- + +Response:: ++ +[source,json] +---- +[ + { + "displayName": "Test User", + "id": "c54b0588-7157-4521-bb52-c1c8ca84ea71", + "mail": "example@example.org", + "onPremisesSamAccountName": "example" + }, + { + "displayName": "Albert Einstein", + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "mail": "einstein@example.org", + "onPremisesSamAccountName": "einstein" + } +] +---- + +=== Creating / Updating Groups + +==== `POST /groups` + +Use this to create a new group. + +===== Request Body + +Note the missing `"id"` Attribute. It will be generated by the server: + +[source,json] +---- +{ + "displayName": "Example Users" +} +---- + +===== Response + +When successful, the response will return the new group including the newly allocated `"id"`: + +[source,json] +---- +{ + "displayName": "Example Users", + "id": "7a20f238-8a22-4458-902d-47674c317e5f" +} +---- + +==== `DELETE /groups/\{id}` + +Example:: ++ +[source,bash] +---- +curl -k --request DELETE 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f' \ + -u user:password +---- + +When successful the API returns no response body and the HTTP status code 204 (No Content) + +==== `PATCH /groups/\{id}` + +Updating attributes of a single group is supposed to be done with a patch request. This is however currently not fully +implemented for our write-enabled backends. The PATCH request can however be used to add multiple members to a group at once. +See below. + +=== Adding a Single Member to a Group + +==== `POST /groups/\{id}/members/$ref` + +The request body contains a single attribute "`@odata.id`" referencing the new member of the group by URI. + +Example:: ++ +[source,bash] +---- +curl -k --header "Content-Type: application/json" \ + --request POST --data \ + '{ "@odata.id": "https://localhost:9200/graph/v1.0/users/4c510ada-c86b-4815-8820-42cdf82c3d51" }' \ + 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f/members/$ref' \ + -u user:password +---- + +When successful the API returns no response body and the HTTP status code 204 (No Content) + +=== Adding Multiple Members in a Single Request + +==== `PATCH /groups/\{id}` + +The request body contains the attribute `members@odata.bind` holding a list of URI references for the new members. + +Example:: ++ +[source,json] +---- +{ + "members@odata.bind": [ + "https://localhost:9200/graph/v1.0/users/4c510ada-c86b-4815-8820-42cdf82c3d51", + "https://localhost:9200/graph/v1.0/users/c54b0588-7157-4521-bb52-c1c8ca84ea71" + ] +} +---- + +When successful the API returns no response body and the HTTP status code 204 (No Content) + +=== Removing a Member + +==== `DELETE /groups/\{groupid}/members/\{id}/$ref` + +Example:: ++ +[source,bash] +---- +curl -k --request DELETE \ + 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f/members/4c510ada-c86b-4815-8820-42cdf82c3d51/$ref' \ + -u user:password +---- + +When successful the API returns no response body and the HTTP status code 204 (No Content) diff --git a/content/ocis/9.0/modules/developer/pages/apis/http/graph/index.adoc b/content/ocis/9.0/modules/developer/pages/apis/http/graph/index.adoc new file mode 100644 index 00000000..97567c94 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/apis/http/graph/index.adoc @@ -0,0 +1,96 @@ += LibreGraph +:toc: right +:toclevels: 3 + +The LibreGraph API is a REST Api which is inspired by the link:https://developer.microsoft.com/en-us/graph[Microsoft Graph API]. It tries to stay compliant with the Microsoft Graph API and aims to be the Next Generation Api in Infinite Scale where we want to support most of the features of the platform. +The link:https://github.com/owncloud/libre-graph-api[API specification] is available in the OpenApi 3 standard and there are generated client and server link:https://github.com/owncloud/libre-graph-api#clients[SDKs] available. You can browse the API with the xref:apis/http/graph/swagger-ui.adoc[Swagger UI]. + +== Calling the LibreGraph API + +[source,bash] +---- +{HTTP method} https://ocis.url/graph/{version}/{resource}?{query-parameters} +---- + +The request component consists of: + +[width="100%", cols="25%,75%", options="header"] +|=== +| Component +| Description + +| {HTTP method} +| The HTTP method which is used in the request. + +| \{version} +| The version of the LibreGraph API used by the client. + +| \{resource} +| The LibreGraph Resource which the client is referencing in the request. + +| \{query-parameters} +| Optional parameters for the request to customize the response. +|=== + +=== HTTP Methods + +[width="100%", cols="25%,75%", options="header"] +|=== +| Method +| Description + +| GET +| Read data from a resource. + +| POST +| Create a new resource. + +| PATCH +| Update an existing resource. + +| PUT +| Replace an existing resource. + +| DELETE +| Delete an existing resource. +|=== + +The methods `GET` and `DELETE` need no request body. The methods `POST`, `PATCH` and `PUT` require a request body, normally in JSON format to provide the needed values. + +=== Version + +Infinite Scale currently provides the version `v1.0`. + +=== Resource + +A resource could be an entity or a complex type and is usually defined by properties. Entities are always recognizable by an `Id` property. The URL contains the resource which you are interacting with e.g. `/me/drives` or `/groups/\{group-id}`. + +Each resource could possibly require different permissions. Usually you need permissions on a higher level for creating or updating an existing resource than for reading. + +=== Query Parameters + +Query parameters can be OData system query options, or other strings that a method accepts to customize its response. + +You can use optional OData system query options to include more or fewer properties than the default response, filter the response for items that match a custom query, or provide additional parameters for a method. + +For example, adding the following filter parameter restricts the drives returned to only those with the driveType property of `project`. + +[source,bash] +---- +GET https://ocis.url/graph/v1.0/drives?$filter=driveType eq 'project' +---- +For more information about OData query options please check the link:https://github.com/owncloud/libre-graph-api[API specification] and the provided examples. + +=== Authorization + +For development purposes Basic Auth can be used. It is disabled by default and should only be enabled by setting `PROXY_ENABLE_BASIC_AUTH` in the xref:admin:{s-path}/proxy.adoc[proxy] for development or test instances. + +To authenticate with a Bearer token or OpenID Connect access token replace the `-u user:password` Basic Auth option of curl with a `-H 'Authorization: Bearer '` header. A `` can be obtained by copying it from a request in the browser, although it will time out within minutes. To automatically refresh the OpenID Connect access token an ssh-agent like solution like link:https://github.com/indigo-dc/oidc-agent[oidc-agent] should be used. The graph endpoints that support a preconfigured token can be found in the link:https://github.com/owncloud/libre-graph-api[API specification] + +== Resources + +* xref:apis/http/graph/spaces.adoc[Spaces] +* xref:apis/http/graph/users.adoc[Users] +* xref:apis/http/graph/groups.adoc[Groups] +* xref:apis/http/graph/permissions.adoc[Permissions] +* xref:apis/http/graph/role.adoc[Role] diff --git a/content/ocis/9.0/modules/developer/pages/apis/http/graph/permissions.adoc b/content/ocis/9.0/modules/developer/pages/apis/http/graph/permissions.adoc new file mode 100644 index 00000000..4a4ac1f7 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/apis/http/graph/permissions.adoc @@ -0,0 +1,193 @@ += Permissions +:toc: right +:toclevels: 3 + +== Permissions API + +The Permissions API is implementing a subset of the functionality of the link:https://learn.microsoft.com/en-us/graph/api/resources/permission?view=graph-rest-1.0[MS Graph Permission resource]. + +=== Example Permissions + +The JSON representation of a Drive, as handled by the Spaces API, looks like this: + +[source,json] +---- +{ + "@libre.graph.permissions.roles.allowedValues": [ + { + "id": "b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5", + "description": "Allows reading the shared file or folder", + "displayName": "Viewer", + "@libre.graph.weight": 1 + }, + { + "id": "fb6c3e19-e378-47e5-b277-9732f9de6e21", + "description": "Allows reading and writing the shared file or folder", + "displayName": "Editor", + "@libre.graph.weight": 2 + }, + { + "id": "312c0871-5ef7-4b3a-85b6-0e4074c64049", + "description": "Allows managing a space", + "displayName": "Manager", + "@libre.graph.weight": 3 + }, + { + "id": "4916f47e-66d5-49bb-9ac9-748ad00334b", + "description": "Allows creating new files", + "displayName": "File Drop", + "@libre.graph.weight": 4 + } + ], + "@libre.graph.permissions.actions.allowedValues": [ + "libre.graph/driveItem/basic/read", + "libre.graph/driveItem/permissions/read", + "libre.graph/driveItem/upload/create", + "libre.graph/driveItem/standard/allTasks", + "libre.graph/driveItem/upload/create" + ], + "value": [ + { + "id": "67445fde-a647-4dd4-b015-fc5dafd2821d", + "link": { + "type": "view", + "webUrl": "https://cloud.example.org/s/fhGBMIkKFEHWysj" + } + }, + { + "id": "34646ab6-be32-43c9-89e6-987e0c237e9b", + "roles": [ + "b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5" + ], + "grantedToV2": [ + { + "user": { + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "displayName": "Albert Einstein" + } + } + ] + }, + { + "id": "81d5bad3-3eff-410a-a2ea-eda2d14d4474", + "roles": [ + "b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5" + ], + "grantedToV2": [ + { + "user": { + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "displayName": "Albert Einstein" + } + } + ] + }, + { + "id": "b470677e-a7f5-4304-8ef5-f5056a21fff1", + "roles": [ + "b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5" + ], + "grantedToV2": [ + { + "user": { + "id": "f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c", + "displayName": "Marie Skłodowska Curie" + } + } + ] + }, + { + "id": "453b02be-4ec2-4e7d-b576-09fc153de812", + "roles": [ + "fb6c3e19-e378-47e5-b277-9732f9de6e21" + ], + "grantedToV2": [ + { + "user": { + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "displayName": "Albert Einstein" + } + } + ], + "expirationDateTime": "2018-07-15T14:00:00.000Z" + }, + { + "id": "86765c0d-3905-444a-9b07-76201f8cf7df", + "roles": [ + "312c0871-5ef7-4b3a-85b6-0e4074c64049" + ], + "grantedToV2": [ + { + "group": { + "id": "167cbee2-0518-455a-bfb2-031fe0621e5d", + "displayName": "Philosophy Haters" + } + } + ] + }, + { + "id": "c42b5cbd-2d65-42cf-b0b6-fb6d2b762256", + "grantedToV2": [ + { + "user": { + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "displayName": "Albert Einstein" + } + } + ], + "@libre.graph.permissions.actions": [ + "libre.graph/driveItem/basic/read", + "libre.graph/driveItem/path/update" + ] + } + ] +} +---- + +## Creating Share Invitation / Link + +### Create a link share + +`POST /drives/\{drive-id}/items/\{item-id}/createLink` + +link:swagger-ui.html#/drives.permissions/CreateLink[LibreGraph: drives.permissions/CreateLink,window=_blank] + +### Create a user/group share + +`POST /drives/\{drive-id}/items/\{item-id}/invite` + +link:swagger-ui.html#/drives.permissions/Invite[LibreGraph: drives.permissions/Invite,window=_blank] + +## Reading Permissions + +### List the effective sharing permissions on a driveitem + +`GET /drives/\{drive-id}/items/\{item-id}/permissions` + +link:swagger-ui.html#/drives.permissions/ListPermissions[LibreGraph: drives.permissions/ListPermissions,window=_blank] + +### List Get sharing permission for a file or folder + +`GET /drives/\{drive-id}/items/\{item-id}/permissions/\{perm-id}` + +link:swagger-ui.html#/drives.permissions/GetPermission[LibreGraph: drives.permissions/GetPermission,window=_blank] + +## Updating Permissions + +### Updating sharing permission + +`POST /drives/\{drive-id}/items/\{item-id}/permissions/\{perm-id}` + +link:swagger-ui.html#/drives.permissions/UpdatePermission[LibreGraph: drives.permissions/UpdatePermission,window=_blank] + +### Set password of permission + +`POST /drives/\{drive-id}/items/\{item-id}/permissions/\{perm-id}/setPassword` + +link:swagger-ui.html#/drives.permissions/SetPermissionPassword[LibreGraph: drives.permissions/SetPermissionPassword,window=_blank] + +### Deleting permission + +`DELETE /drives/\{drive-id}/items/\{item-id}/permissions/\{perm-id}` + +link:swagger-ui.html#/drives.permissions/DeletePermission[LibreGraph: drives.permissions/DeletePermission,window=_blank] diff --git a/content/ocis/9.0/modules/developer/pages/apis/http/graph/role.adoc b/content/ocis/9.0/modules/developer/pages/apis/http/graph/role.adoc new file mode 100644 index 00000000..07f8e927 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/apis/http/graph/role.adoc @@ -0,0 +1,41 @@ += Role +:toc: right +:toclevels: 3 + +== Role API + +The Roles API is implementing a subset of the functionality of the link:https://learn.microsoft.com/en-us/graph/api/resources/rolemanagement?view=graph-rest-1.0[MS Graph Role Management]. + +== Role Management + +=== List roleDefinitions + +`GET /v1beta1/roleManagement/permissions/roleDefinitions` + +link:swagger-ui.html#/roleManagement/ListPermissionRoleDefinitions[LibreGraph: roleManagement/ListPermissionRoleDefinitions,window=_blank] + +=== Get unifiedRoleDefinition + +`GET /drives/\{drive-id}/items/\{item-id}/permissions/\{perm-id}` + +link:swagger-ui.html#/roleManagement/GetPermissionRoleDefinition[LibreGraph: roleManagement/GetPermissionRoleDefinition,window=_blank] + +== Role Assignment + +=== Get appRoleAssignments of a User + +`GET /v1.0/users/\{user-id}/appRoleAssignments` + +link:swagger-ui.html#/user.appRoleAssignment/user.ListAppRoleAssignments[LibreGraph: user.appRoleAssignment/user.ListAppRoleAssignments,window=_blank] + +=== Grant an appRoleAssignment to a User + +`POST /v1.0/users/\{user-id}/appRoleAssignments` + +link:swagger-ui.html#/user.appRoleAssignment/user.CreateAppRoleAssignments[LibreGraph: user.appRoleAssignment/user.CreateAppRoleAssignments,window=_blank] + +=== Delete the appRoleAssignment from a User + +`DELETE /v1.0/users/\{user-id}/appRoleAssignments/\{appRoleAssignment-id}` + +link:swagger-ui.html#/user.appRoleAssignment/user.DeleteAppRoleAssignments[LibreGraph: user.appRoleAssignment/user.DeleteAppRoleAssignments,window=_blank] diff --git a/content/ocis/9.0/modules/developer/pages/apis/http/graph/spaces.adoc b/content/ocis/9.0/modules/developer/pages/apis/http/graph/spaces.adoc new file mode 100644 index 00000000..45ee276d --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/apis/http/graph/spaces.adoc @@ -0,0 +1,554 @@ += Spaces +:toc: right +:toclevels: 3 + +== Spaces API + +The Spaces API is implementing a subset of the functionality of the link:https://learn.microsoft.com/en-us/graph/api/resources/drive?view=graph-rest-1.0[MS Graph Drives resource]. + +=== Example Space + +The JSON representation of a Drive, as handled by the Spaces API, looks like this: + +[source,json] +---- +{ + "driveAlias": "project/mars", + "driveType": "project", + "id": "storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925", + "lastModifiedDateTime": "2023-01-24T21:19:26.417055+01:00", + "name": "Mars", + "owner": { + "user": { + "displayName": "", + "id": "89ad5ad2-5fdb-4877-b8c9-601a9670b925" + } + }, + "quota": { + "remaining": 999853685, + "state": "normal", + "total": 1000000000, + "used": 146315 + }, + "root": { + "eTag": "\"910af0061161c42d8d1224df6c4a2527\"", + "id": "storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925", + "permissions": [ + { + "grantedToIdentities": [ + { + "user": { + "displayName": "Admin", + "id": "some-admin-user-id-0000-000000000000" + } + } + ], + "roles": [ + "manager" + ] + } + ], + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925" + }, + "special": [ + { + "eTag": "\"f97829324f63ce778095334cfeb0097b\"", + "file": { + "mimeType": "image/jpeg" + }, + "id": "storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925!40171bea-3263-47a8-80ef-0ca20c37f45a", + "lastModifiedDateTime": "2022-02-15T17:11:50.000000496+01:00", + "name": "Mars_iStock-MR1805_20161221.jpeg", + "size": 146250, + "specialFolder": { + "name": "image" + }, + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925%2189ad5ad2-5fdb-4877-b8c9-601a9670b925/.space/Mars_iStock-MR1805_20161221.jpeg" + }, + { + "eTag": "\"ff38b31d8f109a4fbb98ab34499a3379\"", + "file": { + "mimeType": "text/markdown" + }, + "id": "storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925!e2167612-7578-46e2-8ed7-971481037bc1", + "lastModifiedDateTime": "2023-01-24T21:10:23.661841+01:00", + "name": "readme.md", + "size": 65, + "specialFolder": { + "name": "readme" + }, + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925%2189ad5ad2-5fdb-4877-b8c9-601a9670b925/.space/readme.md" + } + ], + "webUrl": "https://localhost:9200/f/storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925" +} +---- + +## Creating Spaces + +### Create a single space + +`POST /drives` + +link:swagger-ui.html#/drives/CreateDrive[LibreGraph: drives/CreateDrive,window=_blank] + +### Create a space item (Enable sync) + +`POST /drives/\{drive-id}/root/children` + +link:swagger-ui.html#/drives.root/CreateDriveItem[LibreGraph: drives.root/CreateDriveItem,window=_blank] + +## Reading Spaces + +[source,bash] +---- +GET https://ocis.url/graph/{version}/{me/}drives/?{query-parameters} +---- + +[width="100%", cols="25%,75%", options="header"] +|=== +| Attribute +| Description + +| \{version} +| The version of the LibreGraph API used by the client. + +| {/me} +| The `me` component of the part is optional. If used, you only see spaces where the acting user is a regular member of. + +| \{query-parameters} +| Optional parameters for the request to customize the response. +|=== + +=== List All Spaces + +`GET /drives` + +Returns a list of all available spaces, even ones where the acting user is not a regular member of. You need elevated permissions to do list all spaces. If you don't have the elevated permissions, the result is the same like `GET /me/drives`. + +[NOTE] +==== +The ownCloud spaces concept draws a strict line between users which can work with the content of a space and others who have the permission to manage the space. A user which is able to manage quota and space metadata does not necessarily need to be able to access the content of a space. + +* *Space Admin* + +There is a global user role "Space Admin" which grants users some global permissions to manage space quota and some space metadata. This Role enables the user also to disable, restore and delete spaces. He cannot manage space members. + +* *Space Manager* + +The "Space Manager" is a user which is a regular member of a space because he has been invited. In addition to being part of a space the user can also manage the memberships of the space. +==== + +=== List My Spaces + +`GET /me/drives` + +link:swagger-ui.html#/me.drives/ListMyDrives[LibreGraph: me.drives/ListMyDrives,window=_blank] + +== Modifying Spaces + +Modify the properties of a space. You need elevated permissions to execute this request. + +=== Set the Space Quota to 5GB + +`PATCH /drives/\{drive-id}` + +To limit the quota of a space you need to set the `quota.total` value. The API response will give back all actual quota properties. + +[source,json] +---- +{ + "quota": { + "remaining": 5368709120, + "state": "normal", + "total": 5368709120, + "used": 0 + } +} +---- + +[width="100%", cols="25%,75%", options="header"] +|=== +| Attribute +| Description + +| remaining +| The remaining disk space in `bytes`. If the quota is not limited, this will show the total available disk space. + +| state +| The state of the space in regards to quota usage. This can be used for visual indicators. It can be `normal`(<75%), `nearing`(between 75% and 89%), `critical`(between 90% and 99%) and `exceeded`(100%). + +| total +| The space id. The value needs to be a space ID. + +| used +| The used disk space in bytes. +|=== + +[tabs] +==== +Request:: ++ +-- +[source,bash] +---- +curl -L -k -X PATCH 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff' \ + -H 'Content-Type: application/json' \ + --data-raw '{"quota": { "total": 5368709120}}' +---- +-- + +Response - 200 OK:: ++ +-- +[source,json] +---- +{ + "description": "Marketing team resources", + "driveAlias": "project/marketing", + "driveType": "project", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "lastModifiedDateTime": "2023-01-18T17:13:48.385204589+01:00", + "name": "Marketing", + "owner": { + "user": { + "displayName": "", + "id": "535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + } + }, + "quota": { + "remaining": 5368709120, + "state": "normal", + "total": 5368709120, + "used": 0 + }, + "root": { + "eTag": "\"f91e56554fd9305db81a93778c0fae96\"", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "permissions": [ + { + "grantedToIdentities": [ + { + "user": { + "displayName": "Admin", + "id": "some-admin-user-id-0000-000000000000" + } + } + ], + "roles": [ + "manager" + ] + } + ], + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + }, + "webUrl": "https://localhost:9200/f/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" +} +---- +-- +==== + +### Change the space name, subtitle and alias + +`PATCH /drives/\{drive-id}` + +You can change multiple space properties in one request as long as you submit a valid JSON body. Please be aware that some properties need different permissions. + +[tabs] +==== +Request:: ++ +-- +[source,bash] +---- +curl -L -k -X PATCH 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff' \ + -H 'Content-Type: application/json' \ + --data-raw '{"name": "Mars", "description": "Mission to mars", "driveAlias": "project/mission-to-mars"}' +---- +-- + +Response - 200 OK:: ++ +-- +[source,json] +---- +{ + "description": "Mission to mars", + "driveAlias": "project/mission-to-mars", + "driveType": "project", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "lastModifiedDateTime": "2023-01-19T14:17:36.094283+01:00", + "name": "Mars", + "owner": { + "user": { + "displayName": "", + "id": "535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + } + }, + "quota": { + "remaining": 15, + "state": "normal", + "total": 15, + "used": 0 + }, + "root": { + "eTag": "\"f5fee4fdfeedd6f98956500779eee15e\"", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "permissions": [ + { + "grantedToIdentities": [ + { + "user": { + "displayName": "Admin", + "id": "some-admin-user-id-0000-000000000000" + } + } + ], + "roles": [ + "manager" + ] + } + ], + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + }, + "webUrl": "https://localhost:9200/f/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" +} +---- +-- +==== + +## Disabling / Deleting Spaces + +### Disable a space + +`DELETE /drives/\{drive-id}` + +This operation will make the space content unavailable for all space members. No data will be deleted. + +[tabs] +==== +Request:: ++ +-- +[source,bash] +---- +curl -L -k -X DELETE 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff/' +---- +-- + +Response - 200 OK:: ++ +-- +This response has no body value. + +A disabled space will appear in listings with a `root.deleted.state=trashed` property. The space description and the space image will not be readable anymore. + +[source,json] +---- +{ + "description": "Marketing team resources", + "driveAlias": "project/marketing", + "driveType": "project", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "lastModifiedDateTime": "2023-01-19T14:17:36.094283+01:00", + "name": "Marketing", + "owner": { + "user": { + "displayName": "", + "id": "535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + } + }, + "quota": { + "total": 15 + }, + "root": { + "deleted": { + "state": "trashed" + }, + "eTag": "\"f5fee4fdfeedd6f98956500779eee15e\"", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "permissions": [ + { + "grantedToIdentities": [ + { + "user": { + "displayName": "Admin", + "id": "some-admin-user-id-0000-000000000000" + } + } + ], + "roles": [ + "manager" + ] + } + ], + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + }, + "webUrl": "https://localhost:9200/f/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" +} +---- +-- +==== + +=== Restore a Space + +`PATCH /drives/\{drive-id}` + +This operation will make the space content available again to all members. No content will be changed. + +To restore a space, the Header `Restore: T` needs to be set. + +[tabs] +==== +Request:: ++ +-- +[source,bash] +---- +curl -L -X PATCH 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff/' \ + -H 'Restore: T' \ + -H 'Content-Type: text/plain' \ + --data-raw '{}' +---- + +NOTE: This request needs an empty body (--data-raw '{}') to fulfil the standard libregraph specification even when the body is not needed. +-- + +Response - 200 OK:: ++ +-- +[source,json] +---- +{ + "description": "Marketing team resources", + "driveAlias": "project/marketing", + "driveType": "project", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "lastModifiedDateTime": "2023-01-19T14:17:36.094283+01:00", + "name": "Marketing", + "owner": { + "user": { + "displayName": "", + "id": "535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + } + }, + "quota": { + "remaining": 15, + "state": "normal", + "total": 15, + "used": 0 + }, + "root": { + "eTag": "\"f5fee4fdfeedd6f98956500779eee15e\"", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "permissions": [ + { + "grantedToIdentities": [ + { + "user": { + "displayName": "Admin", + "id": "some-admin-user-id-0000-000000000000" + } + } + ], + "roles": [ + "manager" + ] + } + ], + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + }, + "webUrl": "https://localhost:9200/f/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" +} +---- +-- +==== + +=== Permanently Delete a Space + +`DELETE /drives/\{drive-id}` + +This operation will delete a space and all its data permanently. This is restricted to spaces which are already disabled. + +To delete a space, the Header `Purge: T` needs to be set. + +[tabs] +==== +Request:: ++ +-- +[source,bash] +---- +curl -L -X DELETE 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff' \ + -H 'Purge: T' +---- + +WARNING: This request will delete a space and all its content permanently. This operation cannot be reverted. +-- + +Response - 204 No Content:: ++ +-- +This response has no body value. +-- + +Response - 400 Bad Request:: ++ +-- +The space to be deleted was not disabled before. + +[source,json] +---- +{ + "error": { + "code": "invalidRequest", + "innererror": { + "date": "2023-01-24T19:57:19Z", + "request-id": "f62af40f-bc18-475e-acd7-e9008d6bd326" + }, + "message": "error: bad request: can't purge enabled space" + } +} +---- +-- +==== + +== Sharing Space + +=== Add Member to Space + +`POST /drives/\{drive-id}/root/invite` + +link:swagger-ui.html#/drives.permissions/Invite[LibreGraph: drives.permissions/Invite,window=_blank] + +=== Sharing Space as a Link + +`POST /drives/\{drive-id}/root/createLink` + +link:swagger-ui.html#/drives.root/CreateLinkSpaceRoot[LibreGraph: drives.root/CreateLinkSpaceRoot,window=_blank] + +== Reading Space Permissions + +=== Listing Permissions of a Space + +`GET /drives/\{drive-id}/root/permissions` + +link:swagger-ui.html#/drives.root/ListPermissionsSpaceRoot[LibreGraph: drives.root/ListPermissionsSpaceRoot,window=_blank] + +== Modifying / Deleting Space Permissions + +=== Update Permissions of a Drive + +`PATCH /drives/\{drive-id}/root/permissions/\{perm-id}` + +link:swagger-ui.html#/drives.root/UpdatePermissionSpaceRoot[LibreGraph: drives.root/UpdatePermissionSpaceRoot,window=_blank] + +=== Set Password of a Link Share + +`POST /drives/\{drive-id}/root/permissions/\{perm-id}/setPassword` + +link:swagger-ui.html#/drives.root/SetPermissionPasswordSpaceRoot[LibreGraph: drives.root/SetPermissionPasswordSpaceRoot,window=_blank] + +=== Removing Access to a Space + +`DELETE /drives/\{drive-id}/root/permissions/\{perm-id}` + +link:swagger-ui.html#/drives.root/DeletePermissionSpaceRoot[LibreGraph: drives.root/DeletePermissionSpaceRoot,window=_blank] diff --git a/content/ocis/9.0/modules/developer/pages/apis/http/graph/swagger-ui.adoc b/content/ocis/9.0/modules/developer/pages/apis/http/graph/swagger-ui.adoc new file mode 100644 index 00000000..149aec6f --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/apis/http/graph/swagger-ui.adoc @@ -0,0 +1,7 @@ += Libre Graph API +:toc: right +:description: Interactive API reference for the Libre Graph API used by oCIS. + +The Libre Graph API is the ownCloud-developed REST API for oCIS, aligned with the Microsoft Graph API specification. + +swagger-ui::attachment$v1.0.yaml[] diff --git a/content/ocis/9.0/modules/developer/pages/apis/http/graph/users.adoc b/content/ocis/9.0/modules/developer/pages/apis/http/graph/users.adoc new file mode 100644 index 00000000..be28e4f8 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/apis/http/graph/users.adoc @@ -0,0 +1,300 @@ += Users +:toc: right +:toclevels: 3 + + +== Users API + +The Users API is implementing a subset of the functionality of the link:https://learn.microsoft.com/en-us/graph/api/resources/user?view=graph-rest-1.0[MS Graph User resource] The JSON representation of a User handled by the Users API looks like this: + +[source,json] +---- +{ + "displayName": "Albert Einstein", + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "mail": "einstein@example.org", + "onPremisesSamAccountName": "einstein" +} +---- + +Our implementation currently supports only a limited set of Attributes of Users: + +[width="100%", cols="35%,75%", options="header"] +|=== +| Attribute +| Description + +| displayName +| The full name of the user, usually a combination of given name and last name + +| mail +| The user's email address + +| onPremisesSamAccountName +| The loginname/account name of the user + +| idt +| An unique, stable readonly identifier for the user that stays the same for the whole lifetime of the User, usually a UUID + +| passwordProfile +| Contains the password of the users. This is only present when updating or creating users. It is never returned by the API +|=== + +=== Reading Users + +==== `GET /me` + +Returns the user object of the currently signed-in user + +Example:: ++ +[source,bash] +---- +curl -k 'https://localhost:9200/graph/v1.0/me' \ + -u user:password +---- + +Response:: ++ +[source,json] +---- +{ + "displayName": "Albert Einstein", + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "mail": "einstein@example.org", + "onPremisesSamAccountName": "einstein" +} +---- + +==== `GET /users` + +Returns a list of all users + +Example:: ++ +[source,bash] +---- +curl -k 'https://localhost:9200/graph/v1.0/users' \ + -u user:password +---- + +Response:: ++ +[source,json] +---- +{ + "value": [ + { + "displayName": "Albert Einstein", + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "mail": "einstein@example.org", + "onPremisesSamAccountName": "einstein" + }, + { + "displayName": "Maurice Moss", + "id": "058bff95-6708-4fe5-91e4-9ea3d377588b", + "mail": "moss@example.org", + "onPremisesSamAccountName": "moss" + } + ] +} +---- + +==== `GET /users?$expand=memberOf` + +Returns a list of all users + +Example:: ++ +[source,bash] +---- +curl -k 'https://localhost:9200/graph/v1.0/users?$expand=memberOf' \ + -u user:password + +---- + +Response:: ++ +[source,json] +---- +{ + "value": [ + { + "displayName": "Albert Einstein", + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "mail": "einstein@example.org", + "onPremisesSamAccountName": "einstein", + "memberOf": [ + { + "displayName": "users", + "id": "509a9dcd-bb37-4f4f-a01a-19dca27d9cfa" + }, + { + "displayName": "sailing-lovers", + "id": "6040aa17-9c64-4fef-9bd0-77234d71bad0" + }, + { + "displayName": "violin-haters", + "id": "dd58e5ec-842e-498b-8800-61f2ec6f911f" + }, + { + "displayName": "physics-lovers", + "id": "262982c1-2362-4afa-bfdf-8cbfef64a06e" + } + ], + }, + { + "displayName": "Maurice Moss", + "id": "058bff95-6708-4fe5-91e4-9ea3d377588b", + "mail": "moss@example.org", + "onPremisesSamAccountName": "moss", + "memberOf": [ + { + "displayName": "users", + "id": "509a9dcd-bb37-4f4f-a01a-19dca27d9cfa" + } + ], + } + ] +} +---- + +==== `GET /users/{userid or accountname}` + +Example:: ++ +[source,bash] +---- +curl -k 'https://localhost:9200/graph/v1.0/users/058bff95-6708-4fe5-91e4-9ea3d377588b' \ + -u user:password +---- + +Response:: ++ +[source,json] +---- +{ + "displayName": "Maurice Moss", + "id": "058bff95-6708-4fe5-91e4-9ea3d377588b", + "mail": "moss@example.org", + "onPremisesSamAccountName": "moss" +} +---- + +==== `GET /users/{userid or accountname}?$expand=memberOf` + +Example:: ++ +[source,bash] +---- +curl -k 'https://localhost:9200/graph/v1.0/users/058bff95-6708-4fe5-91e4-9ea3d377588b?$expand=memberOf' \ + -u user:password +---- + +Response:: ++ +[source,json] +---- +{ + "displayName": "Maurice Moss", + "id": "058bff95-6708-4fe5-91e4-9ea3d377588b", + "mail": "moss@example.org", + "onPremisesSamAccountName": "moss", + "memberOf": [ + { + "displayName": "users", + "id": "509a9dcd-bb37-4f4f-a01a-19dca27d9cfa" + } + ], +} +---- + +=== Creating / Updating Users + +==== `POST /users` + +Use this to create a new user. + +===== Request Body + +Note the missing `"id"` Attribute. It will be generated by the server: + +[source,json] +---- +{ + "displayName": "Example User", + "mail": "example@example.org", + "onPremisesSamAccountName": "example", + "passwordProfile": { + "password": "ThePassword" + } +} +---- + +===== Response + +When successful, the response will return the new user, without the password, but including the newly allocated `"id"`: + +[source,json] +---- +{ + "displayName":"Example User", + "id":"c067b139-c91c-4e47-8be6-669156a0587b", + "mail":"example@example.org", + "onPremisesSamAccountName":"example" +} +---- + +==== `DELETE /users/\{id}` + +Example:: ++ +[source,bash] +---- +curl -k --request DELETE 'https://localhost:9200/graph/v1.0/users/c067b139-c91c-4e47-8be6-669156a0587b' \ + -u user:password +---- + +When successful the API returns no response body and the HTTP status code 204 (No Content) + + +==== `PATCH /users/\{id}` + +Updating attributes of a single user can be done with a patch request. The Request Body contains the new values of the attributes +to be updated. E.g. to update the `displayName` Attribute: + +[source,bash] +---- + curl -k --header "Content-Type: application/json" \ + --request PATCH --data '{"displayName": "Test User" }' \ + 'https://localhost:9200/graph/v1.0/users/c54b0588-7157-4521-bb52-c1c8ca84ea71' \ + -u user:password +---- + +Similar to creating a user via `POST`, the `PATCH` request will return the user object containing the new attribute values. + +=== Change Password + +==== `POST /me/changePassword` + +Users can change their own password by sending a POST request to `/me/changePassword` + +===== Request Body + +[source,json] +---- +{ + "currentPassword": "current", + "newPassword": "new" +} +---- + +When successful the API returns no response body and the HTTP status code 204 (No Content) + +[source,bash] +---- + curl -i -k --header "Content-Type: application/json" \ + --request POST --data '{"currentPassword": "current", "newPassword": "new" }' \ + 'https://localhost:9200/graph/v1.0/me/changePassword' \ + -u user:current +---- diff --git a/content/ocis/9.0/modules/developer/pages/apis/http/index.adoc b/content/ocis/9.0/modules/developer/pages/apis/http/index.adoc new file mode 100644 index 00000000..d408d59b --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/apis/http/index.adoc @@ -0,0 +1,7 @@ += HTTP +:toc: right +:toclevels: 3 + +The link:https://www.rfc-editor.org/info/rfc2616/[Hypertext Transfer Protocol (HTTP)] is a stateless application-level protocol for distributed, collaborative, hypertext information systems. HTTP is the foundation of data communication for the World Wide Web, where hypertext documents include hyperlinks to other resources that the user can easily access, for example by a mouse click or by tapping the screen in a web browser. + +Development of HTTP was initiated by Tim Berners-Lee at CERN in 1989 and summarized in a simple document describing the behavior of a client and a server using the first HTTP protocol version that was named 0.9. That first version of HTTP protocol soon evolved into a more elaborated version that was the first draft toward a far future version 1.0 diff --git a/content/ocis/9.0/modules/developer/pages/apis/http/tus_upload.adoc b/content/ocis/9.0/modules/developer/pages/apis/http/tus_upload.adoc new file mode 100644 index 00000000..c5e5d07d --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/apis/http/tus_upload.adoc @@ -0,0 +1,340 @@ += Resumable Upload +:toc: right +:toclevels: 3 + +Infinite Scale supports the tus resumable-upload protocol, which is a robust, modular, and open protocol designed to resume large file uploads reliably over HTTP. In situations where file uploads might be interrupted due to network issues, browser crashes, or other unforeseen interruptions, tus ensures that uploads can be resumed from the point of failure without losing data. This documentation shows some basic examples, refer link:https://tus.io/protocols/resumable-upload[tus official site] for more details. + +== Supported tus Features + +The backend announces certain tus features to clients. WebDAV responses come with tus HTTP headers for the official tus features, and additional, ownCloud specific features are announced via the capabilities endpoint (e.g. `\https://localhost:9200/ocs/v1.php/cloud/capabilities?format=json`). + +The following snippet shows the relevant part of the server capabilities of Infinite Scale that concerns the tus upload: + +[source,json] +---- +{ + "ocs": { + "data": { + "capabilities": { + "files": { + "tus_support": { + "version": "1.0.0", + "resumable": "1.0.0", + "extension": "creation,creation-with-upload", + "max_chunk_size": 10000000, + "http_method_override": "" + } + } + } + } + } + } +} +---- + +[width="100%", cols="25%,25%,25%,25%", options="header"] +|=== +| Parameter +| Environment Variable +| Default Value +| Description + +| max_chunk_size +| FRONTEND_UPLOAD_MAX_CHUNK_SIZE +| 10000000 +| Announces the max chunk sizes in bytes for uploads via the clients. +|=== + +== Upload in Chunks + +=== Create an Upload URL + +The client must send a POST request against a known upload creation URL to request a new upload resource. +The filename has to be provided in base64-encoded format. + +Example: + +[source,bash] +---- +# base64 encoded filename 'tustest.txt' is 'dHVzdGVzdC50eHQ=' +echo -n 'tustest.txt' | base64 +---- + +[tabs] +==== +Request:: ++ +-- +[source,bash] +---- +curl -ks -XPOST https://ocis.test/remote.php/dav/spaces/8d72036d-14a5-490f-889e-414064156402$196ac304-7b88-44ce-a4db-c4becef0d2e0 \ + -H "Authorization: Bearer eyJhbGciOiJQUzI..."\ + -H "Tus-Resumable: 1.0.0" \ + -H "Upload-Length: 10" \ + -H "Upload-Metadata: filename dHVzdGVzdC50eHQ=" +---- +-- + +Response - 201 Created:: ++ +-- +[source,plaintext] +---- +< HTTP/1.1 201 Created +< Access-Control-Allow-Headers: Tus-Resumable, Upload-Length, Upload-Metadata, If-Match +< Access-Control-Allow-Origin: * +< Access-Control-Expose-Headers: Tus-Resumable, Upload-Offset, Location +< Content-Length: 0 +< Content-Security-Policy: default-src 'none'; +< Date: Mon, 16 Oct 2023 08:49:39 GMT +< Location: https://ocis.test/data/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNjk3NTMyNTc5LCJpYXQiOjE2OTc0NDYxNzksInRhcmdldCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE1OC9kYXRhL3R1cy8zYTU3ZWZlMS04MzE0LTQ4MGEtOWY5Ny04N2Q1YzBjYTJhMTgifQ.FbrlY7mdOfsbFgMrP8OtcHlCEq72a2ZVnPD2iBo9MfM +< Tus-Extension: creation,creation-with-upload,checksum,expiration +< Tus-Resumable: 1.0.0 +< Vary: Origin +< X-Content-Type-Options: nosniff +< X-Download-Options: noopen +< X-Frame-Options: SAMEORIGIN +< X-Permitted-Cross-Domain-Policies: none +< X-Request-Id: xxxxxxxxxxxxxxxxxxxxxx +< X-Robots-Tag: none +< +* Connection #0 to host localhost left intact +---- +-- +==== + +The server will return a temporary upload URL in the location header of the response: + +[source,plaintext] +---- +< Location: https://ocis.test/data/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNjk3NTMyNTc5LCJpYXQiOjE2OTc0NDYxNzksInRhcmdldCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE1OC9kYXRhL3R1cy8zYTU3ZWZlMS04MzE0LTQ4MGEtOWY5Ny04N2Q1YzBjYTJhMTgifQ.FbrlY7mdOfsbFgMrP8OtcHlCEq72a2ZVnPD2iBo9MfM +---- + +=== Upload the First Chunk + +Once a temporary upload URL has been created, a client can send a PATCH request to upload a file. The file content should be sent in the body of the request: + +[tabs] +==== +Request:: ++ +-- +[source,bash] +---- +curl -ks -XPATCH https://temporary-upload-url \ + -H "Authorization: Bearer eyJhbGciOiJQUzI..." \ + -H "Tus-Resumable: 1.0.0" \ + -H "Upload-Offset: 0" \ + -H "Content-Type: application/offset+octet-stream" -d "01234" +---- +-- + +Response - 204 No Content:: ++ +-- +[source,plaintext] +---- +< HTTP/1.1 204 No Content +< Date: Tue, 17 Oct 2023 04:10:52 GMT +< Oc-Fileid: 8d72036d-14a5-490f-889e-414064156402$73bb5450-816b-4cae-90aa-1f96adc95bd4!84e319e4-de1d-4dd8-bbd0-e51d933cdbcd +< Tus-Resumable: 1.0.0 +< Upload-Expires: 1697602157 +< Upload-Offset: 5 +< Vary: Origin +< X-Content-Type-Options: nosniff +< X-Request-Id: xxxxxxxxxxxxxxxxxxxxxx +< +* Connection #0 to host localhost left intact +---- +-- +==== + +=== Upload Further Chunks + +After the first chunk is uploaded, the second chunk can be uploaded by pointing `Upload-Offset` to exact position that was returned in the first response. +Upload process will not be marked as complete until the total uploaded content size matches the `Upload-Length` specified during the creation of the temporary URL. + +[tabs] +==== +Request:: ++ +-- +[source,bash] +---- +curl -ks -XPATCH https://temporary-upload-url \ + -H "Authorization: Bearer eyJhbGciOiJQUzI..." \ + -H "Tus-Resumable: 1.0.0" \ + -H "Upload-Offset: 5" \ + -H "Content-Type: application/offset+octet-stream" -d "56789" +---- +-- + +Response - 204 No Content:: ++ +-- +[source,plaintext] +---- +< HTTP/1.1 204 No Content +< Date: Tue, 17 Oct 2023 04:11:00 GMT +< Oc-Fileid: 8d72036d-14a5-490f-889e-414064156402$73bb5450-816b-4cae-90aa-1f96adc95bd4!84e319e4-de1d-4dd8-bbd0-e51d933cdbcd +< Tus-Resumable: 1.0.0 +< Upload-Expires: 1697602157 +< Upload-Offset: 10 +< Vary: Origin +< X-Content-Type-Options: nosniff +< X-Request-Id: xxxxxxxxxxxxxxxxxxxxxx +< +* Connection #0 to host localhost left intact +---- +-- +==== + +CAUTION: `Upload-Offset` header indicates the byte position in the target file where the server should start writing the upload content. It ensures data integrity and order during the upload process. + +== Creation with Upload + +[tabs] +==== +Request:: ++ +-- +[source,bash] +---- +curl -ks -XPOST https://ocis.test/remote.php/dav/spaces/{space-id} \ + -H "Authorization: Bearer eyJhbGciOiJQUzI..." \ + -H "Tus-Resumable: 1.0.0" \ + -H "Upload-Length: 14" \ + -H "Content-Type: application/offset+octet-stream" \ + -H "Upload-Metadata: filename dGVzdC50eHQ=" \ + -H "Tus-Extension: creation-with-upload" \ + -d "upload content" +---- +-- + +Response - 201 Created:: ++ +-- +[source,plaintext] +---- +< HTTP/1.1 201 Created +< Access-Control-Allow-Headers: Tus-Resumable, Upload-Length, Upload-Metadata, If-Match +< Access-Control-Allow-Origin: * +< Access-Control-Expose-Headers: Tus-Resumable, Upload-Offset, Location +< Content-Length: 0 +< Content-Security-Policy: default-src 'none'; +< Content-Type: text/plain +< Date: Mon, 16 Oct 2023 04:18:25 GMT +< Etag: "372c96743f68bc40e789124d30567371" +< Last-Modified: Mon, 16 Oct 2023 04:18:25 +0000 +< Location: https://ocis.test/data/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNjk3NTE2MzA1LCJpYXQiOjE2OTc0Mjk5MDUsInRhcmdldCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE1OC9kYXRhL3R1cy82NjlhODBlZi1hN2VjLTQwYTAtOGNmOS05MTgwNTVhYzlkZjAifQ.yq-ofJYnJ9FLML7Z_jki1FJQ7Ulbt9O_cmLe6V411A4 +< Oc-Etag: "372c96743f68bc40e789124d30567371" +< Oc-Fileid: 44d3e1e0-6c01-4b94-9145-9d0068239fcd$446bdad4-4b27-41f1-afce-0881f202a214!d7c292a6-c395-4e92-bf07-2c1663aec8dd +< Oc-Perm: RDNVWZP +< Tus-Extension: creation,creation-with-upload,checksum,expiration +< Tus-Resumable: 1.0.0 +< Upload-Expires: 1697516305 +< Upload-Offset: 14 +< Vary: Origin +< X-Content-Type-Options: nosniff +< X-Download-Options: noopen +< X-Frame-Options: SAMEORIGIN +* TLSv1.2 (IN), TLS header, Supplemental data (23): +{ [5 bytes data] +< X-Permitted-Cross-Domain-Policies: none +< X-Request-Id: xxxxxxxxxxxxxxxxxxxxxx +< X-Robots-Tag: none +< +* Connection #0 to host localhost left intact +---- +-- +==== + +CAUTION: The `Upload-Length` header of the request has to contain the exact size of the upload content in byte. + +== Supported Upload-Metadata + +Upload-metadata key-value pairs aren't specified in the general tus docs. The following ones are supported in the ownCloud ecosystem: + +[width="100%", cols="20%,45%,100%", options="header"] +|=== +| Parameter (key) +| Example (value, MUST be Base64 encoded) +| Description + +| `name` OR `filename` (mandatory) +| `example.pdf` +| Filename + +| `mtime` (recommended) +| `1701708712` +| Modification time (Unix time format) + +| `checksum` (recommended) +a| `SHA1` + +`a330de5886e5a92d78fb3f8d59fe469857759e72` +| Checksum, computed from the client + +| `type` OR `filetype` +| `application/pdf` +| MIME Type, sent by the web UI + +| `relativePath` +| `undefined` +| File path relative to the folder that is being uploaded, including the filename. Sent by the web UI + +| `spaceId` +| `8748cddf-66b7-4b85-91a7-e6d08d8e1639$a9778d63-21e7-4d92-9b47-1b81144b9993` +| Sent by the web UI + +| `spaceName` +| `Personal` +| Sent by the web UI + +| `driveAlias` +| `personal/admin` +| Sent by the web UI + +| `driveType` +| `personal` +| Sent by the web UI + +| `currentFolder` +| `/` +| Sent by the web UI + +| `currentFolderId` +| `8748cddf-66b7-4b85-91a7-e6d08d8e1639$a9778d63-21e7-4d92-9b47-1b81144b9993!a9778d63-21e7-4d92-9b47-1b81144b9993` +| Sent by the web UI + +| `uppyId` +| `uppy-example/pdf-1e-application/pdf-238300` +| Sent by the web UI + +| `relativeFolder` +| +| File path relative to the folder that is being uploaded, without filename. Sent by the web UI. + +| `tusEndpoint` +| `+https://ocis.ocis-traefik.latest.owncloud.works/remote.php/dav/spaces/8748cddf-66b7-4b85-91a7-e6d08d8e1639$a9778d63-21e7-4d92-9b47-1b81144b9993+` +| Sent by the web UI + +| `uploadId` +| `71d5f878-a96c-4d7b-9627-658d782c93d7` +| Sent by the web UI + +| `topLevelFolderId` +| `undefined` +| Sent by the web UI + +| `routeName` +| `files-spaces-generic` +| Sent by the web UI + +| `routeDriveAliasAndItem` +| `cGVyc29uYWwvYWRtaW4=` +| Sent by the web UI + +| `routeShareId` +| +| Share ID when uploading into a received folder share. Sent by the web UI +|=== diff --git a/content/ocis/9.0/modules/developer/pages/apis/http/webdav/index.adoc b/content/ocis/9.0/modules/developer/pages/apis/http/webdav/index.adoc new file mode 100644 index 00000000..4ce11033 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/apis/http/webdav/index.adoc @@ -0,0 +1,771 @@ += WebDAV +:toc: right +:toclevels: 2 + +**Web D**istributed **A**uthoring and **V**ersioning (WebDAV) consists of a set of methods, headers, and content-types extending HTTP/1.1 for the management of resources and -properties, creation and management of resource collections, URL namespace manipulation, and resource locking (collision avoidance). WebDAV is one of the central APIs that ownCloud uses for handling file resources, metadata and locks. + + +[NOTE] +==== +*WebDAV RFCs* + +RFC 2518 was published in February 1999. link:https://datatracker.ietf.org/doc/html/rfc4918[RFC 4918], published in June 2008 obsoletes RFC 2518 with minor revisions mostly due to interoperability experience. +==== + +== Calling the WebDAV API + +=== Request URI + +[source,bash] +---- +{HTTP method} https://ocis.url/{webdav-base}/{resourceID}/{path} +---- + +The request URI consists of: + +[width="100%", cols="25%,75%", options="header"] +|=== +| Component +| Description + +| {HTTP method} +| The HTTP method which is used in the request. + +| \{webdav-base} +| The WebDAV base path component. Possible options are + +| +| `dav/spaces/` This is the default and optimized endpoint for all WebDAV requests. + +| +| `remote.php/dav/spaces/` ^(1)^ + +| +| `remote.php/webdav/` ^(1)^ + +| +| `webdav/` ^(1)^ + +| +| `dav/` ^(1)^ + +| \{resourceID} +| This resourceID is used as the WebDAV root element. All children are accessed by their relative paths. + +| \{path} +| The relative path to the WebDAV root. In most of the cases, this is the space root. +|=== + +(1) ... these dav endpoints are implemented for legacy reasons and should not be used. Note: The legacy endpoints _do not take the resourceID as an argument._ + +=== HTTP Methods + +[width="100%", cols="20%,75%", options="header"] +|=== +| Method +| Description + +| PROPFIND +| Retrieve properties as XML from a web resource. It is also overloaded to retrieve the collection structure (a.k.a. directory hierarchy) of a remote system. + +| PROPPATCH +| Process instructions specified in the request body to set and/or remove properties defined on the resource identified by the request uri. + +| MKCOL +| Create a WebDAV collection (folder) at the location specified by the request uri. + +| GET +| Retrieve a WebDAV resource. + +| HEAD +| Retrieve a WebDAV resource without reading the body. + +| PUT +| A PUT performed on an existing resource replaces the GET response entity of the resource. + +| POST +| Not part of the WebDAV rfc and has no effect on a WebDAV resource. However, this method is used in the TUS protocol for uploading resources. + +| PATCH +| Not part of the WebDAV rfc and has no effect on a WebDAV resource. However, this method is used in the TUS protocol for uploading resources. + +| COPY +| Creates a duplicate of the source resource identified by the Request-URI, in the destination resource identified by the URI in the Destination header. + +| MOVE +| The MOVE operation on a non-collection resource is the logical equivalent of a copy (COPY), followed by consistency maintenance processing, followed by a delete of the source, where all three actions are performed in a single operation. + +| DELETE +| Delete the resource identified by the Request-URI. + +| LOCK +| A LOCK request to an existing resource will create a lock on the resource identified by the Request-URI, provided the resource is not already locked with a conflicting lock. + +| UNLOCK +| The UNLOCK method removes the lock identified by the lock token in the Lock-Token request header. The Request-URI must identify a resource within the scope of the lock. +|=== + +The methods `MKCOL`, `GET`, `HEAD`, `LOCK`, `COPY`, `MOVE`, `UNLOCK` and `DELETE` need no request body. + +The methods `PROPFIND`, `PROPPATCH`, `PUT` require a request body, normally in XML format to provide the needed values. + +[NOTE] +==== +_WebDAV is not REST_ + +The WebDAV protocol was created before the REST paradigm has become the de-facto standard for API design. WebDAV uses http methods which are not part of REST. Therefore all the tooling around API design and documentation is not usable (like OpenApi 3.0 / Swagger or others). +==== + +=== Authentication + +For development purposes the examples in the developer documentation use Basic Auth. It is disabled by default and should only be enabled by setting the `PROXY_ENABLE_BASIC_AUTH` in xref:admin:{s-path}/proxy.adoc[proxy] for development or test instances. + +To authenticate with a Bearer token or OpenID Connect access token replace the `-u user:password` Basic Auth option of curl with a `-H 'Authorization: Bearer '` header. A `` can be obtained by copying it from a request in the browser, although it will time out within minutes. To automatically refresh the OpenID Connect access token an ssh-agent like solution like link:https://github.com/indigo-dc/oidc-agent[oidc-agent] should be used. + +== Listing Properties + +This method is used to list the properties of a resource in xml. This method can also be used to retrieve the listing of a WebDAV collection which means the content of a remote directory. + +[tabs] +==== +Curl:: ++ +-- +[source,bash] +---- +curl -L -X PROPFIND 'https://localhost:9200/dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/' \ +-H 'Depth: 1' \ +-d ' + + + + + + + + + + + + + + + + + +' +---- +-- + +HTTP:: ++ +-- +[source,bash] +---- +PROPFIND /dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/ HTTP/1.1 +Host: localhost:9200 +Origin: https://localhost +Access-Control-Request-Method: PROPFIND +Depth: 1 +Content-Type: application/xml +Authorization: Basic YWRtaW46YWRtaW4= +Content-Length: 436 + + + + + + + + + + + + + + + + + + + + +---- +-- +==== + +The request consists of a request body and an optional `Depth` Header. + +[NOTE] +==== +_Metadata and Directory listings_ + +Clients can use the `PROPFIND` method to retrieve properties of resources (metadata) and to list the content of a directories. +==== + +=== Response + +[tabs] +==== +207 - Multistatus:: ++ +-- +Multi Status Response:: ++ +A Multi-Status response conveys information about multiple resources in situations where multiple status codes might be appropriate. The default Multi-Status response body is an application/xml HTTP entity with a `multistatus` root element. Further elements contain `200`, `300`, `400`, and `500` series status codes generated during the method invocation. ++ +Although `207` is used as the overall response status code, the recipient needs to consult the contents of the multistatus response body for further information about the success or failure of the method execution. The response MAY be used in success, partial success and also in failure situations. The `multistatus` root element holds zero or more `response` elements in any order, each with information about an individual resource. + +Body:: ++ +[source,xml] +---- + + + /dav/spaces/storage-users-1$some-admin-user-id-0000-000000000000/ + + + RDNVCKZP + 0 + storage-users-1$some-admin-user-id-0000-000000000000!some-admin-user-id-0000-000000000000 + storage-users-1$some-admin-user-id-0000-000000000000!some-admin-user-id-0000-000000000000 + admin + Admin + https://localhost:9200/f/storage-users-1$some-admin-user-id-0000-000000000000%21some-admin-user-id-0000-000000000000 + 10364682 + Mon, 04 Sep 2023 20:10:09 GMT + "c4d3610dfe4fac9b44e1175cfc44b12b" + + + + + HTTP/1.1 200 OK + + + + + + + + + HTTP/1.1 404 Not Found + + + + /dav/spaces/storage-users-1$some-admin-user-id-0000-000000000000/New%20file.txt + + + RDNVWZP + + SHA1:1c68ea370b40c06fcaf7f26c8b1dba9d9caf5dea MD5:2205e48de5f93c784733ffcca841d2b5 ADLER32:058801ab + + 0 + storage-users-1$some-admin-user-id-0000-000000000000!90cc3e73-0c6c-4346-9c4d-f529976d4990 + storage-users-1$some-admin-user-id-0000-000000000000!90cc3e73-0c6c-4346-9c4d-f529976d4990 + admin + Admin + + 0 + 1 + 3 + + https://localhost:9200/f/storage-users-1$some-admin-user-id-0000-000000000000%2190cc3e73-0c6c-4346-9c4d-f529976d4990 + 5 + 5 + Mon, 28 Aug 2023 20:45:03 GMT + "75115347c74701a3be9c635ddebbf5c4" + text/plain + + + HTTP/1.1 200 OK + + + + /dav/spaces/storage-users-1$some-admin-user-id-0000-000000000000/NewFolder/ + + + RDNVCKZP + 0 + storage-users-1$some-admin-user-id-0000-000000000000!5c73ecd9-d9f4-44f4-b685-ca4cb40aa6b7 + storage-users-1$some-admin-user-id-0000-000000000000!5c73ecd9-d9f4-44f4-b685-ca4cb40aa6b7 + admin + Admin + https://localhost:9200/f/storage-users-1$some-admin-user-id-0000-000000000000%215c73ecd9-d9f4-44f4-b685-ca4cb40aa6b7 + 0 + Mon, 28 Aug 2023 20:45:10 GMT + "e83367534cc595a45d706857fa5f03d8" + + + + + HTTP/1.1 200 OK + + + + + + + + + HTTP/1.1 404 Not Found + + + +---- +-- + +400 - Bad Request:: ++ +-- +Body:: ++ +[source,xml] +---- + + + Sabre\DAV\Exception\BadRequest + Invalid Depth header value: 3 + +---- + +This can occur if the request is malformed e.g. due to an invalid xml request body or an invalid depth header value. +-- + +404 - Not Found:: ++ +-- +Body:: ++ +[source,xml] +---- + + + Sabre\DAV\Exception\NotFound + Resource not found + +---- +-- +==== + +=== Request Body + +The `PROPFIND` Request can include an XML request body containing a list of namespaced property names. + +=== Namespaces + +When building the body of your DAV request, you will request properties that are available under a specific namespace URI. It is usual to declare prefixes for those namespace in the `d:propfind` element of the body. + +Available namespaces: + +[width="100%", cols="25%,75%", options="header"] +|=== +| URI +| Prefix + +| DAV: +| d + +| http://sabredav.org/ns +| s + +| http://owncloud.org/ns +| oc + +| http://open-collaboration-services.org/ns +| ocs + +| http://open-cloud-mesh.org/ns +| ocm +|=== + +=== Request Example With Declared Namespaces + +[source,xml] +---- + + + +---- + +=== Supported WebDAV Properties + +[width="100%", cols="35%,30%,35%", options="header"] +|=== +| Property +| Description +| Example + +| `` +| The latest modification time. +| `Fri, 30 Dec 2022 14:22:43 GMT` + +| `` +| The file's etag. +| `"c3a1ee4a0c28edc15b9635c3bf798013"` + +| `` +| The mime type of the file. +| `image/jpeg` + +| `` +| Specifies the nature of the resource. +| `` for a folder + +| `` +| The size if it is a file in bytes. +| `5` bytes + +| `` +| Describes the active locks on a resource. +| Detailed example in Locking. + +| `` +| The globally unique ID of the resource. +| `storage-1$27475553-7fb7-4689-b4cf-bbb635daff79!27475553-7fb7-4689-b4cf-bbb635daff79` + +| `` +| The globally unique ID of the resource. +| `storage-1$27475553-7fb7-4689-b4cf-bbb635daff79!27475553-7fb7-4689-b4cf-bbb635daff79` + +| `` +| Direct URL to download a file from. +| Not implemented. + +| `` +| Determines the actions a user can take on the resource. +a| The value is a string containing letters that clients can use to determine available actions: + +`S`: Shared + +`M`: Mounted + +`D`: Deletable + +`NV`: Updateable, Renameable, Moveable + +`W`: Updateable (file) + +`CK`: Creatable (folders only) + +`Z`: Deniable + +`P`: Trashbin Purgable + +`X`: Securely Viewable + +In the early stages this was indeed a list of permissions. Over time, more flags were added and the term permissions no longer really fits well. + +| `` +| List of user specified tags. +| `test` + +| `` +| The favorite state. +| `0` for not favourited, `1` for favourited + +| `` +| The user id of the owner of a resource. Project spaces have no owner. +| `einstein` + +| `` +| The display name of the owner of a resource. Project spaces have no owner. +| `Albert Einstein` + +| `` +| List of share types. +a| `0` = User Share + +`1` = Group Share + +`2` = Public Link + +| `` +| +a| `` + +`SHA1:1c68ea370b40c06fcaf7f26c8b1dba9d9caf5dea MD5:2205e48de5f93c784733ffcca841d2b5 ADLER32:058801ab` + +`` + +Due to a bug in the very early development of ownCloud, this value is not an array, but a string separated by whitespaces. + +| `` +| Similar to `getcontentlength` but it also works for folders. +| `10` bytes + +| `` +| The ID of the share if the resource is part of such. +| `storage-1$27475553-7fb7-4689-b4cf-bbb635daff79!27475553-7fb7-4689-b4cf-bbb635daff79` + +| `` +| The root path of the shared resource if the resource is part of such. +| `/shared-folder` + +| `` +| The ID of the shared resource if the resource is part of such. +| `storage-1$27475553-7fb7-4689-b4cf-bbb635daff79!27475553-7fb7-4689-b4cf-bbb635daff79` + +| `` +| The type of the resource if it's a public link. +| `folder` + +| `` +| The share permissions of the resource if it's a public link. +| `1` + +| `` +| The expiration date of the public link. +| `Tue, 14 May 2024 12:44:29 GMT` + +| `` +| The date the public link was created. +| `Tue, 14 May 2024 12:44:29 GMT` + +| `` +| The username of the user who created the public link. +| `admin` + +| `` +| The original name of the resource before it was deleted. +| `some-file.txt` + +| `` +| The original location of the resource before it was deleted. +| `some-file.txt` + +| `` +| The date the resource was deleted. +| `Tue, 14 May 2024 12:44:29 GMT` + +| `` +| Audio meta data if the resource contains such. +a| `Metallica` + +`Metallica` + +`Enter Sandman` + +| `` +| Location meta data if the resource contains such. +a| `51.504106` + +`-0.074575` +|=== + +=== Request Headers + +A client executing a `PROPFIND` request MUST submit a Depth Header value. In practice, support for infinite-depth requests MAY be disabled, due to the performance and security concerns associated with this behavior. Servers SHOULD treat a +request without a Depth header as if a `Depth: infinity` header was included. Infinite depth requests are disabled by default in ocis. + +[width="100%", cols="25%,75%", options="header"] +|=== +| Name +| Value + +| Depth +a| `0` = Only return the desired resource. + +`1` = Return the desired resource and all resources one level below in the hierarchy. + +`infinity` = Return all resources below the root. +|=== + +[NOTE] +==== +* *Depth: infinity* + +Using the `Depth: infinity` header value can cause heavy load on the server, depending on the size of the file tree. + +The request can run into a timeout and the server performance could be affected for other users. +==== + +== Create a Directory + +Clients create directories (WebDAV collections) by executing a `MKCOL` request at the location specified by the request url. + +[tabs] +==== +Curl:: ++ +-- +[source,bash] +---- +curl -L -X MKCOL 'https://localhost:9200/dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/NewFolder/' \ +-H 'Authorization: Basic YWRtaW46YWRtaW4=' +---- +-- + +HTTP:: ++ +-- +[source,bash] +---- +MKCOL /dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/NewFolder/ HTTP/1.1 +Host: localhost:9200 +Authorization: Basic YWRtaW46YWRtaW4= +---- +-- +==== + +=== Response + +[tabs] +==== +201 - Created:: ++ +-- +This indicates that the Resource has been created successfully. + +Body:: ++ +The response has no body. +-- + +403 - Forbidden:: ++ +-- +Body:: ++ +[source,xml] +---- + + + Sabre\DAV\Exception\Forbidden + + +---- +-- + +405 - Method not allowed:: ++ +-- +Body:: ++ +[source,xml] +---- + + + Sabre\DAV\Exception\MethodNotAllowed + The resource you tried to create already exists + +---- +-- +==== + +== Upload File + +To upload files to the remote server, clients can use the `PUT` method to create or fully replace the content of the remote file. + +=== Request Headers + +[width="100%", cols="25%,75%", options="header"] +|=== +| Name +| Usage + +| `X-OC-Mtime` +| Send the last modified time of the file to the server in unixtime format. The server applies this mtime to the resource rather than the actual time. + +| `OC-Checksum` +| Provide the checksum of the file content to the server. This is used to prevent corrupted data transfers. + +| `If-Match` +| The If-Match request-header field is used with a method to make it conditional. A client that has one or more entities previously obtained from the resource can verify that one of those entities is current by including a list of their associated entity tags in the `If-Match` header field. +|=== + +[tabs] +==== +Curl:: ++ +-- +[source,shell] +---- +curl -L -X PUT 'https://localhost:9200/dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/test.txt' \ +-H 'X-OC-Mtime: 1692369418' \ +-H 'OC-Checksum: SHA1:40bd001563085fc35165329ea1ff5c5ecbdbbeef' \ +-H 'If-Match: "4436aef907f41f1ac7dfd1ac3d0d455f"' \ +-H 'Content-Type: text/plain' \ +-H 'Authorization: Basic YWRtaW46YWRtaW4=' \ +-d '123' +---- +-- + +HTTP:: ++ +-- +[source,bash] +---- +PUT /dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/test.txt HTTP/1.1 +Host: localhost:9200 +X-OC-Mtime: 1692369418 +OC-Checksum: SHA1:40bd001563085fc35165329ea1ff5c5ecbdbbeef +If-Match: "4436aef907f41f1ac7dfd1ac3d0d455f" +Content-Type: text/plain +Authorization: Basic YWRtaW46YWRtaW4= +Content-Length: 3 + +123 +---- +-- +==== + +=== Response + +[tabs] +==== +201 - Created:: ++ +-- +This indicates that the Resource has been created successfully. + +Body:: ++ +The response has no body. + +Headers:: ++ +[source,yaml] +---- +Oc-Etag: "4436aef907f41f1ac7dfd1ac3d0d455f" +Oc-Fileid: storage-users-1$some-admin-user-id-0000-000000000000!07452b22-0ba9-4539-96e1-3511aff7fd2f +Last-Modified: Fri, 18 Aug 2023 14:36:58 +0000 +X-Oc-Mtime: accepted +---- +-- + +204 - No Content:: ++ +-- +This indicates that the Resource has been updated successfully. + +Body:: ++ +The response has no body. + +Headers:: ++ +[source,yaml] +---- +Oc-Etag: "4436aef907f41f1ac7dfd1ac3d0d455f" +Oc-Fileid: storage-users-1$some-admin-user-id-0000-000000000000!07452b22-0ba9-4539-96e1-3511aff7fd2f +Last-Modified: Fri, 18 Aug 2023 14:36:58 +0000 +X-Oc-Mtime: accepted +---- +-- + +400 - Bad Request:: ++ +-- +This indicates that the checksum, which was sent by the client, does not match the computed one after all bytes have been received by the server. + +Body:: ++ +[source,xml] +---- + + + Sabre\DAV\Exception\BadRequest + The computed checksum does not match the one received from the client. + +---- +-- + +403 - Forbidden:: ++ +-- +The user cannot create files in that remote location. +-- + +404 - Not Found:: ++ +-- +The remote target space cannot be found. +-- + +409 - Conflict:: ++ +-- +This error can occur when the request cannot be executed due to a missing precondition. One example is a PUT into a non-existing remote folder. It can also happen when the client sends the wrong etag in the `If-Match` header. +-- +==== diff --git a/content/ocis/9.0/modules/developer/pages/apis/index.adoc b/content/ocis/9.0/modules/developer/pages/apis/index.adoc new file mode 100644 index 00000000..03ba70b9 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/apis/index.adoc @@ -0,0 +1,42 @@ += APIs +:toc: right +:toclevels: 3 + +Infinite Scale provides a large set of different _application programming interfaces (APIs)_. Infinite Scale is built by microservices. That means many calls to "functions" in the code are remote calls. + +Basically we have two different API "universes": xref:apis/http/index.adoc[HTTP] and xref:apis/grpc_apis/index.adoc[gRPC]. + +[width="100%",cols="1,1"] +|=== +a| image::ocis/apis/http-logo.png[Image sourced from https://commons.wikimedia.org/ free of licenses,width=200] +a| image::ocis/apis/grpc-logo.png[Image sourced from https://grpc.io/ under CC 4.0 BY license,width=200] +|=== + +For inter-service-communication we are using mostly gRPC calls because it has some advantages. In the future, clients may decide to use gRPC directly to make use of these advantages. + +image::ocis/apis/ocis-apis.drawio.svg[] + +== HTTP + +xref:apis/http/index.adoc[HTTP APIs] are mostly used for client <-> server communication. Modern applications are embracing a link:https://en.wikipedia.org/wiki/Representational_state_transfer[RESTful] software architecture style. REST APIs are using the HTTP protocol to transfer data between clients and servers. All our clients talk to the Server using HTTP APIs. This has legacy reasons and is well-supported across many platforms and technologies. Infinite Scale uses an link:../services/proxy[HTTP API gateway] to route client requests to the correct service. + +=== OpenAPI + +It is best practise to define APIs and their behavior by a spec. We are using the OpenAPI standard for all new APIs. The link:https://swagger.io/specification/[OpenAPI Specification], previously known as the Swagger Specification, is a specification for a machine-readable interface definition language for describing, producing, consuming and visualizing RESTful web services. Previously part of the Swagger framework, it became a separate project in 2016, overseen by the OpenAPI Initiative, an open-source collaboration project of the Linux Foundation. Swagger and some other tools can generate code, documentation and test cases from interface files. + +=== RFC + +Some APIs have become a de facto standard and are additionally covered by an link:https://en.wikipedia.org/wiki/Request_for_Comments[RFC]. + +== gRPC + +In xref:apis/grpc_apis/index.adoc[gRPC], a client application can directly call methods on a server application on a different machine as if it was a local object. This makes it easier to create distributed applications based on microservices. In gRPC we can define a service and specify the methods that can be called remotely. A gRPC client has a stub that provides the same methods and types as the server. +Infinite Scale uses a xref:{dev-s-path}/gateway/gateway.adoc[gRPC API Gateway] to route the requests to the correct service. + +=== Protobuf + +gRPC APIs are typically defined by link:https://protobuf.dev/overview/[Protocol buffers]. The different client and server stubs are created from ``*.proto`` files by code generation tools. + +== Versioning + +There are different standards for API versioning: Through URL, through request parameter, through custom header and through content negotiation. Ocis uses the versioning by URL concept although this creates a big code footprint. The versioning follows link:https://semver.org[SemVer]. We update the major version number when breaking changes are needed. Clients can decide which major version they use through the request URL. The specific implementation is documented on each API. diff --git a/content/ocis/9.0/modules/developer/pages/architecture/collaborative-storage.adoc b/content/ocis/9.0/modules/developer/pages/architecture/collaborative-storage.adoc new file mode 100644 index 00000000..a218b86d --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/architecture/collaborative-storage.adoc @@ -0,0 +1,95 @@ += Collaborative Storage +:toc: right + +One of the envisioned design goals of Infinite Scale is to work with so called _collaborative storage_, which means that the file system it is running on is not exclusive for Infinite Scale, but can be manipulated in parallel through third party tools. Infinite Scale is expected to monitor the changes that happen independently and react in a consistent and user friendly way. + +A real world example of that would be a third party "data producer" that submits data directly into a file system path, not going through Infinite Scale APIs. + +This document outlines a few challenges and design concepts for collaborative storage. It is also the base "checklist" for custom storage provider implementations for certain storages, ie. for Ceph- or IBM Storage Scale which provide features that allow more sophisticated and efficient implementations of this goal. + +== Storage Driver Components + +This discusses a few components and sub functions of the storage driver that have relevance for the collaborative storage. + +=== Path Locations + +What is called "the oCIS file system" is defined as the entire filetree underneath a special path in the local POSIX file system, which might either be a real local file system or a mounted net filesystem. It is expected that oCIS is the only consumer of that file tree, except what is expected behaviour with a collaborative file system, that adds and edits files in that tree. + +Underneath the oCIS file system root, there is a collection of different folders containing oCIS specific data. Specific storage driver data is in the directory `storage/users`, organized by spaces. +(TODO: Check again how different storage drivers work together without overwriting data of each other) + +=== Spaces + +Infinite Scale provides spaces as an additional organizational unit for data. Each space is a separate entity with its own attributes such as access patterns and quota. + +A storage driver has to model the separation of spaces and provide a list of spaces in general and also a list of spaces a user can access. Furthermore, it needs to be able to create different types of spaces (Home- or Project space). + +On POSIX, each space could for example be mapped to its own directory in a special spaces folder under the oCIS root folder. + +=== ID to Path Lookup + +Infinite Scale uses file IDs to efficiently identify files within a file tree. The lookup from a given ID to a path within the oCIS file tree is a very basic function that more or less defines the Infinite Scale performance. The functionality to for example query the file path for a given Inode number (which is the nearest equivalent for the Infinite Scale file ID) can not be done with standard POSIX system calls. + +The interface defining the collaborative storage needs an abstraction for this particular function, returning the file id for a given path, and returning the path for given id. + +=== Change Notification + +When a file is changed by a process outside of oCIS, this needs to be monitored by oCIS to quickly maintain internal caches and data structures as required. + +The collaborative storage driver needs a way to achieve that. The easiest way for an POSIX based collaborative storage is inotify, that needs to be set up recursively on a file tree to record changes. Additionally, it is a challenge to distinguish between changes that were done from external activity and the ones that oCIS creates by its own file operations. + +For GPFS, there is a subsystem called delivering that, see link:https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=reference-clustered-watch-folder[Clustered watch folder] for more details. + +=== ETag Propagation + +ownCloud requires that changes which happen "down" in a tree, can be detected in the root element of the tree. That happens through the change of the ETag metadata of each file and/or directory. An ETag is a random, text based tag, that only has one requirement: It has to change its content if a resource further down in the file tree has changed either its content or its metadata. (See link:https://github.com/owncloud/ocis/issues/3782[this issue] for further discussion about the ETag/CTag). + +POSIX file systems do not maintain a change flag like the ETag by default. The file time stamps (atime, ctime, mtime) in general are not fine granular enough (only seconds for some file systems) and depend on the server time, which renders them useless in a distributed environment. + +Infinite Scale needs to implement ETag propagation "up". For the collaborative storage, that needs to be combined with the change notification described above. + +Certain file systems implement this functionality either independently from Infinite Scale (EOS) or at least support proper change notifications (Ceph, GPFS?). + +=== Metadata Management + +Metadata is data "snippets" that are as tightly attached to files as ever possible. In best case, a rename of a file silently keeps the metadata as well. In POSIX, this can be achieved by extended file attributes with certain limitations. + +=== Quota + +Each space has its own quota, thus a storage driver implementation needs to consider that. + +For GPFS for example, there is support for quota handling in the file system, see link:https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=interfaces-gpfs-quotactl-subroutine[gpfs_quotactl() subroutine] for more details. + +Other systems store quota data in the metadata storage and implement propagation of used quota similar to the ETag propagation. + +=== User Management + +With user management it is meant how to handle the users and groups within oCIS and how that reflects to the file system where data is stored. + +==== Exclusive Environment + +In exclusive environments (aka. decomposedFS) all files of oCIS (ie. the entire oCIS filetree) belongs to a system user with the name `ocis` typically. + +==== Collaborative Storage + +For collaborative storages, the approach described above no longer works because users are supposed to be able to manipulate data in "their" file tree parts, and that is identified by ACLs and the owner of the files. + +That requires a few prerequisites that have to be fulfilled: + +. oCIS as one "client" changing data and the system that allows to access the file tree directly have to use the same user provider, to ensure that each user that is available on a shell is also available in oCIS. That ensures that changes are authenticated through system ACLs and users. LDAP based authentication on the system via PAM and the same LDAP as source for the oCIS IDP should be a sufficient setup. + +. oCIS must be able to write as a "different" user than the ocis system user. That means that we somehow have to impersonate file changing operations and run these as the user that is authenticated in oCIS. + +Example: There is a user ben. It has to have an entry in the LDAP that is used by IDP which oCIS is running "behind". With that, ben is able to authenticate through the IDP and work in the oCIS web app. The oCIS linux process will do writes and other changes impersonated as user ben. + +For the access of data on the commandline, the logins to the linux system must be authenticated against the same LDAP - so that ben can authenticate on a terminal using username and password. With that, the user can interactively change data that belongs to user ben (simplified said). + +To give permissions to groups, the linux group management must work accordingly. The same is true for file permissions. + +== Trashbin + +When a user deletes a file in oCIS it is moved to a so called trashbin that allows to restore the file if the deletion was accidental. + +== Versions + +When an existing file is changed, the former file state is to be preserved with data and metadata by oCIS. Some file system types provide this functionality via snapshots on partition or even file level. Other do not and have to implement that via a hidden directory keeping old file versions. diff --git a/content/ocis/9.0/modules/developer/pages/architecture/efficient-stat-polling.adoc b/content/ocis/9.0/modules/developer/pages/architecture/efficient-stat-polling.adoc new file mode 100644 index 00000000..4cc55798 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/architecture/efficient-stat-polling.adoc @@ -0,0 +1,206 @@ += Efficient Stat Polling +:toc: right +:toclevels: 3 + +The fallback sync mechanism uses the ETag to determine which part of a sync tree needs to be checked by recursively descending into folders whose ETag has changed. The ETag can be calculated using a `stat()` call in the filesystem and we are going to explore how many `stat()` calls are necessary and how the number might be reduced. + +== ETag Propagation + +What does ETag propagation mean? Whenever a file changes its content or metadata the ETag or "entity tag" changes. In the early days of ownCloud it was decided to extend this behavior to folders as well, which is outside of any WebDAV RFC specification. Nevertheless, here we are, using the ETag to reflect changes, not only on WebDAV resources but also WebDAV collections. The server will propagate the ETag change up to the root of the tree. + +[mermaid,width=300] +---- +graph TD + linkStyle default interpolate basis + + subgraph final ETag propagation + ert3(( etag:N )) --- el3(( etag:O )) & er3(( etag:N )) + er3 --- erl3(( etag:O )) & err3(( etag:N )) + end + + subgraph first ETag propagation + ert2(( etag:O )) --- el2(( etag:O )) & er2(( etag:N )) + er2 --- erl2(( etag:O )) & err2(( etag:N )) + end + + subgraph initial file change + ert(( etag:O )) --- el(( etag:O )) & er(( etag:O )) + er --- erl(( etag:O )) & err(( etag:N )) + end +---- + +The old `etag:O` is replaced by propagating the new `etag:N` up to the root, where the client will pick it up and explore the tree by comparing the old ETags known to the client with the state of the current ETags on the server. This form of sync is called _state based sync_. + +== Single User Sync + +To let the client detect changes in the drive (a tree of files and folders) of a user, we rely on the ETag of every node in the tree. The discovery phase starts at the root of the tree and checks if the ETag has changed since the last discovery: + +- if it is still the same nothing has changed inside the tree +- if it changed the client will compare the ETag of all immediate children and recursively descend into every node that changed + +This works, because the server side will propagate ETag changes in the tree up to the root. + +[mermaid,width=300] +---- +graph TD + linkStyle default interpolate basis + + ec( client ) -->|"stat()"|ert + + subgraph et[" "] + ert(( )) --- el(( )) & er(( )) + er --- erl(( )) & err(( )) + end +---- + +== Multiple Users + +On an ocis server there is not one user but many. Each of them may have one or more clients running. In the worst case all of them are polling the ETag of their home root node every 30 seconds. + +Keep in mind that etags are only propagated inside each distinct tree. No sharing is considered yet. + +[mermaid,width=300] +---- +graph TD + linkStyle default interpolate basis + + ec( client ) -->|"stat()"|ert + + subgraph et[" "] + ert(( )) --- el(( )) & er(( )) + er --- erl(( )) & err(( )) + end + + mc( client ) -->|"stat()"|mrt + + subgraph mt[" "] + mrt(( )) --- ml(( )) & mr(( )) + mr --- mrl(( )) & mrr(( )) + end + + fc( client ) -->|"stat()"|frt + + subgraph ft[" "] + frt(( )) --- fl(( )) & fr(( )) + fr --- frl(( )) & frr(( )) + end +---- + +== Sharing + +_Storage providers_ are responsible for persisting shares as close to the storage as possible. + +One implementation may persist shares using ACLs, another might use custom extended attributes. The chosen implementation is storage specific and always a tradeoff between various requirements. Yet, the goal is to treat the storage provider as the single source of truth for all metadata. + +If users can bypass the storage provider using e.g. `ssh` additional mechanisms needs to make sure no inconsistencies arise: + +- the ETag must still be propagated in a tree, eg using inotify, a policy engine or workflows triggered by other means +- deleted files should land in the trash (e.g. `rm` could be wrapped to move files to trash) +- overwriting files should create a new version ... other than a fuse fs I see no way of providing this for normal posix filesystems. Other storage backends that use the s3 protocol might provide versions natively. + +The storage provider is also responsible for keeping track of references e.g. using a shadow tree that users normally cannot see or representing them as symbolic links in the filesystem (Beware of symbolic link cycles. The clients are currently unaware of them and would flood the filesystem). + +To prevent write amplification ETags must not propagate across references. When a file that was shared by einstein changes the ETag must not be propagated into any share recipients tree. + +[mermaid,width=300] +---- +graph TD + linkStyle default interpolate basis + + + ec( einsteins client ) -->|"stat()"|ert + + subgraph mt[" "] + ml --- mlr(( )) + mrt(( )) --- ml(( )) & mr(( )) + mr --- mrl(( )) & mrr(( )) + end + + mlr -. reference .-> er + + subgraph et[" "] + ert(( )) --- el(( )) & er(( )) + er --- erl(( )) & err(( )) + end + + mc( maries client ) -->|"stat()"|mrt +---- + +But how can Marie's client detect the change? + +We are trading writes for reads: the client needs to stat the own tree & all shares or entry points into other storage trees. + +It would require client changes that depend on the server side actually having an endpoint that can efficiently list all entry points into storages a user has access to including their current etag. + +But having to list n storages might become a bottleneck anyway, so we are going to have the gateway calculate a virtual root ETag for all entry points a user has access to and cache that. + +== Server Side Stat Polling + +Every client polls the virtual root ETag (every 30 sec). The gateway will cache the virtual root ETag of every storage for 30 sec as well. That way every storage provider is only stated once every 30 sec (can be throttled dynamically to adapt to storage io load). + +[mermaid,width=300] +---- +graph TD + linkStyle default interpolate basis + + ec( client ) -->|"stat()"|evr + + subgraph gateway caching virtual etags + evr(( )) + mvr(( )) + fvr(( )) + end + + evr --- ert + mvr --- mrt + fvr --- frt + + subgraph et[" "] + ert(( )) --- el(( )) & er(( )) + er --- erl(( )) & err(( )) + end + + mc( client ) -->|"stat()"|mvr + + subgraph mt[" "] + mrt(( )) --- ml(( )) & mr(( )) + ml --- mlm(( )) + mr --- mrl(( )) & mrr(( )) + end + + mlm -.- er + mvr -.- er + + fc( client ) -->|"stat()"|fvr + + subgraph ft[" "] + frt(( )) --- fl(( )) & fr(( )) + fr --- frl(( )) & frr(( )) + end +---- + +Since the active clients will poll the etag for all active users the gateway will have their ETag cached. This is where sharing comes into play: The gateway also needs to stat the ETag of all other entry points ... or mount points. That may increase the number of stat like requests to storage providers by an order of magnitude. + +=== RAM Considerations + +For a single machine using a local posix storage the linux kernel already caches the inodes that contain the metadata that is necessary to calculate the ETag (even extended attributes are supported). With 4k inodes 256 nodes take 1Mb of RAM, 1k inodes take 4Mb and 1M inodes take 4Gb to completely cache the file metadata. For distributed filesystems a dedicated cache might make sense to prevent hammering it with stat like requests to calculate ETags. + +=== Bandwidth Considerations + +The bandwidth for a single machine might be another bottleneck. Consider a propfind request with roughly 500 bytes and a response with roughly 800 bytes in size: +- At 100Mbit (~10Mb/s) you can receive 20 000 PROPFIND requests +- At 1000Mbit (~100Mb/s) you can receive 200 000 PROPFIND requests +- At 10Gbit (~1Gb/s) you can receive 2 000 000 PROPFIND requests + +This can be scaled by adding more gateways and sharding users because these components are stateless. + +== Share Mount Point Polling Cache + +What can we do to reduce the number of stat calls to storage providers. Well, the gateway queries the share manager for all mounted shares of a user (or all entry points, not only the users own root/home). The share references contain the storage provider that contains the share. If every user has its own storage provider id the gateway could check in its own cache if the storage root etag has changed. It will be up-to-date because another client likely already polled for its etag. +This would reduce the number of necessary stat requests to active storages. + +=== Active Share Node Cache Invalidation +We can extend the lifetime of share ETag cache entries and only invalidate them when the root of the storage that contains them changes its ETag. That would reduce the number of stat requests to the number of active users. + +=== Push Notifications +We can further enhance this by sending push notifications when the root of a storage changes. Which is becoming increasingly necessary for mobile devices anyway. diff --git a/content/ocis/9.0/modules/developer/pages/architecture/index.adoc b/content/ocis/9.0/modules/developer/pages/architecture/index.adoc new file mode 100644 index 00000000..981b9e4b --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/architecture/index.adoc @@ -0,0 +1,7 @@ += Architecture +:toc: right +:toclevels: 3 + +In the architecture part of the documentation we collect useful developer documentation on different aspects of the architecture. We are using mermaid.js to collaborate on the necessary diagrams. + +_Pictures tell more than a thousand words._ diff --git a/content/ocis/9.0/modules/developer/pages/architecture/posixfs-storage-driver.adoc b/content/ocis/9.0/modules/developer/pages/architecture/posixfs-storage-driver.adoc new file mode 100644 index 00000000..c9b41a37 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/architecture/posixfs-storage-driver.adoc @@ -0,0 +1,205 @@ += PosixFS Storage Driver +:toc: right + +The Posix FS Storage Driver is a new storage driver for Infinite Scale. + +The scope of this document is to give a high level overview to the technical aspects of the Posix FS Storage Driver and guide the setup. + +== Introduction + +The Posix FS Storage Driver is a backend component that manages files on the server utilizing a "real" file tree that represents the data with folders and files in the file system as users are used to it. That is the big difference compared to Decomposed FS which is the default storage driver in Infinite Scale. + +This does not mean that Infinite Scale is trading any of its benefits to this new feature: It still implements simplicity by running without a database, it continues to store metadata in the file system and adds them transparently to caches and search indexes, and it also features the full spaces concept as before, just to name a few examples. + +The architecture of Infinite Scale allows configuring different storage drivers for specific storage types and purposes on a space granularity. The Posix FS Storage Driver is an alternative to the default driver called Decomposed FS. + +However, the clarity of the file structure in the underlying file system is not the only benefit of the Posix FS Storage Driver. This new technology allows users to manipulate the data directly in the file system, and any changes made to files outside of Infinite Scale are monitored and directly reflected in Infinite Scale. For example, a scanner could store its output directly to the Infinite Scale file system, which immediately gets picked up in Infinite Scale. + +For the first time ever with feature rich open source file sync & share systems, users can either choose to work with their data through the clients of the system, its APIs or even directly in the underlying file system on the server. + +That is another powerful vector for integration and enables a new spectrum of use cases across all domains. + +== Technical Aspects + +The Posix FS Storage Driver uses a few features of the underlying file system, which are mandatory and directly contributing to the performance of the system. + +While the simplest form of Posix FS Storage Driver runs with default file systems of every modern Linux system which are directly mounted and thus support inotify, the full power of this unfolds with more capable file systems such as IBM Storage Scale or Ceph. These are recommended as reliable foundations for big installations of Infinite Scale. + +This chapter describes some technical aspects of this storage driver. + +=== Path Locations + +The file tree that is used as storage path for both data and metadata is located under the local path on the machine that is running Infinite Scale. That might either be a real local file system or a mounted net filesystem. It is expected that oCIS is the only consumer of that file tree, except what is expected behaviour with a collaborative file system, that works with files in that tree. + +Underneath the Infinite Scale file system root, there is a collection of different folders containing Infinite Scale specific data storing personal spaces, project spaces and indexes. + +=== Metadata + +Infinite Scale is highly dependent on the efficient usage of meta data which are attached to file resources, but also logical elements such as spaces. + +Metadata is stored in extended attributes (as also supported by decomposed FS) which poses the benefit that metadata is always directly attached to the actual resources. As a result, care has to be taken that extended attributes are considered when working with the file tree however, e.g. when creating or restoring backups. + +NOTE: The maximum number and size of extended attributes are limited depending on the filesystem and block size. See xref:#gpfs-specifics[GPFS Specifics] for more details on GPFS file systems. + +All indexing and caching of metadata is implemented in higher system levels than the storage driver, and thus are not different to the components used with other storage drivers like the decomposed FS. + +=== Monitoring + +To get information about changes such as new files added, files edited or removed, Infinite Scale uses a monitoring system to directly watch the file system. This starts with the Linux inotify system and ranges to much more sophisticated services as for example in Spectrum Scale (see xref:#gpfs-specifics[GPFS Specifics] for more details on GPFS file systems). + +Based on the information transmitted by the watching service, Infinite Scale is able to "register" new or changed files into its own caches and internal management structures. This enables Infinite Scale to deliver resource changes through the "traditional" channels such as APIs and clients. + +Since the most important metadata is the file tree structure itself, it is impossible for the "split brain" situation between data and metadata to cause trouble. + +=== Automatic ETag Propagation + +The ETag of a resource can be understood as a content fingerprint of any file- or folder resource in Infinite Scale. It is mainly used by clients to detect changes of resources. The rule is, that if the content of a file changed the ETag has to change as well, as well as the ETag of all parent folders up to the root of the space. + +Infinite Scale uses a built in mechanism to maintain the ETag for each resource in the file meta data, and also propagates it automatically. + +A sophisticated underlying file system could provide an attribute that fulfills this requirement and changes whenever content or metadata of a resource changes, and - which is most important - also changes the attribute of the parent resource and the parent of the parent etc. + +=== Automatic Tree Size Propagation + +Similar to the ETag propagation described before, Infinite Scale also tracks the accumulated tree size in all nodes of the file tree. A change to any file requires a re-calculation of the size attribute in all parent folders. + +Infinite Scale would benefit from file systems with native tree size propagation. + +=== Quota + +Each space has its own quota, thus every storage driver implementation needs to consider that. + +For example, IBM Spectrum Scale supports quota handling directly in the file system. + +Other systems store quota data in the metadata storage and implement propagation of used quota similar to the tree size propagation. + +=== File ID Resolution + +Infinite Scale uses an ID based approach to work with resources, rather than a file path based mechanism. The reason for that is, that ID based lookups can be done way more efficiently compared to tree traversals, just to name one reason. + +The most important component of the ID is a unique file ID that identifies the resource within a space. Ideally the Inode of a file could be used here. However, some file systems re-use inodes which must be avoided. Infinite Scale thus does not use the file Inode, but generates a UUID instead. + +ID based lookups utilize an ID cache which needs to be shared between all storageprovider and dataprovider instances. During startup a scan of the whole file tree is performed to detect and cache new entities. + +In the future a powerful underlying file system could support Infinite Scale by providing an API that + +. Provides the ID for a given file path referenced resource +. Provides the path for a given ID. + +These two operations are very crucial for the performance of the entire system. + +=== User Management + +With the requirement that data can be manipulated either through the filesystem or the Infinite Scale system, the question under which UID the manipulation happens is important. + +There are a few possible ways for user management: + +. Changes can either be only accepted by the same user that Infinite Scale is running under, for example the user `ocis`. All manipulations in the filesystem have to be done by, and only by this user. + +. Group based: All users who should be able to manipulate files have to be in a unix group. The Infinite Scale user has also to be member of that group. The default umask in the directory used has to allow group writing all over the place. + +. Impersonation: Infinite Scale impersonates the user who owns the folder on the file system to mimic the access as the user. + +All possibilities have pros and cons for operations. + +One for all, it seems reasonable to use LDAP to manage users which is the base for the Infinite Scale IDP as well as the system login system via PAM. + +=== GID Based Space Access + +The Posix FS Storage Driver supports GID based space access to support the problem that project spaces might have to be accessible by multiple users on disk. In order to enable this feature the `ocis` binary needs to have the `setgid` capability and `STORAGE_USERS_POSIX_USE_SPACE_GROUPS` needs to be set to `true`. Infinite Scale will then use the space GID (the gid of the space root) for all file system access using the `setfsgid` syscall, i.e. all files and directories created by Infinite Scale will belong to the same group as the space root. + +== Advanced Features + +Depending on the capabilities of the underlying file system, the Posix FS Storage Driver can benefit from more advanced functionality described here. + +=== Versioning + +If the underlying file system is able to create versions of single resources (imagine a git based file system) this functionality could directly be used by Infinite Scale. + +In the current state of the Posix FS Storage Driver, versioning is not supported. + +=== Trashbin + +If the underlying file system handles deleted files in a trash bin that allows restoring of previously removed files, this functionality could directly be used by Infinite Scale. + +If not available it will follow the link:https://specifications.freedesktop.org/trash/latest/[Free Desktop Trash specification, window=_blank]. + +== Limitations + +As of Q2/2024 the Posix FS Storage Driver is not officially supported and in technical preview state. + +The tech preview comes with the following limitations: + +. Only inotify and GPFS file system change notification methods are supported +. Versioning is not supported yet +. The space/project folders in the filesystem are named after the UUID, not the real space name +. No CephFS support yet +. Postprocessing (ie. anti virus check) does not happen for file actions outside of Infinite Scale + +== Setup + +This describes the steps to use the Posix FS Storage Driver storage driver with Infinite Scale. + +It is possible to use different storage drivers in the same Infinite Scale installation. For example it is possible to set up one space running on Posix FS Storage Driver while others run Decomposed FS. + +=== Prerequisites + +To use the Posix FS Storage Driver, the following prerequisites have to be fulfilled: + +. There must be storage available to store meta data and blobs, available under a root path. +. When using inotify, the storage must be local on the same machine. Network mounts do not work with inotify. `inotifywait` needs to be installed. +. The storage root path must be writeable and executable by the same user Infinite Scale is running under. +. An appropiate version of Infinite Scale is installed, version number 5.0.5 and later. +. `nats-js-kv` as cache service + + +=== Setup Configuration + +This is an example configuration with environment variables that configures Infinite Scale to use Posix FS Storage Driver for all spaces it works with, ie. Personal and Project Spaces: + +[source,bash] +---- +export STORAGE_USERS_DRIVER="posix" +export STORAGE_USERS_POSIX_ROOT="/home/kf/tmp/posix-storage" +export STORAGE_USERS_POSIX_WATCH_TYPE="inotifywait" +export STORAGE_USERS_ID_CACHE_STORE="nats-js-kv" +export STORAGE_USERS_ID_CACHE_STORE_NODES="localhost:9233" + +# Optionally enable gid based space access +export STORAGE_USERS_POSIX_USE_SPACE_GROUPS="true" +---- + +== GPFS Specifics + +When using GPFS as the underlying filesystem the machine running the according `storage-users` service needs to have the GPFS filesystem mounted locally. The mount path is given to ocis as the `STORAGE_USERS_POSIX_ROOT` path. + +Other than that there a few other points to consider: + +=== Extended Attributes + +As described above metadata is stored as extended attributes of the according entities and thus is suspect to their limitations. In GPFS extended attributes are first stored in the inode itself but can then also use an overflow block which is at least 64KB and up to the metadata block size. Inode and metadata block size should be chosen accordingly. + +=== FS Watcher + +The Posix FS Storage Driver supports two different watchers for detecting changes to the filesystem. The watchfolder watcher is better tested and supported at that point. + +==== GPFS File Audit Logging + +The `gpfsfileauditlogging` watcher tails a GPFS file audit log and parses the JSON events to detect relevant changes. + +[source,bash] +---- +export STORAGE_USERS_POSIX_WATCH_TYPE="gpfsfileauditlogging" +export STORAGE_USERS_POSIX_WATCH_PATH="/path/to/current/audit/log" +---- + +==== GPFS Watchfolder + +The `gpfswatchfolder` watcher connects to a kafka cluster which is being filled with filesystem events by the GPFS watchfolder service. + +[source,bash] +---- +export STORAGE_USERS_POSIX_WATCH_TYPE="gpfswatchfolder" +export STORAGE_USERS_POSIX_WATCH_PATH="fs1_audit" # the kafka topic to watch +export STORAGE_USERS_POSIX_WATCH_FOLDER_KAFKA_BROKERS="192.168.1.180:29092" +---- diff --git a/content/ocis/9.0/modules/developer/pages/architecture/protocol-changes.adoc b/content/ocis/9.0/modules/developer/pages/architecture/protocol-changes.adoc new file mode 100644 index 00000000..d23ec1d4 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/architecture/protocol-changes.adoc @@ -0,0 +1,209 @@ += Protocol Changes +:toc: right + +The spaces concept allows clients to look up the space endpoints a user has access to and then do individual sync discoveries. Technically, we introduce an indirection that allows clients to rely on server provided URLs instead of hardcoded `/webdav` or `/dav/files/\{username}` paths, that may change over time. + +== Space Discovery + +[mermaid,width=300] +---- +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant Graph + participant SpaceA + participant SpaceB + links Client: {"web": "https://doc.owncloud.com/ocis/developer/clients/web/index.html", "RClone": ""https://doc.owncloud.com/ocis/developer/clients/rclone/index.html"} + link Graph: Documentation @ "https://doc.owncloud.com/ocis/developer/ + + Note left of Client: First, a clients looks
up the spaces a user has access to + opt space lookup + Client->>+Graph: GET /me/drives + Graph-->>-Client: 200 OK JSON list of spaces, say A, B and C,
each with a dedicated webDavURL, etag and quota + end + + Note left of Client: Then it can do a parallel
sync discovery on spaces
whose etag changed + par Client to Space A + Client->>+SpaceA: PROPFIND {webDavURL for Space A} + SpaceA-->>-Client: 207 Multistatus PROPFIND response + and Client to Space B + Client->>+SpaceB: PROPFIND {webDavURL for space B} + SpaceB-->>-Client: 207 Multistatus PROPFIND response + end +---- + +=== New /dav/spaces/\{spaceid} Endpoint with Spaceid and a Relative Path + +The ocDAV service is responsible for translating ownCloud flavoured WebDAV into CS3 API calls. + +General view:: ++ +-- +A PROPFIND finds its way to a storage provider like this: + +[mermaid,width=300] +---- +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocDAV + participant StorageProvider + + Note right of Client: {spaceid} identifies the space
{relative/path} is relative to the space root + Client->>+ocDAV: PROPFIND /dav/space/{spaceid}/{relative/path} + Note right of ocDAV: translate ownCloud flavoured webdav
into CS3 API requests + ocDAV->>+StorageProvider: ListContainer({spaceid}, path: {relative/path}) + StorageProvider-->>-ocDAV: []ResourceInfo + ocDAV-->>-Client: 207 Multistatus +---- + +While the above is a simplification to get an understanding of what needs to go where, there are several places where sharding can happen. +-- + +Proxy can do user based routing:: ++ +-- +The ocis proxy authenticates requests and can forward requests to different backends, depending on the logged-in user or cookies. For example multiple ocdav services can be configured to shard users based on username or affiliation. + +[mermaid,width=300] +---- +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant proxy + participant ocDAV1 as ocDAV [a-k] + participant ocDAV2 as ocDAV [l-z] + + Note right of Client: {spaceid} identifies the space
{relative/path} is relative to the space root + Client->>+proxy: PROPFIND /dav/space/{spaceid}/{relative/path} + + alt username starting with a-k + proxy->>+ocDAV1: PROPFIND /dav/space/{spaceid}/{relative/path} + Note right of ocDAV1: translate ownCloud flavoured webdav
into CS3 API requests + ocDAV1-->>-Client: 207 Multistatus + else username starting with l-z + proxy->>+ocDAV2: PROPFIND /dav/space/{spaceid}/{relative/path} + ocDAV2-->>-Client: 207 Multistatus + end +---- +-- + +Gateway can do path or storage provider id based routing:: ++ +-- +The reva gateway acts as a facade to multiple storage providers that can be configured with the storage registry: + +[mermaid,width=300] +---- +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant ocDAV + participant Gateway + participant StorageRegistry + participant StorageProvider1 as StorageProvider [a-k] + participant StorageProvider2 as StorageProvider [l-z] + + Note right of ocDAV: translate ownCloud flavoured webdav
into CS3 API requests + ocDAV->>+Gateway: ListContainer({spaceid}, path: {relative/path}) + Note right of Gateway: find address of the storage provider
that is responsible for the space + Gateway->>+StorageRegistry: ListStorageProviders({spaceid}) + StorageRegistry-->>-Gateway: []ProviderInfo + Note right of Gateway: forward request to
correct storage provider + alt username starting with a-k + Gateway->>+StorageProvider1: ListContainer({spaceid}, path: {relative/path}) + StorageProvider1-->>-Gateway: []ResourceInfo + else username starting with l-z + Gateway->>+StorageProvider2: ListContainer({spaceid}, path: {relative/path}) + StorageProvider2-->>-Gateway: []ResourceInfo + end + Gateway-->>-ocDAV: []ResourceInfo +---- +-- + +=== Old /dav/files/\{username} Endpoint with Username and a Path Relative to the Users Home + +PROPFIND request against old webdav endpoints:: ++ +-- +To route a PROPFIND request against the old webdav endpoints like `/dav/files/username`, ocdav first has to build a CS3 namespace prefix, e.g. `/users/{{.Id.OpaqueId}}` to the users home. + +[mermaid,width=300] +---- +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocDAV + participant Gateway + + opt old /dav/files/{username} endpoint with username and a path relative to the users home + Note right of Client: translate ownCloud flavoured webdav
into CS3 API requests + Client->>+ocDAV: PROPFIND /dav/files/{username}/{relative/path} + Note right of ocDAV: translate ownCloud flavoured webdav
into CS3 API requests + ocDAV->>+Gateway: GetUser({username}) + Gateway-->>-ocDAV: User + Note right of ocDAV: build path prefix to user home + ocDAV->>+ocDAV: {namespace/prefix} = ApplyLayout({path layout}, User), eg. /users/e/einstein + Note right of ocDAV: look up the space responsible for a path + ocDAV->>+Gateway: ListStorageSpaces(path: {namespace/prefix}/{relative/path}) + Gateway-->>-ocDAV: []StorageSpace + Note right of ocDAV: make actual request with space and relative path + ocDAV->>+Gateway: ListContainer({spaceid}, path: {relative/path}) + Gateway-->>-ocDAV: []ResourceInfo + ocDAV-->>-Client: 207 Multistatus + end +---- +-- + +Handling legacy global namespace webdav endpoints:: ++ +-- +The reason ocis uses a path based lookup instead of looking up the current users home using the user id and a space type filter is, because there are deployments that use a global namespace at the legacy `/webdav` endpoint. To support these use cases, the gateway allows looking up spaces using their mount path. + +[mermaid,width=300] +---- +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocDAV + participant Gateway + + Note right of Client: translate ownCloud flavoured webdav
into CS3 API requests + alt old /dav/files/{username} endpoint with username and a path relative to the users home + Client->>+ocDAV: PROPFIND /dav/files/{username}/{relative/path} + Note right of ocDAV: look up {username} in URL path + ocDAV->>+Gateway: GetUser({username}) + Gateway-->>-ocDAV: User + Note right of ocDAV:build namespace prefix to user home + ocDAV->>+ocDAV: {namespace/prefix} = ApplyLayout({namespace layout}, User), eg. /users/e/einstein + else legacy /webdav/ endpoint with a path relative to the users home + Client->>+ocDAV: PROPFIND /webdav/{relative/path} + Note right of ocDAV: use currently logged in user + ocDAV->>+ocDAV: ContextGetUser() + Note right of ocDAV: build namespace prefix to user home + ocDAV->>+ocDAV: {namespace/prefix} = ApplyLayout({namespace layout}, User), eg. /users/e/einstein + else legacy /webdav/ endpoint with a path relative to a global namespace + Client->>+ocDAV: PROPFIND /webdav/{relative/path} + Note right of ocDAV: omit namespace prefix by using empty layout template + ocDAV->>+ocDAV: {namespace/prefix} = ApplyLayout("/", u), always returns "/" + end + Note right of ocDAV: look up the space responsible for a path + ocDAV->>+Gateway: ListStorageSpaces(path: {namespace/prefix}/{relative/path}) + Gateway-->>-ocDAV: []StorageSpace + Note right of ocDAV: make actual request with space and relative path + ocDAV->>+Gateway: ListContainer({spaceid}, path: {relative/path}) + Gateway-->>-ocDAV: []ResourceInfo + ocDAV-->>-Client: 207 Multistatus +---- +-- diff --git a/content/ocis/9.0/modules/developer/pages/architecture/services-communication.adoc b/content/ocis/9.0/modules/developer/pages/architecture/services-communication.adoc new file mode 100644 index 00000000..e4436b70 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/architecture/services-communication.adoc @@ -0,0 +1,5 @@ += Services Communication +:toc: right +:toclevels: 3 + +image::ocis/architecture/ocis-services-communication.drawio.svg[] diff --git a/content/ocis/9.0/modules/developer/pages/architecture/upload-processing.adoc b/content/ocis/9.0/modules/developer/pages/architecture/upload-processing.adoc new file mode 100644 index 00000000..5b5b67af --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/architecture/upload-processing.adoc @@ -0,0 +1,124 @@ += Upload Processing +:toc: right +:toclevels: 3 + +Uploads are handled by a dedicated service that uses TUS.io for resumable uploads. When all bytes have been transferred the upload is finalized by making the file available in file listings and for download. + +The finalization may be asynchronous when mandatory workflow steps are involved. + +== Legacy PUT Upload + +[mermaid,width=300] +---- +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocdav + participant storageprovider + participant dataprovider + + Client->>+ocdav: PUT /dav/spaces/{spaceid}/newfile.bin + ocdav->>+storageprovider: InitiateFileUpload + storageprovider-->>-ocdav: OK, Protocol simple, UploadEndpoint: /data, Token: {jwt} + Note right of ocdav: The {jwt} contains the internal actual target, eg.: http://localhost:9158/data/simple/91cc9882-db71-4b37-b694-a522850fcee1 + ocdav->>+dataprovider: PUT /data + Note right of dataprovider: X-Reva-Transfer: {jwt} + dataprovider-->>-ocdav: 201 Created + ocdav-->>-Client: 201 Created +---- + +== TUS Upload + +[mermaid,width=300] +---- +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocdav + participant storageprovider + participant datagateway + participant dataprovider + + Client->>+ocdav: POST /dav/spaces/{spaceid}\nUpload-Metadata: {base64 encoded filename etc}\nTUS-Resumable: 1.0.0 + ocdav->>+storageprovider: InitiateFileUpload + storageprovider-->>-ocdav: OK, Protocol tus, UploadEndpoint: /data, Token: {jwt} + Note right of ocdav: The {jwt} contains the internal actual target, eg.:\nhttp://localhost:9158/data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160 + ocdav-->>-Client: 201 Created\nLocation: /data/{jwt}\nTUS-Resumable: 1.0.0 + + Client->>+datagateway: PATCH /data/{jwt}\nTUS-Resumable: 1.0.0\nUpload-Offset: 0 + Note over datagateway: unwrap the {jwt} target + datagateway->>+dataprovider: PATCH /data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160\nX-Reva-Transfer: {jwt} + Note over dataprovider: storage driver\nhandles request + dataprovider-->>-datagateway: 204 No Content\nTUS-Resumable: 1.0.0\nUpload-Offset: 363976 + datagateway-->>-Client: 204 No Content\nTUS-Resumable: 1.0.0\nUpload-Offset: 363976 +---- + + +== TUS Upload with Async Postprocessing + +[mermaid,width=300] +---- +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocdav + participant storageprovider + participant datagateway + participant dataprovider + participant nats + participant processing + + Client->>+ocdav: POST /dav/spaces/{spaceid} + Note left of Client: Upload-Metadata: {base64 encoded filename etc}\nTUS-Resumable: 1.0.0 + ocdav->>+storageprovider: InitiateFileUpload + storageprovider-->>-ocdav: OK, Protocol tus, UploadEndpoint: /data, Token: {jwt} + Note right of ocdav: The {jwt} contains the internal actual target, eg.: http://localhost:9158/data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160 + ocdav-->>-Client: 201 Created + Note right of Client: Location: /data/{jwt} + Note right of Client: TUS-Resumable: 1.0.0 + + Client->>+datagateway: PATCH /data/{jwt} + Note right of datagateway: TUS-Resumable: 1.0.0\nUpload-Offset: 0 + + Note over datagateway: unwrap the {jwt} target + datagateway->>+dataprovider: PATCH /data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160 + Note over dataprovider: X-Reva-Transfer: {jwt} + Note over dataprovider: storage driver + Note over dataprovider: handles request + dataprovider-)nats: emit all-bytes-received event + nats-)processing: all-bytes-received({uploadid}) event + Note over dataprovider: TODO: A lot of time may pass here, we could use the `Prefer: respond-async` header to return early with a 202 Accepted status and a Location header to a websocket endpoint + alt success + processing-)nats: emit processing-finished({uploadid}) event + nats-)dataprovider: processing-finished({uploadid}) event + dataprovider-->>-datagateway: 204 No Content + Note over datagateway: TUS-Resumable: 1.0.0\nUpload-Offset: 363976 + datagateway-->>-Client: 204 No Content + Note over Client: TUS-Resumable: 1.0.0\nUpload-Offset: 363976 + else failure + activate dataprovider + activate datagateway + processing-)nats: emit processing-aborted({uploadid}) event + nats-)dataprovider: processing-aborted({uploadid}) event + Note over dataprovider: FIXME: What HTTP status code should we report? + Note over dataprovider: 422 Unprocessable Content is just a proposal + Note over dataprovider: see https://httpwg.org/specs/rfc9110.html#status.422 + dataprovider-->>-datagateway: 422 Unprocessable Content + Note over datagateway: TUS-Resumable: 1.0.0\nUpload-Offset: 363976 + datagateway-->>-Client: 422 Unprocessable Content + Note over Client: TUS-Resumable: 1.0.0\nUpload-Offset: 363976 + end +---- + + +== Async TUS Upload with Postprocessing + +This might be a TUS extension or a misunderstanding on our side of what tus can do for us. Clients should send a `Prefer: respond-async` header to allow the server to return early when postprocessing might take longer. The PATCH requests can then return status `202 Accepted` and a `Location` header to a websocket that clients can use to track the processing / upload progress. + +TODO there is a conflict with the TUS.io POST request with the creation extension, as that also returns a `Location` header which carries the upload URL. We would need another header to transport the websocket location. Maybe `Websocket-Location` or `Progress-Location`? diff --git a/content/ocis/9.0/modules/developer/pages/cli-commands/create-new-cli-command.adoc b/content/ocis/9.0/modules/developer/pages/cli-commands/create-new-cli-command.adoc new file mode 100644 index 00000000..8e8ad48b --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/cli-commands/create-new-cli-command.adoc @@ -0,0 +1,18 @@ += Create a New CLI Command +:toc: right +:toclevels: 3 + + +[NOTE] +==== +Existing commands should be checked for commonly used options whenever a new CLI command is created, regardless of whether it is embedded in a service or not. For an example you can see the xref:cli-commands/service_independent_cli.adoc[Service Independent CLI] documentation. +==== + + +== CLI Embedded in a Service + +These commands are usually located in the `/pkg/command` subfolder. + +== CLI Independent of a Service + +These commands are located in the `ocis/pkg/command` subfolder. diff --git a/content/ocis/9.0/modules/developer/pages/cli-commands/document-cli-commands.adoc b/content/ocis/9.0/modules/developer/pages/cli-commands/document-cli-commands.adoc new file mode 100644 index 00000000..dfa4e609 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/cli-commands/document-cli-commands.adoc @@ -0,0 +1,29 @@ += Document CLI Commands +:toc: right +:toclevels: 3 + + +Any CLI command that is added to Infinite Scale must be documented here in the dev docs and the xref:admin:maintenance/commands/commands.adoc[admin docs]. Note that the admin docs primarily distinguish between online and offline commands because the structure of the documentation is different. Typically, any command documented in the developer documentation is integrated into the admin documentation and adapted according to the target audience. The description here is for developers; the admin docs are derived from it. + +[NOTE] +==== +Note that any CLI command requires documentation. However, it may be decided that a CLI command will not be included in the admin documentation. In such a case, the reasons should be valid. +==== + +== Type of CLI Commands + +There are three types of CLI commands that require different documentation locations: + +* Commands that are embedded in a service such as: + +`ocis storage-users uploads` +* Commands that are service independent such as: + +`ocis trash purge-empty-dirs` or `ocis revisions purge` +* `curl` commands that can be one of the above. + +== Rules + +* _Service dependent_ CLI commands: + +Add any CLI command into the repsective `README.md` _of the service_. Use as "template" an existing one such as in `services/storage-users/README.md`. + +* _Service independent_ CLI commands: + +Add any CLI command into the xref:cli-commands/service_independent_cli.adoc[Service Independent CLI] documentation. diff --git a/content/ocis/9.0/modules/developer/pages/cli-commands/index.adoc b/content/ocis/9.0/modules/developer/pages/cli-commands/index.adoc new file mode 100644 index 00000000..816aee46 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/cli-commands/index.adoc @@ -0,0 +1,6 @@ += oCIS CLI Commands +:toc: right +:toclevels: 3 + +CLI commands can either be embedded in a service or operate independently. + diff --git a/content/ocis/9.0/modules/developer/pages/cli-commands/service_dependent_cli.adoc b/content/ocis/9.0/modules/developer/pages/cli-commands/service_dependent_cli.adoc new file mode 100644 index 00000000..d0b3fae0 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/cli-commands/service_dependent_cli.adoc @@ -0,0 +1,41 @@ += Service Dependent CLI +:toc: right +:toclevels: 3 + +This document describes ocis CLI commands that are _embedded in a service_. + +[NOTE] +==== +For all admin-related CLI commands, see the xref:admin:maintenance/commands/commands.adoc#online-only-commands[Maintenance Commands] in the admin documentation. +==== + +== Common Parameters + +The ocis package offers a variety of CLI commands for monitoring or repairing ocis installations. Most of these commands have common parameters such as: + +* `--help` (or `-h`) +Use to print all available options. + +* `--basePath` (or `-p`) + +Needs to point to a storage provider, paths can vary depending on your ocis installation. Example paths are: ++ +[source,bash] +---- +/var/tmp/ocis/storage/users # docker installation +... +---- + +* `--dry-run` + +This parameter, when available, defaults to `true` and must explicitly set to `false`. + +* `--verbose` (or `-v`) + +Get a more verbose output. + +== List of CLI Commands + +For example CLI commands that are _embedded in a service_, see the following services: + +* xref:{dev-s-path}/auth-app/auth-app.adoc[Auth-App] +* xref:{dev-s-path}/graph/graph.adoc[Graph] +* xref:{dev-s-path}/postprocessing/postprocessing.adoc[Postprocessing] +* xref:{dev-s-path}/storage-users/storage-users.adoc[Storage-Users] +* ... diff --git a/content/ocis/9.0/modules/developer/pages/cli-commands/service_independent_cli.adoc b/content/ocis/9.0/modules/developer/pages/cli-commands/service_independent_cli.adoc new file mode 100644 index 00000000..8fea290b --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/cli-commands/service_independent_cli.adoc @@ -0,0 +1,178 @@ += Service Independent CLI +:toc: right +:toclevels: 3 + +This document describes ocis CLI commands that are _service independent_. + +[NOTE] +==== +For all admin-related CLI commands, see the xref:admin:maintenance/commands/commands.adoc#offline-commands[Maintenance Commands] in the admin documentation. +==== + +== Common Parameters + +The ocis package offers a variety of CLI commands for monitoring or repairing ocis installations. Most of these commands have common parameters such as: + +* `--help` (or `-h`) + + Use to print all available options. + +* `--basePath` (or `-p`) + +Needs to point to a storage provider, paths can vary depending on your ocis installation. Example paths are: ++ +[source,plaintext] +---- +/var/tmp/ocis/storage/users # docker installation +... +---- + +* `--dry-run` + + This parameter, when available, defaults to `true` and must explicitly set to `false`. + +* `--verbose` (or `-v`) + + Get a more verbose output. + +== List of Developer Only CLI Commands + +=== Cleanup Orphaned Grants + +Detect and optionally delete storage grants that have no corresponding share-manager entry. + +Sharing in ocis relies on two truths. The share manager and the grants. When a share is created, ocis will + +. Create a grant for the specific file or folder. + +This grant is _checked when access to the file is requested_. + +. Create an entry in the `created.json`/`received.json` files of the specific user. + +These files are _checked whenever shares are listed_. + +The process for creating a share is as follows: first, ocis creates the grant, and then adds the share entry. The reverse order is followed when deleting a share. This means that if the second step fails, the grant will still be present. This can be visually confirmed in the webUI. The webUI details of the "share" section will show an error fetching information for orphan grants. + +The following command fixes the problem of orhaned grants. + +Usage: +[source,bash] +---- +ocis shares clean-orphaned-grants \ + --service-account-id "" \ + --service-account-secret "" \ + [--force] \ + [--space-id ""] \ + [--dry-run=false] +---- + +Notes: +* `--dry-run` + +Defaults to `true` (no deletions). Set to `false` to remove orphaned grants. + +* `--space-id` + +Limit the scan to a specific storage space (opaque ID). + +* `--force` + +Force removal of suspected orphans even when listing shares fails. + +* Public links are not touched. + +=== Cleanup Orphaned Shares + +When a shared space or directory got deleted, use the `shares cleanup` command to remove those share orphans. This can't be done automatically at the moment. + +[source,bash] +---- +ocis shares cleanup +---- + +=== List Unified Roles + +This command simplifies the process of finding out which UID belongs to which role. The command using markdown as output format is: + +[source,bash] +---- +ocis graph list-unified-roles --output-format md +---- + +The output of this command includes the following information for each role: + +* `Name` + +The human readable name of the role. + +* `UID` + +The unique identifier of the role. + +* `Enabled` + +Whether the role is enabled or not. + +* `Description` + +A short description of the role. + +* `Condition` +* `Allowed Resource Actions` + +_Example output (shortned)_ + +[cols="1,2,3,1,3,3,3",options="header"] +|=== +| # +| Label +| UID +| Enabled +| Description +| Condition +| Allowed Resource Actions + +| 1 +| Viewer +| b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5 +| enabled +| View and download. +| exists @Resource.File +| libre.graph/driveItem/path/read + +| +| +| +| +| +| exists @Resource.Folder +| libre.graph/driveItem/quota/read + +| +| +| +| +| +| exists @Resource.File && @Subject.UserType=="Federated" +| libre.graph/driveItem/content/read + +| +| +| +| +| +| exists @Resource.Folder && @Subject.UserType=="Federated" +| libre.graph/driveItem/children/read + +| +| +| +| +| +| +| libre.graph/driveItem/deleted/read + +| +| +| +| +| +| +| libre.graph/driveItem/basic/read + +| 2 +| ViewerListGrants +| d5041006-ebb3-4b4a-b6a4-7c180ecfb17d +| disabled +| View, download and show all invite +| +| +|=== + diff --git a/content/ocis/9.0/modules/developer/pages/clients/index.adoc b/content/ocis/9.0/modules/developer/pages/clients/index.adoc new file mode 100644 index 00000000..9eff1501 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/index.adoc @@ -0,0 +1,5 @@ += Clients +:toc: right +:toclevels: 3 + +This Section describes clients for ocis. diff --git a/content/ocis/9.0/modules/developer/pages/clients/ocis-php-sdk/ocis-php-sdk-docs.adoc b/content/ocis/9.0/modules/developer/pages/clients/ocis-php-sdk/ocis-php-sdk-docs.adoc new file mode 100644 index 00000000..a676a5ff --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/ocis-php-sdk/ocis-php-sdk-docs.adoc @@ -0,0 +1,43 @@ += oCIS PHP SDK — API Reference +:toc: right +:description: phpDocumentor-generated API reference for the oCIS PHP SDK, embedded from the source repository. + +The interactive API reference below is generated by link:https://www.phpdoc.org/[phpDocumentor] directly from the link:https://github.com/owncloud/ocis-php-sdk[ocis-php-sdk source repository]. + +link:attachment$ocis-php-sdk-docs/index.html[Open API Reference in a new tab,window=_blank] + +//// +The passthrough block below embeds the phpDocumentor HTML output in an iframe. + +The iframe src is intentionally set to "about:blank" and resolved at runtime via JavaScript. +This avoids hardcoding the Antora published path, which varies by component, version, and module. +The script derives the attachment URL by stripping everything from "/clients/" onward from the +current page's pathname and appending the known attachment path "/_attachments/ocis-php-sdk-docs/index.html". + +The onload handler resizes the iframe to the full scrollHeight of the loaded document. This works because +both the Antora page and the attachment are served from the same origin (doc.owncloud.com), so the script +can access the iframe's contentWindow. The try/catch silently ignores the resize if cross-origin restrictions +apply in local or preview builds. +//// + +++++ + + +++++ diff --git a/content/ocis/9.0/modules/developer/pages/clients/ocis-php-sdk/ocis-php-sdk.adoc b/content/ocis/9.0/modules/developer/pages/clients/ocis-php-sdk/ocis-php-sdk.adoc new file mode 100644 index 00000000..795a723d --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/ocis-php-sdk/ocis-php-sdk.adoc @@ -0,0 +1,364 @@ += oCIS PHP SDK +:toc: right +:toclevels: 3 +:description: PHP SDK for interacting with ownCloud Infinite Scale (oCIS) storage. + +This SDK allows you to interact with ownCloud Infinite Scale (oCIS) storage using PHP. + +The source code is available at link:https://github.com/owncloud/ocis-php-sdk[github.com/owncloud/ocis-php-sdk]. + +== Installation via Composer + +Add `owncloud/ocis-php-sdk` to the require block in your `composer.json` and then run `composer install`. + +[WARNING] +==== +The ocis-php-sdk currently relies on a development version of the `owncloud/libre-graph-api-php` package. To ensure proper dependency resolution, it is necessary to set `"minimum-stability": "dev"` and `"prefer-stable": true` in your `composer.json` file. +==== + +[source,json] +---- +{ + "minimum-stability": "dev", + "prefer-stable": true, + "require": { + "owncloud/ocis-php-sdk": "^1.0" + } +} +---- + +Alternatively, you can run the following commands from the command line: + +[source,bash] +---- +composer config minimum-stability dev +composer config prefer-stable true +composer require owncloud/ocis-php-sdk +---- + +== Getting Started + +oCIS has two types of access token: + +- An OIDC access token used to interact with drives, groups, users, shares, etc. +- An education access token used to interact with education endpoints. + +Create an `Ocis` object using the service URL and an OIDC access token: + +[source,php] +---- +$ocis = new Ocis('https://example.ocis.com', $accessToken); +---- + +Or create an `Ocis` object to interact with education endpoints using the service URL and an education access token: + +[source,php] +---- +$ocis = new Ocis('https://education.ocis.com', null, [], $educationAccessToken); +---- + +At least one access token must be provided to use the SDK. + +Acquiring an OIDC access token is out of scope of this SDK, but you can find xref:#acquiring-an-access-token[examples for that below]. Refreshing OIDC tokens is also not part of the SDK, but after you receive a new token, you can update the `Ocis` object: + +[source,php] +---- +$ocis->setAccessToken($newAccessToken); +---- + +The education access token is set when starting the oCIS graph server. You can retrieve it using the following snippet: + +[source,php] +---- +$educationAccessToken = getenv("GRAPH_HTTP_API_TOKEN") +---- + +== Drives (Spaces) + +Drives can be listed using the `getAllDrives` or the `getMyDrives` method. The `Drive` class is responsible for most file and folder related actions, such as listing files, creating folders, and uploading files. + +[source,php] +---- +// get the personal drive of the authorized user +// `getMyDrives` returns all drives that the user is a member of +// but in this example the result is filtered to only return +// the personal drive (parameter 3 = DriveType::PERSONAL) +$drives = $ocis->getMyDrives( + DriveOrder::NAME, + OrderDirection::ASC, + DriveType::PERSONAL +); + +// get the drive id +$id = $drives[0]->getId(); + +// get the name of the drive +$name = $drives[0]->getName(); + +// get a link to the drive that can be opened in a browser and will lead the user to the web interface +$webUrl = $drives[0]->getWebUrl(); + +// create a folder inside the drive +$drives[0]->createFolder("/documents"); + +// upload a file to the drive +$drives[0]->uploadFile("/documents/myfile.txt", "Hello World!"); + +// get an array of all resources of the "/documents" folder inside the drive +$resources = $drives[0]->getResources("/documents"); +---- + +=== Drive Permission + +Users and groups can be invited to drives by specifying permissions. The `Drive` class has methods to invite users and groups, update permission roles and expiration dates, remove users and groups, etc. Drive invitations are only possible on project drives. + +[source,php] +---- +// find all users with a specific surname +$users = $ocis->getUsers("einstein")[0]; + +// get all drives of type project +$drives = $ocis->getMyDrives( + DriveOrder::NAME, + OrderDirection::ASC, + DriveType::PROJECT +); + +// get the drive named 'game' +foreach ($drives as $drive) { + if ($drive->getName() === 'game') { + $gameDrive = $drive; + break; + } +} + +// get all roles that are possible for that drive +$driveRoles = $gameDrive->getRoles(); + +// get the role that is allowed to view, download, upload, edit, add, delete and manage members +foreach ($driveRoles as $role) { + if ($role->getDisplayName() === 'Manager') { + $managerRole = $role; + break; + } +} + +// invite user einstein on project drive 'game' with manager permission +$gameDrive->invite($users, $managerRole); +---- + +== Notifications + +Notifications can be listed using the `getNotifications` method, which returns an array of `Notification` objects representing all active notifications. The `Notification` object can retrieve details of the corresponding notification and mark it as read (delete it). + +== Sharing + +Given the correct permissions, an `OcisResource` can be shared with a group or a user. To define the access permissions of the receiver, every share must set `SharingRole`(s). + +[source,php] +---- +// get the resources of a subfolder inside a drive +$resources = $drive->getResources("/documents"); + +// get all roles that are possible for that particular resource +$roles = $resources[0]->getRoles(); + +// find the role that is allowed to read and write the shared file or folder +foreach ($roles as $role) { + if ($role->getDisplayName() === 'Can edit') { + $editorRole = $role; + break; + } +} + +// find all users with a specific surname +$users = $ocis->getUsers("einstein")[0]; + +// share the resource with the users +$resources[0]->invite($users, $editorRole); +---- + +== Education User + +Education users can only be created, listed, and deleted using an education access token. If you want to use other APIs, you need to use the OIDC access token. + +[source,php] +---- +// create education user +$educationUsers = $ocis->createEducationUser() +// list all education users +$educationUsers = $ocis->getEducationUsers() +// list education user by id +$educationUsers = $ocis->getEducationUserById() +// delete education user +$educationUser[0]->delete() +---- + +== Requirements + +- PHP 8.1 or higher +- oCIS 5.0.0 or higher + +== Acquiring an Access Token + +For an easier experience in acquiring an access token, several PHP OIDC client libraries are available. The following code snippet showcases how to retrieve an access token with the link:https://github.com/facile-it/php-openid-client[facile-it/php-openid-client] library. + +=== Install PHP Dependencies + +Install the required libraries using Composer: + +[source,bash] +---- +composer require facile-it/php-openid-client +composer require nyholm/psr7 +---- + +=== Required PHP Libraries + +The following PHP extensions are required: + +- `php-bcmath` +- `php-gmp` + +=== Code Snippet to Fetch an Access Token + +[source,php] +---- +build('https://example.ocis.com'); + +$clientMetadata = ClientMetadata::fromArray([ + 'client_id' => 'client_id', + 'client_secret' => 'client_secret', + 'redirect_uris' => [ + 'http://url-of-this-file', + ], +]); +$client = (new ClientBuilder()) + ->setIssuer($issuer) + ->setClientMetadata($clientMetadata) + ->build(); + +$authorizationService = (new AuthorizationServiceBuilder())->build(); +$redirectAuthorizationUri = $authorizationService->getAuthorizationUri( + $client, + ['scope' => 'openid profile email offline_access'] +); + +if (!isset($_REQUEST["code"])) { + header('Location: ' . $redirectAuthorizationUri); +} else { + $serverRequest = new ServerRequest('GET', $_SERVER['REQUEST_URI']); + + $callbackParams = $authorizationService->getCallbackParams($serverRequest, $client); + + $tokenSet = $authorizationService->callback($client, $callbackParams); + + // store access and refresh token in database + $accessToken = $tokenSet->getAccessToken(); + echo 'AccessToken : ' . $accessToken; + echo '
'; + + $refreshToken = $tokenSet->getRefreshToken(); + echo 'RefreshToken : ' . $refreshToken; + echo '
'; + + // use this code to get a new access token when expired + $tokenSet = $authorizationService->refresh($client, $refreshToken); + $accessToken = $tokenSet->getAccessToken(); + echo 'NewAccessToken : ' . $accessToken; +} +---- + +To test, open a browser and navigate to the URL of this file. + +If you're working in a development environment where you need to bypass SSL verification (not recommended for production): + +[source,php] +---- + false, + 'verify_host' => false +]); +$httpClient = new Psr18Client($symHttpClient); + +$metadataProviderBuilder = (new MetadataProviderBuilder()) + ->setHttpClient($httpClient); +$jwksProviderBuilder = (new JwksProviderBuilder()) + ->setHttpClient($httpClient); +$issuer = (new IssuerBuilder()) + ->setMetadataProviderBuilder($metadataProviderBuilder) + ->setJwksProviderBuilder($jwksProviderBuilder) + ->build('https://example.ocis.com'); +// ... +$client = (new ClientBuilder()) + ->setHttpClient($httpClient) + ->setIssuer($issuer) + ->setClientMetadata($clientMetadata) + ->build(); + +$authorizationService = (new AuthorizationServiceBuilder()) + ->setHttpClient($httpClient) + ->build(); +// ... +---- + +== Development + +=== Integration Tests + +To run the tests locally: + +. Install and set up Docker (minimum version 24) and Docker Compose (minimum version 2.21). + +. Ensure that the following PHP extensions are installed for executing the integration tests: ++ +[source] +---- +php-curl +php-dom +php-phpdbg +php-mbstring +php-ast +---- + +. Add these lines to your `/etc/hosts` file: ++ +[source] +---- +127.0.0.1 ocis.owncloud.test +127.0.0.1 keycloak.owncloud.test +---- + +. Run the whole test suite (this starts a full oCIS server with Keycloak and other services using Docker): ++ +[source,bash] +---- +make test-php-integration +---- + +. Run a single test (this starts a full oCIS server with Keycloak and other services using Docker): ++ +[source,bash] +---- +make test-php-integration testGetResources +---- + +If something goes wrong, use `make clean` to clean up the created containers and volumes. diff --git a/content/ocis/9.0/modules/developer/pages/clients/rclone/index.adoc b/content/ocis/9.0/modules/developer/pages/clients/rclone/index.adoc new file mode 100644 index 00000000..cf816111 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/rclone/index.adoc @@ -0,0 +1,16 @@ += Rclone +:toc: right +:toclevels: 3 + +== About Rclone + +[TIP] +==== +Rclone is a command line program to manage files on cloud storage. It is a feature rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols. + +Rclone has powerful cloud equivalents to the unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm, and cat. Rclone's familiar syntax includes shell pipeline support, and --dry-run protection. It is used at the command line, in scripts or via its API. + +Users call rclone "The Swiss army knife of cloud storage", and "Technology indistinguishable from magic". +==== + +Source: link:https://rclone.org/[Rclone project website] diff --git a/content/ocis/9.0/modules/developer/pages/clients/rclone/webdav-sync-basic-auth.adoc b/content/ocis/9.0/modules/developer/pages/clients/rclone/webdav-sync-basic-auth.adoc new file mode 100644 index 00000000..952ff837 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/rclone/webdav-sync-basic-auth.adoc @@ -0,0 +1,43 @@ += WebDAV with Basic Authentication +:toc: right +:toclevels: 3 + +== WebDAV with Basic Authentication + +[CAUTION] +==== +Basic Authentication is disabled by default in oCIS because of security considerations. In order to make the following Rclone commands work the oCIS administrator needs to enable Basic Authentication e.g. by setting the environment variable `PROXY_ENABLE_BASIC_AUTH` to `true`. + +Please consider to use xref:clients/rclone/webdav-sync-oidc.adoc[Rclone with OpenID Connect] instead. +==== + +For the usage of a WebDAV remote with Rclone see also the link:https://rclone.org/webdav/[Rclone documentation] + +== Configure the WebDAV Remote + +First of all we need to set up our credentials and the WebDAV remote for Rclone. In this example we do this by setting environment variables. You might also set up a named remote or use command line options to achieve the same. + +[source,bash] +---- +export RCLONE_WEBDAV_VENDOR=owncloud +export RCLONE_WEBDAV_URL=https://ocis.owncloud.test/remote.php/webdav/ +export RCLONE_WEBDAV_USER=einstein +export RCLONE_WEBDAV_PASS=$(rclone obscure relativity) +---- + +[NOTE] +==== +Please note that `RCLONE_WEBDAV_PASS` is not set to the actual password, but to the value returned by `rclone obscure `. +==== + +We now can use Rclone to sync the local folder `/tmp/test` to `/test` in your oCIS home folder. + + +=== Sync to the WebDAV Remote + +[source,bash] +---- +rclone sync :local:/tmp :webdav:/test +---- + +If your oCIS doesn't use valid SSL certificates, you may need to use `rclone --no-check-certificate sync ...`. diff --git a/content/ocis/9.0/modules/developer/pages/clients/rclone/webdav-sync-oidc.adoc b/content/ocis/9.0/modules/developer/pages/clients/rclone/webdav-sync-oidc.adoc new file mode 100644 index 00000000..f2dde03f --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/rclone/webdav-sync-oidc.adoc @@ -0,0 +1,71 @@ += WebDAV with OpenID Connect +:toc: right +:toclevels: 3 + +== WebDAV with OpenID Connect + +Rclone itself is not able to open and maintain an OpenID Connect session. But it is able to still use OpenID Connect for authentication by leveraging a so called OIDC-agent. + +=== Setting Up the OIDC-agent + +You need to install the link:https://github.com/indigo-dc/oidc-agent[OIDC-agent] from your OS' package repository (e.g. link:https://github.com/indigo-dc/oidc-agent#debian-packages[Debian] or link:https://github.com/indigo-dc/oidc-agent#debian-packages[MacOS]). + +=== Configuring the OIDC-agent + +Run the following command to add a OpenID Connect profile to your OIDC-agent. It will open the login page of OpenID Connect identity provider where you need to log in if you don't have an active session. + +[source,bash] +---- +oidc-gen \ + --client-id=oidc-agent \ + --client-secret="" \ + --pub \ + --issuer https://ocis.owncloud.test \ + --redirect-uri=http://localhost:12345 \ + --scope max \ + einstein-ocis-owncloud-test +---- + +If you have dynamic client registration enabled on your OpenID Connect identity provider, you can skip the `--client-id`, `--client-secret` and `--pub` options. + +If you're using a dedicated OpenID Connect client for the OIDC-agent, we recommend a public one with the following two redirect URIs: `http://127.0.0.1:_` and `http://localhost:_`. Alternatively you also may use the already existing OIDC client of the ownCloud Desktop Client (`--client-id=xdXOt13JKxym1B1QcEncf2XDkLAexMBFwiT9j6EfhhHFJhs2KM9jbjTmf8JBXE69` and `--client-secret=UBntmLjC2yYCeHwsyj73Uwo9TAaecAetRwMw0xYcvNL9yRdLSUi0hUAHfvCHFeFh`, no `--pub` set, request specific scope for offline access), e.g.: + +[source,bash] +---- +oidc-gen \ + --client-id=xdXOt13JKxym1B1QcEncf2XDkLAexMBFwiT9j6EfhhHFJhs2KM9jbjTmf8JBXE69 \ + --client-secret=UBntmLjC2yYCeHwsyj73Uwo9TAaecAetRwMw0xYcvNL9yRdLSUi0hUAHfvCHFeFh \ + --issuer https://cloud.ocis.test \ + --redirect-uri=http://localhost:12345 \ + --scope="openid offline_access profile email" \ + my-client +---- + +When using a self signed certificate you have to provide the certificate chain using `--cp /etc/ssl/certs/test.cert.pem`. In case oidc-gen cannot determine the flow try with `--flow=code`. + +Please also note that the OIDC-agent will listen on your localhost interface on port 12345 for the time of the initial authentication. If that port is already occupied on your machine, you can easily change that by setting the `--redirect-uri` parameter to a different value. + +After a successful login or an already existing session you will be redirected to success page of the OIDC-agent. +You will now be asked for a password for your account configuration, so that your OIDC session is secured and cannot be used by other people with access to your computer. + +== Configure the WebDAV Remote + +First of all we need to set up our credentials and the WebDAV remote for Rclone. In this example we do this by setting environment variables. You might also set up a named remote or use command line options to achieve the same. + +[source,bash] +---- +export RCLONE_WEBDAV_VENDOR=owncloud +export RCLONE_WEBDAV_URL=https://ocis.owncloud.test/remote.php/webdav/ +export RCLONE_WEBDAV_BEARER_TOKEN_COMMAND="oidc-token einstein-ocis-owncloud-test" +---- + +=== Sync to the WebDAV Remote + +We now can use Rclone to sync the local folder `/tmp/test` to `/test` in your oCIS home folder. + +[source,bash] +---- +rclone sync :local:/tmp :webdav:/test +---- + +If your oCIS doesn't use valid SSL certificates, you may need to use `rclone --no-check-certificate sync ...`. diff --git a/content/ocis/9.0/modules/developer/pages/clients/web/backend/backend-ocis.adoc b/content/ocis/9.0/modules/developer/pages/clients/web/backend/backend-ocis.adoc new file mode 100644 index 00000000..6876a429 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/web/backend/backend-ocis.adoc @@ -0,0 +1,37 @@ += Setup With oCIS +:toc: right +:toclevels: 3 + +== Setting Up Web + +* Clone the link:https://github.com/owncloud/ocis/[repository]. +* Initally install all dependencies by running `pnpm install`. +* Copy `./web/config/config.json.sample-ocis` to `./web/config/config.json` and adjust values if required. + +== Running Web + +* Start bundling web with a watcher by running `pnpm build:w` + +== Setting Up oCIS + +* Setup oCIS by following the xref:ocis/getting-started/index.adoc/[setup instructions] +* Start oCIS with local links to your bundled web frontend and config by running the following command, adjust the paths as necessary: ++ +[source,bash] +---- +WEB_ASSET_CORE_PATH=../../web/dist \ +WEB_UI_CONFIG_FILE=../../web/dist/config.json \ +OCIS_INSECURE=true \ +IDM_CREATE_DEMO_USERS=true \ +./bin/ocis server +---- + +== Start oCIS + +* Open `\https://localhost:9200` and accept the certificate. +* When signing in, use one of the xref:ocis/getting-started//demo-users.adoc[available demo users]. +* Whenever code changes are made, you need to manually reload the browser page (no hot reload). + +== Running Tests + +For testing, please refer to the xref:clients/web/testing/index.adoc[testing docs]. diff --git a/content/ocis/9.0/modules/developer/pages/clients/web/building/building.adoc b/content/ocis/9.0/modules/developer/pages/clients/web/building/building.adoc new file mode 100644 index 00000000..57b5370c --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/web/building/building.adoc @@ -0,0 +1,12 @@ += Building From Source +:toc: right +:toclevels: 3 + +== Building ownCloud Web + +* Run `pnpm install` to install dependencies +* Run `pnpm build` to build Web and all apps included in the `packages` folder + +=== Updating Dependencies + +* *Do not* update dependencies by yourself but use the provided ones by the repo. When rebasing your local clone, you can always run `pnpm install` to be in sync with the defined dependencies. diff --git a/content/ocis/9.0/modules/developer/pages/clients/web/development/conventions.adoc b/content/ocis/9.0/modules/developer/pages/clients/web/development/conventions.adoc new file mode 100644 index 00000000..69757e44 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/web/development/conventions.adoc @@ -0,0 +1,47 @@ += Conventions +:toc: right +:toclevels: 3 + +This is a collection of tips and conventions to follow when working on the link:https://github.com/owncloud/ocis[ownCloud web frontend]. Since it is a living document, please open a PR if you find something missing. + +== Contributing to ownCloud Web + +Everyone is invited to contribute. Simply fork the link:https://github.com/owncloud/ocis/[codebase], check the link:https://github.com/owncloud/ocis/issues?q=is%3Aopen+is%3Aissue+label%3ATopic%3Agood-first-issue[issues] for a suitable one and open a pull request! + +=== Linting and Tests + +To make sure your pull request can be efficiently reviewed and won't need a lot of changes down the road, please run the linter and the unit tests via `pnpm lint --fix` and `pnpm test:unit` locally. `CI` will run on pull requests and report back any problems after that. For a further introduction on how we handle testing, please head to the xref:clients/web/testing/index.adoc[testing docs]. + +=== Changelog Items + +In our project, we follow link:https://semver.org/[SemVer] and keep a changelog for every change that influences the user experience (where "users" can be admins, end-users and developers). Some changes, like refactoring, updating dependencies, writing documentation or adding tests don't require a changelog item. + +Please add a changelog item to the `changelog/unreleased/` folder, referencing the issue and pull request numbers, following the link:https://github.com/owncloud/ocis/blob/master/changelog/TEMPLATE[changelog item template]. + +== Code Conventions + +=== Early Returns + +We're trying to stick with early returns in our code to make it more performant and simpler to reason about it. + +=== Translations + +Use the `v-text` directive in combination with `$gettext` (or a variation of it) inside HTML tags (instead of +a `` or similar) in order to make reasoning about the DOM tree easier. + +=== TypeScript + +We're using TypeScript, which allows us to catch bugs at transpile time. Clean types make sure our IDEs can support us +in reasoning about our (ever growing, complex) codebase. + +=== Vue 3 and Composition API + +We've migrated from Vue 2 to Vue 3 late in 2022 and since then have been investing continuous efforts to move away from the Vue options API +in favor of the Vue composition API. The `web-pkg` helper package provides quite some composables which will help you in +app & extension development, so we encourage you to make use of the Vue composition API as well, even outside of the +ownCloud Web repository. + +=== Dependencies + +To keep the bundle size small and reduce the risk of introducing security problems for our users, we try to limit +the amount of dependencies in our code base and keep them as up-to-date as possible. diff --git a/content/ocis/9.0/modules/developer/pages/clients/web/development/index.adoc b/content/ocis/9.0/modules/developer/pages/clients/web/development/index.adoc new file mode 100644 index 00000000..7e6cae3a --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/web/development/index.adoc @@ -0,0 +1,7 @@ += Development +:toc: right +:toclevels: 3 + +This section is a guide about the development of ownCloud Web *core*, *apps* and *extensions*. This includes *tooling*, *conventions* and the *repo structure*. + +It is of interest for you if you want to contribute to ownCloud Web or develop your own apps and extensions. diff --git a/content/ocis/9.0/modules/developer/pages/clients/web/development/repo-structure.adoc b/content/ocis/9.0/modules/developer/pages/clients/web/development/repo-structure.adoc new file mode 100644 index 00000000..c2c6e2f9 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/web/development/repo-structure.adoc @@ -0,0 +1,110 @@ += Repo Structure and (Published) Packages +:toc: right +:toclevels: 3 + +== Repository Structure + +From a developer's perspective, the most important parts of the link:https://github.com/owncloud/ocis[ownCloud ocis repo] are the following files and folders: + +[NOTE] +=== +All mentioned folders are a subfolder inside the web/ directory of the ocis repo. +=== + +=== dev Folder and docker-compose.yml File + +The `/dev` folder contains all the configuration files that are needed in the `docker-compose.yml` file. This docker compose stack contains all the backend and testing related infrastructure that is needed for an out-of-the-box usable localhost development setup, as described in the xref:clients/web/development/tooling.adoc[tooling section]. + +=== packages Folder + +The `packages/` folder contains a variety of packages. Some of them get published to link:https://npmjs.com[npmjs.com], others define the core packages, apps and extensions that are the foundation of the `ownCloud Web` release artifact. + +Having these packages side by side within the `/packages` folder of the repo is possible because of a `pnpm` feature called `Workspaces`. You can learn more about that by visiting the link:https://pnpm.io/workspaces[pnpm docs]. + +=== tests Folder + +We're using the link:https://playwright.dev[Playwright] for UI testing. The UI tests are located in the `/tests/e2e` + +You're more than welcome to make a pull request and adjust this section of the docs accordingly. You can read more about testing in our xref:clients/web/testing/index.adoc[testing section] + +=== package.json File + +This is probably no surprise: the root level `package.json` file defines the project information, build scripts, dependencies and some more details. Each package in `/packages` can and most likely will contain another `package.json` which does the same for the respective package. + +=== vite.config.ts + +We're working with link:https://vite.dev[Vite] as a local development server and build tool. `vite.config.ts` is the main configuration file for that. You can read more about the usage in our xref:clients/web/development/tooling.adoc[tooling section]. + +== (Published) Packages + +Each package in the `/packages` folder can - not exclusively, but most commonly - consist of + +* source code (`/src`), +* unit tests (`/tests`), +* translations (`/l10n`) and +* a `package.json` file for package specific details and dependencies. + +=== Code Style and Build Config + +Some of our packages in `/packages` are pure helper packages which ensure a common code style and build configuration for all our internal (mono repo) and external packages. We encourage you to make use of the very same packages. This helps the community understand code more easily, even when coming from different developers or vendors in the ownCloud Web ecosystem. + +Namely those packages are + +* `/packages/babel-preset` +* `/packages/eslint-config` +* `/packages/extension-sdk` +* `/packages/prettier-config` +* `/packages/tsconfig/` + +=== ownCloud Design System + +The ownCloud Design System (`/packages/design-system`) is a collection of components, design tokens and styles which ensure a unique and consistent look and feel and branding throughout the ownCloud Web ecosystem. We hope that you use it, too, so that your very own apps and extensions will blend in with all the others. Documentation and code examples can be found in the link:https://owncloud.design[design system documentation]. + +The ownCloud Design System is a standalone project, but to make development easier we have the code in our mono repo. We're planning to publish it on npmjs.com as link:https://www.npmjs.com/package/@ownclouders/design-system[@ownclouders/design-system] as soon as possible. Since it's bundled with ownCloud Web, you should not bundle it with your app or extension. + +=== web-client + +The client package (`/packages/web-client`) serves as an abstraction layer for the various ownCloud APIs, like xref:apis/http/graph/[LibreGraph], xref:server:developer_manual:webdav_api/index.adoc[WebDAV] and xref:server:developer_manual:core/apis/ocs-capabilities.adoc[OCS]. The package provides TypeScript interfaces for various entities (like files, folders, shares and spaces) and makes sure that raw API responses are properly transformed so that you can deal with more useful objects. The web-client package gets published on npmjs.com as link:https://www.npmjs.com/package/@ownclouders/web-client[@ownclouders/web-client]. + +Dedicated documentation for the `web-client` package is not available, yet, since our extension system is still work in progress. However, the package's link:https://github.com/owncloud/ocis/blob/master/web/packages/web-client/README.md[README.md] gives you a few examples on how to use it. + +=== web-pkg + +The web-pkg package (`/packages/web-pkg`) is a collection of opinionated components, composables, types and other helpers that aim +at making your app and extension developer experience as easy and seamless as possible. The web-pkg package gets published on +npmjs.com as link:https://www.npmjs.com/package/@ownclouders/web-pkg[@ownclouders/web-pkg]. + +Dedicated documentation for the `web-pkg` package is not available, yet, since our extension system is still work in progress. + +=== web-runtime + +At the very heart of ownCloud Web, the `web-runtime` is responsible for dependency injection, app bootstrapping, configuration, +authentication, data preloading and much more. +It is very likely that you will never get in touch with it as most of the developer-facing features are exposed via `web-pkg`. If you +have more questions about this package, please join our link:https://matrix.to/#/#ocis:matrix.org[public chat] and simply ask +or write an issue in our link:https://github.com/owncloud/ocis/issues[issue tracker]. + +=== Standalone Core Apps + +Both `web-app-admin-settings` and `web-app-files` are standalone apps which are bundled with the default ownCloud Web release artifact. + +=== Viewer and Editor Apps + +Apps which fall into the categories `viewer` or `editor` can be opened from the context of a file or folder. This mostly happens from +within the `files` app. We currently bundle the following apps with the default ownCloud Web release artifact: + +* `web-app-epub-reader` + +a simple reader for `.epub` files +* `web-app-external` + +an iframe integration of all the apps coming from the xref:{dev-s-path}/app-provider/app-provider.adoc[app provider] (e.g. OnlyOffice, Collabora Online and others) +* `web-app-pdf-viewer` + +a viewer for `.pdf` files, which relies on native PDF rendering support from the browser +* `web-app-preview` + +a viewer for various media files (audio / video / image formats) +* `web-app-text-editor` a simple editor for `.txt`, `.md` and other plain text files + +If you're interested in writing your own viewer or editor app for certain file types, please get in touch with us for more info. + +=== Testing + +Additional unit testing code lives in `test-helpers`. diff --git a/content/ocis/9.0/modules/developer/pages/clients/web/development/tooling.adoc b/content/ocis/9.0/modules/developer/pages/clients/web/development/tooling.adoc new file mode 100644 index 00000000..29b3f7f8 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/web/development/tooling.adoc @@ -0,0 +1,38 @@ += Tooling +:toc: right +:toclevels: 3 + +== Packaging + +Web is using link:https://pnpm.io/[pnpm] as package manager and link:https://vite.dev/[vite] as build tool. The latter is built on top of link:https://rollupjs.org/[rollup] and brings some additional features such as instant hot-reloading. + +== Development Setup + +=== Prerequisites + +* docker +* docker-compose (if not already included in your docker installation) +* pnpm +* node + +If you’re not using Docker Desktop, you might have to modify your `/etc/hosts` and add `127.0.0.1 host.docker.internal` to make `host.docker.internal` links work. + +=== Installing Dependencies + +After cloning the source code, install the dependencies via `pnpm install`. + +=== Starting the Server + +You can start the server by running `docker-compose up ocis`. + +Note that the container needs a short while to start because it is waiting for `tika` to be initialized. This is the case as soon as the `tika-service` container has stopped running. + +=== Building and Accessing Web + +After the docker containers are running (and `tika` is being initialized), run `pnpm build:w` to build Web. This also includes hot-reloading after changes you make, although it will take a while to rebuild the project. See down below for some details on how to enable instant hot-reloading. + +Now you can access Web via https://host.docker.internal:9200. + +=== Using Instant Hot-Reload via Vite + +To work with instant hot-reloading, you can also build Web by running `pnpm vite`. The port to access Web is slightly different then: https://host.docker.internal:9201. Also note that the initial page load may take a bit longer than usual. This is normal and to be expected. diff --git a/content/ocis/9.0/modules/developer/pages/clients/web/embed-mode/index.adoc b/content/ocis/9.0/modules/developer/pages/clients/web/embed-mode/index.adoc new file mode 100644 index 00000000..a0912000 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/web/embed-mode/index.adoc @@ -0,0 +1,174 @@ += Embed Mode +:toc: right +:toclevels: 3 + +The ownCloud Web can be consumed by another application in a stripped down version called "Embed mode". This mode is supposed to be used in the context of selecting or sharing resources. + +== Getting Started + +To integrate ownCloud Web into your application, add an iframe element pointing to your ownCloud Web deployed instance with additional query parameter `embed=true`. + +[source,html] +---- + +---- + +== Communication + +To establish seamless cross-origin communication between the embedded instance and the parent application, our approach involves emitting events using the `postMessage` method. These events can be conveniently captured by utilizing the standard `window.addEventListener('message', listener)` pattern. + +=== Target Origin + +By default, the `postMessage` method does not specify the `targetOrigin` parameter. However, it is recommended best practice to explicitly pass in the URI of the iframe origin (not the parent application). To enhance security, you can specify this value by modifying the config option `options.embed.messagesOrigin`. + +=== Events + +To maintain uniformity and ease of handling, each event encapsulates the same structure within its payload: `{ name: string, data: any }`. + +[cols="1,1,1",options="header"] +|=== +| Name +| Data +| Description + +| *owncloud-embed:select* +| Resource[] +| Gets emitted when user selects resources or location via the select action + +| *owncloud-embed:share* +| string[] +| *DEPRECATED*: Gets emitted when user selects resources and shares them via the "Share links" action. Use `owncloud-embed:share-links` instead. + +| *owncloud-embed:share-links* +| Array<{ url: string; password?: string }> +| Gets emitted when user selects resources and shares them via the "Share link(s)" or "Share link(s) and password(s)" action. Each object contains the link URL and optionally the password (when shared with password). + +| *owncloud-embed:cancel* +| null +| Gets emitted when user attempts to close the embedded instance via "Cancel" action +|=== + +=== Example + +==== Selecting Resources + +[source,html] +---- + + + +---- + +==== Sharing Links with Password + +[source,html] +---- + + + +---- + +== Location Picker + +By default, the Embed mode allows users to select resources. In certain cases (e.g. uploading a file), this needs to be changed to allow selecting a location. This can be achieved by running the embed mode with additional parameter `embed-target=location`. With this parameter, resource selection is disabled and the selected resources array always includes the current folder as the only item. In special scenarios you also want the user to set a file name, this can be achieved by adding the `embed-choose-file-name=true` parameter, or if you also want to set a default file name, you can use `embed-choose-file-name-suggestion=my file.text`. + +=== Example + +[source,html] +---- + + + +---- + +== File Picker + +The File Picker mode in ownCloud Web is designed for embedding an interface that allows users to pick a single file. This mode can be configured to restrict the file types that users can select. To enable the File Picker mode, you need to include the embed-target=file query parameter in the iframe URL. Furthermore, you can specify allowed file types using the embed-file-types parameter. The file types can be specified using file extensions, MIME types, or a combination of both. If the embed-file-types parameter is not provided, all file types will be selectable by default. + +=== Example + +[source,html] +---- + + + + +---- + +== Delegate Authentication + +If you already have a valid `access_token` that can be used to call the API from within the Embed mode and do not want to force the user to authenticate again, you can delegate the authentication. Delegating authentication will disable internal login form in ownCloud Web and will instead use events to obtain the token and update it. + +=== Configuration + +To allow authentication delegation, you need to set the config option `options.embed.delegateAuthentication` to `true`. This can be achieved via query parameter `embed-delegate-authentication=true`. Because we are using the `postMessage` method to communicate across different origins, it is best practice to verify that the event originated from a known origin and not from some malicious site. We highly recommend to allow this check in production environments. You can enable it by setting the config option `options.embed.delegateAuthenticationOrigin` via query parameter `embed-delegate-authentication-origin=my-origin`. The value of this parameter will be compared against the `MessageEvent.origin` value and if they do not match, the token will be rejected. + +=== Events + +==== Opening Embed Mode + +As already mentioned, we're using the `postMessage` method to allow communication between the Embed mode and the parent application. When the Embed mode is opened for the first time, the user gets redirected to the `/web-oidc-callback` page where a message with payload `{ name: 'owncloud-embed:request-token', data: undefined }` is sent to request the `access_token` from the parent application. The parent application should set an event listener before opening the Embed mode and once received, it should send a message with payload `{ name: 'owncloud-embed:update-token', data: { access_token: '' } }`. Once the Embed mode receives this message, it will save the token in the application state and will automatically authenticate the user. + +[NOTE] +==== +When passing the token in the message payload, use only the token itself without `Bearer ` string as that will be added automatically in the Embed mode. +==== + +[NOTE] +==== +To save unnecessary duplication of messages with only different names, the name in the message payload above is exactly the same for both the initial authentication and subsequent token updates after renewal. +==== + +==== Updating the Token + +When authentication is delegated, the automatic renewal of the token inside of ownCloud Web is disabled. In order to update the token, a listener is created which awaits a message with payload `{ name: 'owncloud-embed:update-token', data: { access_token: '' } }`. The token will then be replaced inside of the Embed mode automatically. diff --git a/content/ocis/9.0/modules/developer/pages/clients/web/extension-system/extension-types/actions.adoc b/content/ocis/9.0/modules/developer/pages/clients/web/extension-system/extension-types/actions.adoc new file mode 100644 index 00000000..53f1f9c8 --- /dev/null +++ b/content/ocis/9.0/modules/developer/pages/clients/web/extension-system/extension-types/actions.adoc @@ -0,0 +1,91 @@ += Action Extensions +:toc: right +:toclevels: 3 + +== Action Extension Type + +Actions are one of the possible extension types. Registered actions get rendered in various places across the UI, depending on their scope and targets. + +=== Configuration + +This is what the `ActionExtension` interface looks like: + +[source,typescript] +---- +interface ActionExtension { + id: string + type: 'action' + extensionPointIds?: string[] + action: Action // Please check the Action section below +} +---- + +For `id`, `type`, and `extensionPointIds`, please see xref:clients/web/extension-system/index.adoc#extension-base-configuration[extension base section] in top level docs. + +==== Action + +The most important configuration options are: + +* `icon` - The icon to be displayed, can be picked from https://owncloud.design/#/Design%20Tokens/IconList +* `name` - The name of the action (not displayed in the UI) +* `label` - The text to be displayed +* `route` - The string/route to navigate to. The nav item will be a `` tag. +* `href` - The URL to navigate to. The nav item will be a ``tag. +* `handler` - The action to perform upon click. The nav item will be a `