diff --git a/antora.yml b/antora.yml index a3efc701..14f5a87c 100644 --- a/antora.yml +++ b/antora.yml @@ -4,6 +4,7 @@ version: 'next' start_page: ROOT:index.adoc nav: - modules/ROOT/partials/nav.adoc +- modules/developer/partials/nav.adoc asciidoc: attributes: diff --git a/modules/developer/images/ocis/architecture-overview.drawio.svg b/modules/developer/images/ocis/architecture-overview.drawio.svg new file mode 100644 index 00000000..19102f81 --- /dev/null +++ b/modules/developer/images/ocis/architecture-overview.drawio.svg @@ -0,0 +1,4 @@ + + + +
Storage
Storage
identity management
identity management
reva
reva
proxy
reverse proxy
+ api gateway
proxy...
users
users
idp
konnectd / keycloak
idp...
ldap
AD, openldap, idm
ldap...
web
(aka phoenix)
web...
graph
graph
graph-explorer
graph-explorer
hello
hello
onlyoffice
onlyoffice
settings
settings
store
store
idm
idm
OCIS
OCIS
ocs
ocs
webdav
webdav
registry
registry
authbasic
authbasic
authbearer
authbearer
authprovider
(basic)
authprovider...
authprovider
(oidc)
authprovider...
userprovider
userprovider
gateway
gateway
gateway
gateway
authregistry
authregistry
storageregistry
storageregistry
frontend
frontend
ocs
ocs
ocdav
ocdav
ocm
ocm
datagateway
datagateway
storagehome
storagehome
storageprovider
storageprovider
dataprovider
dataprovider
storageusers
storageusers
storageprovider
storageprovider
dataprovider
dataprovider
storagemeta
storagemeta
storageprovider
storageprovider
dataprovider
dataprovider
storagepubliclink
storagepubliclink
publicstorageprovider
publicstorageprovider
authprovider
(publicshares)
authprovider...
sharing
sharing
usershareprovider
usershareprovider
publicshareprovider
publicshareprovider

single host

- all services run on a single host
- no redundancy
- minimize resource usage
- home setup
single host...
Text is not SVG - cannot display
\ No newline at end of file diff --git a/modules/developer/images/ocis/cs3org.png b/modules/developer/images/ocis/cs3org.png new file mode 100644 index 00000000..6f06e216 Binary files /dev/null and b/modules/developer/images/ocis/cs3org.png differ diff --git a/modules/developer/images/ocis/deployment/ocis_and_wopi_drawio.svg b/modules/developer/images/ocis/deployment/ocis_and_wopi_drawio.svg new file mode 100644 index 00000000..2ceaeddf --- /dev/null +++ b/modules/developer/images/ocis/deployment/ocis_and_wopi_drawio.svg @@ -0,0 +1,4 @@ + + + +
cs3api
cs3api
app registry
app registry
/app/list
/app/open
/app/list...
cs3api
/listapps
/openapp
cs3api...
frontend
frontend
cs3api
/registerapp
cs3api...
GET /hosting/discovery 
GET /hosting/discovery 
/iop/open...
/iop/open...
app provider
(per web office)
app provider...
graph
webdav
ocs
graph...
/space1/foo.docx
/space1/foo.docx
User
User
/space2/bar.docx
/space2/bar.docx
fileid c
fileid c
Web Office
Web Office
Web Office
Web Office
Web Office
Web Office
cs3org/wopiserver
cs3org/wopiserver
cs3org/wopiserver
cs3org/wopiserver
wopi
wopi
fileid a
fileid a
fileid b
fileid b
User
User
Collabora:
for collaborative edit sessions
a sticky route by fileid
(from wopiSrc) is needed

OnlyOffice:
automatic handling
Collabora:...
inter services communication
inter services communication
all outside communication via the proxy service (api gateway)
all outside communication via the proxy service (api gat...
1
1
2
2
8
8
9
9
6
6
4
4
3
3
7
7
19
19
20
20
13
13
11
11
10
10
12
12
17
17
15
15
18
18
14
14
16
16
21
21
5
5
office application
office application
https://office-application.owncloud.test
https://office-application.owncloud....
https://ocis.owncloud.test
https://ocis.owncloud.test
wopi server
wopi server
ocis
ocis
reverse proxy
reverse proxy
User
User
wopi -> general.wopiurl
address an office application can reach the wopi server
wopi -> general.wopiurl...
reverse proxy
reverse proxy
https://wopiserver.owncloud.test
https://wopiserver.owncloud.test
APP_PROVIDER_DRIVER: wopi
  # for display purposes only
  APP_PROVIDER_WOPI_APP_NAME: Office-Application
  # address of the office application server
  APP_PROVIDER_WOPI_APP_URL: https://office-application.owncloud.test
  # where can the app provider reach the wopiserver
  APP_PROVIDER_WOPI_WOPI_SERVER_EXTERNAL_URL: https://wopiserver.owncloud.test
  #  the base URL to access ocis (global OCIS_URL, or overwritten by the below)
  APP_PROVIDER_WOPI_FOLDER_URL_BASE_URL: https://ocis.owncloud.test
APP_PROVIDER_DRIVER: wopi...
wopi -> ocis_wopi/config/wopiserver/wopiserver.conf.dist
wopi -> ocis_wopi/config/wopiserver/wopiserver.conf.dist
a. Open oC web
b. List space
c. Select file
d. Open selected file via office app
e. oC Web opens office url with token in an iframe
a. Open oC web...
a. load:    /app/list => display available office apps
a+b+c                   => obtain file id
e. call:     /app/open
    return: office url to open:
a. load:    /app/list => display available of...
https://office-application.owncloud.test/xyz?
wopiSrc=https://wopiserver.owncloud.test/wopi/xxx/file-id-123

Token management is for security reasons not part of the URL
https://office-application.owncloud.test/xyz?wopiSrc=https:...
reverse proxy
reverse proxy
wopi -> cs3.revagateway
(cs3 api gateway)
wopi -> cs3.revagateway...
Text is not SVG - cannot display
\ No newline at end of file diff --git a/modules/developer/images/ocis/deployment/ubernauten_media/login.png b/modules/developer/images/ocis/deployment/ubernauten_media/login.png new file mode 100644 index 00000000..de76db68 Binary files /dev/null and b/modules/developer/images/ocis/deployment/ubernauten_media/login.png differ diff --git a/modules/developer/images/ocis/grpc-logo.png b/modules/developer/images/ocis/grpc-logo.png new file mode 100644 index 00000000..11ad9b1e Binary files /dev/null and b/modules/developer/images/ocis/grpc-logo.png differ diff --git a/modules/developer/images/ocis/http-logo.png b/modules/developer/images/ocis/http-logo.png new file mode 100644 index 00000000..b37f7467 Binary files /dev/null and b/modules/developer/images/ocis/http-logo.png differ diff --git a/modules/developer/images/ocis/idea.drawio.svg b/modules/developer/images/ocis/idea.drawio.svg new file mode 100644 index 00000000..068b7318 --- /dev/null +++ b/modules/developer/images/ocis/idea.drawio.svg @@ -0,0 +1,489 @@ + + + + + + + + +
+
+
+ storage provider +
+
+
+
+ + storage provider + +
+
+ + + + +
+
+
+ storage provider +
+
+
+
+ + storage provider + +
+
+ + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ a sea of storage spaces +
+
+
+
+ + a sea of storage spaces + +
+
+ + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ einstein +
+
+
+
+ + einst... + +
+
+ + + + + +
+
+
+ marie +
+
+
+
+ + marie + +
+
+ + + + + +
+
+
+ richard +
+
+
+
+ + richa... + +
+
+ + + + + + +
+
+
+ storage space +
+ registry at +
+ + https://cloud.zurich.test + +
+
+
+
+ + storage sp... + +
+
+ + + + +
+
+
+ storage space +
+ registry at +
+ + https://cloud.paris.test + +
+
+
+
+ + storage sp... + +
+
+ + + + +
+
+
+ storage space +
+ registry at +
+ + https://cloud.princeton.test + +
+
+
+
+ + storage sp... + +
+
+ + + + +
+
+
+ oCIS = federated storage +
+
+
+
+ + oCIS = federated storage + +
+
+ + + + +
+
+
+ multiple +
+ storage registries +
+
+
+
+ + multiple... + +
+
+ + + + +
+
+
+ end user devices +
+
+
+
+ + end user devices + +
+
+ + + + +
+
+
+ storage provider +
+
+
+
+ + storage provider + +
+
+ + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + + Viewer does not support full SVG 1.1 + + + +
\ No newline at end of file diff --git a/modules/developer/images/ocis/monitoring_tracing_overview.drawio.svg b/modules/developer/images/ocis/monitoring_tracing_overview.drawio.svg new file mode 100644 index 00000000..79ea6375 --- /dev/null +++ b/modules/developer/images/ocis/monitoring_tracing_overview.drawio.svg @@ -0,0 +1,280 @@ + + + + + + + + + + + + +
+
+
+ Prometheus +
+
+
+
+ + Prometheus + +
+
+ + + + + + + + +
+
+
+ Jaeger Query +
+
+
+
+ + Jaeger Query + +
+
+ + + + + + +
+
+
+ Jaeger Collector +
+
+
+
+ + Jaeger Collector + +
+
+ + + + +
+
+
+ ElasticSearch +
+
+
+
+ + ElasticSearch + +
+
+ + + + + + + + +
+
+
+ Grafana +
+
+
+
+ + Grafana + +
+
+ + + + + + +
+
+
+ oCIS +
+
+
+
+ + oCIS + +
+
+ + + + +
+
+
+ Telegraf +
+
+
+
+ + Telegraf + +
+
+ + + + +
+
+
+ Prometheus output +
+
+
+
+ + Prometheus output + +
+
+ + + + + + + + +
+
+
+ Prometheus input +
+
+
+
+ + Prometheus input + +
+
+ + + + +
+
+
+ Host & Docker metrics +
+
+
+
+ + Host & Docker metrics + +
+
+ + + + + + +
+
+
+ Jaeger Agent +
+
+
+
+ + Jaeger Agent + +
+
+ + + + + + + + +
+
+
+ server for monitoring & tracing infrastructure +
+
+
+
+ + server for monitoring & tracing infrastru... + +
+
+ + + + +
+
+
+ server(s) for oCIS with monitoring & tracing clients +
+
+
+
+ + server(s) for oCIS with monitoring & tracing c... + +
+
+ + + + +
+
+
+ Visualization and querying of +
+ metrics and traces +
+
+
+
+ + Visualization and querying o... + +
+
+
+ + + + + Viewer does not support full SVG 1.1 + + + +
\ No newline at end of file diff --git a/modules/developer/images/ocis/ocis-apis.drawio.svg b/modules/developer/images/ocis/ocis-apis.drawio.svg new file mode 100644 index 00000000..85721884 --- /dev/null +++ b/modules/developer/images/ocis/ocis-apis.drawio.svg @@ -0,0 +1,286 @@ + + + + + + + +
+
+
+ Client 1 +
+
+
+
+ + Client 1 + +
+
+ + + + +
+
+
+ HTTP Gateway +
+ (proxy service) +
+
+
+
+ + HTTP Gateway... + +
+
+ + + + +
+
+
+ Client 2 +
+
+
+
+ + Client 2 + +
+
+ + + + +
+
+
+ Client 3 +
+
+
+
+ + Client 3 + +
+
+ + + + + + +
+
+
+ HTTP +
+
+
+
+ + HTTP + +
+
+ + + + + + +
+
+
+ HTTP +
+
+
+
+ + HTTP + +
+
+ + + + + + +
+
+
+ HTTP +
+
+
+
+ + HTTP + +
+
+ + + + +
+
+
+ ocis service +
+
+
+
+ + ocis service + +
+
+ + + + +
+
+
+ ocis service +
+
+
+
+ + ocis service + +
+
+ + + + +
+
+
+ ocis service +
+
+
+
+ + ocis service + +
+
+ + + + + + +
+
+
+ HTTP +
+
+
+
+ + HTTP + +
+
+ + + + +
+
+
+ GRPC Gateway +
+ (gateway service) +
+
+
+
+ + GRPC Gateway... + +
+
+ + + + + + +
+
+
+ gRPC +
+
+
+
+ + gRPC + +
+
+ + + + + + +
+
+
+ gRPC +
+
+
+
+ + gRPC + +
+
+ + + + + + +
+
+
+ HTTP +
+
+
+
+ + HTTP + +
+
+
+ + + + + Viewer does not support full SVG 1.1 + + + +
\ No newline at end of file diff --git a/modules/developer/images/ocis/ocis-config-redesign.drawio.svg b/modules/developer/images/ocis/ocis-config-redesign.drawio.svg new file mode 100644 index 00000000..e7615dd0 --- /dev/null +++ b/modules/developer/images/ocis/ocis-config-redesign.drawio.svg @@ -0,0 +1,4 @@ + + + +
> ocis server
> ocis server

# global logging config
log:
  pretty: true
  color: true
  level: info
proxy:
  http:
    addr: localhost:9200
  log:
    pretty: false
    color: false
    level: info
accounts:
  log:
    level: debug
    color: false
    pretty: false

# global logging config...
ocis.yaml
ocis.yaml

  http:
    addr: localhost:1234
  log:
    pretty: false
    color: false
    level: info

http:...
proxy.yaml
proxy.yaml

  http:
    addr: localhost:5678


http:...
accounts.yaml
accounts.yaml
resulting proxy config values
resulting proxy config values
resulting accounts config values
resulting accounts config values

  http:
    addr: localhost:1234
  log:
    pretty: false
    color: false
    level: info


http:...
notice how even if ocis supplies
with logging configuration, the values
from the extension overwrite the inherited
values. This can also be achieved using an
environment variable.
notice how even if ocis supplies...

  http:
    addr: localhost:5678
  log:
    pretty: false
    color: false
    level: debug


http:...
on the other hand here we see the extension
inherited the logging information from the parent.
The same holds true from any value. Values from
ocis.yaml are carried down to their respective
extensions.
on the other hand here we see the extension...
Viewer does not support full SVG 1.1
diff --git a/modules/developer/images/ocis/ocis-services-communication.drawio.svg b/modules/developer/images/ocis/ocis-services-communication.drawio.svg new file mode 100644 index 00000000..06d30268 --- /dev/null +++ b/modules/developer/images/ocis/ocis-services-communication.drawio.svg @@ -0,0 +1,4 @@ + + + +
public share auth provider
public share auth pr...
machine auth provider
machine auth provider
proxy
proxy
web
web
grpc
grpc
grpc
grpc
ocs
ocs
grpc
grpc
ocdav
ocdav
grpc
grpc
ocs
ocs
grpc
grpc
settings
settings
LDAP
LDAP
idp
idp
nats
nats
grpc
grpc
LDAP
LDAP
graph
graph
http
http
graph-explorer
graph-explorer
grpc
grpc
webdav
webdav
grpc
grpc
thumbnails
thumbnails
gateway
gateway
grpc
grpc
storage registry
storage registry
grpc
grpc
storage provider
storage provider
share storage provider
share storage provid...
public storage provider
public storage provi...
user provider
user provider
group provider
group provider
http
http
http
http
http
http
http
http
http
http
http
http
http
http
http

http
http
http
grpc
grpc
grpc
grpc
grpc
grpc
grpc
grpc
grpc
grpc
grpc
grpc
Storage
Storage
Reva
Reva
oCIS
oCIS
http
http
libregraph/idm
libregraph/idm
oidc authprovider
oidc authprovider
LDAP
LDAP
basic auth provider
basic auth provider
grpc
grpc
grpc
grpc
Text is not SVG - cannot display
\ No newline at end of file diff --git a/modules/developer/images/ocis/runtime.drawio.svg b/modules/developer/images/ocis/runtime.drawio.svg new file mode 100644 index 00000000..754593d7 --- /dev/null +++ b/modules/developer/images/ocis/runtime.drawio.svg @@ -0,0 +1,4 @@ + + + +
Runtime
Runtime
ocs
ocs
graph
graph
idm
idm
storage
storage
proxy
proxy
idp
idp
thumbnails
thumbnails
webdav
webdav
suture
suture
$ ocis server
$ ocis kill proxy
$ ocis start proxy
$ ocis list
$ ocis server...
the runtime component starts services
and suture then takes over, keeping
track of every started service, restarting
it when needed if an error occurred.
the runtime component starts services...
each service is implemented as a sub-command. The binary essentially calls itself with the right arguments and the side effect is that a
service is started. The service will then be monitored by Suture.
each service is implemented as a sub-command. The binary essentially calls itself with the right arguments and the side e...
the runtime is a service in itself. Starting an
oCIS instance will start a runtime service.
the runtime is a service in itself. Star...
Text is not SVG - cannot display
\ No newline at end of file diff --git a/modules/developer/images/ocis/storage/frontend.drawio.svg b/modules/developer/images/ocis/storage/frontend.drawio.svg new file mode 100644 index 00000000..45289f34 --- /dev/null +++ b/modules/developer/images/ocis/storage/frontend.drawio.svg @@ -0,0 +1,303 @@ + + + + + + + + +
+
+
+ + GET /data/<transfer_token> + +
+
+
+
+ + GET /da... + +
+
+ + + + + +
+
+
+ + PROPFIND /webdav + +
+
+
+
+ + PROPFIN... + +
+
+ + + + + +
+
+
+
+ + POST /ocs/v1/apps/files_sharing/api/v1/shares + +
+
+ + path=/path/to/file + +
+
+ + shareType=0 + +
+
+ + shareWith=<username> + +
+
+
+
+
+ + POST /o... + +
+
+ + + + +
+
+
+ frontend +
+
+
+
+ + frontend + +
+
+ + + + + + + +
+
+
+ datagateway +
+
+
+
+ + datagateway + +
+
+ + + + +
+
+
+ ocdav +
+
+
+
+ + ocdav + +
+
+ + + + +
+
+
+ ocs +
+
+
+
+ + ocs + +
+
+ + + + +
+
+
+ gateway +
+
+
+
+ + gateway + +
+
+ + + + + + + +
+
+
+ storage home | users | ... +
+
+
+
+ + storage home | users | ... + +
+
+ + + + + + + +
+
+
+ storageprovider +
+
+
+
+ + storageprovider + +
+
+ + + + +
+
+
+ dataprovider +
+
+
+
+ + dataprovider + +
+
+ + + + + +
+
+
+ + GET target URL extracted from transfer token (JWT) + +
+
+
+
+ + GET tar... + +
+
+ + + + + +
+
+
+ + Stat +
+ ListContainer +
+
+
+
+
+ + Stat... + +
+
+ + + + + +
+
+
+ + Stat +
+ CreateShare +
+ CreatePublicShare +
+ CreateOCMShare +
+
+
+
+
+ + Stat... + +
+
+
+ + + + + Viewer does not support full SVG 1.1 + + + +
\ No newline at end of file diff --git a/modules/developer/images/ocis/storage/namespaces.drawio.svg b/modules/developer/images/ocis/storage/namespaces.drawio.svg new file mode 100644 index 00000000..b3baa589 --- /dev/null +++ b/modules/developer/images/ocis/storage/namespaces.drawio.svg @@ -0,0 +1,341 @@ + + + + + + + + +
+
+
+ /rel/path/to/file.ext +
+
+
+
+ + /rel/p... + +
+
+ + + + + +
+
+
+
+ / +
+
+
+ /home +
+
+ /users/<user_layout> +
+
+ /public +
+
+ /shares +
+
+
+
+
+ + /... + +
+
+ + + + +
+
+
+ storage home +
+
+
+
+ + storage home + +
+
+ + + + + + +
+
+
+ storageprovider +
+
+
+
+ + storageprovider + +
+
+ + + + +
+
+
+ dataprovider +
+
+
+
+ + dataprovider + +
+
+ + + + +
+
+
+ frontend +
+
+
+
+ + frontend + +
+
+ + + + + + +
+
+
+ ocdav +
+
+
+
+ + ocdav + +
+
+ + + + +
+
+
+ ocs +
+
+
+
+ + ocs + +
+
+ + + + + +
+
+
+ + /webdav +
+ /dav/files/<username> +
+
+
+ + /dav/trash-bin/<username> + +
+
+ + /dav/public-files/<token> + +
+
+ + /dav/meta/<fileid>/v + +
+
+ + /dav/avatars/<username> + +
+
+
+
+
+ + /webdav... + +
+
+ + + + + +
+
+
+ + /ocs/v1.php/apps/files_sharing/api/v1/shares + +
+
+
+
+ + /ocs/v1.ph... + +
+
+ + + + +
+
+
+ gateway +
+
+
+
+ + gateway + +
+
+ + + + + + +
+
+
+ gateway +
+
+
+
+ + gateway + +
+
+ + + + +
+
+
+ authregistry +
+
+
+
+ + authregistry + +
+
+ + + + +
+
+
+ storageregistry +
+
+
+
+ + storageregistry + +
+
+ + + + + + +
+
+
+ oc10 namespace +
+ (all paths aere relative to the users home) +
+
+
+
+ + oc10 names... + +
+
+ + + + +
+
+
+ CS3 global namespace +
+
+
+
+ + CS3 global... + +
+
+
+ + + + + Viewer does not support full SVG 1.1 + + + +
\ No newline at end of file diff --git a/modules/developer/images/ocis/storage/overview.drawio.svg b/modules/developer/images/ocis/storage/overview.drawio.svg new file mode 100644 index 00000000..8dc5bf16 --- /dev/null +++ b/modules/developer/images/ocis/storage/overview.drawio.svg @@ -0,0 +1,685 @@ + + + + + + + + + + + + + + + + + +
+
+
+ proxy +
+
+
+
+ + proxy + +
+
+ + + + + + + + + +
+
+
+ gateway +
+
+
+
+ + gateway + +
+
+ + + + + + + +
+
+
+ gateway +
+
+
+
+ + gateway + +
+
+ + + + +
+
+
+ authregistry +
+
+
+
+ + authregistry + +
+
+ + + + +
+
+
+ storageregistry +
+
+
+
+ + storageregistry + +
+
+ + + + + + + + + + +
+
+
+ storage users +
+
+
+
+ + storage users + +
+
+ + + + + + + +
+
+
+ storageprovider +
+
+
+
+ + storageprovider + +
+
+ + + + +
+
+
+ dataprovider +
+
+
+
+ + dataprovider + +
+
+ + + + +
+
+
+ storage home +
+
+
+
+ + storage home + +
+
+ + + + + + + +
+
+
+ storageprovider +
+
+
+
+ + storageprovider + +
+
+ + + + +
+
+
+ dataprovider +
+
+
+
+ + dataprovider + +
+
+ + + + +
+
+
+ storage public link +
+
+
+
+ + storage public link + +
+
+ + + + + + + +
+
+
+ publicstorageprovider +
+
+
+
+ + publicstorageprovider + +
+
+ + + + +
+
+
+ authprovider +
+ publicshares +
+
+
+
+ + authprovider... + +
+
+ + + + +
+
+
+ storage metadata +
+
+
+
+ + storage metadata + +
+
+ + + + + + + +
+
+
+ storageprovider +
+
+
+
+ + storageprovider + +
+
+ + + + +
+
+
+ dataprovider +
+
+
+
+ + dataprovider + +
+
+ + + + +
+
+
+ sharing +
+
+
+
+ + sharing + +
+
+ + + + + + + +
+
+
+ usershareprovider +
+
+
+
+ + usershareprovider + +
+
+ + + + +
+
+
+ publicshareprovider +
+
+
+
+ + publicshareprovider + +
+
+ + + + +
+
+
+ users +
+
+
+
+ + users + +
+
+ + + + + + + +
+
+
+ userprovider +
+
+
+
+ + userprovider + +
+
+ + + + +
+
+
+ groups +
+
+
+
+ + groups + +
+
+ + + + + + + +
+
+
+ groupprovider +
+
+
+
+ + groupprovider + +
+
+ + + + +
+
+
+ authbasic +
+
+
+
+ + authbasic + +
+
+ + + + + + + +
+
+
+ authprovider +
+
+
+
+ + authprovider + +
+
+ + + + +
+
+
+ authbearer +
+
+
+
+ + authbearer + +
+
+ + + + + + + +
+
+
+ authprovider +
+
+
+
+ + authprovider + +
+
+ + + + +
+
+
+ accounts +
+
+
+
+ + accounts + +
+
+ + + + + + + + +
+
+
+ frontend +
+
+
+
+ + frontend + +
+
+ + + + + + + +
+
+
+ datagateway +
+
+
+
+ + datagateway + +
+
+ + + + +
+
+
+ ocdav +
+
+
+
+ + ocdav + +
+
+ + + + +
+
+
+ ocs +
+
+
+
+ + ocs + +
+
+ + + + +
+
+
+ ocis +
+
+
+
+ + ocis + +
+
+ + + + +
+
+
+ reva +
+
+
+
+ + reva + +
+
+ + + + +
+
+
+ deprecated +
+
+
+
+ + deprec... + +
+
+
+ + + + + Viewer does not support full SVG 1.1 + + + +
\ No newline at end of file diff --git a/modules/developer/images/ocis/storage/semver-schema.png b/modules/developer/images/ocis/storage/semver-schema.png new file mode 100644 index 00000000..f47823e9 Binary files /dev/null and b/modules/developer/images/ocis/storage/semver-schema.png differ diff --git a/modules/developer/images/ocis/storage/spacesprovider.drawio.svg b/modules/developer/images/ocis/storage/spacesprovider.drawio.svg new file mode 100644 index 00000000..d122c58f --- /dev/null +++ b/modules/developer/images/ocis/storage/spacesprovider.drawio.svg @@ -0,0 +1,352 @@ + + + + + + + +
+
+
+ oCIS spaces provider +
+ [Software System] +
+
+
+
+ + oCIS spaces provider... + +
+
+ + + + +
+
+
+ + reva storage provider + +
+ [Component: golang] +
+
+
+ hosts multiple storage spaces using a storage driver +
+
+
+
+
+ + reva storage provider... + +
+
+ + + + +
+
+
+ + reva gateway + +
+ [Component: golang] +
+
+
+ API facade for internal reva services +
+
+
+
+
+ + reva gateway... + +
+
+ + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+
+ + Storage System... + +
+
+ + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+
+ + Reads from and writes to... + +
+
+ + + + +
+
+
+ + reva frontend + +
+ [Component: golang] +
+
+
+ handles protocol translation +
+
+
+
+
+ + reva frontend... + +
+
+ + + + +
+
+
+ + oCIS proxy + +
+ [Component: golang] +
+
+
+ Routes requests to oc10 or ecis +
+
+
+
+
+ + oCIS proxy... + +
+
+ + + + + +
+
+
+
+
+ + Mints an internal JWT +
+ and torwards requests to +
+
+
+ [WebDAV, OCS, OCM, tus] +
+
+
+
+
+
+ + Mints an internal JWT... + +
+
+ + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, +
+ Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+
+ + Client... + +
+
+ + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [WebDAV, libregraph, CS3] +
+
+
+
+
+
+ + Reads from and writes to... + +
+
+ + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [CS3, tus] +
+
+
+
+
+
+ + Reads from and writes to... + +
+
+ + + + + +
+
+
+
+
+ + Forwards to + +
+
+ [CS3, storage registry] +
+
+
+
+
+
+ + Forwards to... + +
+
+ + + + +
+
+
+

+ C4 Component diagram for an oCIS spaces provider +

+

+ An oCIS spaces provider manages resources in storage spaces by persisting them with a specific storage driver in a storage system. +

+

+ Date: 2021-07-22T12:40 +

+
+
+
+
+ + C4 Component diagram for an oCIS spaces provider... + +
+
+
+ + + + + Viewer does not support full SVG 1.1 + + + +
\ No newline at end of file diff --git a/modules/developer/images/ocis/storage/spacesregistry.drawio.svg b/modules/developer/images/ocis/storage/spacesregistry.drawio.svg new file mode 100644 index 00000000..f716b0bb --- /dev/null +++ b/modules/developer/images/ocis/storage/spacesregistry.drawio.svg @@ -0,0 +1,405 @@ + + + + + + + +
+
+
+ oCIS spaces registry +
+ [Software System] +
+
+
+
+ + oCIS spaces registry... + +
+
+ + + + +
+
+
+ + reva storage registry + +
+ [Component: golang] +
+
+
+ manages and caches storage space metadata +
+
+
+
+
+ + reva storage registry... + +
+
+ + + + +
+
+
+ + reva gateway + +
+ [Component: golang] +
+
+
+ API facade for internal reva services +
+
+
+
+
+ + reva gateway... + +
+
+ + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+
+ + Storage System... + +
+
+ + + + + +
+
+
+
+
+ + Provisions and manages spaces in + +
+
+ [CS3] +
+
+
+
+
+
+ + Provisions and manages spaces... + +
+
+ + + + +
+
+
+ + reva frontend + +
+ [Component: golang] +
+
+
+ handles protocol translation +
+
+
+
+
+ + reva frontend... + +
+
+ + + + +
+
+
+ + oCIS proxy + +
+ [Component: golang] +
+
+
+ Routes requests to oc10 or ecis +
+
+
+
+
+ + oCIS proxy... + +
+
+ + + + + +
+
+
+
+
+ + Mints an internal JWT +
+ and torwards requests to +
+
+
+ [libregraph] +
+
+
+
+
+
+ + Mints an internal JWT... + +
+
+ + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, +
+ Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+
+ + Client... + +
+
+ + + + + +
+
+
+
+
+ + polls or gets notified about changes in + +
+
+ [libregraph] +
+
+
+
+
+
+ + polls or gets notified about c... + +
+
+ + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [CS3, tus] +
+
+
+
+
+
+ + Reads from and writes to... + +
+
+ + + + + +
+
+
+
+
+ + Lists spaces using + +
+
+ [CS3] +
+
+
+
+
+
+ + Lists spaces using... + +
+
+ + + + +
+
+
+

+ C4 Component diagram for an oCIS spaces registry +

+

+ An oCIS spaces provider manages resources in storage spaces by persisting them with a specific storage driver in a storage system. +

+

+ Date: 2021-07-22T12:40 +

+
+
+
+
+ + C4 Component diagram for an oCIS spaces registry... + +
+
+ + + + +
+
+
+ + reva storage provider + +
+ [Component: golang] +
+
+
+ hosts multiple storage spaces using a storage driver +
+
+
+
+
+ + reva storage provider... + +
+
+ + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+
+ + Reads from and writes to... + +
+
+
+ + + + + Viewer does not support full SVG 1.1 + + + +
\ No newline at end of file diff --git a/modules/developer/images/ocis/storage/storage.drawio.svg b/modules/developer/images/ocis/storage/storage.drawio.svg new file mode 100644 index 00000000..fd6e759c --- /dev/null +++ b/modules/developer/images/ocis/storage/storage.drawio.svg @@ -0,0 +1,434 @@ + + + + + + + +
+
+
+ oCIS System +
+ [Software System] +
+
+
+
+ + oCIS System... + +
+
+ + + + + + +
+
+
+ + Einstein + +
+ [Person] +
+
+
+ End user +
+
+
+
+
+ + Einstein... + +
+
+ + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+
+ + Client... + +
+
+ + + + +
+
+
+ + Storage Space Registry + +
+ [Container: golang, HTTP, libregraph] +
+
+
+ Manages spaces for users +
+
+
+
+
+ + Storage Space Registry... + +
+
+ + + + +
+
+
+ + Storage Space Provider + +
+ [Container: golang] +
+
+
+ Persists storage spaces using reva +
+
+
+
+
+ + Storage Space Provider... + +
+
+ + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+
+ + Storage System... + +
+
+ + + + + + +
+
+
+ + Moss + +
+ [Person] +
+
+
+ Administrator +
+
+
+
+
+ + Moss... + +
+
+ + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+
+ + Reads from and writes to... + +
+
+ + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [WebDAV, libregraph, CS3, tus] +
+
+
+
+
+
+ + Reads from and writes to... + +
+
+ + + + + +
+
+
+
+
+ + Manages the users Storage Spaces in + +
+
+ [libregraph] +
+
+
+
+
+
+ + Manages the users Storage Spac... + +
+
+ + + + + +
+
+
+
+
+ + Manages resources with + +
+
+ [Web UI or native clients] +
+
+
+
+
+
+ + Manages resources with... + +
+
+ + + + + +
+
+
+
+
+ + Registers itself at and +
+ sends space root etag changes to +
+
+
+ [CS3, libregraph?, PUSH] +
+
+
+
+
+
+ + Registers itself at and... + +
+
+ + + + + +
+
+
+
+
+ + Manages organizational Storage Spaces in + +
+
+ [WebDAV, libregraph, CS3, CLI] +
+
+
+
+
+
+ + Manages organizational Storage... + +
+
+ + + + +
+
+
+ + Identity Management System + +
+ [Software System] +
+
+
+ provides users and groups +
+
+
+
+
+ + Identity Management System... + +
+
+ + + + + +
+
+
+
+
+ + Authenticates users and searches recipients with + +
+
+ [OpenID Connect, LDAP, REST] +
+
+
+
+
+
+ + Authenticates users and search... + +
+
+ + + + +
+
+
+

+ C4 Container diagram for the oCIS System +

+

+ As a platform, the oCIS system may not only includes web, mobile and desktop clients but also the underlying storage system or an identity management system +

+

+ Date: 2021-07-22T16:43 +

+
+
+
+
+ + C4 Container diagram for the oCIS System... + +
+
+
+ + + + + Viewer does not support full SVG 1.1 + + + +
\ No newline at end of file diff --git a/modules/developer/images/ocis/storage/storageprovider-spaces.drawio.svg b/modules/developer/images/ocis/storage/storageprovider-spaces.drawio.svg new file mode 100644 index 00000000..9aed5040 --- /dev/null +++ b/modules/developer/images/ocis/storage/storageprovider-spaces.drawio.svg @@ -0,0 +1,471 @@ + + + + + + + + + +
+
+
+ + CS3 +
+ storage provider +
+ API (GRPC) +
+
+
+
+
+
+ + CS3... + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ storage provider +
+
+
+
+ + storage provider + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + + Viewer does not support full SVG 1.1 + + + +
\ No newline at end of file diff --git a/modules/developer/images/ocis/storage/storageprovider.drawio.svg b/modules/developer/images/ocis/storage/storageprovider.drawio.svg new file mode 100644 index 00000000..e7ba5ea7 --- /dev/null +++ b/modules/developer/images/ocis/storage/storageprovider.drawio.svg @@ -0,0 +1,352 @@ + + + + + + + +
+
+
+ oCIS storage provider +
+ [Software System] +
+
+
+
+ + oCIS storage provider... + +
+
+ + + + +
+
+
+ + reva storage provider + +
+ [Component: golang] +
+
+
+ hosts multiple storage spaces using a storage driver +
+
+
+
+
+ + reva storage provider... + +
+
+ + + + +
+
+
+ + reva gateway + +
+ [Component: golang] +
+
+
+ API facade for internal reva services +
+
+
+
+
+ + reva gateway... + +
+
+ + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+
+ + Storage System... + +
+
+ + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+
+ + Reads from and writes to... + +
+
+ + + + +
+
+
+ + reva frontend + +
+ [Component: golang] +
+
+
+ handles protocol translation +
+
+
+
+
+ + reva frontend... + +
+
+ + + + +
+
+
+ + oCIS proxy + +
+ [Component: golang] +
+
+
+ Routes requests to oc10 or ecis +
+
+
+
+
+ + oCIS proxy... + +
+
+ + + + + +
+
+
+
+
+ + Mints an internal JWT +
+ and torwards requests to +
+
+
+ [WebDAV, OCS, OCM, tus] +
+
+
+
+
+
+ + Mints an internal JWT... + +
+
+ + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, +
+ Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+
+ + Client... + +
+
+ + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [WebDAV, libregraph, CS3] +
+
+
+
+
+
+ + Reads from and writes to... + +
+
+ + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [CS3, tus] +
+
+
+
+
+
+ + Reads from and writes to... + +
+
+ + + + + +
+
+
+
+
+ + Forwards to + +
+
+ [CS3, storage registry] +
+
+
+
+
+
+ + Forwards to... + +
+
+ + + + +
+
+
+

+ C4 Component diagram for an oCIS storage provider +

+

+ An oCIS storage provider manages resources in storage spaces by persisting them with a specific storage driver in a storage system. +

+

+ Date: 2021-07-22T12:40 +

+
+
+
+
+ + C4 Component diagram for an oCIS storage provider... + +
+
+
+ + + + + Viewer does not support full SVG 1.1 + + + +
\ No newline at end of file diff --git a/modules/developer/images/ocis/storage/storageregistry.drawio.svg b/modules/developer/images/ocis/storage/storageregistry.drawio.svg new file mode 100644 index 00000000..2b24b386 --- /dev/null +++ b/modules/developer/images/ocis/storage/storageregistry.drawio.svg @@ -0,0 +1,233 @@ + + + + + + + + +
+
+
+ The storage registry currently maps paths and storageids to the +
+ + address:port + + of the corresponding storage provider +
+
+
+
+ + The storage registry currently maps... + +
+
+ + + + + + + +
+
+
+ storage registry +
+
+
+
+ + storage registry + +
+
+ + + + + + + +
+
+
+ storage providers +
+
+
+
+ + storage providers + +
+
+ + + + + + + + + + + + +
+
+
+ The gateway uses the storage registry to look up the storage provider that is responsible for path and id based references in incoming requests. +
+
+
+
+ + The gateway uses the storage regist... + +
+
+ + + + + + + +
+
+
+ gateway +
+
+
+
+ + gateway + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + / + +
+
+
+
+ + / + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + / + +
+
+
+
+ + / + +
+
+
+ + + + + Viewer does not support full SVG 1.1 + + + +
\ No newline at end of file diff --git a/modules/developer/images/ocis/storage/storagespace.drawio.svg b/modules/developer/images/ocis/storage/storagespace.drawio.svg new file mode 100644 index 00000000..6e60631b --- /dev/null +++ b/modules/developer/images/ocis/storage/storagespace.drawio.svg @@ -0,0 +1,163 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ the root resource of the storage space +
+
+
+
+ + the root resource of... + +
+
+ + + + +
+
+
+ RESOURCE_TYPE_CONTAINER +
+
+
+
+ + RESOURCE_TYPE_CONTAINER + +
+
+ + + + +
+
+
+ RESOURCE_TYPE_FILE +
+
+
+
+ + RESOURCE_TYPE_FILE + +
+
+ + + + + + +
+
+
+ if the gateway encounters a resource type reference it will look it up and replace the reference with the results of the actual node (which can live in another storage provider) +
+
+
+
+ + if the gateway encounters a... + +
+
+ + + + +
+
+
+ every resource can be referenced by path or by id +
+
+
+
+ + every resource can be... + +
+
+ + + + +
+
+
+ RESOURCE_TYPE_SYMLINK +
+
+
+
+ + RESOURCE_TYPE_SYMLINK + +
+
+ + + + + + + + +
+
+
+ RESOURCE_TYPE_REFERENCE +
+
+
+
+ + RESOURCE_TYPE_REFERENCE + +
+
+
+ + + + + Viewer does not support full SVG 1.1 + + + +
\ No newline at end of file diff --git a/modules/developer/images/ocis/tus-public-upload.svg b/modules/developer/images/ocis/tus-public-upload.svg new file mode 100644 index 00000000..ad2a5c5f --- /dev/null +++ b/modules/developer/images/ocis/tus-public-upload.svg @@ -0,0 +1,3 @@ + + +

ocis-reva sharing


REVA_SHARING_ADDR = 0.0.0.0:9150

ocis-reva sharing...

ocis-reva frontend


REVA_FRONTEND_ADDR = 0.0.0.0:9140

REVA_GATEWAY_URL = ocis:9142

ocis-reva frontend...

ocis-proxy


PROXY_HTTP_ADDR = 0.0.0.0:9200

ocis-proxy...
 2  POST http://ocis:9140/remote.php/dav/files/einstein/
2  POST http:/...

ocdav


prefix = ""

timeout = 86400

ocdav...

datagateway


prefix = "data"

timeout = 86400

datagateway...

client



client
 22  PATCH https://oc.example.org/data/{token}
Tus-Resumable: 1.0.0
22  PATCH http...

ocis-reva gateway


REVA_GATEWAY_ADDR = 0.0.0.0:9142

ocis-reva gateway...

storage-registry



storage-registry
Expose: true
Expose: true
 24  PATCH http://ocis:9156/data/u-u-i-d
24  PATCH http...
 4  GetStorageProvider
(ShareReference)
4  GetStorageP...
 5  ProviderInfo
 5  ProviderInfo

storageprovider


REVA_STORAGE_HOME_ADDR = 0.0.0.0:9154

REVA_STORAGE_HOME_DRIVER = eoshome

REVA_STORAGE_HOME_EXPOSE_DATA_SERVER = false

REVA_STORAGE_HOME_DATA_SERVER_URL =

    http://ocis:9156/data

storageprovider...
Expose: false
Expose: false
 6  InitiateFileUpload
(ShareReference)
6  InitiateFil...
EOS
EOS
 15  WriteFile(upload info)
15  WriteFile(...
 7  GetPublicShare
7  GetPublicSh...
 19  UploadEndpoint
https://oc.example.org/data/{token}
19  UploadEndp...
 20  201 Created
Location: https://oc.example.org/data/{token}
20  201 Create...
 21  201 Created
Location: https://oc.example.org/data/{token}
21  201 Create...
 1  POST https://oc.example.org/remote.php/dav/files/einstein/
Upload-Length: 100
Tus-Resumable: 1.0.0
Upload-Metadata: filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg==,
dir d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg==
1  POST https:...
 23  PATCH http://ocis:9140/data/{token}
Tus-Resumable: 1.0.0
23  PATCH http...
 3  InitiateFileUpload
3  InitiateFil...
 25  Write(bytes)
25  Write(byte...
 26  204 No Content
26  204 No Con...
 27  204 No Content
27  204 No Con...
 28  204 No Content
28  204 No Con...

publicstorageprovider


expose-data-server = true

publicstorageprovider...

publicshareprovider



publicshareprovider
 8  GetPublicShare
8  GetPublicSh...
 9  PublicShare
 9  PublicShare
 10  PublicShare
 10  PublicShare
 11  InitiateFileUpload(TargetReference)
11  InitiateFi...
 12  GetStorageProvider
(TargetReference)
12  GetStorage...
 13  ProviderInfo
13  ProviderIn...
 14  InitiateFileUpload(TargetReference)
14  InitiateFi...
 16  UploadEndpoint
http://ocis:9156/data/u-u-i-d
Expose: false
16  UploadEndp...
 17  UploadEndpoint
https://oc.example.org/data/
token: sign(http://ocis:9156/data/u-u-i-d)
17  UploadEndp...
 18  UploadEndpoint
https://oc.example.org/data/{token}
Expose: true
18  UploadEndp...

gateway


REVA_TRANSFER_EXPIRES = 86400

REVA_FRONTEND_URL =

    https://oc.example.org

REVA_DATAGATEWAY_URL =

    https://oc.example.org/data

    

gateway...
When a storage provider
sets the Expose flag of an Upload/Download Endpoint to false the gateway will wrap the url in a JWT and return the URL of the datagateway along with a transfer-token.
When a storage provider...

dataprovider


REVA_STORAGE_HOME_DATA_ADDR = 0.0.0.0:9156

REVA_STORAGE_HOME_DATA_DRIVER = eoshome

dataprovider...
GOAL: transfer bytes from the client up here ...
GOAL: tran...
... to the storage system somewhere down here
... to the storage syst...
Viewer does not support full SVG 1.1
\ No newline at end of file diff --git a/modules/developer/pages/apis/grpc_apis/index.adoc b/modules/developer/pages/apis/grpc_apis/index.adoc new file mode 100644 index 00000000..2902323b --- /dev/null +++ b/modules/developer/pages/apis/grpc_apis/index.adoc @@ -0,0 +1,54 @@ += gRPC +:toc: right +:toclevels: 3 + + +== _R_emote   _P_rocedure   _C_alls + +link:https://grpc.io[gRPC] is a modern open source high performance Remote Procedure Call (RPC) framework that can run in any environment. It can efficiently connect services in and across data centers with pluggable support for load balancing, tracing, health checking and authentication. It is also applicable in last mile of distributed computing to connect devices, mobile applications and browsers to backend services. + +== Advantages of gRPC + +=== {{< icon "gauge-high" >}}   Performance + +gRPC uses http/2 by default and is faster than REST. When using protocol buffers for encoding, the information comes on and off the wire much faster than JSON. Latency is an important factor in distributed systems. JSON encoding creates a noticeable factor of latency. For distributed systems and high data loads, gRPC can actually make an important difference. Other than that, gRPC supports multiple calls via the same channel and the connections are bidirectional. A single connection can transmit requests and responses at the same time. gRPC keeps connections open to reuse the same connection again which prevents latency and saves bandwidth. + +<---> +=== {{< icon "helmet-safety" >}}   Robustness + +gRPC empowers better relationships between clients and servers. The rules of communication are strictly enforced. That is not the case in REST calls, where the client and the server can send and receive anything they like and hopefully the other end understands what to do with it. In gRPC, to make changes to the communication, both client and server need to change accordingly. This prevents mistakes specially in microservice architectures. + +=== {{< icon "magnifying-glass-plus" >}}   Debuggability + +gRPC requests are re-using the same context and can be tracked or traced across multiple service boundaries. +This helps to identify slow calls and see what is causing delays. It is possible to cancel requests which cancels +them on all involved services. + +<---> +=== {{< icon "boxes-stacked" >}}   Microservices + +gRPC has been evolving and has become the best option for communication between microservices because of its unmatched +performance and its polyglot nature. One of the biggest strengths of microservices is the freedom of programming +languages and technologies. By using gRPC we can leverage all the advantages of strictly enforced communication +standards combined with freedom of choice between different programming languages - whichever would fit best. + + +// SHORTCODE: {{< hint type=info title="gRPC Advantages" >}} + +- http/2 +- protocol buffers +- reusable connections +- multi language support +// SHORTCODE: {{< /hint >}} + +== CS3 APIs + +image::ocis/cs3org.png[] + +The link:https://github.com/cs3org/cs3apis[CS3 APIs] connect storages and application providers. + +The CS3 APIs follow Google and Uber API design guidelines, specially on error handling and naming convention. You can read more about these +guidelines at https://cloud.google.com/apis/design/ and https://github.com/uber/prototool/blob/dev/style/README.md. + +The CS3 APIs use link:https://github.com/protocolbuffers/protobuf[Protocol Buffers version 3 (proto3)] as their +Interface Definition Language (IDL) to define the API interface and the structure of the payload messages. diff --git a/modules/developer/pages/apis/http/authorization.adoc b/modules/developer/pages/apis/http/authorization.adoc new file mode 100644 index 00000000..bfe0a1b4 --- /dev/null +++ b/modules/developer/pages/apis/http/authorization.adoc @@ -0,0 +1,142 @@ += Authorization +:toc: right +:toclevels: 3 + +In its default configuration, Infinite Scale supports three authentication methods as outlined on the link:https://openid.net/specs/openid-connect-core-1_0.html#rfc.section.3[OIDC official site]: +1. Authorization Code Flow +2. Implicit Flow +3. Hybrid Flow + +For detailed information on Infinite Scale's support for OpenID Connect (OIDC), please consult the link:https://owncloud.dev/ocis/identity-provider/oidc[OIDC section]. +To authenticate a client app using OIDC, both `client_id` and `client_secret` are essential. Infinite Scale does not offer dynamic registration. The required data for the default link:https://doc.owncloud.com/server/next/admin_manual/configuration/user/oidc/oidc.html#client-ids-secrets-and-redirect-uris[ownCloud clients] can be found in the link and are availble for the following apps: +- Desktop +- Android +- iOS + +While selecting an ownCloud client for authentication, take note of specific limitations such as the `Redirect URI`: + +| Source | Redirect URI | +|------|--------| +|Android|oc://android.owncloud.com| +|iOS|oc://ios.owncloud.com| +|Desktop|http://127.0.0.1
http://localhost | + +In this example, the desktop app's `client_id` and `client_secret` are being used. + +[source,bash] +---- +client_id=xdXOt13JKxym1B1QcEncf2XDkLAexMBFwiT9j6EfhhHFJhs2KM9jbjTmf8JBXE69 +client_secret=UBntmLjC2yYCeHwsyj73Uwo9TAaecAetRwMw0xYcvNL9yRdLSUi0hUAHfvCHFeFh +---- + +== Authorization Code Flow + +1. Requesting authorization + + To initiate the OIDC Code Flow, you can use tools like curl and a web browser. + The user should be directed to a URL to authenticate and give their consent (bypassing consent is against the standard): + + ```plaintext + https://ocis.test/signin/v1/identifier/_/authorize?client_id=client_id&scope=openid+profile+email+offline_access&response_type=code&redirect_uri=http://path-to-redirect-uri + ``` + + After a successful authentication, the browser will redirect to a URL that looks like this: + + ```plaintext + http://path-to-redirect-uri?code=mfWsjEL0mc8gx0ftF9LFkGb__uFykaBw&scope=openid%20profile%20email%20offline_access&session_state=32b08dd...&state= + ``` + + For the next step extract the code from the URL. + + In the above example, + the code is `mfWsjEL0mc8gx0ftF9LFkGb__uFykaBw` + +2. Requesting an access token + + The next step in the OIDC Code Flow involves an HTTP POST request + to the token endpoint of the _Infinite Scale Identity Server_. + + ```bash + curl -vk -X POST https://ocis.test/konnect/v1/token \ + -d "grant_type=authorization_code" \ + -d "code=3a3PTcO-WWXfN3l1mDN4u7G5PzWFxatU" \ + -d "redirect_uri=http:path-to-redirect-uri" \ + -d "client_id=client_id" \ + -d "client_secret=client_secret" + ``` + + Response looks like this: + ```json + { + "access_token": "eyJhbGciOid...", + "token_type": "Bearer", + "id_token": "eyJhbGciOi...", + "refresh_token": "eyJhbGciOiJ...", + "expires_in": 300 + } + ``` + +3. Refreshing an access token + + If the access token has expired, you can get a new one using the refresh token. + ```bash + curl -vk -X POST https://ocis.test/konnect/v1/token \ + -d "grant_type=refresh_token" \ + -d "refresh_token=eyJhbGciOiJ..." \ + -d "redirect_uri=http://path-to-redirect-uri" \ + -d "client_id=client_id" \ + -d "client_secret=client_secret" + ``` + + Response looks like this: + ```json + { + "access_token": "eyJhbGciOi...", + "token_type": "Bearer", + "expires_in": 300 + } + ``` + +== Implicit Code Flow + +When using the implicit flow, tokens are provided in a URI fragment of the redirect URL. +Valid values for the `response_type` request parameter are: + +- token +- id_token token + +// SHORTCODE: {{< hint type=warning title="Important Warning" >}} +If you are using the implicit flow, `nonce` parameter is required in the initial `/authorize` request. +`nonce=pL3UkpAQPZ8bTMGYOmxHY/dQABin8yrqipZ7iN0PY18=` + +bash command to generate cryptographically random value +[source,bash] +---- +openssl rand -base64 32 +---- +// SHORTCODE: {{< /hint >}} + +The user should be directed to a URL to authenticate and give their consent (bypassing consent is against the standard): +[source,bash] +---- +https://ocis.test/signin/v1/identifier/_/authorize?client_id=client_id&scope=openid+profile+email+offline_access&response_type=id_token+token&redirect_uri=http://path-to-redirect-uri&nonce=pL3UkpAQPZ8bTMGYOmxHY/dQABin8yrqipZ7iN0PY18= + ``` + +After a successful authentication, the browser will redirect to a URL that looks like this: +```bash +http://path-to-redirect-uri#access_token=eyJhbGciOiJQUzI...&expires_in=300&id_token=eyJhbGciOiJ...&scope=email%20openid%20profile&session_state=c8a1019f5e054d...&state=&token_type=Bearer +---- + +For the next step, extract the access_token from the URL. +[source,bash] +---- +access_token = 'eyJhbGciOiJQ...' + ``` + +## Hybrid Flow +The Hybrid Flow in OpenID Connect melds features from both the Implicit and Authorization Code flows. It allows clients to directly retrieve certain tokens from the Authorization Endpoint, yet also offers the option to acquire additional tokens from the Token Endpoint. + +The Authorization Server redirects back to the client with appropriate parameters in the response, based on the value of the response_type request parameter: +- code token +- code id_token +- code id_token token diff --git a/modules/developer/pages/apis/http/graph/groups.adoc b/modules/developer/pages/apis/http/graph/groups.adoc new file mode 100644 index 00000000..8c7b12a9 --- /dev/null +++ b/modules/developer/pages/apis/http/graph/groups.adoc @@ -0,0 +1,270 @@ += Groups +:toc: right +:toclevels: 3 + + +== Groups API + +The Groups API is implementing a subset of the functionality of the +link:https://docs.microsoft.com/en-us/graph/api/resources/group?view=graph-rest-1.0[MS Graph Group resource] +The JSON representation of a Group as handled by the Groups API looks like this: + +---- +{ + "displayName": "group", + "id": "f0d97060-da16-4b0d-9fa4-d1ec43afc5f1" +} +---- + +Our implementation currently supports two Attributes for a Group: + +| Attribute | Description | +|---------------|-----------------------------------------------------------------------------------------------------------------------------| +| displayName | The groups name | +| id | An unique, stable readonly identifier for the group that stays the same for the whole lifetime of the Group, usually a UUID | + + +=== Reading groups + +==== `GET /groups` + +Returns a list of all groups + +Example: + +---- +curl -k 'https://localhost:9200/graph/v1.0/groups' -u user:password + +---- + +Response: + +---- +{ + "value": [ + { + "displayName": "group", + "id": "38580a2e-7018-42ed-aff6-b2af0b4e9790" + }, + { + "displayName": "Example Users", + "id": "7a20f238-8a22-4458-902d-47674c317e5f" + } + ] +} +---- + + +==== `GET /groups?$expand=members` + +Returns a list of all groups including its members + +Example: + +---- +curl -k 'https://localhost:9200/graph/v1.0/groups?$expand=members' -u user:password + +---- + +Response: + +---- +{ + "value": [ + { + "displayName": "group", + "id": "38580a2e-7018-42ed-aff6-b2af0b4e9790", + "members": [ + { + "displayName": "user1", + "id": "2e7b7e23-6c42-4d34-81b0-2bed34e51983", + "mail": "user1@example.org", + "onPremisesSamAccountName": "user1" + }, + { + "displayName": "user2", + "id": "b45c9e35-0d95-4165-96bc-68bff4a316ed", + "mail": "user2@example.org", + "onPremisesSamAccountName": "user2" + } + ] + }, + { + "displayName": "Example Users", + "id": "7a20f238-8a22-4458-902d-47674c317e5f", + "members": [ + { + "displayName": "user3", + "id": "026fbfef-79ef-4f5d-887b-9eaf42777239", + "mail": "user3@example.org", + "onPremisesSamAccountName": "user3" + } + ] + } + ] +} +---- + +==== `GET /groups/{groupid}` + +Example: + +---- +curl -k 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f' -u user:password +---- + +Response: + +---- +{ + "displayName": "Example Users", + "id": "7a20f238-8a22-4458-902d-47674c317e5f" +} +---- + +==== `GET /groups/{groupid}?$expand=members` + +Example: + +---- +curl -k 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f?$expand=members' -u user:password +---- + +Response: + +---- +{ + "displayName": "Example Users", + "id": "7a20f238-8a22-4458-902d-47674c317e5f", + "members": [ + { + "displayName": "user3", + "id": "026fbfef-79ef-4f5d-887b-9eaf42777239", + "mail": "user3@example.org", + "onPremisesSamAccountName": "user3" + } + ] +} +---- +=== Getting Group Members + +==== `GET /groups/{groupid}/members` + +Returns a list of User objects that are members of a group. + +Example: + +---- +curl -k 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f/members' -u user:password + +---- + +Response: + +---- +[ + { + "displayName": "Test User", + "id": "c54b0588-7157-4521-bb52-c1c8ca84ea71", + "mail": "example@example.org", + "onPremisesSamAccountName": "example" + }, + { + "displayName": "Albert Einstein", + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "mail": "einstein@example.org", + "onPremisesSamAccountName": "einstein" + } +] +---- + +=== Creating / Updating Groups + +==== `POST /groups` + +Use this to create a new group. +h +===== Request Body + +Note the missing `"id"` Attribute. It will be generated by the server: + +---- +{ + "displayName": "Example Users" +} +---- + +===== Response + +When successful, the response will return the new group including the newly allocated `"id"`: + +---- +{ + "displayName": "Example Users", + "id": "7a20f238-8a22-4458-902d-47674c317e5f" +} +---- + +==== `DELETE /groups/{id}` + +Example: + +---- +curl -k --request DELETE 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f' -u user:password +---- + +When successful the API returns no response body and the HTTP status code 204 (No Content) + +==== `PATCH /groups/{id}` + +Updating attributes of a single group is supposed to be done with a patch request. This is however currently not fully +implemented for our write-enabled backends. The PATCH request can however be used to add multiple members to a group at once. +See below. + +=== Adding a single member to a group + +==== `POST /groups/{id}/members/$ref` + +The request body contains a single attribute "`@odata.id`" referencing the new member of the group by URI. Example: + +---- +curl -k --header "Content-Type: application/json" \ + --request POST --data \ + '{ "@odata.id": "https://localhost:9200/graph/v1.0/users/4c510ada-c86b-4815-8820-42cdf82c3d51" }' \ + 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f/members/$ref' -u user:password + +---- + +When successful the API returns no response body and the HTTP status code 204 (No Content) + +=== Adding multiple members in a single request + +==== `PATCH /groups/{id}` + +The request body contains the attribute `members@odata.bind` holding a list of URI references for the new members. +Example: + +---- +{ + "members@odata.bind": [ + "https://localhost:9200/graph/v1.0/users/4c510ada-c86b-4815-8820-42cdf82c3d51", + "https://localhost:9200/graph/v1.0/users/c54b0588-7157-4521-bb52-c1c8ca84ea71" + ] +} +---- + +When successful the API returns no response body and the HTTP status code 204 (No Content) + +=== Removing a member + +==== `DELETE /groups/{groupid}/members/{id}/$ref` + +Example + +---- +curl -k --request DELETE \ + 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f/members/4c510ada-c86b-4815-8820-42cdf82c3d51/$ref' \ + -u user:password +---- + +When successful the API returns no response body and the HTTP status code 204 (No Content) diff --git a/modules/developer/pages/apis/http/graph/index.adoc b/modules/developer/pages/apis/http/graph/index.adoc new file mode 100644 index 00000000..1a635874 --- /dev/null +++ b/modules/developer/pages/apis/http/graph/index.adoc @@ -0,0 +1,67 @@ += LibreGraph +:toc: right +:toclevels: 3 + +The LibreGraph API is a REST Api which is inspired by the link:https://developer.microsoft.com/en-us/graph[Microsoft Graph API]. It tries to stay compliant with the Microsoft Graph API and aims to be the Next Generation Api in Infinite Scale where we want to support most of the features of the platform. +The link:https://github.com/owncloud/libre-graph-api[API specification] is available in the OpenApi 3 standard and there are generated client and server link:https://github.com/owncloud/libre-graph-api#clients[SDKs] available. You can browse the API with the link:https://owncloud.dev/libre-graph-api/[Swagger UI]. + +== Calling the LibreGraph API + +[source,sh] +---- +{HTTP method} https://ocis.url/graph/{version}/{resource}?{query-parameters} +---- + +The request component consists of: + +| Component | Description | +|--------------------|-------------------------------------------------------------------------| +| {HTTP method} | The HTTP method which is used in the request. | +| {version} | The version of the LibreGraph API used by the client. | +| {resource} | The LibreGraph Resource which the client is referencing in the request. | +| {query-parameters} | Optional parameters for the request to customize the response. | + +=== HTTP methods + +| Method | Description | +|--------|-------------------------------| +| GET | Read data from a resource. | +| POST | Create a new resource. | +| PATCH | Update an existing resource. | +| PUT | Replace an existing resource. | +| DELETE | Delete an existing resource. | + +The methods `GET` and `DELETE` need no request body. The methods `POST`, `PATCH` and `PUT` require a request body, normally in JSON format to provide the needed values. + +=== Version + +Infinite Scale currently provides the version `v1.0`. + +=== Resource + +A resource could be an entity or a complex type and is usually defined by properties. Entities are always recognizable by an `Id` property. The URL contains the resource which you are interacting with e.g. `/me/drives` or `/groups/{group-id}`. + +Each resource could possibly require different permissions. Usually you need permissions on a higher level for creating or updating an existing resource than for reading. + +=== Query parameters + +Query parameters can be OData system query options, or other strings that a method accepts to customize its response. + +You can use optional OData system query options to include more or fewer properties than the default response, filter the response for items that match a custom query, or provide additional parameters for a method. + +For example, adding the following filter parameter restricts the drives returned to only those with the driveType property of `project`. + +[source,shell] +---- +GET https://ocis.url/graph/v1.0/drives?$filter=driveType eq 'project' +---- +For more information about OData query options please check the link:https://github.com/owncloud/libre-graph-api[API specification] and the provided examples. + +=== Authorization + +For development purposes the examples in the developer documentation use Basic Auth. It is disabled by default and should only be enabled by setting `PROXY_ENABLE_BASIC_AUTH` in link:../../../services/proxy/configuration/#environment-variables#environment-variables[the proxy] for development or test instances. + +To authenticate with a Bearer token or OpenID Connect access token replace the `-u user:password` Basic Auth option of curl with a `-H 'Authorization: Bearer '` header. A `` can be obtained by copying it from a request in the browser, although it will time out within minutes. To automatically refresh the OpenID Connect access token an ssh-agent like solution like link:https://github.com/indigo-dc/oidc-agent[oidc-agent] should be used. The graph endpoints that support a preconfigured token can be found in the link:https://github.com/owncloud/libre-graph-api[API specification] + +== Resources + diff --git a/modules/developer/pages/apis/http/graph/permissions.adoc b/modules/developer/pages/apis/http/graph/permissions.adoc new file mode 100644 index 00000000..7ecc22a0 --- /dev/null +++ b/modules/developer/pages/apis/http/graph/permissions.adoc @@ -0,0 +1,179 @@ += Permissions +:toc: right +:toclevels: 3 + + +== Permissions API + +The Permissions API is implementing a subset of the functionality of the +link:https://learn.microsoft.com/en-us/graph/api/resources/permission?view=graph-rest-1.0[MS Graph Permission resource]. + +=== Example Permissions + +The JSON representation of a Drive, as handled by the Spaces API, looks like this: +---- +{ + "@libre.graph.permissions.roles.allowedValues": [ + { + "id": "b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5", + "description": "Allows reading the shared file or folder", + "displayName": "Viewer", + "@libre.graph.weight": 1 + }, + { + "id": "fb6c3e19-e378-47e5-b277-9732f9de6e21", + "description": "Allows reading and writing the shared file or folder", + "displayName": "Editor", + "@libre.graph.weight": 2 + }, + { + "id": "312c0871-5ef7-4b3a-85b6-0e4074c64049", + "description": "Allows managing a space", + "displayName": "Manager", + "@libre.graph.weight": 3 + }, + { + "id": "4916f47e-66d5-49bb-9ac9-748ad00334b", + "description": "Allows creating new files", + "displayName": "File Drop", + "@libre.graph.weight": 4 + } + ], + "@libre.graph.permissions.actions.allowedValues": [ + "libre.graph/driveItem/basic/read", + "libre.graph/driveItem/permissions/read", + "libre.graph/driveItem/upload/create", + "libre.graph/driveItem/standard/allTasks", + "libre.graph/driveItem/upload/create" + ], + "value": [ + { + "id": "67445fde-a647-4dd4-b015-fc5dafd2821d", + "link": { + "type": "view", + "webUrl": "https://cloud.example.org/s/fhGBMIkKFEHWysj" + } + }, + { + "id": "34646ab6-be32-43c9-89e6-987e0c237e9b", + "roles": [ + "b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5" + ], + "grantedToV2": [ + { + "user": { + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "displayName": "Albert Einstein" + } + } + ] + }, + { + "id": "81d5bad3-3eff-410a-a2ea-eda2d14d4474", + "roles": [ + "b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5" + ], + "grantedToV2": [ + { + "user": { + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "displayName": "Albert Einstein" + } + } + ] + }, + { + "id": "b470677e-a7f5-4304-8ef5-f5056a21fff1", + "roles": [ + "b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5" + ], + "grantedToV2": [ + { + "user": { + "id": "f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c", + "displayName": "Marie Skłodowska Curie" + } + } + ] + }, + { + "id": "453b02be-4ec2-4e7d-b576-09fc153de812", + "roles": [ + "fb6c3e19-e378-47e5-b277-9732f9de6e21" + ], + "grantedToV2": [ + { + "user": { + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "displayName": "Albert Einstein" + } + } + ], + "expirationDateTime": "2018-07-15T14:00:00.000Z" + }, + { + "id": "86765c0d-3905-444a-9b07-76201f8cf7df", + "roles": [ + "312c0871-5ef7-4b3a-85b6-0e4074c64049" + ], + "grantedToV2": [ + { + "group": { + "id": "167cbee2-0518-455a-bfb2-031fe0621e5d", + "displayName": "Philosophy Haters" + } + } + ] + }, + { + "id": "c42b5cbd-2d65-42cf-b0b6-fb6d2b762256", + "grantedToV2": [ + { + "user": { + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "displayName": "Albert Einstein" + } + } + ], + "@libre.graph.permissions.actions": [ + "libre.graph/driveItem/basic/read", + "libre.graph/driveItem/path/update" + ] + } + ] +} +```` + +## Creating Share Invitation / Link + +### Create a link share `POST /drives/{drive-id}/items/{item-id}/createLink` + +https://owncloud.dev/libre-graph-api/#/drives.permissions/CreateLink + +### Create a user/group share `POST /drives/{drive-id}/items/{item-id}/invite` + +https://owncloud.dev/libre-graph-api/#/drives.permissions/Invite + +## Reading Permissions + +### List the effective sharing permissions on a driveitem `GET /drives/{drive-id}/items/{item-id}/permissions` + +https://owncloud.dev/libre-graph-api/#/drives.permissions/ListPermissions + +### List Get sharing permission for a file or folder `GET /drives/{drive-id}/items/{item-id}/permissions/{perm-id}` + +https://owncloud.dev/libre-graph-api/#/drives.permissions/GetPermission + +## Updating Permissions + +### Updating sharing permission `POST /drives/{drive-id}/items/{item-id}/permissions/{perm-id}` + +https://owncloud.dev/libre-graph-api/#/drives.permissions/UpdatePermission + +### Set password of permission `POST /drives/{drive-id}/items/{item-id}/permissions/{perm-id}/setPassword` + +https://owncloud.dev/libre-graph-api/#/drives.permissions/SetPermissionPassword + +### Deleting permission `DELETE /drives/{drive-id}/items/{item-id}/permissions/{perm-id}` + +https://owncloud.dev/libre-graph-api/#/drives.permissions/DeletePermission \ No newline at end of file diff --git a/modules/developer/pages/apis/http/graph/role.adoc b/modules/developer/pages/apis/http/graph/role.adoc new file mode 100644 index 00000000..f148c6d8 --- /dev/null +++ b/modules/developer/pages/apis/http/graph/role.adoc @@ -0,0 +1,33 @@ += Role +:toc: right +:toclevels: 3 + + +== Role API + +The Roles API is implementing a subset of the functionality of the +link:https://learn.microsoft.com/en-us/graph/api/resources/rolemanagement?view=graph-rest-1.0[MS Graph Role Management]. + +== Role Management + +=== List roleDefinitions `GET /v1beta1/roleManagement/permissions/roleDefinitions` + +https://owncloud.dev/libre-graph-api/#/roleManagement/ListPermissionRoleDefinitions + +=== Get unifiedRoleDefinition `GET /drives/{drive-id}/items/{item-id}/permissions/{perm-id}` + +https://owncloud.dev/libre-graph-api/#/roleManagement/GetPermissionRoleDefinition + +== Role Assignment + +=== Get appRoleAssignments of a user `GET /v1.0/users/{user-id}/appRoleAssignments` + +https://owncloud.dev/libre-graph-api/#/user.appRoleAssignment/user.ListAppRoleAssignments + +=== Grant an appRoleAssignment to a user `POST /v1.0/users/{user-id}/appRoleAssignments` + +https://owncloud.dev/libre-graph-api/#/user.appRoleAssignment/user.CreateAppRoleAssignments + +=== Delete the appRoleAssignment from a user `DELETE /v1.0/users/{user-id}/appRoleAssignments/{appRoleAssignment-id}` + +https://owncloud.dev/libre-graph-api/#/user.appRoleAssignment/user.DeleteAppRoleAssignments \ No newline at end of file diff --git a/modules/developer/pages/apis/http/graph/spaces.adoc b/modules/developer/pages/apis/http/graph/spaces.adoc new file mode 100644 index 00000000..4d0fa18f --- /dev/null +++ b/modules/developer/pages/apis/http/graph/spaces.adoc @@ -0,0 +1,494 @@ += Spaces +:toc: right +:toclevels: 3 + + +== Spaces API + +The Spaces API is implementing a subset of the functionality of the +link:https://learn.microsoft.com/en-us/graph/api/resources/drive?view=graph-rest-1.0[MS Graph Drives resource]. + +=== Example Space + +The JSON representation of a Drive, as handled by the Spaces API, looks like this: +---- +{ + "driveAlias": "project/mars", + "driveType": "project", + "id": "storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925", + "lastModifiedDateTime": "2023-01-24T21:19:26.417055+01:00", + "name": "Mars", + "owner": { + "user": { + "displayName": "", + "id": "89ad5ad2-5fdb-4877-b8c9-601a9670b925" + } + }, + "quota": { + "remaining": 999853685, + "state": "normal", + "total": 1000000000, + "used": 146315 + }, + "root": { + "eTag": "\"910af0061161c42d8d1224df6c4a2527\"", + "id": "storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925", + "permissions": [ + { + "grantedToIdentities": [ + { + "user": { + "displayName": "Admin", + "id": "some-admin-user-id-0000-000000000000" + } + } + ], + "roles": [ + "manager" + ] + } + ], + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925" + }, + "special": [ + { + "eTag": "\"f97829324f63ce778095334cfeb0097b\"", + "file": { + "mimeType": "image/jpeg" + }, + "id": "storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925!40171bea-3263-47a8-80ef-0ca20c37f45a", + "lastModifiedDateTime": "2022-02-15T17:11:50.000000496+01:00", + "name": "Mars_iStock-MR1805_20161221.jpeg", + "size": 146250, + "specialFolder": { + "name": "image" + }, + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925%2189ad5ad2-5fdb-4877-b8c9-601a9670b925/.space/Mars_iStock-MR1805_20161221.jpeg" + }, + { + "eTag": "\"ff38b31d8f109a4fbb98ab34499a3379\"", + "file": { + "mimeType": "text/markdown" + }, + "id": "storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925!e2167612-7578-46e2-8ed7-971481037bc1", + "lastModifiedDateTime": "2023-01-24T21:10:23.661841+01:00", + "name": "readme.md", + "size": 65, + "specialFolder": { + "name": "readme" + }, + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925%2189ad5ad2-5fdb-4877-b8c9-601a9670b925/.space/readme.md" + } + ], + "webUrl": "https://localhost:9200/f/storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925" +} +```` + +## Creating Spaces + +### Create a single space `POST /drives` + +https://owncloud.dev/libre-graph-api/#/drives/CreateDrive + +### Create a space item (Enable sync) `POST /drives/{drive-id}/root/children` + +https://owncloud.dev/libre-graph-api/#/drives.root/CreateDriveItem + +## Reading Spaces + +```shell +GET https://ocis.url/graph/{version}/{me/}drives/?{query-parameters} +---- + +| Component | Description | +|--------------------|------------------------------------------------------------------------------------------------------------------------| +| {version} | The version of the LibreGraph API used by the client. | +| {/me} | The `me` component of the part is optional. If used, you only see spaces where the acting user is a regular member of. | +| {query-parameters} | Optional parameters for the request to customize the response. | + +=== List all spaces `GET /drives` + +Returns a list of all available spaces, even ones where the acting user is not a regular member of. You need elevated permissions to do list all spaces. If you don't have the elevated permissions, the result is the same like `GET /me/drives`. + + +// SHORTCODE: {{< hint type=info title="Multiple Administration Personas" >}} + +The ownCloud spaces concept draws a strict line between users which can work with the content of a space and others who have the permission to manage the space. A user which is able to manage quota and space metadata does not necessarily need to be able to access the content of a space. + +_Space Admin_\ +There is a global user role "Space Admin" which grants users some global permissions to manage space quota and some space metadata. This Role enables the user also to disable, restore and delete spaces. He cannot manage space members. + +_Space Manager_\ +The "Space Manager" is a user which is a regular member of a space because he has been invited. In addition to being part of a space the user can also manage the memberships of the space. + +// SHORTCODE: {{< /hint >}} + +=== List My Spaces `GET /me/drives` + +https://owncloud.dev/libre-graph-api/#/me.drives/ListMyDrives + +== Modifying Spaces + +Modify the properties of a space. You need elevated permissions to execute this request. + +=== Set the space quota to 5GB `PATCH /drives/{drive-id}` + +To limit the quota of a space you need to set the `quota.total` value. The API response will give back all actual quota properties. + +---- +{ + "quota": { + "remaining": 5368709120, + "state": "normal", + "total": 5368709120, + "used": 0 + } +} +```` + +| Attribute | Description | +|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| remaining | The remaining disk space in `bytes`. If the quota is not limited, this will show the total available disk space. | +| state | The state of the space in regards to quota usage. This can be used for visual indicators. It can be `normal`(<75%), `nearing`(between 75% and 89%), `critical`(between 90% and 99%) and `exceeded`(100%). | +| total | The space id. The value needs to be a space ID. | +| used | The used disk space in bytes. | + +{{< tabs "set-space-quota" >}} +{{< tab "Request" >}} +```shell +curl -L -k -X PATCH 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff' \ +-H 'Content-Type: application/json' \ +--data-raw '{ + "quota": { + "total": 5368709120 + } +}' +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "Response - 200 OK" >}} +---- +{ + "description": "Marketing team resources", + "driveAlias": "project/marketing", + "driveType": "project", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "lastModifiedDateTime": "2023-01-18T17:13:48.385204589+01:00", + "name": "Marketing", + "owner": { + "user": { + "displayName": "", + "id": "535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + } + }, + "quota": { + "remaining": 5368709120, + "state": "normal", + "total": 5368709120, + "used": 0 + }, + "root": { + "eTag": "\"f91e56554fd9305db81a93778c0fae96\"", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "permissions": [ + { + "grantedToIdentities": [ + { + "user": { + "displayName": "Admin", + "id": "some-admin-user-id-0000-000000000000" + } + } + ], + "roles": [ + "manager" + ] + } + ], + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + }, + "webUrl": "https://localhost:9200/f/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" +} +```` +{{< /tab >}} +{{< /tabs >}} + +### Change the space name, subtitle and alias `PATCH /drives/{drive-id}` + +You can change multiple space properties in one request as long as you submit a valid JSON body. Please be aware that some properties need different permissions. + +{{< tabs "change-space-props" >}} +{{< tab "Request" >}} +```shell +curl -L -k -X PATCH 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff' \ +-H 'Content-Type: application/json' \ +--data-raw '{ + "name": "Mars", + "description": "Mission to mars", + "driveAlias": "project/mission-to-mars" +}' +---- +// SHORTCODE: {{< /tab >}} + +// SHORTCODE: {{< tab "Response - 200 OK" >}} +---- +{ + "description": "Mission to mars", + "driveAlias": "project/mission-to-mars", + "driveType": "project", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "lastModifiedDateTime": "2023-01-19T14:17:36.094283+01:00", + "name": "Mars", + "owner": { + "user": { + "displayName": "", + "id": "535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + } + }, + "quota": { + "remaining": 15, + "state": "normal", + "total": 15, + "used": 0 + }, + "root": { + "eTag": "\"f5fee4fdfeedd6f98956500779eee15e\"", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "permissions": [ + { + "grantedToIdentities": [ + { + "user": { + "displayName": "Admin", + "id": "some-admin-user-id-0000-000000000000" + } + } + ], + "roles": [ + "manager" + ] + } + ], + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + }, + "webUrl": "https://localhost:9200/f/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" +} +```` +{{< /tab >}} +{{< /tabs >}} + +## Disabling / Deleting Spaces + +### Disable a space `DELETE /drives/{drive-id}` + +This operation will make the space content unavailable for all space members. No data will be deleted. + +{{< tabs "disable-space" >}} +{{< tab "Request" >}} +```shell +curl -L -k -X DELETE 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff/' +---- +// SHORTCODE: {{< /tab >}} + +// SHORTCODE: {{< tab "Response - 204 No Content" >}} + +This response has no body value. + +A disabled space will appear in listings with a `root.deleted.state=trashed` property. The space description and the space image will not be readable anymore. + +[source,json] +---- +{ + "description": "Marketing team resources", + "driveAlias": "project/marketing", + "driveType": "project", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "lastModifiedDateTime": "2023-01-19T14:17:36.094283+01:00", + "name": "Marketing", + "owner": { + "user": { + "displayName": "", + "id": "535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + } + }, + "quota": { + "total": 15 + }, + "root": { + "deleted": { + "state": "trashed" + }, + "eTag": "\"f5fee4fdfeedd6f98956500779eee15e\"", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "permissions": [ + { + "grantedToIdentities": [ + { + "user": { + "displayName": "Admin", + "id": "some-admin-user-id-0000-000000000000" + } + } + ], + "roles": [ + "manager" + ] + } + ], + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + }, + "webUrl": "https://localhost:9200/f/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" +} +---- + +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< /tabs >}} + +=== Restore a space `PATCH /drives/{drive-id}` + +This operation will make the space content available again to all members. No content will be changed. + +To restore a space, the Header `Restore: T` needs to be set. +// SHORTCODE: {{< tabs "restore-space" >}} +// SHORTCODE: {{< tab "Request" >}} + +[source,shell] +---- +curl -L -X PATCH 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff/' \ +-H 'Restore: T' \ +-H 'Content-Type: text/plain' \ +--data-raw '{}' +---- + +// SHORTCODE: {{< hint type=info title="Body value" >}} + +This request needs an empty body (--data-raw '{}') to fulfil the standard libregraph specification even when the body is not needed. + +// SHORTCODE: {{< /hint >}} +// SHORTCODE: {{< /tab >}} + +// SHORTCODE: {{< tab "Response - 200 OK" >}} + +[source,json] +---- +{ + "description": "Marketing team resources", + "driveAlias": "project/marketing", + "driveType": "project", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "lastModifiedDateTime": "2023-01-19T14:17:36.094283+01:00", + "name": "Marketing", + "owner": { + "user": { + "displayName": "", + "id": "535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + } + }, + "quota": { + "remaining": 15, + "state": "normal", + "total": 15, + "used": 0 + }, + "root": { + "eTag": "\"f5fee4fdfeedd6f98956500779eee15e\"", + "id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff", + "permissions": [ + { + "grantedToIdentities": [ + { + "user": { + "displayName": "Admin", + "id": "some-admin-user-id-0000-000000000000" + } + } + ], + "roles": [ + "manager" + ] + } + ], + "webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" + }, + "webUrl": "https://localhost:9200/f/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff" +} +---- + +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< /tabs >}} + +=== Permanently delete a space `DELETE /drives/{drive-id}` + +This operation will delete a space and all its data permanently. This is restricted to spaces which are already disabled. + +To delete a space, the Header `Purge: T` needs to be set. + +// SHORTCODE: {{< tabs "delete-space" >}} +// SHORTCODE: {{< tab "Request" >}} + +[source,shell] +---- +curl -L -X DELETE 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff' \ +-H 'Purge: T' +---- + +// SHORTCODE: {{< hint type=warning title="Data will be deleted" >}} + +This request will delete a space and all its content permanently. This operation cannot be reverted. + +// SHORTCODE: {{< /hint >}} + +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "Response - 204 No Content" >}} + +This response has no body value. + +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "Response - 400 Bad Request" >}} + +The space to be deleted was not disabled before. + +[source,json] +---- +{ + "error": { + "code": "invalidRequest", + "innererror": { + "date": "2023-01-24T19:57:19Z", + "request-id": "f62af40f-bc18-475e-acd7-e9008d6bd326" + }, + "message": "error: bad request: can't purge enabled space" + } +} +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< /tabs >}} + +== Sharing Space + +=== Add member to space `POST /drives/{drive-id}/root/invite` + +https://owncloud.dev/libre-graph-api/#/drives.permissions/Invite + +=== Sharing space as a link `POST /drives/{drive-id}/root/createLink` + +https://owncloud.dev/libre-graph-api/#/drives.root/CreateLinkSpaceRoot + +== Reading Space Permissions + +=== Listing permissions of a space `GET /drives/{drive-id}/root/permissions` + +https://owncloud.dev/libre-graph-api/#/drives.root/ListPermissionsSpaceRoot + +== Modifying / Deleting Space Permissions + +=== Update permissions of a drive `PATCH /drives/{drive-id}/root/permissions/{perm-id}` + +https://owncloud.dev/libre-graph-api/#/drives.root/UpdatePermissionSpaceRoot + +=== Set password of a link share `POST /drives/{drive-id}/root/permissions/{perm-id}/setPassword` + +https://owncloud.dev/libre-graph-api/#/drives.root/SetPermissionPasswordSpaceRoot + +=== Removing acess to a space `DELETE /drives/{drive-id}/root/permissions/{perm-id}` + +https://owncloud.dev/libre-graph-api/#/drives.root/DeletePermissionSpaceRoot diff --git a/modules/developer/pages/apis/http/graph/users.adoc b/modules/developer/pages/apis/http/graph/users.adoc new file mode 100644 index 00000000..0cb36eb3 --- /dev/null +++ b/modules/developer/pages/apis/http/graph/users.adoc @@ -0,0 +1,265 @@ += Users +:toc: right +:toclevels: 3 + + +== Users API + +The Users API is implementing a subset of the functionality of the +link:https://docs.microsoft.com/en-us/graph/api/resources/user?view=graph-rest-1.0[MS Graph User resource] +The JSON representation of a User handled by the Users API looks like this: + +---- +{ + "displayName": "Albert Einstein", + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "mail": "einstein@example.org", + "onPremisesSamAccountName": "einstein" +} +---- + +Our implementation currently supports only a limited set of Attributes of Users: + +| Attribute | Description | +|---------------|---------------------------------------------------------------------------------------------------------------------------| +| displayName | The full name of the user, usually a combination of given name and last name | +| mail | The user's email address | +| onPremisesSamAccountName | The loginname/account name of the user | +| id | An unique, stable readonly identifier for the user that stays the same for the whole lifetime of the User, usually a UUID | +| passwordProfile | Contains the password of the users. This is only present when updating or creating users. It is never returned by the API | + + +=== Reading users + +==== `GET /me` + +Returns the user object of the currently signed-in user + +Example: +---- +curl -k 'https://localhost:9200/graph/v1.0/me' -u user:password +---- + +Response: +---- +{ + "displayName": "Albert Einstein", + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "mail": "einstein@example.org", + "onPremisesSamAccountName": "einstein" +} +---- + +==== `GET /users` + +Returns a list of all users + +Example: + +---- +curl -k 'https://localhost:9200/graph/v1.0/users' -u user:password + +---- + +Response: + +---- +{ + "value": [ + { + "displayName": "Albert Einstein", + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "mail": "einstein@example.org", + "onPremisesSamAccountName": "einstein" + }, + { + "displayName": "Maurice Moss", + "id": "058bff95-6708-4fe5-91e4-9ea3d377588b", + "mail": "moss@example.org", + "onPremisesSamAccountName": "moss" + } + ] +} +---- + +==== `GET /users?$expand=memberOf` + +Returns a list of all users + +Example: + +---- +curl -k 'https://localhost:9200/graph/v1.0/users?$expand=memberOf' -u user:password + +---- + +Response: + +---- +{ + "value": [ + { + "displayName": "Albert Einstein", + "id": "4c510ada-c86b-4815-8820-42cdf82c3d51", + "mail": "einstein@example.org", + "onPremisesSamAccountName": "einstein", + "memberOf": [ + { + "displayName": "users", + "id": "509a9dcd-bb37-4f4f-a01a-19dca27d9cfa" + }, + { + "displayName": "sailing-lovers", + "id": "6040aa17-9c64-4fef-9bd0-77234d71bad0" + }, + { + "displayName": "violin-haters", + "id": "dd58e5ec-842e-498b-8800-61f2ec6f911f" + }, + { + "displayName": "physics-lovers", + "id": "262982c1-2362-4afa-bfdf-8cbfef64a06e" + } + ], + }, + { + "displayName": "Maurice Moss", + "id": "058bff95-6708-4fe5-91e4-9ea3d377588b", + "mail": "moss@example.org", + "onPremisesSamAccountName": "moss", + "memberOf": [ + { + "displayName": "users", + "id": "509a9dcd-bb37-4f4f-a01a-19dca27d9cfa" + } + ], + } + ] +} +---- + +==== `GET /users/{userid or accountname}` + +Example: + +---- +curl -k 'https://localhost:9200/graph/v1.0/users/058bff95-6708-4fe5-91e4-9ea3d377588b' -u user:password +---- + +Response: + +---- +{ + "displayName": "Maurice Moss", + "id": "058bff95-6708-4fe5-91e4-9ea3d377588b", + "mail": "moss@example.org", + "onPremisesSamAccountName": "moss" +} +---- + +==== `GET /users/{userid or accountname}?$expand=memberOf` + +Example: + +---- +curl -k 'https://localhost:9200/graph/v1.0/users/058bff95-6708-4fe5-91e4-9ea3d377588b?$expand=memberOf' -u user:password +---- + +Response: + +---- +{ + "displayName": "Maurice Moss", + "id": "058bff95-6708-4fe5-91e4-9ea3d377588b", + "mail": "moss@example.org", + "onPremisesSamAccountName": "moss", + "memberOf": [ + { + "displayName": "users", + "id": "509a9dcd-bb37-4f4f-a01a-19dca27d9cfa" + } + ], +} +---- + +=== Creating / Updating Users + +==== `POST /users` + +Use this to create a new user. + +===== Request Body + +Note the missing `"id"` Attribute. It will be generated by the server: + +---- +{ + "displayName": "Example User", + "mail": "example@example.org", + "onPremisesSamAccountName": "example", + "passwordProfile": { + "password": "ThePassword" + } +} +---- + +===== Response + +When successful, the response will return the new user, without the password, but including the newly allocated `"id"`: + +---- +{ + "displayName":"Example User", + "id":"c067b139-c91c-4e47-8be6-669156a0587b", + "mail":"example@example.org", + "onPremisesSamAccountName":"example" +} +---- + +==== `DELETE /users/{id}` + +Example: + +---- +curl -k --request DELETE 'https://localhost:9200/graph/v1.0/users/c067b139-c91c-4e47-8be6-669156a0587b' -u user:password +---- + +When successful the API returns no response body and the HTTP status code 204 (No Content) + + +==== `PATCH /users/{id}` + +Updating attributes of a single user can be done with a patch request. The Request Body contains the new values of the attributes +to be updated. E.g. to update the `displayName` Attribute: + +---- + curl -k --header "Content-Type: application/json" \ + --request PATCH --data '{"displayName": "Test User" }' \ + 'https://localhost:9200/graph/v1.0/users/c54b0588-7157-4521-bb52-c1c8ca84ea71' -u user:password +---- + +Similar to creating a user via `POST`, the `PATCH` request will return the user object containing the new attribute values. + +=== Change password + +==== `POST /me/changePassword` + +Users can change their own password by sending a POST request to `/me/changePassword` + +===== Request Body + +---- +{ + "currentPassword": "current", + "newPassword": "new" +} + +---- + +When successful the API returns no response body and the HTTP status code 204 (No Content) + +---- + curl -i -k --header "Content-Type: application/json" \ + --request POST --data '{"currentPassword": "current", "newPassword": "new" }' \ + 'https://localhost:9200/graph/v1.0/me/changePassword' -u user:current +---- diff --git a/modules/developer/pages/apis/http/index.adoc b/modules/developer/pages/apis/http/index.adoc new file mode 100644 index 00000000..b47f78de --- /dev/null +++ b/modules/developer/pages/apis/http/index.adoc @@ -0,0 +1,7 @@ += Http +:toc: right +:toclevels: 3 + +The link:https://www.rfc-editor.org/rfc/rfc2616[Hypertext Transfer Protocol (HTTP)] is a stateless application-level protocol for distributed, collaborative, hypertext information systems. HTTP is the foundation of data communication for the World Wide Web, where hypertext documents include hyperlinks to other resources that the user can easily access, for example by a mouse click or by tapping the screen in a web browser. + +Development of HTTP was initiated by Tim Berners-Lee at CERN in 1989 and summarized in a simple document describing the behavior of a client and a server using the first HTTP protocol version that was named 0.9. That first version of HTTP protocol soon evolved into a more elaborated version that was the first draft toward a far future version 1.0 diff --git a/modules/developer/pages/apis/http/tus_upload.adoc b/modules/developer/pages/apis/http/tus_upload.adoc new file mode 100644 index 00000000..271d0106 --- /dev/null +++ b/modules/developer/pages/apis/http/tus_upload.adoc @@ -0,0 +1,248 @@ += Resumable Upload +:toc: right +:toclevels: 3 + +Infinite Scale supports the tus resumable-upload protocol, which is a robust, modular, and open protocol designed to resume large file uploads reliably over HTTP. +In situations where file uploads might be interrupted due to network issues, browser crashes, or other unforeseen interruptions, +tus ensures that uploads can be resumed from the point of failure without losing data. +This documentation shows some basic examples, refer link:https://tus.io/protocols/resumable-upload[tus official site] for more details. + +== Supported tus Features + +The backend announces certain tus features to clients. WebDAV responses come with tus HTTP headers for the offical tus features, and additional, ownCloud specific features are announced via the capabilities endpoint (e.g. `https://localhost:9200/ocs/v1.php/cloud/capabilities?format=json`). + +The following snippet shows the relevant part of the server capabilities of Infinite Scale that concerns the tus upload: +[source,json] +---- +{ + "ocs": { + "data": { + "capabilities": { + "files": { + "tus_support": { + "version": "1.0.0", + "resumable": "1.0.0", + "extension": "creation,creation-with-upload", + "max_chunk_size": 10000000, + "http_method_override": "" + } + } + } + } + } + } +} +---- + +| Parameter | Environment Variable | Default Value | Description | +| -------------- | ------------------------------ | ------------- | ------------------------------------------------------------------- | +| max_chunk_size | FRONTEND_UPLOAD_MAX_CHUNK_SIZE | 10000000 | Announces the max chunk sizes in bytes for uploads via the clients. | + +== Upload in Chunks + +=== Create an Upload URL + +The client must send a POST request against a known upload creation URL to request a new upload resource. +The filename has to be provided in base64-encoded format. + +Example: +[source,shell] +---- +# base64 encoded filename 'tustest.txt' is 'dHVzdGVzdC50eHQ=' +echo -n 'tustest.txt' | base64 +---- + +// SHORTCODE: {{< tabs "create-upload-url" >}} +// SHORTCODE: {{< tab "Request" >}} +[source,shell] +---- +curl -ks -XPOST https://ocis.test/remote.php/dav/spaces/8d72036d-14a5-490f-889e-414064156402$196ac304-7b88-44ce-a4db-c4becef0d2e0 \ +-H "Authorization: Bearer eyJhbGciOiJQUzI..."\ +-H "Tus-Resumable: 1.0.0" \ +-H "Upload-Length: 10" \ +-H "Upload-Metadata: filename dHVzdGVzdC50eHQ=" +---- +// SHORTCODE: {{< /tab >}} + +// SHORTCODE: {{< tab "Response - 201 Created" >}} +---- +< HTTP/1.1 201 Created +< Access-Control-Allow-Headers: Tus-Resumable, Upload-Length, Upload-Metadata, If-Match +< Access-Control-Allow-Origin: * +< Access-Control-Expose-Headers: Tus-Resumable, Upload-Offset, Location +< Content-Length: 0 +< Content-Security-Policy: default-src 'none'; +< Date: Mon, 16 Oct 2023 08:49:39 GMT +< Location: https://ocis.test/data/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNjk3NTMyNTc5LCJpYXQiOjE2OTc0NDYxNzksInRhcmdldCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE1OC9kYXRhL3R1cy8zYTU3ZWZlMS04MzE0LTQ4MGEtOWY5Ny04N2Q1YzBjYTJhMTgifQ.FbrlY7mdOfsbFgMrP8OtcHlCEq72a2ZVnPD2iBo9MfM +< Tus-Extension: creation,creation-with-upload,checksum,expiration +< Tus-Resumable: 1.0.0 +< Vary: Origin +< X-Content-Type-Options: nosniff +< X-Download-Options: noopen +< X-Frame-Options: SAMEORIGIN +< X-Permitted-Cross-Domain-Policies: none +< X-Request-Id: xxxxxxxxxxxxxxxxxxxxxx +< X-Robots-Tag: none +< +* Connection #0 to host localhost left intact +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< /tabs >}} + +The server will return a temporary upload URL in the location header of the response: +---- +< Location: https://ocis.test/data/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNjk3NTMyNTc5LCJpYXQiOjE2OTc0NDYxNzksInRhcmdldCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE1OC9kYXRhL3R1cy8zYTU3ZWZlMS04MzE0LTQ4MGEtOWY5Ny04N2Q1YzBjYTJhMTgifQ.FbrlY7mdOfsbFgMrP8OtcHlCEq72a2ZVnPD2iBo9MfM +---- + +=== Upload the First Chunk + +Once a temporary upload URL has been created, a client can send a PATCH request to upload a file. The file content should be sent in the body of the request: +// SHORTCODE: {{< tabs "upload-the-first-chunk" >}} +// SHORTCODE: {{< tab "Request" >}} +[source,shell] +---- +curl -ks -XPATCH https://temporary-upload-url \ +-H "Authorization: Bearer eyJhbGciOiJQUzI..." \ +-H "Tus-Resumable: 1.0.0" \ +-H "Upload-Offset: 0" \ +-H "Content-Type: application/offset+octet-stream" -d "01234" +---- +// SHORTCODE: {{< /tab >}} + +// SHORTCODE: {{< tab "Response - 204 No Content" >}} +---- +< HTTP/1.1 204 No Content +< Date: Tue, 17 Oct 2023 04:10:52 GMT +< Oc-Fileid: 8d72036d-14a5-490f-889e-414064156402$73bb5450-816b-4cae-90aa-1f96adc95bd4!84e319e4-de1d-4dd8-bbd0-e51d933cdbcd +< Tus-Resumable: 1.0.0 +< Upload-Expires: 1697602157 +< Upload-Offset: 5 +< Vary: Origin +< X-Content-Type-Options: nosniff +< X-Request-Id: xxxxxxxxxxxxxxxxxxxxxx +< +* Connection #0 to host localhost left intact +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< /tabs >}} + +=== Upload Further Chunks + +After the first chunk is uploaded, the second chunk can be uploaded by pointing `Upload-Offset` to exact position that was returned in the first response. +Upload process will not be marked as complete until the total uploaded content size matches the `Upload-Length` specified during the creation of the temporary URL. + +// SHORTCODE: {{< tabs "upload-the-second-chunk" >}} +// SHORTCODE: {{< tab "Request" >}} +[source,shell] +---- +curl -ks -XPATCH https://temporary-upload-url \ +-H "Authorization: Bearer eyJhbGciOiJQUzI..." \ +-H "Tus-Resumable: 1.0.0" \ +-H "Upload-Offset: 5" \ +-H "Content-Type: application/offset+octet-stream" -d "56789" +---- +// SHORTCODE: {{< /tab >}} + +// SHORTCODE: {{< tab "Response - 204 No Content" >}} +---- +< HTTP/1.1 204 No Content +< Date: Tue, 17 Oct 2023 04:11:00 GMT +< Oc-Fileid: 8d72036d-14a5-490f-889e-414064156402$73bb5450-816b-4cae-90aa-1f96adc95bd4!84e319e4-de1d-4dd8-bbd0-e51d933cdbcd +< Tus-Resumable: 1.0.0 +< Upload-Expires: 1697602157 +< Upload-Offset: 10 +< Vary: Origin +< X-Content-Type-Options: nosniff +< X-Request-Id: xxxxxxxxxxxxxxxxxxxxxx +< +* Connection #0 to host localhost left intact +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< /tabs >}} +// SHORTCODE: {{< hint type=warning title="Important Warning" >}} +`Upload-Offset` header indicates the byte position in the target file where the server should start writing the upload content. +It ensures data integrity and order during the upload process. +// SHORTCODE: {{< /hint >}} + +== Creation with Upload + +// SHORTCODE: {{< tabs "creation-with-upload" >}} +// SHORTCODE: {{< tab "Request" >}} +[source,shell] +---- +curl -ks -XPOST https://ocis.test/remote.php/dav/spaces/{space-id} \ +-H "Authorization: Bearer eyJhbGciOiJQUzI..." \ +-H "Tus-Resumable: 1.0.0" \ +-H "Upload-Length: 14" \ +-H "Content-Type: application/offset+octet-stream" \ +-H "Upload-Metadata: filename dGVzdC50eHQ=" \ +-H "Tus-Extension: creation-with-upload" \ +-d "upload content" +---- +// SHORTCODE: {{< /tab >}} + +// SHORTCODE: {{< tab "Response - 201 Created" >}} +[source,shell] +---- +< HTTP/1.1 201 Created +< Access-Control-Allow-Headers: Tus-Resumable, Upload-Length, Upload-Metadata, If-Match +< Access-Control-Allow-Origin: * +< Access-Control-Expose-Headers: Tus-Resumable, Upload-Offset, Location +< Content-Length: 0 +< Content-Security-Policy: default-src 'none'; +< Content-Type: text/plain +< Date: Mon, 16 Oct 2023 04:18:25 GMT +< Etag: "372c96743f68bc40e789124d30567371" +< Last-Modified: Mon, 16 Oct 2023 04:18:25 +0000 +< Location: https://ocis.test/data/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNjk3NTE2MzA1LCJpYXQiOjE2OTc0Mjk5MDUsInRhcmdldCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE1OC9kYXRhL3R1cy82NjlhODBlZi1hN2VjLTQwYTAtOGNmOS05MTgwNTVhYzlkZjAifQ.yq-ofJYnJ9FLML7Z_jki1FJQ7Ulbt9O_cmLe6V411A4 +< Oc-Etag: "372c96743f68bc40e789124d30567371" +< Oc-Fileid: 44d3e1e0-6c01-4b94-9145-9d0068239fcd$446bdad4-4b27-41f1-afce-0881f202a214!d7c292a6-c395-4e92-bf07-2c1663aec8dd +< Oc-Perm: RDNVWZP +< Tus-Extension: creation,creation-with-upload,checksum,expiration +< Tus-Resumable: 1.0.0 +< Upload-Expires: 1697516305 +< Upload-Offset: 14 +< Vary: Origin +< X-Content-Type-Options: nosniff +< X-Download-Options: noopen +< X-Frame-Options: SAMEORIGIN +* TLSv1.2 (IN), TLS header, Supplemental data (23): +{ [5 bytes data] +< X-Permitted-Cross-Domain-Policies: none +< X-Request-Id: xxxxxxxxxxxxxxxxxxxxxx +< X-Robots-Tag: none +< +* Connection #0 to host localhost left intact +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< /tabs >}} + +// SHORTCODE: {{< hint type=warning title="Important Warning" >}} +The `Upload-Length` header of the request has to contain the exact size of the upload content in byte. +// SHORTCODE: {{< /hint >}} + +== Supported Upload-Metadata + +Upload-metadata key-value pairs aren't specified in the general tus docs. The following ones are supported in the ownCloud ecosystem: + +| Parameter (key) | Example (value, MUST be Base64 encoded) | Description | +| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | +| `name` OR `filename` (mandatory) | example.pdf | Filename | +| `mtime` (recommended) | 1701708712 | Modification time (Unix time format) | +| `checksum` (recommended) | SHA1 a330de5886e5a92d78fb3f8d59fe469857759e72 | Checksum, computed from the client | +| `type` OR `filetype` | application/pdf | MIME Type, sent by the web UI | +| `relativePath` | undefined | File path relative to the folder that is being uploaded, including the filename. Sent by the web UI | +| `spaceId` | 8748cddf-66b7-4b85-91a7-e6d08d8e1639$a9778d63-21e7-4d92-9b47-1b81144b9993 | Sent by the web UI | +| `spaceName` | Personal | Sent by the web UI | +| `driveAlias` | personal/admin | Sent by the web UI | +| `driveType` | personal | Sent by the web UI | +| `currentFolder` | / | Sent by the web UI | +| `currentFolderId` | 8748cddf-66b7-4b85-91a7-e6d08d8e1639$a9778d63-21e7-4d92-9b47-1b81144b9993!a9778d63-21e7-4d92-9b47-1b81144b9993 | Sent by the web UI | +| `uppyId` | uppy-example/pdf-1e-application/pdf-238300 | Sent by the web UI | +| `relativeFolder` | | File path relative to the folder that is being uploaded, without filename. Sent by the web UI. | +| `tusEndpoint` | https://ocis.ocis-traefik.latest.owncloud.works/remote.php/dav/spaces/8748cddf-66b7-4b85-91a7-e6d08d8e1639$a9778d63-21e7-4d92-9b47-1b81144b9993 | Sent by the web UI | +| `uploadId` | 71d5f878-a96c-4d7b-9627-658d782c93d7 | Sent by the web UI | +| `topLevelFolderId` | undefined | Sent by the web UI | +| `routeName` | files-spaces-generic | Sent by the web UI | +| `routeDriveAliasAndItem` | cGVyc29uYWwvYWRtaW4= | Sent by the web UI | +| `routeShareId` | | Share ID when uploading into a received folder share. Sent by the web UI | diff --git a/modules/developer/pages/apis/http/webdav/index.adoc b/modules/developer/pages/apis/http/webdav/index.adoc new file mode 100644 index 00000000..4a62ee72 --- /dev/null +++ b/modules/developer/pages/apis/http/webdav/index.adoc @@ -0,0 +1,556 @@ += WebDAV +:toc: right +:toclevels: 3 + + +_Web_ _D_istributed _A_uthoring and _V_ersioning (WebDAV) consists of a set of methods, headers, and content-types extending HTTP/1.1 for the management of resources and -properties, creation and management of resource collections, URL namespace manipulation, and resource locking (collision avoidance). WebDAV is one of the central APIs that ownCloud uses for handling file resources, metadata and locks. + + +// SHORTCODE: {{< hint type=info title="RFC" >}} +_WebDAV RFCs_ + +RFC 2518 was published in February 1999. link:https://datatracker.ietf.org/doc/html/rfc4918[RFC 4918], published in June 2008 obsoletes RFC 2518 with minor revisions mostly due to interoperability experience. + +// SHORTCODE: {{< /hint >}} +== Calling the WebDAV API + +=== Request URI + +[source,sh] +---- +{HTTP method} https://ocis.url/{webdav-base}/{resourceID}/{path} +---- + +The request URI consists of: + +| Component | Description | +|---------------|--------------------------------------------------------------------------------------------------------| +| {HTTP method} | The HTTP method which is used in the request. | +| {webdav-base} | The WebDAV base path component. Possible options are | +| | `dav/spaces/` This is the default and optimized endpoint for all WebDAV requests. | +| | `remote.php/dav/spaces/`* | +| | `remote.php/webdav/`* | +| | `webdav/`* | +| | `dav/`* | +| {resourceID} | This resourceID is used as the WebDAV root element. All children are accessed by their relative paths. | +| {path} | The relative path to the WebDAV root. In most of the casese, this is the space root. | + +\* these dav endpoints are implemented for legacy reasons and should not be used. Note: The legacy endpoints _do not take the resourceID as an argument._ + +=== HTTP methods + +| Method | Description | +|-----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| PROPFIND | Retrieve properties as XML from a web resource. It is also overloaded to retrieve the collection structure (a.k.a. directory hierarchy) of a remote system. | +| PROPPATCH | Process instructions specified in the request body to set and/or remove properties defined on the resource identified by the request uri. | +| MKCOL | Create a WebDAV collection (folder) at the location specified by the request uri. | +| GET | Retrieve a WebDAV resource. | +| HEAD | Retrieve a WebDAV resource without reading the body. | +| PUT | A PUT performed on an existing resource replaces the GET response entity of the resource. | +| POST | Not part of the WebDAV rfc and has no effect on a WebDAV resource. However, this method is used in the TUS protocol for uploading resources. | +| PATCH | Not part of the WebDAV rfc and has no effect on a WebDAV resource. However, this method is used in the TUS protocol for uploading resources. | +| COPY | Creates a duplicate of the source resource identified by the Request-URI, in the destination resource identified by the URI in the Destination header. | +| MOVE | The MOVE operation on a non-collection resource is the logical equivalent of a copy (COPY), followed by consistency maintenance processing, followed by a delete of the source, where all three actions are performed in a single operation. | | +| DELETE | Delete the resource identified by the Request-URI. | +| LOCK | A LOCK request to an existing resource will create a lock on the resource identified by the Request-URI, provided the resource is not already locked with a conflicting lock. | +| UNLOCK | The UNLOCK method removes the lock identified by the lock token in the Lock-Token request header. The Request-URI must identify a resource within the scope of the lock. | + +The methods `MKCOL`, `GET`, `HEAD`, `LOCK`, `COPY`, `MOVE`, `UNLOCK` and `DELETE` need no request body. + +The methods `PROPFIND`, `PROPPATCH`, `PUT` require a request body, normally in XML format to provide the needed values. + +// SHORTCODE: {{< hint type=tip title="Tooling" >}} +_WebDAV is not REST_ + +The WebDAV protocol was created before the REST paradigm has become the de-facto standard for API design. WebDAV uses http methods which are not part of REST. Therefore all the tooling around API design and documentation is not usable (like OpenApi 3.0 / Swagger or others). +// SHORTCODE: {{< /hint >}} + +=== Authentication + +For development purposes the examples in the developer documentation use Basic Auth. It is disabled by default and should only be enabled by setting `PROXY_ENABLE_BASIC_AUTH` in link:../../../services/proxy/configuration/#environment-variables#environment-variables[the proxy] for development or test instances. + +To authenticate with a Bearer token or OpenID Connect access token replace the `-u user:password` Basic Auth option of curl with a `-H 'Authorization: Bearer '` header. A `` can be obtained by copying it from a request in the browser, although it will time out within minutes. To automatically refresh the OpenID Connect access token an ssh-agent like solution like link:https://github.com/indigo-dc/oidc-agent[oidc-agent] should be used. + +== Listing Properties + +This method is used to list the properties of a resource in xml. This method can also be used to retrieve the listing of a WebDAV collection which means the content of a remote directory. + +// SHORTCODE: {{< tabs "list-properties" >}} +// SHORTCODE: {{< tab "Curl" >}} +[source,shell] +---- +curl -L -X PROPFIND 'https://localhost:9200/dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/' \ +-H 'Depth: 1' \ +-d ' + + + + + + + + + + + + + + + + + +' +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "HTTP" >}} +[source,shell] +---- +PROPFIND /dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/ HTTP/1.1 +Host: localhost:9200 +Origin: https://localhost +Access-Control-Request-Method: PROPFIND +Depth: 1 +Content-Type: application/xml +Authorization: Basic YWRtaW46YWRtaW4= +Content-Length: 436 + + + + + + + + + + + + + + + + + + + + +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< /tabs >}} + +The request consists of a request body and an optional `Depth` Header. + +// SHORTCODE: {{< hint type=tip title="PROPFIND usage" >}} +_Metadata and Directory listings_ + +Clients can use the `PROPFIND` method to retrieve properties of resources (metadata) and to list the content of a directories. +// SHORTCODE: {{< /hint >}} +=== Response + +// SHORTCODE: {{< tabs "response list properties" >}} +// SHORTCODE: {{< tab "207 - Multistatus" >}} + +==== Multi Status Response + +A Multi-Status response conveys information about multiple resources +in situations where multiple status codes might be appropriate. The +default Multi-Status response body is an application/xml +HTTP entity with a `multistatus` root element. Further elements +contain `200`, `300`, `400`, and `500` series status codes generated during +the method invocation. + +Although `207` is used as the overall response status code, the +recipient needs to consult the contents of the multistatus response +body for further information about the success or failure of the +method execution. The response MAY be used in success, partial +success and also in failure situations. + +The `multistatus` root element holds zero or more `response` elements +in any order, each with information about an individual resource. + +==== Body + +[source,xml] +---- + + + /dav/spaces/storage-users-1$some-admin-user-id-0000-000000000000/ + + + RDNVCKZP + 0 + storage-users-1$some-admin-user-id-0000-000000000000!some-admin-user-id-0000-000000000000 + storage-users-1$some-admin-user-id-0000-000000000000!some-admin-user-id-0000-000000000000 + admin + Admin + https://localhost:9200/f/storage-users-1$some-admin-user-id-0000-000000000000%21some-admin-user-id-0000-000000000000 + 10364682 + Mon, 04 Sep 2023 20:10:09 GMT + "c4d3610dfe4fac9b44e1175cfc44b12b" + + + + + HTTP/1.1 200 OK + + + + + + + + + HTTP/1.1 404 Not Found + + + + /dav/spaces/storage-users-1$some-admin-user-id-0000-000000000000/New%20file.txt + + + RDNVWZP + + SHA1:1c68ea370b40c06fcaf7f26c8b1dba9d9caf5dea MD5:2205e48de5f93c784733ffcca841d2b5 ADLER32:058801ab + + 0 + storage-users-1$some-admin-user-id-0000-000000000000!90cc3e73-0c6c-4346-9c4d-f529976d4990 + storage-users-1$some-admin-user-id-0000-000000000000!90cc3e73-0c6c-4346-9c4d-f529976d4990 + admin + Admin + + 0 + 1 + 3 + + https://localhost:9200/f/storage-users-1$some-admin-user-id-0000-000000000000%2190cc3e73-0c6c-4346-9c4d-f529976d4990 + 5 + 5 + Mon, 28 Aug 2023 20:45:03 GMT + "75115347c74701a3be9c635ddebbf5c4" + text/plain + + + HTTP/1.1 200 OK + + + + /dav/spaces/storage-users-1$some-admin-user-id-0000-000000000000/NewFolder/ + + + RDNVCKZP + 0 + storage-users-1$some-admin-user-id-0000-000000000000!5c73ecd9-d9f4-44f4-b685-ca4cb40aa6b7 + storage-users-1$some-admin-user-id-0000-000000000000!5c73ecd9-d9f4-44f4-b685-ca4cb40aa6b7 + admin + Admin + https://localhost:9200/f/storage-users-1$some-admin-user-id-0000-000000000000%215c73ecd9-d9f4-44f4-b685-ca4cb40aa6b7 + 0 + Mon, 28 Aug 2023 20:45:10 GMT + "e83367534cc595a45d706857fa5f03d8" + + + + + HTTP/1.1 200 OK + + + + + + + + + HTTP/1.1 404 Not Found + + + +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "400 - Bad Request" >}} + +==== Body + +[source,xml] +---- + + + Sabre\DAV\Exception\BadRequest + Invalid Depth header value: 3 + +---- + +This can occur if the request is malformed e.g. due to an invalid xml request body or an invalid depth header value. +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "404 - Not Found" >}} + +==== Body + +[source,xml] +---- + + + Sabre\DAV\Exception\NotFound + Resource not found + +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< /tabs >}} + +=== Request Body + +The `PROPFIND` Request can include an XML request body containing a list of namespaced property names. + +=== Namespaces + +When building the body of your DAV request, you will request properties that are available under a specific namespace URI. It is usual to declare prefixes for those namespace in the `d:propfind` element of the body. + +Available namespaces: + +| URI | Prefix | +|-------------------------------------------|--------| +| DAV: | d | +| http://sabredav.org/ns | s | +| http://owncloud.org/ns | oc | +| http://open-collaboration-services.org/ns | ocs | +| http://open-cloud-mesh.org/ns | ocm | + +=== Request Example with declared namespaces + +[source,xml] +---- + + + +---- + +=== Supported WebDAV Properties + +| Property | Desription | Example | +| ----------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| `` | The latest modification time. | `Fri, 30 Dec 2022 14:22:43 GMT` | +| `` | The file's etag. | `"c3a1ee4a0c28edc15b9635c3bf798013"` | +| `` | The mime type of the file. | `image/jpeg` | +| `` | Specifies the nature of the resource. | `` for a folder | +| `` | The size if it is a file in bytes. | `5` bytes | +| `` | Describes the active locks on a resource. | Detailed Example in [Locking]() | +| `` | The globally unique ID of the resource. | `storage-1$27475553-7fb7-4689-b4cf-bbb635daff79!27475553-7fb7-4689-b4cf-bbb635daff79` | +| `` | The globally unique ID of the resource. | `storage-1$27475553-7fb7-4689-b4cf-bbb635daff79!27475553-7fb7-4689-b4cf-bbb635daff79` | +| `` | Direct URL to download a file from. | Not implemented. | +| `` | Determines the actions a user can take on the resource. | The value is a string containing letters that clients can use to determine available actions. | +| | | `S`: Shared | +| | | `M`: Mounted | +| | | `D`: Deletable | +| | | `NV`: Updateable, Renameable, Moveable | +| | | `W`: Updateable (file) | +| | | `CK`: Creatable (folders only) | +| | | `Z`: Deniable | +| | | `P`: Trashbin Purgable | +| | | `X`: Securely Viewable | +| | | In the early stages this was indeed a list of permissions. Over time, more flags were added and the term permissions no longer really fits well. | +| `` | List of user specified tags. | `test` | +| ` ` | The favorite state. | `0` for not favourited, `1` for favourited | +| `` | The user id of the owner of a resource. Project spaces have no owner. | `einstein` | +| `` | The display name of the owner of a resource. Project spaces have no owner. | `Albert Einstein` | +| `` | List of share types. | `0` = User Share | +| | | `1` = Group Share | +| | | `2` = Public Link | +| `` | | ``
`SHA1:1c68ea370b40c06fcaf7f26c8b1dba9d9caf5dea MD5:2205e48de5f93c784733ffcca841d2b5 ADLER32:058801ab`
`
` | +| | | Due to a bug in the very early development of ownCloud, this value is not an array, but a string separated by whitespaces. | +| `` | Similar to `getcontentlength` but it also works for folders. | `10` bytes | +| `` | The ID of the share if the resource is part of such. | `storage-1$27475553-7fb7-4689-b4cf-bbb635daff79!27475553-7fb7-4689-b4cf-bbb635daff79` | +| `` | The root path of the shared resource if the resource is part of such. | `/shared-folder` | +| `` | The ID of the shared resource if the resource is part of such. | `storage-1$27475553-7fb7-4689-b4cf-bbb635daff79!27475553-7fb7-4689-b4cf-bbb635daff79` | +| `` | The type of the resource if it's a public link. | `folder` | +| `` | The share permissions of the resource if it's a public link. | `1` | +| `` | The expiration date of the public link. | `Tue, 14 May 2024 12:44:29 GMT` | +| `` | The date the public link was created. | `Tue, 14 May 2024 12:44:29 GMT` | +| `` | The username of the user who created the public link. | `admin` | +| `` | The original name of the resource before it was deleted. | `some-file.txt` | +| `` | The original location of the resource before it was deleted. | `some-file.txt` | +| `` | The date the resource was deleted. | `Tue, 14 May 2024 12:44:29 GMT` | +| `` | Audio meta data if the resource contains such. | `MetallicaMetallicaEnter Sandman` | +| `` | Location meta data if the resource contains such. | `51.504106-0.074575` | + +=== Request Headers + +A client executing a `PROPFIND` request MUST submit a Depth Header value. In practice, support for infinite-depth requests MAY be disabled, due to the performance and security concerns associated with this behavior. Servers SHOULD treat a +request without a Depth header as if a `Depth: infinity` header was included. Infinite depth requests are disabled by default in ocis. + +| Name | Value | +|-------------------------------------------|---------------------------------------------------------------------------------------| +| Depth | `0` = Only return the desired resource. | +| | `1` = Return the desired resource and all resources one level below in the hierarchy. | +| | `infinity` = Return all resources below the root. | + +// SHORTCODE: {{< hint type=caution title="Use the Depth header with caution" >}} +_Depth: infinity_ + +Using the `Depth: infinity` header value can cause heavy load on the server, depending on the size of the file tree. + +The request can run into a timeout and the server performance could be affected for other users. + +// SHORTCODE: {{< /hint >}} + +== Create a Directory + +Clients create directories (WebDAV collections) by executing a `MKCOL` request at the location specified by the request url. + +// SHORTCODE: {{< tabs "create-folder" >}} +// SHORTCODE: {{< tab "Curl" >}} +[source,shell] +---- +curl -L -X MKCOL 'https://localhost:9200/dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/NewFolder/' \ +-H 'Authorization: Basic YWRtaW46YWRtaW4=' +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "HTTP" >}} +[source,shell] +---- +MKCOL /dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/NewFolder/ HTTP/1.1 +Host: localhost:9200 +Authorization: Basic YWRtaW46YWRtaW4= +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< /tabs >}} +=== Response + +// SHORTCODE: {{< tabs "response create folder" >}} +// SHORTCODE: {{< tab "201 - Created" >}} +This indicates that the Resource has been created successfully. + +==== Body + +The response has no body. +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "403 - Forbidden" >}} + +==== Body + +[source,xml] +---- + + + Sabre\DAV\Exception\Forbidden + + +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "405 - Method not allowed" >}} + +==== Body + +[source,xml] +---- + + + Sabre\DAV\Exception\MethodNotAllowed + The resource you tried to create already exists + +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< /tabs >}} + +== Upload File + +To upload files to the remote server, clients can use the `PUT` method to create or fully replace the content of the remote file. + +=== Request Headers + +| Name | Usage | +|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `X-OC-Mtime` | Send the last modified
time of the file to the server in unixtime format. The server applies this mtime to the resource rather than the actual time. | +| `OC-Checksum` | Provide the checksum of the
file content to the server.
This is used to prevent corrupted data transfers. | +| `If-Match` | The If-Match request-header field is used with a method to make it
conditional. A client that has one or more entities previously
obtained from the resource can verify that one of those entities is
current by including a list of their associated entity tags in the
If-Match header field. | + +// SHORTCODE: {{< tabs "upload-file" >}} +// SHORTCODE: {{< tab "Curl" >}} +[source,shell] +---- +curl -L -X PUT 'https://localhost:9200/dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/test.txt' \ +-H 'X-OC-Mtime: 1692369418' \ +-H 'OC-Checksum: SHA1:40bd001563085fc35165329ea1ff5c5ecbdbbeef' \ +-H 'If-Match: "4436aef907f41f1ac7dfd1ac3d0d455f"' \ +-H 'Content-Type: text/plain' \ +-H 'Authorization: Basic YWRtaW46YWRtaW4=' \ +-d '123' +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "HTTP" >}} +[source,shell] +---- +PUT /dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/test.txt HTTP/1.1 +Host: localhost:9200 +X-OC-Mtime: 1692369418 +OC-Checksum: SHA1:40bd001563085fc35165329ea1ff5c5ecbdbbeef +If-Match: "4436aef907f41f1ac7dfd1ac3d0d455f" +Content-Type: text/plain +Authorization: Basic YWRtaW46YWRtaW4= +Content-Length: 3 + +123 +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< /tabs >}} + +=== Response + +// SHORTCODE: {{< tabs "response upload file" >}} +// SHORTCODE: {{< tab "201 - Created" >}} +This indicates that the Resource has been created successfully. + +==== Body + +The response has no body. + +==== Headers + +[source,yaml] +---- +Oc-Etag: "4436aef907f41f1ac7dfd1ac3d0d455f" +Oc-Fileid: storage-users-1$some-admin-user-id-0000-000000000000!07452b22-0ba9-4539-96e1-3511aff7fd2f +Last-Modified: Fri, 18 Aug 2023 14:36:58 +0000 +X-Oc-Mtime: accepted +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "204 - No Content" >}} +This indicates that the Resource has been updated successfully. + +==== Body + +The response has no body. + +==== Headers + +[source,yaml] +---- +Oc-Etag: "4436aef907f41f1ac7dfd1ac3d0d455f" +Oc-Fileid: storage-users-1$some-admin-user-id-0000-000000000000!07452b22-0ba9-4539-96e1-3511aff7fd2f +Last-Modified: Fri, 18 Aug 2023 14:36:58 +0000 +X-Oc-Mtime: accepted +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "400 - Bad Request" >}} +This indicates that the checksum, which was sent by the client, does not match the computed one after all bytes have been received by the server. + +==== Body + +[source,xml] +---- + + + Sabre\DAV\Exception\BadRequest + The computed checksum does not match the one received from the client. + +---- +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "403 - Forbidden" >}} + +The user cannot create files in that remote location. +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "404 - Not Found" >}} + +The remote target space cannot be found. +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< tab "409 - Conflict" >}} + +This error can occur when the request cannot be executed due to a missing precondition. One example is a PUT into a non-existing remote folder. It can also happen when the client sends the wrong etag in the `If-Match` header. +// SHORTCODE: {{< /tab >}} +// SHORTCODE: {{< /tabs >}} diff --git a/modules/developer/pages/apis/index.adoc b/modules/developer/pages/apis/index.adoc new file mode 100644 index 00000000..7b8410e2 --- /dev/null +++ b/modules/developer/pages/apis/index.adoc @@ -0,0 +1,47 @@ += APIs +:toc: right +:toclevels: 3 + + +Infinite Scale provides a large set of different _application programming interfaces (APIs)_. Infinite Scale is built by microservices. That means many calls to "functions" in the code are remote calls. + +Basically we have two different API "universes": link:http[HTTP] and link:grpc_apis[gRPC]. + +// SHORTCODE: {{< columns >}} + +image::ocis/http-logo.png[Image sourced from https://commons.wikimedia.org/ free of licenses,width=70%] +<---> + +image::ocis/grpc-logo.png[Image sourced from https://grpc.io/ under CC 4.0 BY license,width=70%] + + + +For inter-service-communication we are using mostly gRPC calls because it has some advantages. In the future, clients may decide to use gRPC directly to make use of these advantages. + +image::ocis/ocis-apis.drawio.svg[] + +== link:http[HTTP] + +HTTP APIs are mostly used for client <-> server communication. Modern applications are embracing a link:https://en.wikipedia.org/wiki/Representational_state_transfer[RESTful] software architecture style. REST APIs are using the HTTP protocol to transfer data between clients and servers. All our clients talk to the Server using HTTP APIs. This has legacy reasons and is well-supported across many platforms and technologies. Infinite Scale uses an link:../services/proxy[HTTP API gateway] to route client requests to the correct service. + +=== OpenAPI + +It is best practise to define APIs and their behavior by a spec. We are using the OpenAPI standard for all new APIs. The link:https://swagger.io/specification/[OpenAPI Specification], previously known as the Swagger Specification, is a specification for a machine-readable interface definition language for describing, producing, consuming and visualizing RESTful web services. Previously part of the Swagger framework, it became a separate project in 2016, overseen by the OpenAPI Initiative, an open-source collaboration project of the Linux Foundation. Swagger and some other tools can generate code, documentation and test cases from interface files. + +=== RFC + +Some APIs have become a de facto standard and are additionally covered by an link:https://en.wikipedia.org/wiki/Request_for_Comments[RFC]. + +== link:grpc_apis[gRPC] + +In gRPC, a client application can directly call methods on a server application on a different machine as if it was a local object. This makes it easier to create distributed applications based on microservices. In gRPC we can define a service and specify the methods that can be called remotely. A gRPC client has a stub that provides the same methods and types as the server. +Infinite Scale uses a link:../services/gateway[gRPC API Gateway] to route the requests to the correct service. + +=== Protobuf + +gRPC APIs are typically defined by link:https://developers.google.com/protocol-buffers/docs/overview[Protocol buffers]. The different client and server stubs are created from ``*.proto`` files by code generation tools. + +== Versioning + +There are different standards for API versioning: Through URL, through request parameter, through custom header and through content negotiation. Ocis uses the versioning by URL concept although this creates a big code footprint. The versioning follows link:https://semver.org[SemVer]. We update the major version number when breaking changes are needed. Clients can decide which major version they use through the request URL. The specific implementation is documented on each API. + diff --git a/modules/developer/pages/architecture/collaborative-storage.adoc b/modules/developer/pages/architecture/collaborative-storage.adoc new file mode 100644 index 00000000..ba347b29 --- /dev/null +++ b/modules/developer/pages/architecture/collaborative-storage.adoc @@ -0,0 +1,105 @@ += Collaborative Storage +:toc: right +:toclevels: 3 + + +One of the envisioned design goals of Infinite Scale is to work with so called _collaborative storage_, which means that the file system it is running on is not exclusive for Infinite Scale, but can be manipulated in parallel through third party tools. Infinite Scale is expected to monitor the changes that happen independently and react in a consistent and user friendly way. + +A real world example of that would be a third party "data producer" that submits data directly into a file system path, not going through Infinite Scale APIs. + +This document outlines a few challenges and design concepts for collaborative storage. It is also the base "checklist" for custom storage provider implementations for certain storages, ie. for Ceph- or IBM Storage Scale which provide features that allow more sophisticated and efficient implementations of this goal. + + += Storage Driver Components + +This discusses a few components and sub functions of the storage driver that have relevance for the collaborative storage. + +== Path Locations + +What is called "the oCIS file system" is defined as the entire filetree underneath a special path in the local POSIX file system, which might either be a real local file system or a mounted net filesystem. It is expected that oCIS is the only consumer of that file tree, except what is expected behaviour with a collaborative file system, that adds and edits files in that tree. + +Underneath the oCIS file system root, there is an collection of different folders containing oCIS specific data. Specific storage driver data is in the directory `storage/users`, organized by spaces. +(TODO: Check again how different storage drivers work together without overwriting data of each other) + +== Spaces + +Infinite Scale provides spaces as an additional organization organizational unit for data. Each space is a separate entity with its own attributes such as access patterns and quota. + +A storage driver has to model the separation of spaces and provide a list of spaces in general and also a list of spaces a user can access. Furthermore, it needs to be able to create different types of spaces (Home- or Project space). + +On POSIX, each space could for example be mapped to it's own directory in a special spaces folder under the oCIS root folder. + +== ID to Path Lookup + +Infinite Scale uses file IDs to efficiently identify files within a file tree. The lookup from a given ID to a path within the oCIS file tree is a very basic function that more or less defines the Infinite Scale performance. The functionality to for example query the file path for a given Inode number (which is the nearest equivalent for the Infinite Scale file ID) can not be done with standard POSIX system calls. + +The interface defining the collaborative storage needs an abstraction for this particular function, returning the file id for a given path, and returning the path for given id. + +== Change Notification + +When a file is changed by a process outside of oCIS, this needs to be monitored by oCIS to quickly maintain internal caches and data structures as required. + +The collaborative storage driver needs a way to achieve that. The easiest way for an POSIX based collaborative storage is inotify, that needs to be set up recursively on a file tree to record changes. Additional it is a challenge to destinguish between changes that were done from external activity and the ones that oCIS creates by its own file operations. + +For GPFS, there is a subsystem called delivering that: + +https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=reference-clustered-watch-folder] + +== ETag Propagation + +ownCloud requires that changes which happen "down" in a tree, can be detected in the root element of the tree. That happens through the change of the ETag metadata of each file and/or directory. An ETag is a random, text based tag, that only has one requirement: It has to change its content if a resource further down in the file tree has changed either its content or its metadata. (See link:https://github.com/owncloud/ocis/issues/3782[this issue] for further discussion about the ETag/CTag). + +POSIX file systems do not maintain a change flag like the ETag by default. The file time stamps (atime, ctime, mtime) in general are not fine granular enough (only seconds for some file systems) and depend on the server time, which renders them useless in a distributed environment. + +Infinite Scale needs to implement ETag propagation "up". For the collaborative storage, that needs to be combined with the change notification described above. + +Certain file systems implement this functionality either independently from Infinite Scale (EOS) or at least support proper change notifications (Ceph, GPFS?). + +== Metadata Management + +Metadata are data "snippets" that are as tightly attached to files as ever possible. In best case, a rename of a file silently keeps the metadata as well. In POSIX, this can be achieved by extended file attributes with certain limitations. + +== Quota + +Each space has it's own quota, thus a storage driver implementation needs to consider that. + +For GPFS for example, there is support for quota handling in the file system. + +https://www.ibm.com/docs/en/gpfs/4.1.0.4?topic=interfaces-gpfs-quotactl-subroutine + +Other systems store quota data in the metadata storage and implement propagation of used quota similar to the ETag propagation. + +== User Management + +With user management it is meant how to handle the users and groups within oCIS and how that reflects to the file system where data is stored. + +=== Exclusive Environment + +In exclusive environments (aka. decomposedFS) all files of oCIS (ie. the entire oCIS filetree) belongs to a system user with the name `ocis` typically. + +=== Collaborative Storage + +For collaborative storages, the approach described above does not longer work because users are supposed to be able to manipulate data in "their" file tree parts, and that is identified by ACLs and the owner of the files. + +That requires a few prerequisites that have to be fulfilled: + +1. oCIS as one "client" changing data and the system that allows to access the file tree directly have to use the same user provider, to ensure that each user that is available on a shell is also available in oCIS. That ensures that changes are authenticated through system ACLs and users. LDAP based authentication on the system via PAM and the same LDAP as source for the oCIS IDP should be a sufficient setup. +2. oCIS must be able to write as a "different" user than the ocis system user. That means that we somehow have to impersonate file changing ooperations and run these as the user that is authenticated in oCIS. + +Example: There is a user ben. It has to have an entry in the LDAP that is used by IDP which oCIS is running "behind". With that, ben is able to authenticate through the IDP and work in the oCIS web app. The oCIS linux process will do writes and other changes impersonated as user ben. + +For the access of data on the commandline, the logins to the linux system must be authenticated against the same LDAP - so that ben can authenticate on a terminal using username and password. With that, the user can interactively change data that belongs to user ben (simplified said). + +To give permissions to groups, the linux group management must work accordingly. The same is true for file permissions. + +== Trashbin + +When a user deletes a file in oCIS it is moved to a so called trashbin that allows to restore the file if the deletion was accidentailly. + +== Versions + +When an existing file is changed, the former file state is to be preserved with data and metadata by oCIS. Some file system types provide this functionality via snapshots on partition or even file level. Other do not and have to implement that via a hidden directory keeping old file versions. + + + + diff --git a/modules/developer/pages/architecture/efficient-stat-polling.adoc b/modules/developer/pages/architecture/efficient-stat-polling.adoc new file mode 100644 index 00000000..93a019bc --- /dev/null +++ b/modules/developer/pages/architecture/efficient-stat-polling.adoc @@ -0,0 +1,197 @@ += Efficient Stat Polling +:toc: right +:toclevels: 3 + +The fallback sync mechanism uses the ETag to determine which part of a sync tree needs to be checked by recursively descending into folders whose ETag has changed. The ETag can be calculated using a `stat()` call in the filesystem and we are going to explore how many `stat()` calls are necessary and how the number might be reduced. + +== ETag propagation + +What does ETag propagation mean? Whenever a file changes its content or metadata the ETag or "entity tag" changes. In the early days of ownCloud it was decided to extend this behavior to folders as well, which is outside of any WebDAV RFC specification. Nevertheless, here we are, using the ETag to reflect changes, not only on WebDAV resources but also WebDAV collections. The server will propagate the ETag change up to the root of the tree. + +// SHORTCODE: {{}} +graph TD + linkStyle default interpolate basis + + subgraph final ETag propagation + ert3(( etag:N )) --- el3(( etag:O )) & er3(( etag:N )) + er3 --- erl3(( etag:O )) & err3(( etag:N )) + end + + subgraph first ETag propagation + ert2(( etag:O )) --- el2(( etag:O )) & er2(( etag:N )) + er2 --- erl2(( etag:O )) & err2(( etag:N )) + end + + subgraph initial file change + ert(( etag:O )) --- el(( etag:O )) & er(( etag:O )) + er --- erl(( etag:O )) & err(( etag:N )) + end +// SHORTCODE: {{}} + +The old `etag:O` is replaced by propagating the new `etag:N` up to the root, where the client will pick it up and explore the tree by comparing the old ETags known to him with the state of the current ETags on the server. This form of sync is called _state based sync_. + +== Single user sync +To let the client detect changes in the drive (a tree of files and folders) of a user, we rely on the ETag of every node in the tree. The discovery phase starts at the root of the tree and checks if the ETag has changed since the last discovery: +- if it is still the same nothing has changed inside the tree +- if it changed the client will compare the ETag of all immediate children and recursively descend into every node that changed + +This works, because the server side will propagate ETag changes in the tree up to the root. + +// SHORTCODE: {{}} +graph TD + linkStyle default interpolate basis + + ec( client ) -->|"stat()"|ert + + subgraph + ert(( )) --- el(( )) & er(( )) + er --- erl(( )) & err(( )) + end +// SHORTCODE: {{}} + +== Multiple users +On an ocis server there is not one user but many. Each of them may have one or more clients running. In the worst case all of them polling the ETag of his home root node every 30 seconds. + +Keep in mind that etags are only propagated inside each distinct tree. No sharing is considered yet. + +// SHORTCODE: {{}} +graph TD + linkStyle default interpolate basis + + ec( client ) -->|"stat()"|ert + + subgraph + ert(( )) --- el(( )) & er(( )) + er --- erl(( )) & err(( )) + end + + mc( client ) -->|"stat()"|mrt + + subgraph + mrt(( )) --- ml(( )) & mr(( )) + mr --- mrl(( )) & mrr(( )) + end + + fc( client ) -->|"stat()"|frt + + subgraph + frt(( )) --- fl(( )) & fr(( )) + fr --- frl(( )) & frr(( )) + end +// SHORTCODE: {{}} + +== Sharing +_Storage providers_ are responsible for persisting shares as close to the storage as possible. + +One implementation may persist shares using ACLs, another might use custom extended attributes. The chosen implementation is storage specific and always a tradeoff between various requirements. Yet, the goal is to treat the storage provider as the single source of truth for all metadata. + +If users can bypass the storage provider using e.g. `ssh` additional mechanisms needs to make sure no inconsistencies arise: +- the ETag must still be propagated in a tree, eg using inotify, a policy engine or workflows triggered by other means +- deleted files should land in the trash (e.g. `rm` could be wrapped to move files to trash) +- overwriting files should create a new version ... other than a fuse fs I see no way of providing this for normal posix filesystems. Other storage backends that use the s3 protocol might provide versions natively. + +The storage provider is also responsible for keeps track of references e.g. using a shadow tree that users normally cannot see or representing them as symbolic links in the filesystem (Beware of symbolic link cycles. The clients are currently unaware of them and would flood the filesystem). + +To prevent write amplification ETags must not propagate across references. When a file that was shared by einstein changes the ETag must not be propagated into any share recipients tree. + +// SHORTCODE: {{}} +graph TD + linkStyle default interpolate basis + + + ec( einsteins client ) -->|"stat()"|ert + + subgraph + ml --- mlr(( )) + mrt(( )) --- ml(( )) & mr(( )) + mr --- mrl(( )) & mrr(( )) + end + + mlr -. reference .-> er + + subgraph + ert(( )) --- el(( )) & er(( )) + er --- erl(( )) & err(( )) + end + + mc( maries client ) -->|"stat()"|mrt + +// SHORTCODE: {{}} + +But how can Marie's client detect the change? + +We are trading writes for reads: the client needs to stat the own tree & all shares or entry points into other storage trees. + +It would require client changes that depend on the server side actually having an endpoint that can efficiently list all entry points into storages a user has access to including their current etag. + +But having to list n storages might become a bottleneck anyway, so we are going to have the gateway calculate a virtual root ETag for all entry points a user has access to and cache that. + +== Server Side Stat Polling +Every client polls the virtual root ETag (every 30 sec). The gateway will cache the virtual root ETag of every storage for 30 sec as well. That way every storage provider is only stated once every 30 sec (can be throttled dynamically to adapt to storage io load). + + +// SHORTCODE: {{}} +graph TD + linkStyle default interpolate basis + + ec( client ) -->|"stat()"|evr + + subgraph gateway caching virtual etags + evr(( )) + mvr(( )) + fvr(( )) + end + + evr --- ert + mvr --- mrt + fvr --- frt + + subgraph + ert(( )) --- el(( )) & er(( )) + er --- erl(( )) & err(( )) + end + + mc( client ) -->|"stat()"|mvr + + subgraph + mrt(( )) --- ml(( )) & mr(( )) + ml --- mlm(( )) + mr --- mrl(( )) & mrr(( )) + end + + mlm -.- er + mvr -.- er + + fc( client ) -->|"stat()"|fvr + + subgraph + frt(( )) --- fl(( )) & fr(( )) + fr --- frl(( )) & frr(( )) + end + +// SHORTCODE: {{}} + +Since the active clients will poll the etag for all active users the gateway will have their ETag cached. This is where sharing comes into play: The gateway also needs to stat the ETag of all other entry points ... or mount points. That may increase the number of stat like requests to storage providers by an order of magnitude. + +=== Ram considerations + +For a single machine using a local posix storage the linux kernel already caches the inodes that contain the metadata that is necessary to calculate the ETag (even extended attributes are supported). With 4k inodes 256 nodes take 1Mb of RAM, 1k inodes take 4Mb and 1M inodes take 4Gb to completely cache the file metadata. For distributed filesystems a dedicated cache might make sense to prevent hammering it with stat like requests to calculate ETags. + +=== Bandwidth considerations + +The bandwidth for a single machine might be another bottleneck. Consider a propfind request with roughly 500 bytes and a response with roughly 800 bytes in size: +- At 100Mbit (~10Mb/s) you can receive 20 000 PROPFIND requests +- At 1000Mbit (~100Mb/s) you can receive 200 000 PROPFIND requests +- At 10Gbit (~1Gb/s) you can receive 2 000 000 PROPFIND requests + +This can be scaled by adding more gateways and sharding users because these components are stateless. + +== Share mount point polling cache +What can we do to reduce the number of stat calls to storage providers. Well, the gateway queries the share manager for all mounted shares of a user (or all entry points, not only the users own root/home). The share references contain the storage provider that contains the share. If every user has its own storage provider id the gateway could check in its own cache if the storage root etag has changed. It will be up-to-date because another client likely already polled for its etag. +This would reduce the number of necessary stat requests to active storages. + +=== Active share node cache invalidation +We can extend the lifetime of share ETag cache entries and only invalidate them when the root of the storage that contains them changes its ETag. That would reduce the number of stat requests to the number of active users. + +=== Push notifications +We can further enhance this by sending push notifications when the root of a storage changes. Which is becoming increasingly necessary for mobile devices anyway. diff --git a/modules/developer/pages/architecture/index.adoc b/modules/developer/pages/architecture/index.adoc new file mode 100644 index 00000000..981b9e4b --- /dev/null +++ b/modules/developer/pages/architecture/index.adoc @@ -0,0 +1,7 @@ += Architecture +:toc: right +:toclevels: 3 + +In the architecture part of the documentation we collect useful developer documentation on different aspects of the architecture. We are using mermaid.js to collaborate on the necessary diagrams. + +_Pictures tell more than a thousand words._ diff --git a/modules/developer/pages/architecture/posixfs-storage-driver.adoc b/modules/developer/pages/architecture/posixfs-storage-driver.adoc new file mode 100644 index 00000000..168c3278 --- /dev/null +++ b/modules/developer/pages/architecture/posixfs-storage-driver.adoc @@ -0,0 +1,201 @@ += PosixFS Storage Driver +:toc: right +:toclevels: 3 + + +The Posix FS Storage Driver is a new storage driver for Infinite Scale. + +The scope of this document is to give a high level overview to the technical aspects of the Posix FS Storage Driver and guide the setup. + +== Introduction + +The Posix FS Storage Driver is a backend component that manages files on the server utilizing a "real" file tree that represents the data with folders and files in the file system as users are used to it. That is the big difference compared to Decomposed FS which is the default storage driver in Infinite Scale. + +This does not mean that Infinite Scale is trading any of its benefits to this new feature: It still implements simplicity by running without a database, it continues to store metadata in the file system and adds them transparently to caches and search indexes, and it also features the full spaces concept as before, just to name a few examples. + +The architecture of Infinite Scale allows configuring different storage drivers for specific storage types and purposes on a space granularity. The Posix FS Storage Driver is an alternative to the default driver called Decomposed FS. + +However, the clarity of the file structure in the underlying file system is not the only benefit of the Posix FS Storage Driver. This new technology allows users to manipulate the data directly in the file system, and any changes made to files outside of Infinite Scale are monitored and directly reflected in Infinite Scale. For example, a scanner could store its output directly to the Infinite Scale file system, which immediately gets picked up in Infinite Scale. + +For the first time ever with feature rich open source file sync & share systems, users can either choose to work with their data through the clients of the system, its APIs or even directly in the underlying file system on the server. + +That is another powerful vector for integration and enables a new spectrum of use cases across all domains. + +== Technical Aspects + +The Posix FS Storage Driver uses a few features of the underlying file system, which are mandatory and directly contributing to the performance of the system. + +While the simplest form of Posix FS Storage Driver runs with default file systems of every modern Linux system which are directly mounted and thus support inotify, the full power of this unfolds with more capable file systems such as IBM Storage Scale or Ceph. These are recommended as reliable foundations for big installations of Infinite Scale. + +This chapter describes some technical aspects of this storage driver. + +=== Path Locations + +The file tree that is used as storage path for both data and metadata is located under the local path on the machine that is running Infinite Scale. That might either be a real local file system or a mounted net filesystem. It is expected that oCIS is the only consumer of that file tree, except what is expected behaviour with a collaborative file system, that works with files in that tree. + +Underneath the Infinite Scale file system root, there is a collection of different folders containing Infinite Scale specific data storing personal spaces, project spaces and indexes. + +=== Metadata + +Infinite Scale is highly dependent on the efficient usage of meta data which are attached to file resources, but also logical elements such as spaces. + +Metadata is stored in extended attributes (as also supported by decompsed FS) which poses the benefit that metadata is always directly attached to the actual resources. As a result, care has to be taken that extended attributes are considered when working with the file tree however, e.g. when creating or restoring backups. + +Note: The maximum number and size of extended attributes are limited depending on the filesystem and block size. See <> for more details on GPFS file systems. + +All indexing and caching of metadata is implemented in higher system levels than the storage driver, and thus are not different to the components used with other storage drivers like the decomposed FS. + +=== Monitoring + +To get information about changes such as new files added, files edited or removed, Infinite Scale uses a monitoring system to directly watch the file system. This starts with the Linux inotify system and ranges to much more sophisticated services as for example in Spectrum Scale (see <> for more details on GPFS file systems). + +Based on the information transmitted by the watching service, Infinite Scale is able to "register" new or changed files into its own caches and internal management structures. This enables Infinite Scale to deliver resource changes through the "traditional" channels such as APIs and clients. + +Since the most important metadata is the file tree structure itself, it is impossible for the "split brain" situation between data and metadata to cause trouble. + +=== Automatic ETag Propagation + +The ETag of a resource can be understood as a content fingerprint of any file- or folder resource in Infinite Scale. It is mainly used by clients to detect changes of resources. The rule is, that if the content of a file changed the ETag has to change as well, as well as the ETag of all parent folders up to the root of the space. + +Infinite Scale uses a built in mechanism to maintain the ETag for each resource in the file meta data, and also propagates it automatically. + +A sophisticated underlying file system could provide an attribute that fulfills this requirement and changes whenever content or metadata of a resource changes, and - which is most important - also changes the attribute of the parent resource and the parent of the parent etc. + +=== Automatic Tree Size Propagation + +Similar to the ETag propagation described before, Infinite Scale also tracks the accumulated tree size in all nodes of the file tree. A change to any file requires a re-calculation of the size attribute in all parent folders. + +Infinite Scale would benefit from file systems with native tree size propagation. + +=== Quota + +Each space has it's own quota, thus every storage driver implementation needs to consider that. + +For example, IBM Spectrum Scale supports quota handling directly in the file system. + +Other systems store quota data in the metadata storage and implement propagation of used quota similar to the tree size propagation. + +=== File ID Resolution + +Infinite Scale uses an ID based approach to work with resources, rather than a file path based mechanism. The reason for that is, that ID based lookups can be done way more efficiently compared to tree traversals, just to name one reason. + +The most important component of the ID is a unique file ID that identifies the resource within a space. Ideally the Inode of a file could be used here. However, some file systems re-use inodes which must be avoided. Infinite Scale thus does not use the file Inode, but generates a UUID instead. + +ID based lookups utilize an ID cache which needs to be shared between all storageprovider and dataprovider instances. During startup a scan of the whole file tree is performed to detect and cache new entities. + +In the future a powerful underlying file system could support Infinite Scale by providing an API that + +1. Provides the ID for a given file path referenced resource +2. Provides the path for a given ID. + +These two operations are very crucial for the performance of the entire system. + +=== User Management + +With the requirement that data can be manipulated either through the filesystem or the Infinite Scale system, the question under which UID the manipulation happens is important. + +There are a few possible ways for user management: +1. Changes can either be only accepted by the same user that Infinite Scale is running under, for example the user `ocis`. All manipulations in the filesystem have to be done by, and only by this user. +2. Group based: All users who should be able to manipulate files have to be in a unix group. The Infinite Scale user has also to be member of that group. The default umask in the directory used has to allow group writing all over the place. +3. Impersonation: Infinite Scale impersonates the user who owns the folder on the file system to mimic the access as the user. + +All possibilities have pros and cons for operations. + +One for all, it seems reasonable to use LDAP to manage users which is the base for the Infinite Scale IDP as well as the system login system via PAM. + +=== GID Based Space Access + +The Posix FS Storage Driver supports GID based space access to support the problem that project spaces might have to be accessible by multiple users on disk. In order to enable this feature the `ocis` binary needs to have the `setgid` capability and `STORAGE_USERS_POSIX_USE_SPACE_GROUPS` needs to be set to `true`. Inifinite Scale will then use the space GID (the gid of the space root) for all file system access using the `setfsgid` syscall, i.e. all files and directories created by Infinite Scale will belong to the same group as the space root. + +== Advanced Features + +Depending on the capabilities of the underlying file system, the Posix FS Storage Driver can benefit from more advanced functionality described here. + +=== Versioning + +If the underlying file system is able to create versions of single resources (imagine a git based file system) this functionality could directly be used by Infinite Scale. + +In the current state of the Posix FS Storage Driver, versioning is not supported. + +=== Trashbin + +If the underlying file system handles deleted files in a trash bin that allows restoring of previously removed files, this functionality could directly be used by Infinite Scale. + +If not available it will follow the link:https://specifications.freedesktop.org/trash-spec/trashspec-latest.html[the Free Desktop Trash specificaton]. + +== Limitations + +As of Q2/2024 the Posix FS Storage Driver is not officially supported and in technical preview state. + +The tech preview comes with the following limitations: + +1. Only inotify and GPFS file system change notification methods are supported +1. Versioning is not supported yet +1. The space/project folders in the filesystem are named after the UUID, not the real space name +1. No CephFS support yet +1. Postprocessing (ie. anti virus check) does not happen for file actions outside of Infinite Scale + +== Setup + +This describes the steps to use the Posix FS Storage Driver storage driver with Infinite Scale. + +It is possible to use different storage drivers in the same Infinite Scale installation. For example it is possible to set up one space running on Posix FS Storage Driver while others run Decomposed FS. + +=== Prerequisites + +To use the Posix FS Storage Driver, the following prerequisites have to be fulfilled: + +1. There must be storage available to store meta data and blobs, available under a root path. +1. When using inotify, the storage must be local on the same machine. Network mounts do not work with inotify. `inotifywait` needs to be installed. +1. The storage root path must be writeable and executable by the same user Infinite Scale is running under. +1. An appropiate version of Infinite Scale is installed, version number 5.0.5 and later. +1. `nats-js-kv` as cache service + + +=== Setup Configuration + +This is an example configuration with environment variables that configures Infinite Scale to use Posix FS Storage Driver for all spaces it works with, ie. Personal and Project Spaces: + +---- +export STORAGE_USERS_DRIVER="posix" +export STORAGE_USERS_POSIX_ROOT="/home/kf/tmp/posix-storage" +export STORAGE_USERS_POSIX_WATCH_TYPE="inotifywait" +export STORAGE_USERS_ID_CACHE_STORE="nats-js-kv" +export STORAGE_USERS_ID_CACHE_STORE_NODES="localhost:9233" + +# Optionally enable gid based space access +export STORAGE_USERS_POSIX_USE_SPACE_GROUPS="true" +---- + +== GPFS Specifics + +When using GPFS as the underlying filesystem the machine running the according `storage-users` service needs to have the GPFS filesystem mounted locally. The mount path is given to ocis as the `STORAGE_USERS_POSIX_ROOT` path. + +Other than that there a few other points to consider: + +=== Extended Attributes + +As described above metadata is stored as extended attributes of the according entities and thus is suspect to their limitations. In GPFS extended attributes are first stored in the inode itself but can then also use an overflow block which is at least 64KB and up to the metadata block size. Inode and metadata block size should be chosen accordingly. + +=== FS Watcher + +The Posix FS Storage Driver supports two different watchers for detecting changes to the filesystem. The watchfolder watcher is better tested and supported at that point. + +==== GPFS File Audit Logging + +The `gpfsfileauditlogging` watcher tails a GPFS file audit log and parses the JSON events to detect relevant changes. + +---- +export STORAGE_USERS_POSIX_WATCH_TYPE="gpfsfileauditlogging" +export STORAGE_USERS_POSIX_WATCH_PATH="/path/to/current/audit/log" +---- + +==== GPFS Watchfolder + +The `gpfswatchfolder` watcher connects to a kafka cluster which is being filled with filesystem events by the GPFS watchfolder service. + +---- +export STORAGE_USERS_POSIX_WATCH_TYPE="gpfswatchfolder" +export STORAGE_USERS_POSIX_WATCH_PATH="fs1_audit" # the kafka topic to watch +export STORAGE_USERS_POSIX_WATCH_FOLDER_KAFKA_BROKERS="192.168.1.180:29092" +---- diff --git a/modules/developer/pages/architecture/protocol-changes.adoc b/modules/developer/pages/architecture/protocol-changes.adoc new file mode 100644 index 00000000..7667d455 --- /dev/null +++ b/modules/developer/pages/architecture/protocol-changes.adoc @@ -0,0 +1,195 @@ += Protocol changes +:toc: right +:toclevels: 3 + +The spaces concept allows clients to look up the space endpoints a user has access to and then do individual sync discoveries. Technically, we introduce an indirection that allows clients to rely on server provided URLs instead of hardcoded `/webdav` or `/dav/files/{username}` paths, that may change over time. + +== Space discovery + +// SHORTCODE: {{}} +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant Graph + participant SpaceA + participant SpaceB + links Client: {"web": "https://owncloud.dev/clients/web/", "RClone": "https://owncloud.dev/clients/rclone/"} + link Graph: Documentation @ https://owncloud.dev/extensions/graph/ + + Note left of Client: First, a clients looks
up the spaces a user has access to + opt space lookup + Client->>+Graph: GET /me/drives + Graph-->>-Client: 200 OK JSON list of spaces, say A, B and C,
each with a dedicated webDavURL, etag and quota + end + + Note left of Client: Then it can do a parallel
sync discovery on spaces
whose etag changed + par Client to Space A + Client->>+SpaceA: PROPFIND {webDavURL for Space A} + SpaceA-->>-Client: 207 Multistatus PROPFIND response + and Client to Space B + Client->>+SpaceB: PROPFIND {webDavURL for space B} + SpaceB-->>-Client: 207 Multistatus PROPFIND response + end +// SHORTCODE: {{
}} + +=== New /dav/spaces/{spaceid} endpoint with spaceid and a relative path + +The ocDAV service is responsible for translating ownCloud flavoured WebDAV into CS3 API calls. + +_General view_ + +A PROPFIND finds its way to a storage provider like this: + +// SHORTCODE: {{}} +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocDAV + participant StorageProvider + + Note right of Client: {spaceid} identifies the space
{relative/path} is relative to the space root + Client->>+ocDAV: PROPFIND /dav/space/{spaceid}/{relative/path} + Note right of ocDAV: translate ownCloud flavoured webdav
into CS3 API requests + ocDAV->>+StorageProvider: ListContainer({spaceid}, path: {relative/path}) + StorageProvider-->>-ocDAV: []ResourceInfo + ocDAV-->>-Client: 207 Multistatus +// SHORTCODE: {{
}} + +While the above is a simplification to get an understanding of what needs to go where, there are several places where sharding can happen. + +_Proxy can do user based routing_ + +The ocis proxy authenticates requests and can forward requests to different backends, depending on the logged-in user or cookies. For example multiple ocdav services can be configured to shard users based on username or affiliation. + +// SHORTCODE: {{}} +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant proxy + participant ocDAV1 as ocDAV [a-k] + participant ocDAV2 as ocDAV [l-z] + + Note right of Client: {spaceid} identifies the space
{relative/path} is relative to the space root + Client->>+proxy: PROPFIND /dav/space/{spaceid}/{relative/path} + + alt username starting with a-k + proxy->>+ocDAV1: PROPFIND /dav/space/{spaceid}/{relative/path} + Note right of ocDAV1: translate ownCloud flavoured webdav
into CS3 API requests + ocDAV1-->>-Client: 207 Multistatus + else username starting with l-z + proxy->>+ocDAV2: PROPFIND /dav/space/{spaceid}/{relative/path} + ocDAV2-->>-Client: 207 Multistatus + end +// SHORTCODE: {{
}} + +_Gateway can do path or storage provider id based routing_ + +The reva gateway acts as a facade to multiple storage providers that can be configured with the storage registry: + +// SHORTCODE: {{}} +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant ocDAV + participant Gateway + participant StorageRegistry + participant StorageProvider1 as StorageProvider [a-k] + participant StorageProvider2 as StorageProvider [l-z] + + Note right of ocDAV: translate ownCloud flavoured webdav
into CS3 API requests + ocDAV->>+Gateway: ListContainer({spaceid}, path: {relative/path}) + Note right of Gateway: find address of the storage provider
that is responsible for the space + Gateway->>+StorageRegistry: ListStorageProviders({spaceid}) + StorageRegistry-->>-Gateway: []ProviderInfo + Note right of Gateway: forward request to
correct storage provider + alt username starting with a-k + Gateway->>+StorageProvider1: ListContainer({spaceid}, path: {relative/path}) + StorageProvider1-->>-Gateway: []ResourceInfo + else username starting with l-z + Gateway->>+StorageProvider2: ListContainer({spaceid}, path: {relative/path}) + StorageProvider2-->>-Gateway: []ResourceInfo + end + Gateway-->>-ocDAV: []ResourceInfo +// SHORTCODE: {{
}} + + +=== Old /dav/files/{username} endpoint with username and a path relative to the users home + +_PROPFIND request against old webdav endpoints_ + +To route a PROPFIND request against the old webdav endpoints like `/dav/files/username`, ocdav first has to build a CS3 namespace prefix, e.g. `/users/{{.Id.OpaqueId}}` to the users home. + +// SHORTCODE: {{}} +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocDAV + participant Gateway + + opt old /dav/files/{username} endpoint with username and a path relative to the users home + Note right of Client: translate ownCloud flavoured webdav
into CS3 API requests + Client->>+ocDAV: PROPFIND /dav/files/{username}/{relative/path} + Note right of ocDAV: translate ownCloud flavoured webdav
into CS3 API requests + ocDAV->>+Gateway: GetUser({username}) + Gateway-->>-ocDAV: User + Note right of ocDAV: build path prefix to user home + ocDAV->>+ocDAV: {namespace/prefix} = ApplyLayout({path layout}, User), eg. /users/e/einstein + Note right of ocDAV: look up the space responsible for a path + ocDAV->>+Gateway: ListStorageSpaces(path: {namespace/prefix}/{relative/path}) + Gateway-->>-ocDAV: []StorageSpace + Note right of ocDAV: make actual request with space and relative path + ocDAV->>+Gateway: ListContainer({spaceid}, path: {relative/path}) + Gateway-->>-ocDAV: []ResourceInfo + ocDAV-->>-Client: 207 Multistatus + end +// SHORTCODE: {{
}} + +_Handling legacy global namespace webdav endpoints_ + +The reason ocis uses a path based lookup instead of looking up the current users home using the user id and a space type filter is, because there are deployments that use a global namespace at the legacy `/webdav` endpoint. To support these use cases, the gateway allows looking up spaces using their mount path. + +// SHORTCODE: {{}} +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocDAV + participant Gateway + + Note right of Client: translate ownCloud flavoured webdav
into CS3 API requests + alt old /dav/files/{username} endpoint with username and a path relative to the users home + Client->>+ocDAV: PROPFIND /dav/files/{username}/{relative/path} + Note right of ocDAV: look up {username} in URL path + ocDAV->>+Gateway: GetUser({username}) + Gateway-->>-ocDAV: User + Note right of ocDAV:build namespace prefix to user home + ocDAV->>+ocDAV: {namespace/prefix} = ApplyLayout({namespace layout}, User), eg. /users/e/einstein + else legacy /webdav/ endpoint with a path relative to the users home + Client->>+ocDAV: PROPFIND /webdav/{relative/path} + Note right of ocDAV: use currently logged in user + ocDAV->>+ocDAV: ContextGetUser() + Note right of ocDAV: build namespace prefix to user home + ocDAV->>+ocDAV: {namespace/prefix} = ApplyLayout({namespace layout}, User), eg. /users/e/einstein + else legacy /webdav/ endpoint with a path relative to a global namespace + Client->>+ocDAV: PROPFIND /webdav/{relative/path} + Note right of ocDAV: omit namespace prefix by using empty layout template + ocDAV->>+ocDAV: {namespace/prefix} = ApplyLayout("/", u), always returns "/" + end + Note right of ocDAV: look up the space responsible for a path + ocDAV->>+Gateway: ListStorageSpaces(path: {namespace/prefix}/{relative/path}) + Gateway-->>-ocDAV: []StorageSpace + Note right of ocDAV: make actual request with space and relative path + ocDAV->>+Gateway: ListContainer({spaceid}, path: {relative/path}) + Gateway-->>-ocDAV: []ResourceInfo + ocDAV-->>-Client: 207 Multistatus +// SHORTCODE: {{
}} diff --git a/modules/developer/pages/architecture/services-communication.adoc b/modules/developer/pages/architecture/services-communication.adoc new file mode 100644 index 00000000..38f426da --- /dev/null +++ b/modules/developer/pages/architecture/services-communication.adoc @@ -0,0 +1,5 @@ += Services Communication +:toc: right +:toclevels: 3 + +image::ocis/ocis-services-communication.drawio.svg[] diff --git a/modules/developer/pages/architecture/upload-processing.adoc b/modules/developer/pages/architecture/upload-processing.adoc new file mode 100644 index 00000000..3c2962a1 --- /dev/null +++ b/modules/developer/pages/architecture/upload-processing.adoc @@ -0,0 +1,128 @@ += Upload processing +:toc: right +:toclevels: 3 + +Uploads are handled by a dedicated service that uses TUS.io for resumable uploads. When all bytes have been transferred the upload is finalized by making the file available in file listings and for download. + +The finalization may be asynchronous when mandatory workflow steps are involved. + +== Legacy PUT upload + +// SHORTCODE: {{}} + +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocdav + participant storageprovider + participant dataprovider + + Client->>+ocdav: PUT /dav/spaces/{spaceid}/newfile.bin + ocdav->>+storageprovider: InitiateFileUpload + storageprovider-->>-ocdav: OK, Protocol simple, UploadEndpoint: /data, Token: {jwt} + Note right of ocdav: The {jwt} contains the internal actual target, eg.: http://localhost:9158/data/simple/91cc9882-db71-4b37-b694-a522850fcee1 + ocdav->>+dataprovider: PUT /data + Note right of dataprovider: X-Reva-Transfer: {jwt} + dataprovider-->>-ocdav: 201 Created + ocdav-->>-Client: 201 Created + +// SHORTCODE: {{}} + +== TUS upload + +// SHORTCODE: {{}} + +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocdav + participant storageprovider + participant datagateway + participant dataprovider + + Client->>+ocdav: POST /dav/spaces/{spaceid}\nUpload-Metadata: {base64 encoded filename etc}\nTUS-Resumable: 1.0.0 + ocdav->>+storageprovider: InitiateFileUpload + storageprovider-->>-ocdav: OK, Protocol tus, UploadEndpoint: /data, Token: {jwt} + Note right of ocdav: The {jwt} contains the internal actual target, eg.:\nhttp://localhost:9158/data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160 + ocdav-->>-Client: 201 Created\nLocation: /data/{jwt}\nTUS-Resumable: 1.0.0 + + Client->>+datagateway: PATCH /data/{jwt}\nTUS-Resumable: 1.0.0\nUpload-Offset: 0 + Note over datagateway: unwrap the {jwt} target + datagateway->>+dataprovider: PATCH /data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160\nX-Reva-Transfer: {jwt} + Note over dataprovider: storage driver\nhandles request + dataprovider-->>-datagateway: 204 No Content\nTUS-Resumable: 1.0.0\nUpload-Offset: 363976 + datagateway-->>-Client: 204 No Content\nTUS-Resumable: 1.0.0\nUpload-Offset: 363976 + +// SHORTCODE: {{}} + + +== TUS upload with async postprocessing + + + +// SHORTCODE: {{}} + +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocdav + participant storageprovider + participant datagateway + participant dataprovider + participant nats + participant processing + + Client->>+ocdav: POST /dav/spaces/{spaceid} + Note left of Client: Upload-Metadata: {base64 encoded filename etc}\nTUS-Resumable: 1.0.0 + ocdav->>+storageprovider: InitiateFileUpload + storageprovider-->>-ocdav: OK, Protocol tus, UploadEndpoint: /data, Token: {jwt} + Note right of ocdav: The {jwt} contains the internal actual target, eg.: http://localhost:9158/data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160 + ocdav-->>-Client: 201 Created + Note right of Client: Location: /data/{jwt} + Note right of Client: TUS-Resumable: 1.0.0 + + Client->>+datagateway: PATCH /data/{jwt} + Note right of datagateway: TUS-Resumable: 1.0.0\nUpload-Offset: 0 + + Note over datagateway: unwrap the {jwt} target + datagateway->>+dataprovider: PATCH /data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160 + Note over dataprovider: X-Reva-Transfer: {jwt} + Note over dataprovider: storage driver + Note over dataprovider: handles request + dataprovider-)nats: emit all-bytes-received event + nats-)processing: all-bytes-received({uploadid}) event + Note over dataprovider: TODO: A lot of time may pass here, we could use the `Prefer: respond-async` header to return early with a 202 Accepted status and a Location header to a websocket endpoint + alt success + processing-)nats: emit processing-finished({uploadid}) event + nats-)dataprovider: processing-finished({uploadid}) event + dataprovider-->>-datagateway: 204 No Content + Note over datagateway: TUS-Resumable: 1.0.0\nUpload-Offset: 363976 + datagateway-->>-Client: 204 No Content + Note over Client: TUS-Resumable: 1.0.0\nUpload-Offset: 363976 + else failure + activate dataprovider + activate datagateway + processing-)nats: emit processing-aborted({uploadid}) event + nats-)dataprovider: processing-aborted({uploadid}) event + Note over dataprovider: FIXME: What HTTP status code should we report? + Note over dataprovider: 422 Unprocessable Content is just a proposal + Note over dataprovider: see https://httpwg.org/specs/rfc9110.html#status.422 + dataprovider-->>-datagateway: 422 Unprocessable Content + Note over datagateway: TUS-Resumable: 1.0.0\nUpload-Offset: 363976 + datagateway-->>-Client: 422 Unprocessable Content + Note over Client: TUS-Resumable: 1.0.0\nUpload-Offset: 363976 + end + +// SHORTCODE: {{}} + + +== Async TUS upload with postprocessing +This might be a TUS extension or a misunderstanding on our side of what tus can do for us. Clients should send a `Prefer: respond-async` header to allow the server to return early when postprocessing might take longer. The PATCH requests can then return status `202 Accepted` and a `Location` header to a websocket that clients can use to track the processing / upload progress. + +TODO there is a conflict with the TUS.io POST request with the creation extension, as that also returns a `Location` header which carries the upload URL. We would need another header to transport the websocket location. Maybe `Websocket-Location` or `Progress-Location`? diff --git a/modules/developer/pages/cli-commands/create-new-cli-command.adoc b/modules/developer/pages/cli-commands/create-new-cli-command.adoc new file mode 100644 index 00000000..228f8f0d --- /dev/null +++ b/modules/developer/pages/cli-commands/create-new-cli-command.adoc @@ -0,0 +1,18 @@ += Create a New CLI Command +:toc: right +:toclevels: 3 + + +[NOTE] +==== +Existing commands should be checked for commonly used options whenever a new CLI command is created, regardless of whether it is embedded in a service or not. For an example you can see the xref:./service_independent_cli.adoc[Service Independent CLI] documentation. +==== + + +== CLI Embedded in a Service + +These commands are usually located in the `/pkg/command` subfolder. + +== CLI Independent of a Service + +These commands are located in the `ocis/pkg/command` subfolder. diff --git a/modules/developer/pages/cli-commands/document-cli-commands.adoc b/modules/developer/pages/cli-commands/document-cli-commands.adoc new file mode 100644 index 00000000..cc0a5c98 --- /dev/null +++ b/modules/developer/pages/cli-commands/document-cli-commands.adoc @@ -0,0 +1,29 @@ += Document CLI Commands +:toc: right +:toclevels: 3 + + +Any CLI command that is added to Infinite Scale must be documented here in the dev docs and the link:https://doc.owncloud.com/ocis/latest/maintenance/commands/commands.html[admin docs]. Note that the admin docs primarily distinguish between online and offline commands because the structure of the documentation is different. Typically, any command documented in the developer documentation is integrated into the admin documentation and adapted according to the target audience. The description here is for developers; the admin docs are derived from it. + +[NOTE] +==== +Note that any CLI command requires documentation. However, it may be decided that a CLI command will not be included in the admin documentation. In such a case, the reasons should be valid. +==== + + +== Type of CLI Commands + +There are three types of CLI commands that require different documentation locations: + +1. Commands that are embedded in a service such as:\ +`ocis storage-users uploads` +2. Commands that are service independent such as:\ +`ocis trash purge-empty-dirs` or `ocis revisions purge` +3. `curl` commands that can be one of the above. + +== Rules + +* _Service dependent_ CLI commands:\ + Add any CLI command into the repsective `README.md` _of the service_. Use as "template" an existing one such as in `services/storage-users/README.md` or `services/auth-app/README.md`. The content created will be transferred automatically to the service in the xref:../services/.adoc[Services] section. +* _Service independent_ CLI commands:\ + Add any CLI command into the xref:./service_independent_cli.adoc[Service Independent CLI] documentation. See the link for an example how to do so. diff --git a/modules/developer/pages/cli-commands/index.adoc b/modules/developer/pages/cli-commands/index.adoc new file mode 100644 index 00000000..4837a222 --- /dev/null +++ b/modules/developer/pages/cli-commands/index.adoc @@ -0,0 +1,7 @@ += oCIS CLI Commands +:toc: right +:toclevels: 3 + + +CLI commands can either be embedded in a service or operate independently. + diff --git a/modules/developer/pages/cli-commands/service_dependent_cli.adoc b/modules/developer/pages/cli-commands/service_dependent_cli.adoc new file mode 100644 index 00000000..cbb081c8 --- /dev/null +++ b/modules/developer/pages/cli-commands/service_dependent_cli.adoc @@ -0,0 +1,36 @@ += Service Dependent CLI +:toc: right +:toclevels: 3 + +This document describes ocis CLI commands that are _embedded in a service_. + + +== Common Parameters + +The ocis package offers a variety of CLI commands for monitoring or repairing ocis installations. Most of these commands have common parameters such as: + +* `--help` (or `-h`)\ + Use to print all available options. + +* `--basePath` (or `-p`)\ + Needs to point to a storage provider, paths can vary depending on your ocis installation. Example paths are: + ```bash + .ocis/storage/users # bare metal installation + /var/tmp/ocis/storage/users # docker installation + ... + ``` + +* `--dry-run`\ + This parameter, when available, defaults to `true` and must explicitly set to `false`. + +* `--verbose` (or `-v`)\ + Get a more verbose output. + +== List of CLI Commands + +For CLI commands that are _embedded in a service_, see the following services: + +* xref:../services/auth-app/.adoc[Auth-App] +* xref:../services/graph/.adoc[Graph] +* xref:../services/postprocessing/.adoc[Postprocessing] +* xref:../services/storage-users/.adoc[Storage-Users] diff --git a/modules/developer/pages/cli-commands/service_independent_cli.adoc b/modules/developer/pages/cli-commands/service_independent_cli.adoc new file mode 100644 index 00000000..78836801 --- /dev/null +++ b/modules/developer/pages/cli-commands/service_independent_cli.adoc @@ -0,0 +1,229 @@ += Service Independent CLI +:toc: right +:toclevels: 3 + +This document describes ocis CLI commands that are _service independent_. + + +For _service dependent_ CLI commands, see the following services: + +* xref:../services/auth-app/.adoc[Auth-App] +* xref:../services/graph/.adoc[Graph] +* xref:../services/postprocessing/.adoc[Postprocessing] +* xref:../services/storage-users/.adoc[Storage-Users] + +== Common Parameters + +The ocis package offers a variety of CLI commands for monitoring or repairing ocis installations. Most of these commands have common parameters such as: + +* `--help` (or `-h`)\ + Use to print all available options. + +* `--basePath` (or `-p`)\ + Needs to point to a storage provider, paths can vary depending on your ocis installation. Example paths are: + ```bash + .ocis/storage/users # bare metal installation + /var/tmp/ocis/storage/users # docker installation + ... + ``` + +* `--dry-run`\ + This parameter, when available, defaults to `true` and must explicitly set to `false`. + +* `--verbose` (or `-v`)\ + Get a more verbose output. + +== List of CLI Commands + +=== Backup CLI + +The backup command allows inspecting the consistency of an ocis storage: + +[source,bash] +---- +ocis backup consistency -p /base/path/storage/users +---- + +This will check the consistency of the storage and output a list of inconsistencies. Inconsistencies can be: + +* _Orphaned Blobs_\ +A blob in the blobstore that is not referenced by any file metadata. +* _Missing Blobs_\ +A blob referenced by file metadata that is not present in the blobstore. +* _Missing Nodes_\ +A node that is referenced by a symlink but doesn't exist. +* _Missing Link_\ +A node that is not referenced by any symlink but should be. +* _Missing Files_\ +A node that is missing essential files (such as the `.mpk` metadata file). +* _Missing/Malformed Metadata_\ +A node that doesn't have any (or malformed) metadata. + +This command provides additional options: + +* `-b` / `--blobstore`\ +Allows specifying the blobstore to use. Defaults to `ocis`. Empty blobs will not be checked. Can also be switched to `s3ng`, but needs addtional envvar configuration (see the `storage-users` service for more details). +* `--fail`\ +Exits with non-zero exit code if inconsistencies are found. Useful for automation. + +=== Cleanup Orphaned Grants + +Detect and optionally delete storage grants that have no corresponding share-manager entry. + +Sharing in ocis relies on two truths. The share manager and the grants. When a share is created, ocis will + +1. Create a grant for the specific file or folder.\ +This grant is _checked when access to the file is requested_. + +2. Create an entry in the `created.json`/`received.json` files of the specific user.\ +These files are _checked whenever shares are listed_. + +The process for creating a share is as follows: first, ocis creates the grant, and then adds the share entry. The reverse order is followed when deleting a share. This means that if the second step fails, the grant will still be present. This can be visually confirmed in the webUI. The webUI details of the "share" section will show an error fetching information for orphan grants. + +The following command fixes the problem of orhaned grants. + +Usage: +[source,bash] +---- +ocis shares clean-orphaned-grants \ + --service-account-id "" \ + --service-account-secret "" \ + [--force] \ + [--space-id ""] \ + [--dry-run=false] +---- + +Notes: +- `--dry-run`\ +Defaults to `true` (no deletions). Set to `false` to remove orphaned grants. +- `--space-id`\ +Limit the scan to a specific storage space (opaque ID). +- `--force`\ +Force removal of suspected orphans even when listing shares fails. +- Public links are not touched. + +=== Cleanup Orphaned Shares + +When a shared space or directory got deleted, use the `shares cleanup` command to remove those share orphans. This can't be done automatically at the moment. + +[source,bash] +---- +ocis shares cleanup +---- + +=== List Unified Roles + +This command simplifies the process of finding out which UID belongs to which role. The command using markdown as output format is: + +[source,bash] +---- +ocis graph list-unified-roles --output-format md +---- + +The output of this command includes the following information for each role: + +* `Name`\ + The human readable name of the role. +* `UID`\ + The unique identifier of the role. +* `Enabled`\ + Whether the role is enabled or not. +* `Description`\ + A short description of the role. +* `Condition` +* `Allowed Resource Actions` + +_Example output (shortned)_ + +| # | LABEL | UID | ENABLED | DESCRIPTION | CONDITION | ALLOWED RESOURCE ACTIONS | +|:--:|:--------------------------------:|:------------------------------------:|:--------:|:------------------------------------------------------------------------------------:|:---------------------------------------------------------:|:----------------------------------------:| +| 1 | Viewer | b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5 | enabled | View and download. | exists @Resource.File | libre.graph/driveItem/path/read | +| | | | | | exists @Resource.Folder | libre.graph/driveItem/quota/read | +| | | | | | exists @Resource.File && @Subject.UserType=="Federated" | libre.graph/driveItem/content/read | +| | | | | | exists @Resource.Folder && @Subject.UserType=="Federated" | libre.graph/driveItem/children/read | +| | | | | | | libre.graph/driveItem/deleted/read | +| | | | | | | libre.graph/driveItem/basic/read | +| 2 | ViewerListGrants | d5041006-ebb3-4b4a-b6a4-7c180ecfb17d | disabled | View, download and show all invite + +=== Move Stuck Uploads + +In some cases of saturated disk usage, Infinite Scale metadata may become stuck. This can occur when file metadata is being moved to its final destination after file operations. This issue was primarily seen with shares, where uploaded files could not be accessed. The required filename parameter aligns with Infinite Scale's internal processes and is used to complete the formerly stuck move action. + +[source,bash] +---- +ocis shares move-stuck-upload-blobs [--dry-run=false] -p /base/path/storage/users +---- + +This command provides additional options: + +* `--dry-run` (default: `true`)\ +Only print found files stuck in transition.\ +Note: This is a safety measure. You must specify `--dry-run=false` for the command to be effective. + +* `--filename` value (default: "received.json")\ +File to move from `uploads/` to share manager metadata `blobs/` + +=== Revisions CLI + +The revisions command allows removing the revisions of files in the storage. + +[source,bash] +---- +ocis revisions purge -p /base/path/storage/users +---- + +It takes the `--resource-id` (or `--r`) parameter which specify the scope of the command: + +* An empty string (default) removes all revisions from all spaces. +* A spaceID (like `d419032c-65b9-4f4e-b1e4-0c69a946181d\$44b5a63b-540c-4002-a674-0e9c833bbe49`) removes all revisions in that space. +* A resourceID (e.g. `d419032c-65b9-4f4e-b1e4-0c69a946181d\$44b5a63b-540c-4002-a674-0e9c833bbe49\!e8a73d49-2e00-4322-9f34-9d7f178577b2`) removes all revisions from that specific file. + +This command provides additional options: + +* `--dry-run` (default: `true`)\ +Do not remove any revisions but print the revisions that would be removed. +* `-b` / `--blobstore`\ +Allows specifying the blobstore to use. Defaults to `ocis`. Can be switched to `s3ng` but needs addtional envvar configuration (see the `storage-users` service for more details). +* `-v` / `--verbose`\ +Prints additional information about the revisions that are removed. +* `--glob-mechanism` (default: `glob`\ +(advanced) Allows specifying the mechanism to use for globbing. Can be `glob`, `list` or `workers`. In most cases the default `glob` does not need to be changed. If large spaces need to be purged, `list` or `workers` can be used to improve performance at the cost of higher cpu and ram usage. `list` will spawn 10 threads that list folder contents in parallel. `workers` will use a special globbing mechanism and multiple threads to achieve the best performance for the highest cost. + +=== Service Health + +The service health CLI command allows checking the health status of a service. If there are no issues found, nothing health related will get printed. + +[source,bash] +---- +ocis health +---- + +_Examples_ + +* The `collaboration` service has been started but not configured and is therefore not in a healthy state: + ```bash + ocis collaboration health + + The WOPI secret has not been set properly in your config for collaboration. Make sure your /root/.ocis/config config contains the proper values (e.g. by using 'ocis init --diff' and applying the patch or setting a value manually in the config/corresponding environment variable). + ``` + +* The `antivirus` service has not been started, the health check responds accordingly: + ```bash + ocis antivirus health + + {"level":"fatal","service":"antivirus","error":"Get \"http://127.0.0.1:9277/healthz\": dial tcp 127.0.0.1:9277: connect: connection refused","time":"2024-10-28T17:47:54+01:00","message":"Failed to request health check"} + ``` + +=== Trash CLI + +The trash cli allows removing empty folders from the trashbin. This should be used to speed up trash bin operations. + +[source,bash] +---- +ocis trash purge-empty-dirs -p /base/path/storage/users +---- + +This command provides additional options: + +* `--dry-run` (default: `true`)\ +Do not remove any empty folders but print the empty folders that would be removed. diff --git a/modules/developer/pages/clients/index.adoc b/modules/developer/pages/clients/index.adoc new file mode 100644 index 00000000..9e11f8b5 --- /dev/null +++ b/modules/developer/pages/clients/index.adoc @@ -0,0 +1,4 @@ += Clients +:toc: right +:toclevels: 3 + diff --git a/modules/developer/pages/clients/rclone/index.adoc b/modules/developer/pages/clients/rclone/index.adoc new file mode 100644 index 00000000..999cc27e --- /dev/null +++ b/modules/developer/pages/clients/rclone/index.adoc @@ -0,0 +1,19 @@ += Rclone +:toc: right +:toclevels: 3 + +== About Rclone + +[TIP] +==== +Rclone is a command line program to manage files on cloud storage. It is a feature rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols. + +Rclone has powerful cloud equivalents to the unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm, and cat. Rclone's familiar syntax includes shell pipeline support, and --dry-run protection. It is used at the command line, in scripts or via its API. + +Users call rclone "The Swiss army knife of cloud storage", and "Technology indistinguishable from magic". +==== + + +Source: link:https://rclone.org/[Rclone project website] + +== Table of Contents diff --git a/modules/developer/pages/clients/rclone/webdav-sync-basic-auth.adoc b/modules/developer/pages/clients/rclone/webdav-sync-basic-auth.adoc new file mode 100644 index 00000000..350bf08e --- /dev/null +++ b/modules/developer/pages/clients/rclone/webdav-sync-basic-auth.adoc @@ -0,0 +1,43 @@ += WebDAV with Basic Authentication +:toc: right +:toclevels: 3 + +== WebDAV with Basic Authentication + +[CAUTION] +==== +Basic Authentication is disabled by default in oCIS because of security considerations. In order to make the following Rclone commands work the oCIS administrator needs to enable Basic Authentication e.g. by setting the environment variable `PROXY_ENABLE_BASIC_AUTH` to `true`. + +Please consider to use xref:webdav-sync-oidc.adoc[Rclone with OpenID Connect] instead. +==== + + +For the usage of a WebDAV remote with Rclone see also the link:https://rclone.org/webdav/[Rclone documentation] + +== Configure the WebDAV remote + +First of all we need to set up our credentials and the WebDAV remote for Rclone. In this example we do this by setting environment variables. You might also set up a named remote or use command line options to achieve the same. + +---- +export RCLONE_WEBDAV_VENDOR=owncloud +export RCLONE_WEBDAV_URL=https://ocis.owncloud.test/remote.php/webdav/ +export RCLONE_WEBDAV_USER=einstein +export RCLONE_WEBDAV_PASS=$(rclone obscure relativity) +---- + +[NOTE] +==== +Please note that `RCLONE_WEBDAV_PASS` is not set to the actual password, but to the value returned by `rclone obscure `. +==== + + +We now can use Rclone to sync the local folder `/tmp/test` to `/test` in your oCIS home folder. + + +=== Sync to the WebDAV remote + +---- +rclone sync :local:/tmp :webdav:/test +---- + +If your oCIS doesn't use valid SSL certificates, you may need to use `rclone --no-check-certificate sync ...`. diff --git a/modules/developer/pages/clients/rclone/webdav-sync-oidc.adoc b/modules/developer/pages/clients/rclone/webdav-sync-oidc.adoc new file mode 100644 index 00000000..d9879894 --- /dev/null +++ b/modules/developer/pages/clients/rclone/webdav-sync-oidc.adoc @@ -0,0 +1,70 @@ += WebDAV with OpenID Connect +:toc: right +:toclevels: 3 + +== WebDAV with OpenID Connect + +Rclone itself is not able to open and maintain an OpenID Connect session. But it is able to still use OpenID Connect for authentication by leveraging a so called OIDC-agent. + +=== Setting up the OIDC-agent + +You need to install the link:https://github.com/indigo-dc/oidc-agent[OIDC-agent] from your OS' package repository (e.g. link:https://github.com/indigo-dc/oidc-agent#debian-packages[Debian] or link:https://github.com/indigo-dc/oidc-agent#debian-packages[MacOS]). + + +=== Configuring the OIDC-agent + +Run the following command to add a OpenID Connect profile to your OIDC-agent. It will open the login page of OpenID Connect identity provider where you need to log in if you don't have an active session. + +---- +oidc-gen \ + --client-id=oidc-agent \ + --client-secret="" \ + --pub \ + --issuer https://ocis.owncloud.test \ + --redirect-uri=http://localhost:12345 \ + --scope max \ + einstein-ocis-owncloud-test +---- + +If you have dynamic client registration enabled on your OpenID Connect identity provider, you can skip the `--client-id`, `--client-secret` and `--pub` options. + +If you're using a dedicated OpenID Connect client for the OIDC-agent, we recommend a public one with the following two redirect URIs: `http://127.0.0.1:_` and `http://localhost:_`. Alternatively you also may use the already existing OIDC client of the ownCloud Desktop Client (`--client-id=xdXOt13JKxym1B1QcEncf2XDkLAexMBFwiT9j6EfhhHFJhs2KM9jbjTmf8JBXE69` and `--client-secret=UBntmLjC2yYCeHwsyj73Uwo9TAaecAetRwMw0xYcvNL9yRdLSUi0hUAHfvCHFeFh`, no `--pub` set, request specific scope for offline access), e.g.: +---- +oidc-gen \ + --client-id=xdXOt13JKxym1B1QcEncf2XDkLAexMBFwiT9j6EfhhHFJhs2KM9jbjTmf8JBXE69 \ + --client-secret=UBntmLjC2yYCeHwsyj73Uwo9TAaecAetRwMw0xYcvNL9yRdLSUi0hUAHfvCHFeFh \ + --issuer https://cloud.ocis.test \ + --redirect-uri=http://localhost:12345 \ + --scope="openid offline_access profile email" \ + my-client +---- + +When using a self signed certificate you have to provide the certificate chain using `--cp /etc/ssl/certs/test.cert.pem`. In case oidc-gen cannot determine the flow try with `--flow=code`. + +Please also note that the OIDC-agent will listen on your localhost interface on port 12345 for the time of the initial authentication. If that port is already occupied on your machine, you can easily change that by setting the `--redirect-uri` parameter to a different value. + +After a successful login or an already existing session you will be redirected to success page of the OIDC-agent. +You will now be asked for a password for your account configuration, so that your OIDC session is secured and cannot be used by other people with access to your computer. + + + +== Configure the WebDAV remote + +First of all we need to set up our credentials and the WebDAV remote for Rclone. In this example we do this by setting environment variables. You might also set up a named remote or use command line options to achieve the same. + +---- +export RCLONE_WEBDAV_VENDOR=owncloud +export RCLONE_WEBDAV_URL=https://ocis.owncloud.test/remote.php/webdav/ +export RCLONE_WEBDAV_BEARER_TOKEN_COMMAND="oidc-token einstein-ocis-owncloud-test" +---- + + +=== Sync to the WebDAV remote + +We now can use Rclone to sync the local folder `/tmp/test` to `/test` in your oCIS home folder. + +---- +rclone sync :local:/tmp :webdav:/test +---- + +If your oCIS doesn't use valid SSL certificates, you may need to use `rclone --no-check-certificate sync ...`. diff --git a/modules/developer/pages/index.adoc b/modules/developer/pages/index.adoc new file mode 100644 index 00000000..0c078833 --- /dev/null +++ b/modules/developer/pages/index.adoc @@ -0,0 +1,45 @@ += ownCloud +:toc: right +:toclevels: 3 + + +== Admin Documentation + +Before you start reading, if you are interested in: + +- learning ocis from an admin perspective, +- different deployment scenarios, +- deployment examples, +- detailed settings and more + +we would recommend to continue with the link:https://doc.owncloud.com/ocis/latest/[ownCloud Admin Documentation for Infinite Scale]. + +== Developer Documentation + +Welcome to our developer documentation. Here you find documentation with focus for _developers_: + +- xref:./ocis.adoc[oCIS] server +- oCIS Services +- Clients like: + - link:https://github.com/owncloud/web[ownCloud Web] - the new web frontend for oCIS and ownCloud + - link:https://github.com/owncloud/android[ownCloud Android app] + - link:https://github.com/owncloud/ios-app[ownCloud iOS app] + - link:https://github.com/owncloud/client[ownCloud Desktop Syncing Client] +- Integrations + +== We love open source + +The oCIS server is Apache v2 licensed. +The lower storage layer of oCIS is defined by the CS3 APIs and implemented in the REVA project. Our goal is to develop the CS3 APIs to an open standard and collaborate on the open source REVA reference implementation for CS3 APIs. + +You can also find all client sources on link:https://github.com/owncloud/[GitHub]. + +== Join the oCIS Community + +The link:https://github.com/owncloud/ocis[server repository] on link:https://www.github.com[GitHub] is a good entry point to the oCIS project. In addition to that there are also ownCloud projects for clients for link:https://github.com/owncloud/ios-app[iOS], link:https://github.com/owncloud/android[Android], the major link:https://github.com/owncloud/desktop[Desktop] platforms and link:https://github.com/owncloud/web[ownCloud Web]. + +To chat about development, join our public chat on link:https://matrix.to/#/#ocis:matrix.org[matrix: ownCloud Infinite Scale]. + +If you want to help and improve ownCloud or oCIS, start coding or open issues on GitHub in the related repository. + +We are very happy to hear your feedback and ideas! diff --git a/modules/developer/pages/ocis/adr/0001-introduce-accounts-service.adoc b/modules/developer/pages/ocis/adr/0001-introduce-accounts-service.adoc new file mode 100644 index 00000000..9fb9617b --- /dev/null +++ b/modules/developer/pages/ocis/adr/0001-introduce-accounts-service.adoc @@ -0,0 +1,45 @@ += 1. Introduce an accounts service +:toc: right +:toclevels: 3 + +* Status: superseded by xref:0003-external-user-management.adoc[ADR-0003] +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/felixboehm[@felixboehm], link:https://github.com/micbar[@micbar], link:https://github.com/pmaier1[@pmaier1] +* Date: link:https://github.com/owncloud/ocis-accounts/pull/34/commits/2fd05e2b6fe2a47c687bd0c0bc5e1b5c48a585b2[2020-06-15] + +Technical Story: link:https://github.com/owncloud/ocis-accounts/pull/34[persist accounts] + +== Context and Problem Statement + +To attach metadata like shares to users ownCloud relies on persistent, non-reassignable, unique identifiers for users (and files). Email und username can change when a user changes his name. But even the OIDC sub+iss combination may change when the IdP changes. While there is link:https://openid.net/specs/openid-connect-account-porting-1_0.html[an account porting protocol] that describes how a relying party (RP) such as ownCloud should behave, it still requires the RP to maintain its own user identifiers. + +== Decision Drivers + +* oCIS should be a single binary that can run out of the box without external dependencies like an LDAP server. +* Time: we want to build a release candidate asap. +* Firewalls need access to guests, typically via LDAP. +* Not all external LDAPs are writeable for us to provision Guest accounts. +* We see multiple LDAP servers in deployments. Being able to handle them is important and should be covered by using OIDC + being able to query multiple LDAP servers. + +== Considered Options + +* Accounts service wraps LDAP +* link:https://github.com/glauth/glauth[GLAuth] wraps accounts service + +== Decision Outcome + +Chosen option: "GLAuth wraps accounts service", because we need write access to provision guest accounts and GLAuth currently has no write support. + +=== Positive Consequences + +* We can build a self-contained user management in the accounts service and can adjust it to our requirements. +* We do not rely on an LDAP server which would only be possible by implementing write support in the LDAP libraries used by GLAuth (hard to estimate effort, when will that be merged upstream). + +=== Negative Consequences + +* We need to spend time on implementing user management + +== Pros and Cons of the Options + +=== Accounts service wraps LDAP + +* Bad, because not all external LDAPs are writeable for us to provision Guest accounts. diff --git a/modules/developer/pages/ocis/adr/0002-persist-accounts-using-cs3-storage.adoc b/modules/developer/pages/ocis/adr/0002-persist-accounts-using-cs3-storage.adoc new file mode 100644 index 00000000..adabafca --- /dev/null +++ b/modules/developer/pages/ocis/adr/0002-persist-accounts-using-cs3-storage.adoc @@ -0,0 +1,31 @@ += 2. Persist accounts in a CS3 storage +:toc: right +:toclevels: 3 + +* Status: accepted +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/felixboehm[@felixboehm] +* Date: 2020-08-21 + +Technical Story: link:https://github.com/owncloud/ocis-accounts/pull/92[File system based indexing] + +== Context and Problem Statement + +To set up High Availability (HA) or a geo-replicated setup we need to persist accounts in a distributed way. To efficiently query the accounts by email or username, and not only by id, they need to be indexed. Unfortunately, the link:https://github.com/blevesearch/bleve[bleve] index we currently store locally on disk cannot be shared by multiple instances, preventing a scale out deployment. + +== Considered Options + +* Look into distributed bleve +* Persist users in a CS3 storage + +== Decision Outcome + +Chosen option: "Persist users in a CS3 storage", because we have one service less running and can rely on the filesystem for geo-replication and HA. + +=== Positive Consequences + +* We can store accounts on the storage using the CS3 API, pushing geo-distribution to the storage layer. +* Backups of users and storage can be implemented without inconsistencies between using snapshots. + +=== Negative Consequences + +* We need to spend time on implementing a reverse index based on files, and symlinks. diff --git a/modules/developer/pages/ocis/adr/0003-external-user-management.adoc b/modules/developer/pages/ocis/adr/0003-external-user-management.adoc new file mode 100644 index 00000000..06ef7e1d --- /dev/null +++ b/modules/developer/pages/ocis/adr/0003-external-user-management.adoc @@ -0,0 +1,110 @@ += 3. Use external User Management +:toc: right +:toclevels: 3 + +* Status: superseded by xref:0017-allow-read-only-external-user-management.adoc[17. Allow read only external User Management] +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/hodyroff[@hodyroff], link:https://github.com/pmaier1[@pmaier1] +* Date: 2022-02-08 + +Technical Story: link:https://github.com/owncloud/ocis/pull/1020[Skip account-service by talking to CS3 user-api] + +== Context and Problem Statement + +To attach metadata like shares to users ownCloud relies on persistent, non-reassignable, unique identifiers for users (and files). Email and username can change when a user changes his name. But even the OIDC sub+iss combination may change when the IdP changes. While there is link:https://openid.net/specs/openid-connect-account-porting-1_0.html[an account porting protocol] that describes how a relying party (RP) such as ownCloud should behave, it still requires the RP to maintain its own user identifiers. + +== Decision Drivers + +* oCIS should be a single binary that can run out of the box without external dependencies like an LDAP server. +* Time: we want to build a release candidate asap. +* oCIS should be able to be easily integrated with standard user management components + +== Considered Options + +* Accounts service wraps LDAP +* link:https://github.com/glauth/glauth[GLAuth] wraps accounts service + +== Decision Outcome + +Chosen option: "Move accounts functionality to GLAuth and name it accounts", by moving the existing accounts service file based persistence to GLAuth and use it as a drop in replacement for an LDAP server. The reverse index and web UI existing in the accounts service will move as well in order to make GLAuth a standalone, small scale user management with write capabilities. + +=== Product summary +- GLAuth is a drop in user management for small scale deployments that do not rely on an actual LDAP server. +- oCIS admins can either use the web UI to manage users in GLAuth or use existing tools in their IDM. +- We hide the complexity by embedding an OpenID Provider, an LDAP server and a user management web UI. + +=== Resulting deployment options +- Use internal user management + - Recommended for small scale use cases and simple deployments + - Users, groups and roles are stored and managed within GLAuth +- Use external user management + - Recommended for mid and large scale use cases + - Users, groups and roles are stored and managed within an external LDAP / AD / IDM + - Separate oCIS and LDAP admin: oCIS admin relies on the LDAP admin to manage users +- User permissions for roles are always managed in oCIS (settings service) because they are specific to oCIS + +=== Resulting technical implications +- Make the file based reverse index a standalone library +- Contribute to GLAuth + - Add ms graph based rest API to manage users, groups and roles (the LDAP lib is currently readonly) + - Add web UI to glauth that uses the ms graph based rest API to manage users + - Add a backend that uses the file based reverse index, currently living in the oCIS accounts service + - Move fallback mechanism from ocis/glauth service to upstream GLAuth to support multiple LDAP servers + - Make it a chain to support more than two LDAP servers + - Document the implications for merging result sets when searching for recipients + - At least one writeable backend is needed to support creating guest accounts +- Make all services currently using the accounts service talk to the CS3 userprovider +- To support multiple LDAP servers we need to move the fallback mechanism in ocis/glauth service to upstream GLAuth +- The current CS3 API for user management should be enriched with pagination, field mask and a query language as needed +- properly register an link:https://github.com/owncloud/ocis/blob/c8668e8cb171860c70fec29e5ae945bca44f1fb7/deployments/examples/cs3_users_ocis/config/ldap/ldif/10_owncloud_schema.ldif[auxiliary LDAP schema that adds an ownCloudUUID attribute to users and groups] + +=== Positive Consequences + +* The accounts service (which is our drop in LDAP solution) can be dropped. The CS3 userprovider service becomes the only service dealing with users. +* No sync + +=== Negative Consequences + +* If users want to store users in their IDM and at the same time guests in a separate user management we need to implement GLAuth backends that support more than one LDAP server. + +== Pros and Cons of the Options + +=== GLAuth wraps accounts service + +Currently, the accounts service is the source of truth and we use it to implement user management. + +* Good, because it solves the problem of storing and looking up an owncloud UUID for a user (and group) +* Good, because we can manage users out of the box +* Good, because we can persist accounts in a CS3 storage provider +* Bad, because it maintains a separate user repository: it needs to either learn or sync users. + +=== Move accounts functionality to GLAuth and name it accounts + +We should use an existing LDAP server and make GLAuth a drop in replacement for it. + +* Good, because we can use an existing user repository (an LDAP server), no need to sync or learn users. +* Good, because admins can rely on existing user management tools. +* Good, because we would have a clear separation of concerns: + - users reside in whatever repository, typically an LDAP server + - could be an existing LDAP server or AD + - could be our embeddable drop in glauth server + - we use a service to wrap the LDAP server with other APIs: + - ms graph API - ODATA based restful API, + - link:http://www.simplecloud.info/[SCIM] - designed to manage user identities, supported by some IDPs, + - the current accounts API (which is a protobuf spec following the ms graph API) + - our account management UI can use the ms graph based API service which can have different backends + - an existing LDAP server + - our drop in glauth server (which might serve the ms graph based API itself) + - the CS3 API + a future guest provisioning API + a future CS3 user provisioning API (or link:https://github.com/cs3org/cs3apis/pull/95[generic space provisioning]) + - all oCIS services can use the service registry to look up the accounts service that provides an internal API + - could be the CS3 user provider (and API) + - could be the internal protobuf accounts API + - introduce a new guest provisioning API to CS3 which properly captures our requirement to have them in the user repository + - guests need to be made available to the firewall + - storages like EOS that integrate with the os for acl based file permissions need a numeric user and group id +* Good, because we can use the CS3 user provider with the existing LDAP / rest driver. +* Bad, because oCIS admins may not have the rights to manage role assignments. (But this is handled at a different department.) +* Bad, because oCIS admins may not have the rights to disable users if an external LDAP is used instead of the drop in GLAuth. + +== Links +* supersedes xref:0001-introduce-accounts-service.adoc[ADR-0001] +* superseded by xref:0017-allow-read-only-external-user-management.adoc[17. Allow read only external User Management] diff --git a/modules/developer/pages/ocis/adr/0004-support-hot-migration.adoc b/modules/developer/pages/ocis/adr/0004-support-hot-migration.adoc new file mode 100644 index 00000000..9135014e --- /dev/null +++ b/modules/developer/pages/ocis/adr/0004-support-hot-migration.adoc @@ -0,0 +1,75 @@ += 4. Support Hot Migration +:toc: right +:toclevels: 3 + +* Status: proposed +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/hodyroff[@hodyroff], link:https://github.com/pmaier1[@pmaier1] +* Date: 2021-03-16 + +Technical Story: \[description | ticket/issue URL\] + +== Context and Problem Statement + +Migration is one of the most important topics of the oCIS story. We need to provide a concept how to migrate from oC10 to oCIS. + +== Decision Drivers + +- Do not lose file blob or metadata. + - To prevent a sync surge from clients the etag for files should be migrated. + - To prevent internal links from breaking or pointing to wrong files the file id of existing files needs to be migrated. + - To prevent data loss trash and version blobs should be migrated. +- Existing shares like public links and federated shares must remain functional after the migration. + - To prevent internal shares the share type, permissions and expiry needs to be migrated. + - To prevent public links from breaking the url token, permissions, expiry and password needs to be migrated. + - _What about federated shares?_ + - _What about additional share permissions, e.g. comment on office files?_ +- Legacy clients need to keep working + - To keep existing clients working the `remote.php/webdav` and `dav/files/` webdav endpoints as well as the ocs API need to be available. +- _What about link:https://doc.owncloud.com/server/user_manual/personal_settings/security.html#app-passwords-tokens[app passwords/tokens]?_ + +== Considered Options + +1. Cold Migration: migrate data while systems are not online, so no user interaction happens in between. +2. Hot Migration: one or both systems are online during migration. + +== Decision Outcome + +Chosen option: "\[option 1\]", because \[justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | … | comes out best (see below)\]. + +=== Positive Consequences + +- \[e.g., improvement of quality attribute satisfaction, follow-up decisions required, …\] +- … + +=== Negative Consequences + +- \[e.g., compromising quality attribute, follow-up decisions required, …\] +- … + +== Pros and Cons of the Options + +=== Cold Migration + +The migration happens while the service is offline. File metadata, blobs and share data is exported from ownCloud 10 and imported in oCIS. This can happen user by user, where every user export would contain the file blobs, their metadata, trash, versions, shares and all metadata that belongs to the user's storage. To prevent group shares from breaking, users in the same groups must be migrated in batch. Depending on the actual group shares in an instance this may effectively require a complete migration in a single batch. + +- Good, because oCIS can be tested in a staging system without writing to the production system. +- Good, because file layout on disk can be changed to support new storage driver capabilities. +- Bad, because the export and import might require significant amounts of storage. +- Bad, because a rollback to the state before the migration might cause data loss of the changes that happened in between. +- Bad, because the cold migration can mean significant downtime. + +=== Hot Migration + +The migration happens in subsequent stages while the service is online. + +- Good, because the admin can migrate users from old to new backend in a controlled way. +- Good, because users and admins can learn to trust the new system. +- Good, because there can be preparations even long before the migrations happens in parallel on the oC10 codebase, i.e. addition of metadata that is needed while the system operates. +- Good, because the downtime of the system can be fairly small. +- Bad, because it is more complex and might drag on for a long time. + + +== Links + + +- link:https://github.com/cs3org/reva/issues/1377[Clarify responsibilities of share providers and storage providers · Issue #1377 · cs3org/reva (github.com)] because the share manager for oCIS should store share information on the storage system. And link:https://github.com/cs3org/cs3apis/issues/93[storage provider should persist share creator · Issue #93 · cs3org/cs3apis (github.com)] finally: link:https://github.com/cs3org/reva/issues/543[eos: store share id in inherited xattr · Issue #543 · cs3org/reva (github.com)] diff --git a/modules/developer/pages/ocis/adr/0005-cs3-api-account-management.adoc b/modules/developer/pages/ocis/adr/0005-cs3-api-account-management.adoc new file mode 100644 index 00000000..3497dda5 --- /dev/null +++ b/modules/developer/pages/ocis/adr/0005-cs3-api-account-management.adoc @@ -0,0 +1,205 @@ += 5. Account Management through CS3 API +:toc: right +:toclevels: 3 + +* Status: proposed +* Deciders: link:https://github.com/refs[@refs], link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/pmaier1[@pmaier1] +* Date: 2021-04-12 + +Technical Story: link:https://github.com/cs3org/cs3apis/pull/119[Write only management API for User and Group resources] + +== Context and Problem Statement + +What would be a more effective way of using network resources and handle account management within the oCIS-Reva ecosystem? Separating account management at the service level is pragmatic and allows for fast iterations, but also steadily accumulates inconsistencies and bloats technical debt. + +== Decision Drivers + +* Reduce number of network calls. +* Reduce number of services (merge Account + GLAuth from ADR-0003). +* Formalize account management at the API level. + +== Considered Options + +* Account management delegated to vendors. +* Add account management to the CS3 API. + +== Decision Outcome + +Chosen option: "Add account management to the CS3 API". Making the API declare an API for account management will not only allow a deployment to fail fast (as in: the management node is not running) but would also centralize all management operations that should happen to be constrained within the Reva context. Constrained operations _SHOULD_ be by definition more secure, or at least as secure as the rest of the system. + +=== Positive Consequences + +* More resilient API. + * Because account management is considered a "first class citizen" changes are forced to go through a more exhaustive revision process. +* Removing Accounts from search users1. +* Replace the provisioning API in favor of the new Reva Admin node. + +(1) the current vendor implementation of searching a user (i.e: when sharing a resource) relies directly on the accounts service, since this is the only source of truth. Searching a user looks like: + +---- +┌────────────────────────────────────────┐ +│user search (no LDAP) │ +│ │ +│ ┌──────────┐ │ +│ │ │ │ +│ │ proxy │ │ +│ │ │ ┌ ─ ─ ─ ─ ─ ┐ │ +│ └──────────┘ go-micro │ +│ ▲ │ │ │ +│ │ Λ │ +│ ▼ │ ╱ ╲ │ │ +│ ┌──────────┐ ╱ ╲ │ +│ │ │ │ ╱ ╲ │ │ +│ │ ocs │◀──(1)───▶registry▏ │ +│ │ │ │ ╲ ╱ │ │ +│ └──────────┘ ╲ ╱ │ +│ ▲ │ ╲ ╱ │ │ +│ │ V │ +│ │ │ │ │ +│ │ │ +│ │ └ ─ ─ ─ ─ ─ ┘ │ +│ │ │ +│ │ │ +│ │ ┌──────────┐ │ +│ │ │ │ │ +│ └─────────────▶│ accounts │ │ +│ │ │ │ +│ └──────────┘ │ +│ │ +│ │ +│(1) ocs requests a connection to the │ +│accounts service to the registry │ +│ │ +└────────────────────────────────────────┘ +---- + +Whereas, as a result of ADR-0003 and this ADR, we can simplify and improve this design: + +---- +┌─────────────────────────────────────────────┐ +│user search │ +│ │ +│ │ +│ ┌──────────┐ │ +│ │ │ │ +│ │ proxy │ │ +│ │ │ │ +│ └──────────┘ │ +│ │ │ +│ ▼ │ +│ ┌──────────┐ │ +│ │ │ │ +│ │ ocs │ │ +│ │ │ │ +│ └──────────┘ │ +│ │ │ +│ │ │ +│ ┌ ─ ─ ─ ─ ─│─ ─ ─ ─ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐ │ +│ reva ▼ │ IDM │ +│ │ ┌──────────┐ │ ┌──────────┐ │ │ +│ │ │ │ │ │ │ +│ │ │ users │◀─────┼──▶│ GLAuth │ │ │ +│ │ │ │ │ │ │ +│ │ └──────────┘ │ └──────────┘ │ │ +│ │ │ +│ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ └ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ +│ │ +└─────────────────────────────────────────────┘ +---- + +And instead rely on the already existing Reva users provider. + + +== Pros and Cons of the Options + +=== Account management delegated to vendors + +* Good, because it allows for fast iterations. +* Bad, because account management happens outside of the Reva process. This can potentially end up in invalid account creation / deletion / updates. + * An example with the existing Accounts service is that any client can fire CRUD accounts requests to the Accounts service as long as the client knows where the server is running and provides with an Authorization header (only required by the proxy). This request totally bypasses Reva middlewares and therefore any security measures that should be enforced by the entire system. +* Bad, because leaves teams the task of designing and implementing a way of dealing with account management. Ideally one schema should be provided / suggested. + +Creating an account using the first option looks currently is implemented in vendors as: + +---- +┌──────────────────────────────────────────────────┐ +│ creating a user (webui) │ +│ │ +│ ┌──────────┐ │ +│ │ │ │ +│ │ proxy │ │ +│ │ │ │ +│ └──────────┘ │ +│ │ │ +│ │ │ +│ /api/v0/accounts/accounts-create │ +│ │ │ +│ │ │ +│ │ │ +│ ▼ │ +│ ┌──────────┐ │ +│ │ │ │ +│ │ accounts │ │ +│ │ │ │ +│ └──────────┘ │ +│ │ +│ note that while doing CRUD operations changes │ +│ are instantly reflected for the IDP since out of │ +│ the box oCIS uses an accounts backend for │ +│ GLAuth. │ +└──────────────────────────────────────────────────┘ +---- + +As explained before, during this flow no Reva middlewares are run. Creating an account will only use the embedded accounts js file alongside a minted jwt token (by the oCIS proxy) to communicate with the accounts service. + +=== Add account management to the CS3 API + +* Good, because it solidifies what the CS3 API can or cannot do, and account management should be handled at the API level since ultimately accounts would contain a mix of required CS3 and vendor-specific attributes. +* Good, because it centralizes account management and constrains it within the Reva boundaries. +* Good, because there is a clear separation of concerns on what is accounts management logic. +* Good, because we already designed link:https://github.com/owncloud/ocis/blob/master/accounts/pkg/proto/v0/accounts.proto#L42-L85[a similar API for the accounts service] the only difference being we (vendors) link:https://github.com/owncloud/ocis/blob/master/accounts/pkg/proto/v0/accounts.proto#L252-L408[define their own messages]. + * The API would fully include CRUD methods +* Bad, because development cycles are larger. + * an example flow will be: `update api > run prototool > publish language specific packages > update dependencies to fetch latest version of the package > utilize the new changes`. + +The new account management workflow will result in: +---- +┌───────────────────────────────────────────────────┐ +│creating a user (webui) │ +│ - maintain the same route for compatibility │ +│ │ +│ ┌──────────┐ │ +│ │ │ │ +│ │ proxy │ │ +│ │ │ │ +│ └──────────┘ │ +│ │ │ +│ │ │ +│ /api/v0/accounts/accounts-create │ +│ │ │ +│ │ │ +│ ┌ ─ ─ ─ ─ ─│─ ─ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ │ +│ Reva │ IDM │ │ +│ │ │ │ │ │ +│ ▼ │ │ +│ │ ┌──────────┐ │ │ ┌──────────┐ │ +│ │ │ │ │ │ │ +│ │ │ admin │───────────┼──┼──▶│ GLAuth │ │ +│ │ │ │ │ │ │ +│ │ └──────────┘ │ │ └──────────┘ │ +│ │ │ +│ │ │ │ │ +│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ +│ │ +│ │ +│ │ +│ │ +│ │ +│ │ +│ an example of a driver could be GLAuth │ +│ implementing the user management portion of the │ +│ GraphAPI │ +└───────────────────────────────────────────────────┘ +---- + +This flow allows Reva and oCIS Proxy to run any middleware logic in order to validate a request. The communication between the proposed Admin api (CS3 API messages) and the IDM (GLAuth) are specific to the _drivers_. diff --git a/modules/developer/pages/ocis/adr/0006-service-discovery.adoc b/modules/developer/pages/ocis/adr/0006-service-discovery.adoc new file mode 100644 index 00000000..9fb2a817 --- /dev/null +++ b/modules/developer/pages/ocis/adr/0006-service-discovery.adoc @@ -0,0 +1,53 @@ += 6. Service Discovery within oCIS and Reva +:toc: right +:toclevels: 3 + +* Status: accepted +* Deciders: link:https://github.com/refs[@refs], link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/pmaier1[@pmaier1] +* Date: 2021-04-19 + +Technical Story: link:https://github.com/cs3org/reva/pull/1509[Introduce Named Services.] + +== Context and Problem Statement + +Reva relies heavily on config files. A known implication of this approach are having to know a-priori where a service is running (host + port). We want to move away from hardcoded values and rely instead on named services for service discovery. Furthermore, we would like both platforms (Reva + oCIS) to have the same source of truth at any given time, not having one to notify the other whenever a service status changes. + +== Decision Drivers + +* Avoid a-priori knowledge of services. +* Ease of scalability. +* Always up-to-date knowledge of the running services on a given deployment (a service registry doesn't have to necessarily be running on the same machine / network) + +== Considered Options + +* Hardcoded tuples of hostname + port +* Dynamic service registration + +== Decision Outcome + +Chosen option: "Dynamic service registration". There were some drawbacks regarding this due to introducing go-micro to Reva was from start an issue. Given the little usage of go-micro we need, we decided instead to define our very own link:https://github.com/refs/reva/blob/58d013a7509d1941834e1bc814e9a9fa8bff00b1/pkg/registry/registry.go#L22-L35[Registry interface] on Reva and extended the runtime arguments to link:https://github.com/refs/reva/blob/58d013a7509d1941834e1bc814e9a9fa8bff00b1/cmd/revad/runtime/option.go#L53-L58[allow for injecting a registry]. + +=== Positive Consequences + +* Having dynamic service registration delegates the entire lifecycle of finding a process to the service registry. +* Removing a-priori knowledge of hostname + port for services. +* Marrying go-micro's registry and a newly defined registry abstraction on Reva. +* We will embrace go-micro interfaces by defining a third merger interface in order to marry go-micro registry and reva registry. +* The ability to fetch a service node relying only on its name (i.e: com.owncloud.proxy) and not on a tuple hostname + port that we rely on being preconfigured during runtime. +* Conceptually speaking, a better framework to tie all the services together. Referring to services by names is less overall confusing than having to add a service name + where it is running. A registry is agnostic to "where is it running" because it, by definition, keeps track of this specific question, so when speaking about design or functionality, it will ease communication. + +== Pros and Cons of the Options + +=== Hardcoded tuples of hostname + port + +* Good, because firewalls are easier to configure since IP are static. +* Good, because the mental model required is easier to grasp as IP addresses can be easily bundled. +* Bad, because it requires thorough planning of ports. + +=== Dynamic service registration + +* Good, because it abstracts the use of service lookup away to registry logic from the admin or developer. +* Good, because it allows for, through interfaces, registry injection + * This means we can have a service registry that we extensively use in oCIS and inject its functionality onto Reva. +* Bad, because it's yet another abstraction. +* Bad, because firewalls are harder to configure with dynamic IPs.f diff --git a/modules/developer/pages/ocis/adr/0007-api-for-spaces.adoc b/modules/developer/pages/ocis/adr/0007-api-for-spaces.adoc new file mode 100644 index 00000000..5225e22e --- /dev/null +++ b/modules/developer/pages/ocis/adr/0007-api-for-spaces.adoc @@ -0,0 +1,163 @@ += 7. Open Graph API for oCIS File Spaces +:toc: right +:toclevels: 3 + +* Status: accepted +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/hodyroff[@hodyroff], link:https://github.com/pmaier1[@pmaier1] +* Date: 2021-03-19 + +Technical Story: API to enable the concept of link:https://github.com/owncloud/enterprise/issues/3863[Spaces] + +== Context and Problem Statement + +As one of the building blocks for Spaces in oCIS we plan to add an API that returns information about available spaces. This ADR discusses the API design oriented on the Microsoft Graph API. + +[quote] +____ +Note: The term "spaces" is used here in the context of "a space where files can be saved", similar to a directory. It is not to be confused with space in the sense of free file space for example. +____ + + +The purpose of this new API is to give clients a very simple way to query the dynamic list of spaces, that the user has access to. Clients can provide a better user experience with that. + +This API is supposed to be queried often, to give clients a condensed view of the available spaces for a user, but also their eTags and cTags. Hence the clients do not have to perform a PROPFIND for every space separately. + +This API would even allow providing (WebDAV-) endpoints depending on the kind and version of the client asking for it. + +== Decision Drivers + +- Make it easy to work with a dynamic list of spaces of a user for the clients. +- No longer the need to make assumptions about WebDAV- and other routes in clients. +- More meta data available about spaces for a better user experience. +- Part of the bigger spaces plan. +- Important to consider in client migration scenarios, i.e. in CERN. + +== Considered Options + +1. link:https://developer.microsoft.com/en-us/graph[Microsoft Graph API] inspired API that provides the requested information. + +== Decision Outcome + +This the DRAFT for the API. + +=== API to Get Info about Spaces + +ownCloud servers provide an API to query for available spaces of an user. + +See the openAPI Specification for the link:https://owncloud.dev/libre-graph-api/[Libre Graph API]. + +Most important, the API returns the WebDAV endpoint for each space. With that, clients do not have to make assumptions about WebDAV routes any more. + +See link:https://docs.microsoft.com/en-us/graph/api/resources/onedrive?view=graph-rest-1.0[Drive item in Microsoft Graph API] for an overview of `drive` and `driveItem` resources. The concrete list of drives / spaces a user has access to can be obtained on multiple endpoints. + +=== Get "Home folder" + +Retrieve information about the home space of a user. Note: The user has access to more spaces. This call only returns the home space to provide API parity with the Graph API. + +API Call: `/me/drive`: Returns the information about the users home folder. + +=== Get All Spaces of a User + +Retrieve a list of available spaces of a user. This includes all spaces the user has access to at that moment, also the home space. + +API Call: `/me/drives`: Returns a list of spaces. + +There is also `/drives`, returning the list of spaces the user has access to. This endpoint is used to access any space by id using `/drives/{drive-id}`. + +=== Common Reply + +The reply to both calls is either one or a list of link:https://docs.microsoft.com/de-de/graph/api/resources/drive?view=graph-rest-1.0[Drive representation objects]: + +---- +{ + "id": "string", + "createdDateTime": "string (timestamp)", + "description": "string", + "driveType": "personal | projectSpaces | shares", + "lastModifiedDateTime": "string (timestamp)", + "name": "string", + "owner": { "@odata.type": "microsoft.graph.identitySet" }, + "quota": { "@odata.type": "microsoft.graph.quota" }, + "root": { "@odata.type": "microsoft.graph.driveItem" }, + "webUrl": "url" +} +---- + +The meaning of the objects in Open Graph API context are: + +1. _id_ - a persistent and unique ID identifying the space, called Storage Space ID. +2. _driveType_ - describing the type of the space. +3. _owner_ - an owner object to whom the space belongs +4. _quota_ - quota information about this space +5. _root_ - the root driveItem object. +6. _webUrl_ - The URL to make this space visible in the browser. + +The following _driveType_ values are available in the first step, but might be enhanced later: + +* _personal_: The users home space +* _projectSpaces_: The project spaces available for the user (*) +* _shares_: The share jail, contains all shares for the user (*) + +Other space types such as backup, hidden etc. can be added later as requested. + +[quote] +____ +Note: The _projectSpaces_ and _shares_ space are virtual spaces. They only contain other spaces, and no regular resources. +____ + + +The (*) marked types are not defined in the official MS API. They are prefixed with `oc` to avoid namespace clashes. + +The `root` object equals a link:https://docs.microsoft.com/de-de/graph/api/resources/driveitem?view=graph-rest-1.0[driveItem] and contains information about the root resource (directory) of the space. + +This is an example object as it can be expected as `root` element. It is not complete, as not all elements will be implemented so far. + +---- +{ + "cTag": "string (etag)", + "webDavUrl": "string", + + /* inherited from baseItem */ + "id": "string (identifier)", + "createdBy": {"@odata.type": "microsoft.graph.identitySet"}, + "createdDateTime": "String (timestamp)", + "eTag": "string", + "lastModifiedBy": {"@odata.type": "microsoft.graph.identitySet"}, + "lastModifiedDateTime": "String (timestamp)", + "name": "string", + "webUrl": "string", +} +---- + +Meaningful fields of the root element in the context of the Open Graph API: + +1. _id_ - a persistent and unique ID identifying the root directory node. +2. _webDavUrl_ - The webdav path of the top item of the space. +3. _eTag_ - an identifier that changes automatically if the content _or_ metadata of the node or the underlying resources changes. +4. _cTag_ - an identifier that changes automatically if the content of the root node or of one of the underlying resources changes. +5. _webUrl_ - The URL to make this space visible in the browser. + +[quote] +____ +Note: To indicate that only the metadata of a resource has changed, the eTag has changed, but the cTag not. +____ + + +=== Positive Consequences + +- A well understood and mature API from Microsoft adopted to our needs. +- Prerequisite for Spaces in oCIS. +- Enables further steps in client development. + +=== Negative Consequences + +- Migration impact on existing installations. Still to be investigated. +- Requires additional webdav endpoint that allows accessing an arbitrary storage space, either + - with an id: `/dav/spaces//relative/path/to/file.ext`, or + - with a global path: `/dav/global////relative/path/to/file.ext`, e.g. `/dav/global/projects/Golive 2021/Resources/slides.odt` + +=== Open Topics + +- What are the WebDAV paths for Trashbin, Versions + + option: additional entries in the reply struct +- The identitySet object used for "owner" and "coowner" require to implement the link:IdentitySet[https://docs.microsoft.com/de-de/graph/api/resources/identityset?view=graph-rest-1.0] JSON object, which contains information that seems to be of limited benefit for oCIS. An alternative would be to implement a simpler identity object for oCIS and use that. diff --git a/modules/developer/pages/ocis/adr/0008-configuration.adoc b/modules/developer/pages/ocis/adr/0008-configuration.adoc new file mode 100644 index 00000000..7b432c60 --- /dev/null +++ b/modules/developer/pages/ocis/adr/0008-configuration.adoc @@ -0,0 +1,191 @@ += 8. Configuration +:toc: right +:toclevels: 3 + +* Status: proposed +* Deciders: link:https://github.com/refs[@refs], link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/pmaier1[@pmaier1] +* Date: 2021-05-03 + +== Context and Problem Statement + +As per urfave/cli's doc: + +>The precedence for flag value sources is as follows (highest to lowest): +> +>0. Command line flag value from user +>1. Environment variable (if specified) +>2. Configuration file (if specified) +>3. Default defined on the flag + +An issue arises in point 2, in the sense that configuration file refers to a single file containing the value for the env variable. The CLI framework we use for flag parsing does not support merging config structs with CLI flags. This introduces an inconsistency with the framework: config structs are not supported, and we cannot hook to the lifecycle of the flags parsing to use a file as source and conform to these rules. + +Because we solely rely on link:https://github.com/owncloud/ocis/blob/master/ocis-pkg/config/config.go[structured configuration] we need a way to modify values in this struct using the provided means urfave/cli gives us (flags, env variables, config files and default value), but since we have different modes of operation (supervised Vs. unsupervised) we have to define a clear line. + +=== Decision Drivers +- Improve experience for the end user. +- Improve experience for developers. +- Sane defaults. +- Sane overrides. + +=== Considered Options + +- Extend link:https://github.com/urfave/cli/blob/master/altsrc/flag.go#L12-L17[FlagInputSourceExtension interface] +- Feature request: support for structured configuration (urfave/cli). +- Clearly defined boundaries of what can and cannot be done. +- Expose structured field values as CLI flags +- Drop support for structure configuration +- Adapt the "structured config files have the highest priority" within oCIS + +=== Decision Outcome + +[STILL UNDECIDED] + +==== Positive Consequences + +[TBD, depends on Decision Outcome] + +=== Pros and Cons of the Options + +==== Extend FlagInputSourceExtension interface +- Good, because we could still use Viper to load from config files here and apply values to the flags in the context. +- Bad, because urfave/cli team are link:https://github.com/urfave/cli/issues/1051#issuecomment-606311923[actively working on v3 of altsrc] and we don't want to maintain yet another slice of the codebase. + +notes: source is link:https://github.com/urfave/cli/blob/master/altsrc/flag.go#L12-L17[FlagInputSourceExtension interface] + +==== Feature request: support for structured configuration (urfave/cli). +- Good, because we could remove Viper off the codebase and solely rely on urfave/cli's native code. +- Bad, because there are no plans to support this upstream. + +==== Clearly defined boundaries of what can and cannot be done. + +- Good, because no changes to the codebase required (not drastic changes.) +- Bad, because we're limited by the framework + +==== Expose structured field values as CLI flags + +- Good, because it has been already taken into account on large projects (kubernetes) link:https://docs.google.com/document/d/1Dvct469xfjkgy3tjWMAKvRAJo4CmGH4cgSVGTDpay6A[here.] in point 5. +- Bad, because it requires quite a bit1 of custom logic. +- Bad, because how should these flags be present in the `-h` menu of a subcommand? Probably some code generation needed. + +*[1] this is a big uncertainty. + +==== Drop support for structure configuration + +- Good, because it makes the integration with the cli framework easier to grasp. +- Good, because it is not encouraged by the 12factor app spec. +- Bad, because we already support if and users make active use of it. At least for development. + +==== Adapt the "structured config files have the highest priority" within oCIS + +- Good, because that would mean little structural changes to the codebase since the Viper config parsing logic already uses the `Before` hook to parse prior to the command's action executes. + +=== Notes + +==== Use Cases and Expected Behaviors + +===== Supervised (`ocis server` or `ocis run extension`) + +image::https:/user-images.githubusercontent.com/6905948/116872568-62b1a780-ac16-11eb-9f29-030a651ee39b.png[grafik] + +- Use a global config file (ocis.yaml) to configure an entire set of services: `> ocis --config-file /etc/ocis.yaml service` +- Use a global config file (ocis.yaml) to configure a single extension: `> ocis --config-file /etc/ocis/yaml proxy` +- When running in supervised mode, config files from extensions are NOT evaluated (only when running `ocis server`, runs with `ocis run extension` do parse individual config files) + - i.e: present config files: `ocis.yaml` and `proxy.yaml`; only the contents of `ocis.yaml` are loaded1. +- Flag parsing for subcommands are not allowed in this mode, since the runtime is in control. Configuration has to be done solely using config files. + +*[1] see the development section for more on this topic. + +====== Known Gotchas +- `> ocis --config-file /etc/ocis/ocis.yaml server` does not work. It currently only supports reading global config values from the predefined locations. + +===== Unsupervised (`ocis proxy`) + +image::https:/user-images.githubusercontent.com/6905948/116872534-54fc2200-ac16-11eb-8267-ffe7b03177b3.png[grafik] + +- `ocis.yaml` is parsed first (since `proxy` is a subcommand of `ocis`) +- `proxy.yaml` is parsed if present, overriding values from `ocis.yaml` and any cli flag or env variable present. + +==== Other known use cases + +- Configure via env + some configuration files like WEB_UI_CONFIG or proxy routes +- Configure via flags + some configuration files like WEB_UI_CONFIG or proxy routes +- Configure via global (single file for all extensions) config file + some configuration files like WEB_UI_CONFIG or proxy routes +- configure via per extension config file + some configuration files like WEB_UI_CONFIG or proxy routes + +Each individual use case DOES NOT mix sources (i.e: when using cli flags, do not use environment variables nor cli flags). + +_Limitations on urfave/cli prevent us from providing structured configuration and framework support for cli flags + env variables._ + +==== Use Cases for Development + +==== Config Loading + +Sometimes is desired to decouple the main series of services from an individual instance. We want to use the runtime to startup all services, then do work only on a single service. To achieve that one could use `ocis server && ocis kill proxy && ocis run proxy`. This series of commands will 1. load all config from `ocis.yaml`, 2. kill the supervised proxy service and 3. start the same service with the contents from `proxy.yaml`. + +==== Start an extension multiple times with different configs (in Supervised mode) + +Flag parsing on subcommands in supervised mode is not yet allowed. The runtime will first parse the global `ocis.yaml` (if any) and run with the loaded configuration. This use case should provide support for having 2 different proxy config files and making use of the runtime start 2 proxy services, with different values. + +For this to work, services started via `Service.Start` need to forward any args as flags: + +[source,go] +---- +if err := client.Call("Service.Start", os.Args[2], &reply); err != nil { + log.Fatal(err) +} +---- + +This should provide with enough flexibility for interpreting different config sources as: `> bin/ocis run proxy --config-file /etc/ocis/unexpected/proxy.yaml` + +==== Developing Considered Alternatives Further + +Let's develop further the following concept: Adapt the "structured config files have the highest priority" within oCIS. + +Of course it directly contradicts urfave/cli priorities. When a command finished parsing its cli args and env variables, only after that `Before` is called. This mean by the time we reach a command `Before` hook, flags have already been parsed and its values loaded to their respective destinations within the `Config` struct. + +This should still not prevent a developer from using different config files for a single service. Let's analyze the following use case: + +1. global config file present (ocis.yaml) +2. single proxy.yaml config file +3. another proxy.yaml config file +4. running under supervision mode + +The outcome of the following set of commands should be having all bootstrapped services running + 2 proxies on different addresses: + +[source,console] +---- +> ocis server +> ocis kill proxy +> ocis run proxy --config-file proxy.yaml +> ocis run proxy --config-file proxy2.yaml +---- + +This is a desired use case that is yet not supported due to lacking of flags forwarding. + +==== Follow-up PR's + +- Variadic runtime extensions to run (development mostly) +- Arg forwarding to command (when running in supervised mode, forward any --config-file flag to supervised subcommands) +- Ability to set `OCIS_URL` from a config file (this would require to extend the ocis-pkg/config/config.go file). + +==== The case for `OCIS_URL` + +`OCIS_URL` is a jack-of-all trades configuration. It is meant to ease up providing defaults and ensuring dependant services are well configured. It is an override to the following env vars: + +---- +OCIS_IDM_ADDRESS +PROXY_OIDC_ISSUER +STORAGE_OIDC_ISSUER +STORAGE_FRONTEND_PUBLIC_URL +STORAGE_LDAP_IDP +WEB_UI_CONFIG_SERVER +WEB_OIDC_AUTHORITY +OCIS_PUBLIC_URL +---- + +Because this functionality is only available as an env var, there is no current way to "normalize" its usage with a config file. That is, there is no way to individually set `OCIS_URL` via config file. This is clear technical debt, and should be added functionality. + +==== State of the Art +- link:https://docs.google.com/document/d/1Dvct469xfjkgy3tjWMAKvRAJo4CmGH4cgSVGTDpay6A[Kubernetes proposal on this very same topic] +- link:https://www.pulumi.com/docs/intro/concepts/config/[Configuration \| Pulumi] + - Configuration can be altered via setters through the CLI. diff --git a/modules/developer/pages/ocis/adr/0009-extension-template.adoc b/modules/developer/pages/ocis/adr/0009-extension-template.adoc new file mode 100644 index 00000000..ece09e6d --- /dev/null +++ b/modules/developer/pages/ocis/adr/0009-extension-template.adoc @@ -0,0 +1,66 @@ += 9. Extension Template +:toc: right +:toclevels: 3 + +* Status: proposed +* Deciders: link:https://github.com/c0rby[@c0rby] +* Date: 2021-06-10 + +Technical Story: [description | ticket/issue URL] + +== Context and Problem Statement + +We want to accelerate and simplify extension development by removing the necessity to type or copy the boilerplate code. Can we provide a template or a similar mechanism to aid when developing new extensions? + + +== Decision Drivers + +* The solution should be easily maintainable. + * It should always be up-to-date. +* The solution should be easy to use. + +== Considered Options + +* Use link:https://github.com/tmrts/boilr[boilr] +* Create a template git repository. +* Use link:https://github.com/owncloud/ocis-hello/[ocis-hello] as a "template" + +== Decision Outcome + +Chosen option: "[option 1]", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | … | comes out best (see below)]. + +=== Positive Consequences: + +* [e.g., improvement of quality attribute satisfaction, follow-up decisions required, …] +* … + +=== Negative consequences: + +* [e.g., compromising quality attribute, follow-up decisions required, …] +* … + +== Pros and Cons of the Options + +=== link:https://github.com/tmrts/boilr[boilr] + +We have a boilr template already. link:https://github.com/owncloud/boilr-ocis-extension/[boilr-ocis-extension] +This approach is nice because it provides placeholders which can be filled during the generation of a new extension from the template. It also provides prompts for the placeholder values during generation. + +* Good, because with the placeholders it is hard to miss values which should be changed +* Bad, because maintaining is more complex + +=== Template git repository + +Create a git repository with an extension containing the boilerplate code. + +* Good, because we can use the usual tools for QA and dependency scanning/updating. +* Good, because it doesn't require any additional tool. + +=== link:https://github.com/owncloud/ocis-hello/[ocis-hello] as a "template" + +We have the ocis-hello repository which acts as an example extension containing a grpc and http service and a web UI. It also demonstrates the usage of the settings service. + +* Good, because it contains a bit more code than just the plain boilerplate +* Good, because the integration into oCIS is already tested for the Hello extension (e.g. with Proxy and Settings). This will ensure, that the example extension is up-to-date. +* Bad, because if you don't require all features you have to delete stuff + diff --git a/modules/developer/pages/ocis/adr/0010-policy-enforcement.adoc b/modules/developer/pages/ocis/adr/0010-policy-enforcement.adoc new file mode 100644 index 00000000..3e002721 --- /dev/null +++ b/modules/developer/pages/ocis/adr/0010-policy-enforcement.adoc @@ -0,0 +1,84 @@ += 10. Extension Policies +:toc: right +:toclevels: 3 + +* Status: proposed +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/hodyroff[@hodyroff], link:https://github.com/pmaier1[@pmaier1], link:https://github.com/fschade[@fschade] +* Date: 2021-06-30 + +== Context and Problem Statement + +There should be a way to impose certain limitations in areas of the code that require licensing. This document researches an approach to achieve this goal, while limiting the scope to the enforcement side of it. The architecture for a policy system must be composed of 2 parts: + +1. License creation and validation +2. Enforcement + +It is desirable to keep both systems isolated, since the implementation of the latter has to be done within the constraints of the codebase. The alternative is running an enforcement service and have each and every single request evaluating whether the request is valid or not. + +== Decision Drivers + +- As a team, we want to have the licensing code concentrated in a central module +- We don't want to stop/start the extension whenever a policy is updated (hot reload). It must happen during runtime. + +== Considered Options + +1. Build the evaluation engine in-house. +2. Use third party libraries such as Open Policy Agent (a CNCF approved project written in Go) + +== Decision Outcome + +Chosen option: option 2; Use third party libraries such as Open Policy Agent (a CNCF approved project written in Go) + +=== Positive Consequences + +- OPA is production battle tested. +- Built around performance - policies evaluations are no longer than 1ms per request. +- Middleware friendly: we use gRPC clients all over our ecosystem; wrappers (or middlewares) is a viable way to solve this problem instead of a dedicated service or its own package. +- Community support. +- Kubernetes friendly. +- Supports envoy, kong, terraform, traefik, php, node and many more. + +=== Negative Consequences + +- More vendor code inside the binary (larger attack surface, larger footprint [to be quantified] ) + +== Chosen option approach + +Make use of link:https://www.openpolicyagent.org/docs/latest/external-data/#option-2-overload-input[overloading Open Policy Agent's input] along with an external storage source (instead of an OPA service) in conjunction with go-micro's gRPC client wrappers (a.k.a. middlewares) to leverage policy rules evaluation. + +=== Terminology + +New terms are defined to refer to new mental models: + +- Policy: self-imposed limitation of a piece of software. i.e: "after 20 users limit the use of thumbnails". +- Checkers: in the context of a middleware, a checker is in charge of defining logical conditions that prevent requests (users) from doing an action. +- Policy file: a link:https://www.openpolicyagent.org/docs/latest/policy-language/[rego file]. +- Policy evaluation: the act of piecing together input (from a request), data (from an external storage) and policies in order to make a decision. + +==== Temporary new Interfaces part of the PoC + +- IStorage: provides means of extracting data from an external source (in case of the POC an etcd storage cluster). + +=== External data storages + +However, for this to be usable it needs state. The Rego engine works with input and data, where data is essentially a database the input is tried against, in order to expand this poc to include functionality such as counters (i.e: give access to the thumbnails only to 50 users) we need an external storage, and consequentially, Rego needs to have an option to load data from an external storage. There is an entire chapter in the documentation regarding external data: https://www.openpolicyagent.org/docs/latest/external-data/. The most "natural" option (option 5) states: + +[quote] +____ +OPA includes functionality for reaching out to external servers during evaluation. This functionality handles those cases where there is too much data to synchronize into OPA, JWTs are ineffective, or policy requires information that must be as up-to-date as possible. +____ + + +This is a natural option because it requires service-to-service communication, and by definition using microservices it should come "natural to us". Another approach is using JWT (which we already use) to encode the necessary data into the JWT and handing it over to rego as "data". The issue with this approach is that depending on the features of the licenses the JWT might grow and be filled with noise and redundancy (this is, unless a new token is issued for licensing purposes). + +=== Future ideas + +link:https://github.com/owncloud/ocis/pull/2236[This proof of concept] is very rigid in the sense that the `IStorage` interface only has one implementation that ties it to etcd, meaning running an oCIS cluster without an etcd service will result in a crash. This is by far ideal and less coupled implementations should be done. There is the case of using the storage metadata as a source to store data necessary to the policies, or even using the go-micro store as a kv store to achieve the exact same, since it already runs as its own service. The implementation of this is trivial and left out of the POC since it requires more time than the allotted for this task. + +==== Message Broker + +This problem perfectly encompasses the use of a message broker, where services such as OCS will emit messages to a bus and only listeners react to them. In this case the following applies: + +image::https:/i.imgur.com/sa1pANQ.jpg[message broker] + +The necessary interfaces are provided to us by go-micro, only implementations are to be done. diff --git a/modules/developer/pages/ocis/adr/0011-global-url-format.adoc b/modules/developer/pages/ocis/adr/0011-global-url-format.adoc new file mode 100644 index 00000000..582f89d1 --- /dev/null +++ b/modules/developer/pages/ocis/adr/0011-global-url-format.adoc @@ -0,0 +1,317 @@ += 11. WebUI URL format +:toc: right +:toclevels: 3 + +* Status: accepted +* Deciders: link:https://github.com/refs[@refs], link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/hodyroff[@hodyroff], link:https://github.com/pmaier1[@pmaier1], link:https://github.com/fschade[@fschade], link:https://github.com/tbsbdr[@tbsbdr], link:https://github.com/kulmann[@kulmann] +* Date: 2021-07-07 + +== Context and Problem Statement + +When speaking about URLs we have to make a difference between browser URLs and API URLs. Only browser URLs are visible to end users and will be bookmarked. The currently existing and bookmarked ownCloud 10 URLs look something like this: + +---- +GET https://demo.owncloud.com/apps/files/?dir=/path/to/resource&fileid=5472225 +303 Location: https://demo.owncloud.com/apps/files/?dir=/path/to/resource +---- + +When the URL contains a `fileid` parameter the server will look up the corresponding `dir`, overwriting whatever was set before the redirect. The `fileid` always takes precedence and the server is responsible for the lookup. + +---- +GET https://demo.owncloud.com/apps/files/?dir=/path/to/resource +---- + +The `dir` parameter is then used to make a WebDAV request against the `/dav/files` endpoint of the currently logged-in user: + +---- +PROPFIND https://demo.owncloud.com/remote.php/dav/files/demo/path/to/resource +---- + +The resulting PROPFIND response is used to render the file listing. All good so far. + +For the new ocis web UI we want to clean up the user visible Browser URLs. They currently look like this: + +---- +https://demo.owncloud.com/#/files/list/all/path/to/resource +---- + +Currently, there is no `fileid` like parameter in the browser URL, making bookmarks of it fragile (they break when a bookmarked folder is renamed). + +The oCIS web UI just takes the path and uses the `/webdav` endpoint of the currently logged-in user: + +---- +PROPFIND https://demo.owncloud.com/remote.php/webdav/path/to/resource +---- + + +With the new ownCloud web client (owncloud/web) + + needs to interpret them to make API calls. With this in mind, this is the current mapping on ownCloud Web with OC10 and OCIS backend: + +| | Browser URL | API URL | +|------|----------------------------------------------------------------|----------------------------------------------------| +| OC10 + classic WebUI | `https://demo.owncloud.com/apps/files/?dir=/path/to/resource&fileid=5472225` | `https://demo.owncloud.com/remote.php/dav/files/demo/path/to/resource` | +| OC10 + OCIS WebUI| `https://web.owncloud.com/index.html#/files/list/all/path%2Fto%2Fresource` | `https://demo.owncloud.com/remote.php/webdav/path/to/resource` | +| OCIS | `https://demo.owncloud.com/#/files/list/all/path/to/resource` | `https://demo.owncloud.com/remote.php/webdav/path/to/resource` | + + +On an OC10 backend the `fileid` query parameter takes precedence over the `dir`. In fact if `dir` is invalid but `fileid` isn't, the resolution will succeed, as opposed to if the `fileid` is wrong (doesn't exist) and `dir` correct, resolution will fail altogether with a 404. + +This ADR is limited to the scope of "how will a web client deal with the browser URL?". The API URLs will change with the spaces concept to `https://demo.owncloud.com/dav/spaces//relative/path/to/resource`. The Web UI can look up a space id and the mount path using the `/graph/v1.0/drives` API: +1. TODO for a given resource id as part of the URL the `https://demo.owncloud.com/v1.0/drive/items/123456A14B0A7750!359?$select=parentReference` can be used to retrieve the drive/space: +---- +{ + "parentReference": { + "driveId": "123456a14b0a7750", + "driveType": "personal", + "id": "123456A14B0A7750!357", + "path": "/drive/root:" + } +} +---- +2. TODO to fetch the list of all spaces with their mount points we need an API endpoint that allows clients (not only the web ui) to 'sync' the list of storages a user has access to from the storage registry on the server side. This allows clients to directly talk to a storage provider on another instance, allowing true storage federation. The MS graph api has no notion of mount points, so we will need to add a `mountpath` _(or `mountpoint`? or `alias`?)_ to our link:https://github.com/owncloud/open-graph-api/blob/dc6da5359eee0345429080b5b59762fd8c57b121/api/openapi-spec/v0.0.yaml#L351-L384[`drive` resource properties in the libreGraph spec]. Tracked in https://github.com/owncloud/open-graph-api/issues/6 + + +// SHORTCODE: {{< hint >}} +@jfd: The graph api returns a `path` in the `parentReference`, which is part of the `root` in a `drive` resource. But it contains a value in the namespace of the `graph` endpoint, e.g.: `/drive/root:/Bilder` for the `/Bilder` folder in the root of the currently logged-in users personal drive/space. Which is again relative to the drive. To give the clients a way to determine the mount point we need to add a new `mountpath/point/alias` property. +// SHORTCODE: {{< /hint >}} + +== Decision Drivers + +* To reveal relevant context to the user URLs should either carry a path component or a meaningful alias +* To prevent bookmarks from breaking URLs should have an id component that can be used by the system to lookup the resource + +== Considered Options + +* Existing ownCloud 10 URLs +* ID based URLs +* Path based URLs +* Space based URLs +* Mixed Global URLs +* Configurable path component in URLs + +== Decision Outcome + +Chosen option: "Mixed global URLs", because it meets the requirement to contain a path and a stable identifier. + +=== Positive Consequences + +* The path makes it "human readable" +* The URL can be bookmarked +* The bookmarked URLs remain stable even if the path changes +* All URLs can be shortened to hide any metadata like path, resource name and query parameters + +=== Negative Consequences + +* the web UI needs to look up the space alias in a registry to build an API request for the `/dav/space` endpoint + +== Pros and Cons of the Options + +=== Existing OwnCloud 10 URLs + +The existing ownCloud 10 URLs look like this + +| URL | comment | +|-----|---------| +| `https:///apps/files/?dir=&fileid=` | pattern | +| `https://demo.owncloud.com/apps/files/?dir=/&fileid=18` | root of the currently logged in user | +| `https://demo.owncloud.com/index.php/apps/files/?dir=/path/to/resource&fileid=192` | sub folder `/path/to/resource` | + +It contains a path and a `fileid` (which takes precedence). + +* Good, because the `fileid` prevents bookmarks from breaking +* Good, because the `dir` reveals context in the form of a path +* Bad, because the web UI needs to look up the space alias in a registry to build an API request for the `/dav/space` endpoint +* Bad, because URLs still contain a long prefix `(/index.php)/apps/files` +* Bad, because the `fileid` needs to be accompanied by a `storageid` to allow efficient routing in ocis +* Bad, because if not configured properly an additional `/index.php` prefixes the route +* Bad, because power users cannot navigate by updating only the path in the URL, as the `fileid` takes precedence. They have to delete the `fileid` to navigate + +=== ID based URLs + +MS OneDrive has URLs like this: + +| URL | comment | +|-----|---------| +| `https:///?id=(&cid=)` | pattern, the `cid` is optional but added automatically | +| `https://onedrive.live.com/?id=root&cid=A12345A14B0A7750` | root of a personal drive | +| `https://onedrive.live.com/?id=A12345A14B0A7750%21359&cid=A12345A14B0A7750` | sub folder in a personal drive | + +It contains only IDs but no folder names. The `fileid` is a URL encoded `!`. Very similar to the CS3 `resourceid` which consists of `storageid` and `nodeid`. + +* Good, because bookmarks cannot break +* Good, because URLs do not disclose unshared path segments +* Bad, because the web UI needs to look up the space id in a registry to build an API request for the `/dav/space` endpoint +* Bad, because URLs reveal no context to users + +=== Path based URLs + +There is a customized ownCloud instance that uses path only based URLs: + +| URL | comment | +|-----|---------| +| `https:///apps/files/?dir=/&` | root of the currently logged in user | +| `https://demo.owncloud.com/apps/files/?dir=/&` | root of the currently logged in user | +| `https://demo.owncloud.com/apps/files/?dir=/path/to/resource&` | sub folder `/path/to/resource` | + +* Good, because the URLs reveal the full path context to users +* Good, because power users can navigate by updating the path in the url +* Bad, because the web UI needs to look up the space id in a registry to build an API request for the `/dav/space` endpoint +* Bad, because the bookmarks break when someone renames a folder in the path +* Bad, because there is no id that can be used as a fallback lookup mechanism +* Bad, because URLs might leak too much context (parent folders of shared files) + +=== Space based URLs + +| URL | comment | +|-----|---------| +| `https:///#/s/(/)(?id=)` | the pattern, relative `path` and `resource_id` are optional | +| `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607` | root of a storage space, might be the currently logged in users home | +| `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607/relative/path/to/resource` | sub folder `/relative/path/to/resource` in the storage with id `b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607`, works _*only*_ if path still exists | +| `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607/relative/path/to/resource?id=ba4c1820-df12-11eb-8dcd-ff21f12c1264:beb78dd6-df12-11eb-a05c-a395505126f6` | sub folder `/relative/path/to/resource` in the storage with id `b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607`, lookup can fall back to the `id` | + +// SHORTCODE: {{< hint >}} +* `/#` is used by the current vue router. +* `/s` denotes that this is a space url. +* `` and `` both consist of `:`, but the `space_id` can be replaced with a shorter id or an alias. See further down below. +* `` takes precedence over the ``, both are optional +// SHORTCODE: {{< /hint >}} + +* Good, because the web UI does not need to look up the space id in a registry to build an API request for the `/dav/space` endpoint +* Good, because the URLs reveal a relevant path context to users +* Good, because everything after the `#` is not sent to the server, building the webdav request to list the folder is offloaded to the clients +* Good, because power users can navigate by updating the path in the url +* Bad, because the current ids are uuid based, leading to very long URLs where the path component nearly vanishes between two very long strings +* Bad, because the `#` in the URL is just a technical requirement +* Bad, because ocis web requires a `/#/files/s` at the root of the route to distinguish the files app from other apps +* Bad, while navigating using the WebUI, the URL has to be updated whenever we change spaces. +* Bad, because the technical `` is meaningless to end users + +With the above explained, let's see some use cases: + +==== Example 1: UserA shares something from her Home folder with UserB + +- open the browser and go to `demo.owncloud.com` +- the browser's url changes to: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607`. You're now in YOUR home folder / personal space. +- you create a new folder `/relative/path/to/resource` and navigate into `/relative/path/to` + - the URL now changes to: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607/relative/path/to` +- You share `resource` with some else +- You navigate into `/relative/path/to/resource` + - now the URL would look like: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:3a9305da-df17-11eb-ab99-abe09d93e08a` + +As you can see, even if you're the owner of `/relative/path/to/resource` and navigate into it, the URL changes due to a new space being entered. This ensures that while working in your home folder, copying URLs and giving them to the person you share the resource with, the receiver can still navigate within the new space. + +In short terms, while navigating using the WebUI, the URL has to constantly change whenever we change spaces to reflect the most explicit one. + +==== Example 2: UserA shares something from a Workspace + +Assuming we only have one storage provider; a consequence of this, all storage spaces will start with the same storage_id. + +- open the browser and go to `demo.owncloud.com` +- the browser's url changes to: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607`. You're now in YOUR home folder / personal space. +- you have access to a workspace called `foo` (created by an admin) +- navigate into workspace `foo` + - the URL now changes to: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:d342f9ce-df18-11eb-b319-1b6d9df4bc74`. You are now at the root of the workspace `foo`. + - because we only have one storage provider, the `space_id` section of the URL only updates the `node_id` part of it. + - had we had more than one storage provider, the `space_id` would depend on which storage provider contains the storage space. +- you create a folder `/relative/path/to/resource` +- you navigate into `/relative/path/to/resource` + - now the URL would look like: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:d342f9ce-df18-11eb-b319-1b6d9df4bc74/relative/path/to/resource` + - or a more robust url: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:d342f9ce-df18-11eb-b319-1b6d9df4bc74/relative/path/to/resource?id=b78c2044-5b51-446f-82f6-907a664d089c:04f1991c-df19-11eb-9cc7-3b09f04f9ca3` + +==== Spaces Registry + +A big drawback against this idea is that the length of the URL is increased by a lot, rendering them almost unreadable. Introducing a Spaces Registry (SR) would shorten them. Let's see how. + +A URL without a SR would look like: `https://ocis.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:d342f9ce-df18-11eb-b319-1b6d9df4bc74/TEST?id=b78c2044-5b51-446f-82f6-907a664d089c:04f1991c-df19-11eb-9cc7-3b09f04f9ca3` +The same URL with a SR `https://ocis.com/#/s/workspaceFoo/TEST?id=b78c2044-5b51-446f-82f6-907a664d089c:04f1991c-df19-11eb-9cc7-3b09f04f9ca3` + +Space Registry resolution can happen at the client side (i.e: the client keeps a list of space name -> space id [where space id = storageid + nodeid]; the client queries a SR) or server side. Server side is more resilient due to clients can have limited networking; for instance if they are running on a tight intranet. + +=== Mixed Global URLs + +While ID based space URLs can be made more readable by shortening the IDs they only start to reveal context when an alias is used instead of the space id. These aliases however have to be unique identifiers. These aliases should live in namespaces like `/workspaces/marketing` and `/personal/marketing` to make phishing attacks harder (in this case a user that registered with the username `marketing`). But namespaced aliases is semantically equivalent to ... a path hierarchy. + +When every space has a namespaced alias and a relative path we can build a global namespace: + +| URL | comment | +|-----|---------| +| `https:///files?id=` | the pattern, `/files` might become optional | +| `https://demo.owncloud.com/files/personal/einstein/?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607` | root of user `einstein` | +| `https://demo.owncloud.com/files/personal/einstein/relative/path/to/resource?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | sub folder `/relative/path/to/resource` | +| `https://demo.owncloud.com/files/shares/einstein/somesharename?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | shared URL for `/relative/path/to/resource` | +| `https://demo.owncloud.com/files/personal/einstein/marie is stupid/and richard as well/resource?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | sub folder `marie is stupid/and richard as well/resource` ... something einstein might not want to reveal | +| `https://demo.owncloud.com/files/shares/einstein/resource (2)?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | named link URL for `/marie is stupid/and richard as well/resource`, does not disclose the actual hierarchy, has an appended counter to avoid a collision | +| `https://demo.owncloud.com/files/shares/einstein/mybestfriends?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | named link URL for `/marie is stupid/and richard as well/resource`, does not disclose the actual hierarchy, has a custom alias for the share | +| `https://demo.owncloud.com/files/public/kcZVYaXr7oZ66bg/relative/path/to/resource` | sub folder `/relative/path/to/resource` in public link with token `kcZVYaXr7oZ66bg` | +| `https://demo.owncloud.com/files/public/kcZVYaXr7oZ66bg/relative/path/to/resource` | sub folder `/relative/path/to/resource` in public link with token `kcZVYaXr7oZ66bg` | +| `https://demo.owncloud.com/s/kcZVYaXr7oZ66bg/` | shortened link to a resource. This is needed to be able to copy a link to a resource without leaking any metadata. | + + +`` is the global path in the CS3 api. The CS3 Storage Registry is responsible by managing the mount points. + +In order to be able to copy and paste URLs all resources must be uniquely identifiable: + +* Instead of `/home` the URL always has to reflect the user: `/personal/einstein` +* Workspaces can use `/workspaces/` or `/workspaces///` where the hierarchy is given by the organization +* Experiments can use `/experiments/` +* Research institutes could set up `/papers//` +* Trash could be accessed by prefixing the namespace alias with `/trash`? or using `/trash/` +* Instead of a namespaced alias a storage space id could be used with a generic `/space/` namespace +* An url shortener can create urls like `/s/` which could be used as a stable link to a resource. +* Links for anonymous users will resolve to `/public/` + +The alias namespace hierarchy and depth can be pre-determined by the admin. Even if aliases change the `id` parameter prevents bookmarks from breaking. A user can decide to build a different hierarchy by using his own registry. + +What about shares? Similar to `/home` it must reflect the user: `/shares/einstein` would list all shares _by_ einstein for the currently logged-in user. The ui needs to apply the same URL rewriting as for space based URLs: when navigating into a share the URL has to switch from `/personal/einstein/relative/path/to/shared/resource` to `/shares/einstein/`. When more than one `resource` was shared a name collision would occur. To prevent this we can use ids `/shares/einstein/id/`. As a default we could take the alias at creation time from the filename. That way two shares to a resource with the same name, e.g.: `/personal/einstein/project AAA/foo` and `/personal/einstein/project BBB/foo` would lead to `/shares/einstein/foo` (a CS3 internal reference to `/personal/einstein/project AAA/foo`) and `/shares/einstein/foo (2)` (a CS3 internal reference to `/personal/einstein/project BBB/foo`). `foo (2)` would keep its name even when `foo` is deleted or renamed. Well an id as the alias might be better then, because users might rename these aliases, which would break URLs if they have been bookmarked. In any case this would make end user more aware of what they share AND it would allow them to choose an arbitrary context for the links they want to send out: personal internal share URLs. + +With these different namespaces the `/files` part in the URL becomes obsolete, because the files application can be registered for multiple namespaces: `/personal`, `/workspaces`, `/shares`, `/trash` ... + +* Good, because it contains a global path +* Good, because spaces with namespaced aliases can be bookmarked and copied into mails or chat without disclosing unshared path segments, as the space is supposed to be shared +* Good, because the UI can detect broken paths and notify the user to update his bookmark if the resource could be found by `id` +* Good, because the `/files` part might only be required for `id` only based lookup to let the web ui know which app is responsible for the route +* Good, because it turns shares into deliberately named spaces in `/shares//` +* Good, because all urls can be shortened to hide any metadata like path, resource name and query parameters +* Bad, because the web UI needs to look up the space alias in a registry to build an API request for the `/dav/space` endpoint + + +=== Configurable path component in URLs + +Not every deployment may have the requirement to have the path in the URL. We could use id only based URLs, similar to onedrive and make showing paths configurable. + + +| URL | comment | +|-----|---------| +| `https:///files?id=` | default id based navigation | +| `https:///files?id=` | optional path based navigation with fallback to id | + +In contrast to ownCloud 10 path takes precedence and the user is warned when the fileid in his bookmark no longer matches the id on the server: sth. like "The path of the resource has changed, please verify and update your bookmark!" + +When a file is selected the filename also becomes part of the URL so individual files can be bookmarked. + +If navigation is id based we need to look up the path for the id so we can make a webdav request, or we need to implement the graph drives and driveItem resources. + +The URL `https:///files?id=̀` is sent to the server. It has to look up the correct path and redirect the request, including the path. But that would make all bookmarks contain tha path again, even if paths were configured to not be part of the URL. + +The `/meta/` webdav endpoint can be used to look up the path with property `meta-path-for-user`. + +For now, we would use path based navigation with URLs like this: + +---- +https:///files?id= +---- + +This means that only the _resource path_ is part of the URL path. Any other parameter, e.g. file `id`, `page` or sort order must be given as URL parameters. + +- [ ] To make lookup by id possible we need to implement the `/meta/` endpoint so the sdk can use it to look up the path. We should not implement a redirect on the ocis server side because the same redirect logic would need to be added to oc10. Having it in ocis web is the right place. + +- [ ] The old sharing links and oc10 urls still need to be redirected by ocis/reva as in oc10. + +Public links would have the same format: `https:///files?id=` The web UI has to detect if the user is logged in or not and adjust the ui accordingly. + +[WARNING] +==== +Since there is no difference between public and private files a logged-in user cannot see the public version of a link unless he logs out. +==== + diff --git a/modules/developer/pages/ocis/adr/0012-tracing.adoc b/modules/developer/pages/ocis/adr/0012-tracing.adoc new file mode 100644 index 00000000..77ace785 --- /dev/null +++ b/modules/developer/pages/ocis/adr/0012-tracing.adoc @@ -0,0 +1,44 @@ += 12. Tracing +:toc: right +:toclevels: 3 + +* Status: proposed +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/mstingl[@mstingl] link:https://github.com/pmaier1[@pmaier1], link:https://github.com/fschade[@fschade] +* Date: 2021-08-17 + +== Context and Problem Statement + +At the time of this writing we are in a situation where our logs have too much verbosity, rendering impossible or rather difficult to debug an instance. For this reason we are giving some care to our traces by updating dependencies from OpenCensus to OpenTelemetry. + +== Decision Drivers + +- We don't want to rely only on logs to debug an instance. +- Logs are too verbose. +- Since we have micro-services, we want to holistically understand a request. + +== Considered Options + +- Trim down logs +- Use OpenCensus +- Migrate to OpenTelemetry + +== Decision Outcome + +Chosen option: option 3; Migrate to OpenTelemetry. OpenCensus is deprecated, and OpenTelemetry is the merger from OpenCensus and OpenTelemetry and the most recent up-to-date spec. + +=== Positive Consequences + +- Fix the current state of the traces on Reva. +- Add more contextual information on a span for a given request. +- Per-request filtering with the `X-Request-Id` header. +- Group the supported tracing backends to support Jaeger only for simplicity. + +== Chosen option approach + +- A trace is a tree, and the proxy will create the root trace and propagate it downstream. +- The Root trace will log the request headers. +- The unit that ultimately does the work will log the result of the operation if success. +- The unit that ultimately does the work will change the state of the span to error if any occurred. + + +With these premises, this is by no means a fixed document and the more we learn about the usage of an instance the more context we can add to the traces. diff --git a/modules/developer/pages/ocis/adr/0013-locking.adoc b/modules/developer/pages/ocis/adr/0013-locking.adoc new file mode 100644 index 00000000..3afc8dbf --- /dev/null +++ b/modules/developer/pages/ocis/adr/0013-locking.adoc @@ -0,0 +1,91 @@ += 13. Locking +:toc: right +:toclevels: 3 + +- Status: accepted +- Deciders: link:https://github.com/hodyroff[@hodyroff], link:https://github.com/pmaier1[@pmaier1], link:https://github.com/jojowein[@jojowein], link:https://github.com/dragotin[@dragotin], link:https://github.com/micbar[@micbar], link:https://github.com/tbsbdr[@tbsbdr], link:https://github.com/wkloucek[@wkloucek] +- Date: 2021-11-03 + +== Context and Problem Statement + +At the time of this writing no locking mechanisms exists in oCIS / REVA for both directories and files. The CS3org WOPI server implements a file based locking in order to lock files. This ADR discusses if this approach is ok for the general availability of oCIS or if changes are needed. + +== Decision Drivers + +- Is the current situation acceptable for the GA +- Is locking needed or can we have oCIS / REVA without locking + +== Considered Options + +1. File based locking +2. No locking +3. CS3 API locking + +== Decision Outcome + +For the GA we chose option 2. Therefore we need to remove or disable the file based locking functionality of the CS3org WOPI server. The decision was taken because the current file based locking does not work on file-only shares. The current locking also does not guarantee exclusive access to a file since other parts of oCIS like the WebDAV API or other REVA services don't respect the locks. + +After the GA we need to implement option 3. + +== Pros and Cons of the Options + +=== File based locking + +The CS3org WOPI server creates a `.sys.wopilock..` and `.~lock.#` file when opening a file in write mode + +_File based locking is good_, because: + +- it is already implemented in the current CS3org WOPI server + +_File based locking is bad_, because: + +- lock files should be checked by all parties manipulating files (e.g. the WebDAV api) +- lock files can be deleted by everyone +- you can not lock files in a file-only share (you need a folder share to create a lock file besides the original file) + +If we have file based locks, we can also sync them with e.g. the Desktop Client. + +_Syncing lock files is good_: because + +- native office applications can notice lock files by the WOPI server and vice versa (LibreOffice also creates `.lock.#` files) + +_Syncing lock files is bad_, because: + +- if lockfile is not deleted, no one can edit the file +- creating lock files in a folder shared with 2000000 users creates a lot of noise and pressure on the server (etag propagation, therefore oC Desktop sync client has an ignore rule for `.~lock.*` files) + +=== No locking + +We remove or disable the file based locking of the CS3org WOPI server. + +_No locking is good_, because: + +- you don't need to release locks +- overwriting a file just creates a new version of it + +_No locking is bad_, because: + +- merging changes from different versions is a pain, since there is no way to calculate differences for most of the files (e.g. docx or xlsx files) +- no locking breaks the WOPI specs, as the CS3 WOPI server won't be capable to honor the WOPI Lock related operations + +=== CS3 API locking + +- Add CS3 API for resource (files, directories) locking, unlocking and checking locks + - locking always with timeout + - lock creation is a "create-if-not-exists" operation + - locks need to have arbitrary metadata (e.g. the CS3 WOPI server is stateless by storing information on / in the locks) +- Implement WebDAV locking using the CS3 API +- Implement Locking in storage drivers +- Change CS3 WOPI server to use CS3 API locking mechanism +- Optional: manual lock / unlock in ownCloud Web (who is allowed to unlock locks of another user?) + +_CS3 API locking is good_, because: + +- you can lock files on the actual storage (if the storage supports that -> storage driver dependent) +- you can lock files in ownCloud 10 when using the ownCloudSQL storage driver in the migration deployment (but oC10 Collabora / OnlyOffice also need to implement locking, to fully leverage that) +- clients can get the lock information via the api without ignoring / hiding lock file changes +- clients can use the lock information to lock the file in their context (e.g. via some file explorer integration) + +_CS3 API locking is bad_, because: + +- it needs to be defined and implemented, currently not planned for the GA diff --git a/modules/developer/pages/ocis/adr/0014-microservices-runtime.adoc b/modules/developer/pages/ocis/adr/0014-microservices-runtime.adoc new file mode 100644 index 00000000..e02dee2e --- /dev/null +++ b/modules/developer/pages/ocis/adr/0014-microservices-runtime.adoc @@ -0,0 +1,58 @@ += 14. Microservices Runtime +:toc: right +:toclevels: 3 + +* Status: proposed +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/mstingl[@mstingl] link:https://github.com/pmaier1[@pmaier1], link:https://github.com/fschade[@fschade] +* Date: 2022-01-21 + +== Context and Problem Statement + +In an environment where shipping a single binary makes it easier for the end user to use oCIS, embedding a whole family of microservices within a package and running it leveraging the use of the Go language has plenty of value. In such environment, a runtime is necessary to orchestrate the services that run within it. Other solutions are hot right now, such as Kubernetes, but for a single deployment this entails orbital measures. + +== Decision Drivers + +- Start oCIS microservices with a single command (`ocis server`). +- Clear separation of concerns between services. +- Control the lifecycle of the running services. +- Services can be distributed across multiple machines and still be controllable somehow. + +== Considered Options + +1.The use of frameworks such as: + - asim/go-micro + - go-kit/kit +2. Build and synchronize all services in-house. +3. A hybrid solution between framework and in-house. + +== Options + +=== go-kit/kit + +Pros +- Large community behind +- The creator is a maintainer of Go, so the code quality is quite high. + +Cons +- Too verbose. Ultimately too slow to make progress. +- Implementing a service would require defining interfaces and a lot of boilerplate. + +=== asim/go-micro + +Pros +- Implementation based in swappable interfaces. +- Multiple implementations, either in-memory or through external services +- Production ready +- Good compromise between high and low level code. + +== Decision Outcome + +Number 3: A hybrid solution between framework and in-house. + +=== Design + +image::ocis/runtime.drawio.svg[] + +First of, every ocis service IS a go-micro service, and because go-micro makes use of urfave/cli, a service can be conveniently wrapped inside a subcommand. Writing a supervisor is then a choice. We do use a supervisor to ensure long-running processes and embrace the "let it crash" mentality. The piece we use for this end is called link:https://github.com/thejerf/suture[Suture]. + +The code regarding the runtime can be found pretty isolated link:https://github.com/owncloud/ocis/blob/d6adb7bee83b58aa3524951ed55872a5f3105568/ocis/pkg/runtime/service/service.go[here]. The runtime itself runs as a service. This is done so messages can be sent to it using the oCIS single binary to control the lifecycle of its services. diff --git a/modules/developer/pages/ocis/adr/0015-events.adoc b/modules/developer/pages/ocis/adr/0015-events.adoc new file mode 100644 index 00000000..1c7673ad --- /dev/null +++ b/modules/developer/pages/ocis/adr/0015-events.adoc @@ -0,0 +1,95 @@ += 15. oCIS Event System +:toc: right +:toclevels: 3 + +* Status: proposed +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/c0rby[@c0rby], link:https://github.com/wkloucek[@wkloucek] +* Date: 2022-01-21 + +== Context and Problem Statement + +=== Overview + +To be able to implement simple, flexible and independent inter service communication there is the idea to implement an event system in oCIS. A service can send out events which are received by one or more other services. The receiving service can cause different kinds of actions based on the event by utilizing the information that the event carries. + +=== Example: Email Notification + +A simple example is the notification feature for oCIS: Users should receive an email when another user shares a file with them. The information, that the file was shared should go out as an event from a storage provider or share manager, carrying the information which file was shared to which receiver. A potential notification service that sends out the email listens to these kinds of events and sends the email out once on every received event of that specific type. + +== Decision Drivers + +* Events are supposed to decouple services and raise flexibility, also considering extensions that are not directly controlled by the ownCloud project. +* Events should bring flexibility in the implementation of sending and receiving services. +* Events should not obsolete other mechanisms to communicate, i.e. grpc calls. +* Sending an event has to be as little resource consuming for the sender as possible. +* Events are never user visible. + +== Considered Options + +1. Lightweight Events with Event Queue and "At most once" QoS +2. As 1., but with "At least once" QoS + +== Options + +=== 1. Lightweight Events with Event Queue and "At most once" QoS + +Reva will get a messaging service that is available to all services within oCIS and Reva. It is considered as one of the mandatory services of the oCIS system. If the messaging backend is not running, neither Reva nor oCIS can be considered healthy and should shut down. + +All oCIS- and Reva-services can connect to the messaging bus and send so-called events. The sender gets an immediate return if handing the event to the message bus was successful or not. + +The sender can not make any assumptions when the message is delivered to any receiving service. Depending on the QoS model (as proposed as alternatives in this ADR) it might even be not guaranteed that the event is delivered at all. Also, the sender can not know if zero, one or many services are listening to that event. + +==== Event Data + +Events are identified by their namespace and their respective name. The namespace is delimited by dots and starts with either "reva" or "ocis" or a future extension name. It is followed by the name of the sending service and an unique name of the event. + +Example: `ocis.ocdav.delete` - an event with that name sent out if an WebDAV DELETE request arrived in the oCDav service. + +An event can carry a payload which is encoded as json object. (See for example link:https://docs.nats.io/using-nats/developer/sending/structure[NATS] ). There are no pre-defined members in that object, it is fully up to the sender which data will be included in the payload. Receivers must be robust to deal with changes. + +==== Quality of Service + +Events are sent with "At most once" quality of service. That means, if a receiver is not present at the moment of publishing it might not receive the event. That requires that the sender and the receiver must have functionality to back up the situation that events were missed. That adds more state to the services because they always need to behave like a link:https://en.wikipedia.org/wiki/Finite-state_machine[FISM]. Given that the event queue can be considered the backbone of the system, it is unlikely that it is not running. + +==== Transactions + +The described way of inter service communication with events is not transactional. It is not supposed to be, but only provides a lightweight, loosely coupled way to "inform". + +If transactions are required, proper synchronous GRPC API calls should be used. Another way would be to build asynchronous flows with request- and reply events as in link:https://microservices.io/patterns/data/saga.html[saga pattern]. That is only recommended for special cases. + +==== Pros + +* Simple setup +* Flexible way of connecting services +* Stateless event queue +* "State of the art" pattern in microservices architectures + +==== Cons + +* Over engineering: Can we do without an extra message queue component? +* Messages might get lost, so that eventual consistency is endangered +* A service needs to hold more state to ensure consistency +* Message queue needs to be implemented in Reva + +=== 2. Lightweight Events with Event Queue and "At-least once" QoS + +Exactly as described above, but with a higher service level quality. + +==== Quality of Service + +Events are sent with "At least once" quality of service. That means the events will remain in the queue until they are received by all receivers. This puts more responsibility on the event bus and adds state to the events. Given that the event queue can be considered the backbone of the system, it is required to be running. + +==== Pros + +* Better service level: Messages do not get lost +* Simplifies the design of the microservices because the events are "fire-and-forget" +* Events would be idempotent. If a service goes down the events will stay in the queue until they are consumed + +==== Cons + +* Stateful event system with higher cost in terms of compute and storage +* The queue could become a bottleneck and needs to be scaled + +== Decision Outcome + +=== Design diff --git a/modules/developer/pages/ocis/adr/0016-files-metadata.adoc b/modules/developer/pages/ocis/adr/0016-files-metadata.adoc new file mode 100644 index 00000000..e2dbbebc --- /dev/null +++ b/modules/developer/pages/ocis/adr/0016-files-metadata.adoc @@ -0,0 +1,89 @@ += 16. Storage for Files Metadata +:toc: right +:toclevels: 3 + +* Status: superseded by xref:0024-msgpack-metadata.adoc[ADR-0024] +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/dragotin[@dragotin], link:https://github.com/micbar[@micbar], link:https://github.com/c0rby[@c0rby] +* Date: 2022-02-04 + +== Context and Problem Statement + +In addition to the file content we need to store metadata which is attached to a file. Metadata describes additional properties of a file. These properties need to be stored as close as possible to the file content to avoid inconsistencies. Metadata are key to workflows and search. We consider them as an additional value which enhances the file content. + +== Decision Drivers + +* Metadata will become more important in the future +* Metadata are key to automated data processing +* Metadata storage should be as close as possible to the file content +* Metadata should be always in sync with the file content + +== Considered Options + +* Database +* Extended file attributes +* Metadata file next to the file content +* Linked metadata in separate file + +== Decision Outcome + +Chosen option: "Extended File Attributes", because we guarantee the consistency of data and have arbitrary simple storage mechanism. + +=== Positive Consequences + +* Metadata is always attached to the file itself +* We can store arbitrary key/values +* No external dependencies are needed + +=== Negative consequences + +* The storage inside extended file attributes has limits +* Changes to extended attributes are not atomic and need file locks + +== Pros and Cons of the Options + +=== Database or Key-Value Store + +Use a Database or an external key/value store to persist metadata. + +* Good, because it scales well +* Good, because databases provide efficient lookup mechanisms +* Bad, because the file content and the metadata could run out of sync +* Bad, because a storage backup doesn't cover the file metadata + +=== Extended File Attributes + +Extended File Attributes allow storing arbitrary properties. There are 4 namespaces `user`, `system`, `trusted` and `security`. We can safely use the `user` namespace. An example attribute name would be `user.ocis.owner.id`. The linux kernel has length limits on attribute names and values. + +From Wikipedia on link:https://en.wikipedia.org/wiki/Extended_file_attributes#Linux[Extended file attributes]: + +[quote] +____ +The Linux kernel allows extended attribute to have names of up to 255 bytes and values of up to 64 KiB,[14] as do XFS and ReiserFS, but ext2/3/4 and btrfs impose much smaller limits, requiring all the attributes (names and values) of one file to fit in one “filesystem block” (usually 4 KiB). Per POSIX.1e,[citation needed] the names are required to start with one of security, system, trusted, and user plus a period. This defines the four namespaces of extended attributes. +____ + + +* Good, because metadata is stored in the filesystem +* Good, because consistency is easy to maintain +* Good, because the data is attached to the file and survives file operations like copy and move +* Good, because a storage backup also covers the file metadata +* Bad, because we could hit the filesystem limit +* Bad, because changes to extended attributes are not atomic + +=== Metadata File + +We could store metadata in a metadata file next to the file content which has a structured content format like .json, .yaml or .toml. That would give us more space to store bigger amounts of metadata. + +* Good, because there are no size limits +* Good, because there is more freedom to the content format +* Good, because a storage backup also covers the file metadata +* Bad, because it doubles the amount of read / write operations +* Bad, because it needs additional measures against concurrent overwriting changes + +=== Link metadata with an id in the extended attributes + +To link metadata to file content a single extended attribute with a file id (unique per storage space) is sufficient. This would also allow putting metadata in better suited storage systems like SQLite or a key value store. + +* Good, because it avoids extended attribute limits +* Good, because the same mechanism could be used to look up files by id, when the underlying filesystem is an existing POSIX filesystem. +* Bad, because backup needs to cover the metadata as well. Could be mitigated by sharing metadata per space and doing space wide snapshots. +* Bad, because it is a bit more effort to access it to read or index it. diff --git a/modules/developer/pages/ocis/adr/0017-allow-read-only-external-user-management.adoc b/modules/developer/pages/ocis/adr/0017-allow-read-only-external-user-management.adoc new file mode 100644 index 00000000..a329d8fd --- /dev/null +++ b/modules/developer/pages/ocis/adr/0017-allow-read-only-external-user-management.adoc @@ -0,0 +1,106 @@ += 17. Allow read only external User Management +:toc: right +:toclevels: 3 + +* Status: proposed +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/rhafer[@rhafer] +* Date: 2022-02-08 + +== Context and Problem Statement + +oCIS needs to be integrated with various external Authentication and Identity Management Systems. +Usually oCIS will have no administrative access to such a system and we will not be allowed to +reconfigure it to suit our needs (e.g. we will not be able to enhance the schema of an already existing +LDAP Directory). In most of the cases our access will be read-only. + +Sidenote: There is a difference between users, identities and accounts: A user may have multiple +identities which he can authenticate with, e.g. his facebook, twitter, microsoft or google +identity. Multiple identities can be linked to an account in ocis, allowing to fall back to another +identity provider should one of them shut down. This also allows migrating from one identity +provider to another. + +There are different cases where oCIS requires access to users: + +1. While we settled on using OpenID Connect (OIDC) as the authentication protocol for oCIS, we + need to build a user object during authentication with at least an account UUID (to identify + the account) and the email or a name (for display purposes). +2. When searching for share recipients we need to be able to query existing users in the external + identity management system +3. When listing files we need to be able to look up a users display properties (username, email, + avatar...) based on the account UUID + +oCIS internally relies on a stable and persistent identifier (e.g. a UUID) for accounts in order to +implement permissions and sharing. Unfortunately, some deployments are unable to deliver this kind +of stable identifier for users: + +- In OIDC itself the only stable identifier that is guaranteed to be provided by the IDP is + combination of the sub and iss claims. IDPs can optionally return other claims, but we cannot + rely on a specific claim being present. +- When no other services (LDAP, SCIM, ...) are available that could be used look up a user UUID + + +== Decision Drivers + +* oCIS should be a single binary that can run out of the box without external dependencies like an + LDAP server. +* Time: we want to build a release candidate asap. +* oCIS should be easy to integrate with standard external identity management systems + +== Considered Options + +There are two case to consider: +* External identity management system provides an OIDC IdP and an interface to query users +* External identity management system provides just an OIDC IdP with no possibility to query users + +== Decision Outcome + +It's not really possible single out any of the options for this ADR. In the end we will likely need +to support both scenarios. + +=== Positive Consequences: + +* Very flexible integration with a wide range of external systems + +=== Negative consequences: + +* configuration complexity, high support efforts +* Increasingly complex code in oCIS + +== Pros and Cons of the Options + +=== External identity management system is read only and provides an interface to query users (e.g. Corporate Active Directory) + +IdP sends sub & iss and mail or username claims, Identity Management System provides APIs (e.g. +LDAP, SCIM, REST ...) to lookup additional user information. All oCIS services use the CS3 API to +look up the account for the given email or username, where CS3 then uses a backend that relies on +the APIs provided by the IdM. + +* Good, because we can rely on the external identity management +* Good, because ocis services only need to know about the CS3 user provider API, which acts as an + abstraction layer for different identity management systems +* Good, because there is only a single source of truth (the external IdM) and we don't need to + implement a synchronization mechanism to maintain an internal user database (we will likely need + some form of caching though, see below) +* Bad, because the identity management needs to provide a stable, persistent, non-reassignable user + identifier for an account, e.g. `owncloudUUID` or `ms-DS-ConsistencyGuid` +* Bad, because we need to implement tools that can change the account id when it did change anyway +* Bad, because without caching we will hammer the identity management system with lookup requests + +=== External identity management system is read only and does NOT provide an API to query users + +Idp sends sub & iss and mail or username claims. We need to provision an internal account mapping, +creating a unique ID, upon the first login of a user to be able to look up user properties by account +id. + +* Good, because this has very little external requirements +* Good, because we have accounts fully under our control +* Bad, because we have to provide the user lookup APIs +* Bad, because users will only a visible after the first login +* Bad, because our internal account mapping might get out of date when user attribute (e.g. name or + mail) change. At least until the next time that user logs in + +== Links + +* [Link type] [Link to ADR] +* … +* supersedes xref:0003-external-user-management.adoc[3. Use external User Management] diff --git a/modules/developer/pages/ocis/adr/0018-file-search-api.adoc b/modules/developer/pages/ocis/adr/0018-file-search-api.adoc new file mode 100644 index 00000000..5fc0d353 --- /dev/null +++ b/modules/developer/pages/ocis/adr/0018-file-search-api.adoc @@ -0,0 +1,56 @@ += 18. File Search API +:toc: right +:toclevels: 3 + +* Status: proposed +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/c0rby[@c0rby] +* Date: 2022-03-18 + +== Context and Problem Statement + +The ability to find files based on certain search terms is a key requirement for a system that provides the ability to store unstructured data on a large scale. + +== Decision Drivers + +* Have a simple yet powerful, scalable and performant way of finding files in oCIS +* Be able to construct intelligent searches based on metadata +* Allow the user to filter the search queries based on metadata + +== Considered Options + +* <> +* <> + +== Decision Outcome + +Chosen option: <> because the current WebUI is compatible with that API. We may use the GraphAPI later in a second iteration. + +=== Positive Consequences + +* The existing Clients can continue to use the well-known API +* There are existing API tests which cover the basic behavior + +=== Negative consequences + +* We have no server side result filtering capabilities + +== Pros and Cons of the Options + +=== Libre Graph API + +* Good, because we try to switch most of our HTTP requests to Libre Graph +* Good, because the Graph API supports scopes, sorting and query language +* Good, because it supports server side result filtering +* Bad, because there are currently no clients which support that + +=== WebDAV API + +* Good, because WebDAV is a well-known and widely adopted Standard +* Good, because existing Clients continue to work without extra efforts +* Bad, because the syntax is limited +* Bad, because we cannot do server side result filtering + +== Links + +* xref:0019-file-search-index.adoc[Search Indexing] +* xref:0020-file-search-query-language.adoc[Search Query Language] diff --git a/modules/developer/pages/ocis/adr/0019-file-search-index.adoc b/modules/developer/pages/ocis/adr/0019-file-search-index.adoc new file mode 100644 index 00000000..92956f79 --- /dev/null +++ b/modules/developer/pages/ocis/adr/0019-file-search-index.adoc @@ -0,0 +1,75 @@ += 19. File Search Index +:toc: right +:toclevels: 3 + +* Status: accepted +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/c0rby[@c0rby] +* Date: 2022-03-18 + +== Context and Problem Statement + +The ability to find files based on certain search terms is a key requirement for a system that provides the ability to store unstructured data on a large scale. + +More sophisticated search capabilities are expected and can be implemented, especially based on metadata. + +To trigger the indexing of a file, the search service listens to create, update and delete events on the internal event bus of oCIS. + +The events need to contain a valid reference that defines the file space and file id of the file in question. The event only must be sent when the file operation (update, creation, removal) is finished. + +Sharing adds more complexity because the index also needs to react to create, delete and modify shares events. Sharing should not duplicate the indexed data, especially within spaces or group shares. + +== Decision Drivers + +* Have a simple yet powerful, scalable and performant way of finding files in oCIS +* Be able to construct intelligent searches based on metadata +* Allow the user to filter the search queries based on metadata +* Basic File Search needs to be implemented out of the box without external dependencies +* The Search Indexing Service should be replaceable with more sophisticated technologies like Elasticsearch +* Make use of the spaces architecture to shard search indexes by space +* The Search Indexing Service needs to deal with multiple users accessing the same resources due to shares +* The Search Service should be compatible with different search indexing technologies + +== Considered Options + +* <> +* <> + +== Decision Outcome + +Chosen option: Bleve Search, because we can fulfill the MVP and include it into the single binary. + +=== Positive Consequences + +* Basic File Search works out of the box +* We do not need heavy external dependencies which need to be deployed alongside + +=== Negative consequences + +* We need to be aware of the scaling limits +* We need to find a way to work with shares and spaces +* It has a limited query language + +== Pros and Cons of the Options + +=== Bleve Search + +* Good, because it is written in GoLang and can be bundled into the single oCIS binary +* Good, because it is a lightweight but powerful solution which could fulfill a lot of use cases +* Bad, because we do not know exactly how we can represent shares in the index without duplicating data +* Bad, because it is a single process +* Bad, because the query language is limited + +=== Elastic Search + +* Good, because it has become an industry standard +* Good, because it supports a rich query language +* Good, because it has built in cluster support and scales well +* Good, because it has a permission system and supports multiple users and groups to access the same resource +* Bad, because it is a heavy setup and needs extra effort and knowledge + +== Links + +* xref:0018-file-search-api.adoc[Search API] +* xref:0020-file-search-query-language.adoc[Search Query Language] +* link:https://github.com/blevesearch/bleve[Bleve Search on GitHub] +* link:https://www.elastic.co/elastic-stack/[ElasticSearch] diff --git a/modules/developer/pages/ocis/adr/0020-file-search-query-language.adoc b/modules/developer/pages/ocis/adr/0020-file-search-query-language.adoc new file mode 100644 index 00000000..de19f4fc --- /dev/null +++ b/modules/developer/pages/ocis/adr/0020-file-search-query-language.adoc @@ -0,0 +1,105 @@ += 20. File Search Query Language +:toc: right +:toclevels: 3 + +* Status: accepted +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin], link:https://github.com/c0rby[@c0rby], link:https://github.com/kulmann[@kulmann], link:https://github.com/felix-schwarz[@felix-schwarz], link:https://github.com/JammingBen[@JammingBen] +* Date: 2023-06-23 + +== Context and Problem Statement + +From the users perspective, the interface to search is just a single form field where the user enters one or more search terms. The minimum expectation is that the search returns file names and links to files that: + +* have a file name that contains at least one of the search terms +* contain at least one of the search terms in the file contents +* have metadata that is equal or contains one of the search terms + +== Decision Drivers + +* The standard user should not be bothered by a query syntax +* The power user should also be able to narrow his search with an efficient and flexible syntax +* We need to consider different backend technologies which we need to access through an abstraction layer +* Using different indexing systems should lead to a slightly different feature set without changing the syntax completely + +== Considered Options + +* <> +* <> +* <> +* <> +* <> + +== Decision Outcome + +Chosen option: <>, because it enables advanced search across all platforms. + +=== Positive Consequences + +* We can use the same query language in all clients + +=== Negative consequences + +* We need to build and maintain a backend connector + +== Pros and Cons of the Options + +=== Keyword Query Language + +The Keyword Query Language (KQL) is used by Microsoft Share Point and other Microsoft Services. It uses very simple query elements, property restrictions and operators. + +* Good, because we can fulfill all our current needs +* Good, because it is very similar to the used query language in iOS +* Good, because it supports date time keywords like "today", "this week" and more +* Good, because it can be easily extended to use "shortcuts" for eg. document types like `:presentation` which combine multiple mime types. +* Good, because it is successfully implemented and used in similar use cases +* Good, because it gives our clients the freedom to always use the same query language across all platforms +* Good, because Microsoft Graph API is using it, we will have an easy transition in the future +* Bad, because we need to build and maintain a connector to different search backends (bleve, elasticsearch or others) + +=== Simplified Query + +Implement a very simple search approach: Return all files which contain at least one of the keywords in their name, path, alias or selected metadata. + +* Good, because that covers 80% of the users needs +* Good, because it is very straightforward +* Good, because it is a suitable solution for GA +* Bad, because it is below the industry standard +* Bad, because it only provides one search query + +=== Lucene Query Language + +The Lucene Query Parser syntax supports advanced queries like term, phrase, wildcard, fuzzy search, proximity search, regular expressions, boosting, boolean operators and grouping. It is a well known query syntax used by the Apache Lucene Project. Popular Platforms like Wikipedia are using Lucene or Solr, which is the successor of Lucene + +* Good, because it is a well documented and powerful syntax +* Good, because it is very close to the Elasticsearch and the Solr syntax which enhances compatibility +* Bad, because there is no powerful and well tested query parser for golang available +* Bad, because it adds complexity and fulfilling all the different query use-cases can be an "uphill battle" + +=== Solr Query Language + +Solr is highly reliable, scalable and fault-tolerant, providing distributed indexing, replication and load-balanced querying, automated failover and recovery, centralized configuration and more. Solr powers the search and navigation features of many of the world's largest internet sites. + +* Good, because it is a well documented and powerful syntax +* Good, because it is very close to the Elasticsearch and the Lucene syntax which enhances compatibility +* Good, because it has a strong community with large resources and knowledge +* Bad, because it adds complexity and fulfilling all the different query use-cases can be an "uphill battle" + +=== Elasticsearch Query Language + +Elasticsearch provides a full Query DSL (Domain Specific Language) based on JSON to define queries. Think of the Query DSL as an AST (Abstract Syntax Tree) of queries, consisting of two types of clauses. It is able to combine multiple query types into compound queries. It is also a successor of Solr. + +* Good, because it is a well documented and powerful syntax +* Good, because it is very close to the Elasticsearch and the Solr syntax which enhances compatibility +* Good, because there is a stable and well tested go client which brings a query builder +* Good, because it could be used as the query language which supports different search backends by just implementing what is needed for our use-case +* Bad, because it adds complexity and fulfilling all the different query use-cases can be an "uphill battle" + +== Links + +* xref:0018-file-search-api.adoc[Search API] +* xref:0019-file-search-index.adoc[Search Indexing] +* link:https://learn.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-syntax-reference[KQL] +* link:https://lucene.apache.org/[Apache Lucene] +* link:https://solr.apache.org/[Apache Solr] +* link:https://solr.apache.org/[Elastic Search] +* link:https://github.com/elastic/go-elasticsearch[Elastic Search for go] diff --git a/modules/developer/pages/ocis/adr/0021-service-accounts.adoc b/modules/developer/pages/ocis/adr/0021-service-accounts.adoc new file mode 100644 index 00000000..7324f4c7 --- /dev/null +++ b/modules/developer/pages/ocis/adr/0021-service-accounts.adoc @@ -0,0 +1,90 @@ += 21. Service accounts +:toc: right +:toclevels: 3 + +* Status: proposed +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/c0rby[@c0rby] +* Date: 2023-01-18 + +== Context and Problem Statement + +There are three levels of security checks in a microservice web application that uses OpenID Connect: +1. _scope claims_ limit the possible operations to what the user (or admin on behalf of the organization) consented to +2. _service authorization_ limit the possible operations to what specific services are allowed to do, on behalf of users or even without them +3. _permission checks_ limit the possible operations to the relationships between subject, permission and resource allow + +This ADR deals with a prerequisite for service authorization: service accounts. + +Some services need access to file content without a user being logged in. We currently pass the owner or manager +of a space in events which allows the search service to impersonate that user to extract metadata from the changed resource. +There are two problems with this: +1. The service could get all permissions of the user and gain write permission +2. There is a race condition where the user in the event might no longer have read permission, causing the index to go stale + +The race condition will become more of an issue when we start working on a workflow engine. + +How can we grant services the least amount of permissions required for their purpose? + +== Decision Drivers + +* It should be possible to represent this as servicePrincipals in the libregraph API, similar to the link:https://learn.microsoft.com/en-us/graph/api/resources/serviceprincipal?view=graph-rest-1.0[MS Graph servicePrincipal]. +* Services should check permissions using the oCIS permissions or reva auth service, we don't want to introduce a new mechanism for this + +== Considered Options + +* <> +* <> + +== Decision Outcome + +Chosen option: <> + +=== Consequences + +* Good, because it allows provisioning permissions for services +* Good, because it uses existing CS3 concepts +* Good, because it uses the existing permissions service +* Good, because it can be mapped to libre graph permissions +* Bad, because we have to make the reva auth manager aware of CS3 link:https://cs3org.github.io/cs3apis/#cs3.identity.user.v1beta1.UserType[`USER_TYPE_SERVICE`] +* Bad, because we have to provision and manage service accounts on init +* Bad, because external APIs may need to filter out service accounts +* Bad, because we need to persist service accounts in addition to normal user accounts + +== Pros and Cons of the Options + +=== Service Accounts + +Make the reva auth manager and registry aware of CS3 users of type link:https://cs3org.github.io/cs3apis/#cs3.identity.user.v1beta1.UserType[`USER_TYPE_SERVICE`]. Then we can provision service accounts at oCIS initialization and use the permissions service to check permissions. +When assigning permissions we use the permission constraints to define the scope of permissions, see <> for more details. + +To authenticate service accounts the static reva auth registry needs to be configured with a new auth provider for type `service`. The actual provider can use a plain JSON file or JSONCS3 that is provisioned once with `ocis init`. TODO Furthermore, the user provider needs to be able to return users for service accounts. + + +* Good, because we could replace machine auth with specific service accounts and no longer have to distribute a shared secret everywhere +* Bad, because we don't know if a there are places in the code that try to look up a user with USER_TYPE_SERVICE at the cs3 users service ... they might not exist there ... or do we have to implement a userregistry, similar to the authregistry? +* Bad, because we have to provision and manage service accounts on init +* Bad, because we have to write code to manage service accounts or at least filter them out in the admin ui + + +=== Impersonate Space-Owners + +We could implement a new auth manager that can authenticate space owners, a CS3 user type we introduced for project spaces which 'have no owner', only one or more managers. + +* Good, because it reuses the space owner user type +* Bad, because the space owner always has write permission +* Bad, because we don't know if a there are places in the code that try to look up a user with USER_TYPE_SPACE_OWNER at the cs3 users service ... they might not exist there ... or do we have to implement a userregistry, similar to the authregistry? +* Bad, because it feels like another hack and does not protect against compromised services that try to execute operations that the user did not consent to. + +== Links + +* link:https://learn.microsoft.com/en-us/graph/api/resources/serviceprincipal?view=graph-rest-1.0[MS Graph servicePrincipal] +* link:https://reva.link/docs/config/packages/auth/manager/[reva auth managers] - lacks docs for `auth_machine`, to be found link:https://github.com/cs3org/reva/blob/edge/pkg/auth/manager/machine/machine.go[in the code] + +== Permission checks +When checking permissions we do not check for global permissions but for the concrete permission. Global permissions describe permissions that are used when assigning permissions, e.g. the index service account has the read permission constrained to tenant. The concrete permission check always contains a resource and a specific permission like `Resource.Read` or `Space.Delete`. That we currently check if a user has the `delete-all-spaces` permission is wrong. It should instead check if the user has the permission `Space.Delete` on a specific space. The permissions service can implement the check by taking the permission constraint into account. + +Another example would be a `Resource.Read` check for a specific resource. Normal users like the demo users Einstein and Marie would have the permission `Resource.ReadWrite` with the constraint ALL (which limits them to all files they own and that have been shared with them). The permissions service can return true. Service accounts like the indexer would have `Resource.Read` with the constraint TENANT and thus be granted read access to all resources. + +In the storage drive implementation we can check the ACLs first (which would allow service accounts that are known to the underlying storage system, e.g. EOS to access the resource) and then make a call to the permissions service. At least for the Read Resource permission. Other permission checks can be introduced as needed. + +The permission names and constraints are different from the MS Graph API. Giving permission like link:https://learn.microsoft.com/en-us/graph/permissions-reference#user-permissions[`Files.ReadWrite.All`] a different meaning, depending on the type of user (for normal users it means all files they have access to, for service accounts it means all files in the organization) is a source of confusion which only gets worse when there are two different UUIDs for this. diff --git a/modules/developer/pages/ocis/adr/0022-sharing-and-space-management-api.adoc b/modules/developer/pages/ocis/adr/0022-sharing-and-space-management-api.adoc new file mode 100644 index 00000000..d3807a2e --- /dev/null +++ b/modules/developer/pages/ocis/adr/0022-sharing-and-space-management-api.adoc @@ -0,0 +1,82 @@ += 22. Sharing and Space Management API +:toc: right +:toclevels: 3 + +* Status: accepted +* Deciders: link:https://github.com/JammingBen[@JammingBen] link:https://github.com/butonic[@butonic] link:https://github.com/theonering[@theonering] link:https://github.com/kobergj[@kobergj] link:https://github.com/micbar[@micbar] +* Date: 2023-08-08 + +Technical Story: link:https://github.com/owncloud/ocis/issues/6993[Public issue] + +== Context and Problem Statement + +In the early days of the rewrite of ownCloud it was an important goal to keep all important APIs compatible with ownCloud 10. Infinite Scale embraced that goal until version 1.0.0. + +After that first release, the focus changed. + +Infinite Scale started the spaces feature which brings a whole new set of APIs and concepts. We made the conscious decision to keep the sharing API as it was, live with its shortcomings and create workarounds to support spaces. We have come a long way so far. Now we need to move on. The Web Client has made the decision to drop the support of ownCloud 10 and keep version 7.0 alive for ownCloud 10 to keep the easy migration path intact. + +The desktop and mobile client platforms were suffering from poor support from the server and can now move forward with a new API implementation. By using openApi 3 and all the needed tooling around it developing the LibreGraph specification, documentaion and SDKs, we now feel confident to move on. + +== Decision Drivers + +* The Path based nature of the OCS API lacks spaces support +* The permissions bitmask is no longer working when using sharing roles +* We want to support server announced sharing roles which are different per instance or scope +* We need to get rid of the currently hardcoded sharing roles in our clients +* New sharing roles and permissions are needed to support secure view and other new features +* Space Memberships are not shares and need to have different semantics +* Elevation of permissions in subfolders or full denials should be possible without creating a new share +* Third party integrations need generated SDKs in different languages to speed up the development + +== Considered Options + +* <> +* <> + +== Decision Outcome + +Chosen option: "<>" + +=== Positive Consequences: + +* We can create a new clean API which fits the spaces concept +* LibreGraph embraces OData which is a known API pattern +* Sharing will be integrated in the existing SDKs and documentation +* Removing the OCS Api reduces complexity +* Removing the OCS Api makes the clients codebases smaller and removes manually maintained parts of the SDKs +* The extra error handling for the OCS API can be dropped from our clients + +=== Negative Consequences: + +* We need to deprecate and remove the OCS API +* Existing third party integrations need to do some refactoring + +== Pros and Cons of the Options + +=== New OCS Api Version + +To overcome the limitations of the OCS 2.0 API we could create a new major version with the spaces concept in mind. This would give us the opportunity to create a new openApi Spec. + +* Good, because the workarounds from version 2.0 could be dropped +* Bad, because we would need to deprecate the version 2.0 +* Bad, because we would need to maintain a separate specification / repository +* Bad, because it would create the need to use two different SDKs in our clients +* Bad, because we would need to implement query parameters and filters on our own +* Bad, because sharing information could not be included in the spaces API via queries or filters + +=== Sharing via LibreGraph + +Integrate Sharing into the link:https://github.com/owncloud/libre-graph-api[LibreGraph API] by using the already existing toolchain and documentation flows. + +* Good, because that reduces the number of SDKs +* Good, because it reduces the number of APIs +* Good, because spaces and shares can be used together in queries and filters +* Good, because we would use the existing OData pattern +* Bad, because we need to deprecate the OCS API + +== Links + +* link:https://github.com/owncloud/libre-graph-api[LibreGraph API] +* link:https://www.odata.org/documentation/[OData] +* link:https://www.openapis.org/[OpenAPI Standard] diff --git a/modules/developer/pages/ocis/adr/0023-index-and-store-metadata.adoc b/modules/developer/pages/ocis/adr/0023-index-and-store-metadata.adoc new file mode 100644 index 00000000..53528b3e --- /dev/null +++ b/modules/developer/pages/ocis/adr/0023-index-and-store-metadata.adoc @@ -0,0 +1,73 @@ += 23. Index and store metadata +:toc: right +:toclevels: 3 + +* Status: accepted +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/dschmidt[@theonering], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin] +* Date: 2023-10-17 + +== Context and Problem Statement + +ownCloud Infinite Scale is supposed to become a data platform and as such it needs to provide access to metadata. +Currently only metadata common to all file types (filesize, mime-type, ...) is stored in the index and the metadata storage. +We want to make other file type specific metadata available to consumers of our internal and external APIs. +Simple examples would be audio metadata like artist, album and title or exif metadata in images. + +== Decision Drivers + +== Considered Options + +* <> +* <> +* <> + +== Decision Outcome + +Chosen option: "<>", because Graph API is a simple common denominator and we want to avoid putting the complexity of mapping non-standardized data from potentially different extractors in several areas of the code base. Storage and index keys are determined by facet and property name, e.g. `audio.artist` for the artist in a music file. Storage keys are additionally prefixed with `libre.graph.`, i.e. `libre.graph.audio.artist`. +Handling Graph API specific metadata is a first step towards handling metadata. More generic and extensible handling of arbitrary metadata can be added later. + +=== Positive Consequences: + +* Graph API endpoint implementation is trivial +* Documented public api and stored data are the same +* Reasonable complexity for the initial implementation + +=== Negative Consequences: + +* Graph API is limited, so not _all_ available metadata can be accessed +* Switching the internal format and adding more metadata later will require re-indexing + +== Pros and Cons of the Options + +=== Store Subset of Extracted Metadata Required for Graph API + +Use Graph API facets and properties for determining the subset of stored metadata and the storage key. +The index key for the `artist` property of the `audio` facet is `audio.artist`, the storage key is additionally prefixed with `libre.graph.`. + +* Good, because central mapping of values happens consistently and only once in a central place + - it happens in the extractor (integration) which likely knows best how to map metadata to standard properties +* Good, because when multiple extractors share a common set of provided values, applications can rely on the mapping and the complexity is kept low +* Bad, because not all metadata is available, not everything can be searched +* Good, because Graph API already chose a reasonable subset of most interesting properties + +=== Store Subset of Extracted Metadata Specified by Another Standard + +There are a bunch of metadata standards but none of them is really universal. There is always something that is only supported in one or the other standard. Tika for example extracts audio metadata using a mixture of Dublin Core and XMP Dynamic Media keys. + +- Bad, because it makes implementing a new extractor integration harder +- Bad, because it makes using the stored data more complicated than a simple standard like discussed above + +=== Store Everything from Extractors + +- Good, because all metadata is available and searchable +- Good, because consuming applications can decide how to map data +- Good, because extractor implementation becomes more trivial +- Bad, because all applications become dependent on the extractor and need to handle different extractors on their own + +== Links + +* https://github.com/owncloud/libre-graph-api/pull/120 / https://learn.microsoft.com/de-de/graph/api/resources/audio?view=graph-rest-1.0 +* https://github.com/owncloud/libre-graph-api/pull/122 / https://learn.microsoft.com/en-us/graph/api/resources/photo?view=graph-rest-1.0 +* https://github.com/owncloud/libre-graph-api/pull/123 / https://learn.microsoft.com/en-us/graph/api/resources/geoCoordinates?view=graph-rest-1.0 +* https://developer.adobe.com/xmp/docs/XMPNamespaces/xmpDM/ +* https://www.dublincore.org/schemas/ diff --git a/modules/developer/pages/ocis/adr/0024-msgpack-metadata.adoc b/modules/developer/pages/ocis/adr/0024-msgpack-metadata.adoc new file mode 100644 index 00000000..aeaa7288 --- /dev/null +++ b/modules/developer/pages/ocis/adr/0024-msgpack-metadata.adoc @@ -0,0 +1,77 @@ += 24. Messagepack metadata +:toc: right +:toclevels: 3 + +* Status: accepted +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/aduffeck[@aduffeck], link:https://github.com/micbar[@micbar], link:https://github.com/dragotin[@dragotin] +* Date: link:https://github.com/cs3org/reva/pull/3711/commits/204253eee9dbb8e7fa93a01f3f94a2d28ce40a06[2023-03-15] + +== Context and Problem Statement + +File metadata management is an important aspect for oCIS as a data platform. While using extended attributes to store metadata allows attaching the metadata to the actual file it causes a significant amount of syscalls that outweigh the benefits. Furthermore, filesystems are subject to different limitations in the number of extended attributes or the value size that is available. + +== Decision Drivers + +Performance of reading extended attributes suffers from the syscall overhead when listing and reading all attributes. Getting rid of limitations imposed by the filesystem used to store decomposedfs metadata. + +== Considered Options + +Going back to the original xref:0016-files-metadata.adoc[ADR-0016 Storage for Files Metadata] we decided to use a dedicated file for metadata storage next to the decomposedfs file representing the node. Several options for the data format were considered: + +* Use JSON files to store metadata +* Use INI files to store metadata +* Use msgpack files to store metadata +* Use protobuf messages to store metadata + +== Decision Outcome + +Chosen option: "<>", because we want to stay with a self describing binary format. This is a performance tradeoff that is faster and more efficient than text based formats and more flexible but less efficient than protobuf. + +Note: directory listings are still read from the storage and remain uncached. + +=== Positive Consequences: + +* Way less syscalls +* Node metadata can easily be cached, avoiding all trips to the storage until a file changes. + +=== Negative Consequences: + +* We need to migrate existing metadata +* We need to build tooling that allows manipulating metadata similar to `setfattr` and `getfattr`. + +== Pros and Cons of the Options + +=== Ini files + +* Good, human readable +* Good, self describing +* Good, widely used and well understood +* Good, suited for key value like content - exactly what we need for extended attributes +* Bad, slower and less efficient than binary formats + +=== JSON files + +* Good, human readable +* Good, self describing +* Good, widely used and well understood +* Good, could be used for more than just key value +* Bad, slower and less efficient than binary formats + +=== Msgpack files + +* Good, self describing +* Good, efficient because it is binary encoded +* Good, could be used for more than just key value +* Bad, not human readable - requires tooling to manipulate safely + +=== protobuf files + +* Good, very efficient because it is binary encoded +* Good, could be used for more than just key value +* Bad, not human readable +* Bad, not self describing - requires tooling to evolve the messages + +== Links + +* supersedes xref:0016-files-metadata.adoc[ADR-0016 Storage for Files Metadata] +* link:https://medium.com/@hugovs/the-need-for-speed-experimenting-with-message-serialization-93d7562b16e4[The need for speed — Experimenting with message serialization] \ No newline at end of file diff --git a/modules/developer/pages/ocis/adr/0025-distributed-search-index.adoc b/modules/developer/pages/ocis/adr/0025-distributed-search-index.adoc new file mode 100644 index 00000000..13eb5ed6 --- /dev/null +++ b/modules/developer/pages/ocis/adr/0025-distributed-search-index.adoc @@ -0,0 +1,71 @@ += 25. Distributed Search Index +:toc: right +:toclevels: 3 + +* Status: draft +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/fschade[@fschade], link:https://github.com/aduffeck[@aduffeck] +* Date: 2024-02-09 + +== Context and Problem Statement + +Search is currently implemented with link:https://github.com/blevesearch/bleve[blevesearch], which internally uses bbolt. bbolt writes to a local file, which prevents scaling out the service. + +The initial implementation used a single blevesearch index for all spaces. While this makes querying all spaces easy because the results do not need to be aggregated from multiple indexes, the single node becomes a bottleneck when answering search queries. Furthermore, indexing is also part of the search service and has to share the resources. + +== Decision Drivers + +* Indexing should be decoupled from the search service +* The search service should be able to scale horizontally +* The solution needs to be embeddable in the single binary + +== Considered Options + +* one index per space +* link:https://github.com/elastic/elasticsearch[elasticsearch] (java) +* link:https://github.com/dgraph-io/dgraph[dgraph] (go) +* link:https://github.com/manticoresoftware/manticoresearch/[manticore] (C++) +* link:https://github.com/meilisearch/meilisearch[meilisearch] (Rust) + +== Decision Outcome + +Chosen option: _???_ + +=== Positive Consequences: + +* TODO + +=== Negative Consequences: + +* TODO + +== Pros and Cons of the Options + +=== one index per space + +Instead of using a single index (current implementation) or a distributed search index like elasticsearch the search service should aggregate queries from dedicated indexes per space. The api to a space index provider should be able to take multiple space ids in the request, similar to how a storage provider can handle multiple spaces. When treating spaces and the corresponding search index to belong together we can also treat them as a single unit for backup and restore. In federated deployments we can send the search queries to all search providers / spaces that the user has access to. + +How a search provider is implemented then depends on the requirements. For a single node deployment bleve might be fine, for a kubernetes deployment a dedicated service might be the better fit. + +=== elasticsearch + +* Good, commercial support available at https://www.elastic.co/de/pricing +* Good, industry standard +* Bad, nobody seems to like it +* Bad, not embeddable (Java) + +=== dgraph + +* Good, commercial support available at https://dgraph.io/pricing +* Good, embeddable? (go) - TODO verify + +=== manticore +* Good, commercial support available at https://manticoresearch.com/services/ +* Bad, not embeddable (C++) + +=== meilisearch +* Good, commercial support available at https://www.meilisearch.com/pricing +* Bad, not embeddable (Rust) + +== Links + +* supersedes xref:0019-file-search-index.adoc[ADR-0019 File Search Index] \ No newline at end of file diff --git a/modules/developer/pages/ocis/adr/0026-application-based-user-settings.adoc b/modules/developer/pages/ocis/adr/0026-application-based-user-settings.adoc new file mode 100644 index 00000000..db11a82d --- /dev/null +++ b/modules/developer/pages/ocis/adr/0026-application-based-user-settings.adoc @@ -0,0 +1,93 @@ += 26. Application based user settings +:toc: right +:toclevels: 3 + +* Status: draft +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/fschade[@fschade], link:https://github.com/kulmann[@kulmann] +* Date: 2024-02-09 + +== Context and Problem Statement + +To share user settings across devices applications want to store user specific settings on the server. The ePUB app wants to remember which page the user is on. The iOS app wants to rember search queries. The Caldav app needs a space to store data. + +== Decision Drivers + +== Considered Options + +* OCS provisioning API +* settings service +* libregraph API + +== Decision Outcome + +Chosen option: _???_ + +=== Positive Consequences: + +* TODO + +=== Negative Consequences: + +* TODO + +== Pros and Cons of the Options + +=== OCS provisioning API + +Nextcloud added a `/ocs/v2.php/apps/provisioning_api/api/v1/config/users/{appId}/{configKey}` endpoint + +* Bad, legacy API we want to get rid of + +=== settings service + +- Bad, yet another API. Always uses POST requests. + +=== libregraph API + +The MS Graph API has link:https://learn.microsoft.com/en-us/graph/api/drive-get-specialfolder?view=graph-rest-1.0&tabs=http[a special approot driveItem] that apps can use to store arbitrary files. See also: +link:https://learn.microsoft.com/en-us/onedrive/developer/rest-api/concepts/special-folders-appfolder?view=odsp-graph-online[Using an App Folder to store user content without access to all files] and a blog post with the section link:https://blog.mastykarz.nl/easiest-store-user-settings-microsoft-365-app/#store-data-in-the-applications-personal-folder[Store data in the application’s personal folder]. + +It basically uses the `/me/drive/special/approot:/{filename}` endpoint to +[source,http] +---- +PUT https://graph.microsoft.com/v1.0/me/drive/special/approot:/settings.json:/content +content-type: text/plain +authorization: Bearer abc + +{"key": "value"} +---- +or +[source,http] +---- +GET https://graph.microsoft.com/v1.0/me/drive/special/approot:/settings.json:/content +authorization: Bearer abc +---- + +On single page apps you need two requests: +[source,http] +---- +GET https://graph.microsoft.com/v1.0/me/drive/special/approot:/settings.json?select=@microsoft.graph.downloadUrl +authorization: Bearer abc +---- +followed by +[source,http] +---- +GET +---- + +Currently, applications have no dedicated tokens that we could use to derive the `appid` from. All apps should have an `appid` and link:https://learn.microsoft.com/en-us/graph/api/application-list?view=graph-rest-1.0&tabs=http[be discoverable under] +[source,http] +---- +GET /applications +---- + +In any case for libregraph we could introduce a `LIBRE_GRAPH_APPID` header to make these requests possible rather soon. + +Then we can decide if we want to store these files in the users personal drive, or if we create a space for every app that then uses the userid as a folder that contains all the files for the user. + +- Good, because clients can remain in libregraph API land +- Bad, we currently have no application tokens + + + +== Links diff --git a/modules/developer/pages/ocis/adr/0027-new-share-jail.adoc b/modules/developer/pages/ocis/adr/0027-new-share-jail.adoc new file mode 100644 index 00000000..3a2301c8 --- /dev/null +++ b/modules/developer/pages/ocis/adr/0027-new-share-jail.adoc @@ -0,0 +1,89 @@ += 27. New Share Jail +:toc: right +:toclevels: 3 + +* Status: draft +* Deciders: link:https://github.com/butonic[@butonic], link:https://github.com/rhafer[@rhafer], link:https://github.com/dragotin[@dragotin] +* Date: 2024-02-21 + +== Context and Problem Statement + +The oCIS share jail is a space that contains all accepted / synced shares of a user. In contrast to a personal or project space that contains actual resources, the share jail space only contains references pointing to shared resources. The root directory only consists of mountpoints that actually represent resources in other spaces. On the WebDAV API clients expect an `oc:fileid` property to identify resources in other API endpoints, eg. the libregraph `/me/sharedWithMe` endpoint. + +Currently, we construct the `oc:fileid` from the pattern `{shareproviderid}${sharespaceid}!{sharemountid}`. `{shareproviderid}`and `{sharespaceid}` are both hardcoded to `a0ca6a90-a365-4782-871e-d44447bbc668`. The `{sharemountid}` itself uses the pattern `{shared-resource-providerid}:{shared-resource-spaceid}:{shareid}`. + +Since a resource can be shared to the same user in multiple ways (a group share and a user share) we deduplicate the two shares and only show one mountpoint in the share jail root. This is where this solution starts to fall apart: +* When accepting, mounting or syncing a share we implicitly have to accept all shares +* Each share has a different `{shareid}`, so we currently look up the oldest share and use it to build the `oc:fileid` +* Consequently, when the oldest share is revoked the `oc:fileid` changes. + +We need to build the `oc:fileid` from a more stable pattern. + +=== Shareid + +The WebDAV PROPFIND response also contains a `oc:shareid` which currently is derived from the path when the spaceid matches the share jail. The jsoncs3 implementation of the share manager currently is the only one using the `{shared-resource-providerid}:{shared-resource-spaceid}:{shareid}` pattern, where `{shareid}` is a uuid that is generated when creating the share. + +Again, the problem is that a resource can be shared multiple times. + +== Decision Drivers + +* We need to change the `oc:fileid` pattern without breaking clients. +* We need to be able to correlate files from WebDAV and the Graph API. + +== Considered Options + +* <> +* <> +* <> +* <> + +== Decision Outcome + +Resource based id: it correctly reflects the semantic meaning of a mount point, by indirectly pointing to the resource, not the share. The permissions on the share have to be checked in the storageprovider itself, anyway. Switching to graph requires more effort and the transition can happen gradually ofter changing the `oc:fileid` pattern in the sharejail. + +=== Positive Consequences: + +* We get rid of mixing share ids with fileids, preventing unexpected `oc:fileid` changes. + +=== Negative Consequences: + +* We need to teach clients about a new share jail space that uses the new `oc:fileid` pattern. They may need to implement a migration strategy to switch from the old share jail space to a new share jail space by replacing the fileid in their internal database. The might be able to just switch over, because the only `oc:fileid` that changes is the one from the mountpoints. The other nodes in the subtree already use the resourceid of the shared resource. +* Clients relying on `oc:shareid` to correlate share jail entries in PROPFIND responses need to either deal with multiple `oc:shareid` as a resource can be shared multiple times, or we deprecate `oc:shareid` and only use the `oc:fileid`. _jfd: Who is using this? why? Please explain and add to the decision drivers above!_ +* The graph api also needs to be able to list entries from the new share jail. _jfd: clients could use a filter to ask for the new share jail id_ + +== Pros and Cons of the Options + +=== Share based id +Follow the pattern `{shareproviderid}${sharespaceid}!{sharemountid}`, where `{sharemountid}` is `{shared-resource-providerid}:{shared-resource-spaceid}:{shareid}`. +Combined patter `{shareproviderid}${sharespaceid}!{shared-resource-providerid}:{shared-resource-spaceid}:{shareid}`. +`{shareproviderid}` and `{sharespaceid}`are both hardcodet to `a0ca6a90-a365-4782-871e-d44447bbc668` to route all id based requests for mountpoints to the share jail space. + ++ Good, the `{shared-resource-providerid}` and `{shared-resource-spaceid}` are used to shard the shares per space. +- Bad, `oc:fileid` changes if the oldest received share to a resource is revoked. + +=== Resource based id +Follow the pattern `{shareproviderid}${sharespaceid}!{shared-resource-providerid}:{shared-resource-spaceid}:{shared-resource-opaqueid}`. +Hardcode `756e6cdf-5630-4b66-9380-55a85188e0f6` as a new `{sharespaceid}` to allow clients to detect the new share jail and change it at their own pace. + ++ Good, stable `oc:fileid` that remains the same, regardless of permission changes. ++ Good, clients can detect the new share jail and deal with it on their terms. + +=== Permission based id +Follow the pattern `{shareproviderid}${sharespaceid}!{shared-resource-providerid}:{shared-resource-spaceid}:{shared-resource-opaqueid}:{permission-id}`. + +- Bad, same instability as the share id +- Bad, we don't even have a permission id. We could construct one from the grantee, but this leads nowhere. + + +=== Use graph for file metadata +Instead of using WebDAV to correlate files with shares fully embrace libregraph to manage file metadata. +Follow the pattern `{shareproviderid}${sharespaceid}!{shared-resource-providerid}:{shared-resource-spaceid}:{shared-resource-opaqueid}`. +WebDAV can be stripped of any ownCloud specific properties and will only be used for file up and download. + +- Bad, more effort ++ Good, clean way of representing mountpoints and the shared resource in one `driveItem` that can include the resource based id. ++ Good, pagination, sorting and filtering cleanly specified ++ Good, WebDAV can be stripped down. ++ Good, Clients could get rid of WebDAV client and XML libs as all endpoints use JSON (all OCS endpoins return JSON when appending a `format=json` query parameter) + +== Links diff --git a/modules/developer/pages/ocis/adr/0028-activities-service.adoc b/modules/developer/pages/ocis/adr/0028-activities-service.adoc new file mode 100644 index 00000000..666b1750 --- /dev/null +++ b/modules/developer/pages/ocis/adr/0028-activities-service.adoc @@ -0,0 +1,101 @@ += 28. Activity Service +:toc: right +:toclevels: 3 + +* Status: draft +* Deciders: link:https://github.com/kobergj[@kobergj], link:https://github.com/fschade[@fschade] +* Date: 2024-05-16 + +== Context and Problem Statement + +The user should be able to see all activities for a resource. +Besides the current resource, the user should also be able to decide if he wants to include child resource activities or not. + +== Decision Drivers + +* The user should be able to see all activities for a resource. +* The user should be able to decide if he wants to include child resource activities. +* Activities should be stored space efficiently. +* Activities should be stored in a way that they can be queried efficiently. +* Activities should stay in place even if the resource is gone. +* Activities reflect the state at a given point in time and not the current state. +* The Service should only store a configurable number of activities per resource. + +== Considered Options + +=== Activity store + +* Use a go-micro store to store the individual activities. +* Use a time series database to store the activities. +* Use a graph database to store the activities. +* Use a relational database to store the activities. +* Use the file system to store the activities. + +=== Activity format + +* Normalize the activities before storing them. +* Only store relevant data to get the related event from the event-history service when needed, e.g., + ```go + package pseudo_code + + import ( + "time" + ) + + type Activity struct { + ResourceId string + EventID string + Depth int64 + Timestamp time.Time + } + ``` +* Store the activity in a human-readable way e.g. "resource A has been shared with user B." +* Store each activity only on the resource itself. +* Store each activity only on the resource itself and all its parents. + +== Decision Outcome + +* Activity store: + * Use a go-micro store to store the individual activities. +* Activity format: + * Store each activity only on the resource itself and all its parents. + * Only store event ids and get the related event from the event-history service when needed. + +=== Positive Consequences: + +* Activity store (go-micro store): + * Reuse existing technology. + * We can use nats-js-kv store which already proved reliable in production. + * No need to introduce any kind of new technology, e.g., a time series database, a relational database. +* Activity Format: + * Having each activity stored on each resource (the resource itself and its parents) + makes it easy to retrieve the timeline of activities for a resource and its children. + * Only storing the event id and getting the related event from the event-history we benefit + from the event-history services capabilities to store and query events. + * Walking the resource tree from the resource to the root is a linear operation and can be done efficiently. + +=== Negative Consequences: + +* Activity store: + * Other database types might be more efficient for storing activities. + * Using the go-micro-store only allows storing the activity in a key-value format. +* Activity Format: + * Storing only the event ids and getting the related data from the event-history service when needed + might introduce additional latency when querying activities. + * Adding each event-id to each resource parent leads to a lot of duplicated data. + +== Pros and Cons of the Options + +* Activity store: + * (PRO) Introducing a new database type might be more efficient for storing activities. + * (CON) Introducing a new database type brings extra complexity and maintenance overhead. + * (CON) Using the file system to store the activities might be inefficient and could be problematic especially in a distributed environment. +* Activity format: + * (PRO) Normalizing the activities before storing them might make it easier and more efficient to query them. + * (PRO) Storing each activity only on the resource itself is more space-efficient. + * (CON) Storing each activity only on the resource itself increases the complexity of querying activities. + * (CON) Storing each activity in a human-readable format is not space-efficient. + +== Links + +* link:https://github.com/owncloud/ocis/issues/8881[Story] diff --git a/modules/developer/pages/ocis/adr/index.adoc b/modules/developer/pages/ocis/adr/index.adoc new file mode 100644 index 00000000..b5182390 --- /dev/null +++ b/modules/developer/pages/ocis/adr/index.adoc @@ -0,0 +1,8 @@ += Architecture Decisions +:toc: right +:toclevels: 3 + +oCIS is documenting architecture decisions using link:https://adr.github.io/madr/[Markdown Architectural Decision Records] (MADR), following link:https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions[Documenting Architecture Decisions by Michael Nygard]. + + +To manage the records we use link:https://github.com/butonic/adr-tools[butonic/adr-tools], a fork of the original link:https://github.com/npryce/adr-tools[npryce/adr-tools], based on link:https://github.com/npryce/adr-tools/pull/43[a pull request that should have added MADR support]. It also supports a YAML header that is used by our Hugo based doc generation \ No newline at end of file diff --git a/modules/developer/pages/ocis/backup.adoc b/modules/developer/pages/ocis/backup.adoc new file mode 100644 index 00000000..92ef68e5 --- /dev/null +++ b/modules/developer/pages/ocis/backup.adoc @@ -0,0 +1,235 @@ += Backup Considerations +:toc: right +:toclevels: 3 + +This small guide aims to shed some light on the internal Infinite Scale data structure. You can refer to it when you are trying to optimize your backups or if you are just curious about how Infinite Scale stores its data. + +Note, as a prerequisite backing up Infinite Scale, the instance has to be fully shut down for the time being. + +== Ocis Data Structure + +Ocis stores its data in a folder that can be configured via the environment variable `OCIS_BASE_DATA_PATH`. Without further configuration, services derive from that path when they store data, though individual settings for certain data types can be configured. + +The default value for the `OCIS_BASE_DATA_PATH` variable is `$HOME/.ocis` (or `/var/lib/ocis` when using the docker container. Note: Configuration data is by default stored in `/etc/ocis/` in the container.). + +Inside this folder, Infinite Scale will store all its data in separate subdirectories. That includes metadata, configurations, queues and stores etc. The actual bytes of files (blobs) are handled by a so called blobstore, which also stores here by default. Depending on the used blobstore, the blobs need to be backed up separately, for example if S3 is used. Note: See special case for the `config` folder in a docker container. + +=== Base Data Path Overview + +Listing the contents of the folder will return the following: +[source,bash] +---- + ~/.ocis/:tree -L 1 +. +├── config +├── idm +├── idp +├── nats +├── proxy +├── search +├── storage +├── thumbnails +└── web + +10 directories, 0 files +---- + +The following sections describe the content and background of the subdirectories to decide if a backup is required or recommended and its effect when it is not backed up. + +=== `config` + +Contains basic Infinite Scale configuration created by `ocis init`(Note: The location of the configuration folder can be specified with the `OCIS_CONFIG_DIR` environment variable, but for this document we will assume this variable is not set and the default is used.) + +[source,bash] +---- + ~/.ocis/config/:tree +. +└── ocis.yaml + +1 directory, 1 file +---- + +* `ocis.yaml`:\ +BACKUP RECOMMENDED. Holds Infinite Scale configuration data. The contents can vary depending on your environment variables. In general, most of this file can be recreated again by running `ocis init`. This will recreate secrets and certificates. However, if not backed up completely, some fields MUST be copied over from the old config manually to regain data access after a restore: + +| Field Name | Envvar Name | Description | If not backed up | +| --- | --- | --- | --- | +| `idp.ldap.bind_password` | `OCIS_LDAP_BIND_PASSWORD` | Password for the idp | no logins possible | +| `idm.service_user_passwords.idp_password`| `IDM_IDPSVC_PASSWORD` | Same as above | no logins possible | +| `system_user_id` | `OCIS_SYSTEM_USER_ID` | The id of storage-system user | no logins possible | +| `idm.service_user_passwords.reva_password`| `IDM_REVASVC_PASSWORD` | The reva password | no logins possible | +| `auth_basic.auth_providers.ldap.bind_password` | `AUTH_BASIC_LDAP_BIND_PASSWORD` | Same as above | no logins possible | +| `users.drivers.ldap.bind_password` | `USERS_LDAP_BIND_PASSWORD` | Same as above | no logins possible | +| `groups.drivers.ldap.bind_password` | `GROUPS_LDAP_BIND_PASSWORD` | Same as above | no logins possible | +| `storage_users.mount_id` | `STORAGE_USERS_MOUNT_ID` | The mountid of the storage_users service | sharing data lost | +| `gateway.storage_registry.storage_users_mount_id` | `GATEWAY_STORAGE_USERS_MOUNT_ID` | Same as above | sharing data lost | + +=== `idm` + +Note: This folder will not appear if you use an external idm. Refer to your idms documentation for backup details in this case. + +Contains the data for the internal Infinite Scale identity management. See the xref:../services/idm/index.adoc[IDM README] for more details. + +[source,bash] +---- + ~/.ocis/idm/:tree +. +├── ldap.crt +├── ldap.key +└── ocis.boltdb + +1 directory, 3 files +---- + +* `ocis.boltdb`:\ +BACKUP REQUIRED. This is the boltdb database that stores user data. Use `IDM_DATABASE_PATH` to specify its path. If not backed up, Infinite Scale will have no users, therefore also all data is lost. +* `ldap.crt`:\ +BACKUP OPTIONAL. This is the certificate for the idm. Use `IDM_LDAPS_CERT` to specify its path. Will be auto-generated if not backed up. +* `ldap.key`:\ +BACKUP OPTIONAL. This is the certificate key for the idm. Use `IDM_LDAPS_KEY` to specify its path. Will be auto-generated if not backed up. + + +=== `idp` + +Note: This folder will not appear if you use an external idp. Refer to your idp's documentation for backup details in this case. + +Contains the data for the internal Infinite Scale identity provider. See the xref:../services/idp/index.adoc[IDP README] for more details. + +[source,bash] +---- + ~/.ocis/idp/:tree +. +├── encryption.key +├── private-key.pem +└── tmp + └── identifier-registration.yaml + +2 directories, 3 files +---- + +* `encryption.key`:\ +BACKUP RECOMMENDED. This is the encryption secret. Use `IDP_ENCRYPTION_SECRET_FILE` to specify its paths. Not backing this up will force users to relogin. +* `private-key.pem`:\ +BACKUP RECOMMENDED. This is the encryption key. Use `IDP_SIGNING_PRIVATE_KEY_FILES` to specify its paths. Not backing this up will force users to relogin. +* `identifier-registration.yml`:\ +BACKUP OPTIONAL. It holds configuration for oidc clients (web, desktop, ios, android). Will be recreated if not backed up. + +=== `nats` + +Note: This folder will not appear if you use an external nats installation. In that case, data has to secured in alignment with the external installation. + +Contains nats data for streams and stores. See the xref:../services/nats/index.adoc[NATS README] for more details. + +[source,bash] +---- + ~/.ocis/nats/:tree -L 1 +. +└── jetstream + +---- + +* `jetstream`:\ +BACKUP RECOMMENDED. This folder contains nats data about streams and key-value stores. Use `NATS_NATS_STORE_DIR` to specify its path. Not backing it up can break history for multiple (non-vital) features such as history or notifications. The Infinite Scale functionality is not impacted if omitted. + +=== `proxy` + +Contains proxy service data. See the xref:../services/proxy/index.adoc[PROXY README] for more details. + +[source,bash] +---- + ~/.ocis/proxy/:tree +. +├── server.crt +└── server.key + +1 directory, 2 files +---- + +* `server.crt`:\ +BACKUP OPTIONAL. This is the certificate for the http services. Use `PROXY_TRANSPORT_TLS_CERT` to specify its path. +* `server.key`:\ +BACKUP OPTIONAL. This is the certificate key for the http services. Use `PROXY_TRANSPORT_TLS_KEY` to specify its path. + +=== `search` + +Contains the search index. See the xref:../services/search/index.adoc[SEARCH README] for more details. + +[source,bash] +---- + ~/.ocis/search/:tree -L 1 +. +└── bleve + +2 directories, 0 files +---- + +* `bleve`:\ +BACKUP RECOMMENDED/OPTIONAL. This contains the search index. Can be specified via `SEARCH_ENGINE_BLEVE_DATA_PATH`. If not backed up, the search index needs to be recreated. This can take a long time depending on the amount of files. + +=== `storage` + +Contains Infinite Scale meta (and blob) data, depending on the blobstore. See the xref:../services/storage-users/index.adoc[STORAGE-USERS README] for more details. + +[source,bash] +---- + ~/.ocis/storage/:tree -L 1 +. +├── metadata +├── ocm +└── users + +4 directories, 0 files +---- + +* `metadata`:\ +BACKUP REQUIRED. Contains system data. Path can be specified via `STORAGE_SYSTEM_OCIS_ROOT`. Not backing it up will remove shares from the system and will also remove custom settings. +* `ocm`:\ +BACKUP REQUIRED/OMITABLE. Contains ocm share data. When not using ocm sharing, this folder does not need to be backed up. +* `users`:\ +BACKUP REQUIRED. Contains user data. Path can be specified via `STORAGE_USERS_OCIS_ROOT`. Not backing it up will remove all spaces and all files. As result, you will have a configured but empty Infinite Scale instance, which is fully functional accepting new data. Old data is lost. + +=== `thumbnails` + +Contains thumbnails data. See the xref:../services/thumbnails/index.adoc[THUMBNAILS README] for more details. + +[source,bash] +---- + ~/.ocis/thumbnails/:tree -L 1 +. +└── files +---- + +* `files`:\ +OPTIONAL/RECOMMENDED. This folder contains prerendered thumbnails. Can be specified via `THUMBNAILS_FILESYSTEMSTORAGE_ROOT`. If not backed up, thumbnails will be regenerated automatically on access which leads to some load on the thumbnails service. + +=== `web` + +Contains web assets such as custom logos, themes etc. See the xref:../services/web/index.adoc[WEB README] for more details. + +[source,bash] +---- + ~/.ocis/web/:tree -L 1 +. +└── assets + +2 directories, 0 files +---- + +* `assets`:\ +BACKUP RECOMMENDED/OMITABLE. This folder contains custom web assets. Can be specified via `WEB_ASSET_CORE_PATH`. If no custom web assets are used, there is no need for a backup. If those exist but are not backed up, they need to be reuploaded. + +=== `external services` + +When using an external idp/idm/nats or blobstore, its data needs to be backed up separately. Refer to your idp/idm/nats/blobstore documentation for backup details. + +== Backup Consistency Command + +Infinite Scale now allows checking an existing backup for consistency. Use the command: +[source,bash] +---- +ocis backup consistency -p "" +---- + +`path-to-base-folder` needs to be replaced with the path to the storage providers base path. Should be same as the `STORAGE_USERS_OCIS_ROOT` + +Use the `-b s3ng` option when using an external (s3) blobstore. Note: When using this flag, the path to the blobstore must be configured via envvars or a yaml file to match the configuration of the original instance. Consistency checks for other blobstores than `ocis` and `s3ng` are not supported at the moment. diff --git a/modules/developer/pages/ocis/build-docs.adoc b/modules/developer/pages/ocis/build-docs.adoc new file mode 100644 index 00000000..a2f9ee40 --- /dev/null +++ b/modules/developer/pages/ocis/build-docs.adoc @@ -0,0 +1,79 @@ += Documentation +:toc: right +:toclevels: 3 + + +== Overview + +The documentation: + +* Is based on the link:https://gohugo.io/[HUGO] framework written in Go. The underlaying language is markdown plus the extensions provided by hugo. +* The framwork including the theme used is maintained in the following link:https://github.com/owncloud/owncloud.github.io[repo]. +* The hugo version, currently at `0.129.0` must match the image version defined in the root makefile and the framework repo, defined in `.drone.yml`.\ +[NOTE] +==== +* Building the theme is part of the loaded framework while building the documentation is done using a separate container. + +* Any intended upgrade of the hugo version implies a check if the theme builds correctly and must be applied to all doc sourcing repos. On errors, stick on the current hugo version or update the theme accordingly! +==== + + +* The documentation sources are distributed over several repos such as `ocis`, `web` and others. When building, these sources act individually and only those sources are build and pushed to the web that come from the respective repo. The definition what is going where is defined in the link:https://github.com/owncloud/owncloud.github.io/blob/main/.batchfile[framework repo]. This means, each merge of a docs change in one of the sourcing repos only pushes that change to the site. There is no global build process and inter doc links must be treated therefore very carefully ! + +* The documentation setup is, with only a few exceptions, identical for both the `ocis` and the `web` repo. This means, for example, running commands locally are the same, the documentation process section below applies for both; only drone uses some specifics. + +* {{< hint info >}} +Any files or folders that need to be excluded from the build process must be defined in the referenced framework repository above, in the file `config.yaml`. +// SHORTCODE: {{< /hint >}} + +== Documentation Process + +When building the ocis documentation, the process differes slightly when building locally or when merging. Though both processes generate they docs, the end up in different locations. + +* _General_\ +Beside building docs, some additional files/folders will be copied required for the process, see the makefile for details. + +* _Locally_\ +The rendered build stays in the branch that is currently active and is NOT part of a git change, though additional changed files show up and may need to be committed. The build can be viewed normally, for details see below. + +* _Merge_\ +When a docs PR gets merged, the content of the `docs/` folder gets "copied". For master the target is the `docs` branch or a respective `docs-stable-` branch otherwise. The latter must be configured on new stable versions. The docs branch is the source of truth for the developer documentation. + +== Building the Documentation + +[NOTE] +==== +* For building the documentation locally, you need to have `make`, `docker` and `go` installed.\ + See the link:https://github.com/owncloud/ocis/#use-the-ocis-repo-as-source[oCIS README] for more details and minimum versions required. +* Regulary, at least once after cloning, branching or rebasing, run `make generate` to update the assets. This ensures that all data that is accessed by the docs build process is up-to-date. +* Run the make commands from the oCIS root, the makefile contains all required targets.\ +Use `make help` for more details. +==== + + +* _For any changes_, you must fully build the documentation first to preview it.\ +For ease of working, open two terminals and change to the oCIS root. The terminals are numberd as (1) and (2). + * At first start, from any terminal, run `sudo make docs-clean` to have a clean build environment.\ + sudo is required for deletion, because files generated by hugo, which runs in a container, are created by root. + * Do all the doc changes required in the docs repository. + * Run (1) `make docs-local`. This will run all steps locally as drone would do but without branch copying or pushing to the web. When omitting this step and just serving content, the documentation might build locally but may fail when creating a PR. + * Run (2) `make docs-serve` to serve the documentation. You can now view the built result in the browser. + * On any more changes made, run (1) `make docs-copy`. The changes will be built on the fly via the running docs-serve command. + +* _To preview the documentation locally_ without changing content, run the following commands in order.\ +This builds the documentation without artifacts and makes it available on link:http://localhost:1313[localhost:1313]. + * `sudo make docs-clean` + * `make docs-local` + * `make docs-serve` + +== Remarks + +[NOTE] +==== +Note that you may get files changed reported like `env_vars.yaml` by just running the make docs commands. You can discard any of these as they come from an internal helper process step necessary. To take care on those changed files, see the xref:../services/general-info/envvars/.adoc[Environment Variables] documentation. +==== + + +Please keep this documentation in sync with the oCIS source code of the master branch. + +Changes on the documentation are automatically applied to xref:../.adoc[this site] when merged to the master branch. diff --git a/modules/developer/pages/ocis/config.adoc b/modules/developer/pages/ocis/config.adoc new file mode 100644 index 00000000..c5508991 --- /dev/null +++ b/modules/developer/pages/ocis/config.adoc @@ -0,0 +1,174 @@ += Configuration +:toc: right +:toclevels: 3 + + +== Configuration Framework + +In order to simplify deployments and development the configuration model from oCIS aims to be simple yet flexible. + +== Overview of the approach + +image::ocis/ocis-config-redesign.drawio.svg[] + +== In-depth configuration + +Since we include a set of predefined extensions within the single binary, configuring an extension can be done in a variety of ways. Since we work with complex types, having as many cli per config value scales poorly, so we limited the options to config files and environment variables. + +The hierarchy is clear enough, leaving us with: + +_(each element above overwrites its precedent)_ + +1. env variables +2. extension config +3. ocis config + +This is manifested in the previous diagram. We can then speak about "configuration file arithmetics", where resulting config transformations happen through a series of steps. An administrator must be aware of these sources, since mis-managing them can be a source of confusion, having undesired transformations on config files believed not to be applied. + +== Flows + +Let's explore the various flows with examples and workflows. + +=== Examples + +Let's explore with examples this approach. + +==== Expected loading locations + +- docker images: `/etc/ocis/` +- binary releases: `$HOME/.ocis/config/` + +followed by the `.yaml`, e.g. `proxy.yaml` for the extension configuration. You also can put an `ocis.yaml` config file to the expected loading location to use a single config file. + +You can set another directory as config path in the environment variable `OCIS_CONFIG_DIR`. It will then pick the same file names, but from the folder you configured. + +==== Only config files + +The following config files are present in the default loading locations: + +_ocis.yaml_ +[source,yaml] +---- +proxy: + http: + addr: localhost:1111 + log: + pretty: false + color: false + level: info +log: + pretty: true + color: true + level: info +---- + +_proxy.yaml_ +[source,yaml] +---- +http: + addr: localhost:3333 +---- + +Note that the extension files will overwrite values from the main `ocis.yaml`, causing `ocis server` to run with the following configuration: + +[source,yaml] +---- +proxy: + http: + addr: localhost:3333 +log: + pretty: true + color: true + level: info +---- + +==== Using ENV variables + +The logging configuration if defined in the main ocis.yaml is inherited by all extensions. It can be, however, overwritten by a single extension file if desired. The same example can be used to demonstrate environment values overwrites. With the same set of config files now we have the following command `PROXY_HTTP_ADDR=localhost:5555 ocis server`, now the resulting config looks like: + +[source,yaml] +---- +proxy: + http: + addr: localhost:5555 +log: + pretty: true + color: true + level: info +---- + +==== Substitute ENV variables in configuration files + +Environment variables can be used in the configurations files and will be replaced by oCIS when loading these. +Default values can be specified after a `|` character - see below. + +[source,yaml] +---- +proxy: + http: + addr: ${PROXY_HTTP_ADDR|localhost:4321} +log: + pretty: true + color: true + level: info +---- + +=== Workflows + +Since one can run an extension using the runtime (supervised) or not (unsupervised), we ensure correct behavior in both modes, expecting the same outputs. + +==== Supervised + +You are using the supervised mode whenever you issue the `ocis server` command. We start the runtime on port `9250` (by default) that listens for commands regarding the lifecycle of the supervised extensions. When an extension runs supervised and is killed, the only way to provide / overwrite configuration values will be through an extension config file. This is due to the parent process has already started, and it already has its own environment. + +==== Unsupervised + +All the points from the priority section hold true. An unsupervised extension can be started with the format: `ocis [extension]` i.e: `ocis proxy`. First, `ocis.yaml` is parsed, then `proxy.yaml` followed by environment variables. + +== Shared Values + +When running in supervised mode (`ocis server`) it is beneficial to have common values for logging, so that the log output is correctly formatted, or everything is piped to the same file without duplicating config keys and values all over the place. This is possible using the global `log` config key: + +_ocis.yaml_ +[source,yaml] +---- +log: + level: error + color: true + pretty: true + file: /var/tmp/ocis_output.log +---- + +There is, however, the option for extensions to overwrite this global values by declaring their own logging directives: + +_ocis.yaml_ +[source,yaml] +---- +log: + level: info + color: false + pretty: false +---- + +One can go as far as to make the case of an extension overwriting its shared logging config that received from the main `ocis.yaml` file. Because things can get out of hands pretty fast we recommend not mixing logging configuration values and either use the same global logging values for all extensions. + +[WARNING] +==== +When overwriting a globally shared logging values, one _MUST_ specify all values. +==== + + +=== Log config keys + +[source,yaml] +---- +log: + level: [ error | warning | info | debug ] + color: [ true | false ] + pretty: [ true | false ] + file: [ path/to/log/file ] # MUST not be used with pretty = true +---- + +== Default config values (in yaml) + +TBD. Needs to be generated and merged with the env mappings. diff --git a/modules/developer/pages/ocis/deployment/basic-remote-setup.adoc b/modules/developer/pages/ocis/deployment/basic-remote-setup.adoc new file mode 100644 index 00000000..f3d9e2fd --- /dev/null +++ b/modules/developer/pages/ocis/deployment/basic-remote-setup.adoc @@ -0,0 +1,50 @@ += Basic Remote Setup +:toc: right +:toclevels: 3 + + +The default configuration of the oCIS binary and the `owncloud/ocis` docker image assume, that you access oCIS on `localhost`. This enables you to do quick testing and development without any configuration. + +If you need to access oCIS running in a docker container, on a VM or a remote machine via another hostname than `localhost`, you need to configure this hostname in oCIS. The same applies if you are not using hostnames but instead an IP (e.g. `192.168.178.25`). + +== Start the oCIS fullstack server from binary + +Initialize the oCIS configuration by running `./bin/ocis init`. + +Upon first start of the oCIS fullstack server with `./bin/ocis server` it will generate a directory tree skeleton in `$HOME/.ocis`. If that is already existing it will not be overwritten as it contains all relevant data for oCIS. + +For the following examples you need to have the oCIS binary in your current working directory, we assume it is named `ocis` and it needs to be marked as executable. See xref:../getting-started/#binaries.adoc[Getting Started] for where to get the binary from. + +=== Using automatically generated certificates + +In order to run oCIS with automatically generated and self-signed certificates please execute following command. You need to replace `your-host` with an IP or hostname. Since you have only self-signed certificates you need to have `OCIS_INSECURE` set to `true`. + +[source,bash] +---- +OCIS_INSECURE=true \ +PROXY_HTTP_ADDR=0.0.0.0:9200 \ +OCIS_URL=https://your-host:9200 \ +./ocis server +---- + +=== Using already present certificates + +If you have your own certificates already in place, you may want to make oCIS use them: + +[source,bash] +---- +OCIS_INSECURE=false \ +PROXY_HTTP_ADDR=0.0.0.0:9200 \ +OCIS_URL=https://your-host:9200 \ +PROXY_TRANSPORT_TLS_KEY=./certs/your-host.key \ +PROXY_TRANSPORT_TLS_CERT=./certs/your-host.crt \ +./ocis server +---- + +If you generated these certificates on your own, you might need to set `OCIS_INSECURE` to `true`. + +For more configuration options check the configuration section in xref:../config.adoc[oCIS] and the oCIS services. + +== Start the oCIS fullstack server with Docker Compose + +Please have a look at our other xref:./.adoc[deployment examples]. diff --git a/modules/developer/pages/ocis/deployment/bridge.adoc b/modules/developer/pages/ocis/deployment/bridge.adoc new file mode 100644 index 00000000..e8213ce2 --- /dev/null +++ b/modules/developer/pages/ocis/deployment/bridge.adoc @@ -0,0 +1,339 @@ += Bridge +:toc: right +:toclevels: 3 + + +We are planning to build a bridge from ownCloud 10 to ocis. The idea is to have a reverse proxy in front of ownCloud 10 that will forward requests to ownCloud 10 or ocis-reva, depending on the migration status of the logged-in user. + +This document is a work in progress of the current setup. + +== Current status + +Using ocis and the ownCloud 10 link:https://github.com/owncloud/graphapi/[graphapi app] it is possible today to use an existing owncloud 10 instance as a user backend and storage backend for ocis. + +== How to do it + +=== Install the owncloud 10 graphapi app + +In an owncloud 10 apps folder +---- +$ git clone git@github.com:owncloud/graphapi.git +$ cd graphapi +$ composer install +---- + +=== Enable the graphapi app + +---- +occ a:e graphapi +---- + +No configuration necessary. You can test with `curl`: +[source,console] +---- +$ curl https://cloud.ocis.test/index.php/apps/graphapi/v1.0/users -u admin -s | jq +Enter host password for user 'admin': +{ + "value": [ + { + "id": "admin", + "displayName": "admin", + "mail": null + }, + { + "id": "demo", + "displayName": "Demo", + "mail": null + }, + ... + + ], + "@odata.nextLink": "https://cloud.ocis.test/apps/graphapi/v1.0/users?$top=10&$skip=10" +} +---- + +// SHORTCODE: {{< hint >}} +The MS graph api actually asks for `Bearer` auth, but in order to check users passwords during an LDAP bind we are exploiting ownClouds authentication implementation that will grant access when `Basic` auth is used. An LDAP Bind you may ask? Read on! +// SHORTCODE: {{< /hint >}} + +=== Grab ocis! + +---- +$ git clone git@github.com:owncloud/ocis.git +$ cd ocis +$ make -C ocis build +---- +This should give you an `ocis/bin/ocis` binary. Try listing the help with `ocis/bin/ocis --help`. + +// SHORTCODE: {{< hint >}} +You can check out a custom branch and build a custom binary which can then be used for the steps below. +// SHORTCODE: {{< /hint >}} + +=== Start ocis glauth + +We are going to use the built binary and ownCloud 10 graphapi app to turn ownCloud 10 into the datastore for an LDAP proxy. + +==== configure it + +While ocis can be configured using environment variables, e.g. for a docker compose setup we are going to use a more traditional config file here. +Create a config file for ocis in either `/etc/ocis`, `$HOME/.ocis` or `./.config`. You can use `.json`, `.yaml` or `.toml`. I will use toml here, because ... reasons. + +[source,toml] +---- +[glauth.backend] +datastore = "owncloud" # switch to the owncloud datastore +servers = ["https://cloud.ocis.test/apps/graphapi/v1.0"] # the graph api endpoint to connect to +basedn = "dc=ocis,dc=test" # base dn to construct the LDAP dn. The user `admin` will become `cn=admin,dc=ocis,dc=test` +---- + +// SHORTCODE: {{< hint >}} +There is a bug in the config merging for environment variables, cli flags and config files causing log settings not to be picked up from the config file when specifying `--extensions`. That is why I will +* configure most of the config in a file, +* adjust logging using `OCIS_LOG_*` environment variables and +* specify which extension to run using `ocis/bin/ocis server --extensions "comma, separated, list, of, extensions"`. +// SHORTCODE: {{< /hint >}} + +==== Run it! + +For now, we only start the glauth extension: +[source,console] +---- +$ OCIS_LOG_PRETTY=true OCIS_LOG_COLOR=true ocis/bin/ocis server --extensions "glauth" +---- + +==== Check it is up and running + +You should now be able to list accounts from your ownCloud 10 oc_accounts table using: +[source,console] +---- +$ ldapsearch -x -H ldap://127.0.0.1:9125 -b dc=ocis,dc=test -D "cn=admin,dc=ocis,dc=test" -W '(objectclass=posixaccount)' +---- + +Groups should work as well: +[source,console] +---- +$ ldapsearch -x -H ldap://127.0.0.1:9125 -b dc=ocis,dc=test -D "cn=admin,dc=ocis,dc=test" -W '(objectclass=posixgroup)' +---- + +// SHORTCODE: {{< hint >}} +This is currently a readonly implementation and minimal to the use-case of authenticating users with an IDP. +// SHORTCODE: {{< /hint >}} + +=== Start ocis storage-gateway, storage-authbasic and storage-userprovider + +We are going to set up reva to authenticate users against our glauth LDAP proxy. This allows us to log in and use the reva cli. The ocis storage-gateway starts the reva gateway which will authenticate basic auth requests using the storage-authbasic service. Furthermore, users have to be available in the storage-userprovider to retrieve displayname, email address and other user metadata. + +To configure LDAP to use our glauth we add this section to the config file: + +[source,toml] +---- +[storage.reva.ldap] +idp = "https://ocis.ocis.test" +basedn = "dc=ocis,dc=test" +binddn = "cn=admin,dc=ocis,dc=test" # an admin user in your oc10 +bindpassword = "secret" +userschema = { uid = "uid", displayname = "givenname" } # TODO make glauth return an ownclouduuid and displayname attribute +---- + +Now we can start all necessary services. + +[source,console] +---- +$ OCIS_LOG_PRETTY=true OCIS_LOG_COLOR=true ocis/bin/ocis server --extensions "glauth, storage-gateway, storage-authbasic, storage-userprovider" +---- + + +[WARNING] +==== +Here I ran out of time. I tried to verify this step with the reva cli: +`cmd/reva/reva -insecure -host localhost:9142` +`login basic` +but it tries to create the user home, which cannot be disabled in a config file: https://github.com/owncloud/ocis/issues/2416#issuecomment-901197053 + +starting `STORAGE_GATEWAY_DISABLE_HOME_CREATION_ON_LOGIN=true OCIS_LOG_LEVEL=debug OCIS_LOG_PRETTY=true OCIS_LOG_COLOR=true ocis/bin/ocis server --extensions "storage-gateway, storage-authbasic, storage-userprovider"` let me login: + +[source,console] +---- +✗ cmd/reva/reva -insecure -host localhost:9142 +reva-cli v1.11.0-27-g95b1f2ee (rev-95b1f2ee) +Please use `exit` or `Ctrl-D` to exit this program. +>> login basic +username: jfd +password: OK +>> whoami +id: username:"jfd" mail:"jfd@butonic.de" display_name:"J\303\266rn" uid_number:99 gid_number:99 +>> exit +---- + +I hope https://github.com/owncloud/ocis/pull/2024 fixes the parsing order of things. + +everything below this is outdated + +... gotta run +==== + + + +=== Start ocis storage-userprovider + +[source,console] +---- +ocis/bin/ocis storage-userprovider --ldap-port 19126 --ldap-user-schema-uid uid --ldap-user-schema-displayName givenName --addr :19144 +---- + +TODO clone `git clone git@github.com:cs3org/cs3apis.git` + +query users using link:https://github.com/fullstorydev/grpcurl[grpcurl] +[source,console] +---- +grpcurl -import-path ./cs3apis/ -proto ./cs3apis/cs3/identity/user/v1beta1/user_api.proto -plaintext localhost:19144 cs3.identity.user.v1beta1.UserAPI/FindUsers +ERROR: + Code: Unauthenticated + Message: auth: core access token not found +---- + + +=== Start ocis idp + +==== Set environment variables + +The built-in link:https://github.com/libregraph/lico[libregraph/lico] needs environment variables to configure the LDAP server: +[source,console] +---- +export OCIS_URL=https://ocis.ocis.test +export IDP_LDAP_URI=ldap://127.0.0.1:9125 +export IDP_LDAP_BASE_DN="dc=ocis,dc=test" +export IDP_LDAP_BIND_DN="cn=admin,dc=ocis,dc=test" +export IDP_LDAP_BIND_PASSWORD="its-a-secret" +export IDP_LDAP_SCOPE=sub +export IDP_LDAP_LOGIN_ATTRIBUTE=uid +export IDP_LDAP_NAME_ATTRIBUTE=givenName +---- +Don't forget to use an existing user with admin permissions (only admins are allowed to list all users via the graph api) and the correct password. + +==== Run it! + +You can now bring up `ocis/bin/ocis idp` with: +[source,console] +---- +$ ocis/bin/ocis idp server --iss http://127.0.0.1:9130 --signing-kid gen1-2020-02-27 +---- + +`ocis/bin/ocis idp` needs to know +- `--iss http://127.0.0.1:9130` the issuer, which must be a reachable http endpoint. For testing an ip works. For openid connect HTTPS is NOT optional. This URL is exposed in the `http://127.0.0.1:9130/.well-known/openid-configuration` endpoint and clients need to be able to connect to it, securely. We will change this when introducing the proxy. +- `--signing-kid gen1-2020-02-27` a signature key id, otherwise the jwks key has no name, which might cause problems with clients. a random key is ok, but it should change when the actual signing key changes. + +[WARNING] +==== +* TODO: the port in the `--iss` needs to be changed when hiding the idp behind the proxy +* TODO: the signing keys and encryption keys should be precreated so they are reused between restarts. Otherwise all client sessions will become invalid when restarting the IdP. +==== + + + +==== Check it is up and running + +1. Try getting the configuration: +[source,console] +---- +$ curl http://127.0.0.1:9130/.well-known/openid-configuration +---- + +2. Check if the login works at http://127.0.0.1:9130/signin/v1/identifier + +// SHORTCODE: {{< hint >}} +If you later get a `Unable to find a key for (algorithm, kid):PS256, )` Error make sure you did set a `--signing-kid` when starting `ocis/bin/ocis idp` by checking it is present in http://127.0.0.1:9130/konnect/v1/jwks.json +// SHORTCODE: {{< /hint >}} + +=== Start ocis proxy + + +// SHORTCODE: {{< hint >}} +Everything below this hint is outdated. Next steps are roughly: +* directly after glauth start the `ocis storage-userporvider`? + - how to verify that works? + - https://github.com/fullstorydev/grpcurl +* start proxy + - the ocis ipd url can be changed to https + - when do we hide oc10 behind ocis? -> advanced bridge at the end? for now run it without touching the existing oc10 instance +* start web + - verify the login works, but how? + - TODO the login works, but then the capabilities requests will fail ... unless we make the proxy answer them by talking to oc10? + +Other ideas: +* the owncloud backend in glauth also works with the user provisioning api ... no changes to a running production instance? db access could be done with a read only account as well... +// SHORTCODE: {{< /hint >}} + + +=== Start ocis-web + +==== Run it! + +Point `ocis-web` to your owncloud domain and tell it where to find the openid connect issuing authority: +[source,console] +---- +$ bin/web server --web-config-server https://cloud.example.com --oidc-authority https://192.168.1.100:9130 --oidc-metadata-url https://192.168.1.100:9130/.well-known/openid-configuration --oidc-client-id ocis +---- + +`ocis-web` needs to know +- `--web-config-server https://cloud.example.com` is ownCloud url with webdav and ocs endpoints (oc10 or ocis) +- `--oidc-authority https://192.168.1.100:9130` the openid connect issuing authority, in our case `oidc-idp`, running on port 9130 +- `--oidc-metadata-url https://192.168.1.100:9130/.well-known/openid-configuration` the openid connect configuration endpoint, typically the issuer host with `.well-known/openid-configuration`, but there are cases when another endpoint is used, e.g. ping identity provides multiple endpoints to separate domains +- `--oidc-client-id ocis` the client id we will register later with `ocis-idp` in idp OIDC client settings + +=== Patch owncloud + +While the UserSession in ownCloud 10 is currently used to test all available IAuthModule implementations, it immediately logs out the user when an exception occurs. However, existing owncloud 10 instances use the oauth2 app to create Bearer tokens for mobile and desktop clients. + +To give the openidconnect app a chance to verify the tokens we need to change the code a bit. See https://github.com/owncloud/core/pull/37043 for a possible solution. + +[quote] +____ +Note: The PR is hot ... as in _younger than this list of steps_. And it messes with authentication. Use with caution. +____ + + +=== Install the owncloud 10 openidconnect app + +In an owncloud 10 apps folder +---- +$ git clone git@github.com:owncloud/openidconnect.git +$ cd openidconnect +$ composer install +---- + +After enabling the app configure it in `config/oidc.config.php` + +[source,php] +---- +$CONFIG = [ + 'openid-connect' => [ + 'provider-url' => 'https://192.168.1.100:9130', + 'client-id' => 'ocis', + 'loginButtonName' => 'OpenId Connect @ Konnectd', + ], + 'debug' => true, // if using self-signed certificates + // allow the different domains access to the ocs and webdav endpoints: + 'cors.allowed-domains' => [ + 'https://cloud.example.com', + 'http://localhost:9100', + ], +]; +---- + +In the above configuration replace +- `provider-url` with the URL to your `ocis-idp` issuer +- `https://cloud.example.com` with the URL to your ownCloud 10 instance +- `http://localhost:9100` with the URL to your ownCloud Web instance + +[quote] +____ +Note: By default the openidconnect app will use the email of the user to match the user from the oidc userinfo endpoint with the ownCloud account. So make sure your users have a unique primary email. +____ + + +== Next steps + +Aside from the above todos these are the next steps +- tie it all together behind `ocis-proxy` +- create an `ocis bridge` command that runs all the ocis services in one step with a properly preconfigured idp OIDC client `ocis-idp` for `ownCloud Web` and the owncloud 10 `openidconnect` app, as well as a randomized `--signing-kid`. diff --git a/modules/developer/pages/ocis/deployment/continuous_deployment.adoc b/modules/developer/pages/ocis/deployment/continuous_deployment.adoc new file mode 100644 index 00000000..3c8cc7a5 --- /dev/null +++ b/modules/developer/pages/ocis/deployment/continuous_deployment.adoc @@ -0,0 +1,107 @@ += Continuous Deployment +:toc: right +:toclevels: 3 + + +We are continuously deploying the following deployment examples. Every example is deployed in two flavors: + +- Master: reflects the current master branch state of oCIS and will be updated with every commit to master +- Rolling: reflects the latest rolling release of oCIS and will be updated with every rolling release +- Production: reflects the latest production release of oCIS and will be updated with every production release + +The configuration for the continuous deployment can be found in the link:https://github.com/owncloud/ocis/tree/master/deployments/continuous-deployment-config[oCIS repository]. + += oCIS with Web Office + +This deployment is based on our modular xref:ocis_full.adoc[ocis_full Example] and uses the default configuration with Collabora Online as the office suite, traefik reverse proxy, cloudimporter and the mailpit mail catching server to showcase the full feature set of oCIS. + +Credentials: + +- oCIS: see xref:../getting-started#login-to-owncloud-web.adoc[default demo users] + +== Master + +- oCIS: link:https://ocis.ocis.master.owncloud.works[ocis.ocis.master.owncloud.works] +- Mail: link:https://mail.ocis.master.owncloud.works[mail.ocis.master.owncloud.works] + +== Rolling Release + +- oCIS: link:https://ocis.ocis.rolling.owncloud.works[ocis.ocis.rolling.owncloud.works] +- Mail: link:https://mail.ocis.rolling.owncloud.works[mail.ocis.rolling.owncloud.works] + +== Production Release + +- oCIS: link:https://ocis.ocis.production.owncloud.works[ocis.ocis.production.owncloud.works] +- Mail: link:https://mail.ocis.production.owncloud.works[mail.ocis.production.owncloud.works] + +== Master with OnlyOffice + +This example is using OnlyOffice as the office suite. + +- oCIS: link:https://ocis.ocis-onlyoffice.master.owncloud.works[ocis.ocis-onlyoffice.master.owncloud.works] + +== Rolling Release with OnlyOffice + +This example is using OnlyOffice as the office suite. + +- oCIS: link:https://ocis.ocis-onlyoffice.rolling.owncloud.works[ocis.ocis-onlyoffice.rolling.owncloud.works] + += oCIS and ownCloud Web with both most recent development versions + +Credentials: + +- oCIS: see xref:../getting-started#login-to-owncloud-web.adoc[default demo users] + +== Master branches + +- oCIS: link:https://ocis.ocis-web.master.owncloud.works[ocis.ocis-web.master.owncloud.works] + += oCIS with Keycloak + +Credentials: + +- oCIS: see xref:../getting-started#login-to-owncloud-web.adoc[default demo users] +- Keycloak: + - username: admin + - password: admin + +== Rolling Release + +- oCIS: link:https://ocis.ocis-keycloak.rolling.owncloud.works[ocis.ocis-keycloak.rolling.owncloud.works] +- Keycloak admin access: link:https://keycloak.ocis-keycloak.rolling.owncloud.works[keycloak.ocis-keycloak.rolling.owncloud.works] +- Keycloak account management: link:https://keycloak.ocis-keycloak.rolling.owncloud.works/realms/oCIS/account/#/[keycloak.ocis-keycloak.rolling.owncloud.works/realms/oCIS/account/#/] + + += oCIS with S3 storage backend (MinIO) + +This deployment is based on our modular xref:ocis_full.adoc[ocis_full Example], see the <> description for the feature set. In addition to that, we deployed a MinIO S3 storage backend. oCIS stores the data in the S3 server and the metadata on the local disk by using the `s3ng` storage driver. + +The MinIO server provides a powerful Web UI for browser-based access to the storage which makes it possible to manage the data stored in the S3 server and understand how different policies and configurations affect the data. + +Credentials: + +- oCIS: see xref:../getting-started/demo-users/.adoc[default demo users] +- MinIO: + - access key: ocis + - secret access key: ocis-secret-key + +== Rolling Release + +- oCIS: link:https://ocis.ocis-s3.rolling.owncloud.works[ocis.ocis-s3.rolling.owncloud.works] +- MinIO: link:https://minio.ocis-s3.rolling.owncloud.works[minio.ocis-s3.rolling.owncloud.works] +- Mail: link:https://mail.ocis-s3.rolling.owncloud.works[mail.ocis-s3.rolling.owncloud.works] + += oCIS with LDAP for users and groups + +Credentials: + +- oCIS: see xref:../getting-started/demo-users/.adoc[default demo users] +- LDAP admin: + - username: cn=admin,dc=owncloud,dc=com + - password: admin + +== Rolling Release + +- oCIS: link:https://ocis.ocis-ldap.rolling.owncloud.works[ocis.ocis-ldap.rolling.owncloud.works] +- LDAP admin: link:https://ldap.ocis-ldap.rolling.owncloud.works[ldap.ocis-ldap.rolling.owncloud.works] + diff --git a/modules/developer/pages/ocis/deployment/index.adoc b/modules/developer/pages/ocis/deployment/index.adoc new file mode 100644 index 00000000..8d672907 --- /dev/null +++ b/modules/developer/pages/ocis/deployment/index.adoc @@ -0,0 +1,24 @@ += Deployment +:toc: right +:toclevels: 3 + + +== Deployments scenarios and examples +This section handles deployments and operations for admins and people who are interested in how versatile oCIS is. If you want to just try oCIS you may also follow xref:../getting-started.adoc[Getting started]. + +=== Setup oCIS on your server +oCIS deployments are super simple, yet there are many configurations possible for advanced setups. + +- xref:basic-remote-setup.adoc[Basic oCIS setup] - configure domain, certificates and port +- xref:ocis_keycloak.adoc[oCIS setup with Keycloak as identity provider] +- xref:ocis_full.adoc[Flexible oCIS setup with WebOffice and Search capabilities] +- xref:oc10_ocis_parallel.adoc[Parallel deployment of oC10 and oCIS] +- xref:ocis_hello.adoc[oCIS with the Hello extension example] + + +== Secure an oCIS instance + +oCIS no longer has any default secrets in versions later than oCIS 1.20.0. Therefore you're no +longer able to start oCIS without generating / setting all needed secrets. + +The recommended way is to use `ocis init` for that. It will generate a secure config file for you. diff --git a/modules/developer/pages/ocis/deployment/kubernetes.adoc b/modules/developer/pages/ocis/deployment/kubernetes.adoc new file mode 100644 index 00000000..675620a7 --- /dev/null +++ b/modules/developer/pages/ocis/deployment/kubernetes.adoc @@ -0,0 +1,71 @@ += Kubernetes +:toc: right +:toclevels: 3 + + +== What is Kubernetes + +Formally described as: + +[quote] +____ +Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. +____ + + +_link:https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/[source]_ + +Without getting too deep in definitions, and for the purpose of compactness, Kubernetes can be summarized as a way of managing containers that run applications to ensure that there is no downtime and an optimal usage of resources. It provides with a framework in which to run distributed systems. + +Kubernetes provides you with: +- _Service discovery and load balancing_: Kubernetes can expose a container using the DNS name or using their own IP address. If traffic to a container is high, Kubernetes is able to load balance and distribute the network traffic so that the deployment is stable. +- _Storage orchestration_: Kubernetes allows you to automatically mount a storage system of your choice, such as local storages, public cloud providers, and more. +- _Automated rollouts and rollbacks_: You can describe the desired state for your deployed containers using Kubernetes, and it can change the actual state to the desired state at a controlled rate. For example, you can automate Kubernetes to create new containers for your deployment, remove existing containers and adopt all their resources to the new container. +- _Automatic bin packing_: You provide Kubernetes with a cluster of nodes that it can use to run containerized tasks. You tell Kubernetes how much CPU and memory (RAM) each container needs. Kubernetes can fit containers onto your nodes to make the best use of your resources. +- _Self-healing_: Kubernetes restarts containers that fail, replaces containers, kills containers that don't respond to your user-defined health check, and doesn't advertise them to clients until they are ready to serve. +- _Secret and configuration management_: Kubernetes lets you store and manage sensitive information, such as passwords, OAuth tokens, and SSH keys. You can deploy and update secrets and application configuration without rebuilding your container images, and without exposing secrets in your stack configuration. + +_link:https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/#why-you-need-kubernetes-and-what-can-it-do[extracted from k8s docs]_ + +If that is still too abstract, link:https://dev.to/miguelmota/comment/filh[here is an ELI5 writeup]. + +=== References and further reads + +- link:https://github.com/Deaddy[Marcel Wunderlich's] link:http://deaddy.net/introduction-to-kubernetes-pt-1.html[4 series articles] on Kubernetes clarifying its declarative nature, deep diving into ingress networking, storage and monitoring. + +=== How does oCIS fit in the Kubernetes model + +oCIS was designed with running on Kubernetes in mind. We set up to adopt the link:https://12factor.net/[Twelve-Factor App] principles regarding configuration, with almost every aspect of oCIS being modifiable via environment variables. This comes in handy when you especially have a look at how a helm chart's (we will introduce this concept shortly) link:https://github.com/owncloud/ocis-charts/blob/d8735e3222d2050504303851d3461909c86fcc89/ocis/values.yaml[list of values] looks like. + +== What is Minikube + +link:https://minikube.sigs.k8s.io/docs/[Minikube] lets you run a Kubernetes cluster locally. It is the most approachable way to test a deployment. It requires no extra configuration on any cloud platform, as everything runs on your local machine. For the purpose of these docs, this is the first approach we chose to run oCIS and will develop on how to set it up. + +== What is `kubectl` + +link:https://kubernetes.io/docs/tasks/tools/[kubectl] is the command-line tool for Kubernetes. It allows users to run commands against a k8s cluster the user has access to. It supports for having multiple contexts for as many clusters as you have access to. In these docs we will setup 2 contexts, a minikube and a GCP context. + +== What are Helm Charts, and why they are useful for oCIS + +link:https://helm.sh/[Helm] is the equivalent of a package manager for Kubernetes. It can be described as a layer on top of how you would write pods, deployments or any other k8s resource declaration. + +=== Installing Helm + +link:https://helm.sh/docs/intro/install/[Follow the official installation guide]. + +== Setting up Minikube + +For a guide on how to set minikube up follow the link:https://minikube.sigs.k8s.io/docs/start/[official minikube start guide] for your specific OS. + +== oCIS charts + +We have not yet published the oCIS Helm charts, therefore you need to clone the git repository manually. It currently also does not support to be run on Kind or Minikube clusters. For known issues and planned features, please have a look at the link:https://github.com/owncloud/ocis-charts/issues[GitHub issue tracker]. + +Configuration options are described link:https://github.com/owncloud/ocis-charts/tree/master/charts/ocis#configuration[here]. + +=== Run oCIS + +1. clone the charts: `git clone https://github.com/owncloud/ocis-charts.git /var/tmp/ocis-charts` +2. cd into the charts root: `cd /var/tmp/ocis-charts/charts/ocis` +3. install the package: `helm install ocis .` (you need to set configuration values in almost all cases) +4. verify the application is running in the cluster: `kubectl get pods` diff --git a/modules/developer/pages/ocis/deployment/monitoring-tracing.adoc b/modules/developer/pages/ocis/deployment/monitoring-tracing.adoc new file mode 100644 index 00000000..88cff22a --- /dev/null +++ b/modules/developer/pages/ocis/deployment/monitoring-tracing.adoc @@ -0,0 +1,36 @@ += Monitoring & Tracing +:toc: right +:toclevels: 3 + + +Monitoring and tracing gives developers and admin insights into a complex system, in this case oCIS. + +If you are a developer and want to trace during developing you should have a look at xref:../development/tracing.adoc[example server setup]. + +This documentation describes how to set up a long-running monitoring & tracing infrastructure for one or multiple oCIS servers or deployments. After reading this guide, you also should know everything needed to integrate oCIS into your existing monitoring and tracing infrastructure. + += Overview about the proposed solution + +image::ocis/monitoring_tracing_overview.drawio.svg[] + +== Monitoring & tracing clients + +We assume that you already have oCIS deployed on one or multiple servers by using our deployment examples (see rectangle on the left). On these servers our monitoring & tracing clients, namely Telegraf and Jaeger agent, need to be added. + +Telegraf will collect host metrics (CPU, RAM, network, processes, ...) and docker metrics (per container CPU, RAM, network, ...). Telegraf is also configured to scrape metrics from Prometheus metric endpoints which oCIS exposes, this is done by the Prometheus input plugin . The metrics from oCIS and all other metrics gathered will be exposed with the Prometheus output plugin and can therefore be scraped by our monitoring & tracing server. + +Jaeger agent is being configured as target for traces in oCIS. It then will receive traces from all oCIS services, add some process tags to them and forward them to our Jaeger collector on our monitoring & tracing server. + +For more information and how to deploy it, see link:https://github.com/owncloud-devops/monitoring-tracing-client[monitoring & tracing client]. + +== Monitoring & tracing server + +A live version of the monitoring and tracing server for our demo instances is available here: link:https://grafana.infra.owncloud.works[Grafana], link:https://prometheus.infra.owncloud.works[Prometheus] and link:https://jaeger.infra.owncloud.works[Jaeger Query]. + +The monitoring & tracing server is considered as shared infrastructure and is normally used for different services. This means that oCIS is not the only software whose metrics and traces are available on the monitoring server. It is also possible that data of multiple oCIS instances are available on the monitoring server. + +Metrics are scraped, stored and can be queried with Prometheus. For the visualization of these metrics Grafana is used. Because Prometheus is scraping the metrics from the oCIS server (pull model instead of a push model), the Prometheus server must have access to the exposed endpoint of the Telegraf Prometheus output plugin. + +Jaeger collector receives traces sent by the Jaeger agent on the oCIS servers and persists them in ElasticSearch. From there the user can query and visualize the traces in Jaeger query or in Grafana. Because Jaeger agent is actively sending traces to the monitoring & tracing server, the server must be reachable from the oCIS server. + +For more information and how to deploy it, see link:https://github.com/owncloud-devops/monitoring-tracing-server[monitoring & tracing server]. diff --git a/modules/developer/pages/ocis/deployment/oc10_ocis_parallel.adoc b/modules/developer/pages/ocis/deployment/oc10_ocis_parallel.adoc new file mode 100644 index 00000000..f872dde4 --- /dev/null +++ b/modules/developer/pages/ocis/deployment/oc10_ocis_parallel.adoc @@ -0,0 +1,162 @@ += Parallel deployment of oC10 and oCIS +:toc: right +:toclevels: 3 + + +== Overview + +- This setup reflects xref:migration#stage-6-parallel-deployment.adoc[stage 6 of the oC10 to oCIS migration plan] +- Traefik generating self-signed certificates for local setup or obtaining valid SSL certificates for a server setup +- OpenLDAP server with demo users +- LDAP admin interface to edit users +- Keycloak as OpenID Connect provider in federation with the LDAP server +- ownCloud 10 with MariaDB and Redis + - ownCloud 10 is configured to synchronize users from the LDAP server + - ownCloud 10 is used to use OpenID Connect for authentication with Keycloak +- oCIS running behind Traefik as reverse proxy + - oCIS is using the ownCloudSQL storage driver on the same files and same database as ownCloud 10 + - oCIS is using Keycloak as OpenID Connect provider + - oCIS is using the LDAP server as user backend +- All requests to both oCIS and oC10 are routed through the oCIS proxy and will be routed based on an OIDC claim to one of them. Therefore admins can change on a user basis in the LDAP which backend is used. + +link:https://github.com/owncloud/ocis/tree/master/deployments/examples/oc10_ocis_parallel[Find this example on GitHub] + +== Server Deployment + +=== Requirements + +- Linux server with docker and docker-compose installed +- four domains set up and pointing to your server + - cloud.\* for serving oCIS + - keycloak.\* for serving Keycloak + - ldap .\* for serving the LDAP management UI + - traefik.\* for serving the Traefik dashboard + +See also xref:preparing_server.adoc[example server setup] + +=== Install this example + +- Clone oCIS repository + + `git clone https://github.com/owncloud/ocis.git` + +- Go to the deployment example + + `cd ocis/deployment/examples/oc10_ocis_parallel` + +- Open the `.env` file in a text editor + The file by default looks like this: + + ```bash + # If you're on a internet facing server please comment out following line. + # It skips certificate validation for various parts of oCIS and is needed if you use self signed certificates. + INSECURE=true + + ### Traefik settings ### + TRAEFIK_LOG_LEVEL= + # Serve Traefik dashboard. Defaults to "false". + TRAEFIK_DASHBOARD= + # Domain of Traefik, where you can find the dashboard. Defaults to "traefik.owncloud.test" + TRAEFIK_DOMAIN= + # Basic authentication for the dashboard. Defaults to user "admin" and password "admin" + TRAEFIK_BASIC_AUTH_USERS= + # Email address for obtaining LetsEncrypt certificates, needs only be changed if this is a public facing server + TRAEFIK_ACME_MAIL= + + ### shared oCIS / oC10 settings ### + # Domain of oCIS / oC10, where you can find the frontend. Defaults to "cloud.owncloud.test" + CLOUD_DOMAIN= + + ### oCIS settings ### + # oCIS version. Defaults to "latest" + OCIS_DOCKER_TAG= + # JWT secret which is used for the storage provider. Must be changed in order to have a secure oCIS. Defaults to "Pive-Fumkiu4" + OCIS_JWT_SECRET= + # JWT secret which is used for uploads to create transfer tokens. Must be changed in order to have a secure oCIS. Defaults to "replace-me-with-a-transfer-secret" + STORAGE_TRANSFER_SECRET= + # Machine auth api key secret. Must be changed in order to have a secure oCIS. Defaults to "change-me-please" + OCIS_MACHINE_AUTH_API_KEY= + + ### oCIS settings ### + # oC10 version. Defaults to "latest" + OC10_DOCKER_TAG= + # client secret which the openidconnect app uses to authenticate to Keycloak. Defaults to "oc10-oidc-secret" + OC10_OIDC_CLIENT_SECRET= + # app which will be shown when opening the ownCloud 10 UI. Defaults to "files" but also could be set to "web" + OWNCLOUD_DEFAULT_APP= + # if set to "false" (default) links will be opened in the classic UI, if set to "true" ownCloud Web is used + OWNCLOUD_WEB_REWRITE_LINKS= + + ### LDAP settings ### + # password for the LDAP admin user "cn=admin,dc=owncloud,dc=com", defaults to "admin" + LDAP_ADMIN_PASSWORD= + # Domain of the LDAP management frontend. Defaults to "ldap.owncloud.test" + LDAP_MANAGER_DOMAIN= + + ### Keycloak ### + # Domain of Keycloak, where you can find the management and authentication frontend. Defaults to "keycloak.owncloud.test" + KEYCLOAK_DOMAIN= + # Realm which to be used with oC10 and oCIS. Defaults to "owncloud" + KEYCLOAK_REALM= + # Admin user login name. Defaults to "admin" + KEYCLOAK_ADMIN_USER= + # Admin user login password. Defaults to "admin" + KEYCLOAK_ADMIN_PASSWORD= + ``` + + You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`. + + If you want to use the Traefik dashboard, set TRAEFIK_DASHBOARD to `true` (default is `false` and therefore not active). If you activate it, you must set a domain for the Traefik dashboard in `TRAEFIK_DOMAIN=` e.g. `TRAEFIK_DOMAIN=traefik.owncloud.test`. + + The Traefik dashboard is secured by basic auth. Default credentials are the user `admin` with the password `admin`. To set your own credentials, generate a htpasswd (e.g. by using link:https://htpasswdgenerator.de/[an online tool] or a cli tool). + + Traefik will issue certificates with LetsEncrypt and therefore you must set an email address in `TRAEFIK_ACME_MAIL=`. + + By default oCIS will be started in the `latest` version. If you want to start a specific version of oCIS set the version to `OCIS_DOCKER_TAG=`. Available versions can be found on link:https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated[Docker Hub]. + + Set your domain for the oC10 and oCIS frontend in `CLOUD_DOMAIN=`, e.g. `CLOUD_DOMAIN=cloud.owncloud.test`. + + By default ownCloud 10 will be started in the `latest` version. If you want to start a specific version of oCIS set the version to `OC10_DOCKER_TAG=`. Available versions can be found on link:https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated[Docker Hub]. + + You can switch the default application of ownCloud 10 by setting`OWNCLOUD_DEFAULT_APP=files` in oder to have the classic UI as frontend, which is also the default. If you prefer ownCloud Web as the default application in ownCloud 10 just set `OWNCLOUD_DEFAULT_APP=web`. + + In oder to change the default link open action which defaults to the classic UI (`OWNCLOUD_WEB_REWRITE_LINKS=false`) you can set it to `OWNCLOUD_WEB_REWRITE_LINKS=true`. This will lead to links being opened in ownCloud Web. + + The OpenLDAP server in this example deployment has an admin users, which is also used as bind user in order to keep these examples simple. You can change the default password "admin" to a different one by setting it to `LDAP_ADMIN_PASSWORD=...`. + + Set your domain for the LDAP manager UI in `LDAP_MANAGER_DOMAIN=`, e.g. `ldap.owncloud.test`. + + Set your domain for the Keycloak administration panel and authentication endpoints to `KEYCLOAK_DOMAIN=` e.g. `KEYCLOAK_DOMAIN=keycloak.owncloud.test`. + + Changing the used Keycloak realm can be done by setting `KEYCLOAK_REALM=`. This defaults to the ownCloud realm `KEYCLOAK_REALM=owncloud`. The ownCloud realm will be automatically imported on startup and includes our demo users. + + You probably should secure your Keycloak admin account by setting `KEYCLOAK_ADMIN_USER=` and `KEYCLOAK_ADMIN_PASSWORD=` to values other than `admin`. + + Now you have configured everything and can save the file. + +- Start the docker stack + + `docker-compose up -d` + +- You now can visit the cloud, oC10 or oCIS depending on the user configuration. Marie defaults to oC10 and Richard and Einstein default to oCIS, but you can change the ownCloud selector at any time in the LDAP management UI. + +== Local setup + +For a more simple local ocis setup see xref:../getting-started.adoc[Getting started] + +This docker stack can also be run locally. One downside is that Traefik can not obtain valid SSL certificates and therefore will create self-signed ones. This means that your browser will show scary warnings. Another downside is that you can not point DNS entries to your localhost. So you have to add static host entries to your computer. + +On Linux and macOS you can add them to your `/etc/hosts` file and on Windows to `C:\Windows\System32\Drivers\etc\hosts` file like this: + +---- +127.0.0.1 cloud.owncloud.test +127.0.0.1 keycloak.owncloud.test +127.0.0.1 ldap.owncloud.test +127.0.0.1 traefik.owncloud.test +---- + +After that you're ready to start the application stack: + +`docker-compose up -d` + +You now can visit the cloud, oC10 or oCIS depending on the user configuration. Marie defaults to oC10 and Richard and Einstein default to oCIS, but you can change the ownCloud selector at any time in the LDAP management UI. diff --git a/modules/developer/pages/ocis/deployment/ocis_full.adoc b/modules/developer/pages/ocis/deployment/ocis_full.adoc new file mode 100644 index 00000000..fa041940 --- /dev/null +++ b/modules/developer/pages/ocis/deployment/ocis_full.adoc @@ -0,0 +1,414 @@ += Full modular oCIS with WebOffice +:toc: right +:toclevels: 3 + + +== Overview + +* oCIS, the collaboration service, Collabora or OnlyOffice running behind Traefik as reverse proxy +* Collabora or OnlyOffice enable you to edit office documents in your browser +* The collaboration server acts as a bridge to make the oCIS storage accessible to Collabora and OnlyOffice +* Traefik generating self-signed certificates for local setup or obtaining valid SSL certificates for a server setup +* The whole deployment acts as a modular toolkit to use different flavors of office suites and ocis features + +link:https://github.com/owncloud/ocis/tree/master/deployments/examples/ocis_full[Find this example on GitHub] + +== Easy Default + +The Infinite Scale Team and product management are providing a default setup for oCIS. + +=== Goal: + - provide a good starting point for a production deployment + - minimal effort to get started with an opinionated setup + - keep it adjustable it to your needs. + +=== Default Components + +- Infinite Scale +- Full Text Search +- Collabora Online Web Office +- Prepared for LetsEncrypt SSL certificates via Traefik Reverse Proxy + +=== Optional Components + +- ClamAV Virusscanner +- Cloud Importer (Experimental) +- OnlyOffice as an alternative to Collabora +- S3 Storage config to connect to an S3 storage backend +- S3 Minio Server as a local S3 storage backend for debugging and development + +=== Important Note + +If you deviate from the configuration setup and let the `collaboration` service run in its own container, you MUST +ensure the ocis configuration is shared as shown in the example deployment. This is because secrets generated +must be accessible for all services. + +== Server Deployment + +=== Requirements + +* Linux server with docker and docker-compose installed +* Three domains set up and pointing to your server + * ocis.* for serving oCIS + * collabora.* for serving Collabora + * onlyoffice.* for serving OnlyOffice + * wopiserver.* for serving the WOPI server + * traefik.* for serving the Traefik dashboard + * companion.* for serving the uppy companion app + +See also xref:preparing_server.adoc[example server setup] + +=== Install oCIS and Traefik + +* Clone oCIS repository + + `git clone https://github.com/owncloud/ocis.git --depth 1` + +* Go to the deployment example + + `cd ocis/deployments/examples/ocis_full` + +* Open the `.env` file in a text editor. + + The file by default looks like this: + + ```shell {linenos=table,hl_lines=[8,24,48,50,135,138]} + ## Basic Settings ## + # Define the docker compose log driver used. + # Defaults to local + LOG_DRIVER= + # If you're on an internet facing server, comment out following line. + # It skips certificate validation for various parts of Infinite Scale and is + # needed when self signed certificates are used. + INSECURE=true + + + ## Traefik Settings ## + # Note: Traefik is always enabled and can't be disabled. + # Serve Traefik dashboard. + # Defaults to "false". + TRAEFIK_DASHBOARD= + # Domain of Traefik, where you can find the dashboard. + # Defaults to "traefik.owncloud.test" + TRAEFIK_DOMAIN= + # Basic authentication for the traefik dashboard. + # Defaults to user "admin" and password "admin" (written as: "admin:admin"). + TRAEFIK_BASIC_AUTH_USERS= + # Email address for obtaining LetsEncrypt certificates. + # Needs only be changed if this is a public facing server. + TRAEFIK_ACME_MAIL= + # Set to the following for testing to check the certificate process: + # "https://acme-staging-v02.api.letsencrypt.org/directory" + # With staging configured, there will be an SSL error in the browser. + # When certificates are displayed and are emitted by # "Fake LE Intermediate X1", + # the process went well and the envvar can be reset to empty to get valid certificates. + TRAEFIK_ACME_CASERVER= + + + ## Infinite Scale Settings ## + # Beside Traefik, this service must stay enabled. + # Disable only for testing purposes. + # Note: the leading colon is required to enable the service. + OCIS=:ocis.yml + # The oCIS container image. + # For production releases: "owncloud/ocis" + # For rolling releases: "owncloud/ocis-rolling" + # Defaults to production if not set otherwise + OCIS_DOCKER_IMAGE=owncloud/ocis-rolling + # The oCIS container version. + # Defaults to "latest" and points to the latest stable tag. + OCIS_DOCKER_TAG= + # Domain of oCIS, where you can find the frontend. + # Defaults to "ocis.owncloud.test" + OCIS_DOMAIN= + # oCIS admin user password. Defaults to "admin". + ADMIN_PASSWORD= + # Demo users should not be created on a production instance, + # because their passwords are public. Defaults to "false". + # Also see: https://doc.owncloud.com/ocis/latest/deployment/general/general-info.html#demo-users-and-groups + DEMO_USERS= + # Define the oCIS loglevel used. + # For more details see: + # https://doc.owncloud.com/ocis/latest/deployment/services/env-vars-special-scope.html + LOG_LEVEL= + # Define the kind of logging. + # The default log can be read by machines. + # Set this to true to make the log human readable. + # LOG_PRETTY=true + # + # Define the oCIS storage location. Set the paths for config and data to a local path. + # Note that especially the data directory can grow big. + # Leaving it default stores data in docker internal volumes. + # For more details see: + # https://doc.owncloud.com/ocis/next/deployment/general/general-info.html#default-paths + # OCIS_CONFIG_DIR=/your/local/ocis/config + # OCIS_DATA_DIR=/your/local/ocis/data + + # S3 Storage configuration - optional + # Infinite Scale supports S3 storage as primary storage. + # Per default, S3 storage is disabled and the local filesystem is used. + # To enable S3 storage, uncomment the following line and configure the S3 storage. + # For more details see: + # https://doc.owncloud.com/ocis/next/deployment/storage/s3.html + # Note: the leading colon is required to enable the service. + #S3NG=:s3ng.yml + # Configure the S3 storage endpoint. Defaults to "http://minio:9000" for testing purposes. + S3NG_ENDPOINT= + # S3 region. Defaults to "default". + S3NG_REGION= + # S3 access key. Defaults to "ocis" + S3NG_ACCESS_KEY= + # S3 secret. Defaults to "ocis-secret-key" + S3NG_SECRET_KEY= + # S3 bucket. Defaults to "ocis" + S3NG_BUCKET= + # + # For testing purposes, add local minio S3 storage to the docker-compose file. + # The leading colon is required to enable the service. + #S3NG_MINIO=:minio.yml + # Minio domain. Defaults to "minio.owncloud.test". + MINIO_DOMAIN= + + # Define SMPT settings if you would like to send Infinite Scale email notifications. + # For more details see: + # https://doc.owncloud.com/ocis/latest/deployment/services/s-list/notifications.html + # NOTE: when configuring mail server, these settings have no effect, see mailserver.yml for details. + # SMTP host to connect to. + SMTP_HOST= + # Port of the SMTP host to connect to. + SMTP_PORT= + # An eMail address that is used for sending Infinite Scale notification eMails + # like "ocis notifications ". + SMTP_SENDER= + # Username for the SMTP host to connect to. + SMTP_USERNAME= + # Password for the SMTP host to connect to. + SMTP_PASSWORD= + # Authentication method for the SMTP communication. + SMTP_AUTHENTICATION= + # Allow insecure connections to the SMTP server. Defaults to false. + SMTP_INSECURE= + + + ## Default Enabled Services ## + + ### Apache Tika Content Analysis Toolkit ### + # Tika (search) is enabled by default, comment if not required. + # Note: the leading colon is required to enable the service. + TIKA=:tika.yml + # Set the desired docker image tag or digest. + # Defaults to "latest" + TIKA_IMAGE= + + + ### Collabora Settings ### + # Collabora web office is default enabled, comment if not required. + # Note: the leading colon is required to enable the service. + COLLABORA=:collabora.yml + # Domain of Collabora, where you can find the frontend. + # Defaults to "collabora.owncloud.test" + COLLABORA_DOMAIN= + # Domain of the wopiserver which handles OnlyOffice. + # Defaults to "wopiserver.owncloud.test" + WOPISERVER_DOMAIN= + # Admin user for Collabora. + # Defaults to "admin". + # Collabora Admin Panel URL: + # https://{COLLABORA_DOMAIN}/browser/dist/admin/admin.html + COLLABORA_ADMIN_USER= + # Admin password for Collabora. + # Defaults to "admin". + COLLABORA_ADMIN_PASSWORD= + # Set to true to enable SSL for Collabora Online. Default is true if not specified. + COLLABORA_SSL_ENABLE=false + # If you're on an internet-facing server, enable SSL verification for Collabora Online. + # Please comment out the following line: + COLLABORA_SSL_VERIFICATION=false + ... + ``` + #### Reverse Proxy and SSL + +// SHORTCODE: {{< hint type=important >}} + _Domains and SSL_\ + Though it may sound strange, most of the setups are failing due to a misconfiguration regarding domains and SSL. Please make sure that you have set up the domains correctly and that they are pointing to your server. Also, make sure that you have set up the email address for the LetsEncrypt certificates in `TRAEFIK_ACME_MAIL=`. +// SHORTCODE: {{< /hint >}} + + You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`. + + Traefik will issue certificates with LetsEncrypt and therefore you must set an email address in `TRAEFIK_ACME_MAIL=`. + + #### Infinite Scale Release and Version + By default oCIS will be started in the `latest` rolling version. Please note that this deployment does currently not work with the 5.x productions releases. + The oCIS "collaboration" service, which is required by this deployment, is not part of the 5.x releases. + + If you want to use a specific version of oCIS, set the version to a dedicated tag like `OCIS_DOCKER_TAG=6.3.0`. The minimal required oCIS Version to run this deployment is 6.3.0. Available production versions can be found on link:https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated[Docker Hub Production] and available rolling releases can be found on link:https://hub.docker.com/r/owncloud/ocis-rolling/tags?page=1&ordering=last_updated[Docker Hub Rolling] + +// SHORTCODE: {{< hint type=info title="oCIS Releases" >}} + You can read more about the different oCIS releases in the xref:../release_roadmap.adoc[oCIS Release Lifecycle]. +// SHORTCODE: {{< /hint >}} + + Set your domain for the oCIS frontend in `OCIS_DOMAIN=`, e.g. `OCIS_DOMAIN=ocis.owncloud.test`. + + Set the initial admin user password in `ADMIN_PASSWORD=`, it defaults to `admin`. + + Web Office needs a public domain for the WOPI server to be set in `WOPISERVER_DOMAIN=`, where the office suite can work on the files via the WOPI protocol. + + Now it's time to set up Collabora and you need to configure the domain of Collabora in `COLLABORA_DOMAIN=`. + + If you want to use the Collabora admin panel you need to set the username and password for the administrator in `COLLABORA_ADMIN_USER=` and `COLLABORA_ADMIN_PASSWORD=`. + +* Start the docker stack + + `docker-compose up -d` + +* You now can visit oCIS and are able to open an office document in your browser. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time. + +== Local Setup + +This docker stack can also be run locally. One downside is that Traefik can not obtain valid SSL certificates and therefore will create self-signed ones. This means that your browser will show scary warnings. Another downside is that you can not point DNS entries to your localhost. So you have to add static host entries to your computer. + +On Linux and macOS you can add them to your `/etc/hosts` file and on Windows to `C:\Windows\System32\Drivers\etc\hosts` file like this: + +---- +127.0.0.1 ocis.owncloud.test +127.0.0.1 traefik.owncloud.test +127.0.0.1 collabora.owncloud.test +127.0.0.1 onlyoffice.owncloud.test +127.0.0.1 wopiserver.owncloud.test +127.0.0.1 mail.owncloud.test +127.0.0.1 companion.owncloud.test +127.0.0.1 minio.owncloud.test +---- + +After that, you're ready to start the application stack: + +`docker-compose pull && docker-compose up -d` + +Open https://collabora.owncloud.test in your browser and accept the invalid certificate warning. + +Open https://ocis.owncloud.test in your browser and accept the invalid certificate warning. You are now able to open an office document in your browser. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time. + +== Additional services + +=== Clamav Virusscanner + +You can add a Clamav Virusscanner to the stack. The service is disabled by default. To enable it, uncomment the `CLAMAV` line in the `.env` file. + +[source,shell] +---- +## Clamav Settings ## +# The leading colon is required to enable the service. +CLAMAV=:clamav.yml +---- + +After enabling that service, you can add the service to the stack with `docker-compose up -d` again. + +=== Traefik Dashboard + +If you want to use the Traefik dashboard, set TRAEFIK_DASHBOARD to `true` (default is `false` and therefore not active). If you activate it, you must set a domain for the Traefik dashboard in `TRAEFIK_DOMAIN=` e.g. `TRAEFIK_DOMAIN=traefik.owncloud.test`. + +The Traefik dashboard is secured by basic auth. Default credentials are the user `admin` with the password `admin`. To set your own credentials, generate a htpasswd (e.g. by using link:https://htpasswdgenerator.de/[an online tool] or a cli tool). + +[source,shell] +---- +### Traefik Settings ### +# Serve Traefik dashboard. +# Defaults to "false". +TRAEFIK_DASHBOARD=true +# Domain of Traefik, where you can find the dashboard. +# Defaults to "traefik.owncloud.test" +TRAEFIK_DOMAIN= +# Basic authentication for the traefik dashboard. +# Defaults to user "admin" and password "admin" (written as: "admin:admin"). +TRAEFIK_BASIC_AUTH_USERS= +---- +=== Cloud Importer + +Cloud importer can provide an Upload Interface to your oCIS instance. It is a separate service that can be enabled in the `.env` file. + +[source,shell] +---- +## Uppy Companion Settings ## +# The leading colon is required to enable the service. +CLOUD_IMPORTER=:cloudimporter.yml +## The docker image to be used for uppy companion. +# owncloud has built a container with public link import support. +COMPANION_IMAGE= +# Domain of Uppy Companion. Defaults to "companion.owncloud.test". +COMPANION_DOMAIN= +# Provider settings, see https://uppy.io/docs/companion/#provideroptions for reference. +# Empty by default, which disables providers. +COMPANION_ONEDRIVE_KEY= +COMPANION_ONEDRIVE_SECRET= +---- + +After Enabling that servive by uncommenting the `CLOUD_IMPORTER` line, you can add the service to the stack with `docker-compose up -d` again. + +=== S3 Storage + +You can use an S3 compatible Storage as the primary data store. The metadatata of the files will still be stored on the local filesystem. + +// SHORTCODE: {{}} +The endpoint, region and keys for your S3 Server need to be provided by the service or company who operates it. Normally you can get these via web portal. +// SHORTCODE: {{}} + +[source,shell] +---- +# S3 Storage configuration - optional +# Infinite Scale supports S3 storage as primary storage. +# Per default, S3 storage is disabled and the local filesystem is used. +# To enable S3 storage, uncomment the following line and configure the S3 storage. +# For more details see: +# https://doc.owncloud.com/ocis/next/deployment/storage/s3.html +# Note: the leading colon is required to enable the service. +# S3NG=:s3ng.yml +# Configure the S3 storage endpoint. Defaults to "http://minio:9000" for testing purposes. +S3NG_ENDPOINT= +# S3 region. Defaults to "default". +S3NG_REGION= +# S3 access key. Defaults to "ocis" +S3NG_ACCESS_KEY= +# S3 secret. Defaults to "ocis-secret-key" +S3NG_SECRET_KEY= +# S3 bucket. Defaults to "ocis" +S3NG_BUCKET= +---- + +==== Use a Local Minio S3 Storage Backend + +For testing purposes, you can use a local minio S3 storage backend. To enable it, uncomment the `S3NG_MINIO` line in the `.env` file. + +The frontend for the minio server is available at `http://minio.owncloud.test` and the access key is `ocis` and the secret key is `ocis-secret`. + +== Local Setup for Web Development + +In case you want to run ownCloud Web from a development branch together with this deployment example (e.g. for feature development for the app provider frontend) you can use this deployment example with the local setup and some additional steps as described below. + +1. Clone the link:https://github.com/owncloud/web[ownCloud Web repository] on your development machine. +2. Run `pnpm i && pnpm build:w` for `web`, so that it creates and continuously updates the `dist` folder for web. +3. Add the dist folder as read only volume to `volumes` section of the `ocis` service in the `docker-compose.yml` file: + ```yaml + - /your/local/path/to/web/dist/:/web/dist:ro + ``` + Make sure to point to the `dist` folder inside your local copy of the web repository. +4. Set the oCIS environment variables `WEB_ASSET_CORE_PATH` and `WEB_ASSET_APPS_PATH` in the `environment` section of the `ocis` service, so that it uses your mounted dist folder for the web assets, instead of the assets that are embedded into oCIS. + ```yaml + WEB_ASSET_CORE_PATH: "/web/dist" + WEB_ASSET_APPS_PATH: "/web/dist" + ``` +5. Start the deployment example as described above in the `Local setup` section. + +For app provider frontend development in `web` you can find the source code in `web/packages/web-app-external`. Some parts of the integration live in `web/packages/web-app-files`. + +== Using Podman + +Podman doesn't have a "local" log driver. Also it's docker-compatibility socket does live in a different location, especially when running a rootless podman. + +Using the following settings you can run the deployment with a recent podman version: + +[source,bash] +---- +LOG_DRIVER=journald \ +DOCKER_SOCKET_PATH=/run/user/1000/podman/podman.sock \ +podman compose start +---- diff --git a/modules/developer/pages/ocis/deployment/ocis_hello.adoc b/modules/developer/pages/ocis/deployment/ocis_hello.adoc new file mode 100644 index 00000000..123a86bc --- /dev/null +++ b/modules/developer/pages/ocis/deployment/ocis_hello.adoc @@ -0,0 +1,115 @@ += oCIS with Hello extension +:toc: right +:toclevels: 3 + + +== Overview + +- oCIS running behind Traefik as reverse proxy +- oCIS Hello extension runs beside the main oCIS stack and providing the Hello functionality +- Traefik generating self-signed certificates for local setup or obtaining valid SSL certificates for a server setup + +link:https://github.com/owncloud/ocis/tree/master/deployments/examples/ocis_hello[Find this example on GitHub] + +The docker stack consists of 3 containers. One of them is Traefik, a proxy which is terminating SSL and forwards the requests to oCIS in the internal docker network. + +The oCIS Hello extension is running in another container and enables you to use its functionality from within ownCloud Web. + +== Server Deployment + +=== Requirements + +- Linux server with docker and docker-compose installed +- two domains set up and pointing to your server + - ocis.\* for serving oCIS + - traefik.\* for serving the Traefik dashboard + +See also xref:preparing_server.adoc[example server setup] + +=== Install oCIS and Traefik + +- Clone oCIS repository + + `git clone https://github.com/owncloud/ocis.git` + +- Go to the deployment example + + `cd ocis/deployment/examples/ocis_hello` + +- Open the `.env` file in a text editor. + + The file by default looks like this: + + ```bash + # If you're on a internet facing server please comment out following line. + # It skips certificate validation for various parts of oCIS and is needed if you use self signed certificates. + INSECURE=true + + ### Traefik settings ### + # Serve Traefik dashboard. Defaults to "false". + TRAEFIK_DASHBOARD= + # Domain of Traefik, where you can find the dashboard. Defaults to "traefik.owncloud.test" + TRAEFIK_DOMAIN= + # Basic authentication for the dashboard. Defaults to user "admin" and password "admin" + TRAEFIK_BASIC_AUTH_USERS= + # Email address for obtaining LetsEncrypt certificates, needs only be changed if this is a public facing server + TRAEFIK_ACME_MAIL= + + ### oCIS settings ### + # oCIS version. Defaults to "latest" + OCIS_DOCKER_TAG= + # Domain of oCIS, where you can find the frontend. Defaults to "ocis.owncloud.test" + OCIS_DOMAIN= + # oCIS admin user password. Defaults to "admin". + ADMIN_PASSWORD= + # The demo users should not be created on a production instance + # because their passwords are public. Defaults to "false". + DEMO_USERS= + + ### oCIS Hello settings ### + # oCIS Hello version. Defaults to "latest" + OCIS_HELLO_DOCKER_TAG= + ``` + + You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`. + + If you want to use the Traefik dashboard, set TRAEFIK_DASHBOARD to `true` (default is `false` and therefore not active). If you activate it, you must set a domain for the Traefik dashboard in `TRAEFIK_DOMAIN=` e.g. `TRAEFIK_DOMAIN=traefik.owncloud.test`. + + The Traefik dashboard is secured by basic auth. Default credentials are the user `admin` with the password `admin`. To set your own credentials, generate a htpasswd (e.g. by using link:https://htpasswdgenerator.de/[an online tool] or a cli tool). + + Traefik will issue certificates with LetsEncrypt and therefore you must set an email address in `TRAEFIK_ACME_MAIL=`. + + By default oCIS will be started in the `latest` version. If you want to start a specific version of oCIS set the version to `OCIS_DOCKER_TAG=`. Available versions can be found on link:https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated[Docker Hub]. + + Set your domain for the oCIS frontend in `OCIS_DOMAIN=`, e.g. `OCIS_DOMAIN=ocis.owncloud.test`. + + Set the initial admin user password in `ADMIN_PASSWORD=`, it defaults to `admin`. + + By default the oCIS Hello extension will be started in the `latest` version. If you want to start a specific version of oCIS Hello set the version to `OCIS_HELLO_DOCKER_TAG=`. Available versions can be found on link:https://hub.docker.com/r/owncloud/ocis-hello/tags?page=1&ordering=last_updated[Docker Hub]. + + Now you have configured everything and can save the file. + +- Start the docker stack + + `docker-compose up -d` + +- You now can visit oCIS and are able to switch to the Hello extension by using the application switcher on the top right corner of ownCloud Web. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time. + +== Local setup + +For a more simple local ocis setup see xref:../getting-started.adoc[Getting started] + +This docker stack can also be run locally. One downside is that Traefik can not obtain valid SSL certificates and therefore will create self-signed ones. This means that your browser will show scary warnings. Another downside is that you can not point DNS entries to your localhost. So you have to add static host entries to your computer. + +On Linux and macOS you can add them to your `/etc/hosts` file and on Windows to `C:\Windows\System32\Drivers\etc\hosts` file like this: + +---- +127.0.0.1 ocis.owncloud.test +127.0.0.1 traefik.owncloud.test +---- + +After that you're ready to start the application stack: + +`docker-compose up -d` + +Open https://ocis.owncloud.test in your browser and accept the invalid certificate warning. You are now able to switch to the Hello extension by using the application switcher on the top right corner of ownCloud Web. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time. diff --git a/modules/developer/pages/ocis/deployment/ocis_keycloak.adoc b/modules/developer/pages/ocis/deployment/ocis_keycloak.adoc new file mode 100644 index 00000000..a65dac0a --- /dev/null +++ b/modules/developer/pages/ocis/deployment/ocis_keycloak.adoc @@ -0,0 +1,130 @@ += oCIS with Keycloak +:toc: right +:toclevels: 3 + + +== Overview + +* oCIS and Keycloak running behind Traefik as reverse proxy +* Keycloak acting as the IDP for oCIS +* Traefik generating self-signed certificates for local setup or obtaining valid SSL certificates for a server setup + +link:https://github.com/owncloud/ocis/tree/master/deployments/examples/ocis_keycloak[Find this example on GitHub] + +The docker stack consists 4 containers. One of them is Traefik, a proxy which is terminating ssl and forwards the requests to oCIS in the internal docker network. It +is also responsible for redirecting requests on the OIDC discovery endpoints (e.g. `.well-known/openid-configuration`) to the correct destination in Keycloak. + +Keycloak add two containers: Keycloak itself and a PostgreSQL as database. Keycloak will be configured as oCIS' IDP instead of the internal IDP xref:../../services/idp.adoc[LibreGraph Connect] + +The other container is oCIS itself, running all services in one container. In this example oCIS uses the xref:../storage/storagedrivers.adoc[oCIS storage driver] + +== Server Deployment + +=== Requirements + +* Linux server with docker and docker-compose installed +* Three domains set up and pointing to your server + - ocis.* for serving oCIS + - keycloak.* for serving Keycloak + - traefik.* for serving the Traefik dashboard + +See also xref:preparing_server.adoc[example server setup] + + +=== Install oCIS and Traefik + +* Clone oCIS repository + + `git clone https://github.com/owncloud/ocis.git` + +* Go to the deployment example + + `cd ocis/deployments/examples/ocis_keycloak` + +* Open the `.env` file in a text editor. + + The file by default looks like this: + + ```bash + # If you're on a internet facing server please comment out following line. + # It skips certificate validation for various parts of oCIS and is needed if you use self signed certificates. + INSECURE=true + + ### Traefik settings ### + # Serve Traefik dashboard. Defaults to "false". + TRAEFIK_DASHBOARD= + # Domain of Traefik, where you can find the dashboard. Defaults to "traefik.owncloud.test" + TRAEFIK_DOMAIN= + # Basic authentication for the dashboard. Defaults to user "admin" and password "admin" + TRAEFIK_BASIC_AUTH_USERS= + # Email address for obtaining LetsEncrypt certificates, needs only be changed if this is a public facing server + TRAEFIK_ACME_MAIL= + + ### oCIS settings ### + # oCIS version. Defaults to "latest" + OCIS_DOCKER_TAG= + # Domain of oCIS, where you can find the frontend. Defaults to "ocis.owncloud.test" + OCIS_DOMAIN= + # ownCloud Web openid connect client id. Defaults to "ocis-web" + OCIS_OIDC_CLIENT_ID= + + ### Keycloak ### + # Domain of Keycloak, where you can find the management and authentication frontend. Defaults to "keycloak.owncloud.test" + KEYCLOAK_DOMAIN= + # Realm which to be used with oCIS. Defaults to "oCIS" + KEYCLOAK_REALM= + # Admin user login name. Defaults to "admin" + KEYCLOAK_ADMIN_USER= + # Admin user login password. Defaults to "admin" + KEYCLOAK_ADMIN_PASSWORD= + + ``` + + You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`. + + If you want to use the Traefik dashboard, set TRAEFIK_DASHBOARD to `true` (default is `false` and therefore not active). If you activate it, you must set a domain for the Traefik dashboard in `TRAEFIK_DOMAIN=` e.g. `TRAEFIK_DOMAIN=traefik.owncloud.test`. + + The Traefik dashboard is secured by basic auth. Default credentials are the user `admin` with the password `admin`. To set your own credentials, generate a htpasswd (e.g. by using link:https://htpasswdgenerator.de/[an online tool] or a cli tool). + + Traefik will issue certificates with LetsEncrypt and therefore you must set an email address in `TRAEFIK_ACME_MAIL=`. + + By default oCIS will be started in the `latest` version. If you want to start a specific version of oCIS set the version to `OCIS_DOCKER_TAG=`. Available versions can be found on link:https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated[Docker Hub]. + + Set your domain for the oCIS frontend in `OCIS_DOMAIN=`, e.g. `OCIS_DOMAIN=ocis.owncloud.test`. + + If you want to change the OIDC client id of th ownCloud Web frontend, you can do this by setting the name to `OCIS_OIDC_CLIENT_ID=`. + + Set your domain for the Keycloak administration panel and authentication endpoints to `KEYCLOAK_DOMAIN=` e.g. `KEYCLOAK_DOMAIN=keycloak.owncloud.test`. + + Changing the used Keycloak realm can be done by setting `KEYCLOAK_REALM=`. This defaults to the oCIS realm `KEYCLOAK_REALM=oCIS`. The oCIS realm will be automatically imported on startup and includes our demo users. + + You probably should secure your Keycloak admin account by setting `KEYCLOAK_ADMIN_USER=` and `KEYCLOAK_ADMIN_PASSWORD=` to values other than `admin`. + + Now you have configured everything and can save the file. + +* Start the docker stack + + `docker-compose up -d` + +* You now can visit oCIS, Keycloak and Traefik dashboard on your configured domains. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time. + +== Local setup +For a more simple local ocis setup see xref:../getting-started.adoc[Getting started] + +This docker stack can also be run locally. One downside is that Traefik can not obtain valid SSL certificates and therefore will create self-signed ones. This means that your browser will show scary warnings. Another downside is that you can not point DNS entries to your localhost. So you have to add static host entries to your computer. + +On Linux and macOS you can add them to your `/etc/hosts` file and on Windows to `C:\Windows\System32\Drivers\etc\hosts` file like this: + +---- +127.0.0.1 ocis.owncloud.test +127.0.0.1 traefik.owncloud.test +127.0.0.1 keycloak.owncloud.test +---- + +After that you're ready to start the application stack: + +`docker-compose up -d` + +Open https://keycloak.owncloud.test in your browser and accept the invalid certificate warning. + +Open https://ocis.owncloud.test in your browser and accept the invalid certificate warning. You now can login to oCIS with the demo users. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time. diff --git a/modules/developer/pages/ocis/deployment/ocis_ldap.adoc b/modules/developer/pages/ocis/deployment/ocis_ldap.adoc new file mode 100644 index 00000000..03ffcf0f --- /dev/null +++ b/modules/developer/pages/ocis/deployment/ocis_ldap.adoc @@ -0,0 +1,127 @@ += oCIS with LDAP +:toc: right +:toclevels: 3 + + +== Overview + +- Traefik generating self-signed certificates for local setup or obtaining valid SSL certificates for a server setup +- OpenLDAP server with demo users +- LDAP admin interface to edit users +- oCIS running behind Traefik as reverse proxy + - oCIS is using the LDAP server as user backend + +link:https://github.com/owncloud/ocis/tree/master/deployments/examples/ocis_ldap[Find this example on GitHub] + +== Server Deployment + +=== Requirements + +- Linux server with docker and docker-compose installed +- four domains set up and pointing to your server + - ocis.\* for serving oCIS + - ldap .\* for serving the LDAP management UI + - traefik.\* for serving the Traefik dashboard + +See also xref:preparing_server.adoc[example server setup] + +=== Install this example + +- Clone oCIS repository + + `git clone https://github.com/owncloud/ocis.git` + +- Go to the deployment example + + `cd ocis/deployment/examples/ocis_ldap` + +- Open the `.env` file in a text editor. + + The file by default looks like this: + + ```bash + # If you're on a internet facing server please comment out following line. + # It skips certificate validation for various parts of oCIS and is needed if you use self signed certificates. + INSECURE=true + + ### Traefik settings ### + # Serve Traefik dashboard. Defaults to "false". + TRAEFIK_DASHBOARD= + # Domain of Traefik, where you can find the dashboard. Defaults to "traefik.owncloud.test" + TRAEFIK_DOMAIN= + # Basic authentication for the dashboard. Defaults to user "admin" and password "admin" + TRAEFIK_BASIC_AUTH_USERS= + # Email address for obtaining LetsEncrypt certificates, needs only be changed if this is a public facing server + TRAEFIK_ACME_MAIL= + + ### oCIS settings ### + # oCIS version. Defaults to "latest" + OCIS_DOCKER_TAG= + # Domain of oCIS, where you can find the frontend. Defaults to "ocis.owncloud.test" + OCIS_DOMAIN= + # JWT secret which is used for the storage provider. Must be changed in order to have a secure oCIS. Defaults to "Pive-Fumkiu4" + OCIS_JWT_SECRET= + # JWT secret which is used for uploads to create transfer tokens. Must be changed in order to have a secure oCIS. Defaults to "replace-me-with-a-transfer-secret" + STORAGE_TRANSFER_SECRET= + # Machine auth api key secret. Must be changed in order to have a secure oCIS. Defaults to "change-me-please" + OCIS_MACHINE_AUTH_API_KEY= + + ### LDAP server settings ### + # Password of LDAP user "cn=admin,dc=owncloud,dc=com". Defaults to "admin" + LDAP_ADMIN_PASSWORD= + + ### LDAP manager settings ### + # Domain of LDAP manager. Defaults to "ldap.owncloud.test" + LDAP_MANAGER_DOMAIN= + ``` + + You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`. + + If you want to use the Traefik dashboard, set TRAEFIK_DASHBOARD to `true` (default is `false` and therefore not active). If you activate it, you must set a domain for the Traefik dashboard in `TRAEFIK_DOMAIN=` e.g. `TRAEFIK_DOMAIN=traefik.owncloud.test`. + + The Traefik dashboard is secured by basic auth. Default credentials are the user `admin` with the password `admin`. To set your own credentials, generate a htpasswd (e.g. by using link:https://htpasswdgenerator.de/[an online tool] or a cli tool). + + Traefik will issue certificates with LetsEncrypt and therefore you must set an email address in `TRAEFIK_ACME_MAIL=`. + + By default oCIS will be started in the `latest` version. If you want to start a specific version of oCIS set the version to `OCIS_DOCKER_TAG=`. Available versions can be found on link:https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated[Docker Hub]. + + Set your domain for the oCIS frontend in `OCIS_DOMAIN=`, e.g. `OCIS_DOMAIN=cloud.owncloud.test`. + + The OpenLDAP server in this example deployment has an admin users, which is also used as bind user in order to keep these examples simple. You can change the default password "admin" to a different one by setting it to `LDAP_ADMIN_PASSWORD=...`. + + Set your domain for the LDAP manager UI in `LDAP_MANAGER_DOMAIN=`, e.g. `ldap.owncloud.test`. + + Grant the oCIS Admin role to the admin user from your LDAP in `OCIS_ADMIN_USER_ID:`. You need to enter the uuid of LDAP user. + +// SHORTCODE: {{< hint type=tip title=Encoding >}} + In the .ldif file in this example, the admin user id is base64 encoded. You need to decode it to make it work. +// SHORTCODE: {{< /hint >}} + + Now you have configured everything and can save the file. + +- Start the docker stack + + `docker-compose up -d` + +- You now can visit oCIS and Traefik dashboard on your configured domains. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time. + +== Local setup + +For a more simple local ocis setup see xref:../getting-started.adoc[Getting started] + +This docker stack can also be run locally. One downside is that Traefik can not obtain valid SSL certificates and therefore will create self-signed ones. This means that your browser will show scary warnings. Another downside is that you can not point DNS entries to your localhost. So you have to add static host entries to your computer. + +On Linux and macOS you can add them to your `/etc/hosts` file and on Windows to `C:\Windows\System32\Drivers\etc\hosts` file like this: + +---- +127.0.0.1 cloud.owncloud.test +127.0.0.1 keycloak.owncloud.test +127.0.0.1 ldap.owncloud.test +127.0.0.1 traefik.owncloud.test +---- + +After that you're ready to start the application stack: + +`docker-compose up -d` + +Open https://ocis.owncloud.test in your browser and accept the invalid certificate warning. You now can login to oCIS with the default users, which also can be found here: xref:../getting-started#login-to-ocis-web.adoc[Getting started]. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time. diff --git a/modules/developer/pages/ocis/deployment/preparing_server.adoc b/modules/developer/pages/ocis/deployment/preparing_server.adoc new file mode 100644 index 00000000..cbee30a9 --- /dev/null +++ b/modules/developer/pages/ocis/deployment/preparing_server.adoc @@ -0,0 +1,61 @@ += Preparing a server +:toc: right +:toclevels: 3 + + + +== Example for Hetzner Cloud +* create server on Hetzner Cloud. Set labels "owner" and "for". Example for hcloud cli: +`hcloud server create --type cx21 --image ubuntu-20.04 --ssh-key admin --name ocis-server --label owner=admin --label for=testing` + +* Configure DNS A-records for needed domains pointing on the servers ip address, for example in CloudFlare + +* Access server via ssh as root + +* Create a new user + + `$ adduser --disabled-password --gecos "" admin` + +* Add user to sudo group + + `$ usermod -aG sudo admin` + +* Install docker + + ``` + apt update + apt install docker.io + ``` + +* Add user to docker group + + `usermod -aG docker admin` + +* Install docker-compose via + + `curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose` + + (docker compose version 1.27.4 as of today) +* Make docker-compose executable + + `chmod +x /usr/local/bin/docker-compose` + + +* Add users pub key to + ``` + mkdir /home/admin/.ssh + echo "" >> /home/admin/.ssh/authorized_keys + chown admin:admin -R /home/admin/.ssh + ``` + +* Secure ssh daemon by editing `/etc/ssh/sshd_config` + ``` + PermitRootLogin no + ChallengeResponseAuthentication no + PasswordAuthentication no + UsePAM no + ``` + +* restart sshd server to apply settings `systemctl restart sshd` + +* Login as the user you created diff --git a/modules/developer/pages/ocis/deployment/systemd.adoc b/modules/developer/pages/ocis/deployment/systemd.adoc new file mode 100644 index 00000000..0220a2f9 --- /dev/null +++ b/modules/developer/pages/ocis/deployment/systemd.adoc @@ -0,0 +1,70 @@ += Systemd service +:toc: right +:toclevels: 3 + + +== Install the oCIS binary + +Download the oCIS binary of your preferred version and for your CPU architecture and operating system from link:https://download.owncloud.com/ocis/ocis[download.owncloud.com]. + +Rename the downloaded binary to `ocis` and move it to `/usr/bin/`. As a next step, you need to mark it as executable with `chmod +x /usr/bin/ocis`. + +When you now run `ocis help` on your command line, you should see the available options for the oCIS command. + +== Systemd service definition + +Create the Systemd service definition for oCIS in the file `/etc/systemd/system/ocis.service` with following content: + +[source,systemd] +---- +[Unit] +Description=OCIS server + +[Service] +Type=simple +User=root +Group=root +EnvironmentFile=/etc/ocis/ocis.env +ExecStart=ocis server +Restart=always + +[Install] +WantedBy=multi-user.target +---- + +[WARNING] +==== +For reasons of simplicity we are using the root user and group to run oCIS which is not recommended. Please use a non-root user in production environments and modify the oCIS service definition accordingly. +==== + + +In the service definition we referenced `/etc/ocis/ocis.env` as our file containing environment variables for the oCIS process. +In order to create the file we need first to create the folder `/etc/ocis/` and then we can add the actual `/etc/ocis/ocis.env` with following content: + +[source,bash] +---- +OCIS_URL=https://some-hostname-or-ip:9200 +PROXY_HTTP_ADDR=0.0.0.0:9200 +OCIS_INSECURE=false + +OCIS_LOG_LEVEL=error + +OCIS_CONFIG_DIR=/etc/ocis +OCIS_BASE_DATA_PATH=/var/lib/ocis +---- + +Since we set `OCIS_CONFIG_DIR` to `/etc/ocis` you can also place configuration files in this directory. + +Please change your `OCIS_URL` in order to reflect your actual deployment. If you are using self-signed certificates you need to set `OCIS_INSECURE=true` in `/etc/ocis/ocis.env`. + +oCIS will store all data in `/var/lib/ocis`, because we configured it so by setting `OCIS_BASE_DATA_PATH`. Therefore you need to create that directory and make it accessible to the user, you use to start oCIS. + +== Starting the oCIS service + +Initialize the oCIS configuration by running `ocis init --config-path /etc/ocis`. + +You can enable oCIS now by running `systemctl enable --now ocis`. It will ensure that oCIS also is restarted after a reboot of the host. + +If you need to restart oCIS because of configuration changes in `/etc/ocis/ocis.env`, run `systemctl restart ocis`. + +You can have a look at the logs of oCIS by issuing `journalctl -f -u ocis`. diff --git a/modules/developer/pages/ocis/deployment/ubernauten.adoc b/modules/developer/pages/ocis/deployment/ubernauten.adoc new file mode 100644 index 00000000..a99b0f89 --- /dev/null +++ b/modules/developer/pages/ocis/deployment/ubernauten.adoc @@ -0,0 +1,137 @@ += Installing ownCloud Infinite Scale at Ubernauten +:toc: right +:toclevels: 3 + + +== Uberspace and ownCloud Infinite Scale in 50 seconds + +This howto shows how to set up ownCloud Infinite Scale for a quick test. For convenience, we will use the free service from the Ubernauten -- Join us here: link:https://dashboard.uberspace.de/register?lang=en[Uberspace Registration Page]. They offer free of charge (for the first month) web hosting. + +In this documentation, we are assuming you already have an account there and it is configured for SSH access. This guide is using "ocis.uber.space" as a domain at Uberspace, version 4.0.3 of Infinite Scale and the local user "owncloud". Make sure you adapt the example code and scripts mentioned in this page to your needs Both username and domain will be set when you sign in to Uberspace. + +Installing ownCloud Infinite Scale on Ubernauten is pretty straigt-forward, you can do it in three steps that take less than a minute: Here's a short video that shows how fast the whole process actually is - thanks to Infinite Scale's cloud native architecture. link:/ocis/deployment/ubernauten_media/001-OCIS-in-50-seconds-2023-10-17.mkv[ownCloud Infinite Scale in 50 seconds] + +=== Three Steps to your Infinite UberSpace + +* Download the Infinite Scale binary and make it executable +---- +curl https://download.owncloud.com/ocis/ocis/stable/4.0.3/ocis-4.0.3-linux-amd64 --output ocis +chmod +x ocis +---- + +* Set some environment variables related to Uberspace (_Make sure you fill in YOUR domain!_) +---- +uberspace web backend set / --http --port 9200 +export OCIS_URL=https://ocis.uber.space +export PROXY_TLS=false +export PROXY_HTTP_ADDR=0.0.0.0:9200 +export PROXY_LOG_LEVEL=debug +---- + +* Start the `ocis` binary, first with the parameter `init` for initial configuration. This will also give you your unique login password for the user `admin`. Once finished, call `ocis start`: +---- +./ocis init +./ocis server +---- +Wait a few seconds, now you can visit the url of your uberspace server and login: + +image::ocis/deployment/ubernauten_media/login.png[,width=70%] + +To make it easier (and faster), here's the commands in a script called `ocis.install`: + +---- +#!/bin/bash +# This file is named ocis.install +# It downloads ocis, configures the environment varibles and starts +# ownCloud Infinite Scale on a ubernauten account. You can run it in your home directory + +curl https://download.owncloud.com/ocis/ocis/stable/4.0.3/ocis-4.0.3-linux-amd64 --output ocis +chmod +x ocis +uberspace web backend set / --http --port 9200 +export OCIS_URL=https://ocis.uber.space +export PROXY_TLS=false +export PROXY_HTTP_ADDR=0.0.0.0:9200 +export PROXY_LOG_LEVEL=debug +./ocis init +./ocis server +---- + +=== Service Management with Supervisord + +If you want `ocis` to run continuously, you need to configure `supervisord` (http://supervisord.org) which is the tool Uberspace is using for service management. + +You can start and stop services with `supervisorctl`, it will (re)read configuration files it finds in your home directory, under `etc/services.d/`, in `.ini` files. The content of these files is very simple, you only have to enter three lines, here is the example for Infinite Scale in `/home/owncloud/etc/services.d/ocis.ini`. + +---- +[program:ocis] +command="/home/owncloud/ocis.start" +startsecs=60 +---- + +`ocis.start` is a script that combines all of the commands above except for the download of the ocis binary. It looks like this: + +---- +#!/bin/bash +# This file is named ocis.start. +# It sets environment variables needed for uber.space needed for Infinite Scale +/usr/bin/uberspace web backend set / --http --port 9200 & +export OCIS_URL=https://ocis.uber.space +export PROXY_TLS=false +export PROXY_HTTP_ADDR=0.0.0.0:9200 +export PROXY_LOG_LEVEL=debug +/home/owncloud/ocis server +---- + +There are four supervisorctl commands that you will find useful (many more can be found in its documentation). You can use `supervisorctl status` to check which services managed by supervisorctl are running, a `supervisorctl reread` will be necessary after you changed the `ini` files, an `update` is applying changes, and `supervisorctl stop` will stop a running service: + +---- +[owncloud@ocis ~]$ supervisorctl status +ocis RUNNING pid 9813, uptime 0:01:40 +[owncloud@ocis ~]$ supervisorctl reread +No config updates to processes +[owncloud@ocis ~]$ supervisorctl update +---- +You can find all information on Supervisord and `supervisorctl` on its website: link:http://supervisord.org/running.html[Running Supervisord]. + +=== Updating ownCloud Infinite Scale + +Updating the ocis binary is simple: When a new version comes to life, just download the new `ocis` binary from the download server, replacing the old `ocis` executable on your uberspace server. + +Make a backup of your data and make sure you have read and understood the release notes of your new version , especially the "breaking changes" section before starting the binary. + +Don't worry, you can always go back to the older version you had installed, there's a long list of older versions available for download. + +Mind that if you want to re-configure, re-install or start a new version of ocis, make sure you have stopped supervisorctl from restarting ocis. Enter `supervisorctl stop ocis`, followed by a `killall -9 ocis` to make sure no ocis version is still running. + +=== Wiping and Clean Restart from Scratch + +This little script is removing your ocis installation (and _all of your data!_), replacing it with a new, clean ocis installation. Be careful and only use it for testing purposes. Specify your desired ocis version in the curl command. + +---- +#!/bin/bash +# This file is named ocis.reinstall +# It deletes the old ocis installation, fetches a new binary and starts ocis. +rm -rf .ocis +curl https://download.owncloud.com/ocis/ocis/stable/4.0.3/ocis-4.0.3-linux-amd64 --output ocis +chmod +x ocis +uberspace web backend set / --http --port 9200 +export OCIS_URL=https://ocis.uber.space +export PROXY_TLS=false +export PROXY_HTTP_ADDR=0.0.0.0:9200 +export PROXY_LOG_LEVEL=debug +./ocis init +./ocis server +---- + +=== Troubleshooting + +* SSL/TLS Certificates: Every Uberspace comes with its own HTTPS certificate via Let's Encrypt. See the link:https://manual.uberspace.de/web-https/[Uberspace - HTTPS] documentation for more details. + +* Error message about `jwt_secret`: If you get the following error message, then you probably forgot to run the `ocis init` command. If `ocis server` find an configuration hasn't been set up, it will complain like this: +---- +[owncloud@ocis ~]$ ./ocis server +The jwt_secret has not been set properly in your config for ocis. +Make sure your /home/mfeilner/.ocis/config config contains the proper values +(e.g. by running ocis init or setting it manually in the config/corresponding +environment variable). +---- diff --git a/modules/developer/pages/ocis/development/beta-testplan.adoc b/modules/developer/pages/ocis/development/beta-testplan.adoc new file mode 100644 index 00000000..71508b78 --- /dev/null +++ b/modules/developer/pages/ocis/development/beta-testplan.adoc @@ -0,0 +1,313 @@ += Beta testplan +:toc: right +:toclevels: 3 + += Beta Testing + +This document is supposed to give you some ideas how and what to test on ocis. It's not meant to be an extensive list of all tests to be done, rather it should help you, as beta-tester, to get started and enable you to get creative and create your own test-cases. link:https://twitter.com/sempf/status/514473420277694465[Derive from these examples, be creative, do unusual and unconventional things, to try to break things]. + +One option to create new test-cases and to stress the system is to examine what the link:https://owncloud.dev/ocis/development/testing/#testing-with-test-suite-natively-installed[API acceptance-tests] or the <> does, <> and do something a bit different with curl. This is also a good way to find out how APIs work that are not already fully documented. + +Some cases have suggested setup steps, but feel free to use other setups. This can include: +- different deployment methods (e.g. running single binary, docker-container, docker-compose setup, link:https://owncloud.dev/ocis/deployment/ocis_individual_services/[individual services in own docker containers]) +- different identity managers (e.g. link:https://owncloud.dev/ocis/deployment/ocis_ldap/[different external LDAP], internal IDM) +- different reverse proxies (e.g. link:https://owncloud.dev/ocis/deployment/ocis_traefik/[traefik]) +- different OpenIDConnect IDPs (e.g builtin IDP, link:https://owncloud.dev/ocis/deployment/ocis_keycloak/[keycloak], AzureAD) + - for some functionalities you will need an link:https://owncloud.dev/ocis/deployment/ocis_ldap/[LDAP server] where the IDP and oCIS both get the users from + - link:https://owncloud.dev/ocis/deployment/ocis_keycloak/[keycloak example] + - link:https://owncloud.dev/extensions/idp/configuration/[service configuration] +- different storage systems (decomposedFS on local POSIX (default), link:https://owncloud.dev/ocis/storage-backends/dcfsnfs/[decomposedFS on NFS], link:https://owncloud.dev/ocis/deployment/ocis_s3/[S3] ) + +It's a good idea to test ocis in the same environment where you are planning to use it later (with the LDAP server, storage system, etc. of your organisation). + += run oCIS +For a quick start, please have a look into the xref:../getting-started/#run-ocis.adoc[getting started documentation of oCIS] +If you would like +to access oCIS remotely please refer to the xref:../deployment/basic-remote-setup.adoc[Basic Remote Setup] section. + +== additional tips +- to allow basic auth (e.g. to easily access oCIS using curl commands) you have to set `PROXY_ENABLE_BASIC_AUTH=true` environment variable +- if you cannot use real SSL Certificates set `OCIS_INSECURE=true` as environment variable + + += Testplan + +== user / groups from LDAP + +Prerequisite: +- connect ocis to your preferred LDAP server +- create users and groups in LDAP +- start ocis with basic auth + +documentation resources: + - link:https://owncloud.dev/ocis/deployment/ocis_ldap/[configure ocis with LDAP] + - link:https://doc.owncloud.com/server/10.9/developer_manual/core/apis/ocs-share-api.html[sharing API is compatible to ownCloud 10] + - <> + +| Test Case | Expected Result | Comment | +|-------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|---------| +| share file / folder to a group | member of the group can access shared item | | +| share file / folder to a group, remove member from group in LDAP | removed member should not have access to the shared item | | +| share file / folder to a group with different permissions, as receiver try to violate the permissions | receiver should not be able to violate the permissions | | +| try to login with wrong credentials | login should not be possible | | +| set a quota in LDAP, upload files till the quota is exceeded | upload should work till quota is full, uploads should not work when quota is full | | +| try to access files / folders of other users | access should not be possible | | +| try to share with non-existing users and groups | sharing should not be possible | | +| try to share with user/groups-names that contain special characters | sharing should be possible, access shares with that user does not create any problems | | + +== other sharing + +should be tried in various ways and in different environments + +documentation resources: +- link:https://doc.owncloud.com/server/10.9/developer_manual/core/apis/ocs-share-api.html[sharing API is compatible to ownCloud 10] +- <> + +| Test Case | Expected Result | Comment | +|---------------------------------------------------------------------------------------|-----------------------------------------------------------------------|-----------------------------------------------------------| +| share a file/folder with the same name from different users | receiver can accept and access both file/folders and distinguish them | | +| share a file/folder with the same name but different permissions from different users | receiver can access both file/folders according to the permissions | | +| share a file/folder with the same name but different locations from one user | receiver can accept and access both file/folders and distinguish them | | +| share a file/folder back to the sharer | sharing back should not be possible | | +| re-share a file/folder with different permissions | sharing with lower permissions is possible, but not with higher | | +| decline received share | shared resource should not be shown to the receiver | | + + +== parallel deployment + +- link:https://owncloud.dev/ocis/deployment/ocis_ldap/[configure ocis with LDAP] +- link:https://owncloud.dev/ocis/deployment/oc10_ocis_parallel/[setup oC10 and ocis in parallel] +- create users and groups in LDAP + +documentation resources: +- link:https://doc.owncloud.com/server/10.9/developer_manual/core/apis/ocs-share-api.html[sharing API is compatible to ownCloud 10] + +| Test Case | Expected Result | Comment | +|------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------|---------| +| share file / folder to a group in one implementation (use different permissions), access the items with the other implementation, try to violate the permissions | receiver should not be able to violate the permissions | | +| share file / folder to a group, remove member from group in LDAP, try to access items with the removed member from both implementations | removed member should not have access to the shared item | | + +== Spaces + +Prerequisite: +- start ocis with basic auth +- variable declaration, for curl examples: +[source,shell] +---- +SERVER_URI=https://localhost:9200 +GRAPH_API_PATH=graph/v1.0 +SHARE_API_PATH=ocs/v2.php/apps/files_sharing/api/v1/shares +USER=admin +PASSWORD=admin +---- +- create a new user `curl -k -u $USER:$PASSWORD "$SERVER_URI/$GRAPH_API_PATH/users" -X POST -d'{"displayName":"Example User","mail":"example@example.org","onPremisesSamAccountName":"example","passwordProfile":{"password":"ThePassword"}}'` +- give the user the "Space Admin" role + 1. get the id of the user: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/users/ | jq '.id'` + 2. assign role user to role: `curl -k -u $USER:$PASSWORD $SERVER_URI/api/v0/settings/assignments-add -d'{"account_uuid":"","role_id":"2aadd357-682c-406b-8874-293091995fdd"}` +- create a space: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$GRAPH_API_PATH/drives" -X POST -d'{"Name":"Space for finance","driveType":"project", "description":"we need some space"}'` +- list spaces: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/me/drives` +- disable a space: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/ -X DELETE` +- delete a space: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/ -X DELETE -H'Purge: T'` +- restore a space: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/ -X PATCH -d"{}" -H"Restore: true"` +- rename the space: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/ -X PATCH -d'{"name":"नेपालि नाम"}'` +- change description of the space: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/ -X PATCH -d'{"description":"this contains important data"}'` +- change quota of the space `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/ -X PATCH -d'{"quota":{"total":}}'` +- change image of space: + 1. upload an image file to the `.spaces` folder: `curl -k -u $USER:$PASSWORD https://localhost:9200/dav/spaces//.space/ -T -v` + 2. note the id provided in the `Oc-Fileid` header + 3. set the image as "special-folder": `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/ -X PATCH -d'{"special":[{"specialFolder":{"name":"image"},"id":""}]}'` +- share a space: `curl -k -u $USER:$PASSWORD $SERVER_URI/$SHARE_API_PATH -d'space_ref=&shareType=7&shareWith=&role='` + - Roles: + - viewer + - editor + - manager +- share a resource within a space `curl -k -u $USER:$PASSWORD $SERVER_URI/$SHARE_API_PATH -d'shareType=0&shareWith=&space_ref=%2F&permissions=` + - possible permissions + - 1 = read + - 2 = update + - 4 = create + - 8 = delete + - 15 = read/write + - 16 = share + - 31 = All permissions +- WebDAV + - root: `https:///dav/spaces/` + - <> + +| Test Case | Expected Result | Comment | +|-------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|---------| +| create a space | space should exist | | +| create a space with special characters as a name & description | space should exist | | +| create a space, delete the space | space should not exist | | +| create a space, share the space with a user | space should be accessible | | +| create a space, share the space with a group | space should be accessible, space content is shared among all users | | +| create a space, share the space with a group, disable the space | space should not be accessible | | +| create a space, share the space with a user, disable the space, restore the space | space should be accessible | | +| create a space, disable the space, delete the space, restore the space | it should not be possible to restore the space | | +| create a space, disable the space, try to share the space | sharing the space should not be possible | | +| create a space, try to delete the space | it should not be possible to delete an enabled space | | +| create & share a space with a group with viewer role, do CRUD file/folder operations with WebDAV | space content is readable but neither space not content should not be writable | | +| create & share a space with a group with editor role, do CRUD file/folder operations with WebDAV | space and content should be writable | | +| create a space, try CRUD file/folder operations with WebDAV on the space with a user that its not shared with | space and content should not be accessible | | +| create a space with a quota, share the space, upload files till the quota is exceeded | upload should work till quota is full, uploads should not work when quota is full | | +| share file/folders from inside a space (see other sharing section) | sharing works and obeys the permissions | | +| create a space, rename the space | new name should be displayed in API calls and web | | +| create a space, change description of the space | new description should be displayed in API calls and web | | +| create a space, set quota, change quota of the space | new quota is obeyed | | +| create a space, set quota, change quota of the space to a value that is lower than the sum of data already stored | new quota is obeyed, new files cannot be uploaded | | +| try the various space operations with invalid data | good error output, server does not crash | | +| try the various space operations without the correct permissions | operations are not possible without sufficient permissions | | +| try the various space operations on personal and virtual spaces | server should not crash, good error responses | | +| try the various space operations sending invalid data (invalid numbers, wrong types, invalid JSON, etc.) | server should not crash, good error responses | | + + +== Web + +Prerequisite: +- link:https://owncloud.dev/ocis/deployment/ocis_ldap/[connect ocis to your preferred LDAP server] +- create users and groups in LDAP +- Use your preferred browser (Firefox 94-96, Chrome 92-97, Opera 81-82, Edge 96-97, Safari 14-15) to access the built-in webUI (by default: https://localhost:9200) + +| Test Case | Expected Result | Comment | +|--------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|---------| +| Login with the created user | User logs in. | | +| Create a text file. | Text editor can open, file is saved. | | +| Create a text file with special characters as name | file is created if the name is legal otherwise an error is displayed | | +| Modify a text file. | File can be modified, no problems found. | | +| Rename a file. | File is renamed. | | +| Upload a file. | File is uploaded, no problems found. | | +| Upload multiple files at once. | Files are uploaded, no problems found. | | +| delete all content of a folder at once. | Folder is cleaned, items are visible in the trashbin | | +| Overwrite a file by uploading a new version. | File is uploaded and overwritten, file versions are displayed | | +| Overwrite a file by uploading a new version, restore the original version. | File is restored correctly | | +| upload a huge file | File is uploaded, no problems found. | | +| upload a huge file, cancel the upload, restart the upload | Upload continues at the position where it was cancelled, file is uploaded completely. | | +| Remove a file. | File is removed correctly, it appears in the trashbin. | | +| Restore the deleted file from trashbin | File is restored correctly | | +| Remove multiple files that have the same name but are located in different folders | Files are removed correctly, they appear in the trashbin. | | +| Restore some of the deleted files from trashbin | Files are restored correctly in the correct folders. | | +| Restore some of the deleted files from trashbin, but delete the original containing folder before | Files are restored correctly | | +| Clean files from the trashbin | files are permanently deleted | | +| Create a lot of files, delete a lot of files, empty the trashbin | trashbin is cleaned | | +| Move a file inside a folder. | There are not problems on the process. | | +| Move a file inside a folder that already contains a file with the same name | File is not moved, content in the destination is not overwritten | | +| Create a folder. | Folder is created, no MKCOL problems appear. | | +| Create a folder with special characters as name | Folder is created if the name is legal otherwise an error is displayed | | +| Create a folder with a name of an already existing file/folder | Folder is not created, an error is displayed | | +| Create a folder with a lot of subfolders, use special characters in the name | Folder is created, no MKCOL problems appear. | | +| Delete a folder. | Folder is removed. | | +| Move a folder inside another. | No problems while moving the folder. | | +| open images in mediaviewer | files are displayed correctly. | | +| open videos in mediaviewer | files are displayed correctly. | | +| switch through videos and images in mediaviewer | files are displayed correctly. | | +| Share a file by public link. | Link is created and can be accessed. | | +| Share a folder by public link. | Link is created and can be accessed. | | +| Share a file with another user. | It is shared correctly. | | +| Share a folder with another user. | It is shared correctly. | | +| Share a file with a group. | It is shared correctly. | | +| Share a folder with a group. | It is shared correctly. | | +| Share a folder with userB giving edit permissions. As userB do CRUD operations on items inside the received folder | userB doesn't find any problem while interacting with files. | | +| Use your mobile device to access the UI | All elements reachable | | +| do tests mentioned in the <> section using the web-UI | | | + +== Desktop Client + +Prerequisite: +- link:https://owncloud.com/desktop-app/[install the desktop client on your preferred OS] +- start ocis +- connect a new account in the desktop client to ocis e.g. `https://localhost:9200` if you are running the server and the client on the same machine +- accept the self-signed certificate +- you will be redirected to the browser, accept the certificate there also +- login as any user +- allow the "ownCloud desktop app" to access ocis +- select a folder to sync + +| Test Case | Expected Result | Comment | +|---------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|---------| +| Set up two clients with the same user. Change files, add some, delete some, move some, create folders. | Changes sync properly in both clients without errors. | | +| Share a file using contextual menu with userB. | Option to share appears in the contextual menu and file is correctly shared. | | + + +== Mobile Clients (iOS || Android) + +Prerequisite: +- link:https://owncloud.com/mobile-apps/[install the ownCloud app on your mobile] +- start oCIS as described in xref:../deployment/basic-remote-setup.adoc[Basic Remote Setup] +- connect a new account in the mobile client to ocis +- accept the self-signed certificate +- you will be redirected to the browser, accept the certificate there also +- login as any user +- allow the mobile app to access ocis + + +| Test Case | Expected Result | Comment | +|-----------------------------------------------|------------------------------------------|---------| +| Connect to server, see files, download one. | No problems while downloading. | | +| Upload a file using mobile client. | No problems while uploading. | | +| Share a file with userB using mobile client. | File is correctly shared. | | + +== other WebDAV clients + +Prerequisite: +- start ocis with basic auth + +| Test Case | Expected Result | Comment | +|---------------------------------------------------------------|-------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------| +| use ocis as webDAV external storage in ownCloud 10 | resource access works | | +| access webDAV with your file-manager | that will not give you a good UX, but ocis should not crash | Urls: https://\/remote.php/webdav & https://\/remote.php/dav/files/\ | +| access webDAV with the "remote-files" function of LibreOffice | files are accessible and can be written back | | + += Tips for testing + +== WebDav +WebDav is accessible under different path +- old: https://\/remote.php/webdav +- new: https://\/remote.php/dav/files/\ +- spaces: https://\/dav/spaces/\ + +WebDav specifications can be found on http://webdav.org/ + +here some general WebDav request examples: + +variable declaration: +[source,shell] +---- +SERVER_URI=https://localhost:9200 +API_PATH=remote.php/webdav +USER=admin +PASSWORD=admin +---- +- list content of root folder: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$API_PATH/" -X PROPFIND` +- list content of sub-folder: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$API_PATH/f1" -X PROPFIND` +- create a folder: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$API_PATH/folder" -X MKCOL` +- delete a resource: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$API_PATH/folder" -X DELETE` +- rename / move a resource: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$API_PATH/folder" -X MOVE -H "Destination: $SERVER_URI/$API_PATH/renamed"` +- copy a resource: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$API_PATH/folder" -X COPY -H "Destination: $SERVER_URI/$API_PATH/folder-copy"` + +== decode HTTPS traffic with wireshark +To decode the HTTPS traffic we need the keys that were used to encrypt the traffic. Those keys are kept secret by the clients, but we can request the clients to save them in a specific file, so that wireshark can use them to decrypt the traffic again. + +1. create key file: `touch /tmp/sslkey.log` +2. start wireshark +3. set log filename + - navigate to Edit=>Preferences=>Protocols=>TLS + - in the field `(Pre)-Master-Secret log filename` enter `/tmp/sslkey.log` +4. decode as HTTP + - navigate to Analyze=>Decode As... + - click the + button + - set Field: `TLS Port; Value=9200; Type: Integer, base 10; Default (none); Current HTTP` (adjust the port if you are using another one than 9200) +5. start recording + - use `port 9200` as capture filter to only record ocis packages + - use `http` as display filter to see only decoded traffic +6. run test-software with `SSLKEYLOGFILE=/tmp/sslkey.log` as env. variable e.g. + - curl: `SSLKEYLOGFILE=/tmp/sslkey.log curl -k -u admin:admin https://localhost:9200/ocs/v1.php/cloud/users` + - Browser: `SSLKEYLOGFILE=/tmp/sslkey.log firefox` + - LibreOffice: `SSLKEYLOGFILE=/tmp/sslkey.log libreoffice` + - acceptance tests: `SSLKEYLOGFILE=/tmp/sslkey.log make test-acceptance-api ...` + +== format output +- piping _xml_ results to `xmllint` gives you nice formats. E.g. `curl -k --user marie:radioactivity "https://localhost:9200/ocs/v1.php/apps/files_sharing/api/v1/shares" | xmllint --format -` +- piping _json_ results to `jq` gives you nice formats. E.g. `curl -k --user marie:radioactivity "https://localhost:9200/ocs/v1.php/apps/files_sharing/api/v1/shares?format=json" | jq` + +== create edge cases +- link:https://github.com/minimaxir/big-list-of-naughty-strings[Big List of Naughty Strings] diff --git a/modules/developer/pages/ocis/development/build.adoc b/modules/developer/pages/ocis/development/build.adoc new file mode 100644 index 00000000..fd23f49b --- /dev/null +++ b/modules/developer/pages/ocis/development/build.adoc @@ -0,0 +1,46 @@ += Build +:toc: right +:toclevels: 3 + + +== Build requirements + +see xref:getting-started/#requirements.adoc[Development - Getting Started] + +== Get the sources + +git clone https://github.com/owncloud/ocis.git +cd ocis +// SHORTCODE: {{< / highlight >}} + +== Build the oCIS binary + +You only need to run following command if you have changed protobuf definitions or the frontend part in one of the extensions. Run the command in the root directory of the repository. Otherwise you can skip this step and proceed to build the oCIS binary. +This will usually modify multiple `embed.go` files because we embed the frontend build output in these `embed.go` files and a timestamp will be updated and also minor differences are expected between different Node.js versions. + +make generate +// SHORTCODE: {{< / highlight >}} + +The next step is to build the actual oCIS binary. Therefore you need to navigate to the subdirectory `ocis` and start the build process. + +cd ocis +make build +// SHORTCODE: {{< / highlight >}} + +After the build process finished, you can find the binary within the `bin/` folder (in `ocis/bin` relative to the oCIS repository root folder). + +Try to run it: `./bin/ocis h` + +== Build a local oCIS docker image + +If you are developing and want to run your local changes in a docker or docker-compose setup, you have to build an image locally. + +Therefore run following commands in the root of the oCIS repository: + +docker build -t owncloud/ocis:dev . +// SHORTCODE: {{< / highlight >}} + +Then you can test as usual via + +docker run --rm -ti owncloud/ocis:dev +// SHORTCODE: {{< / highlight >}} diff --git a/modules/developer/pages/ocis/development/continuous-integration.adoc b/modules/developer/pages/ocis/development/continuous-integration.adoc new file mode 100644 index 00000000..20d05494 --- /dev/null +++ b/modules/developer/pages/ocis/development/continuous-integration.adoc @@ -0,0 +1,52 @@ += Continuous Integration +:toc: right +:toclevels: 3 + + +oCIS uses link:https://www.drone.io/[DRONE] as CI system. You can find the pipeline logs link:https://drone.owncloud.com/owncloud/ocis[here] or in your PR. + +== Concepts + +The pipeline is defined in link:https://github.com/bazelbuild/starlark[Starlark] and transformed to YAML upon pipeline run. This enables us to do a highly dynamic and non repeating pipeline configuration. We enforce Starlark format guidelines with Bazel Buildifier. You can format the .drone.star file by running `make ci-format`. + +Upon running the pipeline, your branch gets merged to the master branch. This ensures that we always test your changeset if as it was applied to the master of oCIS. Please note that this does not apply to the pipeline definition (`.drone.star`). + +== Things done in CI + +- static code analysis +- linting +- running UI tests +- running ownCloud 10 test suite against oCIS +- build and release docker images +- build and release binaries +- build and release documentation + +== Flags in commit message and PR title + +You may add flags to your commit message or PR title in order to speed up pipeline runs and take load from the CI runners. + +- `[CI SKIP]`: no CI is run on the commit or PR + +- `[full-ci]`: deactivates the fail early mechanism and runs all available test (as default only smoke tests are run) + +=== Knowledge base + +- My pipeline fails because some CI related files or commands are missing. + + Please make sure to rebase your branch onto the latest master of oCIS. It could be that the pipeline definition (`.drone.star`) was changed on the master branch. This is the only file, that will not be auto merged to master upon pipeline run. So things could be out of sync. + +- How can I see the YAML drone pipeline definition? + + In order to see the Yaml pipeline definition you can use the drone-cli to convert the Starlark file. + + ``` + drone starlark + ``` + +[NOTE] +==== + If you experience a `"build" struct has no .title attribute` you need a newer version of drone-cli. + + You currently need to build it yourself from this link:https://github.com/drone/drone-cli[source code]. If you are not using master as source, please ensure that this link:https://github.com/drone/drone-cli/pull/175[PR] is included. +==== + diff --git a/modules/developer/pages/ocis/development/debugging.adoc b/modules/developer/pages/ocis/development/debugging.adoc new file mode 100644 index 00000000..a19061d0 --- /dev/null +++ b/modules/developer/pages/ocis/development/debugging.adoc @@ -0,0 +1,289 @@ += Debugging +:toc: right +:toclevels: 3 + + +== Debugging + +As a single binary for easy deployment running `ocis server` just forks itself to start all the services, which makes debugging those processes a little harder. + +Ultimately, we want to be able to stop a single service using e.g. `ocis kill web` so that you can start the service you want to debug in debug mode. We need to link:https://github.com/owncloud/ocis/issues/77[change the way we fork processes] though, otherwise the runtime will automatically restart a service if killed. + +=== Start ocis + +For debugging there are two workflows that work well, depending on your preferences. + +==== Use the debug binary and attach to the process as needed + +Run the debug binary with `OCIS_LOG_LEVEL=debug bin/ocis-debug server` and then find the service you want to debug using: + +[source,console] +---- +# ps ax | grep ocis +12837 pts/1 Sl+ 0:00 bin/ocis-debug server +12845 pts/1 Sl 0:00 bin/ocis-debug graph +12847 pts/1 Sl 0:00 bin/ocis-debug reva-auth-bearer +12849 pts/1 Sl 0:00 bin/ocis-debug ocs +12850 pts/1 Sl 0:00 bin/ocis-debug reva-storage-oc-data +12863 pts/1 Sl 0:00 bin/ocis-debug webdav +12874 pts/1 Sl 0:00 bin/ocis-debug reva-frontend +12897 pts/1 Sl 0:00 bin/ocis-debug reva-sharing +12905 pts/1 Sl 0:00 bin/ocis-debug reva-gateway +12912 pts/1 Sl 0:00 bin/ocis-debug reva-storage-home +12920 pts/1 Sl 0:00 bin/ocis-debug reva-users +12929 pts/1 Sl 0:00 bin/ocis-debug glauth +12940 pts/1 Sl 0:00 bin/ocis-debug reva-storage-home-data +12948 pts/1 Sl 0:00 bin/ocis-debug idp +12952 pts/1 Sl 0:00 bin/ocis-debug proxy +12961 pts/1 Sl 0:00 bin/ocis-debug thumbnails +12971 pts/1 Sl 0:00 bin/ocis-debug reva-storage-oc +12981 pts/1 Sl 0:00 bin/ocis-debug web +12993 pts/1 Sl 0:00 bin/ocis-debug api +12998 pts/1 Sl 0:00 bin/ocis-debug registry +13004 pts/1 Sl 0:00 bin/ocis-debug web +13015 pts/1 Sl 0:00 bin/ocis-debug reva-auth-basic +---- + +Then you can set a breakpoint in the service you need and attach to the process via processid. To debug the `reva-sharing` service the VS Code `launch.json` would look like this: + +[source,json] +---- +{ + "version": "0.2.0", + "configurations": [ + { + "name": "ocis attach", + "type": "go", + "request": "attach", + "mode": "local", + "processId": 12897 + } + ] +} +---- + +==== Start all services independently to replace one of them with a debug process + +1. You can use this `./ocis.sh` script to start all services independently, so they don't get restarted by the runtime when you kill them: + +[source,bash] +---- +#/bin/sh +LOG_LEVEL="debug" + +bin/ocis --log-level=$LOG_LEVEL micro & + +bin/ocis --log-level=$LOG_LEVEL glauth & +bin/ocis --log-level=$LOG_LEVEL graph & +#bin/ocis --log-level=$LOG_LEVEL hello & +bin/ocis --log-level=$LOG_LEVEL idp & +#bin/ocis --log-level=$LOG_LEVEL ocs & +bin/ocis --log-level=$LOG_LEVEL web & +bin/ocis --log-level=$LOG_LEVEL reva-auth-basic & +bin/ocis --log-level=$LOG_LEVEL reva-auth-bearer & +bin/ocis --log-level=$LOG_LEVEL reva-frontend & +bin/ocis --log-level=$LOG_LEVEL reva-gateway & +bin/ocis --log-level=$LOG_LEVEL reva-sharing & +bin/ocis --log-level=$LOG_LEVEL reva-storage-home & +bin/ocis --log-level=$LOG_LEVEL reva-storage-home-data & +bin/ocis --log-level=$LOG_LEVEL reva-storage-oc & +bin/ocis --log-level=$LOG_LEVEL reva-storage-oc-data & +bin/ocis --log-level=$LOG_LEVEL reva-storage-root & +bin/ocis --log-level=$LOG_LEVEL reva-users & +#bin/ocis --log-level=$LOG_LEVEL webdav + +bin/ocis --log-level=$LOG_LEVEL proxy & +---- + +2. Get the list of running processes: + +[source,console] +---- +# ps ax | grep ocis +12837 pts/1 Sl+ 0:00 bin/ocis-debug server +12845 pts/1 Sl 0:00 bin/ocis-debug graph +12847 pts/1 Sl 0:00 bin/ocis-debug reva-auth-bearer +12849 pts/1 Sl 0:00 bin/ocis-debug ocs +12850 pts/1 Sl 0:00 bin/ocis-debug reva-storage-oc-data +12863 pts/1 Sl 0:00 bin/ocis-debug webdav +12874 pts/1 Sl 0:00 bin/ocis-debug reva-frontend +12897 pts/1 Sl 0:00 bin/ocis-debug reva-sharing +12905 pts/1 Sl 0:00 bin/ocis-debug reva-gateway +12912 pts/1 Sl 0:00 bin/ocis-debug reva-storage-home +12920 pts/1 Sl 0:00 bin/ocis-debug reva-users +12929 pts/1 Sl 0:00 bin/ocis-debug glauth +12940 pts/1 Sl 0:00 bin/ocis-debug reva-storage-home-data +12948 pts/1 Sl 0:00 bin/ocis-debug idp +12952 pts/1 Sl 0:00 bin/ocis-debug proxy +12961 pts/1 Sl 0:00 bin/ocis-debug thumbnails +12971 pts/1 Sl 0:00 bin/ocis-debug reva-storage-oc +12981 pts/1 Sl 0:00 bin/ocis-debug web +12993 pts/1 Sl 0:00 bin/ocis-debug api +12998 pts/1 Sl 0:00 bin/ocis-debug registry +13004 pts/1 Sl 0:00 bin/ocis-debug web +13015 pts/1 Sl 0:00 bin/ocis-debug reva-auth-basic +---- + +3. Kill the service you want to start in debug mode: + +[source,console] +---- +# kill 17628 +---- + +4. Start the service you are interested in debug mode. When using make to build the binary there is already a `bin/ocis-debug` binary for you. When running an IDE tell it which service to start by providing the corresponding sub command, e.g. `bin\ocis-debug reva-frontend`. + +=== Debugging the ocis in a docker container + +Remote debugging is the debug mode commonly used to work with a debugger and target running on a remote machine or a container for example a wopi stack `deployments/examples/ocis_full/docker-compose.yml`. Docker compose lets us define a compose application model through multiple compose files. When doing so, compose follows certain rules to merge compose files. See link:https://docs.docker.com/compose/compose-file/13-merge/[Merge and override] in the Compose Specification. Based on this rules, we added the extra files `deployments/examples/ocis_full/debug-ocis.yml`, `deployments/examples/ocis_full/debug-collaboration-collabora.yml`, `deployments/examples/ocis_full/debug-collaboration-onlyoffice.yml` that overwrites the `command` attribute and extends the `ports` attribute. +Below we describe the steps how to build the image, run the docker-compose and connect via remote debugger. +1. Build the image: +[source,bash] +---- +cd github.com/owncloud/ocis/ocis +make debug-docker +---- +2. Change the tag label: +[source,bash] +---- +export OCIS_DOCKER_TAG=debug +---- +3. Run docker compose +Building the docker compose command depends on what you want to debug, for example `ocis` and `collaboration` with the `collabora` supports. +[source,bash] +---- +docker compose -f docker-compose.yml -f ocis.yml -f collabora.yml -f debug-ocis.yml -f debug-collaboration-collabora.yml up -d +---- +4. Connect to remote `delve` +* For the VS Code add the configuration to the `.vscode/launch.json` link:https://github.com/golang/vscode-go/wiki/debugging#remote-debugging[https://github.com/golang/vscode-go/wiki/debugging#remote-debugging] +[source,json] +---- + { + "name": "Debug remote ocis :40000", + "type": "go", + "request": "attach", + "mode": "remote", + "port": 40000, + "host": "localhost", // optional + "trace": "verbose", // optional + "showLog": true // optional +}, +{ + "name": "Debug remote collaboration collabora :40001", + "type": "go", + "request": "attach", + "mode": "remote", + "port": 40001, + "host": "localhost", // optional + "trace": "verbose", // optional + "showLog": true // optional +}, +{ + "name": "Debug remote collaboration onlyoffice :40002", + "type": "go", + "request": "attach", + "mode": "remote", + "port": 40002, + "host": "localhost", // optional + "trace": "verbose", // optional + "showLog": true // optional +}, +---- +* For the Jetbrains Goland add the configuration following the docs link:https://www.jetbrains.com/help/go/go-remote.html[https://www.jetbrains.com/help/go/go-remote.html] + + +=== Gather error messages + +We recommend you collect all related information in a single file or in a GitHub issue. Let us start with an error that pops up in the Web UI: + +[quote] +____ +Error while sharing. +error sending a grpc stat request +____ + + +This popped up when I tried to add `marie` as a collaborator in ownCloud Web. That triggers a request to the server which I copied as curl. We can strip a lot of headers and the gist of it is: + +[source,console] +---- +# curl 'https://localhost:9200/ocs/v1.php/apps/files_sharing/api/v1/shares' -d 'shareType=0&shareWith=marie&path=%2FNeuer+Ordner&permissions=1' -u einstein:relativity -k -v | xmllint -format - +[... headers ...] + + + + error + 998 + error sending a grpc stat request + + +---- + +[NOTE] +==== +The username and password only work when basic auth is available. Otherwise you have to obtain a bearer token, e.g. by grabbing it from the browser. +==== + +[CAUTION] +==== +TODO add ocis cli tool to obtain a bearer token. +==== + + +We also have a few interesting log entries: + +---- +0:43PM INF home/jfd/go/pkg/mod/github.com/cs3org/reva@v0.0.2-0.20200318111623-a2f97d4aa741/internal/grpc/interceptors/log/log.go:69 > unary code=OK end="18/Mar/2020:22:43:40 +0100" from=tcp://[::1]:44078 pid=17836 pkg=rgrpc start="18/Mar/2020:22:43:40 +0100" time_ns=95841 traceid=b4eb9a9f45921f7d3632523ca32a42b0 uri=/cs3.storage.registry.v1beta1.RegistryAPI/GetStorageProvider user-agent=grpc-go/1.26.0 +10:43PM ERR home/jfd/go/pkg/mod/github.com/cs3org/reva@v0.0.2-0.20200318111623-a2f97d4aa741/internal/grpc/interceptors/log/log.go:69 > unary code=Unknown end="18/Mar/2020:22:43:40 +0100" from=tcp://[::1]:43910 pid=17836 pkg=rgrpc start="18/Mar/2020:22:43:40 +0100" time_ns=586115 traceid=b4eb9a9f45921f7d3632523ca32a42b0 uri=/cs3.gateway.v1beta1.GatewayAPI/Stat user-agent=grpc-go/1.26.0 +10:43PM ERR home/jfd/go/pkg/mod/github.com/cs3org/reva@v0.0.2-0.20200318111623-a2f97d4aa741/internal/http/services/owncloud/ocs/reqres.go:94 > error sending a grpc stat request error="rpc error: code = Unknown desc = gateway: error calling Stat: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp [::1]:9152: connect: connection refused\"" pid=17832 pkg=rhttp traceid=b4eb9a9f45921f7d3632523ca32a42b0 +---- + +[CAUTION] +==== +TODO return the trace id in the response so we can correlate easier. For reva tracked in https://github.com/cs3org/reva/issues/587 +==== + + +The last line gives us a hint where the log message originated: `.../github.com/cs3org/reva@v0.0.2-0.20200318111623-a2f97d4aa741/internal/http/services/owncloud/ocs/reqres.go:94`. Which looks like this: + +[source,go] +---- +89: // WriteOCSResponse handles writing ocs responses in json and xml +90: func WriteOCSResponse(w http.ResponseWriter, r *http.Request, res *Response, err error) { +91: var encoded []byte +92: +93: if err != nil { +94: appctx.GetLogger(r.Context()).Error().Err(err).Msg(res.OCS.Meta.Message) +95: } +---- + +Ok, so this seems to be a convenience method that is called from multiple places and also handles errors. Unfortunately, this hides the actual source of the error. We could set a breakpoint in line 94 and reproduce the problem, which can be a lot harder than just clicking the share button or sending a curl request again. So let us see what else the log tells us. + +The previous line tells us that a Stat request failed: `uri=/cs3.gateway.v1beta1.GatewayAPI/Stat`. This time the line is written by the grpc log interceptor. What else is there? + +The first line tells us that looking up the responsible storage provider seems to have succeeded: `uri=/cs3.storage.registry.v1beta1.RegistryAPI/GetStorageProvider`. + +At this point it your familiarity with the codebase starts to become a factor. If you are new you should probably go back to setting a break point on the log line and check the stack trace. + +Debug wherever the call trace leads you to ... good luck! + +=== Managing dependencies and testing changes + +You can either run and manage the services independently, or you can update the `go.mod` file and replace dependencies with your local version. + +To debug the reva frontend we need to add two replacements: + +---- +// use the local ocis-reva repo +replace github.com/owncloud/ocis-reva => ../ocis-reva +// also use the local reva repo +replace github.com/cs3org/reva => ../reva +---- + +[NOTE] +==== +The username and password only work when basic auth is available. Otherwise you have to obtain a bearer token, e.g. by grabbing it from the browser. +==== + + +Rebuild ocis to make sure the dependency is used. It should be sufficient to just restart the service you want to debug. diff --git a/modules/developer/pages/ocis/development/envvars.adoc b/modules/developer/pages/ocis/development/envvars.adoc new file mode 100644 index 00000000..2a20f90e --- /dev/null +++ b/modules/developer/pages/ocis/development/envvars.adoc @@ -0,0 +1,8 @@ += Environment Variables +:toc: right +:toclevels: 3 + +Environment variables are an essential part of configuring services. + +If you are going to create new ones or deprecate existing ones, you must read the xref:services/general-info/envvars/envvar-naming-scopes.adoc[Envvar Naming Scope] and the +xref:services/general-info/envvars/deprecating-variables.adoc[Deprecating Variables] documentation for more details first before doing so. diff --git a/modules/developer/pages/ocis/development/extensions.adoc b/modules/developer/pages/ocis/development/extensions.adoc new file mode 100644 index 00000000..99baf678 --- /dev/null +++ b/modules/developer/pages/ocis/development/extensions.adoc @@ -0,0 +1,60 @@ += Extensions +:toc: right +:toclevels: 3 + +oCIS is all about files, sync and share - but most of the time there is more you want to do with your files, e.g. having a different view on your photo collection or editing your offices files in an online file editor. ownCloud 10 faced the same problem and solved it with `applications`, which can extend the functionality of ownCloud 10 in a wide range. Since oCIS is different in its architecture compared to ownCloud 10, we had to come up with a similar (yet slightly different) solution. To extend the functionality of oCIS, you can write or install `extensions`. An extension is basically any running code which integrates into oCIS and provides functionality to oCIS and its users. Because extensions are just microservices providing an API, you can technically choose any programming language you like - a huge improvement to ownCloud 10, where it was nearly impossible to use a different programming language than PHP. + +We will now introduce you to the oCIS extension system and show you how you can create a custom extension yourself. + +== Extension examples + +Technically every service in oCIS is an extension, even if oCIS would not really work without some of them. Therefore, you can draw inspiration from any of the plenty of extensions in the link:https://github.com/owncloud/ocis[oCIS monorepo]. + +Besides these "default" extensions in the oCIS monorepo, there are two more extensions you should be aware of: + +- link:https://github.com/owncloud/ocis-hello[Hello] +- link:https://github.com/owncloud/ocis-wopiserver[WOPI server] + +Differences between the extensions maintained inside the oCIS monorepo and the ones maintained in their own repository are: + +- extensions inside the link:https://github.com/owncloud/ocis[oCIS monorepo] are all written in Go, whereas other extensions may choose the programming language freely +- extensions inside the oCIS monorepo heavily share tooling to reduce maintenance efforts, whereas other extensions may use different tooling (e.g. a different CI system) +- extensions inside the oCIS monorepo will be all build into one binary and started with the `ocis server` command. All other extensions must be started individually besides oCIS. + + +For quickstart purposes we also offer a link:https://github.com/owncloud/boilr-ocis-extension[template project] which can be used to generate all the boilerplate code for you. But you also can decide to use your own project layout or even a different programming language. + + +== Integration into oCIS + +Depending on the functionality of your extension, you might need to integrate with one or multiple of the components of oCIS mentioned below. + +=== ownCloud Web + +If your extension is not just doing something in the background, you will need a UI in order to allow the user to interact with your extension. You could just provide your own web frontend for that purpose, but for a better user experience you can easily integrate into the web frontend of oCIS, the new link:https://github.com/owncloud/web[ownCloud Web]. + +ownCloud Web allows you to write an extension for itself and therefore offers a seamless user experience. Upon login, the user will be able to use the application switcher to switch between the files view, settings and other available and installed extensions, yours included. Furthermore it is also possible to register your extension for different file actions. As an example, you could offer your extension to the user for creating and editing office documents. The user will then be able to create or open a file with your application directly from the files view. How to provide create an extension for ownCloud Web can be seen best in link:https://github.com/owncloud/ocis-hello/blob/master/ui/app.js[the Hello extension], whereas plain file handling without any web frontend is available in the link:https://github.com/owncloud/ocis-wopiserver/blob/master/ui/app.js[WOPI server extension]. + +To make ownCloud Web pick up your extension, you need to activate it in the configuration like seen in the link:https://owncloud.dev/extensions/ocis_hello/running/#configure-and-start-ocis[Hello extension]. + +For a consistent look and feel, ownCloud Web uses an external design library, the link:https://github.com/owncloud/owncloud-design-system[ownCloud design system]. Since its classes and components are available through the wrapping `web runtime`, we highly recommend you to leverage it in your extension as well. + +=== Settings + +An extension likely has some behaviour which the user can configure. Fundamental configuration will often be done by administrators during deployment, via configuration files or by setting environment variables. But for other settings, which are supposed to change more often or which are even user specific, this is not a viable way. Therefore you need to offer the users a UI where they can configure your extension to their liking. Because implementing something like this is a repetitive task among extensions, oCIS already offers the settings extensions which does that for your extension. Your extension just needs to register settings bundles, respective permissions and finally read the current values from the settings service. You can read more on that on the xref:../../services/settings.adoc[settings extension] and see how link:https://owncloud.dev/extensions/ocis_hello/settings/[oCIS Hello uses these settings]. + +=== Proxy + +The Proxy is an API gateway and acts as the single connection point where all external request from users and devices need to pass through. + +To make sure that requests can reach your extension's API, you need to register one or multiple endpoints at the proxy. The registration is an easy task and can be seen best on the link:https://owncloud.dev/extensions/ocis_hello/running/#configure-and-start-ocis[oCIS Hello example]. + +As files in ownCloud must always stay private (unless you share them with your friends or coworkers), requests to oCIS have an authenticated user context. This user context is also available to your extension and can be used to interact with the user's files. How to get the user context and authentication can be seen on the link:https://owncloud.dev/extensions/ocis_hello/settings/#account-uuid[oCIS Hello example]. + +=== Storage + +oCIS leverages the CS3 APIs and link:https://github.com/cs3org/reva[CS3 REVA] as a storage system because it offers a very flexible setup and supports a variety of storage backends like EOS, S3 and of course your local hard drive. REVA makes it easy to support more storage backends as needed. + +If you need to interact with files directly, you have the full power of the link:https://cs3org.github.io/cs3apis/[CS3 APIs] in your hand. With the user context and the user's authentication token, which your extensions gets from the proxy, your extension can make these request in behalf of the user. + +If your extension needs to store persistent data which is not supposed to live in the user's home folder, there is also a so-called metadata storage, intended for exactly that purpose. You should always use the metadata storage in favor of the local filesystem for persistent files, because your extension will then automatically use the storage backend the oCIS admin decides to use. For a temporary cache it is perfectly fine to use the local filesystem. diff --git a/modules/developer/pages/ocis/development/getting-started.adoc b/modules/developer/pages/ocis/development/getting-started.adoc new file mode 100644 index 00000000..d3ad69b1 --- /dev/null +++ b/modules/developer/pages/ocis/development/getting-started.adoc @@ -0,0 +1,49 @@ += Getting Started +:toc: right +:toclevels: 3 + + +== Requirements + +We want contribution to oCIS and the creation of extensions to be as easy as possible. +So we are trying to reflect this in the tooling. It should be kept simple and quick to be set up. + +Besides standard development tools like git and a text editor, you need the following software for development: + +- Go >= v1.22 (link:https://golang.org/doc/install[install instructions]) +- pnpm (link:https://pnpm.io/installation[install instructions]) +- docker (link:https://docs.docker.com/get-docker/[install instructions]) +- docker-compose (link:https://docs.docker.com/compose/install/[install instructions]) + +If you find tools needed besides the mentioned above, please feel free to open an issue or open a PR. + +== Repository structure + +oCIS consists of multiple microservices, also called extensions. We started by having standalone repositories for each of them, but quickly noticed that this adds a time-consuming overhead for developers. So we ended up with a monorepo housing all the extensions in one repository. + +Each extension lives in a subfolder (e.g. `accounts` or `settings`) within this repository as an independent Go module, following the link:https://github.com/golang-standards/project-layout[golang-standard project-layout]. They have common Makefile targets and can be used to change, build and run individual extensions. This allows us to version and release each extension independently. + +The `ocis` folder contains our link:https://github.com/asim/go-micro/[go-micro] and link:https://github.com/thejerf/suture[suture] based runtime. It is used to import all extensions and implements commands to manage them, similar to a small orchestrator. With the resulting oCIS binary you can start single extensions or even all extensions at the same time. + +The `docs` folder contains the source for the xref:../.adoc[oCIS documentation]. + +The `deployments` folder contains documented deployment configurations and templates. On a single node, running a single ocis runtime is a resource efficient way to deploy ocis. For multiple nodes docker compose or helm charts for kubernetes examples can be found here. + +The `scripts` folder contains scripts to perform various build, install, analysis, etc. operations. + +== Starting points + +Depending on what you want to develop there are different starting points. These will be described below. + +=== Developing oCIS + +If you want to contribute to oCIS: + +- see link:https://github.com/owncloud/ocis#contributing[contribution guidelines] +- make sure the tooling is set up by xref:build.adoc[building oCIS] and xref:build-docs.adoc[building the docs] +- create or pick an link:https://github.com/owncloud/ocis/issues[open issue] to develop on and mention in the issue that you are working on it +- open a PR and get things done + +=== Developing extensions + +If you want to develop an extension, start here: xref:extensions.adoc[Extensions] diff --git a/modules/developer/pages/ocis/development/index.adoc b/modules/developer/pages/ocis/development/index.adoc new file mode 100644 index 00000000..4d32858f --- /dev/null +++ b/modules/developer/pages/ocis/development/index.adoc @@ -0,0 +1,4 @@ += Development +:toc: right +:toclevels: 3 + diff --git a/modules/developer/pages/ocis/development/profiling.adoc b/modules/developer/pages/ocis/development/profiling.adoc new file mode 100644 index 00000000..839953ff --- /dev/null +++ b/modules/developer/pages/ocis/development/profiling.adoc @@ -0,0 +1,140 @@ += Profiling +:toc: right +:toclevels: 3 + + += 0. Prerequisites + +- Go development kit of a link:https://golang.org/doc/devel/release.html#policy[supported version]. + Follow link:http://golang.org/doc/code.html[these instructions] to install the + go tool and set up GOPATH. +- Graphviz: http://www.graphviz.org/. Used to generate graphic visualizations of profiles, which this example setup does. + +The only way to enable the profiler currently is to explicitly select which areas to collect samples for. In order to do this, the following steps have to be followed. + +== 1. Clone Reva + +Reva is the reference implementation of the CS3 APIs that we use for our daily business between oCIS and its storages. It is in charge of accessing the storage, as well as managing shares. Because of this fact, the examples will modify code in this dependency. You can think of Reva as the framework we use in order to interface with different storage providers. + +`git clone github.com/cs3org/reva` + +== 2. Patch reva with the area that you want sampled. + +For the purposes of these docs let's use the WebDAV `PROPFIND` path. This patch is needed in order to have the WebDAV process reporting profiling traces to the `pprof`. + +[source,diff] +---- +diff --git a/internal/http/services/owncloud/ocdav/propfind.go b/internal/http/services/owncloud/ocdav/propfind.go +index 0e9c99be..f271572f 100644 +--- a/internal/http/services/owncloud/ocdav/propfind.go ++++ b/internal/http/services/owncloud/ocdav/propfind.go +@@ -32,6 +32,8 @@ import ( + "strings" + "time" + ++ _ "net/http/pprof" ++ + userv1beta1 "github.com/cs3org/go-cs3apis/cs3/identity/user/v1beta1" + rpc "github.com/cs3org/go-cs3apis/cs3/rpc/v1beta1" + link "github.com/cs3org/go-cs3apis/cs3/sharing/link/v1beta1" +@@ -311,6 +313,12 @@ func requiresExplicitFetching(n *xml.Name) bool { + return true + } + ++func init() { ++ go func() { ++ http.ListenAndServe(":1234", nil) ++ }() ++} ++ + // from https://github.com/golang/net/blob/e514e69ffb8bc3c76a71ae40de0118d794855992/webdav/xml.go#L178-L205 + func readPropfind(r io.Reader) (pf propfindXML, status int, err error) { + c := countingReader{r: r} +---- + +The previous patch will: + +1. import `net/http/pprof`, which will register debug handlers in `DefaultServeMux`. +2. define a `init()` function that starts an HTTP server with the previously registered handlers. + +With everything running one should have access to http://localhost:1234/debug/pprof/ + +== 3. Replace reva in oCIS go.mod with local version and build a new binary + +In Go, the `go.mod` file controls the dependencies of your module. Because we patched an external library, Go provides with a mechanism to overwrite an existing dependency with one on your local machine, which we previously installed. + +[source,diff] +---- +diff --git a/go.mod b/go.mod +index 131d14d7b..9668c38e4 100644 +--- a/go.mod ++++ b/go.mod +@@ -78,6 +78,7 @@ require ( + + replace ( + github.com/crewjam/saml => github.com/crewjam/saml v0.4.5 ++ github.com/cs3org/reva => path/to/your/reva + go.etcd.io/etcd/api/v3 => go.etcd.io/etcd/api/v3 v3.0.0-20210204162551-dae29bb719dd + go.etcd.io/etcd/pkg/v3 => go.etcd.io/etcd/pkg/v3 v3.0.0-20210204162551-dae29bb719dd + ) +---- + +Make sure to replace `github.com/cs3org/reva => path/to/your/reva` with the correct location of your reva. + +== 4. Build a new ocis binary + +Using the new dependency with the pprof patch. + +From owncloud/ocis root: + +[source,console] +---- +$ cd ocis +$ make clean build +---- + +== 5. Start oCIS server + +From owncloud/ocis root: + +[source,console] +---- +$ ocis/bin/ocis server +---- + +== 6. Run `pprof` + +link:https://github.com/google/pprof[Pprof] is a tool developed at Google. It is a tool for visualization and analysis of profiling data. It will take the reported profiled data from our server, and represent it in a meaningful manner. + +=== Install pprof + +If `pprof` is not installed make sure to get it; one way of installing it is using the Go tools: + +[source,console] +---- +$ go get -u github.com/google/pprof +---- + +=== Collecting samples + +Collect 30 seconds of samples: + +[source,console] +---- +$ pprof -web http://:1234/debug/pprof/profile\?seconds\=30 +---- + +Once the collection is done a browser tab will open with the result `svg`, looking similar to this: + +image::https:/i.imgur.com/vo0EbcX.jpg[img] + +For references on how to interpret this graph, link:https://github.com/google/pprof/blob/master/doc/README.md#interpreting-the-callgraph[continue reading here]. + +== Room for improvement + +Because these docs are intended to be read by developers they are quite technical in content. Requiring the user to alter the code. This is done so that we do not include, or assume, third party dependencies such as Graphviz in our binary, making it heavier. Having said this, the profiler is only meant to be used in development + +== References + +- https://medium.com/swlh/go-profile-your-code-like-a-master-1505be38fdba +- https://dave.cheney.net/2013/07/07/introducing-profile-super-simple-profiling-for-go-programs diff --git a/modules/developer/pages/ocis/development/testing.adoc b/modules/developer/pages/ocis/development/testing.adoc new file mode 100644 index 00000000..32ed7e89 --- /dev/null +++ b/modules/developer/pages/ocis/development/testing.adoc @@ -0,0 +1,730 @@ += Acceptance Testing +:toc: right +:toclevels: 3 + + +To run tests in the test suite you have two options. You may go the easy way and just run the test suite in docker. But for some tasks you could also need to install the test suite natively, which requires a little more setup since PHP and some dependencies need to be installed. +[NOTE] +==== +To run the tests PHP version ">= 8.2.0" is required. +==== + + +Both ways to run tests with the test suites are described here. + +== Running Test Suite in Docker + +Let's see what is available. Invoke the following command from within the root of the oCIS repository. + +[source,bash] +---- +make -C tests/acceptance/docker help +---- + +Basically we have two sources for feature tests and test suites: + +- link:https://github.com/owncloud/ocis/tree/master/tests/acceptance/features[oCIS feature test and test suites] +- link:https://github.com/owncloud/ocis/tree/master/tests/acceptance/features[tests and test suites transferred from ownCloud core, they have prefix coreApi] + +At the moment, both can be applied to oCIS since the api of oCIS is designed to be compatible with ownCloud. + +As a storage backend, we offer oCIS native storage, also called `ocis`. This stores files directly on disk. Along with that we also provide `S3` storage driver. + +You can invoke two types of test suite runs: + +- run a full test suite, which consists of multiple feature tests +- run a single feature or single scenario in a feature + +=== Run Full Test Suite + +==== Local oCIS Tests (prefix `api`) + +The names of the full test suite make targets have the same naming as in the CI pipeline. See the available local oCIS specific test suites link:https://github.com/owncloud/ocis/tree/master/tests/acceptance/features[here]. They can be run with `ocis` storage and `S3` storage. + +For example, command: + +[source,bash] +---- +make -C tests/acceptance/docker localApiTests-apiGraph-ocis +---- + +runs the same tests as the `localApiTests-apiGraph-ocis` CI pipeline, which runs the oCIS test suite "apiGraph" against the oCIS server with ocis storage. + +And command: + +[source,bash] +---- +make -C tests/acceptance/docker localApiTests-apiGraph-s3ng +---- + +runs the oCIS test suite `apiGraph` against the oCIS server with s3 storage. + +[NOTE] +==== +While running the tests, oCIS server is started with link:https://github.com/owncloud/ocis/blob/master/tests/ociswrapper/README.md[ociswrapper] (i.e. `WITH_WRAPPER=true`) by default. In order to run the tests without ociswrapper, provide `WITH_WRAPPER=false` when running the tests. For example: + +[source,bash] +---- +WITH_WRAPPER=false \ +BEHAT_FEATURE='tests/acceptance/features/apiGraphUser/createUser.feature:26' \ +make -C tests/acceptance/docker test-ocis-feature-ocis-storage +---- + +But some test suites that are tagged with `@env-config` require the oCIS server to be run with ociswrapper. So, running those tests require `WITH_WRAPPER=true` (default setting). +==== + + +[NOTE] +==== +To run the tests that require an email server (tests tagged with `@email`), you need to provide `START_EMAIL=true` while running the tests. + +[source,bash] +---- +START_EMAIL=true \ +OCIS_ADD_RUN_SERVICES=notifications \ +BEHAT_FEATURE='tests/acceptance/features/apiNotification/notification.feature' \ +make -C tests/acceptance/docker test-ocis-feature-ocis-storage +---- + +==== + + +[NOTE] +==== +To run the tests that require tika service (tests tagged with `@tikaServiceNeeded`), you need to provide `START_TIKA=true` while running the tests. + +[source,bash] +---- +START_TIKA=true \ +BEHAT_FEATURE='tests/acceptance/features/apiSearchContent/contentSearch.feature' \ +make -C tests/acceptance/docker test-ocis-feature-ocis-storage +---- + +==== + + +[NOTE] +==== +To run the tests that require an antivirus service (tests tagged with `@antivirus`), you need to provide the following environment variables while running the tests. + +[source,bash] +---- +START_ANTIVIRUS=true \ +OCIS_ASYNC_UPLOADS=true \ +OCIS_ADD_RUN_SERVICES=antivirus \ +POSTPROCESSING_STEPS=virusscan \ +BEHAT_FEATURE='tests/acceptance/features/apiAntivirus/antivirus.feature' \ +make -C tests/acceptance/docker test-ocis-feature-ocis-storage +---- + +==== + + +==== Tests Transferred From ownCloud Core (prefix `coreApi`) + +Command `make -C tests/acceptance/docker Core-API-Tests-ocis-storage-3` runs the same tests as the `Core-API-Tests-ocis-storage-3` CI pipeline, which runs the third (out of ten) test suite groups transferred from ownCloud core against the oCIS server with ocis storage. + +And `make -C tests/acceptance/docker Core-API-Tests-s3ng-storage-3` runs the third (out of ten) test suite groups transferred from ownCloud core against the oCIS server with s3 storage. + +=== Run Single Feature Test + +The tests for a single feature (a feature file) can also be run against the different storage backends. To do that, multiple make targets with the schema _test-_\_-feature-_\__ are available. To select a single feature you have to add an additional `BEHAT_FEATURE=` parameter when invoking the make command. + +For example; + +[source,bash] +---- +BEHAT_FEATURE='tests/acceptance/features/apiGraphUser/createUser.feature' \ +make -C tests/acceptance/docker test-ocis-feature-ocis-storage +---- + +[NOTE] +==== +`BEHAT_FEATURE` must be pointing to a valid feature file +==== + + +And to run a single scenario in a feature, you can do: + +[NOTE] +==== +A specific scenario from a feature can be run by adding `:` at the end of the feature file path. For example, to run the scenario at line 26 of the feature file `apiGraphUser/createUser.feature`, simply add the line number like this: `apiGraphUser/createUser.feature:26`. Note that the line numbers mentioned in the examples might not always point to a scenario, so always check the line numbers before running the test. +==== + + +[source,bash] +---- +BEHAT_FEATURE='tests/acceptance/features/apiGraphUser/createUser.feature:26' \ +make -C tests/acceptance/docker test-ocis-feature-ocis-storage +---- + +Similarly, with S3 storage; + +[source,bash] +---- +# run a whole feature +BEHAT_FEATURE='tests/acceptance/features/apiGraphUser/createUser.feature' \ +make -C tests/acceptance/docker test-ocis-feature-s3ng-storage + +# run a single scenario +BEHAT_FEATURE='tests/acceptance/features/apiGraphUser/createUser.feature:26' \ +make -C tests/acceptance/docker test-ocis-feature-s3ng-storage +---- + +In the same way, tests transferred from ownCloud core can be run as: + +[source,bash] +---- +# run a whole feature +BEHAT_FEATURE='tests/acceptance/features/coreApiAuth/webDavAuth.feature' \ +make -C tests/acceptance/docker test-core-feature-ocis-storage + +# run a single scenario +BEHAT_FEATURE='tests/acceptance/features/coreApiAuth/webDavAuth.feature:15' \ +make -C tests/acceptance/docker test-core-feature-ocis-storage +---- + +[NOTE] +==== +The test suites transferred from ownCloud core have `coreApi` prefixed +==== + + +=== oCIS Image to Be Tested (Skip Local Image Build) + +By default, the tests will be run against the docker image built from your current working state of the oCIS repository. For some purposes it might also be handy to use an oCIS image from Docker Hub. Therefore, you can provide the optional flag `OCIS_IMAGE_TAG=...` which must contain an available docker tag of the link:https://hub.docker.com/r/owncloud/ocis[owncloud/ocis registry on Docker Hub] (e.g. 'latest'). + +[source,bash] +---- +OCIS_IMAGE_TAG=latest \ +make -C tests/acceptance/docker localApiTests-apiGraph-ocis +---- + +=== Test Log Output + +While a test is running or when it is finished, you can attach to the logs generated by the tests. + +[source,bash] +---- +make -C tests/acceptance/docker show-test-logs +---- + +[NOTE] +==== +The log output is opened in `less`. You can navigate up and down with your cursors. By pressing "F" you can follow the latest line of the output. +==== + + +=== Cleanup + +During testing we start a redis and oCIS docker container. These will not be stopped automatically. You can stop them with: + +[source,bash] +---- +make -C tests/acceptance/docker clean +---- + +== Running Test Suite in Local Environment + +=== Run oCIS + +Create an up-to-date oCIS binary by xref:build.adoc[building oCIS] + +To start oCIS: + +[source,bash] +---- +IDM_ADMIN_PASSWORD=admin \ +ocis/bin/ocis init --insecure true + +OCIS_INSECURE=true PROXY_ENABLE_BASIC_AUTH=true \ +ocis/bin/ocis server +---- + +`PROXY_ENABLE_BASIC_AUTH` will allow the acceptance tests to make requests against the provisioning api (and other endpoints) using basic auth. + +==== Run Local oCIS Tests (prefix `api`) and Tests Transferred From ownCloud Core (prefix `coreApi`) + +[source,bash] +---- +make test-acceptance-api \ +TEST_SERVER_URL=https://localhost:9200 \ +---- + +Useful environment variables: + +`TEST_SERVER_URL`: oCIS server url. Please, adjust the server url according to your setup. + +`BEHAT_FEATURE`: to run a single feature + +[NOTE] +==== +A specific scenario from a feature can be run by adding `:` at the end of the feature file path. For example, to run the scenario at line 26 of the feature file `apiGraphUser/createUser.feature`, simply add the line number like this: `apiGraphUser/createUser.feature:26`. Note that the line numbers mentioned in the examples might not always point to a scenario, so always check the line numbers before running the test. +==== + + +[quote] +____ +Example: +____ + +> +[quote] +____ +BEHAT_FEATURE=tests/acceptance/features/apiGraphUser/createUser.feature +____ + +> +[quote] +____ +Or +____ + +> +[quote] +____ +BEHAT_FEATURE=tests/acceptance/features/apiGraphUser/createUser.feature:13 +____ + + +`BEHAT_SUITE`: to run a single suite + +[quote] +____ +Example: +____ + +> +[quote] +____ +BEHAT_SUITE=apiGraph +____ + + +`STORAGE_DRIVER`: to run tests with a different user storage driver. Available options are `ocis` (default), `owncloudsql` and `s3ng` + +[quote] +____ +Example: +____ + +> +[quote] +____ +STORAGE_DRIVER=owncloudsql +____ + + +`STOP_ON_FAILURE`: to stop running tests after the first failure + +[quote] +____ +Example: +____ + +> +[quote] +____ +STOP_ON_FAILURE=true +____ + + +=== Use Existing Tests for BDD + +As a lot of scenarios are written for oC10, we can use those tests for Behaviour driven development in oCIS. +Every scenario that does not work in oCIS with "ocis" storage, is listed in `tests/acceptance/expected-failures-API-on-OCIS-storage.md` with a link to the related issue. + +Those scenarios are run in the ordinary acceptance test pipeline in CI. The scenarios that fail are checked against the +expected failures. If there are any differences then the CI pipeline fails. + +The tests are not currently run in CI with the OWNCLOUD or EOS storage drivers, so there are no expected-failures files for those. + +If you want to work on a specific issue + +1. locally run each of the tests marked with that issue in the expected failures file. + + E.g.: + + ```bash + make test-acceptance-api \ + TEST_SERVER_URL=https://localhost:9200 \ + STORAGE_DRIVER=OCIS \ + BEHAT_FEATURE='tests/acceptance/features/coreApiVersions/fileVersions.feature:141' + ``` + +2. the tests will fail, try to understand how and why they are failing +3. fix the code +4. go back to 1. and repeat till the tests are passing. +5. remove those tests from the expected failures file +6. make a PR that has the fixed code, and the relevant lines removed from the expected failures file. + +== Running Tests With And Without `remote.php` + +By default, the tests are run with `remote.php` enabled. If you want to run the tests without `remote.php`, you can disable it by setting the environment variable `WITH_REMOTE_PHP=false` while running the tests. + +[source,bash] +---- +WITH_REMOTE_PHP=false \ +TEST_SERVER_URL="https://localhost:9200" \ +make test-acceptance-api +---- + +== Running ENV Config Tests (@env-Config) + +Test suites tagged with `@env-config` are used to test the environment variables that are used to configure oCIS. These tests are special tests that require the oCIS server to be run using link:https://github.com/owncloud/ocis/blob/master/tests/ociswrapper/README.md[ociswrapper]. + +=== Run oCIS With ociswrapper + +[source,bash] +---- +# working dir: ocis repo root dir + +# init oCIS +IDM_ADMIN_PASSWORD=admin \ +ocis/bin/ocis init --insecure true + +# build the wrapper +cd tests/ociswrapper +make build + +# run oCIS +PROXY_ENABLE_BASIC_AUTH=true \ +./bin/ociswrapper serve --bin=../../ocis/bin/ocis +---- + +=== Run the Tests + +[source,bash] +---- +OCIS_WRAPPER_URL=http://localhost:5200 \ +TEST_SERVER_URL="https://localhost:9200" \ +BEHAT_FEATURE=tests/acceptance/features/apiAsyncUpload/delayPostprocessing.feature \ +make test-acceptance-api +---- + +=== Writing New ENV Config Tests + +While writing tests for a new oCIS ENV configuration, please make sure to follow these guidelines: + +1. Tag the test suite (or test scenarios) with `@env-config` +2. Use `OcisConfigHelper.php` for helper functions - provides functions to reconfigure the running oCIS instance. +3. Recommended: add the new step implementations in `OcisConfigContext.php` + +== Running Test Suite With Email Service (@email) + +Test suites that are tagged with `@email` require an email service. We use mailpit as the email service in our tests. + +=== Setup Mailpit + +Run the following command to setup mailpit + +[source,bash] +---- +docker run -d --restart unless-stopped --name=mailpit -p 8025:8025 -p 1025:1025 axllent/mailpit:v1.22.3 +---- + +=== Run oCIS + +Documentation for environment variables is available link:https://owncloud.dev/services/notifications/#environment-variables[here] + +[source,bash] +---- +# init oCIS +IDM_ADMIN_PASSWORD=admin \ +ocis/bin/ocis init --insecure true + +# run oCIS +PROXY_ENABLE_BASIC_AUTH=true \ +OCIS_ADD_RUN_SERVICES=notifications \ +NOTIFICATIONS_SMTP_HOST=localhost \ +NOTIFICATIONS_SMTP_PORT=1025 \ +NOTIFICATIONS_SMTP_INSECURE=true \ +NOTIFICATIONS_SMTP_SENDER="owncloud " \ +ocis/bin/ocis server +---- + +=== Run the Acceptance Test + +Run the acceptance test with the following command: + +[source,bash] +---- +TEST_SERVER_URL="https://localhost:9200" \ +EMAIL_HOST="localhost" \ +EMAIL_PORT=8025 \ +BEHAT_FEATURE="tests/acceptance/features/apiNotification/notification.feature" \ +make test-acceptance-api +---- + +== Running Test Suite With Tika Service (@tikaServiceNeeded) + +Test suites that are tagged with `@tikaServiceNeeded` require tika service. + +=== Setup Tika Service + +Run the following docker command to setup tika service + +[source,bash] +---- +docker run -d -p 127.0.0.1:9998:9998 apache/tika +---- + +=== Run oCIS + +Documentation related to the content based search and tika extractor can be found link:https://doc.owncloud.com/ocis/next/deployment/services/s-list/search.html#content-extraction[here] + +[source,bash] +---- +# init oCIS +IDM_ADMIN_PASSWORD=admin \ +ocis/bin/ocis init --insecure true + +# run oCIS +PROXY_ENABLE_BASIC_AUTH=true \ +OCIS_INSECURE=true \ +SEARCH_EXTRACTOR_TYPE=tika \ +SEARCH_EXTRACTOR_TIKA_TIKA_URL=http://localhost:9998 \ +SEARCH_EXTRACTOR_CS3SOURCE_INSECURE=true \ +ocis/bin/ocis server +---- + +=== Run the Acceptance Test + +Run the acceptance test with the following command: + +[source,bash] +---- +TEST_SERVER_URL="https://localhost:9200" \ +BEHAT_FEATURE="tests/acceptance/features/apiSearchContent/contentSearch.feature" \ +make test-acceptance-api +---- + +== Running Test Suite With Antivirus Service (@antivirus) + +Test suites that are tagged with `@antivirus` require antivirus service. The available antivirus and the configuration related to them can be found link:https://doc.owncloud.com/ocis/next/deployment/services/s-list/antivirus.html[here]. This documentation is only going to use `clamAv` as antivirus. + +=== Setup clamAV + +==== 1. Setup Locally + +===== Linux OS User + +Run the following command to set up calmAV and clamAV daemon + +[source,bash] +---- +sudo apt install clamav clamav-daemon -y +---- + +Make sure that the clamAV daemon is up and running + +[source,bash] +---- +sudo service clamav-daemon status +---- + +[NOTE] +==== +The commands are ubuntu specific and may differ according to your system. You can find information related to installation of clamAV in their official documentation link:https://docs.clamav.net/manual/Installing/Packages.html[here]. +==== + + +===== Mac OS User + +Install ClamAV using link:https://gist.github.com/mendozao/3ea393b91f23a813650baab9964425b9[here] +Start ClamAV daemon + +[source,bash] +---- +/your/location/to/brew/Cellar/clamav/1.1.0/sbin/clamd +---- + +==== 2. Setup clamAV With Docker + +===== Linux OS User + +Run `clamAV` through docker + +[source,bash] +---- +docker run -d -p 3310:3310 owncloudci/clamavd +---- + +===== Mac OS User + +[source,bash] +---- +docker run -d -p 3310:3310 -v /your/local/filesystem/path/to/clamav/:/var/lib/clamav mkodockx/docker-clamav:alpine +---- + +=== Run oCIS + +As `antivirus` service is not enabled by default we need to enable the service while running oCIS server. We also need to enable `async upload` and as virus scan is performed in post-processing step, we need to set it as well. Documentation for environment variables related to antivirus is available link:https://owncloud.dev/services/antivirus/#environment-variables[here] + +[source,bash] +---- +# init oCIS +IDM_ADMIN_PASSWORD=admin \ +ocis/bin/ocis init --insecure true + +# run oCIS +PROXY_ENABLE_BASIC_AUTH=true \ +ANTIVIRUS_SCANNER_TYPE="clamav" \ +ANTIVIRUS_CLAMAV_SOCKET="tcp://host.docker.internal:3310" \ +POSTPROCESSING_STEPS="virusscan" \ +OCIS_ASYNC_UPLOADS=true \ +OCIS_ADD_RUN_SERVICES="antivirus" \ +ocis/bin/ocis server +---- + +[NOTE] +==== +The value for `ANTIVIRUS_CLAMAV_SOCKET` is an example which needs adaption according your OS. + +For antivirus running localy on Linux OS, use `ANTIVIRUS_CLAMAV_SOCKET= "/var/run/clamav/clamd.ctl"`. +For antivirus running localy on Mac OS, use `ANTIVIRUS_CLAMAV_SOCKET= "/tmp/clamd.socket"`. +For antivirus running with docker, use `ANTIVIRUS_CLAMAV_SOCKET= "tcp://host.docker.internal:3310"` +==== + + +==== Run the Acceptance Test + +Run the acceptance test with the following command: + +[source,bash] +---- +TEST_SERVER_URL="https://localhost:9200" \ +BEHAT_FEATURE="tests/acceptance/features/apiAntivirus/antivirus.feature" \ +make test-acceptance-api +---- + +== Running Test Suite With Federated Sharing (@ocm) + +Test suites that are tagged with `@ocm` require running two different ocis instances. More detailed information and configuration related to it can be found link:https://doc.owncloud.com/ocis/5.0/deployment/services/s-list/ocm.html[here]. + +=== Setup First oCIS Instance + +[source,bash] +---- +# init oCIS +IDM_ADMIN_PASSWORD=admin \ +ocis/bin/ocis init --insecure true + +# run oCIS +OCIS_URL="https://localhost:9200" \ +PROXY_ENABLE_BASIC_AUTH=true \ +OCIS_ENABLE_OCM=true \ +OCM_OCM_PROVIDER_AUTHORIZER_PROVIDERS_FILE="tests/config/local/providers.json" \ +OCIS_ADD_RUN_SERVICES="ocm" \ +OCM_OCM_INVITE_MANAGER_INSECURE=true \ +OCM_OCM_SHARE_PROVIDER_INSECURE=true \ +OCM_OCM_STORAGE_PROVIDER_INSECURE=true \ +WEB_UI_CONFIG_FILE="tests/config/local/ocis-web.json" \ +ocis/bin/ocis server +---- + +The first oCIS instance should be available at: + +=== Setup Second oCIS Instance + +You can run the second oCIS instance in two ways: + +==== Using `.vscode/launch.json` + +From the `Run and Debug` panel of VSCode, select `Fed oCIS Server` and start the debugger. + +==== Using env File + +[source,bash] +---- +# init oCIS +source tests/config/local/.env-federation && ocis/bin/ocis init + +# run oCIS +source tests/config/local/.env-federation && ocis/bin/ocis server +---- + +The second oCIS instance should be available at: + +[NOTE] +==== +To enable ocm in the web interface, you need to set the following envs: +`OCIS_ENABLE_OCM="true"` +`OCIS_ADD_RUN_SERVICES="ocm"` +==== + + +==== Run the Acceptance Test + +Run the acceptance test with the following command: + +[source,bash] +---- +TEST_SERVER_URL="https://localhost:9200" \ +TEST_SERVER_FED_URL="https://localhost:10200" \ +BEHAT_SUITE=apiOcm \ +make test-acceptance-api +---- + +== Running Text Preview Tests Containing Unicode Characters + +There are some tests that check the text preview of files containing Unicode characters. The oCIS server by default cannot generate the thumbnail of such files correctly but it provides an environment variable to allow the use of custom fonts that support Unicode characters. So to run such tests successfully, we have to run the oCIS server with this environment variable. + +[source,bash] +---- +... +THUMBNAILS_TXT_FONTMAP_FILE="/path/to/fontsMap.json" +ocis/bin/ocis server +---- + +The sample `fontsMap.json` file is located in `tests/config/drone/fontsMap.json`. + +[source,json] +---- +{ + "defaultFont": "/path/to/ocis/tests/config/drone/NotoSans.ttf" +} +---- + +== Running Test on Helm Setup + +Refer to the xref:../../../tests/config/k8s/README.adoc[Running API Tests in Kubernetes Cluster] documentation for running tests against a K8s setup. + +== Generating Code Coverage Report by Running Acceptance Tests + +To find out what oCIS code is covered by the API tests, first create a debug build of oCIS. + +[source,shell] +---- +make -c ocis build-debug +---- + +oCIS should be served by the debug binary. + +[source,shell] +---- +ocis/bin/ocis-debug server +---- + +Then define a folder to store the coverage report. + +[source,shell] +---- +export GOCOVERDIR=coveragedatafiles +---- + +Running the tests will generate the coverage report inside the `coveragedatafiles` directory. + +To view the report in human readable form, enter the following command or refer to the official link:https://go.dev/doc/build-cover#working[documentation] for more formats. + +---- +go tool covdata textfmt -i=coveragedatafiles -o=cov.txt +---- + +You can also view the report in a web UI using the following command. + +---- +go tool cover -html=cov.txt +---- + +This command should open a browser with the code report. diff --git a/modules/developer/pages/ocis/development/tracing.adoc b/modules/developer/pages/ocis/development/tracing.adoc new file mode 100644 index 00000000..ab0465ca --- /dev/null +++ b/modules/developer/pages/ocis/development/tracing.adoc @@ -0,0 +1,44 @@ += Tracing +:toc: right +:toclevels: 3 + + +By default, we use link:https://www.jaegertracing.io[Jaeger] for request tracing within oCIS. You can follow these steps +to get started: + +1. Start Jaeger by using the all-in-one docker image: + ```console + docker run -d --name jaeger \ + -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \ + -p 5775:5775/udp \ + -p 6831:6831/udp \ + -p 6832:6832/udp \ + -p 5778:5778 \ + -p 16686:16686 \ + -p 14268:14268 \ + -p 14250:14250 \ + -p 9411:9411 \ + jaegertracing/all-in-one:1.17 + ``` +2. Every single oCIS service has its own environment variables for enabling and configuring tracing. + 1. You can enable and configure tracing on each service individually. For example, enable tracing + in Reva when starting the oCIS single binary like this: + ```console + REVA_TRACING_ENABLED=true \ + REVA_TRACING_ENDPOINT=localhost:6831 \ + REVA_TRACING_COLLECTOR=http://localhost:14268/api/traces \ + ./bin/ocis server + ``` + 2. Enabling and configuring tracing on oCIS itself will forward the configuration to all services: + ```console + OCIS_TRACING_ENABLED=true \ + OCIS_TRACING_ENDPOINT=localhost:6831 \ + OCIS_TRACING_COLLECTOR=http://localhost:14268/api/traces \ + ./bin/ocis server + ``` + If you want to set individual tracing configuration for each service, make sure to set + `OCIS_TRACING_ENABLED=false`. +3. Make the actual request that you want to trace. +4. Open up the link:http://localhost:16686[Jaeger UI] to analyze request traces. + +For more information on Jaeger, please refer to their link:https://www.jaegertracing.io/docs/1.17/[Documentation]. diff --git a/modules/developer/pages/ocis/development/unit-testing/index.adoc b/modules/developer/pages/ocis/development/unit-testing/index.adoc new file mode 100644 index 00000000..3c42a364 --- /dev/null +++ b/modules/developer/pages/ocis/development/unit-testing/index.adoc @@ -0,0 +1,82 @@ += Unit Testing +:toc: right +:toclevels: 3 + + +Go is a statically typed language, which makes it easy to write unit tests. The Go standard library provides a `testing` package that allows you to write tests for your code. The testing package provides a framework for writing tests, and the `go test` command runs the tests. Other than that there are a lot of libraries and tools available to make testing easier. + +- link:https://github.com/stretchr/testify[Testify] - A toolkit with common assertions and mocks that plays nicely with the standard library. +- link:https://onsi.github.io/ginkgo/[Ginkgo] - A BDD-style testing framework for Go. +- link:https://onsi.github.io/gomega/[Gomega] - A matcher/assertion library for Ginkgo. +- link:https://github.com/cucumber/godog[GoDog] - A Behavior-Driven Development framework for Go which uses Gherkin. + +In oCIS we generally use link:https://onsi.github.io/ginkgo/[Ginkgo] framework for testing. To keep things consistent, we would encourage you to use the same. In some cases, where you feel the need for a more verbose or more "code oriented" approach, you can also use the testing package from the standard library without ginkgo. + +== 1 Ginkgo + +Using a framework like link:https://onsi.github.io/ginkgo/[Ginkgo] brings many advantages. + +=== Pros + +- Provides a BDD-style syntax which makes it easier to write reusable and understandable tests +- Together with link:https://onsi.github.io/gomega/[Gomega] it provides a powerful and expressive framework with assertions in a natural language +- Natural Language Format empowers testing in a way that resembles user interactions with the system +- In the context of microservices it is particularly well suited to test individual services and the interactions between them +- Offers support for asynchronous testing which makes it easier to test code that involves concurrency +- Nested and structured containers and setup capabilities make it easy to organize tests and adhere to the DRY principle +- Provides helpful error messages to identify and fix issues +- Very usable for Test Driven Development following the link:https://en.wikipedia.org/wiki/Test-driven_development["Red, Green, Cleanup, Repeat"] workflow. + +=== Cons + +- Sometimes it can be difficult to get started with +- Asynchronous behaviour brings more complexity to tests. +- Not compatible with broadly known `testify` package + +=== Example + +As you can see, _Ginkgo_ and _Gomega_ together provide the foundation to write understandable and maintainable tests which can mimic user interaction and the interactions between microservices. + +[source,go] +---- +Describe("Public Share Provider", func() { + Context("When the user has no share permission", func() { + BeforeEach(func() { + // downgrade user permissions to have no share permission + resourcePermissions.AddGrant = false + }) + It("should return grpc invalid argument", func() { + req := &link.CreatePublicShareRequest{} + + res, err := provider.CreatePublicShare(ctx, req) + Expect(err).ToNot(HaveOccurred()) + Expect(res.GetStatus().GetCode()).To(Equal(rpc.Code_CODE_INVALID_ARGUMENT)) + Expect(res.GetStatus().GetMessage()).To(Equal("no share permission")) + }) +}) +---- + +=== How to use it in oCIS + +// SHORTCODE: {{< button relref="testing-ginkgo" size="large" >}}{{< icon "gdoc_arrow_right_alt" >}} Read more{{< /button >}} + +== 2 Testing Package + +For smaller straight-forward tests of some packages it might feel more natural to use the testing package that comes with the go standard library. + +=== Pros + +- Straightforward approach +- Naming conventions +- Built-in tooling + +=== Cons + +- Difficult to reuse code in larger and more complex packages +- Difficult to create clean and isolated setups for the test steps +- No natural language resemblance + + +=== How to use it in ocis + +// SHORTCODE: {{< button relref="testing-pkg" size="large" >}}{{< icon "gdoc_arrow_right_alt" >}} Read more{{< /button >}} diff --git a/modules/developer/pages/ocis/development/unit-testing/testing-ginkgo.adoc b/modules/developer/pages/ocis/development/unit-testing/testing-ginkgo.adoc new file mode 100644 index 00000000..f6c86f91 --- /dev/null +++ b/modules/developer/pages/ocis/development/unit-testing/testing-ginkgo.adoc @@ -0,0 +1,405 @@ += Testing with Ginkgo +:toc: right +:toclevels: 3 + + +In this section we try to enable developers to write tests in oCIS using Ginkgo and Gomega and explain how to mock other microservices to also cover some integration tests. The full documentation of the tools can be found on the link:https://onsi.github.io/ginkgo/[Ginkgo] and link:https://onsi.github.io/gomega/[Gomega] websites. + +// SHORTCODE: {{% hint type=tip icon=gdoc_link title="Reading the documentation" %}} +This page provides only a basic introduction to get started with Ginkgo and Gomega. For more detailed information, please refer to the official documentation. + +_Useful Links:_ + +- link:https://onsi.github.io/ginkgo/[Ginkgo] +- link:https://onsi.github.io/gomega/[Gomega] +- link:https://vektra.github.io/mockery/latest/[Mockery] + +// SHORTCODE: {{% /hint %}} + +== Prerequisites + +To use Ginkgo, you need to install the Ginkgo CLI. You can install it using the following command: + +[source,bash] +---- +go install github.com/onsi/ginkgo/v2/ginkgo +go get github.com/onsi/gomega/... +---- + +== Getting Started + +Navigate to the directory where you want to write your tests and run the following command: + +=== Bootstrap + +[source,bash] +---- +cd ocis/ocis-pkg/config/parser +ginkgo bootstrap +Generating ginkgo test suite bootstrap for parser in: + parser_suite_test.go + +---- + +This command creates a `parser_suite_test.go` file in the parser directory. This file contains the test suite for the parser package. + +[source,go] +---- +package parser_test + +import ( + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestParser(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "Parser Suite") +} +---- + +Ginkgo defaults to setting up the suite as a `*_test` package to encourage you to only test the external behavior of your package, not its internal implementation details. + +After the package `parser_test` declaration we import the ginkgo and gomega packages into the test's top-level namespace by performing a `.` dot-import. Since Ginkgo and Gomega are DSLs (Domain-specific Languages) this makes the tests more natural to read. If you prefer, you can avoid the dot-import via `ginkgo bootstrap --nodot`. Throughout this documentation we'll assume dot-imports. + +With the bootstrap complete, you can now run your tests using the `ginkgo` command: + +[source,bash] +---- +ginkgo + +Running Suite: Parser Suite - /ocis/ocis-pkg/config/parser +=============================================================================================== +Random Seed: 1714076559 + +Will run 0 of 0 specs + +Ran 0 of 0 Specs in 0.000 seconds +SUCCESS! -- 0 Passed | 0 Failed | 0 Pending | 0 Skipped +PASS + +Ginkgo ran 1 suite in 7.0058606s +Test Suite Passed +---- + +Under the hood, ginkgo is simply calling `go test`. While you can run `go test` instead of the ginkgo CLI, Ginkgo has several capabilities that can only be accessed via `ginkgo`. We generally recommend users embrace the ginkgo CLI and treat it as a first-class member of their testing toolchain. + +=== Adding Specs to the Suite + +[source,bash] +---- +ginkgo generate parser +Generating ginkgo test for Parser in:  ✔  7s  22:22:46  + parser_test.go +---- + +This will generate a `parser_test.go` file in the parser directory. This file contains the test suite for the parser package. + +[source,go] +---- +package parser_test + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/owncloud/ocis/v2/ocis-pkg/config/parser" +) + +var _ = Describe("Parser", func() { + +}) +---- + +== Writing Specs + +=== Describe + +The `Describe` block is used to describe the behavior of a particular component of your code. It is a way to group together related specs. The `Describe` block takes a string and a function. The string is a description of the component you are describing, and the function contains the specs that describe the behavior of that component. + +[source,go] +---- +var _ = Describe("Parser", func() { + // Specs go here +}) +---- + +=== Context + +The `Context` block is used to further describe the behavior of a component. It is a way to group together related specs within a `Describe` block. The `Context` block takes a string and a function. The string is a description of the context you are describing, and the function contains the specs that describe the behavior of that context. + +[source,go] +---- +var _ = Describe("Parser", func() { + Context("when the input is valid", func() { + // Specs go here + }) +}) +---- + +=== It + +The `It` block is used to describe a single spec. It takes a string and a function. The string is a description of the behavior you are specifying, and the function contains the code that exercises that behavior. + +[source,go] +---- +var _ = Describe("Parser", func() { + Context("when the input is valid", func() { + It("parses the input", func() { + // Spec code goes here + }) + }) +}) +---- + +=== Expect + +The `Expect` function is used to make assertions in your specs. It takes a value and returns an `_Expectation`. You can then chain methods on the `_Expectation` to make assertions about the value. + +[source,go] +---- +var _ = Describe("Parser", func() { + Context("when the input is valid", func() { + It("parses the input", func() { + result := parser.Parse("valid input") + Expect(result).To(Equal("expected output")) + }) + }) +}) +---- + +=== BeforeEach + +The `BeforeEach` block is used to run a setup function before each spec in a `Describe` or `Context` block. It takes a function that contains the setup code. + +[source,go] +---- +package parser_test + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "github.com/owncloud/ocis/v2/ocis-pkg/config" + + p "github.com/owncloud/ocis/v2/ocis-pkg/config/parser" +) + +var _ = Describe("Parser", func() { + var c *config.Config + + BeforeEach(func() { + c = config.DefaultConfig() + }) + + Context("when the input is valid", func() { + It("parses the input", func() { + err := p.ParseConfig(c, false) + Expect(err).ToNot(HaveOccurred()) + Expect(c.Commons.OcisURL).To(Equal("https://localhost:9200")) + }) + }) +}) +---- + +Let us take a closer look at the code above: + +We are following the recommended practise on variables to _"declare in container nodes"_ and _"initialize in setup nodes"_. This is why we are declaring the `c` variable at the top of the `Describe` block and initializing it in the `BeforeEach` block. This is important to get isolated test steps which can be run in any order and even in parallel. + +Let us take a look at a bad example where we are polluting the spec by not following this recommended practise: + +[source,go] +---- +package parser_test + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "github.com/owncloud/ocis/v2/ocis-pkg/config" + + p "github.com/owncloud/ocis/v2/ocis-pkg/config/parser" +) + + +var _ = Describe("Parser", func() { + c := config.DefaultConfig() + + Context("when the defaults are applied", func() { + It("fails to parse the input", func() { + c.TokenManager.JWTSecret = "" // bam! we have changed the closure variable and it will never be reset + err := p.ParseConfig(c, false) + Expect(err).To(HaveOccurred()) + }) + It("parses the input", func() { + err := p.ParseConfig(c, false) + Expect(err).ToNot(HaveOccurred()) + Expect(c.Commons.OcisURL).To(Equal("https://localhost:9200")) + }) + }) +}) +---- + +// SHORTCODE: {{% hint type="warning" title="Specs MUST be clean and independent"%}} +Always _declare variables in the container node_(which are basically `Describe()` and `Context()`) + +and _initialize your variables in the setup nodes._ (which are basically `BeforeEach()` and `JustBeforeEach()`). + +This will ensure that your specs are clean and independent of each other. +// SHORTCODE: {{% /hint %}} + +=== Focused Specs + +You can focus on a single spec by adding an `F` in front of the `It` block. This will run only the focused spec. + +[source,go] +---- +var _ = Describe("Parser", func() { + Context("when the input is valid", func() { + FIt("parses the input", func() { + result := parser.Parse("valid input") + Expect(result).To(Equal("expected output")) + }) + }) +}) +---- + +=== Pending Specs + +You can mark a spec as pending by adding a `P` in front of the `It` block. This will skip the spec. + +[source,go] +---- +var _ = Describe("Parser", func() { + Context("when the input is valid", func() { + PIt("parses the input", func() { + result := parser.Parse("valid input") + Expect(result).To(Equal("expected output")) + }) + }) +}) +---- + +=== Test Driven Development + +You can run the tests in watch mode to follow a test-driven development approach. This will run the tests every time you save a file. + +[source,bash] +---- +ginkgo watch +---- + +== Mocking + +In oCIS, we use the `mockery` tool to generate mocks for interfaces. link:https://vektra.github.io/mockery/latest/[Mockery] is a simple tool that generates mock implementations of Go interfaces. It is useful for writing tests against interfaces instead of concrete types. We can use it to mock requests to other microservices to cover some integration tests. We should already have a number of mocks in the project. The mocks are configured on the packages level in the `.mockery.yaml` files. + +_Example file:_ + +[source,yaml] +---- +with-expecter: true +filename: "{{.InterfaceName | snakecase }}.go" +dir: "{{.PackageName}}/mocks" +mockname: "{{.InterfaceName}}" +outpkg: "mocks" +packages: + github.com/owncloud/ocis/v2/ocis-pkg/oidc: + interfaces: + OIDCClient: +---- + +We should add missing mocks to this file and define the interfaces we want to mock. After that, we can generate the mocks by running `mockery` in the repo, it will find all the `.mockery.yaml` files and generate the mocks for the interfaces defined in them. + +Our mocks are generated with the setting `with-expecter: true`. This allows us to use type-safe methods to generate the call expectations by simply calling `EXPECT()` on the mock object. + +// SHORTCODE: {{% hint type="tip" title="Type safe mock identifiers" %}} +By using `EXPECT()` on the mock object, we can work with type-safe methods to generate the call expectations. +// SHORTCODE: {{% /hint %}} + +_Example of a mocked gateway client_ + +In our oCIS services we need to use a gateway pool selector to get the gateway client. + +We should always use the constructor on a new mock like `gatewayClient = cs3mocks.NewGatewayAPIClient(GinkgoT())`. This brings us two advantages: + +- The `AssertExpectations` method is registered to be called at the end of the tests via `t.Cleanup()` method. +- The `testing.TB` interface is registered on the `mock.Mock` so that tests don't panic when a call on the mock is unexpected. + +[source,go] +---- +package publicshareprovider_test + +import ( + "context" + "time" + + + "github.com/owncloud/reva/v2/pkg/rgrpc/todo/pool" + cs3mocks "github.com/owncloud/reva/v2/tests/cs3mocks/mocks" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "google.golang.org/grpc" +) + +var _ = Describe("PublicShareProvider", func() { + // declare in container nodes + var ( + gatewayClient *cs3mocks.GatewayAPIClient + gatewaySelector pool.Selector + ) + + BeforeEach(func() { + // initialize in setup nodes + pool.RemoveSelector("GatewaySelector" + "any") + // create a new mock client + gatewayClient = cs3mocks.NewGatewayAPIClient(GinkgoT()) + gatewaySelector = pool.GetSelector[gateway.GatewayAPIClient]( + "GatewaySelector", + "any", + func(cc *grpc.ClientConn) gateway.GatewayAPIClient { + return gatewayClient + }, + ) + }) + Context("The user has the permission to create public shares", func() { + BeforeEeach(func() { + // set up the mock + // this is implicitly creating the expectation that it will be called Once() + // this will throw an error if the method is not called + gatewayClient. + EXPECT(). + CheckPermission( + mock.Anything, + mock.Anything, + ). + Return(checkPermissionResponse, nil) + }) + It("should return a public share", func() { + // call the method + req := &link.CreatePublicShareRequest{ + ResourceInfo: &providerpb.ResourceInfo{ + Owner: &userpb.UserId{ + OpaqueId: "alice", + }, + Path: "./NewFolder/file.txt", + }, + Grant: &link.Grant{ + Permissions: &link.PublicSharePermissions{ + Permissions: linkPermissions, + }, + Password: "SecretPassw0rd!", + }, + Description: "test", + } + res, err := provider.CreatePublicShare(ctx, req) + Expect(err).ToNot(HaveOccurred()) + Expect(res.GetStatus().GetCode()).To(Equal(rpc.Code_CODE_OK)) + Expect(res.GetShare()).To(Equal(createdLink)) + }) + }) +}) +---- + +// SHORTCODE: {{% hint type="tip" title="Mocking in oCIS" %}} +Use the constructor on new mocks to register the `AssertExpectations` method to be called at the end of the tests via the `t.Cleanup()` method. +// SHORTCODE: {{% /hint %}} diff --git a/modules/developer/pages/ocis/development/unit-testing/testing-pkg.adoc b/modules/developer/pages/ocis/development/unit-testing/testing-pkg.adoc new file mode 100644 index 00000000..725ecb65 --- /dev/null +++ b/modules/developer/pages/ocis/development/unit-testing/testing-pkg.adoc @@ -0,0 +1,111 @@ += Standard Library Testing +:toc: right +:toclevels: 3 + +== Using the standard library + +To write a unit test for your package, create a file with the `_test.go` suffix. For example, if you have a package `foo` with a file `foo.go`, you can create a file `foo_test.go` in the same directory. The test file should have the same package name as the package being tested. By doing this, you can access all exported and unexported identifiers of the package. It is a good practice to keep the test file in the same package as the code being tested. + +=== Simple Example + +We are using an oversimplified example from link:https://codingdojo.org/kata/FooBarQix/[FooBarQuix] to demonstrate how to use the `testing` package. + +[source,go] +---- +package divide + +import "strconv" + +// If the number is divisible by 3, write "Yes" otherwise, the number +func IsDivisible(input int) string { + if (input % 3) == 0 { + return "Yes" + } + return strconv.Itoa(input) +} +---- + +To test the `IsDivisible` function, create a file `divide_test.go` in the same directory as `divide.go`. The test file should have the same package name as the package being tested. + +A test function in Go starts with `Test` and takes `*testing.T` as the only parameter. In most cases, you will name the unit test `Test[NameOfFunction]`. The testing package provides tools to interact with the test workflow, such as `t.Errorf`, which indicates that the test failed by displaying an error message on the console. + +The test function for the `IsDivisible` function could look like this + +[source,go] +---- +package divide + +import "testing" + +func TestDivide3(t *testing.T) { + result := IsDivisible(3) + if result != "Yes" { + t.Errorf("Result was incorrect, got: %s, want: %s.", result, "Yes") + } +} +---- + +To run the test, use the `go test` command in the directory where the test file is located. + +=== Use a helper package for assertions + +You could make the test more readable by using testify. The `assert` package provides a lot of helper functions to make the test more readable. + +[source,go] +---- +package divide + +import ( + "testing" + "github.com/stretchr/testify/assert" +) + +func TestDivide3(t *testing.T) { + result := IsDivisible(3) + assert.Equal(t, "Yes", result) +} +---- + +=== Table Driven Example + +Write Table Driven Tests to test multiple inputs. + +[source,go] +---- +package divide + +import ( + "testing" + "github.com/stretchr/testify/assert" +) + + +func TestIsDivisibleTableDriven(t *testing.T) { + // Defining the columns of the table + var tests = []struct { + name string + input int + want string + }{ + // the table itself + {"9 should be Yes", 9, "Yes"}, + {"3 should be Yes", 3, "Yes"}, + {"1 is not Yes", 1, "1"}, + {"0 should be Yes", 0, "Yes"}, + } + + // The execution loop + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + answer := IsDivisible(tt.input) + assert.Equal(t, tt.want, answer) + }) + } +} +---- + +A table-driven test starts by defining the input structure. This can be seen like defining the columns of the table. Each row of the table lists a test case to execute. Once the table is defined, the execution loop can be created. + +The execution loop calls `t.Run()`, which defines a subtest. In our example each row of the table defines a subtest named `[NameOfTheFuction]/[NameOfTheSubTest]`. + +This way of writing tests is very popular, and considered the canonical way to write unit tests in Go. diff --git a/modules/developer/pages/ocis/flow-docs/index.adoc b/modules/developer/pages/ocis/flow-docs/index.adoc new file mode 100644 index 00000000..fe99918d --- /dev/null +++ b/modules/developer/pages/ocis/flow-docs/index.adoc @@ -0,0 +1,4 @@ += Flow documentation +:toc: right +:toclevels: 3 + diff --git a/modules/developer/pages/ocis/flow-docs/login-flow.adoc b/modules/developer/pages/ocis/flow-docs/login-flow.adoc new file mode 100644 index 00000000..b236c68d --- /dev/null +++ b/modules/developer/pages/ocis/flow-docs/login-flow.adoc @@ -0,0 +1,54 @@ += Login Flow +:toc: right +:toclevels: 3 + +== Login Flow + +The following sequence diagram describes the link:https://openid.net/specs/openid-connect-core-1_0.html#CodeFlowAuth[openid connect auth code flow]. The eight numbered steps and notes correspond to the link:https://openid.net/specs/openid-connect-core-1_0.html#CodeFlowSteps[openid connect auth code flow steps]. Example requests are based on the spec as well: + +// SHORTCODE: {{< mermaid class="text-center">}} +sequenceDiagram + %% we have comments!! \o/ + %% this documents the login workflow + %% examples taken from the oidc spec https://openid.net/specs/openid-connect-core-1_0.html#CodeFlowAuth + %% TODO add PKCE, see https://developer.okta.com/blog/2019/08/22/okta-authjs-pkce#use-pkce-to-make-your-apps-more-secure + participant user as User + participant client as Client + participant proxy as ocis-proxy + participant idp as IdP + participant idm as LibreIDM + participant ldap as External User Directory + + user->>+client: What is the content of my home? + client->>+proxy: PROPFIND no (or expired) auth + Note over client,proxy: ocis needs to know the IdP that is used to authenticate users. The proxy will redirect unauthenticated requests to that IdP. + proxy-->>-client: 401 Unauthorized + client->>+proxy: 1. The client starts a new openIDConnect Flow + Note over client, proxy: GET /.well-known/openid-configuration + proxy-->>-client: Return openidConnect configuration for the IdP + client-->>client: 2. Client prepares an Authentication Request containing the desired request parameters and generates the code challenge (PKCE). + client->>+idp: 3. Client sends the request and the code challenge to the Authorization Server. + Note over client, idp: GET /authorize? flow=oidc&response_type=code &scope=openid%20profile%20email &code_challenge=Y2SGoq9vtAp7YAavTaO0B550H_Rsj9DypiL7xZuFjOE &code_challenge_method=S25&client_id=s6BhdRkqt3 &state=af0ifjsldkj &redirect_uri=https%3A%2F%2Fclient.example.org%2Fcb HTTP/1.1 Host: server.example.com + Note over user, idp: 3. Authorization Server Authenticates the End-User. + alt all users managed by idp/ocis idm + idp->>+idm: LDAP query/bind + idm-->>-idp: LDAP result + Note over idp,ldap: In case users are managed in an external ldap they have to be autoprovisioned in the ocis IdM when they are loggin in. + else all users authenticated by an external idp + idp->>+ldap: Lookup of the user in the directory + ldap-->>-idp: Lookup result + end + idp-->>-user: Idp presents the user an authentication prompt. + user->>+idp: 5. User authenticates and gives consent. + idp-->>-client: 6. Authorization Server sends the End-User back to the Client with an Authorization Code. + Note over client, idp: HTTP/1.1 302 Found Location: https://client.example.org/cb? code=SplxlOBeZQQYbYS6WxSbIA&state=af0ifjsldkj + client->>+idp: 7. Client requests a response using the Authorization Code and the code verifier at the Token Endpoint. + Note over client, idp: POST /token HTTP/1.1 Host: server.example.com Content-Type: application/x-www-form-urlencoded grant_type=authorization_code&code=SplxlOBeZQQYbYS6WxSbIA &redirect_uri=https%3A%2F%2Fclient.example.org &code_verifier=a98ccbe253754259963e6e2b67b5a044929446d7a15046cc8e3194022ad061d9d667dce91876418d9e6fe9f54819332e + idp->>+idp: 8. IdP checks the code verifier (PKCE) + idp-->>-client: 9. Client receives a response that contains an ID Token and Access Token in the response body. If offline access is requested, the client also receives a refresh token. + Note over client, idp: HTTP/1.1 200 OK Content-Type: application/json Cache-Control: no-store Pragma: no-cache { "access_token": "SlAV32hkKG", "token_type": "Bearer", "refresh_token": "8xLOxBtZp8", "expires_in": 3600, "id_token": "a ... b.c ... d.e ... f" // must be a JWT } + client-->>client: 10. Client validates the ID token and retrieves the End-User's Subject Identifier. + client->>+proxy: PROPFIND With access token + proxy-->>-client: 207 Multi-Status + client-->>-user: List of Files X, Y, Z ... +// SHORTCODE: {{< /mermaid >}} diff --git a/modules/developer/pages/ocis/flow-docs/public-upload-flow.adoc b/modules/developer/pages/ocis/flow-docs/public-upload-flow.adoc new file mode 100644 index 00000000..15daf40c --- /dev/null +++ b/modules/developer/pages/ocis/flow-docs/public-upload-flow.adoc @@ -0,0 +1,9 @@ += Public upload Flow +:toc: right +:toclevels: 3 + +== Public Upload flow + +The following diagram describes the flow of requests: + +image::ocis/tus-public-upload.svg[] diff --git a/modules/developer/pages/ocis/flow-docs/request-flow.adoc b/modules/developer/pages/ocis/flow-docs/request-flow.adoc new file mode 100644 index 00000000..5775e646 --- /dev/null +++ b/modules/developer/pages/ocis/flow-docs/request-flow.adoc @@ -0,0 +1,90 @@ += Request Flow +:toc: right +:toclevels: 3 + +== Request Flow + +The following sequence diagram describes the general request flow. It shows where account provisioning and token minting are happening: + +// SHORTCODE: {{< mermaid class="text-center">}} +sequenceDiagram + %% we have comments!! \o/ + participant user as User + participant client as Client + participant proxy as ocis-proxy + participant idp as IdP + participant accounts as ocis-accounts + participant ldap as corporate LDAP server + + user->>+client: What is the content of my home? + + client->>+proxy: PROPFIND Bearer auth using oidc auth token + Note over client,proxy: What is in a bearer token? The spec recommends opaque tokens. Treat it as random byte noise. + Note over client,proxy: the proxy MUST authenticate users using ocis-accounts because it needs to decide where to send the request + %% Mention introspection endpoint for opaque tokens + %% idp uses jwt, so we can save a request + %% either way the token can be used to look up the sub and iss of the user + + %% or is token check enough? + proxy->>+idp: GET /userinfo + alt userinfo succeeds + + idp-->>proxy: 200 OK + Note over proxy,accounts: Content-Type: application/json{"sub": "248289761001","name": "Jane Doe","given_name": "Jane","family_name": "Doe","preferred_username": "j.doe","email": "janedoe@example.com","picture": "http://example.com/janedoe/me.jpg"} + %% see: https://openid.net/specs/openid-connect-core-1_0.html#UserInfoResponse + + else userinfo fails + + idp-->>-proxy: 401 Unauthorized + Note over proxy,accounts: WWW-Authenticate: error="invalid_token",error_description="The Access Token expired" + + proxy-->>client: 401 Unauthorized or 302 Found with redirect to idp + Note over client: start at login flow or refresh the token + + end + + proxy->>+accounts: TODO API call to exchange sub@iss with account UUID + Note over proxy,accounts: does not autoprovision users. They are explicitly provisioned later. + + alt account exists or has been migrated + + accounts-->>proxy: existing account UUID + else account does not exist + + opt oc10 endpoint is configured + Note over proxy,oc10: Check if user exists in oc10 + proxy->>+oc10: GET /apps/graphapi/v1.0/users/<uuid> + opt user exists in oc10 + oc10-->>-proxy: 200 + %% TODO auth using internal token + proxy->>+oc10: PROPFIND + Note over proxy,oc10: forward existing bearer auth + oc10-->>-proxy: Multistatus response + proxy-->>client: Multistatus response + client-->>user: List of Files X, Y, Z ... + end + end + + Note over proxy,accounts: provision a new account including displayname, email and sub@iss TODO only if the user is allowed to login, based on group membership in the ldap server + proxy->>proxy: generate new uuid + proxy->>+accounts: TODO create account with new generated uuid + accounts-->>-proxy: OK / error + + else account has been disabled + + accounts-->>-proxy: account is disabled + proxy-->>client: 401 Unauthorized or 302 Found with redirect to idp + Note over client: start at login flow or refresh the token + + end + proxy->>proxy: store uuid in context + + %% what if oc10 does not support a certain request / API + + proxy->>proxy: mint an internal jwt that includes the UUID and username using revas `x-access-token` header + proxy->>+reva: PROPFIND Token auth using internal JWT + reva-->>-proxy: Multistatus response + proxy-->>-client: Multistatus response + + client-->>-user: List of Files X, Y, Z ... +// SHORTCODE: {{< /mermaid >}} diff --git a/modules/developer/pages/ocis/getting-started/demo-users.adoc b/modules/developer/pages/ocis/getting-started/demo-users.adoc new file mode 100644 index 00000000..c8b046aa --- /dev/null +++ b/modules/developer/pages/ocis/getting-started/demo-users.adoc @@ -0,0 +1,25 @@ += Demo Users +:toc: right +:toclevels: 3 + +oCIS has the option to create demo users during the first startup. These enable you to do quick testing and developing. + +[NOTE] +==== +To create the demo users, run the initial setup step with an additional environment variable. +`IDM_CREATE_DEMO_USERS=true ./bin/ocis server` will generate the demo users listed in the table below. By default, it only generates the admin and one user for IDP and Reva respectively. +==== + + +Following users are available in the demo set: + +| username | password | email | role | groups | +| --------- | ----------------------------------------- | --------------------- | ----------- | ----------------------------------------------------------------------- | +| admin | admin or the one generated by `ocis init` | admin@example.org | admin | users | +| einstein | relativity | einstein@example.org | user | users, philosophy-haters, physics-lovers, sailing-lovers, violin-haters | +| marie | radioactivity | marie@example.org | user | users, physics-lovers, polonium-lovers, radium-lovers | +| moss | vista | moss@example.org | space admin | users | +| richard | superfluidity | richard@example.org | user | users, philosophy-haters, physics-lovers, quantum-lovers | +| katherine | gemini | katherine@example.org | space admin | users, sailing-lovers, physics-lovers, quantum-lovers | + +You may also want to run oCIS with only your custom users by xref:../deployment#delete-demo-users.adoc[deleting the demo users]. diff --git a/modules/developer/pages/ocis/getting-started/index.adoc b/modules/developer/pages/ocis/getting-started/index.adoc new file mode 100644 index 00000000..e68e8837 --- /dev/null +++ b/modules/developer/pages/ocis/getting-started/index.adoc @@ -0,0 +1,136 @@ += Getting Started +:toc: right +:toclevels: 3 + + +== oCIS online demo + +We have an oCIS demo instance running on link:https://ocis.owncloud.com[ocis.owncloud.com] where you can get a first impression of it. + +We also have some more variations of oCIS running and xref:../deployment/continuous_deployment.adoc[continuously deployed] to reflect different scenarios in that oCIS might be used. + +== Run oCIS + +We are distributing oCIS as binaries and Docker images. + +[WARNING] +==== +The examples in this document assume that oCIS is accessed from the same host as it is running on (`localhost`). If you would like +to access oCIS remotely please refer to the xref:../deployment/basic-remote-setup.adoc[Basic Remote Setup] section. Especially +to the notes about setting the `PROXY_HTTP_ADDR` and `OCIS_URL` environment variables. +==== + + +You can find more deployment examples in the xref:../deployment.adoc[deployment section]. + +=== Binaries + +You can find the latest official release of oCIS at link:https://download.owncloud.com/ocis/ocis/stable/[our download mirror] or on link:https://github.com/owncloud/ocis/releases[GitHub]. +The latest build from the master branch can be found at link:https://download.owncloud.com/ocis/ocis/daily/[our download mirrors daily section]. Pre-Releases are available at link:https://download.owncloud.com/ocis/ocis/testing/[our download mirrors testing section]. + +To run oCIS as binary you need to download it first and then run the following commands. +For this example, assuming version 2.0.0-beta.5 of oCIS running on a Linux AMD64 host: + +[source,console] +---- +# download +curl https://download.owncloud.com/ocis/ocis/testing/2.0.0-beta.5/ocis-2.0.0-beta.5-linux-amd64 --output ocis + +# make binary executable +chmod +x ocis + +# initialize a minimal oCIS configuration +./ocis init + +# run with demo users +IDM_CREATE_DEMO_USERS=true ./ocis server +---- + +The default primary storage location is `~/.ocis` or `/var/lib/ocis` depending on the packaging format and your operating system user. You can change that value by configuration. + +[NOTE] +==== +When you're using oCIS with self-signed certificates, you need to answer the question for certificate checking with "yes" or set the environment variable `OCIS_INSECURE=true`, in order to make oCIS work. +==== + + +[WARNING] +==== +oCIS by default relies on Multicast DNS (mDNS), usually via avahi-daemon. If your system has a firewall, make sure mDNS is allowed in your active zone. +==== + + +[WARNING] +==== + +==== Open Files on macOS + +The start command `./ocis server` starts a runtime which runs all oCIS services in one process. On MacOS we have very low limits for open files. oCIS needs more than the default 256. Please raise the limit to 1024 by typing `ulimit -n 1024` within the same cli session where you start ocis from. +==== + + +=== Docker + +Docker images for oCIS are available on link:https://hub.docker.com/r/owncloud/ocis[Docker Hub]. + +The `latest` tag always reflects the current master branch. + +[source,console] +---- +docker pull owncloud/ocis +docker run --rm -it -v ocis-config:/etc/ocis owncloud/ocis init +docker run --rm -p 9200:9200 -v ocis-config:/etc/ocis -v ocis-data:/var/lib/ocis -e IDM_CREATE_DEMO_USERS=true owncloud/ocis +---- + +[NOTE] +==== +When you're using oCIS with self-signed certificates, you need to set the environment variable `OCIS_INSECURE=true`, in order to make oCIS work. +==== + + +[NOTE] +==== +When you're creating the xref:./demo-users.adoc[demo users] by setting `IDM_CREATE_DEMO_USERS=true`, you need to be sure that this instance is not used in production because the passwords are public. +==== + + +[WARNING] +==== +We are using named volumes for the oCIS configuration and oCIS data in the above example (`-v ocis-config:/etc/ocis -v ocis-data:/var/lib/ocis`). You could instead also use host bind-mounts instead, eg. `-v /some/host/dir:/var/lib/ocis`. + +You cannot use bind mounts on MacOS, since extended attributes are not supported (link:https://github.com/owncloud/ocis/issues/182[owncloud/ocis#182], link:https://github.com/moby/moby/issues/1070[moby/moby#1070]). +==== + + +== Usage + +=== Login to ownCloud Web + +Open link:https://localhost:9200[https://localhost:9200] and xref:./demo-users.adoc[login using one of the demo accounts]. + +=== Basic Management Commands + +The oCIS single binary contains multiple extensions and the `ocis` command helps you to manage them. You already used `ocis server` to run all available extensions in the xref:#run-ocis.adoc[Run oCIS] section. We now will show you some more management commands, which you may also explore by typing `ocis --help` or going to the xref:../config.adoc[docs]. + +To initialize the oCIS configuration: + +ocis init +// SHORTCODE: {{< / highlight >}} + +To start oCIS server: + +ocis server +// SHORTCODE: {{< / highlight >}} + +The list command prints all running oCIS services. +ocis list +// SHORTCODE: {{< / highlight >}} + +The version command prints the version of your installed oCIS. +ocis --version +// SHORTCODE: {{< / highlight >}} + +The health command is used to execute a health check, if the exit code equals zero the service should be up and running, if the exit code is greater than zero the service is not in a healthy state. Generally this command is used within our Docker containers, it could also be used within Kubernetes. + +ocis health --help +// SHORTCODE: {{< / highlight >}} diff --git a/modules/developer/pages/ocis/guides/ai_assisted_dev.adoc b/modules/developer/pages/ocis/guides/ai_assisted_dev.adoc new file mode 100644 index 00000000..e9c85155 --- /dev/null +++ b/modules/developer/pages/ocis/guides/ai_assisted_dev.adoc @@ -0,0 +1,448 @@ += Building oCIS Web Extensions with AI-Assisted Development +:toc: right +:toclevels: 3 + + +== Introduction + +This guide demonstrates how to build Infinite Scale (oCIS) web extensions using AI-assisted development (sometimes called "vibe coding") with Claude AI. By following this approach, review-ready extensions can be created without writing code manually—the AI handles syntax, boilerplate, and implementation while the developer focuses on architecture and user experience. + +_What you'll learn:_ +- Setting up MCP connectors so Claude can access your server +- Using Claude to install development tools and manage your environment +- The five-phase workflow for AI-assisted extension development +- Practical tips for supervising AI-generated code +- How to contribute extensions back to the oCIS community + +_Time investment:_ ~10-14 hours per extension (concept to review-ready) + +_Cost:_ ~$100 USD for two extensions (using the Claude Max subscription) + +== Prerequisites + +Before starting, ensure you have the following: + +- A running oCIS instance, see the link:https://doc.owncloud.com/ocis/latest/depl-examples/ubuntu-compose/ubuntu-compose-prod.html[Admin Docs] for a deployment example +- link:https://claude.com/pricing/max[Claude Max subscription] (starts with $100 US/month at the time of writing, other plans availabe) +- SSH access to your server (for MCP connectivity) +- Full root access to the server unless Claude Code, Claude MCP connector, oCIS and other tools all run as another user + +That's it. Claude will help install everything else. + +== Step 1: Configure MCP Connector + +// SHORTCODE: {{< hint type=important title="This Is the Hardest Part" >}} +Setting up the MCP connector is the most challenging step. Once this is done, Claude handles everything else—including installing development tools, managing git, and building extensions. +// SHORTCODE: {{< /hint >}} + +MCP (Model Context Protocol) allows Claude to connect directly to a server, enabling both the Claude web interface and Claude Code to access files, run commands, and debug oCIS deployments. + +=== Installation + +Follow the official MCP documentation to set up a connector: + +- _link:https://modelcontextprotocol.io/quickstart[MCP Quickstart Guide]_ +- _link:https://modelcontextprotocol.io/docs/develop/connect-local-servers[Connect to Local Servers]_ +- _link:https://support.claude.com/en/articles/10949351-getting-started-with-local-mcp-servers-on-claude-desktop[Claude Desktop MCP Setup]_ + +=== Verify Connection + +Once configured, verify Claude can access the server by asking: + +[source,text] +---- +Can you list the contents of /home on my server? +---- + +If Claude successfully lists directories, proceed to the next step. + +== Step 2: Let Claude Set Up Your Environment + +With MCP connected, Claude can install the development tools needed. Simply ask: + +[source,text] +---- +Please install Claude Code and Git on my server. I'll need these for +developing an oCIS web extension. +---- + +Claude will: +- Detect the operating system +- Install Claude Code using the appropriate method +- Install Git if not already present +- Configure any necessary paths or permissions + +== The Five-Phase Workflow + +This workflow has been refined over several projects and consistently produces review-ready code. + +=== Phase 1: Architecture (~15-30 minutes) + +Start in the Claude web interface (not Claude Code). Describe the project and have a conversation: + +[source,text] +---- +I want to build an oCIS web extension that displays photos in a timeline +grouped by their EXIF capture date. It should have infinite scroll starting +from today and going backwards, show camera metadata when you click on a photo, +and include an interactive map view showing where photos were taken. + +What questions do you have about the requirements? +---- + +Claude will ask clarifying questions about scope, user experience, and technical constraints. This back-and-forth shapes the architecture before any code is written. + +=== Phase 2: Deep Research (~30 minutes) + +Ask Claude to research the existing oCIS architecture: + +[source,text] +---- +Please research how oCIS web extensions work. I need to understand: +- The extension manifest format +- How extensions integrate with the oCIS web UI +- What APIs are available (Graph API, WebDAV, etc.) +- How existing extensions like json-viewer are structured +- The AMD module format requirements + +Examine the oCIS documentation and existing extensions to synthesise +a technical approach for my photo gallery extension. +---- + +Claude reads documentation, examines code patterns, and produces an architectural plan. + +=== Phase 3: Scaffolding (~15 minutes) + +Ask Claude to create the project structure on the server: + +[source,text] +---- +Please create the project structure for my photo-addon extension: +- Git repository initialised +- Build configuration (Vite with AMD output) +- TypeScript configuration +- manifest.json for oCIS +- Basic Vue 3 component structure +- A starter CLAUDE.md file documenting the project for future sessions +---- + +By the end of this phase, a working (empty) extension exists. + +=== Phase 4: Implementation (~5-6 hours) + +Login to the server and change to the folder where the new project was created, then run: + +[source,text] +---- +claude "Please review the CLAUDE.md project file, build the web extension, +and deploy it to our server. Install and configure any dependencies required. +Use 'sudo' where needed to modify or configure the oCIS environment." +---- +Once complete, work through features incrementally. Example prompts: + +_Feature requests:_ + +[source,text] +---- +Please adjust the logic to start by searching for today's date (unless +overwritten by the dropdowns at the top), then display them on the screen. +If the images don't fill the screen, load up another day, until the screen +is filled plus some buffer. Then when I scroll, bring in more images. +---- + +[source,text] +---- +In the map view, when I hover over a dot, the thumbnail shows up above +off the screen. Can you fix the positioning? +---- + +_Bug fixes:_ + +[source,text] +---- +I'm seeing this error in the console: +[paste error here] + +Can you investigate and fix it? +---- + +=== Phase 5: Polish (~3 hours) + +Once core features work, request comprehensive cleanup: + +_Performance optimisation:_ + +[source,text] +---- +Are there any performance improvements that we can make? +---- + +_Bug hunting:_ + +[source,text] +---- +Are there any memory leaks or bugs that can be found? +---- + +_Code quality:_ + +[source,text] +---- +Are there any inefficiencies we can improve on? Unused variables? +Duplicate functions? Excessive if statements? Overwriting CSS styles? +Overly complex logic? +---- + +_Error handling:_ + +[source,text] +---- +Are there any error handling gaps or functions that may fall through +without handling all scenarios? +---- + +_Documentation:_ + +[source,text] +---- +Are there any missing comments? Or complex code that could use +clear documentation? +---- + +_Internationalisation:_ + +[source,text] +---- +Are there any visible text strings that are not yet set up for translation? +---- + +_Comprehensive cleanup:_ + +[source,text] +---- +Pulling from the backlog list, can you do a good code refactor to simplify, +add all required unit tests, and add i18n support, then commit and push the +code with the right comments. +---- + +== Active Supervision + +// SHORTCODE: {{< hint type=warning title="This is NOT 'Set It and Forget It'" >}} +While no coding is required, this approach demands active supervision. Developers cannot simply give Claude a prompt and walk away. +// SHORTCODE: {{< /hint >}} + +=== When to Intervene + +There are times when Claude Code may: +- Deviate from requirements (with good intentions) +- Undo something it had just done +- Go down a path that doesn't match the architecture + +The decisions are always logical, but sometimes the harder path is required. Expect to intervene and redirect approximately ~5% of the time. + +_Watch for:_ +- Changes that don't match architectural decisions +- Unnecessary complexity being added +- Repeated undo/redo cycles +- Deviation from oCIS extension patterns + +=== Code Cleanup Is Essential + +AI-generated code works, but it can accumulate: +- Redundant logic +- Unused variables +- Overly complex conditionals +- Duplicate CSS styles +- Missing good practice for the orders of directives +- Missing the definition of required interactive handlers + +When submitting a pull request (PR), issues highlighted during CI steps, such as linting, can be resolved by explicitly asking for code refactoring and optimisation. + +=== UI Testing Is Manual + +The AI can write unit tests, but exploratory testing requires human judgment: +- Click around the interface +- Try unexpected inputs +- Test with different user accounts (search in oCIS is space-scoped!) +- Check mobile layouts +- Look for edge cases + +== Debugging Techniques + +When issues arise, these techniques help Claude understand and fix problems: + +=== Console Output + +Open Chrome DevTools, copy console errors, and paste into Claude: + +[source,text] +---- +I'm seeing this error in the console: +[paste error here] + +Can you investigate and fix it? +---- + +=== HAR Files + +For performance issues, export HAR files (network traffic captures): + +1. Open DevTools > Network tab +2. Reproduce the issue +3. Right-click > Save all as HAR +4. Share the HAR file with Claude for analysis + +=== HTML/CSS Inspection + +Copy specific HTML/CSS snippets from DevTools: + +[source,text] +---- +The thumbnail hover popup is positioned incorrectly. Here's the current +CSS and HTML structure: +[paste code here] + +Can you fix the positioning so the popup appears above the cursor +but stays within the viewport? +---- + +// SHORTCODE: {{< hint type=tip title="Chrome Extension Limitations" >}} +Claude has a Chrome extension, but it may be limited for this use case. Manually copying console output and HTML/CSS snippets into Claude is often more effective. +// SHORTCODE: {{< /hint >}} + +== Example Web Extensions Built with This Approach + +=== Photo Gallery Extension (web-app-photo-addon) + +Features: +- _Timeline View_: Photos grouped by EXIF capture date with infinite scroll +- _Pinch-to-zoom calendar_: Navigate between day, month, and year views +- _EXIF Metadata Panel_: Camera make/model, aperture, ISO, focal length +- _Map View_: Interactive Leaflet map with marker clustering +- _Lightbox_: Full-screen viewer with keyboard navigation + +Tech stack: Vue 3 Composition API, TypeScript, Leaflet.js, AMD module format + +=== Advanced Search Extension (web-app-advanced-search) + +Features: +- _Photo Metadata Filters_: Search by camera, date range, aperture, ISO +- _Filter Chips_: Visual indicators with one-click removal +- _Search Saving_: Save and retrieve search queries +- _KQL Search_: Direct advanced query editing +- _Index Statistics_: View indexing status and configuration + +=== Backend Changes in oCIS + +To enable photo metadata search, changes were contributed to oCIS core: +- Photo field mappings in the KQL parser +- `Store=true` configuration in Bleve for field retrieval +- WebDAV properties for photo metadata + +This was the trickiest part—understanding why searches returned empty results. The answer: Bleve was indexing fields but not storing them for retrieval. + +== Tips for oCIS Development + +=== 1. Start with Conversation, Then Architecture + +Don't jump straight into implementation. Spend ~30-60 minutes in regular chat, discussing the problem space and having Claude research the codebase. Those early conversations shape the architecture that makes implementation smooth. + +=== 2. Work Incrementally + +Ask for one feature at a time. Get it working, then move to the next. This creates natural checkpoints and makes debugging much easier. + +=== 3. Use Browser DevTools Liberally + +Console errors, network requests, and performance profiles are incredibly valuable context. Copy and paste them into AI conversations. + +=== 4. Understand oCIS Extension Requirements + +Extensions must: +- Use CJS module format (not ES modules) +- Have a proper `manifest.json` +- Use the oCIS web SDK for API access + +Claude learns these constraints during the research phase and applies them consistently. + +=== 5. Test with Multiple Accounts + +Search in oCIS is space-scoped. What works for an admin account might not work for regular users. + +=== 6. Read Maintainer Feedback Carefully + +PR reviews teach things no documentation mentions. The oCIS team's suggestions can significantly improve code quality. + +== Contributing to oCIS or web-extensions + +Once a web-extension is ready, Claude can handle the entire contribution workflow: + +- Install and configure Git +- Create forks of the oCIS repositories +- Create feature branches +- Generate commits with appropriate messages +- Push changes to GitHub +- Create pull requests with descriptions and screenshots + +Example prompts: + +[source,text] +---- +Please fork the owncloud/web-extensions repository to my GitHub account +and clone it to my server. +---- + +[source,text] +---- +Create a new branch called feat/web-app-photo-addon and commit our +extension with an appropriate commit message. +---- + +[source,text] +---- +Please create a pull request for our photo-addon extension. Include +screenshots from the /screenshots directory and a clear description +of what the extension does. +---- + +_Contributions made by this approach:_ +- _oCIS core_: Photo metadata search backend +- _web-extensions_: Photo gallery extension +- _web-extensions_: Advanced search extension + +// SHORTCODE: {{< hint type=note title="CLA Requirement" >}} +Contributors need to sign the ownCloud Contributor License Agreement (CLA) for PRs to be accepted. Claude can help find the CLA link and explain the process. +// SHORTCODE: {{< /hint >}} + +== Notes + +Additional steps can be required in the oCIS repository for changes in the web-extensions repository, in particular: + +- If the web extension requires it, add proposals to the back-end code in the oCIS codebase. +- In order to make the new web extension more easily accessible to the public, the `ocis_full` deployment example in the oCIS repository should be updated. The new oCIS version will then reference it automatically and the documentation team will update the release notes and documentation accordingly. + +== Summary + +Using this approach, complete web-extensions can be built without writing code manually. No git commands need to be typed manually. No pnpm commands either. + +Every commit, every push, every PR—all generated through AI assistance. The backend changes to oCIS core were approved and merged. The web-extensions demonstrate the viability of this development approach. + +The barrier to entry for meaningful open source contribution is significantly lower with AI-assisted development. For those interested in contributing to oCIS but feeling intimidated by the codebase, this approach is worth considering. + +Please note that, as with all contributions, this approach requires a full review process from the code maintainers, with iterations required to finalise all steps. To ensure stability and maintainability, all feedback must be incorporated into the proposed changes. + +''' + +== Resources + +| Resource | URL | +|----------|-----| +| MCP Quickstart | https://modelcontextprotocol.io/quickstart | +| MCP Local Servers | https://modelcontextprotocol.io/docs/develop/connect-local-servers | +| Claude Desktop MCP | https://support.claude.com/en/articles/10949351-getting-started-with-local-mcp-servers-on-claude-desktop | +| oCIS Web Extensions | https://github.com/owncloud/web-extensions | +| oCIS Admin Documentation | https://doc.owncloud.com/ocis/ | +| oCIS Developer Documentation | https://owncloud.dev | +''' + +_This guide was written with AI assistance and reviewed by the ownCloud documentation team for accuracy._ + diff --git a/modules/developer/pages/ocis/guides/index.adoc b/modules/developer/pages/ocis/guides/index.adoc new file mode 100644 index 00000000..205ba798 --- /dev/null +++ b/modules/developer/pages/ocis/guides/index.adoc @@ -0,0 +1,16 @@ += Guides +:toc: right +:toclevels: 3 + + +== Step-by-step Guides + +These guides should extend the generic documentation and help admins to learn Infinite Scale through digestable "hands-on" sessions. + +=== Available Guides + +oCIS deployments are super simple, yet there are many configurations possible for advanced setups. + +- {{< icon "docker" >}} xref:ocis-mini-eval.adoc[Minimalistic Evaluation Guide for oCIS with Docker] - minimalistic evaluation +- {{< icon "docker" >}} xref:ocis-local-docker.adoc[Discover oCIS with Docker] - configure domain, certificates and port +- {{< icon "person-walking-luggage" >}} xref:migrate-data-rclone.adoc[Migrate data to Infinite Scale] diff --git a/modules/developer/pages/ocis/guides/migrate-data-rclone.adoc b/modules/developer/pages/ocis/guides/migrate-data-rclone.adoc new file mode 100644 index 00000000..3e07eb12 --- /dev/null +++ b/modules/developer/pages/ocis/guides/migrate-data-rclone.adoc @@ -0,0 +1,109 @@ +:toc: right +:toclevels: 3 + + +''' +title: "Migrate Data using rclone" +date: 2020-06-12T14:35:00+01:00 +weight: 10 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/ocis/guides +geekdocFilePath: migrate-data-rclone.md +geekdocCollapseSection: true +''' + +People keep asking on how to migrate data from other cloud storage to Infinite Scale. + +There are too many cloud variants and use cases out there to have a migration path for all at hand, but let's see what we can start with: There is the famous Swiss army knife for clouds called rclone available. + +The awesome rclone tool makes it easy to migrate data from one installation to another on a user per user base. A very good first step. + +This article explains by the example of Nextcloud how you would migrate your data from an running NC to Infinite Scale. A prerequisite is that you have Infinite Scale already set up and running on a different domain. + +[WARNING] +==== +Be prepared that migration can take some time. Also, check the size of your data. This example was around 1.5 GiB of data, that obviously went fast. + +And of course: Have a backup! Even if this method only reads from the source, you never know. +==== + + +== Install rclone + +Check the link:https://rclone.org/install/[rclone website on how to install] rclone. + +== Create Users + +First, decide on a user that you want to migrate. To create the user on Infinite Scale, log in as an admin user and create the desired user. Create and set groups accordingly. + +For the next step, you need the user credentials on both the source- and the destination cloud. + +== Configure rclone Remotes + +To be able to address the clouds in rclone, you need to configure so called _remotes_. It is nothing else than a shortcut for the combination of +- which kind of cloud are you talking to +- the URL +- the username +- the password, if one is set + +You need to add a configuration for both the source cloud (Nextcloud) and the target (Infinite Scale). As both talk WebDAV, the link:https://rclone.org/webdav/[rclone manual page] is accurate to follow. + +For both, use an URL in the form of `https://my.host.de/remote.php/webdav`. + +Once that is finished, the command `rclone config show` should give output similar to this: + +[source,bash] +---- +[:~/] ± rclone config show + +[NCoC] +type = webdav +url = https://nc.this.de/remote.php/webdav +vendor = nextcloud +user = wilma +pass = zfdsaiewrafdskfjdasfxdasffdafdsafas + +[ocis] +type = webdav +url = https://infinitescale.works/remote.php/webdav +vendor = owncloud +user = wilma +pass = cdsfasrefdsadaGkxTXjksfpqQFI5nQawqs + +---- + +Now, for example the directories on the Nextcloud root can be checked with `rclone lsd NCoC:/`. + +== Copy Data + +To migrate the data, rclone provides the command `copy`. It transfers data from one remote to the other. Use the following command example to transfer the entire cloud data from Nextcloud to Infinite Scale: +---- +rclone copy NCoC:/ ocis:/ --no-check-certificate -P +---- +The --no-check-certificate can and should be skipped if your clouds have proper certificates. The `-P` however, provides you with interesting statistics about the copy progress. +Once you are finished, this might be the result: +---- +[:~/] $ rclone copy NCoC:/ ocis:/ --no-check-certificate -P +Transferred: 1.228 GiB / 1.228 GiB, 100%, 10.170 MiB/s, ETA 0s +Transferred: 411 / 411, 100% +Elapsed time: 2m19.3s +---- + +Note that while testing this, occasionally the Nextcloud was returning a `404 not found` for some files. While the reason for that was not completely clear, it does not matter, because the rclone command can be repeated. It is clever enough to only copy what has changed! + +== Enjoy! + +All done! Now you have your data on Infinite Scale. + +Obviously this method has a few downsides, such as: +- This migration requires a little of "quiet time" for migrating data. +- It is a user by user method where provisioning of users has to be done manually. +- Only data is migrated, and there is probably a data size limit in real life using this way. +- Private- and public shares are not migrated +- The trashbin, versions, comments and favorites are not migrated + +These are shortcomings but this is a good first step to start investigating. The other parts will be sorted as we move along. + +''' +To improve this guide, you are welcome to file an issue or even send a pull request. See the link:https://owncloud.dev/ocis/development/build-docs/[getting started guide] how easy it is to build the documentation. + diff --git a/modules/developer/pages/ocis/guides/ocis-and-containers.adoc b/modules/developer/pages/ocis/guides/ocis-and-containers.adoc new file mode 100644 index 00000000..352d242a --- /dev/null +++ b/modules/developer/pages/ocis/guides/ocis-and-containers.adoc @@ -0,0 +1,36 @@ += oCIS and Containers +:toc: right +:toclevels: 3 + +== Cloud Native + +Why do we recommend to work with containers? + + +=== {{< icon "scale-balanced" >}}   Lightweight + +Containers are more lightweight than VMs. It is easier to work with shared volumes and networks because they are isolated from the host system. +<---> + +=== {{< icon "shield-halved" >}}   Dependencies + +The container images have all dependencies installed and the maintainer takes care for keeping them up-to-date. + +<---> + +=== {{< icon "gauge-high" >}}   Scaling + +In addition to that, containers help with scaling. You can run multiple instances of one container and distribute them across hosts. + + +== Docker compose + +For oCIS deployments you often need multiple services. These services need to share resources like volumes and networks. If you do not use any orchestration tool, you would end up writing bash scripts to create and update containers and volumes and connect them via networks. This is what orchestration tools like docker compose can do for you. You define a service mesh using .yaml files and the tool tries to run and maintain that. You gain more value and a version history by using a version control system. Your deployment configuration is fully written down as a spec and you will never touch any system directly and change the config manually. + +== Kubernetes + +Containers are also used in link:https://kubernetes.io/[kubernetes]. Kubernetes is part of a huge ecosystem and is founded on best-of-breed practises to orchestrate large scale container applications and services. + +== oCIS and Containers + +oCIS was developed as microservices. We do not scale the whole system as a monolith but we scale the individual services. diff --git a/modules/developer/pages/ocis/guides/ocis-local-docker.adoc b/modules/developer/pages/ocis/guides/ocis-local-docker.adoc new file mode 100644 index 00000000..89bded3e --- /dev/null +++ b/modules/developer/pages/ocis/guides/ocis-local-docker.adoc @@ -0,0 +1,344 @@ += Discover oCIS with Docker +:toc: right +:toclevels: 3 + + +== Prerequisites + +- Local docker installation (e.g. Docker for Desktop) +- Check xref:ocis-and-containers.adoc[oCIS and Containers] + +== Start oCIS with docker compose + +=== Create the project + +Use the following skeleton as a docker-compose.yml: + +[source,bash] +---- +mkdir simple-ocis && \ +cd simple-ocis && \ +touch docker-compose.yml +---- + +Copy the following file content into `docker-compose.yml` and save it. + +[source,yaml] +---- +version: "3.7" + +services: + ocis: + image: owncloud/ocis:latest + environment: + # INSECURE: needed if oCIS / Traefik is using self generated certificates + OCIS_INSECURE: "true" + + # OCIS_URL: the external domain / ip address of oCIS (with protocol, must always be https) + OCIS_URL: "https://localhost:9200" + + # OCIS_LOG_LEVEL: error / info / ... / debug + OCIS_LOG_LEVEL: info +---- + +=== Prepare Paths + +Create directories if not exists: + +[source,bash] +---- +mkdir -p $(pwd)/ocis-config \ +mkdir -p $(pwd)/ocis-data +---- + + +Set the user for the directories to be the same as the user inside the container: + +[source,bash] +---- +sudo chown -Rfv 1000:1000 $(pwd)/ocis-config/ \ +sudo chown -Rfv 1000:1000 $(pwd)/ocis-data +---- + +=== Initialize + +[source,bash] +---- +docker run --rm -it -v $(pwd):/etc/ocis/ owncloud/ocis:latest init +---- + +You will get the following output: + +[source,txt] +---- +Do you want to configure Infinite Scale with certificate checking disabled? + This is not recommended for public instances! [yes | no = default] yes + +========================================= + generated OCIS Config +========================================= + configpath : /etc/ocis/ocis.yaml + user : admin + password : t3p4N0jJ47LbhpQ04s9W%u1$d2uE3Y.3 +---- + +Check your local folder. We just generated a default ocis configuration file with random passwords and secrets. + +[source,bash] +---- +ls # list the current folder +docker-compose.yml ocis.yaml # ocis.yaml has been generated +---- + +Run `cat ocis.yaml` + +[source,yaml] +---- +token_manager: + jwt_secret: X35rffWpS9BR.=^#LDt&z3ykYOd7h@w* +machine_auth_api_key: -0$4ieu5+t6HD6Ui^0PpKU6B0qxisv.m +system_user_api_key: oVxICwMR9YcKXTau+@pqKZ0EO-OHz8sF +transfer_secret: e%3Sda=WFBuy&ztBUmriAbBR$i2CmaDv +system_user_id: b7d976a1-7300-4db7-82df-13502d6b5e18 +admin_user_id: c59a6ae9-5f5e-4eef-b82e-0e5c34f93e52 +graph: + spaces: + insecure: false + identity: + ldap: + bind_password: wElKpGjeH0d.E4moXh=.dc@s2CtB0vy% +idp: + ldap: + bind_password: Ft2$2%#=6Mi22@.YPkhh-c6Kj=3xBZAb +idm: + service_user_passwords: + admin_password: t3p4N0jJ47LbhpQ04s9W%u1$d2uE3Y.3 + idm_password: wElKpGjeH0d.E4moXh=.dc@s2CtB0vy% + reva_password: pJAdZ2fU!SFKgcdDPRW%ruIiNM6GnN1D + idp_password: Ft2$2%#=6Mi22@.YPkhh-c6Kj=3xBZAb +proxy: + insecure_backends: false +frontend: + archiver: + insecure: false +auth_basic: + auth_providers: + ldap: + bind_password: pJAdZ2fU!SFKgcdDPRW%ruIiNM6GnN1D +auth_bearer: + auth_providers: + oidc: + insecure: false +users: + drivers: + ldap: + bind_password: pJAdZ2fU!SFKgcdDPRW%ruIiNM6GnN1D +groups: + drivers: + ldap: + bind_password: pJAdZ2fU!SFKgcdDPRW%ruIiNM6GnN1D +storage_system: + data_provider_insecure: false +storage_users: + data_provider_insecure: false +ocdav: + insecure: false +thumbnails: + thumbnail: + transfer_secret: z-E%G8MTeFpuT-ez2o8BjfnG1Jl2yLLm + webdav_allow_insecure: false + cs3_allow_insecure: false +---- + +// SHORTCODE: {{< hint type=tip title="Admin password" >}} +_Password initialisation_\ +During the run of `./ocis init`, the password for the `admin` user has been set to a random string. + +You can override that later by setting `IDM_ADMIN_PASSWORD=secret`. The ENV variable setting always overrides the config file. +// SHORTCODE: {{< /hint >}} + +=== Mount the config file + +Add the config file as a bind mount. + +[source,yaml] +---- + volumes: + # mount the ocis config file inside the container + - "./ocis.yaml:/etc/ocis/ocis.yaml" +---- + +=== Apply the changes + +[source,bash] +---- +docker compose up -d +---- + +The service should be running. + +[source,bash] +---- +docker compose ps +NAME COMMAND SERVICE STATUS PORTS +simple-ocis-ocis-1 "/usr/bin/ocis server" ocis running 9200/tcp +---- + +=== Open the port 9200 to the outside + +[source,yaml] +---- +ports: + - 9200:9200 +---- + +Add a port mapping to your docker compose file and run `docker compose up -d` again. You should now be able to access and log in. You will get a warning from your browser that the connection is not safe because we are using a self-signed certificate. Accept this warning message to continue. Use the user `admin` and the password which has been initialized before. + +Congratulations! You have successfully set up a simple Infinite Scale locally. + +// SHORTCODE: {{< hint type=tip title="Docker Compose Helps you" >}} +_docker compose up_\ +You do not need to shut down your service to apply changes from the docker-compose.yml file. Running `docker compose up -d` again is enough. Docker compose always tries to bring all services to the desired state. + +_docker compose ps_\ +This gives you a status of the services of the project. + +_docker compose exec \ \_\ +This command is handy to run specific commands inside your service. Try `docker compose exec ocis ocis version`. +// SHORTCODE: {{< /hint >}} + +=== Persist data, restart and logging + +The key to a successful container setup is the persistence of the application data to make the data survive a re-boot. Docker normally uses link:https://docs.docker.com/storage/volumes/[volumes] for this purpose. A volume can either be a "named volume" which are completely managed by docker and have many advantages (see the linked docker documentation), or "bind mounts" which are using the directory structure and OS of the host system. In our example we already use a bind mount for the config file. We will now add a named volume for the oCIS data directory. + +This is the way we should configure the ocis service: + +[source,yaml] +---- + volumes: + # mount the ocis config file inside the container + - "./ocis.yaml:/etc/ocis/ocis.yaml" + # short syntax for using a named volume + # in the form : + # use a named volume for the ocis data directory + - "ocis-data:/var/lib/ocis" + # or the more verbose syntax + #- type: volume + # source: ocis-data # name of the volume + # target: /var/lib/ocis # the mount path inside the container +---- + +The docker-compose.yml needs to declare the named volumes globally, add this at the bottom of the file: + +[source,yaml] +---- +# https://docs.docker.com/compose/compose-file/compose-file-v3/#volumes +# this declares the named volume with all default settings +# you can also see the volume when running `docker volume list` +volumes: + ocis-data: +---- + +Now let us configure the restart policy and the logging settings for the ocis service: + +[source,yaml] +---- + # https://docs.docker.com/compose/compose-file/compose-file-v3/#restart + restart: always # or on-failure / unless-stopped + + # https://docs.docker.com/config/containers/logging/configure/ + # https://docs.docker.com/compose/compose-file/compose-file-v3/#logging + # the default log driver does no log rotation + # you can switch to the "local" log driver which does rotation by default + logging: + driver: local + # otherwise you could specify log rotation explicitly + # driver: "json-file" # this is the default driver + # options: + # max-size: "200k" # limit the size of the log file + # max-file: "10" # limit the count of the log files +---- + +Apply your changes! Just run `docker compose up -d` again. + +Now you have an oCIS which will survive reboots, restart automatically and has log rotation by default. + +Access the logs via `docker compose logs -f` and do some actions in the frontend to see the effect. Create data by uploading files and adding more users. Then run `docker compose down` to shut oCIS down. Start it again `docker compose up -d`, log in again and check that your data has survived the reboot. + +=== Pin the oCIS version + +Last but not least, it is never a good idea to use the `latest` docker tag. Pin your container image to a released version. + +[source,yaml] +---- + image: owncloud/ocis:latest@sha256:5ce3d5f9da017d6760934448eb207fbaab9ceaf0171b4122e791e292f7c86c97 + # the latest tag is not recommended, because you don't know which version you'll get + # but even if you use `owncloud/ocis:1.16.0` you cannot be sure that you'll get + # the exact same image if you pull it at a later point in time (because docker image tags are not immutable). + # To be 100% that you always get the same image, you can pin the digest (hash) of the + # image. If you do a `docker pull owncloud/ocis:latest`, it also will also show you the digest. + # see also https://docs.docker.com/engine/reference/commandline/images/#list-image-digests +---- + +== Wrapping up + +If you have completed this guide, your docker-compose.yml should look like the following example: + +// SHORTCODE: {{< expand "Solution" "..." >}} +[source,yaml] +---- +version: "3.7" + +services: + ocis: + image: owncloud/ocis:latest@sha256:5ce3d5f9da017d6760934448eb207fbaab9ceaf0171b4122e791e292f7c86c97 + # the latest tag is not recommended, because you don't know which version you'll get + # but even if you use `owncloud/ocis:1.16.0` you cannot be sure that you'll get + # the exact same image if you pull it at a later point in time (because docker image tags are not immutable). + # To be 100% that you always get the same image, you can pin the digest (hash) of the + # image. If you do a `docker pull owncloud/ocis:latest`, it also will also show you the digest. + # see also https://docs.docker.com/engine/reference/commandline/images/#list-image-digests + environment: + # INSECURE: needed if oCIS / Traefik is using self generated certificates + OCIS_INSECURE: "true" + + # OCIS_URL: the external domain / ip address of oCIS (with protocol, must always be https) + OCIS_URL: "https://localhost:9200" + + # OCIS_LOG_LEVEL: error / info / ... / debug + OCIS_LOG_LEVEL: info + volumes: + # mount the ocis config file inside the container + - "./ocis.yaml:/etc/ocis/ocis.yaml" + # short syntax for using a named volume + # in the form : + # use a named volume for the ocis data directory + - "ocis-data:/var/lib/ocis" + # or the more verbose syntax + #- type: volume + # source: ocis-data # name of the volume + # target: /var/lib/ocis # the mount path inside the container + ports: + - 9200:9200 + # https://docs.docker.com/compose/compose-file/compose-file-v3/#restart + restart: always # or on-failure / unless-stopped + + # https://docs.docker.com/config/containers/logging/configure/ + # https://docs.docker.com/compose/compose-file/compose-file-v3/#logging + # the default log driver does no log rotation + # you can switch to the "local" log driver which does rotation by default + logging: + driver: local + # otherwise you could specify log rotation explicitly + # driver: "json-file" # this is the default driver + # options: + # max-size: "200k" # limit the size of the log file + # max-file: "10" # limit the count of the log files + +# https://docs.docker.com/compose/compose-file/compose-file-v3/#volumes +# this declares the named volume with all default settings +# you can also see the volume when running `docker volume list` +volumes: + ocis-data: +---- +// SHORTCODE: {{< /expand >}} diff --git a/modules/developer/pages/ocis/guides/ocis-mini-eval.adoc b/modules/developer/pages/ocis/guides/ocis-mini-eval.adoc new file mode 100644 index 00000000..812b033e --- /dev/null +++ b/modules/developer/pages/ocis/guides/ocis-mini-eval.adoc @@ -0,0 +1,103 @@ += Minimalistic Evaluation Guide for oCIS with Docker +:toc: right +:toclevels: 3 + + +_IMPORTANT_ This type of installation is _ONLY_ intended for a quick evaluation and to access the instance from your local machine using internal certificates only. We recommend using test data only for this evaluation. + +== Prerequisites + +As a prerequisite, we assume you have Docker installed locally on a machine that has a GUI. A headless one will not work. You know how to use Docker commands and what they mean. This guide is not intended to be a detailed explanation. + +== What you can Expect + +By passing the commands as described, you will be able to get a very first look and feel of Infinite Scale, which can be accessed from your browser using `localhost`. + +== Setup Procedure + +The setup process is quite simple and is done from a terminal. + +=== Prepare Paths + +Create directories if not exists: + +[source,bash] +---- +mkdir -p $HOME/ocis/ocis-config \ +mkdir -p $HOME/ocis/ocis-data +---- + +Set the user for the directories to be the same as the user inside the container: + +[source,bash] +---- +sudo chown -Rfv 1000:1000 $HOME/ocis/ +---- + +=== Pull the Image + +[source,bash] +---- +docker pull owncloud/ocis +---- + +=== First Time Initialisation + +[source,bash] +---- +docker run --rm -it \ + --mount type=bind,source=$HOME/ocis/ocis-config,target=/etc/ocis \ + --mount type=bind,source=$HOME/ocis/ocis-data,target=/var/lib/ocis \ + owncloud/ocis init --insecure yes +---- + +You will get an output like the following: + +[source,txt] +---- +========================================= + generated OCIS Config +========================================= + configpath : /etc/ocis/ocis.yaml + user : admin + password : t3p4N0jJ47LbhpQ04s9W%u1$d2uE3Y.3 +---- + +Note that the password displayed is the one that will be used when you first log in or until it is changed. + +=== Recurring Start of Infinite Scale + +[source,bash] +---- +docker run \ + --name ocis_runtime \ + --rm \ + -it \ + -p 9200:9200 \ + --mount type=bind,source=$HOME/ocis/ocis-config,target=/etc/ocis \ + --mount type=bind,source=$HOME/ocis/ocis-data,target=/var/lib/ocis \ + -e OCIS_INSECURE=true \ + -e PROXY_HTTP_ADDR=0.0.0.0:9200 \ + -e OCIS_URL=https://localhost:9200 \ + owncloud/ocis +---- + +== Access Infinite Scale with the Browser + +To access Infinite Scale, open your browser and type as url: + +---- +https://localhost:9200 +---- + +== Remove the Evaluation + +[source,bash] +---- +sudo docker rmi owncloud/ocis \ +sudo rm -r $HOME/ocis +---- + +== Next Steps + +After evaluation, we strongly recommend that you use the link:https://doc.owncloud.com/ocis/next/depl-examples/ubuntu-compose/ubuntu-compose-prod.html[Install Infinite Scale on a Server] documentation which is ready for production and remove this evaluation setup completely. diff --git a/modules/developer/pages/ocis/identity-provider/index.adoc b/modules/developer/pages/ocis/identity-provider/index.adoc new file mode 100644 index 00000000..ffca481f --- /dev/null +++ b/modules/developer/pages/ocis/identity-provider/index.adoc @@ -0,0 +1,7 @@ += Identity Provider +:toc: right +:toclevels: 3 + +== Overview + +oCIS provides out of the box a minimal OpenID Connect provider via the link:../../services/idp/[IDP service] and a minimal LDAP service via the link:../../services/idm/[IDM service]. Both services are limited in the provided functionality, see the link:https://doc.owncloud.com/ocis/next/deployment/services/s-list/idp.html[admin documentation] for details, and can be used for small environments like up to a few hundred users. For enterprise environments, it is highly recommended using enterprise grade external software like KeyCloak plus openLDAP or MS ADFS with Active Directory, which can be configured in the respective service. Entrada ID (formerly Azure AD) is in preparation, but not yet released or documented and might need some small fixes, and for certain functions a LDAP/AD connection. diff --git a/modules/developer/pages/ocis/identity-provider/ldap-active-directory.adoc b/modules/developer/pages/ocis/identity-provider/ldap-active-directory.adoc new file mode 100644 index 00000000..e478ab0a --- /dev/null +++ b/modules/developer/pages/ocis/identity-provider/ldap-active-directory.adoc @@ -0,0 +1,37 @@ += LDAP - Active Directory +:toc: right +:toclevels: 3 + +== Overview + +oCIS can be configured using Active Directory as identity provider. + +== Configuration Example + +This configuration is an _example_ for using Samba4 AD as well as a Windows Server 2022 as the LDAP backend for oCIS. It is intended as guideline and first starting point. + +[source,text] +---- +OCIS_LDAP_URI=ldaps://xxxxxxxxx +OCIS_LDAP_INSECURE="true" +OCIS_LDAP_BIND_DN="cn=administrator,cn=users,xxxxxxxxxx" +OCIS_LDAP_BIND_PASSWORD=xxxxxxx +OCIS_LDAP_DISABLE_USER_MECHANISM="none" +OCIS_LDAP_GROUP_BASE_DN="dc=owncloud,dc=test" +OCIS_LDAP_GROUP_OBJECTCLASS="group" +OCIS_LDAP_GROUP_SCHEMA_ID="objectGUID" +OCIS_LDAP_GROUP_SCHEMA_ID_IS_OCTETSTRING="true" +OCIS_LDAP_GROUP_SCHEMA_GROUPNAME="cn" +OCIS_LDAP_USER_BASE_DN="dc=owncloud,dc=test" +OCIS_LDAP_USER_OBJECTCLASS="user" +OCIS_LDAP_USER_SCHEMA_ID="objectGUID" +OCIS_LDAP_USER_SCHEMA_ID_IS_OCTETSTRING="true" +OCIS_LDAP_USER_SCHEMA_USERNAME="sAMAccountName" +OCIS_LDAP_LOGIN_ATTRIBUTES="sAMAccountName" +IDP_LDAP_LOGIN_ATTRIBUTE="sAMAccountName" +IDP_LDAP_UUID_ATTRIBUTE="objectGUID" +IDP_LDAP_UUID_ATTRIBUTE_TYPE=binary +GRAPH_LDAP_SERVER_WRITE_ENABLED="false" +OCIS_EXCLUDE_RUN_SERVICES=idm +OCIS_ADMIN_USER_ID="" +---- diff --git a/modules/developer/pages/ocis/identity-provider/oidc.adoc b/modules/developer/pages/ocis/identity-provider/oidc.adoc new file mode 100644 index 00000000..44782dc4 --- /dev/null +++ b/modules/developer/pages/ocis/identity-provider/oidc.adoc @@ -0,0 +1,101 @@ += OIDC +:toc: right +:toclevels: 3 + +Infinite Scale has implemented OpenID Connect (OIDC) for authentication. +OIDC defines a discovery mechanism, called OpenID Connect Discovery, +where an OpenID server publishes its metadata at a well-known URL, typically: + +`https://ocis.test/.well-known/openid-configuration` + +This URL returns a JSON listing of the OpenID/OAuth endpoints, supported scopes and claims, public keys used to sign the tokens, and other details. +The clients can use this information to construct a request to the OpenID server. +The field names and values are defined in the link:https://openid.net/specs/openid-connect-discovery-1_0.html[OpenID Connect Discovery Specification]. +Here is an example of data returned: +[source,json] +---- +{ + "issuer": "https://ocis.test", + "authorization_endpoint": "https://ocis.test/signin/v1/identifier/_/authorize", + "token_endpoint": "https://ocis.test/konnect/v1/token", + "userinfo_endpoint": "https://ocis.test/konnect/v1/userinfo", + "end_session_endpoint": "https://ocis.test/signin/v1/identifier/_/endsession", + "check_session_iframe": "https://ocis.test/konnect/v1/session/check-session.html", + "jwks_uri": "https://ocis.test/konnect/v1/jwks.json", + "scopes_supported": [ + "openid", + "offline_access", + "profile", + "email", + "LibgreGraph.UUID", + "LibreGraph.RawSub" + ], + "response_types_supported": [ + "id_token token", + "id_token", + "code id_token", + "code id_token token" + ], + "subject_types_supported": [ + "public" + ], + "id_token_signing_alg_values_supported": [ + "RS256", + "RS384", + "RS512", + "PS256", + "PS384", + "PS512" + ], + "userinfo_signing_alg_values_supported": [ + "RS256", + "RS384", + "RS512", + "PS256", + "PS384", + "PS512" + ], + "request_object_signing_alg_values_supported": [ + "ES256", + "ES384", + "ES512", + "RS256", + "RS384", + "RS512", + "PS256", + "PS384", + "PS512", + "none", + "EdDSA" + ], + "token_endpoint_auth_methods_supported": [ + "client_secret_basic", + "none" + ], + "token_endpoint_auth_signing_alg_values_supported": [ + "RS256", + "RS384", + "RS512", + "PS256", + "PS384", + "PS512" + ], + "claims_parameter_supported": true, + "claims_supported": [ + "iss", + "sub", + "aud", + "exp", + "iat", + "name", + "family_name", + "given_name", + "email", + "email_verified" + ], + "request_parameter_supported": true, + "request_uri_parameter_supported": false +} +---- + +Refer to the link:https://owncloud.dev/apis/http/authorization[Authorization] section for example usages and details. diff --git a/modules/developer/pages/ocis/index.adoc b/modules/developer/pages/ocis/index.adoc new file mode 100644 index 00000000..5b5c7a95 --- /dev/null +++ b/modules/developer/pages/ocis/index.adoc @@ -0,0 +1,61 @@ += oCIS - ownCloud Infinite Scale +:toc: right +:toclevels: 3 + +image::media/is.png[,width=70%] + +== ownCloud Infinite Scale +Welcome to oCIS, the modern file-sync and share platform, which is based on our knowledge and experience with the PHP based link:https://owncloud.com/#server[ownCloud server]. + +=== The idea of federated storage +To create a truly federated storage architecture oCIS breaks down the old ownCloud 10 user specific namespace, which is assembled on the server side, and makes the individual parts accessible to clients as storage spaces and storage space registries. + +The diagram below shows the core concepts that are the foundation for the new architecture: +- End user devices can fetch the list of _storage spaces_ a user has access to, by querying one or multiple _storage space registries_. The list contains a unique endpoint for every _storage space_. +- xref:./storage/terminology#storage-space-registries.adoc[_Storage space registries_] manage the list of storage spaces a user has access to. They may subscribe to _storage spaces_ in order to receive notifications about changes on behalf of an end users mobile or desktop client. +- xref:./storage/terminology#storage-spaces.adoc[_Storage spaces_] represent a collection of files and folders. A users personal files are contained in a _storage space_, a group or project drive is a _storage space_, and even incoming shares are treated and implemented as _storage spaces_. Each with properties like owners, permissions, quota and type. +- xref:./storage/terminology#storage-providers.adoc[_Storage providers_] can hold multiple _storage spaces_. At an oCIS instance, there might be a dedicated _storage provider_ responsible for users personal storage spaces. There might be multiple, either to shard the load, provide different levels of redundancy or support custom workflows. Or there might be just one, hosting all types of _storage spaces_. + +image::ocis/idea.drawio.svg[] + +As an example, Einstein might want to share something with Marie, who has an account at a different identity provider and uses a different storage space registry. The process makes use of link:https://openid.net/specs/openid-connect-core-1_0.html[OpenID Connect (OIDC)] for authentication and would look something like this: + +To share something with Marie, Einstein would open `https://cloud.zurich.test`. His browser loads oCIS web and presents a login form that uses the link:https://openid.net/specs/openid-connect-discovery-1_0.html#EmailSyntax[OpenID Connect Discovery] to look up the OIDC issuer. For `einstein@zurich.test` he will end up at `https://idp.zurich.test`, authenticate and get redirected back to `https://cloud.zurich.test`. Now, oCIS web will use a similar discovery to look up the _storage space registry_ for the account, based on the email (or username). He will discover that `https://cloud.zurich.test` is also his _storage registry_ that the web UI will use to load the list of _storage spaces_ that are available to him. + +After locating a folder that he wants to share with Marie he enters her email `marie@paris.test` in the sharing dialog to grant her the editor role. This, in effect, creates a new _storage space_ that is registered with the _storage space registry_ at `https://cloud.zurich.test`. + +Einstein copies the URL in the browser (or an email with the same URL is sent automatically, or the storage registries use a back-channel mechanism). It contains the most specific `storage space id` and a path relative to it: `https://cloud.zurich.test/#/spaces/716199a6-00c0-4fec-93d2-7e00150b1c84/a/rel/path`. + +When Marie enters that URL she will be presented with a login form on the `https://cloud.zurich.test` instance, because the share was created on that domain. If `https://cloud.zurich.test` trusts her OpenID Connect identity provider `https://idp.paris.test` she can log in. This time, the _storage space registry_ discovery will come up with `https://cloud.paris.test` though. Since that registry is different than the registry tied to `https://cloud.zurich.test` oCIS web can look up the _storage space_ `716199a6-00c0-4fec-93d2-7e00150b1c84` and register the WebDAV URL `https://cloud.zurich.test/dav/spaces/716199a6-00c0-4fec-93d2-7e00150b1c84/a/rel/path` in Marie's _storage space registry_ at `https://cloud.paris.test`. When she accepts that share her clients will be able to sync the new _storage space_ at `https://cloud.zurich.test`. + +=== oCIS microservice runtime + +The oCIS runtime allows us to dynamically manage services running in a single process. We use link:https://github.com/thejerf/suture[suture] to create a supervisor tree that starts each service in a dedicated goroutine. By default oCIS will start all built-in oCIS services in a single process. Individual services can be moved to other nodes to scale-out and meet specific performance requirements. A link:https://github.com/asim/go-micro/blob/master/registry/registry.go[go-micro] based registry allows services in multiple nodes to form a distributed microservice architecture. + +=== oCIS Services + +Every oCIS service uses link:https://github.com/owncloud/ocis/tree/master/ocis-pkg[ocis-pkg], which implements the link:https://go-micro.dev/[go-micro] interfaces for link:https://github.com/asim/go-micro/blob/v3.5.0/server/server.go#L17-L37[servers] to register and link:https://github.com/asim/go-micro/blob/v3.5.0/client/client.go#L11-L23[clients] to lookup nodes with a service link:https://github.com/asim/go-micro/blob/v3.5.0/registry/registry.go[registry]. +We are following the link:https://12factor.net/[12 Factor] methodology with oCIS. The uniformity of services also allows us to use the same command, logging and configuration mechanism. Configurations are forwarded from the +oCIS runtime to the individual extensions. + + +=== go-micro + +While the link:https://go-micro.dev/[go-micro] framework provides abstractions as well as implementations for the different components in a microservice architecture, it uses a more developer focused runtime philosophy: It is used to download services from a repo, compile them on the fly and start them as individual processes. For oCIS we decided to use a more admin friendly runtime: You can download a single binary and start the contained oCIS services with a single `bin/ocis server`. This also makes packaging easier. + +We use link:https://github.com/owncloud/ocis/tree/master/ocis-pkg[ocis-pkg] to configure the default implementations for the go-micro link:https://github.com/asim/go-micro/tree/v3.5.0/plugins/server/grpc[grpc server], link:https://github.com/asim/go-micro/tree/v3.5.0/plugins/client/grpc[client] and link:https://github.com/asim/go-micro/blob/v3.5.0/registry/mdns_registry.go[mdns registry], swapping them out as needed, e.g. to use the link:https://github.com/asim/go-micro/tree/v3.5.0/plugins/registry/kubernetes[kubernetes registry plugin]. + +=== REVA +A lot of embedded services in oCIS are built upon the link:https://reva.link/[REVA] runtime. We decided to bundle some of the link:https://github.com/cs3org/cs3apis[CS3 services] to logically group them. A link:https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/storagehome.go#L93-L108[home storage provider], which is dealing with link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ProviderAPI[metadata], and the corresponding link:https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/storagehome.go#L109-L123[data provider], which is dealing with link:https://cs3org.github.io/cs3apis/#cs3.gateway.v1beta1.FileUploadProtocol[up and download], are one example. The link:https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/frontend.go[frontend] with the link:https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/frontend.go#L132-L138[oc flavoured webdav], link:https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/frontend.go#L139-L148[ocs handlers] and a link:https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/frontend.go#L126-L131[data-gateway] are another. + +=== Protocol driven development +Interacting with oCIS involves a multitude af APIs. The server and all clients rely on link:https://openid.net/connect/[OpenID Connect] for authentication. The link:https://owncloud.dev/services/idp/[embedded LibreGraph Connect] can be replaced with any other OpenID Connect Identity Provider. Clients use the link:http://webdav.org/[WebDAV] based link:https://github.com/cernbox/smashbox/blob/master/protocol/protocol.md[oc sync protocol] to manage files and folders, link:https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html[ocs to manage shares] and link:https://tus.io/protocols/resumable-upload.html[TUS] to upload files in a resumable way. On the server side link:https://reva.link/[REVA] is the reference implementation of the link:https://github.com/cs3org/cs3apis[CS3 apis] which is defined using link:https://developers.google.com/protocol-buffers/[protobuf]. By embedding link:https://github.com/libregraph/idm[libregraph/idm], oCIS provides a link:https://tools.ietf.org/html/rfc2849[LDAP] interface to make accounts, including guests available to firewalls and other systems. In the future, we are looking into link:https://docs.microsoft.com/en-us/graph/api/overview?view=graph-rest-1.0[the Microsoft Graph API], which is based on link:http://docs.oasis-open.org/odata/odata/v4.0/odata-v4.0-part1-protocol.html[odata], as a well-defined REST/JSON dialect for the existing endpoints. + +=== Acceptance test suite +We run a huge link:https://github.com/owncloud/core/tree/master/tests[test suite], which originated in ownCloud 10 and continues to grow. A detailed description can be found in the developer docs for xref:development/testing.adoc[testing]. + +=== Architecture Overview + +Running `bin/ocis server` will start the following services, all of which can be scaled and deployed on a single node or in a cloud native environment, as needed. + +image::ocis/architecture-overview.drawio.svg[] diff --git a/modules/developer/pages/ocis/license.adoc b/modules/developer/pages/ocis/license.adoc new file mode 100644 index 00000000..e7057062 --- /dev/null +++ b/modules/developer/pages/ocis/license.adoc @@ -0,0 +1,9 @@ += License +:toc: right +:toclevels: 3 + +The source code of the project is licensed under the link:https://github.com/owncloud/ocis/blob/master/LICENSE[Apache 2.0] license. For the license of the used libraries you have to check the respective sources. + +Stable, supported binary builds of ownCloud Infinite Scale that will be distributed by ownCloud GmbH are covered by a non OSS Freemium License link:https://owncloud.com/license-owncloud-infinite-scale/[EULA]. This protects additional efforts that ownCloud GmbH is putting into these builds. + +Since the source code of Infinite Scale is available under free licenses, the free usage can, should and will not be limited in general. diff --git a/modules/developer/pages/ocis/metrics.adoc b/modules/developer/pages/ocis/metrics.adoc new file mode 100644 index 00000000..7caef118 --- /dev/null +++ b/modules/developer/pages/ocis/metrics.adoc @@ -0,0 +1,103 @@ += Metrics +:toc: right +:toclevels: 3 + +== Metrics + +This service provides some link:https://prometheus.io/[Prometheus] metrics through the debug endpoint, you can optionally secure the metrics endpoint by some random token, which has to be configured either through the flag `--debug-token` or the environment variable `OCIS_DEBUG_TOKEN` mentioned above. By default, the metrics endpoint is bound to `http://0.0.0.0:8001/metrics`. + +go_gc_duration_seconds +: A summary of the GC invocation durations + +go_gc_duration_seconds_sum +: A summary of the GC invocation durations + +go_gc_duration_seconds_count +: A summary of the GC invocation durations + +go_goroutines +: Number of goroutines that currently exist + +go_info +: Information about the Go environment + +go_memstats_alloc_bytes +: Number of bytes allocated and still in use + +go_memstats_alloc_bytes_total +: Total number of bytes allocated, even if freed + +go_memstats_buck_hash_sys_bytes +: Number of bytes used by the profiling bucket hash table + +go_memstats_frees_total +: Total number of frees + +go_memstats_gc_cpu_fraction +: The fraction of this program's available CPU time used by the GC since the program started + +go_memstats_gc_sys_bytes +: Number of bytes used for garbage collection system metadata + +go_memstats_heap_alloc_bytes +: Number of heap bytes allocated and still in use + +go_memstats_heap_idle_bytes +: Number of heap bytes waiting to be used + +go_memstats_heap_inuse_bytes +: Number of heap bytes that are in use + +go_memstats_heap_objects +: Number of allocated objects + +go_memstats_heap_released_bytes +: Number of heap bytes released to OS + +go_memstats_heap_sys_bytes +: Number of heap bytes obtained from system + +go_memstats_last_gc_time_seconds +: Number of seconds since 1970 of last garbage collection + +go_memstats_lookups_total +: Total number of pointer lookups + +go_memstats_mallocs_total +: Total number of mallocs + +go_memstats_mcache_inuse_bytes +: Number of bytes in use by mcache structures + +go_memstats_mcache_sys_bytes +: Number of bytes used for mcache structures obtained from system + +go_memstats_mspan_inuse_bytes +: Number of bytes in use by mspan structures + +go_memstats_mspan_sys_bytes +: Number of bytes used for mspan structures obtained from system + +go_memstats_next_gc_bytes +: Number of heap bytes when next garbage collection will take place + +go_memstats_other_sys_bytes +: Number of bytes used for other system allocations + +go_memstats_stack_inuse_bytes +: Number of bytes in use by the stack allocator + +go_memstats_stack_sys_bytes +: Number of bytes obtained from system for stack allocator + +go_memstats_sys_bytes +: Number of bytes obtained from system + +go_threads +: Number of OS threads created + +promhttp_metric_handler_requests_in_flight +: Current number of scrapes being served + +promhttp_metric_handler_requests_total +: Total number of scrapes by HTTP status code diff --git a/modules/developer/pages/ocis/migration.adoc b/modules/developer/pages/ocis/migration.adoc new file mode 100644 index 00000000..d6e36d14 --- /dev/null +++ b/modules/developer/pages/ocis/migration.adoc @@ -0,0 +1,773 @@ += Migration +:toc: right +:toclevels: 3 + +The migration happens in subsequent stages while the service is online. First all users need to migrate to the new architecture, then the global namespace needs to be introduced. Finally, the data on disk can be migrated user by user by switching the storage driver. + +
+ +[WARNING] +==== +@jfd: It might be easier to introduce the spaces api in oc10 and then migrate to oCIS. We cannot migrate both at the same time, the architecture to oCIS (which will change fileids) and introduce a global namespace (which requires stable fileids to let clients handle moves without redownloading). Either we implement arbitrary mounting of shares in oCIS / reva or we make clients and oc10 spaces aware. +==== + + +
+ +== Migration Stages + +=== Stage 0: pre migration +Is the pre-migration stage when having a functional ownCloud 10 instance. + +
+ +==== FAQ +_Feel free to add your question as a PR to this document using the link at the top of this page!_ + +
+ +
+ +=== Stage 1: introduce ownCloud Web +Install and introduce link:https://github.com/owncloud/web/[ownCloud Web] and let users test it voluntarily to gain early feedback on the new UI. + +==== Steps +Deploy web and enable switching to and from it. +For more details see: link:https://owncloud.dev/clients/web/deployments/oc10-app/[ownCloud 10 with ownCloud Web] + +
+ +_TODO allow limiting the web ui switch to an 'early adopters' group_ + +
+ +==== Validation +Ensure switching back and forth between the classic ownCloud 10 web UI and ownCloud web works as at our https://demo.owncloud.com. + +==== Rollback +Should there be problems with ownCloud web at this point it can simply be removed from the menu and be undeployed. + +==== Notes +
+The ownCloud 10 demo instance uses OAuth to obtain a token for ownCloud web and currently always requires explicit consent. In oCIS the token is provided by the OpenID Connect Identity Provider, which may skip the consent step for trusted clients for a more seamless login experience. You may want to introduce OpenID Connect before enabling the new web UI. + +
+ +_TODO make oauth2 in oc10 trust the new web ui, based on `redirect_uri` and CSRF so no explicit consent is needed?_ + +==== FAQ +_Feel free to add your question as a PR to this document using the link at the top of this page!_ + +
+ +
+ +=== Stage 2: introduce OpenID Connect + +Basic auth requires us to properly store and manage user credentials. Something we would rather like to delegate to a tool specifically built for that task. +While SAML and Shibboleth are protocols that solve that problem, they are limited to web clients. Desktop and mobile clients were an afterthought and keep running into timeouts. For these reasons, we decided to move to link:https://owncloud.com/news/openid-connect-oidc-app/[OpenID Connect as our primary authentication protocol]. + +
+ +_TODO @butonic add ADR for OpenID Connect and flesh out pros and cons of the above_ + +
+ +==== User impact + +When introducing OpenID Connect, the clients will detect the new authentication scheme when their current way of authenticating returns an error. Users will then have to +reauthorize at the OpenID Connect IdP, which again, may be configured to skip the consent step for trusted clients. + +==== Steps +1. There are multiple products that can be used as an OpenID Connect IdP. We test with link:https://github.com/libregraph/lico[LibreGraph Connect], which is also link:https://github.com/owncloud/web/[embedded in oCIS]. Other alternatives include link:https://www.keycloak.org/[Keycloak] or link:https://www.pingidentity.com/[Ping]. Please refer to the corresponding setup instructions for the product you intend to use. + +
+ +_TODO @butonic flesh out oCIS IDP documentation_ + +
+ +2. Add link:https://doc.owncloud.com/server/admin_manual/configuration/user/oidc/[Openid Connect (OIDC)] support to ownCloud 10. + +==== Validation +When OpenID Connect support is enabled verify that all clients can login: +- web classic +- ownCloud web +- desktop +- android +- iOS + +==== Rollback +Should there be problems with OpenID Connect at this point you can disable the app. Users will have to reauthenticate in this case. + +==== Notes +
+Legacy clients relying on Basic auth or app passwords need to be migrated to OpenId Connect to work with oCIS. For a transition period Basic auth in oCIS can be enabled with `PROXY_ENABLE_BASIC_AUTH=true`, but we strongly recommend adopting OpenID Connect for other tools as well. + +While OpenID Connect providers will send an `iss` and `sub` claim that relying parties (services like oCIS or ownCloud 10) can use to identify users we recommend introducing a dedicated, globally unique, persistent, non-reassignable user identifier like a UUID for every user. This `ownclouduuid` should be sent as an additional claim to save additional lookups on the server side. It will become the user id in oCIS, e.g. when searching for recipients the `ownclouduuid` will be used to persist permissions with the share manager. It has a different purpose than the ownCloud 10 username, which is used to login. Using UUIDs we can not only mitigate username collisions when merging multiple instances but also allow renaming usernames after the migration to oCIS has been completed. + +
+ +==== FAQ +_Feel free to add your question as a PR to this document using the link at the top of this page!_ + +
+ +
+ +=== Stage 3: introduce oCIS internally + +Before letting oCIS handle end user requests we will first make it available in the internal network. By subsequently adding services we can add functionality and verify the services work as intended. + +Start oCIS backend and make read only tests on existing data using the `owncloudsql` storage driver which will read (and write) +- blobs from the same datadirectory layout as in ownCloud 10 +- metadata from the ownCloud 10 database: +The oCIS share manager will read share information from the ownCloud database using an `owncloud` driver as well. + +
+ +_TODO @butonic add guide on how to configure `owncloudsql`_ + +_TODO we need a share manager that can read from the ownCloud 10 database as well as from whatever new backend will be used for a pure oCIS setup. Currently, that would be the json file. Or that is migrated after all users have switched to oCIS. -- jfd_ + +
+ +==== User impact +None, only administrators will be able to explore oCIS during this stage. + +==== Steps and verifications + +We are going to run and explore a series of services that will together handle the same requests as ownCloud 10. For initial exploration the oCIS binary is recommended. The services can later be deployed using a single oCIS runtime or in multiple containers. + + +===== Storage provider for file metadata +1. Deploy OCIS storage provider with owncloudsql driver. +2. Set `read_only: true` in the storage provider config.
_TODO @butonic add read only flag to storage drivers_
+3. Use cli tool to list files using the CS3 api + +===== File ID alternatives +Multiple ownCloud instances can be merged into one oCIS instance. To prevent the numeric ids from colliding, the file ids will be prefixed with a new storage space id which is used by oCIS to route requests to the correct storage provider. See Stage 8 below. + +
+ +[WARNING] +==== +_Alternative 1_ +Add a routable prefix to fileids in oc10, and replicate the prefix in oCIS. +=== Stage-3.1 +Let oc10 render file ids with prefixes: `$!`. This will allow clients to handle moved files. + +=== Stage-3.2 +Roll out new clients that understand the spaces API and know how to convert local sync pairs for legacy oc10 `/webdav` or `/dav/files/` home folders into multiple sync pairs. +One pair for `/webdav/home` or `/dav/files//home` and another pair for every accepted share. The shares will be accessible at `/webdav/shares/` when the server side enables the spaces API. +Files can be identified using `$!` and moved to the correct sync pair. + +=== Stage-3.3 +Enable spaces API in oc10: +- New clients will get a response from the spaces API and can set up new sync pairs. +- Legacy clients will still poll `/webdav` or `/dav/files/` where they will see new subfolders instead of the users home. They will move down the users files into `/home` and shares into `/shares`. Custom sync pairs will no longer be available, causing the legacy client to leave local files in place. They can be picked up manually when installing a new client. + +==== + + +[WARNING] +==== +_Alternative 2_ +An additional `uuid` property used only to detect moves. A lookup by uuid is not necessary for this. The `/dav/meta` endpoint would still take the fileid. Clients would use the `uuid` to detect moves and set up new sync pairs when migrating to a global namespace. +=== Stage-3.1 +Generate a `uuid` for every file as a file property. Clients can submit a `uuid` when creating files. The server will create a `uuid` if the client did not provide one. + +=== Stage-3.2 +Roll out new clients that understand the spaces API and know how to convert local sync pairs for legacy oc10 `/webdav` or `/dav/files/` home folders into multiple sync pairs. +One pair for `/webdav/home` or `/dav/files//home` and another pair for every accepted share. The shares will be accessible at `/webdav/shares/` when the server side enables the spaces API. Files can be identified using the `uuid` and moved to the correct sync pair. + +=== Stage-4.1 +When reading the files from oCIS return the same `uuid`. It can be migrated to an extended attribute or it can be read from oc10. If users change it the client will not be able to detect a move and maybe other weird stuff happens. _What if the uuid gets lost on the server side due to a partial restore?_ + +==== + +
+ + +
+ +===== graph API endpoint +1. Deploy graph api to list spaces +2. Use curl to list spaces using graph drives endpoint + +===== owncloud flavoured WebDAV endpoint +1. Deploy ocdav +2. Use curl to send PROPFIND + +===== data provider for up and download +1. Deploy dataprovider +2. Use curl to up and download files +3. Use tus to upload files + +Deploy ... + +===== share manager +Deploy share manager with ownCloud driver + +===== reva gateway +1. Deploy gateway to authenticate requests? I guess we need that first... Or we need the to mint a token. Might be a good exercise. + +===== automated deployment +Finally, deploy oCIS with a config to set up everything running in a single oCIS runtime or in multiple containers. + +==== Rollback +You can stop the oCIS process at any time. + +==== Notes +
+Multiple ownCloud instances can be merged into one oCIS instance. The file ids will be prefixed with a new storage space id which is used to route requests to the correct storage provider. + +
+ +==== FAQ +_Feel free to add your question as a PR to this document using the link at the top of this page!_ + +
+ +
+ +=== Stage 4: internal write access with oCIS +Test writing data with oCIS into the existing ownCloud 10 data directory using the `owncloudsql` storage driver. + +==== User impact +Only administrators will be able to explore oCIS during this stage. End users should not be affected if the testing is limited to test users. + +==== Steps +Set `read_only: false` in the storage provider config. + +
+ +_TODO @butonic add read only flag to storage drivers_ + +
+ +==== Verification +==== Rollback +Set `read_only: true` in the storage provider config. + +
+ +_TODO @butonic add read only flag to storage drivers_ + +
+ +==== Notes +
+With write access it becomes possible to manipulate existing files and shares. + +
+ +==== FAQ +_Feel free to add your question as a PR to this document using the link at the top of this page!_ + +
+ +
+ +=== Stage-5: introduce user aware proxy +In the previous stages oCIS was only accessible for administrators with access to the network. To expose only a single service to the internet, oCIS comes with a user aware proxy that can be used to route requests to the existing ownCloud 10 installation or oCIS, based on the authenticated user. The proxy uses OIDC to identify the logged-in user and route them to the configured backend. + +==== User impact +The IP address of the ownCloud host changes. There is no change for the file sync and share functionality when requests are handled by the oCIS codebase as it uses the same database and storage system as owncloud 10. + +==== Steps and verifications + +===== Deploy oCIS proxy +1. Deploy the `ocis proxy` +2. Verify the requests are routed based on the ownCloud 10 routing policy `oc10` by default + +===== Test user based routing +1. Change the routing policy for a user or an early adopters group to `ocis`
_TODO @butonic currently, the migration selector will use the `ocis` policy for users that have been added to the accounts service. IMO we need to evaluate a claim from the IdP._
+2. Verify the requests are routed based on the oCIS routing policy `oc10` for 'migrated' users. + +At this point you are ready to rock & roll! + +===== Let ownCloud domain point to proxy +1. Update the dns to use the oCIS proxy instead of the ownCloud application servers directly. +2. Let DNS propagate the change and monitor requests moving from the ownCloud application servers to the oCIS proxy. +3. Verify the DNS change has propagated sufficiently. All requests should now use the oCIS Proxy. + +==== Rollback +Should there be a problem with the oCIS routes the user can be routed to ownCloud by changing his routing policy. In case of unfixable problems with the proxy the DNS needs to be updated to use the ownCloud 10 application servers directly. This could also be done in a load balancer. + +==== Notes +
+The proxy is stateless, multiple instances can be deployed as needed. + +
+ +==== FAQ +_Feel free to add your question as a PR to this document using the link at the top of this page!_ + +
+ +
+ +=== Stage-6: parallel deployment +Running ownCloud 10 and oCIS in parallel is a crucial stage for the migration: it allows users access to group shares regardless of the system that is being used to access the data. A user by user migration with multiple domains would technically break group shares when users vanish because they (and their data) are no longer available in the old system. + +Depending on the amount of power users on an instance, the admin may want to allow users to voluntarily migrate to the oCIS backend. A monitoring system can be used to visualize the behavior for the two systems and gain trust in the overall stability and performance. + +==== User impact +Since the underling data is still stored in the same systems, a similar or performance can be expected. +
+ +See _TODO hmpf outdated didn't we want to run them nightly? ..._ +_TODO @butonic update performance comparisons nightly_ + +
+ +==== Steps +There are several options to move users to the oCIS backend: +- Use a canary app to let users decide themselves +- Use an early adopters group with an opt-in +- Force migrate users in batch or one by one at the administrators will + +==== Verification +The same verification steps as for the internal testing stage apply. Just from the outside. + +==== Rollback +Until now, the oCIS configuration mimics ownCloud 10 and uses the old data directory layout and the ownCloud 10 database. Users can seamlessly be switched from ownCloud 10 to oCIS and back again. +
+ +_TODO @butonic we need a canary app that allows users to decide for themselves which backend to use_ + +
+ +
+ +==== Notes +Running the two systems in parallel stage +Try to keep the duration of this stage short. Until now we only added services and made the system more complex. oCIS aims to reduce the maintenance cost of an ownCloud instance. You will not get there if you keep both systems alive. + +
+ +==== FAQ +_Feel free to add your question as a PR to this document using the link at the top of this page!_ + +
+ +
+ +=== Stage-7: introduce spaces using ocis +To encourage users to switch you can promote the workspaces feature that is built into oCIS. The ownCloud 10 storage backend can be used for existing users. New users and group or project spaces can be provided by storage providers that better suit the underlying storage system. + +==== Steps +First, the admin needs to +- deploy a storage provider with the storage driver that best fits the underlying storage system and requirements. +- register the storage in the storage registry with a new storage id (we recommend a uuid). + +Then a user with the necessary create storage space role can create a storage space and assign Managers. + +
+ +_TODO @butonic a user with management permission needs to be presented with a list of storage spaces where he can see the amount of free space and decide on which storage provider the storage space should be created. For now a config option for the default storage provider for a specific type might be good enough._ + +
+ +==== Verification +The new storage space should show up in the `/graph/drives` endpoint for the managers and the creator of the space. + +==== Notes +Depending on the requirements and acceptable tradeoffs, a database less deployment using the ocis or s3ng storage driver is possible. There is also a link:https://github.com/cs3org/reva/pull/1209[cephfs driver] on the way, that directly works on the API level instead of POSIX. + +=== Stage-8: shut down ownCloud 10 +Disable ownCloud 10 in the proxy, all requests are now handled by oCIS, shut down oc10 web servers and redis (or keep for calendar & contacts only? rip out files from oCIS?) + +==== User impact +All users are already sent to the oCIS backend. Shutting down ownCloud 10 will remove the old web UI, apps and functionality that is not yet present in ownCloud web. For example contacts and calendar. + +
+ +_TODO @butonic recommend alternatives_ + +
+ +==== Steps +1. Shut down the apache servers that are running the ownCloud 10 PHP code. +2. DO NOT SHUT DOWN THE DATABASE, YET! + +==== Verification +The ownCloud 10 classic web UI should no longer be reachable. + +==== Rollback +Redeploy ownCloud 10. + +==== Notes +
+The database needs to remain online until the storage layer and share metadata have been migrated as well. One thing at a time. + +
+ +==== FAQ +_Feel free to add your question as a PR to this document using the link at the top of this page!_ + +
+ +
+ +=== Stage 9: storage migration +To get rid of the database we will move the metadata from the old ownCloud 10 database into dedicated storage providers. This can happen in a user by user fashion. group drives can properly be migrated to group, project or workspaces in this stage. + +==== User impact +Noticeable performance improvements because we effectively shard the storage logic and persistence layer. + +==== Steps +1. User by user storage migration from `owncloud` or `ownclouds3` driver to `ocis`/`s3ng`/`cephfs`... currently this means copying the metadata from one storage provider to another using the cs3 api. +2. Change the responsible storage provider for a storage space (e.g. a user home, a group or project space are a workspace) in the storage registry. + +
+ +_TODO @butonic implement `ownclouds3` based on `s3ng`_ +_TODO @butonic implement tiered storage provider for seamless migration_ +_TODO @butonic document how to manually do that until the storage registry can discover that on its own._ + +
+ +==== Verification +Start with a test user, then move to early adopters and finally migrate all users. + +==== Rollback +To switch the storage provider again the same storage space migration can be performed again: copy metadata and blob data using the CS3 api, then change the responsible storage provider in the storage registry. + +==== Notes +
+Multiple ownCloud instances can be merged into one oCIS instance. The file ids will be prefixed with a new storage space id which is used to route requests to the correct storage provider. + +The storage space migration will become a seamless feature in the future that allows administrators to move users to storage systems with different capabilities, to implement premium features, deprovisioning strategies or archiving. + +
+ +==== FAQ +_Feel free to add your question as a PR to this document using the link at the top of this page!_ + +
+ +
+ +=== Stage-10: share metadata migration +Migrate share data to _yet to determine_ share manager backend and shut down ownCloud database. + +The ownCloud 10 database still holds share information in the `oc_share` and `oc_share_external` tables. They are used to efficiently answer queries about who shared what with whom. In oCIS shares are persisted using a share manager and if desired these grants are also sent to the storage provider so it can set ACLs if possible. Only one system should be responsible for the shares, which in case of treating the storage as the primary source effectively turns the share manager into a cache. + +==== User impact +Depending on chosen the share manager provider some sharing requests should be faster: listing incoming and outgoing shares is no longer bound to the ownCloud 10 database but to whatever technology is used by the share provider: + - For non HA scenarios they can be served from memory, backed by a simple json file. + - TODO: implement share manager with redis / nats / ... key value store backend: use the micro store interface please ... + +==== Steps +1. Start new share manager +2. Migrate metadata using the CS3 API (copy from old to new) +3. Shut down old share manager +4. Shut down ownCloud 10 database + +
+ +_TODO for HA implement share manager with redis / nats / ... key value store backend: use the micro store interface please ..._ +_TODO for batch migration implement share data migration cli with progress that reads all shares via the cs3 api from one provider and writes them into another provider_ +_TODO for seamless migration implement tiered/chained share provider that reads share data from the old provider and writes new shares to the new one_ +_TODO for storage provider as source of truth persist ALL share data in the storage provider. Currently, part is stored in the share manager, part is in the storage provider. We can keep both, but the share manager should directly persist its metadata to the storage system used by the storage provider so metadata is kept in sync_ + +
+ +==== Verification +After copying all metadata start a dedicated gateway and change the configuration to use the new share manager. Route a test user, a test group and early adopters to the new gateway. When no problems occur you can start the desired number of share managers and roll out the change to all gateways. + +
+ +_TODO let the gateway write updates to multiple share managers ... or rely on the tiered/chained share manager provider to persist to both providers_ + +
+ +==== Rollback +To switch the share manager to the database one revert routing users to the new share manager. If you already shut down the old share manager start it again. Use the tiered/chained share manager provider in reverse configuration (new share provider as read only, old as write) and migrate the shares again. You can also restore a database backup if needed. + +
+ +=== Stage-11 +Profit! Well, on the one hand you do not need to maintain a clustered database setup and can rely on the storage system. On the other hand you are now in microservice wonderland and will have to relearn how to identify bottlenecks and scale oCIS accordingly. The good thing is that tools like jaeger and prometheus have evolved and will help you understand what is going on. But this is a different topic. See you on the other side! + +==== FAQ +_Feel free to add your question as a PR to this document using the link at the top of this page!_ + +
+ +
+ +== Architectural differences + +The fundamental difference between ownCloud 10 and oCIS is that the file metadata is moved from the database in the `oc_filecache` table (which is misnamed, as it actually is an index) to the storage provider who can place metadata as close to the underlying storage system as possible. In effect, the file metadata is sharded over multiple specialized services. + + +== Data that will be migrated + +Currently, oCIS focuses on file sync and share use cases. + +=== Blob data + +In ownCloud 10 the files are laid out on disk in the _data directory_ using the following layout: +---- +data +├── einstein +│ ├── cache +│ ├── files +│ │ ├── Photos +│ │ │ └── Portugal.jpg +│ │ ├── Projects +│ │ │ └── Notes.md +│ │ └── ownCloud Manual.pdf +│ ├── files_external +│ ├── files_trashbin +│ │ ├── files +│ │ │ ├── Documents.d1564687985 +│ │ │ ├── TODO.txt.d1565721976 +│ │ │ └── welcome.txt.d1564775872 +│ │ └── versions +│ │ │ ├── TODO.txt.v1564605543.d1565721976 +│ │ │ └── TODO.txt.v1564775936.d1565721976 +│ ├── files_versions +│ │ ├── Projects +│ │ │ ├── Notes.md.v1496912691 +│ │ │ └── Notes.md.v1540305560 +│ │ └── ownCloud Manual.pdf.v1396628249 +│ ├── thumbnails +│ │ └── 123 +│ │ │ ├── 2048-1536-max.png +│ │ │ └── 32-32.png // the file id, e.g. of /Photos/Portugal.jpg +│ └── uploads +├── marie +│ ├── cache +│ ├── files +│ ├── files_external +│ ├── files_trashbin +│ ├── files_versions +│ └── thumbnails +│ … +├── moss +… +---- + +The _data directory_ may also contain subfolders for ownCloud 10 applications like `avatars`, `gallery`, `files_external` and `cache`. + +When an object storage is used as the primary storage all file blobs are stored by their file id and a prefix, e.g.: `urn:oid:`. + +The three types of blobs we need to migrate are stored in +- `files` for file blobs, the current file content, +- `files_trashbin` for trashed files (and their versions) and +- `files_versions` for file blobs of older versions. + +
+ +=== Filecache table + +In both cases the file metadata, including a full replication of the file tree, is stored in the `oc_filecache` table of an ownCloud 10 database. The primary key of a row is the file id. It is used to attach additional metadata like shares, tags, favorites or arbitrary file properties. + +The `filecache` table itself has more metadata: + +| Field | Type | Null | Key | Default | Extra | Comment | Migration | +|--------------------|---------------|------|-----|---------|----------------|----------------|----------------| +| `fileid` | bigint(20) | NO | PRI | NULL | auto_increment | | MUST become the oCIS `opaqueid` of a file reference. `ocis` driver stores it in extended attributes and can use numbers as node ids on disk. for eos see note below table | +| `storage` | int(11) | NO | MUL | 0 | | _the filecache holds metadata for multiple storages_ | corresponds to an oCIS _storage space_ | +| `path` | varchar(4000) | YES | | NULL | | _the path relative to the storages root_ | MUST become the `path` relative to the storage root. `files` prefix needs to be trimmed. | +| `path_hash` | varchar(32) | NO | | | | *mysql once had problems indexing long paths, so we stored a hash for lookup by path. | - | +| `parent` | bigint(20) | NO | MUL | 0 | | _used to implement the hierarchy and listing children of a folder by id. redundant with `path`_ | - | +| `name` | varchar(250) | YES | | NULL | | _basename of `path`_ | - | +| `mimetype` | int(11) | NO | | 0 | | _joined with the `oc_mimetypes` table. only relevant for object storage deployments_ | can be determined from blob / file extension | +| `mimepart` | int(11) | NO | | 0 | | _"_ | can be determined from blob / file extension | +| `size` | bigint(20) | NO | | 0 | | _same as blob size unless encryption is used_ | MAY become size, can be determined from blob | +| `mtime` | bigint(20) | NO | | 0 | | _same as blob mtime_ | for files MAY become mtime (can be determined from blob as well), for directories MUST become tmtime | +| `encrypted` | int(11) | NO | | 0 | | _encrypted flag_ | oCIS currently does not support encryption | +| `etag` | varchar(40) | YES | | NULL | | _used to propagate changes in a tree_ | MUST be migrated (or calculated in the same way) to prevent clients from syncing unnecessarily | +| `unencrypted_size` | bigint(20) | NO | | 0 | | _same as blob size_ | oCIS currently does not support encryption | +| `storage_mtime` | bigint(20) | NO | | 0 | | _used to detect external storage changes_ | oCIS delegates that to the storage providers and drivers | +| `permissions` | int(11) | YES | | 0 | | *used as the basis for permissions. synced from disk when running a file scan. * | oCIS delegates that to the storage providers and drivers | +| `checksum` | varchar(255) | YES | | NULL | | _same as blob checksum_ | SHOULD become the checksum in the storage provider. eos calculates it itself, `ocis` driver stores it in extended attributes | + + +[quote] +____ +Note: for EOS a hot migration only works seamlessly if file ids in oc10 are already read from eos. Otherwise, either a mapping from the oc10 filecache file id to the new eos file id has to be created under the assumption that these id sets do not intersect or files and corresponding shares need to be exported and imported offline to generate a new set of ids. While this will preserve public links, user, group and even federated shares, old internal links may still point to different files because they contain the oc10 fileid +____ + + +
+ +=== share table + +used to store +- Public links +- Private shares with users and groups +- Federated shares _partly_ +- Guest shares + +| Field | Type | Null | Key | Default | Extra | Comment | link:https://cs3org.github.io/cs3apis/[CS3 API] | +|---------------|--------------|------|-----|---------|----------------|---------|-| +| `id` | int(11) | NO | PRI | NULL | auto_increment | | `ShareId.opaqueid` string | +| `share_type` | smallint(6) | NO | | 0 | | _in CS3 every type is handled by a dedicated API. See below the table_ | does NOT map to link:https://cs3org.github.io/cs3apis/#cs3.sharing.ocm.v1beta1.Share.ShareType[`Share.ShareType`] _TODO clarify_ | +| `share_with` | varchar(255) | YES | MUL | NULL | | | `Share.grantee` link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Grantee[`Grantee`] | +| `uid_owner` | varchar(64) | NO | | | | | `ShareId.owner` link:https://cs3org.github.io/cs3apis/#cs3.identity.user.v1beta1.UserId[`UserID`] | +| `parent` | int(11) | YES | | NULL | | | - | +| `item_type` | varchar(64) | NO | MUL | | | | `Share.resource_id` link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId[`ResourceId`] | +| `item_source` | varchar(255) | YES | MUL | NULL | | | `Share.resource_id` link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId[`ResourceId`] | +| `item_target` | varchar(255) | YES | | NULL | | | `Share.resource_id` link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId[`ResourceId`] | +| `file_source` | bigint(20) | YES | MUL | NULL | | _cannot store uuid style file ids from oCIS. when all users have migrated to oCIS the share manager needs to be updated / migrated to a version that does._ | `Share.resource_id` link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId[`ResourceId`] | +| `file_target` | varchar(512) | YES | | NULL | | | `Share.resource_id` link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId[`ResourceId`] | +| `permissions` | smallint(6) | NO | | 0 | | | `Share.Permissions` link:https://cs3org.github.io/cs3apis/#cs3.sharing.ocm.v1beta1.SharePermissions[`SharePermissions`] | +| `stime` | bigint(20) | NO | | 0 | | | `Share.ctime`, `Share.mtime` | +| `accepted` | smallint(6) | NO | | 0 | | | `ReceivedShare.ShareState` link:https://cs3org.github.io/cs3apis/#cs3.sharing.collaboration.v1beta1.ShareState[`ShareState`] | +| `expiration` | datetime | YES | | NULL | | _only used for the Link API and storage provider api, currently cannot be added using the Collaboration or OCM API_ | link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Grant[`Grant`] | +| `token` | varchar(32) | YES | MUL | NULL | | | link:https://cs3org.github.io/cs3apis/#cs3.sharing.link.v1beta1.PublicShare[`PublicShare.token`] | +| `mail_send` | smallint(6) | NO | | 0 | | | - | +| `uid_initiator` | varchar(64) | YES | | NULL | | | `ShareId.creator` link:https://cs3org.github.io/cs3apis/#cs3.identity.user.v1beta1.UserId[`UserID`] | +| `share_name` | varchar(64) | YES | | NULL | | _only exists for public shares_ | link:https://cs3org.github.io/cs3apis/#cs3.sharing.link.v1beta1.PublicShare[`PublicShare.display_name`] | +| `attributes` | longtext | YES | | NULL | | _additional share attributes_ | _could be implemented using opaque data, but should be added to the CS3 api_ | + +In the CS3 API +1. public links are handled by the PublicShareProvider using the link:https://cs3org.github.io/cs3apis/#cs3.sharing.link.v1beta1.LinkAPI[Link API] +2. internal shares are handled by the UserShareProvider using the link:https://cs3org.github.io/cs3apis/#cs3.sharing.collaboration.v1beta1.CollaborationAPI[Collaboration API]. This covers user and group shares. +3. federated shares are handled by the OcmShareProvider using the link:https://cs3org.github.io/cs3apis/#cs3.sharing.ocm.v1beta1.OcmAPI[OCM Share Provider AP] aka. Open Cloud Mesh. + + +
+ +=== share_external + +Used to store additional metadata for federated shares. + +| Field | Type | Null | Key | Default | Extra | Comment | +|-----------------|---------------|------|-----|---------|----------------|---------| +| `id` | bigint(20) | NO | PRI | NULL | auto_increment | | +| `remote` | varchar(512) | NO | | NULL | | Url of the remote owncloud instance | +| `share_token` | varchar(64) | NO | | NULL | | Public share token | +| `password` | varchar(64) | YES | | NULL | | Optional password for the public share | +| `name` | varchar(64) | NO | | NULL | | Original name on the remote server | +| `owner` | varchar(64) | NO | | NULL | | User that owns the public share on the remote server | +| `user` | varchar(64) | NO | MUL | NULL | | Local user which added the external share | +| `mountpoint` | varchar(4000) | NO | | NULL | | Full path where the share is mounted | +| `mountpoint_hash` | varchar(32) | NO | | NULL | | md5 hash of the mountpoint | +| `remote_id` | varchar(255) | NO | | -1 | | | +| `accepted` | int(11) | NO | | 0 | | | + +
+ +_TODO document how the reva OCM service currently persists the data_ + +
+ +
+ +=== trusted_servers + +used to determine if federated shares can automatically be accepted + +| Field | Type | Null | Key | Default | Extra | Comment | +|---------------|--------------|------|-----|---------|----------------|---------| +| `id` | int(11) | NO | PRI | NULL | auto_increment | | +| `url` | varchar(512) | NO | | NULL | | Url of trusted server | +| `url_hash` | varchar(255) | NO | UNI | | | sha1 hash of the url without the protocol | +| `token` | varchar(128) | YES | | NULL | | token used to exchange the shared secret | +| `shared_secret` | varchar(256) | YES | | NULL | | shared secret used to authenticate | +| `status` | int(11) | NO | | 2 | | current status of the connection | +| `sync_token` | varchar(512) | YES | | NULL | | cardDav sync token | + +
+ +_TODO clarify how OCM handles this and where we store / configure this. It seems related to trusted IdPs_ + +
+ +
+ +=== user data + +Users are migrated in two steps: +1. They should all be authenticated using OpenID Connect, which already moves them to a common identity management system. +2. To search share recipients, both, ownCloud 10 and oCIS need access to the same user directory using e.g. LDAP. + +
+ +_TODO add state to CS3 API, so we can 'disable' users_ +_TODO how do we map (sub) admins? -> map to roles & permissions_ + +
+ +accounts: + +| Field | Type | Null | Key | Default | Extra | Comment | +|---------------|---------------------|------|-----|---------|----------------|---------| +| `id` | bigint(20) unsigned | NO | PRI | NULL | auto_increment | | +| `email` | varchar(255) | YES | MUL | NULL | | | +| `user_id` | varchar(255) | NO | UNI | NULL | | | +| `lower_user_id` | varchar(255) | NO | UNI | NULL | | | +| `display_name` | varchar(255) | YES | MUL | NULL | | | +| `quota` | varchar(32) | YES | | NULL | | | +| `last_login` | int(11) | NO | | 0 | | | +| `backend` | varchar(64) | NO | | NULL | | | +| `home` | varchar(1024) | NO | | NULL | | | +| `state` | smallint(6) | NO | | 0 | | | + +users: + +| Field | Type | Null | Key | Default | Extra | Comment | +|-------------|--------------|------|-----|---------|-------|---------| +| `uid` | varchar(64) | NO | PRI | | | +| `password` | varchar(255) | NO | | | | +| `displayname` | varchar(64) | YES | | NULL | | + +groups: + +The groups table really only contains the group name. + +| Field | Type | Null | Key | Default | Extra | +|-------|-------------|------|-----|---------|-------| +| `gid` | varchar(64) | NO | PRI | | | + +
+ +=== LDAP + +
+ +_TODO clarify if metadata from ldap & user_shibboleth needs to be migrated_ + +
+ +The `dn` -> _owncloud internal username_ mapping that currently lives in the `oc_ldap_user_mapping` table needs to move into a dedicated `ownclouduuid` attribute in the LDAP server. The idp should send it as a claim so the proxy does not have to look up the user using LDAP again. The username cannot be changed in ownCloud 10 and the oCIS provisioning API will not allow changing it as well. When we introduce the graph api we may allow changing usernames when all clients have moved to that api. + +The problem is that the username in owncloud 10 and in oCIS also need to be the same, which might not be the case when the ldap mapping used a different column. In that case we should add another owncloudusername attribute to the ldap server. + + +
+ +=== activities + +_dedicated service, not yet implemented, requires decisions about an event system -- jfd_ + +| Field | Type | Null | Key | Default | Extra | Comment | +|---------------|---------------|------|-----|---------|----------------|---------| +| `activity_id` | bigint(20) | NO | PRI | NULL | auto_increment | +| `timestamp` | int(11) | NO | MUL | 0 | | +| `priority` | int(11) | NO | | 0 | | +| `type` | varchar(255) | YES | | NULL | | +| `user` | varchar(64) | YES | | NULL | | +| `affecteduser` | varchar(64) | NO | MUL | NULL | | +| `app` | varchar(255) | NO | | NULL | | +| `subject` | varchar(255) | NO | | NULL | | +| `subjectparams` | longtext | NO | | NULL | | +| `message` | varchar(255) | YES | | NULL | | +| `messageparams` | longtext | YES | | NULL | | +| `file` | varchar(4000) | YES | | NULL | | +| `link` | varchar(4000) | YES | | NULL | | +| `object_type` | varchar(255) | YES | MUL | NULL | | +| `object_id` | bigint(20) | NO | | 0 | | + +== Links + +The link:https://github.com/owncloud/data_exporter[data_exporter] has logic that allows exporting and importing users, including shares. The link:https://github.com/owncloud/data_exporter/tree/master/lib/Model[model classes] contain the exact mapping. + +
diff --git a/modules/developer/pages/ocis/release_notes.adoc b/modules/developer/pages/ocis/release_notes.adoc new file mode 100644 index 00000000..d7eb514a --- /dev/null +++ b/modules/developer/pages/ocis/release_notes.adoc @@ -0,0 +1,5 @@ += Release Notes +:toc: right +:toclevels: 3 + +The Infinite Scale Release Notes have been moved and are published with the link:https://doc.owncloud.com/docs/next/ocis_release_notes.html[ownCloud Documentation]. diff --git a/modules/developer/pages/ocis/release_roadmap.adoc b/modules/developer/pages/ocis/release_roadmap.adoc new file mode 100644 index 00000000..be5a6499 --- /dev/null +++ b/modules/developer/pages/ocis/release_roadmap.adoc @@ -0,0 +1,136 @@ += Release Life Cycle +:toc: right +:toclevels: 3 + + +This page is designed to provide clarity and transparency regarding the scheduling, nature, and support of our releases, ensuring you have all the information you need to stay updated and manage your deployments effectively. Find here an overview of our plans for future enhancements and features: link:https://owncloud.com/roadmap[owncloud.com/roadmap] + +== Dates + +Note that Version and Notes tagged with UNRELEASED are only prepared and cannot be resolved by design. + + +| Release Date | Type | Version | Notes | +|----------------|----------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------| +| UNRELEASED | _Production_ | _link:https://download.owncloud.com/ocis/ocis/stable/7.1.0/[Infinite Scale 7 (7.1.0)]_ | _link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-7-1-0-production[Release Notes]_ | +| 07.02.2025 | _Production_ | _link:https://download.owncloud.com/ocis/ocis/stable/7.0.1/[Infinite Scale 7 (7.0.1)]_ | _link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-7-0-1-production[Release Notes]_ | +| 07.02.2025 | Rolling | link:https://download.owncloud.com/ocis/ocis/rolling/6.5.1/[Infinite Scale 6 (6.5.1)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-6-5-1-rolling[Release Notes] | +| _17.12.2024_ | _Production_ | _link:https://download.owncloud.com/ocis/ocis/stable/7.0.0/[Infinite Scale 7 (7.0.0)]_ | _link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-7-0-0-production[Release Notes]_ | +| _14.11.2024_ | _Production_ | _link:https://download.owncloud.com/ocis/ocis/stable/5.0.9/[Infinite Scale 5 (5.0.9)]_ | _link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-5-0-9-production[Release Notes]_ | +| 24.10.2024 | Rolling | link:https://download.owncloud.com/ocis/ocis/rolling/6.6.1/[Infinite Scale 6 (6.6.1)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-6-6-1-rolling[Release Notes] | +| 22.10.2024 | Rolling | link:https://download.owncloud.com/ocis/ocis/rolling/6.6.0/[Infinite Scale 6 (6.6.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-6-6-0-rolling[Release Notes] | +| 01.10.2024 | Rolling | link:https://download.owncloud.com/ocis/ocis/rolling/6.5.0/[Infinite Scale 6 (6.5.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-6-5-0-rolling[Release Notes] | +| 30.09.2024 | Production | link:https://download.owncloud.com/ocis/ocis/stable/5.0.8/[Infinite Scale 5 (5.0.8)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-5-0-8-production[Release Notes] | +| 12.09.2024 | Rolling | link:https://download.owncloud.com/ocis/ocis/rolling/6.4.0/[Infinite Scale 6 (6.4.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-6-4-0-rolling[Release Notes] | +| 04.09.2024 | Production | link:https://download.owncloud.com/ocis/ocis/stable/5.0.7/[Infinite Scale 5 (5.0.7)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-5-0-7-production[Release Notes] | +| 19.08.2024 | Rolling | link:https://download.owncloud.com/ocis/ocis/rolling/6.3.0/[Infinite Scale 6 (6.3.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-6-3-0-rolling[Release Notes] | +| 30.07.2024 | Rolling | link:https://download.owncloud.com/ocis/ocis/rolling/6.2.0/[Infinite Scale 6 (6.2.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-6-2-0-rolling[Release Notes] | +| 08.07.2024 | Rolling | link:https://download.owncloud.com/ocis/ocis/rolling/6.1.0/[Infinite Scale 6 (6.1.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-6-1-0-rolling[Release Notes] | +| 19.06.2024 | Rolling | link:https://download.owncloud.com/ocis/ocis/rolling/6.0.0/[Infinite Scale 6 (6.0.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-6-0-0-rolling[Release Notes] | +| 17.07.2024 | Production | link:https://download.owncloud.com/ocis/ocis/stable/5.0.6/[Infinite Scale 5 (5.0.6)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-5-0-6[Release Notes] | +| 22.05.2024 | Production | link:https://download.owncloud.com/ocis/ocis/stable/5.0.5/[Infinite Scale 5 (5.0.5)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-5-0-5[Release Notes] | +| 14.05.2024 | Production | link:https://download.owncloud.com/ocis/ocis/stable/5.0.4/[Infinite Scale 5 (5.0.4)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-5-0-4[Release Notes] | +| 02.05.2024 | Production | link:https://download.owncloud.com/ocis/ocis/stable/5.0.3/[Infinite Scale 5 (5.0.3)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-5-0-3[Release Notes] | +| 17.04.2024 | Production | link:https://download.owncloud.com/ocis/ocis/stable/5.0.2/[Infinite Scale 5 (5.0.2)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-5-0-2[Release Notes] | +| 11.04.2024 | Production | link:https://download.owncloud.com/ocis/ocis/stable/5.0.1/[Infinite Scale 5 (5.0.1)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-5-0-1[Release Notes] | +| 18.03.2024 | Production | link:https://download.owncloud.com/ocis/ocis/stable/5.0.0/[Infinite Scale 5 (5.0.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-5-0-0[Release Notes] | +| 27.03.2024 | Production | link:https://download.owncloud.com/ocis/ocis/stable/4.0.7/[Infinite Scale 4 (4.0.7)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-4-0-7[Release Notes] | +| 07.02.2024 | Production | link:https://download.owncloud.com/ocis/ocis/stable/4.0.6/[Infinite Scale 4 (4.0.6)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-4-0-6[Release Notes] | +| 21.12.2023 | Production | link:https://download.owncloud.com/ocis/ocis/stable/4.0.5/[Infinite Scale 4 (4.0.5)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-4-0-5[Release Notes] | +| 07.12.2023 | Production | link:https://download.owncloud.com/ocis/ocis/stable/4.0.4/[Infinite Scale 4 (4.0.4)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-4-0-4[Release Notes] | +| 24.11.2023 | Production | link:https://download.owncloud.com/ocis/ocis/stable/4.0.3/[Infinite Scale 4 (4.0.3)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-4-0-3[Release Notes] | +| 06.10.2023 | Production | link:https://download.owncloud.com/ocis/ocis/stable/4.0.2/[Infinite Scale 4 (4.0.2)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-4-0-2[Release Notes] | +| 01.09.2023 | Production | link:https://download.owncloud.com/ocis/ocis/stable/4.0.1/[Infinite Scale 4 (4.0.1)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-4-0-1[Release Notes] | +| 25.08.2023 | Production | link:https://download.owncloud.com/ocis/ocis/stable/4.0.0/[Infinite Scale 4 (4.0.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-4-0-0[Release Notes] | +| 07.06.2023 | Production | link:https://download.owncloud.com/ocis/ocis/stable/3.0.0/[Infinite Scale 3 (3.0.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-3-0-0[Release Notes] | +| 30.11.2022 | Production | link:https://download.owncloud.com/ocis/ocis/stable/2.0.0/[Infinite Scale 2 (2.0.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#infinite-scale-2-0-0[Release Notes] | +| 04.04.2022 | Beta | link:https://download.owncloud.com/ocis/ocis/stable/1.9.0/[Infinite Scale 1 (1.9.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#beta-releases[Release Notes] | +| 04.04.2022 | Beta | link:https://download.owncloud.com/ocis/ocis/stable/1.8.0/[Infinite Scale 1 (1.8.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#beta-releases[Release Notes] | +| 04.04.2022 | Beta | link:https://download.owncloud.com/ocis/ocis/stable/1.7.0/[Infinite Scale 1 (1.7.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#beta-releases[Release Notes] | +| 04.04.2022 | Beta | link:https://download.owncloud.com/ocis/ocis/stable/1.6.0/[Infinite Scale 1 (1.6.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#beta-releases[Release Notes] | +| 04.04.2022 | Beta | link:https://download.owncloud.com/ocis/ocis/stable/1.5.0/[Infinite Scale 1 (1.5.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#beta-releases[Release Notes] | +| 04.04.2022 | Beta | link:https://download.owncloud.com/ocis/ocis/stable/1.4.0/[Infinite Scale 1 (1.4.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#beta-releases[Release Notes] | +| 04.04.2022 | Beta | link:https://download.owncloud.com/ocis/ocis/stable/1.3.0/[Infinite Scale 1 (1.3.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#beta-releases[Release Notes] | +| 13.04.2022 | Beta | link:https://download.owncloud.com/ocis/ocis/stable/1.20.0/[Infinite Scale 1 (1.20.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#beta-releases[Release Notes] | +| 04.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.2.0/[Infinite Scale 1 (1.2.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +| 05.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.19.1/[Infinite Scale 1 (1.19.1)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +| 05.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.19.0/[Infinite Scale 1 (1.19.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +| 04.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.18.0/[Infinite Scale 1 (1.18.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +| 04.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.17.0/[Infinite Scale 1 (1.17.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +| 04.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.16.0/[Infinite Scale 1 (1.16.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +| 04.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.15.0/[Infinite Scale 1 (1.15.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +| 04.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.14.0/[Infinite Scale 1 (1.14.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +| 04.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.13.0/[Infinite Scale 1 (1.13.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +| 04.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.12.0/[Infinite Scale 1 (1.12.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +| 04.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.11.0/[Infinite Scale 1 (1.11.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +| 04.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.10.0/[Infinite Scale 1 (1.10.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +| 04.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.1.0/[Infinite Scale 1 (1.1.0)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +| 04.04.2022 | Tech Preview | link:https://download.owncloud.com/ocis/ocis/stable/1.0.0/[Infinite Scale 1 (1.0.06)] | link:https://doc.owncloud.com/ocis_release_notes.html#technology-preview-releases[Release Notes] | +Please find the daily release here: link:https://download.owncloud.com/ocis/ocis/daily/[https://download.owncloud.com/ocis/ocis/daily/]. + + +== Release Types + +ownCloud Infinite Scale is released in three different release types: _Production_, _Rolling_ and _Daily_. Each of them is targeted to a specific use case and audience group: + +// SHORTCODE: {{< hint type=tip title=Production >}} +- Stable and tested release, suitable for critical data. +- Slow cycle, but most stable with slow feature additions. +- Patch releases are provided for critical- and security-bugs as defined by ownCloud support regulations. +// SHORTCODE: {{< /hint >}} +<---> +// SHORTCODE: {{< hint title=Rolling icon=gdoc_check >}} +- Chances are high that some manual testing has happened, yet not structured. +- Features are mostly completed. An upgrade path from the previous rolling release is provided and tested. Great release to use with non critical data. +- Critical bugs are guaranteed to be fixed with the next rolling release. +// SHORTCODE: {{< /hint >}} +## +<---> +// SHORTCODE: {{< hint type=important title=Daily >}} +- Mainly for test use cases. As the releases are done completely unattended, only the automatic test suite has tested the release. Manual testing was only applied by chance. +- Based on ownClouds strong test suite the daily releases are pretty stable, but the risk of unfinished changes is high. +- For example, if a feature requires three commits, and only one was committed, the daily is cut anyway.{{< /columns >}} +// SHORTCODE: {{< /hint >}} + +| Type | Production | Rolling | Daily | +| ----------------------------- | ------------------------------------------------------------------------------------------------------ | -------------------------------------------------- | ---------------------------------------- | +| Frequency | About every 6 month | Every 3 weeks | Daily | +| Audience | Everyone | Early adopters | Developers | +| Risk | Low (known) risk | Low-medium risk | High (unknown) risk | +| Support | Commercially supported | Company support on special agreement with ownCloud | Community support | +| Documentation | Official | Moving documentation | Moving documentation, engineering output | +| Updates | Patch releases based on last stable | Rolling every three weeks | None | +| Update path | Incremental: from previous production to new production, from last rolling before a production release | from previous rolling to new rolling | Clean slate | +| Downgrade | No | No | No | +| Service-level agreement (SLA) | Yes | No | No | +| Overlapping support | Yes | No | No | + + + + +== Documentation +- _Production_ will come with released documentation that is specific for the release. It will remain valid throughout the whole livetime of the release. If patches require documentation changes, addendums will be delivered. + +- _Daily_ and _Rolling_ have access to the documentation as it moves forward in the development process along with the product which will be available on ownClouds web site. There wont be specific releases. Changelog entries, PR comments and similar engineering output can complement the information. + +== Updating and Overlap + +- _Production_ provides a guaranteed upgrade path from the last Production release, as well as from the previous Rolling release before the new Production. For that, support from ownCloud is required. Upgrades between two Production releases are only supported to tested patch releases provided by ownCloud. In an upgrade process, all released patch releases have to be installed in the correct sequence. + +- _Rolling_ is guaranteed to upgrade from the last rolling release or from the previous daily before the new Rolling release. If a critical bug is found in a rolling release, it is guaranteed to be fixed in the next Rolling. There are no backports to the Rolling. In critical cases, an upgrade to a daily release in between has to be done at own risk. + +- _Daily_ does not come with any guaranteed update path. Chances are good that updates will go smooth, but that might have hickups in cases where the upgrade code is not finalized in time. + +Only production gives a reasonable overlap time between releases, for example if version 9 was released, version 8 will still receive a patch release for critical bugs for a reasonable time frame. That time frame will be announced separately and will be aligned with customer needs. +== Support +ownCloud only offers commercial support for _Production_. Rolling might be considered for customer installations in the sales process but always requires an individual agreement between all parties. + +_Daily_ and _Rolling_ are supported on best effort provided by community and ownCloud staff. There is _no SLA_ and no guarantee for attention. + +As ownCloud understands that the effort taken to report a problem is significant and results benefit all users in the community and ownCloud customers, there are resources available to respectfully work on community issues. + +Please consider the link:https://github.com/owncloud/ocis/blob/master/CONTRIBUTING.md[Contribution guidelines] for this. + +== Versioning Scheme +image::../storage/semver-schema.png[Schema of Semantic Versioning,width=70%] +We adhere to _Semantic Versioning_ (SemVer) to ensure clarity and predictability in each of our releases. Learn how Semantic Versioning works on link:https://semver.org/ [https://semver.org/]. diff --git a/modules/developer/pages/ocis/releasing_guide/index.adoc b/modules/developer/pages/ocis/releasing_guide/index.adoc new file mode 100644 index 00000000..58e348db --- /dev/null +++ b/modules/developer/pages/ocis/releasing_guide/index.adoc @@ -0,0 +1,7 @@ += Releasing Processes +:toc: right +:toclevels: 3 + +== General Information + +This section provides information how to prepare a new Infinite Scale release. diff --git a/modules/developer/pages/ocis/releasing_guide/releasing_guide.adoc b/modules/developer/pages/ocis/releasing_guide/releasing_guide.adoc new file mode 100644 index 00000000..fbd06dcf --- /dev/null +++ b/modules/developer/pages/ocis/releasing_guide/releasing_guide.adoc @@ -0,0 +1,200 @@ += Releasing Guide +:toc: right +:toclevels: 3 + + +To prepare an oCIS production release, you must follow a specific sequence of steps in the correct order. + +== New ocis Version + +Use this detailed copy/paste checklists for Jira when starting a new release. + +[WARNING] +==== +* The _examples_ use the following versions which will need to be _adapted_ for the planned release: + * For Major and Minor versions: `7.2.0` + * For Patch releases: `7.2.1` +* The process differs slighly when creating a patch release only. +* Changes applied to a `stable-7.2` branch valid for documentation will be processed to the `docs-stable-x.y` branch automatically after merging. +==== + + +=== Steps for Major or Minor Releases + +[WARNING] +==== +* A new production release is being prepared and the implementation of new features has finished in master. +* `web`, `reva` and `dependencies` have already been bumped. +* Translations + * For oCIS, all required translations changes are included. + * For Web, all translation changes have been applied and are part of the web version provided. +* _Only_ bug fixes to release candidates are allowed. +==== + + +==== Overview + +1. Start the Releasing Process +1. Start the Release Candidates Phase +1. Prepare Release\ +Integrate all RC changes and set the final version +1. Sign-off the Releasing Process +1. Release the new Version +1. Postprocessing and Finalization\ +Backport changes to `master` and create other required stable branches, update cronjobs and clean up + +==== Details + +---- +https://owncloud.dev/ocis/releasing_guide/ + +* [ ] Start the Releasing Process + * [ ] Create a `stable-7.2` branch based on current `master` + * [ ] Inform documentation that the releasing process has started + +* [ ] Start the Release Candidates Phase + * [ ] Create a new feature branch based on `stable-7.2` + * [ ] Move all changelog items from `unreleased` to `7.2.0-rc.1_2025-06-12` + * [ ] Bump oCIS version in `ocis-pkg/version/version.go` + * [ ] Bump oCIS version in `sonar-project.properties` + * [ ] Create PR with `[full-ci][k6-test]` against `stable-7.2` + * [ ] Get PR approved. **DO NOT MERGE YET** + * [ ] Wait for pipeline to be green + * [ ] Create tag: `git tag -s v7.2.0-rc.1 -m "Release 7.2.0-rc.1"` + * [ ] Push tag: `git push origin v7.2.0-rc.1` + * [ ] Watch the PR to see pipeline succeed (can be restarted) + * [ ] Merge PR + * [ ] Sync with DevOps and Product\ + Repeat process with `rc.2`,`rc.3`, ... + * [ ] All required translations from oCIS are included + * [ ] All issues are fixed, RC phase has finished + +* [ ] Prepare Release + * [ ] Create a new feature branch based on `stable-7.2` + * [ ] Move all changelog items from `7.2.0-rc.*` to `7.2.0_2025-04-01` + * [ ] Bump oCIS version in `ocis-pkg/version/version.go` + * [ ] Bump oCIS version in `sonar-project.properties` + * [ ] Create PR with `[full-ci][k6-test]` against `stable-7.2` + * [ ] Mark PR as **Draft** to avoid accidentially merging + * [ ] Get PR approved. **DO NOT MERGE YET**\ + Info: merging will be done in step *Release the new Version* + * [ ] Wait for pipeline to be green + +* [ ] Get new Release Sign-off (jira) + * [ ] **EITHER (preferred):** Find someone who wants the release more than you do, and have them sign-off + * [ ] **OR (not recommended):** Have the appropriate people sign the *release sign-off* document + +* [ ] Release the new Version + * [ ] Create tag: `git tag -s v7.2.0 -m "Release 7.2.0"`\ + Note the tag name scheme is important + * [ ] Push tag: `git push origin v7.2.0` + * [ ] Watch the PR to see pipeline succeed (can be restarted) + * [ ] Smoke test docker image `owncloud/ocis@v7.2.0` + * [ ] Choose any docker-compose example from oCIS repository + * [ ] Export `OCIS_DOCKER_IMAGE=owncloud/ocis` + * [ ] Export `OCIS_DOCKER_TAG=7.2.0` + * [ ] `docker compose up -d` + * [ ] Confirm oCIS version in browser and start the `upload-download-awesome` test + * [ ] Remove the **Draft** state and Merge PR\ + This is the PR from step *Prepare Release* + * [ ] Delete the feature branch + * [ ] Announce release in *Teams channel: oCIS* + +* [ ] Postprocessing and Finalization + * [ ] Backport `stable-7.2` to the master branch + * [ ] Create a `docs-stable-7.2` branch + * [ ] Create orphan branch: `git checkout --orphan docs-stable-7.2` + * [ ] Initial commit: `git commit --allow-empty -m "initial commit"` + * [ ] Push it: `git push` + * [ ] Adjust the `.drone.star` file to write to `docs-stable-7.2` + * [ ] Find target_branch value in the docs section and change it to `docs-stable-7.2` + * [ ] Example: https://github.com/owncloud/ocis/blame/56f7645f0b11c9112e15ce46f6effd2fea01d6be/.drone.star#L2249 + * [ ] Add `stable-7.2` to the nightly cron jobs in drone (`Settings` -> `Cron Jobs`) +---- + +=== Steps for Patch Releases Only + +[WARNING] +==== +* A patch branch is prepared, based off the appropriate stable branch, and contains all the changes including `web`, `reva` and `dependencies` bumps. The patch branch is merged into the corresponding stable branch. No release candidates are used because the changes are known. +* Translations + * For oCIS, all required translations changes are included. + * For Web, all translation changes have been applied and are part of the web version provided. +* _Only_ bug fixes to the patch branch are allowed. +==== + + +==== Overview + +1. Prepare Release\ +Integrate all patches and set the final version +1. Sign-off the Releasing Process +1. Release the new Version +1. Check Forward Porting to Master + +==== Details + +---- +https://owncloud.dev/ocis/releasing_guide/ + +* [ ] Prepare Release + * [ ] Create a new feature branch based on `stable-7.2` + * [ ] Move all changelog items from `unreleased` to `7.2.1_2025-04-01` + * [ ] Bump oCIS version in `ocis-pkg/version/version.go` + * [ ] Bump oCIS version in `sonar-project.properties` + * [ ] Create PR with `[full-ci][k6-test]` against `stable-7.2` + * [ ] Mark PR as **Draft** to avoid accidentially merging + * [ ] Get PR approved **requires 2 approvals**, **DO NOT MERGE YET**\ + Info: merging will be done in step *Release the new Version* + * [ ] Wait for pipeline to be green + +* [ ] Get new Release Sign-off (confluence) + * [ ] **EITHER (preferred):** Find someone who wants the release more than you do, and have them sign-off + * [ ] **OR (not recommended):** Have the appropriate people sign the *release sign-off* document + +* [ ] Release the new Version + * [ ] Create tag: `git tag -s v7.2.1 -m "Release 7.2.1"`\ + Note the tag name scheme is important + * [ ] Push tag: `git push origin v7.2.1` + * [ ] Watch the PR to see pipeline succeed (can be restarted) + * [ ] Smoke test docker image `owncloud/ocis@v7.2.1` + * [ ] Choose any docker-compose example from ocis repository + * [ ] Export `OCIS_DOCKER_IMAGE=owncloud/ocis` + * [ ] Export `OCIS_DOCKER_TAG=7.2.1` + * [ ] `docker compose up -d` + * [ ] Confirm oCIS version in browser and start the `upload-download-awesome` test + * [ ] Remove the **Draft** state and Merge PR\ + This is the PR from step *Prepare Release* + * [ ] Delete the feature branch + * [ ] Announce release in *Teams channel: oCIS* + +* [ ] Check if Forward Porting to Master is Required +---- + +== Update Documentation Related Data + +=== Envvar Changes + +Follow the link:https://owncloud.dev/services/general-info/envvars/new-release-process/[Release Process for Envvars] documentation to update the required environment variable data. These changes _must_ be present in the appropriate `stable-7.2` branch and should be applied to the respective feature branch during the development cycle. This _avoids manually backporting_ to `stable-7.2` but may be required if changes go to master instead. Changes to envvars are typically found in the `docs` folder. + +[WARNING] +==== +The admin docs processes access the data via the branch name, not the tag set. Therefore, any changes applied to the stable branch are accessible to the documentation process without the need of a new version. +==== + + +=== Prepare Admin Docs + +The admin documentation must be prepared for the new release. + +Note that this section is only _informational_ for oCIS developers and is done by the documentation team. This is why they need to be informed in a timely manner. + +* [ ] Create _overview release notes_ in the `docs-main` repo based on the changelog\ +The source is most likely the unreleased folder during the development cycle +* [ ] For Major and Minor versions: + * [ ] Add all _documentation changes_ including the upgrade guide in the `docs-ocis` repo in master + * [ ] Create a new release branch in the `docs-ocis` repo based off master + * [ ] _Update oCIS branches and versions_ in the `docs` repo +* [ ] For Patch releases: + * [ ] _Update oCIS versions_ in the `docs` repo + diff --git a/modules/developer/pages/ocis/releasing_guide/troubleshooting.adoc b/modules/developer/pages/ocis/releasing_guide/troubleshooting.adoc new file mode 100644 index 00000000..95a607a0 --- /dev/null +++ b/modules/developer/pages/ocis/releasing_guide/troubleshooting.adoc @@ -0,0 +1,70 @@ += Troubleshooting +:toc: right +:toclevels: 3 + + +This document covers some of the issues that can happen during the release process. + +Use this detailed copy/paste checklists for the PR when starting a backport. + +== Backport Missing Commits + +Once the `stable-x.y` branch has been created but work has been merged into master, these changes will not appear in the stable branch unless backporting is used. + +[WARNING] +==== +* Any change applied to `stable-7.2` must be made _before_ the tag for this release is created to be part of which is mandatory for any code change. +* There is one exception. Changes applied for documentation purposes are accessed via the branch name by the doc process and not via the tag and therefore accessible to the documentation processes. +==== + + +Though any changes should ideally be targeted to the correct branch from the beginning, this may sometimes be impossible, especially for the following folder locations: + +- `docs` and +- `deployments/examples/ocis_full` + +=== Documentation + +Changes to documentation must be present in the respective stable branch because the documentation process accesses the data for further processing. Follow these steps to apply all doc related changes from master to the stable branch _after_ the tag has been created: + +---- +https://owncloud.dev/ocis/releasing_guide/ + +* [ ] Check all relevant commits made to the master branch that are not in stable\ +`git log stable-7.2..master -- docs/` +* [ ] Double check all changes if they apply to stable\ +`git diff stable-7.2..master -- docs/` +* [ ] Create a new feature branch based on `stable-7.2` +* [ ] Cherry pick missing commits +* [ ] Create PR with leading `[docs-only]` against `stable-7.2` +* [ ] Get PR approved **requires 2 approvals** and merge it +* [ ] Delete feature branch +---- + +=== Deployment Example + +Changes to the `ocis_full` deployment example are less critical. But as this is referenced by the documentation, changes must be present in the respective stable branch. Follow these steps to apply all deployment example related changes from master to the stable branch _after_ the tag has been created: + +==== Web Extensions + +Update all dependencies and extension related data in the `web-extensions` repo and create for each extension a new release if applicapable. + +==== ocis_full + +Update the `ocis_full` deployment example as required in master which includes to update the versions of the web extensions used. Test the functionality if the example works. + +Follow these steps to apply all `ocis_full` deployment example related changes from master to the stable branch: + +---- +https://owncloud.dev/ocis/releasing_guide/ + +* [ ] Check all relevant commits made to the master branch that are not in stable\ +`git log stable-7.2..master -- deployments/examples/ocis_full/` +* [ ] Double check all changes if they apply to stable\ +`git diff stable-7.2..master -- deployments/examples/ocis_full/` +* [ ] Create a new feature branch based on `stable-7.2` +* [ ] Cherry pick missing commits +* [ ] Create PR with leading `[docs-only]` against `stable-7.2` +* [ ] Get PR approved **requires 2 approvals** and merge it +* [ ] Delete feature branch +---- diff --git a/modules/developer/pages/ocis/storage-backends/cephfs.adoc b/modules/developer/pages/ocis/storage-backends/cephfs.adoc new file mode 100644 index 00000000..cb8d450a --- /dev/null +++ b/modules/developer/pages/ocis/storage-backends/cephfs.adoc @@ -0,0 +1,71 @@ += cephfs +:toc: right +:toclevels: 3 + + +oCIS intends to make the aspects of existing storage systems available as transparently as possible, but the static sync algorithm of the desktop client relies on some form of recursive change time propagation on the server side to detect changes. While this can be bolted on top of existing file systems with inotify, the kernel audit or a fuse based overlay filesystem, a storage system that already implements this aspect is preferable. Aside from EOS, cephfs supports a recursive change time that oCIS can use to calculate an etag for the webdav API. + +== Development + +The cephfs development happens in a link:https://github.com/cs3org/reva/pull/1209[Reva branch] and is currently driven by CERN. + +== Architecture + +In the original approach the driver was based on the link:https://github.com/cs3org/reva/blob/a8c61401b662d8e09175416c0556da8ef3ba8ed6/pkg/storage/utils/localfs/localfs.go[localfs] driver, relying on a locally mounted cephfs. It would interface with it using the POSIX apis. This has been changed to directly call the Ceph API using https://github.com/ceph/go-ceph. It allows using the ceph admin APIs to create sub-volumes for user homes and maintain a file id to path mapping using symlinks. + +== Implemented Aspects +The recursive change time built ino cephfs is used to implement the etag propagation expected by the ownCloud clients. This allows oCIS to pick up changes that have been made by external tools, bypassing any oCIS APIs. + +Like other filesystems cephfs uses inodes and like most other filesystems inodes are reused. To get stable file identifiers the current cephfs driver assigns every node a file id and maintains a custom fileid to path mapping in a system directory: +---- +/tmp/cephfs $ tree -a +. +├── reva +│ └── einstein +│ ├── Pictures +│ └── welcome.txt +└── .reva_hidden + ├── .fileids + │ ├── 50BC39D364A4703A20C58ED50E4EADC3_570078 -> /tmp/cephfs/reva/einstein + │ ├── 571EFB3F0ACAE6762716889478E40156_570081 -> /tmp/cephfs/reva/einstein/Pictures + │ └── C7A1397524D0419B38D04D539EA531F8_588108 -> /tmp/cephfs/reva/einstein/welcome.txt + └── .uploads +---- + +Versions are not file but snapshot based, a link:https://docs.ceph.com/en/latest/dev/cephfs-snapshots/[native feature of cephfs]. The driver maps entries in the native cephfs `.snap` folder to the CS3 api recycle bin concept and makes them available in the web UI using the versions sidebar. Snapshots can be triggered by users themselves or on a schedule. + +Trash is not implemented, as cephfs has no native recycle bin and instead relies on the snapshot functionality that can be triggered by end users. It should be possible to automatically create a snapshot before deleting a file. This needs to be explored. + +Shares link:https://github.com/cs3org/reva/pull/1209/files#diff-5e532e61f99bffb5754263bc6ce75f84a30c6f507a58ba506b0b487a50eda1d9R168-R224[are mapped to ACLs] supported by cephfs. The share manager is used to persist the intent of a share and can be used to periodically verify or reset the ACLs on cephfs. + +== Future work +- The spaces concept matches cephfs sub-volumes. We can implement the CreateStorageSpace call with that, keep track of the list of storage spaces using symlinks, like for the id based lookup. +- The share manager needs a persistence layer. +- Currently we persist using a single json file. +- As it basically provides two lists, _shared with me_ and _shared with others_, we could persist them directly on cephfs! + - If needed for redundancy, the share manager can be run multiple times, backed by the same cephfs + - To save disk io the data can be cached in memory, and invalidated using stat requests. +- A good tradeoff would be a folder for each user with a json file for each list. That way, we only have to open and read a single file when the user want's to list the shares. +- To allow deprovisioning a user the data should be sharded by userid. That way all share information belonging to a user can easily be removed from the system. If necessary it can also be restored easily by copying the user specific folder back in place. +- For consistency over metadata any file blob data, backups can be done using snapshots. +- An example where einstein has shared a file with marie would look like this on disk: +---- +/tmp/cephfs $ tree -a +. +├── reva +│ └── einstein +│ ├── Pictures +│ └── welcome.txt +├── .reva_hidden +│ ├── .fileids +│ │ ├── 50BC39D364A4703A20C58ED50E4EADC3_570078 -> /tmp/cephfs/reva/einstein +│ │ ├── 571EFB3F0ACAE6762716889478E40156_570081 -> /tmp/cephfs/reva/einstein/Pictures +│ │ └── C7A1397524D0419B38D04D539EA531F8_588108 -> /tmp/cephfs/reva/einstein/welcome.txt +│ └── .uploads +└── .reva_share_manager + ├── einstein + │ └── sharedWithOthers.json + └── marie + └── sharedWithMe.json +---- +- The fileids should link:https://github.com/cs3org/reva/pull/1209/files#diff-eba5c8b77ccdd1ac570c54ed86dfa7643b6b30e5625af191f789727874850172R125-R127[not be based on the path] and instead use a uuid that is also persisted in the extended attributes to allow rebuilding the index from scratch if necessary. diff --git a/modules/developer/pages/ocis/storage-backends/dcfsnfs.adoc b/modules/developer/pages/ocis/storage-backends/dcfsnfs.adoc new file mode 100644 index 00000000..6f37d7a9 --- /dev/null +++ b/modules/developer/pages/ocis/storage-backends/dcfsnfs.adoc @@ -0,0 +1,60 @@ += Decomposed FS on NFS +:toc: right +:toclevels: 3 + + +oCIS' default storage backend is the Decomposed FS. The Decomposed FS can be set up to run on a NFS share. That way the same storage can be provided over the network to other nodes running oCIS. + +This document summarizes a few important considerations of the NFS setup and describes a tested setup. The test has not covered concurrent access of data from different nodes yet. + +== NFS Server Setup + +This document covers the linux kernel NFS server on a standard Linux running on x86_64. + +The NFS server needs to be set up in a way that it supports link:https://en.wikipedia.org/wiki/Extended_file_attributes[extended file attributes]. + +Extended attributes are supported by NFS starting with Kernel version 5.9, which means that the server with the NFS server has to run a kernel with that or a higher version number. To check that, run the command `uname -a` on the NFS server and compare the displayed version number. + +The NFS server in the test setup was configured with the following line in the config file `/etc/exports`: + +`/space/nfstest 192.168.178.0/24(rw,root_squash,async,subtree_check,anonuid=0,anongid=100,all_squash)` + +This exports the directory `/space/nfstest` to the internal network with certain options. + +Important: + +- The share needs to be exported with the `async` option for proper NFS performance. + +== NFS Client Mount + +The nodes that run oCIS need to mount the NFS storage to a local mount point. + +The test setup uses the client mount command: `mount -t nfs -o nfsvers=4 192.168.178.28:/space/nfstest /mnt/ocisdata/` + +It sets the NFS version to 4, which is important to support extended attributes. + +After successfully mounting the storage on the client, it can be checked if the NFS setup really supports extended attributes properly using the following commands. + +`setfattr -n user.test -v "xattr test string" ocisdata/foo` to write an extended attribute to a file, and `getfattr -d ocisdata/foo` to list all the attributes a file has set. + +[NOTE] +==== +The NFS server setup can be optimized considering system administrative-, performance- and security options. This is not (yet) covered in this documentation. +==== + + +== oCIS Start using the NFS Share + +The oCIS server can be instructed to set up the decomposed FS at a certain path by setting the environment variable `OCIS_BASE_DATA_PATH`. + +The test setup started an oCIS tech preview single binary release using this start command: + +[source,bash] +---- +./ocis init +OCIS_BASE_DATA_PATH=/mnt/ocisdata/ OCIS_LOG_LEVEL=debug OCIS_INSECURE=true PROXY_HTTP_ADDR=0.0.0.0:9200 OCIS_URL=https://hostname:9200 ./ocis server +---- + +This starts oCIS and a decomposed FS skeleton file system structure is set up on the NFS share. + +The oCIS instance is passing a smoke test. diff --git a/modules/developer/pages/ocis/storage-backends/eos.adoc b/modules/developer/pages/ocis/storage-backends/eos.adoc new file mode 100644 index 00000000..b8758e66 --- /dev/null +++ b/modules/developer/pages/ocis/storage-backends/eos.adoc @@ -0,0 +1,245 @@ += EOS +:toc: right +:toclevels: 3 + + +oCIS can be configured to run on top of link:https://eos.web.cern.ch/[eos]. While the link:http://eos-docs.web.cern.ch/[eos documentation] does cover a lot of topics, it leaves out some details that you may have to either pull from various link:https://gitlab.cern.ch/eos/eos-docker[docker containers], the link:https://eos-community.web.cern.ch/[forums] or even the link:https://github.com/cern-eos/eos[source] itself. + +This document is a work in progress of the current setup. + +== Docker dev environment for eos storage + +We begin with the docker-compose.yml found in https://github.com/owncloud/ocis/tree/master/ocis/ and +switch it to eos-storage. + +=== 1. Start eos & ocis containers + +Start the eos cluster and ocis via the compose stack. +---- +docker-compose up -d +---- + +[NOTE] +==== +The first time the _ocis_ container starts up, it will compile ocis from scratch which can take a while. +To follow progress, run `docker-compose logs -f --tail=10 ocis` +==== + + +=== 2. LDAP support + +Configure the OS to resolve users and groups using ldap + +---- +docker-compose exec -d ocis /start-ldap +---- + +Check that the OS in the ocis container can now resolve einstein or the other demo users + +---- +$ docker-compose exec ocis id einstein +uid=20000(einstein) gid=30000(users) groups=30000(users),30001(sailing-lovers),30002(violin-haters),30007(physics-lovers) +---- + +[NOTE] +==== +If the user is not found at first you might need to wait a few more minutes in case the ocis container is still compiling. +==== + + +We also need to restart the storage-userprovider service, so it picks up the changed environment. Without a restart it is not able to resolve users from LDAP. +---- +docker-compose exec ocis ./bin/ocis kill storage-userprovider +docker-compose exec ocis ./bin/ocis run storage-userprovider +---- + +=== 3. Home storage + +Kill the home storage. By default it uses the `ocis` storage driver. We need to switch it to the `eoshome` driver: + +---- +docker-compose exec ocis ./bin/ocis kill storage-home +docker-compose exec -e STORAGE_HOME_DRIVER=eoshome ocis ./bin/ocis run storage-home +---- + +=== 4. Users storage + +Kill the users storage. By default it uses the `ocis` storage driver. We need to switch it to the `eos` driver: + +---- +docker-compose exec ocis ./bin/ocis kill storage-users +docker-compose exec -e STORAGE_USERS_DRIVER=eos ocis ./bin/ocis run storage-users +---- + +=== 5. Metadata storage + +First we need to create the metadata root in eos and set an owner: +---- +docker-compose exec ocis eos mkdir -p /eos/dockertest/ocis/metadata +docker-compose exec ocis eos chown 2:2 /eos/dockertest/ocis/metadata +---- + +[NOTE] +==== +The uid and gid `2` are referencing the user `daemon` inside the ocis container. That user is also configured when restarting the accounts service later. For production systems you should create a dedicated user for the metadata storage. +==== + + +Kill the metadata storage. By default it uses the `ocis` storage driver. We need to switch it to the `eos` driver: + +---- +docker-compose exec ocis ./bin/ocis kill storage-system +docker-compose exec -e STORAGE_SYSTEM_DRIVER=eos -e STORAGE_SYSTEM_ROOT=/eos/dockertest/ocis/metadata ocis ./bin/ocis run storage-system +---- + + +=== 6. Accounts service + +Kill the accounts service. By default it uses the `ocis` storage driver. We need to switch it to the `eos` driver: + +---- +docker-compose exec ocis ./bin/ocis kill accounts +docker-compose exec -e ACCOUNTS_SERVICE_USER_USERNAME=daemon -e ACCOUNTS_SERVICE_USER_UID=2 -e ACCOUNTS_SERVICE_USER_GID=2 ocis ./bin/ocis run accounts +---- + +== Verification + +Login with `einstein / relativity`, upload a file to einsteins home and verify the file is there using + +---- +docker-compose exec ocis eos ls -l /eos/dockertest/reva/users/4/4c510ada-c86b-4815-8820-42cdf82c3d51/ +-rw-r--r-- 1 einstein users 10 Jul 1 15:24 newfile.txt +---- + +If the problem persists, please check the <>. + +== Further exploration + +EOS has a built-in shell that you can enter using +---- +$ docker-compose exec mgm-master eos +# --------------------------------------------------------------------------- +# EOS Copyright (C) 2011-2019 CERN/Switzerland +# This program comes with ABSOLUTELY NO WARRANTY; for details type `license'. +# This is free software, and you are welcome to redistribute it +# under certain conditions; type `license' for details. +# --------------------------------------------------------------------------- +EOS_INSTANCE=eostest +EOS_SERVER_VERSION=4.6.5 EOS_SERVER_RELEASE=1 +EOS_CLIENT_VERSION=4.6.5 EOS_CLIENT_RELEASE=1 +EOS Console [root://localhost] |/> help +access Access Interface +accounting Accounting Interface +acl Acl Interface +archive Archive Interface +attr Attribute Interface +backup Backup Interface +clear Clear the terminal +cd Change directory +chmod Mode Interface +chown Chown Interface +config Configuration System +console Run Error Console +cp Cp command +debug Set debug level +exit Exit from EOS console +file File Handling +fileinfo File Information +find Find files/directories +newfind Find files/directories (new implementation) +fs File System configuration +fsck File System Consistency Checking +fuse Fuse Mounting +fusex Fuse(x) Administration +geosched Geoscheduler Interface +group Group configuration +health Health information about system +help Display this text +info Retrieve file or directory information +inspector Interact with File Inspector +io IO Interface +json Toggle JSON output flag for stdout +license Display Software License +ls List a directory +ln Create a symbolic link +map Path mapping interface +member Check Egroup membership +mkdir Create a directory +motd Message of the day +mv Rename file or directory +node Node configuration +ns Namespace Interface +pwd Print working directory +quit Exit from EOS console +quota Quota System configuration +reconnect Forces a re-authentication of the shell +recycle Recycle Bin Functionality +rmdir Remove a directory +rm Remove a file +role Set the client role +route Routing interface +rtlog Get realtime log output from mgm & fst servers +silent Toggle silent flag for stdout +space Space configuration +stagerrm Remove disk replicas of a file if it has tape replicas +stat Run 'stat' on a file or directory +squash Run 'squashfs' utility function +test Run performance test +timing Toggle timing flag for execution time measurement +touch Touch a file +token Token interface +tracker Interact with File Tracker +transfer Transfer Interface +version Verbose client/server version +vid Virtual ID System Configuration +whoami Determine how we are mapped on server side +who Statistics about connected users +? Synonym for 'help' +.q Exit from EOS console +EOS Console [root://localhost] |/> +---- + +But this is a different adventure. See the links at the top of this page for other sources of information on eos. + +== Cleaning up + +To clean up and start completely from scratch, run `docker-compose down -v`. +Then delete the local "bin" folder as root which contains the ocis binaries compiled by the "ocis" docker. + +== Troubleshooting + +=== Docker-compose exits right away + +When running `docker-compose up -d` ocis exits right away. + +You can check the error code using `docker-compose ps` and investigate further by running only ocis again using `docker-compose up ocis` (without `-d` so you can see what is going on in the foreground). +One reason might be that the binary was already built but does not match the container env. Try running `make clean` before running `docker-compose up ocis` so it gets built inside the container. + +=== Where are the logs ? + +The ocis logs can be accessed using `docker-compose logs ocis`. Add `-f` for following. + +=== How do I update a service in the ocis container? + +1. `docker-compose exec ocis make clean build` to update the binary +2. `docker-compose exec ocis ./bin/ocis kill ` to kill the service +3. `docker-compose exec ocis ./bin/ocis run ` to start the service. Do not forget to set any env vars, e.g. + `docker-compose exec -e STORAGE_HOME_DRIVER=eoshome -e STORAGE_DRIVER_EOS_LAYOUT="{{substr 0 1 .Id.OpaqueId}}/{{.Id.OpaqueId}}" ocis ./bin/ocis run storage-home` + +=== Creation and upload of files does not work + +If the upload did not work, please check the status of the eos space using the command `docker-compose exec mgm-master eos fs ls`. +In case the default space appears as offline, run `docker-compose exec mgm-master eos space set default on`. + +=== Uploading big files appears to hang + +Please note that the uploads first go into the "ocis" docker and land in its "/tmp" folder, then gets copied over to the EOS docker using `xrdcopy`. +This is why uploading first transfers all bytes and then seem to hang for a while during the final copy. + +=== Running out of space quickly + +The EOS dockers are configured with replication, so every file uploaded there will be replicated 4 times, +so make sure there is enough physical space on disk when testing. + +Also please note that older failed uploads might still be present in the "/tmp" directory of the "ocis" container. + diff --git a/modules/developer/pages/ocis/storage-backends/index.adoc b/modules/developer/pages/ocis/storage-backends/index.adoc new file mode 100644 index 00000000..95a08843 --- /dev/null +++ b/modules/developer/pages/ocis/storage-backends/index.adoc @@ -0,0 +1,4 @@ += Storage backends +:toc: right +:toclevels: 3 + diff --git a/modules/developer/pages/ocis/storage/index.adoc b/modules/developer/pages/ocis/storage/index.adoc new file mode 100644 index 00000000..824d7eca --- /dev/null +++ b/modules/developer/pages/ocis/storage/index.adoc @@ -0,0 +1,30 @@ += Storage +:toc: right +:toclevels: 3 + +== Overview + +OCIS wraps link:https://github.com/cs3org/reva/[reva] and adds an opinionated configuration to provide two core services for the oCIS platform: +1. A xref:./spacesregistry.adoc[_Spaces Registry_] that acts as a dictionary for storage _Spaces_ and their metadata +2. A xref:./spacesprovider.adoc[_Spaces Provider_] that organizes _Resources_ in storage _Spaces_ and persists them in an underlying _Storage System_ + +_Clients_ will use the _Spaces Registry_ to poll or get notified about changes in all _Spaces_ a user has access to. Every _Space_ has a dedicated `/dav/spaces/` WebDAV endpoint that is served by a _Spaces Provider_ which uses a specific reva storage driver to wrap an underlying _Storage System_. + +image::ocis/storage/overview.drawio.svg[] + +The dashed lines in the diagram indicate requests that are made to authenticate requests or lookup the storage provider: +1. After authenticating a request, the proxy may either use the CS3 `userprovider` or the accounts service to fetch the user information that will be minted into the `x-access-token`. +2. The gateway will verify the JWT signature of the `x-access-token` or try to authenticate the request itself, e.g. using a public link token. + +[WARNING] +==== +The bottom part is lighter because we will deprecate it in favor of using only the CS3 user and group providers after moving some account functionality into reva and glauth. The metadata storage is not registered in the reva gateway to separate metadata necessary for running the service from data that is being served directly. +==== + + +== Endpoints and references + +In order to reason about the request flow, two aspects in the architecture need to be understood well: +1. What kind of xref:./namespaces.adoc[_namespaces_] are presented at the different WebDAV and CS3 endpoints? +2. What kind of xref:./terminology.md#resources.adoc[_resource_] xref:./terminology.md#references.adoc[_references_] are exposed or required: path or id based? +image::ocis/storage/storage.drawio.svg[] diff --git a/modules/developer/pages/ocis/storage/namespaces.adoc b/modules/developer/pages/ocis/storage/namespaces.adoc new file mode 100644 index 00000000..39687710 --- /dev/null +++ b/modules/developer/pages/ocis/storage/namespaces.adoc @@ -0,0 +1,100 @@ += Namespaces +:toc: right +:toclevels: 3 + +A _namespace_ is a set of paths with a common prefix. Depending on the endpoint you are talking to you will encounter a different kind of namespace: +In ownCloud 10 all paths are considered relative to the users home. The CS3 API uses a global namespace and the _storage providers_ use a local namespace with paths relative to the storage providers root. + +image::ocis/storage/namespaces.drawio.svg[] + +The different paths in the namespaces need to be translated while passing xref:./terminology.md#references.adoc[_references_] from service to service. While the oc10 endpoints all work on paths we internally reference shared resources by id, so the shares don't break when a file is renamed or moved inside a storage xref:./spaces.adoc[_space_]. The following table lists the various namespaces, paths and id based references: + +| oc10 namespace | CS3 global namespace | storage provider | reference | content | +|--------------------------------------------------|----------------------------------------|------------------|-----------|---------| +| `/webdav/path/to/file.ext` `/dav/files//path/to/file.ext` | `/home/path/to/file.ext` | home | `//path/to/file.ext` | currently logged in users home | +| `/webdav/Shares/foo` `/dav/files//Shares/foo` | `/home/Shares/foo` | users | id based access | all users, used to access collaborative shares | +| `/dav/public-files//rel/path/to/file.ext` | `/public//rel/path/to/file.ext` | public | id based access | publicly shared files, used to access public links | + + +[CAUTION] +==== +oCIS currently is configured to jail users into the CS3 `/home` namespace in the oc10 endpoints to mimic ownCloud 10. CernBox has been exposing a global namespace on `/webdav` for years already. The ocs service returns urls that are relative to the CS3 global namespace which makes both scenarios work, but only one of them at a time. Which is why the testsuite hiccups when trying to link:https://github.com/cs3org/reva/pull/1605[Allow full paths targets in reva#1605]. +==== + + + +[WARNING] +==== +In the global CS3 namespaces we plan to move `/home/Shares`, which currently lists all mounted shares of the currently logged-in user to a dedicated `/shares` namespace. See xref:#cs3-namespaces.adoc[below] and link:https://github.com/cs3org/reva/pull/1584[Move shares folder out from home directory to a separate mount reva#1584]. +==== + + +== ownCloud namespaces + +In contrast to the global namespace of CS3, ownCloud always presented a user specific namespace on all endpoints. It will always list the users private files under `/`. Shares can be mounted at an arbitrary location in the users private spaces. See the xref:./spacesprovider#webdav.adoc[webdav] and xref:./spacesprovider#sharing.adoc[ocs] sections for more details end examples. + +With the spaces concept we are planning to introduce a global namespace to the ownCloud webdav endpoints. This will push the users private space down in the hierarchy: it will move from `/webdav` to `/webdav/home` or `/webdav/users/`. The related xref:../../ocis/migration.adoc[migration stages] are subject to change. + +== CS3 global namespaces + +The _CS3 global namespace_ in oCIS is configured in the storage xref:./spacesregistry.adoc[_spaces registry_]. oCIS uses these defaults: + +| global namespace | description | +|-|-| +| `/home` | an alias for the currently logged in uses private space | +| `/users/` | user private spaces | +| `/shares` | a virtual listing of share spaces a user has access to | +| `/public/` | a virtual folder listing public shares | +| `/spaces/` | _TODO: project or group spaces_ | + +Technically, the `/home` namespace is not necessary: the storage xref:./spacesregistry.adoc[_spaces registry_] knows the path to a users private space in the `/users` namespace and the gateway can forward the requests to the responsible storage provider. + +[WARNING] +==== +*@jfd: Why don't we use `/home/` instead of `/users/`. Then the paths would be consistent with most unix systems. +==== + + +The `/shares` namespace is used to solve two problems: +- To query all shares the current user has access to the _share manager_ can be used to list the resource ids. While the shares can then be navigated by resource id, they will return the relative path in the actual xref:./terminology.md#storage-providers.adoc[_storage provider_], leaking parent folders of the shared resource. +- When accepting a remote share e.g., for OCM the resource does not exist on the local instance. They are made accessible in the global namespace under the `/shares` namespace. + +[WARNING] +==== +_@jfd: Should we split `/shares` into `/collaborations`, `/ocm` and `/links`? We also have `/public` which uses token based authentication. They may have different latencies or polling strategies? Well, I guess we can cache them differently regardless of the mount point._ +==== + + +== Browser URLs vs API URLs +In ownCloud 10 you can not only create _public links_ but also _private links_. Both can be copy pasted into an email or chat to grant others access to a file. Most often though, end users will copy and paste the URL from their browsers location bar. + +| URL | description | +|-|-| +| https://demo.owncloud.com/apps/files/?dir=/Photos/Vacation&fileid=24 | The normal browser URL | +| https://demo.owncloud.com/apps/files/?fileid=24 | the `dir` is actually not used to find the directory and will be filled when pasting this URL | +| https://demo.owncloud.com/f/24 | _private links_ are the shortened version of this and work in the same way | +| https://demo.owncloud.com/s/piLdAAt1m3Bg0Fk | public link | + +// SHORTCODE: {{< hint >}} +The `dir` parameter alone cannot be used to look up the directory, because the path for a file may be different depending on the currently logged-in user: +- User A shares his `/path/to/Photos` with User X. +- User B shares his `/other/path/to/Photos` with User X and Y. +- User A shares his `/path/to/Photos` with User Y. + +(Depending on the order in which they accept the shares) X and Y now have two folders `/shares/Photos` and `/shares/Photos (1)`. But if they were to copy paste a link with that path in the URL and if the directory were only looked up by path X and Y would end up in different folders. + +You could argue that the path should always use a global path in the CS3 namespace: +- User A shares his `/users/a/path/to/Photos` with User X. +- User B shares his `/users/b/other/path/to/Photos` with User X and Y. +- User A shares his `/users/a/path/to/Photos` with User Y. + +By using a global path like this X and Y would always end up in the correct folder. However, there are two caveats: +- This only works for resources that reside on the instance (because only they have unique and global path). Shares from other instances need to be identified by id, or they cannot be uniquely addressed +- User A may not want to leak path `path/to` segments leading to `Photos`. They might contain things like `low-priority` or personal data. + +That is the reason why URLs always have to contain some kind of stable identifier. By introducing the concept of _storage spaces_ and treating user homes, project drives and shares we can create a URL that contains an identifier for the _storage space_ and a path relative to the root of it. +// SHORTCODE: {{< /hint >}} + +In ocis we will unify the way links sharing works, however there will always be at least two types of URLs: +1. the URL you see in the browsers location bar, and +2. the URL that a client uses to actually access a file. diff --git a/modules/developer/pages/ocis/storage/proposedchanges.adoc b/modules/developer/pages/ocis/storage/proposedchanges.adoc new file mode 100644 index 00000000..46f3902e --- /dev/null +++ b/modules/developer/pages/ocis/storage/proposedchanges.adoc @@ -0,0 +1,174 @@ += Proposed Changes +:toc: right +:toclevels: 3 + +Some architectural changes still need to be clarified or changed. Maybe an ADR is in order for all of the below. + +== Reva Gateway changes + +== A dedicated shares storage provider + +Currently, when a user accepts a share, a cs3 reference is created in the users `/home/shares` folder. This reference represents the mount point of a share and can be renamed, similar to the share jail in ownCloud 10. This spreads the metadata of a share in two places: +- the share is persisted in the _share manager_ +- the mount point of a share is persisted in the home _storage provider_ + +Furthermore, the _gateway_ treats `/home/shares` different than any other path: it will stat all children and calculate an etag to allow clients to discover changes in accepted shares. This requires the storage provider to cooperate and provide this special `/shares` folder in the root of a users home when it is accessed as a home storage. That is the origin of the `enable_home` config flag that needs to be implemented for every storage driver. + +In order to have a single source of truth we need to make the _share manager_ aware of the mount point. We can then move all the logic that aggregates the etag in the share folder to a dedicated _shares storage provider_ that is using the _share manager_ for persistence. The _shares storage provider_ would provide a `/shares` namespace outside of `/home` that lists all accepted shares for the current user. As a result the storage drivers no longer need to have a `enable_home` flag that jails users into their home. The `/home/shares` folder would move outside of the `/home`. In fact `/home` will no longer be needed, because the home folder concept can be implemented as a space: `CreateHome` would create a `personal` space on the. + +Work on this is done in https://github.com/cs3org/reva/pull/2023 + +[WARNING] +==== +What about copy pasting links from the browser? Well this storage is only really needed to have a path to ocm shares that actually reside on other instances. In the UI the shares would be listed by querying a _share manager_. It returns ResourceIds, which can be stated to fetch a path that is then accessible in the CS3 global namespace. Two caveats: +- This only works for resources that are actually hosted by the current instance. For those it would leak the parent path segments to a shared resource. +- For accepted OCM shares there must be a path in the xref:./namespaces.md#cs3-global-namespaces.adoc[_CS3 global namespace_] that has to be the same for all users, otherwise they cannot copy and share those URLs. +==== + + +=== The gateway should be responsible for path transformations + +Currently, storage providers are aware af their mount point, coupling them tightly with the gateway. + +Tracked in https://github.com/cs3org/reva/issues/578 + +Work is done in https://github.com/cs3org/reva/pull/1866 + +== URL escaped string representation of a CS3 reference + +For the spaces concept we introduced the `/dav/spaces/` endpoint. It encodes a cs3 _reference_ in a URL compatible way. +1. We can separate the path using a `/`: `/dav/spaces//` +2. The `spaceid` currently is a cs3 resourceid, consisting of `` and ``. Since the opaqueid might contain `/` e.g. for the local driver we have to urlencode the spaceid. + +To access resources by id we need to make the `/dav/meta/` able to list directories... Otherwise id based navigation first has to look up the path. Or we use the libregraph api for id based navigation. + +A _reference_ is a logical concept. It identifies a xref:#resources.adoc[_resource_] and consists of a `` and a ``. A `` consists of a `` and a ``. They can be concatenated using the separators `!` and `:`: +---- +!: +---- +While all components are optional, only three cases are used: +| format | example | description | +|-|-|-| +| `!:` | `!:/absolute/path/to/file.ext` | absolute path | +| `!:` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:path/to/file.ext` | path relative to the root of the storage space | +| `!:` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!c3cf23bb-8f47-4719-a150-1d25a1f6fb56:to/file.ext` | path relative to the specified node in the storage space, used to reference resources without disclosing parent paths | + +`` should be a UUID to prevent references from breaking when a _user_ or xref:#storage-spaces.adoc[_storage space_] gets renamed. But it can also be derived from a migration of an oc10 instance by concatenating an instance identifier and the numeric storage id from oc10, e.g. `oc10-instance-a$1234`. + +A reference will often start as an absolute/global path, e.g. `!:/home/Projects/Foo`. The gateway will look up the storage provider that is responsible for the path + +| Name | Description | Who resolves it? | +|------|-------------|-| +| `!:/home/Projects/Foo` | the absolute path a client like davfs will use. | The gateway uses the storage registry to look up the responsible storage provider | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:/Projects/Foo` | the `storage_space` is the same as the `root`, the path becomes relative to the root | the storage provider can use this reference to identify this resource | + +Now, the same file is accessed as a share +| Name | Description | +|------|-------------| +| `!:/users/Einstein/Projects/Foo` | `Foo` is the shared folder | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a` is the id of `Foo`, the path is empty | + + +The `:`, `!` and `$` are chosen from the set of link:https://tools.ietf.org/html/rfc3986#section-2.2[RFC3986 sub delimiters] on purpose. They can be used in URLs without having to be encoded. In some cases, a delimiter can be left out if a component is not set: +| reference | interpretation | +|-|-| +| `/absolute/path/to/file.ext` | absolute path, all delimiters omitted | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!path/to/file.ext` | relative path in the given storage space, root delimiter `:` omitted | +| `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:to/file.ext` | relative path in the given root node, storage space delimiter `!` omitted | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | node id in the given storage space, `:` must be present | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62` | root of the storage space, all delimiters omitted, can be distinguished by the `/` | + +== space providers +When looking up an id based resource the reference must use a logical space id, not a CS3 resource id. Otherwise id based requests, which only have a resourceid consisting of a storage id and a node id cannot be routed to the correct storage provider if the storage has moved from one storage provider to another. + +if the registry routes based on the storageid AND the nodeid it has to keep a cache of all nodeids in order to route all requests for a storage space (which consists of storage it + nodeid) to the correct storage provider. the correct resourceid for a node in a storage space would be `$!`. The `$` part allow the storage registry to route all id based requests to the correct storage provider. This becomes relevant when the storage space was moved from one storage provider to another. The storage space id remains the same, but the internal address and port change. + +TODO discuss to clarify further + +== Storage drivers + +=== allow clients to send a uuid on upload +iOS clients can only queue single requests to be executed in the background. They queue an upload and need to be able to identify the uploaded file after it has been uploaded to the server. The disconnected nature of the connection might cause workflows or manual user interaction with the file on the server to move the file to a different place or changing the content while the device is offline. However, on the device users might have marked the file as favorite or added it to other iOS specific collections. To be able to reliably identify the file the client can generate a `uuid` and attach it to the file metadata during the upload. While it is not necessary to look up files by this `uuid` having a second file id that serves exactly the same purpose as the `file id` is redundant. + +Another aspect for the `file id` / `uuid` is that it must be a logical identifier that can be set, at least by internal systems. Without a writeable fileid we cannot restore backups or migrate storage spaces from one storage provider to another storage provider. + +Technically, this means that every storage driver needs to have a map of a `uuid` to an internal resource identifier. This internal resource identifier can be +- an eos fileid, because eos can look up files by id +- an inode if the filesystem and the storage driver support looking up by inode +- a path if the storage driver has no way of looking up files by id. + - In this case other mechanisms like inotify, kernel audit or a fuse overlay might be used to keep the paths up to date. + - to prevent excessive writes when deep folders are renamed a reverse map might be used: it will map the `uuid` to `:`, in order to trade writes for reads + - as a fallback a sync job can read the file id from the metadata of the resources and populate the uuid to internal id map. + +The TUS upload can take metadata, for PUT we might need a header. + +=== Space id vs resource id vs storage id + +We have `/dav/meta/` where the `fileid` is a string that was returned by a PROPFIND or by the `/graph/v1.0/me/drives/` endpoint? That returns a space id and the root drive item which has an `id` + +Does that `id` have a specific format? We currently concatenate as `!`. + +A request against `/dav/meta/fileid` will use the reva storage registry to look up a path. + +What if the storage space is moved to another storage provider. This happens during a migration: + +1. the current oc10 fileids need to be prefixed with at least the numeric storage id to shard them. + +`123` becomes `instanceprefix$345!123` if we use a custom prefix that identifies an instance (so we can merge multiple instances into one ocis instance) and append the numeric storageid `345`. The pattern is `$!`. + +Every `$` identifies a space. + +- [ ] the owncloudsql driver can return these spaceids when listing spaces. + +Why does it not work if we just use the fileid of the root node in the db? + +Say we have a space with three resources: +`$!` +`instanceprefix$345!1` +`instanceprefix$345!2` +`instanceprefix$345!3` + +All users have moved to ocis and the registry contains a regex to route all `instanceprefix.*` references to the storageprovider with the owncloudsql driver. It is up to the driver to locate the correct resource by using the filecache table. In this case the numeric storage id is unnecessary. + +Now we migrate the space `345` to another storage driver: +- the storage registry contains a new entry for `instanceprefix$345` to send all resource ids for that space to the new storage provider +- the new storage driver has to take into account the full storageid because the nodeid may only be unique per storage space. + +If we now have to fetch the path on the `/dav/meta/` endpoint: +`/dav/meta/instanceprefix$345!1` +`/dav/meta/instanceprefix$345!2` +`/dav/meta/instanceprefix$345!3` + +This would work because the registry always sees `instanceprefix$345` as the storageid. + +Now if we use the fileids directly and leave out the numeric storageid: +`!` +`instanceprefix!1` +`instanceprefix!2` +`instanceprefix!3` + +This is the current `!` format. + +The reva storage registry contains a `instanceid` entry pointing to the storage provider with the owncloudsql driver. + +Resources can be looked up because the oc_filecache has a unique fileid over all storages. + +Now we again migrate the space `345` to another storage driver: +- the storage registry contains a new entry for `instanceprefix!1` so the storage space root now points to the new storage provider +- The registry needs to be aware of node ids to route properly. This is a no-go. We don't want to keep a cache of _all_ nodeids in the registry. Only the root nodes of spaces. +- The new storage driver only has a nodeid which might collide with other nodeids from other storage spaces, e.g. when two instances are imported into one ocis instance. Although it would be possible to just set up two storage providers extra care would have to be taken to prevent nodeid collisions when importing a space. + +If we now have to fetch the path on the `/dav/meta/` endpoint: +`/dav/meta/instanceprefix!1` would work because it is the root of a space +`/dav/meta/instanceprefix!2` would cause the gateway to poll all storage providers because the registry has no way to determine the responsible storage provider +`/dav/meta/instanceprefix!3` same + +The problem is that without a part in the storageid that allows differentiating storage spaces we cannot route them individually. + +Now, we could use the nodeid of the root of a storage space as the spaceid ... if it is a uuid. If it is numeric it needs a prefix to distinguish it from other spaces. +`!` would be easy for the decomposedfs. +eos might use numeric ids: `$!`, but it needs a custom prefix to distinguish multiple eos instances. + +Furthermore, when migrating spaces between storage providers we want to stay collision free, which is why we should recommend uuids. + +All this has implications for the decomposedfs, because it needs to split the nodes per space to prevent them from colliding. diff --git a/modules/developer/pages/ocis/storage/spaces.adoc b/modules/developer/pages/ocis/storage/spaces.adoc new file mode 100644 index 00000000..fe2755a0 --- /dev/null +++ b/modules/developer/pages/ocis/storage/spaces.adoc @@ -0,0 +1,212 @@ += Spaces +:toc: right +:toclevels: 3 + + +== Editing a Storage Space + +The OData specification allows for a mirage of ways of addressing an entity. We will support addressing a Drive entity by its unique identifier, which is the one the graph-api returns when listing spaces, and its format is: + +[source,json] +---- +{ + "id": "1284d238-aa92-42ce-bdc4-0b0000009157!b6e2c9cc-9dbe-42f0-b522-4f2d3e175e9c" +} +---- + +This is an extract of an element of the list spaces response. An entire object has the following shape: + +[source,json] +---- +{ + "driveType": "project", + "id": "1284d238-aa92-42ce-bdc4-0b0000009157!b6e2c9cc-9dbe-42f0-b522-4f2d3e175e9c", + "lastModifiedDateTime": "2021-10-07T11:06:43.245418+02:00", + "name": "marketing", + "owner": { + "user": { + "id": "ddc2004c-0977-11eb-9d3f-a793888cd0f8" + } + }, + "quota": { + "total": 65536 + }, + "root": { + "id": "1284d238-aa92-42ce-bdc4-0b0000009157!b6e2c9cc-9dbe-42f0-b522-4f2d3e175e9c", + "webDavUrl": "https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!b6e2c9cc-9dbe-42f0-b522-4f2d3e175e9c" + } +} +---- + +=== Updating a space property + +Having introduced the above, one can refer to a Drive with the following URL format: + +[source,console] +---- +'https://localhost:9200/graph/v1.0/drives/1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570 +---- + +Updating an entity attribute: + +[source,console] +---- +curl -X PATCH 'https://localhost:9200/graph/v1.0/drives/1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570' -d '{"name":"42"}' -v +---- + +The previous URL resource path segment (`1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570`) is parsed and handed over to the storage registry in order to apply the patch changes in the body, in this case update the space name attribute to `42`. Since space names are not unique we only support addressing them by their unique identifiers, any other query would render too ambiguous and explode in complexity. + + +=== Updating a space description + +Since every space is the root of a webdav directory, following some conventions we can make use of this to set a default storage description and image. In order to do so, every space is created with a hidden `.space` folder at its root, which can be used to store such data. + +[source,curl] +---- +curl -k -X PUT https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157\!07c26b3a-9944-4f2b-ab33-b0b326fc7570/.space/description.md -d "Add a description to your spaces" -u admin:admin +---- + +Verify the description was updated: + +[source,curl] +---- +❯ curl -k https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157\!07c26b3a-9944-4f2b-ab33-b0b326fc7570/.space/description.md -u admin:admin +Add a description to your spaces +---- + +This feature makes use of the internal storage layout and is completely abstracted from the end user. + +=== Quotas + +Spaces capacity (quota) is independent of the Storage quota. As a Space admin you can set the quota for all users of a space, and as such, there are no limitations and is up to the admin to make a correct use of this. + +It is possible to have a space quota greater than the storage quota. A Space may also have "infinite" quota, meaning a single space without quota can occupy the entirety of a disk. + +==== Quota Enforcement + +Creating a Space with a quota of 10 bytes: + +`curl -k -X POST 'https://localhost:9200/graph/v1.0/drives' -u admin:admin -d '{"name":"marketing", "quota": {"total": 10}}' -v` + +[source,console] +---- +/var/tmp/ocis/storage/users +├── blobs +├── nodes +│   ├── 627981c2-2a71-4adf-b680-177e245afdda +│   ├── 9541e7c3-8fda-4b49-b697-e7e51457cf5a +│   ├── b5692345-108d-4b80-9747-3a7e9739ad57 +│   └── root +│   ├── 118351d7-67a4-4cdf-b495-6093d1e572ed -> ../627981c2-2a71-4adf-b680-177e245afdda +│   └── ddc2004c-0977-11eb-9d3f-a793888cd0f8 -> ../b5692345-108d-4b80-9747-3a7e9739ad57 +├── spaces +│   ├── personal +│   │   └── b5692345-108d-4b80-9747-3a7e9739ad57 -> ../../nodes/b5692345-108d-4b80-9747-3a7e9739ad57 +│   ├── project +│   │   └── 627981c2-2a71-4adf-b680-177e245afdda -> ../../nodes/627981c2-2a71-4adf-b680-177e245afdda +│   └── share +├── trash +└── uploads +---- + +Verify the new space has 10 bytes, and none of it is used: + +[source,json] +---- +{ + "driveType": "project", + "id": "1284d238-aa92-42ce-bdc4-0b0000009157!627981c2-2a71-4adf-b680-177e245afdda", + "lastModifiedDateTime": "2021-10-15T11:16:26.029188+02:00", + "name": "marketing", + "owner": { + "user": { + "id": "ddc2004c-0977-11eb-9d3f-a793888cd0f8" + } + }, + "quota": { + "remaining": 10, + "total": 10, + "used": 0 + }, + "root": { + "id": "1284d238-aa92-42ce-bdc4-0b0000009157!627981c2-2a71-4adf-b680-177e245afdda", + "webDavUrl": "https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157!627981c2-2a71-4adf-b680-177e245afdda" + } +} +---- + +Upload a 6 bytes file: + +`curl -k -X PUT https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157\!627981c2-2a71-4adf-b680-177e245afdda/6bytes.txt -d "012345" -u admin:admin -v` + +Query the quota again: + +[source,json] +---- +{ + "quota": { + "remaining": 4, + "total": 10, + "used": 6 + } +} +---- + +Now attempt to upload 5 bytes to the space: + +`curl -k -X PUT https://localhost:9200/dav/spaces/1284d238-aa92-42ce-bdc4-0b0000009157\!627981c2-2a71-4adf-b680-177e245afdda/5bytes.txt -d "01234" -u admin:admin -v` + +The request will fail with `507 Insufficient Storage`: + +---- + HTTP/1.1 507 Insufficient Storage +< Access-Control-Allow-Origin: * +< Content-Length: 0 +< Content-Security-Policy: default-src 'none'; +< Date: Fri, 15 Oct 2021 09:24:46 GMT +< Vary: Origin +< X-Content-Type-Options: nosniff +< X-Download-Options: noopen +< X-Frame-Options: SAMEORIGIN +< X-Permitted-Cross-Domain-Policies: none +< X-Robots-Tag: none +< +* Connection #0 to host localhost left intact +* Closing connection 0 +---- + +===== Considerations + +- If a Space quota is updated to unlimited, the upper limit is the entire available space on disk +[WARNING] +==== + +The current implementation in oCIS might not yet fully reflect this concept. Feel free to add links to ADRs, PRs and Issues in short warning boxes like this. + +==== + + +== Storage Spaces +A storage _space_ is a logical concept. It organizes a set of xref:#resources.adoc[_resources_] in a hierarchical tree. It has a single _owner_ (_user_ or _group_), +a _quota_, _permissions_ and is identified by a `storage space id`. + +image::ocis/storage/storagespace.drawio.svg[] + +Examples would be every user's personal storage _space_, project storage _spaces_ or group storage _spaces_. While they all serve different purposes and may or may not have workflows like antivirus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A storage xref:./spacesregistry.adoc[_Spaces Registry_] then allows listing the capabilities of storage _spaces_, e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ... + +Finally, a logical `storage space id` is not tied to a specific xref:./spacesprovider.adoc[_spaces provider_]. If the xref:./storagedrivers.adoc[_storage driver_] supports it, we can import existing files including their `file id`, which makes it possible to move storage _spaces_ between xref:./spacesprovider.adoc[_spaces providers_] to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs. + +== Shares +_To be clarified: we are aware that xref:#storage-spaces.adoc[_storage spaces_] may be too 'heavyweight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like storage xref:#storage-spaces.adoc[_spaces_] that users can provision themselves. They would share the quota with the users home or personal storage xref:#storage-spaces.adoc[_space_] and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new storage xref:#storage-spaces.adoc[_space_] would be the same. This obviously also extends to user shares and even file individual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album._ + +== Notes + +We can implement link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ListStorageSpacesRequest[ListStorageSpaces] by either +- iterating over the root of the storage and treating every folder following the `` as a `home` _storage space_, +- iterating over the root of the storage and treating every folder following a new `` as a `project` _storage space_, or +- iterating over the root of the storage and treating every folder following a generic `` as a _storage space_ for a configurable space type, or +- we allow configuring a map of `space type` to `layout` (based on the link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.CreateStorageSpaceRequest[CreateStorageSpaceRequest]) which would allow things like +---- +home=/var/lib/ocis/storage/home/{{substr 0 1 .Owner.Username}}/{{.Owner.Username}} +spaces=/spaces/var/lib/ocis/storage/projects/{{.Name}} +---- diff --git a/modules/developer/pages/ocis/storage/spacesprovider.adoc b/modules/developer/pages/ocis/storage/spacesprovider.adoc new file mode 100644 index 00000000..e40eea2d --- /dev/null +++ b/modules/developer/pages/ocis/storage/spacesprovider.adoc @@ -0,0 +1,111 @@ += Spaces Provider +:toc: right +:toclevels: 3 + +[WARNING] +==== + +The current implementation in oCIS might not yet fully reflect this concept. Feel free to add links to ADRs, PRs and Issues in short warning boxes like this. + +==== + + +== Spaces Provider +A _storage provider_ manages xref:#resources.adoc[_resources_] identified by a xref:#references.adoc[_reference_] +by accessing a xref:#storage-systems.adoc[_storage system_] with a xref:./storagedrivers.adoc[_storage driver_]. + +image::ocis/storage/spacesprovider.drawio.svg[] + + +== Frontend + +The oCIS frontend service starts all services that handle incoming HTTP requests: +- _ocdav_ for ownCloud flavoured WebDAV +- _ocs_ for sharing, user provisioning, capabilities and other OCS API endpoints +- _datagateway_ for up and downloads +- TODO: _ocm_ + +image::ocis/storage/frontend.drawio.svg[] + +=== WebDAV + +The ocdav service not only handles all WebDAV requests under `(remote.php/)(web)dav` but also some other legacy endpoints like `status.php`: + +| endpoint | service | CS3 api | CS3 namespace | description | TODO | +|----------|---------|-------------|------|------|------| +| _ownCloud 10 / current ocis setup:_ ||||| +| `status.php` | ocdav | - | - | currently static | should return compiled version and dynamic values | +| `(remote.php/)webdav` | ocdav | storageprovider | `/home` | the old webdav endpoint | | +| `(remote.php/)dav/files/` | ocdav | storageprovider | `/home` | the new webdav endpoint | | +| `(remote.php/)dav/meta//v` | ocdav | storageprovider | id based | versions | | +| `(remote.php/)dav/trash-bin/` | ocdav | recycle | - | trash | should aggregate the trash of xref:./terminology.md#storage-spaces.adoc[_storage spaces_] the user has access to | +| `(remote.php/)dav/public-files/` | ocdav | storageprovider | `/public/` | public links | | +| `(remote.php/)dav/avatars/` | ocdav | - | - | avatars, hardcoded | look up from user provider and cache | +| _CernBox setup:_ ||||| +| `(remote.php/)webdav` | ocdav | storageprovider | `/` | | | +| _Note: existing folder sync pairs in legacy clients will break when moving the user home down in the path hierarchy_ ||||| +| `(remote.php/)webdav/home` | ocdav | storageprovider | `/home` | | | +| `(remote.php/)webdav/users` | ocdav | storageprovider | `/users` | | | +| `(remote.php/)dav/files/` | ocdav | storageprovider | `/users/` | | | +| _Spaces concept also needs a new endpoint:_ ||||| +| `(remote.php/)dav/spaces//` | ocdav | storageregistry & storageprovider | bypass path based namespace and directly talk to the responsible storage provider using a relative path | link:https://github.com/owncloud/ocis/pull/1827[spaces concept] needs to point to storage xref:./spaces.adoc[_spaces_] | allow accessing spaces, listing is done by the graph api | + + +The correct endpoint for a users home storage xref:./spaces.adoc[_space_] in oc10 is `remote.php/dav/files/`. In oc10 all requests at this endpoint use a path based reference that is relative to the users home. In oCIS this can be configured and defaults to `/home` as well. Other API endpoints like ocs and the web UI still expect this to be the users home. + +In oc10 we originally had `remote.php/webdav` which would render the current users home xref:./terminology.md#storage-spaces.adoc[_storage space_]. The early versions (pre OC7) would jail all received shares into a `remote.php/webdav/shares` subfolder. The semantics for syncing such a folder are link:https://github.com/owncloud/core/issues/5349[not trivially predictable], which is why we made shares link:https://github.com/owncloud/core/pull/8026[freely mountable] anywhere in the users home. + +The current reva implementation jails shares into a `remote.php/webdav/Shares` folder for performance reasons. Obviously, this brings back the link:https://github.com/owncloud/product/issues/7[special semantics for syncing]. In the future we will follow link:https://github.com/owncloud/product/issues/302[a different solution] and jail the received shares into a dedicated `/shares` space, on the same level as `/home` and `/spaces`. We will add a dedicated link:https://github.com/owncloud/ocis/pull/1827[API to list all _storage spaces_] a user has access to and where they are mounted in the users _namespace_. + +[WARNING] +==== +TODO rewrite this hint with `/dav/spaces` +Existing folder sync pairs in legacy clients will break when moving the user home down in the path hierarchy like CernBox did. +For legacy clients the `remote.php/webdav` endpoint will no longer list the users home directly, but instead present the different types of storage spaces: +- `remote.php/webdav/home`: the users home is pushed down into a new `home` xref:./terminology.md#storage-spaces.adoc[_storage space_] +- `remote.php/webdav/shares`: all mounted shares will be moved to a new `shares` xref:./terminology.md#storage-spaces.adoc[_storage space_] +- `remote.php/webdav/spaces`: other xref:./terminology.md#storage-spaces.adoc[_storage spaces_] the user has access to, e.g. group or project drives +==== + + + +=== Sharing + +The link:https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html[OCS Share API] endpoint `/ocs/v1.php/apps/files_sharing/api/v1/shares` returns shares, which have their own share id and reference files using a path relative to the users home. They API also lists the numeric storage id as well as the string type `storage_id` (which is confusing ... but yeah) which would allow constructing combined references with a `storage space id` and a `path` relative to the root of that xref:./terminology.md#storage-spaces.adoc[_storage space_]. The web UI however assumes that it can take the path from the `file_target` and append it to the users home to access it. + +// SHORTCODE: {{< hint >}} +The API link:https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html#example-request-response-payloads-4[already returns the storage id] (and numeric id) in addition to the file id: +---- + home::auser + 993 + 3994486 + 3994486 + 3994485 + /Shared/Paris.jpg +---- +link:https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html#function-arguments[Creating shares only takes the _path_ as the argument] so creating and navigating shares only needs the path. When you update or delete a share it takes the `share id` not the `file id`. +// SHORTCODE: {{< /hint >}} + +The OCS service makes a stat request to the storage provider to get a link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceInfo[ResourceInfo] object. It contains both, a link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId[`ResourceId`] _and_ an absolute path. If the _resource_ exists a request is sent to the gateway. Depending on the type of share the link:https://cs3org.github.io/cs3apis/#cs3.sharing.collaboration.v1beta1.CollaborationAPI[Collaboration API], the link:https://cs3org.github.io/cs3apis/#cs3.sharing.link.v1beta1.LinkAPI[Link API] or the link:https://cs3org.github.io/cs3apis/#cs3.sharing.ocm.v1beta1.OcmAPI[Open Cloud Mesh API] endpoints are used. + +| API | Request | Resource identified by | Grant type | Further arguments | +|-----|---------|------------------------|------------|-------------------| +| Collaboration | link:https://cs3org.github.io/cs3apis/#cs3.sharing.collaboration.v1beta1.CreateShareRequest[CreateShareRequest] | link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceInfo[ResourceInfo] | link:https://cs3org.github.io/cs3apis/#cs3.sharing.collaboration.v1beta1.ShareGrant[ShareGrant] | - | +| Link | link:https://cs3org.github.io/cs3apis/#cs3.sharing.link.v1beta1.CreatePublicShareRequest[CreatePublicShareRequest] | link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceInfo[ResourceInfo] | Link link:https://cs3org.github.io/cs3apis/#cs3.sharing.link.v1beta1.Grant[Grant] | We send the public link `name` in the `ArbitraryMetadata` of the `ResourceInfo` | +| Open Cloud Mesh | link:https://cs3org.github.io/cs3apis/#cs3.sharing.ocm.v1beta1.CreateOCMShareRequest[CreateOCMShareRequest] | link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId[ResourceId] | OCM link:https://cs3org.github.io/cs3apis/#cs3.sharing.ocm.v1beta1.ShareGrant[ShareGrant] | OCM link:https://cs3org.github.io/cs3apis/#cs3.ocm.provider.v1beta1.ProviderInfo[ProviderInfo] | + + +// SHORTCODE: {{< hint >}} +The user and public share provider implementations identify the file using the link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId[`ResourceId`]. The link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceInfo[`ResourceInfo`] is passed so the share provider can also store who the owner of the resource is. The _path_ is not part of the other API calls, e.g. when listing shares. +The OCM API takes an id based reference on the CS3 api, even if the OCM HTTP endpoint takes a path argument. _@jfd: Why? Does it not need the owner? It only stores the owner of the share, which is always the currently logged-in user, when creating a share. Afterwards only the owner can update a share ... so collaborative management of shares is not possible. At least for OCM shares._ +// SHORTCODE: {{< /hint >}} + + +== REVA Storage Registry + +The reva _storage registry_ manages the xref:./namespaces.md#cs3-global-namespaces.adoc[_CS3 global namespace_]: +It is used by the reva _gateway_ +to look up `address` and `port` of the xref:#storage-providers.adoc[_storage provider_] +that should handle a xref:#references.adoc[_reference_]. + +image::ocis/storage/storageregistry.drawio.svg[] diff --git a/modules/developer/pages/ocis/storage/spacesregistry.adoc b/modules/developer/pages/ocis/storage/spacesregistry.adoc new file mode 100644 index 00000000..abdc962c --- /dev/null +++ b/modules/developer/pages/ocis/storage/spacesregistry.adoc @@ -0,0 +1,17 @@ += Spaces Registry +:toc: right +:toclevels: 3 + +[WARNING] +==== + +The current implementation in oCIS might not yet fully reflect this concept. Feel free to add links to ADRs, PRs and Issues in short warning boxes like this. + +==== + + +== Storage Space Registries + +A storage _spaces registry_ manages the xref:./namespaces.adoc[_namespace_] for a _user_: it is used by _clients_ to look up storage spaces a user has access to, the `/dav/spaces` endpoint to access it via WabDAV, and where the client should mount it in the users personal namespace. + +image::ocis/storage/spacesregistry.drawio.svg[] diff --git a/modules/developer/pages/ocis/storage/storagedrivers.adoc b/modules/developer/pages/ocis/storage/storagedrivers.adoc new file mode 100644 index 00000000..b70f9ea5 --- /dev/null +++ b/modules/developer/pages/ocis/storage/storagedrivers.adoc @@ -0,0 +1,203 @@ += Storage drivers +:toc: right +:toclevels: 3 + +A _storage driver_ implements access to a xref:#storage-systems.adoc[_storage system_]: + +It maps the _path_ and _id_ based CS3 _references_ to an appropriate xref:#storage-systems.adoc[_storage system_] specific reference, e.g.: +- eos file ids +- posix inodes or paths +- deconstructed filesystem nodes + +== Storage providers + +To manage the file tree oCIS uses _storage providers_ that are accessing the underlying storage using a _storage driver_. The driver can be used to change the implementation of a storage aspect to better reflect the actual underlying storage capabilities. As an example a move operation on a POSIX filesystem (link:https://danluu.com/deconstruct-files/[theoretically]) is an atomic operation. When trying to implement a file tree on top of S3 there is no native move operation that can be used. A naive implementation might fall back on a COPY and DELETE. Some S3 implementations provide a COPY operation that uses an existing key as the source, so the file at least does not need to be reuploaded. In the worst case scenario, which is renaming a folder with hundreds of thousands of objects, a reupload for every file has to be made. Instead of hiding this complexity a better choice might be to disable renaming of files or at least folders on S3. There are however implementations of filesystems on top of S3 that store the tree metadata in dedicated objects or use a completely different persistence mechanism like a distributed key value store to implement the file tree aspect of a storage. + + +[NOTE] +==== +While the _storage provider_ is responsible for managing the tree, file up- and downloads are delegated to a dedicated _data provider_. See below. +==== + + +== Storage aspects +A lot of different storage technologies exist, ranging from general purpose file systems with POSIX semantics to software defined storage with multiple APIs. Choosing any of them is making a tradeoff decision. Or, if a storage technology is already in place it automatically predetermines the capabilities that can be made available. _Not all storage systems are created equal._ + +Unfortunately, no POSIX filesystem natively supports all storage aspects that ownCloud 10 requires: + +=== A hierarchical file tree +An important aspect of a filesystem is organizing files and directories in a file hierarchy, or tree. It allows you to create, move and delete nodes. Beside the name a node also has well known metadata like size and mtime that are persisted in the tree as well. + +[NOTE] +==== +_Folders are not directories_ +There is a difference between _folder_ and _directory_: a _directory_ is a file system concept. A _folder_ is a metaphor for the concept of a physical file folder. There are also _virtual folders_ or _smart folders_ like the recent files folder which are no file system _directories_. So, every _directory_ and every _virtual folder_ is a _folder_, but not every _folder_ is a _directory_. See link:https://en.wikipedia.org/wiki/Directory_(computing[the folder metaphor in wikipedia]#Folder_metaphor). Also see the activity history below. +==== + + +==== Id based lookup +While traditionally nodes in the tree are reached by traversing the path the tree persistence should be prepared to look up a node by an id. Think of an inode in a POSIX filesystem. If this operation needs to be cached for performance reasons keep in mind that cache invalidation is hard and crawling all files to update the inode to path mapping takes O(n), not O(1). + +==== ETag propagation +For the state based sync a client can discover changes by recursively descending the tree and comparing the ETag for every node. If the storage technology supports propagating ETag changes up the tree, only the root node of a tree needs to be checked to determine if a discovery needs to be started and which nodes need to be traversed. This allows using the storage technology itself to persist all metadata that is necessary for sync, without additional services or caches. + +==== Subtree size accounting +The tree can keep track of how many bytes are stored in a folder. Similar to ETag propagation a change in file size is propagated up the hierarchy. + +[NOTE] +==== +_ETag and Size propagation_ +When propagating the ETag (mtime) and size changes up the tree the question is where to stop. If all changes need to be propagated to the root of a storage then the root or busy folders will become a hotspot. There are two things to keep in mind: 1. propagation only happens up to the root of a single space (a user private drive or a single group drive), 2. no cross storage propagation. The latter was used in oc10 to let clients detect when a file in a received shared folder changed. This functionality is moving to the storage registry which caches the ETag for every root so clients can discover if and which storage changed. +==== + + +==== Rename +Depending on the underlying storage technology some operations may either be slow, up to a point where it makes more sense to disable them entirely. One example is a folder rename: on S3 a _simple_ folder rename translates to a copy and delete operation for every child of the renamed folder. There is an exception though: this restriction only applies if the S3 storage is treated like a filesystem, where the keys are the path and the value is the file content. There are smarter ways to implement file systems on top of S3, but again: there is always a tradeoff. + +[NOTE] +==== +_S3 has no rename_ +Technically, link:https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3-objects.html#copy-object[S3 has no rename operation at all]. By design, the location of the value is determined by the key, so it always has to do a copy and delete. Another example is the link:https://redis.io/commands/rename[redis RENAME operation]: while being specified as O(1) it _executes an implicit DEL operation, so if the deleted key contains a very big value it may cause high latency..._ +==== + + +==== Arbitrary metadata persistence +In addition to well known metadata like name size and mtime, users might be able to add arbitrary metadata like tags, comments or link:https://en.wikipedia.org/wiki/Dublin_Core[dublin core]. In POSIX filesystems this maps to extended attributes. + +=== Grant persistence +The CS3 API uses grants to describe access permissions. Storage systems have a wide range of permissions granularity and not all grants may be supported by every storage driver. POSIX ACLs for example have no expiry. If the storage system does not support certain grant properties, e.g. expiry, then the storage driver may choose to implement them in a different way. Expiries could be persisted in a different way and checked periodically to remove the grants. Again: every decision is a tradeoff. + +=== Trash persistence +After deleting a node the storage allows listing the deleted nodes and has an undo mechanism for them. + +=== Versions persistence +A user can restore a previous version of a file. + +[NOTE] +==== +_Snapshots are not versions_ +Modern POSIX filesystems support snapshotting of volumes. This is different from keeping track of versions to a file or folder, but might be another implementation strategy for a storage driver to allow users to restore content. +==== + + +=== Activity History +The storage keeps an activity history, tracking the different actions that have been performed. This does not only include file changes but also metadata changes like renames and permission changes. + +== Storage drivers + +Reva currently has several storage driver implementations that can be used for _storage providers_ as well as _data providers_. + +=== OCIS and S3NG Storage Driver + +The oCIS storage driver is the default storage driver. It decomposes the metadata and persists it in a POSIX filesystem. Blobs are stored on the filesystem as well. The layout makes extensive use of symlinks and extended attributes. A filesystem like xfs or zfs without practical inode size limitations is recommended. We will evolve this to further integrate with file systems like cephfs or gpfs. + +[WARNING] +==== +Ext4 limits the number of bytes that can be used for extended attribute names and their values to the size of a single block (by default 4k). This reduces the number of shares for a single file or folder to roughly 20-30, as grants have to share the available space with other metadata. +==== + + +The S3NG storage driver uses the same metadata layout on a POSIX storage as the oCIS driver, but it uses S3 as the blob storage. + +==== Tradeoffs +➕ Efficient ID based lookup + +➕ Leverages Kernel VFS cache + +➕ No database needed + +➖ Not intended to be accessed by end users on the server side as it does not reflect a normal filesystem on disk + +➖ Metadata limited by Kernel VFS limits (see below) + +==== Related Kernel limits +The Decomposed FS currently stores CS3 grants in extended attributes. When listing extended attributes the result is currently limited to 64kB. Assuming a 20 byte uuid a grant has ~40 bytes. Which would limit the number of extended attributes to ~1630 entries or ~1600 shares. This can be extended by moving the grants from extended attributes into a dedicated file and is tracked in link:https://github.com/owncloud/ocis/issues/4638[ocis/issues/4638]. + +From link:https://en.wikipedia.org/wiki/Extended_file_attributes#Linux[Wikipedia on Extended file attributes]: +[quote] +____ +The Linux kernel allows extended attribute to have names of up to 255 bytes and values of up to 64 KiB,[14] as do XFS and ReiserFS, but ext2/3/4 and btrfs impose much smaller limits, requiring all the attributes (names and values) of one file to fit in one "filesystem block" (usually 4 KiB). Per POSIX.1e,[citation needed] the names are required to start with one of security, system, trusted, and user plus a period. This defines the four namespaces of extended attributes.[15] +____ + + +And from the link:https://www.man7.org/linux/man-pages/man2/listxattr.2.html[man page on listxattr]: +[quote] +____ +As noted in xattr(7), the VFS imposes a limit of 64 kB on the size of the extended attribute name list returned by listxattr(7). If the total size of attribute names attached to a file exceeds this limit, it is no longer possible to retrieve the list of attribute names. +____ + + +=== Local Storage Driver + +The _minimal_ storage driver for a POSIX based filesystem. It literally supports none of the storage aspect other than basic file tree management. Sharing can - to a degree - be implemented using POSIX ACLs. + +- tree provided by a POSIX filesystem + - inefficient path by id lookup, currently uses the file path as id, so ids are not stable + - can store a uuid in extended attributes and use a cache to look them up, similar to the ownCloud driver + - no native ETag propagation, five options are available: + - built in propagation (changes bypassing ocis are not picked up until a rescan) + - built in inotify (requires 48 bytes of RAM per file, needs to keep track of every file and folder) + - external inotify (same RAM requirement, but could be triggered by external tools, e.g. a workflow engine) + - kernel audit log (use the linux kernel audit to capture file events on the storage and offload them to a queue) + - fuse filesystem overlay + - no subtree accounting, same options as for ETag propagation + - efficient rename + - arbitrary metadata using extended attributes +- grant persistence + - using POSIX ACLs + - requires an LDAP server to make guest accounts available in the OS + - an existing LDAP could be used if guests ar provisioned in another way + - using extended attributes to implement expiry or sharing that does not require OS level integration + - fuse filesystem overlay +- no native trash + - could use link:https://specifications.freedesktop.org/trash-spec/trashspec-latest.html[The FreeDesktop.org Trash specification] + - fuse filesystem overlay +- no native versions, multiple options possible + - git for folders + - rcs for single files + - rsnapshot for hourly / daily / weekly / monthly backups ... but this is not versioning as known from oc10 + - design new freedesktop spec, basically what is done in oc10 without the limitations or borrow ideas from the freedesktop trash spec + - fuse filesystem overlay + +To provide the other storage aspects we plan to implement a FUSE overlay filesystem which will add the different aspects on top of local filesystems like ext4, btrfs or xfs. It should work on NFSv4 as well, although NFSv4 supports RichACLs and we will explore how to leverage them to implement sharing at a future date. The idea is to use the storages native capabilities to deliver the best user experience. But again: that means making the right tradeoffs. + +=== EOS Storage Driver + +The CERN eos storage has evolved with ownCloud and natively supports id based lookup, ETag propagation, subtree size accounting, sharing, trash and versions. To use it you need to change the default configuration of the `storage storage-home` command (or have a look at the Makefile ̀ eos-start` target): + +---- +export STORAGE_DRIVER_EOS_NAMESPACE=/eos +export STORAGE_DRIVER_EOS_MASTER_URL="root://eos-mgm1.eoscluster.cern.ch:1094" +export STORAGE_DRIVER_EOS_ENABLE_HOME=true +export STORAGE_DRIVER_EOS_LAYOUT="dockertest/{{.Username}}" +---- + +Running it locally also requires the `eos` and `xrootd` binaries. Running it using `make eos-start` will use CentOS based containers that already have the necessary packages installed. + +[NOTE] +==== +Pull requests to add explicit `storage storage-(s3|custom|...)` commands with working defaults are welcome. +==== + + +=== S3 Storage Driver + +A naive driver that treats the keys in an S3 capable storage as `/` delimited path names. While it does not support MOVE or ETag propagation it can be used to read and write files. Better integration with native capabilities like versioning is possible but depends on the Use Case. Several storage solutions that provide an S3 interface also support some form of notifications that can be used to implement ETag propagation. + +== Data Providers + +Clients using the CS3 API use an link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.InitiateFileDownloadRequest[InitiateFileDownload] and link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.InitiateFileUploadRequest[InitiateUpload] request at the link:https://cs3org.github.io/cs3apis/#cs3.gateway.v1beta1.GatewayAPI[storage gateway] to obtain a URL endpoint that can be used to either GET the file content or upload content using the resumable link:https://tus.io[tus.io] protocol. + +The _data provider_ uses the same _storage driver_ as the _storage provider_ but can be scaled independently. + +The dataprovider allows uploading the file to a quarantine area where further data analysis may happen before making the file accessible again. One use case for this is antivirus scanning for files coming from untrusted sources. + +== Future work + +=== FUSE overlay filesystem +We are planning to further separate the concerns and use a local storage provider with a FUSE filesystem overlaying the actual POSIX storage that can be used to capture deletes and writes that might happen outside of ocis/reva. + +It would allow us to extend the local storage driver with missing storage aspects while keeping a tree like filesystem that end users are used to see when sshing into the machine. + +=== Upload to Quarantine area +Antivirus scanning of random files uploaded from untrusted sources and executing metadata extraction or thumbnail generation should happen in a sandboxed system to prevent malicious users from gaining any information about the system. By spawning a new container with access to only the uploaded data we can further limit the attack surface. diff --git a/modules/developer/pages/ocis/storage/terminology.adoc b/modules/developer/pages/ocis/storage/terminology.adoc new file mode 100644 index 00000000..13f04869 --- /dev/null +++ b/modules/developer/pages/ocis/storage/terminology.adoc @@ -0,0 +1,99 @@ += Terminology +:toc: right +:toclevels: 3 + +Communication is hard. And clear communication is even harder. You may encounter the following terms throughout the documentation, in the code or when talking to other developers. Just keep in mind that whenever you hear or read _storage_, that term needs to be clarified, because on its own it is too vague. PR welcome. + +== Logical concepts + +=== Resources +A _resource_ is the basic building block that oCIS manages. It can be of link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceType[different types]: +- an actual _file_ +- a _container_, e.g. a folder or bucket +- a _symlink_, or +- a xref:#references.adoc[_reference_] which can point to a resource in another xref:#storage-providers.adoc[_storage provider_] + +=== References +A _reference_ identifies a xref:#resources.adoc[_resource_]. A link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Reference[_CS3 reference_] can carry a _path_ and a link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId[CS3 _resource id_]. The references come in two flavors: absolute and combined. +Absolute references have either the _path_ or the _resource id_ set: +- An absolute _path_ MUST start with a `/`. The _resource id_ MUST be empty. +- An absolute _resource id_ uniquely identifies a xref:#resources.adoc[_resource_] and is used as a stable identifier for sharing. The _path_ MUST be empty. +Combined references have both, _path_ and _resource id_ set: +- the _resource id_ identifies the root xref:#resources.adoc[_resource_] +- the _path_ is relative to that root. It MUST start with `.` +== References + +A _reference_ is a logical concept that identifies a xref:#resources.adoc[_resource_]. A link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Reference[_CS3 reference_] consists of either +- a _path_ based reference, used to identify a xref:#resources.adoc[_resource_] in the xref:./namespaces.adoc[_namespace_] of a xref:#storage-providers.adoc[_storage provider_]. It must start with a `/`. +- a link:https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId[CS3 _id_ based reference], uniquely identifying a xref:#resources.adoc[_resource_] in the xref:./namespaces.adoc[_namespace_] of a xref:#storage-providers.adoc[_storage provider_]. It consists of a `storage provider id` and an `opaque id`. The `storage provider id` must NOT start with a `/`. + +[NOTE] +==== +The `/` is important because currently the static xref:#storage-space-registries.adoc[_storage registry_] uses a map to look up which xref:#storage-providers.adoc[_storage provider_] is responsible for the resource. Paths must be prefixed with `/` so there can be no collisions between paths and storage provider ids in the same map. +==== + + +== Storage Drivers + +A _storage driver_ implements access to a xref:#storage-systems.adoc[_storage system_]: + +It maps the _path_ and _id_ based CS3 _references_ to an appropriate xref:#storage-systems.adoc[_storage system_] specific reference, e.g.: +- eos file ids +- posix inodes or paths +- deconstructed filesystem nodes + +[WARNING] +==== +_Proposed Change_ +iOS clients can only queue single requests to be executed in the background. The queue an upload and need to be able to identify the uploaded file after it has been uploaded to the server. The disconnected nature of the connection might cause workflows or manual user interaction with the file on the server to move the file to a different place or changing the content while the device is offline. However, on the device users might have marked the file as favorite or added it to other iOS specific collections. To be able to reliably identify the file the client can generate a `uuid` and attach it to the file metadata during the upload. While it is not necessary to look up files by this `uuid` having a second file id that serves exactly the same purpose as the `file id` is redundant. + +Another aspect for the `file id` / `uuid` is that it must be a logical identifier that can be set, at least by internal systems. Without a writeable fileid we cannot restore backups or migrate storage spaces from one storage provider to another storage provider. + +Technically, this means that every storage driver needs to have a map of a `uuid` to in internal resource identifier. This internal resource identifier can be +- an eos fileid, because eos can look up files by id +- an inode if the filesystem and the storage driver support looking up by inode +- a path if the storage driver has no way of looking up files by id. + - In this case other mechanisms like inotify, kernel audit or a fuse overlay might be used to keep the paths up to date. + - to prevent excessive writes when deep folders are renamed a reverse map might be used: it will map the `uuid` to `:`, allowing to trade writes for reads + +==== + +== Storage Providers + +== Technical concepts + +=== Storage Systems +image::ocis/storage/storageprovider.drawio.svg[] + +A _storage provider_ manages multiple xref:#storage-space.adoc[_storage spaces_] +by accessing a xref:#storage-systems.adoc[_storage system_] with a xref:#storage-drivers.adoc[_storage driver_]. + +image::ocis/storage/storageprovider-spaces.drawio.svg[] + +== Storage Space Registries + +A xref:./spacesregistry.adoc[_storage spaces registry_] manages the xref:./namespaces.adoc[_namespace_] for a _user_ + +== Storage Spaces +A _storage space_ is a logical concept: +It is a tree of xref:#resources.adoc[_resources_]_resources_ +with a single _owner_ (_user_ or _group_), +a _quota_ and _permissions_, identified by a `storage space id`. + +image::ocis/storage/storagespace.drawio.svg[] + +Examples would be every user's home storage space, project storage spaces or group storage spaces. While they all serve different purposes and may or may not have workflows like antivirus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A xref:#storage-space-registries.adoc[_storage space registry_] then allows listing the capabilities of xref:#storage-spaces.adoc[_storage spaces_], e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ... + +Finally, a logical `storage space id` is not tied to a specific xref:#storage-providers.adoc[_storage provider_]. If the xref:#storage-drivers.adoc[_storage driver_] supports it, we can import existing files including their `file id`, which makes it possible to move xref:#storage-spaces.adoc[_storage spaces_] between xref:#storage-providers.adoc[_storage providers_] to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs. + +== Shares +_To be clarified: we are aware that xref:#storage-spaces.adoc[_storage spaces_] may be too 'heavyweight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like xref:#storage-spaces.adoc[_storage spaces_] that users can provision themselves. They would share the quota with the users home xref:#storage-spaces.adoc[_storage space_] and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new xref:#storage-spaces.adoc[_storage space_] would be the same. This obviously also extends to user shares and even file individual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album._ + + +== Storage Systems +Every _storage system_ has different native capabilities like id and path based lookups, recursive change time propagation, permissions, trash, versions, archival and more. +A xref:#storage-providers.adoc[_storage provider_] makes the storage system available in the CS3 API by wrapping the capabilities as good as possible using a xref:./storagedrivers.adoc[_storage driver_]. +There might be multiple xref:./storagedrivers.adoc[_storage drivers_] for a _storage system_, implementing different tradeoffs to match varying requirements. + +=== Gateways +A _gateway_ acts as a facade to the storage related services. It authenticates and forwards API calls that are publicly accessible. diff --git a/modules/developer/pages/ocis/storage/users.adoc b/modules/developer/pages/ocis/storage/users.adoc new file mode 100644 index 00000000..7c3eab74 --- /dev/null +++ b/modules/developer/pages/ocis/storage/users.adoc @@ -0,0 +1,61 @@ += Users +:toc: right +:toclevels: 3 + +TODO add this to the storage overview? or is this a different part? That should be started as a separate service ? And documented elsewhere, e.g. in the accounts? + +=== User and Group provisioning + +In oc10 users are identified by a username, which cannot change, because it is used as a foreign key in several tables. For oCIS we are internally identifying users by a UUID, while using the username in the WebDAV and OCS APIs for backwards compatability. To distinguish this in the URLs we are using `` instead of ``. You may have encountered ``, which refers to a template that can be configured to build several path segments by filling in user properties, e.g. the first character of the username (`{{substr 0 1 .Username}}/{{.Username}}`), the identity provider (`{{.Id.Idp}}/{{.Username}}`) or the email (`{{.Mail}}`) + +[WARNING] +==== +Make no mistake, the link:https://doc.owncloud.com/server/developer_manual/core/apis/provisioning-api.html[OCS Provisioning API] uses `userid` while it actually is the username, because it is what you use to login. +==== + + +We are currently working on adding link:https://github.com/owncloud/ocis/pull/1930[user management through the CS3 API] to handle user and group provisioning (and deprovisioning). + +=== Demo driver + +This is a simple user driver for testing. It contains three users: +---- +einstein:relativity +marie:radioactivity +richard:superfluidity +---- +In order to use the `demo` driver you need to export the relevant environment variable: +---- +export STORAGE_USERS_DRIVER=demo +---- + +=== JSON driver + +In order to switch from the `ldap` driver to JSON based users you need to export the relevant environment variables: +---- +export STORAGE_USERS_DRIVER=json +export STORAGE_USERS_JSON=/path/to/users.json +---- + +For the format of the users.json have a look at the link:https://github.com/cs3org/reva/blob/master/examples/oc-phoenix/users.demo.json[reva examples] + +=== LDAP driver + +This is the default user driver. + +If the following defaults don't match your environment then change them accordingly: +---- +export STORAGE_LDAP_HOSTNAME=localhost +export STORAGE_LDAP_PORT=9126 +export STORAGE_LDAP_BASE_DN='dc=ocis,dc=test' +export STORAGE_LDAP_USER_FILTER='(&(objectclass=posixAccount)(cn=%s))' +export STORAGE_LDAP_GROUP_FILTER='(&(objectclass=posixGroup)(cn=%s))' +export STORAGE_LDAP_BIND_DN='cn=reva,ou=sysusers,dc=ocis,dc=test' +export STORAGE_LDAP_BIND_PASSWORD=reva +export STORAGE_LDAP_USER_SCHEMA_UID=uid +export STORAGE_LDAP_USER_SCHEMA_MAIL=mail +export STORAGE_LDAP_USER_SCHEMA_DISPLAYNAME=sn +export STORAGE_LDAP_USER_SCHEMA_CN=cn +---- + +Then restart the `bin/storage users` and `bin/storage auth-basic` services for the changes to take effect. diff --git a/modules/developer/pages/ocis/storage_registry_discovery.adoc b/modules/developer/pages/ocis/storage_registry_discovery.adoc new file mode 100644 index 00000000..3fa16c5e --- /dev/null +++ b/modules/developer/pages/ocis/storage_registry_discovery.adoc @@ -0,0 +1,104 @@ += Storage Registry Discovery +:toc: right +:toclevels: 3 + +== Introduction + +In order for an oCIS client to access oCIS storage spaces for an End-User, the client needs to know where the oCIS instance is. oCIS uses WebFinger link:http://tools.ietf.org/html/rfc7033[RFC7033] to locate the oCIS instance for an End-User. + +This discovery is optional. If the client has another way of discovering the OpenID instance, e.g. when logging in with a username a static domain might be configured or the domain in the URL might be used. + +For guest accounts that do not have an OIDC issuer or whose IdP is not part of a trusted federation clients may fall back to a local IdP. + +== User Input using E-Mail Address Syntax + +To find the oCIS instance for the given user input in the form of an e-mail address `joe@example.com`, the WebFinger parameters are as follows: + +| WebFinger Parameter | Value | +|-|-| +| `resource` | `acct:joe@example.com` | +| `host` | `example.com` | +| `rel` | http://owncloud.com/specs/ocis/1.0/instance | + +Note that in this case, link:http://tools.ietf.org/html/draft-ietf-appsawg-acct-uri-07[the `acct:` scheme] is prepended to the identifier. + +The client (relying party) would make the following WebFinger request to discover the oCIS instance location (with line wraps within lines for display purposes only): + +---- + GET /.well-known/webfinger + ?resource=acct%3Ajoe%40example.com + &rel=http%3A%2F%2Fowncloud.com%2Fspecs%2Focis%2F1.0%2Finstance + HTTP/1.1 + Host: example.com + + HTTP/1.1 200 OK + Content-Type: application/jrd+json + + { + "subject": "acct:joe@example.com", + "links": + [ + { + "rel": "http://owncloud.com/specs/ocis/1.0/instance", + "href": "https://cloud.example.com" + } + ] + } +---- + +// SHORTCODE: {{< hint >}} +Note: the `example.com` domain is derived from the email. +// SHORTCODE: {{< /hint >}} + +[CAUTION] +==== +The `https://cloud.example.com` domain above would point to the ocis instance. +TODO that ins ocis web ... not the registry ... hmmmm +maybe introduce an ocis provider which then has an `/.well-known/ocis-configuration`, similar to `/.well-known/openid-configuration`? +It would contain +- the ocis domain, e.g. `https://cloud.example.com` +- the web endpoint, e.g. `https://cloud.example.com` +- the registry / drives endpoint, e.g. `https://cloud.example.com/graph/v0.1/drives/me` see link:https://github.com/owncloud/ocis/pull/1827[Add draft of adr for spaces API. #1827] + + +example: +---- +HTTP/1.1 200 OK + Content-Type: application/json + + { + "instance": "https://cloud.example.com", + "graph_endpoint": "https://cloud.example.com/graph/v0.0", + "ocis_web_config": "https://cloud.example.com/web/config.json", + "issuer": "https://idp.example.com", + } +---- + +`graph_endpoint` is the open-graph-api endpoint that is used to list storage spaces at e.g. `https://cloud.example.com/graph/v0.1/me/drives`. + +`ocis_web_config` points ocis web to the config for the instance. Maybe we can add more config in the `/.well-known/ocis-configuration` to replace the config.json? Is this the new status.php? How safe is it to expose all this info ...? + +The `issuer` could be used to detect the issuer that is used if no other issuer is found ... might be a fallback_issuer, but actually we may decide to skid the OIDC discovery and rely on this property. Maybe we need it if no IdP is present yet or the `/.well-known/openid-configuration` is not set up / reachable. + + +==== + + +== Obtaining oCIS Provider Configuration Information +Using the `instance` location discovered as described above or by other means, the oCIS Provider's configuration information can be retrieved. + +oCIS Providers supporting Discovery MUST make a JSON document available at the path formed by concatenating the string `/.well-known/openid-configuration` to the `instance`. The syntax and semantics of `.well-known` are defined in link:http://tools.ietf.org/html/rfc5785[RFC5785] and apply to the `instance` value when it contains no path component. `ocis-configuration` MUST point to a JSON document compliant with this specification and MUST be returned using the `application/json` content type. + +=== oCIS Provider Configuration Request + +An oCIS Provider Configuration Document MUST be queried using an HTTP GET request at the previously specified path. + +The client (relying party) would make the following request to the instance https://example.com to obtain its Configuration information, since the Issuer contains no path component: + + GET /.well-known/openid-configuration HTTP/1.1 + Host: example.com +If the Issuer value contains a path component, any terminating / MUST be removed before appending /.well-known/openid-configuration. The RP would make the following request to the Issuer https://example.com/issuer1 to obtain its Configuration information, since the Issuer contains a path component: + + GET /issuer1/.well-known/openid-configuration HTTP/1.1 + Host: example.com +Using path components enables supporting multiple issuers per host. This is required in some multi-tenant hosting configurations. This use of .well-known is for supporting multiple issuers per host; unlike its use in RFC 5785 [RFC5785], it does not provide general information about the host. \ No newline at end of file diff --git a/modules/developer/pages/service-independent/index.adoc b/modules/developer/pages/service-independent/index.adoc new file mode 100644 index 00000000..20706acb --- /dev/null +++ b/modules/developer/pages/service-independent/index.adoc @@ -0,0 +1,6 @@ += Service Independent Settings +:toc: right +:toclevels: 3 + + +Some components programmed are not bound to a service but are valid for all of ocis. These are for example environment variables that are used before any service gets started. For ocis CLI commands that are service independent, see the xref:../cli-commands/service_independent_cli.adoc[Service Independent CLI] documentation. diff --git a/modules/developer/pages/service-independent/service_independent_envvars.adoc b/modules/developer/pages/service-independent/service_independent_envvars.adoc new file mode 100644 index 00000000..b593102f --- /dev/null +++ b/modules/developer/pages/service-independent/service_independent_envvars.adoc @@ -0,0 +1,37 @@ += Service Independent Envvars +:toc: right +:toclevels: 3 + +The following environment variables are service independent. You will find the respective code in the `ocis-pkg/` directory. See the link:https://doc.owncloud.com/ocis/7.3/deployment/services/env-vars-special-scope.html[Admin Documentation - Environment Variables with Special Scopes] for a comprehensive list and explanation. + + +[NOTE] +==== +See the xref:../services/general-info/envvars/.adoc[Environment Variables] documentation for common and important details on envvars. +==== + + +== Service Registry + +This package configures the service registry which will be used to look up for example the service addresses. + +Available registries are: + +- nats-js-kv (default) +- memory + +To configure which registry to use, you have to set the environment variable `MICRO_REGISTRY`, and for all except `memory` you also have to set the registry address via `MICRO_REGISTRY_ADDRESS` and other envvars. + +== Startup Related Envvars + +These envvars define the startup of ocis and can for example add or remove services from the startup process such as `OCIS_ADD_RUN_SERVICES`. + +== Memory Limits + +[NOTE] +==== +Note that this envvar is for development purposes only and not described in the admin docs. +==== + + +oCIS will automatically set the go native `GOMEMLIMIT` to `0.9`. To disable the limit set `AUTOMEMLIMIT=off`. For more information take a look at the official link:https://go.dev/doc/gc-guide[Guide to the Go Garbage Collector]. diff --git a/modules/developer/pages/services/antivirus/configuration.adoc b/modules/developer/pages/services/antivirus/configuration.adoc new file mode 100644 index 00000000..9cbf78c5 --- /dev/null +++ b/modules/developer/pages/services/antivirus/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/antivirus-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/antivirus-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/antivirus_configvars.md diff --git a/modules/developer/pages/services/app-provider/configuration.adoc b/modules/developer/pages/services/app-provider/configuration.adoc new file mode 100644 index 00000000..0c6e4bb0 --- /dev/null +++ b/modules/developer/pages/services/app-provider/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/app-provider-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/app-provider-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/app-provider_configvars.md diff --git a/modules/developer/pages/services/app-provider/index.adoc b/modules/developer/pages/services/app-provider/index.adoc new file mode 100644 index 00000000..74d27518 --- /dev/null +++ b/modules/developer/pages/services/app-provider/index.adoc @@ -0,0 +1,9 @@ += App-Provider +:toc: right +:toclevels: 3 + +== Abstract + + +== Table of Contents + diff --git a/modules/developer/pages/services/app-registry/configuration.adoc b/modules/developer/pages/services/app-registry/configuration.adoc new file mode 100644 index 00000000..749e1ab4 --- /dev/null +++ b/modules/developer/pages/services/app-registry/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/app-registry-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/app-registry-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/app-registry_configvars.md diff --git a/modules/developer/pages/services/audit/configuration.adoc b/modules/developer/pages/services/audit/configuration.adoc new file mode 100644 index 00000000..9f8491d5 --- /dev/null +++ b/modules/developer/pages/services/audit/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/audit-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/audit-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/audit_configvars.md diff --git a/modules/developer/pages/services/auth-basic/configuration.adoc b/modules/developer/pages/services/auth-basic/configuration.adoc new file mode 100644 index 00000000..42e6ebdf --- /dev/null +++ b/modules/developer/pages/services/auth-basic/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/auth-basic-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/auth-basic-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/auth-basic_configvars.md diff --git a/modules/developer/pages/services/auth-bearer/configuration.adoc b/modules/developer/pages/services/auth-bearer/configuration.adoc new file mode 100644 index 00000000..dd0db6b3 --- /dev/null +++ b/modules/developer/pages/services/auth-bearer/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/auth-bearer-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/auth-bearer-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/auth-bearer_configvars.md diff --git a/modules/developer/pages/services/auth-machine/configuration.adoc b/modules/developer/pages/services/auth-machine/configuration.adoc new file mode 100644 index 00000000..f4daa999 --- /dev/null +++ b/modules/developer/pages/services/auth-machine/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/auth-machine-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/auth-machine-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/auth-machine_configvars.md diff --git a/modules/developer/pages/services/eventhistory/configuration.adoc b/modules/developer/pages/services/eventhistory/configuration.adoc new file mode 100644 index 00000000..3f289b45 --- /dev/null +++ b/modules/developer/pages/services/eventhistory/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/eventhistory-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/eventhistory-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/eventhistory_configvars.md diff --git a/modules/developer/pages/services/frontend/configuration.adoc b/modules/developer/pages/services/frontend/configuration.adoc new file mode 100644 index 00000000..25b90b7d --- /dev/null +++ b/modules/developer/pages/services/frontend/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/frontend-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/frontend-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/frontend_configvars.md diff --git a/modules/developer/pages/services/gateway/configuration.adoc b/modules/developer/pages/services/gateway/configuration.adoc new file mode 100644 index 00000000..18546f3d --- /dev/null +++ b/modules/developer/pages/services/gateway/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/gateway-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/gateway-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/gateway_configvars.md diff --git a/modules/developer/pages/services/general-info/add-translations.adoc b/modules/developer/pages/services/general-info/add-translations.adoc new file mode 100644 index 00000000..b44b99fa --- /dev/null +++ b/modules/developer/pages/services/general-info/add-translations.adoc @@ -0,0 +1,107 @@ += Add Translations +:toc: right +:toclevels: 3 + +Services can have texts that need to be translated. These translations will be shown in the ownCloud Web UI. Compared to web, these translations: + +* Are located in the same Transifex project as the web, which is named link:https://app.transifex.com/owncloud-org/owncloud-web/translate/[ownCloud Web]. +* Have a name starting with `ocis-` for ease of identification. + +The process for _synchronisation_ with Transifex is already setup and nothing needs to be done here. For any translation, it is necessary to set it up in the respective service and tell to sync it. + +_IMPORTANT_\ +Translations are automatically synced on a daily basis in the night. To do so, there is an own repo that covers the process for ALL translations from all configured repos: link:https://github.com/owncloud/translation-sync[translation-sync]. If there is a manual "emergency" sync necessary, you only need to trigger link:https://drone.owncloud.com/owncloud/translation-sync[drone] via cli + +[source,bash] +---- +drone cron exec owncloud/translation-sync nightly +---- +Note that you need to be logged on in drone to execute the command. + +== Implementing ocis Translations + +The implementation example is a guide and shall show how to do it. You can derive at any time according to your needs. + +Note that paths are examples and can be adapted based on requirements.\ +Replace `` with the name of the respective service.\ +Translations have a `context` and a `translatable string`. The context is shown on Transifex but not translated and helps translators to get a context for the string to be translated. + +* Add the `OCIS_DEFAULT_LANGUAGE` envvar in `services//pkg/config/config.go`.\ + For details see the userlog or notifications service code. + +* Add the `_TRANSLATION_PATH` envvar in `services//pkg/config/config.go`.\ + For details see the userlog or notifications service code. + +* Use `"github.com/owncloud/ocis/v2/ocis-pkg/l10n"` for the translation.\ + Use `l10n.Template` to define the translation string.\ + Use `l10n.NewTranslator` or `l10n.NewTranslatorFromCommonConfig` to get the translator.\ + Use `t.Get` to translate the string. See package for more advanced usage. + +* Create a config in `services//pkg/service/l10n/.tx/config` with the following content. Note that it is important to stick with `ocis-` to easily identify all ocis translations on Transifex: + ``` + [main] + host = https://www.transifex.com + + [o:owncloud-org:p:owncloud-web:r:ocis-] + file_filter = locale//LC_MESSAGES/.po + minimum_perc = 75 + resource_name = ocis- + source_file = .pot + source_lang = en + type = PO + ``` + Note: o: organization, p: project, r: resource + +* Create an empty file `services//pkg/service/l10n/locale/en/LC_MESSAGES/.po`. This is required for ocis to build. This file will be replaced nightly with the latest translations from Transifex. + +* Create a go file like `templates.go` in e.g. `ocis/services//pkg/service` that will define your translation sources like the following: + ``` + // this comment will appear in transifex as context + var yourString = l10n.Template("Translation String") + ``` + +* In the `Makefile` in the _ocis root_, add in the following section the service you want to synchronize translations with Transifex: + ``` + # add a service here when it uses transifex + L10N_MODULES := \ + services/notifications \ + services/userlog \ + services/graph \ + services/activitylog \ + services/ + ``` + +* In the `Makefile` of the _``_ add:\ + _At the beginning:_ + ``` + # Where to write the files generated by this makefile. + OUTPUT_DIR = ./pkg/service/<...>/l10n + TEMPLATE_FILE = ./pkg/service/<...>/l10n/.pot + ``` + _In the `.PHONY` list:_ + ``` + ############ translations ######## + .PHONY: l10n-pull + l10n-pull: + cd $(OUTPUT_DIR) && tx pull --all --force --skip --minimum-perc=75 + + .PHONY: l10n-push + l10n-push: + cd $(OUTPUT_DIR) && tx push -s --skip + + .PHONY: l10n-read + l10n-read: $(GO_XGETTEXT) + go-xgettext -o $(OUTPUT_DIR)/.pot \ + --keyword=l10n.Template --add-comments -s \ + pkg/service/templates.go + + .PHONY: l10n-write + l10n-write: + + .PHONY: l10n-clean + l10n-clean: + rm -f $(TEMPLATE_FILE); + ``` + +* Add Description Text to README\ + Add the full `Translations` and `Default Language` text blocks including their sub sections to the service readme. You can derive from the `activitylog` or `userlog` service for easy copy/paste. diff --git a/modules/developer/pages/services/general-info/additional-information.adoc b/modules/developer/pages/services/general-info/additional-information.adoc new file mode 100644 index 00000000..876df8a3 --- /dev/null +++ b/modules/developer/pages/services/general-info/additional-information.adoc @@ -0,0 +1,13 @@ += Additional Information +:toc: right +:toclevels: 3 + +This section contains information on general topics + +== GRPC Maximum Message Size + +ocis is using grpc for inter-service communication. When having a folder with a lot of files (25.000+, the size does not matter) and doing a `PROPFIND` on that folder, the server will run into errors. This is because the grpc message body becomes to big. With introduction of the envvar `OCIS_GRPC_MAX_RECEIVED_MESSAGE_SIZE`, the max size for the grpc body can be raised. + +NOTE: With a certain amount of files even raising the grpc message size will not suffice as the requests will run into network timeouts. Also generally the more files are in a folder, the longer it will take time to load. + +It is recommended to use `OCIS_GRPC_MAX_RECEIVED_MESSAGE_SIZE` only _temporary_ to copy files out of the folder (like via the web ui) and use the default value in general. diff --git a/modules/developer/pages/services/general-info/envvars/deprecating-variables.adoc b/modules/developer/pages/services/general-info/envvars/deprecating-variables.adoc new file mode 100644 index 00000000..6d27b578 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/deprecating-variables.adoc @@ -0,0 +1,57 @@ += Envvar Deprecation +:toc: right +:toclevels: 3 + + +== Deprecating Environment Variables + +Sometimes it is necessary to deprecate an environment variable to align the naming with conventions or remove it completely. We therefore added annotations to automate the _documentation_ process. + +The relevant annotations in the envvar struct tag are: + +* `deprecationVersion`\ + The release an envvar is announced for deprecation. +* `removalVersion`\ + The version it is finally going to be removed is defined via the mandatory placeholder `%%NEXT_PRODUCTION_VERSION%%`, not an actual version number. +* `deprecationInfo`\ + The reason why it was deprecated. +* `deprecationReplacement`\ + Only if it is going to be replaced, not necessary if removed. + +[WARNING] +==== +* During the development cycle, the value for the `removalVersion` must be set to `%%NEXT_PRODUCTION_VERSION%%`. This placeholder will be replaced by the real semantic-version number during the production releasing process. +* Compared when introducing new envvars where you can use arbitrary alphabetic identifyers, the string for deprecation is fixed and cannot be altered. +==== + + +For the documentation to show the correct value for the `removalVersion`, our docs helper scripts will automatically generate the correct version to be printed in the documentation. If `%%NEXT_PRODUCTION_VERSION%%` is found in the query, it will be replaced with `next-prod`, else the value found is used. + +=== Example + +[source,golang] +---- +// Notifications defines the config options for the notifications service. +type Notifications struct { +RevaGateway string `yaml:"reva_gateway" env:"OCIS_REVA_GATEWAY;REVA_GATEWAY" desc:"CS3 gateway used to look up user metadata" deprecationVersion:"3.0" removalVersion:"%%NEXT_PRODUCTION_VERSION%%" deprecationInfo:"REVA_GATEWAY changing name for consistency" deprecationReplacement:"OCIS_REVA_GATEWAY"` +... +} +---- + +There are four different annotation variables that need to be filled: + +| Annotation |Description| Format| +|---|---|---| +| deprecationVersion | The version the variable will be deprecated | semver (e.g. 3.0)| +| removalVersion | The version the variable will be removed from the codebase. Consider semver rules when finally removing a deprecated ennvar | `%%NEXT_PRODUCTION_VERSION%%` | +| deprecationInfo | Information why the variable is deprecated, must start with the name of the variable in order to avoid confusion, when there are multiple options in the `env:`-field | string (e.g. NATS_NATS_HOST is confusing) | +| deprecationReplacement | The name of the variable that is going to replace the deprecated one.| string (e.g. NATS_HOST_ADDRESS) | + +=== What Happens Next? + +To remove an environment variable, which needs to be done before the planned release has codefreeze: + +* Check if the envvar is also present in the REVA code and adapt if so. +* The envvar needs to be removed from any occurrences in the ocis code. +* The envvar must also be removed from the `docs/helpers/env_vars.yaml` file. This should be done in the same PR that removes the envvar from the code. +* Notify docs. The added/deprecated/removed envvar tables need to be created/updated. The release notes will get a note about this change. diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/4.0.0-5.0.0-added.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/4.0.0-5.0.0-added.adoc new file mode 100644 index 00000000..02a2ce28 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/4.0.0-5.0.0-added.adoc @@ -0,0 +1,266 @@ +:toc: right +:toclevels: 3 + += Added Variables between oCIS 4.0.0 and oCIS 5.0.0 + +| File | Variable | Description | Default | +|-----------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------| +| ocis-pkg/shared/shared_types.go | `OCIS_CACHE_DISABLE_PERSISTENCE` | Disables persistence of the cache. Only applies when store type 'nats-js-kv' is configured. | `false` | +| ocis-pkg/registry/registry.go | `MICRO_REGISTRY` (important change) | The Go micro registry type to use. Supported types are: 'memory', 'nats-js-kv' (default) and 'kubernetes'. Note that 'nats', 'etcd', 'consul' and 'mdns' are deprecated and will be removed in a later version. Only change on supervision of ownCloud Support. | `nats-js-kv` | +| ocis-pkg/natsjsregistry/registry.go | `MICRO_REGISTRY_AUTH_USERNAME` | Optional when using nats to authenticate with the nats cluster. | | +| | `MICRO_REGISTRY_AUTH_PASSWORD` | Optional when using nats to authenticate with the nats cluster. | | +| services/antivirus/pkg/config/config.go | `OCIS_EVENTS_AUTH_USERNAME` | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_EVENTS_AUTH_PASSWORD` | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `ANTIVIRUS_ICAP_SCAN_TIMEOUT` | Scan timeout for the ICAP client. Defaults to '5m' (5 minutes). See the Environment Variable Types description for more details. | 5m0s | +| services/audit/pkg/config/config.go | `OCIS_EVENTS_AUTH_USERNAME` | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_EVENTS_AUTH_PASSWORD` | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| services/auth-service/pkg/config/config.go | `OCIS_LOG_LEVEL;AUTH_SERVICE_LOG_LEVEL` | The log level. Valid values are: 'panic', 'fatal', 'error', 'warn', 'info', 'debug', 'trace'." | | +| | `OCIS_LOG_PRETTY;AUTH_SERVICE_LOG_PRETTY` | Activates pretty log output. | | +| | `OCIS_LOG_COLOR;AUTH_SERVICE_LOG_COLOR` | Activates colorized log output. | | +| | `OCIS_LOG_FILE;AUTH_SERVICE_LOG_FILE` | The path to the log file. Activates logging to this file if set. | | +| | `AUTH_SERVICE_DEBUG_ADDR` | Bind address of the debug server, where metrics, health, config and debug endpoints will be exposed. | | +| | `AUTH_SERVICE_DEBUG_PPROF` | Enables pprof, which can be used for profiling. | | +| | `AUTH_SERVICE_DEBUG_TOKEN` | Enables pprof, which can be used for profiling. | | +| | `AUTH_SERVICE_DEBUG_ZPAGES` | Enables zpages, which can be used for collecting and viewing in-memory traces. | | +| | `AUTH_SERVICE_GRPC_ADDR` | The bind address of the GRPC service. | | +| | `AUTH_SERVICE_GRPC_PROTOCOL` | The transport protocol of the GRPC service. | | +| | `OCIS_SERVICE_ACCOUNT_ID;AUTH_SERVICE_SERVICE_ACCOUNT_ID` | The ID of the service account the service should use. See the 'auth-service' service description for more details. | | +| | `OCIS_SERVICE_ACCOUNT_SECRET;AUTH_SERVICE_SERVICE_ACCOUNT_SECRET` | The service account secret. | | +| services/auth-service/pkg/config/reva.go | `OCIS_JWT_SECRET;AUTH_SERVICE_JWT_SECRET` | The secret to mint and validate jwt tokens. | | +| services/auth-service/pkg/config/tracing.go | `OCIS_TRACING_ENABLED;AUTH_SERVICE_TRACING_ENABLED` | Activates tracing. | | +| | `OCIS_TRACING_TYPE;AUTH_SERVICE_TRACING_TYPE` | The type of tracing. Defaults to '', which is the same as 'jaeger'. Allowed tracing types are 'jaeger' and '' as of now."` | | +| | `OCIS_TRACING_ENDPOINT;AUTH_SERVICE_TRACING_ENDPOINT` | The endpoint of the tracing agent. | | +| | `OCIS_TRACING_COLLECTOR;AUTH_SERVICE_TRACING_COLLECTOR` | The HTTP endpoint for sending spans directly to a collector, i.e. http://jaeger-collector:14268/api/traces. Only used if the tracing endpoint is unset. | | +| services/clientlog/pkg/config/config.go | `OCIS_REVA_GATEWAY;CLIENTLOG_REVA_GATEWAY` | CS3 gateway used to look up user metadata | | +| | `OCIS_EVENTS_ENDPOINT;CLIENTLOG_EVENTS_ENDPOINT` | The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. | | +| | `OCIS_EVENTS_CLUSTER;CLIENTLOG_EVENTS_CLUSTER` | The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Mandatory when using NATS as event system. | | +| | `OCIS_INSECURE;CLIENTLOG_EVENTS_TLS_INSECURE` | Whether to verify the server TLS certificates. | | +| | `OCIS_EVENTS_TLS_ROOT_CA_CERTIFICATE;CLIENTLOG_EVENTS_TLS_ROOT_CA_CERTIFICATE` | The root CA certificate used to validate the server's TLS certificate. If provided NOTIFICATIONS_EVENTS_TLS_INSECURE will be seen as false. | | +| | `OCIS_EVENTS_ENABLE_TLS;CLIENTLOG_EVENTS_ENABLE_TLS` | Enable TLS for the connection to the events broker. The events broker is the ocis service which receives and delivers events between the services.. | | +| | `OCIS_EVENTS_AUTH_USERNAME;CLIENTLOG_EVENTS_AUTH_USERNAME` | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services.. | | +| | `OCIS_EVENTS_AUTH_PASSWORD;CLIENTLOG_EVENTS_AUTH_PASSWORD` | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services.. | | +| | `OCIS_JWT_SECRET;CLIENTLOG_JWT_SECRET` | The secret to mint and validate jwt tokens. | | +| | `OCIS_SERVICE_ACCOUNT_ID;CLIENTLOG_SERVICE_ACCOUNT_ID` | The ID of the service account the service should use. See the 'auth-service' service description for more details. | | +| | `OCIS_SERVICE_ACCOUNT_SECRET;CLIENTLOG_SERVICE_ACCOUNT_SECRET` | The service account secret. | | +| services/clientlog/pkg/config/debug.go | `CLIENTLOG_DEBUG_ADDR` | Bind address of the debug server, where metrics, health, config and debug endpoints will be exposed. | | +| | `CLIENTLOG_DEBUG_TOKEN` | Token to secure the metrics endpoint. | | +| | `CLIENTLOG_DEBUG_PPROF` | Enables pprof, which can be used for profiling. | | +| | `CLIENTLOG_DEBUG_ZPAGES` | Enables zpages, which can be used for collecting and viewing in-memory traces. | | +| services/clientlog/pkg/config/log.go | `OCIS_LOG_LEVEL;CLIENTLOG_USERLOG_LOG_LEVEL` | The log level. Valid values are: 'panic', 'fatal', 'error', 'warn', 'info', 'debug', 'trace'." | | +| | `OCIS_LOG_PRETTY;CLIENTLOG_USERLOG_LOG_PRETTY` | Activates pretty log output. | | +| | `OCIS_LOG_COLOR;CLIENTLOG_USERLOG_LOG_COLOR` | Activates colorized log output. | | +| | `OCIS_LOG_FILE;CLIENTLOG_USERLOG_LOG_FILE` | The path to the log file. Activates logging to this file if set. | | +| services/clientlog/pkg/config/tracing.go | `OCIS_TRACING_ENABLED;CLIENTLOG_TRACING_ENABLED` | Activates tracing. | | +| | `OCIS_TRACING_TYPE;CLIENTLOG_TRACING_TYPE` | The type of tracing. Defaults to '', which is the same as 'jaeger'. Allowed tracing types are 'jaeger' and '' as of now."` | | +| | `OCIS_TRACING_ENDPOINT;CLIENTLOG_TRACING_ENDPOINT` | The endpoint of the tracing agent. | | +| | `OCIS_TRACING_COLLECTOR;CLIENTLOG_TRACING_COLLECTOR` | The HTTP endpoint for sending spans directly to a collector, i.e. http://jaeger-collector:14268/api/traces. Only used if the tracing endpoint is unset. | | +| services/eventhistory/pkg/config/config.go | `OCIS_PERSISTENT_STORE_AUTH_USERNAME;EVENTHISTORY_STORE_AUTH_USERNAME` | The username to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_PERSISTENT_STORE_AUTH_PASSWORD;EVENTHISTORY_STORE_AUTH_PASSWORD` | The password to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_EVENTS_AUTH_USERNAME;EVENTHISTORY_EVENTS_AUTH_USERNAME` | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_EVENTS_AUTH_PASSWORD;EVENTHISTORY_EVENTS_AUTH_PASSWORD` | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| services/frontend/pkg/config/config.go | `OCIS_ENABLE_RESHARING` | Changing this value is NOT supported. Enables the support for resharing in the clients. | | +| | `FRONTEND_DEFAULT_LINK_PERMISSIONS` | Defines the default permissions a link is being created with. Possible values are 0 (= internal link, for instance members only) and 1 (= public link with viewer permissions). Defaults to 1. | | +| | `FRONTEND_AUTO_ACCEPT_SHARES` | Defines if shares should be auto accepted by default. Users can change this setting individually in their profile. | true | +| | `OCIS_CACHE_DISABLE_PERSISTENCE;FRONTEND_OCS_STAT_CACHE_DISABLE_PERSISTENCE` | Disable persistence of the cache. Only applies when using the 'nats-js-kv' store type. Defaults to false. | false | +| | `OCIS_CACHE_AUTH_USERNAME;FRONTEND_OCS_STAT_CACHE_AUTH_USERNAME` | The username to use for authentication. Only applies when using the 'nats-js-kv' store type. | | +| | `OCIS_CACHE_AUTH_PASSWORD;FRONTEND_OCS_STAT_CACHE_AUTH_PASSWORD` | The password to use for authentication. Only applies when using the 'nats-js-kv' store type. | | +| | `FRONTEND_OCS_LIST_OCM_SHARES` | Include OCM shares when listing shares. See the OCM service documentation for more details. | | +| | `OCIS_SHARING_PUBLIC_SHARE_MUST_HAVE_PASSWORD;FRONTEND_OCS_PUBLIC_SHARE_MUST_HAVE_PASSWORD` | Set this to true if you want to enforce passwords on all public shares. | | +| | `OCIS_SHARING_PUBLIC_WRITEABLE_SHARE_MUST_HAVE_PASSWORD;FRONTEND_OCS_PUBLIC_WRITEABLE_SHARE_MUST_HAVE_PASSWORD` | Set this to true if you want to enforce passwords on Uploader, Editor or Contributor shares. | | +| | `FRONTEND_OCS_INCLUDE_OCM_SHAREES` | Include OCM sharees when listing sharees. | | +| | `OCIS_EVENTS_ENDPOINT;FRONTEND_EVENTS_ENDPOINT` | The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. | | +| | `OCIS_EVENTS_CLUSTER;FRONTEND_EVENTS_CLUSTER` | The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Mandatory when using NATS as event system. | | +| | `OCIS_INSECURE;FRONTEND_EVENTS_TLS_INSECURE` | Whether to verify the server TLS certificates. | | +| | `FRONTEND_EVENTS_TLS_ROOT_CA_CERTIFICATE;OCS_EVENTS_TLS_ROOT_CA_CERTIFICATE` | The root CA certificate used to validate the server's TLS certificate. If provided NOTIFICATIONS_EVENTS_TLS_INSECURE will be seen as false. | | +| | `OCIS_EVENTS_ENABLE_TLS;FRONTEND_EVENTS_ENABLE_TLS` | Enable TLS for the connection to the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_EVENTS_AUTH_USERNAME;FRONTEND_EVENTS_AUTH_USERNAME` | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services.. | | +| | `OCIS_EVENTS_AUTH_PASSWORD;FRONTEND_EVENTS_AUTH_PASSWORD` | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services.. | | +| | `OCIS_SERVICE_ACCOUNT_ID;FRONTEND_SERVICE_ACCOUNT_ID` | The ID of the service account the service should use. See the 'auth-service' service description for more details. | | +| | `OCIS_SERVICE_ACCOUNT_SECRET;FRONTEND_SERVICE_ACCOUNT_SECRET` | The service account secret. | | +| | `OCIS_PASSWORD_POLICY_DISABLED;FRONTEND_PASSWORD_POLICY_DISABLED` | Disable the password policy. Defaults to false if not set. | | +| | `OCIS_PASSWORD_POLICY_MIN_CHARACTERS;FRONTEND_PASSWORD_POLICY_MIN_CHARACTERS` | Define the minimum password length. Defaults to 0 if not set. | 0 | +| | `OCIS_PASSWORD_POLICY_MIN_LOWERCASE_CHARACTERS;FRONTEND_PASSWORD_POLICY_MIN_LOWERCASE_CHARACTERS` | Define the minimum number of lowercase characters. Defaults to 0 if not set. | 0 | +| | `OCIS_PASSWORD_POLICY_MIN_UPPERCASE_CHARACTERS;FRONTEND_PASSWORD_POLICY_MIN_UPPERCASE_CHARACTERS` | Define the minimum number of uppercase characters. Defaults to 0 if not set. | 0 | +| | `OCIS_PASSWORD_POLICY_MIN_DIGITS;FRONTEND_PASSWORD_POLICY_MIN_DIGITS` | Define the minimum number of digits. Defaults to 0 if not set. | 0 | +| | `OCIS_PASSWORD_POLICY_MIN_SPECIAL_CHARACTERS;FRONTEND_PASSWORD_POLICY_MIN_SPECIAL_CHARACTERS` | Define the minimum number of special characters. Defaults to 0 if not set. | 0 | +| | `OCIS_PASSWORD_POLICY_BANNED_PASSWORDS_LIST;FRONTEND_PASSWORD_POLICY_BANNED_PASSWORDS_LIST` | Path to the 'banned passwords list' file. See the documentation for more details. | | +| services/gateway/pkg/config/config.go | `OCIS_CACHE_DISABLE_PERSISTENCE;GATEWAY_STAT_CACHE_DISABLE_PERSISTENCE` | Disables persistence of the stat cache. Only applies when store type 'nats-js-kv' is configured. Defaults to false. | false | +| | `OCIS_CACHE_DISABLE_PERSISTENCE;GATEWAY_PROVIDER_CACHE_DISABLE_PERSISTENCE` | Disables persistence of the provider cache. Only applies when store type 'nats-js-kv' is configured. Defaults to false. | false | +| | `OCIS_CACHE_AUTH_USERNAME;GATEWAY_PROVIDER_CACHE_AUTH_USERNAME` | The username to use for authentication. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_CACHE_AUTH_PASSWORD;GATEWAY_PROVIDER_CACHE_AUTH_PASSWORD` | The password to use for authentication. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_CACHE_DISABLE_PERSISTENCE;GATEWAY_CREATE_HOME_CACHE_DISABLE_PERSISTENCE` | Disables persistence of the create home cache. Only applies when store type 'nats-js-kv' is configured. Defaults to false. | | +| | `OCIS_CACHE_AUTH_USERNAME;GATEWAY_CREATE_HOME_CACHE_AUTH_USERNAME` | The username to use for authentication. Only applies when store type 'nats-js-kv' is configured. | false | +| | `OCIS_CACHE_AUTH_PASSWORD;GATEWAY_CREATE_HOME_CACHE_AUTH_PASSWORD` | The password to use for authentication. Only applies when store type 'nats-js-kv' is configured. | | +| services/graph/pkg/config/cache.go | `OCIS_CACHE_DISABLE_PERSISTENCE;GRAPH_CACHE_DISABLE_PERSISTENCE` | Disables persistence of the cache. Only applies when store type 'nats-js-kv' is configured. Defaults to false. | false | +| | `OCIS_CACHE_AUTH_USERNAME;GRAPH_CACHE_AUTH_USERNAME` | The username to authenticate with the cache. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_CACHE_AUTH_PASSWORD;GRAPH_CACHE_AUTH_PASSWORD` | The password to authenticate with the cache. Only applies when store type 'nats-js-kv' is configured. | | +| services/graph/pkg/config/config.go | `GRAPH_IDENTITY_SEARCH_MIN_LENGTH` | The minimum length the search term needs to have for unprivileged users when searching for users or groups. | | +| | `OCIS_EVENTS_AUTH_USERNAME;GRAPH_EVENTS_AUTH_USERNAME` | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_EVENTS_AUTH_PASSWORD;GRAPH_EVENTS_AUTH_PASSWORD` | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_SERVICE_ACCOUNT_ID;GRAPH_SERVICE_ACCOUNT_ID` | The ID of the service account the service should use. See the 'auth-service' service description for more details. | | +| | `OCIS_SERVICE_ACCOUNT_SECRET;GRAPH_SERVICE_ACCOUNT_SECRET` | The service account secret. | | +| | `OCIS_ENABLE_RESHARING;GRAPH_ENABLE_RESHARING` | Changing this value is NOT supported. Enables the support for resharing in the clients. | | +| services/idp/pkg/config/config.go | `IDP_LOGIN_BACKGROUND_URL` | Configure an alternative URL to the background image for the login page. | | +| services/notifications/pkg/config/config.go | `OCIS_DEFAULT_LANGUAGE` | The default language used by services and the WebUI. If not defined, English will be used as default. See the documentation for more details. | | +| | `OCIS_EVENTS_AUTH_USERNAME;NOTIFICATIONS_EVENTS_AUTH_USERNAME` | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_EVENTS_AUTH_PASSWORD;NOTIFICATIONS_EVENTS_AUTH_PASSWORD` | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_SERVICE_ACCOUNT_ID;NOTIFICATIONS_SERVICE_ACCOUNT_ID` | The ID of the service account the service should use. See the 'auth-service' service description for more details. | | +| | `OCIS_SERVICE_ACCOUNT_SECRET;NOTIFICATIONS_SERVICE_ACCOUNT_SECRET` | The service account secret. | | +| services/ocdav/pkg/config/config.go | `OCDAV_OCM_NAMESPACE` | The human readable path prefix for the ocm shares. | | +| services/ocm/pkg/config/config.go | `SHARING_OCM_PROVIDER_AUTHORIZER_DRIVER` | Driver to be used to persist ocm invites. Supported value is only 'json'. | `json` | +| | `OCM_HTTP_ADDR` | The bind address of the HTTP service. | | +| | `OCM_HTTP_PROTOCOL` | The transport protocol of the HTTP service. | | +| | `OCM_HTTP_PREFIX` | The path prefix where OCM can be accessed (defaults to /). | | +| | `OCIS_SERVICE_ACCOUNT_ID;OCM_SERVICE_ACCOUNT_ID` | The ID of the service account the service should use. See the 'auth-service' service description for more details. | | +| | `OCIS_SERVICE_ACCOUNT_SECRET;OCM_SERVICE_ACCOUNT_SECRET` | The service account secret. | | +| | `OCIS_CORS_ALLOW_ORIGINS;OCM_CORS_ALLOW_ORIGINS` | A list of allowed CORS origins. See following chapter for more details: _Access-Control-Allow-Origin_ at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin. See the Environment Variable Types description for more details. | | +| | `OCIS_CORS_ALLOW_METHODS;OCM_CORS_ALLOW_METHODS` | A list of allowed CORS methods. See following chapter for more details: _Access-Control-Allow-Methods_ at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Methods. See the Environment Variable Types description for more details. | | +| | `OCIS_CORS_ALLOW_HEADERS;OCM_CORS_ALLOW_HEADERS` | A list of allowed CORS headers. See following chapter for more details: _Access-Control-Allow-Headers_ at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers. See the Environment Variable Types description for more details. | | +| | `OCIS_CORS_ALLOW_CREDENTIALS;OCM_CORS_ALLOW_CREDENTIALS` | Allow credentials for CORS.See following chapter for more details: _Access-Control-Allow-Credentials_ at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Credentials. | | +| | `OCM_GRPC_ADDR` | The bind address of the GRPC service. | | +| | `OCM_GRPC_PROTOCOL` | The transport protocol of the GRPC service. | | +| | `OCM_SCIENCEMESH_PREFIX` | URL path prefix for the ScienceMesh service. Note that the string must not start with '/'. | | +| | `OCM_MESH_DIRECTORY_URL ` | URL of the mesh directory service. | | +| | `OCM_OCMD_PREFIX` | URL path prefix for the OCMd service. Note that the string must not start with '/'. | | +| | `OCM_OCMD_EXPOSE_RECIPIENT_DISPLAY_NAME` | Expose the display name of OCM share recipients. | | +| | `OCM_OCM_INVITE_MANAGER_DRIVER` | Driver to be used to persist ocm invites. Supported value is only 'json'. | `json` | +| | `OCM_OCM_INVITE_MANAGER_INSECURE` | Disable TLS certificate validation for the OCM connections. Do not set this in production environments. | | +| | `OCM_OCM_INVITE_MANAGER_JSON_FILE` | Path to the JSON file where OCM invite data will be stored. If not defined, the root directory derives from $OCIS_BASE_DATA_PATH:/storage. | | +| | `OCM_OCM_PROVIDER_AUTHORIZER_PROVIDERS_FILE` | Path to the JSON file where ocm invite data will be stored. If not defined, the root directory derives from $OCIS_BASE_DATA_PATH:/storage. | | +| | `OCM_OCM_PROVIDER_AUTHORIZER_VERIFY_REQUEST_HOSTNAME` | Verify the hostname of the request against the hostname of the OCM provider. | | +| | `OCM_OCM_CORE_DRIVER` | Driver to be used to persist ocm shares. Supported value is only 'json'. | `json` | +| | `OCM_OCM_STORAGE_PROVIDER_INSECURE` | Disable TLS certificate validation for the OCM connections. Do not set this in production environments. | | +| | `OCM_OCM_STORAGE_PROVIDER_STORAGE_ROOT` | Directory where the ocm storage provider persists its data like tus upload info files. | | +| | `OCM_OCM_CORE_JSON_FILE` | Path to the JSON file where OCM share data will be stored. If not defined, the root directory derives from $OCIS_BASE_DATA_PATH:/storage. | | +| | `OCM_OCM_SHARE_PROVIDER_DRIVER` | Driver to be used to persist ocm shares. Supported value is only 'json'. | `json` | +| | `OCM_OCM_SHARE_PROVIDER_INSECURE` | Disable TLS certificate validation for the OCM connections. Do not set this in production environments. | | +| | `OCM_WEBAPP_TEMPLATE` | Template for the webapp url. | | +| | `OCM_OCM_SHAREPROVIDER_JSON_FILE` | Path to the JSON file where OCM share data will be stored. If not defined, the root directory derives from $OCIS_BASE_DATA_PATH:/storage. | | +| services/ocm/pkg/config/debug.go | `OCM_DEBUG_ADDR` | Bind address of the debug server, where metrics, health, config and debug endpoints will be exposed. | | +| | `OCM_DEBUG_TOKEN` | Token to secure the metrics endpoint. | | +| | `OCM_DEBUG_PPROF` | Enables pprof, which can be used for profiling. | | +| | `OCM_DEBUG_ZPAGES` | Enables zpages, which can be used for collecting and viewing in-memory traces. | | +| services/ocm/pkg/config/log.go | `OCIS_LOG_LEVEL;OCM_LOG_LEVEL` | The log level. Valid values are: 'panic', 'fatal', 'error', 'warn', 'info', 'debug', 'trace'." | | +| | `OCIS_LOG_PRETTY;OCM_LOG_PRETTY` | Activates pretty log output. | | +| | `OCIS_LOG_COLOR;OCM_LOG_COLOR` | Activates colorized log output. | | +| | `OCIS_LOG_FILE;OCM_LOG_FILE` | The path to the log file. Activates logging to this file if set. | | +| services/ocm/pkg/config/tracing.go | `OCIS_TRACING_ENABLED;OCM_TRACING_ENABLED` | Activates tracing. | | +| | `OCIS_TRACING_TYPE;OCM_TRACING_TYPE` | The type of tracing. Defaults to '', which is the same as 'jaeger'. Allowed tracing types are 'jaeger' and '' as of now."` | | +| | `OCIS_TRACING_ENDPOINT;OCM_TRACING_ENDPOINT` | The endpoint of the tracing agent. | | +| | `OCIS_TRACING_COLLECTOR;OCM_TRACING_COLLECTOR` | The HTTP endpoint for sending spans directly to a collector, i.e. http://jaeger-collector:14268/api/traces. Only used if the tracing endpoint is unset. | | +| services/ocs/pkg/config/config.go | `OCIS_CACHE_STORE;OCS_PRESIGNEDURL_SIGNING_KEYS_STORE` | The type of the signing key store. Supported values are: 'redis-sentinel' and 'nats-js-kv'. See the text description for details. | | +| | `OCIS_CACHE_STORE_NODES;OCS_PRESIGNEDURL_SIGNING_KEYS_STORE_NODES` | A list of nodes to access the configured store. Note that the behaviour how nodes are used is dependent on the library of the configured store. See the Environment Variable Types description for more details. | | +| | `OCIS_CACHE_TTL;OCS_PRESIGNEDURL_SIGNING_KEYS_STORE_TTL` | Default time to live for signing keys. See the Environment Variable Types description for more details. | | +| | `OCIS_CACHE_AUTH_USERNAME;OCS_PRESIGNEDURL_SIGNING_KEYS_STORE_AUTH_USERNAME` | The username to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_CACHE_AUTH_PASSWORD;OCS_PRESIGNEDURL_SIGNING_KEYS_STORE_AUTH_PASSWORD` | The password to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| services/policies/pkg/config/config.go | `OCIS_EVENTS_AUTH_USERNAME;POLICIES_EVENTS_AUTH_USERNAME` | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_EVENTS_AUTH_PASSWORD;POLICIES_EVENTS_AUTH_PASSWORD` | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| services/policies/pkg/config/tracing.go | `OCIS_TRACING_ENABLED;POLICIES_TRACING_ENABLED` | Activates tracing. | | +| | `OCIS_TRACING_TYPE;POLICIES_TRACING_TYPE` | The type of tracing. Defaults to '', which is the same as 'jaeger'. Allowed tracing types are 'jaeger' and '' as of now."` | | +| | `OCIS_TRACING_ENDPOINT;POLICIES_TRACING_ENDPOINT` | The endpoint of the tracing agent. | | +| | `OCIS_TRACING_COLLECTOR;POLICIES_TRACING_COLLECTOR` | The HTTP endpoint for sending spans directly to a collector, i.e. http://jaeger-collector:14268/api/traces. Only used if the tracing endpoint is unset. | | +| services/postprocessing/pkg/config/config.go | `POSTPROCESSING_RETRY_BACKOFF_DURATION` | The base for the exponential backoff duration before retrying a failed postprocessing step. See the Environment Variable Types description for more details. | | +| | `POSTPROCESSING_MAX_RETRIES` | The maximum number of retries for a failed postprocessing step. | | +| | `OCIS_EVENTS_AUTH_USERNAME;POSTPROCESSING_EVENTS_AUTH_USERNAME` | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_EVENTS_AUTH_PASSWORD;POSTPROCESSING_EVENTS_AUTH_PASSWORD` | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_PERSISTENT_STORE_AUTH_USERNAME;POSTPROCESSING_STORE_AUTH_USERNAME` | The username to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_PERSISTENT_STORE_AUTH_PASSWORD;POSTPROCESSING_STORE_AUTH_PASSWORD` | The password to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| services/postprocessing/pkg/config/tracing.go | `OCIS_TRACING_ENABLED;POSTPROCESSING_TRACING_ENABLED` | Activates tracing. | | +| | `OCIS_TRACING_TYPE;POSTPROCESSING_TRACING_TYPE` | The type of tracing. Defaults to '', which is the same as 'jaeger'. Allowed tracing types are 'jaeger' and '' as of now."` | | +| | `OCIS_TRACING_ENDPOINT;POSTPROCESSING_TRACING_ENDPOINT` | The endpoint of the tracing agent. | | +| | `OCIS_TRACING_COLLECTOR;POSTPROCESSING_TRACING_COLLECTOR` | The HTTP endpoint for sending spans directly to a collector, i.e. http://jaeger-collector:14268/api/traces. Only used if the tracing endpoint is unset. | | +| services/proxy/pkg/config/config.go | `OCIS_CACHE_AUTH_USERNAME;PROXY_OIDC_USERINFO_CACHE_AUTH_USERNAME` | The username to authenticate with the cache. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_CACHE_AUTH_PASSWORD;PROXY_OIDC_USERINFO_CACHE_AUTH_PASSWORD` | The password to authenticate with the cache. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_CACHE_STORE;PROXY_PRESIGNEDURL_SIGNING_KEYS_STORE` | The type of the signing key store. Supported values are: 'redis-sentinel' and 'nats-js-kv'. See the text description for details. | | +| | `OCIS_CACHE_STORE_NODES;PROXY_PRESIGNEDURL_SIGNING_KEYS_STORE_NODES` | A list of nodes to access the configured store. Note that the behaviour how nodes are used is dependent on the library of the configured store. See the Environment Variable Types description for more details. | | +| | `OCIS_CACHE_TTL;PROXY_PRESIGNEDURL_SIGNING_KEYS_STORE_TTL` | Default time to live for signing keys. See the Environment Variable Types description for more details. | | +| | `OCIS_CACHE_DISABLE_PERSISTENCE;PROXY_PRESIGNEDURL_SIGNING_KEYS_STORE_DISABLE_PERSISTENCE` | Disables persistence of the store. Only applies when store type 'nats-js-kv' is configured. Defaults to true. | | +| | `OCIS_CACHE_AUTH_USERNAME;PROXY_PRESIGNEDURL_SIGNING_KEYS_STORE_AUTH_USERNAME` | The username to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_CACHE_AUTH_PASSWORD;PROXY_PRESIGNEDURL_SIGNING_KEYS_STORE_AUTH_PASSWORD` | The password to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_SERVICE_ACCOUNT_ID;PROXY_SERVICE_ACCOUNT_ID` | The ID of the service account the service should use. See the 'auth-service' service description for more details. | | +| | `OCIS_SERVICE_ACCOUNT_SECRET;PROXY_SERVICE_ACCOUNT_SECRET` | The service account secret. | | +| services/search/pkg/config/config.go | `OCIS_SERVICE_ACCOUNT_ID;SEARCH_SERVICE_ACCOUNT_ID` | The ID of the service account the service should use. See the 'auth-service' service description for more details. | | +| | `OCIS_SERVICE_ACCOUNT_SECRET;SEARCH_SERVICE_ACCOUNT_SECRET` | The service account secret. | | +| services/search/pkg/config/content.go | `SEARCH_EXTRACTOR_TIKA_CLEAN_STOP_WORDS` | Defines if stop words should be cleaned or not. See the documentation for more details. | | +| services/search/pkg/config/search.go | `OCIS_EVENTS_AUTH_USERNAME;SEARCH_EVENTS_AUTH_USERNAME` | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_EVENTS_AUTH_PASSWORD;SEARCH_EVENTS_AUTH_PASSWORD` | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| services/settings/pkg/config/config.go | `SETTINGS_SERVICE_ACCOUNT_IDS;OCIS_SERVICE_ACCOUNT_ID` | The list of all service account IDs. These will be assigned the hidden 'service-account' role. Note: When using 'OCIS_SERVICE_ACCOUNT_ID' this will contain only one value while 'SETTINGS_SERVICE_ACCOUNT_IDS' can have multiple. See the 'auth-service' service description for more details about service accounts. | | +| | `OCIS_DEFAULT_LANGUAGE` | The default language used by services and the WebUI. If not defined, English will be used as default. See the documentation for more details. | | +| | `OCIS_CACHE_DISABLE_PERSISTENCE;SETTINGS_CACHE_DISABLE_PERSISTENCE` | Disables persistence of the cache. Only applies when store type 'nats-js-kv' is configured. Defaults to false. | | +| | `OCIS_CACHE_AUTH_USERNAME;SETTINGS_CACHE_AUTH_USERNAME` | The username to authenticate with the cache. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_CACHE_AUTH_PASSWORD;SETTINGS_CACHE_AUTH_PASSWORD` | The password to authenticate with the cache. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_EVENTS_AUTH_USERNAME;SETTINGS_EVENTS_AUTH_USERNAME` | The username to authenticate with the cache. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_EVENTS_AUTH_PASSWORD;SETTINGS_EVENTS_AUTH_PASSWORD` | The password to authenticate with the cache. Only applies when store type 'nats-js-kv' is configured. | | +| services/sharing/pkg/config/config.go | `OCIS_SHARING_PUBLIC_WRITEABLE_SHARE_MUST_HAVE_PASSWORD;SHARING_PUBLIC_WRITEABLE_SHARE_MUST_HAVE_PASSWORD` | Set this to true if you want to enforce passwords on Uploader, Editor or Contributor shares. If not using the global OCIS_SHARING_PUBLIC_WRITEABLE_SHARE_MUST_HAVE_PASSWORD, you must define the FRONTEND_OCS_PUBLIC_WRITEABLE_SHARE_MUST_HAVE_PASSWORD in the frontend service. | | +| | `OCIS_SHARING_PUBLIC_SHARE_MUST_HAVE_PASSWORD;SHARING_PUBLIC_SHARE_MUST_HAVE_PASSWORD` | Set this to true if you want to enforce passwords on all public shares. | | +| | `OCIS_EVENTS_AUTH_USERNAME;SHARING_EVENTS_AUTH_USERNAME` | Username for the events broker. | | +| | `OCIS_EVENTS_AUTH_PASSWORD;SHARING_EVENTS_AUTH_PASSWORD` | Password for the events broker. | | +| | `OCIS_PASSWORD_POLICY_DISABLED;SHARING_PASSWORD_POLICY_DISABLED` | Disable the password policy. Defaults to false if not set. | | +| | `OCIS_PASSWORD_POLICY_MIN_CHARACTERS;SHARING_PASSWORD_POLICY_MIN_CHARACTERS` | Define the minimum password length. Defaults to 0 if not set. | 0 | +| | `OCIS_PASSWORD_POLICY_MIN_LOWERCASE_CHARACTERS;SHARING_PASSWORD_POLICY_MIN_LOWERCASE_CHARACTERS` | Define the minimum number of lowercase characters. Defaults to 0 if not set. | 0 | +| | `OCIS_PASSWORD_POLICY_MIN_UPPERCASE_CHARACTERS;SHARING_PASSWORD_POLICY_MIN_UPPERCASE_CHARACTERS` | Define the minimum number of uppercase characters. Defaults to 0 if not set. | 0 | +| | `OCIS_PASSWORD_POLICY_MIN_DIGITS;SHARING_PASSWORD_POLICY_MIN_DIGITS` | Define the minimum number of digits. Defaults to 0 if not set. | 0 | +| | `OCIS_PASSWORD_POLICY_MIN_SPECIAL_CHARACTERS;SHARING_PASSWORD_POLICY_MIN_SPECIAL_CHARACTERS` | Define the minimum number of special characters. Defaults to 0 if not set. | 0 | +| | `OCIS_PASSWORD_POLICY_BANNED_PASSWORDS_LIST;SHARING_PASSWORD_POLICY_BANNED_PASSWORDS_LIST` | Path to the 'banned passwords list' file. See the documentation for more details. | | +| services/sse/pkg/config/config.go | `OCIS_LOG_LEVEL;SSE_LOG_LEVEL` | The log level. Valid values are: 'panic', 'fatal', 'error', 'warn', 'info', 'debug', 'trace'." | | +| | `OCIS_LOG_PRETTY;SSE_LOG_PRETTY` | Activates pretty log output. | | +| | `OCIS_LOG_COLOR;SSE_LOG_COLOR` | Activates colorized log output. | | +| | `OCIS_LOG_FILE;SSE_LOG_FILE` | The path to the log file. Activates logging to this file if set. | | +| | `SSE_DEBUG_ADDR` | Bind address of the debug server, where metrics, health, config and debug endpoints will be exposed. | | +| | `SSE_DEBUG_TOKEN` | Token to secure the metrics endpoint. | | +| | `SSE_DEBUG_PPROF` | Enables pprof, which can be used for profiling. | | +| | `SSE_DEBUG_ZPAGES` | Enables zpages, which can be used for collecting and viewing in-memory traces. | | +| | `OCIS_EVENTS_ENDPOINT;SSE_EVENTS_ENDPOINT` | The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. | | +| | `OCIS_EVENTS_CLUSTER;SSE_EVENTS_CLUSTER` | The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Mandatory when using NATS as event system. | | +| | `OCIS_INSECURE;SSE_EVENTS_TLS_INSECURE` | Whether to verify the server TLS certificates. | | +| | `OCIS_EVENTS_TLS_ROOT_CA_CERTIFICATE;SSE_EVENTS_TLS_ROOT_CA_CERTIFICATE` | The root CA certificate used to validate the server's TLS certificate. If provided NOTIFICATIONS_EVENTS_TLS_INSECURE will be seen as false. | | +| | `OCIS_EVENTS_ENABLE_TLS;SSE_EVENTS_ENABLE_TLS` | Enable TLS for the connection to the events broker. The events broker is the ocis service which receives and delivers events between the services.. | | +| | `OCIS_EVENTS_AUTH_USERNAME;SSE_EVENTS_AUTH_USERNAME` | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services.. | | +| | `OCIS_EVENTS_AUTH_PASSWORD;SSE_EVENTS_AUTH_PASSWORD` | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services.. | | +| | `OCIS_CORS_ALLOW_ORIGINS;SSE_CORS_ALLOW_ORIGINS` | A list of allowed CORS origins. See following chapter for more details: _Access-Control-Allow-Origin_ at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin. See the Environment Variable Types description for more details. | | +| | `OCIS_CORS_ALLOW_METHODS;SSE_CORS_ALLOW_METHODS` | A list of allowed CORS methods. See following chapter for more details: _Access-Control-Allow-Methods_ at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Methods. See the Environment Variable Types description for more details. | | +| | `OCIS_CORS_ALLOW_HEADERS;SSE_CORS_ALLOW_HEADERS` | A list of allowed CORS headers. See following chapter for more details: _Access-Control-Allow-Headers_ at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers. See the Environment Variable Types description for more details. | | +| | `OCIS_CORS_ALLOW_CREDENTIALS;SSE_CORS_ALLOW_CREDENTIALS` | Allow credentials for CORS.See following chapter for more details: _Access-Control-Allow-Credentials_ at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Credentials. | | +| | `SSE_HTTP_ADDR` | The bind address of the HTTP service. | | +| | `SSE_HTTP_ROOT` | The root path of the HTTP service. | | +| | `OCIS_JWT_SECRET;SSE_JWT_SECRET` | The secret to mint and validate jwt tokens. | | +| services/sse/pkg/config/tracing.go | `OCIS_TRACING_ENABLED;SSE_TRACING_ENABLED` | Activates tracing. | | +| | `OCIS_TRACING_TYPE;SSE_TRACING_TYPE` | The type of tracing. Defaults to '', which is the same as 'jaeger'. Allowed tracing types are 'jaeger' and '' as of now."` | | +| | `OCIS_TRACING_ENDPOINT;SSE_TRACING_ENDPOINT` | The endpoint of the tracing agent. | | +| | `OCIS_TRACING_COLLECTOR;SSE_TRACING_COLLECTOR` | The HTTP endpoint for sending spans directly to a collector, i.e. http://jaeger-collector:14268/api/traces. Only used if the tracing endpoint is unset. | | +| services/storage-system/pkg/config/config.go | `OCIS_CACHE_DISABLE_PERSISTENCE;STORAGE_SYSTEM_CACHE_DISABLE_PERSISTENCE` | Disables persistence of the cache. Only applies when store type 'nats-js-kv' is configured. Defaults to false. | false | +| | `OCIS_CACHE_AUTH_USERNAME;STORAGE_SYSTEM_CACHE_AUTH_USERNAME` | Username for the configured store. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_CACHE_AUTH_PASSWORD;STORAGE_SYSTEM_CACHE_AUTH_PASSWORD` | TPassword for the configured store. Only applies when store type 'nats-js-kv' is configured. | | +| services/storage-users/pkg/config/config.go | `OCIS_GATEWAY_GRPC_ADDR;STORAGE_USERS_GATEWAY_GRPC_ADDR` | The bind address of the gateway GRPC address. | | +| | `OCIS_MACHINE_AUTH_API_KEY;STORAGE_USERS_MACHINE_AUTH_API_KEY` | Machine auth API key used to validate internal requests necessary for the access to resources from other services. | | +| | `STORAGE_USERS_CLI_MAX_ATTEMPTS_RENAME_FILE` | The maximum number of attempts to rename a file when a user restores a file to an existing destination with the same name. The minimum value is 100. | | +| | `OCIS_EVENTS_AUTH_USERNAME;STORAGE_USERS_EVENTS_AUTH_USERNAME` | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_EVENTS_AUTH_PASSWORD;STORAGE_USERS_EVENTS_AUTH_PASSWORD` | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_CACHE_DISABLE_PERSISTENCE;STORAGE_USERS_STAT_CACHE_DISABLE_PERSISTENCE` | Disables persistence of the stat cache. Only applies when store type 'nats-js-kv' is configured. Defaults to false. | false | +| | `OCIS_CACHE_DISABLE_PERSISTENCE;STORAGE_USERS_FILEMETADATA_CACHE_DISABLE_PERSISTENCE` | Disables persistence of the file metadata cache. Only applies when store type 'nats-js-kv' is configured. Defaults to false. | false | +| | `OCIS_CACHE_AUTH_USERNAME;STORAGE_USERS_FILEMETADATA_CACHE_AUTH_USERNAME` | The username to authenticate with the cache store. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_CACHE_AUTH_PASSWORD;STORAGE_USERS_FILEMETADATA_CACHE_AUTH_PASSWORD` | The password to authenticate with the cache store. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_CACHE_DISABLE_PERSISTENCE;STORAGE_USERS_ID_CACHE_DISABLE_PERSISTENCE` | Disables persistence of the id cache. Only applies when store type 'nats-js-kv' is configured. Defaults to false. | false | +| | `OCIS_CACHE_AUTH_USERNAME;STORAGE_USERS_ID_CACHE_AUTH_USERNAME` | The username to authenticate with the cache store. Only applies when store type 'nats-js-kv' is configured | | +| | `OCIS_CACHE_AUTH_PASSWORD;STORAGE_USERS_ID_CACHE_AUTH_PASSWORD` | The password to authenticate with the cache store. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_SERVICE_ACCOUNT_ID;STORAGE_USERS_SERVICE_ACCOUNT_ID` | The ID of the service account the service should use. See the 'auth-service' service description for more details. | | +| | `OCIS_SERVICE_ACCOUNT_SECRET;STORAGE_USERS_SERVICE_ACCOUNT_SECRET` | The service account secret. | | +| services/userlog/pkg/config/config.go | `OCIS_DEFAULT_LANGUAGE` | The default language used by services and the WebUI. If not defined, English will be used as default. See the documentation for more details. | | +| | `OCIS_PERSISTENT_STORE_AUTH_USERNAME;USERLOG_STORE_AUTH_USERNAME` | The username to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_PERSISTENT_STORE_AUTH_PASSWORD;USERLOG_STORE_AUTH_PASSWORD` | The password to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| | `OCIS_EVENTS_AUTH_USERNAME;USERLOG_EVENTS_AUTH_USERNAME` | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_EVENTS_AUTH_PASSWORD;USERLOG_EVENTS_AUTH_PASSWORD` | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | `OCIS_SERVICE_ACCOUNT_ID;USERLOG_SERVICE_ACCOUNT_ID` | The ID of the service account the service should use. See the 'auth-service' service description for more details. | | +| | `OCIS_SERVICE_ACCOUNT_SECRET;USERLOG_SERVICE_ACCOUNT_SECRET` | The service account secret. | | +| services/web/pkg/config/options.go | `WEB_OPTION_LOGIN_URL` | Specifies the target URL to the login page. This is helpful when an external IdP is used. This option is disabled by default. Example URL like: https://www.myidp.com/login. | | | +| | `WEB_OPTION_DISABLED_EXTENSIONS` | A list to disable specific Web extensions identified by their ID. The ID can e.g. be taken from the 'index.ts' file of the web extension. Example: 'com.github.owncloud.web.files.search,com.github.owncloud.web.files.print'. See the Environment Variable Types description for more details. | | +| | `WEB_OPTION_USER_LIST_REQUIRES_FILTER` | Defines whether one or more filters must be set in order to list users in the Web admin settings. Set this option to 'true' if running in an environment with a lot of users and listing all users could slow down performance. Defaults to 'false'. | false | +| | `WEB_OPTION_EMBED_ENABLED` | Defines whether Web should be running in 'embed' mode. Setting this to 'true' will enable a stripped down version of Web with reduced functionality used to integrate Web into other applications like via iFrame. Setting it to 'false' or not setting it (default) will run Web as usual with all functionality enabled. See the text description for more details. | | +| | `WEB_OPTION_EMBED_TARGET` | Defines how Web is being integrated when running in 'embed' mode. Currently, the only supported options are '' (empty) and 'location'. With '' which is the default, Web will run regular as defined via the 'embed.enabled' config option. With 'location', Web will run embedded as location picker. Resource selection will be disabled and the selected resources array always includes the current folder as the only item. See the text description for more details. | | +| | `WEB_OPTION_EMBED_MESSAGES_ORIGIN` | Defines a URL under which Web can be integrated via iFrame in 'embed' mode. Note that setting this is mandatory when running Web in 'embed' mode. Use '*' as value to allow running the iFrame under any URL, although this is not recommended for security reasons. See the text description for more details. | | +| | `WEB_OPTION_EMBED_DELEGATE_AUTHENTICATION` | Defines whether Web should require authentication to be done by the parent application when running in 'embed' mode. If set to 'true' Web will not try to authenticate the user on its own but will require an access token coming from the parent application. Defaults to being unset. | | +| | `WEB_OPTION_EMBED_DELEGATE_AUTHENTICATION_ORIGIN` | Defines the host to validate the message event origin against when running Web in 'embed' mode with delegated authentication. Defaults to event message origin validation being omitted, which is only recommended for development setups. | | +| | `WEB_OPTION_CONCURRENT_REQUESTS_RESOURCE_BATCH_ACTIONS` | Defines the maximum number of concurrent requests per file/folder/space batch action. Defaults to 4. | | +| | `WEB_OPTION_CONCURRENT_REQUESTS_SSE` | Defines the maximum number of concurrent requests in SSE event handlers. Defaults to 4. | | +| | `WEB_OPTION_CONCURRENT_REQUESTS_SHARES_CREATE` | Defines the maximum number of concurrent requests per sharing invite batch. Defaults to 4. | | +| | `WEB_OPTION_CONCURRENT_REQUESTS_SHARES_LIST` | Defines the maximum number of concurrent requests when loading individual share information inside listings. Defaults to 2. | | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/4.0.0-5.0.0-removed.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/4.0.0-5.0.0-removed.adoc new file mode 100644 index 00000000..02a9cd46 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/4.0.0-5.0.0-removed.adoc @@ -0,0 +1,16 @@ +:toc: right +:toclevels: 3 + += Removed Variables between oCIS 4.0.0 and oCIS 5.0.0 + +| File | Variable | Replacement | Description | +|------------------------------------------|--------------------------------------|-------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| services/auth-basic/pkg/config/config.go | `LDAP_BIND_PASSWORD` | `OCIS_LDAP_BIND_PASSWORD` | Password to use for authenticating the 'bind_dn'. | +| services/graph/pkg/config/config.go | `LDAP_BIND_PASSWORD` | `OCIS_LDAP_BIND_PASSWORD` | Password to use for authenticating the 'bind_dn'. | +| services/groups/pkg/config/config.go | `LDAP_BIND_PASSWORD` | `OCIS_LDAP_BIND_PASSWORD` | Password to use for authenticating the 'bind_dn'. | +| services/idp/pkg/config/config.go | `LDAP_BIND_PASSWORD` | `OCIS_LDAP_BIND_PASSWORD` | Password to use for authenticating the 'bind_dn'. | +| services/sharing/pkg/config/config.go | `SHARING_EVENTS_TLS_ROOT_CA_CERT` | `SHARING_EVENTS_TLS_ROOT_CA_CERTIFICATE` | The root CA certificate used to validate the server's TLS certificate. If provided SHARING_EVENTS_TLS_INSECURE will be seen as false. | +| services/users/pkg/config/config.go | `LDAP_BIND_PASSWORD` | `OCIS_LDAP_BIND_PASSWORD` | Password to use for authenticating the 'bind_dn'. | +| | `LDAP_USER_SCHEMA_ID_IS_OCTETSTRING` | `OCIS_LDAP_USER_SCHEMA_ID_IS_OCTETSTRING` | Set this to true if the defined 'ID' attribute for users is of the 'OCTETSTRING' syntax. This is e.g. required when using the 'objectGUID' attribute of Active Directory for the user ID's. | +| services/web/pkg/config/options.go | `WEB_OPTION_IMPRINT_URL` | To be configured via the `theme.json` file | Specifies the target URL for the imprint link valid for the ocis instance in the account menu. | +| | `WEB_OPTION_PRIVACY_URL` | To be configured via the `theme.json` file | Specifies the target URL for the privacy link valid for the ocis instance in the account menu. | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/5.0.0-7.0.0-added.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/5.0.0-7.0.0-added.adoc new file mode 100644 index 00000000..be1d5704 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/5.0.0-7.0.0-added.adoc @@ -0,0 +1,132 @@ +:toc: right +:toclevels: 3 + +Note that the links provided in the service column are non functional when clicked. + +| Service | Variable | Description | Default | +| --- | --- | --- | --- | +| xref:deployment/services/env-vars-special-scope.adoc[Special Scope Envvars] | OCIS_ASSET_THEMES_PATH | Serve ownCloud themes from a path on the filesystem instead of the builtin assets. If not defined, the root directory derives from $OCIS_BASE_DATA_PATH/web/assets/themes | /var/lib/ocis/web/assets/themes | +| | OCIS_DISABLE_VERSIONING | Disables versioning of files. When set to true, new uploads with the same filename will overwrite existing files instead of creating a new version. | false | +| | OCIS_SHOW_USER_EMAIL_IN_RESULTS | Include user email addresses in responses. If absent or set to false emails will be omitted from results. Please note that admin users can always see all email addresses. | false | +| | OCIS_TRANSLATION_PATH | (optional) Set this to a path with custom translations to overwrite the builtin translations. Note that file and folder naming rules apply, see the documentation for more details. | | +| | OCIS_WOPI_DISABLE_CHAT | Disable chat in the office web frontend. This feature applies to OnlyOffice and Microsoft. | false | +| xref:{s-path}/activitylog.adoc[Activitylog] | ACTIVITYLOG_TRANSLATION_PATH | (optional) Set this to a path with custom translations to overwrite the builtin translations. Note that file and folder naming rules apply, see the documentation for more details. | | +| xref:{s-path}/antivirus.adoc[Antivirus] | ANTIVIRUS_WORKERS | The number of concurrent go routines that fetch events from the event queue. | 10 | +| xref:{s-path}/auth-app.adoc[Auth-App] | AUTH_APP_DEBUG_ADDR | Bind address of the debug server, where metrics, health, config and debug endpoints will be exposed. | 127.0.0.1:9245 | +| | AUTH_APP_DEBUG_PPROF | Enables pprof, which can be used for profiling. | false | +| | AUTH_APP_DEBUG_TOKEN | Token to secure the metrics endpoint. | | +| | AUTH_APP_DEBUG_ZPAGES | Enables zpages, which can be used for collecting and viewing traces in-memory. | false | +| | AUTH_APP_ENABLE_IMPERSONATION | Allows admins to create app tokens for other users. Used for migration. Do NOT use in productive deployments. | false | +| | AUTH_APP_GRPC_ADDR | The bind address of the GRPC service. | 127.0.0.1:9246 | +| | AUTH_APP_GRPC_PROTOCOL | The transport protocol of the GRPC service. | tcp | +| | AUTH_APP_JWT_SECRET | The secret to mint and validate jwt tokens. | | +| | AUTH_APP_LOG_COLOR | Activates colorized log output. | false | +| | AUTH_APP_LOG_FILE | The path to the log file. Activates logging to this file if set. | | +| | AUTH_APP_LOG_LEVEL | The log level. Valid values are: 'panic', 'fatal', 'error', 'warn', 'info', 'debug', 'trace'. | | +| | AUTH_APP_LOG_PRETTY | Activates pretty log output. | false | +| | AUTH_APP_MACHINE_AUTH_API_KEY | The machine auth API key used to validate internal requests necessary to access resources from other services. | | +| | AUTH_APP_SKIP_USER_GROUPS_IN_TOKEN | Disables the encoding of the user’s group memberships in the access token. This reduces the token size, especially when users are members of a large number of groups. | false | +| | AUTH_APP_TRACING_COLLECTOR | The HTTP endpoint for sending spans directly to a collector, i.e. http://jaeger-collector:14268/api/traces. Only used if the tracing endpoint is unset. | | +| | AUTH_APP_TRACING_ENABLED | Activates tracing. | false | +| | AUTH_APP_TRACING_ENDPOINT | The endpoint of the tracing agent. | | +| | AUTH_APP_TRACING_TYPE | The type of tracing. Defaults to '', which is the same as 'jaeger'. Allowed tracing types are 'jaeger' and '' as of now. | | +| xref:{s-path}/collaboration.adoc[Collaboration] | COLLABORATION_APP_ADDR | The URL where the WOPI app is located, such as https://127.0.0.1:8080. | https://127.0.0.1:9980 | +| | COLLABORATION_APP_DESCRIPTION | App description | Open office documents with Collabora | +| | COLLABORATION_APP_ICON | Icon for the app | image-edit | +| | COLLABORATION_APP_INSECURE | Skip TLS certificate verification when connecting to the WOPI app | false | +| | COLLABORATION_APP_LICENSE_CHECK_ENABLE | Enable license checking to edit files. Needs to be enabled when using Microsoft365 with the business flow. | false | +| | COLLABORATION_APP_NAME | The name of the app which is shown to the user. You can chose freely but you are limited to a single word without special characters or whitespaces. We recommend to use pascalCase like 'CollaboraOnline'. | Collabora | +| | COLLABORATION_APP_PRODUCT | The WebOffice app, either Collabora, OnlyOffice, Microsoft365 or MicrosoftOfficeOnline. | Collabora | +| | COLLABORATION_APP_PROOF_DISABLE | Disable the proof keys verification | false | +| | COLLABORATION_APP_PROOF_DURATION | Duration for the proof keys to be cached in memory, using time.ParseDuration format. If the duration can’t be parsed, we’ll use the default 12h as duration | 12h | +| | COLLABORATION_CS3API_DATAGATEWAY_INSECURE | Connect to the CS3API data gateway insecurely. | false | +| | COLLABORATION_DEBUG_ADDR | Bind address of the debug server, where metrics, health, config and debug endpoints will be exposed. | 127.0.0.1:9304 | +| | COLLABORATION_DEBUG_PPROF | Enables pprof, which can be used for profiling. | false | +| | COLLABORATION_DEBUG_TOKEN | Token to secure the metrics endpoint. | | +| | COLLABORATION_DEBUG_ZPAGES | Enables zpages, which can be used for collecting and viewing in-memory traces. | false | +| | COLLABORATION_GRPC_ADDR | The bind address of the GRPC service. | 127.0.0.1:9301 | +| | COLLABORATION_GRPC_PROTOCOL | The transport protocol of the GRPC service. | tcp | +| | COLLABORATION_HTTP_ADDR | The bind address of the HTTP service. | 127.0.0.1:9300 | +| | COLLABORATION_LOG_COLOR | Activates colorized log output. | false | +| | COLLABORATION_LOG_FILE | The path to the log file. Activates logging to this file if set. | | +| | COLLABORATION_LOG_LEVEL | The log level. Valid values are: 'panic', 'fatal', 'error', 'warn', 'info', 'debug', 'trace'. | | +| | COLLABORATION_LOG_PRETTY | Activates pretty log output. | false | +| | COLLABORATION_STORE | The type of the store. Supported values are: 'memory', 'nats-js-kv', 'redis-sentinel', 'noop'. See the text description for details. | nats-js-kv | +| | COLLABORATION_STORE_AUTH_PASSWORD | The password to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| | COLLABORATION_STORE_AUTH_USERNAME | The username to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| | COLLABORATION_STORE_DATABASE | The database name the configured store should use. | collaboration | +| | COLLABORATION_STORE_NODES | A list of nodes to access the configured store. This has no effect when 'memory' store is configured. Note that the behaviour how nodes are used is dependent on the library of the configured store. See the Environment Variable Types description for more details. | [127.0.0.1:9233] | +| | COLLABORATION_STORE_TABLE | The database table the store should use. | | +| | COLLABORATION_STORE_TTL | Time to live for events in the store. Defaults to '30m' (30 minutes). See the Environment Variable Types description for more details. | 30m0s | +| | COLLABORATION_TRACING_COLLECTOR | The HTTP endpoint for sending spans directly to a collector, i.e. http://jaeger-collector:14268/api/traces. Only used if the tracing endpoint is unset. | | +| | COLLABORATION_TRACING_ENABLED | Activates tracing. | false | +| | COLLABORATION_TRACING_ENDPOINT | The endpoint of the tracing agent. | | +| | COLLABORATION_TRACING_TYPE | The type of tracing. Defaults to '', which is the same as 'jaeger'. Allowed tracing types are 'jaeger' and '' as of now. | | +| | COLLABORATION_WOPI_DISABLE_CHAT | Disable chat in the office web frontend. This feature applies to OnlyOffice and Microsoft. | false | +| | COLLABORATION_WOPI_PROXY_SECRET | Optional, the secret to authenticate against the ownCloud Office365 WOPI proxy. This secret can be obtained from ownCloud via the office365 proxy subscription. | | +| | COLLABORATION_WOPI_PROXY_URL | The URL to the ownCloud Office365 WOPI proxy. Optional. To use this feature, you need an office365 proxy subscription. If you become part of the Microsoft CSP program (https://learn.microsoft.com/en-us/partner-center/enroll/csp-overview), you can use WebOffice without a proxy. | | +| | COLLABORATION_WOPI_SECRET | Used to mint and verify WOPI JWT tokens and encrypt and decrypt the REVA JWT token embedded in the WOPI JWT token. | | +| | COLLABORATION_WOPI_SHORTTOKENS | Use short access tokens for WOPI access. This is useful for office packages, like Microsoft Office Online, which have URL length restrictions. If enabled, a persistent store must be configured. | false | +| | COLLABORATION_WOPI_SRC | The WOPI source base URL containing schema, host and port. Set this to the schema and domain where the collaboration service is reachable for the wopi app, such as https://office.owncloud.test. | https://localhost:9300 | +| xref:{s-path}/frontend.adoc[Frontend] | FRONTEND_APP_HANDLER_SECURE_VIEW_APP_ADDR | Service name or address of the app provider to use for secure view. Should match the service name or address of the registered CS3 app provider. | com.owncloud.api.collaboration | +| | FRONTEND_MAX_CONCURRENCY | Maximum number of concurrent go-routines. Higher values can potentially get work done faster but will also cause more load on the system. Values of 0 or below will be ignored and the default value will be used. | 1 | +| xref:{s-path}/gateway.adoc[Gateway] | GATEWAY_APP_REGISTRY_ENDPOINT | The endpoint of the app-registry service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | com.owncloud.api.app-registry | +| | GATEWAY_AUTH_APP_ENDPOINT | The endpoint of the auth-app service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | com.owncloud.api.auth-app | +| | GATEWAY_AUTH_BASIC_ENDPOINT | The endpoint of the auth-basic service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | com.owncloud.api.auth-basic | +| | GATEWAY_AUTH_BEARER_ENDPOINT | The endpoint of the auth-bearer service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | | +| | GATEWAY_AUTH_MACHINE_ENDPOINT | The endpoint of the auth-machine service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | com.owncloud.api.auth-machine | +| | GATEWAY_AUTH_SERVICE_ENDPOINT | The endpoint of the auth-service service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | com.owncloud.api.auth-service | +| | GATEWAY_GROUPS_ENDPOINT | The endpoint of the groups service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | com.owncloud.api.groups | +| | GATEWAY_OCM_ENDPOINT | The endpoint of the ocm service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | com.owncloud.api.ocm | +| | GATEWAY_PERMISSIONS_ENDPOINT | The endpoint of the permissions service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | com.owncloud.api.settings | +| | GATEWAY_SHARING_ENDPOINT | The endpoint of the shares service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | com.owncloud.api.sharing | +| | GATEWAY_STORAGE_PUBLIC_LINK_ENDPOINT | The endpoint of the storage-publiclink service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | com.owncloud.api.storage-publiclink | +| | GATEWAY_STORAGE_SHARES_ENDPOINT | The endpoint of the storage-shares service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | com.owncloud.api.storage-shares | +| | GATEWAY_STORAGE_USERS_ENDPOINT | The endpoint of the storage-users service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | com.owncloud.api.storage-users | +| | GATEWAY_USERS_ENDPOINT | The endpoint of the users service. Can take a service name or a gRPC URI with the dns, kubernetes or unix protocol. | com.owncloud.api.users | +| xref:{s-path}/graph.adoc[Graph] | GRAPH_AVAILABLE_ROLES | A comma separated list of roles that are available for assignment. | [b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5 a8d5fe5e-96e3-418d-825b-534dbdf22b99 fb6c3e19-e378-47e5-b277-9732f9de6e21 58c63c02-1d89-4572-916a-870abc5a1b7d 2d00ce52-1fc2-4dbc-8b95-a73b73395f5a 1c996275-f1c9-4e71-abdf-a42f6495e960 312c0871-5ef7-4b3a-85b6-0e4074c64049] | +| | GRAPH_MAX_CONCURRENCY | The maximum number of concurrent requests the service will handle. | 20 | +| | GRAPH_TRANSLATION_PATH | (optional) Set this to a path with custom translations to overwrite the builtin translations. Note that file and folder naming rules apply, see the documentation for more details. | | +| xref:{s-path}/ocm.adoc[OCM] | OCM_OCM_INVITE_MANAGER_TIMEOUT | Timeout specifies a time limit for requests made to OCM endpoints. | 30s | +| | OCM_OCM_INVITE_MANAGER_TOKEN_EXPIRATION | Expiry duration for invite tokens. | 24h0m0s | +| | OCM_OCM_STORAGE_DATA_SERVER_URL | URL of the data server, needs to be reachable by the data gateway provided by the frontend service or the user if directly exposed. | http://localhost:9280/data | +| xref:{s-path}/postprocessing.adoc[Postprocessing] | POSTPROCESSING_WORKERS | The number of concurrent go routines that fetch events from the event queue. | 3 | +| xref:{s-path}/proxy.adoc[Proxy] | PROXY_AUTOPROVISION_CLAIM_DISPLAYNAME | The name of the OIDC claim that holds the display name. | name | +| | PROXY_AUTOPROVISION_CLAIM_EMAIL | The name of the OIDC claim that holds the email. | email | +| | PROXY_AUTOPROVISION_CLAIM_GROUPS | The name of the OIDC claim that holds the groups. | groups | +| | PROXY_AUTOPROVISION_CLAIM_USERNAME | The name of the OIDC claim that holds the username. | preferred_username | +| | PROXY_CSP_CONFIG_FILE_LOCATION | The location of the CSP configuration file. | | +| | PROXY_ENABLE_APP_AUTH | Allow app authentication. This can be used to authenticate 3rd party applications. Note that auth-app service must be running for this feature to work. | false | +| | PROXY_EVENTS_AUTH_PASSWORD | The password to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | PROXY_EVENTS_AUTH_USERNAME | The username to authenticate with the events broker. The events broker is the ocis service which receives and delivers events between the services. | | +| | PROXY_EVENTS_CLUSTER | The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. | ocis-cluster | +| | PROXY_EVENTS_ENABLE_TLS | Enable TLS for the connection to the events broker. The events broker is the ocis service which receives and delivers events between the services. | false | +| | PROXY_EVENTS_ENDPOINT | The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Set to a empty string to disable emitting events. | 127.0.0.1:9233 | +| | PROXY_EVENTS_TLS_INSECURE | Whether to verify the server TLS certificates. | false | +| | PROXY_EVENTS_TLS_ROOT_CA_CERTIFICATE | The root CA certificate used to validate the server’s TLS certificate. If provided PROXY_EVENTS_TLS_INSECURE will be seen as false. | | +| xref:{s-path}/sharing.adoc[Sharing] | SHARING_USER_JSONCS3_MAX_CONCURRENCY | Maximum number of concurrent go-routines. Higher values can potentially get work done faster but will also cause more load on the system. Values of 0 or below will be ignored and the default value will be used. | 1 | +| xref:{s-path}/sse.adoc[SSE] | SSE_KEEPALIVE_INTERVAL | To prevent intermediate proxies from closing the SSE connection, send periodic SSE comments to keep it open. | 0s | +| xref:{s-path}/storage-users.adoc[Storage-Users] | STORAGE_USERS_OCIS_GENERAL_SPACE_PATH_TEMPLATE | Template string to construct the paths of the projects space roots. | | +| | STORAGE_USERS_OCIS_MAX_CONCURRENCY | Maximum number of concurrent go-routines. Higher values can potentially get work done faster but will also cause more load on the system. Values of 0 or below will be ignored and the default value will be used. | 1 | +| | STORAGE_USERS_OCIS_PERSONAL_SPACE_PATH_TEMPLATE | Template string to construct the paths of the personal space roots. | | +| | STORAGE_USERS_PERMISSION_ENDPOINT | Endpoint of the permissions service. The endpoints can differ for 'ocis', 'posix' and 's3ng'. | com.owncloud.api.settings | +| | STORAGE_USERS_POSIX_GENERAL_SPACE_PATH_TEMPLATE | Template string to construct the paths of the projects space roots. | projects/{{.SpaceId}} | +| | STORAGE_USERS_POSIX_PERMISSIONS_ENDPOINT | Endpoint of the permissions service. The endpoints can differ for 'ocis', 'posix' and 's3ng'. | com.owncloud.api.settings | +| | STORAGE_USERS_POSIX_PERSONAL_SPACE_PATH_TEMPLATE | Template string to construct the paths of the personal space roots. | users/{{.User.Username}} | +| | STORAGE_USERS_POSIX_ROOT | The directory where the filesystem storage will store its data. If not defined, the root directory derives from $OCIS_BASE_DATA_PATH/storage/users. | /var/lib/ocis/storage/users | +| | STORAGE_USERS_POSIX_SCAN_DEBOUNCE_DELAY | The time in milliseconds to wait before scanning the filesystem for changes after a change has been detected. | 1s | +| | STORAGE_USERS_POSIX_USE_SPACE_GROUPS | Use space groups to manage permissions on spaces. | false | +| | STORAGE_USERS_POSIX_WATCH_FOLDER_KAFKA_BROKERS | Comma-separated list of kafka brokers to read the watchfolder events from. | | +| | STORAGE_USERS_POSIX_WATCH_PATH | Path to the watch directory/file. Only applies to the 'gpfsfileauditlogging' and 'inotifywait' watcher, in which case it is the path of the file audit log file/base directory to watch. | | +| | STORAGE_USERS_POSIX_WATCH_TYPE | Type of the watcher to use for getting notified about changes to the filesystem. Currently available options are 'inotifywait' (default), 'gpfswatchfolder' and 'gpfsfileauditlogging'. | | +| | STORAGE_USERS_S3NG_GENERAL_SPACE_PATH_TEMPLATE | Template string to construct the paths of the projects space roots. | | +| | STORAGE_USERS_S3NG_PERSONAL_SPACE_PATH_TEMPLATE | Template string to construct the paths of the personal space roots. | | +| | STORAGE_USERS_SERVICE_NAME | Service name to use. Change this when starting an additional storage provider with a custom configuration to prevent it from colliding with the default 'storage-users' service. | storage-users | +| xref:{s-path}/thumbnails.adoc[Thumbnails] | THUMBNAILS_MAX_CONCURRENT_REQUESTS | Number of maximum concurrent thumbnail requests. Default is 0 which is unlimited. | 0 | +| | THUMBNAILS_MAX_INPUT_HEIGHT | The maximum height of an input image which is being processed. | 7680 | +| | THUMBNAILS_MAX_INPUT_IMAGE_FILE_SIZE | The maximum file size of an input image which is being processed. Usable common abbreviations: [KB, KiB, MB, MiB, GB, GiB, TB, TiB, PB, PiB, EB, EiB], example: 2GB. | 50MB | +| | THUMBNAILS_MAX_INPUT_WIDTH | The maximum width of an input image which is being processed. | 7680 | +| xref:{s-path}/userlog.adoc[Userlog] | USERLOG_MAX_CONCURRENCY | Maximum number of concurrent go-routines. Higher values can potentially get work done faster but will also cause more load on the system. Values of 0 or below will be ignored and the default value will be used. | 1 | +| xref:{s-path}/web.adoc[Web] | WEB_ASSET_APPS_PATH | Serve ownCloud Web apps assets from a path on the filesystem instead of the builtin assets. If not defined, the root directory derives from $OCIS_BASE_DATA_PATH/web/assets/apps | /var/lib/ocis/web/assets/apps | +| | WEB_ASSET_CORE_PATH | Serve ownCloud Web assets from a path on the filesystem instead of the builtin assets. If not defined, the root directory derives from $OCIS_BASE_DATA_PATH/web/assets/core | /var/lib/ocis/web/assets/core | +| | WEB_ASSET_THEMES_PATH | Serve ownCloud themes from a path on the filesystem instead of the builtin assets. If not defined, the root directory derives from $OCIS_BASE_DATA_PATH/web/assets/themes | /var/lib/ocis/web/assets/themes | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/5.0.0-7.0.0-deprecated.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/5.0.0-7.0.0-deprecated.adoc new file mode 100644 index 00000000..5090b9b0 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/5.0.0-7.0.0-deprecated.adoc @@ -0,0 +1,22 @@ +:toc: right +:toclevels: 3 + +Note that the links provided in the service column are non functional when clicked. + +| Service | Variable | Description | Removal Version | Deprecation Info | +| --- | --- | --- | --- | --- | +| xref:{s-path}/clientlog.adoc[Clientlog] | CLIENTLOG_REVA_GATEWAY | CS3 gateway used to look up user metadata | %%NEXT_PRODUCTION_VERSION%% | CLIENTLOG_REVA_GATEWAY removed for simplicity. | +| xref:{s-path}/frontend.adoc[Frontend] | FRONTEND_OCS_ADDITIONAL_INFO_ATTRIBUTE | Additional information attribute for the user like {{.Mail}}. | %%NEXT_PRODUCTION_VERSION%% | The OCS API is deprecated | +| | FRONTEND_OCS_ENABLE_DENIALS | EXPERIMENTAL: enable the feature to deny access on folders. | %%NEXT_PRODUCTION_VERSION%% | The OCS API is deprecated | +| | FRONTEND_OCS_INCLUDE_OCM_SHAREES | Include OCM sharees when listing sharees. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_INCLUDE_OCM_SHAREES, the OCS API is deprecated | +| | FRONTEND_OCS_LIST_OCM_SHARES | Include OCM shares when listing shares. See the OCM service documentation for more details. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_LIST_OCM_SHARES, the OCS API is deprecated | +| | FRONTEND_OCS_PERSONAL_NAMESPACE | Home namespace identifier. | %%NEXT_PRODUCTION_VERSION%% | The OCS API is deprecated | +| | FRONTEND_OCS_PREFIX | URL path prefix for the OCS service. Note that the string must not start with '/'. | %%NEXT_PRODUCTION_VERSION%% | The OCS API is deprecated | +| | FRONTEND_OCS_SHARE_PREFIX | Path prefix for shares as part of an ocis resource. Note that the path must start with '/'. | %%NEXT_PRODUCTION_VERSION%% | The OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_AUTH_PASSWORD | The password to use for authentication. Only applies when using the 'nats-js-kv' store type. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_STAT_CACHE_AUTH_PASSWORD, the OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_AUTH_USERNAME | The username to use for authentication. Only applies when using the 'nats-js-kv' store type. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_STAT_CACHE_AUTH_USERNAME, the OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_DISABLE_PERSISTENCE | Disable persistence of the cache. Only applies when using the 'nats-js-kv' store type. Defaults to false. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_STAT_CACHE_DISABLE_PERSISTENCE, the OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_STORE | The type of the cache store. Supported values are: 'memory', 'redis-sentinel', 'nats-js-kv', 'noop'. See the text description for details. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_STAT_CACHE_STORE, the OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_STORE_NODES | A list of nodes to access the configured store. This has no effect when 'memory' store is configured. Note that the behaviour how nodes are used is dependent on the library of the configured store. See the Environment Variable Types description for more details. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_STAT_CACHE_STORE_NODES, the OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_TABLE | The database table the store should use. | %%NEXT_PRODUCTION_VERSION%% | The OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_TTL | Default time to live for user info in the cache. Only applied when access tokens has no expiration. See the Environment Variable Types description for more details. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_STAT_CACHE_TTL, the OCS API is deprecated | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/5.0.0-7.0.0-removed.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/5.0.0-7.0.0-removed.adoc new file mode 100644 index 00000000..ac2b4c28 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/5.0.0-7.0.0-removed.adoc @@ -0,0 +1,56 @@ +:toc: right +:toclevels: 3 + +Note that the links provided in the service column are non functional when clicked. + +| Service | Variable | Description | Default | +| --- | --- | --- | --- | +| xref:deployment/services/env-vars-special-scope.adoc[Special Scope Envvars] | OCIS_CACHE_SIZE | The maximum quantity of items in the user info cache. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived from the ocmem package though not exclicitly set as default. | 0 | +| | OCIS_DECOMPOSEDFS_METADATA_BACKEND | The backend to use for storing metadata. Supported values are 'messagepack' and 'xattrs'. The setting 'messagepack' uses a dedicated file to store file metadata while 'xattrs' uses extended attributes to store file metadata. Defaults to 'messagepack'. | messagepack | +| | OCIS_ENABLE_RESHARING | Changing this value is NOT supported. Enables the support for re-sharing in the clients. | false | +| | OCIS_PERSISTENT_STORE_SIZE | The maximum quantity of items in the store. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived from the ocmem package though not exclicitly set as default. | 0 | +| xref:{s-path}/antivirus.adoc[Antivirus] | ANTIVIRUS_ICAP_TIMEOUT | Timeout for the ICAP client. | 0 | +| xref:{s-path}/eventhistory.adoc[Eventhistory] | EVENTHISTORY_STORE_SIZE | The maximum quantity of items in the store. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived and used from the ocmem package though no explicit default was set. | 0 | +| xref:{s-path}/frontend.adoc[Frontend] | FRONTEND_ENABLE_RESHARING | Changing this value is NOT supported. Enables the support for re-sharing in the clients. | false | +| | FRONTEND_OCS_STAT_CACHE_SIZE | Max number of entries to hold in the cache. | 0 | +| xref:{s-path}/gateway.adoc[Gateway] | GATEWAY_CREATE_HOME_CACHE_SIZE | The maximum quantity of items in the cache. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived from the ocmem package though not explicitly set as default. | 0 | +| | GATEWAY_PROVIDER_CACHE_SIZE | The maximum quantity of items in the cache. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived from the ocmem package though not explicitly set as default. | 0 | +| xref:{s-path}/graph.adoc[Graph] | GRAPH_CACHE_SIZE | The maximum quantity of items in the store. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived from the ocmem package though not explicitly set as default. | 0 | +| | GRAPH_ENABLE_RESHARING | Changing this value is NOT supported. Enables the support for re-sharing. | false | +| xref:{s-path}/ocm.adoc[OCM] | OCM_OCM_PROVIDER_AUTHORIZER_VERIFY_REQUEST_HOSTNAME | Verify the hostname of the incoming request against the hostname of the OCM provider. | false | +| xref:{s-path}/postprocessing.adoc[Postprocessing] | POSTPROCESSING_STORE_SIZE | The maximum quantity of items in the store. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived from the ocmem package though not exclicitly set as default. | 0 | +| xref:{s-path}/proxy.adoc[Proxy] | PROXY_OIDC_USERINFO_CACHE_SIZE | The maximum quantity of items in the user info cache. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived from the ocmem package though not explicitly set as default. | 0 | +| xref:{s-path}/settings.adoc[Settings] | SETTINGS_CACHE_SIZE | The maximum quantity of items in the cache. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived from the ocmem package though not exclicitly set as default. | 0 | +| | SETTINGS_DATA_PATH | The directory where the filesystem storage will store ocis settings. If not defined, the root directory derives from $OCIS_BASE_DATA_PATH:/settings. | /var/lib/ocis/settings | +| | SETTINGS_STORE_TYPE | Store type configures the persistency driver. Supported values are 'metadata' and 'filesystem'. Note that the value 'filesystem' is considered deprecated. | metadata | +| xref:{s-path}/sharing.adoc[Sharing] | SHARING_ENABLE_RESHARING | Changing this value is NOT supported. Enables the support for resharing. | false | +| xref:{s-path}/storage-system.adoc[Storage-System] | STORAGE_SYSTEM_CACHE_SIZE | The maximum quantity of items in the user info cache. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived from the ocmem package though not exclicitly set as default. | 0 | +| | STORAGE_SYSTEM_OCIS_METADATA_BACKEND | The backend to use for storing metadata. Supported values are 'messagepack' and 'xattrs'. The setting 'messagepack' uses a dedicated file to store file metadata while 'xattrs' uses extended attributes to store file metadata. Defaults to 'messagepack'. | messagepack | +| xref:{s-path}/storage-users.adoc[Storage-Users] | STORAGE_USERS_FILEMETADATA_CACHE_SIZE | The maximum quantity of items in the user info cache. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived from the ocmem package though not exclicitly set as default. | 0 | +| | STORAGE_USERS_ID_CACHE_SIZE | The maximum quantity of items in the user info cache. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived from the ocmem package though not exclicitly set as default. | 0 | +| | STORAGE_USERS_OCIS_METADATA_BACKEND | The backend to use for storing metadata. Supported values are 'messagepack' and 'xattrs'. The setting 'messagepack' uses a dedicated file to store file metadata while 'xattrs' uses extended attributes to store file metadata. Defaults to 'messagepack'. | messagepack | +| | STORAGE_USERS_S3NG_METADATA_BACKEND | The backend to use for storing metadata. Supported values are 'xattrs' and 'messagepack'. The setting 'xattrs' uses extended attributes to store file metadata while 'messagepack' uses a dedicated file to store file metadata. Defaults to 'xattrs'. | messagepack | +| The `Store` service has been removed completely | STORE_DATA_PATH | The directory where the filesystem storage will store ocis settings. If not defined, the root directory derives from $OCIS_BASE_DATA_PATH:/store. | /var/lib/ocis/store | +| | STORE_DEBUG_ADDR | Bind address of the debug server, where metrics, health, config and debug endpoints will be exposed. | 127.0.0.1:9464 | +| | STORE_DEBUG_PPROF | Enables pprof, which can be used for profiling. | false | +| | STORE_DEBUG_TOKEN | Token to secure the metrics endpoint. | | +| | STORE_DEBUG_ZPAGES | Enables zpages, which can be used for collecting and viewing in-memory traces. | false | +| | STORE_GRPC_ADDR | The bind address of the GRPC service. | 127.0.0.1:9460 | +| | STORE_LOG_COLOR | Activates colorized log output. | false | +| | STORE_LOG_FILE | The path to the log file. Activates logging to this file if set. | | +| | STORE_LOG_LEVEL | The log level. Valid values are: 'panic', 'fatal', 'error', 'warn', 'info', 'debug', 'trace'. | | +| | STORE_LOG_PRETTY | Activates pretty log output. | false | +| | STORE_TRACING_COLLECTOR | The HTTP endpoint for sending spans directly to a collector, i.e. http://jaeger-collector:14268/api/traces. Only used if the tracing endpoint is unset. | | +| | STORE_TRACING_ENABLED | Activates tracing. | false | +| | STORE_TRACING_ENDPOINT | The endpoint of the tracing agent. | | +| | STORE_TRACING_TYPE | The type of tracing. Defaults to '', which is the same as 'jaeger'. Allowed tracing types are 'jaeger' and '' as of now. | | +| xref:{s-path}/userlog.adoc[Userlog] | USERLOG_STORE_SIZE | The maximum quantity of items in the store. Only applies when store type 'ocmem' is configured. Defaults to 512 which is derived from the ocmem package though not exclicitly set as default. | 0 | +| xref:{s-path}/web.adoc[Web] | WEB_ASSET_PATH | Serve ownCloud Web assets from a path on the filesystem instead of the builtin assets. | | +| | WEB_OPTION_DISABLE_PREVIEWS | Set this option to 'true' to disable previews in all the different web file listing views. This can speed up file listings in folders with many files. The only list view that is not affected by this setting is the trash bin, as it does not allow previewing at all. | false | +| | WEB_OPTION_HOME_FOLDER | Specifies a folder that is used when the user navigates 'home'. Navigating home gets triggered by clicking on the 'All files' menu item. The user will not be jailed in that directory, it simply serves as a default location. A static location can be provided, or variables of the user object to come up with a user specific home path can be used. This uses the twig template variable style and allows picking a value or a substring of a value of the authenticated user. Examples are '/Shares', '/{{.Id}}' and '/{{substr 0 3 .Id}}/{{.Id}'. | | +| | WEB_OPTION_HOVERABLE_QUICK_ACTIONS | Set this option to 'true' to hide quick actions (buttons appearing on file rows) and only show them when the user hovers over the row with his mouse. Defaults to 'false'. | false | +| | WEB_OPTION_OPEN_LINKS_WITH_DEFAULT_APP | Specifies whether single file link shares should be opened with the default app or not. If not opened by the default app, the Web UI just displays the file details. | true | +| | WEB_OPTION_PREVIEW_FILE_MIMETYPES | A list of mimeTypes to specify which ones will be previewed in the UI. For example, to only preview jpg and text files, set this option to 'image/jpeg,text/plain'. See the Environment Variable Types description for more details. | [image/gif image/png image/jpeg text/plain image/tiff image/bmp image/x-ms-bmp application/vnd.geogebra.slides] | +| | WEB_OPTION_ROUTING_ID_BASED | Enable or disable fileIds being added to the URL. Defaults to 'true', because otherwise spaces with name clashes cannot be resolved correctly. Note: Only disable this if you can guarantee on the server side, that spaces of the same namespace cannot have name clashes. | true | +| | WEB_OPTION_SHARING_RECIPIENTS_PER_PAGE | Sets the number of users shown as recipients in the dropdown menu when sharing resources. | 200 | +| | WEB_OPTION_SIDEBAR_SHARES_SHOW_ALL_ON_LOAD | Sets the list of the (link) shares list in the sidebar to be initially expanded. Default is a collapsed state, only showing the first three shares. | false | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.0.0-7.1.0-added.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.0.0-7.1.0-added.adoc new file mode 100644 index 00000000..16e11618 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.0.0-7.1.0-added.adoc @@ -0,0 +1,16 @@ +:toc: right +:toclevels: 3 + +Note that the links provided in the service column are non functional when clicked. + +| Service | Variable | Description | Default | +| --- | --- | --- | --- | +| xref:{s-path}/frontend.adoc[Fontend] | FRONTEND_CONFIGURABLE_NOTIFICATIONS | Allow configuring notifications via web client. | false | +| xref:{s-path}/notifications.adoc[Notifications] | NOTIFICATIONS_STORE | The type of the store. Supported values are: 'memory', 'nats-js-kv', 'redis-sentinel', 'noop'. See the text description for details. | nats-js-kv | +| | NOTIFICATIONS_STORE_AUTH_PASSWORD | The password to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| | NOTIFICATIONS_STORE_AUTH_USERNAME | The username to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| | NOTIFICATIONS_STORE_DATABASE | The database name the configured store should use. | notifications | +| | NOTIFICATIONS_STORE_NODES | A list of nodes to access the configured store. This has no effect when 'memory' store is configured. Note that the behaviour how nodes are used is dependent on the library of the configured store. See the Environment Variable Types description for more details. | [127.0.0.1:9233] | +| | NOTIFICATIONS_STORE_TABLE | The database table the store should use. | | +| | NOTIFICATIONS_STORE_TTL | Time to live for notifications in the store. Defaults to '336h' (2 weeks). See the Environment Variable Types description for more details. | 336h0m0s | +| xref:{s-path}/settings.adoc[Settings] | SETTINGS_TRANSLATION_PATH | (optional) Set this to a path with custom translations to overwrite the builtin translations. Note that file and folder naming rules apply, see the documentation for more details. | | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.0.0-7.1.0-deprecated.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.0.0-7.1.0-deprecated.adoc new file mode 100644 index 00000000..0e2b8a5e --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.0.0-7.1.0-deprecated.adoc @@ -0,0 +1,24 @@ +:toc: right +:toclevels: 3 + +Note that the links provided in the service column are non functional when clicked. + +Note that these environment variables were listed for deprecation in 7.0.0 already. Removal has been delayed for an upcoming production version. They are listed here for consistency. + +| Service | Variable | Description | Removal Version | Deprecation Info | +| --- | --- | --- | --- | --- | +| xref:{s-path}/clientlog.adoc[Clientlog] | CLIENTLOG_REVA_GATEWAY | CS3 gateway used to look up user metadata | %%NEXT_PRODUCTION_VERSION%% | CLIENTLOG_REVA_GATEWAY removed for simplicity. | +| xref:{s-path}/frontend.adoc[Frontend] | FRONTEND_OCS_ADDITIONAL_INFO_ATTRIBUTE | Additional information attribute for the user like {{.Mail}}. | %%NEXT_PRODUCTION_VERSION%% | The OCS API is deprecated | +| | FRONTEND_OCS_ENABLE_DENIALS | EXPERIMENTAL: enable the feature to deny access on folders. | %%NEXT_PRODUCTION_VERSION%% | The OCS API is deprecated | +| | FRONTEND_OCS_INCLUDE_OCM_SHAREES | Include OCM sharees when listing sharees. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_INCLUDE_OCM_SHAREES, the OCS API is deprecated | +| | FRONTEND_OCS_LIST_OCM_SHARES | Include OCM shares when listing shares. See the OCM service documentation for more details. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_LIST_OCM_SHARES, the OCS API is deprecated | +| | FRONTEND_OCS_PERSONAL_NAMESPACE | Home namespace identifier. | %%NEXT_PRODUCTION_VERSION%% | The OCS API is deprecated | +| | FRONTEND_OCS_PREFIX | URL path prefix for the OCS service. Note that the string must not start with '/'. | %%NEXT_PRODUCTION_VERSION%% | The OCS API is deprecated | +| | FRONTEND_OCS_SHARE_PREFIX | Path prefix for shares as part of an ocis resource. Note that the path must start with '/'. | %%NEXT_PRODUCTION_VERSION%% | The OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_AUTH_PASSWORD | The password to use for authentication. Only applies when using the 'nats-js-kv' store type. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_STAT_CACHE_AUTH_PASSWORD, the OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_AUTH_USERNAME | The username to use for authentication. Only applies when using the 'nats-js-kv' store type. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_STAT_CACHE_AUTH_USERNAME, the OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_DISABLE_PERSISTENCE | Disable persistence of the cache. Only applies when using the 'nats-js-kv' store type. Defaults to false. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_STAT_CACHE_DISABLE_PERSISTENCE, the OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_STORE | The type of the cache store. Supported values are: 'memory', 'redis-sentinel', 'nats-js-kv', 'noop'. See the text description for details. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_STAT_CACHE_STORE, the OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_STORE_NODES | A list of nodes to access the configured store. This has no effect when 'memory' store is configured. Note that the behaviour how nodes are used is dependent on the library of the configured store. See the Environment Variable Types description for more details. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_STAT_CACHE_STORE_NODES, the OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_TABLE | The database table the store should use. | %%NEXT_PRODUCTION_VERSION%% | The OCS API is deprecated | +| | FRONTEND_OCS_STAT_CACHE_TTL | Default time to live for user info in the cache. Only applied when access tokens has no expiration. See the Environment Variable Types description for more details. | %%NEXT_PRODUCTION_VERSION%% | FRONTEND_OCS_STAT_CACHE_TTL, the OCS API is deprecated | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.0.0-7.1.0-removed.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.0.0-7.1.0-removed.adoc new file mode 100644 index 00000000..e09f6cd0 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.0.0-7.1.0-removed.adoc @@ -0,0 +1,7 @@ +:toc: right +:toclevels: 3 + +Note that the links provided in the service column are non functional when clicked. + +| Service | Variable | Description | Default | +| --- | --- | --- | --- | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.1.0-7.2.0-added.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.1.0-7.2.0-added.adoc new file mode 100644 index 00000000..a1711022 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.1.0-7.2.0-added.adoc @@ -0,0 +1,13 @@ +:toc: right +:toclevels: 3 + +Note that the links provided in the service column are non functional when clicked. + +| Service | Variable | Description | Default | +| --- | --- | --- | --- | +| xref:deployment/services/env-vars-special-scope.adoc[Special Scope Envvars] | OCIS_CLAIM_MANAGED_SPACES_CLAIMNAME | The name of the claim used to manage Spaces. | | +| | OCIS_CLAIM_MANAGED_SPACES_ENABLED | Enables Space management through OIDC claims. See the text description for more details. | false | +| | OCIS_CLAIM_MANAGED_SPACES_MAPPING | (Optional) Mapping of OIDC roles to ocis Space roles. Example: 'oidcroleA:viewer,oidcroleB:manager' | [] | +| | OCIS_CLAIM_MANAGED_SPACES_REGEXP | The regular expression that extracts Space IDs and roles from a claim. | | +| | OCIS_MAX_TAG_LENGTH | Define the maximum tag length. Defaults to 100 if not set. Set to 0 to not limit the tag length. Changes only impact the validation of new tags. | 100 | +| xref:{s-path}/search.adoc[search] | SEARCH_ENGINE_BLEVE_SCALE | Enable scaling of the search index (bleve). If set to 'true', the instance of the search service will no longer have exclusive write access to the index. Note when scaling search, all instances of the search service must be set to true! For 'false', which is the default, the running search service has exclusive access to the index as long it is running. This locks out other search processes tying to access the index. | false | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.1.0-7.2.0-deprecated.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.1.0-7.2.0-deprecated.adoc new file mode 100644 index 00000000..92f69c83 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.1.0-7.2.0-deprecated.adoc @@ -0,0 +1,7 @@ +:toc: right +:toclevels: 3 + +Note that the links provided in the service column are non functional when clicked. + +| Service | Variable | Description | Removal Version | Deprecation Info | +| --- | --- | --- | --- | --- | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.1.0-7.2.0-removed.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.1.0-7.2.0-removed.adoc new file mode 100644 index 00000000..d920e658 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.1.0-7.2.0-removed.adoc @@ -0,0 +1,24 @@ +:toc: right +:toclevels: 3 + +Note that the links provided in the service column are non functional when clicked. + +| Service | Variable | Description | Default | +| --- | --- | --- | --- | +| xref:{s-path}/clientlog.adoc[clientlog] | CLIENTLOG_REVA_GATEWAY | CS3 gateway used to look up user metadata | com.owncloud.api.gateway | +| xref:{s-path}/frontend.adoc[frontend] | FRONTEND_OCS_ADDITIONAL_INFO_ATTRIBUTE | Additional information attribute for the user like {{.Mail}}. | {{.Mail}} | +| | FRONTEND_OCS_ENABLE_DENIALS | EXPERIMENTAL: enable the feature to deny access on folders. | false | +| | FRONTEND_OCS_INCLUDE_OCM_SHAREES | Include OCM sharees when listing sharees. | false | +| | FRONTEND_OCS_LIST_OCM_SHARES | Include OCM shares when listing shares. See the OCM service documentation for more details. | true | +| | FRONTEND_OCS_PERSONAL_NAMESPACE | Home namespace identifier. | /users/{{.Id.OpaqueId}} | +| | FRONTEND_OCS_PREFIX | URL path prefix for the OCS service. Note that the string must not start with '/'. | ocs | +| | FRONTEND_OCS_PUBLIC_SHARE_MUST_HAVE_PASSWORD | Set this to true if you want to enforce passwords on all public shares. | true | +| | FRONTEND_OCS_PUBLIC_WRITEABLE_SHARE_MUST_HAVE_PASSWORD | Set this to true if you want to enforce passwords for writable shares. Only effective if the setting for 'passwords on all public shares' is set to false. | false | +| | FRONTEND_OCS_SHARE_PREFIX | Path prefix for shares as part of an ocis resource. Note that the path must start with '/'. | /Shares | +| | FRONTEND_OCS_STAT_CACHE_AUTH_PASSWORD | The password to use for authentication. Only applies when using the 'nats-js-kv' store type. | | +| | FRONTEND_OCS_STAT_CACHE_AUTH_USERNAME | The username to use for authentication. Only applies when using the 'nats-js-kv' store type. | | +| | FRONTEND_OCS_STAT_CACHE_DISABLE_PERSISTENCE | Disable persistence of the cache. Only applies when using the 'nats-js-kv' store type. Defaults to false. | false | +| | FRONTEND_OCS_STAT_CACHE_STORE | The type of the cache store. Supported values are: 'memory', 'redis-sentinel', 'nats-js-kv', 'noop'. See the text description for details. | memory | +| | FRONTEND_OCS_STAT_CACHE_STORE_NODES | A list of nodes to access the configured store. This has no effect when 'memory' store is configured. Note that the behaviour how nodes are used is dependent on the library of the configured store. See the Environment Variable Types description for more details. | [127.0.0.1:9233] | +| | FRONTEND_OCS_STAT_CACHE_TABLE | The database table the store should use. | | +| | FRONTEND_OCS_STAT_CACHE_TTL | Default time to live for user info in the cache. Only applied when access tokens has no expiration. See the Environment Variable Types description for more details. | 5m0s | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.2.0-7.3.0-added.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.2.0-7.3.0-added.adoc new file mode 100644 index 00000000..14dd46f1 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.2.0-7.3.0-added.adoc @@ -0,0 +1,7 @@ +:toc: right +:toclevels: 3 + +| Service | Variable | Description | Default | +| --- | --- | --- | --- | +| collaboration | COLLABORATION_APP_PRODUCT_EDITION | The WebOffice app edition defines the capabilities specific to the product such as CE - Community Edition, EE - Enterprise Edition DE - Developer Edition, etc. Currently supported values are limited to OnlyOffice and are: 'ce', 'ee' or 'de' and default to empty which is equal to ce). See the documentation for more details. | | +| | COLLABORATION_WOPI_ENABLE_MOBILE | Enable the mobile web view for office app. This feature applies to OnlyOffice. See the documentation for more details. | false | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.2.0-7.3.0-deprecated.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.2.0-7.3.0-deprecated.adoc new file mode 100644 index 00000000..48a88a18 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.2.0-7.3.0-deprecated.adoc @@ -0,0 +1,5 @@ +:toc: right +:toclevels: 3 + +| Service | Variable | Description | Removal Version | Deprecation Info | +| --- | --- | --- | --- | --- | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.2.0-7.3.0-removed.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.2.0-7.3.0-removed.adoc new file mode 100644 index 00000000..285e239b --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.2.0-7.3.0-removed.adoc @@ -0,0 +1,5 @@ +:toc: right +:toclevels: 3 + +| Service | Variable | Description | Default | +| --- | --- | --- | --- | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.3.0-8.0.0-added.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.3.0-8.0.0-added.adoc new file mode 100644 index 00000000..76ea44ff --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.3.0-8.0.0-added.adoc @@ -0,0 +1,32 @@ +:toc: right +:toclevels: 3 + +| Service | Variable | Description | Default | +| --- | --- | --- | --- | +| xref:{s-path}/collaboration.adoc[Collaboration] | COLLABORATION_WOPI_DISABLED_EXTENSIONS | List of extensions to disable: Disabling an extension will make it unavailable to the Office web front end. The list is comma-separated with no spaces between the items, such as 'docx,xlsx,pptx'. | [] | +| xref:{s-path}/graph.adoc[Graph] | GRAPH_LDAP_EXTERNAL_ID_ATTRIBUTE | LDAP attribute that references the external ID of users during the provisioning process. The final ID is provided by an external identity provider. If it is not set, a default attribute will be used instead. | owncloudExternalID | +| | GRAPH_LDAP_REQUIRE_EXTERNAL_ID | If enabled, the 'OCIS_LDAP_USER_SCHEMA_EXTERNAL_ID' is used as primary identifier for the provisioning API. | false | +| | OCIS_LDAP_USER_SCHEMA_EXTERNAL_ID | LDAP attribute that references the external ID of users during the provisioning process. The final ID is provided by an external identity provider. If it is not set, a default attribute will be used instead. | owncloudExternalID | +| | OCIS_LDAP_INSTANCE_MAPPER_BASE_DN | BaseDN of the 'instancename to instanceid' mapper in LDAP. Requires OCIS_MULTI_INSTANCE_ENABLED. | | +| | OCIS_LDAP_INSTANCE_MAPPER_ENABLED | The InstanceMapper allows mapping instance names (user readable) to instance IDs (machine readable) based on an LDAP query. See other _INSTANCE_MAPPER_ env vars. Requires OCIS_MULTI_INSTANCE_ENABLED. | false | +| | OCIS_LDAP_INSTANCE_MAPPER_ID_ATTRIBUTE | LDAP Attribute of the instance ID. Requires OCIS_MULTI_INSTANCE_ENABLED. | | +| | OCIS_LDAP_INSTANCE_MAPPER_NAME_ATTRIBUTE | LDAP Attribute of the instance name. Requires OCIS_MULTI_INSTANCE_ENABLED. | | +| | OCIS_LDAP_CROSS_INSTANCE_REFERENCE_TEMPLATE | Template for the users unique reference across oCIS instances. Requires OCIS_MULTI_INSTANCE_ENABLED. | | +| | OCIS_LDAP_INSTANCE_URL_TEMPLATE | Template for the instance URL. Requires OCIS_MULTI_INSTANCE_ENABLED. | | +| | OCIS_LDAP_GROUP_AFFILIATION_ATTRIBUTE | LDAP Attribute to signal which instance the group is belonging to. Requires OCIS_MULTI_INSTANCE_ENABLED. | | +| | OCIS_LDAP_PRECISE_SEARCH_ATTRIBUTE | LDAP Attribute to be used for searching users on other instances. Requires OCIS_MULTI_INSTANCE_ENABLED. | | +| | OCIS_LDAP_USER_GUEST_ATTRIBUTE | LDAP Attribute to signal the user is guest of an instance. Requires OCIS_MULTI_INSTANCE_ENABLED. | | +| | OCIS_LDAP_USER_MEMBER_ATTRIBUTE | LDAP Attribute to signal the user is member of an instance. Requires OCIS_MULTI_INSTANCE_ENABLED. | | +| | OCIS_MULTI_INSTANCE_QUERY_TEMPLATE | The regular expression extracting username and instancename from a user provided search. | | +| xref:{s-path}/proxy.adoc[Proxy] | PROXY_FORCE_STRICT_TRANSPORT_SECURITY | Force emission of the Strict-Transport-Security header on all responses, including plain HTTP requests. See also: PROXY_TLS | false | +| | OCIS_MULTI_INSTANCE_ENABLED | Enable multiple instances of Infinite Scale. | false | +| | OCIS_MULTI_INSTANCE_GUEST_CLAIM | The claim name for the 'guestOf' property | guestOf | +| | OCIS_MULTI_INSTANCE_GUEST_ROLE | The role that should be assigned to a guest user | user-light | +| | OCIS_MULTI_INSTANCE_INSTANCEID | The unique id of this instance | c1f28cf2-d363-4671-a8fe-8d7a80b36b87 | +| | OCIS_MULTI_INSTANCE_MEMBER_CLAIM | The claim name for the 'memberOf' property | memberOf | +| xref:{s-path}/storage-publiclink.adoc[Storage-Publiclink] | STORAGE_PUBLICLINK_STORE_STORE | The type of the store. Supported values are: 'memory', 'nats-js-kv', 'redis-sentinel', 'noop'. See the text description for details. | nats-js-kv | +| | STORAGE_PUBLICLINK_STORE_NODES | A list of nodes to access the configured store. This has no effect when 'memory' store is configured. Note that the behaviour how nodes are used is dependent on the library of the configured store. See the Environment Variable Types description for more details. | [127.0.0.1:9233] | +| | STORAGE_PUBLICLINK_STORE_DATABASE | The database name the configured store should use. | | +| | STORAGE_PUBLICLINK_STORE_TABLE | The database table the store should use. | | +| | STORAGE_PUBLICLINK_STORE_AUTH_USERNAME | The username to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | +| | STORAGE_PUBLICLINK_STORE_AUTH_PASSWORD | The password to authenticate with the store. Only applies when store type 'nats-js-kv' is configured. | | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.3.0-8.0.0-deprecated.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.3.0-8.0.0-deprecated.adoc new file mode 100644 index 00000000..48a88a18 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.3.0-8.0.0-deprecated.adoc @@ -0,0 +1,5 @@ +:toc: right +:toclevels: 3 + +| Service | Variable | Description | Removal Version | Deprecation Info | +| --- | --- | --- | --- | --- | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.3.0-8.0.0-removed.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.3.0-8.0.0-removed.adoc new file mode 100644 index 00000000..7b58b12e --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/7.3.0-8.0.0-removed.adoc @@ -0,0 +1,6 @@ +:toc: right +:toclevels: 3 + +| Service | Variable | Description | Default | +| --- | --- | --- | --- | +| xref:{s-path}/frontend.adoc[Frontend] | OCIS_SHOW_USER_EMAIL_IN_RESULTS | Include user email addresses in responses. If absent or set to false emails will be omitted from results. Please note that admin users can always see all email addresses. | false | diff --git a/modules/developer/pages/services/general-info/envvars/env-var-deltas/index.adoc b/modules/developer/pages/services/general-info/envvars/env-var-deltas/index.adoc new file mode 100644 index 00000000..291cef62 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/env-var-deltas/index.adoc @@ -0,0 +1,37 @@ += Deltas Between Versions +:toc: right +:toclevels: 3 + +== General Information + +This section provides information about `added`, `removed` and `deprecated` environment variables between two major/minor versions. + +[WARNING] +==== +* When creating a new release, this step should be completed _before_ the new `stable-x.y` branch is created. Then, all changes will go cleanly into this stable branch. +* If the changes required are not part of the stable branch created for the release process, you must backport all `added`, `removed` and `deprecated` files created from the described process below into the stable branch. Backporting `env_vars.yaml`to the stable branch is not required and can be omitted. +==== + + +To create the changed envvar tables, you must proceed the following steps in order: + +1. Install, if not already done, the converter for adoc to markdown tables: `npm install -g downdoc`\ +This is only required when converting adoc to markdown tables but it is highly recommended to show them in the dev docs too! + +1. Run `make docs-local` from the ocis root.\ +Usually, a file named `env_vars.yaml` gets changed. Check for validity. If issues are found, fix them in the service sources first which need to be merged before you rerun make. For details how to do so, see xref:../new-release-process.md#maintain-the-env_varsyaml-file.adoc[Maintain the 'env_vars.yaml' File]. Any delta information is based on an actual `env_vars.yaml` file which is pulled _from master_ by the python script described below! + +1. Configure the Python script `docs/helpers/changed_envvars.py` variables for the new version.\ +Note that you _must_ use semver and not code names! + +1. Run the python script from the ocis root such as `python3 docs/helpers/changed_envvars.py`.\ +Note that the script pulls data from the master branch as a base reference, therefore the `env_vars.yaml` file must be kept up to date. The adoc tables generated are used for the admin documentation and form the basis for Markdown. + +1. As the script cannot determine the link target (xref:) in the `Service` column, you must adapt these manually in the generated adoc files according to the file name and printed name. Envvars starting with `OCIS_` are displayed differently in the `Service` column because the file name and printed name cannot be easily identified and must be resolved differently. You have to check where they are defined, unlike the others where the name provides a clue. The final xref path must be corrected manually in all cases. Only one entry per identical service is required to generate an easy block view. Delete the cell content except for the pipe symbol (`|`) to make it easier to read. See existing files for an example. + +1. Change into the directory that contains the generated adoc files and run `npx downdoc ` for each of the newly generated `added`, `removed` and `deprecated` files. This will generate markdown files for the dev docs. + +1. Add in each markdown file on top the following sentence:\ +`Note that the links provided in the service column are non functional when clicked.`, including a newline. + +1. Create a PR and merge it. diff --git a/modules/developer/pages/services/general-info/envvars/envvar-naming-scopes.adoc b/modules/developer/pages/services/general-info/envvars/envvar-naming-scopes.adoc new file mode 100644 index 00000000..495dbe75 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/envvar-naming-scopes.adoc @@ -0,0 +1,76 @@ += Envvar Naming Scopes +:toc: right +:toclevels: 3 + + +The scope of an environment variable can be derived from its name. Therefore, it is important to follow the correct naming scheme to enable easy and proper identification. This is important when either: + +- a new local envvar is introduced. +- a new global envvar is added to an existing local envvar. + +== Envvar Definition + +- A variable that is only used in a particular service is a _local envvar_. +- A variable that is used in more than one service is a _global envvar_. +- If applicapable, a global envvar has a local counterpart. +- Variables that are not limited to any service are by definition global. + +== Naming Scope + +=== Local Envvars + +A local envvar always starts with the service name such as `POSTPROCESSING_LOG_FILE`. + +=== Global Envvars + +A global envvar always starts with `OCIS_` like `OCIS_LOG_FILE`. + +Note that this envvar is the global representation of the local example from above. + +To get a list of global envvars used in all services, see the link:https://doc.owncloud.com/ocis/next/deployment/services/env-vars-special-scope.html#global-environment-variables[Global Environment Variables] table in the ocis admin documentation. + +=== Reserved Envvar Names + +Services and their local envvars _MUST NOT_ be named `extended` or `global`. These are reserved names for the automated documentation process. + +== Lifecycle of Envvars + +The envvar struct tag contains at maximum the following key/value pairs to document the lifecycle of a config variable: + +* `introductionVersion` +* `deprecationVersion` +* `removalVersion` +* `deprecationInfo` +* `deprecationReplacement` + +=== Introduce new Envvars + +* If a _new_ envvar is introduced, the entire structure must be added, including the `introductionVersion` field. Note that 'introduced' means, that the new envvar was not present in any of the services. + +[NOTE] +==== + * During development, set the `introductionVersion` to a short, _alphabetic code name_ that represents the upcoming release such as `releaseX` or the project name for that release such as `Daledda`. + * This identifier stays constant until the release receives its final production semantic-version number. + * Although the pipeline checks the semver string when a PR is created, you can perform this check upfront manually by entering the following command from the ocis root: + + ```bash + .make/check-env-var-annotations.sh + ``` +==== + + + The doc helper scripts render these alphabetic identifiers verbatim. They appear in the next (master) branch of the admin documentation exactly as they are entered. + +* See the xref:./new-release-process/.adoc[Set the Correct IntroductionVersion] documentation before starting a new release candidate. + +=== Adding Envvars to Existing Ones + +If an envvar has been introduced with a particular release, the `introductionVersion` got a semver value accordingly. If an additional envvar is added to this existing one such as a global envvar, the introduction version _must not_ be changed. + +=== Deprecate Existing Envvars + +See the xref:./deprecating-variables.adoc[deprecation rules] documentation for more details. + +== Separating Multiple Envvars + +When multiple envvars are defined for one purpose like a global and local one, use `;` (semicolon) to properly separate the envvars in go code. Though it is possible to separate with `,` (comma) according go rules, the current implementation of the docs generation process only recognizes semicolons as separator. diff --git a/modules/developer/pages/services/general-info/envvars/index.adoc b/modules/developer/pages/services/general-info/envvars/index.adoc new file mode 100644 index 00000000..1e94ff73 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/index.adoc @@ -0,0 +1,9 @@ += Environment Variables +:toc: right +:toclevels: 3 + +== General Information + +This section provides general information how to manage environment variables. + +Additionally read the link:https://github.com/owncloud/ocis/tree/master/docs/helpers#readme[helper README] for internal details how envvars are managed. diff --git a/modules/developer/pages/services/general-info/envvars/new-release-process.adoc b/modules/developer/pages/services/general-info/envvars/new-release-process.adoc new file mode 100644 index 00000000..0544fe36 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/new-release-process.adoc @@ -0,0 +1,62 @@ += Release Process for Envvars +:toc: right +:toclevels: 3 + + +_IMPORTANT_\ +For a new ocis release, some tasks are necessary to be done _before_ releasing. Follow the steps carefully to avoid issues. Most of the docs related tasks are not part of the CI. With each step finished successfully, the next step can be started. Sometimes, due to last minute changes, steps need to be redone! + +The following can be done at any time but it must be done _latest_ when no envvar changes are made which is just before a new release gets finally tagged. The data generated _must_ be part of the upcoming release and be merged before tagging/branching! + +== Special Scope Envvars + +Ask the developers if envvars of this type have been changed (added or removed). See the xref:./special-envvars.md#special-scope-envvars.adoc[Special Envvars] documentation for more details on how to manage such a change. + +== Extended Envvars + +* From the ocis root run:\ +`sudo make docs-clean`\ +`make docs-generate`\ +Discard any changes in `env_vars.yaml`! +* Check if there is a change in the `extended-envars.yaml` output.\ +If so, process xref:./special-envvars.md#fixing-changed-items.adoc[Extended Envvars - Fixing Changed Items]. +* When done, re-run `make docs-generate` and check if the output matches the expectations in `./docs/services/_includes/adoc/extended_configvars.adoc`. + +== Ordinary Envvars + +=== Set the Correct IntroductionVersion + +* Once the release is cut, _before_ creating the first release candidate, replace them with the actual semantic version (e.g. `releaseX` → `8.1.0`). To find these placeholders in `introductionVersion` keys, you can run a helper script by issuing the following command: + ```bash + docs/ocis/helpers/identify_envvar_placeholder_names.sh + ``` + +[NOTE] +==== + A new production version _MUST NOT_ contain any alphabetic identifyers but the semantic version only, using _major, minor and a patch version, which is always 0!_. +==== + + +* Create a PR and merge it _before_ taking the next step maintaining the `env_vars.yaml` file! Do not forget to rebase your local git repo. + +=== Maintain the 'env_vars.yaml' File + +This is _mandatory for a new release_ ! + +* From the ocis root run:\ +`sudo make docs-clean`\ +`make docs-generate`\ +Any changes in `env_vars.yaml` are now considered. +* This file will most likely show changes and merging them is _essential_ as base for _added/removed or deprecated envvars_ (envvar deltas). Note that this file will get additions/updates only, but items never get deleted automatically !!\ +[NOTE] +==== +Note that due to how the code is currently designed, _things may get shifted_ around though no real changes have been introduced. +==== + +* With a new branch, remove all envvars from the `env_vars.yaml` file manually that have formerly been deprecated _and removed_ from the code. +* Commit the changes and merge it.\ +Now, `env_vars.yaml` is up to date in the repo in master. Next steps depend on this updated file! + +=== Create Envvar Delta Files + +* Create xref:./env-var-deltas/.adoc[Envvar Deltas Between Versions] files according the linked description. diff --git a/modules/developer/pages/services/general-info/envvars/special-envvars.adoc b/modules/developer/pages/services/general-info/envvars/special-envvars.adoc new file mode 100644 index 00000000..a470e840 --- /dev/null +++ b/modules/developer/pages/services/general-info/envvars/special-envvars.adoc @@ -0,0 +1,82 @@ += Special Envvars +:toc: right +:toclevels: 3 + + +Handling these envvars properly is very important for the automated doc process for both the developer and the admin docs! + +== Special Scope Envvars + +Variables with special scope are only related to a deployment method such as `OCIS_RUN_SERVICES`. These variables cannot be gathered automatically, rarely change, can be viewed and must be maintained manually via the link:https://doc.owncloud.com/ocis/next/deployment/services/env-vars-special-scope.html[admin documentation]. + +== Extended Envvars + +Environment variables with extended scope are not included in a service. They are rarely added or removed, though their code location can change during development. + +They are variables that must be present before the core or services start up because they depend on information such as the path to configuration files. Therefore, they are not bound to services like other environment variables. Extended environment variables are identified via `os.Getenv` and are usually defined via a subfolder of `ocis-pkg`. The real envvar name cannot be automatically assigned and must be defined manually via the code in the generated `extended_vars.yaml` file. + +While generating the ocis Developer Documentation, the `extended_vars.yaml` file located in `/docs/helpers/` might be updated which needs special care to finalize the process. When merged, an `extended_configvars.adoc` file is internally generated from it. This file, along with the others, is moved to the `doc` branch. The process runs automatically, and no additional effort is required. The admin docs picks this .adoc file for further processing. + +The file for the master (`docs`) branch is located at:\ +`https://github.com/owncloud/ocis/tree/docs/services/_includes/adoc/extended_configvars.adoc`\ +respectivle in any `docs-stable-x.y` branch. + +=== General Info + +The process behind updating the `extended_vars.yaml` is non-destructive. This means, that the process checks the code if values found match those in the file already present. If differences occur, only _new_ content blocks are added, independent if it is new or moved code. The file is recreated when deleted - try to avoid this and maintain the changed one. + +This also means, that if generating the docs result in a change in the `extended_vars.yaml` file, manual action _must_ be taken and the final changes need to be committed/pushed/merged. If this is not done, the `extended_configvars.adoc` will contain invalid and/or corrupt data. + +It can happen that extended envvars are found but do not need to be published as they are for internal use only. Those envvars can be defined to be ignored for further processing. + +_IMPORTANT:_ + +- _First Time Identification_\ + Once an extended envvar has been identified, it is added to the `extended_vars.yaml` file found, but never changed or touched by the process anymore. There is one exception with respect to single/double quote usage. While you can (and will) manually define a text like: `"'/var/lib/ocis'"`, quotes are transformed by the process in the .yaml file to: `'''/var/lib/ocis'''`. There is no need to change this back, as the final step transforms this correctly for the adoc table. + +- _Item Naming_\ + An extended envvar may not have the right naming. It may appear as `name: _registryEnv`. In case, this envvar needs to be named properly like `name: MICRO_REGISTRY` which can only be done in close alignment with development. + +- _Automatic Data Population_:\ + `rawname`, `path` and `foundincode` are automatically filled by the program.\ + _IMPORTANT_: DO NOT EDIT THESE VALUES MANUALLY - except the line number in the `path` key. + +- _Manual Data Population_:\ + The following keys can and must be updated manually: `name`, `type`, `default_value`, `description`, `do_ignore`\ + For the `path` key, _only_ the line number of the value may be changed, see fixing values below. + +- _Item Uniqueness_\ + The identification, if an envvar is already present in the yaml file, is made via the `rawname` and the `path` identifier which includes the line number. _If there is a change in the source file shifting line numbers, new items will get added and old ones do not get touched._ Though technically ok, this can cause confusion to identify which items are correctly present or just added additionally just be cause code location has changed. If there are multiple occurrences of the same `rawname` value, check which one contains relevant data and set `do_ignore` to `false` and all others to `true`. When there are two identical blocks with different source references, mostly the one containing a proper `default_value` is the active one. Populate the false block with the envvar data to be used. + +- _Sort Ordering_\ + Do not change the sort order of extended envvar blocks as they are automatically reordered alphabetically. + +- _Mandatory Key Values_\ + Because extended envvars do not have the same structural setup as "normal" envvars (like type, description or defaults), this info needs to be provided manually once - for each valid block. Any change of this info will be noticed during the next CI run, the corresponding `extended_configvars.adoc` file will be generated, changes moved to the docs branch and published in the next admin docs build. See the following example with all keys listed and populated: + ```yaml + rawname: registryAddressEnv + path: ocis-pkg/registry/registry.go:44 + foundincode: true + name: MICRO_REGISTRY_ADDRESS + type: string + default_value: "" + description: The bind address of the internal go micro framework. Only change on + supervision of ownCloud Support. + do_ignore: false + ``` + +=== Fixing Changed Items + +If there is a change in `extended_vars.yaml` which you can identify via git when running e.g. `make -C docs docs-generate`, read the <> section first and follow the items listed below afterwards. + +- _Fixing Items_\ + If an item has been identified as additionally added such as there was a change in the code location only, it is mostly sufficient to just fix the line number in the `path` key of the existing/correct one and double check by removing the newly added ones. Then, re-run `make -C docs docs-generate`. If the fix was correct, no new items of the same will re-appear. + +- _Remove Orphaned Items_\ + To get rid of items with wrong line numbers, check `rawname` the `path` and correct the _existing ones_, especially the one containing the description and which is marked `do_ignore` false. Only items that have a real line number match need to be present, orphaned items can safely be removed. + + You can double-check valid items by creating a dummy branch, delete the `extended_vars.yaml` and run `make -C docs docs-generate` to regenerate the file having only items with valid path references. With that info, you can remove orphaned items from the live file. Note to be careful on judging only on `foundincode` set to false indicating an item not existing anymore. Fix all items first, when rerunning `make -C docs docs-generate`, this may change back to true! + + I an envvar has been removed from the code completely, you can also remove the respective entry block from the file. + +- When all is set, create a PR and merge it. diff --git a/modules/developer/pages/services/general-info/index.adoc b/modules/developer/pages/services/general-info/index.adoc new file mode 100644 index 00000000..42909dbd --- /dev/null +++ b/modules/developer/pages/services/general-info/index.adoc @@ -0,0 +1,7 @@ += General Information +:toc: right +:toclevels: 3 + +== General Information + +This section provides general information valid for all services. diff --git a/modules/developer/pages/services/general-info/new-service-checklist.adoc b/modules/developer/pages/services/general-info/new-service-checklist.adoc new file mode 100644 index 00000000..0e28f824 --- /dev/null +++ b/modules/developer/pages/services/general-info/new-service-checklist.adoc @@ -0,0 +1,49 @@ += New Service Checklist +:toc: right +:toclevels: 3 + +When a new service gets introduced, this checklist is a good starting point for things that need to be completed before the service gets published (merged). This list is without claim of completeness or correct sort order. + +== New Service Checklist + +Use this checklist with copy/paste in your PR - right from the beginning. It renders correctly in your PR. + +[source,markdown] +---- +- [ ] Services MUST NOT be named `extended` or `global`. These are reserved names for the automated documentation process. +- [ ] Provide a README.md for that service in the root folder of that service. + - Use CamelCase for section headers. +- [ ] For images and example files used in README.md: + - Create a folder named `md-sources` on the same level where README.md is located. Put all the images and example files referenced by README.md into this folder. + - Use absolute references like `https://raw.githubusercontent.com/owncloud/ocis/master/services//md-sources/file` to make the content accessible for both README.md and owncloud.dev + bad `` + good `` +- [ ] If new CLI commands are introduced, those commands must be described in the README.md. + - Commands are added to `ocis/pkg/command` +- [ ] If new global envvars are introduced, the name must start with `OCIS_`. +- [ ] Add the service to the makefile in the ocis repo root. +- [ ] Service startup: + - add it to `ocis/pkg/command/services.go` + - Include for automatic startup: + - add it to `ocis/pkg/runtime/service/service.go` + - Exclude from automatic startup: + - add it to the `populate optional services` block in `ocis/pkg/runtime/service/service.go` + - Add the service config to `ocis-pkg/config/defaultconfig.go` +- [ ] If the service is using service accounts, add it to `ocis/pkg/init/init.go` +- [ ] Check that the service properly responds to `ocis health` and has `/healthz` and `/readyz` endpoints +- [ ] Add the service to `.drone.star` to enable CI. +- [ ] Inform doc team in an *early stage* to review the readme AND the environment variables created. + - The description must reflect the behaviour AND usually has a positive code quality impact. +- [ ] Create proper description strings for envvars - see other services for examples, especially when it comes to multiple values. This must include: + - base description, set of available values, description of each value. +- [ ] When suggested commits are created for text changes, and you agree, collect them to a batch and commit them. Do not forget to rebase locally to avoid overwriting the changes made. +- [ ] If new envvars are introduced which serve the same purpose but in multiple services, an additional envvar must be added at the beginning of the list starting with `OCIS_` (global envvar). +- [ ] Ensure that a service has a debug port +- [ ] If the new service introduces a new port: + - The port must be added to [port-ranges.md](https://github.com/owncloud/ocis/blob/master/docs/services/general-info/port-ranges.md) and to the README.md file. +- [ ] Make sure to have a function `FullDefaultConfig()` in `pkg/config/defaults/defaultconfig.go` of your service. It is needed to create the documentation. +- [ ] Add metrics to the code to enable monitoring. See the proxy service for implementation details. + - Plus add documentation about monitoring in the README.md file +- [ ] When the service requires translations that have to be covered by the service and are not sourced by web, see the [add translation]({{< ref "./add-translations.md" >}}) documentation for more details. +- [ ] If the service requires a `cache` or `store`, check existing services for implementation and add a documentation in the README.md +---- diff --git a/modules/developer/pages/services/general-info/port-ranges.adoc b/modules/developer/pages/services/general-info/port-ranges.adoc new file mode 100644 index 00000000..125ff47d --- /dev/null +++ b/modules/developer/pages/services/general-info/port-ranges.adoc @@ -0,0 +1,207 @@ += Port Ranges +:toc: right +:toclevels: 3 + +oCIS services often need a port to expose their services to other services or the outside world. As users may have many different extensions running on the same machine, we should track port usage in the oCIS ecosystem. In the best case, we ensure that each extension uses a non-colliding port range, to make life easier for users. + +This page tracks the knowingly used port ranges. + +Feel free to "reserve" a free port range when you're developing an extension by adding your extension to the list (see the edit button in the top right corner). + +If you're developing a non-public extension, we recommend using ports outside of the ranges listed below. + +We also suggest using the last port in your extensions' range as a debug/metrics port. + +== Allocations + +| Port range | Service | +|------------|----------------------------------------------------------------------------------------| +| 9000-9010 | xref:../../../ocis/index.adoc[reserved for Infinite Scale] | +| 9100-9104 | xref:../web/index.adoc[web] | +| 9105-9109 | link:https://github.com/owncloud/ocis-hello[hello] | +| 9110-9114 | xref:../ocs/index.adoc[ocs] | +| 9115-9119 | xref:../webdav/index.adoc[webdav] | +| 9120-9124 | xref:../graph/index.adoc[graph] | +| 9125-9129 | xref:../policies/index.adoc[policies] | +| 9130-9134 | xref:../idp/index.adoc[idp] | +| 9135-9139 | xref:../sse/index.adoc[sse] | +| 9140-9141 | xref:../frontend/index.adoc[frontend] | +| 9142-9143 | xref:../gateway/index.adoc[gateway] | +| 9144-9145 | xref:../users/index.adoc[users] | +| 9146-9147 | xref:../auth-basic/index.adoc[auth-basic] | +| 9148-9149 | xref:../auth-bearer/index.adoc[auth-bearer] | +| 9150-9153 | xref:../sharing/index.adoc[sharing] | +| 9154-9156 | xref:../storage-shares/index.adoc[storage-shares] | +| 9157-9159 | xref:../storage-users/index.adoc[storage-users] | +| 9160-9162 | xref:../groups/index.adoc[groups] | +| 9163 | xref:../ocdav/index.adoc[ocdav] | +| 9164 | xref:../groups/index.adoc[groups] | +| 9165 | xref:../app-provider/index.adoc[app-provider] | +| 9166-9169 | xref:../auth-machine/index.adoc[auth-machine] | +| 9170-9174 | xref:../notifications/index.adoc[notifications] | +| 9175-9179 | xref:../storage-publiclink/index.adoc[storage-publiclink] | +| 9180-9184 | FREE (formerly used by accounts) | +| 9185-9189 | xref:../thumbnails/index.adoc[thumbnails] | +| 9190-9194 | xref:../settings/index.adoc[settings] | +| 9195-9197 | xref:../activitylog/index.adoc[activitylog] | +| 9198-9199 | xref:../auth-service/index.adoc[auth-service] | +| 9200-9204 | xref:../proxy/index.adoc[proxy] | +| 9205-9209 | xref:../proxy/index.adoc[proxy] | +| 9210-9214 | xref:../userlog/index.adoc[userlog] | +| 9215-9219 | xref:../storage-system/index.adoc[storage-system] | +| 9220-9224 | xref:../search/index.adoc[search] | +| 9225-9229 | xref:../audit/index.adoc[audit] | +| 9230-9234 | xref:../nats/index.adoc[nats] | +| 9235-9239 | xref:../idm/index.adoc[idm] | +| 9240-9244 | xref:../app-registry/index.adoc[app-registry] | +| 9245-9249 | xref:../auth-app/index.adoc[auth-app] | +| 9250-9254 | link:https://github.com/owncloud/ocis/tree/master/ocis/pkg/runtime[ocis server (runtime)] | +| 9255-9259 | xref:../postprocessing/index.adoc[postprocessing] | +| 9260-9264 | xref:../clientlog/index.adoc[clientlog] | +| 9265-9269 | xref:../clientlog/index.adoc[clientlog] | +| 9270-9274 | xref:../eventhistory/index.adoc[eventhistory] | +| 9275-9279 | xref:../webfinger/index.adoc[webfinger] | +| 9280-9284 | xref:../ocm/index.adoc[ocm] | +| 9285-9289 | FREE | +| 9290-9294 | FREE | +| 9295-9299 | FREE | +| 9300-9304 | xref:../collaboration/index.adoc[collaboration] | +| 9305-9309 | FREE | +| 9310-9314 | FREE | +| 9315-9319 | FREE | +| 9320-9324 | FREE | +| 9325-9329 | FREE | +| 9330-9334 | FREE | +| 9335-9339 | FREE | +| 9340-9344 | FREE | +| 9345-9349 | FREE | +| 9350-9354 | xref:../ocdav/index.adoc[ocdav] | +| 9355-9359 | FREE | +| 9360-9364 | FREE | +| 9365-9369 | FREE | +| 9370-9374 | FREE | +| 9375-9379 | FREE | +| 9380-9384 | FREE | +| 9385-9389 | FREE | +| 9390-9394 | FREE | +| 9395-9399 | FREE | +| 9400-9404 | FREE | +| 9405-9409 | FREE | +| 9410-9414 | FREE | +| 9415-9419 | FREE | +| 9420-9424 | FREE | +| 9425-9429 | FREE | +| 9430-9434 | FREE | +| 9435-9439 | FREE | +| 9440-9444 | FREE | +| 9445-9449 | FREE | +| 9450-9454 | FREE | +| 9455-9459 | FREE | +| 9460-9464 | FREE (formerly used by store-service) | +| 9465-9469 | FREE | +| 9470-9474 | FREE | +| 9475-9479 | FREE | +| 9480-9484 | FREE | +| 9485-9489 | FREE | +| 9490-9494 | FREE | +| 9495-9499 | FREE | +| 9500-9504 | FREE | +| 9505-9509 | FREE | +| 9510-9514 | FREE | +| 9515-9519 | FREE | +| 9520-9524 | FREE | +| 9525-9529 | FREE | +| 9530-9534 | FREE | +| 9535-9539 | FREE | +| 9540-9544 | FREE | +| 9545-9549 | FREE | +| 9550-9554 | FREE | +| 9555-9559 | FREE | +| 9560-9564 | FREE | +| 9565-9569 | FREE | +| 9570-9574 | FREE | +| 9575-9579 | FREE | +| 9580-9584 | FREE | +| 9585-9589 | FREE | +| 9590-9594 | FREE | +| 9595-9599 | FREE | +| 9600-9604 | FREE | +| 9605-9609 | FREE | +| 9610-9614 | FREE | +| 9615-9619 | FREE | +| 9620-9624 | FREE | +| 9625-9629 | FREE | +| 9630-9634 | FREE | +| 9635-9639 | FREE | +| 9640-9644 | FREE | +| 9645-9649 | FREE | +| 9650-9654 | FREE | +| 9655-9659 | FREE | +| 9660-9664 | FREE | +| 9665-9669 | FREE | +| 9670-9674 | FREE | +| 9675-9679 | FREE | +| 9680-9684 | FREE | +| 9685-9689 | FREE | +| 9690-9694 | FREE | +| 9695-9699 | FREE | +| 9700-9704 | FREE | +| 9705-9709 | FREE | +| 9710-9714 | FREE | +| 9715-9719 | FREE | +| 9720-9724 | FREE | +| 9725-9729 | FREE | +| 9730-9734 | FREE | +| 9735-9739 | FREE | +| 9740-9744 | FREE | +| 9745-9749 | FREE | +| 9750-9754 | FREE | +| 9755-9759 | FREE | +| 9760-9764 | FREE | +| 9765-9769 | FREE | +| 9770-9774 | FREE | +| 9775-9779 | FREE | +| 9780-9784 | FREE | +| 9785-9789 | FREE | +| 9790-9794 | FREE | +| 9795-9799 | FREE | +| 9800-9804 | FREE | +| 9805-9809 | FREE | +| 9810-9814 | FREE | +| 9815-9819 | FREE | +| 9820-9824 | FREE | +| 9825-9829 | FREE | +| 9830-9834 | FREE | +| 9835-9839 | FREE | +| 9840-9844 | FREE | +| 9845-9849 | FREE | +| 9850-9854 | FREE | +| 9855-9859 | FREE | +| 9860-9864 | FREE | +| 9865-9869 | FREE | +| 9870-9874 | FREE | +| 9875-9879 | FREE | +| 9880-9884 | FREE | +| 9885-9889 | FREE | +| 9890-9894 | FREE | +| 9895-9899 | FREE | +| 9900-9904 | FREE | +| 9905-9909 | FREE | +| 9910-9914 | FREE | +| 9915-9919 | FREE | +| 9920-9924 | FREE | +| 9925-9929 | FREE | +| 9930-9934 | FREE | +| 9935-9939 | FREE | +| 9940-9944 | FREE | +| 9945-9949 | FREE | +| 9950-9954 | FREE | +| 9955-9959 | FREE | +| 9960-9964 | FREE | +| 9965-9969 | FREE | +| 9970-9974 | FREE | +| 9975-9979 | FREE | +| 9980-9984 | FREE | +| 9985-9989 | FREE | +| 9990-9994 | FREE | +| 9995-9999 | FREE | diff --git a/modules/developer/pages/services/general-info/registry.adoc b/modules/developer/pages/services/general-info/registry.adoc new file mode 100644 index 00000000..cc107425 --- /dev/null +++ b/modules/developer/pages/services/general-info/registry.adoc @@ -0,0 +1,39 @@ += Registry +:toc: right +:toclevels: 3 + +To be able to communicate with each other, services need to register in a common registry. + + +== Configuration + +The type of registry to use can be configured with the `MICRO_REGISTRY` environment variable. Supported values are: + +=== `nats-js-kv` (Default) + +Set the environment variable to `nats-js-kv` or leave it empty to use a nats-js key value store as registry. + +- Note: If not running build-in nats, `MICRO_REGISTRY_ADDRESS` needs to be set to the address of the nats-js cluster, which is the same value as `OCIS_EVENTS_ENDPOINT`. +- Optional: Use `MICRO_REGISTRY_AUTH_USERNAME` and `MICRO_REGISTRY_AUTH_PASSWORD` to authenticate with the nats cluster. + +=== `kubernetes` + +When deploying in a kubernetes cluster, the Kubernetes registry can be used. Additionally, the `MICRO_REGISTRY_ADDRESS` environment variable needs to be set to the url of the Kubernetes registry. + +=== `memory` + +Setting the environment variable to `memory` starts an in-memory registry. This only works with the single binary deployment. + +=== Deprecated Registries + +These registries are currently working but will be removed in a later version. It is recommended to switch to a supported one. + +- `nats`\ + Uses a registry based on nats streams. Requires `MICRO_REGISTRY_ADDRESS` to be set. +- `etcd`\ + Uses an etcd cluster as the registry. Requires `MICRO_REGISTRY_ADDRESS` to be set. +- `consul`\ + Uses `HashiCorp Consul` as registry. Requires `MICRO_REGISTRY_ADDRESS` to be set. +- `mdns`\ + Uses multicast dns for registration. This type can have unwanted side effects when other devices in the local network use mdns too. + diff --git a/modules/developer/pages/services/graph/configuration.adoc b/modules/developer/pages/services/graph/configuration.adoc new file mode 100644 index 00000000..eca9b3a7 --- /dev/null +++ b/modules/developer/pages/services/graph/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/graph-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/graph-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/graph_configvars.md \ No newline at end of file diff --git a/modules/developer/pages/services/graph/unified-roles.adoc b/modules/developer/pages/services/graph/unified-roles.adoc new file mode 100644 index 00000000..9b40c794 --- /dev/null +++ b/modules/developer/pages/services/graph/unified-roles.adoc @@ -0,0 +1,37 @@ += Unified Roles +:toc: right +:toclevels: 3 + + +== Create a New Built-in Role + +To create a new built-in role, it is necessary to: + +- Create role in Reva +- Update Reva in oCIS +- Add newly created role to oCIS + +=== Add Role to Reva + +In the link:https://github.com/owncloud/reva[Reva repository], add the role into the `/pkg/conversions/role.go` file: + +1. Add a role name constant for this role. See the existing ones for how this is setup +1. Add a new function to create a new role struct with the role name constant and desired permissions +1. Add the role to the `RoleFromName` function +1. In `/pkg/conversions/role_test.go`, extend unit tests to cover the new role + +=== Add Role to oCIS + +After adding the role to Reva and updating the Reva in oCIS, it is necessary to add the role to oCIS as well: + +1. Generate UUID for the role +1. In `/services/graph/pkg/unifiedrole/roles.go`, add the role ID generated in the first step as a constant +1. In `/services/graph/pkg/unifiedrole/roles.go`, add translatable role display name and description variables +1. In `/services/graph/pkg/unifiedrole/roles.go`, add role variable with a function to create the role\ +The function should first create the role struct using the function from Reva and return the `UnifiedRoleDefinition` struct. +1. In `/services/graph/pkg/unifiedrole/filter.go`, add the role into the `buildInRoles` function +1. In `/services/graph/pkg/config/defaults/defaultconfig.go`, if the role is not intended to be enabled by default, add the role ID into the `_disabledByDefaultUnifiedRoleRoleIDs` constant +1. In `/services/graph/pkg/unifiedrole/export_test.go`, add the role variable +1. In `/services/graph/pkg/unifiedrole/roles_test.go`, extend unit tests to cover the new role +1. In `/services/web/pkg/theme/theme.go`, add the role into the `common.shareRoles` +2. In `/services/graph/README.md`, add the role to the list of built-in roles diff --git a/modules/developer/pages/services/groups/configuration.adoc b/modules/developer/pages/services/groups/configuration.adoc new file mode 100644 index 00000000..937a28b0 --- /dev/null +++ b/modules/developer/pages/services/groups/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/groups-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/groups-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/groups_configvars.md diff --git a/modules/developer/pages/services/groups/index.adoc b/modules/developer/pages/services/groups/index.adoc new file mode 100644 index 00000000..b147093f --- /dev/null +++ b/modules/developer/pages/services/groups/index.adoc @@ -0,0 +1,9 @@ += Groups +:toc: right +:toclevels: 3 + +== Abstract + + +== Table of Contents + diff --git a/modules/developer/pages/services/idm/admin_password_reset.adoc b/modules/developer/pages/services/idm/admin_password_reset.adoc new file mode 100644 index 00000000..650cd0b6 --- /dev/null +++ b/modules/developer/pages/services/idm/admin_password_reset.adoc @@ -0,0 +1,18 @@ += Resetting a lost administrator password +:toc: right +:toclevels: 3 + +== Resetting a lost administrator password +By default, when using oCIS with the builtin IDM an ad generates the +user `admin` (DN `uid=admin,ou=users,o=libregraph-idm`) if, for any +reason, the password of that user is lost, it can be reset using +the `resetpassword` sub-command: + +---- +ocis idm resetpassword +---- + +It will prompt for a new password and set the password of that user +accordingly. Note: As this command is accessing the idm database directly +will only work while ocis is not running and nothing else is accessing +database. diff --git a/modules/developer/pages/services/idm/configuration.adoc b/modules/developer/pages/services/idm/configuration.adoc new file mode 100644 index 00000000..8ae2c786 --- /dev/null +++ b/modules/developer/pages/services/idm/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/idm-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/idm-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/idm_configvars.md \ No newline at end of file diff --git a/modules/developer/pages/services/idm/configuration_hints.adoc b/modules/developer/pages/services/idm/configuration_hints.adoc new file mode 100644 index 00000000..e33d5993 --- /dev/null +++ b/modules/developer/pages/services/idm/configuration_hints.adoc @@ -0,0 +1,43 @@ += Configuration Hints +:toc: right +:toclevels: 3 + +== TLS Server Certificates +By default IDM generates a self-signed certificate and key on first startup to be +able to provide TLS protected services. The certificate is stored in +`idm/ldap.crt` inside the oCIS base data directory. The key is in +`idm/ldap.key` in the same directory. You can use a custom server +certificate by setting the `IDM_LDAPS_CERT` and `IDM_LDAPS_KEY`. + +== Default / Demo Users +On startup IDM creates a set of default services users that are needed +internally to provide access to IDM to other oCIS services. These users are stored +in a separate subtree. The base DN of that subtree is: +`ou=sysusers,o=libregraph-idm`. The service users are: + +* `uid=libregraph,ou=sysusers,o=libregraph-idm`: This is the only user with write + access to the LDAP tree. It is used by the Graph service to look up, create, delete and + modify users and groups. +* `uid=idp,ou=sysusers,o=libregraph-idm`: This user is used by the IDP service to + perform user lookups for authentication. +* `uid=reva,ou=sysusers,o=libregraph-idm`: This user is used by the "reva" services + `user`, `group` and `auth-basic`. + +IDM is also able to create link:../../../ocis/getting-started/demo-users[Demo Users] +upon startup. + +== Access via LDAP command line tools +For testing purposes it is sometimes helpful to query IDM using the ldap +command line clients. To e.g. list all users, this command can be used: + +---- +ldapsearch -x -H ldaps://127.0.0.1:9235 -x -D uid=libregraph,ou=sysusers,o=libregraph-idm -w idm -b o=libregraph-idm objectclass=inetorgperson +---- + +When using the default configuration with the self-signed server certificate, +you might need to switch off the certificate validation using the `LDAPTL_REQCERT` env +variable: + +---- +LDAPTLS_REQCERT=never ldapsearch -x -H ldaps://127.0.0.1:9235 -x -D uid=libregraph,ou=sysusers,o=libregraph-idm -w idm -b o=libregraph-idm objectclass=inetorgperson +---- diff --git a/modules/developer/pages/services/idp/configuration.adoc b/modules/developer/pages/services/idp/configuration.adoc new file mode 100644 index 00000000..86e24287 --- /dev/null +++ b/modules/developer/pages/services/idp/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/idp-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/idp-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/idp_configvars.md \ No newline at end of file diff --git a/modules/developer/pages/services/idp/theming.adoc b/modules/developer/pages/services/idp/theming.adoc new file mode 100644 index 00000000..0ca2f1c8 --- /dev/null +++ b/modules/developer/pages/services/idp/theming.adoc @@ -0,0 +1,45 @@ += Theming +:toc: right +:toclevels: 3 + + +== Intro +Our default IDP UI is built with the link:https://github.com/libregraph/lico[LibreGraph Connect] React app. Even though this app comes already with a simple theming options, we have compiled our own edited version of the app with more advanced changes than the default theming offers. Because of that, it is not possible at the moment to do any kind of easy theming and including custom theme means again compiling custom assets. + +== Customizing assets +Depending on what changes you wish to do with the theme, there are several files you can edit. All of them are located in the `idp/ui` folder. + +=== Static assets +If you wish to add static assets like images, CSS, etc., you can add them to `idp/ui/public/static`. The `public` folder also contains the `index.html` file which can be adjusted to your needs. + +=== CSS +LibreGraph Connect is built with link:https://github.com/Kopano-dev/kpop[kpop], a collection of React UI components. To include any custom styles on top of that collection, you can define them in the `idp/ui/src/app.css` file. These rules will take precedence over the kpop. + +=== Containers +Layouts of all pages are located in the `idp/ui/src/containers` folder. By editing any of files in that folder, you can do any kind of changes in the layout and create advanced themes. It is, however, important to be careful when touching this code as it imports also actions which are responsible for the login flow. + +==== What pages to theme +- Login + - Login - login form used to authenticate the users + - Consent - consent page used to authorise apps for already signed-in users + - Chooseaccount - page with a list of accounts to choose from +- Goodbye + - Goodbyescreen - goodbye message displayed to users after they signed out +- Welcome + - Welcomescreen - welcome message displayed to users after they signed in + +=== Components +`idp/ui/src/components` folder contains all custom components which are then imported into containers. + +=== Images +Every image placed in `idp/ui/src/images` can be directly import into components or containers and will be optimized when compiling assets. + +=== Locales +If you need to edit or add new locales, you can do so with json files in the `idp/ui/src/locales` folder. If adding new locale, make sure to add it also in the `index.js` file in the same folder. + +== Building assets +In order to build all assets, run `pnpm build` in the `idp` folder. This script will compile all assets and output them into `idp/assets` folder. + +At this point, you have two possible ways how to deploy your new theme: +- run `make generate` in the root folder of your oCIS clone and generate the new assets +- start the IDP service directly with custom assets by specifying the env var `IDP_ASSET_PATH` diff --git a/modules/developer/pages/services/index.adoc b/modules/developer/pages/services/index.adoc new file mode 100644 index 00000000..085cfc00 --- /dev/null +++ b/modules/developer/pages/services/index.adoc @@ -0,0 +1,11 @@ += Services +:toc: right +:toclevels: 3 + +The documentation of services is intended for developers and only reflects the state of the master branch of the ocis repo. + +[WARNING] +==== +See the link:https://doc.owncloud.com/ocis/next/deployment/services/services.html[admin documentation] which provides versioned content suitable for administrators. This documentation also offers other useful information including deployment guides. +==== + diff --git a/modules/developer/pages/services/invitations/configuration.adoc b/modules/developer/pages/services/invitations/configuration.adoc new file mode 100644 index 00000000..8aa3c673 --- /dev/null +++ b/modules/developer/pages/services/invitations/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/invitations-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/invitations-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/invitations_configvars.md \ No newline at end of file diff --git a/modules/developer/pages/services/nats/configuration.adoc b/modules/developer/pages/services/nats/configuration.adoc new file mode 100644 index 00000000..65dbe2a4 --- /dev/null +++ b/modules/developer/pages/services/nats/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/nats-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/nats-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/nats_configvars.md \ No newline at end of file diff --git a/modules/developer/pages/services/notifications/configuration.adoc b/modules/developer/pages/services/notifications/configuration.adoc new file mode 100644 index 00000000..227be8c7 --- /dev/null +++ b/modules/developer/pages/services/notifications/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/notifications-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/notifications-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/notifications_configvars.md \ No newline at end of file diff --git a/modules/developer/pages/services/ocdav/configuration.adoc b/modules/developer/pages/services/ocdav/configuration.adoc new file mode 100644 index 00000000..2105c668 --- /dev/null +++ b/modules/developer/pages/services/ocdav/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/ocdav-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/ocdav-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/ocdav_configvars.md diff --git a/modules/developer/pages/services/ocm/configuration.adoc b/modules/developer/pages/services/ocm/configuration.adoc new file mode 100644 index 00000000..7eacce57 --- /dev/null +++ b/modules/developer/pages/services/ocm/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/ocm-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/ocm-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/ocm.md diff --git a/modules/developer/pages/services/ocm/create_share_flow.adoc b/modules/developer/pages/services/ocm/create_share_flow.adoc new file mode 100644 index 00000000..3bc7f5d9 --- /dev/null +++ b/modules/developer/pages/services/ocm/create_share_flow.adoc @@ -0,0 +1,59 @@ += Create Share Flow +:toc: right +:toclevels: 3 + +== OCM Create Share Flow + +// SHORTCODE: {{< mermaid class="text-center">}} +sequenceDiagram + box Instance A + participant osp as ocmsharesprovider + participant gwa as Gateway A + participant httpa as ocs + end + actor usera as User A + box Instance B + participant httpb as ocmd + participant gwb as Gateway B + participant ocmc as OCMCore + end + + Note over usera: A shares a resource with B + usera->>+httpa: CreateShare + httpa->>+gwa: GetInfoByDomain + Note left of gwa: GetInfoByDomain (ocmproviderauthorizer) + gwa-->>-httpa: return + + httpa->>+gwa: GetAcceptedUser + Note left of gwa: GetAcceptedUser (ocminvitemanager) + gwa-->>-httpa: return + + httpa->>+gwa: CreateOCMShare + gwa->>+osp: CreateOCMShare + osp->>+gwa: Stat + gwa-->>-osp: return + + Note left of osp: store share in repo + + osp->>+httpb: POST /shares + httpb->>+gwb: IsProviderAllowed + Note right of gwb: IsProviderAllowed (ocmproviderauthorizer) + gwb-->>-httpb: return + + httpb->>+gwb: GetUser + Note right of gwb: GetUser (userprovider) + gwb-->>-httpb: return + + httpb->>+gwb: CreateOCMCoreShare + gwb->>+ocmc: CreateOCMCoreShare + Note right of ocmc: StoreReceivedShare + ocmc-->>-gwb: return + gwb-->>-httpb: return + httpb-->>-osp: return + osp-->>-gwa: return + gwa-->>-httpa: return + httpa->>+gwa: Stat + Note left of gwa: Stat (storageprovider) + gwa-->>-httpa: return + httpa-->>-usera: return +// SHORTCODE: {{< /mermaid >}} diff --git a/modules/developer/pages/services/ocm/invitation_flow.adoc b/modules/developer/pages/services/ocm/invitation_flow.adoc new file mode 100644 index 00000000..acac02d4 --- /dev/null +++ b/modules/developer/pages/services/ocm/invitation_flow.adoc @@ -0,0 +1,49 @@ += Invitation flow +:toc: right +:toclevels: 3 + +== OCM Invitation Flow + +// SHORTCODE: {{< mermaid class="text-center">}} +sequenceDiagram + box Instance A + participant ima as InviteManager A + participant gwa as Gateway A + participant httpa as HTTP Api A (ocm, sm) + end + actor usera as User A + actor userb as User B + box Instance B + participant httpb as HTTP Api B (ocm, sm) + participant gwb as Gateway B + participant imb as InviteManager B + end + + Note over usera: A creates invitation token + usera->>+httpa: POST /generate-invite (sciencemesh) + httpa->>+gwa: GenerateInviteToken + gwa->>+ima: GenerateInviteToken + Note left of ima: store token in repo + ima-->>-gwa: return token + gwa-->>-httpa: return token + httpa-->>-usera: return token + + Note over usera,userb: A passes token to B + + Note over userb: B accepts invitation + userb->>+httpb: POST /accept-invite (sciencemesh) + httpb->>+gwb: ForwardInvite + gwb->>+imb: ForwardInvite + imb->>+httpa: POST /invite-accepted (ocm) + httpa->>+gwa: AcceptInvite + gwa->>+ima: AcceptInvite + Note left of ima: get token from repo + Note left of ima: add remote user + ima-->>-gwa: return + gwa-->>-httpa: return remote user + httpa->>-imb: return remote user + Note right of imb: add remote user + imb-->>-gwb: return + gwb-->>-httpb: return + httpb-->>-userb: return +// SHORTCODE: {{< /mermaid >}} \ No newline at end of file diff --git a/modules/developer/pages/services/ocs/configuration.adoc b/modules/developer/pages/services/ocs/configuration.adoc new file mode 100644 index 00000000..410f32bc --- /dev/null +++ b/modules/developer/pages/services/ocs/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/ocs-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/ocs-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/ocs_configvars.md \ No newline at end of file diff --git a/modules/developer/pages/services/policies/configuration.adoc b/modules/developer/pages/services/policies/configuration.adoc new file mode 100644 index 00000000..1719bb08 --- /dev/null +++ b/modules/developer/pages/services/policies/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/policies-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/policies-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/policies_configvars.md diff --git a/modules/developer/pages/services/postprocessing/configuration.adoc b/modules/developer/pages/services/postprocessing/configuration.adoc new file mode 100644 index 00000000..0962e293 --- /dev/null +++ b/modules/developer/pages/services/postprocessing/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/postprocessing-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/postprocessing-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/postprocessing_configvars.md diff --git a/modules/developer/pages/services/proxy/configuration.adoc b/modules/developer/pages/services/proxy/configuration.adoc new file mode 100644 index 00000000..3974f58f --- /dev/null +++ b/modules/developer/pages/services/proxy/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/proxy-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/proxy-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/proxy_configvars.md \ No newline at end of file diff --git a/modules/developer/pages/services/search/configuration.adoc b/modules/developer/pages/services/search/configuration.adoc new file mode 100644 index 00000000..d2875e05 --- /dev/null +++ b/modules/developer/pages/services/search/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/search-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/search-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/search_configvars.md diff --git a/modules/developer/pages/services/settings/bundles.adoc b/modules/developer/pages/services/settings/bundles.adoc new file mode 100644 index 00000000..2cae71cc --- /dev/null +++ b/modules/developer/pages/services/settings/bundles.adoc @@ -0,0 +1,71 @@ += Settings Bundles +:toc: right +:toclevels: 3 + +A _Settings Bundle_ is a collection of settings, uniquely identified by the key of the +extension registering the bundle and the key of the bundle itself. Its purpose is to let +oCIS services define settings and make them available to users. They are dynamically +rendered into forms, available in the frontend. + +As of now we support five different types of settings: +- boolean +- integer +- string +- single choice list of integers or strings +- multiple choice list of integers or strings + +Each _Setting_ is uniquely identified by a key within the bundle. Some attributes +depend on the chosen type of setting. Through the information provided with the +attributes of the setting, the settings frontend dynamically renders form elements, +allowing users to change their settings individually. + +== Example + +[source,json] +---- +{ + "identifier": { + "extension": "ocis-accounts", + "bundleKey": "profile" + }, + "displayName": "Profile", + "settings": [ + { + "settingKey": "lastname", + "displayName": "Lastname", + "description": "Input for lastname", + "stringValue": { + "placeholder": "Set lastname" + } + }, + { + "settingKey": "age", + "displayName": "Age", + "description": "Input for age", + "intValue": { + "min": "16", + "max": "200", + "step": "2", + "placeholder": "Set age" + } + }, + { + "settingKey": "timezone", + "displayName": "Timezone", + "description": "User timezone", + "singleChoiceValue": { + "options": [ + { + "stringValue": "Europe/Berlin", + "displayValue": "Europe/Berlin" + }, + { + "stringValue": "Asia/Kathmandu", + "displayValue": "Asia/Kathmandu" + } + ] + } + } + ] +} +---- diff --git a/modules/developer/pages/services/settings/configuration.adoc b/modules/developer/pages/services/settings/configuration.adoc new file mode 100644 index 00000000..39fbbca5 --- /dev/null +++ b/modules/developer/pages/services/settings/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/settings-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/settings-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/settings_configvars.md \ No newline at end of file diff --git a/modules/developer/pages/services/settings/glossary.adoc b/modules/developer/pages/services/settings/glossary.adoc new file mode 100644 index 00000000..33749670 --- /dev/null +++ b/modules/developer/pages/services/settings/glossary.adoc @@ -0,0 +1,35 @@ += Glossary +:toc: right +:toclevels: 3 + +In the context of this extension and oCIS in general, we are using the following terminology. + +=== Configuration + +- System configuration +- e.g. service host names and ports +- Changes need to be propagated to other services +- Typically modified on the CLI + +=== Settings + +- Application level settings +- e.g. default language +- Can be modified at runtime without restarting the service + +=== Preferences + +- User settings +- Subset of "Settings" +- e.g. preferred language of a user + +=== Settings Bundle + +- Collection of related settings +- Registered by an oCIS extension + +=== Settings Value + +- Manifestation of a setting for a specific user +- E.g. used for customization (at runtime) in `ocis-web` +- Can be queried and modified by other oCIS services diff --git a/modules/developer/pages/services/settings/values.adoc b/modules/developer/pages/services/settings/values.adoc new file mode 100644 index 00000000..77548559 --- /dev/null +++ b/modules/developer/pages/services/settings/values.adoc @@ -0,0 +1,71 @@ += Settings Values +:toc: right +:toclevels: 3 + +A _Settings Value_ is the value an authenticated user has chosen for a specific setting, defined in a +_settings bundle_. + +== Identifying settings values + +A _settings value_ is uniquely identified by four attributes. Three of them are coming from the definition of +the setting within it's settings bundle (see xref:bundles.adoc[Settings Bundles] +for an example). The fourth identifies the user. +- extension: Key of the extension that registered the settings bundle, +- bundleKey: Key of the settings bundle, +- settingKey: Key of the setting as defined within the bundle, +- accountUuid: The UUID of the authenticated user who has saved the setting. + +[NOTE] +==== +When requests are going through `ocis-proxy`, the accountUuid attribute can be set to the static keyword `me` +instead of using a real UUID. `ocis-proxy` will take care of minting the UUID of the authenticated user into +a JWT, providing it in the HTTP header as `x-access-token`. That UUID is then used in this service, to replace +`me` with the actual UUID of the authenticated user. +==== + + +== Example of stored settings values + +[source,json] +---- +{ + "values": { + "language": { + "identifier": { + "extension": "ocis-accounts", + "bundleKey": "profile", + "settingKey": "language", + "accountUuid": "5681371f-4a6e-43bc-8bb5-9c9237fa9c58" + }, + "listValue": { + "values": [ + { + "stringValue": "de" + } + ] + } + }, + "timezone": { + "identifier": { + "extension": "ocis-accounts", + "bundleKey": "profile", + "settingKey": "timezone", + "accountUuid": "5681371f-4a6e-43bc-8bb5-9c9237fa9c58" + }, + "listValue": { + "values": [ + { + "stringValue": "Europe/Berlin" + } + ] + } + } + } +} +---- + +== gRPC endpoints +Services can use gRPC endpoints of the `ValueService` to query and modify _settings values_. +The gRPC endpoints require the same identifier attributes as described above, so for making a request to +the `ValueService` you will have to make sure that the accountUuid of the authenticated user is available in +your service at the time of the request. diff --git a/modules/developer/pages/services/sharing/configuration.adoc b/modules/developer/pages/services/sharing/configuration.adoc new file mode 100644 index 00000000..1b204a23 --- /dev/null +++ b/modules/developer/pages/services/sharing/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/sharing-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/sharing-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/sharing_configvars.md diff --git a/modules/developer/pages/services/sharing/index.adoc b/modules/developer/pages/services/sharing/index.adoc new file mode 100644 index 00000000..bcef766f --- /dev/null +++ b/modules/developer/pages/services/sharing/index.adoc @@ -0,0 +1,10 @@ += Sharing +:toc: right +:toclevels: 3 + +== Abstract + +This service implements the CS3 link:https://cs3org.github.io/cs3apis/#cs3.sharing.link.v1beta1.LinkAPI[LinkAPI] to manage public links as well as the link:https://cs3org.github.io/cs3apis/#cs3.sharing.collaboration.v1beta1.CollaborationAPI[CollaborationAPI] to manage user and group shares. + +== Table of Contents + diff --git a/modules/developer/pages/services/sse/configuration.adoc b/modules/developer/pages/services/sse/configuration.adoc new file mode 100644 index 00000000..0c6e4bb0 --- /dev/null +++ b/modules/developer/pages/services/sse/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/app-provider-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/app-provider-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/app-provider_configvars.md diff --git a/modules/developer/pages/services/storage-publiclink/configuration.adoc b/modules/developer/pages/services/storage-publiclink/configuration.adoc new file mode 100644 index 00000000..05f7ae85 --- /dev/null +++ b/modules/developer/pages/services/storage-publiclink/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/storage-publiclink-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/storage-publiclink-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/storage-publiclink_configvars.md diff --git a/modules/developer/pages/services/storage-publiclink/index.adoc b/modules/developer/pages/services/storage-publiclink/index.adoc new file mode 100644 index 00000000..68d0cae5 --- /dev/null +++ b/modules/developer/pages/services/storage-publiclink/index.adoc @@ -0,0 +1,52 @@ += storage-publiclink +:toc: right +:toclevels: 3 + + + +== Abstract + + + +== Table of Contents + +* <> +* <> + +== Brute Force Protection + +The brute force protection will prevent access to public links if wrong +passwords are entered. The implementation is very similar to a rate limiter, +but taking into account only wrong password attempts. + +This feature is enabled by default with predefined settings. + +If you want to disable this feature, set the related configuration values to 0. + +By default, you're allowed a maximum of 5 failed attempts in 1 hour: + +* `STORAGE_PUBLICLINK_BRUTEFORCE_MAXATTEMPTS=5` +* `STORAGE_PUBLICLINK_BRUTEFORCE_TIMEGAP=1h` + +You can adjust those values to your liking in order to define the failure rate +threshold (5 failures per hour, by default). + +If the failure rate threshold is exceeded, the public link will be blocked +until such rate goes below the threshold. This means that it will remain +blocked for an undefined time: a couple of seconds in the best case, or up +to `STORAGE_PUBLICLINK_BRUTEFORCE_TIMEGAP` in the worst case. + +If the public link is blocked by the brute force protection, it will be blocked +for all the users. Any attempt with a false password will restart the trigger. + +Note that this feature uses the configurable store. No setting changes need to +be made when using defaults. +== Example Yaml Config +// AUTO-GENERATED CONTENT: services/_includes/storage-publiclink-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/storage-publiclink-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/storage-publiclink_configvars.md + diff --git a/modules/developer/pages/services/storage-shares/configuration.adoc b/modules/developer/pages/services/storage-shares/configuration.adoc new file mode 100644 index 00000000..5f3c1317 --- /dev/null +++ b/modules/developer/pages/services/storage-shares/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/storage-shares-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/storage-shares-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/storage-shares_configvars.md diff --git a/modules/developer/pages/services/storage-shares/index.adoc b/modules/developer/pages/services/storage-shares/index.adoc new file mode 100644 index 00000000..9400f04d --- /dev/null +++ b/modules/developer/pages/services/storage-shares/index.adoc @@ -0,0 +1,9 @@ += Storage-Shares +:toc: right +:toclevels: 3 + +== Abstract + + +== Table of Contents + diff --git a/modules/developer/pages/services/storage-system/configuration.adoc b/modules/developer/pages/services/storage-system/configuration.adoc new file mode 100644 index 00000000..5bc28f0f --- /dev/null +++ b/modules/developer/pages/services/storage-system/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/storage-system-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/storage-system-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/storage-system_configvars.md diff --git a/modules/developer/pages/services/storage-users/configuration.adoc b/modules/developer/pages/services/storage-users/configuration.adoc new file mode 100644 index 00000000..73816226 --- /dev/null +++ b/modules/developer/pages/services/storage-users/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/storage-users-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/storage-users-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/storage-users_configvars.md diff --git a/modules/developer/pages/services/thumbnails/configuration.adoc b/modules/developer/pages/services/thumbnails/configuration.adoc new file mode 100644 index 00000000..d0b298bc --- /dev/null +++ b/modules/developer/pages/services/thumbnails/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/thumbnails-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/thumbnails-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/thumbnails_configvars.md \ No newline at end of file diff --git a/modules/developer/pages/services/userlog/configuration.adoc b/modules/developer/pages/services/userlog/configuration.adoc new file mode 100644 index 00000000..5557b8a5 --- /dev/null +++ b/modules/developer/pages/services/userlog/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/userlog-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/userlog-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/userlog_configvars.md diff --git a/modules/developer/pages/services/users/configuration.adoc b/modules/developer/pages/services/users/configuration.adoc new file mode 100644 index 00000000..92ca65fb --- /dev/null +++ b/modules/developer/pages/services/users/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/users-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/users-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/users_configvars.md diff --git a/modules/developer/pages/services/users/index.adoc b/modules/developer/pages/services/users/index.adoc new file mode 100644 index 00000000..12e5065b --- /dev/null +++ b/modules/developer/pages/services/users/index.adoc @@ -0,0 +1,9 @@ += Users +:toc: right +:toclevels: 3 + +== Abstract + + +== Table of Contents + diff --git a/modules/developer/pages/services/web/configuration.adoc b/modules/developer/pages/services/web/configuration.adoc new file mode 100644 index 00000000..6fd15a4e --- /dev/null +++ b/modules/developer/pages/services/web/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/web-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/web-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/web_configvars.md \ No newline at end of file diff --git a/modules/developer/pages/services/web/releasing.adoc b/modules/developer/pages/services/web/releasing.adoc new file mode 100644 index 00000000..d40c5944 --- /dev/null +++ b/modules/developer/pages/services/web/releasing.adoc @@ -0,0 +1,32 @@ += Releasing +:toc: right +:toclevels: 3 + + +== Releasing + +The next generation Web Frontend is shipped as an oCIS Extension. The `ocis-web` extension is also embedded in the single binary and part of the `ocis server` command. + +To update this package within all the deliveries, we need to update the package in the following chain from the bottom to the top. + +=== Package Hierarchy + +- link:https://github.com/owncloud/ocis[ocis] + - link:https://github.com/owncloud/ocis/tree/master/web[ocis-web] + - link:https://github.com/owncloud/ocis/tree/master/ocis-pkg[ocis-pkg] + - link:https://github.com/owncloud/web[ownCloud Web] + +==== Prerequisites + +Before updating the assets, make sure that link:https://github.com/owncloud/web[ownCloud Web] has been released first +and take note of its release tag name. + +==== Updating ocis-web + +1. Create a branch `update-web-$version` in the link:https://github.com/owncloud/ocis[ocis repository] +2. Change into web package folder via `cd web` +3. Inside `web/`, update the `Makefile` so that the WEB_ASSETS_VERSION variable references the currently released version of https://github.com/owncloud/web +4. Move to the changelog (`cd ../changelog/`) and add a changelog file to the `unreleased/` folder (You can copy an old web release changelog item as a template) +5. Move to the repo root (`cd ..`)and update the WEB_COMMITID in the `/.drone.env` file to the commit id from the released version (unless the existing commit id is already newer) +6. _Optional:_ Test the changes locally by running `cd ocis && go run cmd/ocis/main.go server`, visiting link:https://localhost:9200[https://localhost:9200] and confirming everything renders correctly +7. Commit your changes, push them and link:https://github.com/owncloud/ocis/pulls[create a PR] diff --git a/modules/developer/pages/services/webdav/configuration.adoc b/modules/developer/pages/services/webdav/configuration.adoc new file mode 100644 index 00000000..4718840b --- /dev/null +++ b/modules/developer/pages/services/webdav/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/webdav-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/webdav-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/webdav_configvars.md \ No newline at end of file diff --git a/modules/developer/pages/services/webfinger/configuration.adoc b/modules/developer/pages/services/webfinger/configuration.adoc new file mode 100644 index 00000000..77cabb8e --- /dev/null +++ b/modules/developer/pages/services/webfinger/configuration.adoc @@ -0,0 +1,13 @@ += Service Configuration +:toc: right +:toclevels: 3 + +== Example YAML Config + +// AUTO-GENERATED CONTENT: services/_includes/webfinger-config-example.yaml +[source,yaml] +---- +// Include from: services/_includes/webfinger-config-example.yaml +---- + +// AUTO-GENERATED CONTENT: services/_includes/webfinger_configvars.md diff --git a/modules/developer/partials/nav.adoc b/modules/developer/partials/nav.adoc new file mode 100644 index 00000000..fb63924e --- /dev/null +++ b/modules/developer/partials/nav.adoc @@ -0,0 +1,191 @@ +* xref:index.adoc[Developer Documentation] +* xref:ocis/index.adoc[oCIS] +** xref:ocis/development/index.adoc[Development] +*** xref:ocis/development/getting-started.adoc[Getting Started] +*** xref:ocis/development/build.adoc[Building] +*** xref:ocis/development/testing.adoc[Testing] +*** xref:ocis/development/debugging.adoc[Debugging] +*** xref:ocis/development/profiling.adoc[Profiling] +*** xref:ocis/development/tracing.adoc[Tracing] +*** xref:ocis/development/extensions.adoc[Extensions] +*** xref:ocis/development/envvars.adoc[Environment Variables] +*** xref:ocis/development/continuous-integration.adoc[Continuous Integration] +*** xref:ocis/development/beta-testplan.adoc[Beta Test Plan] +*** Unit Testing +**** xref:ocis/development/unit-testing/index.adoc[Overview] +**** xref:ocis/development/unit-testing/testing-ginkgo.adoc[Ginkgo] +**** xref:ocis/development/unit-testing/testing-pkg.adoc[Testing Package] +** xref:ocis/adr/index.adoc[Architecture Decision Records] +*** xref:ocis/adr/0001-introduce-accounts-service.adoc[0001 Accounts Service] +*** xref:ocis/adr/0002-persist-accounts-using-cs3-storage.adoc[0002 CS3 Account Persistence] +*** xref:ocis/adr/0003-external-user-management.adoc[0003 External User Management] +*** xref:ocis/adr/0004-support-hot-migration.adoc[0004 Hot Migration] +*** xref:ocis/adr/0005-cs3-api-account-management.adoc[0005 CS3 API Account Management] +*** xref:ocis/adr/0006-service-discovery.adoc[0006 Service Discovery] +*** xref:ocis/adr/0007-api-for-spaces.adoc[0007 API for Spaces] +*** xref:ocis/adr/0008-configuration.adoc[0008 Configuration] +*** xref:ocis/adr/0009-extension-template.adoc[0009 Extension Template] +*** xref:ocis/adr/0010-policy-enforcement.adoc[0010 Policy Enforcement] +*** xref:ocis/adr/0011-global-url-format.adoc[0011 Global URL Format] +*** xref:ocis/adr/0012-tracing.adoc[0012 Tracing] +*** xref:ocis/adr/0013-locking.adoc[0013 Locking] +*** xref:ocis/adr/0014-microservices-runtime.adoc[0014 Microservices Runtime] +*** xref:ocis/adr/0015-events.adoc[0015 Events] +*** xref:ocis/adr/0016-files-metadata.adoc[0016 Files Metadata] +*** xref:ocis/adr/0017-allow-read-only-external-user-management.adoc[0017 Read-Only External User Management] +*** xref:ocis/adr/0018-file-search-api.adoc[0018 File Search API] +*** xref:ocis/adr/0019-file-search-index.adoc[0019 File Search Index] +*** xref:ocis/adr/0020-file-search-query-language.adoc[0020 File Search Query Language] +*** xref:ocis/adr/0021-service-accounts.adoc[0021 Service Accounts] +*** xref:ocis/adr/0022-sharing-and-space-management-api.adoc[0022 Sharing and Space Management API] +*** xref:ocis/adr/0023-index-and-store-metadata.adoc[0023 Index and Store Metadata] +*** xref:ocis/adr/0024-msgpack-metadata.adoc[0024 Msgpack Metadata] +*** xref:ocis/adr/0025-distributed-search-index.adoc[0025 Distributed Search Index] +*** xref:ocis/adr/0026-application-based-user-settings.adoc[0026 Application-Based User Settings] +*** xref:ocis/adr/0027-new-share-jail.adoc[0027 New Share Jail] +*** xref:ocis/adr/0028-activities-service.adoc[0028 Activities Service] +** xref:ocis/flow-docs/index.adoc[Flow Documentation] +*** xref:ocis/flow-docs/login-flow.adoc[Login Flow] +*** xref:ocis/flow-docs/public-upload-flow.adoc[Public Upload Flow] +*** xref:ocis/flow-docs/request-flow.adoc[Request Flow] +** xref:ocis/storage/index.adoc[Storage] +*** xref:ocis/storage/terminology.adoc[Terminology] +*** xref:ocis/storage/spaces.adoc[Spaces] +*** xref:ocis/storage/users.adoc[Users] +*** xref:ocis/storage/namespaces.adoc[Namespaces] +*** xref:ocis/storage/storagedrivers.adoc[Storage Drivers] +*** xref:ocis/storage/spacesprovider.adoc[Spaces Provider] +*** xref:ocis/storage/spacesregistry.adoc[Spaces Registry] +*** xref:ocis/storage/proposedchanges.adoc[Proposed Changes] +*** xref:ocis/storage-backends/index.adoc[Storage Backends] +**** xref:ocis/storage-backends/cephfs.adoc[CephFS] +**** xref:ocis/storage-backends/dcfsnfs.adoc[DCFS NFS] +**** xref:ocis/storage-backends/eos.adoc[EOS] +** xref:ocis/guides/index.adoc[Guides] +*** xref:ocis/guides/ocis-mini-eval.adoc[Mini Evaluation] +*** xref:ocis/guides/ocis-local-docker.adoc[Local Docker Setup] +*** xref:ocis/guides/ocis-and-containers.adoc[Containers] +*** xref:ocis/guides/migrate-data-rclone.adoc[Migrate Data with rclone] +*** xref:ocis/guides/ai_assisted_dev.adoc[AI-Assisted Development] +** xref:ocis/identity-provider/index.adoc[Identity Provider] +*** xref:ocis/identity-provider/oidc.adoc[OIDC] +*** xref:ocis/identity-provider/ldap-active-directory.adoc[LDAP / Active Directory] +** xref:ocis/deployment/index.adoc[Deployment (Developer)] +*** xref:ocis/deployment/ocis_hello.adoc[Hello World] +*** xref:ocis/deployment/ocis_full.adoc[Full Setup] +*** xref:ocis/deployment/ocis_ldap.adoc[With LDAP] +*** xref:ocis/deployment/ocis_keycloak.adoc[With Keycloak] +*** xref:ocis/deployment/oc10_ocis_parallel.adoc[OC10 Parallel] +*** xref:ocis/deployment/kubernetes.adoc[Kubernetes] +*** xref:ocis/deployment/systemd.adoc[Systemd] +*** xref:ocis/deployment/continuous_deployment.adoc[Continuous Deployment] +*** xref:ocis/deployment/basic-remote-setup.adoc[Basic Remote Setup] +*** xref:ocis/deployment/bridge.adoc[Bridge] +*** xref:ocis/deployment/preparing_server.adoc[Preparing the Server] +*** xref:ocis/deployment/monitoring-tracing.adoc[Monitoring and Tracing] +*** xref:ocis/deployment/ubernauten.adoc[Ubernauten Example] +** xref:ocis/releasing_guide/index.adoc[Releasing Guide] +*** xref:ocis/releasing_guide/releasing_guide.adoc[Process] +*** xref:ocis/releasing_guide/troubleshooting.adoc[Troubleshooting] +** xref:ocis/config.adoc[Configuration] +** xref:ocis/metrics.adoc[Metrics] +** xref:ocis/migration.adoc[Migration] +** xref:ocis/backup.adoc[Backup] +** xref:ocis/build-docs.adoc[Building the Docs] +** xref:ocis/release_notes.adoc[Release Notes] +** xref:ocis/release_roadmap.adoc[Release Roadmap] +** xref:ocis/license.adoc[License] +** xref:ocis/storage_registry_discovery.adoc[Storage Registry Discovery] +** xref:ocis/getting-started/index.adoc[Getting Started] +*** xref:ocis/getting-started/demo-users.adoc[Demo Users] +* xref:architecture/index.adoc[Architecture] +** xref:architecture/upload-processing.adoc[Upload Processing] +** xref:architecture/efficient-stat-polling.adoc[Efficient Stat Polling] +** xref:architecture/protocol-changes.adoc[Protocol Changes] +** xref:architecture/posixfs-storage-driver.adoc[PosixFS Storage Driver] +** xref:architecture/collaborative-storage.adoc[Collaborative Storage] +** xref:architecture/services-communication.adoc[Services Communication] +* xref:apis/index.adoc[APIs] +** xref:apis/grpc_apis/index.adoc[gRPC APIs] +** xref:apis/http/index.adoc[HTTP APIs] +*** xref:apis/http/authorization.adoc[Authorization] +*** xref:apis/http/tus_upload.adoc[TUS Upload] +*** xref:apis/http/graph/index.adoc[Graph API] +**** xref:apis/http/graph/users.adoc[Users] +**** xref:apis/http/graph/groups.adoc[Groups] +**** xref:apis/http/graph/permissions.adoc[Permissions] +**** xref:apis/http/graph/role.adoc[Roles] +**** xref:apis/http/graph/spaces.adoc[Spaces] +*** xref:apis/http/webdav/index.adoc[WebDAV API] +* xref:cli-commands/index.adoc[CLI Commands] +** xref:cli-commands/create-new-cli-command.adoc[Create a New CLI Command] +** xref:cli-commands/document-cli-commands.adoc[Document CLI Commands] +** xref:cli-commands/service_dependent_cli.adoc[Service-Dependent CLI] +** xref:cli-commands/service_independent_cli.adoc[Service-Independent CLI] +* xref:clients/index.adoc[Clients] +** xref:clients/rclone/index.adoc[rclone] +*** xref:clients/rclone/webdav-sync-basic-auth.adoc[WebDAV Sync — Basic Auth] +*** xref:clients/rclone/webdav-sync-oidc.adoc[WebDAV Sync — OIDC] +* xref:services/index.adoc[Services] +** xref:services/general-info/index.adoc[General Information] +*** xref:services/general-info/new-service-checklist.adoc[New Service Checklist] +*** xref:services/general-info/port-ranges.adoc[Port Ranges] +*** xref:services/general-info/registry.adoc[Registry] +*** xref:services/general-info/additional-information.adoc[Additional Information] +*** xref:services/general-info/add-translations.adoc[Adding Translations] +*** Environment Variables +**** xref:services/general-info/envvars/index.adoc[Overview] +**** xref:services/general-info/envvars/envvar-naming-scopes.adoc[Naming and Scopes] +**** xref:services/general-info/envvars/special-envvars.adoc[Special Env Vars] +**** xref:services/general-info/envvars/deprecating-variables.adoc[Deprecating Variables] +**** xref:services/general-info/envvars/new-release-process.adoc[Release Process] +**** xref:services/general-info/envvars/env-var-deltas/index.adoc[Version Deltas] +** xref:services/antivirus/configuration.adoc[Antivirus] +** xref:services/app-provider/index.adoc[App Provider] +** xref:services/app-registry/configuration.adoc[App Registry] +** xref:services/audit/configuration.adoc[Audit] +** xref:services/auth-basic/configuration.adoc[Auth Basic] +** xref:services/auth-bearer/configuration.adoc[Auth Bearer] +** xref:services/auth-machine/configuration.adoc[Auth Machine] +** xref:services/eventhistory/configuration.adoc[Event History] +** xref:services/frontend/configuration.adoc[Frontend] +** xref:services/gateway/configuration.adoc[Gateway] +** xref:services/graph/configuration.adoc[Graph] +*** xref:services/graph/unified-roles.adoc[Unified Roles] +** xref:services/groups/index.adoc[Groups] +** xref:services/idm/configuration.adoc[IDM] +*** xref:services/idm/admin_password_reset.adoc[Admin Password Reset] +*** xref:services/idm/configuration_hints.adoc[Configuration Hints] +** xref:services/idp/configuration.adoc[IDP] +*** xref:services/idp/theming.adoc[Theming] +** xref:services/invitations/configuration.adoc[Invitations] +** xref:services/nats/configuration.adoc[NATS] +** xref:services/notifications/configuration.adoc[Notifications] +** xref:services/ocdav/configuration.adoc[OCDAV] +** xref:services/ocm/configuration.adoc[OCM] +*** xref:services/ocm/invitation_flow.adoc[Invitation Flow] +*** xref:services/ocm/create_share_flow.adoc[Create Share Flow] +** xref:services/ocs/configuration.adoc[OCS] +** xref:services/policies/configuration.adoc[Policies] +** xref:services/postprocessing/configuration.adoc[Postprocessing] +** xref:services/proxy/configuration.adoc[Proxy] +** xref:services/search/configuration.adoc[Search] +** xref:services/settings/configuration.adoc[Settings] +*** xref:services/settings/bundles.adoc[Bundles] +*** xref:services/settings/values.adoc[Values] +*** xref:services/settings/glossary.adoc[Glossary] +** xref:services/sharing/index.adoc[Sharing] +** xref:services/sse/configuration.adoc[SSE] +** xref:services/storage-publiclink/index.adoc[Storage Public Link] +** xref:services/storage-shares/index.adoc[Storage Shares] +** xref:services/storage-system/configuration.adoc[Storage System] +** xref:services/storage-users/configuration.adoc[Storage Users] +** xref:services/thumbnails/configuration.adoc[Thumbnails] +** xref:services/userlog/configuration.adoc[Userlog] +** xref:services/users/index.adoc[Users] +** xref:services/web/configuration.adoc[Web] +*** xref:services/web/releasing.adoc[Releasing] +** xref:services/webdav/configuration.adoc[WebDAV] +** xref:services/webfinger/configuration.adoc[Webfinger] +* xref:service-independent/index.adoc[Service-Independent Config] +** xref:service-independent/service_independent_envvars.adoc[Environment Variables]