diff --git a/content/product/control_plane_configuration/large-scale_deployment/grpc.md b/content/product/control_plane_configuration/large-scale_deployment/grpc.md new file mode 100644 index 000000000..f9ad54955 --- /dev/null +++ b/content/product/control_plane_configuration/large-scale_deployment/grpc.md @@ -0,0 +1,118 @@ +--- +title: "gRPC Integration" +linkTitle: "gRPC Integration" +date: "2025-02-28" +description: +categories: +pageintoc: "44" +tags: +weight: "5" +--- + + + + + +## Overview + +gRPC offers a high-performance alternative to the legacy XML-RPC protocol. By using binary serialization in place of text-based XML, gRPC significantly reduces CPU overhead and network payload sizes, particularly in large-scale deployments. It simplifies development by providing typed contracts and multi-language support through Protocol Buffers for a scalable and easy to maintain code-base. + +## Configuration + +To enable and use gRPC within an OpenNebula environment, you must configure both the daemon (server-side) and the client. + +### Enabling the gRPC Service + +The oned daemon manages the gRPC server, which is enabled by default. To modify the default values, edit `/etc/one/oned.conf` and define the listening port and address: + +> ```none +> GRPC_PORT = 2634 +> GRPC_LISTEN_ADDRESS = "0.0.0.0" +> ``` + +### Using the gRPC Clients + +OpenNebula 7.2 provides gRPC support for Ruby (CLI), Go, and Python (partial). When using the Command Line Interface (CLI), you can toggle the protocol using one of the following methods: + +* Flag-based: Append the `--grpc` flag to any supported command. +* Environment-based: Set `ONEAPI_PROTOCOL=grpc` in your shell profile to make gRPC the default for all commands. + +By default, the client attempts to connect to the local endpoint. You can override this by specifying the target server address: + +> ```Bash +> export ONE_GRPC=":" +> ``` + +### Integration with OneFlow + +To configure the OneFlow service to communicate with oned via gRPC, update the `:one_xmlrpc` setting in `/etc/one/oneflow-server.conf` to point to the gRPC endpoint: + +> ```yaml +> :one_xmlrpc: 127.0.0.1:2634 +> ``` + +### High Availability (HA) and Federation + +In distributed environments, explicit endpoint definitions are required to ensure the client can reach the active leader or the correct regional zone. + +* High availability: Define the `ENDPOINT_GRPC` attribute for every node within the cluster. +* Federation: Define the `ENDPOINT_GRPC` attribute for each zone in the federation. + +{{< alert title="Warning" color="warning" >}} +Important: If a client is configured to use gRPC but the `ENDPOINT_GRPC` is missing in an HA or Federated setup, commands may fail to route correctly, resulting in connection errors. +{{< /alert >}} + +## Performance + +This section provides a comparative analysis of response times between the legacy XML-RPC and the new gRPC protocol. + +### Benchmark Methodology + +To ensure realistic results, the protocols were tested against a synthetic workload designed to simulate a medium-to-large OpenNebula deployment. The core oned service was stressed in a single-zone configuration with the following environment specifications: + +| Metric | Value | +|--------------------------|---------| +| Number of hosts | 1,250 | +| Number of VMs | 20,000 | +| Average VM template size | 10 KB | + +The workload consisted of the four most common API calls, executed simultaneously to mirror the request ratio observed in production environments: `host.info`, `hostpool.info`, `vm.info`, and `vmpool.info`. + +### Comparative Results + +The following table illustrates the average response times (in seconds) under two different load intensities: 10 requests per second (req/s) and 30 req/s. + +| API Method | XML-RPC (10 req/s) | gRPC (10 req/s) | XML-RPC (30 req/s) | gRPC (30 req/s) | +| :--- | :---: | :---: | :---: | :---: | +| `host.info` | 0.01 | 0.01 | 0.07 | 0.02 | +| `hostpool.info` | 0.07 | 0.02 | 0.17 | 0.03 | +| `vm.info` | 0.02 | 0.01 | 0.07 | 0.02 | +| `vmpool.info` | 0.97 | 0.43 | 2.15 | 0.94 | + +For data-intensive calls like `vmpool.info`, **gRPC reduces latency by over 50%**, demonstrating the efficiency of binary serialization over text-based XML. The performance advantage of gRPC becomes more significant as the API load increases. At 30 req/s, gRPC consistently delivers responses **2 to 3 times faster than XML-RPC**. + +## Development + +A primary advantage of gRPC is the ability to generate native client libraries for virtually any programming language using the core service definitions. + +### Protocol Buffers (.proto) + +The contract between the OpenNebula server and any client is defined in `.proto` files. These files describe the services, RPC methods, and data structures. You can find these definitions in the OpenNebula source tree: `src/rm/grpc/proto/` + +### Generating Language Bindings + +If you require support for a language not currently provided by the core team such as Rust, Java, or C#, you can generate your own bindings using the `protoc` compiler and the appropriate plugin for your language. + +### Example: Generating C++ Bindings + +To generate the header and source files manually, use a command similar to the following: + +> ```Bash +> protoc -I=src/rm/grpc/proto --cpp_out=./output --grpc_out=./output \ +> --plugin=protoc-gen-grpc=`which grpc_cpp_plugin` \ +> src/rm/grpc/proto/user.proto +> ``` + +### Contributing + +We encourage you to explore the existing Go and Ruby implementations within the source code as templates for building new language providers. \ No newline at end of file diff --git a/content/software/installation_process/build_from_source_code/build_deps.md b/content/software/installation_process/build_from_source_code/build_deps.md index 406938804..949be5108 100644 --- a/content/software/installation_process/build_from_source_code/build_deps.md +++ b/content/software/installation_process/build_from_source_code/build_deps.md @@ -27,48 +27,32 @@ This page lists the **build** dependencies for OpenNebula. ## Ubuntu 22.04, 24.04 * **bash-completion** -* **bower** -* **debhelper (>= 7.0.50~)** -* **default-jdk** -* **freerdp2-dev** -* **grunt** -* **javahelper (>= 0.32)** -* **libaugeas-dev** -* **libcairo2-dev** * **libcurl4-openssl-dev** -* **libmysql++-dev** * **libmysqlclient-dev** * **libnode-dev (>= 10)** -* **libossp-uuid-dev** -* **libpango1.0-dev** -* **libpulse-dev** +* **libnsl-dev** * **libsqlite3-dev** -* **libssh2-1-dev** * **libssl-dev** * **libsystemd-dev** -* **libtool** * **libvncserver-dev** -* **libvorbis-dev** -* **libwebp-dev** -* **libws-commons-util-java** * **libxml2-dev** * **libxmlrpc-c++8-dev** -* **libxslt1-dev** -* **libzmq3-dev** -* **libzmq5** * **nodejs (>= 10)** * **npm** * **python3** * **python3-pip** * **python3-setuptools** * **rake** -* **ruby-dev** * **scons** * **unzip** +* **protobuf-compiler-grpc** +* **libgrpc++-dev** +* **libabsl-dev** +* ruby gem **grpc-tools** For Ubuntu 22.04 is recommended a updated version of Node.js: -```default +```shell apt -y remove nodejs libnode-dev || true apt -y autoremove curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash - @@ -76,103 +60,101 @@ curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash - Install all requirements using: -```default -apt install bash-completion debhelper default-jdk freerdp2-dev grunt javahelper libaugeas-dev libcairo2-dev libcurl4-openssl-dev libmysql++-dev libmysqlclient-dev libnode-dev libossp-uuid-dev libpango1.0-dev libpulse-dev libsqlite3-dev libssh2-1-dev libssl-dev libsystemd-dev libtool libvncserver-dev libvorbis-dev libwebp-dev libws-commons-util-java libxml2-dev libxmlrpc-c++8-dev libxslt1-dev libzmq3-dev libzmq5 nodejs npm python3 python3-pip python3-setuptools rake ruby-dev scons unzip libnsl-dev && npm install -g bower +```shell +apt install bash-completion libcurl4-openssl-dev libmysqlclient-dev \ + libnode-dev libnsl-dev libsqlite3-dev libssl-dev libsystemd-dev \ + libvncserver-dev libxml2-dev libxmlrpc-c++8-dev nodejs npm \ + python3 python3-pip python3-setuptools rake scons unzip \ + protobuf-compiler-grpc libgrpc++-dev libabsl-dev +gem install grpc-tools ``` -## Debian 11, Debian 12 +## Debian 12, 13 * **bash-completion** * **bower** -* **debhelper (>= 7.0.50~)** -* **default-jdk** * **default-libmysqlclient-dev** -* **freerdp2-dev** -* **grunt** -* **javahelper (>= 0.32)** -* **libaugeas-dev** -* **libcairo2-dev** * **libcurl4-openssl-dev** * **libnode-dev (>= 10)** -* **libossp-uuid-dev** -* **libpango1.0-dev** -* **libpulse-dev** +* **libnsl-dev** * **libsqlite3-dev** -* **libssh2-1-dev** * **libssl-dev** * **libsystemd-dev** -* **libtool** * **libvncserver-dev** -* **libvorbis-dev** -* **libwebp-dev** -* **libws-commons-util-java** * **libxml2-dev** -* **libxmlrpc-c++8-dev** -* **libxslt1-dev** -* **libzmq3-dev** -* **libzmq5** +* **libxmlrpc-c++8-dev** -> Debian 12, +* **libxmlrpc-c++9-dev** -> Debian 13 * **nodejs (>= 10)** * **npm** * **python3** +* **python3-pip** * **python3-setuptools** * **rake** -* **ruby-dev** * **scons** * **unzip** +* **protobuf-compiler-grpc** +* **libgrpc++-dev** +* **libabsl-dev** +* ruby gem **grpc-tools** Install all requirements using: -```default -apt install bash-completion debhelper default-jdk default-libmysqlclient-dev freerdp2-dev grunt javahelper libaugeas-dev libcairo2-dev libcurl4-openssl-dev libnode-dev libossp-uuid-dev libpango1.0-dev libpulse-dev libsqlite3-dev libssh2-1-dev libssl-dev libsystemd-dev libtool libvncserver-dev libvorbis-dev libwebp-dev libws-commons-util-java libxml2-dev libxmlrpc-c++8-dev libxslt1-dev libzmq3-dev libzmq5 nodejs npm python3 python3-setuptools rake ruby-dev scons unzip && npm install -g bower +```shell +# Debian 12 +apt install bash-completion default-libmysqlclient-dev libcurl4-openssl-dev \ + libnode-dev libnsl-dev libsqlite3-dev libssl-dev libsystemd-dev \ + libvncserver-dev libxml2-dev libxmlrpc-c++8-dev nodejs npm \ + python3 python3-pip python3-setuptools rake scons unzip \ + protobuf-compiler-grpc libgrpc++-dev libabsl-dev +gem install grpc-tools +``` + +```shell +# Debian 13 +apt install bash-completion default-libmysqlclient-dev libcurl4-openssl-dev \ + libnode-dev libnsl-dev libsqlite3-dev libssl-dev libsystemd-dev \ + libvncserver-dev libxml2-dev libxmlrpc-c++9-dev nodejs npm \ + python3 python3-pip python3-setuptools rake scons unzip \ + protobuf-compiler-grpc libgrpc++-dev libabsl-dev +gem install grpc-tools ``` -## AlmaLinux/RHEL 8,9 +## AlmaLinux/RHEL 9, 10 * **gcc-c++** -* **augeas-devel** -* **cairo-devel** -* **curl-devel** -* **epel-rpm-macros** -* **expat-devel** -* **freerdp-devel** * **gnutls-devel** -* **java-1.7.0-openjdk-devel** # java-1.8.0-openjdk-devel needs to be installed for AlmaLinux 9 * **libcurl-devel** -* **libffi-devel** * **libjpeg-turbo-devel** * **libnsl2-devel** -* **libpq-devel** -* **libssh2-devel** -* **libtool** * **libvncserver-devel** -* **libvorbis-devel** -* **libwebp-devel** * **libxml2-devel** -* **libxslt-devel** -* **mysql-devel** +* **mariadb-devel** * **nodejs >= 10** -* **nodejs-devel >= 10** * **npm** * **openssh** * **openssl-devel** -* **pango-devel** * **pkgconfig** -* **pulseaudio-libs-devel** * **python3** -* **python3-devel** -* **python3-rpm-macros** * **python3-scons** * **python3-setuptools** -* **python3-wheel** -* **ruby-devel** -* **rubygem-rake** * **rubygems** * **sqlite-devel** * **systemd** * **systemd-devel** * **xmlrpc-c-devel** -* **uuid-devel** -* **zeromq-devel** +* **grpc-devel** +* ruby gem **grpc-tools** + +Install all requirements using: + +```shell +dnf config-manager --set-enabled crb +dnf install gcc-c++ gnutls-devel libcurl-devel libjpeg-turbo-devel \ + libnsl2-devel libvncserver-devel libxml2-devel mariadb-devel nodejs npm \ + openssh openssl-devel pkgconfig python3 python3-scons python3-setuptools \ + rubygems sqlite-devel systemd systemd-devel xmlrpc-c-devel grpc-devel +gem install grpc-tools +``` ## Arch diff --git a/content/software/installation_process/build_from_source_code/compile.md b/content/software/installation_process/build_from_source_code/compile.md index d78cec40c..e56aa0453 100644 --- a/content/software/installation_process/build_from_source_code/compile.md +++ b/content/software/installation_process/build_from_source_code/compile.md @@ -46,14 +46,16 @@ The argument expression [OPTION=VALUE] is used to set non-default values for : | sqlite | yes | **no** if you don’t want to build Sqlite support | | mysql | no | **yes** if you want to build MySQL support | | parsers | no | **yes** if you want to rebuild Flex/Bison files. | -| sunstone | no | **yes** if you want to build Ruby Sunstone minified files | | fireedge | no | **yes** if you want to build FireEdge minified files | | systemd | no | **yes** if you want to build systemd support | | rubygems | no | **yes** if you want to generate Ruby gems | | svncterm | yes | **no** to skip building VNC support for LXD drivers | | context | no | **yes** to download guest contextualization packages | +| strict | no | Strict C++ compiler, more warnings, treat warnings as errors | | download | no | **yes** to download 3rd-party tools (Restic, Prometheus…) | -| xmlrpc_pkgconf | no | **el8/el9/el10** if you want to use pkg-config for xmlrpc-c libs dependency (otherwise xmlrpc-c-config is used) | +| grpc | yes | **yes** to build gRPC support | +| grpcproto | no | **yes** to generate C++ sources from .proto files | +| xmlrpc_pkgconf | no | **yes** to use pkg-config to discover xmlrpc libs dependencies, otherwise xmlrpc-c-config is used. Needed for Alma9 and RHEL9 | - OpenNebula can be installed in two modes: `system-wide` or in `self-contained` directory. In either case, you do not need to run OpenNebula as root. These options can be specified when running the install script: @@ -77,15 +79,13 @@ where ** can be one or more of: | **-k** | keep configuration files of existing OpenNebula installation, useful when upgrading. This flag should not be set when installing OpenNebula for the first time | | **-d** | target installation directory. If defined, it will specified the path for the **self-contained** install. If not defined, the installation will be performed **system wide** | | **-c** | only install client utilities: OpenNebula cli and ec2 client files | -| **-s** | install OpenNebula Ruby Sunstone | -| **-p** | do not install OpenNebula Ruby Sunstone non-minified files | -| **-F** | install OpenNebula FireEdge | +| **-F** | install only OpenNebula FireEdge | | **-P** | do not install OpenNebula FireEdge non-minified files | -| **-G** | install OpenNebula Gate | -| **-6** | install only OpenNebula Gate Proxy | -| **-f** | install OpenNebula Flow | +| **-G** | install only OpenNebula Gate | +| **-f** | install only OpenNebula Flow | | **-r** | remove Opennebula, only useful if -d was not specified, otherwise `rm -rf $ONE_LOCATION` would do the job | | **-l** | creates symlinks instead of copying files, useful for development | +| **-a** | architecture of downloaded vendor artifacts, default: x86_64" | | **-h** | prints installer help | {{< alert title="Note" color="success" >}}