-
Notifications
You must be signed in to change notification settings - Fork 925
pipelines: document KFP MLflow plugin #4423
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| +++ | ||
| title = "MLflow Plugin Configuration" | ||
| weight = 4 | ||
| +++ | ||
|
|
||
| ## KFP MLflow Plugin | ||
|
|
||
| Kubeflow Pipelines supports an MLflow plugin that enables automatic experiment tracking. When enabled, the plugin registers each KFP run with MLflow, allowing users to view and analyze their pipeline runs in the MLflow UI. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Admin access to a Kubernetes cluster | ||
| - Kubeflow Pipelines installed on your cluster | ||
| - kubectl configured to access your cluster | ||
|
|
||
| ## Deploying MLflow | ||
|
|
||
| You need an MLflow instance running on your Kubernetes cluster. One recommended option is the [OpenDataHub MLflow Operator](https://github.com/opendatahub-io/mlflow-operator#mlflow-operator), which simplifies MLflow deployment and management on Kubernetes. | ||
|
|
||
| To deploy using the MLflow Operator: | ||
|
|
||
| 1. Follow the operator [deployment instructions](https://github.com/opendatahub-io/mlflow-operator#to-deploy-on-the-cluster) | ||
| 2. Note the MLflow service endpoint and port (typically `https://<service-name>:8443`) | ||
| 3. If using TLS, retrieve the CA certificate for your MLflow deployment | ||
|
|
||
| ## Security and TLS | ||
|
|
||
| The KFP MLflow plugin uses TLS to secure communication with the MLflow server. Configure the following TLS settings in your [API server configuration](#configuring-the-kfp-mlflow-plugin): | ||
|
|
||
| - **caBundlePath**: Path to the CA certificate bundle used to verify the MLflow server's certificate | ||
| - **insecureSkipVerify**: Set to `false` (default) to enforce certificate verification | ||
|
|
||
| For production deployments, always use valid TLS certificates and keep `insecureSkipVerify` set to `false`. | ||
|
|
||
| ## MLflow Experiments | ||
|
|
||
| When the plugin is enabled, pipeline runs are logged to MLflow experiments: | ||
|
|
||
| - **Default experiment**: If no experiment is specified, runs are logged to an experiment named "default" (created automatically if it doesn't exist) | ||
| - **Custom experiments**: Users can specify a custom experiment name when submitting a run. KFP will create the experiment if it doesn't already exist | ||
| - **RBAC**: For enhanced security, you can enforce Kubernetes RBAC for MLflow requests using the [Kubeflow MLflow extension](https://github.com/kubeflow/mlflow-integration#mlflow-kubeflow-integration) | ||
|
|
||
| ## MLflow Workspaces | ||
|
|
||
| MLflow workspaces provide an optional organizational layer and permission framework, similar to Kubernetes namespaces. A workspace can contain multiple experiments, but each experiment belongs to only one workspace. | ||
|
|
||
| **Configuration:** | ||
| - When `authType` is set to `kubernetes`, workspaces are **enabled by default** | ||
| - For other authentication types, workspaces are **disabled by default** | ||
| - You can explicitly control this behavior by setting `workspacesEnabled` to `true` or `false` in the plugin configuration | ||
|
|
||
| **Requirements:** | ||
| To use MLflow workspaces, you must deploy the [Kubeflow MLflow extension](https://github.com/kubeflow/mlflow-integration#mlflow-kubeflow-integration), which adds Kubernetes-native authentication and multi-tenancy support to MLflow. | ||
|
|
||
| ## Configuring the KFP MLflow Plugin | ||
|
|
||
| To enable the MLflow plugin, add the following configuration to your KFP API server `config.json` file: | ||
|
|
||
| ```json | ||
| { | ||
| "plugins": { | ||
| "mlflow": { | ||
| "endpoint": "<schema>://<mlflow-service>:<mlflow-port>", | ||
| "timeout": 30, | ||
| "tls": { | ||
| "insecureSkipVerify": <boolean>, | ||
| "caBundlePath": "<path-to-ca-bundle>" | ||
| }, | ||
| "settings": {} | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### Configuration Values | ||
|
|
||
| Replace the placeholder values with your deployment-specific values: | ||
|
|
||
| - **endpoint**: The full URL of your MLflow server (e.g., `https://mlflow-service.mlflow.svc.cluster.local:8443`) | ||
| - **timeout**: Timeout in seconds for MLflow API calls (default: `30`) | ||
| - **tls**: TLS configuration for MLflow communication | ||
| - **insecureSkipVerify**: Set to `true` to skip TLS certificate verification (not recommended for production) | ||
| - **caBundlePath**: Path to the CA certificate for your MLflow server | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you also add the ConfigMap creation steps for better clarity? If MLflow uses a custom/internal CA, create a ConfigMap containing the CA certificate, mount it in the API server pod (for example at /kfp/certs/ca.crt), and set plugins.mlflow.tls.caBundlePath to that mounted path. |
||
| - **settings**: See the [user guide](/docs/components/pipelines/user-guides/integrations/mlflow-plugin/) for a complete list of MLflow plugin settings | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you verify if the hyperlink is working ? |
||
|
|
||
| ### Restart the API Server | ||
|
|
||
| After updating the configuration, restart the KFP API server to apply the changes: | ||
|
|
||
| ```bash | ||
| kubectl rollout restart deployment/ml-pipeline -n kubeflow | ||
| ``` | ||
|
|
||
| Verify the plugin is enabled by checking the API server logs: | ||
|
|
||
| ```bash | ||
| kubectl logs -n kubeflow deployment/ml-pipeline | grep mlflow | ||
| ``` | ||
|
|
||
| You should see messages indicating the MLflow plugin has been initialized successfully. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| +++ | ||
| title = "Integrations" | ||
| description = "Kubeflow Pipelines supports plugins with external services." | ||
| weight = 5 | ||
| +++ |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| +++ | ||
| title = "MLflow Plugin for KFP" | ||
| description = "Deploy MLflow on your cluster and configure the KFP MLflow plugin to use it." | ||
| weight = 1 | ||
| +++ | ||
|
|
||
| ## MLflow & Experiment Tracking | ||
|
|
||
| [MLflow](https://mlflow.org/) is an open-source platform for managing the end-to-end machine learning lifecycle. | ||
| It provides tools for experiment tracking, model versioning, and deployment. | ||
| The MLflow UI is a powerful interface for organizing, querying, and visualizing experiments and their corresponding runs, including run parameters and scalar metric artifacts. | ||
|
|
||
|
|
||
| ## Integration Overview | ||
|
|
||
| The KFP MLflow integration enables automatic experiment tracking for your Kubeflow Pipelines. When enabled, each pipeline run is automatically registered in MLflow, allowing you to: | ||
|
|
||
| - Track pipeline runs as MLflow experiments | ||
| - View pipeline parameters and metrics in the MLflow UI | ||
| - Organize related runs under custom experiments | ||
| - Maintain a complete audit trail of your ML workflows | ||
|
|
||
| This integration bridges the gap between pipeline orchestration and experiment tracking, giving you a unified view of your machine learning workflows. | ||
| All tracked data is viewable in the MLflow UI, organized by experiments. You can specify a custom experiment name for each run, or use the default experiment configured by your administrator. | ||
| This page provides detailed instructions on how to set up the MLflow plugin for Kubeflow Pipelines at the user level, once it has been deployed and configured at the [admin level](/docs/components/pipelines/operator-guides/mlflow-plugin/). | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could verify if the hyperlink is working? |
||
|
|
||
| ## Prerequisites | ||
|
|
||
| Before using the MLflow plugin, ensure your cluster administrator has: | ||
| - Deployed MLflow on your cluster | ||
| - Enabled and configured the KFP MLflow plugin | ||
|
|
||
| See the [operator guide](/docs/components/pipelines/operator-guides/mlflow-plugin/) for setup instructions. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could verify if the hyperlink is working? |
||
|
|
||
| ## User Configuration | ||
|
|
||
| To use the MLflow plugin with your pipelines, configure the plugin in your KFP API `config.json` file: | ||
|
|
||
| ```json | ||
| { | ||
| "plugins": { | ||
| "mlflow": { | ||
| "endpoint": "<schema>://<mlflow-service>:<mlflow-port>", | ||
| "timeout": 30, | ||
| "tls": { | ||
| "insecureSkipVerify": <boolean>, | ||
| "caBundlePath": "<path-to-ca-bundle>" | ||
| }, | ||
| "settings": {} | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Configuring Plugin Settings | ||
| See the [operator guide](/docs/components/pipelines/operator-guides/mlflow-plugin/#configuring-the-kfp-mlflow-plugin) for instructions on configuring the `endpoint`, `timeout` and `TLS` fields. | ||
| The following `settings` fields are available for further configuration: | ||
|
|
||
| #### authType | ||
| Authentication method for the MLflow server. Options: | ||
| - `kubernetes` (default): Use Kubernetes-based authentication | ||
| - `bearer`: Use bearer token authentication | ||
| - `basic-auth`: Use username/password authentication | ||
| - `none`: No authentication | ||
|
|
||
| #### credentialSecretRef | ||
| When using `bearer` or `basic-auth` authentication, this field references a Kubernetes secret containing the credentials. | ||
|
|
||
| #### workspacesEnabled | ||
| Enable MLflow workspaces for multi-tenant organization. | ||
| When `authType` is `kubernetes`, this defaults to `true`. Otherwise, it defaults to `false`. | ||
| Learn more about workspaces in MLflow in the [operator guide](/docs/components/pipelines/operator-guides/mlflow-plugin/#mlflow-workspaces). | ||
|
|
||
| #### defaultExperimentName | ||
| Default experiment name when none is specified for a pipeline run. Default is `"default"`. | ||
|
|
||
| #### experimentDescription | ||
| Default description for newly created experiments. Default is `"Created by Kubeflow Pipelines"`. | ||
|
|
||
| #### kfpBaseURL | ||
| If set, this URL is added as a tag to MLflow runs for traceability back to KFP. | ||
|
|
||
| #### mlflowBaseURL | ||
| Base URL for linking to the MLflow UI from pipeline runs. | ||
|
|
||
| #### mlflowUIPathPrefix | ||
| Path prefix for constructing MLflow UI links. | ||
|
|
||
| #### injectUserEnvVars | ||
| When set to `true`, injects MLflow environment variables into user containers, enabling a component's user code to interact with MLflow directly: | ||
|
|
||
| **General variables:** | ||
| - `MLFLOW_RUN_ID`: The MLflow run ID for the pipeline | ||
| - `MLFLOW_TRACKING_URI`: The MLflow endpoint URL | ||
| - `MLFLOW_EXPERIMENT_ID`: The MLflow experiment ID | ||
| - `MLFLOW_WORKSPACE`: The MLflow workspace name (if workspaces are enabled) | ||
|
|
||
| **Authentication variables:** | ||
| - `MLFLOW_TRACKING_TOKEN`: Bearer token (when using bearer auth) | ||
| - `MLFLOW_TRACKING_USERNAME` and `MLFLOW_TRACKING_PASSWORD`: Credentials (when using basic auth) | ||
|
|
||
| ## Using the Plugin | ||
|
|
||
| Once configured, pipeline runs are automatically registered in MLflow. You can optionally specify a custom experiment name when submitting a run through the KFP UI or SDK. If no experiment is specified, runs are logged to the default experiment. | ||
|
|
||
| ### Visualizing Pipeline Artifacts with MLflow | ||
| Current plugin artifact support extends to scalar Metric artifacts, which are logged to MLflow runs and can be visualized in the MLflow console. | ||
|
|
||
| ### Visualizing Pipeline Parameters with MLflow | ||
| Pipeline parameters are automatically logged to MLflow runs, making it easy to track and analyze the configuration of your pipeline runs. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.