Databricks instance pool Terraform module

One instance pool.

Creates one instance pool and, when permissions is not empty, one databricks_permissions block on it.

Resources

• databricks_instance_pool.this
• databricks_permissions.this[0] (count = 1 only when permissions is not empty)

The resource addresses above are part of the DataTF import contract. Do not rename them.

Usage

module "instance_pool" {
  source  = "536tech/instance-pool/databricks"
  version = "1.0.0"

  name                                  = "shared-pool"
  node_type_id                          = "Standard_DS3_v2"
  min_idle_instances                    = 1
  max_capacity                          = 10
  idle_instance_autotermination_minutes = 15
  enable_elastic_disk                   = true
  preloaded_spark_versions              = ["15.4.x-scala2.12"]

  azure_attributes = {
    availability       = "ON_DEMAND_AZURE"
    spot_bid_max_price = -1
  }

  permissions = [{
    permission_level = "CAN_MANAGE"
    user_name        = "jon@example.com"
  }]
}

Compatibility

Configure the Databricks provider in the calling root with a workspace endpoint. This resource module is also used by the workspace pattern module. Each repository has its own releases. Consumers select an exact tested module version.

The resource addresses match the original workspace submodule in version 0.1.1. To migrate a direct submodule call, change its source and version. Keep the module block name. Run terraform init and require a plan with no resource changes. DataTF exports continue to use the workspace pattern module and its existing import addresses.

Development

Use Terraform 1.7 or later for the mock tests. The module supports Terraform 1.5 or later.

prek install
terraform init -backend=false -lockfile=readonly
terraform validate
terraform test
tflint --recursive
prek run --all-files

CI tests the committed provider version and the minimum supported provider, 1.128.0. The workspace pattern module checks the complete DataTF contract and its integration behavior.

License

Apache-2.0.

Requirements

The following requirements are needed by this module:

• terraform (>= 1.5.0)

• databricks (>= 1.128.0, < 2.0.0)

Providers

The following providers are used by this module:

• databricks (>= 1.128.0, < 2.0.0)

Resources

The following resources are used by this module:

• databricks_instance_pool.this (resource)
• databricks_permissions.this (resource)

Required Inputs

The following input variables are required:

enable_elastic_disk

Description: Add disk space to pool instances when they run low.

Type: bool

idle_instance_autotermination_minutes

Description: Minutes an idle instance stays in the pool above min_idle_instances.

Type: number

min_idle_instances

Description: Instances the pool keeps ready.

Type: number

name

Description: Instance pool name.

Type: string

node_type_id

Description: Azure VM size for the pool, for example Standard_DS3_v2.

Type: string

preloaded_spark_versions

Description: Databricks Runtime versions cached on pool instances.

Type: list(string)

Optional Inputs

The following input variables are optional (have default values):

azure_attributes

Description: Azure placement settings for pool instances.

Type:

object({
    availability       = optional(string)
    spot_bid_max_price = optional(number)
  })

Default: null

custom_tags

Description: Tags applied to pool instances.

Type: map(string)

Default: null

max_capacity

Description: Maximum number of instances in the pool.

Type: number

Default: null

permissions

Description: Direct permissions on the pool. Each element names exactly one principal.

Type:

list(object({
    permission_level       = string
    group_name             = optional(string)
    user_name              = optional(string)
    service_principal_name = optional(string)
  }))

Default: []

Outputs

The following outputs are exported:

id

Description: Instance pool id.

name

Description: Instance pool name.