feat(index): add num_segments for distributed index builds

[ddupg]

Background

Distributed index builds previously used num_workers for two different concerns: the number of fragment batches / index segments to create, and the maximum Ray Pool worker concurrency. These values do not always need to match. For example, a caller may want more index segments for the resulting index layout while keeping worker concurrency lower to fit available cluster resources. Coupling the two means increasing segment count also requires increasing concurrent workers, or limiting workers also limits segment count.

Summary

• add a num_segments option to distributed create_index and create_scalar_index
• keep num_workers as the maximum Ray Pool worker concurrency
• default num_segments to num_workers for backwards compatibility
• update distributed indexing docs and integration coverage for scalar/vector segment counts

Testing

• uv run pytest tests/test_vector_index_options.py
• uv run pytest tests/test_distributed_indexing.py::TestDistributedIndexing::test_distributed_fts_index_new_api tests/test_distributed_indexing.py::test_build_distributed_vector_index
• uv run ruff check lance_ray/index.py tests/test_vector_index_options.py tests/test_distributed_indexing.py

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.