[VL] ORC column mapping mode (`orc.force.positional.evolution` / `orcUseColumnNames`) is global, but must be decided per-file — a query joining a `_col*` table and a named-column table cannot be read correctly

[beliefer]

Backend

VL (Velox)

Bug description

Summary

Gluten decides ORC column-mapping mode (by-name vs by-position) from a single global config (spark.hadoop.orc.force.positional.evolution, which flips spark.gluten.sql.columnar.backend.velox.orcUseColumnNames). Vanilla Spark decides it per ORC file, based on whether that file's physical schema uses placeholder names (_col0, _col1, …). As a result, a single query that reads two ORC tables with different physical schemas — one with real column names, one with Hive _col* names — cannot be read correctly under Gluten with any value of the global config: whichever value is chosen, one of the two tables is read wrong.

Vanilla Spark's per-file behavior (for reference)

OrcUtils.requestedColumnIds decides mapping mode file-by-file:

// spark/sql/core/.../orc/OrcUtils.scala
val orcFieldNames = reader.getSchema.getFieldNames.asScala
val forcePositionalEvolution = OrcConf.FORCE_POSITIONAL_EVOLUTION.getBoolean(conf)
...
if (forcePositionalEvolution || orcFieldNames.forall(_.startsWith("_col"))) {
  // map physical schema -> data schema by INDEX
} else {
  // map by NAME
}

The key part Gluten is missing is orcFieldNames.forall(_.startsWith("_col")): even when forcePositionalEvolution=false, Spark still switches to positional mapping automatically for any file whose physical schema is all _col*. This is a per-file decision, so Spark reads both a _col* table and a named-column table correctly in the same query.

Gluten's current behavior

Gluten resolves one boolean for the whole query:

• backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala:85

  def orcUseColumnNames: Boolean = getConf(ORC_USE_COLUMN_NAMES) &&
    !conf.getConfString(GlutenConfig.SPARK_ORC_FORCE_POSITIONAL_EVOLUTION, "false").toBoolean

• gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala:579 writes the global override into the native conf map when orc.force.positional.evolution=true.
• backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala:57 applies it to all local files in the split, regardless of each file's actual physical schema:

  if (((fileFormat == OrcReadFormat || fileFormat == DwrfReadFormat) && !VeloxConfig.get.orcUseColumnNames) || ...) {
    localFilesNode.setFileSchema(fileSchema)   // force by-position for every file
  }

There is no per-file / per-table detection of _col* schemas, so the mode is uniform for the entire query.

This is a follow-up to #12234 ("[VL] Respect orc.force.positional.evolution"), which made Gluten honor the global flag but did not add Spark's per-file _col* auto-detection.

Reproduction

A query joins two ORC tables:

• Table A — physical ORC schema has real column names:

  struct<account_id:string, account_uid:int, account_ccid:int, ..., account_type:string, ..., platform_type:string, ..., dt:int>
  

  → must be mapped by name.

• Table B — physical ORC schema is Hive _col* placeholders (File Version: 0.12 with HIVE_8732):

  struct<_col0:string, _col1:string, _col2:string, _col3:int, ..., _col13:int>
  

  → must be mapped by position.

-- simplified
SELECT b.game_urs, min(...)
FROM table_a a
JOIN table_b b ON a.account_id = b.cc_uid
WHERE a.dt BETWEEN 20260501 AND 20260630
  AND a.platform_type = 'cc' AND a.account_type = 'login_video'
  AND b.dt = 20260630 AND b.flag_role = 1
GROUP BY b.game_urs;

Case 1 — orc.force.positional.evolution=true (global):
Table B reads correctly, but Table A's real-name columns get mapped by position and land on the wrong (integer) file column. Query fails:

• Without a pushed filter (plain projection of account_type), read init fails:

  VeloxUserError SCHEMA_MISMATCH: Schema mismatch, From Kind: INTEGER, To Kind: VARCHAR
  Function: checkTypeCompatibility   (velox/dwio/common/TypeUtils.cpp)
  

  (account_type:string at request index 1 is mapped to file position 1 = account_uid:int.)
• With the string filter platform_type = 'cc' pushed down, it fails even earlier during stats pruning:

  VeloxUserError UNSUPPORTED: Filter(BytesValues, deterministic, no nulls): testInt64Range() is not supported.
  Function: testInt64Range   (velox/type/Filter.h)
  

  (A string BytesValues filter is dispatched against an integer file column in DwrfData::filterMatches → testFilter → testIntFilter.)

Case 2 — orc.force.positional.evolution=false (global):
Table A reads correctly, but Table B's _col* file is now mapped by name. The metastore names (cc_uid, flag_role, …) don't exist in the file (only _col0..._col13), so every column reads NULL. The filter flag_role = 1 AND cc_uid IS NOT NULL removes all rows → the join side is empty → AQE folds the whole plan to LocalTableScan <empty>, number of output rows: 0. Confirmed:

SELECT count(*) FROM table_b
WHERE dt=20260630 AND flag_role=1 AND cc_uid IS NOT NULL;
-- Gluten: 0     (vanilla Spark: > 0)

SELECT cc_uid, flag_role, game_urs FROM table_b WHERE dt=20260630 LIMIT 10;
-- Gluten: all NULL

Vanilla Spark 3.5.2 returns correct, non-empty results for the same query because it picks the mapping mode per file.

Impact: any query touching a mix of _col* and real-name ORC tables (very common in Hive-origin warehouses) either crashes or silently returns wrong/empty results under Gluten, with no single config that fixes both. The silent-empty case (Case 2) is especially dangerous — no error, just missing rows.

Proposed fix (align with Spark's per-file logic):
Decide the mapping mode per ORC file rather than from a global conf. When building the split (VeloxIteratorApi.setFileSchemaForLocalFiles) or in the native ORC reader, detect whether a file's physical schema is entirely _col* (equivalently, whether the physical field names match the requested schema names); use positional mapping for that file when forcePositionalEvolution is set or the file schema is all _col*, and by-name mapping otherwise. This makes mixed-schema joins read correctly without any user config, matching vanilla Spark.

Workaround today: run such queries with SET spark.gluten.enabled=false; (vanilla Spark handles both tables), or physically rewrite the _col* table to real column names so a single global positional=false satisfies all tables.

Gluten version

main branch

Spark version

Spark-3.5.x

Spark configurations

Reproduced on Spark 3.5.2, Gluten 1.6.0 (Velox backend), Hive/Iceberg ORC tables.

System information

No response

Relevant logs