Repository navigation
Top-level map columns reflect as String #857
Description
Problem
AthenaDialect._get_column_type now reflects array<...> and nested map/struct types, but a top-level map<string,int> column still reflects as String, so the key and value types are lost for inspectors and autoload_with tables.
Reproduction
from pyathena.sqlalchemy.base import AthenaDialect
AthenaDialect()._get_column_type('map<string,int>') # String()
AthenaDialect()._get_column_type('array<map<string,int>>').item_type # AthenaMap(...)Environment
PyAthena 3.35.4 and master (c01c56f), Python 3.11, SQLAlchemy 2.0.52, Athena engine version 3.
Proposed fix (optional)
Apply the nested map branch at the top level too, returning AthenaMap(key_type, value_type).
Activity
laughingman7743 commentedon Sep 26, 2026
Confirmed.
_get_column_typeonly takes themapandrow/structbranches when_nestedis set; #755 / PR #774 deliberately limited typed reflection to ARRAY columns, so top-levelmap<...>still reflects asString. The same applies to top-levelstruct<...>/row(...)columns, which this issue does not mention. Please widen the scope to cover both so reflection is consistent, returningAthenaMap(...)andAthenaStruct(...)respectively.This changes the reflected type of existing columns from
String, so it will be release-noted as a behavior change.Per CONTRIBUTING.md, please validate against a real table (Glue metadata and the
information_schemafallback both feed this code) in your own AWS environment and record before/after results in the PR's TEST section. External-fork PRs are not run in the project's AWS CI.laughingman7743 commentedon Sep 26, 2026
Target
masterfor the fix; the maintainers will port it to the3.xmaintenance branch afterwards, so please do not open a PR against3.x.laughingman7743 commentedon Sep 26, 2026
If you plan to send a PR for this, please say so here. If the issue stays open for a long time without a PR, the maintainers may pick it up and implement the fix themselves.
laughingman7743 commentedon Oct 3, 2026
Fixed on
masterby #995 (6be7de3): top-levelmap<...>columns now reflect asAthenaMap(key_type, value_type), and top-levelstruct<...>/row(...)columns asAthenaStruct(...), as nested ones already did. For example,AthenaDialect()._get_column_type("map<string,int>")returnsAthenaMap(key_type=String(), value_type=INTEGER()). The change will be release-noted as a behavior change, since these columns previously reflected asString. Thank you for the report.Backport #995: Reflect the field and value types of top-level STRUCT …
laughingman7743 commentedon Oct 6, 2026
The fix is released in v3.38.0, backported to the 3.x line in #1104.
Reflected top-level
map<...>columns areAthenaMap(key_type, value_type), and top-levelstruct<...>/row(...)columns areAthenaStructwith their field types, instead ofString.On 3.x, an unrecognized field, key, or value type is reflected as
NullTypewith a warning and still renders asNULLinCREATE TABLE. In 4.0.0,NullTyperaisesCompileErrorin DDL instead (#995, #961). Thank you again for the report.