Skip to content

[R][C++] terminate called after throwing an instance of 'std::length_error' #45872

Closed

Description

@rafapereirabr

Describe the bug, including details regarding any error messages, version, and platform.

Problem

I'm using {arrow} as a dependency in my package {geocodebr}. In one particular case, I use {arrow} to keep only the distinct rows of a large table originally with 50,240,061 rows. When I run the code of the reprex (see below), it simply crashes Rstudio. When I use this code inside my package, though, it throws the following error messag, which seems to come from an error in C++:

terminate called after throwing an instance of 'std::length_error'
what(): vector::_M_default_append

ps. I just wanted to add that {arrow} is an incredible package and that the R community really appreciates your work on it ! Thanks !

Reprex

remotes::install_github("ipeaGIT/geocodebr")

library(geocodebr)
library(dplyr)
library(arrow)

# download parquet file
geocodebr::download_cnefe(tabela = 'municipio_logradouro_numero_cep_localidade')

path_to_parquet <- geocodebr::listar_dados_cache()[1]

key_cols <- c("estado", "municipio", "logradouro", "numero", "cep" )

unique_logradouros <- arrow::open_dataset(path_to_parquet) |>
  dplyr::select(dplyr::all_of(key_cols)) |>
  dplyr::distinct() |>
  dplyr::compute()


## remove files from cache
# geocodebr::deletar_pasta_cache()

This error only occurs when I use all five key columns. When I remove the column "numero", though, it works fine.

key_cols <- c("estado", "municipio", "logradouro", "numero", "cep" )

Environment

I'm running this on machine with 250GB of RAM. I'm using the latest version of {arrow} in a Windows OS. See below.

> sessionInfo()
R version 4.3.2 (2023-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows Server 2022 x64 (build 20348)

Matrix products: default


locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8   
[3] LC_MONETARY=English_United States.utf8 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.utf8    

time zone: America/Sao_Paulo
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] arrow_19.0.1    dplyr_1.1.4     geocodebr_0.2.0 testthat_3.2.3 

loaded via a namespace (and not attached):
 [1] rappdirs_0.3.3     generics_0.1.3     class_7.3-22       KernSmooth_2.23-22 digest_0.6.37     
 [6] magrittr_2.0.3     grid_4.3.2         nanoarrow_0.6.0    pkgload_1.4.0      fastmap_1.2.0     
[11] rprojroot_2.0.4    pkgbuild_1.4.6     sessioninfo_1.2.3  e1071_1.7-16       backports_1.5.0   
[16] brio_1.1.5         DBI_1.2.3          urlchecker_1.0.1   promises_1.3.2     purrr_1.0.4       
[21] httr2_1.1.1        duckdb_1.2.0       cli_3.6.4          shiny_1.10.0       rlang_1.1.5       
[26] units_0.8-7        ellipsis_0.3.2     bit64_4.6.0-1      remotes_2.5.0      withr_3.0.2       
[31] cachem_1.1.0       devtools_2.4.5     parallel_4.3.2     tools_4.3.2        sfheaders_0.4.4   
[36] memoise_2.0.1      checkmate_2.3.2    httpuv_1.6.15      curl_6.2.1         assertthat_0.2.1  
[41] vctrs_0.6.5        R6_2.6.1           mime_0.12          proxy_0.4-27       classInt_0.4-11   
[46] lifecycle_1.0.4    bit_4.6.0          fs_1.6.5           htmlwidgets_1.6.4  usethis_3.1.0     
[51] miniUI_0.1.1.1     pkgconfig_2.0.3    desc_1.4.3         pillar_1.10.1      later_1.4.1       
[56] data.table_1.17.0  glue_1.8.0         profvis_0.4.0      Rcpp_1.0.14        sf_1.0-19         
[61] tibble_3.2.1       tidyselect_1.2.1   rstudioapi_0.17.1  xtable_1.8-4       htmltools_0.5.8.1 
[66] compiler_4.3.2     enderecobr_0.4.1  

Component(s)

R, C++

Activity

  1. changed the title [-]terminate called after throwing an instance of 'std::length_error'[/-] [+][R][C++] terminate called after throwing an instance of 'std::length_error'[/+] on Mar 20, 2025
  2. thisisnic commented on Nov 14, 2026

    @thisisnic
    Member

    Had Claude look into this and could reproduce it but looks like it has been fixed. Please open a new issue if not:

    🤖 :

    This reproduces on arrow 26.0.0 with a synthetic reprex: calling distinct() on a large_string column with more than 2GB of distinct values aborts with the same std::length_error / vector::_M_default_append message.

    library(arrow)
    library(dplyr)
    
    # 2300 distinct ~1MB large_string values, written as small files
    dir.create("big_keys")
    pad <- strrep("a", 2^20)
    for (f in 1:46) {
      ids <- (f - 1L) * 50L + 1:50
      tab <- arrow_table(key = Array$create(paste0(pad, ids), type = large_utf8()))
      write_parquet(tab, sprintf("big_keys/part-%02d.parquet", f), chunk_size = 10L)
    }
    
    open_dataset("big_keys") |>
      distinct() |>
      compute()
    #> terminate called after throwing an instance of 'std::length_error'
    #>   what():  vector::_M_default_append

    The cause is that the logradouro column in the file from the original report is large_string (checked from the Parquet footer of the v0.2.0 data release), which sends the query through a grouper that used 32-bit offsets for the key data, so it overflowed once the distinct keys passed 2GB. That code was changed to 64-bit offsets in #55170 (GH-54885), and the same reprex completes on a dev build which includes that change (C++ 27.0.0-SNAPSHOT), so the fix should be in the 27.0.0 release.

    Until then, casting the large_string column to string() before distinct() avoids the problem, e.g. mutate(logradouro = cast(logradouro, string())). This was checked on 26.0.0 with the same reprex.

    Not checked: the original file itself (only its schema) and arrow 19.0.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions