Colombian Idioms Under Two Prompt Languages

Reproducibility artifact — EMNLP 2026. Dataset, model responses, scores, human annotations and figures for our study of how large language models handle Colombian modismos.

Looking for something specific?

If you want	Go to
a number from the paper, traced to its file	docs/RESULTS_INDEX.md
to re-run any stage	docs/PIPELINE.md
what the data is and where it came from	docs/DATA_CARD.md
what we know is broken	below

---

The question

In Colombia, Raimundo is not only a name. Raimundo y todo el mundo means everybody, and the dictionary gloss is plain: "Todos, mucha gente."

We put it to twenty-three models. Google's Gemini 3 Flash answered "alusión genérica que representa a la totalidad de las personas" — correct. DeepSeek Chat v3.1 called it "una persona rica o con abundantes recursos económicos." Wrong, but at least decisively wrong.

Then we asked the same model the same thing again, changing only the language the instructions were written in. In English it answered "persona o cosa usada como un ejemplo anónimo, genérico y común, similar a 'Fulano'." Still not right, yet far closer, and no longer the same mistake.

Both answers sit in the committed files, and you can pull them yourself:

import json
rows = json.load(open("04_metrics_es/LLMs_Results/prompt_2_metrics_data.json"))
[r for r in rows if r["modismo"] == "Raimundo"]      # then the same in 05_metrics_en/

One idiom proves nothing. Twenty-three models across three tasks, scored twice, is a different matter. This repository holds that, and whether prompt language moves the numbers systematically is settled in 06_cross_language/ — not by the anecdote above.

The three tasks

	The model is asked to	Scored by
Task 1	Say whether an expression is a Colombian idiom (Sí / No)	accuracy
Task 2	Define the idiom in one sentence, 60 words maximum	BERTScore, SBERT, chrF
Task 3	Read the idiom in context, give the most plausible paraphrase in 10 words or fewer, then define it	BERTScore, SBERT, chrF

Every prompt is in 02_responses_es/Straico/prompts.py and its English twin in 03_responses_en/Straico/prompts.py. Read them side by side; the wording is a deliberate translation, nothing else changes.

---

Read this before you read the folder names

The suffixes _es and _en mark the language the prompt was written in. They do not mark the language of the data.

The idioms are Spanish in both arms. The reference definitions are Spanish in both arms. The models answer in Spanish in both arms — the DeepSeek quotations above are both from the raw files, and both are Spanish. Nothing is translated except the instructions.

Treat 05_metrics_en/ as "English data" and every cross-language number will look inexplicable.

---

The chain of evidence

Six stages, each consuming the one before it. Nothing in the chain is regenerated on the fly: every intermediate file is committed, so you can enter at any stage and check the step from its own inputs.

                  Spanish prompts                 Spanish scores
              +-- 02_responses_es -------------> 04_metrics_es --+
              |                                                  |
01_dataset ---+                                                  +--> 06_cross_language
              |                                                  |
              +-- 03_responses_en -------------> 05_metrics_en --+
                  English prompts                 English scores

Stage	What happens	Size
01_dataset	Two Colombian dictionaries become one curated dataset	19 MB
02_responses_es	23 models answer the three tasks, prompted in Spanish	347 MB
03_responses_en	The same 23 models, prompted in English	346 MB
04_metrics_es	Scores for the Spanish arm, plus human and geographic analysis	669 MB
05_metrics_en	Scores for the English arm	676 MB
06_cross_language	The two arms compared	1.5 MB

Each stage has its own README with the detail. Paths inside a stage are relative to that stage, so run its notebooks from its own directory:

cd 04_metrics_es && jupyter lab

The last stage is small because it stores no copy of its inputs; it reads the two Metrics_Results/ directories directly. See Deduplication.

---

Checking it yourself

git clone https://github.com/SBobi/Modismos_EMNLP_2026.git
cd Modismos_EMNLP_2026
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python tools/verify_integrity.py

That script does three things: it checksums all 481 files against tools/MANIFEST.sha256, confirms that the files we deliberately kept in two places are still byte-identical, and greps the notebooks for credentials. It prints PASS or tells you which file disagrees.

You need no API key and no GPU to audit this work. Every response and every score is already committed. Keys buy you one thing only: re-querying the models from scratch, which costs money and will not reproduce the committed files, because these endpoints are non-deterministic and the identifiers behind them drift over time.

To recompute the analysis from the committed responses:

cd 04_metrics_es    && jupyter lab ComputeMetrics.ipynb   # then RankingModels.ipynb
cd 06_cross_language && jupyter lab LanguageAnalysis.ipynb

ComputeMetrics.ipynb is the one heavy step: it downloads four encoders and takes hours on a CPU. Everything downstream runs in minutes on a laptop.

Stage order, inputs and outputs in full: docs/PIPELINE.md.

---

Tracing a number back to the paper

docs/RESULTS_INDEX.md maps every figure and table to the file that produced it. Start there rather than grepping, because five things do not mean what they appear to mean:

• Task 3 ran on 3,916 idioms, though its input file has 4,998 rows. The file repeats idioms; the notebook does not.
• LLMs_Results/ and Metrics_Results/ use different denominators. The first keeps only responses that parsed; the second scores the unparsable ones 0.0.
• Score Promedio averages thirteen values, though only twelve sit beside it.
• Sonar sits at 0.0, ranked last, its Task 2 rows duplicated.
• The heatmap's colours are normalised per column; the printed numbers are raw.

None is a defect in the study, and each one caught us first. That document explains every one, with the code to work around it.

For the dataset itself — its sources, its schema, which two of the ten variants the paper uses, and how thin the regional coverage is — see docs/DATA_CARD.md.

---

What we know is wrong or missing

1. Sonar's Task 2 rows are duplicated. The paper queries 23 models and analyses 22, excluding perplexity/sonar because external retrieval could not be ruled out. ProcessResults.ipynb drops it; ComputeMetrics.ipynb writes it back as zeros, so the scored files list 23 with Sonar last at 0.0. Ranks are unaffected, but its Task 2 rows appear twice, 10,572 for 5,286 items, in both arms. Tasks 1 and 3 are clean.
2. gpt-5.1 was queried and never used. all_models.json lists 24 models: the 23 of models.txt plus gpt-5.1 from Azure. It reaches no scored file, no ranking and no table in the paper. Its raw responses are committed because they exist, not because they count.
3. The English arm has no human evaluation. 05_metrics_en/Human_Metrics/ holds the sampling scripts and the Label Studio spreadsheets, but no annotations. Only the Spanish arm was annotated.
4. The English arm has no maps. Its Geo_Analysis.ipynb writes map_task2.pdf; that file was never committed. The Spanish arm has all twenty.
5. claude_prompt3_annotated.csv reports 2,488 rows. A malformed quoted export doubled them. The true count is 1,245, matching the other three models. Deduplicate on annotation_id first.
6. 04_metrics_es/Geo_Analysis.ipynb was committed with its outputs stripped. Its interactive Plotly widgets weighed 80 MB, far past what GitHub will render. The twenty maps it draws are committed as PDFs beside it; re-run the notebook for the interactive versions.

---

Deduplication

Assembling six working repositories into one produced 1.1 GB of files that were byte-for-byte identical in two places. We kept one copy of each and repointed the consumer:

Deleted	Kept	Recovered
06_cross_language/Metrics_ES/	04_metrics_es/Metrics_Results/	546 MB
06_cross_language/Metrics_EN/	05_metrics_en/Metrics_Results/	562 MB

Two Path(...) constants at the top of LanguageAnalysis.ipynb changed; the edit is commented in place.

Smaller duplicates we left alone on purpose, so that each stage still runs by itself: 04_metrics_es/LLMs_Results/ repeats 02_responses_es/Results/Clean/ (77 MB, and the same for the English pair), and each querying stage carries the slice of the dataset it needs. verify_integrity.py checks that these copies have not drifted apart.

---

Where the data comes from

Two lexicographic sources, both third-party works:

• the Breve Diccionario de Colombianismos (2,693 entries), extracted from PDF with marker;
• the Diccionario de Colombianismos of the Instituto Caro y Cuervo (7,800 entries), retrieved through its REST API.

Our compilation, annotations and figures are CC BY 4.0 (LICENSE-DATA); the code is MIT (LICENSE). Neither licence can cover the dictionaries themselves. We redistribute extracted entries so the study can be checked; confirm terms with the publishers before any other use.

---

Credentials

No key is stored here, and the analysis needs none. The querying notebooks read the environment:

cp .env.example .env      # then fill it in

Two keys were hardcoded in the private repositories this artifact was assembled from: one for Straico, the aggregator serving the 23 models of models.txt, and one for Azure OpenAI, which served gpt-5.1. They are gone from these files, but they remain in that history and should be treated as compromised.

---

Citation

CITATION.cff, in Citation File Format.