OCR Corpora Early 2025 Publication Thread

[ctschroeder]

OCR publication thread for early 2025

Do not close this issue until all checkboxes below are complete or have been rescheduled:

List of corpora:

In Processed OCR folder (need chapter divisions + sentence splitting+full automatic NLP processing like the bible corpora)

• Giron Legendes (11 docs)
  • chapter divisions should follow
  • Daughters Zenon 132
    • chapter divisions added/checked
    • pb xml:id numbers corrected
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
    • this document should be divided into 3: 1) BNF 132.1 ff. 19-20, 2) Leiden fragment, 3) BNF 132.1 f 21
  • Daughters Zenon 78
    • chapter divisions added/checked
    • pb xml:id numbers corrected
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Daughters Zenon Crawford
    • chapter divisions added/checked
    • pb xml:id numbers corrected
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Emperors Daughter 1
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Emperors Daughter 2
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Emperors Daughter 3
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Emperors Daughter 4
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Eve Serpent
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Marina BNF
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Marina Clarendon
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py
  • Sacrifice Abraham
    • chapter divisions added/checked
    • lb n tags changed to ed_line n
    • ed_pg n tags changed to ed_page n
    • pb xml:id n tags changed to pb xml:id
    • metadata updated
    • final validation at https://gucorpling.org/gitdox/validate_sgml.py

all documents above need to be moved to https://github.com/CopticScriptorium/auto-corpora when done

In GitDox

• timothy.alexandria corpus prayer.athanasius

  • translation span (work by E. Sturgis via @amir-zeldes in progress)
  • entities and identities
  • metadata

• apocalypse.paul (2)
  - [ ] corpus name needed
  - [ ] other metadata updated
  - possibly error in data -- translation on p. 1043 begins with folio 24a but OCR coptic begins in the middle of folio 6a p. 533

• pscyril.alexandria

  • On Mary still in XML mode (auto tagging?)
  • entities and identities

• pscyril.jerusalem

  • on the cross
    • needs corpus name
    • metadata updated
    • chapter & verse need to be updated in spreadsheet based on open tags in XML
  • on Mary
    • needs corpus name
    • metadata updated
    • chapter & verse need to be updated in spreadsheet based on open tags in XML

• psepiphanius on Mary

  • translation span
  • entities and identities
  • metadata

• pschrysostom

  • translation span
  • entities and identities
  • metadata

• pscelestinus

• pstimothy.alex

• psote.psoi

• timothy.discourse

[ctschroeder]

I know we were going to put Giron in the "automatically processed straight from XML like the big bible corpora" bucket, but with the sentence splitting and chapter divisions required, I think it will be easier to do both in GitDox. Chapter divisions are easier to see in the spreadsheet mode, and you can use the entity mode to do the sentence-splitting. So I'm making the executive decision to put this stuff in GitDox. @amir-zeldes @EarlyCodices

[amir-zeldes]

Fine by me - 11 docs should still be manageable, I'd be more worried for really big stuff. Keep in mind that one potential advantage of "XML Bible mode" is that tokenization is done fresh each time as tokenizers improve, so if you're correcting any segmentation manually, GD can make more sense (or you could introduce pipes in the XML repo). If you want fresh re-tokenization each release, you should not put things in GD - that would be the other main reason not to choose it.

[ctschroeder]

I don't think we are checking segmentation manually, but this is a good point. We have lots to do so I'll ponder this. They already have pipes from Lydia's work. I found when I was working on other documents for the full-auto-from-xml mode, validation was a real pain for reasons I don't fully remember and probably repressed.

[ctschroeder]

Daughers of Zeno(n) is actually Life of Hilaria. Doc titles updated. @amir-zeldes can you confirm that we only need chapter not sentence divisions for the automatically processed docs? thanks!

[amir-zeldes]

Yes, that's right - if there are chapter tags I can automatically predict sentences within those (I just need to remember to switch that option on, but unless I'm very distracted I think I'll do it the right way 😅)

[ctschroeder]

Great. Thanks @amir-zeldes! I think I will leave the documents in with pipes but remove the pipes for the auto folder. I'll validate everything and fix the metadata here and then remove the pipes when copying over.

[amir-zeldes]

You can but you don't have to - I have a setting that auto-removes them if I know a document needs re-tokenizing (which is most of auto-corpora)

[ctschroeder]

@amir-zeldes what do you do about identities for these? (I assume entities=automatic)

[ctschroeder]

Note to self: Discovered that in Life of Hilaria ("Daughters of Zeno") Giron inserted text from aLeiden ms between folios 20 and 21. The text is continuous and in the proper place, but according to our guidelines this document should be divided into 3: 1) BNF 132.1 ff. 19-20, 2) Leiden fragment, 3) BNF 132.1 f 21.