You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Atom id and the retention cache key raise UnicodeEncodeError for an atom statement or artifact text with a lone surrogate #758
Atom.__post_init__ (dokimi/dokimi/novelty/atoms.py:120) hashes self.statement.strip().encode("utf-8") when no id is given, and retention._key (dokimi/dokimi/novelty/retention.py:258,260) hashes the statement and the text the same way. A statement parsed from a model reply or an atoms file can hold a lone surrogate from a JSON escape. Repro: Atom(statement=json.loads('"n\\ud800"'), question='q', answer='a') and retention._key(Atom(statement=<that>, question='q', answer='a', id='a1'), 't') each raise UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position N: surrogates not allowed; the same happens with a surrogate in the text argument (line 260). Fix: .encode('utf-8', 'surrogatepass'); the id and the key of every statement that encodes today stay the same, which matters because cached verdicts are keyed on them.
Activity