Skip to content

Revert "Remove greedy regexes all together, no idea what they origina… - #356

Merged
amercader merged 2 commits into
ckan:masterfrom
benjwadams:revert_strip_xml
Oct 27, 2026
Merged

amercader merged 2 commits into
ckan:masterfrom
benjwadams:revert_strip_xml

Conversation

@benjwadams

Copy link
Copy Markdown
Contributor

…lly were trying to do"

This reverts commit a1f77d1.

Regex were not greedy, but rather the regex was matching XML declarations starting with "<?xml". This had downstream effects when calling etree.fromstring since the XML string was already str/Unicode and encoding declarations in the XML declaration would cause that function to fail and throw an exception.

…lly were trying to do"

This reverts commit a1f77d1.

Regex were not greedy, but rather the regex was matching XML
declarations starting with "<?xml".  This had downstream effects
when calling `etree.fromstring` since the XML string was already
`str`/Unicode and encoding declarations in the XML declaration would
cause that function to fail and throw an exception.
@benjwadams

Copy link
Copy Markdown
Contributor Author

For context, was initially receiving errors like this on some harvest objects that had XML declarations that declared an encoding:

2026-10-14 18:25:32,990 WARNI [ckan.common] Option smtp.port is not declared                                                                                                                                                                                     14:25 [4/1142]
2026-10-14 18:25:33,896 DEBUG [ckanext.harvest.queue] Fetch queue consumer registered
2026-10-14 18:25:33,912 INFO  [ckanext.harvest.queue] Received harvest object id: 028e0fbf-e448-411d-ab5e-edcb4fe64ad0
2026-10-14 18:25:34,791 WARNI [ckan.common] Option ckanext.spatial.harvest.user_name is not declared
2026-10-14 18:25:34,791 DEBUG [ckanext.spatial.harvesters.base.import] Import stage for harvest object: 028e0fbf-e448-411d-ab5e-edcb4fe64ad0
2026-10-14 18:25:34,801 WARNI [ckan.common] Option ckan.spatial.validator.profiles is not declared
Traceback (most recent call last):
  File "/usr/local/bin/ckan", line 6, in <module>
    sys.exit(ckan())
             ~~~~^^
  File "/usr/local/lib/python3.14/site-packages/click/core.py", line 1569, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.14/site-packages/click/core.py", line 1490, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.14/site-packages/click/core.py", line 1970, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/usr/local/lib/python3.14/site-packages/click/core.py", line 1970, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/usr/local/lib/python3.14/site-packages/click/core.py", line 1353, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.14/site-packages/click/core.py", line 907, in invoke
    return callback(*args, **kwargs)
  File "/srv/app/src/ckanext-harvest/ckanext/harvest/cli.py", line 241, in fetch_consumer
    utils.fetch_consumer()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/srv/app/src/ckanext-harvest/ckanext/harvest/utils.py", line 339, in fetch_consumer
    fetch_callback(consumer, method, header, body)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/app/src/ckanext-harvest/ckanext/harvest/queue.py", line 504, in fetch_callback
    fetch_and_import_stages(harvester, obj)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/srv/app/src/ckanext-harvest/ckanext/harvest/queue.py", line 522, in fetch_and_import_stages
    success_import = harvester.import_stage(obj)
  File "/srv/app/src/ckanext-spatial/ckanext/spatial/harvesters/base.py", line 568, in import_stage
    is_valid, profile, errors = self._validate_document(harvest_object.content, harvest_object)
                                ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/app/src/ckanext-spatial/ckanext/spatial/harvesters/base.py", line 888, in _validate_document
    xml = etree.fromstring(document_string)
  File "src/lxml/etree.pyx", line 3433, in lxml.etree.fromstring
  File "src/lxml/parser.pxi", line 2076, in lxml.etree._parseMemoryDocument
ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

@benjwadams

Copy link
Copy Markdown
Contributor Author

I think that the call to lxml.etree.fromstring should possibly use a bytes string instead of str/Unicode. If there's a different encoding under the current code it's obviously going to fail. Under the reverted code, however, if there's some encoding aside from UTF-8 this that goes into byte ranges > 127, (i.e. not ASCII) it has the potential to introduce decoding errors.

@amercader

Copy link
Copy Markdown
Member

Not sure what is going on with the 2.10 tests but it is unrelated

@amercader
amercader merged commit bac5bb9 into ckan:master Oct 27, 2026
6 of 8 checks passed
@benjwadams
benjwadams deleted the revert_strip_xml branch October 28, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants