Regression on parsing invalid URLs

[kamil-certat]

As a continuation of #2377, we have a regression on parsing invalid URLs. Previously, the urllib was mach more liberal in processing URLs, now it rejects much more cases.

We use it for sanitize the URLs, and html_parser is an example of bot that uses the liberal behavior in tests:

intelmq/intelmq/tests/bots/parsers/html_table/test_parser_column_split.py

Line 47 in 61c45ac

EXAMPLE_EVENT2['source.url'] = "http://[D] lingvaworld.ru/media/system/css/messg.jpg"

intelmq/intelmq/tests/bots/parsers/html_table/test_parser_column_split.py

Lines 73 to 80 in 61c45ac

	def test_event_without_split(self):
	self.sysconfig = {"columns": ["time.source", "source.url", "malware.hash.md5",
	"source.ip", "__IGNORE__"],
	"skip_head": True,
	"default_url_protocol": "http://",
	"type": "malware-distribution"}
	self.run_bot()
	self.assertMessageEqual(0, EXAMPLE_EVENT2)

In patched Python versions (e.g. 3.11.4), this URL is rejected. We need to either decide against allowing such URLs, or redesign our sanitization.

Temporally, the test is skipped to unlock other work.

[sebix]

We now have a PR #2714 by @0xTaoZ to allow such URLs. I find this weird. They are syntactically just invalid. Any bots further down the processing order can't anymore rely on the URLs being valid and will fail in unexpected ways.

What such bots and users can do: If the URL is not valid, use it as extra.source.url. In the case of the generic parsers configuring it with columns "source.url|extra.source.url"

[0xTaoZ]

@sebix thanks for clarifying. I read #2382 as leaving both approaches open, so I initially went with preserving the previous permissive behavior. Your point about downstream bots being able to rely on source.url being valid makes sense.

The HTML table parser already supports fallback columns, so I can update #2714 to keep the strict URL validation and change the skipped regression case to use source.url|extra.source.url, preserving the invalid original value in extra.source.url. If that is the preferred resolution, I can make that change now.

[sebix]

I'd like to hear others' opinions on this, especially from @kamil-certat