gh-83273: Rewrite csv.Sniffer dialect detection using trial parsing

Guess the dialect by parsing the sample with every plausible
combination of delimiter, quotechar and escapechar, using the actual
CSV parser in strict mode, and choosing the combination which splits
the sample into rows with the most consistent number of fields.  The
old heuristics, which guessed the delimiter from characters adjacent
to quotes and from character frequencies, are removed.

A large sample is parsed incrementally: first only its beginning,
then, after eliminating the combinations which are clearly worse than
the leader, a several times larger part, and so on.

* csv.Sniffer can now detect escapechar='\\'.
* Explicitly requested delimiters are no longer restricted to ASCII.
* A delimiter inside a quoted field no longer wins over the actual
  delimiter, and sniffing no longer takes quadratic time on quoted
  samples.
* The sample can be cut off at an arbitrary point: in the middle of a
  row, of a quoted field or of an escaped sequence.
* Only '\r', '\n' and '\r\n' are treated as row separators, so
  characters like '\x1c' can now be detected as a delimiter.
* A preamble (title or comment lines) before the data does not prevent
  detection if the data rows outnumber the preamble lines.
* A sample consisting of a single column of quoted fields now raises
  csv.Error instead of guessing a delimiter from the content of the
  fields, and Sniffer.has_header() no longer raises ValueError for
  such samples.
* doublequote is detected by comparing the two readings of the sample.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: serhiy-storchaka

Doc/library/csv.rst

@@ -312,6 +312,22 @@ The :mod:`!csv` module defines the following classes:
       is given, it is interpreted as a string containing possible valid
       delimiter characters.
 
+      The dialect is deduced by parsing the sample with every plausible
+      combination of parameters
+      and choosing the combination which splits the sample into rows
+      with the most consistent number of fields,
+      so the returned dialect is consistent with how :func:`reader`
+      will parse the sample.
+      Raise :exc:`Error` if no combination fits the sample,
+      in particular if it is a single column,
+      so there is no delimiter to find.
+
+      .. versionchanged:: next
+         The dialect is now deduced by trial parsing
+         and the results may differ from those of earlier Python versions.
+         The *escapechar* parameter can now be detected,
+         and the requested *delimiters* are not restricted to ASCII.
+
 
    .. method:: has_header(sample)
 

Doc/whatsnew/3.16.rst

@@ -96,6 +96,21 @@ codecs
   even when a built-in codec of the same name exists.
   (Contributed by Serhiy Storchaka in :gh:`152997`.)
 
+csv
+---
+
+* :meth:`csv.Sniffer.sniff` now deduces the dialect by trial parsing
+  instead of heuristics based on matching isolated fragments
+  and on character frequencies,
+  so the result is consistent with how :func:`csv.reader` will parse the sample.
+  It can now detect the *escapechar* parameter,
+  accepts non-ASCII delimiters in the *delimiters* argument,
+  no longer misdetects the delimiter
+  when the sample contains delimiter characters inside quoted fields,
+  and no longer takes quadratic time on quoted samples.
+  The results may differ from those of earlier Python versions.
+  (Contributed by Serhiy Storchaka in :gh:`83273`.)
+
 curses
 ------