Underscore in numbers are not handled correctly

[fh-yuxiao-zeng]

YAML allows underscores in numbers to increase readability (see https://yaml.org/type/int.html. "Any “_” characters in the number are ignored, allowing a readable representation of large values.")

However, it seems that this library can't handle it properly. Given a standard ASP.NET Core application and a config like this:

Kestrel:
  Limits:
    MaxRequestLineSize: 16_384
    MaxRequestBodySize: 104_857_600

I see an exception like this:

Unhandled exception. System.InvalidOperationException: Failed to convert configuration value at 'Kestrel:Limits:MaxRequestLineSize' to type 'System.Int32'.
 ---> System.ArgumentException: 16_384 is not a valid value for Int32. (Parameter 'value')

[andrewlock]

Unfortunately this is a limitation with how the .NET configuration system works.

The .NET configuration system is inherently string based. Wherever the configuration data is read from, it's stored as a string in the system. This is then bound to a strongly-typed object, at which point the strings are parsed to int etc.

The Yaml configuration system reads all the config values using YamlDotNet, using the YamlStream abstraction. Unfortunately, AFAICT, the YamlDotNet deserializer also requires a "target" type to deserialize these values, otherwise they're converted to strings e.g. "16_384", and then the binding happens later. I can't see an obvious way atm to make YamlDotNet parse the value as an int when it doesn't "know" that it should be an int...

[fh-yuxiao-zeng]

@andrewlock thanks for looking into this issue. Today I spent some time to understand the libraries. Let me describe my understanding.

As you mentioned, YamlStream is used to parse the yaml content. YamlStream can return the YAML nodes (scalars, tags, etc), and we need to construct the value from the nodes.

On the other hand, in Java world, Spring supports YAML config as well, and underscore in numbers works well there. What's the difference between their implementation and this library? It turns out that Spring doesn't deal with yaml nodes directly. Instead, it directly parses a yaml file into an object (source). The type of the object is already handled by the underlying YAML library, so 12_34 will be parsed as the integer 1234.

It's worth mentioning that Java config is also string based. After Spring gets the YAML object, it would flatten the dictionary and store all configs into string based key value pairs. Then later when a value is used, it will be converted to the right type.

If we follow their logic, then instead of YamlStream, we can use Deserializer (and turn on the attemptUnknownTypeDeserialization option so that it can detect the right types) to parse the YAML documents.

BTW while testing, I found that YamlDotNet is not parsing numbers with underscores consistently: aaubry/YamlDotNet#1058. But once their bug is fixed, I feel the Deserializer approach should allow underscroes to be used in configs.

What do you think?

[fh-yuxiao-zeng]

@andrewlock We have implemented this internally. I've extracted the implementation into the PR #84.

The new implementation is more complaint with YAML spec. For example, it can handle yes and no as boolean values (see the test case LoadMethodCanHandleBooleanValue). It can handle YAML tags (see the test case LoadMethodCanHandleYamlTags). It can handle underscores in numbers (for now when a tag is used. still waiting for YamlDotNet to fix their issue). It also supports YAML anchors.

The new implementation also allows parsing customization using tags, by providing a callback to modify the deserializer.

Could you have a look at the PR?