Release official Java API

[headius]

This is a parent issue for all tasks needed to declare the Java API "official" and release it for the general JVM public.

Subtasks:

• Split artifact into share API, native backend, WASM backend, and WASM backend with full non-semantic data (see discussions in Conflicting requirements between Ruby and Java builds #3962).
• Consider depending on jcodings Encoding to align with CRuby's supported encodings (Java's default set of Charsets does not cover all cases).
• Consider storing regular expressions as compiled JOni Regex to verify them at parse time.
• Consider shifting to or adding a zero-allocation protobuf-style API.
• Documentation.
• Bump version to 1.0.0.

[kddnewton]

Should the version just align with Prism version?

[eregon]

Consider depending on jcodings Encoding to align with CRuby's supported encodings (Java's default set of Charsets does not cover all cases).

With #4016 I think Charset isn't used anywhere, so seems unnecessary.

Consider storing regular expressions as compiled JOni Regex to verify them at parse time.

It seems too big a dependency to always have it and it's also pretty expensive to create Regex objects (something I would actually like to make lazy in TruffleRuby, currently they are created eagerly).
It could be nice if Prism could warn/error for all issues in Regexp, or maybe JOni have a mode to do that without compiling the regexp to JOni bytecode (which I guess is the expensive part).

Consider shifting to or adding a zero-allocation protobuf-style API.

Could you elaborate?

[headius]

JCodings Encoding alongside identifer bytes

With #4016 I think Charset isn't used anywhere, so seems unnecessary.

Why is it there for TruffleRuby? Can't you infer the Charset from the source encoding?

too big a dependency

Perhaps, yes. I again come back to wishing we would design this API to accept a factory for creating things like identifiers and regular expression objects.

pretty expensive to create Regex objects

Regular expression syntax errors are supposed to be reported at parse time, so at least the parsing needs to happen eagerly.

$ cx 4 ruby -e 'def foo; puts /*/; end'
-e: -e:1: syntax error found (SyntaxError)
> 1 | def foo; puts /*/; end
    |                ^ target of repeat operator is not specified

if Prism could warn/error for all issues in Regexp

Can Prism parse the Ruby regular expression grammar?

zero-allocation protobuf-style API

For identifiers, I see a lot of allocation of byte[] and a lot of copying of bytes out of the constant pool and it makes me want to replace those byte[] with a byte[] + offset + length. Instead of getting byte[] name directly from a CallNode, you would call something along the lines of

interface ByteStringFunction<T> {
     T apply(byte[] buffer, int offset, int length);
}

public <T> T constructName(ByteStringFunction<T> func) {
    return func(bytes, offset, length);
}

Expanding out from that idea would be either making that data copying lazier in other places, including regular strings and symbols. The advantage would be limiting copying of data out of the serialized buffer until it's actually needed. The disadvantage would be the need to hold onto the serialized buffer much longer.

A full protobuf-style system would make nodes essentially stateless, so instead of constructing nodes you'd access node-specific data directly out of the serialized buffer. That's probably not feasible for this API but it's an idea I've wanted to try.

void processCallNode(Buffer buffer, int pointer) {
    RubySymbol name = CallNode.name(buffer, pointer, ruby::newSymbol);
    Operand r = processReceiver(buffer, CallNode.receiver(buffer, pointer));
    Operand a = processArgs(buffer, CallNode.args(buffer, pointer));
    addInstr(new CallInstr(name, r, a));

This is pseudocode based on JRuby's IR compiler. The only allocation that would be forced to happen would be the RubySymbol, the two Operand instances, and the CallInstr instance, since that's needed for our IR representation. The Prism nodes would not actually exist other than as a stream of data.

There's docs out there about how protocol buffers work, if you aren't familiar. I'm not an expert.

[headius]

@kddnewton:

Should the version just align with Prism version?

It certainly can, if you aren't concerned about us having to rev that version for Java-specific changes. The alternative that @eregon and I have hinted at would be something like ###.x.y where ### is a shorthand form of the base Prism version and we'd make our iterative changes to the Java logic in x.y.

Of course this also ignores whether the Java API version should reflect the version of Ruby syntax it supports...

[eregon]

Why is it there for TruffleRuby? Can't you infer the Charset from the source encoding?

It's there so it works without extra dependencies for conversion to j.l.String.
TruffleRuby overrides those like this: https://github.com/truffleruby/truffleruby/blob/3c830c7eb0f70bba9f54197ba09e8bf9dc3efbd7/src/main/java/org/truffleruby/parser/YARPLoader.java#L61-L71

I'll add a comment on #4016, I think we can remove encodingCharset entirely.

Can Prism parse the Ruby regular expression grammar?

It can parse it, but I think it doesn't emit warnings/errors for it.

[kddnewton]

It emits some warnings/errors for it. See all of the pm_regexp_parse_error calls in regexp.c. I'm happy to add more.

[eregon]

replace those byte[] with a byte[] + offset + length.

I see. Since these are identifiers I expect in several cases we'd still need a byte[] without offset to look them up, but if converting e.g. to org.jruby.RubySymbol then yeah you can do the lookup with that triplet.
A downside is the non-trivial overhead of that extra triplet object in many Node instances.

It'd be nice to actually go buffer->j.l.String for TruffleRuby without an extra byte[] in between. Could be done by passing offset and length to bytesToName().

Another solution here is to give more access to the constant pool, so we could create such triplet objects, etc.
Maybe the simplest would be if loadConstant just returns a constant pool index and nothing else?
See #4009 (comment)

[eregon]

A full protobuf-style system would make nodes essentially stateless, so instead of constructing nodes you'd access node-specific data directly out of the serialized buffer. That's probably not feasible for this API but it's an idea I've wanted to try.

I thought about something somewhat similar a long time ago, we could avoid building nodes and instead pass the value of the fields to the visitor methods. For child nodes, we'd visit them before and return the result of the visit method.
In practice it seems hard to use because it implies visiting all leafs before other nodes, but compilation might need to visit non-leaf nodes first, e.g. to know local variables and pass some information down.

The processCallNode() example is different because it does allow visiting non-leak nodes first. But then it seems to require knowing the order of the fields for each node which seems too brittle and hard to maintain if not generated code (and if generated then I guess no way to use the fields without e.g. an Object[] or some object to pass the values between code in Prism and code in the compiler).

[headius]

give more access to the constant pool,

loadConstant just returns a constant pool index

I was thinking along these lines myself. The nodes could simply aggregate a constant pool index/offset for their identifier field. Getting the actual constant could be either a method on the node that does the lookup (rather than a raw field access as it is now) or users would be expected to request the identifier from the ConstantPool construct themselves.

This is easier if ConstantPool is generic and can cache whatever identifier type the caller wants. TruffleRuby would just put String in there, JRuby could put RubySymbol, and no transient byte[] would be living in the AST.

Again, though, for this to be lazy the bytes of the constant pool (at least) would have to remain in memory. And generics have some checkcast overhead (though for this case, it would always be exactly one type for JRuby and TruffleRuby's uses).

hard to use because it implies visiting all leafs

I think that's where I got stuck in past attempts to do this with the JRuby IR serialization dumper.

There's also the argument that a protobuf-style API is super cumbersome to work with even in the best case. Best performance, bar none, but not a lot of fun to use.

Probably safe to file this under YAGNI and if someone in the future wants to (needs to) do a protobuf version of this, great.

[eregon]

Probably safe to file this under YAGNI and if someone in the future wants to (needs to) do a protobuf version of this, great.

Yep, agreed on YAGNI for protobuf-style. BTW I've been doing some measurements of parse/serialize/deserialize and the compiler (Prism AST to Truffle AST) in TruffleRuby and the compiler is taking more time than deserialize.
IOW deserialize is not really the bottleneck right now.
EDIT: parse it taking most of the time.
It's like 55.5% parse, 8% serialize, 5.5% deserialize, 7.35% compile.
pm_parser_free is taking 2%.

Hence why I'm looking at using the Prism depth fields and possibly the constant pool index to avoid extra hash lookups in the compiler.