Support doc query

[liufengyun]

Fix #84 : Support doc query

Summary

Support query symbol information from command line:

jo compile --doc --query jo.List.map

It outputs

[
  {
    "name": "jo.List.map",
    "kind": "def",
    "signature": "def map[S](f: T => S receives none): List[S] receives none",
    "source": { "file": "lib/List.jo", "line": 162 },
    "visibility": "public",
    "flags": ["loaded", "fun", "method"],
    "annotations": [],
    "doc": "Apply `f` to each element and return the resulting list."
  }
]

It is also possible to filter by file name:

jo compile --doc --query file:main.jo src/*.jo

Only symbols defined in the specified will be included.

Checklist

• Added / updated tests under tests/pos/ or tests/warn/
• Docs updated if the change affects user-visible behavior
• All commits are signed off (why?)

How to sign off commits

Use git commit -s to add the Signed-off-by line automatically:

To add a sign-off to the last commit retroactively:

git commit --amend -s --no-edit

To add sign-off to the last 3 commits:

git rebase --signoff HEAD~3

Security impact

No

Compatibility impact

No

• Source compatibility: existing code continue to compile
• SAST compatibility
  • forward compatibility: new libraries can be used by old compiler
  • backward comopatibility: old libraries can be used by new compiler
• Standard Library compatibility:
  • forward compatibility: new code can work with old stdlib
  • backward compatibility: old code work with the new stdlib
• Runtime Library compatibility:
  • forward compatibility: new code can work with old runtime
  • backward compatibility: old code work with the new runtime
• Build tool
  • Build spec compatibility: old projects continue to build
  • Joy package compatibility: old .joy can be consumed by new build tool

[necto]

That's a good idea to include the source line

"source": { "file": "lib/List.jo", "line": 162 },

This made me thing that "endLine" could probably also be useful. I imagine, an LLM would want to read the actual definition of some symbol, in that case it could issue a "read" command with the begin and end lines to read only a slice of a file.

[necto]

Does --query work only with --format json?
If so, then you essentially give only two options:

• "--query --format json"
• "--format html" and no --query (or is it permitted and ignored?)

Thus you could simplify the API to accept either --query and then output json or no --query and output HTML and require --out, and you don't need the --format parameter.

Additionally, I am not sure --query is needed. If you output all available symbols in JSON format to stdout, and especially in JSONL (i.e., just a stream of json objects, rather than a JSON array), then the consumer can use a plethora of standard tools to filter/query/transform the json on-the-fly. For example they could use jq to select the symbol of interest and discard the rest.

Having your own query means you have to implement and explain the query language, and if it is even slightly non-trivial it means headache for the users to learn it and use correctly.

[necto]

This is a bit of a ranaway sentence. If I understood it correctly, here is how I would make it easier to read:

Suggested change

	is just an ordinary symbol query. A query may omit positional source files; in
	that case it searches the loaded SAST libraries, including stdlib unless
	`--no-stdlib` is set and any `--lib` paths.
	is just an ordinary symbol query. A query may omit positional source files; in
	that case it searches any `--lib` paths and the loaded SAST libraries (including stdlib unless
	`--no-stdlib` is set).

But correct me if I misunderstood it.

[necto]

what is ~ symbol? It looks like some kind of special symbol. Is it actually a Jo operator ~? If so, it is a confusing example for the documentation.

[liufengyun]

Yes, it's a little confusing, I'll remove it. ~ is defined in standard library:

val x = 3 ~ "a"
val y = "a" ~ 4 ~ 6

val a ~ b = x
val c ~ h ~ w = y

[necto]

Do you have a use case in mind for the file:<path> selector? Can't the same be achieved by providing just the <path> as the compilation target?
I would err on the simplicity side adding things only if they have a use case.

[necto]

does jo.py refer to some jo symbol , or to a python file?

[liufengyun]

jo.py is a symbolic selector, file:src/API.jo is a file selector.

[necto]

I think using a symbolic selector that resembles a file with a classic extension together with a file selector is confusing, especially for people that are not aware that jo.py exists in the jo library.

[liufengyun]

Does --query work only with --format json? If so, then you essentially give only two options:

• "--query --format json"
• "--format html" and no --query (or is it permitted and ignored?)

Thus you could simplify the API to accept either --query and then output json or no --query and output HTML and require --out, and you don't need the --format parameter.

This is a good point, given that --doc and --query do not share code, we can decouple them.

The only case where --query is optional when source files are provided can be supported with a special selector such as all or *.

Currently, it kind of works as you mentioned: if --query is given, --format json is implied.

Additionally, I am not sure --query is needed. If you output all available symbols in JSON format to stdout, and especially in JSONL (i.e., just a stream of json objects, rather than a JSON array), then the consumer can use a plethora of standard tools to filter/query/transform the json on-the-fly. For example they could use jq to select the symbol of interest and discard the rest.

Having your own query means you have to implement and explain the query language, and if it is even slightly non-trivial it means headache for the users to learn it and use correctly.

The reason why --query is needed is that the name universe is huge: it by default contains all names in standard library, but may also contain names from 3rd party libraries. Without --query, it is just inconvenient for simple use cases --- users and LLMs should not need to resort to another tool for common use cases.

In addition, without --query, it's not clear how to support a simple use case like the following:

jo compie --query jo.List.map

[necto]

Currently, it kind of works as you mentioned: if --query is given, --format json is implied.

Sure, but it invites confusion. As a language designer you should know that a better designed language makes something inexpressible rather than simply forbidden at compile or at runtime. CLI is a form of a language, so making unwanted things inexpressible also plays in its favor.

The reason why --query is needed is that the name universe is huge: it by default contains all names in standard library, but may also contain names from 3rd party libraries. Without --query, it is just inconvenient for simple use cases --- users and LLMs should not need to resort to another tool for common use cases.

Indeed it is large by default, but there is always --no-stdlib. If the most common case turns out to be looking for non-stdlib symbols, we could make stdlib opt-in rather than opt-out.

In addition, without --query, it's not clear how to support a simple use case like the following:

jo compie --query jo.List.map

Here is how I would do that in shell (note that I cannot avoid --query in the current implementation because jo errors):

jo compile --doc --query jo.List | jq '.. | objects | select(.name == "jo.List.map")'

Or, what is more likely, I (or LLM) would invoke jo compile --doc in a python script, and use its json parsing to parse and select the objects I'm interested in.

As an aside, --query jo.List gives me documentation of jo.List (class, section, pattern, and def), --query jo.List.map does not find anything:

$ ./bin/jo compile --doc --format json --query jo.List.map
[Error] No documentation entries match symbol selector jo.List.map
1 error(s), 0 warning(s)

BTW should jo/docs/usage/commands/doc.md also mention this new mode?

[liufengyun]

Currently, it kind of works as you mentioned: if --query is given, --format json is implied.

Sure, but it invites confusion. As a language designer you should know that a better designed language makes something inexpressible rather than simply forbidden at compile or at runtime. CLI is a form of a language, so making unwanted things inexpressible also plays in its favor.

Yes, we can go even further here to decouple the --query from --doc.

The reason why --query is needed is that the name universe is huge: it by default contains all names in standard library, but may also contain names from 3rd party libraries. Without --query, it is just inconvenient for simple use cases --- users and LLMs should not need to resort to another tool for common use cases.

Indeed it is large by default, but there is always --no-stdlib. If the most common case turns out to be looking for non-stdlib symbols, we could make stdlib opt-in rather than opt-out.

I think --no-stdlib is only useful for compiling the standard library.

For normal compilation, standard library is indispensable.

In addition, without --query, it's not clear how to support a simple use case like the following:

jo compie --query jo.List.map

Here is how I would do that in shell (note that I cannot avoid --query in the current implementation because jo errors):

jo compile --doc --query jo.List | jq '.. | objects | select(.name == "jo.List.map")'

This is painful, I never remember the syntax of jq. Think the whole-program (source, stdlibs, 3rd-party libs) as a database of symbols, targeted query is a natural and inherent need which the interface design need to address, IMHO. Otherwise, it's an awkward interface to be used.

Or, what is more likely, I (or LLM) would invoke jo compile --doc in a python script, and use its json parsing to parse and select the objects I'm interested in.

As an aside, --query jo.List gives me documentation of jo.List (class, section, pattern, and def), --query jo.List.map does not find anything:

$ ./bin/jo compile --doc --format json --query jo.List.map
[Error] No documentation entries match symbol selector jo.List.map
1 error(s), 0 warning(s)

This is a bug, I'll investigate.

BTW should jo/docs/usage/commands/doc.md also mention this new mode?

jo doc is the tool interface, not compiler interface. The compiler interface begins with jo compile.

[necto]

My use case: retriever all the symbols defined in a file and their doc string (#84 ). Does this implementation support it?
I couldn't find the syntax to do that (--query expects an argument and I don't what I can pass to it to get everything from a file).

[necto]

I never remember the syntax of jq.

Luckily we have LLMs for that now.

Think the whole-program (source, stdlibs, 3rd-party libs) as a database of symbols, targeted query is a natural and inherent need which the interface design need to address, IMHO. Otherwise, it's an awkward interface to be used.

You can implement a separate jo filter or something command that would filter and select necessary entries from the full doc produced by jo doc --json.

I think adding a query language before anyone asks for it is premature, but I do not insist, as long as it does not get in the way.

[necto]

I think --no-stdlib is only useful for compiling the standard library.

For normal compilation, standard library is indispensable.

I am speaking only about documentation extraction. --no-stdlib would allow me not to worry about the huge list of definitions in stdlib when I am querying a single code file.

[liufengyun]

My use case: retriever all the symbols defined in a file and their doc string (#84 ). Does this implementation support it? I couldn't find the syntax to do that (--query expects an argument and I don't what I can pass to it to get everything from a file).

If the namespace of the file is know, it can be done by querying the namespace symbol. Otherwise, query with the file name file:xxx.jo, it will output symbols in the file.

[liufengyun]

I think --no-stdlib is only useful for compiling the standard library.
For normal compilation, standard library is indispensable.

I am speaking only about documentation extraction. --no-stdlib would allow me not to worry about the huge list of definitions in stdlib when I am querying a single code file.

--no-stdlib is only intended for compiling the standard library, it does not have other uses. It is impossible to compile other files without standard library.

[liufengyun]

Think the whole-program (source, stdlibs, 3rd-party libs) as a database of symbols, targeted query is a natural and inherent need which the interface design need to address, IMHO. Otherwise, it's an awkward interface to be used.

You can implement a separate jo filter or something command that would filter and select necessary entries from the full doc produced by jo doc --json.

I think adding a query language before anyone asks for it is premature, but I do not insist, as long as it does not get in the way.

I think "query language" is not an accurate characterization: it's just an input format. If a user or LLM wants to know the interface of String, it simply queries jo.String. The query is always there, it's inherent with all the use cases I am aware of.

I'm not sure how to tell whether something is "premature" or not. But as principled design we never take a requirement at its surface value and for information-retrival interface, narrowing/querying is essential IMHO.

[necto]

If the namespace of the file is know, it can be done by querying the namespace symbol. Otherwise, query with the file name file:xxx.jo, it will output symbols in the file.

Here is how I tried to do it:

 ~/proj/jo $ ./bin/jo compile --doc --query "file:Byte.jo"
[Error] No documentation entries match file selector `file:Byte.jo`

1 error(s), 0 warning(s)
 ~/proj/jo $ ./bin/jo compile --doc --query "file:Byte.jo" lib/Byte.jo 
---------- Error at lib/Byte.jo:10:7 ---------------
| class Byte
|       ^^^^
|       Redefinition of Byte
|
| The name `Byte` is already defined at lib/Byte.jo:10:7
| class Byte
|       ^^^^

1 error(s), 0 warning(s)

[necto]

--no-stdlib is only intended for compiling the standard library, it does not have other uses. It is impossible to compile other files without standard library.

Then you can add an equivalent for docs. e.g. --no-stdlib-doc

[liufengyun]

--no-stdlib is only intended for compiling the standard library, it does not have other uses. It is impossible to compile other files without standard library.

Then you can add an equivalent for docs. e.g. --no-stdlib-doc

That won't work, because without standard library, it is impossible generate docs.