Add doc discovery/fetch for AI tooling

[MUFFANUJ]

Follow-up to #115. We should add a way for AI tooling to discover and fetch plugin docs.

[MUFFANUJ]

Maybe having a max limit for characters to fetch would make sense in here to make sure it is not consuming more tokens? ideally the user should have the option to define this limit? @krassowski

[krassowski]

Yes, definitely. There is some prior art here worth reviewing but at a basic level we could have arguments like:

interface IArgs {
  package: string;
  query: string;
  detailLevel: number;
}

and response at a lower detail level could contain hint at how much many more matching references there are. Or we could use some pagination to avoid feeding to much context at once but allowing agent to fetch more if it needs to.

[krassowski]

Maybe via LSP?

[krassowski]

I opened jupyter-lsp/jupyterlab-lsp#1184 which discusses how we could get relevant information for AI from LSP servers.