Some ideas for package manager-independent dep resolution

[pombredanne]

After a discussion with the original author and designer of the Eclipse P2 dep resolver, both p2 and libsolv could be candidates for universal dep resolution.

[sschuberth]

FYI, here's what package managers / resolvers we're looking into:

• npm / yarn for Node.js dependencies
• maven-resolver / aether for Maven dependencies
• coursier for Scala / SBT dependencies

(I'll edit this post to update.)

[roscopecoltran]

FYI,

Found this one:
https://github.com/jeremylong/DependencyCheck (More details here)

[pombredanne]

@sschuberth FWIW the latest scancode develop includes https://github.com/sassoftware/pymaven (with some patches and some contributions that will eventually end upstream) which provides a decent pure python maven client. This eventually can resolve (as in fetch) deps even if crudely.
One possibility here would be to feed additional inputs besides POM: gradle, sbt, leningen, etc

On the dep resolution side, I have been looking at https://github.com/ContinuumIO/pycosat and Conda by @ilanschnell which provides a solid dependencies resolver https://github.com/conda/libconda/blob/master/libconda/resolve.py ... My idea would be to go from:

1. collect direct dependencies with scancode package manifests
2. here, provide "clients" that know how to fetch manifests and artifacts from various repos (e.g. maven, pypi, etc)
3. here, based on a set packages and clients, resolve (and eventually fetch) the deps

The point is that I want a solution that would be as much as possible neutral with regards to package formats and provides a good enough resolution (albeit not guaranteed to be exactly the same as the regular resolver for a given package manager)

[pombredanne]

@roscopecoltran Richard: thanks for the link. I have been tracking DependencyCheck for quite a while and borrowing some (or all) of its code is an option too. I wish it were written in Python though ;)
The think I like about it is that it is based on a static analysis of deps (and does not require all the package managers to be installed)

[sschuberth]

latest scancode develop includes https://github.com/sassoftware/pymaven

Hmm, looks like that code hasn't been touched in years, and quote "fetch artifacts and artifact dependencies from maven2 repositories". I haven't seen Maven 2 projects since years...

FYI, our currently approach is to do 1. and 2. in a single step, and later do 3., all using whatever is the native package manager for the respective language. More details later 😉

[pombredanne]

@sschuberth you wrote:

Hmm, looks like that code hasn't been touched in years,

Not entirely correct and not correct of late: This is what's cooking for instance: https://github.com/sassoftware/pymaven/pull/8

and quote "fetch artifacts and artifact dependencies from maven2 repositories". I haven't seen Maven 2 projects since years...

Fair enough! but this is in reference to the maven2 repository layout for Maven POMs 4.0.0 as opposed to the maven1 repository layout (which was used with older POM versions 3.0 and earlier) .... this is not about Maven (the software) v2.x.... pymaven handles the current Maven 3.x things alright (and actually only support current Maven POMs)

Now in all cases having both static-analysis-based dependencies resolution and dynamic-package-manager-based resolution are not in conflict. In fact the best approach is likely to support both with the dynamic being the preferred, first try and the static a fall-back solution if the first is not working (for instance if the package manager too is not installed or if only a static resolution is requested)

[sschuberth]

Back Duck started with hub-detect a somewhat similar project, though tailored towards sending the results to their hub platform, whereas we're targeting at store results in ABCD format.

[pombredanne]

@sschuberth neat! is this standalone? or tied to having some stuffs fetched from their servers?
and data formats can be converted alright

[sschuberth]

./gradlew bootRun yields ERROR--- Your Hub configuration is not valid, so it does not seem to be able to run the detection stand-alone before uploading anything to their hub.

[pombredanne]

Some ideas of tools that could help with dependency resolution:

• https://github.com/sarugaku/resolvelib now used in pip
• https://github.com/ContinuumIO/pycosat used on Conda
• https://github.com/sdispater/mixology (now at https://github.com/python-poetry/poetry/tree/master/poetry/mixology) from the author of Poetry

Also related, these are universal package manager tools:

• a universal package manager idea: https://gist.github.com/pombredanne/d3585617882f91d9316be5ce5eddf190
• and UPT that can convert mostly all package formats https://framagit.org/upt
• https://github.com/sigoden/upt

And of course for dynamic resolution of dependencies, this:

• https://github.com/oss-review-toolkit/ort (which if it could be run without Java may be great here)