pkg: Add CLI interface for enabling and disabling package management during single command executions

[shonfeder]

Motivation

Currently the only way to enable package management requires a change on disk. I.e., either

• creating a lock directory
• writing configurations in the dune-workspace

However, there are cases where a user may want to enable or disable package management temporarily, from the CLI. E.g.,

• When writing packaging recipes, a user may want to ensure that package management is (or is not) used during a command invocation (for instance, see the discussion at Add doc about making brew package with dune pkg #12744 (comment)).
• When doing an exploratory build of a project that is usually managed by opam, a user may want to have dune manage its dependencies instead for exploratory purposes or for some CI process
• When working on a project that generally uses dune package by default, a user may want to use a different package system, either for exploration, in CI, or some other reason (see, e.g., Add dune pkg binary release workflow quantifyearth/container-image#7 (comment))

In such cases, it would be convenient to be able to run a dune command with package management enabled (or disabled).

Interface ideas

A possible interfaces to consider (I am not attached to any particular interface):

Via env var

DUNE_PKG=1 dune build # enabled
DUNE_PKG=0 dune build # disabled
DUNE_PKG=  dune build # will run with the default behavior given the file systems

This should apply to any other relevant subcommands.

Via flag

dune build --pkg=enabled  # enabled
dune build --pkg=disabled # disabled
dune build                # will run with the default behavior given the file system

This should apply to any other relevant subcommands.

[ElectreAAS]

An already existing option is what I did on my PR adding a dune pkg CI run to irmin:
Add a file dune-workspace.for-ci containing (pkg enabled), and, when relevant, pass --workspace ./dune-workspace.for-ci.

Obviously that still require some disk operations and isn't a complete fix

[shonfeder]

@ElectreAAS I agree this is a workaround and not a fix :)

[shonfeder]

Following up on some discussion offline and in other places...

As @ElectreAAS and I discussed, the decision about configuration precedence should be based on principles, taking into account both existing precedent in Dune, the OCaml ecosystem, and the wider world. This may require factoring out an antecedent issue to specify and formalized Dune's configuration precedence (which would at least make its way into the docs, but could also have implications for existing functionality that violates the rule).

A couple useful resources from the wider world to consider:

• https://docs.spring.io/spring-boot/reference/features/external-config.html
• https://github.com/spf13/viper?tab=readme-ov-file#putting-values-in-viper

We should also look at what opam does, among other things.

[shonfeder]

In discussion with @Alizter, @ElectreAAS, and @art-w today, they explained that at some point they looked at the configuration code needed to add this and realized that the configuration precedence precedence was enforced at the code level already (and, IIUC, the enforced order is sensible).

That obviates the need for what was discussed in #12819 (comment) and in some meetings.

We also agreed that the fact that this precedence is enforced in the code (if used correctly) should be documented.

Moreover, as I write this now, I realize users also should be able to find documentation of the standard configuration order.

I will open issues for these.