display Experiment URL to stdout automatically through Experiment object instead of specifying it manually within script

[philhetzel]

I'm trying to get similar DX to the py/TS SDKs where the URL is sent to stdout automatically when Eval is initiated

Next steps after this:

• Conform around datasets being input/expected/metadata. Right now Evaluation.java is looking for "question" instead of input.
• Fix trivia example to declare expected instead of expectedAnswer in dataset
• Abstract Eval summary and reporting into Experiments or Evaluations object instead of example scripts

[Ankur Goyal (ankrgyl)]

In python we don't display the URL from within the SDK. Meaning if you just do

import braintrust

experiment = braintrust.init_experiment(...)

it won't print the URL. If im reading this correctly, we would do that here?

[philhetzel]

I may be thinking about this the wrong way - Here's an example in python where I'm running an Eval. I think the behavior of sending the results summary / experiment URL to stdout is nice DX. Stdout is used both on experiment start and experiment end. I was trying to replicate some of that behavior.

Is the issue that the logic to write to stdout in this PR was added to the Experiment class rather than the Evaluation class?

[Ankur Goyal (ankrgyl)]

That is the behavior when you run braintrust eval but it's implemented in the code that actually kicks off the experiment, not in the experiment's constructor. So if you run braintrust.init_experiment for example, it won't do that. The main thing I want to make sure of is that if you're just writing code to run an experiment, you are not forced to have all kinds of stuff printed to your screen.

[Andrew Kent (realark)]

Backlogging this issue. Will support this when we add async execution for evals. See BRA-3582