Print failed test results at the end for the spec test suite

[stevenfontanella]

• Print all failed test stdouts/stderrs at the end so that the output isn't lost among the successful tests
• Make shared.options.abort_on_first_failure work with the spec test suite and add aliases for --fail-fast and --no-fail-fast
• Re-raise in run_one_spec_test since fail_with_error may no longer throw an exception when abort_on_first_failure is false (the remaining code shouldn't execute because then actual isn't defined).

With --no-fail-fast:

With --fail-fast (the default):

[sbc100]

Awesome!

[sbc100]

Single quotes here.

Also, I think we can just remove the old name here... the number folks who run this script is small enough we don't need to worry about back compat.

Also, you can use argparse.BooleanOptionalAction here maybe?

[stevenfontanella]

Is there any harm in keeping the old name? It seems better to avoid breaking users if we can, and it doesn't look like a maintenance burden in this case.

[sbc100]

Another nice addition (ideas for followups).

1. Print the number of failing tests.
2. Print the test number and progress with each result. e.g. [N/M] test_name..

Even better, only print this single-line status (i.e. no stdout from the test) unless --verbose or the test fails.

[stevenfontanella]

Another nice addition (ideas for followups).

1. Print the number of failing tests.
2. Print the test number and progress with each result. e.g. [N/M] test_name..

Even better, only print this single-line status (i.e. no stdout from the test) unless --verbose or the test fails.

1 is already there actually, but the way that it counts errors is a little strange (this is with fail-fast enabled): . I think it's because some types of errors get caught (while already increasing the error count), and then the error count is increased again in the catch, e.g. here.

Agreed on simplifying the output. For now we can actually redirect stdout to /dev/null and we'll only see failed outputs: