Skip to content

chore(operations): Compare test harness output with nightly/latest #2204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
binarylogic merged 1 commit into from
Aug 1, 2020
+1 −1
Merged

chore(operations): Compare test harness output with nightly/latest #2204

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test-harness.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ jobs:
with:
args: |
bash -o pipefail -xc "cd /vector-test-harness \
&& bin/compare -s vector -c default $(echo '${{ github.event.comment.body }}' | head -n 1 | sed 's|^/test ||') | tee \"$GITHUB_WORKSPACE/output\""
&& bin/compare -s vector -c default -v nightly/latest -v ${{ steps.cloned-repo-info.outputs.test_harness_vector_version }} $(echo '${{ github.event.comment.body }}' | head -n 1 | sed 's|^/test ||') | tee \"$GITHUB_WORKSPACE/output\""

@binarylogic binarylogic Apr 2, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MOZGIII I'm curious how you think we should go about carrying over the -c flag to here?

@MOZGIII MOZGIII Apr 2, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we make -c flag settable via env vars, we'll be able to assign it by default to default just like at the test command above.

I'm also thinking about adding a command to the test harness repo that would be dedicated to handling the Github Action invocations. This would better decouple the test harness logic from the Github Action logic - I believe it'd be better if the latter is scoped to just triggering test harness, so that we offload unrelated logic from it as much as possible.

@MOZGIII MOZGIII Apr 2, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we make -c flag settable via env vars, we'll be able to assign it by default to default just like at the test command above.

Btw, the same applies to -s.

env:
GENERATE_SSH_KEY: true
VECTOR_TEST_RESULTS_S3_BUCKET_NAME: test-results.vector.dev
Expand Down