Skip to content

Handle private repositories dependencies #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rvignolo wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Handle private repositories dependencies #31

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2ccf279
feat: :sparkles: add private repositories support (#1)
rvignolo May 10, 2024
edc6ed2
feat: ✨ add support for multiple private repos (#2)
rvignolo May 16, 2024
fb75097
feat: :construction: try deploy keys as unique var (#3)
rvignolo May 17, 2024
eba5357
Merge branch 'multiversx:main' into main
rvignolo May 17, 2024
f59e983
fix: :art: format desc (#4)
rvignolo May 17, 2024
94294c5
fix: :bug: no need to install rust if no private dependencies (#5)
rvignolo May 20, 2024
03660b6
chore: :fire: remove wabt (#6)
rvignolo Jan 7, 2025
2f512bb
fix: :arrow_up: upgrade upload-artifact version (#7)
rvignolo Jan 16, 2025
b9d7fc6
fix: :bug: don't fail if .cargo exists, also append (#8)
rvignolo Mar 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 57 additions & 32 deletions .github/workflows/contracts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,59 +4,57 @@ on:
workflow_call:
inputs:
rust-toolchain:
description: 'Rust toolchain to use'
default: 'nightly'
description: "Rust toolchain to use"
default: "nightly"
required: false
type: string
pip-mxpy-args:
description: 'pip mxpy install arguments'
default: 'multiversx-sdk-cli==v9.5.2'
description: "pip mxpy install arguments"
default: "multiversx-sdk-cli==v9.5.2"
required: false
type: string
sc-meta-version:
description: 'multiversx-sc-meta version'
default: ''
description: "multiversx-sc-meta version"
default: ""
required: false
type: string
mx-scenario-go-version:
description: 'sc-scenario-go version'
default: ''
description: "sc-scenario-go version"
default: ""
required: false
type: string
path-to-sc-meta:
description: 'multiversx-sc-meta from local'
default: ''
description: "multiversx-sc-meta from local"
default: ""
required: false
type: string
clippy-args:
description: 'cargo clippy arguments'
default: '--all-targets --all-features'
description: "cargo clippy arguments"
default: "--all-targets --all-features"
required: false
type: string
enable-contracts-size-report:
description: 'Enable contracts size report'
description: "Enable contracts size report"
default: true
required: false
type: boolean
coverage-args:
description: 'sc-meta test-coverage arguments'
default: '--output ./coverage.md'
description: "sc-meta test-coverage arguments"
default: "--output ./coverage.md"
required: false
type: string
binaryen-version:
description: 'binaryen (wasm-opt) version to use'
default: 'version_112'
required: false
type: string
wabt-version:
description: 'wabt version to use'
default: '1.0.27-1'
description: "binaryen (wasm-opt) version to use"
default: "version_112"
required: false
type: string
secrets:
token:
description: 'Github token'
description: "Github token"
required: true
deploy-keys:
description: "Deploy SSH private keys"
required: false

jobs:
wasm_test:
Expand All @@ -65,6 +63,14 @@ jobs:
steps:
- uses: actions/checkout@v3

- name: Setup Credentials
uses: webfactory/ssh-agent@v0.9.0
env:
SUPER_SECRET: ${{ secrets.deploy-keys }}
if: ${{ env.SUPER_SECRET != '' }}
with:
ssh-private-key: ${{ secrets.deploy-keys }}

Comment on lines +66 to +73
Copy link
Contributor

@andreibancioiu andreibancioiu May 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd like to avoid using an additional third party step, if possible (and only have it as a last resort).

- name: Install rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
Expand All @@ -79,8 +85,6 @@ jobs:
tar -xf binaryen.tar.gz
cp binaryen-${{ inputs.binaryen-version }}/bin/wasm-opt $HOME/.local/bin

sudo apt install -y wabt=${{ inputs.wabt-version }}

cargo install wasm-opt
cargo install twiggy

Expand Down Expand Up @@ -110,24 +114,27 @@ jobs:
- name: Build the wasm contracts
env:
RUSTFLAGS: ""
CARGO_NET_GIT_FETCH_WITH_CLI: true
Copy link
Contributor

@andreibancioiu andreibancioiu May 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reviewers: https://doc.rust-lang.org/nightly/cargo/reference/config.html#netgit-fetch-with-cli

run: sc-meta all build --no-imports --target-dir $(pwd)/target --path .

- name: Run the wasm tests
env:
RUSTFLAGS: ""
CARGO_NET_GIT_FETCH_WITH_CLI: true
run: cargo test --features multiversx-sc-scenario/run-go-tests

- name: Generate the contract report
if: ${{ inputs.enable-contracts-size-report }}
env:
RUSTFLAGS: ""
CARGO_NET_GIT_FETCH_WITH_CLI: true
run: |
sc-meta all build-dbg --twiggy-paths --target-dir $(pwd)/target --path .
mxpy contract report --skip-build --skip-twiggy --output-format json --output-file report.json

- name: Upload the report json
if: ${{ inputs.enable-contracts-size-report }}
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: report
path: report.json
Expand Down Expand Up @@ -166,7 +173,7 @@ jobs:
head: ${{ github.event.pull_request.head.sha }}

- name: Upload the report markdown
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
if: ${{ github.event_name == 'pull_request' && inputs.enable-contracts-size-report }}
with:
name: report-markdown
Expand All @@ -178,8 +185,8 @@ jobs:
if: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && inputs.enable-contracts-size-report }}
with:
issue-number: ${{ github.event.pull_request.number }}
comment-author: 'github-actions[bot]'
body-includes: 'Contract comparison'
comment-author: "github-actions[bot]"
body-includes: "Contract comparison"

- name: Create or update the report comment
uses: peter-evans/create-or-update-comment@v2
Expand All @@ -196,6 +203,14 @@ jobs:
steps:
- uses: actions/checkout@v3

- name: Setup Credentials
Copy link
Contributor

@andreibancioiu andreibancioiu May 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For contracts.yml (not for reproducible-build.yml), the solution you've previously provided (without altering contracts.yml) seems sufficient (an additional step in the workflow file of the repository itself):

multiversx/mx-sdk-rust-contract-builder#59 (comment)

Maybe we can promote and document that solution, and have contracts.yml untouched (at least, from the perspective of setting up credentials)?

uses: webfactory/ssh-agent@v0.9.0
env:
SUPER_SECRET: ${{ secrets.deploy-keys }}
if: ${{ env.SUPER_SECRET != '' }}
with:
ssh-private-key: ${{ secrets.deploy-keys }}

- name: Install rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
Expand Down Expand Up @@ -224,11 +239,12 @@ jobs:
- name: Run tests and generate report
env:
RUSTFLAGS: ""
CARGO_NET_GIT_FETCH_WITH_CLI: true
run: |
sc-meta test-coverage ${{ inputs.coverage-args }}

- name: Upload the report
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: coverage
path: coverage.md
Expand Down Expand Up @@ -256,15 +272,24 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3

- name: Setup Credentials
uses: webfactory/ssh-agent@v0.9.0
env:
SUPER_SECRET: ${{ secrets.deploy-keys }}
if: ${{ env.SUPER_SECRET != '' }}
with:
ssh-private-key: ${{ secrets.deploy-keys }}

- uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: ${{ inputs.rust-toolchain }}

- name: Run the rust tests
env:
RUSTFLAGS: ""
run:
cargo test
CARGO_NET_GIT_FETCH_WITH_CLI: true
run: cargo test

clippy_check:
name: Clippy linter check
Expand Down
56 changes: 48 additions & 8 deletions .github/workflows/reproducible-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ on:
description: Image multiversx/sdk-rust-contract-builder
project_path:
type: string
default: '.'
default: "."
required: false
description: A specific project path
contract_name:
Expand All @@ -27,7 +27,11 @@ on:
description: Skip preliminary checks. Never set this in production!
package_whole_project_src:
type: boolean
description: Include all project files in the packaged source (*.source.json)
description: Include all project files in the packaged source (*.source.json)
secrets:
deploy-keys:
description: "Deploy SSH private keys"
required: false

jobs:
build:
Expand Down Expand Up @@ -86,6 +90,42 @@ jobs:
fetch-depth: 0
repository: ${{ env.GITHUB_REPOSITORY }}

- name: Setup Credentials
Copy link
Contributor

@andreibancioiu andreibancioiu May 27, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We understand the need of a closed-source contract to depend on private dependencies, e.g. for development phase 👍 - of course, this is not wise for contracts deployed on mainnet - such contracts cannot pass the verified and reproducible build check.

Though, how would you feel about the options below?

(A) have your private dependencies brought as git submodules in your main private repository, then alter the Cargo.toml to point to them (of course, making your project open-source would reveal the source code of the dependencies, as well).

(B) in the Docker images for reproducible builds, we can enable ssh-agent, and run ssh-add (if any key is provided in the Docker environment) - I did not yet test this approach, though.

(C) allow reproducible-build.yml to be parametrized with the fully qualified-name of a Docker image (instead of image_tag only) - this way, you can use a custom Docker image to perform the build of a closed-source contract.

uses: webfactory/ssh-agent@v0.9.0
env:
SUPER_SECRET: ${{ secrets.deploy-keys }}
if: ${{ env.SUPER_SECRET != '' }}
with:
ssh-private-key: ${{ secrets.deploy-keys }}

- name: Install rust
uses: actions-rust-lang/setup-rust-toolchain@v1
env:
SUPER_SECRET: ${{ secrets.deploy-keys }}
if: ${{ env.SUPER_SECRET != '' }}
with:
toolchain: ${{ inputs.rust-toolchain }}
target: wasm32-unknown-unknown

- name: Vendored dependencies (if private dependencies are used)
Copy link
Contributor

@andreibancioiu andreibancioiu May 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thus, private dependencies are brought locally, before starting the build, correct (sorry if I misunderstood)?

If so, the usual contract verification flow (which relies on the Docker container to perform the whole build) would not be able to cover this case (however, for verified contracts, private dependencies would not be acceptable, anyway).

env:
CARGO_NET_GIT_FETCH_WITH_CLI: true
SUPER_SECRET: ${{ secrets.deploy-keys }}
if: ${{ env.SUPER_SECRET != '' }}
run: |
mkdir -p .cargo
shopt -s globstar
cargos=($(echo **/Cargo.toml))
cargos_s=$(printf -- '-s %s ' "${cargos[@]}")
cargo vendor ${cargos_s} >> .cargo/config.toml

# Prevent mx-sdk-rust-contract-builder from bulding vendored dependencies
rm -f vendor/**/multiversx.json

for checksum_file in **/.cargo-checksum.json; do
jq 'del(.files."multiversx.json")' "$checksum_file" > "$checksum_file.tmp" && mv "$checksum_file.tmp" "$checksum_file"
done

- name: Preliminary checks
if: ${{ inputs.skip_preliminary_checks == false }}
run: |
Expand All @@ -110,7 +150,7 @@ jobs:
else:
logging.error(f"wasm/Cargo.lock file not found: {cargo_lock}")
missing_cargo_lock = True

if missing_cargo_lock:
sys.exit(f"ERROR: One or more 'wasm/Cargo.lock' files are missing. They are essential for reproducible builds.")
EOF
Expand All @@ -120,18 +160,18 @@ jobs:
- name: Download build script
run: |
wget https://raw.githubusercontent.com/multiversx/mx-sdk-rust-contract-builder/${{ inputs.image_tag }}/build_with_docker.py

- name: Build contracts
run: |
flag_package_whole_project_src=""
if ${{ inputs.package_whole_project_src }}; then
flag_package_whole_project_src="--package-whole-project-src"
fi

python3 ./build_with_docker.py --no-docker-tty --image=multiversx/sdk-rust-contract-builder:${{ inputs.image_tag }} --project=${{ inputs.project_path }} --contract=${{ inputs.contract_name }} --output=/home/runner/work/output-from-docker ${flag_package_whole_project_src}

- name: Save artifacts
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: build-output
path: |
Expand Down Expand Up @@ -167,7 +207,7 @@ jobs:

try:
logging.info(f"Fetching existing release notes, if any, for repository = {repository}, tag = {ref_name} ...")

request = urllib.request.urlopen(f"https://api.github.com/repos/{repository}/releases/tags/{ref_name}")
data = json.loads(request.read())
existing_notes = data["body"]
Expand Down