Skip to content

ci: containerize CI pipeline with pre-built Docker image #2529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
BrendanWalsh wants to merge 1 commit into
base: master
Choose a base branch
from
Open

ci: containerize CI pipeline with pre-built Docker image #2529

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
target
.idea
.git
tools/docker/Dockerfile.dev
pipeline.yaml
.dockerignore
# Allowlist: only send files referenced by COPY instructions in Dockerfiles.
# CI Dockerfile: environment.yml, project/, build.sbt, sonatype.sbt
# Demo Dockerfile: tools/docker/demo/init_notebook.py, docs/
*
!environment.yml
!project/
!build.sbt
!sonatype.sbt
!tools/docker/demo/init_notebook.py
!docs/
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,3 +95,4 @@ condaenv.*.requirements.txt
# Dev-loop artifacts
.dev-loop/
.dev-loop-artifacts/
pipeline.yaml.bak
7 changes: 6 additions & 1 deletion build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,12 @@ getDatasetsTask := {
val f = new File(d, datasetName)
if (!d.exists()) d.mkdirs()
if (!f.exists()) {
FileUtils.copyURLToFile(datasetUrl, f)
val cached = new File(sys.env.getOrElse("DATASET_CACHE", "/opt/datasets"), datasetName)
if (cached.exists()) {
java.nio.file.Files.copy(cached.toPath, f.toPath)
} else {
FileUtils.copyURLToFile(datasetUrl, f)
}
UnzipUtils.unzip(f, d)
}
}
Expand Down
15 changes: 13 additions & 2 deletions core/src/test/scala/com/microsoft/azure/synapse/ml/core/test/base/TestBase.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import org.scalatest.time.{Seconds, Span}

import java.io.File
import java.nio.file.{Files, Path}
import scala.concurrent._
import scala.concurrent.blocking
import scala.reflect.ClassTag

trait SparkSessionManagement {
Expand Down Expand Up @@ -147,7 +147,18 @@ object TestBase extends SparkSessionManagement {

}

abstract class TestBase extends AnyFunSuite with BeforeAndAfterEachTestData with BeforeAndAfterAll {
abstract class TestBase extends AnyFunSuite with BeforeAndAfterEachTestData with BeforeAndAfterAll with TimeLimits {

// Global per-test timeout (10 minutes). Override in subclass if needed.
val testTimeoutInSeconds: Int = 10 * 60

override def test(testName: String, testTags: Tag*)(testFun: => Any)(implicit pos: Position): Unit = {
super.test(testName, testTags: _*) {
failAfter(Span(testTimeoutInSeconds, Seconds)) {
testFun
}
}
Comment on lines +152 to +160
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description mentions a 30-minute hard timeout for test suites, but the new global per-test timeout here is set to 10 minutes. If 30 minutes is the intended limit, update testTimeoutInSeconds accordingly (or make it configurable via an env var/system property); otherwise the PR description should be updated to avoid confusion.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Collaborator Author

@BrendanWalsh BrendanWalsh Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR description updated. The 10-minute per-test timeout is correct — the "30 minutes" in the description referred to the sbt setup timeout, not the per-test limit. Subclasses can override testTimeoutInSeconds for longer-running tests (e.g., DatabricksTestHelper).

}

lazy val sparkProvider: SparkSessionManagement = TestBase

Expand Down
Loading
Loading