Ziggy Build failed

[jcsmithhere]

Hi Ziggy developers. I'm trying to build Ziggy and quickly ran into a problem.

Note that I am not a Java developer and don't know anything about Gradle. First thing I needed to do was install JDK. Looks like OpenJDK is the standard on Linux, so I installed that with:

sudo apt install default-jdk

Then I set the three environment variables JAVA_HOME, CC and CXX. This is on Ubuntu 20.04.6. Then I tried to build and got the below error. Not being a Java developer I figure I better just send the error to you. It looks like there is an error in interpreting the Java code.

(base) bohr@stan:~/dev_TOLIMAN/ziggy$ ./gradlew
Downloading https://services.gradle.org/distributions/gradle-8.12-bin.zip
.............10%.............20%.............30%.............40%.............50%.............60%.............70%.............80%.............90%.............100%

Welcome to Gradle 8.12!

Here are the highlights of this release:
 - Enhanced error and warning reporting with the Problems API
 - File-system watching support on Alpine Linux
 - Build and test Swift 6 libraries and apps

For more details see https://docs.gradle.org/8.12/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)
> Task :buildSrc:compileJava FAILED

[Incubating] Problems report is available at: file:///home/bohr/dev_toliman/ziggy/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':buildSrc:compileJava'.
> Compilation failed; see the compiler output below.
  /home/bohr/dev_toliman/ziggy/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java:67: error: : expected
              case LINUX_INTEL -> "mexa64";
                                          ^
  /home/bohr/dev_toliman/ziggy/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java:68: error: : expected
              case MAC_M1 -> "mexmaca64";
                                        ^
  /home/bohr/dev_toliman/ziggy/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java:69: error: : expected
              case MAC_INTEL -> "mexmaci64";
                                           ^
  /home/bohr/dev_toliman/ziggy/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java:80: error: : expected
              case LINUX_INTEL -> "glnxa64";
                                           ^
  /home/bohr/dev_toliman/ziggy/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java:81: error: : expected
              case MAC_M1 -> "maca64";
                                     ^
  /home/bohr/dev_toliman/ziggy/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java:82: error: : expected
              case MAC_INTEL -> "maci64";
                                        ^
  /home/bohr/dev_toliman/ziggy/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/SystemArchitecture.java:87: error: : expected
                  case LINUX_INTEL -> linuxIntelObject;
                                                      ^
  /home/bohr/dev_toliman/ziggy/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/SystemArchitecture.java:88: error: : expected
                  case MAC_INTEL -> macIntelObject;
                                                  ^
  /home/bohr/dev_toliman/ziggy/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/SystemArchitecture.java:89: error: : expected
                  case MAC_M1 -> macM1Object;
                                            ^/home/bohr/dev_toliman/ziggy/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java:66: error: illegal start of expression
          return switch (getArchitecture()) {
                 ^
  /home/bohr/dev_toliman/ziggy/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java:79: error: illegal start of expression
          return switch (getArchitecture()) {
                 ^
  /home/bohr/dev_toliman/ziggy/buildSrc/src/main/java/gov/nasa/ziggy/buildutil/SystemArchitecture.java:86: error: illegal start of expression
              return switch (architecture()) {
                     ^
  12 errors

* Try:
> Check your code and dependencies to fix the compilation error(s)
> Run with --scan to get full insights.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.12/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD FAILED in 22s
1 actionable task: 1 executed
(base) bohr@stan:~/dev_TOLIMAN/ziggy$ 

[wohler]

Hi Jeff, from the error, I'm guessing Ubuntu is installing an older Java version. You need a minimum of Java 17. On my Debian system, I've installed the openjdk-17-jdk package. What does "java --version" give you?

[jcsmithhere]

Thanks for the quick response, Bill! I installed the same JDK version as you with:

sudo apt install openjdk-17-jdk

That resolved the problem and Ziggy now builds successfully.

I recommend updating the build instructions page to summarize how to install JDK on Linux and Mac (and Windows?). I see your "Prerequisites and System Requirements" page does state JDK 17 or higher, but I think it's reasonable for someone new to JDK to first try:

sudo apt install default-jdk

which a quick Google search recommends.

Also, it took me some searches to find where to point JAVA_HOME. Your could provide some simple guidance on where to find that. I know this is all stuff a typical Java engineer would know, but I think these additions would help the uninitiated, like me.

[wohler]

Glad to hear it, Jeff! I'll look forward to hearing that you completed the sample pipeline as well.

Once we've implemented your suggestion in ZIGGY-551, Add quick start to manual, setting JAVA_HOME should be a lot more obvious. We have another issue, ZIGGY-530, Clarify unclear portions of the Ziggy user manual, that should help as well.

Describing the dozens of possible ways to install Java, especially on Windows which is unsupported, is beyond the scope of the manual. However, the manual update should make the Java version more obvious.

[wohler]

Adding, here is the quick start to run the sample pipeline:

$ sample-pipeline/build-env.sh
$ ziggy cluster init && ziggy cluster start
$ ziggy console 

Then follow the instructions in Starting a Pipeline.

I'll keep this open until you've successfully run the sample pipeline.

[jcsmithhere]

Got this error when trying to build sample pipeline:

(base) bohr@stan:~/.../sample_pipeline/sample-pipeline$ ./build-env.sh 
ERROR: Invalid requirement: '/src/main/python/ziggy'
Hint: It looks like a path. File '/src/main/python/ziggy' does not exist.

./build-env.sh: line 41: /bin/ziggy: No such file or directory

Looks like the build-env.sh file uses $ZIGGY_HOME but the user's guides uses $ZIGGY_ROOT. Changed the build script to use $ZIGGY_ROOT and I got further this time:

(base) bohr@stan:~/.../sample_pipeline/sample-pipeline$ ./build-env.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing ./build/src/main/python/sample_pipeline
  Preparing metadata (setup.py) ... done
Processing /home/bohr/dev_toliman/ziggy/src/main/python/ziggy
  Preparing metadata (setup.py) ... done
Collecting Pillow
  Downloading pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.6/6.6 MB 37.2 MB/s eta 0:00:00
Collecting numpy
  Downloading numpy-2.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.5/19.5 MB 76.3 MB/s eta 0:00:00
Collecting h5py
  Downloading h5py-3.14.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 88.8 MB/s eta 0:00:00
Installing collected packages: Pillow, numpy, sample-pipeline, h5py, ziggy
  DEPRECATION: sample-pipeline is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for sample-pipeline ... done
  DEPRECATION: ziggy is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for ziggy ... done
Successfully installed Pillow-11.3.0 h5py-3.14.0 numpy-2.0.2 sample-pipeline-0.10.0 ziggy-0.10.0


./build-env.sh: line 41: /home/bohr/dev_toliman/ziggy/bin/ziggy: No such file or directory

The last line is an error. Looks like there is no bin directory in the ziggy. It's in the build subdriectory. Changed last line of build-env.sh to:

$ZIGGY_ROOT/build/bin/ziggy generate-build-info --home $sample_home

Now after I re-ran, I got no error, but the last line above to run ziggy generated no output. Here's another run of it from the shell prompt:

(base) bohr@stan:~/.../sample_pipeline/sample-pipeline$ ~/dev_toliman/ziggy/build/bin/ziggy generate-build-info --home /home/bohr/data/TOLIMAN/learning_ziggy/users_guide/sample_pipeline/sample-pipeline/build
(base) bohr@stan:~/.../sample_pipeline/sample-pipeline$ 

No output. I would expect some output. It didn't error, so perhaps everything is now fine.

[jcsmithhere]

Maybe it didn't work correctly. My build directory does not look correct:

(base) bohr@stan:~/.../sample_pipeline/sample-pipeline$ ls
build  build-env.sh  build-env.sh~  clean-env.sh  data  etc  multi-data  src
(base) bohr@stan:~/.../sample_pipeline/sample-pipeline$ ls build
env  etc  pipeline-results  src
(base) bohr@stan:~/.../sample_pipeline/sample-pipeline$ 

Comparing this to the Build System for your Pipeline page, I see I get different directories. The User's Guide says I should get:

sample-pipeline$ ls build
bin    env    pipeline-results

So, is somethign wrong, or does the User's Guide need to be updated?

[jcsmithhere]

Ah, reading on in Build System for your Pipeline I see the command is not supposed to generate output. It generates a file in build/etc:

(base) bohr@stan:~/.../build/etc$ cat pipeline-build.properties 
# This file is automatically generated by Ziggy.
# Do not edit.
pipeline.version = 
pipeline.version.branch = 
pipeline.version.commit = 

My file does not agree with the User's Guide. But the file was generated, so maybe everything is fine.

[jcsmithhere]

I followed the remaining steps to run the sample pipeline in the console. After starting, I see my run in the "instances" panel, but it seems to be hung. Below is what I see. The worker idiot light is grey and there are no pipeline tasks showing up.

Is something wrong, or do I just need to wait longer?

[jcsmithhere]

My pipeline run is clearly hung. I can't find a way to stop the run. The Halt all incomplete tasks returns an error that there are no tasks to halt. Restarting the run also returns an error. I guess I just close the console and stop the cluster with ziggy cluster stop.

[wohler]

You definitely need to define the environment variable ZIGGY_HOME to /path/to/ziggy/build. I'd also suggest that you add $ZIGGY_HOME/bin to your path. I see now that the user manual doesn't say to do that. It will.

ZIGGY_ROOT is only used for the sample pipeline. You won't use it for your own pipeline. You will use ZIGGY_HOME, however, since that's how your pipeline will find Ziggy's binaries and libraries.

I'd suggest reverting your changes to the build script and running sample-pipeline/clean-env.sh. Then you can try my quick start below that I developed from an empty environment. Note that you'll have to substitute "/path/to" with your own paths. I hope this goes smoother!

export ZIGGY_ROOT=/path/to/ziggy
export ZIGGY_HOME=$ZIGGY_ROOT/build
export PIPELINE_CONFIG_PATH=$ZIGGY_ROOT/sample-pipeline/etc/sample.properties 
export JAVA_HOME=/path/to/java-17
export CC=/path/to/gcc
export CXX=/path/to/g++
export PATH=$ZIGGY_HOME/bin:$JAVA_HOME/bin:`dirname $CC`:/path/to/python3>=3.8/bin:$PATH
./gradlew
sample-pipeline/build-env.sh
ziggy cluster init && ziggy cluster start && ziggy console start sample
while true; do ziggy console display --instance 1 --displayType full; sleep 10; done 

Be sure the USER environment variable is defined. I also found that Python 3.6 does not work. Python 3.11 works and PT tells me that Python 3.8 probably works.

In this example, I used the command line to start and display the progress of the pipeline since you asked about that. Try "ziggy console --help".

[wohler]

To answer some of your other questions:

The manual is out of date. My sample-pipeline/build directory matches what you see:

[bwohler@lmc ziggy (release/ZIGGY-612-ziggy-0.11.0:=)]$ ls sample-pipeline/build
env/  etc/  pipeline-results/  src/

Regarding pipeline-build.properties, I would not expect to see empty properties. Given that you hadn't defined ZIGGY_HOME, I'm rather surprised this file was generated at all. When you follow my quick start, you should have a file with valid properties that contain your Git tag, branch, and commit.

As we mentioned yesterday, the pipeline completes in about 30 seconds. I typically see the behavior you report when build-env failed in some way. Resolving the issue and performing a scorched earth restart usually does the trick. That means: ziggy cluster stop && gradle clean && gradle && sample-pipeline/clean-env.sh && sample-pipeline/build-env.sh && ziggy cluster init && ziggy cluster start && ziggy console start sample.

Thanks for the feedback!