-
Notifications
You must be signed in to change notification settings - Fork 282
Build FATES container for PEcAn and create job.sh to run fates in the container #3465
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Hhh-hyc
wants to merge
3
commits into
PecanProject:develop
Choose a base branch
from
Hhh-hyc:develop_3.1
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,137 @@ | ||
| # this needs to be at the top, what version are we building | ||
| ARG IMAGE_VERSION="latest" | ||
|
|
||
| # ---------------------------------------------------------------------- | ||
| # BUILD MODEL | ||
| # ---------------------------------------------------------------------- | ||
| FROM pecan/models:${IMAGE_VERSION} | ||
|
|
||
| # load packages before adding CTSM | ||
| RUN apt update && apt upgrade -y | ||
| RUN apt install -y \ | ||
| sudo \ | ||
| build-essential \ | ||
| python3-dev \ | ||
| cmake \ | ||
| gfortran-9 \ | ||
| zlib1g-dev \ | ||
| byacc \ | ||
| libblas-dev \ | ||
| liblapack-dev \ | ||
| libxml2-utils \ | ||
| libxml-libxml-perl \ | ||
| libgdal-dev \ | ||
| xmlstarlet \ | ||
| csh \ | ||
| diffutils \ | ||
| rsync | ||
|
|
||
| RUN update-alternatives --install /usr/bin/gfortran gfortran /usr/bin/gfortran-9 100 | ||
|
|
||
| RUN echo '/usr/local/lib' > /etc/ld.so.conf.d/local.conf && ldconfig | ||
|
|
||
| WORKDIR /tmp/sources | ||
|
|
||
| RUN wget -q http://www.mpich.org/static/downloads/3.3.2/mpich-3.3.2.tar.gz && \ | ||
| tar zxf mpich-3.3.2.tar.gz && \ | ||
| cd mpich-3.3.2 && \ | ||
| ./configure --prefix=/usr/local && \ | ||
| make -j 2 install | ||
|
|
||
| RUN wget -q https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.0/src/hdf5-1.12.0.tar.gz && \ | ||
| tar zxf hdf5-1.12.0.tar.gz && \ | ||
| cd hdf5-1.12.0 && \ | ||
| ./configure --prefix=/usr/local && \ | ||
| make -j 2 install | ||
|
|
||
| RUN wget -q https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.7.4.tar.gz && \ | ||
| tar zxf v4.7.4.tar.gz && \ | ||
| cd netcdf-c-4.7.4 && \ | ||
| ./configure --prefix=/usr/local && \ | ||
| make -j 2 install && \ | ||
| ldconfig | ||
|
|
||
| RUN wget -q https://github.com/Unidata/netcdf-fortran/archive/refs/tags/v4.5.3.tar.gz && \ | ||
| tar zxf v4.5.3.tar.gz && \ | ||
| cd netcdf-fortran-4.5.3 && \ | ||
| ## original netcdf(version=4.7.3): /usr, ; fates netcdf(version>=4.7.4): /usr/local; --enable-fortran to ensure netcdf.mod | ||
| ./configure --prefix=/usr/local LDFLAGS="-L/usr/local/lib" CPPFLAGS="-I/usr/local/include" --enable-fortran && \ | ||
| make -j 2 install && \ | ||
| ldconfig | ||
|
|
||
| RUN wget -q https://parallel-netcdf.github.io/Release/pnetcdf-1.12.1.tar.gz && \ | ||
| tar zxf pnetcdf-1.12.1.tar.gz && \ | ||
| cd pnetcdf-1.12.1 && \ | ||
| ./configure MPIF77=/usr/local/bin/mpif77 MPIF90=/usr/local/bin/mpif90 MPICXX=/usr/local/bin/mpicxx MPICC=/usr/local/bin/mpicc && \ | ||
| make -j 2 install && \ | ||
| ldconfig | ||
|
|
||
| RUN cd /tmp/sources \ | ||
| wget -q https://github.com/esmf-org/esmf/archive/refs/tags/ESMF_8_2_0.tar.gz && \ | ||
| tar zxf ESMF_8_2_0.tar.gz && \ | ||
| cd esmf-ESMF_8_2_0 && \ | ||
| export ESMF_DIR='/tmp/sources/esmf-ESMF_8_2_0' && \ | ||
| export ESMF_INSTALL_PREFIX='/usr/local' && \ | ||
| export ESMF_LAPACK="netlib" && \ | ||
| export ESMF_LAPACK_LIBS="-llapack -lblas" && \ | ||
| export ESMF_LAPACK_LIBPATH="/usr/lib/x86_64-linux-gnu" && \ | ||
| export ESMF_NETCDF="nc-config" && \ | ||
| make && \ | ||
| make install && \ | ||
| ldconfig | ||
|
|
||
| RUN rm -rf /tmp/sources | ||
| RUN pip install --upgrade pip setuptools | ||
|
|
||
| ## Above has been tested in the pecan/model container | ||
|
|
||
| ######################################################################## | ||
|
|
||
| ## Below needs further discussion. | ||
|
|
||
| ## Load CTSM from https://github.com/NorESMhub/noresm-land-sites-platform/blob/main/docker/api/entrypoint_api.sh | ||
| ## Following the structure of ED, download CTSM | ||
| WORKDIR /src | ||
| RUN git clone https://github.com/ESCOMP/CTSM.git && \ | ||
| ## go to model folder | ||
| cd CTSM && \ | ||
| ## Checkout specific CTSM version, which has been tested in the container, more recent version than this needs to be tested in the future. | ||
| git checkout ctsm5.1.dev108 && \ | ||
| ## Download fates (tag = sci.1.58.1_api.24.1.0), the version can be modified in Externals_CLM.cfg & CISM should be manually cloned | ||
| ./manage_externals/checkout_externals | ||
|
|
||
| RUN set -e && \ | ||
| ## set home folder for running CTSM-fates. | ||
| USER=root && \ | ||
| HOME=/root && \ | ||
| ## create .bashrc file for setting basic compiling parameters for CTSM-fates | ||
| export USER="$USER" && \ | ||
| ## 1 /usr/bin/python3 | ||
| export PYTHONPATH='/usr/bin/python3' && \ | ||
| export CIME_MACHINE=container && \ | ||
| export MPICC=mpicc && \ | ||
| export MPIFC=mpif90 && \ | ||
| export MPIF90=mpif90 && \ | ||
| export MPIF77=mpif77 && \ | ||
| ## if CESMDATAROOT is changed, also change in config.py and other relevant places. | ||
| ## 2 /src/data | ||
| export CESMDATAROOT=${CESMDATAROOT:-/src/data} && \ | ||
| ## Copy dotcime ((https://github.com/NorESMhub/ctsm-api/tree/main/docker/dotcime)) to pecan/models/fates folder | ||
| ## Dotcime is a key file for setting the compiling environment for CTSM-fates, SHOULD be kept in the home directory of the fates model conatiner as '.cime' | ||
| cd .. && \ | ||
| git clone https://github.com/NorESMhub/ctsm-api.git && \ | ||
| cp -r ./ctsm-api/docker/dotcime "$HOME"/.cime && \ | ||
| rm -rf /src/ctsm-api | ||
|
|
||
|
|
||
| ######################################################################## | ||
|
|
||
| # Some variables that can be used to set control the docker build | ||
| ARG MODEL_VERSION=git | ||
| ENV MODEL_VERSION=${MODEL_VERSION} | ||
|
|
||
| # Setup model_info file | ||
| ## model.json | ||
| COPY model_info.json /work/model.json | ||
| RUN sed -i -e "s/@VERSION@/${MODEL_VERSION}/g" \ | ||
| -e "s#@BINARY@#/usr/local/bin/fates.${MODEL_VERSION}#g" /work/model.json | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given that there already existing CESM docker container efforts, e.g., https://hub.docker.com/r/openeuler/cesm, I wonder if it would be easier to start from that image and just add the required PEcAn and rabbitmq bits from pecan/models
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @mdietze for pointing this. It is interesting to try. I am not sure if it is easier to install PEcAn related libraries or not.
There are two additional concerns: (1) this is a CESM docker container, I am not sure if PEcAn want the whole cesm or just CLM-FATES. (2) the CLM version used in this image is already two years old (see its dockerfile). Some recent development of CLM-FATES will be missed. I am pretty sure that we need to update the installed libraries such as PIO, netcdf and ESMF, if we want to run the newer version of CLM-FATES.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, not required to go this route, just wanted to bring it up as an option. If you think we can set up the GH Actions to keep a FATES container continuously up-to-date, and other projects aren't doing that, then that's a strong argument in favor of the current route
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure if @glemieux, @rosiealice and @serbinsh have any comments about this? Setting up the GH Actions to keep a FATES container continuously up-to-date sounds like a good plan to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it'd be good to get @briandobbins insight on what efforts on the NCAR side of things might exist that you could leverage. I know there are container recipes for e3sm that are being developed, although I'm not sure about the details of their goals. The FATES team recently built off of @serbinsh containers for a tutorial that uses a more up-to-date version of e3sm, but we don't have funding currently to keep these up-to-date: https://github.com/NGEET/fates-containers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@glemieux the NGEET fates contains similarly seem to be 6 mo old, similar to the other efforts that @huitang-earth concluded were lagging too far behind model development.
When you say it's built off of @serbinsh container does that mean this PEcAn FATES container (which Shawn helped engineer) or a different container? We definitely can't set up a circular dependency. If a different container, which one?
Finally, I'm not sure what you mean by "funding to keep these up-to-date" as all of our containers are built by GH Actions -- the containers update automatically every time the mainline of the code is updated on github. We're OK with linking our container to our develop branch, but its just as easy to link to updates to the stable release version. But the key point is that it's no anyone's job to update the containers so it doesn't take much time/funding (just occasional tweaks to GH Actions)
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mdietze the containers aren't strictly built off of Shawn's containers in the way you mean; I copied the container recipes and updated/modified them to our NGEET org repo. I realized also that I pointed to the wrong repo in my previous message. This repo is actually a fork of the BNL Test group NGEEA workshop containers that Shawn developed: https://github.com/NGEET/tutorial-containers
Wrt funding, we didn't have time to set up our tutorial containers with github actions at the time is all. I fully agree that setting up a development pipeline to keep things up to date is the way to go, it's just currently not high on the priority list relative to other funded goals. All that said, we hope to run this training again in the future and I would like to use the time in preparation for that to set that up.