Workflows for Running & Developing SOG

Set Up Your Mercurial Configuration

Before you use Mercurial to track changes that you make the the SOG code, documentation, or initialization and forcing data you need to do some initial setup of your Mercurial configuration.

Note

It is essential that these configuration steps be completed before you push changes to the central, shared repositories so that the email notification and buildbot hooks will work properly.

These instructions assume that you use emacs as your editor.

  1. Set the EDITOR and VISUAL environment variables to emacs to ensure that Mercurial will take you into emacs, not vi for creation of commit messages, etc. If you use bash as your shell use:

    $ export EDITOR=emacs
    $ export VISUAL=emacs
    

    and add the same commands to your ~/.bash_profile file so that the environment variables are set whenever you log in. If you use csh as your shell use:

    $ setenv EDITOR emacs
    $ setenv VISUAL emacs
    

    and add the same commands to your ~/.cshrc file so that the environment variables are set whenever you log in.

  2. If you have not already done so, you should create a .hgrc file in your home directory containing:

    [ui]
    username = Your Name <your_userid@eos.ubc.ca>
    ignore = ~/.hgignore
    merge = emacs-merge.sh
    

    These lines:

    • Set how your changes will be attributed to you in commit messages, etc.
    • Tell Mercurial to find your list of globally ignored files in ~/.hgignore (see below)
    • Tell Mercurial to use the emacs-merge.sh shell script (see below) to hook into emacs as your merge resolution tool
  3. Create a .hgignore file in your home directory containing:

    syntax: glob
    *~
    
    syntax: regexp
    (.*/)?\#[^/]*\#$

    These lines will cause Mercurial to ignore emacs temporary and backup files in all of your Mercurial repositories (not just the SOG ones).

  4. If you don’t already have one, create a bin directory in your home directory:

    $ mkdir ~/bin
    

    Add ~/bin to your path. If you use bash as your shell use:

    $ export PATH=$PATH:$HOME/bin
    

    and add the same command to your ~/.bash_profile file so that ~/bin is added to your path whenever you log in. If you use csh as your shell use:

    $ setenv PATH ${PATH}:${HOME}/bin
    

    and add the same command to your ~/.cshrc file so that ~/bin is added to your path whenever you log in.

  5. Create an emacs-merge.sh file in your ~/bin directory containing:

    #!/bin/sh
    
    # Enable use of emacs ediff mode as merger program for mercurial
    
    # Hook to mercurial in ~/.hgrc is:
    #  [ui]
    #  merge = emacs-merge.sh
    
    # Copied from http://www.selenic.com/mercurial/wiki/index.cgi/MergingWithEmacs
    
    # bail out quickly on failure
    set -e
    
    LOCAL="$1"
    BASE="$2"
    OTHER="$3"
    
    BACKUP="$LOCAL.orig"
    
    Restore ()
    {
        cp "$BACKUP" "$LOCAL"
    }
    
    ExitOK ()
    {
        exit $?
    }
    
    # Back up our file
    cp "$LOCAL" "$BACKUP"
    
    # Attempt to do a non-interactive merge
    if which merge > /dev/null 2>&1 ; then
        if merge "$LOCAL" "$BASE" "$OTHER" 2> /dev/null; then
        # success!
        ExitOK
        fi
        Restore
    elif which diff3 > /dev/null 2>&1 ; then
        if diff3 -m "$BACKUP" "$BASE" "$OTHER" > "$LOCAL" ; then
        # success
        ExitOK
        fi
        Restore
    fi
    
    if emacs -q --no-site-file --eval "(ediff-merge-with-ancestor \"$BACKUP\" \"$OTHER\" \"$BASE\" nil \"$LOCAL\")"
    then
        ExitOK
    fi
    
    echo "emacs-merge: failed to merge files"
    exit 1
    

    Make emacs-merge.sh executable with:

    $ chmod u+x ~/bin/emacs-merge.sh
    
  6. Add /ocean/dlatorne/.virtualenvs/SOG-hg-buildbot to your PYTHONPATH environment variable, and make the Mercurial instance installed there your default. This ensures that the email notification and buildbot hooks will work properly when you push changes to any of the SOG repositories. If you use bash as your shell use:

    $ export PYTHONPATH=$PYTHONPATH:/ocean/dlatorne/.virtualenvs/SOG-hg-buildbot/lib/python2.6/site-packages
    $ alias hg="/ocean/dlatorne/.virtualenvs/SOG-hg-buildbot/bin/hg"
    

    and add the same 1st command to your ~/.bash_profile, and the end to your ~/.bashrc file so that they take effect whenever you log in. If you use csh as your shell use:

    $ setenv PYTHONPATH ${PYTHONPATH}:/ocean/dlatorne/.virtualenvs/SOG-hg-buildbot/lib/python2.6/site-packages
    $ alias hg "/ocean/dlatorne/.virtualenvs/SOG-hg-buildbot/bin/hg"
    

    and add the same commands to your ~/.cshrc file so that they take effect whenever you log in.

Using Mercurial

If you’re not familiar with Mercurial you should read at least Chapter 2 of the Mercurial - The Definitive Guide book right now.

Shared Repositories in the SOG Environment

Your SOG working environment includes clones of 4 shared Mercurial repositories from the EOS ocean file server:

  • SOG

    The top level repository that contains the Makefile for managing your SOG environment, the source files for this documentation, and the other 3 repository clones as sub-directories.

  • SOG-code

    The SOG source code which is cloned to SOG-code-ocean in tour environment.

  • SOG-initial

    The SOG initial conditions data repository that contains CTD and nutrient data files used to initialize SOG runs.

  • SOG-forcing

    The SOG forcing data repository that contains meteorological, river flow, wind, and bottom conditions data files used provide forcing terms at each time-step in SOG runs.

Subscribing to Email Notifications

To keep tabs on changes that other users are making to Shared Repositories in the SOG Environment you should subscribe to email notifications for each repository. You can ask Doug or Susan to do that for you, or you can do it yourself with the following commands:

$ cd SOG
$ hg clone /ocean/sallen/hg_repos/notify
$ cd notify
$ emacs notify.conf

Add your email address to the comma separated list of addresses for each repository in the [reposubs] section of notify.conf and save the file. Commit the change in your local repository and push it to ocean:

$ hg commit
$ hg push

A Mercurial hook in the /ocean/sallen/hg_repos/notify repository will update the notify.conf file there and you should start to receive an changeset notification messages.

notify.conf also contains a subscription list for changes to the SOG buildbot Automated Testing System code and docs that you can join if you are interested.

Once subscribed you will receive an message from hg_repos@eos.ubc.ca for each changeset that you or any other user pushes to the shared repositories on ocean. The subject of the message will indicate that it is a Mercurial update message, tell you which repositories it applies to, and give you 50 characters (or so) of the commit message; example:

HG update: SOG-code: Delete unused new_year.f90 source code file.

The message contains details of the changeset and a link to the changeset in the SOG trac Code Browser and Issue Tracker.

Keeping Your Repository Clones Up To Date

The Mercurial commands that you need to help you keep your repository clones up to date are:

  • hg incoming

    Gives you a list of changesets in a remote repository that have not been pulled into your local repository

  • hg pull

    Pull changes from a remote repository to your local one

  • hg update

    Update the files in your local repository with all of the changes you have pulled in

  • hg merge

    Merge the changesets pull into your local repository with changes that you have made since you last updated from the remote repository

  • hg status

    Show that status of files in your local repository

You can shorten Mercurial commands to the fewest letters that make them unique; e.g. hg inco, hg up, hg stat.

You may not need to use hg incoming if you know from having received changeset notification messages that there have been changes to the remote repository. On the other hand, you may want to use it just to check.

To update your local clone of the SOG-code repository with changes that other users have pushed to the shared repository on ocean, recall that SOG-code is cloned to SOG-code-ocean in your local SOG environment and use the commands:

$ cd SOG-code-ocean
$ hg incoming
... list of changesets that will be pulled ...
$ hg pull
... status of pull request ...
$ hg update
... status of update request ...

From the hg pull command onward Mercurial will tell you what you need to do next.

hg incoming and hg pull look at the remote repository that your local repository was cloned from by default. You can check where that repository is with the hg paths command. You can operate against a different remote repository by giving its path explicitly; e.g.:

$ hg incoming /ocean/dlatornell/SoG/SOG/SOG-code-dev

If there are changes in your local repository that you have made, or pulled from another repository since you last updated from the remote repository Mercurial will tell you that you need to do a merge. It will refuse to do the merge if you have any uncommitted changes in your repository, so clean up before you pull. Mercurial is pretty good at merging files automatically, but sometimes it needs help and it will open emacs in ediff mode for you to manually merge the changes in a file where it find unresolvable conflicts. Once a merge is finished, Mercurial will remind you to commit the result of the merge. Use a commit message something like:

Merge changes from ocean repository.

For more information about merging see Chapter 3 of the Mercurial - The Definitive Guide book.

Using gfortran

These are work in progress notes on changing the SOG build tool chain from using g95 to using gfortran.

Note

gfortran is not (yet) installed on the ocean machines. For now these notes documents Doug’s development work on OS/X 10.6 (Snow Leopard).

Development of the g95 Fortran 95 compiler appears to have ceased (or at least gone on hiatus) in late-2008. On the other hand, compared to its moribund state in 2006, the gfortran compiler is under active development and appears to have become the free/open-source Fortran 95 compiler of choice.

Installing gfortran on OS/X

There are a variety of options described on the gfortran binaries page. These notes are based on an installation from the

  • Snow Leopard (10.6) on Intel 64-bit processors (gfortran 4.6.2): download (released on 2011-10-21)

link.

Installation of the disk image from that link puts the ditribution in /usr/local/gfortran with executable links in /usr/local/bin. Make sure that /usr/local/bin is on your path.

Using gfortran

You can do a clean gfortran build with:

$ make clean && make F90=gfortran LD=gfortran

If/when we adopt gfortran as the standard compiler for the SOG project the Mafefile will be changed to set F90 and LD variable values to gfortran.

Makefile Changes

  • gfortran doesn’t recognize the g95 -ftrace=full option in the FFLAGS_DEV variable, it uses -fbacktrace instead
  • The g95 -fbounds-check in the FFLAGS-DEV needs to be changed to the more general -fcheck=all that includes bounds checking

Code Changes

The code at Mercurial changeset 8ff28b078730 builds cleanly with g95. The changes below were required to get a clean build with gfortran.

Errors

Compiling with gfortran produces:

air_sea_fluxes.f90:160.29:

    h_latent = max(h_latent, 0)
                             1
Error: 'a2' argument of 'max' intrinsic at (1) must be REAL(8)

Changing that line to:

h_latent = max(h_latent, 0.0d0)

resolves the compile time error and diffs clean for a 300 hour infile run when compiled with g95. However, running the gfortran compiled code produces the runtime error:

At line 443 of file grid.f90
Fortran runtime error: Array bound mismatch for dimension 1 of array 'indices' (82/81)

That error is from interp_value() and arises when that subroutine is used to interpolate on quantities defined at grid interfaces rather than grid points. Fixing it causes the g95 compiled code to not diff clean.

The next runtime error that arises in the gfortran compiled code is:

At line 546 of file turbulence.f90
Fortran runtime error: Index '81' of dimension 1 of array 'nu' outside of expected range (80:1)

That error is due to a mistake in the code for the vectorized calculation of the smoothed sheer diffusivity. The g95 compiled code diffs clean with this error fixed.

The next runtime error from the gfortran compiled code is:

At line 299 of file physics_eqn_builder.f90
Fortran runtime error: Array bound mismatch for dimension 1 of array 'p_grad' (80/81)

That error is due to a mistake in the code for the calculation of the Coriolis and baroclinic pressure gradient components of the RHS vector for the physics equations. The g95 compiled code diffs clean with this error fixed.

Warnings

Compiling with gfortran produces several warnings.

forcing.f90:172.66:

    use_average_forcing_data = getpars("use average/hist forcing")
                                                                  1
Warning: CHARACTER expression will be truncated in assignment (8/80) at (1)

Changing the declaration of use_average_forcing_data to character*80 resolves that.

baroclinic_pressure.f90:183.18:

    gridbotsurf = 15./0.25
                  1
Warning: Possible change of value in conversion from REAL(4) to INTEGER(4) at (1)

This warning was resolved by refactoring the baroclinic_P_gradient() subroutine in baroclinic_pressure.f90 to change the surface layer depth to a parameter and make the calculation of its grid point index valid for any grid spacing; see changeset adf1c51fb6e4.

upwelling.f90:64.37:

  subroutine upwelling_profile(Qriver)
                                     1
Warning: Unused dummy argument 'qriver' at (1)

This was one of many warnings about unused dummy arguments. They were all resolved by deleting the unused arguments.

Noted in Passing

  • air_sea_fluxes.f90 contains lots of real literals that are not explicitly specified to be double precision