Differences between revisions 59 and 60
Revision 59 as of 2022-03-29 07:39:49
Size: 11332
Editor: GötzWaschk
Revision 60 as of 2022-05-12 17:07:20
Size: 11378
Editor: GötzWaschk
Deletions are marked like this. Additions are marked like this.
Line 16: Line 16:
 * 2022-05-12
   * AlmaLinux updated to 8.6


Singularity allows instantiating an alternative operating system environment on our Linux systems.

In other words:

  • If you need to develop software in a Scientific Linux 7 environment on your Ubuntu 16.04 desktop and then execute it on a Scientific Linux 6 system, Singularity makes that possible.
  • If you need to run software from Ubuntu 16.04 (like an up to date PDF viewer) or other software requiring recent versions of some shared libraries (like the Atom editor) on the ancient SL6, Singularity makes that possible.
  • If you still have software for the discontinued SL5 around and need to run, build, develop or debug it on a supported system, Singularity makes that possible.

And all this without significant overhead. Sounds like fiction? It's fact. Right now. Some caveats do apply though.

Credit: Singularity was and is being developed at Berkeley Lab. Please see https://sylabs.io/docs/ for upstream information.


  • 2022-05-12
  • 2022-03-29
  • 2022-02-21
  • 2021-12-09
  • 2021-11-22
  • 2021-11-15
  • 2021-10-06
  • 2021-09-02
    • SL7 and pax images updated to Scientific Linux 7.9
  • 2021-07-01
  • 2021-06-11
  • 2021-06-03
  • 2021-04-27
  • 2021-04-12
    • AlmaLinux 8.3: initial image of the RHEL8 clone

  • 2021-03-31
  • 2020-12-10
  • 2020-12-03
    • CentOS7 image updated to 7.9
  • 2020-10-26
  • 2020-10-05
  • 2020-08-11
  • 2020-05-20
    • CentOS, SL7 and pax images updated to 7.8
  • 2020-03-05
  • 2020-01-29
  • 2019-09-24
    • CentOS image updated to 7.7
  • 2019-08-30
    • SL-based images update to 7.7
  • 2019-06-12
    • SL-based images update to 7.6
  • 2018-12-12
  • 2018-10-29
    • SL-based images updated to 6.10 and 7.5
  • 2018-09-04
    • A new image for CentOS 7.5 was created
  • 2018-09-03
    • Singularity will be updated to release 2.6.0
  • 2018-08-01
    • A new image pax was created with an environment compatible with the parallel farm
  • 2018-07-05
    • Singularity will be updated to the security release 2.5.2 tonight - no negative impact on users is epected
    • the SL7 image was updated and no longer provides the "cluster" software/environment
      • this should enhance compatibility with ordinary WGSs and the (non-pax) batch farm
  • 2018-06-21
    • Gridengine directory /usr/gridengine is bind mounted on SL hosts to provide UGE batch system access
  • 2018-06-06
    • Overlayfs was reenabled
  • 2018-06-05
    • Initial Ubuntu 18.04 image released, needs host newer than SL6 to run
  • 2018-05-28
    • Singularity was updated to release 2.5.1 on all supported systems
  • 2018-05-01
    • Singularity was updated to release 2.5.0 on all supported systems
  • 2018-04-03
    • We had to disable the use of overlayfs for security reasons for the time being. Consequently, bind mounting additional paths will not work unless the mount point exists in the image. This will inhibit CUDA support as described below, at least until we have rolled out images containing /etc/OpenCL/vendors .
  • 2018-02-21
    • Singularity was updated to release 2.4.2 on all supported systems
    • A set of updated standard images was provided - now in squashfs format which makes them even snappier to start and use


There are several ways to instantiate a Singularity container and run software in it. The two most important ones are:

  • "Execute" an image
    Singularity images are executable. Executing one of the standard images provided by DV will give you an interactive shell in the container environment:

    [wgs03] ~ % lsb_release -d                     
    Description:    Red Hat Enterprise Linux Server release 6.9 (Santiago)
    [wgs03] ~ % /project/singularity/images/U16.img
    * You are now running a shell in a  Singularity container providing an  *
    * Ubuntu 16.04 environment. Most things should just work as if you had  *
    * just logged in to  such a system, but there are differences.          *
    *                                                                       *
    * For more info, please see https:://dvinfo.zeuthen.desy.de/Singularity *
    [wgs03:U16] ~ % lsb_release -d
    Description:    Ubuntu 16.04.2 LTS
    [wgs03:U16] ~ %
    Note the modified prompt. It's an attempt to make it more transparent in which environment you're currently working.
  • Run an arbitrary command in the context of the image.

    [wgs03] ~ % lsb_release -d
    Description:    Red Hat Enterprise Linux Server release 6.9 (Santiago)
    [wgs03] ~ % singularity exec /project/singularity/images/SL5.img lsb_release -d
    Description:    Scientific Linux release 5.11 (Boron)
    [wgs03] ~ % 


Singularity containers can be instantiated on any current user accessible Linux system at DESY Zeuthen. In particular, Singularity is available on all

  • Linux Desktops
  • Workgroup Servers
  • Computes Nodes (Farm & Cluster)

The current set of standard images available in /project/singularity/images includes

  • Scientific Linux 5 (SL5.img)

  • Scientific Linux 6 (SL6.img)

  • Scientific Linux 7 (SL7.img)

  • Ubuntu 16.04 LTS (U16.img)

  • Ubuntu 18.04 LTS (U18.img)

These short image names are symbolic links to the current versions of the standard images for these operating systems. The specific images are being kept around for a few months, but not forever.

Note users can copy a current image to any location to make sure this exact version will remain available.

Also note users can bring their own images and thus don't have to rely on the ones provided by DV. Creating those is rather simple and documented upstream. It currently requires root access though, so cannot be done on supported Linux systems in Zeuthen. But have a look at https://singularity-hub.org .

Since Release 2.3, it's also possible to import docker images without root privileges. Note that the environment variable SINGULARITY_CACHEDIR is used to place downloaded image layers, and it defaults to ~/.singularity. Change the variable to something more sensible to avoid wasting a lot of home space.

Batch system support

In case you want to run your batch jobs directly under the standard images available in /project/singularity/images, it is possible via the qsub switch -l singularity_os=<os>, whereas <os> defines the image to be used. Example:

[wgs] ~ % qrsh -w n -l singularity_os=sl6 lsb_release -d
Description:    Scientific Linux release 6.9 (Carbon)
[wgs] ~ % qrsh -w n -l singularity_os=u18 lsb_release -d
Description:    Ubuntu 18.04 LTS

/!\ Note: The -w n switch is recommended for qsub/qrsh in this case to avoid the warning "warning: no suitable queues".

Filesystems shared between Host and Container

Subject to availability on the host:

  • $HOME

  • /tmp

  • /afs

  • /lustre

  • /cvmfs

  • /batch

  • /acs, /pnfs (dCache)

  • /net (dCache)

  • /nfs

  • /run/user

  • /proc

  • /dev

  • /usr/gridengine

  • $PWD (the current working directory of the process invoking singularity)

Users can specify additional folders on the host to be bind mounted inside the container when the container is started. This shouldn't be required. But if it is, please see singularity help run for info how to do it - and let DV know why you need to do this.

CUDA support

To run CUDA code in singularity, the CUDA libraries must be available both inside and outside the container. Singularity can automatically bind the needed directories into the container like this:

singularity exec --nv -B /etc/OpenCL/vendors yourprogram


  • The "standard images" are probably not final. If you need additional software or configuration, please let us know.

  • While we made an effort to provide applications running in a Singularity container with the same set of significant environment variables - and in particular PATH and OS_ARCH should make sense - , environment variables inside the container may differ from expectation. DV can improve here if required.

  • While AFS just works, and the credentials (=tokens) inside a container are just the same as outside, the AFS sysname is always that of the host. There is no way to change this. Which is yet another reason not to rely on AFS sysname magic. To reiterate: using paths like /afs/zeuthen.desy.de/group/software/@sys/bin will not work as expected.

  • Automounted filesystems on SL6 hosts will have to be mounted before the container starts, thus the mount may have to be triggered outside the container. This affects cvmfs, lustre and dCache

  • Some special applications require compatibility between userland (tied to the container environment) and kernel (tied to the host). In particular:
    • CUDA
    • InfiniBand

    • MPI across hosts, especially over InfiniBand

    While all of those use cases are objectives of the Singularity project, actually making them work hasn't been explored yet. In particular, our standard images currently don't have CUDA installed - but see the note on CUDA above.
  • Setuid/Setgid bits have no effect in the container. We haven't found a case where this causes real problems, but it can be surprising. Affected:
    • ping

    • screen (workaround: use tmux instead)

Singularity (last edited 2022-05-12 17:07:20 by GötzWaschk)