SKA Regional Centre Training Event Series

📡 SKA hands-on containerization


This is the first event in a series of workshops that will lead attendees to understand the basics and dive into more advanced knowledge of the technologies and tools that will be useful for training on the working model in the years to come. The objective of this training is to develop a complete workflow with data, ranging from the use of a repository with version control where the code and data will be stored, to how to run some experiments using container technology that will ensure its reproducibility.  In order to complete this workflow it will be necessary to have installed all the tools that we indicate in this document and that basically are three: git, docker and singularity.

Table of contents:

🚀 Get everything installed        

📦 1. All-in-one installation        

1.1. Vagrant - Virtual Machine        

1.1.1 Install components        

1.2. Deploy SKA-Training environment        

1.3. Test the environment        

📋 2. Manual installation        

2.1 For Linux        

2.1.1. Git        

2.1.2. Docker        

2.1.3. Singularity        

2.2 For MacOS        

2.2.1. Git        

2.2.2. Docker        

2.2.3. Singularity        

2.3 For Windows        

2.3.1. Git        

2.3.2. Docker        

2.3.3. Singularity        



🚀 Get everything installed

To work on the SKA Regional Centre Training Event Series it is necessary to have these three components installed: Git, Docker and Singularity, which you can use our all-in-one environment or you can install manually (Git, Docker, Singularity).

📦 1. All-in-one installation

With this installation you will be able to use all the tools that we are going to use in this SKA-Training from an environment that has everything up and ready to work. Please refer to 'Manual installation' instructions if you would rather install packages individually.

Minimum requirements:

A 64-bit Intel CPU.
At least 2 GBytes of available RAM and 50 GB minimum storage.
Access to your computer as Administrator or Root.
Support for Virtual Machine from BIOS setting.

1.1. Vagrant - Virtual Machine

To enable this environment, it is necessary to install some components. First you need to install the components and then deploy the environment.

1.1.1 Install components

First download and then install VirtualBox for your system (Linux, MacOSX or Windows):

Then, install Vagrant for your Operating System:

For Debian/Ubuntu

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -


sudo apt-add-repository \

 "deb [arch=amd64] https://apt.releases.hashicorp.com \

  $(lsb_release -cs) main"


sudo apt-get update && sudo apt-get install vagrant

For Fedora/CentOS

sudo yum install -y yum-utils

sudo yum-config-manager --add-repo \

    https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo

sudo yum -y install vagrant

For Windows

Vagrant for 64-bit

For MacOS X

Minimum requirements:

A 64-bit Intel CPU or Apple Silicon CPU macOS Catalina (10.15) (or higher) Command Line Tools (CLT) for Xcode (from xcode-select --install or https://developer.apple.com/download/all/) or Xcode

Install from brew (first, you will need to install brew, once installed type the following):

brew install vagrant

From native application, download the following file and install it.

1.2. Deploy SKA-Training environment

For Linux and MacOS X

Open a Terminal and type the following, we will create our environment project folder and the shared folder to work with :

mkdir ska-training && cd ska-training

And then, the following to confirm you are working in a new environment (just your first time):

vagrant destroy && rm Vagrantfile

If there was not a previous environment, you will get a warning, which is OK.

Create the SKA-Training environment for vagrant:

export VM=ska-training/containers && vagrant init $VM && vagrant up && vagrant ssh

After that, you will see:

vagrant@ska-training:~$

Finally from there you can use the git, docker and singularity commands, to work on the SKA-Training.

Once in this environment, you can access

vagrant@ska-training:~$ cd /vagrant

Where you will be able to share and view/work with the files stored in that directory on your host machine.

For Windows

Open a Command prompt and type the following, we will create our environment project folder and the shared folder to work with :

md ska-training

cd ska-training

And then, the following to confirm you are working in a new environment (just your first time):

vagrant destroy

del Vagrantfile

Create the SKA-Training environment for vagrant:

set VM=ska-training/containers

vagrant init %VM%

vagrant up

vagrant ssh

Remember that after this point you will be executing the commands from that virtual environment. To return back to your host machine, just type "exit".

Last command will allow you access to the created VM, you will see:

vagrant@ska-training:~$

Finally from there you can use the git, docker and singularity commands, to work on the SKA-Training.

Once in this environment, you can access

cd /vagrant

Where you will be able to share and view/work with the files stored in that directory on your host machine.

1.3. Test the environment

Finally, to confirm that everything is installed and working, run the following commands in this environment:

Check git

git --version

And setting-up git configuration

Once you have verified or installed git, you have to configure your credentials.

To do that, configure your Git username (create your account on GitHub / GitLab) and email using the following commands, replacing Manuel’s name with your own. These details will be associated with any commits that you create (remember to use the same name and email that you usually register with, for example the one you will use in GitLab):

git config --global user.name "Manuel"

git config --global user.email "manuel@email.com"

Check docker

docker --version

Check singularity

singularity --version

That’s all, 😄 enjoy our 🌌 🚀 SKA training on git and containers.

📋 2. Manual installation

Choose your operating system: Linux, MacOS or Windows and then follow the instructions for each component.

2.1 For Linux

2.1.1. Git

Depending on your Linux distribution, you can use the following options.

Debian / Ubuntu (with apt-get)

From your shell, install Git using apt-get:

sudo apt-get update

sudo apt-get install git

Verify the installation was successful by typing git --version

Fedora (dnf or yum)

From your shell, install Git using dnf (or yum, on older versions of Fedora and CentOS):

sudo dnf install git or sudo yum install git

Verify the installation was successful by typing git --version 

2.1.1.1. Setting-up git configuration

Once you have verified and installed git, you have to configure your credentials.

To do that, configure your Git username and email using the following commands, replacing Manuel’s name with your own. These details will be associated with any commits that you create (remember to use the same name and email that you usually register with, for example the one you will use in GitLab):

git config --global user.name "Manuel"

git config --global user.email "manuel@email.com"

2.1.2. Docker

Depending on your Linux distribution, you can use the following options.

Debian / Ubuntu

Older versions of Docker were called docker, docker.io, or docker-engine. If these are installed, uninstall them:

sudo apt-get remove docker docker-engine docker.io containerd runc

Update the apt package index and install packages to allow apt to use a repository over HTTPS:

sudo apt-get update

sudo apt-get install \

    ca-certificates \

    curl \

    gnupg \

    lsb-release

Add Docker’s official GPG key:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg  | sudo gpg \

   --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Use the following command to set up the stable repository:

echo "deb [arch=$(dpkg --print-architecture) \

   signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] \

   https://download.docker.com/linux/ubuntu \

   $(lsb_release -cs) stable" | sudo tee \

   /etc/apt/sources.list.d/docker.list > /dev/null

Update the apt package index, and install the latest version of Docker Engine and containerd

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io

After that, verify if Docker Engine is installed correctly by running the hello-world image.

sudo docker run hello-world

CentOS / Fedora

Older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies. Note: if you are in CentOS, please install dnf package with sudo yum install dnf

sudo dnf remove docker \

                  docker-client \

                  docker-client-latest \

                  docker-common \

                  docker-latest \

                  docker-latest-logrotate \

                  docker-logrotate \

                  docker-selinux \

                  docker-engine-selinux \

                  docker-engine

Install the dnf-plugins-core package (which provides the commands to manage your DNF repositories) and set up the stable repository.

sudo dnf -y install dnf-plugins-core

For Fedora:

sudo dnf config-manager --add-repo \

     https://download.docker.com/linux/fedora/docker-ce.repo

For CentOs:

sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo

Then:

sudo dnf update

Install the latest version of Docker Engine and containerd:

sudo dnf install docker-ce docker-ce-cli containerd.io

Start Docker

sudo systemctl start docker

After that, verify if Docker Engine is installed correctly by running the hello-world image.

sudo docker run hello-world

Enable this post-installation procedure if you don't have sudo access on your machine:

https://docs.docker.com/engine/install/linux-postinstall/

2.1.3. Singularity

Debian / Ubuntu

sudo apt-get update && \

     sudo apt-get install -y build-essential \

     libssl-dev uuid-dev libgpgme11-dev \

     squashfs-tools libseccomp-dev pkg-config

CentOS / Fedora

sudo yum update -y && \

     sudo yum groupinstall -y 'Development Tools' && \

     sudo yum install -y openssl-devel \

     libuuid-devel libseccomp-devel wget squashfs-tools

For all flavours, download and install Go (Go Language):

wget https://go.dev/dl/go1.17.6.linux-amd64.tar.gz

Then run the following as root or through sudo:

sudo rm -rf /usr/local/go && \

sudo tar -C /usr/local -xzf go1.17.6.linux-amd64.tar.gz

Add /usr/local/go/bin to the PATH environment variable:

export PATH=$PATH:/usr/local/go/bin

Then fix changes:

echo 'export GOPATH=${HOME}/go' >> ~/.bashrc

echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc

source ~/.bashrc

Verify that you’ve installed Go by opening a command prompt and typing the following command:

go version

You will see a line similar to:

go version go1.17.6 linux/amd64

Since we are installing Singularity v3.x you will also need to install dep for dependency resolution.

go get -u github.com/golang/dep/cmd/dep

Download, compile and install Singularity:

mkdir -p $GOPATH/src/github.com/sylabs

cd $GOPATH/src/github.com/sylabs

wget https://github.com/sylabs/singularity/releases/download/v3.9.0/singularity-ce-3.9.0.tar.gz

tar -xzf singularity-ce-3.9.0.tar.gz

Compiling and installing Singularity

cd ./singularity-ce-3.9.0 && \

    ./mconfig && \

    make -C ./builddir && \

    sudo make -C ./builddir install

After that verify if Singularity Engine is installed correctly by running:

singularity --version

2.2 For MacOS

2.2.1. Git

There are several ways to install Git on a Mac. If you’ve installed XCode (or it’s Command Line Tools), Git may already be installed. To find out, open a terminal and enter git --version. If the command returns the git version, then it is installed, otherwise you can install Xcode from the App Store or use the other methods below.

Install Git on a Mac is via the stand-alone installer

  1. Download the latest Git for Mac installer here and then install it.
  2. Open a terminal and verify the installation was successful by typing git --version.

Install Git with Homebrew

First you need to have HomeBrew installed, after that follow the next:

  1. Open your terminal and install Git using Homebrew: brew install git.
  2. Verify the installation was successful by typing git --version.

2.2.1.1. Setting-up git configuration

Once you have verified and installed git, you have to configure your credentials.

To do that, configure your Git username and email using the following commands, replacing Manuel’s name with your own. These details will be associated with any commits that you create (remember to use the same name and email that you usually register with, for example the one you will use in GitLab):

git config --global user.name "Manuel"

git config --global user.email "manuel@email.com"

2.2.2. Docker

Requirements:

MacOS must be version 10.15 (Catalina, Big Sur, or Monterey) or newer and at least 4GB of RAM.

Download Docker Desktop for Mac (for Mac with Intel Architecture) here (approx. 550 MB).

Double-click Docker.dmg to open the installer, then drag the Docker icon to the Applications folder.

Double-click Docker.app in the Applications folder to start Docker (you will see an icon in the Mac bar).

After that, open a terminal and verify the installation was successful by typing:

docker --version

2.2.3. Singularity

You need to install several programs. This example uses Homebrew but you can also install these tools using the GUI.

First, optionally install Homebrew.

/bin/bash -c "$(curl -fsSL \

https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Next, install Vagrant and the necessary bits (either using this method or by downloading and installing the tools manually).

Update brew repository:

brew update

Install VirtualBox, Vagrant, and Vagrant-manager for Catalina, BigSur:

brew install --cask virtualbox && brew install --cask vagrant && brew install --cask vagrant-manager

Install VirtualBox, Vagrant, and Vagrant-manager for Monterrey:

brew install virtualbox && brew install vagrant && brew install vagrant-manager

Note that here you will be asked for permissions to install some system extension(s)

Open a Terminal and create and enter a directory to be used with your Vagrant VM.

mkdir vm-singularity && cd vm-singularity

If you have already created and used this folder for another VM, you will need to destroy the VM and delete the Vagrantfile.

vagrant destroy && rm Vagrantfile

Then issue the following commands to bring up the Virtual Machine. (Substitute a different value for the $VM variable if you like.)

export VM=sylabs/singularity-3.0-ubuntu-bionic64 && vagrant init $VM && vagrant up && vagrant ssh

You can check the installed version of Singularity with the following:

singularity version

2.3 For Windows

2.3.1. Git

Download the latest Git for Windows installer and then install it.

Then, open a Command Prompt (or Git Bash if during installation you elected not to use Git from the Windows Command Prompt), and verify the installation was successful by typing git --version and checking it’s showing a version greater or equal than 2.34.

2.3.1.1. Setting-up git configuration

Once you have verified and installed git, you have to configure your credentials.

To do that, configure your Git username and email using the following commands, replacing Manuel’s name with your own. These details will be associated with any commits that you create (remember to use the same name and email that you usually register with, for example the one you will use in GitLab):

git config --global user.name "Manuel"

git config --global user.email "manuel@email.com"

2.3.2. Docker

Requirements:

Windows 11 64-bit, Windows 10 64-bit, at least 4GB system RAM, and BIOS-level hardware virtualization support must be enabled in the BIOS settings.

Note: Linux kernel for Windows (WSL2) and Docker requires a Windows restart once installed.

Download and install the Linux kernel (WSL2) update package.

Then, download and install Docker Desktop here.

Double-click Docker Desktop Installer.exe to run the installer. Docker Desktop does not start automatically after installation. To start Docker Desktop: Search for Docker and execute it (you will see a Docker icon within the taskbar).

After that verify if Docker is installed, up and running correctly by opening a Command Prompt and running the hello-world image, to do this:

Open a Command Prompt and type the following:

docker run hello-world

2.3.3. Singularity

Singularity cannot run natively on Windows or Mac because of basic incompatibilities with the host kernel. For this reason, the Singularity community maintains a set of Vagrant Boxes via Vagrant Cloud, one of Hashicorp’s open source tools.

Note: Singularity for Windows requires a restart once installed.

Install the following programs:

Run GitBash (Windows) and create and enter a directory to be used with your Vagrant VM.

mkdir vm-singularity && cd vm-singularity

If you have already created and used this folder for another VM, you will need to destroy the VM and delete the Vagrantfile.

vagrant destroy && rm Vagrantfile

Then issue the following commands to bring up the Virtual Machine. (Substitute a different value for the $VM variable if you like.)

export VM=sylabs/singularity-3.0-ubuntu-bionic64 && \

     vagrant init $VM && \

     vagrant up && \

     vagrant ssh

Note: This command needs some time to complete 😄

You can check the installed version of Singularity with the following:

singularity version

Your version should be greater or equal than 3.0