SKA Regional Centre Training Event Series
📡 SKA hands-on containerization
This is the first event in a series of workshops that will lead attendees to understand the basics and dive into more advanced knowledge of the technologies and tools that will be useful for training on the working model in the years to come. The objective of this training is to develop a complete workflow with data, ranging from the use of a repository with version control where the code and data will be stored, to how to run some experiments using container technology that will ensure its reproducibility. In order to complete this workflow it will be necessary to have installed all the tools that we indicate in this document and that basically are three: git, docker and singularity.
Table of contents:
1.1. Vagrant - Virtual Machine
1.2. Deploy SKA-Training environment
To work on the SKA Regional Centre Training Event Series it is necessary to have these three components installed: Git, Docker and Singularity, which you can use our all-in-one environment or you can install manually (Git, Docker, Singularity).
With this installation you will be able to use all the tools that we are going to use in this SKA-Training from an environment that has everything up and ready to work. Please refer to 'Manual installation' instructions if you would rather install packages individually.
Minimum requirements:
A 64-bit Intel CPU.
At least
2 GBytes of available RAM and 50 GB minimum storage.
Access
to your computer as Administrator or Root.
Support for
Virtual Machine from BIOS setting.
To enable this environment, it is necessary to install some components. First you need to install the components and then deploy the environment.
First download and then install VirtualBox for your system (Linux, MacOSX or Windows):
Then, install Vagrant for your Operating System:
For Debian/Ubuntu
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo
apt-add-repository \
"deb [arch=amd64] https://apt.releases.hashicorp.com \
$(lsb_release -cs) main"
sudo apt-get update &&
sudo apt-get install vagrant
For Fedora/CentOS
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo \
https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install vagrant
For Windows
Vagrant for 64-bit
For MacOS X
Minimum requirements:
A 64-bit Intel CPU or Apple Silicon CPU macOS Catalina (10.15) (or higher) Command Line Tools (CLT) for Xcode (from xcode-select --install or https://developer.apple.com/download/all/) or Xcode
Install from brew (first, you will need to install brew, once installed type the following):
brew install vagrant
From native application, download the following file and install it.
For Linux and MacOS X
Open a Terminal and type the following, we will create our environment project folder and the shared folder to work with :
mkdir ska-training && cd ska-training
And then, the following to confirm you are working in a new environment (just your first time):
vagrant destroy && rm Vagrantfile
If there was not a previous environment, you will get a warning, which is OK.
Create the SKA-Training environment for vagrant:
export VM=ska-training/containers && vagrant init $VM && vagrant up && vagrant ssh
After that, you will see:
vagrant@ska-training:~$
Finally from there you can use the git, docker and singularity commands, to work on the SKA-Training.
Once in this environment, you can access
vagrant@ska-training:~$ cd /vagrant
Where you will be able to share and view/work with the files stored in that directory on your host machine.
For Windows
Open a Command prompt and type the following, we will create our environment project folder and the shared folder to work with :
md ska-training
cd ska-training
And then, the following to confirm you are working in a new environment (just your first time):
vagrant destroy
del Vagrantfile
Create the SKA-Training environment for vagrant:
set VM=ska-training/containers
vagrant init %VM%
vagrant up
vagrant ssh
Remember that after this point you will be executing the commands from that virtual environment. To return back to your host machine, just type "exit".
Last command will allow you access to the created VM, you will see:
vagrant@ska-training:~$
Finally from there you can use the git, docker and singularity commands, to work on the SKA-Training.
Once in this environment, you can access
cd /vagrant
Where you will be able to share and view/work with the files stored in that directory on your host machine.
Finally, to confirm that everything is installed and working, run the following commands in this environment:
Check git
git --version
And setting-up git configuration
Once you have verified or installed git, you have to configure your credentials.
To do that, configure your Git username (create your account on GitHub / GitLab) and email using the following commands, replacing Manuel’s name with your own. These details will be associated with any commits that you create (remember to use the same name and email that you usually register with, for example the one you will use in GitLab):
git config --global user.name "Manuel"
git config --global user.email "manuel@email.com"
Check docker
docker --version
Check singularity
singularity --version
That’s all, 😄 enjoy our 🌌 🚀 SKA training on git and containers.
Choose your operating system: Linux, MacOS or Windows and then follow the instructions for each component.
Depending on your Linux distribution, you can use the following options.
Debian / Ubuntu (with apt-get)
From your shell, install Git using apt-get:
sudo apt-get update
sudo apt-get install git
Verify the installation was successful by typing git --version
Fedora (dnf or yum)
From your shell, install Git using dnf (or yum, on older versions of Fedora and CentOS):
sudo dnf install git or sudo yum install git
Verify the installation was successful by typing git --version
2.1.1.1. Setting-up git configuration
Once you have verified and installed git, you have to configure your credentials.
To do that, configure your Git username and email using the following commands, replacing Manuel’s name with your own. These details will be associated with any commits that you create (remember to use the same name and email that you usually register with, for example the one you will use in GitLab):
git config --global user.name "Manuel"
git config --global user.email "manuel@email.com"
Depending on your Linux distribution, you can use the following options.
Debian / Ubuntu
Older versions of Docker were called docker, docker.io, or docker-engine. If these are installed, uninstall them:
sudo apt-get remove docker docker-engine docker.io containerd runc
Update the apt package index and install packages to allow apt to use a repository over HTTPS:
sudo apt-get update
sudo apt-get install \
ca-certificates \
curl \
gnupg \
lsb-release
Add Docker’s official GPG key:
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg \
--dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
Use the following command to set up the stable repository:
echo "deb [arch=$(dpkg --print-architecture) \
signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] \
https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee \
/etc/apt/sources.list.d/docker.list > /dev/null
Update the apt package index, and install the latest version of Docker Engine and containerd
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
After that, verify if Docker Engine is installed correctly by running the hello-world image.
sudo docker run hello-world
CentOS / Fedora
Older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies. Note: if you are in CentOS, please install dnf package with sudo yum install dnf
sudo dnf remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine
Install the dnf-plugins-core package (which provides the commands to manage your DNF repositories) and set up the stable repository.
sudo dnf -y install dnf-plugins-core
For Fedora:
sudo dnf config-manager --add-repo \
https://download.docker.com/linux/fedora/docker-ce.repo
For CentOs:
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
Then:
sudo dnf update
Install the latest version of Docker Engine and containerd:
sudo dnf install docker-ce docker-ce-cli containerd.io
Start Docker
sudo systemctl start docker
After that, verify if Docker Engine is installed correctly by running the hello-world image.
sudo docker run hello-world
Enable this post-installation procedure if you don't have sudo access on your machine:
https://docs.docker.com/engine/install/linux-postinstall/
Debian / Ubuntu
sudo apt-get update && \
sudo apt-get install -y build-essential \
libssl-dev uuid-dev libgpgme11-dev \
squashfs-tools libseccomp-dev pkg-config
CentOS / Fedora
sudo yum update -y && \
sudo yum groupinstall -y 'Development Tools' && \
sudo yum install -y openssl-devel \
libuuid-devel libseccomp-devel wget squashfs-tools
For all flavours, download and install Go (Go Language):
wget https://go.dev/dl/go1.17.6.linux-amd64.tar.gz
Then run the following as root or through sudo:
sudo rm -rf /usr/local/go && \
sudo tar -C /usr/local -xzf go1.17.6.linux-amd64.tar.gz
Add /usr/local/go/bin to the PATH environment variable:
export PATH=$PATH:/usr/local/go/bin
Then fix changes:
echo 'export GOPATH=${HOME}/go' >> ~/.bashrc
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc
source ~/.bashrc
Verify that you’ve installed Go by opening a command prompt and typing the following command:
go version
You will see a line similar to:
go version go1.17.6 linux/amd64
Since we are installing Singularity v3.x you will also need to install dep for dependency resolution.
go get -u github.com/golang/dep/cmd/dep
Download, compile and install Singularity:
mkdir -p $GOPATH/src/github.com/sylabs
cd $GOPATH/src/github.com/sylabs
wget https://github.com/sylabs/singularity/releases/download/v3.9.0/singularity-ce-3.9.0.tar.gz
tar -xzf singularity-ce-3.9.0.tar.gz
Compiling and installing Singularity
cd ./singularity-ce-3.9.0 && \
./mconfig && \
make -C ./builddir && \
sudo make -C ./builddir install
After that verify if Singularity Engine is installed correctly by running:
singularity --version
There are several ways to install Git on a Mac. If you’ve installed XCode (or it’s Command Line Tools), Git may already be installed. To find out, open a terminal and enter git --version. If the command returns the git version, then it is installed, otherwise you can install Xcode from the App Store or use the other methods below.
Install Git on a Mac is via the stand-alone installer
Install Git with Homebrew
First you need to have HomeBrew installed, after that follow the next:
2.2.1.1. Setting-up git configuration
Once you have verified and installed git, you have to configure your credentials.
To do that, configure your Git username and email using the following commands, replacing Manuel’s name with your own. These details will be associated with any commits that you create (remember to use the same name and email that you usually register with, for example the one you will use in GitLab):
git config --global user.name "Manuel"
git config --global user.email "manuel@email.com"
Requirements:
MacOS must be version 10.15 (Catalina, Big Sur, or Monterey) or newer and at least 4GB of RAM.
Download Docker Desktop for Mac (for Mac with Intel Architecture) here (approx. 550 MB).
Double-click Docker.dmg to open the installer, then drag the Docker icon to the Applications folder.
Double-click Docker.app in the Applications folder to start Docker (you will see an icon in the Mac bar).
After that, open a terminal and verify the installation was successful by typing:
docker --version
You need to install several programs. This example uses Homebrew but you can also install these tools using the GUI.
First, optionally install Homebrew.
/bin/bash -c "$(curl -fsSL \
https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Next, install Vagrant and the necessary bits (either using this method or by downloading and installing the tools manually).
Update brew repository:
brew update
Install VirtualBox, Vagrant, and Vagrant-manager for Catalina, BigSur:
brew install --cask virtualbox && brew install --cask vagrant && brew install --cask vagrant-manager
Install VirtualBox, Vagrant, and Vagrant-manager for Monterrey:
brew install virtualbox && brew install vagrant && brew install vagrant-manager
Note that here you will be asked for permissions to install some system extension(s)
Open a Terminal and create and enter a directory to be used with your Vagrant VM.
mkdir vm-singularity && cd vm-singularity
If you have already created and used this folder for another VM, you will need to destroy the VM and delete the Vagrantfile.
vagrant destroy && rm Vagrantfile
Then issue the following commands to bring up the Virtual Machine. (Substitute a different value for the $VM variable if you like.)
export VM=sylabs/singularity-3.0-ubuntu-bionic64 && vagrant init $VM && vagrant up && vagrant ssh
You can check the installed version of Singularity with the following:
singularity version
Download the latest Git for Windows installer and then install it.
Then, open a Command Prompt (or Git Bash if during installation you elected not to use Git from the Windows Command Prompt), and verify the installation was successful by typing git --version and checking it’s showing a version greater or equal than 2.34.
2.3.1.1. Setting-up git configuration
Once you have verified and installed git, you have to configure your credentials.
To do that, configure your Git username and email using the following commands, replacing Manuel’s name with your own. These details will be associated with any commits that you create (remember to use the same name and email that you usually register with, for example the one you will use in GitLab):
git config --global user.name "Manuel"
git config --global user.email "manuel@email.com"
Requirements:
Windows 11 64-bit, Windows 10 64-bit, at least 4GB system RAM, and BIOS-level hardware virtualization support must be enabled in the BIOS settings.
Note: Linux kernel for Windows (WSL2) and Docker requires a Windows restart once installed.
Download and install the Linux kernel (WSL2) update package.
Then, download and install Docker Desktop here.
Double-click Docker Desktop Installer.exe to run the installer. Docker Desktop does not start automatically after installation. To start Docker Desktop: Search for Docker and execute it (you will see a Docker icon within the taskbar).
After that verify if Docker is installed, up and running correctly by opening a Command Prompt and running the hello-world image, to do this:
Open a Command Prompt and type the following:
docker run hello-world
Singularity cannot run natively on Windows or Mac because of basic incompatibilities with the host kernel. For this reason, the Singularity community maintains a set of Vagrant Boxes via Vagrant Cloud, one of Hashicorp’s open source tools.
Note: Singularity for Windows requires a restart once installed.
Install the following programs:
Run GitBash (Windows) and create and enter a directory to be used with your Vagrant VM.
mkdir vm-singularity && cd vm-singularity
If you have already created and used this folder for another VM, you will need to destroy the VM and delete the Vagrantfile.
vagrant destroy && rm Vagrantfile
Then issue the following commands to bring up the Virtual Machine. (Substitute a different value for the $VM variable if you like.)
export VM=sylabs/singularity-3.0-ubuntu-bionic64 && \
vagrant init $VM && \
vagrant up && \
vagrant ssh
Note: This command needs some time to complete 😄
You can check the installed version of Singularity with the following:
singularity version
Your version should be greater or equal than 3.0