title

Preface

This handbook contains documentation on using GNU/linux® systems, common workflows in scientific computing and few scripts to helps you during your day to day job. It is intended to support a medium-advanced GNU/Linux® user during the usage of the software. Keep in mind it should be seen as a co-pilot for your workflow, instead of a place to study on, it is designed to be used via keywords search.

How to interpret the text:

StructureMeaning
AbcNormal text
AbcShell command
<Abc>User input required

User management

To check the users listing read the content of /etc/passwd file:

getent passwd | awk -F: '{ print $1}'

User accounts are used not only for actual, human users, but also to run system services and sometimes as owners of system files. This is done because the separation between human users' resources (processes, files, etc.) and the separation between system services' resources requires the same mechanisms under the hood.

Add a user

To add a user to the system execute the following command:

sudo adduser <username>

To give administrator privileges:

sudo adduser <username> sudo    ## in debian based systems
sudo adduser <username> wheel   ## in RHEL based system

then to elude the need to insert the password each time you need to use super-user privileges, excute the command sudo visudo and change its configuration file:

...

# User privilege specification
root    ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL

# Group without password
%nopwd  ALL=NOPASSWD: ALL

Concatenate inside this file the following statememt:

yourUsername ALL=(All) NOPASSWD:ALL

Users system login

Show who is logged into the machine and what they are doing

w

User ownership specification

chown -R admin /opt/script

Groups

To check how many groups are present:

groups

Group management

To add a user into one the above groups, execute:

sudo usermod -a -G <groupName> <username>

While to change the ownership of files/directories:

chgrp -R <groupName> <fileOrDirectory>

Environment management

As environment configuration, it is intened the binary path, library path configuration, plus environment variable saved in the machine state.

Binary path modification

The execution path is essential to permit the system to find the executables in the disks.

echo $PATH

To modify it run:

export PATH="/home/$USER/new/bin/location:$PATH"

Library path modification

As per the execution path, the library location path can be identified and modified at occurrence. To see where the machine is looking for a library:

echo $LD_LIBRARY_PATH

while for the execution path check:

echo $PATH

To add a custom path run for a new library location run:

export LD_LIBRARY_PATH="/path/to/newlibrary:$LD_LIBRARY_PATH"

Directory structure

The root directory (/) is the top-level directory in the file system hierarchy. It is the parent directory for all other directories on the system. Here are some common directories found within the root directory:

Root directoryDescription
/binbinary: conatins binaries essential to the OS
/etcetcetera: contains configuration files for the system and installed applications
/sbinsystem binary: binary accessible only from the sudo user
/usruser directory, contains files (binaries/libraries/shared files/manuals) non essential to the OS
/usr/localcontains manually managed files, hence without using a package manager
/bootcontains files needed for booting the system, such as the kernel and boot loader.
/devdevice files, contains device files, which represent hardware devices on the system
/optoptional
/tmptemporary files, contains homonymous files
/varvariable, contains log and system generated file
/liblibrary, contains essential library files.
/procsvirtual folder created by the kernel to take trace of the running process

Standard streams

Strandard stream allows the machine to interact with the external periferals. Hence they are file that represents streams stored usually in the /dev (device) directory. They can handle input, output and error streams produced by the processes i.e.

  • Standard input|STDIN - file handle that your process reads to get information from you i.e. keyboard or mouse input.
  • Standard output|STDOUT - your process writes conventional output to this file handle.
  • Standard error|STDERR - your process writes diagnostic output to this file handle.

That's mostly by convention .

Examples

executable   <inputfile.txt    2>errorfile.txt  |  grep nice_work

which will:

  • create a process for executable.
  • open ./inputfile.txt as your standard input (file handle 0).
  • open ./errorfile.txt as your standard error (file handle 2).
  • create another process for grep.
  • attach the standard output of executable to the standard input of grep.

Shell scripting

The first line to initialize shell script tells at the system which shell environment has to be used. The following code snippet calls a bash interpreter.

#!/bin/bash

... # bash code

This code can be either sourced with source ./script.sh or execuded in shell ./script.sh

Sourcing shell script

When we call source we means that the contents of the file are read and executed in the current shell environment. source is designed to work with shell scripts (such as those written in bash) and not directly with scripts written in other languages like Python.

Shell script execution

While when we directly execute with the shell, we run a separate process with its own environment.

As an example, the following script call the python interpreter:

#! /usr/bin/python

... # pyhton code

And it can only be executed, not sourced.

When you try to use the source utility on a shell script that has Python code, it will likely result in a syntax error because the Python interpreter does not understand the source command or the shell-specific syntax used in the script.

To summarize, execution runs a script or command as a separate process, while sourcing reads and executes the contents of a file in the current shell session, allowing the changes to directly affect the current environment.

Shell variables

Regarding a shell script written in bash, variables can be expressed as:

name="John"
echo $name  # see below
echo "$name"
echo "${name}!"

Generally quote your variables unless they contain wildcards to expand or command fragments.

wildcard="*.txt"
option="iv"
cp -$options $wildcard /tmp

String quotes

String quotes have different effect on how the text is interpreted.

name="John"
echo "Hi $name"  #=> Hi John
echo 'Hi $name'  #=> Hi $name

Conditional execution

git commit && git push
git commit || echo "Commit failed"

Functions

get_name() {
  echo "John"
}

echo "You are $(get_name)"

Cycles

#!/bin/bash

for region in $(foamListRegions solid);
do
    <commands>;
done

File system

A file system is a programme that recognize the avaible memory of the disks in the machine. Organize, its memory in allocable and dynamic blocks of bits and create a pattern to organize hirarcally the file allocation. Different file systems differ from each other for the way they handle this organization, and the presence of functionality such as expansion in other disk, compression, backup capabilities or corruption check ability.

Mounting a device refers to the process of making a storage device, accessible and usable by the OS file system. A disk is mounted in a specific directory (mount point) in the file system hierarchy. By mounting a device, you make its contents accessible to the operating system and users.

Storage devices such as hard drives, solid-state drives, USB drives, network shares, and remote file systems are represented as device files. These device files are located in the /dev directory and have names like /dev/sda, or /dev/nvme0n1.

The device file acts as an interface between the operating system and the storage device, allowing the system to read from and write to the device.

Disk management

Disk free (df)

Displaying all filesystems and their disk usage in human-readable form can be done through the df utility:

df --human

To check what type of file system you are using use the -T flag as follows:

df -T

To get a good description divided in Megabyte and Gygabyte:

sudo du -cha --max-depth=2 / | grep -E "M|G"

Disk usage (du)

Instead to estimate and summarize file and directory space usage, use the disk usage du utility as follows:

du -hT

A common usage of the du utility is to check the root directory usage as follows:

du -h --max-depth=1 / | sort -n

Enlarge the disk volume

If allocated a new disk, run the following command to list all storage devices in a tree-like format:

lsblk

To enlarge a current disk perform the following commands:

sudo growpart /dev/nvme0n1 1
lsblk

Extended the file system

The commands to extend the file system differ depending on the file system type. For XFS file system usd

sudo xfs_growfs -d /

For Ext4 file system use the correspective:

sudo resize2fs /dev/<nvme0n1p...>

OS Kernel

The kernel takes care of recognizing hardware, networking perifericals and setting up the CPU registers, memory and file system.

The kernel is loaded into memory during the boot process (GRUB direct your CPU at the address at which the kernel is stored) and initializes it.

After the kernel is loaded, it start the first process of the machine namely the init service, responsible for starting and managing other processes in the system.

To check what the kernel is loading after booting, follow a simlar command to parse the first of the process:

ps -ef > process_list && head -n 2 process_list

Which will return a similar result if you are using systemd as your init system.

UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 11:46 ?        00:00:02 /usr/lib/systemd/systemd rhgb --switched-root --system --deserialize=35

Find the current kernel version

The uname utility print information about the machine and operating system is running:

uname -r      # -r stands for --kernel-release

List all installed kernels

Use the rpm command or dpkg command on the terminal:

rpm -q kernel

Remove old kernels

Do not remove the kernel the system is currently running. These examples may result into unstable system if not executed with care. Choose which kernel you want to uninstall from the list of those installed. Type the following command to remove the kernel package under RHEL/CentOS/Fedora Linux:

rpm -e kernel-2.6.12-1.el5

Please note that on newer system all obsolete kernels and headers should automatically be flagged as no more needed, and thus can be purged with the following single command:

sudo dnf autoremove

Change the default kernel

If you experience problems with the latest kernel you can rollback to a previous one (usually your system store more than one in the disks at /boot). Telling grub to point into a different location in memory, i.g.

sudo grubby --info=ALL | grep -E "^kernel|^index"

From the output above select one index and the use it as follows:

sudo grubby --set-default-index=1

Hardware

A big shout out must be given to Lexi Mattick & Hack Club for writing this article on how a CPU works for normal humans. To understand the basics functioning of a CPU cycle we invite to read through the following documentation

Putting the “You” in CPU

  • https://cpu.land/

Hardware scan

Specs of your CPU

lscpu

Return its architecture:

uname -m

where aarch64 stands for ARM64.

Bluetooth peripherical installation

To connect a Bluetooth device, use a systemd shortcut to manage the bluetooth control service:

sudo bluetoothctl

Then:

scan on
#Take note of the device you would like to pair
scan off
pair <deviceCode>
exit

Keyboard settings

Change keyboard layout in Italian

sudo loadkeys it

NVIDIA

Check your Nvidia device:

lspci | grep -i nvidia

Check if CUDA is installed

Run which nvcc to find if the Nvidia CUDA Compiler (NVCC) is installed properly.

which nvcc

You should see something like /usr/bin/nvcc. If that appears, your NVCC is installed in the standard directory.

You can check

nvcc --version

to get the CUDA compiler version, which matches the toolkit version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0

This means that we have CUDA version 11.5.119 installed.

Libraries

A library is code enclosed in external binary files, which contains functionality to be included in a different programme. With a focus on the C programming language, the source code can be a compilaion target for:

  • Static library
  • Dynamic library
  • Executable

.a - Static Library

An archive .a file contains a library that may contain few functions or include an entire library of functions. Archive files are typically created by the GNU ar utility. They are used as a compilation artifact and consequently embedded inside the target binary. The final binary that get produced from this libraries does not depend on the .a to work.

.so - Shared Object | Dynamic Library

A shared object .so file is a shared library used by programs. It contains common programme functions and logic that multiple programs require access to, .so files allow programs to access common functions from one shared place in a computer's system memory, rather than implementing their own versions of the functions. They must be present in the system and reachable from the linked ld.

Dynamic libraries are loaded at runtime, which means that they are not linked into a program at compile time like static libraries.

What's inside a library files

To view a list of the functions an .so file contains, use:

nm objectfile.o
nm -D path/to/filename.so

To understand the dynamic libraries that an executable requires you can run:

ldd </path/to/executable>

Which will print all shared object dependencies.

Where are the libraries?

Libraries manageg by package managerts are usually installed at LD_LIBRARY_PATH. To see it:

echo $LD_LIBRARY_PATH

Often resulting into the following directories:

  • /lib
  • /usr/lib
  • /usr/local/lib

While to add locations execute:

export LD_LIBRARY_PATH="/path/to/your/nanaged/library/directory:$LD_LIBRARY_PATH"

Libraries workflows

A mention to the GNU C Compiler (gcc) is worthwhile. Therefore we few basic source file to explain few concepts.

Preprocessor directives

Preprocessor directives for libraries are used to make the code base aware of additional external code.

#include <mylib.h>

// use your library functions here

These lines are not program statements but directives for the preprocessor. The preprocessor examines the code before actual compilation of code begins and resolves all these directives before any code is actually generated by regular statements.

Source files

Create a touch ~/add.c file such as:

#include "add.h"

double add(double a, double b) {
	return(a + b)
}

With an associated header file:

echo "double add(double x, double y);" > ~/add.h

and a main source file main.c like

#include <stdio.h>
#include "add.h"

double addition_result = add(double 1., double 2.)
printf("The additon result is: %d \n", addition_result);

Compilation procedure

Compilers first transform all the program's source code files into object .o files. The compiler then links the compiled object files into an executable file.

An example is shown below that address the compiler to include header files at the set location (-I), libraries names (-l) located at a give location (-L)

gcc path/to/source.cpp -o ouputName -Ipath/to/header/directory -Lpath/to/library -llibrary_name

.o - Object file generation

An .o file is a compiled C program object. Some C compilers create .o files during the executable creation process. .o files themselves are typically not executable.

Create and use a static library

Then treat ~/add.c as a source file of the library:

gcc -c add.cpp -o add.o

The -c flag is indicating that you want to compile but not link the file to anything else.

Wrap the object as a static library.

In Linux/GNU systems, it involves using the ar application (an archive utility tool) to create the static library file (.a : archive file) such as:

ar -cr libmymath.a add.o

The -cr flag is to indicate creating a new static library file.

Notice the name of output is “libmymath.a”. It is a convention to name the file libXXX.a. The compiler relies on this convention for the linker to work properly.

Compile the main c file into an object file as follwows and link it to the static library to create final executable with the library embedded into it:

gcc main.cpp libmymath.a -o final_executable

# A more explicit and preferred  way to link a library is:
gcc main.cpp -L. -lmymath

The flag -L is used to explicitly specify the library path while -l the library name, as convetion the lib prefix is expressed in this way.

Delete the library for verification

We can verify that the library has been copied into the executable by deleting the library and running:

rm lib_mymath.a 
./a.out

Create and use a dynamic library

Similar to static library, create an object file:

gcc -c my_math.cpp -o my_math.o

Then create a shared object out of it

gcc -shared -o libmymath.so my_math.o

and either move the .so file into the library path, or enlarge the existing one:

export LD_LIBRARY_PATH="/path/to/your/project/root:$LD_LIBRARY_PATH"

Create the executable:

gcc main.o -L. -lmymath -o final_executable

Position independent code

In this example:

mylibrary.so is the name of the output shared library. mysource1.c is the source file you want to compile.

gcc -shared -fPIC -o ouputName.so mysource1.c -lmylib -L/path/to/library

The first two flags stands for:

  • shared: instructs GCC to create a shared library instead of an executable.
  • fPIC: This stands for "Position Independent Code."

When creating shared libraries, it's crucial to generate position-independent code because the library can be loaded into any part of the address space at runtime. -fPIC ensures that the generated code does not depend on a fixed memory location, making it suitable for dynamic linking.

Dynamic library dependencies

Dynamic libraries are loaded at runtime, which means that they are not linked into a program at compile time like static libraries. To understand the dynamic library that an executable requires you can run:

ldd </path/to/the/executable>

Which will print all shared object (*.so) dependencies. Dynamic libraries are often used to p rovide common functionality that is needed by multiple programs, such as system functions, graphical user interface (GUI) libraries, and database connectivity libraries.

Advantages and Limitations of Shared library

Dynamic libraries have several advantages over static libraries:

  • The executabel take up less space, because multiple programs can share the same copy of the library.
  • They are easier to update, because you only have to update the library file itself instead of rebuilding all the programs that use it.
  • They allow programs to use new functionality without being recompiled, as long as the interface to the library remains the same.

Dynamic libraries also have some disadvantages:

  • They can cause runtime errors if the library file is not found or is incompatible with the program.
  • They can be slower to load than static libraries, because they need to be loaded into memory at runtime.

Checkout object files (*.o)

The compilations steps create an object file where symbols represent the entry point for the underline machine code for the linking phase. Symbols are used by functions, variable and all the rest of the machinery needed to link correclty the object (.o) files.

To list symbol names in object files, you can use the:

nm file.o

to list function symbol and respective definition of object files.

Python library

A python library can be source code (to be interpreted) or precompiled. To call a library, it is used the import XXX keyword. The python utility will look in several location for several files.

  • A file named XXX.py or XXX/__init__.py in the directory where python is run.
  • A file named XXX.py or XXX/__init__.py in the PYTHONPATH environment variable, if it is set.
  • A file named XXX.py or XXX/__init__.py in an installation-dependent list of directories configured at the time Python is installed

The last one is where external library are stored usually, to check where the file actually is use the sys python module:

python 
>>> import dolfinx
>>> dolfinx.__file__
'/home/mattia/miniconda3/envs/fenicsx-env/lib/python3.10/site-packages/dolfinx/__init__.py'

Linear Algebra BLAS/LAPACK

LAPACK is written such that the majority of the computation is performed by calls to the Basic Linear Algebra Subprograms (BLAS)

Basic Linear Algebra Subprograms (BLAS) and Linear Algebra PACKage (LAPACK)

As a consequence, you need to install BLAS before LAPACK. Download the packages from the following websites.

  • BLAS: http://www.netlib.org/blas/
  • LAPACK: http://www.netlib.org/lapack/

Locate BLAS Library

Check installation

The first step is to determine where is the BLAS library on your system. Use the command locate libblas.so to find the library. If several results are reported, look for the version usually under /usr/lib/ or /usr/lib64 or something similar to that path. For example, on my machine the library is located at /usr/lib/libblas.so. That location is most likely a symbolic link to another location in your filesystem.

Compile BLAS library and manage the output

Switch to the BLAS folder and execute:

make

to compile all fortran files. After that rename the created library.

mv blas_LINUX.a libblas.a

After creating the library called libblas.a, copy that file to your library folder by executing the following command

sudo cp libblas.a /usr/local/lib/

The above directory can be replaced by any library path in your systems. BLAS and LAPACK are then installed on your system.

When dealing with C/C++ compilation, do not forget to point out your search directory for header files with option “-I”, and add your library path with “-L” for libraries with “-l” if the search paths for the header files and libraries are not included.

PETSc

To compile a programme that include the usage of the library:

PETSC_DIR=$HOME/software/petsc
PETSC_ARCH=linux-gnu-c-debug

# load appropriate modules  (e.g. openmpi)
mpicc -I${PETSC_DIR}/include -I${PETSC_DIR}/${PETSC_ARCH}/include \
      -L${PETSC_DIR}/${PETSC_ARCH}/lib \
      -o test_main \
      main.c

For VS Code user that uses the C/C++ extension for static analysis, few modification should take place to index files/programme correctly on ./.vscode/c_cpp_properties:

{
    "configurations": [
        {
            "name": "PETSc",
            "includePath": [
                "${workspaceFolder}/**",
                "/home/user/Projects/petsc/linux-gnu-c-debug/include",
                "/home/user/Projects/petsc/include"
            ],
            "compilerPath": "/usr/lib64/openmpi/bin/mpicc",
            "intelliSenseMode": "linux-gcc-x86",
        }
    ],
}

Processes

A process is an instance of a particular executable program running. A process can communicate with other devices or services through a network port, moreover it can create other processes which are known as "Child Processes".

The main utilities to investigate processes are shown below:

ss - Investigate network sockets.
ps - Information all running processes.
nc - Working with TCP or UDP data.

Scan ongoin processes

To report a snapshot of the current processes use:

ps -ef

It scan all processes that are running on your system (not all processes use a network port). To check processes by name the following utilts will return the process ID:

pgrep <process_name>

Manage process sockets

To see processes state, socket they are using and check if they are listening on it, use the ss command:

sudo ss -lptn

Which will returning this table header:

State      Recv-Q Send-Q      Local Address:Port       Peer Address:Port
  • Recv-Q : This field represents the amount of data that is currently in the receiving queue (buffer) of the socket. It indicates the number of bytes that have been received by the socket but not yet read by the application.
  • Send-Q : This field represents the amount of data that is currently in the sending queue (buffer) of the socket. It indicates the number of bytes that have been sent by the application but not yet acknowledged by the remote peer.
  • Local Address:Port : This field represents the local network address and port number associated with the socket. The local address refers to the network interface on the local machine that the socket is bound to. The port number is a unique identifier that allows multiple connections to exist simultaneously on a single machine.
  • Peer Address:Port : This field represents the network address and port number of the remote peer (the other end of the connection) that the socket is connected to. It identifies the destination or source of the connection.

Commonly used in conjuction with the grep command:

sudo ss -lptn | grep <processDescriptionOrPort>

Stop processes

To kill/stop all processes which match their full command instead of writing the process name use the -f flag:

pkill -f "command_name"

Another interesting flag is -9; it refer to the signal that we will send to the process, and we pass in the process ID within that process group we want to kill.

The signals you can send are:

  • 1 — HUP (hang up)
  • 2 — INT (interrupt)
  • 3 — QUIT (quit)
  • 6 — ABRT (abort)
  • 9 — KILL (non-catchable, non-ignorable kill)
  • 14 — ALRM (alarm clock)
  • 15 — TERM (software termination signal)

Threads

Processes are binaries instructions that are dispatched from the ready state and scheduled in the CPU for execution. The PCB(Process Control Block) holds the concept of process. The process takes more time to terminate and it is isolated means it does not share the memory with any other process.

The process can have the following states:

  • new
  • ready
  • running
  • waiting
  • terminated
  • suspended

Meanwhile, a thread is the segment of a process which means a process can have multiple threads and these multiple threads are contained within a process. A thread has three states:

  • new
  • runnable
  • blocked (this state do not use any cpu)
  • terminated

The thread takes less time to terminate as compared to the process but unlike the process, threads do not isolate memory.

Services and Init System

A service (or daemon), is an example of a process, but it has some additional properties:

  • It runs in the background, in the sense that it has no direct interaction with the user
  • It is often started during bootup, or the first time the service is required, and after that it often stays running indefinitely, it is never finished.
  • They are often used for events that might happen at any time and when they do, need handling of some kind. An example is the internet daemon, which handles incoming network traffic.

The defacto standard GNU/Linux system for a system/service manager and initialization tool is named systemd.

systemd

systemd is responsible for starting and managing services. Often for a user, the interface is the sevice management system via the systemctl utility.

systemctl

Is a command-line utility used to control the system service manager (responsible for starting, stopping, and managing system services). systemctl can be used to start, stop, restart, enable, or disable services, as well as view the status and log files of those.

Here are some examples of common systemctl commands:

systemctl start <service> 	# start a service
systemctl stop <service> 	# stop a service
systemctl restart <service> # restart a service
systemctl enable <service>  # enable a service to start automatically at boot
systemctl disable <service> # disable a service from starting automatically at boot
systemctl status <service>  # check the status of a service
systemctl list-units --type=service    # list all available services
systemctl cat <service>     #view the configuration file for a service

You may have noticed that the system services end with .service extension, these are nothing but files that are used to define a service in systemd. systemctl is smart enough to understand that you’re looking for a service and correctly display the status, however, you can add in the .service extension as a good practice.

For example, you can view the contents of the service file like this:

systemctl cat postgresql.service

To restart systemd to be sure that all deamons are laoded:

systemctl deamon-reload
systemctl restart <service>.service

Instanciate services

Usually, system services run at boot and they keep running in background, to run the machine properly. You can instantiate a new service to start at boot, storing the executable in:

/etc/init.d

And then running the following:

chkconfig –add <executable>
chkconfig <exec> on

Networking

There are various tools available for managing network settings and connections. Some common tasks related to networking include:

Managing network connections:

You can use the nmcli (Network Manager CLI) utility to manage network connections. It allows to configure the network settings for various types of connections, such as Ethernet, WiFi, and VPN.

nmcli connection show
# or if the device is not known
nmcli dev wifi list

To connect to one of the listed networks run:

nmcli device wifi connect "Your SSID" password "network-password"

Viewing networks statistic:

Network interfaces are the physical or logical connections that allow a device to communicate with other devices on a network. You can use the ifconfig utility to view network interfaces, while with the netstat command you can view the network statistics, such as the number of packets sent and received.

Debugging network issues:

If you encounter any issues with your network connections, you can use the ping, traceroute, and mtr commands to troubleshoot the problem. These commands allow you to test the connectivity and performance of a network connection.

IP Address

An IP address is a unique numerical label assigned to every device connected to a network, for being identified.

Private

192.168.0.0 is the beginning of the private IP address range (that includes all IP addresses through 192.168.255.255) assigned by . One common IP address assigned to home routers is 192.168.1.1. This IP address is used because the router is on the 192.168.1.0 network. In the same way, routers on the 192.168.0.0 network are usually assigned the local, private IP address of 192.168.0.1.

To find out your private IP address, run:

ip route get 1.1.1.1 | awk '{print $7}'

If you are interested into visualize the private IP addresses of your network run:

nmap -sP 192.168.1.0/24

To return a panoramic of the system you want to investigate

nmap -sT -O <private_IP_address>

An address like 192.168.0.0 becomes unusable for any other purpose after it's established as a network number. If an administrator assigns 192.168.0.0 to any device on the network as a static IP address, the network stops functioning until that device is taken offline.

Local IP address

localhost is the default name of the computer you are working on. The term is a pseudo name for the IP address127.0.0.1. This IP address allows the machine to connect to and communicate with itself. Therefore, localhost (127.0.0.1) is used to establish an IP connection to the same device used by the end-user.

Although using 127.0.0.1 is the most common practice, the IPv4 network standard reserves the range 127.0.0.1 – 127.255.255.255 for localhost. Using another IP address within the range results in the same or similar manner. The IPv6 standard assigns one address to loopback - :: 1.

Public

A public IP address is a unique numerical label assigned to every device connected to the Internet. It allows devices to communicate with each other and with servers over the Internet. Your public IP address is the address that is assigned to your device by your Internet Service Provider (ISP) and is used to identify your device on the Internet. It is a unique identifier for your device, similar to your home address, and is used to route traffic to your device from other devices on the Internet.

You can find out your public IP address by using:

curl ifconfig.co

The public IP address can change over time, especially if you have a dynamic IP address assignment from your Internet Service Provider (ISP). If you have a dynamic IP address, it may change every time you connect to the Internet or after a certain period of time. If you have a static IP address, it will remain the same unless you change it manually or through your ISP.

Sockets

A socket is a endpoint for network communication in a computer network. It represents a combination of an IP address and a port number that uniquely identifies a specific process or service on a networked device. A socket acts as a bidirectional communication channel between applications, allowing them to send and receive data over the network. Sockets are used in many networking protocols, such as TCP, UDP, and SCTP.

Connect to a socket

nc -vz <host> <port>

Check socket reachability

From client side, you can use the netcat (nc) utility. The syntax to check a network socket is:

nc -vz <host> <port>

Where -v flag strand for verbose and -z activate the port scanner mode, which only listen services are scanned (no data is sent) From a Windows client side, to see if a socket can be reached use:

telnet <ipAddressServer> <portNumber>

Manage sockets from client

The nc (or netcat) utility is used for just about anything under the sun involving TCP, UDP, or UNIX-domain sockets. The syntax is:

# Listen on a specified port and print any data received:
nc -l port

# Connect to a certain port:
nc ip_address port

# Keep the server up after the client detaches:
nc -k -l port

# Keep the client up even after EOF:
nc -q timeout ip_address

# Act as proxy and forward data from a local TCP port to the given remote host:
nc -l local_port | nc hostname remote_port

Firewall

First check which firewall program is installed in your system:

sudo which nft >/dev/null && echo nftables is enabled in this system || echo ufw is enabled in this system

Open ports

If ufw (uncomplicated firewall) is the firewall program enabled in your machine, execute the following command to open a different port, replacing the PORT placeholder with the number of the port to be opened:

ufw status
ufw allow 8080/tcp
ufw status
sudo systemctl restart ufw.service

If nft is the firewall program enabled in your machine, modify the /etc/nftables.conf file and add the following line inside the chain inbound block, replacing the PORT placeholder with the number of the port to be opened:

...

  chain inbound {
      ...
      tcp dport PORT accept
  }
...

Make

make is a build system, designed to automate tasks. It reads a Makefile which is structured as follows, remember that indentation matter. A Makefile follows this pattern:

final_executable: main.c
	$(CC) main.c -o final_executable -Wall -Wextra -pedantic -std=c99

Where the first line says that final_executable is what we want to build, and main.c is required to be present to start the build procedure. The second line specifies the command to run in order to actually build final_executable out of main.c. Looking at the flags:

- ```$(CC)``` is a variable that make expands to cc by default.
- ```Wall``` stands for all warnings,
- ```Wextra``` and ```pedantic``` turns pon even more warning 
- ```std=c99``` sspecifies the standard

Makefile

A dive in into the make utility is useful to understand the step to compile a library/program. An example of a Makefile follows:

final_executable: libmymath.so main.o
	g++ main.o -L . -l_mymath -o final_executable

lib_mymath.so: my_math.o
	g++ -shared -o libmymath.so my_math.o

my_math.o: mymath.cpp
	g++ -c my_math.cpp -o my_math.o

main.o: main.cpp
	g++ -c main.cpp -o main.o

clean:
	rm -r main.o my_math.o

There are lots of different makefile solutions but a simple config might look like the follows:

SRCS = main.o mario_game.o sprites.o sfx.o
OBJS = $(SRCS:.cpp=.o)
EXE = mario_game

$(EXE): $(OBJS)
    $(CC) $(CFLAGS) -o $(EXE) $(OBJS)

.cpp.o:
    $(CC) $(CFLAGS) -c $< -o $@

Overcoming Make weirdness

The .PHONY target is a special target that is used to specify a list of counterfait targets. Phony targets do not represent actual files, but rather represent an action or command that needs to be executed regardless.

.PHONY: clean

clean:
    rm -rf *.o

On the above example, clean is declared a phony target, and it will be always executed regardless if a file name clean exist.

Running multiple commands at once

What if we want to run the tests, and if they pass, then deploy our code? You can combine commands in long chains. If one fails, the rest of them going to be skipped.

make test deploy

Making Make less verbose

When you don't want a command to be printed, and just want Make to execute it, prepend it with @ :

hello:
 @echo “Hi, Bob!”

Package

A package is a collection of files. Then, a package manager takes care of its installation, with a well defined set of rules to place the files into the correct place within the system.

Package manager

Using Debian based systems, the package manager is:

apt --version

To understand the specific files/changes made by apt, you can use the dpkg command (debian packages only), which is the underlying package management tool that apt utilizes. The dpkg command allows you to query information about installed packages, including their files, configuration details, and more.

dpkg -L <package_name>

Install software without the help of the package manager

Once you have obtained a software without using the package manager (i.g. downloading a pre-build version of it or after having built it ), you have to decide where to store the artifacts (binaries and libraries). A plausible option is to organize the files in the /opt directory and then provide a symbolic link into an executable path or library path such as /usr/local/... (directory location used to store manually built binaries[/usr/local/bin]/library[/usr/local/lib]).

In the following snippet, the HDFView programme is placed and linkd in the above mentioned location.

# HDFView is a directory structure containing binaries
sudo mv HDFView /opt
sudo ln -s "/opt/HDFView/bin/HDFView" /usr/local/bin/HDFView

SSH

SSH (Secure Shell) is a network protocol that allows secure remote login and other secure network services over an unsecured network. It is commonly used to log into servers remotely, transfer files between systems, and execute commands on remote systems.

To use instanciate an SSH connection, you need an SSH client and the IP address of the SSH server. Before any connection take place, install the necessary packages on client:

sudo apt install openssh-server net-tools

Configuration

You can find the configurations of the service at sudo vim /etc/ssh/sshd_config. An option to always check is: port 22. Then ensure the deamon is running, via systemd:

systemctl ssh start

SSH tunnelling

SSH tunneling (also known as SSH port forwarding) is a way to create a secure connection between a local computer and a remote server by tunneling through an intermediate server. It allows you to access resources on the remote server as if you were directly connected to it, even if the connection between your local system and the remote server is not directly accessible.

To set up an SSH tunnel, you will need to specify the local and remote port numbers that you want to use for the tunnel, as well as the intermediate server that you want to use as the tunnel. You can then use your SSH client to connect to the intermediate server and create the tunnel.

ssh -R 8001:<ipAddressMachine1>:8002 <userName>@<ipAddressMachine2>

In this particular example if we connect to the machine 1 at port 8001, we are connected to machine 2 via port 8002.

Key management

Creta a brand new SSH keys pair:

ssh-keygen -t ed25519 -C <your_email@example.com>

This will genrate a key pair in ~/.ssh. However to make a public key out of the private one (usually to reuse the key for different user of machines):

ssh-keygen -y -f ~/path_to_key/<key> ~/path_to_key/<key>.pub

Then run the following commands to transfer the keypairs of the system you know the credential:

ssh-copy-id -i ~/path_to_key/<key> user@hostaname

This will generate a .ssh/authorized_keys file with the credential filled for the specified user. Permitting the usage of the key instead of the password.

Automated connection

Save the credential for automating access into ~/.ssh/config, and write a similar content to the following:

Host machine1
    HostName 123.311.23.24
    User mattia

Host machine2
    HostName example.com
    User root

Host machine3
    HostName 64.233.160.0
    User mattia
    Port 56000

Host machine4
    HostName 123.31.23.23
    User ec2-user
    IdentityFile /path/to/special/privatekey/ssh-key.pem

Then you can directly connect using:

ssh machine1
# Or ssh machine2 3 or 4 depending where you would like to connect

HPC

HPC systems are typically made up of a large number of processors (such as CPUs or GPUs) that work together to perform calculations in parallel. They may also have a lot of memory and fast interconnects to allow the processors to communicate and share data quickly.

HPC users connection

On a server side, once a new user is set up, create a /home/<newUser>/.ssh/authorized_keys file.

mkdir ~/.ssh
chmod 700 .ssh

The next command will generate a pair key, namely private (never to be shared) and a public key:

ssh-keygen –y
chmod 600 .ssh/authorized_keys
chmod 400 <sshKeyName>.pem

Select the .pem and copy the output in the /home//.ssh/authorized_keys

cat <sshKeyName>.pem > /home/<user>/.ssh/authorized_keys

After that as a sudo user add a line at the end of /etc/ssh/sshd_config

addUsers <userName1> <userName2> <userName3> …
addGroups sudo  # If your account has sudo privileges

And to make reachable this user via the SSH protocol without rebooting the machine, restart the SSH service:

sudo systemctl restart sshd

Add path to all the users

When a new user is created, it can happen that some applications are not recognized as installed, Make all the users aware of the installed executables working with the file /etc/profile which will open the files /etc/profile.d/*

With the user (it is likely that root return this) that can use the executable run:

which <executableYouNeed>

Add this path into the /etc/profile.d/path.sh using

sudo nano /etc/profile.d/path.d 
PATH_REQUIRED_DIRECTORIES=(… … <pathToTheExecutableYouNeed>)
source /etc/profile

Hardware description

sudo apt-get -y install hwloc

The below will show how the architecture of your machine looks like:

lstopo

Environment management

Especially in HPC environments, the system is clogged with applications and libraries that cannot be all included at the same time into an environment path in a mantainable way. A way to include only a necessary portion of paths is via the lmod utility:

lmod (Environmental module system)

lmod provides a convenient way to dynamically change a user's environment through modulefiles.

A modulefile contains the necessary information to allow a user to run a particular application or provide access to a particular library. This includes easily adding or removing directories to the PATH environment variable.

  • Modulefiles for applications modify the user's path to make access easy.
  • Modulefiles for Library packages provide environment variables that specify where the library and header files can be found.

Usage

Check which modules are loaded into the system:

module list

Check which module are available to the system:

module avail

To get an overview of the environment modifications, run:

module show <module_name>

Adding modulefile

To write a custom loadable environment you can add a modulefile. These file must be present on MODULE_PATH following:

touch $MODULE_PATH/<modulefile_name>.lua

The content of the modulefile you state should look similar to:

help([[
For detailed instructions, go to:
   https://...

]])
whatis("Version: 5.0.1")
whatis("Keywords: System, Utility")
whatis("URL: http://content.allinea.com/downloads/userguide.pdf")
whatis("Description: Parallel, graphical, symbolic debugger")

setenv(       "DDTPATH",        "/opt/apps/ddt/5.0.1/bin")
prepend_path( "PATH",           "/opt/apps/ddt/5.0.1/bin")
prepend_path( "LD_LIBRARY_PATH","/opt/apps/ddt/5.0.1/lib")

Loadable with:

module load <modulefile_name>

Different package version

Sometimes the need for a different package version arise. The best way to deal with is with the usage of the package manager spack. spack is non-destructive, that's imply that installing a new version does not break existing installations, so many configurations can coexist on the same system.

Spack installation

Designed for HPCs, where users share common installations of software with exotic architectures, using libraries that do not have a standard Application Binary Interface (ABI). To install it follwows:

git clone https://github.com/spack/spack.git
source ~/spack/share/spack/setup-env.sh        # Shell support for bash/zsh

Sourcing will place spack command in your PATH, set up your MODULEPATH to use Spack's packages, and add other useful shell integration for certain commands, environments, and modules. To install an application (i.g. gcc):

spack info gcc
spack install gcc@12.1.0 # We are installing a specific version

Then load and run the binary as follow:

spack load gcc@12.1.0  # Loading the module
gcc --version

If you don’t specify the version of a package, Spack will install the default version of that package.

Multiple users, same packages

For users to be able to use shared packages provided by the system administrators and to create/ install their own packages with spack , one of the ways is to create a chained spack installation. To do that, individual users will need to install a private copy of spack in their home directory and connect it with the shared (“upstream”).

Install local copy of spack

Log on to the cluster as normal user and clone the repository: and load the local environment:

git clone https://github.com/spack/spack.git
source spack/share/spack/setup-env.sh

Chain to a share spack instance

Note the location of the shared install (make sure the directory is readable by users):

SPACK_SHARED_DIR=/home/master_spackuser/spack

Create file touch ~/spack/etc/spack/upstreams.yaml with following contents:

upstreams:
  spack-shared:
    install_tree: $SPACK_SHARED_DIR/opt/spack

The chained spack is now ready to use. You can check that all upstream compilers and packages are available:

MPI (Message Passage Interface)

MPI is a popular library for distributed-memory parallel programming in SPMD (single program, multiple data) style.

MPI offers both point-to-point message passing and group communication operations (broadcast, scatter/gather, etc).

Several implementations of MPI are available (OpenMPI, MPICH, InteMPI, etc...).

Compilation:

Regular applications

gcc test.c -o test

MPI applications:

mpicc test.c -o test

Execution:

Regular applications

./test

MPI applications (running with 16 processes):

mpiexec –n 16 ./test

System setup

Here it is presented an example on how to set up a Fedora system with MPI compilers wrapper. you’ll need to install the openmpi package, as well as set up the envionrment paths correctly (the module utility becomes handy).

To install a packaged library:

sudo dnf install openmpi openmpi-devel

Now you can load in the OpenMPI module, prepared as indicated by the lmod utility

module load mpi/openmpi-x86_64

With the module now loaded the system environment is modified, so you can use the compiler tools and runner. Do note that you’ll have to source and load the OpenMPI module for every shell you open unless you add it within $HOME/.bashrc.

SLURM scheduler

Prepare a programme that can run on a distributed computin system.

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main() {
    int  step, node, hostlen;
    char hostname[256];
    hostlen = 255;

    MPI_Init(NULL, NULL);
    MPI_Comm_rank(MPI_COMM_WORLD, &node);
    MPI_Get_processor_name(hostname, &hostlen);

    for (step = 1; step < 5; step++) {
        printf("Hello World from Step %d on Node %d, (%s)\n", step, node, hostname);
        sleep(2);
    }

    MPI_Finalize();

    return 0;
}

Compile it with the following:

mpicc main.c -o mpi_exec

Then you can instruct the scheduler on how to run the programme, such as:

#!/bin/bash
#SBATCH --job-name=hello-world-job
#SBATCH --ntasks=6
#SBATCH --output=%x_%j.out

mpirun ./mpiexec

To start the Slurm scheduler alongside with the script that specifies the resources needed and the commands to be executed:

sbatch directives_slurm.sbatch

General utilities

Instead to run directly a single job or a script on allocated compute nodes, use:

srun

This can be useful for quick tasks or testing purposes As the name suggests, the following command allows you to cancel submitted jobs. It can be used for jobs that are pending, running, or even to send signals to running jobs

scancel

The following command is for checking the status of your submitted jobs. It displays information like job ID, name, user, partition, state, and resource use:

squeue

Instead the next command provides details about the available compute nodes and partitions (queues) on the Slurm cluster. It's helpful for understanding the resources available and choosing the right nodes for your jobs

sinfo

SLURM Installation

First install munge

Munge

# Verified user
sudo useradd --system --no-create-home --shell /usr/sbin/nologin --user-group munge
grep munge /etc/passwd
grep munge /etc/group

# Get tarball of munge
tar xJf munge-0.5.16.tar.xz
cd munge-0.5.16
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --runstatedir=/run
make
make check
sudo chown munge:munge /usr/sbin/mungekey
sudo chown -R munge:munge /etc/munge /var/lib/munge /var/log/munge /run/munge
sudo -u munge /usr/sbin/mungekey --verbose
sudo systemctl start munge.service
sudo systemctl enable munge.service
sudo /etc/init.d/munge start

SLURM

wget https://download.schedmd.com/slurm/slurm-24.11.0.tar.bz2
tar -xaf slurm*tar.bz2 
sudo apt-get install build-essential fakeroot devscripts equivs
cd slurm-24.11.0
sudo mk-build-deps -i debian/control
sudo  debuild -b -uc -us # Build the packages
cd ..
dpkg -i on_necessary_packages

# Write the config
micro /etc/slurm/slurm.conf
sudo mkdir -p /var/spool/slurmctld

sudo systemctl start slurmctld                                                     
sudo systemctl start slurmd

The packages will be in the parent directory after debuild completes.

Test

Given this script:

#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --output=output_%j.log
#SBATCH --error=error_%j.log
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G

echo "Date start                = $(date)"
echo "Initiating Host           = $(hostname)"
echo "Working Directory         = $(pwd)"
echo ""
echo "Number of Nodes Allocated = ${SLURM_JOB_NUM_NODES}"
echo "Number of Tasks Allocated = ${SLURM_NTASKS}"
echo ""

echo "Hello from host"

RETURN=${?}

echo ""
echo "Exit code                 = ${RETURN}"
echo "Date end                  = $(date)"
echo ""

run the below command:

chmod +x script.sh
sbatch script.sh

Web

The web communication work thanks to a procol implementation: HTML. Which stands for Hyper Text Tranfer Protocol.

HTTP

A “Protocol” is a definition of behaviour between entities, in this case client and server.

“Protocol Transfer” stands for defining the rules of communication with the purpose of transferring data.

“Hyper Text Tranfer Protocol” define a protocol to transfer hyper text

An http connection is stateless, which means there is no record of previous interaction, if you want to keep the login state you need to include your login data at every request (on the Header) from a client point of view.

HTTP Verbs

There are 4 basic and of common usage HTTP verbs for having interactions over a network that supports a REST system (architectural style for providing standards between machines):

  • GET — retrieve a specific resource (by id) or a collection of resources
  • POST — create a new resource
  • PUT — update a specific resource (by id)
  • DELETE — remove a specific resource by id

For each HTTP verb, there are expected status codes a server should return upon success:

  • GET — return 200 (OK)
  • POST — return 201 (CREATED)
  • PUT — return 200 (OK)
  • DELETE — return 204 (NO CONTENT)

If operations fail, the return status code should be specific to the problem that was encountered.

Other less common verbs are :

REST APIs act on top of the HTTP protocol hence the possile actions that is possible to communicate are:

  • GET: The GET method requests a representation of the specified resource. Requests using GET should only retrieve data.
  • HEAD: The HEAD method asks for a response identical to a GET request, but without the response body.
  • POST: The POST method submits an entity to the specified resource, often causing a change in state or side effects on the server.
  • PUT: The PUT method replaces all current representations of the target resource with the request payload.
  • DELETE: The DELETE method deletes the specified resource.
  • CONNECT: The CONNECT method establishes a tunnel to the server identified by the target resource.
  • OPTIONS: The OPTIONS method describes the communication options for the target resource.
  • TRACE: The TRACE method performs a message loop-back test along the path to the target resource.
  • PATCH: The PATCH method applies partial modifications to a resource.

SSH

SSH (Secure Shell) is a network protocol that allows secure remote login and other secure network services over an unsecured network. It is commonly used to log into servers remotely, transfer files between systems, and execute commands on remote systems.

To use instanciate an SSH connection, you need an SSH client and the IP address of the SSH server. Before any connection take place, install the necessary packages on client:

sudo apt install openssh-server net-tools

Configuration

You can find the configurations of the service at sudo vim /etc/ssh/sshd_config. An option to always check is: port 22. Then ensure the deamon is running, via systemd:

systemctl ssh start

SSH tunnelling

SSH tunneling (also known as SSH port forwarding) is a way to create a secure connection between a local computer and a remote server by tunneling through an intermediate server. It allows you to access resources on the remote server as if you were directly connected to it, even if the connection between your local system and the remote server is not directly accessible.

To set up an SSH tunnel, you will need to specify the local and remote port numbers that you want to use for the tunnel, as well as the intermediate server that you want to use as the tunnel. You can then use your SSH client to connect to the intermediate server and create the tunnel.

ssh -R 8001:<ipAddressMachine1>:8002 <userName>@<ipAddressMachine2>

In this particular example if we connect to the machine 1 at port 8001, we are connected to machine 2 via port 8002.

Key management

Creta a brand new SSH keys pair:

ssh-keygen -t ed25519 -C <your_email@example.com>

This will genrate a key pair in ~/.ssh. However to make a public key out of the private one (usually to reuse the key for different user of machines):

ssh-keygen -y -f ~/path_to_key/<key> ~/path_to_key/<key>.pub

Then run the following commands to transfer the keypairs of the system you know the credential:

ssh-copy-id -i ~/path_to_key/<key> user@hostaname

This will generate a .ssh/authorized_keys file with the credential filled for the specified user. Permitting the usage of the key instead of the password.

Automated connection

Save the credential for automating access into ~/.ssh/config, and write a similar content to the following:

Host machine1
    HostName 123.311.23.24
    User mattia

Host machine2
    HostName example.com
    User root

Host machine3
    HostName 64.233.160.0
    User mattia
    Port 56000

Host machine4
    HostName 123.31.23.23
    User ec2-user
    IdentityFile /path/to/special/privatekey/ssh-key.pem

Then you can directly connect using:

ssh machine1
# Or ssh machine2 3 or 4 depending where you would like to connect

REST API

A REST API (Representational State Transfer API) is a type of API that follows a set of architectural principles for building web services. REST APIs use HTTP verbs (such as: GET, POST, PUT, DELETE) to define the operations that can be performed on resources. Each resource is identified by a unique URL, and the API defines the operations that can be performed on that resource and the data that can be exchanged.

An API can be considered REST when it respects the below constrains.

  • Statelessness: REST APIs are stateless, which means that they do not store client state in the server. This allows the API to be more scalable, because it does not need to maintain a session state for each client.
  • Cacheability: REST APIs are designed to be cacheable, which means that they can be stored in a cache and reused to reduce network traffic and improve performance.
  • Layered system: REST APIs can be used by multiple clients and servers, and they can be layered on top of each other to create a distributed system.

REST APIs are often used to build APIs for web-based applications, mobile apps, and other systems that need to access data or services over the Internet.

Licence server

Almost evert floating licence, if does not have a proprietary licence manger will use lmgrd and the utils that comes with it

./lmgrd -c <license_file>.lic -l <license>.log

Remember to open the port you specify in the licence file.

Utilities

A list of a useful software for UNIX like environment. For more complex tools follow the continuation of this chapter.

tldr

Abbreviation for "Too Long; Didn't Read"; it will return a better man page. It return a coincise manual page, with suggestion and brief explanation of the utility you are searching for.

ltdr <utilityName>

curl

Which stands for "Client for URL (Uniform Resource Locator)". It transfers data from or to a server. Supports most protocols, including HTTP, FTP, and POP3. For example to test a REST-API you can run:

curl https://pokeapi.co/api/v2/pokemon/ \> pokemon_return_rest_api.json

Git

Git is a programme that does version control on code bases. The usual workflow is to download (clone in the git jargon) the code, apport modification (commit) and upload the code (push).

If you already have a git server, configure the client running:

git config --global user.email "<yourEmail>"
git config --global user.name "<yourName>"   # Otherwise you can configure the account with your username

Get a repository from server

Once cloned the repository you are interest with on your local device:

git clone <sshAddress>  # Or repo URL

Using the SSH protocol, you must have a key the access the repository if the repository is private. To get the last version of the remote code, run:

git pull

Create a repository

Initialized empty Git repository locally,

git init -b main
git add . && git commit -m "initial commit"

Look at the local branches in your repository:

git branch

To see all other branches hiding in your repository and you can use the --all/-a flag.

git branch --all

If you want to work on a specific branch, you'll need to create a local tracking branch which is done by:

git checkout <branchName>

Modify the repository branch

First, always check the status of the local repository to see which file changed and which has not:

git status

To not consider the changes made since the last commit, you can stash your changes with:

git stash

However if the data are needed in the future, you can recover the last stashed repository state with:

# Only the last stashed modifications 
git stash pop

Add code

Add some code and then do a commit into the base level of the repository. Stages all changes:

git add . 
# Or a particolar file 
git add path/to/file

Commit code

After having added new file/modifications into the git queue. You can commit these with:

git commit -m “comment”

A commit file will appear where you need to make visible the changes you want to commission.

Manage branches

To create a new branch, execute the following:

git branch <newBranch>

Remove a branch run one of these two command:

# Remote branch
git push -d <remoteBranchName> <branch_name>
# Local branch
git branch -d <localBranchName>

the remote_name is usually: origin

Contribute to a project - GitHub

You can contribute to an existing project, opening a pull request. The overall process should look somewhat like this:

  1. Fork the repo.

  2. Clone your fork locally: git clone https://github.com/your_username/forked_project.git

  3. Create a feature branch, e.g. named after the command you plan to edit: git checkout -b branch_name

It is bad practice to submit a pull request from the main branch of your forked repository.

  1. Make your changes

  2. Commit the changes: git commit --all -m "commit_message"

  3. Push the commit(s) to your fork: git push origin branch_name

Please avoid force-pushing since it makes the review process harder.

  1. Go to the GitHub page for your fork and click the green "Compare & pull request" button.

GitHub SSH keys

To acceed in Github using the SSH protocol, you need to make aware the platform of your public SSH key, which you can generate with:

ssh-keygen -t ed25519 -C <your_email@example.com>

Git worktree

A bare git repository is needed to build a worktree:

git clone --bare <repositoryAddress>

To create a worktree that is based on an existitng branch run:

cd repositoryAddress.git 
git worktree add --checkout <remoteBranch>

Conda

Conda is a general purpose package manager widely used in the scientific community. Other than that, it serves as an envirnoment management tool to provide encapslutated environment where specific programmes, variables and libraries are defined.

To check the presence of Conda environment execute:

conda info --envs

Then to activate one of them with:

conda activate <environment_name>

C/C++ develpment with Conda

An example of how to use Conda to get a C/C++ development environment.

Package managers usually focus on delivering the libraries that we use in our project, but there are actually many more, they can be useful at different time and for different purposes (non exclusively):

  • to build (compile) our project,
  • to run our project (our executable our library),
  • to test our project,
  • to benchmark our project,
  • to generate the documentation for our project,
  • to perform quality analysis of our project (e.g. code check and formatting).

Example creation C dev environment

Create a file called environment.yaml as follows:

channels:
  - conda-forge
dependencies:
  - cxx-compiler
  - make
  - cmake
  - boost-cpp

Then create and activate the environemnt:

conda env create --name c_dev --file environment.yaml
conda activate c_dev

If you would like to check what is installed in the environment:

conda list

Web server

A web server is a program that serves web pages to clients through web request. When a client, such as a web browser, makes a request to a web server, the latter sends the requested packages back to the client.

Web servers can be set up to host a website, and they are often used to store and manage the files that make up a website, as well as to process requests from clients and serve appropriate files or data in response.

Python https module

Using the http.server module, running the following command in a terminal:

python -m http.server

This will start an HTTP server on your local machine and listen for incoming requests on port 8000.

Nginx

The following install nginx, which is a web server that runs as a deamon.

sudo dnf install nginx

Nginx is managed by service management system (systemd) hence using the systemctl utility nginx can be controlled in several different ways. Such as:

sudo systemctl <systemctlAction> nginx

Nginx settings

Setting how nginx is going to work is done though /etc/nginx/nginx.conf Then prepare a config file to serve you site, using as a base a config file already present in /etc/nginx/conf.d. Adding a new .conf file and restarting the service:

sudo vi /etc/nginx/conf.d/<yourWebsiteConfigFile>.conf
sudo systemctl restart nginx

As an example of a config file, pointing to an html file to be served:

server {
    server_name    gnulinux-handbook.adigecalculations.com www.gnulinux-handbook.adigecalculations.com;
    root           /var/www/html/OpenFOAM-handbook/book;
    index          index.html;
    proxy_set_header Host      $host;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/gnulinux-handbook.adigecalculations.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/gnulinux-handbook.adigecalculations.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host = gnulinux-handbook.adigecalculations.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen         80;
    listen         [::]:80;
    server_name    gnulinux-handbook.adigecalculations.com www.gnulinux-handbook.adigecalculations.com;
    return 404; # managed by Certbot
}

Verify the syntax of your configuration edits with:

sudo nginx -t

Next, an example on how to insert reverse proxy to an internal working webserver listening at port 8000:

server {
    server_name    fc.adigecalculations.com www.fc.adigecalculations.com;
    location / {
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_pass http://localhost:8000;
     }

    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/fc.adigecalculations.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/fc.adigecalculations.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    if ($host = fc.adigecalculations.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;
    server_name    fc.adigecalculations.com www.fc.adigecalculations.com;

    listen [::]:80;
    return 404; # managed by Certbot
}

Certificates management

To use a HTTPS protocol the web-server must be able to send off certificates to proof that you are the owner of the IP address.

Installing certificate autority client with Nginx feautures

The first step to using Let’s Encrypt to obtain an SSL certificate is to install the certbot software on your server. You can obtain the certbot-nginx package by typing:

sudo dnf install certbot-nginx

Set/Grub Nginx addresses

Using a client to communicat with Let's Encrypt (a certificate authority) called certbox, it is possible to automatically configure SSL for Nginx. It does this by looking for the server_name directive that matches the domain you’re requesting a certificate for.

Find the existing server_name line in /etc/nginx/nginx.conf such as:

server_name adigecalculations.com www.adigecalculations.com;

If that runs with no errors, reload Nginx to load the new configuration:

sudo systemctl reload nginx

Certbot will now be able to find the correct server block and update it. Then a firewall update allows HTTPS traffic.

Updating the Firewall

If you have a firewall enabled, make sure port 80 and 443 are open to incoming traffic. If you are not running a firewall, you can skip ahead, otherwise you can open these ports by typing:

sudo firewall-cmd --add-service=http
sudo firewall-cmd --add-service=https
sudo firewall-cmd --runtime-to-permanent

If an iptables firewall is running, the commands you need to run are highly dependent on your current rule set. For an initial rule set, you can add HTTP and HTTPS access by typing:

sudo iptables -I INPUT -p tcp -m tcp --dport 80 -j ACCEPT
sudo iptables -I INPUT -p tcp -m tcp --dport 443 -j ACCEPT

Now the system is ready to run Certbot and fetch our certificates.

Obtaining a certificate

Certbot provides a variety of ways to obtain SSL certificates, through various plugins. The Nginx plugin will take care of reconfiguring Nginx and reloading the config whenever necessary:

sudo certbot --nginx -d example.com -d www.example.com

This runs certbot with the --nginx plugin, using -d to specify the names we’d like the certificate to be valid for.

Certbot will communicate with the Let’s Encrypt server, then run a challenge to verify that you control the domain you’re requesting a certificate for. The configuration will be updated, and Nginx will reload to pick up the new settings. certbot will wrap up with a message telling you the process was successful and where your certificates are stored. Your certificates are downloaded, installed, and loaded.

Setting Up Auto Renewal

Let’s Encrypt’s certificates are only valid for 90 days.

To run the renewal-check daily, we will use cron, a standard system service for running periodic jobs. Editing a file called a crontab.

sudo crontab -e

The default crontab config file will open. Paste in the following line:

15 3 * * * /usr/bin/certbot renew --quiet

The 15 3 * * * part of this line means “run the following command at 3:15 am, every day”.

The renew command for certbot will check all certificates installed on the system and update any that are set to expire in less than thirty days. The --quiet flag tells Certbot not to output information or wait for user input.

To check that all is set up:

crontab -l

All installed certificates will be automatically renewed and reloaded when they have thirty days or less before they expire.

Docker

Docker is a software that automate the deployment of applications inside containers. It works with images and container, the main difference between these two follwos:

  • An image is piece of memory with the os and applicaiton data inside
  • A container instead is a running image

To visualize a resume of the system run:

docker system df  # use -v to get further informations

System images

An image becomes a container when you execute it. Check the images that are present in your system after the build docker images, the command to build an image is:

docker build <DirectoryWhereTheDockerfileIsLocated>

First create a docker file, which is a file named Dockerfile with a content similar to the followng script snippet (depending on your needs) on which a docker image will be built;

# Start from the official Ubuntu official image in DockeHub (last LTS version)
FROM ubuntu:latest

# Setting the environment variable
ENV DEBIAN_FRONTEND=noninteractive

# Install software from apt
RUN apt-get -y update && \
  apt-get install -y \
  curl \
  libssl-dev \
  pkg-config \
  build-essential


# Install rust toolchain
RUN curl https://sh.rustup.rs -sSf | bash -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
RUN rustup target add wasm32-unknown-unknown
RUN cargo install wasm-bindgen-cli
RUN cargo install trunk

# Execute the script to serve the site
RUN mkdir -p /var/www/html/Application
COPY . /var/www/html/Application/
WORKDIR /var/www/html/Application

CMD ["/bin/bash", "/var/www/html/Application/start.sh"]

To build the image from the Dockerfile use:

docker build -t <nameOfTheImage> <DockerfileDirectory>

To visualise how many images you have in your system run:

docker images

Running the container

Running the container means that the docker file will be read and executed, the execution can be modified by the flags that follows the utility such as:

docker run -ti -–rm <imageID>

Here's some more info on the flags:

  • ti → make you access to the terminal
  • rm → remove the container once you exit
  • d → (deamon) run the container in detached mode (in the background)
  • p 80:80 → port exposition, the local OS can connect to the docker image via port 80

To see if the container is running check:

docker ps

Then to start and/or stop the container run the following self explatory commands:

docker start <imageID>
docker stop <imageID>

Mount a file system directory inside a docker container

Do not store your simulation data in the container. Instead let’s give our container access to just a little bit of our local filesystem. From your project directory in a Windows environment, run:

docker run -ti --rm -v ${PWD}:/data -w /data <imageID>

By adding the -v option we’re asking Docker to mount our current working directory ($PWD) as /data in the container. We’ve also added the -w switch to tell Docker that we’d like to be in /data when the container starts.

Creating an image from a container

You can convert a container into an image by using the command

docker commit

Delete container

The following command will delete the container:

docker ps			  	   # To visualise them
docker rmi <imageID>

Or you can delete all of them piping two commands:

docker rm -vf $(docker ps -aq)

Delete images

To delete the images that does not run a container above them run;

docker images  				# To visualise them
docker image rm <imageID>

Or you can delete all of them piping two commands:

docker rmi -f $(docker images -aq)

Reclaim space from machine

Docker saves container chaces on var/lib/docker, this can clog the machine disk if there is not enough storage available. A quick solution is to run the following command to clean the system cache:

docker system prune -a -f

Docker compose

Docker Compose is a tool for running multi-container applications on Docker defined using the Compose file format. A Compose file is used to define how one or more containers that make up your application are configured, such as:

version: '3.4'

services:
  restsvr:
    image: imageName
    ports:
      - 443:8000      # HOST / CONTAINER
    build:
      context: .
      dockerfile: ./Dockerfile

    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "5"

Once you have a Compose file, you can create and start your application with a single command:

docker-compose up --build -d

Benchmark for performances

An utility to benchmark progammes isperf, a profiling tool. To spin it up:

perf stat ./my_programme

The best resource I know is on Brendan Gregg’s site,

Hardware counters

So, let’s imagine you want to know exactly how many CPU instructions happen when you run ls. It turns out that your CPU stores information about this kind of thing, and perf can tell you. Here’s what the answer looks like, from perf stat.

sudo perf stat ls
       1,482,453 instructions              #    0.48  insns per cycle

One of the first processors to implement such counter and an associated instruction RDPMC to access it was the Intel Pentium. We can use http://livegrep.com to search the Linux kernel for the RDPMC instruction. Here’s it being used in a cryptic header file called msr.h:

static inline unsigned long long native_read_pmc(int counter)
{
    DECLARE_ARGS(val, low, high);

    asm volatile("rdpmc" : EAX_EDX_RET(val, low, high) : "c" (counter));
    return EAX_EDX_VAL(val, low, high);
}

This is AWESOME. Maybe this is how Linux reads hardware counters and gives them back to us in perf stat!! Further grepping for uses of native_read_pmc reveals that we read hardware counters via rdpmcl in x86/kernel/cpu/perf_event.c.

This code is a little impenetrable to me, but here’s a hypothesis for how this could work. Let’s say we’re running ls. This code might get scheduled on and off the CPU a few times.

So! Here’s what I think this looks like.

  • kernel: ok let's run ls for a while
  • kernel: CPU! Start counting CPU instructions!
  • CPU:
  • kernel:
  • ls: yayyyyyyyyyy
  • kernel:
  • kernel: CPU! How many instructions was that! (rdpmc)
  • CPU: 10,200!
  • kernel: <increments counter by 10,200>

One important outcome of this, if I understand correctly is – hardware counters are exact counters, and they’re low enough overhead that the kernel can just always run RDPCM every time it’s done running a piece of code. There’s no sampling or approximations involved.

Sampling software events

The core of perf events looks like it’s in kernel/events/core.c. This file includes the definition of the perf_event_open system call, on line 8107. Files with 10,000 lines of C code are not my specialty, but I’m going to try to make something of this.

My goal: understand how perf does sampling of CPU events. For the sake of argument, let’s pretend we only wanted to save the state of the CPU’s registers every time we sample.

We know from the perf_event_open man page that perf writes out events to userspace (“hi! I am in julia’s awesome function right now!”). It writes events to a mmap’d ring buffer. Which is some data structure in memory. Okay.

Further inspection of this 10,000 line core.c file reveals that the code outputs data to userspace in the perf_event_update_userpage function.

So, let’s find the code that copies all the x86 registers into userspace! It turns out it’s not too hard to find – it’s in this file called perf_regs.c. There are like 15 registers to copy! Neat.

In this case it makes sense that we sample – we definitely couldn’t save all the registers every instruction. That would be way too much work!

So now I can see a little tiny bit of the code that perf uses to do sampling. This isn’t terribly enlightening, but it does make me feel better.

Questions

when does perf do its sampling? is it when the process gets scheduled onto the CPU? how is the sampling triggered? I am completely confused about this. what is the relationship between perf and kprobes? if I just want to sample the registers / address of the instruction pointer from ls’s execution, does that have anything to do with kprobes? with ftrace? I think it doesn’t, and that I only need kprobes if I want to instrument a kernel function (like a system call), but I’m not sure. are kprobes and ftrace the same kernel system? I feel like they are but I am confused.

Cross compilation

Cross compilation is the process of building a program on one platform (the host) to be run on another platform (the target). Cross compilation can be useful when the target platform does not have the necessary tools or resources to build the program.

To perform cross compilation, you will need a cross-compiler, which is a toolchain that includes a compiler, linker, and other tools that are specific to the target platform. You will also need to set up the build system to use the cross-compiler and specify the target platform. The installation of the necessary packages and tools for cross compiling from an x64 GNU/Linux machine to an GNU/Linux ARM architecture, the following standard tools are:

sudo apt install gcc make gcc-aarch64-linux-gnu binutils-aarch64-linux-gnu