Linux: Cgroups Resource Limiting

This document describes the process of a basic installation of the arbiter2 tool, that can be used to apply CPU & Memory limits on logged-in Linux users, using cgroups accounting. The official documentation can be found at https://github.com/subfission/arbiter2/blob/master/INSTALL.md

Installing Python

$ sudo yum install python36 

Installing external modules

$ python3 -m ensurepip --default-pip
$ pip3 install matplotlib toml requests  # requests may not be necessary

Acquiring the Arbiter2 source files

$ sudo yum install git

# RECOMMENDED : Clone arbiter2 to /etc/arbiter2
$ git clone https://gitlab.chpc.utah.edu/arbiter2/arbiter2.git optional-destination-directory

Setting up 'arbiter' user to run the script

$ useradd -M -N -r -s /bin/false -c "System account for Arbiter2" arbiter

# Create group arbiter
$ groupadd arbiter 

# Add user arbiter to group arbiter
$ usermod -a -G arbiter arbiter

Allowing cgroup files to be edited without root via sudo

$ sudo python3 tools/make_sudoers.py -u arbiter -g arbiter -r <ARBITER_UID> > /etc/sudoers.d/arbiter2

Create logs directory

$ mkdir -p /etc/arbiter2/logs/`hostname`/plots
$ chmod 773 /etc/arbiter2/logs/`hostname`

# Make arbiter the owner of cloned arbiter2 directory
$ chown -R arbiter /etc/arbiter2

Setup arbiter2 service file

Modify /etc/arbiter2/arbiter2.service file

[Service]

# Username to run the arbiter2 service as. Recommended to run as arbiter.
User=arbiter

# Find arbiter uid with `id -u arbiter`
Slice=user-<ARBITER UID>.slice

# Set arbiter2 directory path
Environment=ARBITER_DIR=/etc/arbiter2
WorkingDirectory=/etc/arbiter2

ExecStart=<python-absolute-path>/python3.6 ${ARBITER_DIR}/arbiter/arbiter.py -g /etc/arbiter2/etc/config.toml -s  

Copy service file to systemd dir.

$ cp /etc/arbiter2/arbiter2.service /etc/systemd/system/

Edit arbiter2 config file

Modify /etc/arbiter2/etc/config.toml file

[self]
# Arbiter's groupname
groupname = "arbiter"

[email]
plot_location = '/etc/arbiter2/logs/%H/plots'  

[database]
log_location = '/etc/arbiter2/logs/%H'

[processes]
whitelist_file = '/etc/arbiter2/etc/whitelist.txt'

OPTIONAL: Assign users to status groups. By default each non-service user(uid greater than min_uid) is by default allocated to the 'normal' status group. If required, each user can be explicitly assigned to a 'normal', 'admin' or 'invincible' user groups by adding the concerned uid in the 'uids' list below.

[general]
# All processes owned by uids <=1000 will not be tracked by arbiter2
min_uid = 1000  # i.e. non-service accounts

[status.normal]
uids = []
cpu_quota = 800  # 8 virtual cores
mem_quota = 2  # Gigabyte

[status.admin]
uids = [1000]
cpu_quota = 1600  # 16 virtual cores
mem_quota = 6   # Gigabyte

[status.invincible]
uids = []
cpu_quota = 1e5
mem_quota = 1e5   # Gigabyte

Advanced config documentation can be found at https://github.com/subfission/arbiter2/blob/master/CONFIG.md

Run arbiter2 service

IMPORTANT: Arbiter2 service will only start if there is a user slice present in /sys/fs/cgroup/memory/user.slice directory for a user with uid greater than 'min_uid'(default value is 1000) in config.toml. This can be done by ensuring that atleast one user with uid > min_uid has or is logged on to the system('sudo su <user>' does not work).

# systemctl daemon-reload
# systemctl enable arbiter2
# systemctl start arbiter2

Corralling processes

Start monitoring already running processes.

# Modify allusers_corraller.sh
$ sed -i "s/w -h/who/g" /etc/arbiter2/tools/allusers_corraller.sh

# Run allusers_corraller.sh
$ /etc/arbiter2/tools/allusers_corraller.sh