Difference between revisions of "Linux: Cgroups Resource Limiting"

From Define Wiki
Jump to navigation Jump to search
(edit)
 
(20 intermediate revisions by the same user not shown)
Line 4: Line 4:
 
== Installing Python ==
 
== Installing Python ==
 
  <nowiki>
 
  <nowiki>
−
$ yum install python36  
+
$ sudo yum install python36  
 
</nowiki>
 
</nowiki>
  
 
== Installing external modules ==
 
== Installing external modules ==
−
<nowiki>
+
<nowiki>
 
$ python3 -m ensurepip --default-pip
 
$ python3 -m ensurepip --default-pip
 
$ pip3 install matplotlib toml requests  # requests may not be necessary
 
$ pip3 install matplotlib toml requests  # requests may not be necessary
 +
</nowiki>
 +
 +
== Acquiring the Arbiter2 source files ==
 +
<nowiki>
 +
$ sudo yum install git
 +
 +
# RECOMMENDED : Clone arbiter2 to /etc/arbiter2
 +
$ git clone https://gitlab.chpc.utah.edu/arbiter2/arbiter2.git optional-destination-directory
 +
 +
</nowiki>
 +
 +
== Setting up 'arbiter' user to run the script ==
 +
<nowiki>
 +
$ useradd -M -N -r -s /bin/false -c "System account for Arbiter2" arbiter
 +
 +
# Create group arbiter
 +
$ groupadd arbiter
 +
 +
# Add user arbiter to group arbiter
 +
$ usermod -a -G arbiter arbiter
 +
</nowiki>
 +
 +
== Allowing cgroup files to be edited without root via sudo ==
 +
<nowiki>
 +
$ sudo python3 tools/make_sudoers.py -u arbiter -g arbiter -r <ARBITER_UID> > /etc/sudoers.d/arbiter2
 +
</nowiki>
 +
 +
== Create logs directory ==
 +
<nowiki>
 +
$ mkdir -p /etc/arbiter2/logs/`hostname`/plots
 +
$ chmod 773 /etc/arbiter2/logs/`hostname`
 +
 +
# Make arbiter the owner of cloned arbiter2 directory
 +
$ chown -R arbiter /etc/arbiter2
 +
</nowiki>
 +
 +
== Setup arbiter2 service file ==
 +
 +
Modify /etc/arbiter2/arbiter2.service file
 +
<nowiki>
 +
[Service]
 +
 +
# Username to run the arbiter2 service as. Recommended to run as arbiter.
 +
User=arbiter
 +
 +
# Find arbiter uid with `id -u arbiter`
 +
Slice=user-<ARBITER UID>.slice
 +
 +
# Set arbiter2 directory path
 +
Environment=ARBITER_DIR=/etc/arbiter2
 +
WorkingDirectory=/etc/arbiter2
 +
 +
ExecStart=<python-absolute-path>/python3.6 ${ARBITER_DIR}/arbiter/arbiter.py -g /etc/arbiter2/etc/config.toml -s 
 +
 +
</nowiki>
 +
 +
Copy service file to systemd dir.
 +
 +
<nowiki>
 +
$ cp /etc/arbiter2/arbiter2.service /etc/systemd/system/
 +
</nowiki>
 +
 +
== Edit arbiter2 config file ==
 +
Modify /etc/arbiter2/etc/config.toml file
 +
 +
<nowiki>
 +
[self]
 +
# Arbiter's groupname
 +
groupname = "arbiter"
 +
 +
[email]
 +
plot_location = '/etc/arbiter2/logs/%H/plots' 
 +
 +
[database]
 +
log_location = '/etc/arbiter2/logs/%H'
 +
 +
[processes]
 +
whitelist_file = '/etc/arbiter2/etc/whitelist.txt'
 +
 +
</nowiki>
 +
 +
OPTIONAL: Assign users to status groups. By default each non-service user(uid greater than min_uid) is by default allocated to the 'normal' status group. If required, each user can be explicitly assigned to a 'normal', 'admin' or 'invincible' user groups by adding the concerned uid in the 'uids' list below.
 +
 +
<nowiki>
 +
[general]
 +
# All processes owned by uids <=1000 will not be tracked by arbiter2
 +
min_uid = 1000  # i.e. non-service accounts
 +
 +
[status.normal]
 +
uids = []
 +
cpu_quota = 800  # 8 virtual cores
 +
mem_quota = 2  # Gigabyte
 +
 +
[status.admin]
 +
uids = [1000]
 +
cpu_quota = 1600  # 16 virtual cores
 +
mem_quota = 6  # Gigabyte
 +
 +
[status.invincible]
 +
uids = []
 +
cpu_quota = 1e5
 +
mem_quota = 1e5  # Gigabyte
 +
</nowiki>
 +
 +
Advanced config documentation can be found at https://github.com/subfission/arbiter2/blob/master/CONFIG.md
 +
 +
== Run arbiter2 service ==
 +
IMPORTANT: Arbiter2 service will only start if there is a user slice present in /sys/fs/cgroup/memory/user.slice directory for a user with uid greater than 'min_uid'(default value is 1000) in config.toml. This can be done by ensuring that atleast one user with uid > min_uid has or is logged on to the system('sudo su <user>' does not work). 
 +
<nowiki>
 +
# systemctl daemon-reload
 +
# systemctl enable arbiter2
 +
# systemctl start arbiter2
 +
</nowiki>
 +
 +
== Corralling processes ==
 +
 +
Start monitoring already running processes.
 +
 +
<nowiki>
 +
# Modify allusers_corraller.sh
 +
$ sed -i "s/w -h/who/g" /etc/arbiter2/tools/allusers_corraller.sh
 +
 +
# Run allusers_corraller.sh
 +
$ /etc/arbiter2/tools/allusers_corraller.sh
 
</nowiki>
 
</nowiki>

Latest revision as of 09:32, 2 September 2020

This document describes the process of a basic installation of the arbiter2 tool, that can be used to apply CPU & Memory limits on logged-in Linux users, using cgroups accounting. The official documentation can be found at https://github.com/subfission/arbiter2/blob/master/INSTALL.md

Installing Python

$ sudo yum install python36

Installing external modules

$ python3 -m ensurepip --default-pip
$ pip3 install matplotlib toml requests  # requests may not be necessary

Acquiring the Arbiter2 source files

$ sudo yum install git

# RECOMMENDED : Clone arbiter2 to /etc/arbiter2
$ git clone https://gitlab.chpc.utah.edu/arbiter2/arbiter2.git optional-destination-directory

Setting up 'arbiter' user to run the script

$ useradd -M -N -r -s /bin/false -c "System account for Arbiter2" arbiter

# Create group arbiter
$ groupadd arbiter 

# Add user arbiter to group arbiter
$ usermod -a -G arbiter arbiter

Allowing cgroup files to be edited without root via sudo

$ sudo python3 tools/make_sudoers.py -u arbiter -g arbiter -r <ARBITER_UID> > /etc/sudoers.d/arbiter2

Create logs directory

$ mkdir -p /etc/arbiter2/logs/`hostname`/plots
$ chmod 773 /etc/arbiter2/logs/`hostname`

# Make arbiter the owner of cloned arbiter2 directory
$ chown -R arbiter /etc/arbiter2

Setup arbiter2 service file

Modify /etc/arbiter2/arbiter2.service file 

[Service]

# Username to run the arbiter2 service as. Recommended to run as arbiter.
User=arbiter

# Find arbiter uid with `id -u arbiter`
Slice=user-<ARBITER UID>.slice

# Set arbiter2 directory path
Environment=ARBITER_DIR=/etc/arbiter2
WorkingDirectory=/etc/arbiter2

ExecStart=<python-absolute-path>/python3.6 ${ARBITER_DIR}/arbiter/arbiter.py -g /etc/arbiter2/etc/config.toml -s

Copy service file to systemd dir. 

$ cp /etc/arbiter2/arbiter2.service /etc/systemd/system/

Edit arbiter2 config file

Modify /etc/arbiter2/etc/config.toml file 

[self]
# Arbiter's groupname
groupname = "arbiter"

[email]
plot_location = '/etc/arbiter2/logs/%H/plots'  

[database]
log_location = '/etc/arbiter2/logs/%H'

[processes]
whitelist_file = '/etc/arbiter2/etc/whitelist.txt'

OPTIONAL: Assign users to status groups. By default each non-service user(uid greater than min_uid) is by default allocated to the 'normal' status group. If required, each user can be explicitly assigned to a 'normal', 'admin' or 'invincible' user groups by adding the concerned uid in the 'uids' list below. 

[general]
# All processes owned by uids <=1000 will not be tracked by arbiter2
min_uid = 1000  # i.e. non-service accounts

[status.normal]
uids = []
cpu_quota = 800  # 8 virtual cores
mem_quota = 2  # Gigabyte

[status.admin]
uids = [1000]
cpu_quota = 1600  # 16 virtual cores
mem_quota = 6   # Gigabyte

[status.invincible]
uids = []
cpu_quota = 1e5
mem_quota = 1e5   # Gigabyte

Advanced config documentation can be found at https://github.com/subfission/arbiter2/blob/master/CONFIG.md

Run arbiter2 service

IMPORTANT: Arbiter2 service will only start if there is a user slice present in /sys/fs/cgroup/memory/user.slice directory for a user with uid greater than 'min_uid'(default value is 1000) in config.toml. This can be done by ensuring that atleast one user with uid > min_uid has or is logged on to the system('sudo su <user>' does not work). 

# systemctl daemon-reload
# systemctl enable arbiter2
# systemctl start arbiter2

Corralling processes

Start monitoring already running processes. 

# Modify allusers_corraller.sh
$ sed -i "s/w -h/who/g" /etc/arbiter2/tools/allusers_corraller.sh

# Run allusers_corraller.sh
$ /etc/arbiter2/tools/allusers_corraller.sh
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=Linux:_Cgroups_Resource_Limiting&oldid=31804"