I just tried to implement cgroups on an Ubuntu machine. It was the most horrendous experience. In the hope of helping others, or myself in the future, here is some of what I've learned. A lot of Googling around and trial and error was needed. The most valuable resource on the web (which is still not fully helpful...) is at https://wiki.archlinux.org/index.php/cgroups.
Setting up simple cgroups
To start, here's what I was hoping to do. I had three processes running on boot that kept slurping up all of my CPU and memory, which meant I couldn't ssh in, which meant I kept having to reboot every time I entered the server. This was not very good!
I'm guessing in reality it's just the memory I need to limit (since your server should be able to use 100% of CPU and intelligently let sshd use some of that CPU when it needs it) but just to be safe I've left both in.
The first step, on Ubuntu, is to install the cgroups packages:
sudo apt-get install cgroup-bin cgroup-lite libcgroup1
This creates a few files for you. Notably:
- /sys/fs/cgroup is created, which has a bunch of folders in a sort of virtual filesystem that "represents" your cgroup.
- /etc/init/cgroup-lite.conf. cgroup-lite.conf creates the /sys/fs/cgroup directory, based on the contents of
- /proc/cgroups, which specifies what groups cgroup-lite should create.
Take a look at these files to see what's going on inside.
OK now let's create a cgroup. The cgcreate command is helpful, but also annoying because the group disappears on a reboot. Let's assume your sudo-enabled account is called devin; it's a good practice (I guess?) to specify the admin permissions as root and the "task" permissions to the unprivileged account.
cgcreate -a root -t devin cpu,memory:limitgroup
Now look at the two relevant directories to see what has appeared:
ls /sys/fs/cgroup/cpu ls /sys/fs/cgroup/memory
Bam! There are new directories in there for your new group! I'm interested in two specific files for my use case; take a look at the others if you like.
head /sys/fs/cgroup/cpu/limitgroup/cpu.shares /sys/fs/cgroup/memory/limitgroup/memory.limit_in_bytes
You'll get 1024 for your cpu.shares (the maximum & default) and a huge number for your memory usage limit in bytes. To adjust things is easy. Let's set the maximum to 3 gigabytes of memory and 3/4 of CPU usage:
echo 768 > /sys/fs/cgroup/cpu/limitgroup/cpu.shares echo 3G > /sys/fs/cgroup/memory/limitgroup/memory.limit_in_bytes
Wow! At this point, you've got groups that will work. There is a complicated system to classify processes into groups, but for now, just run
cgexec -g cpu,memory:limitgroup cat /dev/urandom > /dev/null
This process will be put in the limitgroup and won't take up too much cpu or memory.
For me, this was pretty close to what I wanted. I had a script in my user's crontab that ran three commands, and I just prefixed each one with cgexec. If the arguments got too complicated, I made another bash script and then called that bash script via cgexec.
The issue left was having this group be created on boot. I originally thought about calling cgcreate on boot, and this still doesn't seem like a bad idea. But the idea I eventually came up with was to use /etc/cgconfig.conf, just like the Archlinux Wiki page said. The problem is that every page on the Internet that talks about this file assumes you're mounting your cgroups manually with mount {} directives. To see a bit about this, run
man cgconfig.conf
and then look through to the examples. They're not super helpful, since we're using this cgroup-lite package that mounts a bunch of cgroups for us.
Here's what I ended up putting in my /etc/cgconfig.conf:
group limitgroup {
perm {
admin {
uid = root;
gid = root;
}
task {
uid = 1001;
gid = 1001;
}
}
cpu {
cpu.shares = "768";
}
memory {
memory.limit_in_bytes = "3G";
}
}There are a few things different than in the Arch wiki. One, I specified the UID of my unprivileged user (1001) manually, but did NOT specify a UID for root. So weird. You can run
id devin
to get the UID and GID of the user you want to use. You need to let a user have task permissions to be able to create tasks in this group with cgexec, by the way. Other things that are different are the quotes around the cpu.shares and the memory.limitinbytes. I stopped questioning things once they worked. Finally, there's not a mount directive - see below for where the actual mounting happens. I don't think having a mount directive makes sense on Debian.
The way to verify that it's working (or get error messages, which don't seem helpful when you first see them but eventually you realize they are) is to run
cgconfigparser -l /etc/cgconfig.conf
This parses and then "executes" your cgconfig file; that is, if it parses correctly, it'll create your groups. I fought with creating mount directives in this file for quite a few reboots, but finally I just modified my /etc/init/cgroup-lite.conf and added the cgconfigparser -l /etc/cgconfig.conf line below the /bin/cgroups-mount line. So the modified contents of the pre-start section in /etc/init/cgroup-lite.conf file looked like
pre-start script
test -x /bin/cgroups-mount || { stop; exit 0; }
test -d /sys/fs/cgroup || { stop; exit 0; }
/bin/cgroups-mount
cgconfigparser -l /etc/cgconfig.conf
end script
It's kind of interesting to look at the contents of /bin/cgroups-mount, if you are interested.
There is a lot more to cgroups than what I set up here. But for my use case - setting up cgroups that limit a set of processes across reboots on Ubuntu - this is what I needed.