Implementing cgroups on Ubuntu or Debian

I just tried to implement cgroups on an Ubuntu machine. It was the most horrendous experience. In the hope of helping others, or myself in the future, here is some of what I've learned. A lot of Googling around and trial and error was needed. The most valuable resource on the web (which is still not fully helpful...) is at https://wiki.archlinux.org/index.php/cgroups.

Setting up simple cgroups

To start, here's what I was hoping to do. I had three processes running on boot that kept slurping up all of my CPU and memory, which meant I couldn't ssh in, which meant I kept having to reboot every time I entered the server. This was not very good!

I'm guessing in reality it's just the memory I need to limit (since your server should be able to use 100% of CPU and intelligently let sshd use some of that CPU when it needs it) but just to be safe I've left both in.

The first step, on Ubuntu, is to install the cgroups packages:

sudo apt-get install cgroup-bin cgroup-lite libcgroup1

This creates a few files for you. Notably:

  • /sys/fs/cgroup is created, which has a bunch of folders in a sort of virtual filesystem that "represents" your cgroup.
  • /etc/init/cgroup-lite.conf. cgroup-lite.conf creates the /sys/fs/cgroup directory, based on the contents of
  • /proc/cgroups, which specifies what groups cgroup-lite should create.

Take a look at these files to see what's going on inside.

OK now let's create a cgroup. The cgcreate command is helpful, but also annoying because the group disappears on a reboot. Let's assume your sudo-enabled account is called devin; it's a good practice (I guess?) to specify the admin permissions as root and the "task" permissions to the unprivileged account.

cgcreate -a root -t devin cpu,memory:limitgroup # may need '-g cpu,memory:limitgroup'

Now look at the two relevant directories to see what has appeared:

ls /sys/fs/cgroup/cpu
ls /sys/fs/cgroup/memory

Bam! There are new directories in there for your new group! I'm interested in two specific files for my use case; take a look at the others if you like.

head /sys/fs/cgroup/cpu/limitgroup/cpu.shares /sys/fs/cgroup/memory/limitgroup/memory.limit_in_bytes

You'll get 1024 for your cpu.shares (the maximum & default) and a huge number for your memory usage limit in bytes. To adjust things is easy. Let's set the maximum to 3 gigabytes of memory and 3/4 of CPU usage:

echo 768 > /sys/fs/cgroup/cpu/limitgroup/cpu.shares
echo 3G > /sys/fs/cgroup/memory/limitgroup/memory.limit_in_bytes

Wow! At this point, you've got groups that will work. There is a complicated system to classify processes into groups, but for now, just run

cgexec -g cpu,memory:limitgroup cat /dev/urandom > /dev/null

This process will be put in the limitgroup and won't take up too much cpu or memory.

For me, this was pretty close to what I wanted. I had a script in my user's crontab that ran three commands, and I just prefixed each one with cgexec. If the arguments got too complicated, I made another bash script and then called that bash script via cgexec.

The issue left was having this group be created on boot. I originally thought about calling cgcreate on boot, and this still doesn't seem like a bad idea. But the idea I eventually came up with was to use /etc/cgconfig.conf, just like the Archlinux Wiki page said. The problem is that every page on the Internet that talks about this file assumes you're mounting your cgroups manually with mount {} directives. To see a bit about this, run

man cgconfig.conf

and then look through to the examples. They're not super helpful, since we're using this cgroup-lite package that mounts a bunch of cgroups for us.

Here's what I ended up putting in my /etc/cgconfig.conf:

group limitgroup {
  perm {
    admin {
      uid = root;
      gid = root;
    }
    task {
      uid = 1001;
      gid = 1001;
    }  
  }
  cpu {
    cpu.shares = "768";
  }
  memory {
    memory.limit_in_bytes = "3G";
  }            
}

There are a few things different than in the Arch wiki. One, I specified the UID of my unprivileged user (1001) manually, but did NOT specify a UID for root. So weird. You can run

id devin

to get the UID and GID of the user you want to use. You need to let a user have task permissions to be able to create tasks in this group with cgexec, by the way. Other things that are different are the quotes around the cpu.shares and the memory.limitinbytes. I stopped questioning things once they worked. Finally, there's not a mount directive - see below for where the actual mounting happens. I don't think having a mount directive makes sense on Debian.

The way to verify that it's working (or get error messages, which don't seem helpful when you first see them but eventually you realize they are) is to run

cgconfigparser -l /etc/cgconfig.conf

This parses and then "executes" your cgconfig file; that is, if it parses correctly, it'll create your groups. I fought with creating mount directives in this file for quite a few reboots, but finally I just modified my /etc/init/cgroup-lite.conf and added the cgconfigparser -l /etc/cgconfig.conf line below the /bin/cgroups-mount line. So the modified contents of the pre-start section in /etc/init/cgroup-lite.conf file looked like

pre-start script
        test -x /bin/cgroups-mount || { stop; exit 0; }
        test -d /sys/fs/cgroup || { stop; exit 0; }
        /bin/cgroups-mount
        cgconfigparser -l /etc/cgconfig.conf
end script

It's kind of interesting to look at the contents of /bin/cgroups-mount, if you are interested.

There is a lot more to cgroups than what I set up here. But for my use case - setting up cgroups that limit a set of processes across reboots on Ubuntu - this is what I needed.

Tags: 

Comments

This is bloody amazing. I have been going mad trying to understand cgroups under Ubuntu... and rueing my decision to use Ubuntu instead of Red Hat which seems to have much better support for cgroups. Confident I can get things working now... I hope! Thanks!

That is great. For me, getting just this far with the stuff available online was a nightmare, so I'm glad I could save you some work!

Thank you man! As the Ubuntu 14.04 here still doesn't uses SystemD, I've struggled a lot trying to understand how to use CGroups to limit CPU/MEM usage of Firefox until I found your article. I mentioned SystemD because as far as I've seen, it makes a lot easier to deal with CGroups.

Devin, enormous thanks. I was going nuts for a week. This article is amazing. Thank you so much for sharing this.

I feel your pain, and thanks for posting to let me know it helped!

this is simply fantastic! You sir, saved me a ton of time and your instructions are a pleasure to follow, easy to understand and very informative. Thanks! p.s I'm printing these pages and saving as .pdfs

Add new comment