Portage TMPDIR on tmpfs

From Gentoo Linux Wiki
Jump to: navigation, search

During emerge, you can build packages in tmpfs (aka 'ram disk' or 'virtual memory file system'), instead of having massive intermediate, temporary files on an HDD. Let RAM replace HDD.

Reduction of compilation time may very well not be in any way significant. The usefulness of TMPDIR on a tmpfs is often discussed, such as on this forum thread.

This page also describes some basic aspects of tmpfs itself.

Contents

[edit] Preparation

[edit] Kernel config

You need to have a kernel with swap and Virtual memory file system support enabled. Almost all default kernel configurations have these enabled.

Linux Kernel Configuration: Enable the Virtual memory file system
File systems --->
    Pseudo filesystems ---> 
        [*] Virtual memory file system support (former shm fs)

[edit] fstab

Add the tmpfs entry to /etc/fstab,

File: /etc/fstab
tmpfs      /var/tmp/portage   tmpfs  size=2G         0 0
Then mount the tmpfs,
mount -a

[edit] Size caveat

The size is very important, and fully explained here.

Tmpfs should be big enough, because once it gets full, emerge fails at that point. Probably you can't resume it, since files in tmpfs may be killed by oom killer, just like processes.

If you don't specify the tmpfs size, it'll be half of your physical RAM. It's merely the limit; only the actually used size occupies the RAM. Tmpfs grows and shrinks dynamically, so to say.

Swap can be used, so tmpfs can be bigger than the physical RAM. However, if the use of tmpfs exceeds the half of physical RAM, it slows down considerably. [1] [2]

If you have parallel merge, take 1 or 2GB. For single merge most packages suffice with 500MB, but some packages need huge area, for example:

[edit] Per-package choice for tmpfs / HDD

There're two methods to switch between tmpfs and HDD per packages.

[edit] By portage configuration

In this method, the packages you don't want to use tmpfs are specified in /etc/portage/package.env. This requires portage >= 2.1.9.

First create the following file in which you tell portage where to place the temporary files directory portage instead of default /var/tmp.

File: /etc/portage/env/notmpfs.conf
PORTAGE_TMPDIR="/var/tmp/notmpfs"

Next create the file in which you will list all the packages that need some special environment variable settings:

File: /etc/portage/package.env
app-office/libreoffice notmpfs.conf
mail-client/mozilla-thunderbird notmpfs.conf
Note: The thunderbird package was renamed into mail-client/thunderbird

And you are done! Portage will now use default settings for all packages except for ones listed in /etc/portage/package.env.

(This section is adapted from the blog post by Gentoo developer Jeremy Olexa which is available under CC-BY-SA license.)

[edit] Script to use tmpfs temporarily

This method is the opposite. You don't use tmpfs by default, but the script below automatically does all to emerge with tmpfs temporarily. This doesn't need the line in /etc/fstab. It will

  1. checks if /var/tmp/portage is already mounted (then quits, because it would confuse the other emerge process)
  2. mounts tmpfs to /var/tmp/portage
  3. runs emerge
  4. unmounts /var/tmp/portage
File: /usr/local/sbin/tmerge
#!/bin/bash
MEMSIZE=512M
mounted=false

. /etc/make.conf
. /etc/init.d/functions.sh

if [ -z "$PORTAGE_TMPDIR" ]; then
  PORTAGE_TMPDIR="/var/tmp/portage"
fi


mounttmpfs() {
                mount -t tmpfs -o size=$MEMSIZE,nr_inodes=1M tmpfs ${PORTAGE_TMPDIR}
                mounted="true"
}

compile() {
        einfo "running emerge ${*}"
                emerge ${*}
}

unmount() {
        ebegin "unmounting tmpfs"
                umount -f ${PORTAGE_TMPDIR}
        eend $?
}

ebegin "Mounting $MEMSIZE of memory to ${PORTAGE_TMPDIR}"
        if [ -z "$(pgrep -f /usr/bin/emerge)" ];then
                if [ -z "$(mount | grep ${PORTAGE_TMPDIR})" ];then
                     mounttmpfs
                else
                     eerror "tmpfs already mounted!"
                     exit 0
                fi
        else
                eerror "emerge already running!"
                exit 0
        fi
eend $?

# the next line would change the cpu-governour, if available, to the highest frequency
#[ -f /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor ] && echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# run emerge
compile $@

# unmount tmpfs
$mounted && umount ${PORTAGE_TMPDIR}

# and set the scheduler back to "ondemand"
#[ -f /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor ] && echo ondemand > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
To use the script, copy it to /usr/local/sbin/tmerge, make it executable using chmod and perform a
echo 'export PATH="$PATH":/usr/local/sbin' >> /root/.bashrc
so that it can be found by the shell. Then, simply use tmerge instead of emerge.

[edit] See Also

Personal tools
In other languages