Mar 292013
 

Overview

This post covers building a server that will be used to update both ports and the base system on FreeBSD backend servers that don’t have access to the Internet. For ports it will use poudriere combined with the new pkgng package manager in order to build packages that will be distributed to the other servers using Nginx. Since Nginx is already there it will be used as a forward proxy, as opposed to reverse proxy as it’s usually used, to http://updates.freebsd.org and http://pkgbeta.freebsd.org. This covers freebsd-update and installing pkg on FreeBSD 9.X. For no good reason, just to do something different, there’s a cron job that runs every day and fetches auditfile.tbz from http://portaudit.freebsd.org if it’s changed, so that pkg audit can do it’s job properly. In order to remotely install the servers they are PXE booted into a net install image, mfsbsd, base install files are mirrored on the package repository and served by Nginx. With that, the environment is complete, servers can be installed and kept up to date without giving them any kind of access to the Internet.

Server – poudriere

The Documentation. Poudriere knows how to use ZFS (used to need it, that’s not longer the case), we installed FreeBSD on a ZFS mirror using this script.

To compile packages for different releases and architectures from different ports trees poudriere uses “jails”, which are actually just chroot, not FreeBSD jails. Each jail or ports tree is a different zfs filesystem so they can be easily added/removed/restored, etc. When compiling packages any combination of jail+ports can be used. A jail represents a base system used to compile the ports. For example, we could have FreeBSD 8 and FreeBSD 9, some of them using the latest php version from a portstree fetched with portsnap while others use an year old version from a portstree checked out by svn. By chrooting inside a jail any modification to the base system can be made to mirror the real server environments, although that’s probably rarely necessary. So yes, it’s very versatile.

Packages are stored in different repositories under /poudriere/data/packages/$JAIL-$PORTS, using our configuration. If WITH_PKGNG is defined each repository will have repo.txz created, which is an sqlite database that pkgng downloads and uses to see what packages are available on the repository and check for updates.

First steps

When I first wrote this, it was on FreeBSD 9, with gcc and ZFS was mandatory. Things changed, so I give two versions below. Note that ZFS or not ZFS has nothing to do with the version of FreeBSD, both work in either version.

For FreeBSD 9 and ZFS:
Like the documentation says, install poudriere, pkgng if necessary and ccache from ports. Make sure dialog4ports is installed too, it will be needed in order to set options. If it isn’t pulled in as a dependency, install it. Copy poudriere.conf.sample to poudriere.conf and edit as necessary. Our config looks like this, comments filtered out for brevity:

FreeBSD 10, no ZFS:
There are two things that are different in FreeBSD 10 (ZFS isn’t one of them). One is that the default compiler changed to clang, the other is that pkgng is in base now. The default compiler affects the use of ccache. I am no expert, but clang is not officially supported in ccache 3.1.9, the current stable version, because there are some issues with it. In my opinion the risk of a port not compiling because of ccache doesn’t justify the benefits. If, however, someone would like to use it, pay attention to the port config option that installs symlinks for clang and the message after install. There is also a shell script called ‘ccache-update-links.sh’, which creates links for different compilers. As for pkgng, it is now in base in FreeBSD 10, so doesn’t need to be installed anymore. However, ports might have a newer version.

Using this configuration there are two directories of note, $BASEFS, which is /poudriere in the ZFS version and /usr/local/poudriere in the non-ZFS version, and /usr/local/etc/poudriere.d. From this point forward $BASEFS will be assumed to be /poudriere. The working directory is $BASEFS (/poudriere), logs and packages are written here, while /usr/local/etc/poudriere.d is where various settings are, like make.conf options and ports options for each jail. We also store the lists of packages to be compiled and cron scripts for updates here.

Create a jail:

Create make.conf for this jail:

When compiling, the contents of this file will be added to the contents of the /etc/make.conf file that is inside the jail, in this case /poudriere/jails/srv_91r_amd64/etc/make.conf. There is also /usr/local/etc/poudriere.d/make.conf that can be used for options that should be applied to ALL jails.

Create a ports tree:

Passing -p name creates a tree with that particular name. If -p is not specified, a default ports tree called “default” will be used. This is true for other poudriere commands, too, like poudriere bulk, used to build packages.

The -m parameter stands for “method” and it can be used to specify how that ports tree will be created and updated. For a ports tree that will be downloaded by svn for example, -m svn+http could be used. Obviously, subversion needs to be installed in that case.

Create a list of packages to be compiled by poudriere:

This file accepts comments marked with a hash tag (#).

Set compilation options for all ports in pkglist:

Build the packages:

Enjoy watching the damn thing actually do some work for a change!

Daily usage

Adding a package to the list and setting options for the port and it’s dependencies. Like nagios:

Changing options for a single port:

Removing options for a port and all dependecies:

Use -n to set/remove options for nagios only. Note that, at least at the time of this writing, -r removes options for all dependencies, even if they are not direct dependencies of the respective port. For example, for nagios, it also removes options for apache22, because php5 depends on it and nagios depends on php5. But when using -c only options for things nagios directly depends on are set, like php5, but not apache22.

Updating the ports tree and rebuilding ports that were updated:

As noted before, if -p is omitted a ports tree named “default” is assumed.

Passing -c or -C to poudriere bulk will clean all built packages or packages from the list respectively before compiling.

Crontab

To have crontab update our repositories we use this script:

It’s run from /etc/crontab at 1:30 every night:

A few hours later the other servers update their local repo.txz and check for updates.

Server – nginx

Nginx is used to make the compiled packages available to the other servers. The default website is set to /poudriere/data/packages and autoindex is turned on, that’s pretty much all there is to it. In addition to serving packages from that website it also mirrors whatever distribution we need from ftp.freebsd.org, like ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/9.1-RELEASE/. auditfile.tbz is also on this website, updated every day using this crontab entry:

It’s also used as a forward proxy for update.freebsd.org and pkgbeta.freebsd.org. Apparently the guys behind update.freebsd.org don’t like mirrors for security reasons and are likely to block repeated calls from wget and the like, using a proxy is the recommended method. Nginx is already installed and it does the job just fine. Another reason is that freebsd-update uses phttpget and squid&co might not play well with it. pkgbeta.freebsd.org could be mirrored, just like base.

nginx.conf:

Nginx has three vhosts defined, pkg-repo.example.org, fbsd-update.example.org and pkg-bootstrap.example.org. These names need to be defined in example.org’s DNS and they should point to this server.

Clients – pkgng

Since pkg 1.2, the format of the config file changed radically, basically pkg.conf contains only a few general settings and each repository gets it’s own config. Also, pkg now uses VuXML file by default. This article has not been updated to reflect that, partially because it seems that the dust hasn’t settled yet as I’m writing this.

FreeBSD 10 will have pkg installed in base by default, but in FreeBSD 9 it’s still in ports. In order to install it, a bare ‘pkg’ is provided which will download the real files from pkgbeta.freebsd.org. Nginx is proxy-ing connections to that site, so in order to bootstrap pkg in FreeBSD 9.1, assuming cshrc, the default root shell:

Hopefully it’s clear enough.

Next, edit /usr/local/etc/pkg.conf:

Ready to install packages now:

Cron job to check for updates every night:

Check UPDATING before upgrading packages:

by default it looks for /usr/ports/UPDATING which has no reason to be there on client computers, but it can easily be given out by Nginx on the pkg builder.

When pkgng is installed it also creates the periodic script /usr/local/etc/periodic/security/410.pkg-audit, which runs pkg audit, a replacement for portaudit. This is why we need auditfile in Nginx and the reason for the PORTAUDIT_SITE setting in pkg.conf.

Mandatory link to pkgng page in FreeBSD’s handbook

Clients – freebsd-update

In /etc/freebsd-update.conf the line

should be replaced with

That’s all, freebsd-update should be working now. Provided the DNS entry is in place, of course.

/etc/crontab:

  10 Responses to “Building a package repository for FreeBSD with poudriere and pkgng”

  1. Hey thank you for this really cool and in depth primer on poudriere. Everything is working without a hitch so far. Coolest port ever. The added cron scripts were also a very welcome nugget. Thanks again.

  2. Apparently Poudriere does use FreeBSD jails now instead of chroot:

    # jls

    JID IP Address Hostname Path

    388 127.0.0.1 release83amd64-default /usr/local/poudriere/data/build/release83amd64-default/ref

    • Thanks for the info, my old install still doesn’t show jails, but I’ve updated the article.

  3. It seems to me this would be a great plan for most any group of production servers as well….

  4. Is there a specific way to use ccache with poudiere?

    On FreeBSD 10.0-RELEASE, no matter what I did, ‘ccache -s’ shows:

    # ccache -s
    cache directory /root/.ccache
    cache hit (direct) 0
    cache hit (preprocessed) 0
    cache miss 0
    files in cache 0
    cache size 0 Kbytes
    max cache size 1.0 Gbytes

    What I did:

    ATTEMPT A:
    §1. Installed ccache and dialog4ports from pkg(ng) repos with ‘pkg install ccache’.
    §2 Appended ‘CCACHE_DIR=/var/cache/ccache’ to /usr/local/etc/poudriere.conf’
    §3 ‘ccache -s’ shows the same result as above. Seems like ccache is not in effect.

    ATTEMPT B:
    §4. Then I appended the following three lines to /etc/make.conf to mount ccache to tmpfs
    WRKDIRPREFIX=/ccache
    WITH_CCACHE_BUILD=yes
    CCACHE_COMPRESS=yes
    §5 Appended ‘none /compcache tmpfs rw,size=4294967296 0 0′ to /etc/fstab’ and mounted /compcache.
    §6 Replaced CCACHE_DIR=/ccache’ to /usr/local/etc/poudriere.conf’
    §7 ‘ccache -s’ shows the same result as above. No changes. Seems like ccache is not effect this time too.

    What is the right way to use CCACHE with Poudriere in FreeBSD 10-RELEASE?

    • If I am not mistaken, FreeBSD 10 uses clang as it’s default compiler. Right? So, without actually testing anything, I’m guessing that’s the problem, clang and ccache don’t work together by default. I don’t have a FreeBSD 10 yet and I don’t have access to the machine where I set up the repository anymore, but I will try to upgrade this server and set up a repo here, see if that works.

    • I don’t know if it makes a difference but when I followed the above guide, I built ccache with CLANG link support also my ccache didn’t work until I created the directory ‘/var/cache/ccache’. This directory gets mounted and shared between all of your jails. Everything is working great now, FreeBSD-10.0-RELEASE

    • Got the time to look into this a little. Updated the post to reflect the changes in FreeBSD 10. I chose not to install ccache this time, because it looks like it has some issues with clang. It was probably written with gcc in mind and clang didn’t get much coverage. When 3.2 is relased, it should contain improvements in this area, but until then the 3.1 branch doesn’t officially support it.

      If you want to use it however, the port’s config has an option to install symlinks for clang which should probably be checked. It will install a shell script called ‘ccache-update-links.sh’ and it also contains this message that should be displayed if ccache is compiled “with clang support”:

      =============================

      You’ve chosen to create symlinks to the clang compiler binaries if they exist.
      While it’s known to be safe to build world with clang/ccache, it is not fully
      supported yet. You have been warned.

      You should define CCACHE_CPP2=1 in your /etc/make.conf until bug 8460 is fixed.

      See:
      https://bugzilla.samba.org/show_bug.cgi?id=8460
      https://bugzilla.samba.org/show_bug.cgi?id=8118

      =============================

  5. Instead of @daily for your cron job, pick a random value. Distribute the load. Avoid having the entire time zone slamming the servers between midnight and 1am.

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">

(required)

(required)