This post covers building a server that will be used to update both ports and the base system on FreeBSD backend servers that don’t have access to the Internet. For ports it will use poudriere combined with the new pkgng package manager in order to build packages that will be distributed to the other servers using Nginx. Since Nginx is already there it will be used as a forward proxy, as opposed to reverse proxy as it’s usually used, to http://updates.freebsd.org and http://pkgbeta.freebsd.org. This covers freebsd-update and installing pkg on FreeBSD 9.X. For no good reason, just to do something different, there’s a cron job that runs every day and fetches auditfile.tbz from http://portaudit.freebsd.org if it’s changed, so that pkg audit can do it’s job properly. In order to remotely install the servers they are PXE booted into a net install image, mfsbsd, base install files are mirrored on the package repository and served by Nginx. With that, the environment is complete, servers can be installed and kept up to date without giving them any kind of access to the Internet.
The Documentation. Poudriere knows how to use ZFS (used to need it, that’s not longer the case), we installed FreeBSD on a ZFS mirror using this script.
To compile packages for different releases and architectures from different ports trees poudriere uses “jails”, which are actually just chroot, not FreeBSD jails. Each jail or ports tree is a different zfs filesystem so they can be easily added/removed/restored, etc. When compiling packages any combination of jail+ports can be used. A jail represents a base system used to compile the ports. For example, we could have FreeBSD 8 and FreeBSD 9, some of them using the latest php version from a portstree fetched with portsnap while others use an year old version from a portstree checked out by svn. By chrooting inside a jail any modification to the base system can be made to mirror the real server environments, although that’s probably rarely necessary. So yes, it’s very versatile.
Packages are stored in different repositories under /poudriere/data/packages/$JAIL-$PORTS, using our configuration. If WITH_PKGNG is defined each repository will have repo.txz created, which is an sqlite database that pkgng downloads and uses to see what packages are available on the repository and check for updates.
When I first wrote this, it was on FreeBSD 9, with gcc and ZFS was mandatory. Things changed, so I give two versions below. Note that ZFS or not ZFS has nothing to do with the version of FreeBSD, both work in either version.
For FreeBSD 9 and ZFS:
Like the documentation says, install poudriere, pkgng if necessary and ccache from ports. Make sure dialog4ports is installed too, it will be needed in order to set options. If it isn’t pulled in as a dependency, install it. Copy poudriere.conf.sample to poudriere.conf and edit as necessary. Our config looks like this, comments filtered out for brevity:
ZPOOL=zroot
# CHANGE THIS TO A HOST CLOSE TO YOU
FREEBSD_HOST=ftp://ftp.freebsd.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/poudriere
USE_PORTLINT=yes
USE_TMPFS=yes
DISTFILES_CACHE=/usr/ports/distfiles
CHECK_CHANGED_OPTIONS=yes
CCACHE_DIR=/var/cache/ccacheFreeBSD 10, no ZFS:
There are two things that are different in FreeBSD 10 (ZFS isn’t one of them). One is that the default compiler changed to clang, the other is that pkgng is in base now. The default compiler affects the use of ccache. I am no expert, but clang is not officially supported in ccache 3.1.9, the current stable version, because there are some issues with it. In my opinion the risk of a port not compiling because of ccache doesn’t justify the benefits. If, however, someone would like to use it, pay attention to the port config option that installs symlinks for clang and the message after install. There is also a shell script called ‘ccache-update-links.sh’, which creates links for different compilers. As for pkgng, it is now in base in FreeBSD 10, so doesn’t need to be installed anymore. However, ports might have a newer version.
NO_ZFS=yes
# CHANGE THIS TO A HOST CLOSE TO YOU
FREEBSD_HOST=ftp://ftp.freebsd.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_PORTLINT=no
DISTFILES_CACHE=/usr/ports/distfiles
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
URL_BASE=http://neant.ro/poudriere/Using this configuration there are two directories of note, $BASEFS, which is /poudriere in the ZFS version and /usr/local/poudriere in the non-ZFS version, and /usr/local/etc/poudriere.d. From this point forward $BASEFS will be assumed to be /poudriere. The working directory is $BASEFS (/poudriere), logs and packages are written here, while /usr/local/etc/poudriere.d is where various settings are, like make.conf options and ports options for each jail. We also store the lists of packages to be compiled and cron scripts for updates here.
Create a jail:
# poudriere jails -c -j srv_91r_amd64 -v 9.1-RELEASE -a amd64Create make.conf for this jail:
# echo "WITH_PKGNG=yes" > /usr/local/etc/poudriere.d/srv_91r_amd64-make.conf
# echo "WITHOUT_X11=yes" >> /usr/local/etc/poudriere.d/srv_91r_amd64-make.confWhen compiling, the contents of this file will be added to the contents of the /etc/make.conf file that is inside the jail, in this case /poudriere/jails/srv_91r_amd64/etc/make.conf. There is also /usr/local/etc/poudriere.d/make.conf that can be used for options that should be applied to ALL jails.
Create a ports tree:
# poudriere ports -c -p defaultPassing -p name creates a tree with that particular name. If -p is not specified, a default ports tree called “default” will be used. This is true for other poudriere commands, too, like poudriere bulk, used to build packages.
The -m parameter stands for “method” and it can be used to specify how that ports tree will be created and updated. For a ports tree that will be downloaded by svn for example, -m svn+http could be used. Obviously, subversion needs to be installed in that case.
Create a list of packages to be compiled by poudriere:
# vim /usr/local/etc/poudriere.d/srv_91r_amd64.pkglistThis file accepts comments marked with a hash tag (#).
Set compilation options for all ports in pkglist:
# poudriere options -f /usr/local/etc/poudriere.d/srv_91r_amd64.pkglist -j srv_91r_amd64Build the packages:
# poudriere bulk -f /usr/local/etc/poudriere.d/srv_91r_amd64.pkglist -j srv_91r_amd64Enjoy watching the damn thing actually do some work for a change!
Adding a package to the list and setting options for the port and it’s dependencies. Like nagios:
# echo "net-mgmt/nagios" >> /usr/local/etc/poudriere.d/srv_91r_amd64.pkglist
# poudriere options -j srv_91r_amd64 net-mgmt/nagiosChanging options for a single port:
# poudriere options -c -j srv_91r_amd64 net-mgmt/nagiosRemoving options for a port and all dependecies:
# poudriere options -r -j srv_91r_amd64 net-mgmt/nagiosUse -n to set/remove options for nagios only. Note that, at least at the time of this writing, -r removes options for all dependencies, even if they are not direct dependencies of the respective port. For example, for nagios, it also removes options for apache22, because php5 depends on it and nagios depends on php5. But when using -c only options for things nagios directly depends on are set, like php5, but not apache22.
Updating the ports tree and rebuilding ports that were updated:
# poudriere ports -u -p default
# poudriere bulk -f /usr/local/etc/poudriere.d/srv_91r_amd64.pkglist -j srv_91r_amd64 -p defaultAs noted before, if -p is omitted a ports tree named “default” is assumed.
Passing -c or -C to poudriere bulk will clean all built packages or packages from the list respectively before compiling.
To have crontab update our repositories we use this script:
#!/bin/sh
JAILNAME=$1
PORTSTREE=$2
POUDRIERE_DIR="/usr/local/etc/poudriere.d/"
PATH="$PATH:/usr/local/bin"
if [ -z "$JAILNAME" ]; then
printf "Provide a jail name please.\n"
exit 1
fi
if [ -z "$PORTSTREE" ]; then
PORTSTREE="default"
fi
# check that the ports tree exists
poudriere ports -l | grep "^$PORTSTREE" > /dev/null
if [ $? -gt 0 ]; then
printf "No such ports tree ($PORTSTREE)\n"
exit 2
fi
## check that there's a list of packages for that jail
if [ ! -f "$POUDRIERE_DIR$JAILNAME.pkglist" ]; then
printf "No such file (list of packages: $POUDRIERE_DIR$PORTSTREE)\n"
exit 3
fi
# check that the jail is there
poudriere jails -l | grep "^$JAILNAME" > /dev/null
if [ $? -gt 0 ]; then
printf "No such jail ($JAILNAME)\n"
exit 4
fi
# update the ports tree
poudriere ports -u -p $PORTSTREE
# build new packages
poudriere bulk -f /usr/local/etc/poudriere.d/$JAILNAME.pkglist -j $JAILNAME -p $PORTSTREE It’s run from /etc/crontab at 1:30 every night:
# update pkg jails
30 1 * * * root /usr/local/etc/poudriere.d/scripts/repo-update.sh srv_91r_amd64 defaultA few hours later the other servers update their local repo.txz and check for updates.
Nginx is used to make the compiled packages available to the other servers. The default website is set to /poudriere/data/packages and autoindex is turned on, that’s pretty much all there is to it. In addition to serving packages from that website it also mirrors whatever distribution we need from ftp.freebsd.org, like ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/9.1-RELEASE/. auditfile.tbz is also on this website, updated every day using this crontab entry:
# update the audit file for 'pkg audit'
# 15 minutes before @daily
45 23 * * * root cd /poudriere/data/packages/ && fetch -mq http://portaudit.freebsd.org/auditfile.tbzIt’s also used as a forward proxy for update.freebsd.org and pkgbeta.freebsd.org. Apparently the guys behind update.freebsd.org don’t like mirrors for security reasons and are likely to block repeated calls from wget and the like, using a proxy is the recommended method. Nginx is already installed and it does the job just fine. Another reason is that freebsd-update uses phttpget and squid&co might not play well with it. pkgbeta.freebsd.org could be mirrored, just like base.
nginx.conf:
#user nobody;
worker_processes 4; # adjust to the number of CPU cores
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
server {
listen 80 default;
server_name pkg-repo.example.org "";
#access_log logs/host.access.log main;
location / {
root /poudriere/data/packages/;
index index.html index.htm;
autoindex on;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/local/www/nginx-dist;
}
}
# proxy connections to update.freebsd.org for
# servers that don't have internet access
server {
listen 80;
server_name fbsd-update.example.org;
location / {
proxy_pass http://update.freebsd.org;
proxy_http_version 1.1;
root /;
}
}
# proxy connections to pkgbeta.freebsd.org for
# servers that don't have internet access
# used for pkg bootstrap on new FBSD9 servers
server {
listen 80;
server_name pkg-bootstrap.example.org;
location / {
proxy_pass http://pkgbeta.freebsd.org;
proxy_http_version 1.1;
root /;
}
}
}Nginx has three vhosts defined, pkg-repo.example.org, fbsd-update.example.org and pkg-bootstrap.example.org. These names need to be defined in example.org’s DNS and they should point to this server.
FreeBSD 10 will have pkg installed in base by default, but in FreeBSD 9 it’s still in ports. In order to install it, a bare ‘pkg’ is provided which will download the real files from pkgbeta.freebsd.org. Nginx is proxy-ing connections to that site, so in order to bootstrap pkg in FreeBSD 9.1, assuming cshrc, the default root shell:
# setenv PACKAGEROOT "http://pkg-bootstrap.example.org"
# pkgHopefully it’s clear enough.
Next, edit /usr/local/etc/pkg.conf:
PACKAGESITE: http://pkg-repo-eu.perfectworld.eu/srv_91r_amd64-default/
PORTAUDIT_SITE: http://pkg-repo-eu.perfectworld.eu/auditfile.tbzReady to install packages now:
# pkg update
Updating repository catalogue
repo.txz 100% 1836 1.8KB/s 1.8KB/s 00:00
# pkg install sudo vim-lite
(...)Cron job to check for updates every night:
# check for updated packages
30 5 * * * root pkg update -q && pkg version -vRL=Check UPDATING before upgrading packages:
# pkg updatingby default it looks for /usr/ports/UPDATING which has no reason to be there on client computers, but it can easily be given out by Nginx on the pkg builder.
When pkgng is installed it also creates the periodic script /usr/local/etc/periodic/security/410.pkg-audit, which runs pkg audit, a replacement for portaudit. This is why we need auditfile in Nginx and the reason for the PORTAUDIT_SITE setting in pkg.conf.
Mandatory link to pkgng page in FreeBSD’s handbook
In /etc/freebsd-update.conf the line
ServerName update.FreeBSD.orgshould be replaced with
ServerName fbsd-update.example.orgThat’s all, freebsd-update should be working now. Provided the DNS entry is in place, of course.
/etc/crontab:
# check for OS updates
@daily root freebsd-update cron