Package Building Procedures

In order to provide pre-compiled binaries of third-party applications for FreeBSD, the Ports Collection is regularly built on one of the “Package Building Clusters.” Currently, the main cluster in use is at http://pointyhat.FreeBSD.org.

This article documents the internal workings of the cluster.

You may set up clients to either netboot from the master (connected nodes) or have them either self-hosted or netboot from some other pxe host (disconnected nodes). In all cases they set themselves up at boot-time to prepare to build packages.

The cluster master rsyncs the interesting data (ports and src trees, bindist tarballs, scripts, etc.) to disconnected nodes during the node-setup phase. Then, the disconnected portbuild directory is nullfs-mounted for jail builds.

The portbuild user can ssh(1) to the client nodes to monitor them. Use sudo and check the portbuild.hostname.conf for the user and access details.

The scripts/allgohans script can be used to run a command on all of the ${arch} clients.

Package builds are performed by the clients in a jail populated by the portbuild script using the ${arch}/${branch}/builds/${buildid}/bindist.tar file.

On the server, use the makeworld command to build a world from the ${arch}/${branch}/builds/${buildid}/src/ tree and install it into ${arch}/${branch}/builds/${buildid}/bindist.tar. The tree will be updated first unless -novcs is specified.

# /a/portbuild/admin/scripts/makeworld ${arch} ${branch} ${buildid} [-novcs]

Similiarly on the server, the bindist.tar tarball is created from the previously installed world by the mkbindist script.

# /a/portbuild/admin/scripts/mkbindist ${arch} ${branch} ${buildid}

The per-machine tarballs are located on the server in ${arch}/clients.

The bindist.tar file is extracted onto each client at client boot time, and at the start of each pass of the dopackages script.

For both commands above, if ${buildid} is latest, it may be omitted.

You can customize your build by providing local versions of make.conf and/or src.conf, named ${arch}/${branch}/builds/${buildid}/make.conf.server and ${arch}/${branch}/builds/${buildid}/src.conf.server, respectively. These will be used in lieu of the default-named files on the server side.

Similarly, if you wish to also affect the client-side make.conf, you may use ${arch}/${branch}/builds/${buildid}/make.conf.client.

RUBY_DEFAULT_VER= 1.9

.if !defined(CC) || ${CC} == "cc"
CC=clang
.endif
.if !defined(CXX) || ${CXX} == "c++"
CXX=clang++
.endif
.if !defined(CPP) || ${CPP} == "cpp"
CPP=clang-cpp
.endif
# Do not die on warnings
NO_WERROR=
WERROR=

WITH_PKGNG=yes
PKG_BIN=/usr/local/sbin/pkg

WITH_PKGNG=yes

WITH_BSD_SORT=yes

Separate builds for various combinations of architecture and branch are supported. All data private to a build (ports tree, src tree, packages, distfiles, log files, bindist, Makefile, etc) are located under the ${arch}/${branch}/builds/${buildid}/ directory. The most recently created build can be alternatively referenced using buildid latest, and the one before using previous.

New builds are cloned from the latest, which is fast since it uses ZFS.

# dopackages.wrapper ${arch} ${branch} ${buildid} [-options]

% build cleanup ${arch} ${branch} ${buildid} -full

don't know how to make dns-all(continuing)

% dopackages.wrapper i386 8 latest -nosrc -norestr -nofinish

% dopackages.wrapper amd64 8 latest -nosrc -noports -norestr -continue -noindex -noduds -nofinish

% dopackages.wrapper sparc64 8 -finish

# path/qmanager/packagebuild amd64 7-exp 20080904212103 aclock-0.2.3_2.tbz

A full build without any -no options performs the following operations in the specified order:

There are several cases where you will need to manually clean up a build:

% /a/portbuild/scripts/build cleanup i386 8-exp 20080714120411 -full

% ssh gohan24 df

% allgohans "umount -f pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports"
% allgohans "umount -f pointyhat.freebsd.org:/var/portbuild/i386/8-exp/src"

umount: pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports: statfs: No such file or directory
umount: pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports: unknown file system
umount: Cleanup of /x/tmp/8-exp/chroot/53837/compat/linux/proc failed!
/x/tmp/8-exp/chroot/53837/compat/linux/proc: not a file system root directory

You can use qclient command to monitor the status of build nodes, and to list the currently scheduled jobs:

% python path/qmanager/qclient jobs
% python path/qmanager/qclient status

The scripts/stats ${branch} command shows the number of packages already built.

Running cat /a/portbuild/*/loads/* shows the client loads and number of concurrent builds in progress. The files that have been recently updated are the clients that are online; the others are the offline clients.

Running tail -f ${arch}/${branch}/build.log shows the overall build progress.

If a port build is failing, and it is not immediately obvious from the log as to why, you can preserve the WRKDIR for further analysis. To do this, touch a file called .keep in the port's directory. The next time the cluster tries to build this port, it will tar, compress, and copy the WRKDIR to ${arch}/${branch}/wrkdirs/.

If you find that the system is looping trying to build the same package over and over again, you may be able to fix the problem by rebuilding the offending package by hand.

If all the builds start failing with complaints that they cannot load the dependent packages, check to see that httpd is still running, and restart it if not.

Keep an eye on df(1) output. If the /a/portbuild file system becomes full then Bad Things™ happen.

The status of all current builds is generated periodically into the packagestats.html file, e.g., http://pointyhat.FreeBSD.org/errorlogs/packagestats.html. For each buildenv, the following is displayed:

The easiest way to track build failures is to receive the emailed logs and sort them to a folder, so you can maintain a running list of current failures and detect new ones easily. To do this, add an email address to ${branch}/portbuild.conf. You can easily bounce the new ones to maintainers.

After a port appears broken on every build combination multiple times, it is time to mark it BROKEN. Two weeks' notification for the maintainers seems fair.

When building packages for a release, it may be necessary to manually update the ports and src trees to the release tag and use -novcs and -noportsvcs.

To build package sets intended for use on a CD-ROM, use the -cdrom option to dopackages.

If the disk space is not available on the cluster, use -nodistfiles to avoid collecting distfiles.

After the initial build completes, restart the build with -restart -fetch-original to collect updated distfiles as well. Then, once the build is post-processed, take an inventory of the list of files fetched:

% cd ${arch}/${branch}
% find distfiles > distfiles-${release}

You should use that output to periodically clean out the distfiles from ftp-master. When space gets tight, distfiles from recent releases can be kept while others can be thrown away.

Once the distfiles have been uploaded (see below), the final release package set must be created. Just to be on the safe side, run the ${arch}/${branch}/cdrom.sh script by hand to make sure all the CD-ROM restricted packages and distfiles have been pruned. Then, copy the ${arch}/${branch}/packages directory to ${arch}/${branch}/packages-${release}. Once the packages are safely moved off, contact the re@FreeBSD.org and inform them of the release package location.

Remember to coordinate with the re@FreeBSD.org about the timing and status of the release builds.

Once a build has completed, packages and/or distfiles can be transferred to ftp-master for propagation to the FTP mirror network. If the build was run with -nofinish, then make sure to follow up with dopackages -finish to post-process the packages (removes RESTRICTED and NO_CDROM packages where appropriate, prunes packages not listed in INDEX, removes from INDEX references to packages not built, and generates a CHECKSUM.MD5 summary); and distfiles (moves them from the temporary distfiles/.pbtmp directory into distfiles/ and removes RESTRICTED and NO_CDROM distfiles).

It is usually a good idea to run the restricted.sh and/or cdrom.sh scripts by hand after dopackages finishes just to be safe. Run the restricted.sh script before uploading to ftp-master, then run cdrom.sh before preparing the final package set for a release.

The package subdirectories are named by whether they are for release, stable, or current. Examples:

If you are doing a completely new package set (e.g., for a new release), copy packages to the staging area on ftp-master with something like the following:

# cd /a/portbuild/${arch}/${branch}
# tar cfv - packages/ | ssh portmgr@ftp-master tar xfC - w/ports/${arch}/tmp/${subdir}

Then log into ftp-master, verify that the package set was transferred successfully, remove the package set that the new package set is to replace (in ~/w/ports/${arch}), and move the new set into place. (w/ is merely a shortcut.)

For incremental builds, packages should be uploaded using rsync so we do not put too much strain on the mirrors.

ALWAYS use -n first with rsync and check the output to make sure it is sane. If it looks good, re-run the rsync without the -n option.

Example rsync command for incremental package upload:

# rsync -n -r -v -l -t -p --delete packages/ portmgr@ftp-master:w/ports/${arch}/${subdir}/ | tee log

Distfiles should be transferred with the cpdistfiles script:

# /a/portbuild/scripts/cpdistfiles ${arch} ${branch} ${buildid} [-yesreally] | tee log2

Doing it by hand is deprecated.

Experimental patches builds are run from time to time to new features or bugfixes to the ports infrastructure (i.e. bsd.port.mk), or to test large sweeping upgrades. At any given time there may be several simultaneous experimental patches branches, such as 8-exp on the amd64 architecture.

In general, an experimental patches build is run the same way as any other build, except that you should first update the ports tree to the latest version and then apply your patches. To do the former, you can use the following:

% cvs -R update -dP > update.out
% date > .updated

This will most closely simulate what the dopackages script does. (While .updated is merely informative, it can be a help.)

You will need to edit update.out to look for lines beginning with ^M, ^C, or ^? and then deal with them.

It is always a good idea to save original copies of all changed files, as well as a list of what you are changing. You can then look back on this list when doing the final commit, to make sure you are committing exactly what you tested.

Since the machine is shared, someone else may delete your changes by mistake, so keep a copy of them in e.g., your home directory on freefall. Do not use tmp/; since pointyhat itself runs some version of -CURRENT, you can expect reboots (if nothing else, for updates).

In order to have a good control case with which to compare failures, you should first do a package build of the branch on which the experimental patches branch is based for the i386™ architecture (currently this is 8). Then, when preparing for the experimental patches build, checkout a ports tree and a src tree with the same date as was used for the control build. This will ensure an apples-to-apples comparison later.

Once the build finishes, compare the control build failures to those of the experimental patches build. Use the following commands to facilitate this (this assumes the 8 branch is the control branch, and the 8-exp branch is the experimental patches branch):

% cd /a/portbuild/i386/8-exp/errors
% find . -name \*.log\* | sort > /tmp/8-exp-errs
% cd /a/portbuild/i386/8/errors
% find . -name \*.log\* | sort > /tmp/8-errs

% comm -3 /tmp/8-errs /tmp/8-exp-errs | less

This last command will produce a two-column report. The first column is ports that failed on the control build but not in the experimental patches build; the second column is vice versa. Reasons that the port might be in the first column include:

Reasons for a port appearing in the second column include:

Both columns should be investigated and the reason for the errors understood before committing the experimental patches set. To differentiate between broken by experimental patches and broken by upgrading above, you can do a rebuild of the affected packages under the control branch:

% cd /a/portbuild/i386/8/ports

The following command will set up the control branch for the partial build:

% /a/portbuild/scripts/dopackages.wrapper i386 8 latest -noportsvcs -nobuild -novcs -nofinish

The builds must be performed from the packages/All directory. This directory should initially be empty except for the Makefile symlink. If this symlink does not exist, it must be created:

% cd /a/portbuild/i386/8/packages/All
% ln -sf ../../Makefile .
% make -k -j<#> <list of packages to build>

This will build only those packages listed as well as all of their dependencies.

You can check the progress of this partial build the same way you would a regular build.

Once all the errors have been resolved, you can commit the package set. After committing, it is customary to send a HEADS UP email to ports@FreeBSD.org and copy ports-developers@FreeBSD.org informing people of the changes. A summary of all changes should also be committed to /usr/ports/CHANGES.

Before following these steps, please coordinate with portmgr.

When an old branch goes out of support, there are some things to garbage-collect.

As of 2011, the philosophy of package building is to build packages based on the earliest supported release of each branch. e.g.: if on RELENG-8, the following releases are supported: 8.1, 8.2, 8.3; then packages-8-stable should be built from 8.1.

As releases go End-Of-Life (see chart), a full (not incremental!) package build should be done and uploaded.

The procedure is as follows:

When a machine has a disk failure (e.g., panics due to read errors, etc), then we should do the following steps:

When a disk has failed, please inform the cluster administrators so we can try to get it replaced.