This document summarizes installation and use of BroControl, Bro's interactive shell for operating Bro installations. Bro Control has two modes of operation: a stand-alone mode for managing a traditional, single-system Bro setup; and a cluster mode for maintaining a multi-system setup of coordinated Bro instances load-balancing the work across a set of independent machines. Below, we describe the installation process separately for the two modes. Once installed, the operation is pretty similar for both types; just keep in mind that if this documents refers to "nodes" and you're in a stand-alone setup, there's is only a single one and no worker/proxies.)

Prerequisites

Running BroControl requires the following prerequisites:

Note
If you're using a development version, you might need to apply a patch to Bro. Please see below.

Installing a Stand-alone Bro

This is the default installation. Configure and compile Bro as usual, specifying the target installation path as `prefix` (we use /usr/local/bro as an example):

> cd /path/to/bro/source/distribution
> ./configure --prefix=/usr/local/bro
> make

Installing a Bro Cluster

A Bro Cluster is a set of systems jointly analyzing the traffic of a network link in a coordinated fashion. BroControl is able to operate such a setup from a central manager system pretty much transparently, hiding much of the complexity of the multi-machine installation.

A cluster consists of four types of components:

  1. One or more frontends. Frontends load-balance the traffic across a set of worker machines.

  2. Worker nodes. Workers are doing the actual analysis, with each seeing a slice of the overall traffic as splitted up by the frontends.

  3. One or more proxies. Proxies relay the communication between worker nodes.

  4. One manager. The manager provides the cluster's user-interface for controlling and logging. During operation, the user only interacts with the manager; this is where BroControl is running.

See this RAID Paper for more information about the general cluster architecture. This document focusses on the installation of manager, workers, and the proxies. See <somewhere different> for a discussion of how to setup a frontend. If not otherwise stated, in the following we use the terms "manager", "worker", and "proxy" to refer to Bro instances, not to physical machines; rather, we use the term "node" to refer to physical machines. There may be multiple Bro instances running on the same node. For example, it's possible to run a proxy on the same node as the manager is operating on.

In the following, as an example setup, we will assume that our cluster consists of four nodes (not counting the frontend). The host names of the systems will be host1, host2, host3, and host4. We will configure the cluster so that host1 runs the manager and the (only) proxy, and host{2,3,4} are each running one worker. This is a typical setup, which will work well for many sites.

When installing a cluster, in addition to the prerequisites mentioned above, you need to

With all prerequisites in place, as the Bro user, perform the following steps to install a Bro cluster:

Getting Started

BroControl is an interactive interface to the cluster which allows you to, e.g., start/stop the monitoring or update its configuration. It is started with the broctl script and then expects commands on its command-line (alternatively, broctl can also be started with a single command directly on the shell's command line):

> broctl
Welcome to BroControl 0.2
Type "help" for help.
[BroControl] >

As the message says, type help to see a list of all commands. We will now briefly summarize the most important commands. A full reference follows below.

Once broctl.cfg and node.cfg are set up as described above, the monitoring can be started with the start command. In the cluster setup, this will successively start manager, proxies, and workers. The status command should then show all nodes as operating. To stop the monitoring, issue the stop command. exit leaves the shell.

On the manager system (and on the standalone system), you find the current set of (aggregated) logs in logs/current (which is a symlink to the corresponding spool directory.) The workers and proxies log into spool/proxy/ and spool/<worker-name>/, respectively. The manager/stand-alone logs are archived in logs/, by default once a day. Logs files of workers and proxies are discarded at the same rotation interval.

Whenenver the BroControl configuration is modified in any way (including changes to configuration files and site-specific policy scripts), install installs the new version. No changes will take effect until install is run. Before you run install, check can be used to check for any potential erros in the new configuration, e.g., typos in scripts. If check does not report any problems, doing install will pretty likely not break anything.

Note that generally configuration changes only take effect after a restart of the affected nodes. The restart command triggers this. Some changes however can be put into effect on-the-fly without restarting any of the nodes by using the update command (again only after doing install first). Such dynamic updates work with all changes done via the analysis command (see below) as well as generally with all policy which only modify global variables declared as redefinable (i.e., with Bro's &redef attribute).

Generally, site-specific tuning needs to be done with local policy scripts, as in any Bro setup. This is described below. Some general types of analysis can however be enabled/disabled via the analysis command.

BroControl provides various options to control the behaviour of the setup. These options can be set by editing etc/broctl.cfg. The config command gives list of all options with their current values. A list of the most important options also follows below.

Site-specific Customization

You'll most likely want to adapt the Bro policy to the local environment. While some types of analysis can be customized via the analysis command, much of the more specific tuning requires writing local policy files.

By default, it is assumed that you put site-specific policy scripts into the share/bro/site directory. To change the location of site policies, set the option SitePolicyPath in broctl.cfg to a different path.

During the initial install, sample policy scripts are installed in share/bro/site, which you can edit as appropiate. In the stand-alone setup, this is just a single file called local.bro. In the cluster setup, there are three: local-manager.bro and local-worker.bro are loaded by the manager and the workers (plus proxies), respectively. In turn, both of these load local.bro, which contains all configuration code shared by all nodes. If in doubt, put your customizations into local.bro so that all nodes see it. If you want to change which local scripts are loaded by the nodes, you can set SitePolicyManager for the manager and SitePolicyWorker for the workers.

In the cluster setup, the main exception to putting everything into local.bro is notice filtering, which should be done only on the manager. The example local-manager.bro comes with an example setup to configure notice policy and notice actions. You should to adapt thiese to the local environment.

In general, all of BroControl's policy scripts are loaded before any site-specific policy so that you can redefine any of the defaults locally. The scripts shows precisely which policy scripts get loaded by a node; that can be very helpful for debugging.

Please note that enabling a particular kind of analysis via the analysis command only has an effect if the corresponding Bro scripts are loaded by the local site policy in local.bro.

It is also possible to add additional scripts to individual nodes only. This works by setting the option aux_scripts for the corresponding node(s) in etc/nodes.cfg. For example, one could add a script experimental.bro to a single worker for trying out new experimental code.

Command Reference

The following summary lists all commands supported by BroControl. All commands may be either entered interactively or specificed on the shell's command line. If not specified otherwise, commands taking [<nodes>] as arguments apply their action either to the given set of nodes, or to all nodes if none is given.

analysis enable|disable <type>

This command enables or disables certain kinds of analysis without the need for doing any changes to Bro scripts. Currently, the analyses shown in the table below can be controlled (in parentheses the corresponding Bro scripts; the effect of enabling/disabling is similar to loading or not loading these scripts, respectively). The list might be extended in the future. Any changes performed via this command are applied by update and therefore do not require a restart of the nodes.

Type Description
dns DNS analysis (dns.bro)
ftp FTP analysis (ftp.bro)
http-body HTTP body analysis (http-body.bro).
http-request Client-side HTTP analysis only (http-request.bro)
http-reply Client- and server-side HTTP analysis (http-request.bro/http-reply.bro)
http-header HTTP header analysis (http-headers.bro)
scan Scan detection (scan.bro)
smtp SMTP analysis (smtp.bro)
attachgdb [<nodes>]

Primarily for debugging, the command attaches a gdb to the main Bro process on the given nodes.

capstats [<interval>] [<nodes>]

Determines the current load on the network interfaces monitored by each of the given worker nodes. The load is measured over the specified interval (in seconds), or by default over 10 seconds. This command uses the capstats tool, which is installed along with broctl.

(Note: When using a CFlow and the CFlow command line utility is installed as well, the capstats command can also query the device for port statistics. TODO: document how to set this up.)

check [<nodes>]

Verifies a modified configuration in terms of syntactical correctness (most importantly correct syntax in policy scripts). This command should be executed for each configuration change before install is used to put the change into place. Note that check is the only command which operates correctly without a former install command; check uses the policy files as found in SitePolicyPath to make sure they compile correctly. If they do, install will then copy them over to an internal place from where the nodes will read them at the next start. This approach ensures that new errors in a policy scripts will not affect currently running nodes, even when one or more of them needs to be restarted.

cleanup [—all] [<nodes>]

Clears the nodes' spool directories (if they are not running currently). This implies that their persistent state is flushed. Nodes that were crashed are reset into stopped state. If —all is specified, this command also removes the content of the node's TmpDir, in particular deleteing any data potentially saved there for reference from previous crashes. Generally, if you want to reset the installation back into a clean state, you can first stop all nodes, then execute cleanup —all, and finally start all nodes again.

config

Prints all configuration options with their current values.

cron [<nodes>]

As the name implies, this command should be executed regularly via cron, as described above. It performs a set of maintainance tasks, including the logging of various statistical information, expiring old log files, checking for dead hosts, and restarting nodes which terminated unexpectedly. While not intended for interactive use, no harm will be caused by executing the command manually: all the maintainance tasks will then just be performed one more time.

df [<nodes>]

Reports the amount of disk space available on the nodes. Shows only paths relevant to the broctl installation.

diag [<nodes>]

If a node has terminated unexpectedly, this command prints a (somewhat cryptic) summary of its final state including excerpts of any stdout/stderr output, resource usage, and also a stack backtrace if a core dump is found. The same information is sent out via mail when a node is found to have crashed (the "crash report"). While the information is mainly intended for debugging, it can also help to find misconfigurations (which are usually, but not always, caught by the check command).

exec <command line>

Executes the given Unix shell command line on all nodes configured to run at least one Bro instance. This is handy to quickly perform an action across all systems.

exit

Terminates the shell.

help

Prints a brief summary of all commands understood by the shell.

install

Reinstalls the given nodes, including all configuration files and local policy scripts. This command must be executed after all changes to any part of the broctl configuration, otherwise the modifications will not take effect. Usually all nodes should be reinstalled at the same time, as any inconsistencies between them will lead to strange effects. Before executing install, it is recommended to verify the configuration with check.

netstats [<nodes>]

Queries each of the nodes for their current counts of captured and dropped packets.

nodes

Prints a list of all configured nodes.

peerstatus [<nodes>]

Primarily for debugging, peerstatus reports statistics about the network connections cluster nodes are using to communicate with other nodes.

print <id> [<nodes>]

Reports the current live value of the given Bro script ID on all of the specified nodes (which obviously must be running). This can for example be useful to (1) check that policy scripts are working as expected, or (2) confirm that configuration changes have in fact been applied. Note that IDs defined inside a Bro namespace must be prefixed with <namespace>:: (e.g., print SSH::did_ssh_version to print the corresponding table from ssh.bro.)

quit

Terminates the shell.

restart [—clean] [<nodes>]

Restarts the given nodes, or all nodes if none are specified. The effect is the same as first executing stop followed by a start, giving the same nodes in both cases. This command is most useful to activate any changes made to Bro policy scripts (after running install first). Note that a subset of policy changes can also be installed on the fly via the update, without requiring a restart.

If —clean is given, the installation is reset into a clean state before restarting. More precisely, a restart —clean turns into the command sequence stop, cleanup —all, check, install, and start.

scripts [-p|-c] [<nodes>]

Primarily for debugging Bro configurations, the script command lists all the Bro scripts loaded by each of the nodes in the order as they will be parsed by the node at startup. If -p is given, all scripts are listed with their full paths. If -c is given, the command operates as check does: it reads the policy files from their original location, not the copies installed by install. The latter option is useful to check a not yet installed configuration.

start [<nodes>]

Starts the given nodes, or all nodes if none are specified. Nodes already running are left untouched.

status [<nodes>]

Prints the current status of the given nodes.

stop [<nodes>]

Stops the given nodes, or all nodes if none are specified. Nodes not running are left untouched.

top [<nodes>]

For each of the nodes, prints the status of the two Bro processes (parent process and child process) in a top-like format, including CPU usage and memory consumption. If executed interactively, the display is updated frequently until key q is pressed. If invoked non-interactively, the status is printed only once.

update [<nodes>]

After a change to Bro policy scripts, this command updates the Bro processes on the given nodes while they are running (i.e., without requiring a restart). However, such dynamic updates work only for a subset of Bro's full configuration. The following changes can be applied on the fly: (1) The value of all script variables defined as &redef can be changed; and (2) all configuration changes performed via the analysis command can be put into effect. More extensive script changes are not possible during runtime and always require a restart; if you change more than just the values of &redef variables and still issue update, the results are undefined and can lead to crashes. Also note that before running update, you still need to do an install (preferably after check), as otherwise update will not see the changes and resend the old configuration.

Option Reference

This section summarizes the options that can be set in etc/broctl.cfg for customizing the behaviour of BroControl. Usually, one only needs to change the "user options", which are listed first. The "internal options" are, as the name suggests, primarily used internally and set automatically. They are documented here only for reference.

User Options

AuxPostProcessors (string, default empty)

Additional log postprocessors, with paths separated by spaces.

AuxScriptsManager (string, default empty)

Additional Bro scripts loaded on the manager, separated by spaces.

AuxScriptsStandalone (string, default empty)

Additional Bro scripts loaded on a standalone Brothe manage, separated by spaces.

AuxScriptsWorker (string, default empty)

Additional Bro scripts loaded on the workers, separated by spaces.

BroArgs (string, default empty)

Additional arguments to pass to Bro on the command-line.

CFlowAddr (string, default empty)

If a cFlow load-balander is used, the address of the device (format: <ip>:<port>).

CFlowPassword (string, default empty)

If a cFlow load-balander is used, the password for accessing its configuration interface.

CFlowUser (string, default empty)

If a cFlow load-balander is used, the user name for accessing its configuration interface.

CronCmd (string, default empty)

A custom command to run everytime the cron command has finished.

CustomInstallBin (string, default empty)

Additional executables to be installed into ${BinDir}, including full path and separated by spaces.

Debug (bool, default 0)

Enable extensive debugging output in spool/debug.log.

DevMode (bool, default 0)

Enable development mode, which changes how things are installed by the install command.

HaveNFS (bool, default 0)

True if shared files are mounted across all nodes via NFS (see FAQ).

LogDir (string, default "${BroBase}/logs")

Directory for archived log files.

LogExpireInterval (int, default 30)

Number of days log files are kept.

MailAlarmPrefix (string, default "ALERT:")

Subject prefix for individual alerts triggered by NOTICE_EMAIL.

MailAlarms (bool, default 1)

True if Bro should send mails for NOTICE_EMAIL alerts.

MailAlarmsTo (string, default "${MailTo}")

Destination address for broctl-generated alarm mails.

MailFrom (string, default "Big Brother <>")

Originator address for broctl-generated mails.

MailReplyTo (string, default empty)

Reply-to address for broctl-generated mails.

MailSubjectPrefix (string, default "[Bro]")

General Subject prefix for broctl-generated mails.

MailTo (string, default "<user>")

Destination address for broctl-generated non-alarm mails. Default is to use the same address as MailTo.

MakeArchiveName (string, default "${BroBase}/share/broctl/scripts/make-archive-name")

Script to generate filenames for archived log files.

MemLimit (string, default "unlimited")

Maximum amount of memory for Bro processes to use (in KB, or the string unlimited).

MinDiskSpace (int, default 5)

Percentage of minimum disk space available before warning is mailed.

Prefixes (string, default "local")

Additional script prefixes for Bro, separated by colons. Use this instead of @prefix.

SaveTraces (bool, default 0)

True to let backends capture short-term traces via -w. These are not archived but might be helpful for debugging.

SendMail (bool, default 1)

True if shell may send mails.

SitePolicyManager (string, default "local-manager.bro")

Local policy file for manager.

SitePolicyPath (string, default "${PolicyDir}/site")

Directories to search for local policy files, separated by colons.

SitePolicyStandalone (string, default "local.bro")

Local policy file for standalone Bro.

SitePolicyWorker (string, default "local-worker.bro")

Local policy file for workers.

TimeFmt (string, default "%d %b %H:%M:%S")

Format string to print data/time specifications (see man strftime).

TimeMachineHost (string, default empty)

If the manager should connect to a Time Machine, the address of the host it is running on.

TimeMachinePort (string, default "47757/tcp")

If the manager should connect to a Time Machine, the port it is running on (in Bro syntax, e.g., 47757/tcp.

Internal Options

AnalysisCfg (string, default "${CfgDir}/analysis.dat")

Configuration file defining types of analysis which can be toggled on-the-fly.

BinDir (string, default "${BroBase}/bin")

Directory for executable files.

BroBase (string, default empty)

Base path of broctl installation on all nodes.

Capstats (string, default "${bindir}/capstats")

Path to capstats binary; empty if not available.

CfgDir (string, default "${BroBase}/etc")

Directory for configuration files.

DebugLog (string, default "${SpoolDir}/debug.log")

Log file for debugging information.

DistDir (string, default empty)

Path to Bro distribution directory.

HaveBroccoli (bool)

True if Broccoli interface is available.

HelperDir (string, default "${BroBase}/share/broctl/scripts/helpers")

Directory for broctl helper scripts.

LibDir (string, default "${BroBase}/lib")

Directory for library files.

LibDirInternal (string, default "${BroBase}/lib/broctl")

Directory for broctl-specific library files.

LocalNetsCfg (string, default "${CfgDir}/networks.cfg")

File definining the local networks.

LockFile (string, default "${SpoolDir}/lock")

Lock file preventing concurrent shell operations.

NodeCfg (string, default "${CfgDir}/node.cfg")

Node configuration file.

OS (string, default empty)

Name of operation systems as reported by uname.

PolicyDir (string, default "${BroBase}/share/bro")

Directory for standard policy files.

PolicyDirBroCtl (string, default "${PolicyDir}/broctl")

Directory where the shell copies the additional broctl policy scripts when installing.

PolicyDirSiteInstall (string, default "${PolicyDir}/.site")

Directory where the shell copies local policy scripts when installing.

PolicyDirSiteInstallAuto (string, default "${PolicyDir}/.site/auto")

Directory where the shell copies auto-generated local policy scripts when installing.

PostProcDir (string, default "${BroBase}/share/broctl/scripts/postprocessors")

Directory for log postprocessors.

Scripts-Manager (string, default "cluster-manager")

Bro scripts loaded on the manager, separated by spaces.

Scripts-Proxy (string, default "cluster-proxy")

Bro scripts loaded on the proxies, separated by spaces.

Scripts-Standalone (string, default "standalone")

Bro scripts loaded on a standalone Bro, separated by spaces.

Scripts-Worker (string, default "cluster-worker")

Bro scripts loaded on the workers, separated by spaces.

ScriptsDir (string, default "${BroBase}/share/broctl/scripts")

Directory for executable scripts shipping as part of broctl.

SpoolDir (string, default "${BroBase}/spool")

Directory for run-time data.

StandAlone (bool, default 0)

True if running in stand-alone mode (see elsewhere).

StateFile (string, default "${SpoolDir}/broctl.dat")

File storing the current broctl state.

StaticDir (string, default "${BroBase}/share/broctl")

Directory for static, arch-independent files.

StatsDir (string, default "${LogDir}/stats")

Directory where statistics are kepts.

StatsLog (string, default "${SpoolDir}/stats.log")

Log file for statistics.

TemplateDir (string, default "${BroBase}/share/broctl/templates")

Directory where the *.in templates are copied into.

Time (string, default empty)

Path to time binary.

TmpDir (string, default "${SpoolDir}/tmp")

Directory for temporary data.

TmpExecDir (string, default "${SpoolDir}/tmp")

Directory where binaries are copied before execution.

TraceSummary (string, default "${bindir}/trace-summary")

Path to trace-summary script; empty if not available.

Version (string, default empty)

Version of the broctl.

Miscellanous

Mails

BroControl sents four types of mails to the address given in MailTo:

  1. When logs are rotated (per default once a day), a list of all alerts during the last rotation interval is sent. This can be disabled by setting MailAlarms=0.

  2. When the cron command noticies that a node has crashed, it restarts it and sends a notification. It may also send a more detailed crash report containing information about the crash.

  3. NOTICES with a notice action of NOTICE_EMAIL; see the Bro documentation for how to configure NOTICE priorities.

  4. If trace-summary is installed, a traffic summary is sent each rotation interval.

Performance Analysis

TODO: broctl cron logs a number of statistics, which can be analyzed/plotted for understanding the cluster's run-time behaviour.

Questions and Answers

  1. Can I use an NFS-mounted partition as the cluster's base directory to avoid the rsync'ing?

    Yes. BroBase can be on an NFS partition. Configure and install the shell as usual with —prefix=<BroBase>. Then add HaveNFS=1+ and SpoolDir=<spath> to etc/broctl.cfg, where <spath> is a path on the local disks of the nodes; <spath> will be used for all non-shared data (make sure that the parent directory exists and is writable on all nodes!). Then run make install-broctl again. Finally, you can remove <BroBase>/spool (or link it to <spath>). In addition, you might want to keep the log files locally on the nodes as well by setting LogDir+ to a non-NFS directory. (Only the manager's logs will be kept permanently, the logs of workers/proxies are discarded upon rotation.)

  2. When I'm using the standalone mode, do I still need to have ssh and rsync installed and configured?

    No. The standalone performs all operations directly on the local file system.

  3. What do I need to do when the something in the Bro distribution changes?

    Just re-run make broctl-install. It will reinstall all the files from the distribution, except for any of the template files, as they might have been edited.

    In addition, BroControl has a "development mode", which makes installation easier if distribution files change frequently (e.g., becayse you're working from SVN). Development mode activated by adding DevMode=1 to etc/broctl.cfg. Once done, BroControl's install command will also install all the files from the distribution, just as make broctl-install does. Note that you obviously need to keep the distribution around to have this work.

    Note for folks who have used the old "cluster shell": the development mode corresponds to the old default behaviour, which worked with any make install-broctl.

  4. Anything special to consider when using development versions?

    If you are using a development version, BroControl might require patching Bro itself to work correctly. A "development version" is defined as anything you have downloaded from Bro's Subversion repository (as opposed to downloading a release tar-ball). If so, see if there's a file called aux/broctl/patch-bro.diff. If there is, apply it as follows before you proceed with any of the steps below:

    > cd /path/to/bro/source/distribution
    > patch -p0 <aux/broctl/patch-bro.diff
    > ./autogen.sh

Can I change the naming scheme that BroControl uses for archived log files? Yes, set MakeArchiveName to a script that outputs the desired destination file name for an archived log file. The default script for that task is <BroBase>/share/broctl/scripts/make-archive-name+, which you can use that as a template for creating your own version. See the beginning of that script for instructions.