Wed 31 December 2014

Using Warden to setup jails in FreeBSD

This is a guide on installing jails on FreeBSD using Warden. Warden is a new system for managing jails, originating from the PC-BSD project. I switched to Warden after using ezjail for some time. The reason for switch was that ezjail started showing some errors on FreeBSD 10.1, and Warden seems more streamlined.

Installation and initial setup

Installing Warden is as simple as:

cd /usr/ports/sysutils/pcbsd-utils && make install clean

Afterwards, I modified /usr/local/etc/warden.conf to contain the following:

#!/bin/sh
# Configuration options for the Warden
######################################################################

# Network Interface for the jails to use
NIC: lo1

# Directory to use for compressing / decompressing files
WTMP: /usr/jails

# Location of the jails
JDIR: /usr/jails

# When automatically creating jails with unspecified IPv4 addresses, use this
# address at the starting point for new addresses
IP4POOL: 10.10.10.0

The reason for using lo1 as the networking card is because I want to have a greater control over the traffic to and from jails. Now, in order to get lo1 on my system, I created a /etc/start_if.lo1 file:

cat >/etc/start_if.lo1 <<EOF
#!/bin/sh
/sbin/ifconfig \$1 create
EOF

Also, I included lo1 in the network_interfaces setting in /etc/rc.conf file.

Jail time!

The first jail I created on my FreeBSD NAS was for btsync, a file syncing tool that I use to sync many things in my household.

I had to decide first on the IP for this jail, and I chose 10.10.2.30. This is a is a private network, RFC1918 address, local to the machine only, as I don’t want to bind this jail to any address on my home network. Thus, pf on the host machine will do all the NAT/redirecting.

warden create btsync --ipv4=10.10.2.30/32
warden start btsync
warden chroot btsync
# do the installation and setup of btsync
...

In addition, I wanted btsync to store all synced data on a specific ZFS filesystem. This can be achieved by calling:

warden fstab btsync

and adding the following in the editor:

# Device        Mountpoint      FStype          Options Dump Pass
/mnt/btsync_data /usr/jails/btsync/btsync_data nullfs rw 0 0

Since ZFS filesystem is mounted on /mnt/btsync_data on the host, this line causes Warden to mount it to /usr/jails/btsync/btsync_data. As btsync jail resides at /usr/jails/btsync, this effectively means that the jail will see this ZFS filesystem at /btsync_data.

Finaly, one needs to set up pf to NAT the traffic correctly. To do so, first ensure that pf is enabled in /etc/rc.conf, that is, these 2 lines are present:

pf_enable="YES"
pflog_enable="YES"

Then, add something like this to /etc/pf.conf file:

ext_if="re0"
# btsync stuff
BTSYNC_IP="10.10.2.30"
BTSYNC_PORTS="{51551,50543}"

set skip on lo0
scrub in all

nat on $ext_if from $BTSYNC_IP to any -> ($ext_if)

rdr on $ext_if inet proto tcp from any to port $BTSYNC_PORTS -> $BTSYNC_IP
rdr on $ext_if inet proto udp from any to port $BTSYNC_PORTS -> $BTSYNC_IP

pass all

Et voilá! The jail is ready to get some data in :)

Additional tricks for easier handling of pf.conf

Modifying /etc/pf.conf to accommodate all new jails can be tedious and error-prone task. Thus, I created this little perl script that can take an input file (I keep it at /usr/local/etc/jailnet.conf) and produces a nice set of settings that can be put in /etc/pf.conf (brave souls can try to directly overwrite it).

The script will take an input describing mappings for jails, in the form:

btsync;udp:51551,tcp:51551,tcp:50543
owncloud;tcp:8081/80,tcp:6000-6010,udp:2000-2010/3000-*,tcp:2011-2015/8112

and output two files, /etc/pf/jails.options.conf and /etc/pf/jails.translation.conf. These two files should be included in /etc/pf.conf via the include directive.

The input contains one line per jail, with two main fields separated by ‘;‘: jail name and port settings. Port settings are a ‘,‘-separated list of ports, in form protocol:portrange. Protocol can be either ‘udp’ or ‘tcp’. Port range can be either a sole port number, or a range in the same format pf that understands, just with ‘:‘ being replaced by ‘-‘ here. An addition that the script understands is useful when redirecting a different port on the host to a port on the jail. The format is ‘rangehost/rangejail’. Again, just like with pf.conf, one can write ‘begin_port-*’ in ‘rangejail’, and pf will do a one-to-one mapping of port in ‘rangehost’ to ‘rangejail’. Naturally, one can have a single port in ‘rangejail’ (and/or ‘rangehost’), and all ports from ‘rangehost’ will be redirected to a single port defined in ‘rangejail’.

Finally, the script is as follows (an can be fetched from GitHub):

#!/usr/bin/env perl

use strict;
use warnings;
use v5.10; # because we use some new stuff
use feature qw(say);

my $card = 're0';
my $opt_file = "/etc/pf/jails.options.conf";
my $nat_file = "/etc/pf/jails.translation.conf";

my %ips = ();
my %ports = ();

# Parses an input string containing the port list the format of the
# list is:
# 'proto':port[-portend][/jail_port[-jail_port_end]]
#
# If portend is given, function will return all ports in range port to
# portend If jail_port is given, function will return values that
# notify the caller that translation should map host ports port to
# portend (on host) to jail_port to jail_port_end (on jail)
#
# jail_port_end might be omitted, in which case all input ports will
# map to a single port. If jail_port_end is '*', there will be a
# one-to-one mapping from the range of ports on host to the range of
# ports on jail, starting with 'port' and 'jail_port'.
#
# This function returns a reference to a hash whose key '$proto'
# contains a hash whose keys are the ports that translate to the same
# ports on the host. The returned hash reference might also contain
# another key, '${proto}_special' that describes port mapping through
# a hash whose keys represent host ports, while values represent host
# ports.
sub get_ports($$) {
  my ($proto, $ports) = @_;
  my @ports = map { s/$proto://; $_ }
              grep { /$proto:/ }
              split /,\s*/, $ports;

  my %out = ();
  foreach my $port (@ports) {
    my ($rangeh, $rangej) = split /\//, $port;
    $rangej = $rangej // $rangeh;
    $rangeh =~ tr/-/:/;
    $rangej =~ tr/-/:/;

    say STDERR "comparing '$rangeh' to '$rangej'";
    $out{$proto . ($rangeh eq $rangej?'':'_special')}->{$rangeh} = $rangej;
  }
  return \%out;
}

# Prints the option part of pf.conf
# This function creates variables 'jail_proto_ports' containing the
# list of ports that will be used verbatim on host, and a list of pairs of
# variables named 'jail_proto_ports_JX'/'jail_proto_ports_HX', where X
# is a number, which represents pairings of ports used on the jail and
# on the host, respectively.
sub print_opt($$$$) {
  my ($fout, $JN, $proto, $ports) = @_;
  my @normal = keys %{$ports->{$proto}};
  my $special = $ports->{$proto . '_special'} // {};
  say $fout qq(${JN}_\U$proto\E_PORTS="{) . (join ',', @normal) . qq(}") if 0+@normal;
  my $cnt = 0;
  foreach my $port (keys %$special) {
    $cnt++;
    say $fout qq(${JN}_\U$proto\E_PORTS_H${cnt}="$port");
    say $fout qq(${JN}_\U$proto\E_PORTS_J${cnt}=") . $special->{$port} . qq(");
  }
  return (0+@normal, $cnt);
}

while (<>) {
  # skip comments
  next if /^#/;
  chomp;
  my ($jail, $ports) = split /;\s*/;
  $ips{$jail} = `warden get ipv4 $jail`;
  chomp($ips{$jail});
  print STDERR "JAIL: $jail -> $ips{$jail}\n";
  my $udp_ports = get_ports('udp', $ports);
  my $tcp_ports = get_ports('tcp', $ports);
  $ports{$jail} = { %$udp_ports, %$tcp_ports };
}

print STDERR "Writing data to output files\n";
open my $fopt, '>', $opt_file or die "Can't write to $opt_file: $!";
open my $fnat, '>', $nat_file or die "Can't write to $nat_file: $!";

foreach my $jail (keys %ips) {
  my $JN = uc($jail);
  say $fopt qq(${JN}_IP="$ips{$jail}");

  my (%ok, %cnt);
  ($ok{udp}, $cnt{udp}) = print_opt($fopt, $JN, 'udp', $ports{$jail});
  ($ok{tcp}, $cnt{tcp}) = print_opt($fopt, $JN, 'tcp', $ports{$jail});

  say $fnat qq,nat on $card from \$${JN}_IP to any -> ($card),;
  foreach my $proto (qw(udp tcp)) {
    say $fnat qq,rdr on $card inet proto $proto from any to port \$${JN}_\U$proto\E_PORTS -> \$${JN}_IP, if $ok{$proto};
    foreach my $c (1..$cnt{$proto}) {
      say $fnat qq,rdr on $card inet proto $proto from any to port \$${JN}_\U$proto\E_PORTS_H$c -> \$${JN}_IP port \$${JN}_\U$proto\E_PORTS_J$c,;
    }
  }
}

close $fopt or die "Can't close $fopt: $!\n";
close $fnat or die "Can't close $fnat: $!\n";