Welcome to The Machine Emulator

This is TME: The Machine Emulator, an extensible platform for creating & implementing dynamic machine specifications and running/testing/debugging them. Originally conceived by Matthew Fredette at MIT, this is hoped to be the continuation & expansion of that groundbreaking project. Under the auspices of the Phabrics project - to be unveiled at some unspecified later time - it is meant as an independent, standalone product. But, enough with the pleasantries -- on to the nitty-gritty!

What is TME?

TME is a generic machine emulation system for emulating one or more machines. Each of the machines is specified in a special, extensible machine description language (MDL) called tmesh. It includes an interpreter for this language that is used to translate the description into a working, dynamical system that emulates the machine. The elements of the language are keywords that name individual components of the machine. Each component is implemented using dynamically loaded modules that are compiled directly into native machine code. In this way, it is very similar to a virtual machine and is what makes it so efficient compared to many other systems which may interpret the individual component descriptions.

The machine description itself is interpreted, but it creates a statically connected system comprised of multiple dynamic modules that are themselves compiled. As each component keyword is translated, the components they describe get instantiated. This means that the component module is dynamically loaded into the main running program. In addition, components are connected via the "at" keyword. When they are connected, a communications path is established between the components. The path is dependent upon the interfaces presented by the components. This means that the components must have compatible interfaces in order to be connected using "at". The communications protocols are determined by the components as well, and must obviously be compatible. In this way, the machine description language is very close to how an actual technical specification might be accomplished. This makes it a prime candidate for use in prototyping systems as well as for distributing dynamic specifications of production systems. It could also be a potential candidate for a future board-level (or network-level) ESL system (like SystemC) that actually compiles the MDL into an HDL such as Verilog or VHDL and from there into actual hardware systems with associated software toolchains. The possibilities are limitless!

A good way to think of TME is that it is to machines what SPICE is to circuits. In SPICE, the components can be standard, primitive devices like resistors, diodes, and transistors, described using physical constraints. They can be connected to form complex electronic circuits which may then be simulated. Models of new components can be created in the SPICE language to provide novel circuit elements. In the same way, the elements in TME can be standard "primitive" components like ICs, memories, busses, or external devices like NICs and disk drives. New types of components can be modelled by coding their behaviour directly in C source code which, when compiled, exports the required interfaces as dynamically loaded modules using the extremely portable GNU libltdl. They are combined to form more complex systems such as computers that may then be emulated.

Supported Hosts:

One of the major goals of TME is to work on as many hosts as possible. This is done by design when test machines are available, but it is obviously impossible to cover all platforms, even the major ones. Nonetheless, I have tried to cover Linux & the major BSDs in use today, but resource constraints & time considerations limit the testing to just a few as follows -

  1. NetBSD - the original host platform of choice for developing TME. Runs on the most host machine platforms. Now includes support for autonat using NPF. A source package has been added to their pkgsrc-wip repo under tme. See the pkgsrc-wip project page for more info on using it to install and try out source packages.
  2. OpenBSD - the most secure OS and arguably the hardest to develop for as well. If it works here, it should work anywhere.
  3. FreeBSD - the most popular BSD. Good for getting the word out.
  4. PC-BSD - a FreeBSD derivative with a user-friendly graphical interface. Very nice for beginners, but powerful enough for advanced system administrators.
  5. DragonflyBSD - a BSD with many modern features. A good testbed for trying out optimizations.
  6. Fedora Linux - For now, the only Linux testbed, but other distros should follow.
  7. Arch Linux - TME is now available in the Archlinux User Repository! It should install without any problems. See Arch Linux-specific info in the NFTables section below for a caveat, if you wish to use TAP/NAT for Ethernet access.
  8. Windows - Preliminary support for building/running on Windows platforms is provided through Cygwin. It should work out of the box. See below for details. It should also build (natively or cross) on MinGW, and run under the Wine emulator (use --without-x --enable-threads=glib configure options). Prelinary Ethernet support is available for Windows, using OpenVPN.
  9. Other - It should work on any POSIX platforms, sans whatever features aren't available on them. Illumos (a free version of Solaris) is one example that has been tried, but doesn't provide Ethernet support, yet. OS X has NOT been tried, but it should be fairly simple to port to that platform.

Another goal is to work on the latest versions of all these & more. Expanding to other platforms is highly encouraged, and any successes or patches to make it work on others is encouraged. Please send any patches or info about other platforms to phabrics@phabrics.com. As it expands, we would like to incorporate any core patches that expand functionality to other platforms, while keeping the others working. This requires a constant QA cycle across all supported platforms, so may not always be possible. In particular, any expansion should not break existing functionality on a known-good host. This is another goal.

It should be relatively easy to install directly from a source tarball (see below) on any of the hosts mentioned above. Every effort has been made to make this possible with minimal fuss and no extra configuration. Another one of the main goals was to make this as self-contained a package as possible. No other tools should be needed outside of the initial build process to get tmesh working. All configuration is done with the tmesh scripts to keep things as simple as possible.

Packages may be made available at some point for some of the hosts above, but should download original source from the site(s) documented below. The original TME package (which stopped at 0.8 as of this writing) is still packaged on most of these platforms.

Prerequisites:

If building from source tarball, you must have certain packages installed on your host platform of choice. At a minimum, if not already installed, you should install these packages (latest versions should work) -

Required:

Build-time

Link-time

Optional:

Build-time

Link-time

Obtaining:

Source tarball available via anonymous access here. Current version: 0.12beta34

To run sparc64 under pthreads, you can try using the realtime extensions for setting the scheduler to round-robin/fifo combo as such: tmesh -a r1 -m f1 -i e --cpus 1 ./MY-SUN4U. More info to come shortly in the docs.

Note that the only documentation is the README, which is a (currently outdated) copy of the instructions here. We are working to improve the documentation. In particular, note that the tap device configuration should be set up as described in the Network Descriptions section.

On NetBSD, you can get the pkgsrc-wip repo and install it as documented on their pkgsrc-wip project page. The project directory is tme.

On Arch Linux, you can get the AUR package tarball and install it as documented in their guidelines.

Installation:

This should be the standard GNU install as documented in INSTALL. Some general steps ($srcdir refers to location that the source package is extracted to.) -

  1. tar xJf tme-$version.tar.xz
  2. mkdir $builddir
  3. cd $builddir
  4. $srcdir/configure --prefix=$installprefix # non-default configure options: --disable-recode --enable-threads=<threadmodel> --disable-shared --disable-nat --without-x --enable-debug // see notes or output of --help for more details
  5. make
  6. su
  7. make install

Installation step should usually be done as root if you want to configure tap devices with ip parameters in the tmesh descriptions. On *BSD, it will run setuid to allow this, but gives up privilege immediately after. On Linux, it will install with cap_net_admin capability instead. Plugin modules are normally libtool shared libraries, unless the package is configured with --disable-shared, in which case they are static libtool libraries. Installable executable is normally done with shared enabled. Setting LD_LIBRARY_PATH will not work for setuid executables, as it is simply ignored.

Note: If you have trouble linking against dependent libraries, like libltdl or libnftnl, it could be because they are installed in non-standard locations. Try setting LD_LIBRARY_PATH to the location $installprefix/lib or ldconfig $installprefix/lib. You know the drill.

Note: to run setuid, be sure it is allowed on the installation filesystem, i.e., the nosuid bit should not be set there, which it is usually not for standard install locations such as /usr/local

Note: recode doesn't seem to currently work on 64-bit platforms with threading enabled, so --disable-recode is required on these hosts when using multithreading. SJLJ threading (i.e., single-threaded multitasking using setjmp/longjmp) should work with recode on all platforms that it builds on.

Note: --disable-threads is an alternative way to use the classic SJLJ threading model. Also specified by --enable-threads=sjlj

Note: To debug, or if you otherwise have issues running TME, you may try the --enable-debug configure option.

Note: starting with 1.0beta_4, it is no longer necessary to set TME_MODULE_PATH. The program uses the install path directly, but it can still be set to allow alternative directories to be used as well. This may be taken out at some point as it might be a security risk.

Running:

tmesh can run in one of two modes: headless or GUI. To run headless, modify the tmesh script to create the machine with the terminal connected to a serial port and disable the screen. To run GUI, keep the screen configured. See the original tmesh site for more details. tmesh usage, as documented on there should still work fine. More info to come on the phabrics.com website.

Guest platforms:

tmesh is meant to be an extensible platform, meaning that it should be good enough to run any machine description coded according to its standards. The standards have yet to be documented outside of the source code. Right now, the only guest machines available are sun2, sun3, sun4c, and sun4u (ultra-1). These work reasonably well right now, but stability improvements are an ongoing affair. They are meant for demonstration purposes alone, but will run relatively complete emulations of certain configurations as documented on the original TME site.

Some guest operating systems have been tested as well. We will try to document these as we test things out further. It is known that both NetBSD & OpenBSD (latest versions) are known to work pretty well on all the guest machines. In addition, ethernet configuration has been one of the major points of effort in this latest version of TME (the other being upgrading from GTK+2 to GTK+3).

Graphics:

Certain guest platforms can run X11 directly in the TME emulated framebuffer. In particular, I've successfully run X with versions of NetBSD/sparc <= 5.2.3 on sun4c. With 6.0, the move to wscons seems to have caused issues with running the graphics. This is because of some bugs in the bwtwo/cgthree drivers. I have patched these in my own copies to allow it to run and will provide precompiled kernels at some point. In the meantime, the patches can be viewed with NetBSD bug report #49639.

Screenshots:

Here are some screenshots of TME running a modified NetBSD/sparc 7.99.4 beta kernel guest with a cgthree framebuffer (patched to allow X to run).

An X session using fvwm window manager with a picture open using feh:

An X session using fvwm window manager with a pdf open using xpdf:

An X session using ratpoison window manager with a web page open using dillo:

An X session using fluxbox window manager with emacs:

The configuration above also shows a sample network configuration with a gateway (tap) device address set to 10.0.77.1. It is equivalent to what is shown in the Network Descriptions section below. In addition, the network is NAT so we can connect to an external network like the Internet, as can be seen by the ftp session to the netbsd site. That is the topic of the next section.

Networking:

Ethernet:

Ethernet configuration can be done in a couple of ways: BPF or TAP device.

The BPF device is the original method for ethernet configuration. It still works & is still supported. Support for this method has also been added on the Linux platform through the equivalent Linux Socket Filter (LSF) facility. It allows a direct connection to an adapter, but may not work on all adapters, particularly on Linux hosts. Because of this limitation, support for TAP devices was added.

TAP devices make a tap device, which is basically a pseudo-ethernet adapter that is created on demand. This is as opposed to a real hardware ethernet adapter. It is basically an ethernet adapter that runs in software only in the kernel. See the sample tmesh scripts for examples. It can be configured with an ip address, which basically acts as a gateway to the host machine. The guest machine can be configured with an ip address in the same subnet as the tap device, thus allowing for communication between host & guest. Afterwards, a bridge or a NAT can be set up to the adapter, if communication with the external net is desired.

It is possible to combine BPF and TAP devices to create NAT networks. See the section below on Network Descriptions.

PPP:

If you lack an Ethernet pseudo-device like tap or bpf, it is still possible to network the machine via the serial port using the PPP protocol. You must configure one of the serial ports; on Sun machines, these are usually enumerated as ttya or ttyb, depending on which one is configured in your machine description. It also must not be connected to keyboard, mouse, or console of course. In this case, we simulate a NULL modem via the same mechanism used for the console. You must configure a serial port to use a posix serial device. See the sample line for the console, for example. Afterwards, both host and guest must run a PPP program that will allow them to talk over the serial line. Most Unix-y platforms have a PPPD or equivalent; see your system documentation/packages for details.

For example, using a NetBSD guest on a Linux host, you can use the same pppd package on both, with corresponding options. They both offer the same options, so you can use, e.g.:

pppd nodetach noauth local /dev/pts/2 38400 192.168.1.2:192.168.1.1

on the host (replacing pts/2 with whatever slave device is used by TME) and:

pppd nodetach noauth local /dev/ttya 38400 192.168.1.1:192.168.1.2

on the guest (using ttya or ttyb depending on which is configured for your guest machine).

Note that 38400 is the max speed supported by Sun machines (ref?). This configures static IP addresses of 192.168.1.2 on the host and 192.168.1.1 on the guest. Alternatively, you may omit that addresses and get them assigned automatically. If you have an interface to another LAN on the host machine, you can even use Proxy ARP to forward packets from machines on the LAN to this machine and back by using an available address from the LAN subnet. Besides NAT, this is another way to get on an external network pretty quickly. On FreeBSD, there is a user-space PPP program that will set up NAT to an external network as well. See the next section for more info about NAT.

It should be possible to set up a P2P link between two machines by following the above steps on both guests and then using, e.g., socat to set up a relay between the machines to simulate a NULL modem. I haven't successfully done this yet, but I see no reason why it can't be done. In fact, socat can be used for general packet relay and is generally useful for non-standard network setups where other options are not available. See the socat docs for details.

IPv4 Network Address Translation:

tmesh machines support IPv4 and thus require NAT between network segments (although not strictly required if subnetting - see Proxy ARP). IPv6 should work as well, but must be set up manually (and is currently untested). The way NAT is done is host-dependent. There are multiple NAT alternatives from host to host, and even on the same host platforms. We try to support the major ones here, either through autoconfiguration within TME itself, or by providing instructions to allow you to manually customize it to your own setup. Currently, three NAT technologies are supported: NFTables, NPF and PF. NFTables is a Linux-only solution, and is the next-generation packet filtering technology to replace IPTables in the future. NPF is a NetBSD-only tech that serves a similar purpose on that platform, supplanting PF at some point presumably. They are both very similar in that they work in the same way as BPF. They take a set of compiled filter rules that specify what to do with individual packets. PF is the traditional BSD filter.

All will write the rules automatically, enable IPv4 forwarding and start filtering right away. You can disable this behaviour at compile-time (using --disable-nat) or run-time (by specifying a nat interface that doesn't exist on the tap nat option). Alternatively, you may manually add or modify the rules after starting tme to suit your own needs, but tmesh will overwrite them if you run it again -- it does not persist, but uses a static set of rules in the current implementation.

Auto NAT

Linux/NFTABLES NAT Configuration

NAT support is now integrated directly into TME via the NFTables API. TME will directly write a table into the NFTables Netfilter kernel module using its native instructions as compiled through the nft tool and using libnftnl. At a minimum, libnftnl and associated headers (development package) must be installed on the host system to get this functionality. In addition, the nftables kernel module support must be configured into the kernel, but the nft tool is not required unless further manual configuration is required. Without these, it will revert to the same behaviour as 0.9, where configuration must be done manually. It may still be necessary to do further manual configuration depending on your particular host system configuration. For example, if you are still having trouble with NAT forwarding from the tap interface, you may still have to flush the iptables forward table (iptables -F FORWARD) or something similar. Further information about NFTables is available here.

To see the table written out to NFTables, run "nft list table tme". The output should be similar to the following:

table ip tme {
chain prerouting {
type nat hook prerouting priority 0;
}
chain postrouting {
type nat hook postrouting priority 0;
ip saddr $int_net oifname $ext_if snat $host
}
}

where $int_net is the internal network number in CIDR format, $ext_if is the external interface to NAT to, and $host is the host name or address.

Arch Linux Users: TME should install right out-of-the-box using the AUR source package builder. However, if the libnftnl package is installed, it will not compile correctly. This version of TME does not work with the libnftnl package version installed on Arch. You will have to install the latest version of libnftnl manually as documented on the NFTables Wiki. Either GIT version or snapshot should work.

NetBSD/NPF

NAT support is now integrated directly into TME via the NPF API. TME will directly write a ruleset configurtion into the NPF kernel module using its native instructions as compiled through the npfctl tool and using libnpf. At a minimum, libnpf and associated headers (development package) must be installed on the host system to get this functionality. In addition, the npf kernel module support must be configured into the kernel with the same version, but the npfctl tool is not required unless further manual configuration is required. Without these, NAT will not be set up. It may still be necessary to do further manual configuration depending on your particular host system configuration. NPF is included with NetBSD 6.0 or later. Further information available at www.netbsd.org or in the NetBSD man pages.

To see the ruleset written out to NPF, run "npfctl show". The output should be similar to the following:

map $ext_if dynamic any -> $host pass from { $int_net }

group (name "external", interface $ext_if) {
pass stateful out final all
}

group (default) {
pass final all
}

where $int_net is the internal network number in CIDR format, $ext_if is the external interface to NAT to, and $host is the host name or address.

*BSD/PF

The fallback for most other BSD platforms is PF, which is supported on all the major BSD platforms. TME writes the rules directly using pfctl as documented in the next section.

Manual NAT

As root or superuser, run the following commands from a shell as required. $ext_if is the external, physical network card to NAT to and $int_if is the internal, tap device created by the TME configuration. To learn these, use "ifconfig -a" or "ip addr show" and note the names to use here.

Linux/IPTABLES NAT Configuration (tme-0.9)

  1. modprobe iptables_nat
  2. echo 1 > /proc/sys/net/ipv4/ip_forward
  3. * iptables -F FORWARD
  4. iptables -t nat -A POSTROUTING -o $ext_if -j MASQUERADE
  5. iptables -A FORWARD -i $ext_if -o $int_if -m state --state RELATED,ESTABLISHED -j ACCEPT
  6. iptables -A FORWARD -i $int_if -o $ext_if -m state -j ACCEPT

*Note that step 3 is optional; it flushes the FORWARD chain of the filter table; this is to ensure that there are no rules that will block the NAT from working. You may or may not need to do this depending on your setup. If you omit this step and find that there are problems communicating to the external net, e.g., DNS is not working, this is probably why. This is the case, e.g., with the default iptables config on Fedora Linux.

Be sure you know what you are doing here, or consult your nearest system administrator or guru. There's a lot of documentation for IPTABLES available on the web and in manual pages. Unfortunately, IPTABLES does not have a public API with a stable interface for programming, although many have reported success with programming it using the undocumented API. But, this is just a very basic setup to get you going with connecting your TME instance to the Internet.

BSD/PF NAT Configuration

  1. ({Free,DragonFly}BSD) kldload pf
  2. ({Open,Free,DragonFly}BSD) sysctl net.inet.ip.forwarding=1 (NetBSD) sysctl -w net.inet.ip.forwarding=1
  3. * pfctl -F rules
  4. ({Net,Free,DragonFly}BSD) echo "pass from $int_if:network to any keep state" | pfctl -f- (OpenBSD) echo "pass out on $ext_if from $int_if:network to any nat-to $ext_if" | pfctl -f-
  5. ({Net,Free,DragonFly}BSD) echo "nat on $ext_if from $int_if:network to any -> ($ext_if)" | pfctl -f-
  6. ({Net,Free,DragonFly}BSD) pfctl -e

*Again, step 3 is optional depending on your setup. It simply flushes the rules modifier to ensure that nothing gets blocked; it's usually not required. If there are problems with communicating to the external net, this might be why.

Your mileage may vary; this is what worked for me, but you may have a different setup/needs, so use your own discretion and consult who and whatever documentation is required. Again, there is much documentation on PF available. There are also other NAT solutions available on the BSDs, but this seems to be the most flexible and stable. It also has a stable ioctl API for directly programming the rules into a program. I'm also looking into using the new NPF facility in NetBSD, which seems to have an even nicer, functional programming API for direct integration into the tool.

Note that OpenBSD requires only two steps (2 & 4). Again, the goal here is to get the user up and running as quickly as possible with minimal fuss, so this is by no means a comprehensive way to do IP forwarding with NAT. Much documentation exists to assist you there, but hopefully we will have a minimal function built into the tool itself so that these steps won't be required to be done outside the tool.

After NAT is setup, make sure your routes are set correctly in your guests. In particular, make sure the default gateway is set to the ip address of the tap device. Also, if you want to access the external network or Internet, you will have to set up DNS. Usually, you fill in the /etc/resolv.conf with the "nameserver xx.xx.xx.xx" line where xx.xx.xx.xx. is the ip address of the nameserver - usually the same as the host machine's. This is usually all done as part of the process of installing or configuring the guest OS; refer to the guest OS documentation for more details. It is basically the same as setting it up for an internal network as specified by the TME configuration.

Proxy ARP/NDP

An alternative to NAT in certain situations, is to use Proxy ARP. In fact, if you are already behind a NAT firewall, this may be preferable as it allows you to use a subset of the addresses in the subnet to which the host belongs, rather than having to set up a new subnet to NAT to/from.

This is done by allocating some of the address space to the internal network set up by the TAP device and then broadcasting the ARP addresses through Proxy ARP, on the external network. The way this is done is host-dependent and must currently be set up manually. The details can be found in the documentation for your platform-of-choice, if available.

The high-level details are as-follows (using commands on Linux for illustration):

  1. Set up the networking on TAP device as before, without NAT (nat=none), giving it a select pool of addresses from the external network. Make sure they are available first.
  2. Set proxy_arp on the TAP interface, and any interface that packets should be forwarded to/from: echo 1 > /proc/sys/net/ipv4/conf/<if>/proxy_arp.
  3. Enable forwarding, if not already: echo 1 > /proc/sys/net/ipv4/conf/all/forwarding.

In theory, that should be sufficient to get it working; the interfaces should "learn" how to route the packets. But other steps may be required if not working. For instance, you might have to explicitly set the proxy ARP addresses on the external interface, if they don't get set automatically (ip neighbor add <address> dev <interface>). Linux documentation indicates this should not be needed, but I have seen some cases where it doesn't seem to work. Default routes may also need to be set explicitly (ip route add <addresses> via <address> dev <interface>).

NDP is the IPv6 generalization of ARP. The above instructions should apply, with commands used for IPv6 instead (ip -6). If your router doesn't have a route to your host, this may be needed to use IPv6 networking.

OpenVPN

TME networking capabilities are expanded using the OpenVPN tools. These come included with TME to provide alternate methods for network access, besides the built-in BPF and TAP. One advantage of using these is that, in some cases, they allow for greater portability to platforms that don’t have native BPF or TAP. Another pro is that OpenVPN provides value-added capabilities such as encryption, fragmentation, compression, virtual networking, and scripting. These can be used to greatly expand the domain of application to larger systems. There is an ongoing effort to bring in more of this functionality over time.

To begin with, some OpenVPN capability has been imported using modules that map to certain elements in the OpenVPN program. In particular, TUN/TAP devices and link sockets are two new TME elements that can be used to network machines and hosts. The elements can be thought of as subsets of OpenVPN itself, with the capabilities specified using the same command-line options that it provides. Some will work and some won’t, depending on what has been implemented, and what is needed for that element.

The following sections outline these methods.

OpenVPN TUN/TAP Devices

Preliminary support for OpenVPN is now provided for the purposes of creating and using TAP devices on multiple platforms. OpenVPN is compiled in as another host module, with a special tap leaf interface, that uses the same options as OpenVPN itself. See the OpenVPN docs for further information.

In particular, it can be used to configure tap interfaces with IP or IPv6 addresses. It can also be used to run postconfiguration scripts that can, e.g., set up NAT using the filter rules appropriate to the platform. Here is a sample line that can be used to accomplish these:

tap0 at le0: tme/host/openvpn/tap --dev tap --ifconfig 10.0.77.1 255.255.255.0 --ifconfig-ipv6 "2001:db8::1/64" "2001:db8::2" --tun-ipv6 --up nat.sh

Note that this is currently the only way to get networking support under Windows.

OpenVPN Socket Link Devices

OpenVPN provides network links via sockets, for the purposes of creating private networks over existing L3 networks, using IP protocols. We use this to expand ethernet networking capability with P2P links between network elements of machines on different host machines or networks. These links are also known as tunnels because they can be used to make virtual networks through physical networks.

The usage is quite simple: network elements are connected to link elements in the same fashion as they are connected to other devices. You can think of them as virtual ethernet or TAP devices. The link elements are paired together, and can be in the same or other machine configurations to provide a bridge for the ethernet traffic to flow between them.

Link elements are paired by IP address. They are implemented using sockets, so they have a local and remote address. The local address must be specified; the remote may be specified or floating. The encapsulating protocol may be specified as UDP (stateless) or TCP (session-oriented). In the latter case, both machines must be running and connected for either to proceed.

The sample machine configurations provide examples of different scenarios where they might be used. The simplest is a peer-to-peer connection between two machines (or even two elements theoretically). This demonstrates the extreme flexibility of TME in specifying different network configurations over other networks which might not have the needed resources or topology. In particular, TAP devices aren’t available on every platform (ex: Cygwin). You can bridge between a platform that doesn’t and one that does to give external network access. Once again, the possibilities are limited only by the firewalls. You can also use proxies to overcome this (not yet implemented, but coming soon), or customize the firewall rules.

Here is the peer-to-peer connection specification:

Machine A - link0 at le0: tme/host/openvpn/socket/link --dev tap --local 192.168.0.1 --remote 192.168.0.2

Machine B - link1 at le0: tme/host/openvpn/socket/link --dev tap --local 192.168.0.2 --remote 192.168.0.1

This creates a network of two machines, but no external access.

Here’s a network description that allows for two remote machines, each on different hosts, to be connected to two local machines, networked using the BPF/TAP combo, and connected externally via AutoNAT.

Host 1: Machine A - tap1: tme/host/tun/tap inet 10.0.77.1 netmask 255.255.255.0 bcast 10.0.77.255 link2 at tap1: tme/host/openvpn/socket/link --dev tap --local 192.168.0.1 --remote 192.168.0.2 --proto tcp-server bpf1 at le0: tme/host/bsd/bpf interface tap0

Machine B - link3: tme/host/openvpn/socket/link --dev tap --local 192.168.0.1 --lport 1195 --remote 192.168.0.3 bpf1 at le0: tme/host/bsd/bpf interface tap0 bpf2 at link3: tme/host/bsd/bpf interface tap0

Host 2: Machine B - link4 at le0: tme/host/openvpn/socket/link --dev tap --local 192.168.0.2 --remote 192.168.0.1 --proto tcp-client

Machine D - link5 at le0: tme/host/openvpn/socket/link --dev tap --local 192.168.0.3 --remote 192.168.0.1 1195

To avoid confusion, element names in the samples are unique, but the actual names of elements are unimportant, so long as they are unique within a single machine configuration. Also, when connecting more than one link, different ports must be used (or the addresses must otherwise be unique, perhaps using different interfaces). OpenVPN allows for shared ports, but it hasn’t been tested yet. The default protocol is UDP, but TCP can be used as well, using the –proto argument to the openvpn element as shown above.

As you can tell, this can get pretty complicated. The following section is a discussion of the BPF/TAP network bridge, for making a local network on the host. The capabilities and documentation will continue to improve, but TME is a great platform for experimentation. Perhaps, there are other configurations that have yet to be thought of... or better ways of accomplishing these tasks. There are no correct methods, but these idioms are a good starting point.

Network Descriptions

Using the tmesh machine description language, it's possible to describe not only a single machine, but also an entire network of machines. This is done by creating a clever configuration of TAP and BPF network devices which allow multiple machines described by different tmesh scripts to communicate. You can connect the machines via their network devices using the very same statements that are used to connect their other components. In particular, you create a "master" machine that includes a tap connection that is then connected to by other "slave" machines using bpf filters. The master's connection can act as an application-level gateway by including a nat configuration as well. You may then nat it to other subnets or to the external network as described above. The slaves simply connect to the tap device the same way they would to a real NIC. All slaves connected in this way can directly communicate with each other or the master machine using ethernet protocols. When configured correctly, they may also communicate with the external machine or other networks outside the internal network via nat.

The naive approach is to create the master tap system the same way you would any ordinary nat machine. In other words, the emulated NIC connected directly to a tap device that may be nat'd to an external network. Other machines could then be added to the internal network thus created, by using bpf to set a filter on the tap device. This almost works. All the machines in the network can communicate with each other. But only the master machine can communicate with the external network. Why is this? The answer is that writing to the bpf does not write back to the tap device. To communicate outside the network, it is necessary to write to the tap. But remember that the tap device is actually a software NIC that kind of works in reverse of how the bpf devices do. Writing to tap sends packets to the machine. Writing to bpf sends to the network. Reading is the opposite in both cases, i.e., tap receives packets from the network, bpf receives them from the machine. (It's actually a little more complicated than that as bpf reads also read from the network, but that's not an issue.) So, why is this a problem? Well, think about what happens in the case where one of the bpf machines sends a packet with a route through the tap device's address (i.e., a default, gateway route). The tap will never see this packet! This is because the write to bpf sent it "out" on the tap device's network, so tap never saw it.

Confusing, isn't it? How do we resolve this dilemma? The solution is a little tricky, but it actually works out quite well. More importantly, it works within the original methodology of TME, once again validating its approach to describing complex systems through simple components and working without requiring any updates to it. Plus, once you figure it out, it's quite straightforward and provides a way to build up ever more complex configurations from simple connections. You just have to understand how the information flows between network components in tmesh. Each network component is one side of a paired connection. There is a read path and a write path and they are both connected to the same other side. So when network components are connected, a bidirectional path is set up between them, much like a pipe. Writes to one end will be read from the other. This is why they must be paired. As it turns out, this is all you need to set up just about any network configuration. The key lies in how you set up the master tap connection to the network. The sample tap machine description simply paired the emulated NIC with the tap device. But, this is not good enough. We also have to write back packets sent through bpf filters set on the tap. As alluded to, this can already be accomplished by setting up another pairing. Instead of pairing the tap directly with the emulated NIC, we pair the tap with itself instead! What this accomplishes is to make the packets written out to tap get written back to the tap device, creating a loopback scenario. This allows tap to see the packets written by bpf. Here is a sample machine description that establishes the writeback tap necessary to create a fully nat'ing network of machines! Note that it is a simple modification of the standard sun4c description included with the latest tme distribution. In particular, the relevant lines describing this master configuration are changed to this:

tap0: tme/host/tun/tap interface tapA inet 10.0.77.1 netmask 255.255.255.0 bcast 10.0.77.255
tap0 at tap0
bpf0 at le0: tme/host/bsd/bpf interface tapA

There are several things to note here. First, the tap device, tap0, is no longer connected directly to the emulated NIC, le0. Instead, it is now a root node. Second, you may see duplicate ICMP packets when doing pings. This is not because of suboptimal routing, but because the bpf device detects both reads and writes of packets to the tap device. Since all the packets are written back, they are all duplicated. This is inefficient, since only packets destined for tap need be written back. Unfortunately, there's no way around this in the current version of TME, because we can't yet set a filter on tap devices directly. Future revisions should fix this issue. Finally, the first two lines are unique to this master configuration. The last line is shared among all the slave configurations as well as this one. The bpf filter is set on the tap device by naming its interface the same as tap0's. It is paired with the emulated NIC (le0), effectively connecting the each slave machine to the tap network.

Note: The sample master config was for Linux. The interface name in particular might need to be changed depending on whether the platform supports that or not. In NetBSD, for instance, the default names start at tap0 and go up from there.

Note: The different machines should run in their own processes in order to be efficient. When looked at this way, the ethernet trunk is analogous to an IPC bus connecting the processes. Indeed, it might make a nice alternate mechanism for just such a purpose. The tap/bpf devices each run in a separate "thread" within each process, just like all the other components. The more components, the more threads needed; currently the threading model is cooperative using a coprocess technique that does not take full advantage of the multithreading capabilities of modern processors. This is one area for future improvement.

Network Services

tmesh does not currently integrate any support services such as DHCP or DNS. This must be done using external, third-party programs. For instance, dnsmasq provides a pretty simple way to set up a DHCP/DNS server by setting it up on the tap device or somewhere on the network connected to which the machine(s) are connected. For example, you could set it up on a tap master to provide these services to the machines in the internal network. Another possibility is to run it on a machine inside the network itself. dnsmasq is a good solution since it is lightweight and aimed at embedded devices. You might also consider using it for a netboot configuration using its BOOTP capabilities. You can also use ISC's dhcpcd or mdnsd which are usually included with BSD hosts. But for now, if you don't set these up, you can still set up static configurations in your guests using ifconfig to set up static ip addresses and route to set up routes. You can also set up DNS statically using the hosts and resolv.conf files in the guests. You can set up a static gateway/nat as described above using tap. If you are using IPv6, then you can also set up router advertisement to the TAP interface, using radvd (Linux), rtadvd (BSD), or dnsmasq (portably). See docs for details...