connecting the cloud with ethernet

One day I was bored at work. I was stuck in a training program I didn't need to be in and had literally nothing better to do than piss around. I also had recently read through Xe's excellent article Anything can be a message queue if you use it wrongly enough.

If you haven't heard of it before, go read the post. It's got a lot more and a lot deeper background than I'm going to go into here. The relevant parts for me, though, are the bits about Unix philosophy and TUN/TAP devices.

TUN/TAP devices

In Unix, you get access to network hardware through syscalls. You can go pretty far down the network stack, but at the end of the day you're handing data to the kernel that you want packed up and sent through a wire. But what if you don't want to use a wire but you still want to use the kernel's networking stack?

Enter the Universal TUN/TAP Driver! This Linux kernel driver creates a virtual network device that still uses the kernel's networking stack, but hands outputs to (and accepts inputs from) user mode programs instead of physical hardware. A TUN device operates with IP packets while a TAP device carries Ethernet frames.

You'll know from reading it, that Xi's post used a TUN driver to send IP packets through Amazon S3. And me being me, I wasn't happy with just recreating that. No, I had to do something new.

Amazon Kinesis

Kinesis is a data streaming service offered by AWS. You can set up a series of "shards" to handle a high volume of streamed data from multiple sources with low latency. Additionally, Kinesis has a feature called-

Ok I promise this is not a sales pitch, I just really like Kinesis.

-enhanced fan-out that lets stream consumers subscribe to specific shards and get data pushed down to them. This is great because it means I don't have to deal the problem Xi ran into of pulling network packets!

one more thing: Ethernet frames

An Ethernet frame is what is contained in an Ethernet packet. They don't hold a lot of data, just a couple MAC addresses, client data (the IP packet), and some extra verification data that I don't care about. This is all the data necessary to get an Ethernet frame from one computer to another, provided that the other computer is directly connected to your Ethernet adapter hardware.

Anyway, why does that matter? The short answer is that we need to know what MAC address to insert into our Ethernet frame when sending it to Kinesis. The long answer is that it actually does not matter at all! Ethernet is a solved problem, so we can just rely on higher layer elements of the system to handle address resolution. Since what I'm doing here is essentially plugging two Ethernet physical adapters into each other, all I have to do is give each side its own static IP address and let the TAP driver do its ARP magic.

However, I think it's important to consider what's actually going on here. The "interface" (in this case the TAP driver) is asking around to figure out the MAC address that corresponds to a specific IP address. It knows what IP address to ask for based on what IP address we're trying to send things to. There's no automatic address assignment going on here, this is plain Ethernet baby!

the program

Ok, enough preamble.

Xe originally used Golang for their S3 implementation. I had never used Go before, but that has never stopped me. Plus Go has a couple really nice libraries for working with Ethernet frames as well as a complete Kinesis Client Library (KCL) package.

Unfortunately, like Xe, I also cannot share any code samples. I sure as hell was not going to pay for Kinesis myself so all of the code I wrote for this belongs to my employer. I regret to inform you this project is unlikely to end up on my employer's GitHub.

The program starts by creating a virtual TAP adapter using the water Go library. It does a little configuration of the adapter using the netlink package; things like naming the interface and setting an IP address.

One important thing it does though, is grab the generated MAC address for this new interface. This is important later to figure out which Ethernet frames are destined for our TAP interface.

When it comes to interfacing with Kinesis, I did a little back and forth querying to figure out what partition keys I needed to ensure I'm sending data to a specific shard. Unfortunately, since Kinesis operates on that shard model you can't just throw data at it and expect it to be everywhere all at once. Well, you kind of can, but not with Enhanced Fan-out. Data submitted to the Kinesis stream is distributed to a single shard in the stream based on a "partition key". Partition keys are designed to be a set of keys uniformly spread from 0 to some value, and each shard handles data partitioned to a subset of that range.

Since Enhanced Fan-out requires you to subscribe to a specific shard we need to make sure our partition key is within the range that our target shard deals with. That's actually pretty simple, a quick query of the available shards can give us that information. We just have to provide a shard ID (called a shard ARN (Amazon Resource Name)) to the program and use that to select which shard's partition key range we use.

With all that done, the program just has to set up two concurrent tasks: transmit packets coming from the kernel to Kinesis and receive packets from Kinesis and give them to the kernel! The transmit task is dirt simple and the receive task is only a little more complicated since it has to deal with subscribing to a shard based on its shard ARN. Though note that the receive task also has to ensure it does not receive the device's own Ethernet frames (I did some quick filtering using another package by the creator of water).

This performed shockingly well! Unfortunately I lost my screenshot, but I was getting somewhere in the neighborhood of 200ms of latency.

limitations

I want to address a couple limitations here, starting with the solvable ones.

Technically, my implementation does not actually give two shits about the target MAC address. Because I was testing in a very limited environment I simply filtered out any packet that came from a device's own TAP interface and just never connected more than two devices to any given Kinesis shard. A better implementation that makes the shard more akin to a layer 2 switch is left as an exercise for the reader (hint: be sure to keep in mind broadcast addresses!).

Additionally, Enhanced Fan-out subscriptions time out after 5 minutes. Not 5 minutes of inactivity, 5 minutes of subscription. After that you need to resubscribe to the shard, which I did not care enough to implement.

There are a couple unsolvable limitations though, and they are to do with Kinesis' throughput. While yes, technically there is no upper limit to Kinesis' overall throughput, there is a limit to the throughput of individual shards and that is 1MB/sec or 1,000 records/sec (write). You can also only subscribe 20 registered consumers to each data stream, not even just to each shard. Gigabit Ethernet this is not.

anyway

Don't do this. And when you do, don't use it for anything practical.