Building a static flanneld binary on Ubuntu

03 Dec 2016

I just spent some time trying to build flannel and since there were some nuances, I decided to list the instructions here.

Install build dependencies:

 sudo apt-get install linux-libc-dev golang gcc

Make Go directories:

 mkdir -p ~/go/src
 cd ~/go/src
 export GOPATH=~/go

Clone flannel:

 git clone https://github.com/coreos/flannel.git

Install Go dependencies:
```
 cd flannel
 go install
```

Since I wanted a statically linked binary, I edited the Makefile and updated the build instruction like this:

 dist/flanneld: $(shell find . -type f  -name '*.go')
     go build -o dist/flanneld \
        -ldflags '-extldflags "-static" -X github.com/coreos/flannel/version.Version=$(TAG)'

Now build the binary:
```
 make dist/flanneld
```
flanneld binary should now be created in the dist directory. You can strip it to make it smaller:
```
 strip dist/flanneld
```

One other problem I encountered was that you need at least 2GB of RAM for this. I was trying this in a VM with 1GB and I ran out of memory.

DPDK, tamed!

16 Apr 2016

DPDK is a fantastic piece of software. I’ve used it both for work and in my hobby projects (yes, I crunch packets as a hobby, say what you will about me!) and it works great. My only grievance with it has always been the complicated build system it forces on you. It’s ugly and inflexible and it sometimes drives you crazy.

Yesterday, I saw that DPDK is in Ubuntu’s repositories. It took me some time to realize wha that means. I was looking for the value of the RTE_SDK variable when it clicked in my head; I could just use the damn thing like any normal library; with -l, -L, -I and all that. I just needed to add a -msse4.2 flag for the compilation to work properly (I’ve been adding that flag to the TOOLCHAIN_CFLAGS make variable on some of my machines to make DPDK compile anyway).

Does it have any drawbacks? I have no idea, as I never managed to penetrate the many layers of DPDK make files to see what they really do. It’s quite possible that this is not as efficient as compiling DPDK yourself so it can detect and use the full capabilities of your machine, but at the very least I learned a thing or two about building DPDK by looking at the the Git repository of the Ubuntu maintainers. Just do a git clone https://git.launchpad.net/~ubuntu-server/dpdk. Switch to the ubuntu-xenial branch to see the interesting bits.

This seems to be a Canonical endeavor, so thank you kind folks in Canonical!

Controlling rtorrent with XML-RPC

10 Feb 2016

I’m a huge fan of rtorrent. It’s simple, lean and elegant. I have so far only used it as an interactive BitTorrent client inside screen/tmux or as a batch downloader with a watch directory.

There is more to rtorrent however. It supports XML-RPC which means you can control it programmatically. As I was in dire need of a client I can manipulate from a script, I spent some time today to setup rtorrent correctly and call it remotely.

First we’re going to need something like this line in our .rtorrent.rc file.

scgi_port = localhost:5005

You can also make rtorrent listen to a UNIX socket:

scgi_local = /tmp/rtorrent-listening-here.sock

After that, we need a web server to forward our requests to rtorrent, and responses back to us. The web server will be communicating with rtorrent through SCGI. I use nginx for this purpose. Here’s the relevant nginx config for this:

server {
    listen 8000;
    server_name localhost;
    root /rtorrent/;
    location ^~ /RPC2 {
        include scgi_params;
#        scgi_pass  unix:/tmp/rtorrent-listening-here.sock;
        scgi_pass   127.0.0.1:5005;
    }
}

After this, reload nginx config, make sure rtorrent is running, and you’re ready to go.

In order to test your setup, or contact rtorrent from a shell script, you can use the xmlrpc utility accompanied by the libxmlrpc library. You can get this in Ubuntu by running sudo apt-get install libxmlrpc-core-c3-dev. After that, run this:

xmlrpc localhost:8000 system.listMethods

If everything is done correctly, you will get a list of RPC methods supported by rtorrent.

Some useful commands:

Add a torrent file and start downloading it:

 xmlrpc localhost:8000 load_start ~/foo.torrent

See list of current downloads:
```
 xmlrpc localhost:8000 download_list
```

Start a torrent:

 xmlrpc localhost:8000 d.start 07BF391725A0891F3A1E84784B078D5EF337F82F

Stop a torrent:

 xmlrpc localhost:8000 d.stop 07BF391725A0891F3A1E84784B078D5EF337F82F

Get the status of a torrent:

 xmlrpc localhost:8000 d.state 07BF391725A0891F3A1E84784B078D5EF337F82F

There are a lot more commands. Try some of them for yourself. I admit that it’s difficult to find out how each command works this way, but in the absence of a comprehensive documentation, it seems to be the only way. There are a few examples here.

Complete Vagrant Setup

I have put the complete setup for use with vagrant in this github repository. Simply clone the repo, and run vagrant up in its directory. After the VM is up, you can access rtorrent on the forwarded port 8080:

xmlrpc localhost:8080 load_start /vagrant/foo.torrent

Python/asyncio based BitTorrent tracker

24 Jan 2016

I just uploaded a Python/asyncio-based UDP BitTorrent tracker to my github account. It’s called pybtracker and you can find it here.

You can install pybtracker using pip by running pip3 install pybtracker. You’ll need Python 3.5. After installing pybtracker you can simply run it like pybtracker -b 127.0.0.1:8000 -O. An interactive client is also included which can be used by running pybtracker-client udp://127.0.0.1:8000 (update server address as you wish).

Letting AsyncIO choose a port for you

19 Dec 2015

Say you are writing a test case for an AsyncIO-based network function. You want to write a test server and have the code being tested connect to it. You can choose a port number and hope it’s not taken when the test is run, or you can have a free port chosen for you each time the test is run. Simply pass 0 as the local port number:

    transport, protocol = await loop.create_datagram_endpoint(
        ServerProto, local_addr=('127.0.0.1', 0))

Afterwards, you can query the transport object for the port number chosen for you:

    server_addr, server_port = transport._sock.getsockname()

And that’s it; now you know the port number. I’m not sure if using a member variable with an underscore at the start of its name is the best or the official way to do this but it’s the best method I’ve found so far.

Older Newer