build a bittorrent tracker

So there are quite a bit of posts and guides that tell you how to build a BitTorrent client, but there’s practically nothing out there on how to build another important piece of the protocol: the tracker. Why is that? Perhaps it’s because of their possible use in illicit operations, even though they can (and have) been used to distribute completely legal media across the internet. In any case, it’s not illegal to tell you how to create one, so this guide will show you how to make a basic HTTP tracker.

To be clear, I completely understand that BitTorrent trackers have been used by some as a means for distribution of potentially copyrighted material. I do not condone said actions and I encourage others to abide by the laws and regulations of their country of residence.

To begin, you should have a basic understanding about BitTorrent and its concepts. If you’ve ever used a BitTorrent client, you can probably skip this section. I’m going to be using Rust for this project. If you don’t know Rust, that’s okay; all of the basic requirements are the same regardless of programming language, and I’ll be sure to indicate when something is Rust-specific. Also, if you need it, I’ll include links to the relevant commit for each section.

bittorrent specs

Okay, so we want to make a BitTorrent tracker, that’s great. But how do we know what to implement in order to properly interact with clients? Specifications are helpful in that regard, and there happens to be one for exactly this purpose: the BitTorrent protocol specification (BEP 003). This document should contain all of the information we need to create a tracker.

But it doesn’t.

Don’t get me wrong, it’s fairly descriptive. But it leaves a good bit of the implementation up to interpretation, and when different people start working on a vague spec, differences are bound to arise. Thankfully, there’s a secondary document that’s been created by the development community with the intention of being a formal specification, and it’s much more prescriptive. We’ll use both of these documents to create our tracker.

spec breakdown

After carefully reading the specs, it seems that the tracker should (at a minimum) do a few things:

• the tracker should support an announce operation in which clients update the tracker with their most recent information regarding a particular torrent
• the tracker should support a scrape operation in which clients request up-to-date information regarding all or a particular set of torrents
• the tracker should parse GET requests that have certain keys and values
• the tracker should respond to GET requests with responses containing certain keys and values
• tracker responses should be bencoded

Let’s take a moment to consider what set of components we would need to implement to satisfy the minimum requirements. In my estimation, that would be a webserver, announce/scrape routes, stores for peer swarms and torrent information, and a bencoder (whatever that is).

So let’s get started on the foundation of the tracker: the webserver.

webserver

commit: 8a38f2a

Without the webserver, we won’t be able to accept requests nor send responses, which would really put on a damper on the rest of our plans. So let’s create the smallest working example and progress from there.

use axum::{
    routing::get,
    Router,
};

#[tokio::main]
async fn main() {

    let app = Router::new().route("/", get(|| async { "Hello, World!" }));

    axum::Server::bind(&"0.0.0.0:3000".parse().unwrap())
        .serve(app.into_make_service())
        .await
        .unwrap();
}

In the first few lines, we’re importing from the axum crate. axum is a framework for creating web applications. There are quite a few options for Rust developers and many more in other languages; feel free to pick the one that makes you comfortable. After that, we have our main() function. And if we run our server with cargo run and visit the page at localhost:3000, we should see “Hello, World!”

❯ curl localhost:3000
Hello, World!

So we know the webserver works. Let’s extend it by talking about routing.

routing

Looking back at that breakdown that I shared in the previous sections, we see that the tracker needs to support two different kinds of operations: announce and scrape. An announce is a periodic GET request sent from a BitTorrent client to a tracker and contains recent data about a particular torrent (included as query parameters following the announce URL); the tracker responds with the corresponding metadata along with a list of peers that are also seeding or leeching the same torrent (more info on that in a moment). Similarly, a scrape is a GET request that queries the state of a given torrent or all torrents that the tracker is currently managing (this is up to the tracker itself); the corresponding response includes a smaller set of metadata than that of the announce response.

Additionally, as we said in the breakdown, tracker responses should be bencoded before they’re sent to the client. Now you may be asking “what is bencode?”; I’ll talk about it more in a later section but essentially, bencode (bee-encode) is an encoding scheme used to transmit loosely structured data.

announces

Let’s focus on understanding and implementing announces.

requests

commit: 9500ee2

Announce requests are generally comprised of the following data:

• info_hash: a URL-encoded 20-byte SHA1 hash of the info key of a torrent’s Metainfo file
• peer_id: a URL-encoded 20-byte string that serves as a unique ID for the client
• port: the port number that the client listens on
• uploaded: the total amount (in bytes) uploaded by the client since sending the started event to the tracker
• downloaded: the total amount (in bytes) downloaded by the client since sending the started event to the tracker
• left: the number of bytes the client still has to download to be 100% complete
• compact: a boolean value indicating that the client accepts a “compact” response
• no_peer_id: a boolean value indicating that the tracker can omit the peer ID field in the peers part of the response

They can also optionally include the following data:

• event: a value informing the tracker whether the client has started, stopped, or completed downloading
• ip: the true IP address of the client
• numwant: the number of peers that the client would like to receive from the tracker
• key: additional identification to help the tracker to prove the identity of a client if their IP address changes
• trackerid: additional identification to identify the tracker to the client

Let’s stop for a moment and turn this into Rust.

enum Event {
    Started,
    Stopped,
    Completed,
}

struct AnnounceRequest {
    info_hash: Vec<u8>,  // 20-byte URL-encoded SHA-1 hash
    peer_id: Vec<u8>,    // 20-byte URL-encoded string
    port: u16,
    uploaded: u64,
    downloaded: u64,
    left: u64,
    compact: bool,      // 1 = true, 0 = false
    no_peer_id: bool,   // 1 = true, 0 = false
    event: Option<Event>,
    ip: Option<IpAddr>,
    numwant: Option<usize>,
    key: Option<String>,
    trackerid: Option<String>,
}

In this excerpt, we’ve encoded each key into its corresponding type and wrapped the optional keys in Option types. Each request that comes in the /announce route will be parsed into this type according to its query parameters; eventually, we’ll implement error handling for when requests don’t match this type. With that being said, let’s move onto creating an initial version of a handler for announce requests.

Trackers receive request data through the use of query parameters, which are extensions of a URL that allow for values to be assigned to specific parameters. These can be used to inform a web server to use special logic according to those parameters. For example, if I send a request to http://localhost:3000/example?a=1&b=2, I’m interacting with the /example route and sending key-value pairs of a: 1 and b: 2.

So we need to ensure that the announce handler can properly parse the tracker request from the query parameters. The most straightforward way to do that would be taking the query string (the part after the ? in a URL), splitting the string on the & separator, and parsing each of the elements; you could then ensure that each of the individual elements is well-formed, meaning that an element conforms to the expectations of the specification. I’m not going to do that for this guide, but if you want an example, I have an old implementation here.

Speaking of well-formed parameters, a number of the request parameters have special expectations:

• info_hash is a 20-byte URL-encoded SHA-1 hash
• peer_id is a 20-byte URL-encoded string
• compact and peer_id need to be boolean values, but typically use values of 1 or 0
• optional values need to be supported

In order to safely and successfully interact with clients, your chosen implementation needs to meet these expectations at a minimum. Clients can transmit malformed data (sometimes even intentionally!), so proper checking of these parameters is paramount. Broadly speaking, proper checking of anything and everything is a good idea™ in the context of distributed computing.

Here’s an excerpt of the Rust implementation that I created, which leverages the serde crate:

#[derive(Deserialize)]
struct AnnounceRequest {
    #[serde(default, deserialize_with = "deserialize_url_encode")]
    info_hash: Vec<u8>,
    #[serde(default, deserialize_with = "deserialize_url_encode")]
    peer_id: Vec<u8>,
    port: u16,
    uploaded: u64,
    downloaded: u64,
    left: u64,
    #[serde(default, deserialize_with = "deserialize_bool")]
    compact: bool,
    #[serde(default, deserialize_with = "deserialize_bool")]
    no_peer_id: bool,
    #[serde(default, deserialize_with = "deserialize_optional_fields")]
    event: Option<Event>,
    #[serde(default, deserialize_with = "deserialize_optional_fields")]
    ip: Option<Ipv4Addr>,
    #[serde(default, deserialize_with = "deserialize_optional_fields")]
    numwant: Option<usize>,
    #[serde(default, deserialize_with = "deserialize_optional_fields")]
    key: Option<String>,
    #[serde(default, deserialize_with = "deserialize_optional_fields")]
    trackerid: Option<String>,
}

fn deserialize_url_encode<'de, D>(deserializer: D) -> Result<Vec<u8>, D::Error>
where
    D: Deserializer<'de>,
{
    let buf: &[u8] = de::Deserialize::deserialize(deserializer)?;
    let decoded = urlencoding::decode_binary(buf).into_owned();
    if decoded.len() == 20 {
        return Ok(decoded);
    } else {
        return Err(de::Error::custom(
            "URL-encoded parameters should be 20 bytes in length",
        ));
    }
}

fn deserialize_bool<'de, D>(deserializer: D) -> Result<bool, D::Error>
where
    D: Deserializer<'de>,
{
    let s: &str = de::Deserialize::deserialize(deserializer)?;
    match s {
        "1" | "true" | "TRUE" => Ok(true),
        "0" | "false" | "FALSE" => Ok(false),
        _ => Err(de::Error::unknown_variant(s, &["1", "0", "true", "false"])),
    }
}

fn deserialize_optional_fields<'de, D, T>(deserializer: D) -> Result<Option<T>, D::Error>
where
    D: Deserializer<'de>,
    T: FromStr,
    T::Err: fmt::Display,
{
    let opt = Option::<String>::deserialize(deserializer)?;
    match opt.as_deref() {
        None | Some("") => Ok(None),
        Some(s) => FromStr::from_str(s).map_err(de::Error::custom).map(Some),
    }
}

This bit may look a bit weird, but each of the deserialization functions is doing the same two-step process: deserialize the string and parse it according to its special expectations. Then, a macro is applied to each of the fields requiring deserialization.

Let’s check the parsing code now by returning the parameters of a request as a respone for the announce handler.

async fn handle_announce(announce: Query<AnnounceRequest>) -> impl IntoResponse {
    let announce: AnnounceRequest = announce.0;

    (
        StatusCode::OK,
        [(header::CONTENT_TYPE, "text/plain")],
        announce.to_string(),
    )
}

In this blurb, we’re using axum’s Query extractor to parse the request’s query parameters into an AnnounceRequest struct and returning said struct as a string.

We’ll add this new handler to our router.

#[tokio::main]
async fn main() {
    let app = Router::new()
        .route("/announce", get(handle_announce))
        .route("/", get(|| async { "Hello, World!" }));

    axum::Server::bind(&"0.0.0.0:3000".parse().unwrap())
        .serve(app.into_make_service())
        .await
        .unwrap();
}

And now we’re ready to test. First, let’s send a request with the required bits as well-formed values.

❯ curl -X GET "http://localhost:3000/announce?info_hash=aaaaaaaaaaaaaaaaaaaa&peer_id=bbbbbbbbbbbbbbbbbbbb&port=1234&uploaded=100&downloaded=200&left=50&compact=1&no_peer_id=1"
{ info_hash: aaaaaaaaaaaaaaaaaaaa, peer_id: bbbbbbbbbbbbbbbbbbbb, port: 1234, uploaded: 100, downloaded: 200, left: 50, compact: true, no_peer_id: true }

Then with a few parts as malformed values.

❯ curl -X GET "http://localhost:3000/announce?info_hash=abc&peer_id=xyz&port=1234&uploaded=100&downloaded=200&left=50&compact=1&no_peer_id=1"
Failed to deserialize query string: URL-encoded parameters should be 20 bytes in length

And finally with some optional bits included.

❯ curl -X GET "http://localhost:3000/announce?info_hash=aaaaaaaaaaaaaaaaaaaa&peer_id=bbbbbbbbbbbbbbbbbbbb&port=1234&uploaded=100&downloaded=200&left=50&compact=1&no_peer_id=1&event=started&numwant=30"
{ info_hash: aaaaaaaaaaaaaaaaaaaa, peer_id: bbbbbbbbbbbbbbbbbbbb, port: 1234, uploaded: 100, downloaded: 200, left: 50, compact: true, no_peer_id: true, event: started, numwant: 30 }

Let’s switch gears for awhile and talk about the SwarmStore.

Technically speaking, a BitTorrent tracker doesn’t store any files at all. No media, no torrents, nothing at all. However, it does typically store metadata about the swarms associated with those torrents, which exist to share media. To transmit this data, we need to be able maintain it so that we can augment it over time, and that’s where our stores come in.

SwarmStore

commit(s): 8d485b8, 524ac34

Maintaining swarms is arguably the most important job of the tracker. Without healthy swarms, clients are unable to connect to other peers and share files. So we should be careful in implementing this part of the tracker as mistakes at this stage will lead to unforeseen consequences at scale.¹

To start, swarms are comprised of peers. In order to store peers, we need a peer type. How do we decide what goes into the type? That’s right, it’s time yet again to look at the specification. After skimming through the document, it seems that for announce responses, three pieces of information are communicated about a peer: its ID, IP address, and port number.

type InfoHash = Vec<u8>
type PeerId = Vec<u8>;

#[derive(Hash, Eq, PartialEq)]
struct Peer {
    id: PeerId,
    ip: IpAddr,
    port: u16,
}

We can now use the Peer type to construct our Swarm type. The two types of peers in a swarm are seeders and leechers.

struct Swarm {
    seeders: HashSet<Peer>,
    leechers: HashSet<Peer>,
}

I’ve chosen to store the peers in hash sets rather than hash maps in order to ensure that there’s no redundant peers in either part of our swarm. In general, when using hashmaps or hashsets, you should verify the uniqueness of your keys in order to prevent collisions.² And even though the other parts of the tracker won’t be interacting with the swarms directly, we should add some helper methods to make things a bit easier on us later:

impl Swarm {
    fn new() -> Swarm {
        Swarm {
            seeders: HashSet::new(),
            leechers: HashSet::new(),
        }
    }

    fn add_seeder(&mut self, peer: Peer) {
        self.seeders.insert(peer);
    }

    fn add_leecher(&mut self, peer: Peer) {
        self.leechers.insert(peer);
    }

    fn remove_seeder(&mut self, peer: Peer) {
        self.seeders.remove(&peer);
    }

    fn remove_leecher(&mut self, peer: Peer) {
        self.leechers.remove(&peer);
    }

    fn promote_leecher(&mut self, peer: Peer) {
        match self.leechers.take(&peer) {
            Some(leecher) => self.seeders.insert(leecher),
            None => self.seeders.insert(peer),
        };
    }
}

Since peers come together to transmit data about torrents, we should find a way to map a torrent to its respective swarm. So now we need to find a unique property of each torrent to allow for us to store that information in a hashmap. If we look back at BEP 003 again, we’ll find that announce requests use a torrent’s info_hash to communicate data about the torrent. That will work, but you should verify that.³

Finally, let’s create the SwarmStore type:

#[derive(Clone)]
struct SwarmStore(Arc<RwLock<HashMap<InfoHash, Swarm>>>);

If you’re not used to Rust, then this signature probably looks pretty nasty. The innermost type is a hashmap that maps info hashes to swarms, which lets us access a particular torrent’s set of peers. The RwLock type that wraps it allows for multiple simultaneous immutable readers or at most one mutable writer, which prevents data races as threads access the store. Lastly, the outermost Arc type provides thread-safe shared ownership.

Why all the fuss over locks, threads, and the like? In production scenarios, the tracker can receive tens of thousands of requests per second. As such, we want our webserver to be able to serve more than one request at a time by using threads; otherwise using it would be a pretty miserable experience. And because we’re allowing for parallel access, we need to ensure that the data is correct at all times lest we send useless data to a client and waste bandwidth. By allowing for thread-safe mutable access, we satisfy both of these desires.

Using announce requests, the SwarmStore will be the only way to query and modify swarms, so we should focus on the functionality that it will provide.

Consider the announce request and its dual purpose:

• notify the tracker of client events and metadata
• request a list of peers for the client to use

We need to ensure the store covers both of these characteristics; otherwise, as it currently stands, the tracker has no usable functionality. First, let’s focus on the first point and revisit the Event field of the announce request. It’s optional, and can be one of three values: Started, Stopped, or Completed.

According to the specs, Stopped must be sent if the client is shutting down gracefully. This allows the client to inform the tracker that they are no longer part of that torrent’s swarm and peers will be unable to connect to them.⁴ We’ll include functionality to remove peers from a swarm.

Completed must be sent when the client finishes a download and switches from being a leeching peer to a seeding peer. However, it must not be sent if the download was fully complete at client startup. This allows the tracker to increment a metric for completed downloads (also known as “snatches”) in a clear way. So we should also include functionality to move a peer from the leechers set to the seeders set.

Finally, the Started event must be sent as part of the first request to the tracker. Thus, when a client starts or resumes a download, Started will be sent to the tracker and the client should be considered a leeching peer. But here’s a question: what happens if a user starts their client and a download is already fully complete? How do we know that they’re a seeder and not a leecher? We can’t use the Completed event as that only gets sent when the download actually completes.

If you look back at the announce request, you’ll see that there’s a left field that communicates how many bytes the client has left to download. We can use that to ensure that we add the client to the correct part of the swarm.⁵

Let’s synthesize all of that into a few methods:

impl SwarmStore {
    fn new() -> SwarmStore {
        SwarmStore(Arc::new(RwLock::new(HashMap::new())))
    }

    async fn add_peer(&mut self, info_hash: InfoHash, peer: Peer, is_download_complete: bool) {
        let mut write_locked_store = self.0.write().await;
        match write_locked_store.get_mut(&info_hash) {
            Some(swarm) => {
                if is_download_complete {
                    swarm.add_seeder(peer);
                } else {
                    swarm.add_leecher(peer);
                }
            }
            None => {
                let mut swarm = Swarm::new();
                if is_download_complete {
                    swarm.add_seeder(peer);
                } else {
                    swarm.add_leecher(peer);
                }
                write_locked_store.insert(info_hash, swarm);
            }
        }
    }

    async fn remove_peer(&mut self, info_hash: InfoHash, peer: Peer) {
        let mut write_locked_store = self.0.write().await;
        if let Some(swarm) = write_locked_store.get_mut(&info_hash) {
            swarm.remove_seeder(peer.clone());
            swarm.remove_leecher(peer);
        }
    }

    async fn promote_peer(&mut self, info_hash: InfoHash, peer: Peer) {
        let mut write_locked_store = self.0.write().await;
        if let Some(swarm) = write_locked_store.get_mut(&info_hash) {
            swarm.promote_leecher(peer)
        }
    }

	...

Now onto peer selection. An announce response contains a list of peers and the client uses that list to find or supply pieces of files. In distributed systems, it’s important to ensure that data is redundant and resistant to network partitions. Otherwise, if the desired data is concentrated in one part of the network and the link to that part disappears, there’s no way to procure the data. The way to make that happen is through a robust distribution method, and for this purpose, that’s done through peer selection.

According to the spec, we’re free to implement whatever selection mechanism we deem appropriate. For example, we could choose to only send leechers in response to a seeder’s announce or send a weighted mix of both seeders and leechers according to how much a peer has left to download. However, for this guide, we’ll just use good ol’ random selection.

	...
	
    async fn get_peers(&self, info_hash: InfoHash, numwant: usize) -> (Vec<Peer>, Vec<Peer>) {
        let (mut peers, mut peers6): (Vec<Peer>, Vec<Peer>) = (Vec::new(), Vec::new());
        let read_locked_store = self.0.read().await;
        if let Some(swarm) = read_locked_store.get(&info_hash) {
            for peer in swarm.seeders.clone().into_iter() {
                if peer.ip.is_ipv4() {
                    peers.push(peer);
                } else {
                    peers6.push(peer);
                }
            }

            for peer in swarm.leechers.clone().into_iter() {
                if peer.ip.is_ipv4() {
                    peers.push(peer);
                } else {
                    peers6.push(peer);
                }
            }

            if (swarm.seeders.len() + swarm.leechers.len()) > numwant {
                let mut rng = rand::thread_rng();
                peers.shuffle(&mut rng);
                peers6.shuffle(&mut rng);
                peers.truncate(numwant);
                peers6.truncate(numwant);
            }
        }

        (peers, peers6)
    }

Let’s finish up our SwarmStore implementation by also implementing a method for getting the seeder and leecher counts for the complete and incomplete fields of the announce response, respectively.

	...
	
    async fn get_announce_stats(&self, info_hash: InfoHash) -> (usize, usize) {
        let read_locked_store = self.0.read().await;
        if let Some(swarm) = read_locked_store.get(&info_hash) {
            return (swarm.seeders.len(), swarm.leechers.len());
        } else {
            return (0, 0);
        }
    }
}

It’s now time to test our implementation!

responses

commit(s): 524ac34, 1908345

Let’s start off by introducing the announce response type. Looking back at our specifications, we can see that our tracker should respond with a “text/plain” bencoded dictionary consisting of the following keys:

• failure reason: a human-readable error message explaining why the request failed
  • if this key is present, then no other keys may be present
• interval: the amount of time in seconds that the client should wait before sending another request
• tracker id: a string that the client should send back in its next announcement
• complete: number of seeders with the entire file
• incomplete: number of non-seeder peers, also known as leechers
• peers: information about other peers in this torrent’s swarm; it can take the form of two different models:
  • dictionary: the value is a list of dictionaries; each one with keys for the peer ID string, IP as a string, and port number
  • binary: the value is a string consisting of multiples of six (6) bytes; the first four bytes are the IP address and the last two bytes are the port number, all in big endian notation
• peers6: same as peers, but for IPv6 addresses⁶
• warning message: (optional) a human-readable message warning the end user of a potential issue
• min interval: (optional) the minimum amount of time that a client should wait before re-announcing
  ⁶ IPv6 support isn’t part of the original specifiction; it’s described a draft BEP specification. However, I’ve included it here in case someone wanted to support them.

Here is that type in Rust:

enum AnnounceResponse {
    Failure {
        failure_reason: String,
    },
    Success {
        interval: u64,
        complete: usize,
        incomplete: usize,
        peers: Vec<Peer>,
        peers6: Vec<Peer>,
        tracker_id: String,
        warning_message: Option<String>,
        min_interval: Option<u64>,
    },
}

As you can see, I’ve chosen to separate the response type into two distinct sub-types: failure and success. This isn’t entirely necessary, but it will keep things clean when we implement our bencoder. And with the exception of the interval field, we have all of the required fields covered by our SwarmStore implementation. We should now adjust the handle_announce function to do two things:

• parse announce requests and modify the swarm store, if necessary
• query the swarm store and generate an announce response

async fn handle_announce(
    announce: Query<AnnounceRequest>,
    State(mut swarm_store): State<SwarmStore>,
    ConnectInfo(addr): ConnectInfo<SocketAddr>,
) -> impl IntoResponse {
    let announce: AnnounceRequest = announce.0;

    let info_hash = announce.info_hash;

    if let Some(event) = announce.event {
        let ip = if let Some(client_ip) = announce.ip {
            client_ip
        } else {
            addr.ip()
        };

        let peer = Peer {
            id: announce.peer_id,
            ip,
            port: announce.port,
        };

        let is_download_complete = announce.left == 0;

        match event {
            Event::Started => {
                swarm_store
                    .add_peer(info_hash.clone(), peer, is_download_complete)
                    .await;
            }
            Event::Stopped => {
                swarm_store.remove_peer(info_hash.clone(), peer).await;
            }
            Event::Completed => {
                swarm_store.promote_peer(info_hash.clone(), peer).await;
            }
        }
    }

    let numwant = if let Some(n) = announce.numwant {
        n
    } else {
        30 // 30 peers is generally a good amount
    };

    let (peers, peers6) = swarm_store.get_peers(info_hash.clone(), numwant).await;
    let (complete, incomplete) = swarm_store.get_announce_stats(info_hash).await;

    let response = AnnounceResponse::Success {
        interval: 1800, // 30-minute announce interval
        complete,
        incomplete,
        peers,
        peers6,
        tracker_id: String::from("test"),
        min_interval: None,
        warning_message: None,
    };

    (
        StatusCode::OK,
        [(header::CONTENT_TYPE, "text/plain")],
        response.to_string(),
    )
}

There’s several adjustments here:

• Two new parameters have been added. The first one allows the handler to safely access the swarm store. The second parameter is an object from our webserver choice (axum) that allows us to get the IP of the client; this enables us to be able to create the peer if the announce request doesn’t include it.
• The announce request is checked for an event. If present, we extract the requisite info and modify the swarm.
• A swarm’s seeder and leecher values along with randomized peer lists are used to make the announce response.

Finally, let’s adjust the main() function to supply the necessary bits to the handler:

#[tokio::main]
async fn main() {
    let swarm_store: SwarmStore = SwarmStore::new();

    let app = Router::new()
        .route("/announce", get(handle_announce))
        .route("/", get(|| async { "Hello, World!" }))
        .with_state(swarm_store);

    axum::Server::bind(&"0.0.0.0:3000".parse().unwrap())
        .serve(app.into_make_service_with_connect_info::<SocketAddr>())
        .await
        .unwrap();
}

Here’s the moment of truth. Run the server again with cargo run and start sending requests to the tracker. Start with the Started event; there should be one leecher.

❯ curl -X GET "http://localhost:3000/announce?info_hash=aaaaaaaaaaaaaaaaaaaa&peer_id=bbbbbbbbbbbbbbbbbbbb&port=1234&uploaded=100&downloaded=200&left=50&compact=1&no_peer_id=1&event=started&numwant=30"
interval: 1800, complete: 0, incomplete: 1, tracker_id: test, peers: [ { peer_id: bbbbbbbbbbbbbbbbbbbb, ip: 127.0.0.1, port: 1234 } ], peers6: []

Then, the Stopped event; we should see that there are no peers.

❯ curl -X GET "http://localhost:3000/announce?info_hash=aaaaaaaaaaaaaaaaaaaa&peer_id=bbbbbbbbbbbbbbbbbbbb&port=1234&uploaded=100&downloaded=200&left=50&compact=1&no_peer_id=1&event=stopped&numwant=30"
interval: 1800, complete: 0, incomplete: 0, tracker_id: test, peers: [], peers6: []

Next, let’s test Started again, but with left=0 as a parameter this time; we should see one seeder.

❯ curl -X GET "http://localhost:3000/announce?info_hash=aaaaaaaaaaaaaaaaaaaa&peer_id=bbbbbbbbbbbbbbbbbbbb&port=1234&uploaded=100&downloaded=200&left=0&compact=1&no_peer_id=1&event=started&numwant=30"
interval: 1800, complete: 1, incomplete: 0, tracker_id: test, peers: [ { peer_id: bbbbbbbbbbbbbbbbbbbb, ip: 127.0.0.1, port: 1234 } ], peers6: []

Finally, the Completed event. Either repeat the Stopped event or restart the server to remove the peer from the swarm, and then send the request; there should be one seeder.

❯ curl -X GET "http://localhost:3000/announce?info_hash=aaaaaaaaaaaaaaaaaaaa&peer_id=bbbbbbbbbbbbbbbbbbbb&port=1234&uploaded=100&downloaded=200&left=50&compact=1&no_peer_id=1&event=completed&numwant=30"
interval: 1800, complete: 1, incomplete: 0, tracker_id: test, peers: [ { peer_id: bbbbbbbbbbbbbbbbbbbb, ip: 127.0.0.1, port: 1234 } ], peers6: []

bencode

commit: 5c12f38

Currently, the announce route returns the response in a human-readable format. That’s nice and all, but users aren’t going to be using these responses to manually connect to peers; that’s what they have clients for. And note that this response has just one peer; with the peer list sizes that clients typically see at the rate that they usually announce, there’s potential for a lot of wasted bandwidth. If a human isn’t going to be reading the responses, then we can probably do away with the whitespace and use something a bit more compact, right?

That’s where bencode comes in. It allows the transmission of loosely structured data in a fairly terse format. For example, the dictionary { "cow": "moo", "duck": "quack" } would be bencoded as d3:cow3:moo4:duck5:quacke. The letters d and e indicate the start and end of a dictionary, respectively; each key and value is preceded by its length and placed in between the ends of the string. I won’t go over the whole format in this post, but you can read more about it here. In addition to its terseness, it also works great as a cross-platform format as there is a one-to-one mapping between each possible complex value and its encoding. With bencode, there’s only one correct way to interpret encoded values.

As much as I waste my own time, I draw the line at writing a bencode library, especially when there are several great ones already available. For Rust, I recommend bendy as it makes it very easy to implement a bencoding method for custom types. If you’re using a different language, try to find a library that makes adding the functionality as easy as possible.

With bendy, all you have to do to get a to_bencode() method is implement the ToBencode trait for that type and any other types that will have to be encoded as part of said type. We want to bencode the AnnounceResponse type, so we should ensure that the types for each value are able to be encoded. If we look at the documentation, it appears that bendy supports encoding of a lot of built-in types, so the only other type that needs ToBencode implemented is Peer.

impl ToBencode for Peer {
    const MAX_DEPTH: usize = 1;

    fn encode(&self, encoder: SingleItemEncoder) -> Result<(), Error> {
        encoder.emit_dict(|mut e| {
            e.emit_pair(b"ip", self.ip.to_string())?;
            e.emit_pair(b"peer_id", self.id.clone())?;
            e.emit_pair(b"port", self.port)?;
            Ok(())
        })?;

        Ok(())
    }
}

And now that we’ve done that, bendy will automatically use that implementation to encode the peer list fields! Here’s the implementation of ToBencode for AnnounceResponse:

impl ToBencode for AnnounceResponse {
    const MAX_DEPTH: usize = 5;
    fn encode(&self, encoder: SingleItemEncoder) -> Result<(), Error> {
        match self {
            Self::Failure { failure_reason } => {
                encoder.emit_dict(|mut e| {
                    e.emit_pair(b"failure_reason", failure_reason)?;
                    Ok(())
                })?;
            }
            Self::Success {
                interval,
                complete,
                incomplete,
                tracker_id,
                peers,
                peers6,
                warning_message,
                min_interval,
            } => {
                encoder.emit_dict(|mut e| {
                    e.emit_pair(b"complete", complete)?;
                    e.emit_pair(b"incomplete", incomplete)?;
                    e.emit_pair(b"interval", interval)?;

                    if let Some(min_interval) = min_interval {
                        e.emit_pair(b"min_interval", min_interval)?;
                    }

                    e.emit_pair(b"peers", peers)?;
                    e.emit_pair(b"peers6", peers6)?;

                    e.emit_pair(b"tracker_id", tracker_id)?;


                    if let Some(warning_message) = warning_message {
                        e.emit_pair(b"warning_message", warning_message)?;
                    }

                    Ok(())
                })?;
            }
        }

        Ok(())
    }
}

To use this new bencoding functionality in our tracker, all we need to do is to go to handle_announce and adjust our returned response:

    (
        StatusCode::OK,
        [(header::CONTENT_TYPE, "text/plain")],
        response.to_bencode().unwrap(),
    )

Let’s test it out.

❯ curl -X GET "http://localhost:3000/announce?info_hash=aaaaaaaaaaaaaaaaaaaa&peer_id=bbbbbbbbbbbbbbbbbbbb&port=1234&uploaded=100&downloaded=200&left=0&compact=1&no_peer_id=1&event=started&numwant=30"
d8:completei1e10:incompletei0e8:intervali1800e5:peersld2:ip9:127.0.0.17:peer_idli98ei98ei98ei98ei98ei98ei98ei98ei98ei98ei98ei98ei98ei98ei98ei98ei98ei98ei98ei98ee4:porti1234eee6:peers6le10:tracker_id4:teste

Neat!

Whew. If you’re still here up to this point, thanks for sticking around. And also, congratulations! We’ve reached the minimally viable point for a tracker; you’ve made a basic BitTorrent TCP tracker that responds to client announces. That was a lot of information to consume, so feel free to stop reading here. Or if you’re feeling up to it, continue reading to learn how to implement scrapes and find suggestions for how you can improve upon this example.

scrapes

Trackers also support another type of request called a scrape in which a client queries for information about the state of a torrent and its swarm but not for a list of its peers. This is typically done to determine whether to send an announce to request more peers or to update a torrent’s metadata in the user interface; thus, you can think of it as a client doing a sort of “pulse check” upon the tracker. If supplied with a torrent’s info hash, the scrape operation will respond with the state of only that torrent; however, if no info hash is given, it allows for a user to query the state of all torrents on a tracker.

It’s worth noting that some trackers do not support scraping. This decision is usually made for a number of reasons. First, there are some trackers where access is restricted to members only (also known as private trackers); these trackers tend to take privacy and security very seriously, and are hesitant to allow for bulk data collection in any way. Secondly, for complete scrapes, a tracker has to iterate through every torrent that it has which can become a time-consuming operation; if enough users were to do this simultaenously, it could very well bring the tracker to a halt.

requests

commit: 7bb0245

Scrape requests are GET requests sent to the /scrape endpoint of a tracker and can be supplemented by one or more optional info_hash query parameters. In Rust, the type would look something like this:

struct ScrapeRequest {
    info_hashes: Option<Vec<InfoHash>>,
}

We’ll need to create another handler to support requests on this endpoint. It should check for any query parameters and create the corresponding ScrapeRequest. Remember that info hashes are URL-encoded so they’ll need to be decoded prior to any downstream use. Here’s an example implementation:

async fn handle_scrape(scrape: Query<Vec<(String, String)>>) {
    let info_hashes: Option<Vec<InfoHash>> = if scrape.0.is_empty() {
        None
    } else {
        let raw_info_hashes: Vec<&(String, String)> = scrape
            .0
            .iter()
            .filter(|(key, _)| key.to_lowercase() == "info_hash")
            .collect();
        if raw_info_hashes.is_empty() {
            None
        } else {
            let decoded_info_hashes = raw_info_hashes
                .into_iter()
                .map(|(_, raw_val)| urlencoding::decode_binary(raw_val.as_bytes()).into_owned())
                .filter(|buf| buf.len() == 20)
                .collect();
            Some(decoded_info_hashes)
        }
    };

    let scrape = ScrapeRequest { info_hashes };
}

Note that any query parameter whose key doesn’t correspond to the expected “info_hash” string is filtered out before the URL-decoding step. Additionally, info hashes that don’t match the correct length are filtered out; you would typically return an error here but this is just an example.

Also, let’s add this new handler to the router:

#[tokio::main]
async fn main() {
    let swarm_store: SwarmStore = SwarmStore::new();

    let app = Router::new()
        .route("/announce", get(handle_announce))
        .route("/scrape", get(handle_scrape))
        .route("/", get(|| async { "Hello, World!" }))
        .with_state(swarm_store);

    axum::Server::bind(&"0.0.0.0:3000".parse().unwrap())
        .serve(app.into_make_service_with_connect_info::<SocketAddr>())
        .await
        .unwrap();
}

If we add some logging to print out requests, we’ll see the following:

❯ curl -X GET "http://localhost:3000/scrape?info_hash=aaaaaaaaaaaaaaaaaaaa"
ScrapeRequest { info_hashes: Some([[97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97]]) }

❯ curl -X GET "http://localhost:3000/scrape?info_hash=aaaaaaaaaaaaaaaaaaaa&info_hash=bbbbbbbbbbbbbbbbbbbb"
ScrapeRequest { info_hashes: Some([[97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97], [98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98]]) }

❯ curl -X GET "http://localhost:3000/scrape"
ScrapeRequest { info_hashes: None }

Looks like it works! Now, if we’re going to query the state of some torrents, we’ll need to find a place to store that information. I wonder where we could put it…

TorrentStore

commit: fc07bc4

A TorrentStore should store information about torrents. Upon revisiting the specifications and looking at the scrape section, we can see that we’ll need to report one thing about a torrent: how many times it’s been fully downloaded.

Now that we’ve found a unique key for our torrent store, let’s implement the types. Firstly, the Torrent:

struct Torrent {
    info_hash: InfoHash,
    downloaded: u64,
}

struct TorrentStore(Arc<RwLock<HashMap<InfoHash, Torrent>>>);

Once again, let’s find ways to make things easier for us. We know that we’ll need to return download count for scrapes, so we should add a method for that. Additionally, we should add helper functions for times where the store data needs to be updated; those would be when when a new torrent needs to be added or when a download completes. Let’s add those now.

impl TorrentStore {
    async fn add_torrent(&mut self, info_hash: InfoHash) {
        let mut write_locked_store = self.0.write().await;
        write_locked_store.insert(
            info_hash.clone(),
            Torrent {
                info_hash,
                downloaded: 0,
            },
        );
    }

    async fn increment_downloaded(&mut self, info_hash: InfoHash) {
        let mut write_locked_store = self.0.write().await;
        if let Some(torrent) = write_locked_store.get_mut(&info_hash) {
            torrent.downloaded += 1;
        }
    }

    async fn get_stats_for_scrapes(&self, info_hashes: Vec<InfoHash>) -> Vec<u64> {
        let read_locked_store = self.0.read().await;
        let stats = info_hashes.iter().map(|info_hash| if let Some(torrent) = read_locked_store.get(info_hash) {
            return torrent.downloaded;
        } else {
            return 0;
        }).collect();
        stats
    }

    async fn get_global_scrape_stats(&self) -> HashMap<InfoHash, usize> {
        let read_locked_store = self.0.read().await;
        let mut global_stats = HashMap::new();

        for (info_hash, torrent) in read_locked_store.iter() {
            global_stats.insert(info_hash.clone(), torrent.downloaded);
        }

        global_stats
    }
}

And with that, TorrentStore is finished.

A quick side note: since scrapes also report seeder and leecher counts, we should also add the scraping methods to SwarmStore and adjust them accordingly.

	...
	
    async fn get_stats_for_scrapes(&self, info_hashes: Vec<InfoHash>) -> (Vec<usize>, Vec<usize>) {
        let read_locked_store = self.0.read().await;
        let (complete_vec, incomplete_vec) = info_hashes
            .iter()
            .map(|info_hash| {
                if let Some(swarm) = read_locked_store.get(info_hash) {
                    return (swarm.seeders.len(), swarm.leechers.len());
                } else {
                    return (0, 0);
                }
            })
            .unzip();
        (complete_vec, incomplete_vec)
    }

    async fn get_global_scrape_stats(&self) -> HashMap<InfoHash, (usize, usize)> {
        let read_locked_store = self.0.read().await;
        let mut global_stats = HashMap::new();

        for (info_hash, swarm) in read_locked_store.iter() {
            global_stats.insert(
                info_hash.clone(),
                (swarm.seeders.len(), swarm.leechers.len()),
            );
        }

        global_stats
    }

responses

commit(s): 8116fd4, 930ed09

To finish up the scrape implementation, we need to create a type for the response. In the specs, the scrape response is described as having the following keys:

• files: a dictionary containing one key-value pair for each supplied info hash (or all, if not supplied); each key is an info hash and the value is another dictionary containing the following:
  • complete: number of seeders
  • downloaded: total number of times that the tracker has registered a completion
  • incomplete: number of leechers

Let’s create the corresponding types. Here’s the implementation that I went with:

#[derive(Clone)]
struct ScrapeData {
    complete: usize,
    downloaded: usize,
    incomplete: usize,
}

#[derive(Clone)]
struct Scrape {
    info_hash: InfoHash,
    data: ScrapeData,
}

#[derive(Clone)]
struct ScrapeResponse {
    files: Vec<Scrape>,
}

I’ve chosen to separate the inner dictionary into a separate ScrapeData type and use that to compose the Scrape type. This isn’t strictly necessary, but it does make it easier to implement bencoding. The ScrapeResponse type is then composed with the Scrape type.

Speaking of bencoding, we’ll need to implement the ToBencode trait for each type:

impl ToBencode for ScrapeData {
    const MAX_DEPTH: usize = 2;

    fn encode(&self, encoder: SingleItemEncoder) -> Result<(), Error> {
        encoder.emit_dict(|mut e| {
            e.emit_pair(b"complete", self.complete)?;
            e.emit_pair(b"downloaded", self.downloaded)?;
            e.emit_pair(b"incomplete", self.incomplete)?;

            Ok(())
        })?;
        Ok(())
    }
}

impl ToBencode for Scrape {
    const MAX_DEPTH: usize = 2;

    fn encode(&self, encoder: SingleItemEncoder) -> Result<(), Error> {
        encoder.emit_dict(|mut e| {
            e.emit_pair(&self.info_hash, self.data.clone())?;

            Ok(())
        })?;
        Ok(())
    }
}

impl ToBencode for ScrapeResponse {
    const MAX_DEPTH: usize = 5;

    fn encode(&self, encoder: SingleItemEncoder) -> Result<(), Error> {
        encoder.emit_dict(|mut e| {
            e.emit_pair(b"files", self.files.clone())?;

            Ok(())
        })?;

        Ok(())
    }
}

Since each type implements ToBencode, bendy can use each implementation when it needs to encode the sub-types.

Now we need to adjust our main() function and handlers to take advantage of our new torrent store and scrape response types. For the main() function, we should make both the swarm and torrent stores available to the handlers. To that end, we can wrap them in a struct and include that in the router. Let’s also add some dummy torrents into the store before we start the server.

#[tokio::main]
async fn main() {
    let mut state = AppState {
        swarm_store: SwarmStore::new(),
        torrent_store: TorrentStore::new(),
    };

    state
        .torrent_store
        .add_torrent("aaaaaaaaaaaaaaaaaaaa".as_bytes().to_vec())
        .await;
    state
        .torrent_store
        .add_torrent("bbbbbbbbbbbbbbbbbbbb".as_bytes().to_vec())
        .await;

    let app = Router::new()
        .route("/announce", get(handle_announce))
        .route("/scrape", get(handle_scrape))
        .route("/", get(|| async { "Hello, World!" }))
        .with_state(state);

    axum::Server::bind(&"0.0.0.0:3000".parse().unwrap())
        .serve(app.into_make_service_with_connect_info::<SocketAddr>())
        .await
        .unwrap();
}

Now let’s make a couple adjustments to handle_announce. First, we’ll need to change the parameters to use the new AppState type.

async fn handle_announce(
    announce: Query<AnnounceRequest>,
    State(mut state): State<AppState>,
    ConnectInfo(addr): ConnectInfo<SocketAddr>,
) -> impl IntoResponse {
	...

Note that all references to swarm_store in the handler will need to be changed to state.swarm_store.

And the part of the handler that handles the Completed event should be adjusted to increment the downloaded metric of the corresponding torrent.

	...
			
            Event::Completed => {
                state
                    .swarm_store
                    .promote_peer(info_hash.clone(), peer)
                    .await;
                state
                    .torrent_store
                    .increment_downloaded(info_hash.clone())
                    .await;
            }
    ...

Finally, we need to add functionality to handle_scrape that generates a ScrapeResponse. Here’s how I chose to implement it.

async fn handle_scrape(
    scrape: Query<Vec<(String, String)>>,
    State(state): State<AppState>,
) -> impl IntoResponse {
    let info_hashes: Option<Vec<InfoHash>> = if scrape.0.is_empty() {
        None
    } else {
        let raw_info_hashes: Vec<&(String, String)> = scrape
            .0
            .iter()
            .filter(|(key, _)| key.to_lowercase() == "info_hash")
            .collect();
        if raw_info_hashes.is_empty() {
            None
        } else {
            let decoded_info_hashes = raw_info_hashes
                .into_iter()
                .map(|(_, raw_val)| urlencoding::decode_binary(raw_val.as_bytes()).into_owned())
                .filter(|buf| buf.len() == 20)
                .collect();
            Some(decoded_info_hashes)
        }
    };

    let scrape = ScrapeRequest { info_hashes };

    let scrapes = match scrape.info_hashes {
        Some(info_hashes) => {
            let (seeder_stats, leecher_stats) = state
                .swarm_store
                .get_stats_for_scrapes(info_hashes.clone())
                .await;
            let downloaded_stats = state
                .torrent_store
                .get_stats_for_scrapes(info_hashes.clone())
                .await;
            downloaded_stats
                .iter()
                .enumerate()
                .map(|(idx, downloaded)| {
                    let data = ScrapeData {
                        complete: seeder_stats[idx],
                        incomplete: leecher_stats[idx],
                        downloaded: *downloaded,
                    };
                    Scrape {
                        info_hash: info_hashes[idx].clone(),
                        data,
                    }
                })
                .collect::<Vec<Scrape>>()
        }
        None => {
            let seeder_leecher_map = state.swarm_store.get_global_scrape_stats().await;
            let downloaded_map = state.torrent_store.get_global_scrape_stats().await;

            downloaded_map
                .into_iter()
                .map(|(info_hash, downloaded)| {
                    let (complete, incomplete) = match seeder_leecher_map.get(&info_hash) {
                        Some((c, i)) => (*c, *i),
                        None => (0, 0),
                    };
                    let data = ScrapeData {
                        complete,
                        incomplete,
                        downloaded,
                    };
                    Scrape { info_hash, data }
                })
                .collect::<Vec<Scrape>>()
        }
    };

    let response = ScrapeResponse { files: scrapes };

    (
        StatusCode::OK,
        [(header::CONTENT_TYPE, "text/plain")],
        response.to_bencode().unwrap(),
    )
}

Admittedly, it is a little dense. Generally, it checks whether there were any info hashes included in the scrape request. If so, it requests the stats for each info hash and associates them together in a scrape, and that’s all collected into a list of scrapes. If there were no info hashes supplied, then a map for each set of stats is used to associate the data together into a list of scrapes. The response is then bencoded and returned to the user.

It’s time to test our scrape implementation. Let’s scrape for information on the first dummy torrent that we added:

❯ curl -X GET "http://localhost:3000/scrape?info_hash=aaaaaaaaaaaaaaaaaaaa"
d5:filesld20:aaaaaaaaaaaaaaaaaaaad8:completei0e10:downloadedi0e10:incompletei0eeeee

Looking at the bencode string, we can see that all of the stats are zero as expected. Let’s send an announce with a Completed event and scape the same torrent again:

❯ curl -X GET "http://localhost:3000/scrape?info_hash=aaaaaaaaaaaaaaaaaaaa"
d5:filesld20:aaaaaaaaaaaaaaaaaaaad8:completei1e10:downloadedi1e10:incompletei0eeeee

As expected, the complete and downloaded metrics have increased by one. Finally, let’s do a global scrape, which should include information on all torrents in the torrent store.

❯ curl -X GET "http://localhost:3000/scrape"
d5:filesld20:aaaaaaaaaaaaaaaaaaaad8:completei1e10:downloadedi1e10:incompletei0eeed20:bbbbbbbbbbbbbbbbbbbbd8:completei0e10:downloadedi0e10:incompletei0eeeee

conclusion

And that’s it! We’ve created a basic functional BitTorrent tracker together that adheres to the specifications that a client should expect. You can use it to do anything you like, such as distributing home movies between family members or sharing a large collection of public domain works; the possibilities are endless!

That being said, this implementation truly is a basic tracker. Firstly, it works over the TCP protocol rather than UDP, which requires established connections between the client and server. Many performant trackers choose to go a bit lower-level and decode UDP for more efficiency. Secondly, this is an unsecured tracker, meaning that there is no encryption of the data sent between the client and tracker. In the modern age of the internet, one should generally try to interact with secure services.

Lastly, in addition to addressing the caveats above, there are a number of ways that this example implementation could be improved, if you’re so inclined:

• add unit and integration tests in order to ensure that expected functionality continues to work as you add new features
• add a way to programmatically add torrent records from a database so that the tracker is aware of existing torrents
• add metrics so that you can see how well the tracker is performing and if there are any bottlenecks in the service
• ensure swarm health by monitoring peer activity and removing stale peers from swarms
• implement proper error handling so that the tracker doesn’t panic and shut down unexpectedly

If you have any questions or comments, feel free to send me an email; you can find the address on my about page. Cheers!