For a while now, clients on the torrent protocol can find other clients that happen to have the same torrents as them through the Distributed Hash Table. The simplest explanation i can manage is that if you know a torrent’s hash (kind of an ID number specific to that torrent), you can derive through math and a bit of networking magic (and some public servers) where to find other people who also have this torrent’s hash and ideally the files you want. i’m skipping over a lot but this system means that if all trackers on a torrent go down, you can still hopefully do this process and find other people who have done this process without the aid of a tracker which is a point of centralization.

This is all pretty bonkers but it gets better: consider the previous, and now ask the question:“who’s to say i have to know a torrent’s hash before finding people who have it? what is stopping me from simply doing the derivation math on random numbers and finding out if people do have any torrent with that hash?”. that’s the basis behind DHT crawlers. i have a server at home that runs Magnetico, which generates random hashes, finds out if there is a torrent with that hash that people are seeding, and if so gets the file list and title and saves it to a database that i can query with a web interface.

Of course, most of the time there isn’t a torrent with that hash (there is math you can do to have better odds but ultimately i’m carried by my fibre connection), but if you spam hard enough you do find some torrents, most of which are porn, and if you find enough you start finding things you care about. i have been crawling for 6 months (only during night time. the networking specifics mean that my home router gets really bogged down by the amount of connections i make, and using the network normally becomes really slow) and to date i have found about a million and a half torrents total. far from being a complete collection but a great way of finding both popular and niche things.