Today we heard about an interesting development in P2P file sharing technology from Harvard University researchers and a bunch of geeks in Amsterdam from a company called Tribler. Potentially these guys can come up with what I’d consider the next generation of file sharing technology. Previously we’ve had programs such as eMule and Kazaa which were able to download from multiple sources at once. Then bit torrent came along and showed that it was just as easy to download your file from others even if they haven’t received the complete file. Theoretically, bit-torrent uses close to the most efficient method of transfer possible for getting a file from one box to x number of other boxes.
But bit-torrent is not ideal. The problem lies in the fact that 90% of us have asymmetric connections as for some reason most ISPs think they can get away with giving us 1:16 upload/download ratios. If this is the average ratio within a swarm it means that for a client to receive a file at their maximum possible download speed they need at least 16 people to download 16 different parts of the file from. Or, in other words, for each file you download, you need to spend 16 times the time it took to download uploading it back into the swarm.
The problem with bit-torrent is that it doesn’t reach anywhere near its maximum potential because people are not willing to spend that much time with their PCs running if they can get away with just leeching a file. Private torrent trackers are a good way of solving this problem because everyone has a ratio that they have to maintain. The tracker enforces that you upload at least 70% of what you download. This means you can get some awesome speeds with some popular files. The problem is that the system is centralised and there are only so many users and torrents that it can keep track of and so rare or unpopular files do not get the same benefits.
Tribler appear to be making a lot of effort to defeat the problems of public and private torrents. They are trying to decentralise the user score system. A user will be given points by other users depending on how much they upload. The problem they need to solve however is how many points people give for very popular files and for very rare ones. More points for popular files will mean more and more of the very few people with gigabit connections will be able to max out their download speed. More points for rare files will mean faster transfers for rare files but slower transfers for those with high-speed connections.
The goal Tribler have at the moment is to develop a fast and efficient file transfer process but geared specifically towards streaming video. The reason for this of course is that they can then sell the technology for millions to TV companies or video distribution services such as YouTube. But TV and video on demand is quite a specific use case. Files must be transfered head first rather than in random chunks; most files are unlikely to remain popular after a few weeks and most of the time people will be wanting the same relatively small set of video files. Tribler will have to tailor their system to be most efficient for this particular kind of usage.
Ideally we’d have a completely universal file transfer system which would work equally well with streaming video as with distributing rare random files. Perhaps it might even be possible for each user to tailor how they want their bandwidth to be used based on what services they desire. That way people with similar interests would automatically form their own mini swarm of files. In any case I can see a very interesting future in the field of file sharing. We’ve only just begun to scratch the surface of what can be done with our increasingly speedy broadband connections.