David Vorick expressed:  We are excited today to announce official support for 100 GB files on Skynet. This was a surprisingly difficult thing to support, and is in fact NOT supported on the vast majority of centralized filesharing services.

Most of the web works using a protocol called “TCP”. Beginner network engineers are told “UDP is unreliable, and TCP is reliable”, which is only somewhat true. A more true true statement would be “UDP is highly unreliable, and TCP is mildly unreliable”.

When you connect to a server over the internet, you are bouncing through often dozens of different routers, all which are handling tons of connections at once. If at any time (extremely common) a router has more messages incoming than it can handle, it’ll just delete some.

This is called “dropping packets”, and is the main reason that UDP is unreliable. UDP is a naive protocol that just sends packets to the destination, and doesn’t worry if stuff gets dropped or reordered along the way.

TCP is considered “reliable” because it does care. The server and the sender work together to ensure that all the packets made it, and if some packets get dropped TCP will resend them.

But TCP can only handle so much chaos. Occasionally, The number of packets that get dropped (among other sources of network turbulence) gets so high that TCP also fails.

On the open Internet today, those failures usually happen somewhere between 1 GB and 4 GB of data transferred. And that’s why you can’t just “upload a bigger file” on most centralized services. TCP isn’t stable enough to make it work.

It helps to think about the Internet as a stormy sea. In fact, you will often see network engineers using weather metaphors for the Internet. Packet floods, request storms, etc. Packets are tiny ships crossing a vast and chaotic ocean.

So we got more reliability by adding another layer on top of TCP called “TUS”

TUS creates a longer lasting connection between the client and the server that can detect when a TCP connection has failed and retry. TUS can also handle short bursts of things like losing Internet entirely, or switching from wifi to cellular data, etc.

With TUS, we were able to get stable, reliable uploads to 100 GB and more… kinda. Out of the box TUS is a protocol that talks to one server. And so you can get a super reliable connection to that server, but for some of our users on slower connections, a 100 GB upload can take several days. And for maintenance purposes, we restart our servers about once a week. If you are mid-upload when a server restarts, your upload will fail because TUS itself will get interrupted.

And after lot of explanations Vorick states, After all of this, we’ve gotten to a place where Skynet, the decentralized service, is faster and more reliable for large files (>10 GB) than almost every other centralized service. It’s a hard problem, and one that we internally insisted on doing correctly.