I recently downloaded two sizable content files, each 1.1GB. My FIOS is 20/5.
The first file took X minutes to download. The second download link (from the
same company) to me to a “content downloader” and the file of the same size
took less than half the time. I mean, it was quite noticeably faster. I wonder
how their “downloader” program can cut the download time of a huge file in half
because my internet connection is still 20/5.
Download accelerators certainly don’t make your internet connection any
faster, but they do sometimes seem like they do.
A normal file download is essentially a file copy operation. A file on some
remote server is copied to your computer.
A download accelerator does that too, but it can make use of a few tricks to
simply be more efficient at it; sometimes, much more efficient.
Become a Patron of Ask Leo! and go ad-free!
Download Accelerators
There are many download accelerators and I’m not going to speak to any one
specific instance.
Rather, I’m going to discuss the techniques that they may or may not use to
improve your download speeds.
Some accelerators may use some, others may use all of the techniques here.
Heck, there are probably some accelerators that are using techniques that haven’t
even thought of.
But they all do share one thing in common.
They do not make your internet connection any faster.
Parallelism
Parallelism is just a fancy word meaning doing two (or more) things at once.
To understand why that might be preferable to doing only one thing at a time, we need to understand how – at a very grossly over-simplified level – a normal download works.
Normally, when you download a file, it’s sent in chunks, one chunk at a time. You can think of it something like this:
| Your Computer |
Download Server |
|
| I’d like “wondefullprogram.exe” please |
||
| Here’s chunk #1 | ||
| … delay … | ||
| OK, got it! | ||
| Here’s chunk #2 | ||
| … delay … | ||
| OK, got it! | ||
| Here’s chunk #3 | ||
| … delay … | ||
| OK, got it! |
You get the idea. Each chunk is sent in turn; after each chunk, your computer spends a little time making sure that it got it correctly. Couple that with other technical overhead in the communications protocol and files rarely travel at “full speed” in a normal download.
The most common way that download accelerators overcome this is to do two
(or more) things in parallel.
| Connection #1 | Connection #2 | ||
| Your Computer |
Download Server |
Your Computer |
Download Server |
| I’d like “wondefullprogram.exe” please |
|||
| Here’s chunk #1 | send chunk #2 please | ||
| Got chunk #1, send #3 | Here’s chunk #2 | ||
| Here’s chunk #3 | Got #2, send #4 | ||
| Got #3, send #5 | Here’s chunk #4 | ||
| Here’s chunk #5 | Got #4, send #6 | ||
| Got #5, send #7 | Here’s chunk #6 | ||
While the two downloads compete with each other for the full speed of your internet connection, the delays that we saw earlier are effectively eliminated; if one of the two download connections delays for any reason, that simply means that the other one will continue at a faster pace.
In effect, this is also what your web browser does when it renders a web page. Separate connections are created for each of the elements on the page – one for the page itself, and then as they are encountered, one for each item, such as an image, a style sheet, a JavaScript file – and so on to some pre-configured maximum number of connections.
Normally, in a traditional file, download only one connection is used. Download accelerators most commonly speed up downloads by creating more than one. Interestingly, the impact of this technique appears to be greatest the faster your internet connection, where the delays in a normal download become a proportionally larger percentage of the time spent downloading.
Compression
If a file being download is not itself compressed – say it’s a text file or word document, or even an executable “.exe” file – the act of downloading it does not necessarily compress it.
Some download accelerators cause the file to be compressed by the server before sending it to your computer and then automatically decompress it when it arrives. Because compression makes the file smaller in most cases, that means that fewer bytes are being downloaded across your internet connection. The net effect is that the download happens faster.
This doesn’t work for all file types; files that are already compressed, like .jpg pictures or .zip archives don’t benefit from this last-minute compression. Because “compressing” an already-compressed file can actually make it bigger, a good download accelerator will pay attention to the type of file being downloaded and only compress it if there would be a net savings.
Resumability
This isn’t so much a true speed up in the downloading technology, but it represents a huge speed up if you’ve ever been disconnected in the middle of a large download.
Normally, if your computer somehow loses its connection to the download server in the middle of a large download, you have no choice but to start over from the beginning. Imagine being 95% of the way through that 1.1GB download, only to have it fail. Now that 1.1GB download is going to end up taking almost twice as long because you had to download it 1.95 times – once to get it to work, plus that 0.95 that failed (assuming the second attempt is successful).
Many download accelerators not only magically remember where you were when a download is disconnected for some reason, but they’ll often automatically re-connect. You might never even know that anything was amiss. Your download simply happens.
What’s the Catch?
So if this acceleration is so great, why don’t all downloads just do it normally?
Two reasons combine:
- Most of the techniques used to speed up downloads, such as those which I describe above, require compatible support from the server that hosts the download.
- There’s no standard.†
What that means is that you can’t just install an accelerator and have it work everywhere – it requires support from the sites from which you are downloading.
It also means that the most common tool to perform downloads – your web browser – can’t just add this feature.
† One can argue that the BitTorrent protocol is a standard. Technically, it’s not a download in the “one server to your machine” sense, but rather, it’s a technique that collects the various “chunks” of a file from other computers – all peers – hosting a copy of that same file. But the protocol definitely makes heavy use of parallel connections to maximize throughput.
This must be how Microsoft updates works. It seems to pause the downloads when you go off line and continue when you come back on line. I would think this “pause” method would save a whole lot of traffic across the Internet. So much that some administrative group would be looking into it.
To understand the question, it might help to know what “My FIOS is 20/5” means. Huh?
10-Feb-2012
I’ve noticed that when I buy an MP3 disk from Amazon or ClassicsOnline they both ask me first to download and use their own download program which in turn downloads a whole CD very fast indeed – much faster than other downloads. Clearly they’re using an accelerator that’s compatible with their end of things.
Hi Leo! Thank you for your article.
A couple of questions.
There is something quite peculiar going on when downloading any YouTube file using one of the download add-ons (Download Helper) to my Firefox.
Throughout the download, the Firefox’s download window displays the progress: a sliding bar indicator, time remaining, how many MBs downloaded out of how many – and the download speed at the moment, in KB/sec or MB/sec. There is also a button to pause, and a button to cancel.
In the beginning of the file download, the download speed may quickly reach certain value (say … 378 KB/sec … though in each case it is quite different). Then the speed starts dropping … and dropping… Until it reaches some ridiculously low number (let’s say … 67 KB/sec). If (if!) I do not disturb anything, from this point on, the speed dances around that low number till the end… that may come in a few hours… ;-)
(Mind you, my broadband connection has nothing to do with it; the size of my download pipe is always in excess of 20 MB/sec.)
However, through accident and experimentation, I discovered that I could immensely speed-up the above download “manually”, as follows:
“Technique 1” (some “technique”…): Observe the download speed, and as soon as the speed numbers start dropping off, pause the download, and then resume it after a second or so.
“Technique 2” (even faster): Each time the progress sliding bar stops moving, pause the download, and then resume it after a second or so.
Quite primitive, I admit, but it _works_. In fact, a download that would take two hours undisturbed, I can thus cut down to below ten minutes. The drawback, of course, is that it is “labour intensive” (… just imagine clicking the mouse like mad for ten full minutes…), and so, I shall not patent this “royal pain download accelerator”… :-)
But I do have questions:
1) My (uneducated) guess is that the above stop-start technique “fools” the YouTube server, which is perhaps deliberately set to give high speeds for small files only, and restrict the speed beyond certain quota. Is this indeed so?
2) And if it is so, do those “turbo-accelerators” work exactly as the above – “fooling” the server with start-stop, except that they do it automatically, instead of manually?
Thank you!
Johan
P.S. I now use another add-on for YouTube (while reserving the Download Helper for other web sites, where it is indeed useful), which is very fast by itself, and am wondering about it’s design…
11-Feb-2012
From one perspective all web browsers have a mini accelerators built in called the cache. A copy of the web page being made on your computer first then loading up from your hard drive giving the impression of extra speed.
I believe this is how Google’s web accelerator worked in the “dial-up” a days. And speaking of dial-up, when I had it and was still using IE I use to stop the download and start it up again if it dragged to slow. I would always get a little burst of speed when the download started.
I used download managers then that always promised faster downloads, they did not but still where very convenient when downloading through dial-up.
I do believe the “throttling” on UToob (heh! heh!) is simply their way of controlling the enormous bandwidth demands on their system. They send enough data quickly to build a good bit of buffered video, but then throttle back the speed to only slightly faster than the buffer cache empties. This way, YouTube’s servers aren’t overwhelmed (or at least, so they would hope!) with all the traffic they receive.
Leo, I agree that a download managers cannot increase your connection speed (in my case, DSL 6). But my experience with one, Internet Download Manager (by Tonec), is that it makes a substantial difference in effective download speed. The benefit can be dramatic when downloading a list of files, something that browsers typically do not handle well. The IDM interface shows that it processes several files at once and if your download is interrupted it will automatically resume when the connection is restored. Another nice feature is the ability to delay download(s) until a set time, such as when your computer(s) will not be using the internet connection for backup, streaming, etc. In short, I’ve found IDM to be a *very* worthwhile utility that has given rock-solid service since purchase about a year ago and I would buy it again in a heartbeat.
@Johan. A few years ago I had the same ‘stop and go’ job for almost every download. The problem arose from a cable modem–new modem, no trouble anymore.
The initially fast, then slow download speed reported by Johan with YouTube is normal with any download in Firefox, in my experience. I must try the pause/resume trick. I find that Reload will kick (re)start a stalled web page download.