Automate all your downloads with curl

Curl is well known on Linux as a flexible command line tool which can download files using a wide range of protocols: HTTP, HTTPS, FTP, FTPS, POP3, IMAP, RTMP, RTSP and many more.

The program is also available on Windows, though (and Mac, iOS, NetWare, Solaris, AmigaOS and more), and is ideal for automating regular downloads in your own custom scripts.

Curl’s basic rules are very simple. Just start by giving it the name of something to download.

curl http://www.example.com

curl http://ift.tt/1tjxaID

By default the contents of whatever you’re downloading is displayed on the screen, but you can redirect this to a file, or use the -o Output option to save it somewhere.

curl -o downloadcrew.html http://ift.tt/1fttVG8

Your resource password might be password-protected, but curl can still access it.

curl http://ift.tt/1tjxaIJ

You can download several files at once.

curl -o http://ift.tt/1oRICLI http://ift.tt/1oRICLK

(As we’re downloading specific files, the -o option will this time save them locally as their online names, file1.txt and file2.txt).

Maybe you only need part of a file? Here’s how to get the first and last 256 bytes of a file.

curl -r 0-255 http://ift.tt/1tjxc34

curl -r -256 http://ift.tt/1tjxc34

Curl can be smarter that you might expect. You’ll often want to download files only if they’re newer than a local copy, for example, and something like this can help.

curl -z local.html http://ift.tt/1tjxaYZ

Remote.html will be compared to Local.html, and only downloaded if it’s more recent.

This is just the start. Curl can send emails. Upload files. Post forms. Resume file transfers. Set upper or lower transfer speed limits, set a custom referrer or user agent, use cookie authentication and more.

And although it is a command line tool, curl does a good job of keeping you in touch with whatever it’s downloading. A progress bar displays the total download size, the bytes downloaded so far, the average transfer speed, time elapsed, expected completion time and more.

Put it all together and curl has more power in its tiny portable executable (<600KB) than you’ll find in many commercial download managers. If you ever need to script downloads -- or you just want to check it out and see what’s possible -- then go grab a copy, and take a look.