How to download a file with cURL

Whether you’re currently using cURL for file downloads or you’re aspiring to utilize it for this purpose, this guide is for you. We'll explore the following:

What is cURL?

cURL (Client for URL) provides a library and command-line tool for transferring data using various protocols. Its versatility makes it a popular tool for automating website interactions, API testing, and file downloads. It can communicate over a wide range of protocols, including HTTP, HTTPS, FTP, FTPS, SMTP, and many more.

Basic file download processes with cURL

Downloading a file with cURL

The core cURL command for file downloads looks like this:

curl [URL] -o [output_file_name]

Replace [URL] with the file's URL and [output_file_name] with your desired file name. For example, to download Apify’s logo (logo.svg) from a content distribution network (CDN) server, you would use this:

curl https://blog.apify.com/content/images/2023/12/logo-light.svg -o logo.svg

This command tells cURL to download the file from the specified URL and save it as the provided filename.

Understanding the output argument `o`

When you use the -o option, cURL writes the downloaded data to the specified file. It also displays handy information during the process, including download progress, speed, and estimated time.

Following redirects

Sometimes, files hide behind redirects. Use the -L option to tell cURL to follow those sneaky redirects and snag the file at its final destination.

Output without `o`

If you omit the -o option, cURL still displays the downloaded data on your screen. This works for text files but not so well for images or other binary data. Think garbled mess instead of a cute cat.

💡

When downloading binary files, printing the content to the terminal may not be human-readable, leading to potential loss of data.

To avoid these issues and ensure the integrity of binary files, it is recommended to use the -o option to specify an output file where the downloaded content will be saved.

Note that the lowercase -o option lets you specify a custom filename for saving the downloaded file, whereas the uppercase -O option instructs curl to use the filename provided by the server for saving the file.

Making cURL silent

For automation and scripting, passing the -s or --silent option to the command suppresses progress output and error messages to make the operation tidy and interruption-free. This is especially useful in scripts and logs where only essential information is preferred.

curl -s https://blog.apify.com/content/images/2023/12/logo-light.svg -o logo.svg

Advanced file download techniques

Limiting download speed in cURL

The --limit-rate option provides control over the bandwidth used for downloads, ensuring other network activities remain unaffected. This feature is particularly beneficial in environments with bandwidth constraints or to avoid network congestion during large file transfers.

curl --limit-rate 1M https://example.com/large-file.zip -O

In this example, the download speed was limited to 1MB/s.

Secure file transfers in cURL

cURL supports HTTPS to provide encrypted connections. You can use the --cacert option to specify a Certificate Authority (CA) bundle for SSL certificate verification. You can also verify SSL certificates to prevent man-in-the-middle attacks.

curl https://secure.example.com/file.zip -O

Downloading portions of a file with `curl --range`

cURL supports range requests that allow you to download specific portions of a file. This is handy for resuming interrupted downloads or fetching only a portion of a large file.

curl -r 0-999 https://example.com/large-file.zip -o partial-file.zip

Seeing download progress in cURL

The -# or --progress-bar option displays a progress bar for the download so you can have a visual representation of the transfer status. This real-time feedback is great for monitoring the progress of large or lengthy downloads.

# Displaying download progress
curl --progress-bar https://example.com/large-file.zip -O

Following redirects in cURL

cURL automatically follows redirects by default, but you can control this behavior with the -L or --location option. This is helpful when dealing with URLs that redirect to another location.

curl -L https://example.com/redirecting-url -o redirected-file.txt

Handling various file download scenarios in cURL

Downloading multiple files

cURL facilitates batch downloading, meaning multiple files can be downloaded in sequence or parallel. You can do this by issuing separate cURL commands for each file to make the downloading process efficient or by appending all URLs to the -O option.

# Downloading multiple files
curl -O https://example.com/file1.zip https://example.com/file2.zip

Resuming interrupted file downloads

The -C or --continue-at option helps recover from interrupted downloads by attempting to resume file transfers from where they were halted. This feature sees to it that downloads are completed, which you'll be thankful for when dealing with files or working under unreliable connection conditions. For example, if file2.zip from above was interrupted, you can use the command below to resume download from the point of interruption.

# Resuming file2 interrupted download
curl --continue-at - https://example.com/file2.zip -O

Downloading files from authenticated URLs

cURL simplifies downloading from URLs requiring authentication by using the -u option along with credentials or -H "Authorization: Bearer [token]" for token-based authentication. This capability is key for accessing restricted files securely.

# Downloading a file with basic authentication
curl -u username:password <https://example.com/secure-file.zip> -O

Customizing cURL requests

Specifying a client in cURL

Changing the User-Agent with the -A option in cURL commands is sometimes necessary for accessing web resources that require certain client identification. This customization facilitates testing and access to resources under specific conditions.

# Setting a custom User-Agent header
curl -A "Mozilla/5.0" https://example.com/file.zip -O

Using cURL with cookies

Cookie management is essential for maintaining sessions or preserving state across requests. cURL’s ability to send and store cookies (with the -b and -c options respectively) plays a significant role in automated login sequences, data scraping, and more.

# Sending cookies with the request
curl --cookie "session_id=123456" https://example.com/protected-resource -O

Debugging issues with file downloads in cURL

When encountering issues with file downloads in cURL, debugging techniques can help identify and resolve problems effectively. Here are a few debugging methods.

Using verbose mode

Using the -v or --verbose option provides verbose output, including detailed information about the HTTP request and response, such as headers and status codes.

curl -v https://example.com/problematic-file.zip -o debugged-file.zip

Retrying requests

In case of transient network issues or server errors, you can retry failed requests using the --retry option with a specified number of retries.

curl --retry 3 https://example.com/problematic-file.zip -o debugged-file.zip

Sending outputs to a file

Saving debugging output to a file using the -o option allows for further analysis and sharing with others for troubleshooting assistance.

curl -v https://example.com/problematic-file.zip -o debugged-output.txt

Conclusion and next steps

We've introduced you to the basics of using cURL for file downloads, but the potential applications of cURL extend far beyond what we've covered here. If you want to learn more about cURL and how to use it, check out the articles below: