How to use cURL with a proxy (2024 step-by-step guide)

What is cURL?

cURL is a command-line tool for transferring data using various protocols. It's commonly used for downloading data and testing APIs.

What is a proxy in cURL?

Before we let you in on what a proxy is in cURL, it's important to clarify what a proxy is in general. A proxy sits in between your computing device and the internet. So, a proxy in cURL serves as the intermediary between the cURL client running on your computer and the internet. By using a proxy with cURL, you're able to route your requests through a different internet protocol (IP) address.

What do you need to use cURL through a proxy?

Here are the key resources you need in order to use cURL through a proxy.

cURL installed

cURL comes pre-installed on all major operating systems. But if it isn’t already installed on your operating system, kindly follow this quick guide to installing cURL on various platforms.

• Windows

If you use the Chocolatey package manager, you can run the choco install curl command from the command prompt. Otherwise, follow the steps below.

1. Download the cURL executable from the official website. You can search for “windows” to jump to the section with Windows-compatible versions.
2. Extract the downloaded ZIP file.
3. Move the curl.exe file to a directory included in your system's PATH environment variable, such as C:\\Windows\\System32.

Confirm your cURL installation with the command curl --version.

• Linux

Open your terminal of choice, and run the command below.

# update your distro's apt repo to ensure it has an up-to-date URL to the latest version of curl
sudo apt update

# download the curl package
sudo apt install curl

• macOS

Open your terminal and run the command.

brew install curl

This example assumes you have a version of Homebrew running on your macOS. Otherwise, you'll need to run the following command from your Mac’s terminal before running the brew install curl command.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

For more information on installing Homebrew, kindly go to the installation page.

Proxy information

In order to use cURL with a proxy, you'll need the following:

• Protocol: The type of proxy (HTTP, HTTPS, or SOCKS).
• Proxy server address: The IP address or hostname of the proxy server.
• Port: The port number the proxy server uses (usually 80 for HTTP or 443 for HTTPS).
• Username (if authentication is required): A valid username to access the proxy server.
• Password (if authentication is required): The password to the above username.

Using the command line argument to set a cURL proxy

To set a proxy using the command line argument, use the --proxy option followed by the proxy server's address and port number. If the proxy requires authentication, add the authentication credentials of the username and password to the proxy server option like below.

curl --proxy https://username:password@proxy-server:port URL

In the example, the proxy-server:port is the proxy server address with its port number, and the URL is the target server address we want to access.

Ultimately, the command will use the remote proxy server you specify (with the specified port number) for proxy-server:port to access the target URL.

Using environment variables

Environment variables provide a way to set configuration options for cURL. To use a proxy with environment variables, you can set the variables for:

• HTTP protocol: Export a value for the http_proxy variable
• HTTPS protocol: Export a value for the https_proxy variable
• All protocols: Export a value for the all_proxy variable

Here's an example of using a proxy through the HTTPS protocol.

export https_proxy=https://173.212.237.43:52366
curl https://httpbin.org

In this example, cURL will know to automatically use the https_proxy as our proxy server for the target URL since they run on the same HTTPS protocol.

Using an alias

An alias is a shortcut for a command or a series of commands. To use a proxy with an alias, create an alias that includes the -x or --proxy option. Aliases let you create shortcuts for your frequently used cURL commands. An alias is a useful option for proxy settings.

alias curl_proxy='curl --proxy https://35.185.196.38:3128'
curl_proxy https://httpbin.org

With aliases, you can also refer to an environment variable that has been previously defined. So, with an environment variable, the above command can be expressed like this:

http_proxy=https://35.185.196.38:3128
alias curl_proxy='curl --proxy $http_proxy’
curl_proxy https://httpbin.org

Using a .curlrc file

The .curlrc file is a config file for cURL that allows you to set default options. This file is read by cURL when it starts, and it can save you time and effort by avoiding the need to repeatedly enter the same options and settings.

# .curlrc file
proxy = [https://35.185.196.38:3128](https://35.185.196.38:3128/)

Using a rotating proxy with cURL

A rotating proxy is a type of proxy that changes the IP address for each request. To use a rotating proxy with cURL, you can use a proxy service that provides rotating proxies, like Apify’s Proxy service, or write a script that rotates the proxy list. It's advisable to use proxy services as they're more convenient than proxy lists. If you’d prefer proxy lists, here are the steps involved in creating a rotating proxy using proxy lists.

• Create a list of proxy addresses

Create a txt file (e.g., proxies.txt) that contains the IP addresses or proxies to rotate, with each proxy on a separate line in this format [proxy_address]:[port]. Your list can contain proxies from different providers.

// proxies.txt

proxy.apify.com:8000
35.185.196.38:3128
// add others

• Create your proxy rotation script

This sample cURL script will iterate through the proxy list and use each proxy in the list for each request to be made to the target URL.

#!/bin/bash

# Target URL
target_url="https://www.google.com"  # your target server URL

# Proxy list file
proxy_list="proxies.txt"

while IFS=':' read -r proxy_address proxy_port; do
  # Try the current proxy in the loop
  output=$(curl -x "$proxy_address:$proxy_port" -s -o output.html "$target_url" 2>&1)

  # Check if curl encountered an error while using the current proxy
  if [ $? -ne 0 ]; then
    echo "Error using proxy: $proxy_address:$proxy_port"
    echo "Error message: $output"
  else
    echo "Request successful using proxy: $proxy_address:$proxy_port"
  fi

  # Delay (optional) - Adjust delay between requests to avoid overwhelming servers
  sleep 2
done < "$proxy_list"

• Run the Script

Make the script executable with chmod +x curl_proxy_rotate.sh (replace with your script’s filename) and run it with ./curl_proxy_rotate.sh.

Using SOCKS proxies with cURL

SOCKS proxies are a type of proxy that can handle any type of network traffic, including HTTP, HTTPS, FTP, and SSH. They offer a more advanced way to anonymize your traffic compared to HTTP/HTTPS proxies, making your IP address mostly invisible. To use a SOCKS proxy with cURL, use the -x or --proxy option, followed by the socks4:// or socks5:// protocol and the proxy server's address and port number.

curl -x socks5://45.79.189.110:16148 https://apify.com

This command uses an Akamai SOCKS5 proxy server at 45.79.189.110 on port 16148 to access the Apify homepage.

Best proxies for cURL

There are various proxy providers, and for these providers, a range of proxy options are available which are tailored to specific use cases. Let's explore the best proxy options available through providers like Apify Proxy.

• Datacenter proxies

Datacenter proxies are IP addresses hosted on servers in data centers. They're fast and cost-effective, making them suitable for tasks that require high speed and moderate anonymity. However, they're more likely to be detected and blocked by websites that actively monitor for proxy usage. Below is an example that demonstrates the usage of the Apify datacenter proxy.

curl -x http://auto:<your_password>@proxy.apify.com:8000
https://google.com

Replace your_password with the password provided in Apify Console. Note that we haven't specified a username, as the Apify Proxy service utilizes the username field to pass parameters like groups.

This is useful when you want to modify your proxy options. The auto option specifies that the default group setting for proxies should be used, which is the datacenter option by default.

• Residential proxies

Residential proxies use IP addresses assigned to real residential users, making them appear more legitimate and very less likely to be blocked. They offer higher anonymity and are highly suitable for tasks that require scraping without being detected.

This example demonstrates the usage of the Apify residential proxy to anonymize our request to get data on Apify from Google servers.

curl -x http://groups-RESIDENTIAL,country-UK:<your_password>@proxy.apify.com:8000 https://www.google.com/search?q=Apify

• Google SERP Proxies

Apify offers Google SERP proxies, which are specialized proxy services designed to retrieve the HTML code of Google search result pages.

The Apify Google SERP proxy also allows you to route your HTTP requests through a proxy server from a selected country, providing a way to access Google search results from different regions.

This proxy service is particularly useful for web scraping and data collection tasks that require accessing Google search results from various countries.

curl -x http://groups-GOOGLE_SERP:<your_password>@proxy.apify.com:8000 https://www.google.com/search?q=><query

Replace your_password with your proxy password, and query with your actual query. If your query is more than one word, use URL encoding to encode characters like space.

• ISP proxies

ISP proxies use IP addresses directly assigned by internet service providers (ISPs). They're reliable and offer good performance. ISP proxies are suitable for tasks that require stability and consistency.

• Mobile proxies

Mobile proxies use IP addresses assigned by mobile carriers. They're highly anonymous and are most suitable for tasks that require accessing location-specific content or apps.

Tips for using proxies with cURL

Ignore the proxy for a single cURL request

If you need to bypass the proxy for a specific cURL request, you can use the --noproxy option:

curl --noproxy https://blog.apify.com https://apify.com

This command will ensure that both target URLs are accessed without proxy settings, even if set.

Quickly turning proxies on and off

Toggle proxy usage by setting or unsetting the  HTTP_PROXY and HTTPS_PROXY environment variables in your shell.

To turn on the proxy:

# set HTTP proxy
export HTTP_PROXY=https://proxy.apify.com:8000

# set HTTPS proxy
export HTTPS_PROXY=https://proxy.apify.com:8000

To turn off the proxy:

# unset HTTP proxy
unset HTTP_PROXY

# unset HTTPS proxy
unset HTTPS_PROXY

The other way to turn off or unset a proxy is by setting both HTTP & HTTPS proxies as empty variables

 # unset HTTP proxy
export HTTP_PROXY=''

# unset HTTPS proxy
export HTTPS_PROXY=''

Bypass SSL certificate errors

If you're using a proxy and encounter SSL certificate errors, you can use the --insecure option to bypass the certificate verification when you are using self-signed certificates.

curl --insecure --proxy http://173.212.237.43:52366
https://www.google.com/search?q=Apify

Get more information about the request

To get more information about a cURL request, including proxy usage, you can use the --verbose or -v option:

curl --verbose --proxy http://173.212.237.43:52366
https://www.google.com/search?q=Apify

Frequently asked questions (FAQs)

Conclusion

This guide has covered the various ways to configure cURL to work with proxy settings, including command-line arguments, environment variables, aliases, and configuration files. By understanding these techniques, you can use cURL to do web scraping and make API calls while making the most of a proxy server.