How to send a GET request with cURL

Intro to cURL

cURL lets you transfer data through the command line interface (CLI), making it one of the various tools for making HTTP requests. Its support for various network protocols makes it one of the most used tools for data transfer.

cURL is very useful for sending GET requests over HTTP, which will be the focus of this guide - where we'll explore various ways to send GET requests effectively using curl.

For a more in-depth understanding of cURL, you can refer to our previous article on What is cURL?

What is a GET request?

Before we jump into the specifics of sending GET requests, it's important to understand what a GET request really is.

GET requests are primarily used to retrieve data from a server and are a fundamental part of the HTTP protocol. They're basically the foundation of communication between web browsers and web servers, and they don't alter the state of the (web) server’s resources. That makes them ideal for scenarios where read-only operations are required.

cURL syntax for GET requests

The basic syntax for sending GET requests with curl follows the structure below.

curl https://apify.com

This uses the format curl [URL], which may include optional flags (options) like -X GET which explicitly specifies the request method as GET.

The other part to that is the -X POST which specifies a request as POST.

📌

POST requests send data to a server and are useful when you need to alter the state of the server by submitting data to it.

When the -X option isn’t included in the command, the default method curl uses is -X GET. This is why you would normally use the curl [URL]syntax for GET requests with curl.

Another optional flag is the -H flag, which allows you to send headers with your request.

Headers -H in cURL are typically HTTP headers that help you provide extra context and information about your request to the server. This way, the server has a clear grasp of how to respond to your request.

There are various forms of headers in cURL, which are covered in my article on how to send HTTP headers with cURL.

So, based on what has been discussed, the actual cURL syntax for sending GET requests is:

curl [options] [URL]

Where [options] represent the common options you would send with your GET request, which could be -X GET or -H (appending your types of headers) or both.

How to send a GET request

Now that you have a nice grasp of the syntax for sending a GET request, let's go through a step-by-step illustration of how to send a GET request to an API that returns JSON for a sample post.

Step 1: Open your terminal or command prompt (CMD)

Navigate to where your CMD is on your operating system. This is the application we'll use to enter our cURL commands to send the GET request.

Step 2: Type the GET request

Enter the command below, which also specifies the URL of the JSONPlaceholder resource that we’ll retrieve our data from.

curl https://jsonplaceholder.typicode.com/posts/1

Press Enter to execute the command. You can replace https://jsonplaceholder.typicode.com/posts/1 with the URL of any resource or server you want to access.

Step 3: Review the response

After sending the GET request, you'll receive a response from the server. The response typically includes a status code indicating the success or failure of the request, as well as the content returned by the server.

If you run the example command and the request is successful, you'll see the response data below in your terminal or command prompt.

{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\\nsuscipit recusandae consequuntur expedita et cum\\nreprehenderit molestiae ut ut quas totam\\nnostrum rerum est autem sunt rem eveniet architecto"
}

Parameters in GET requests

You can further customize the response data retrieved from the server by including parameters in the GET request.

These parameters (params) are appended to the URL after a question mark (?), formatted as key-value pairs, and multiple parameters are separated by ampersands (&).

Using the JSONPlaceholder server as a sample, you could query the same post we queried from the previous example using this command:

curl https://jsonplaceholder.typicode.com/posts?userId=1&id=1

This example will also return the same response data as the previous example. In this example, we’re sending a GET request to https://jsonplaceholder.typicode.com/posts with two parameters: userId and id.

The ?userId=1&id=1 is a query parameter filtering posts by user ID and post ID.

💡

The supported parameters of any given endpoint can usually be found in the API reference documentation for that endpoint. The documentation for the example above can be found here.

Handling headers in GET requests

Headers provide additional information about the client making the request or the request itself. They can be included in cURL requests with the -H flag, followed by the details of the header.

While some headers are automatically included in requests by cURL, such as the User-Agent header, you may need to include custom headers in your GET requests for various purposes, such as authentication, specifying content types, or controlling caching behavior.

For example, you can specifically request response data in JSON format using the Accept header:

curl -H "Accept: application/json" https://api.apify.com/v2/users/apify

While the inclusion of custom headers is an essential aspect of HTTP requests, a detailed discussion on this topic exceeds the scope of this article. For more information, you should refer to How to send HTTP headers with cURL.

Interpreting server responses

After sending a GET request with cURL, the server responds with content, including status code and content (response body). We'll look at these two parts of the HTTP response data.

Status code

HTTP status codes provide information about the success or failure of a request. These codes are grouped into five categories.

1xx (Informational): Indicates that the server has received the request and is processing it.
2xx (Success): Indicates that the request was successfully received, understood, and accepted.
3xx (Redirection): Indicates that further action needs to be taken to complete the request.
4xx (Client Error): Indicates that there was an error on the client's side, such as invalid request syntax or authentication failure.
5xx (Server Error): Indicates that there was an error on the server's side, such as server overload or internal server error.

Status codes that you’ll frequently encounter include:

200 OK: The request was successful.
404 Not Found: The requested resource could not be found on the server.
401 Unauthorized: The request requires user authentication.
500 Internal Server Error: The server encountered an unexpected condition that prevented it from fulfilling the request.

Content

As a part of the server’s response, the data will also include the requested content. This content can vary depending on the nature of the request and the configuration of the server. The types of content you can request from the server include the following:

JSON: JavaScript Object Notation - a lightweight data-interchange format.
HTML: Hypertext Markup Language - used for structuring web pages.
XML: Extensible Markup Language - used for storing and transporting data.

We can instruct the server to return the content in the preferred format by specifying the desired format in the request. Going back to our JSONPlaceholder example:

curl -i -H "Accept: application/json" https://jsonplaceholder.typicode.com/posts?userId=1&id=1

In this example, we introduced the -i flag, which ensures that the response output we get from the jsonplaceholderserver includes the HTTP response headers, along with the status code of the response (200 OK in our case).

HTTP/1.1 200 OK
Content-Type: application/json

{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\\nsuscipit recusandae consequuntur expedita et cum\\nreprehenderit molestiae ut ut quas totam\\nnostrum rerum est autem sunt rem eveniet architecto"
}

Advanced cURL options for GET requests

Other than the functionalities covered so far, cURL offers several advanced options that are useful for fine-tuning GET requests to meet your requirements.

Here are some of the advanced options that are available for GET requests.

Ignore errors with SSL certificates

When you make HTTPS GET requests to servers with self-signed or expired SSL certificates, cURL will sometimes throw SSL certificate verification errors. But, you can bypass these errors by using the -k or --insecure flag (not recommended for sensitive data).

curl -k https://apify.com

This command disregards SSL certificate errors and makes the GET request to https://apify.com

Integrate with Apify Actors

You can integrate cURL with Apify Actors by using the Apify API. This integration allows developers to use the capabilities of Actors and the Apify platform directly within cURL commands.

As an example, we'll use cURL to run an Apify Actor called "compass/crawler-google-places" using the Apify API.

curl "https://api.apify.com/v2/acts/compass~crawler-google-places/runs?token=$API_TOKEN" \\
-X POST \\
-d @input.json \\
-H 'Content-Type: application/json'

Set the API_TOKEN variable with your Apify API token, and prepare the input for the Apify Actor by creating a JSON file named input.json with the desired input parameters.

Use basic auth

When accessing resources protected by basic authentication, you can include the username and password in the GET request using the -u or --user flag.

curl -u username:password http://example.com/auth/data

The command sends a GET request to http://example.com/auth/data with the specified username and password for basic authentication. Basic auth in curl is covered in-depth in this article.

Cookies are small pieces of data that are sent by a server and stored on the client’s side. They can help you with user authentication and session state.

You can handle cookies during GET requests by using the -b flag followed by the cookie data, as specified in this example.

curl -b "cookie1=value1; cookie2=value2" https://apify.com

Rate limiting

In situations where you need to control the rate of GET requests sent to a server, cURL provides the --limit-rate option. This option allows you to specify the maximum transfer rate in bytes per second.

For example, to limit the download speed to 100 KB/s:

curl --limit-rate 100k http://example.com

This command restricts the download speed of the response to 100 kilobytes per second.

Troubleshooting cURL GET requests

Despite cURL’s versatility and reliability, it may encounter issues when sending GET requests, ranging from connection timeouts to SSL errors. Let's now review some of the workarounds to some common issues with GET requests in curl.

SSL certificate errors

SSL certificate errors occur when there are issues with the server's SSL certificate, such as expired certificates, mismatched hostnames, or self-signed certificates. These errors can prevent curl from establishing a secure connection with the server.

Depending on the specific error encountered, there are several solutions or workarounds:

Ignore SSL certificate errors: As mentioned previously, you can use the k or -insecure flag to bypass SSL certificate verification if the SSL certificate errors are non-critical or if you trust the server.

curl -k https://apify.com

Specify CA certificates: If the server's SSL certificate is self-signed or issued by a custom certificate authority (CA), you can specify the path to the CA certificates using the -cacert flag.

curl --cacert /path/to/ca-cert.pem https://apify.com

Connection timeouts

Connection timeouts occur when cURL fails to establish a connection with the server within the specified time limit. This can happen due to network issues, server unavailability, or firewall restrictions.

To solve this, you’d need to increase the timeout value using the --connect-timeout flag to allow more time for the connection to be established.

For a request to apify.com, you can set the connection timeout to 10 seconds, providing more time for the connection to be established before timing out.

curl --connect-timeout 10 https://apify.com

Learn more about cURL

Whether you're accessing APIs or downloading files, cURL offers a powerful and flexible way to send GET requests to servers via the command line. To further explore cURL usage in conjunction with Python, check out our detailed guide on How to use cURL in Python.

Cheat sheet for cURL GET requests

Task	cURL Command
Basic GET Request	curl https://apify.com
Include Headers in Request	curl -H "HeaderName: Value" https://apify.com
Verbose Mode (Include Request & Response Headers)	curl -v https://apify.com
Follow Redirects	curl -L http://example.com
Save Response to File	curl -o index.html https://apify.com
Send GET Request with Query Parameters	curl "http://example.com/data?key1=value1&key2=value2"
Use a Proxy	curl -x proxy-server:port http://example.com
Specify a User Agent	curl -A "User-Agent: Mozilla/5.0" http://example.com
Accept a Specific Format (e.g., JSON)	curl -H "Accept: application/json" http://example.com
Limit Download Speed	curl --limit-rate 100k http://example.com
Use Cookies	curl -b "name1=value1; name2=value2" http://example.com
Store Response Headers	curl -D headers.txt http://example.com
Use Basic Authentication	curl -u username:password http://example.com
Ignore SSL Certificate Errors	curl -k https://example.com
Show Request/Response with Timing	curl -w "@timing-format.txt" -o /dev/null -s http://example.com
Output Response Body Only	curl -o /dev/null -s http://example.com
Retry Failed Requests	curl --retry 3 https://apify.com
Resume a Download	curl -C - -o filename http://example.com/largefile
Disable Output	curl -s -o /dev/null http://example.com
Disable Progress Meter	curl -sS http://example.com
Include IP Address in Output	curl -w "\nLookup Time:\t%{time_namelookup}\nConnect Time:\t%{time_connect}\nAppCon Time:\t%{time_appconnect}\nRedirect Time:\t%{time_redirect}\nPretransfer Time:\t%{time_pretransfer}\nStarttransfer Time:\t%{time_starttransfer}\n\nTotal Time:\t%{time_total}\n" -o /dev/null -s http://example.com
Send Custom HTTP Method	curl -X METHOD http://example.com
Display TLS/SSL Certificate Info	curl --cert-status --cert http://example.com
Set Timeout for Request	curl --max-time 10 http://example.com
Set DNS Server	curl --dns-servers 8.8.8.8 http://example.com
Disable Expect Header	curl -H "Expect:" http://example.com
Include Remote IP Address	curl -H "X-Forwarded-For: client_ip" http://example.com
Send Custom HTTP Header	curl -H "HeaderName: Value" http://example.com
Include Request Body	curl -X POST -d "param1=value1&param2=value2" http://example.com
Save Cookies to File	curl -c cookies.txt http://example.com
Load Cookies from File	curl -b cookies.txt http://example.com
Include Referer Header	curl -e http://referrer.com http://example.com
Follow Links (Recursive)	curl -O -L -r -nd http://example.com
Set User-Agent String	curl -A "User-Agent String" http://example.com
Use SSL Version	curl --sslv2 https://apify.com
Set Source IP Address	curl --interface eth0 http://example.com
Display Version Information	curl --version
Send HEAD Request	curl -I http://example.com
Debug Output	curl -v -o output.txt http://example.com

FAQs

#1. How can I confirm the installation of cURL?

You can confirm your curl installation by running the command below.

curl --version

#2. What is a GET request?

A GET request is an HTTP method to retrieve server data. It's commonly used to fetch information from a server without modifying any resources on the server's side.

#3. Can cURL follow redirects automatically in a GET request?

Yes, cURL can automatically follow redirects in a GET request using the L or -location flag.

curl -L https://apify.com

#4. How can you include headers in a cURL GET request?

Headers can be included in a cURL GET request using the H or -header flag followed by the header details.

#5. How do you save the output of a cURL GET request to a file?

You can save the output of a cURL GET request to a file using the -o or --output flag followed by the filename.

curl https://apify.com -o output.html

#6. How can you send GET requests with query parameters using cURL?

GET requests with query parameters can be sent using cURL by appending them to the URL.

#7. How do you interpret the response of a GET request in cURL?

The response of a GET request in cURL can be interpreted by analyzing the status codes and content returned by the server. The status code indicates the success or failure of the request, while the content contains the data returned by the server.

#8. Can cURL handle internationalized domain names (IDN) in GET requests?

Yes, cURL can handle internationalized domain names (IDN) in GET requests. IDNs allow you to use characters from different languages and alphabets directly in your domain name instead of being restricted to the standard ASCII characters. cURL supports IDNs using Punycode encoding for domain names.