Content
Intro to cURL
cURL lets you transfer data through the command line interface (CLI), making it one of the various tools for making HTTP requests. Its support for various network protocols makes it one of the most used tools for data transfer.
cURL is very useful for sending GET requests over HTTP, which will be the focus of this guide - where we'll explore various ways to send GET requests effectively using curl.
For a more in-depth understanding of cURL, you can refer to our previous article on What is cURL?
What is a GET request?
Before we jump into the specifics of sending GET requests, it's important to understand what a GET request really is.
GET requests are primarily used to retrieve data from a server and are a fundamental part of the HTTP protocol. They're basically the foundation of communication between web browsers and web servers, and they don't alter the state of the (web) server’s resources. That makes them ideal for scenarios where read-only operations are required.
cURL syntax for GET requests
The basic syntax for sending GET requests with curl follows the structure below.
curl https://apify.com
This uses the format curl [URL], which may include optional flags (options) like -X GET which explicitly specifies the request method as GET.
The other part to that is the -X POST which specifies a request as POST.
When the -X option isn’t included in the command, the default method curl uses is -X GET. This is why you would normally use the curl [URL]syntax for GET requests with curl.
Another optional flag is the -H flag, which allows you to send headers with your request.
Headers -H in cURL are typically HTTP headers that help you provide extra context and information about your request to the server. This way, the server has a clear grasp of how to respond to your request.
There are various forms of headers in cURL, which are covered in my article on how to send HTTP headers with cURL.
So, based on what has been discussed, the actual cURL syntax for sending GET requests is:
curl [options] [URL]
Where [options] represent the common options you would send with your GET request, which could be -X GET or -H (appending your types of headers) or both.
How to send a GET request
Now that you have a nice grasp of the syntax for sending a GET request, let's go through a step-by-step illustration of how to send a GET request to an API that returns JSON for a sample post.
Step 1: Open your terminal or command prompt (CMD)
Navigate to where your CMD is on your operating system. This is the application we'll use to enter our cURL commands to send the GET request.
Step 2: Type the GET request
Enter the command below, which also specifies the URL of the JSONPlaceholder resource that we’ll retrieve our data from.
curl https://jsonplaceholder.typicode.com/posts/1
Press Enter to execute the command. You can replace https://jsonplaceholder.typicode.com/posts/1 with the URL of any resource or server you want to access.
Step 3: Review the response
After sending the GET request, you'll receive a response from the server. The response typically includes a status code indicating the success or failure of the request, as well as the content returned by the server.
If you run the example command and the request is successful, you'll see the response data below in your terminal or command prompt.
{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\\nsuscipit recusandae consequuntur expedita et cum\\nreprehenderit molestiae ut ut quas totam\\nnostrum rerum est autem sunt rem eveniet architecto"
}
Parameters in GET requests
You can further customize the response data retrieved from the server by including parameters in the GET request.
These parameters (params) are appended to the URL after a question mark (?), formatted as key-value pairs, and multiple parameters are separated by ampersands (&).
Using the JSONPlaceholder server as a sample, you could query the same post we queried from the previous example using this command:
curl https://jsonplaceholder.typicode.com/posts?userId=1&id=1
This example will also return the same response data as the previous example. In this example, we’re sending a GET request to https://jsonplaceholder.typicode.com/posts with two parameters: userId and id.
The ?userId=1&id=1 is a query parameter filtering posts by user ID and post ID.
Handling headers in GET requests
Headers provide additional information about the client making the request or the request itself. They can be included in cURL requests with the -H flag, followed by the details of the header.
While some headers are automatically included in requests by cURL, such as the User-Agent header, you may need to include custom headers in your GET requests for various purposes, such as authentication, specifying content types, or controlling caching behavior.
For example, you can specifically request response data in JSON format using the Accept header:
curl -H "Accept: application/json" https://api.apify.com/v2/users/apify
While the inclusion of custom headers is an essential aspect of HTTP requests, a detailed discussion on this topic exceeds the scope of this article. For more information, you should refer to How to send HTTP headers with cURL.
Interpreting server responses
After sending a GET request with cURL, the server responds with content, including status code and content (response body). We'll look at these two parts of the HTTP response data.
Status code
HTTP status codes provide information about the success or failure of a request. These codes are grouped into five categories.
- 1xx (Informational): Indicates that the server has received the request and is processing it.
- 2xx (Success): Indicates that the request was successfully received, understood, and accepted.
- 3xx (Redirection): Indicates that further action needs to be taken to complete the request.
- 4xx (Client Error): Indicates that there was an error on the client's side, such as invalid request syntax or authentication failure.
- 5xx (Server Error): Indicates that there was an error on the server's side, such as server overload or internal server error.
Status codes that you’ll frequently encounter include:
- 200 OK: The request was successful.
- 404 Not Found: The requested resource could not be found on the server.
- 401 Unauthorized: The request requires user authentication.
- 500 Internal Server Error: The server encountered an unexpected condition that prevented it from fulfilling the request.
Content
As a part of the server’s response, the data will also include the requested content. This content can vary depending on the nature of the request and the configuration of the server. The types of content you can request from the server include the following:
- JSON: JavaScript Object Notation - a lightweight data-interchange format.
- HTML: Hypertext Markup Language - used for structuring web pages.
- XML: Extensible Markup Language - used for storing and transporting data.
We can instruct the server to return the content in the preferred format by specifying the desired format in the request. Going back to our JSONPlaceholder example:
curl -i -H "Accept: application/json" https://jsonplaceholder.typicode.com/posts?userId=1&id=1
In this example, we introduced the -i flag, which ensures that the response output we get from the jsonplaceholderserver includes the HTTP response headers, along with the status code of the response (200 OK in our case).
HTTP/1.1 200 OK
Content-Type: application/json
{
"userId": 1,
"id": 1,
"title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
"body": "quia et suscipit\\nsuscipit recusandae consequuntur expedita et cum\\nreprehenderit molestiae ut ut quas totam\\nnostrum rerum est autem sunt rem eveniet architecto"
}
Advanced cURL options for GET requests
Other than the functionalities covered so far, cURL offers several advanced options that are useful for fine-tuning GET requests to meet your requirements.
Here are some of the advanced options that are available for GET requests.
Ignore errors with SSL certificates
When you make HTTPS GET requests to servers with self-signed or expired SSL certificates, cURL will sometimes throw SSL certificate verification errors. But, you can bypass these errors by using the -k or --insecure flag (not recommended for sensitive data).
curl -k https://apify.com
This command disregards SSL certificate errors and makes the GET request to https://apify.com
Integrate with Apify Actors
You can integrate cURL with Apify Actors by using the Apify API. This integration allows developers to use the capabilities of Actors and the Apify platform directly within cURL commands.
As an example, we'll use cURL to run an Apify Actor called "compass/crawler-google-places" using the Apify API.
curl "https://api.apify.com/v2/acts/compass~crawler-google-places/runs?token=$API_TOKEN" \\
-X POST \\
-d @input.json \\
-H 'Content-Type: application/json'
Set the API_TOKEN variable with your Apify API token, and prepare the input for the Apify Actor by creating a JSON file named input.json with the desired input parameters.
Use basic auth
When accessing resources protected by basic authentication, you can include the username and password in the GET request using the -u or --user flag.
curl -u username:password http://example.com/auth/data
The command sends a GET request to http://example.com/auth/data with the specified username and password for basic authentication. Basic auth in curl is covered in-depth in this article.
Cookie handling
Cookies are small pieces of data that are sent by a server and stored on the client’s side. They can help you with user authentication and session state.
You can handle cookies during GET requests by using the -b flag followed by the cookie data, as specified in this example.
curl -b "cookie1=value1; cookie2=value2" https://apify.com
Rate limiting
In situations where you need to control the rate of GET requests sent to a server, cURL provides the --limit-rate option. This option allows you to specify the maximum transfer rate in bytes per second.
For example, to limit the download speed to 100 KB/s:
curl --limit-rate 100k http://example.com
This command restricts the download speed of the response to 100 kilobytes per second.
Troubleshooting cURL GET requests
Despite cURL’s versatility and reliability, it may encounter issues when sending GET requests, ranging from connection timeouts to SSL errors. Let's now review some of the workarounds to some common issues with GET requests in curl.
SSL certificate errors
SSL certificate errors occur when there are issues with the server's SSL certificate, such as expired certificates, mismatched hostnames, or self-signed certificates. These errors can prevent curl from establishing a secure connection with the server.
Depending on the specific error encountered, there are several solutions or workarounds:
- Ignore SSL certificate errors: As mentioned previously, you can use the
kor-insecureflag to bypass SSL certificate verification if the SSL certificate errors are non-critical or if you trust the server.
curl -k https://apify.com
- Specify CA certificates: If the server's SSL certificate is self-signed or issued by a custom certificate authority (CA), you can specify the path to the CA certificates using the
-cacertflag.
curl --cacert /path/to/ca-cert.pem https://apify.com
Connection timeouts
Connection timeouts occur when cURL fails to establish a connection with the server within the specified time limit. This can happen due to network issues, server unavailability, or firewall restrictions.
To solve this, you’d need to increase the timeout value using the --connect-timeout flag to allow more time for the connection to be established.
For a request to apify.com, you can set the connection timeout to 10 seconds, providing more time for the connection to be established before timing out.
curl --connect-timeout 10 https://apify.com
Learn more about cURL
Whether you're accessing APIs or downloading files, cURL offers a powerful and flexible way to send GET requests to servers via the command line. To further explore cURL usage in conjunction with Python, check out our detailed guide on How to use cURL in Python.
Cheat sheet for cURL GET requests
| Task | cURL Command |
|---|---|
| Basic GET Request | curl https://apify.com |
| Include Headers in Request | curl -H "HeaderName: Value" https://apify.com |
| Verbose Mode (Include Request & Response Headers) | curl -v https://apify.com |
| Follow Redirects | curl -L http://example.com |
| Save Response to File | curl -o index.html https://apify.com |
| Send GET Request with Query Parameters | curl "http://example.com/data?key1=value1&key2=value2" |
| Use a Proxy | curl -x proxy-server:port http://example.com |
| Specify a User Agent | curl -A "User-Agent: Mozilla/5.0" http://example.com |
| Accept a Specific Format (e.g., JSON) | curl -H "Accept: application/json" http://example.com |
| Limit Download Speed | curl --limit-rate 100k http://example.com |
| Use Cookies | curl -b "name1=value1; name2=value2" http://example.com |
| Store Response Headers | curl -D headers.txt http://example.com |
| Use Basic Authentication | curl -u username:password http://example.com |
| Ignore SSL Certificate Errors | curl -k https://example.com |
| Show Request/Response with Timing | curl -w "@timing-format.txt" -o /dev/null -s http://example.com |
| Output Response Body Only | curl -o /dev/null -s http://example.com |
| Retry Failed Requests | curl --retry 3 https://apify.com |
| Resume a Download | curl -C - -o filename http://example.com/largefile |
| Disable Output | curl -s -o /dev/null http://example.com |
| Disable Progress Meter | curl -sS http://example.com |
| Include IP Address in Output | curl -w "\nLookup Time:\t%{time_namelookup}\nConnect Time:\t%{time_connect}\nAppCon Time:\t%{time_appconnect}\nRedirect Time:\t%{time_redirect}\nPretransfer Time:\t%{time_pretransfer}\nStarttransfer Time:\t%{time_starttransfer}\n\nTotal Time:\t%{time_total}\n" -o /dev/null -s http://example.com |
| Send Custom HTTP Method | curl -X METHOD http://example.com |
| Display TLS/SSL Certificate Info | curl --cert-status --cert http://example.com |
| Set Timeout for Request | curl --max-time 10 http://example.com |
| Set DNS Server | curl --dns-servers 8.8.8.8 http://example.com |
| Disable Expect Header | curl -H "Expect:" http://example.com |
| Include Remote IP Address | curl -H "X-Forwarded-For: client_ip" http://example.com |
| Send Custom HTTP Header | curl -H "HeaderName: Value" http://example.com |
| Include Request Body | curl -X POST -d "param1=value1¶m2=value2" http://example.com |
| Save Cookies to File | curl -c cookies.txt http://example.com |
| Load Cookies from File | curl -b cookies.txt http://example.com |
| Include Referer Header | curl -e http://referrer.com http://example.com |
| Follow Links (Recursive) | curl -O -L -r -nd http://example.com |
| Set User-Agent String | curl -A "User-Agent String" http://example.com |
| Use SSL Version | curl --sslv2 https://apify.com |
| Set Source IP Address | curl --interface eth0 http://example.com |
| Display Version Information | curl --version |
| Send HEAD Request | curl -I http://example.com |
| Debug Output | curl -v -o output.txt http://example.com |
FAQs
#1. How can I confirm the installation of cURL?
You can confirm your curl installation by running the command below.
curl --version
#2. What is a GET request?
A GET request is an HTTP method to retrieve server data. It's commonly used to fetch information from a server without modifying any resources on the server's side.
#3. Can cURL follow redirects automatically in a GET request?
Yes, cURL can automatically follow redirects in a GET request using the L or -location flag.
curl -L https://apify.com
#4. How can you include headers in a cURL GET request?
Headers can be included in a cURL GET request using the H or -header flag followed by the header details.
#5. How do you save the output of a cURL GET request to a file?
You can save the output of a cURL GET request to a file using the -o or --output flag followed by the filename.
curl https://apify.com -o output.html
#6. How can you send GET requests with query parameters using cURL?
GET requests with query parameters can be sent using cURL by appending them to the URL.
#7. How do you interpret the response of a GET request in cURL?
The response of a GET request in cURL can be interpreted by analyzing the status codes and content returned by the server. The status code indicates the success or failure of the request, while the content contains the data returned by the server.
#8. Can cURL handle internationalized domain names (IDN) in GET requests?
Yes, cURL can handle internationalized domain names (IDN) in GET requests. IDNs allow you to use characters from different languages and alphabets directly in your domain name instead of being restricted to the standard ASCII characters. cURL supports IDNs using Punycode encoding for domain names.
