The internet is a vast, interconnected network of computers that are in constant communication. At its core, this communication relies on the exchange of HTTP messages between different clients and servers. These messages are the fundamental blocks that allow client applications to access and retrieve various resources from servers—be it webpages, data files, or other digital resources. These applications utilize HTTP clients like Axios (which provide the capability to handle HTTP requests and responses) to send HTTP messages to servers, specifying the required resources.
HTTP messages consist of the actual data being transmitted, HTTP headers, and other elements. HTTP headers, particularly, carry additional information about the requests, such as authentication details and the preferred connection method with the recipient resource. Understanding the mechanics behind these headers is essential to making successful API calls.
In this article, we will explore the fundamentals of Axios, as well as the significance of HTTP headers when making HTTP requests
What is Axios?
Axios is a popular HTTP client library—with over 50 million package downloads at the time of writing—that works well in both server ( Node.js) environments and web browsers. It makes managing HTTP requests to servers a breeze, providing an easy-to-use interface for sending requests and effectively handling the subsequent responses.
Here's a simplified diagram illustrating the interaction between Axios and applications when making API calls:
Axios offers a comprehensive range of features and functionalities for handling HTTP requests and responses, such as:
- Browser compatibility: Axios has extensive browser support, thanks to its internal use of the browser's
XMLHttpRequestmechanism. This enables Axios to work effectively across different web browsers. The Caniuse website lists the specific browser versions it supports. - Interceptors: Axios interceptors allow your applications to intercept HTTP requests and responses. This feature provides a flexible mechanism for modifying or enhancing the behavior of these requests and responses, such as adding specific headers to a particular HTTP request.
- Request and response transformation: Axios simplifies the process of transforming request and response data. It provides support for various data formats, allowing users to work with different data representations.
- Automatic JSON data handling: When working with JSON payloads, Axios automatically parses the response data as JSON. This eliminates the need for manual parsing.
Why use Axios?
While Axios excels at handling HTTP requests, other HTTP clients exist, and they are just as good; the native Fetch API is the most popular alternative.
Both of these HTTP clients offer efficient ways to manage HTTP requests, their responses, and their interaction with servers. Nonetheless, there are certain features that Axios has that, arguably (we all do have our favorite toolsets!) make it a better HTTP client. They include:
- Request cancellation: Axios provides an easy way to cancel pending requests using the
AbortControllerobject. This capability provides greater control over network requests, making it easy to abort or stop requests as needed. - Error handling: Axios provides a clean and simple interface for handling HTTP errors that automatically converts HTTP response codes to errors.
- Setting timeouts: Axios makes it easy to set request timeouts by allowing you to specify a timeout duration in milliseconds. This prevents requests from waiting indefinitely for slow or unavailable servers, avoiding potential performance issues and application crashes.
- Efficient header management: Axios makes it simple to work with HTTP request and response headers. You can easily set, modify, or remove headers to customize communication with servers.
What are HTTP headers?
HTTP headers are additional information (metadata) included in every HTTP request and response exchanged between clients and servers. They act like message labels, providing additional context about the data being sent or received beyond the basic request or response body.
For instance, a Content-Type request header specifies the type of data being sent (text, image, etc.), while a Cache-Control response header tells the browser how to store the response.
Common HTTP headers
HTTP headers come in various types, each serving a specific purpose in the communication between clients and servers. The commonly used types include:
- Request headers: These headers are included in the client's request to the server and provide additional information about the request. Some examples include:
- User-Agent: This header identifies the client application or browser making the request. It helps the server understand the type of client accessing the resource.
- Accept: This header specifies the preferred media types that the client can handle for the response. It allows the server to provide the appropriate content format.
Most API clients, e.g., Postman, provide a convenient Headers section where you can easily configure the values of these headers if you're testing your requests and need to experiment with different header configurations.
- Response headers: These headers are sent by the server in response to the client's request and provide additional information about the response. Examples include:
- Content-Type: Specifies the media type of the response body, such as application/json or text/html.
- Content-Length: Indicates the size of the response body in bytes.
- Server: Identifies the server software handling the request.
- Entity headers: Entity headers provide information about the content of the request or response body, e.g., the Content-Encoding, which specifies the encoding applied to the response body, such as gzip or deflate.
- CORS headers: Cross-Origin Resource Sharing (CORS) headers, such as Access-Control-Allow-Origin and Access-Control-Allow-Methods, control which domains are allowed to access application resources. They help enforce security restrictions on cross-origin requests.
Here are some examples of response headers received after making GET HTTP requests to the JSONPlaceholder API.
These are just a few examples of the many different headers that are included in both requests initiated by clients and responses returned by servers.
Getting started with Axios
We've explored what Axios is and the importance of HTTP headers. Now, let's see Axios in action with some code.
The beauty of Axios is its incredibly versatile; you can use it in both frontend projects utilizing frameworks like React or Angular, as well as backend projects built with Express.js (which we'll be using in this tutorial to demonstrate a few examples).
Now, before you start experimenting with Axios locally, ensure you have the following:
- Node.js version 18 or later installed
- npm (Node Package Manager) installed—comes bundled with Node.js. Alternatively, you can opt for the yarn package manager, which functions similarly to npm.
To install Node.js and npm, there are several ways you can opt for, depending on your operating system:
For macOS:
- If you already use Homebrew, you can install Node.js with the following command in your terminal:
brew install node
- Alternatively, you can download the Node.js installer package from the official Node.js website and follow the installer's on-screen installation instructions.
For Windows:
- Visit the official Node.js website and download the Windows installer. Then, follow the on-screen installation instructions to complete the setup.
Once Node.js is installed, npm will also be installed. You can verify that both Node and npm are available by running the command:
node -v
npm -v
After completing the installation steps, you can start creating and running Node.js applications.
To use Yarn on both macOS and Windows, simply use the following command to install it:
npm install --global yarn
Installing Axios
Now, go ahead and create a working project directory. Inside this folder, create a new index.js file as follows:
#windows
echo. > index.js
#macOS
touch index.js
To use Axios in your project, you'll need to install it as a dependency using npm. To do that, first, initialize a npm as follows:
npm init --y
This command will automatically generate a package.json file in the current directory with default settings. Then, install Axios.
npm install axios
Once the package is installed, you can use it to make HTTP requests and handle responses.
Setting headers with Axios
There are two main approaches for setting headers with Axios:
- On a per-request basis.
- Globally, for all requests.
Let's explore these methods in detail and understand their differences.
Setting headers globally vs. request
What exactly is setting headers on a per-request basis? Ideally, HTTP is a stateless protocol, meaning each request-response interaction is treated independently. The server doesn't inherently remember anything about previous requests from the same client.
Since the server has no built-in memory, HTTP messages rely on headers to convey crucial information for processing the request. These headers include details like the requested resource, content type, etc, hence, the setting headers on a per-request basis. Essentially, it allows you to tailor them based on the unique requirements of each request to a particular resource.
On the other hand, setting headers globally means simply configuring them once, and they will be included in all requests made with the HTTP client. This approach is useful when you have common headers, such as authentication tokens or content types, that need to be included in every request automatically or in each subsequent request.
1. Setting headers per requests
To set headers individually for each request, you can create a configuration object that includes the desired headers and then pass it along with the request. Let's take a look at an example where you can set a header containing authentication information.
In the index.js file, create a header configuration object, and pass it as a parameter in the Axios GET request method, as shown in this code snippet:
import axios from 'axios';
const config = {
headers: {
'Authorization': 'Bearer YOUR_AUTH_ACCESS_TOKEN',
'Content-Type': 'application/json'
}
};
const getData = async () => {
try {
const response = await axios.get('http://localhost:5000/api/users', config);
// Handle the response
} catch (e) {
// Handle errors
}
};
In this example, we set two headers: Authorization and Content-Type. Once you make GET requests to the api/usersendpoint, for instance, these headers will be automatically included with the requests.
2. Setting headers globally for all requests
Setting headers globally for all requests can streamline your workflow, especially when numerous requests share common headers. In Axios, global headers are configured using the defaults.headers.common configuration object.
A common global header you might set is the Accept-Language header. Here's an example illustrating how to set the Accept-Language header globally:
import axios from 'axios';
axios.defaults.headers.common['Accept-Language'] = 'en-US';
axios.get('http://localhost:3000/api/users')
.then(response => {
//handle respose
})
.catch(error => {
// Handle errors
});
In this example, since this header is set globally, all subsequent Axios requests will automatically include it unless overridden in specific request configurations.
It's important to note that you should set headers globally only once, preferably during application initialization or configuration. Any changes made to the defaults.headers.common object will affect all subsequent requests made with Axios.
Handling CORS with Axios
Cross-Origin Resource Sharing (CORS) is a security mechanism implemented by web browsers to restrict cross-origin requests between servers and clients. By default, browsers block requests made from a different domain than the one that served the application, preventing unauthorized access to resources. CORS allows servers to specify which origins (domains) are permitted to access their resources.
When making HTTP requests (e.g., from a client app running on one port to an API service on a different port or domain), the browser first sends a preflight request to the server to check if the requested domain is allowed to access the requested resource. The server responds with CORS headers indicating whether the request is allowed or not. If the server doesn't implement the CORS policy correctly, the browser will block the request, typically throwing the "request to a particular URI is blocked by CORS policy" error.
To correctly implement the CORS policy, you need to enable CORS in the server to allow cross-origin Axios requests. You can either manually build a custom Express middleware to manage which requests are allowed and from which domain or configure the CORS package—which simplifies the process—in your server-side code (in this case, an Express.js application).
Let's take a look at an example. Make sure to install Express.js and the CORS package in your project:
npm install express cors
Make sure to include “type”: “module” In the package.json file to enable the use of ES6 modules in your Express app.
In your server file, the index.js file, import Express, and CORS create an Express app and use the cors() middleware to enable CORS for all routes in your API as follows:
import express from 'express';
import cors from 'cors';
const app = express();
let allowedOrigins = [];
// whitelisted URLs
const allowedClientOrigins = [process.env.PROD_CLIENT_URL_1, process.env.PROD_CLIENT_URL_2];
if (process.env.NODE_ENV === 'development' || process.env.NODE_ENV === 'DEV') {
allowedOrigins = [process.env.CLIENT_URL || 'http://localhost:3000'];
} else if (process.env.NODE_ENV === 'PROD') {
if (allowedClientOrigins.includes(process.env.PROD_CLIENT_URL)) {
allowedOrigins = [process.env.PROD_CLIENT_URL];
} else {
console.error('PROD_CLIENT_URL does not match allowed origins.');
}
} else {
console.error('Invalid NODE_ENV value.');
}
const corsOptions = {
origin: allowedOrigins,
methods: ['GET', 'POST'],
allowedHeaders: ['X-Requested-With'],
};
app.use(cors(corsOptions));
app.options(allowedOrigins, cors(corsOptions));
app.get('/', (req, res) => {
res.send('Hello World!');
});
const PORT = process.env.PORT || 5000;
app.listen(PORT, () => {
console.log(`Server listening on port ${PORT}`);
});
Now, assuming you have configured Axios within a client-side application to make HTTP requests to the Express server as shown below:
import axios from 'axios';
axios.get('http://localhost:5000', {
headers: {
'X-Custom-Auth': 'token123'
})
.then(response => {
//response
})
.catch(error => {
//error
});
The Express server utilizes the CORS middleware to enable CORS (Cross-Origin Resource Sharing) support on all routes. The middleware manages preflight OPTIONS requests and appends CORS headers to the server's responses.
When a client initiates a cross-origin request that includes a custom header in the Axios request, the browser triggers a preflight check. In response, the server adds the necessary CORS headers to the response, allowing the browser to recognize and authorize the intended cross-origin requests.
Advanced Axios header techniques
Axios provides several advanced techniques for managing request and response headers. These techniques allow you to perform advanced functionalities, including adding, updating, or removing headers dynamically before a request is sent or after a response is received. They are particularly useful in scenarios where you need to handle headers based on specific conditions, contexts, or application requirements.
For instance, let's assume you want to implement an authentication mechanism that refreshes the access token when it expires. In this case, you can utilize Axios interceptors to intercept the response, check for an authentication error (e.g., a 401 Unauthorized status code), and automatically refresh the access token. After obtaining a new token, you can then update the Authorization header with the new token and retry the original request.
Now, let's take a look at how we can implement each of these functionalities in code.
Adding, updating, and removing headers
Axios interceptors allow you to intercept requests and responses before they are handled by then/catch promise handlers. Interceptors act like middleware for outgoing Axios calls and incoming responses.
There are two types of interceptors - request and response interceptors. Request interceptors let you modify the request configuration, headers, etc., before the request is sent out. On the other hand, response interceptors allow you to intercept the response object before it is passed back to the then() code block. For example, if you are trying to access a protected route, a request interceptor can add authentication headers to every request. While, a response interceptor can handle 401 responses by refreshing a token automatically.
Axios interceptors are implemented using this interceptors.request/response.use() method. The function will be called sequentially for every outgoing or incoming call. You can even chain multiple interceptors together for different preprocessing tasks. Essentially, interceptors provide a clean way to handle cross-cutting concerns like auth tokens, logging, and more without cluttering your application code.
For demonstration purposes, we'll only explore request interceptors. Let's highlight a few examples.
Adding headers to requests
Here's an example of how you can use Axios interceptors to add headers to requests automatically:
import axios from 'axios';
const config = {
headers: {
'Content-Type': 'application/json'
}
};
axios.interceptors.request.use(function (config) {
config.headers['X-Custom-Header'] = 'Custom Value';
return config;
}, function (error) {
// Handle error
return Promise.reject(error);
});
axios.get('URL', config)
.then(function (response) {
// Handle successful response
})
.catch(function (error) {
// Handle request error
});
In this example, an interceptor is added to the request pipeline using axios.interceptors.request.use. The interceptor function receives the config object, which contains the request configuration. You can then add new headers to the config.headers object. In this case, a custom header 'X-Custom-Header' with the value 'Custom Value'.
Updating a header
Interceptors can also be used to update existing headers. For example, you might need to update the Authorizationheader with a new access token after it has been refreshed:
axios.interceptors.request.use(function (config) {
const token = localStorage.getItem('auth_token');
if (token) {
config.headers['Authorization'] = `Bearer ${token}`;
}
return config;
}, function (error) {
// Handle request error
});
In this example, the interceptor updates the token value of the 'Authorization' header with a new access token.
Removing a header
To remove a header, you can use the delete operator on the config.headers object, specifying the header key you want to remove:
axios.interceptors.request.use(config => {
delete config.headers['X-Custom-Header'];
return config;
});
Using Axios interceptors to alter headers dynamically
Now, in some cases, you may need to modify the HTTP headers based on dynamic conditions or factors that may vary from one request to another. This approach allows you to adjust headers in real time based on the context of each individual request.
Dynamically altering headers is a totally different function from just updating headers. While updating headers involves directly modifying the header values, dynamically altering headers using interceptors allows you to intercept the request before it is sent and make changes to the headers based on custom logic or conditions.
For example, you might add or remove headers based on user authentication status, the type of content being requested, or the current state of the application.
Let's take a look at an example where we include a language header to serve content in an application based on the user's default language settings.
const axios = require('axios');
axios.interceptors.request.use(function (config) {
const userLanguage = getUserLanguage();
config.headers['Accept-Language'] = userLanguage;
return config;
});
function getUserLanguage() {
const language = navigator.languages[0] || navigator.language;
return language;
}
In this example, the interceptor will dynamically alter the Accept-Language header based on the user's preferred language.
Handling conditional headers
Conditional HTTP headers allow you to make requests that are executed differently based on the specified preconditions. This is useful when you want to optimize the performance of your applications, as well as reduce server load.
For instance, let's assume you want to access users’ profile data from your user API service. Instead of blindly fetching the entire profile data every time, you can use conditional headers to check if the data has changed on the server since your last access and then fetch and update the information on the client (only if there are updates).
There are several HTTP conditional headers, including:
- If-Match: This header specifies an
expectedETag (entity tag) value for the resource being requested. The server responds with a304 (Not Modified) status code if the resource ETag matches the one provided in the header, indicating no changes. This saves unnecessary data transfer if the requested information is already up-to-date. - If-None-Match: This header works opposite to
If-Match. It provides a list of ETags that the resource should not match. The server responds with a 200 (OK) and the full resource data if none of the listed ETags match the server's ETag for the resource. This is helpful for efficient updates, fetching only the modified parts. - If-Modified-Since: This header specifies a date and time. When attached to HTTP requests, the server will respond with a
304 (Not Modified)if the resource's last modification date on the server is earlier than the provided date, indicating that there were no changes since the last access. This tells the client that its cached copy is still valid and no re-download is necessary.
Here's an example of how you can use these headers in Axios to handle conditional requests:
import axios from 'axios';
const userId = "user1";
const lastModifiedDate = localStorage.getItem(`user-${userId}-lastModified`);
async function getUserProfile(userId) {
const url = `http://localhost:5000/api/users/${userId}`;
const headers = {};
if (lastModifiedDate) {
headers['If-Modified-Since'] = lastModifiedDate;
}
try {
const response = await axios.get(url, { headers });
if (response.status === 200) {
console.log('User profile updated:', response.data);
localStorage.setItem(`user-${userId}-lastModified`, response.headers['last-modified']);
} else if (response.status === 304) {
console.log('User profile unchanged.');
} else {
console.error('Error fetching user profile:', response.statusText);
}
} catch (error) {
console.error('Error:', error);
}
}
getUserProfile(userId);
In this example, we want to conditionally fetch a user profile from an API endpoint based on the data modification status. First, the code retrieves a previously stored last modified date for the user profile from the local storage. If a date is found, it sets the If-Modified-Since header with that value before making the API request using Axios.
Now, based on the response status code received from the server, you can determine the next course of action. If the status is 200 (OK), meaning the resource has been modified since the last modified date provided. In this case, we log the updated user profile data and store the new last modified date from the response headers in the local storage.
However, if the status code is 304 (Not Modified), indicating that the resource has not been modified since the last modified date provided. You can log a message stating that the user profile is unchanged. Therefore, there's no need to update the data or the local storage.
Using HTTP headers with Axios for web scraping
Web scraping is the automated process of collecting data from websites in a structured format. Normally, websites tend to block scrapers because they appear like bots sending automated requests.
To bypass bot detection systems and minimize the risk of being blocked during web scraping, it's essential to include appropriate HTTP headers in your requests. These headers make the scrapers' requests look like regular users on a browser, appearing more legitimate and reducing the likelihood of being flagged as bots.
Although Axios isn't built specifically for it, Axios' features can be helpful for web scraping in Node.js. For instance, you can use Axios interceptors to add, modify, and set headers dynamically based on the website you're scraping. This allows you to customize headers for different websites and reduce the chances of getting blocked. Ideally, by adjusting headers, Axios can disguise scraping bots as regular traffic to evade blockers.
While there are many HTTP headers you can attach to requests, only a few are crucial for web scraping. Here are some that are essential:
- User-Agent header: This header identifies the client making the request, including the operating system, version, and so on. Setting a proper User-Agent header helps make your requests appear more legitimate.
- Accept-Language header: This header specifies the user's preferred language. Including this header can make the scraper resemble a genuine user, decreasing the risk of detection and blocking by websites.
Referer header: The Referer header (yes, referer is intentionally misspelled - see the Wikipedia page on HTTP referer for more details on the history of this header name) specifies the URL of the web page that linked to the current page being requested. Some websites may expect a valid Referer header and use it as a security measure or to track the source of the request. Here's an example of how you can achieve this:
import axios from 'axios';
function generateReferer() {
// If running in a browser environment
const referer = window.document.referrer;
return referer;
}
const headers = {
'Referer': generateReferer(),
};
axios.get('URL', { headers })
.then(response => {
// Handle the response here
})
.catch(error => {
// Handle the error here
});
In this code example, the window.document.referrer line retrieves the referrer URL from the browser. It works by accessing the document object of the current page, which is available via window.document property. The referrer property on the document contains the URL of the previous page that linked to the current page. The generated referrer is then added to a headers object and passed along with an Axios GET request. This way, by including the referrer header in their requests, scrapers can make their web scraping tasks appear more authentic and mimic typical user behavior, as the requests appear to originate from a regular browser session rather than from automated scripts.
Depending on the website you're scraping, you may need to set the correct user-agents and browser headers, such as cookies, authorization tokens, or custom headers required by the website's API.
To properly learn how to scrape websites in Node.js, you can refer to this guide to web scraping using Node.js and Cheerio.
Troubleshooting common HTTP header and Axios issues
Here are some common problems that you can encounter when working with HTTP headers and Axios and troubleshooting steps to resolve them.
- CORS policy errors: These errors are common when making HTTP requests, even with Axios, because, by default, browsers implement CORS protections that restrict HTTP access to resources hosted on different domains. To resolve these errors, implement CORS policies on the server code—the CORS middleware library should be sufficient for this task; it will automatically attach the required CORS headers to responses.
- Headers not being sent: When encountering a situation where headers are not being sent with your Axios requests, first ensure that you've configured the headers correctly within the Axios request setup. Double-check the syntax and values to ensure accuracy. In addition, check whether any interceptors or middleware are altering or removing the headers.
- Unexpected header values: If you encounter unexpected header values in the response or encounter issues with header parsing, it could be due to various factors, such as misconfigurations or server-side discrepancies. One way to address this is by inspecting the server response headers. Simply log them out. This way, you can see all the headers received from the server, including their names and values.
Conclusion and next steps
Throughout this article, we've explored the fundamentals of Axios, as well as HTTP headers. We covered how to use Axios to send HTTP headers, along with various other advanced techniques and use cases.
If you're curious and want to explore HTTP headers further, consider following other guides on the topic, such as learning how to send HTTP headers using cURL, a command-line tool for transferring data via URLs.
