Content
Over the past two decades, the way businesses advertise and candidates apply for jobs has radically changed. Gone are the days of reading the classified section in the local newspaper for job vacancies, applicants now search for job opportunities almost exclusively via the web. With the average job seeker visiting at least one job board and four company websites in their hunt for the next role, it's clear the search for talent has moved online. With this change, companies now have a massive, public source of job data, including open positions, candidates, and salary data available at the click of a button.
With the labor market experiencing unprecedented changes in the past few years and businesses feeling the pressure of staff shortages, rising wages and changing patterns of work, this rich source of job market data can help businesses adapt and even thrive amidst the turmoil.
Searching across the various job boards, career websites and aggregator sites, provides a highly detailed view of jobs being created, average wages, and positions in demand across the world. By focusing on a particular sector or group of employers, we can also gain powerful intelligence on which companies are expanding, what products they might need and salary benchmarks for similar roles they’re hiring.
Data-driven companies are using job postings to help them achieve the following;
- Collect job postings for populating niche job boards with job positions.
- Competitor analysis, monitoring hiring trends for skills and technology gaps, and tracking sentiment from current and past employees.
- Lead generation, finding companies hiring for key positions relevant for recruitment and software companies.
- Salary benchmarking, monitoring the salaries of similar roles in the industry, and calculating their fair market rate.
- Investment decision-making, helping local authorities and property development companies predict and plan for future office space requirements.
- Collecting candidate information and automating the applicant management process.
- Hiring trend analysis, predicting demand for skills and job roles, and planning for these today.
However, there is one critical challenge that stands in the way of leveraging this data: the sheer volume of job postings added to job boards each day makes it almost impossible to sift through, collect, and analyze the data to get the insights needed in real time. To make better decisions, you need to analyze large amounts of data, and this requires the right tools to help you scale.
To help them tackle the challenges of collecting and analyzing job postings at scale, data-driven companies are increasingly turning to web scraping and robotic process automation (RPA) tools, which use computers to automatically collect, process and analyze the data, removing the need for humans to spend countless hours of manual data collection. These tools provide the informational edge to help companies excel at recruitment, competitor analysis, lead generation, and even investment planning.
If you’re looking to leverage job postings to give your company a competitive edge, let's take a look at developing your very own job ad web scraping system.
How to use web scraping to collect job data
To begin using web scraping to collect and process job data, you need to first start with a data source. For job postings, this would entail finding one or more sites that contain the job information that you want to collect. Job sites you find on the internet tend to fall into one of three categories:
National or international job sites
Large, popular job sites such as Glassdoor, Indeed or LinkedIn are used by companies across the world to list vacant positions and find candidates. Their size and popularity makes them great sources of job data and return listings from thousands of companies across the country. They can be a great source of data due to their size, but the sheer volume of job data on these sites can make them more difficult to work with if you need to extract large amounts of job data, with some such as LinkedIn employing anti-web scraping technology and even legal challenges to block requests from web scrapers.
Niche or industry-specific job sites
Industry, role, or location-based job sites are becoming increasingly popular these days, helping companies find candidates that are looking for a specific role, want to work remotely or are looking for companies that align with their values, such as sustainability jobs. These sites have fewer jobs and candidates by comparison, but offer more relevant opportunities to both employers and employees. If you’re looking for highly targeted sources of job data in tech, remote or green job listing websites such as Otta, We Work Remotely, or Green Jobs Online can be a great alternative to more popular sites.
Company career pages
If you’re only interested in job information from one or maybe a handful of companies, you can look for the career pages on their websites directly. Rather than be limited to the format of a job board, companies can list much more information about the role and the company on their own career pages and usually update these listings first before external job boards. These can be a rich, high-frequency source of job information, but it will require web scrapers to be built per company site, adding more complexity.
Natasha Lekh
How are you going to use the job listings data?
Once you have selected your data sources, the next step is to decide on how you intend to use the job ad data and how you need your web scraping tool to work for you. Consider the following questions when designing your data collection system;
- How often will you need to collect your job data? Is it a one-off project or will you need data collected weekly, daily or even multiple times per day?
- Can you collect all the information you need from job postings themselves, or will you require additional information as well as business contact information?
- How will you store and access the data? Will you store the data in an Excel spreadsheet or does it need to be entered into a database?
- Finally, how much data do you need to collect? Do you require information on a handful of jobs per month, or are you likely to need thousands of records for your project?
Generally speaking, the more data you need to collect, more sites you need to collect from, or other tools that are to be integrated with, the more complex the project will be. This is why businesses are increasingly turning to web scraping platforms to handle this complexity, rather than building these themselves in house. A web scraping platform such as Apify handles all the web scraping architecture, data formatting, and even much of the integration work, leaving you and your team with more time to focus on the rest of the project.
Building a job ad scraping system using the Apify platform
Using the Apify platform to build a web scraping system for job ads couldn’t be simpler for large, popular job boards such as Indeed or Glassdoor using Apify’s pre-built library of Actors. Apify has one of the largest developer communities in the world, who build and maintain web scrapers that you can use for free. Simply create an Apify account and visit Apify Store to copy one of these ready-made Actors into your account.
1. You can find all job-related scrapers in the library of Actors.
2. Let's take a look at the Indeed Scraper. This is what its page looks like. Click on Try me button to proceed.
3. You'll be redirected to the Actor's tab in Apify Console. From there you can configure your web scraper based on your search terms, location, and other job specs. Once you’re happy with your configuration, simply hit run and sit back while the Apify platform extracts the data for you.
4. Here's what running an Indeed scraper in Apify Console looks like.
5. Those are the results you get from scraping Indeed. When the data has been collected, you will have the option to export the data out into a format of your choice such as CSV, Excel or JSON, or connect a database to have Apify deliver the data automatically.
6. The same workflow goes for all the other job scrapers such as the Actor for scraping Glassdoor. Here's what the page for Glassdoor Scraper looks like.
7. This is what working with a Glassdoor scraping Actor looks like.
But what if you’re not able to find a pre-built web scraper for your target website? You have two options, firstly, you can order a custom web scraper to be built for you directly by requesting a custom tool, or you can add your idea to the Actor ideas board. Second, if you don’t want to wait for a tool to be developed or you have a development team of your own, the Apify platform is a great tool for building your own, custom web scrapers.
Using the Web Scraper Actor on the Apify Store, you can configure your own scraper for bespoke niche or company-specific job sites, configuring the scraper to collect the information you need, while the Apify platform takes care of the web scraping infrastructure, navigating through the website and exporting the data in a file format of your choice. Even non-developers can learn to build scrapers on the Apify platform using the getting started guide or looking at tutorials.
Mariachiara Faraon
Leveraging job data for a competitive edge
Whether your business is looking to adapt to the rapid changes in the employment market, target potential customers, or make investment decisions, there has never been a better time to start using job market data readily available on modern job sites. Whether you’re looking to extract data from large, national scale job boards, niche sites or individual careers pages, using a web scraping platform will help you cut down the manual data collection, processing and analysis time.
But building a web scraping system that can extract data from multiple sources at scale is complex, requiring you to build extensive IT infrastructure if you were to do this in-house. This is where the Apify platform comes in, providing pre-build web scrapers for popular job boards and enterprise grade infrastructure that can help you scale from hundreds, to hundreds of thousands of jobs.
Try the Apify platform using one of our pre-build web scrapers in Apify Store today for free or request a custom solution to discuss a bespoke system built for you.
Author: Bryce Davies
Bryce is the Head of Growth at SeedLever, a growth consultancy specializing in scaling tech companies. A lover of all things data, Bryce writes about and tests the newest innovations in web scraping and robotic process automation.
Website: seedlever.com
