Content
What is fintech?
Fintech, or financial technology, is a relatively new industry that uses technology to improve financial services or activities. The industry includes established financial companies and banks using technology such as internet and mobile banking to modernize customer services, but also startups that can dramatically alter the way things work – think Robinhood, Stripe, and Revolut.
COVID-19 has had its own effect on fintech: the lockdowns around the world have pushed more customers than ever to rely on digital and especially mobile banking. And as traditional banks wake up to the technological gap between them and early fintech companies, both competition and cooperation are likely to intensify.
Artificial intelligence (AI), big data, and robotic process automation (RPA) are all some of the many technologies associated with fintech. Even cryptocurrencies like Bitcoin and Ethereum are part of the fintech revolution, but not just because of their volatile spikes in prices. The decentralized blockchain technology has the potential to make wealth truly digital and change how it will be transferred.
But what other technologies might be able to benefit or revolutionize fintech? Is there a role for web scraping in fintech?
What is web scraping?
Web scraping is the process of automatically extracting data and content from websites. Websites were designed to be used by humans, but a web scraper can take the information intended for people and make it readable by computers. That data can then be downloaded or exported and used for any purpose completely separate from the original website.
You can also watch our YouTube video on web scraping and its various use cases.
Is web scraping legal?
Web scraping is legal, but you should be aware that your results may contain personal or copyrighted data. It’s therefore essential to comply with the regulations set by the EU or the US and only scrape data that is publicly available. To learn more about these regulations and the ethics of web scraping, check out our legality blog post.
How web scraping can be used in fintech
- Market sentiment
- Compliance
- Risk mitigation
- Monitoring ratings changes
- Investing wisely
- Understanding customers
Financial decisions rely on good data. If you don’t have enough up-to-date information on a sector or a company, you risk making bad decisions. Those decisions need not be made by humans, but even AI needs data. So what are some of the ways that web scraping can help businesses?
Market sentiment
The stock market is not entirely based on fundamentals. Sometimes stocks do surprising things, as the recent Reddit-fueled GameStop surges have shown. While it might have been challenging for any investor or hedge fund to be prescient enough to monitor Reddit for discussions related to GameStop, that kind of monitoring now seems prudent (just recently, Brooklyn ImmunoTherapeutics (BTX) surged, again probably because of Reddit).
Web scraping is ideal for watching websites for the shifts and tides in attitude that may herald erratic stock market behavior. The same goes for keeping an eye on social media to see what brands are being plugged by influencers. Traditional indicators, such as moving averages or the high-low index can also be efficiently and continuously monitored by web scraping. Plenty of websites offer these kinds of trackers, but any banking institution or fintech that wants to maintain its edge has good reason to roll their own and customize exactly what data they get and when they get it.
Apify tools to monitor sentiment:
- An obvious choice here is to keep an eye on Reddit. We have Reddit Scraper for that.
- But it might also be a good idea to use our Instagram Scraper and Facebook Scraper to watch for potential wider waves of social media sentiment.
Or order a complete custom solution for your use case: Apify for Enterprise.
Compliance
At one time, fintech companies existed in a sort of internet Wild West, where they were, if not beyond the law, at least ignored by it. This is no longer the case, with fintech companies already being hit by fines of hundreds of thousands of dollars for not being compliant with financial regulations.
Luckily, governments and regulators are usually meticulous about publishing the rules online, which makes them available to web scraping tools. Apify recently helped a company to download 740,000 PDFs from over 100 government websites in less than 14 hours. While that was aimed at identifying economic and financial incentives for forest and landscape restoration in Latin America, it illustrates that government websites, designed to be easily accessible and public, can be web scraped. There’s no need to be caught out by changes in the rules if you have tireless bots keeping an eye on updates and sending you automatic notifications.
Recommended Apify tool to keep an eye on regulations and requirements:
- Google is already indexing lots of official documents, so you can use our Google Search Results Scraper to find and collect the information you need. Gather the data and feed it into spreadsheets or databases to keep ahead of changes in the rules.
Risk mitigation
Events in the real world can have devastating effects on companies and industries. Investors, insurance companies, and banks need to know when there are impending emergencies, such as freak weather or flooding, that could unexpectedly impact value. Web scraping can provide alerts on the first suggestion of unusual weather patterns from meteorological agencies or local news reports.
Recommended Apify tool designed to monitor weather:
- Weather can destroy assets and impact morale. Our Weather Scraper can be used to give you advance warning of unusual weather patterns based on specific criteria and localization.
Monitoring ratings changes
Standard & Poor’s (S&P), Moody’s and Fitch are the three biggest rating agencies that regularly evaluate the credit worthiness of companies and governments. It may seem a small change, but if Moody’s degrades a country by a single notch and it goes from ‘A3’ to ‘Baa3’, that will have a knock-on effect across the markets. It stands to reason that it would be a good idea to track these rating changes as they happen, without any delays in waiting for analysts to digest the information. Even better, there are other indices that appear to correlate with rating changes, such as the Corruption Perceptions Index. Gather enough information and you might find your tools predicting what Fitch is going to do. Even a margin of minutes could make all the difference.
Recommended Apify tool to watch rating agencies and indices:
- Keeping an eye on ratings changes can be as straightforward as just watching a single value on a single web page. Our Content Checker does this with no frills and just sends a screenshot and the value to you by email when the value changes. Set it to check as often as you like and feed the result into your own database.
Investing wisely
Due diligence is one side of checking out a company for potential investment, but online research can also provide useful information on both the company’s news flow and the activities of its founders or staff. Social media profiles used to be considered hands-off for companies, but the rise of cancel culture means that increased attention is being paid to unacceptable online behavior and ignoring warning signs could be costly. Most investors would like to know where a company stands on touchy subjects, or even whether there are rumors of risky attitudes.
Recommended Apify tool to track news flow and public opinion:
- Twitter has become one of the platforms that brands need to watch for unexpected surges in public opinion. Use our Twitter Scraper to track tweets, replies, favorites, and threads without running up against Twitter’s limits on downloading data.
Arbitrage and algo-trading
Traders engaging in arbitrage need to move fast, so fast that they have to rely on automated trading systems. Arbitrage is the buying of a financial instrument on one market and the selling of the same financial instrument on another market at a higher price. While that may sound like any trade, the difference is that arbitrage is risk-free, because it happens in the split millisecond (or less) before the second market has updated the price to reflect current market conditions. As you can imagine, arbitrage might once have been possible without computers, but now it is dominated by them. Algorithmic trading, or algo-trading, is another form of trading that relies on speed to carry out trades. Both arbitrage and algo-trading depend on complex, often custom-built software that sometimes does not even operate transparently to its users. Web scraping can’t provide these machines with price data at the required speeds, but it can be used to prepare them for probable trades, by gathering data ahead of any opportunities. Sometimes that kind of information is only available on sites that can’t be accessed or easily understood by machines. Web scraping can be used as a component of these systems, to give them an extra edge.
Recommended Apify tool to watch crypto activity:
- Crypto is still the new frontier and you can always make money on new frontiers. Use our new Bitcoin/Ethereum Address Watcher to track transactions in real time to specific addresses and wallets.
Lead generation
Just like any technology company, fintech companies need to market their solutions. While inbound and content marketing are going to be cornerstones of any fintech marketer, tracking down leads, especially when some of those leads might become huge, well-paying clients, is an approach that isn’t going away. Reaching out to potential leads means that you need to know how to contact them and web scraping is a very efficient way to gather contact information. Rather than manually wading through websites to work out who your sales team should call, you can use a scraper to regularly crawl through hundreds of sites and extract the relevant details. If the person or any contact details change, your scraper will catch it and automatically flow the data right into your CRM system. Phone numbers, email addresses, social media profiles - anything relevant to keeping in touch with your customers can be scraped and stored.
Recommended Apify tool to gather contact details and generate leads:
- Our Contact Details Scraper is a good place to start scraping email addresses, phone numbers, and social media profiles from any website. You can scrape as many websites as you like, using simple lists or even feeding the URLs in from a spreadsheet.
- Google Maps Email Extractor is built for getting contact data from Google Maps. This tool can extract contact details from any Google Maps business listings helping you find potential leads in specific areas.
- Facebook Pages Scraper can be used to extract contact details from the official Facebook Pages and expand your lead database with social media data.
Natasha Lekh
Understanding customers
Scraping personal data is a minefield, especially after the European GDPR came into force in 2018. Recent EU regulations also restrict the kind of data fintechs can get through what is usually called screen scraping. Internet users might not mind sharing their personal details with companies like Facebook or Instagram, but they can get a little squeamish about the idea of other companies tracking them or keeping information about their habits in a database.
But fintechs thrive on serving customers better than old-fashioned financial institutions. While access to detailed financial data might be something to be extra careful about, fintechs don’t have to make web scraping personal. Fintechs are newcomers to the financial scene so they need to keep up with how their brands and industry are seen across the web. Web scraping social media to see what customers are worried about, interested in, and how they talk about financial services is a non-invasive, ethical use of web scraping that pays dividends without the risk of negative press.
David Barton
Recommended Apify tools to enhance customer experience:
- Apify has an ever-expanding suite of social media scrapers you can use in parallel to track your brand and industry. You can scrape Instagram, Facebook, Twitter, YouTube, Yelp, Reddit and many more.
- To see what people are searching for, you can use our Google Search Scraper or Google Trends Scraper.
Putting the tech in fintech: robotic process automation will be the next in our series of posts about how technology is transforming the world of finance. RPA is the really big thing in fintech, so don’t miss it. Subscribe to the blog, follow us on Twitter or just sign up for a free Apify account to automatically receive our regular newsletter, including links to articles and tips from the world of web scraping and RPA.
If you couldn’t find a ready-made Apify tool for what you need and we don’t cover something you need to scrape, you can request a custom solution. We partner with global companies of all sizes, so you can rely on us to complete and maintain your project, no matter the scale. Just email hello@apify.com and tell us what you need.
