Hello there, web scraping enthusiast! Our September edition of updates offers you a great read on the legality of web scraping, 2 excellent use cases of web scraping for real estate research, a couple of handy guides on how to scrape Google, Instagram and set up a watchdog, as well as first out of many upcoming technical blogposts.
There are many nuances to the legal side of web scraping and we're here to clear them up. Read our blog post about the legality of web scraping if you ever wondered whether it's on the dodgy side of web activities (spoiler: it's not). You can also expect a video on this topic coming up next month. Stay tuned and keep it legal!
For Apify, MongoDB is a crucial element that can affect both UX and our platform's performance. In early 2021, our users started reporting degraded performance of our UI. The cause? Over-utilized drives in our MongoDB cluster. It was time to take action and improve our overall usage of MongoDB.
Read about some of the techniques and MongoDB Cloud features we've used to debug performance issues and expose sub-optimal queries.
Looking at real estate prices may hurt, but not if you're armed with real data. Check out how one American student used our Zillow Scraper to analyze the real estate market in white picket fence areas of Boston, MA. Or see for yourself how easy it is to compare prices of thousands of cottages all over Czechia, just like these ladies from Czechitas did. This is some pretty impressive research done via data extraction - try it yourself👨🔬👩🔬
That's right, you can scrape the whole Google now. If you're feeling lucky today 😉 here's two tutorials - one on how to scrape Google SERPs with a small trip down memory lane, and one for scraping Google Trending Searches for staying ahead of the curve in whatever your web-related goals are. There's also a collection of SEO-oriented actors that can replace many SEO tools. Enjoy!
- Make good use of the IMDb scraper for anything cinema-related
- Use our GIF Scroll Animation actor for testing UI or showcasing your work
- Check out our new Shopify scraper💥 for keeping an eye on the most precious items you'd like to add to your collection
As you can see, we've started a series of dev-oriented blog posts. Keep an eye on our blog, since we're planning to publish articles on the following subjects:
- How to use Apify from PHP
- Handling IPv4-mapped IPv6 addresses in Node.js
... and many more! We'll keep you posted 🗞️
Minor and major UI improvements ✅
Last but not least, we also rolled out some tweaks to the app UI:
- New Actor UI version has been perfected with the help of your feedback, ready to be relaunched
- List inputs now support large amounts of data (by switching to json editor)
- You can now access your own actors&tasks by referring to them as ~resource-name
- Enabled token editing validation
- You can now see a Python example in the API tab in public actor pages
- We've added API endpoints for actor environment variables
Are you our next amazing teammate? 👨💻 👩💻 You might be exactly who we're looking for:
We kicked off this month by making #HackerCamp happen. If you haven't heard of it yet - perhaps you will next time, as it was our first year this September. You can find some snapshots of this epic getaway on our LinkedIn, but be warned: they don't reflect the whole vibe. Do join us in 2022 😉🌲 🏕️ The only thing running faster than our scrapers this month was our amazing #VltavaRun team, who spent 32h running in total, and took 71st place out of 272 teams. Look at them go! 🏃🏃♀️🏃♂️🏃♀️