Crawlee won't fix broken selectors for you (yet), but it helps you build and maintain your crawlers faster. When a website adds JavaScript rendering, you don't have to rewrite everything, only switch to one of the browser crawlers.

The browser fires an OPTIONS request to https://albertsons.okta.com/api/v1/authn. Since we’re doing this programmatically, we don’t need to worry about this (as it’s for cross origin request safety, a browser safety feature).

Using this, in conjunction with AWS Step Functions, Lambda, and ECS, it became merely cents a month to run a headless scraper task in the cloud. What does your workflow look like?

Puppeteer is official tool for Chrome Headless by Google Chrome team. Since the official announcement of Chrome Headless, many of the industry standard libraries for automated testing have been discontinued by their maintainers. Including PhantomJS.

The full code for the completed scraper can be found in the companion repository on github. I wouldn’t really consider web scraping one of my hobbies or anything but I guess I sort of do a lot of it.