Crawler

Using Playwright to crawl data

Simple example for using Playwright to crawl data from websites.

Photo by Lucas Ludwig on Unsplash

In software world today, there’s a lot of existing data in the web that we could benefit from. So crawling data is often a necessary step in software development, especially for data driven applications. In this article, we will go over for how Playwright can help us to do this task. We will try to crawl data from AWS application which requires authentication step.

Playwright has a feature that can reuse cookies data, we can setup states for our application beforehand and reuse it later. This can save us a lot of time from set things up.

Then reuse it

Crawl data from site by accessing to its elements and save data to csv file

Run the test with playwright cli

npx playwright test tests/crawler.spec.ts

Hope this helps.

Happy coding guys.

Cheers!