PrWeb Crawler

Fake IP when doing web crawling in Go

There are sites that check whether their contents are crawled from applications. That’s when we need to do some tricks to fake our IP addresses, or continually change the IP.

Photo by Alexander Kaunas on Unsplash

For example we want to crawl from site “endpoint_crawler:80”, and the fake IP address we will be using is “216.58.198.206:80”.

We can change up a little bit for an array of IP addresses use for each call request.

~~Happy coding~~