PrWeb Crawler

Fake IP when doing web crawling in Go

There are sites that check whether their contents are crawled from applications. That’s when we need to do some tricks to fake our IP addresses, or continually change the IP.

Image for post
Image for post
Photo by Alexander Kaunas on Unsplash

For example we want to crawl from site “endpoint_crawler:80”, and the fake IP address we will be using is “216.58.198.206:80”.

We can change up a little bit for an array of IP addresses use for each call request.

~~Happy coding~~

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store