Fake IP when doing web crawling in Go
There are sites that check whether their contents are crawled from applications. That’s when we need to do some tricks to fake our IP addresses, or continually change the IP.
For example we want to crawl from site “endpoint_crawler:80”, and the fake IP address we will be using is “220.127.116.11:80”.
We can change up a little bit for an array of IP addresses use for each call request.