0. Web Crawling Blockage Occurs, What Is the Cause?
Developers who have experienced web crawling have inevitably experienced blockages.
Your crawler was perfect, but were you frustrated because you didn't know where the problem occurred?
In this post, we will discuss the typical causes of blockages with a focus on solutions.
Crawling is similar to going into a store and fetching the desired products. When we enter a store, there are implicit rules to follow. For example, removing the dirt from your shoes before entering, properly folding and placing umbrellas in the stand, and dressing in a way that does not inconvenience others.
In some places, there are rules specific to the owner. In such cases, you must strictly adhere to those rules to use the store. Just like cafes that are trending these days, where you must "inquire via DM," or in restaurants where you must return the tray yourself, or leaving your seat without clearing it.
Similarly, there are rules that must be followed in crawling. One typical case is when requests lack User-Agent or other parameters or if they are set strangely, causing the website to block the request, assuming it is from a bot.
Therefore, the most basic step is to set the User-Agent so that it does not appear as a bot.
1. Let's Set the User-Agent
Setting the User-Agent involves putting the User-Agent value in the HTTP request header. The User-Agent value is a string that represents the type and version of the web browser or HTTP client.
For example, the Chrome browser uses the following User-Agent value.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3
By setting the User-Agent value in the crawler, the website will recognize the request as coming from a browser. Therefore, if the User-Agent value is not set, there is a higher possibility that the website will block the request, assuming it is from a crawler.
The method of setting the User-Agent value varies depending on the HTTP request library, but generally, it is possible to include the User-Agent value in the HTTP request header. In Python, when using the requests library, you can set the User-Agent value as follows.
import requests
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
}
response = requests.get("http://example.com", headers=headers)
In the requests library, you can set the User-Agent value in the HTTP request header using the headers parameter.
However, nowadays, websites are increasingly attempting to block crawling using not only User-Agent but also various other methods. Therefore, it is advisable to use other methods in addition to setting the User-Agent. For example, frequently changing IP addresses or adjusting the crawling speed.
2. Change IP Addresses Frequently
Continuously requesting connections with the same IP address may appear very suspicious to the website.
One simple way to frequently change IP addresses is to use a VPN. VPN stands for Virtual Private Network, a service that hides the user's IP address and changes it to an IP address from a different region through an internet connection.
By using a VPN, the request is perceived as connecting to the internet through the VPN, reducing the likelihood of being blocked. Additionally, using a VPN allows you to avoid using IP addresses that have been previously blocked, enabling you to perform crawling more reliably.
3. Irregularly Adjust the Crawling Speed
Requesting with precise timing and intervals like a machine can increase the likelihood of being recognized as a bot. Therefore, it is necessary to configure settings that irregularly adjust the crawling speed.
There are two main methods to adjust the crawling speed.
The first method is to adjust the interval between crawling requests. Sending crawling requests at short intervals may increase the likelihood of the server recognizing and blocking them. Therefore, it is advisable to send crawling requests at regular intervals.
The second method is to use various IP addresses. If you continue to send crawling requests from the same IP address, the server may recognize and block them. Therefore, it is recommended to send crawling requests using multiple IP addresses. You can use proxy servers for this purpose.
4. Set Parameters Accurately
Earlier, we set the header when setting the User-Agent.
When crawling, problems can arise if there are parameters that should be set in the HTTP request header or if there are parameters that should not be included.
For example, if the User-Agent value is not set, some websites may block the request, assuming it is not sent from a browser. Also, if you set and send cookie values that should not be included in the HTTP request header, the website may recognize and block them.
Therefore, when crawling, it is essential to carefully check and adjust what parameters should be set in the HTTP request header and if there are any parameters that should not be included.
5. Proper Exception Handling
Continuously sending the same request during crawling when an error occurs can increase the likelihood of being recognized as a bot and blocked by the website. Therefore, proper exception handling is necessary.
Exception handling refers to handling unexpected situations that occur during program execution.
For example, if the server does not respond when sending an HTTP request, the request fails. In this case, the program should recognize this and handle it by waiting for a certain period before sending the request again.
Moreover, if you send crawling requests that violate the website's rules, the website will block them. In this case, the program should recognize this through exception handling and stop the request. Therefore, when sending an HTTP request, it is essential to check the status code returned by the website and handle exceptions accordingly.
Exception handling is a crucial part of crawling. By implementing proper exception handling, you can perform crawling reliably.
Conclusion
To perform web crawling reliably, it is essential to avoid website blockages by using various methods such as setting the User-Agent, changing IP addresses, irregularly adjusting crawling speed, setting parameters accurately, and implementing proper exception handling.
If you experience blockages, check if the above five principles are well reflected to achieve good results.
