"Can I ask ChatGPT to do it?"
These days, the most common question among those considering data collection. It's a valid question. If you ask ChatGPT or Claude to "write a Python code to collect product names and prices from this site," you'll get a fairly plausible crawler code within minutes. It used to be something that required hiring a developer a few years ago.
So, this article is not saying "AI can't do it." There is a clear scope of what can be done with AI, and if it falls within that scope, doing it yourself is the right approach. However, when you actually operate it, there are certain points where you may encounter obstacles, so knowing how far AI can go and where other issues start can significantly reduce trial and error.
Let's start with what can be done
For simple structured sites, collecting a small amount of data once, an AI-generated crawler is sufficient. You can work on fixing it while conversing with AI without knowing the code, and it practically costs nothing. In fact, there is a blog post on our blog introducing how to create a crawling bot using ChatGPT.
Here's what AI does well:
- Writing crawler code itself — finding selectors, parsing logic, and even saving to Excel
- Providing a code fix suggestion when given an error message
- Code for organizing and processing collected data
Up to this point, things have certainly become easier in the AI era. The issue is that in crawling, code is only half the battle.
Five points where you may encounter obstacles
1. Blocking is not a code issue but an infrastructure issue
The code generated by AI mostly uses basic request methods. While it may work on small sites, it quickly gets blocked on sites like large e-commerce, portals, and social media platforms with strong bot detection. Your IP gets blocked, captchas appear, and blank screens are returned.
To overcome this, you need better infrastructure, not just better code. Dynamic IPs (residential proxies), browser fingerprint management, captcha handling — these are areas that cannot be solved through code generation alone; they require investment and operational efforts. Even if you ask AI to "bypass the block," it cannot be executed without the necessary infrastructure.
2. Silent breakdowns — AI doesn't have the ability to detect failures
Site screen structures change unexpectedly, causing the crawler to stop or start collecting empty values at that moment. While AI can fix it when asked, it does not proactively detect that something has broken.
The most frightening aspect in operations is this point. Discovering two weeks of empty data during monthly analysis — data from the past period cannot be recovered. Automated collection requires monitoring systems such as data volume monitoring, anomaly detection, and alerts, which are outside the scope of the crawler code.
3. In complex structures, humans ultimately debug
In structures like dynamic pages where content loads as you scroll, login sessions, nested frames, the success rate of AI-generated code drops significantly. When you find yourself going back and forth with AI on whether "it works" or "it doesn't," at some point, you'll need someone who can read and understand the code. If you started without coding knowledge, this is the practical barrier.
4. Regular and bulk collection requires a system, not just a script
To reliably collect tens of thousands of data daily, you need mechanisms like scheduling, retry on failure, duplicate removal, and value format validation. A script that runs once and a collection system that runs daily are different entities. While AI can generate code for these components, assembling and operating them still require human intervention.
5. Legal judgment and responsibility cannot be borne by AI
Whether the data being collected is public, free from personal information or copyright issues, and whether there are no issues from terms of service and server load perspectives — AI can provide reference opinions but judgment and responsibility lie with the user. Especially if you plan to use the collected data for business purposes, this aspect needs to be addressed before starting.
Changing perspectives: AI reduced development costs, shifting the value to operations
The conclusion that permeates these five points is clear. What AI has reduced is the cost of crawler development, while the actual cost of crawling increasingly lies in operations. Infrastructure for handling blocks, detection and repair of breakdowns, data quality management — while these were challenging in the past, the gap has become more pronounced now that development has become easier.
That's why we also use AI. If the crawler stops due to site changes, AI first analyzes the cause, generates a fix, and we operate a system where humans review and implement it. Thanks to AI, repairs are faster, but what makes that AI work is the monitoring system, infrastructure, and review processes outside the AI. The difference lies between standalone AI and AI within an operational system.
So, when to do it with AI directly and when to delegate
| Situation | Recommendation |
|---|---|
| Simple site, small amount, one-time | Directly with AI — sufficient and most cost-effective |
| Someone to handle code, target not too challenging | AI assistance + trying to operate directly is worth considering |
| Sites with strong blocking (large e-commerce, SNS, etc.) | Infrastructure is crucial — falls under data collection services |
| Daily/weekly recurring collection, data linked to work | Monitoring and maintenance are crucial — falls under data collection services |
In short: Use AI for one-time code runs, delegate continuous collection to an operational system for a rational approach.
Frequently Asked Questions
Q. As AI advances, won't it eventually handle crawler operations automatically?
The repair speed will continue to increase. We are also utilizing AI in that direction. However, overcoming blocks (proxies, etc.) and having a system to "detect failures" are issues on a different axis than code generation, so as AI improves, the emphasis on this operational foundation tends to increase.
Q. If I get stuck while making with AI, can I delegate at that point?
Yes, you can. In fact, many people do that, and those who have tried it themselves progress faster as the requirements are clearer. Sharing the code you've tried or the points where you got stuck can be helpful for review.
Q. What's different if I delegate after using AI to create?
When it comes to crawler development alone, the difference is diminishing. The difference lies in operations — infrastructure for handling blocks, monitoring and automatic alerts, free repairs on site changes, and data quality verification are included in the service.
Recommended Reads
- Creating a Coupang crawling bot with ChatGPT: Extracting product information from search results
- Why do corporate data teams give up manual crawling?
- How to start web data extraction — comparing 4 methods from manual copying to data collection services
- Subscription-based crawling vs. individual billing — Not comparing the total cost of ownership (TCO) for a year results in losses
Start Now
Have you tried with AI and got stuck? If you provide the target site and where you got stuck, we will diagnose the feasibility and recommend a suitable approach for free.
