"I tried it myself, but it didn't work."
This is the most common phrase I hear in consultations. People have tried free tools, asked colleagues who know a bit of coding, and even had AI write code for them these days — but they all end up getting stuck at some point.
What's interesting is that the content of that "it didn't work" is similar for everyone. The points where web scraping projects collapse are generally predetermined, and they even come in a specific order. One at the beginning, another two weeks later, one when trying to analyze the data, and two more a few months later. This post shows you those five points in advance. If you know where you might get stuck before you start, at least you won't stumble at the same spot.
Reason 1: Not verifying the feasibility of data collection before starting
The first failure comes right at the beginning. Choosing tools without checking what the target site is like — how strong the blocking is, if login is required, if the desired items are actually on the screen.
Methods that worked well on small sites don't work on large e-commerce, portals, or social media sites. The defenses against automated access are tight, so the tools or basic code generated by AI either receive blank screens or get blocked quickly. Assuming "web scraping works," setting a schedule and budget, only to realize later that it doesn't work.
Prevention: Regardless of the method used, test a small amount on the target site before the actual work. We also conduct feasibility assessments (accessibility, item presence, structural stability) before providing an estimate, and we don't take on projects that skip this step. This is where half of the projects diverge.
Reason 2: Assuming "once it's built, it's done"
The second failure comes about two weeks later. Initially, everything runs smoothly. But the moment the site changes its layout, the web scraper either stops or, worse, quietly starts collecting empty values. And no one knows — because no mechanism was set up to alert that it's broken.
A month later, when you open the data, you discover two weeks' worth of missing data. Since you can't recollect the screens from the past, that gap becomes permanent.
Prevention: You need to plan monitoring along with the web scraper. Is there an alert when the amount collected is different from usual? Does it automatically retry if it fails? We set automatic retries as the default for collection failures, and we make sure that the system, not the web scraper, detects anomalies exceeding the expected amount. The lesson at this point is that web scraping is not about building but running.
Reason 3: Nobody verifies the data quality
The third failure occurs when you open the data for analysis. The collection itself was successful, but you find that the same product appears three times (duplicates), some pages are missing (omissions), and the price column contains both "12,900 won" and "12900" (format inconsistency). Judging that everything is going well based solely on the number of items collected is the cause.
Prevention: Always inspect the initial collection results — visually check for duplicates, missing essential items, and inconsistent value formats. We inspect samples with clients during onboarding to finalize fields and formats, deliver data that has undergone deduplication and format cleanup. Data ready for analysis is the completion of the collection.
Reason 4: Unable to continue maintenance
The fourth failure comes about four months later. Even if you're able to detect malfunctions with a monitoring system, you may encounter obstacles on the repair side this time. The target site doesn't change just once or twice but continuously. While the responsible person may fix it the first couple of times, maintaining the web scraper isn't anyone's main job, so it starts to get postponed. If there are multiple target sites, the repair workload multiplies.
As the responsible person gets busier or changes positions, the person who can fix it disappears, and the web scraper gradually transitions to a state of being "left broken." If you've been unable to fix a stalled web scraper for two months, the project is essentially over.
It's not just maintenance. Monitoring is not continuous either. Initially, someone may check the collection status daily, but after a few weeks, the person in charge may disappear, and even if alerts are set up, the recipient may change when the responsible person changes. Both maintenance and monitoring are tasks that need to be done continuously, and the first thing to collapse in internal operations is this continuity.
Prevention: Before starting, determine "who will monitor and how often they will fix." Site changes are not exceptions but constants. If it's difficult to handle internally, it's best to start with a structure that includes maintenance — we include site change responses in a monthly subscription to prevent situations where repairs are delayed. We're reducing the repair speed by having AI analyze the cause first, create a fix, and have a person review and implement it.
Reason 5: Calculating costs only as development expenses
The fifth failure comes from the budget. Initially, only the cost of building the web scraper is considered — but in reality, monthly server and proxy operation costs accrue, repair costs arise each time the site changes, and development costs reappear when expanding the collection targets. If billing repeatedly exceeds the confirmed budget, the project starts to be called a "money-eating pit" regardless of its performance, and discussions about expansion come to a halt.
Prevention: Calculate the total cost of ownership (TCO) for one year before starting. Include development costs, operating costs, maintenance costs, additional development costs, and compare two methods (pay-per-use vs. monthly subscription) to see which structure suits our situation. The TCO article in the "Recommended Readings" section below summarizes the calculation framework.
Commonalities of the five reasons
To summarize:
| Stage | Failure | Cause |
|---|---|---|
| At the beginning | Collection doesn't work | Lack of feasibility verification |
| Two weeks later | Gaps in data | Lack of monitoring system |
| During analysis | Distrust in data | Lack of quality inspection |
| Four months later | Left broken | Inability to sustain maintenance and monitoring |
| Budgeting | Cost explosion | Failure to calculate operating costs |
Do you see the commonalities? All five failures occur because people view web scraping as "writing code once." In reality, it involves verifying, monitoring, inspecting, continuously fixing, and managing operating costs — ongoing operations. The real reason behind most cases of "I tried it myself, but it didn't work" is the weight of these operations, not lack of technical skills.
Frequently Asked Questions
Q. If I got stuck while doing it myself, do I have to start over from the beginning?
No. In fact, those who have tried it themselves progress faster because the target and items are clear. If you tell us where you got stuck, we'll review from that point.
Q. Which of the five reasons is the most common?
At the beginning, it's reason 1 (lack of feasibility verification), and during operation, it's reason 2 (lack of monitoring). Especially reason 2 often goes unnoticed and causes significant damage.
Q. Will handing it over solve all five reasons?
Yes, the five reasons are precisely what a data collection service does — feasibility assessment (1), monitoring and automatic retries (2), sample inspection and refinement (3), ongoing maintenance and monitoring (4), a predictable monthly subscription structure (5). However, how the collected data will be used for decision-making is the client's responsibility, so we first inquire about the purpose in consultations and narrow down the collection scope together.
Recommended Readings
- Building web scrapers with AI: what works and where it gets stuck
- Outsourcing web scraping process — from inquiry to the first data, organized step by step
- Web scraping subscription vs. pay-per-use — you'll incur losses without comparing the total cost of ownership (TCO)
- Why do large companies give up on web scraping themselves?
Start Now
Where are you currently stuck? If you let us know the target site and the issue you're facing, we'll diagnose for free from the perspective of the five reasons.
