The Pipeline That Was Fine Until Yesterday Is Empty This Morning
This rarely happens with internal sources. Changing a schema requires alignment, removing a column comes with advance notice, and when something goes wrong, there is an accountable owner. But sources coming from external websites—such as competitor prices, reviews, and market data—are different.
One day, a target site rebuilds its review section in React, moves where prices are displayed, or hides items that were previously visible only after logging in. Without notice. At that moment, parsing that worked perfectly until yesterday starts returning empty values, dbt models pass them through as-is, and yesterday's data remains as a hole in the dashboard. Business teams ask, "Why doesn't the data match?" and the data team fixes the parser at dawn.
This article explains why this situation keeps recurring and how to design accountability on the assumption that breakage will happen, from the perspective of a Data Contract.
Why the Web Keeps Breaking in Particular
A data contract is an explicit agreement between a source and its consumers that says, "This data will be provided in this shape, with this meaning, and at this cadence." It is not merely a schema definition; it is also an agreement covering who does what and when when changes occur.
In internal systems, this contract operates as an organizational norm. Source teams notify consumers before changing schemas, provide a deprecation period before removing columns, and have someone within the organization who is responsible for fixing breakages. The key is that the source is a cooperative party. You can communicate with it, coordinate schedules, and hold it accountable.
External web data is difficult because this premise breaks down. Target sites are not parties to our contract. They do not take our pipeline circumstances into account, provide no notification, and cannot be asked to roll back. Moreover, web pages were not originally created as pipeline inputs. They are interfaces designed for people to view, and extracting data means reverse-engineering those interfaces. As a result, the following characteristics destabilize the contract.
- Unannounced redesigns: Marketing renewals and framework replacements can change DOM structures and rendering methods (SSR→CSR) overnight.
- A/B tests and condition-specific pages: The same URL may return a different page on each visit, while displayed items vary by region, login status, membership tier, and device. If the conditions under which the data was viewed are not recorded alongside it, the meaning of values in the same field can quietly change.
- Partial missingness caused by anti-bot measures and blocking: When blocks, CAPTCHAs, or rate limits are triggered, the entire dataset may not disappear—only parts of it do. This creates "reduced data that looks normal," causing downstream systems to silently produce incorrect aggregations.
The common issue is that the assumption of stable schemas, which holds for internal sources, does not hold for the external web. Therefore, an external web data contract should not promise that "nothing will break." It must instead address who is responsible for detection, remediation, and recollection when breakage occurs.
Why a Minor Redesign Becomes an Organization-Wide Problem
A small redesign on a single site propagates downstream. When parsing rules no longer match the actual DOM, fields become missing or values are transformed during collection. If this is not caught in the loading table, it moves into dbt transformations, creating silent NULL propagation and incorrect aggregations, ultimately resulting in dashboard gaps and flawed decisions.
Two factors expand the blast radius. First is silent failure. It is actually better when parsing throws an exception and stops. The dangerous case is when it returns empty strings or NULLs as if they were valid values, allowing downstream systems to pass them through unchanged. The pipeline remains green, but the numbers are wrong. Second is an overly broad field contract. If downstream models and dashboards directly reference every collected field, changing just one site element can destabilize every output that depends on it.
Buffer Design — Preventing Downstream Failures Even When Things Break
The instability of the external web cannot be eliminated. Instead, place a buffer layer in between so that instability does not propagate downstream.
Preserve raw data. Store collected originals—HTML snapshots, raw responses, images before OCR, and so on—separately from parsing results. If parsing is later found to be incorrect, retained originals allow historical data to be restored through reparsing without recollection. If originals are discarded, a parsing bug becomes permanent data loss. Raw preservation is the common foundation for recollection, auditing, and lineage.
Minimize the field contract. Narrow the contract to only the essential fields that downstream systems depend on. Do not include secondary elements that sites frequently change in the contract; retain them only in raw data. Validate the type, required status, and value range of contracted core fields, and prevent violations from flowing downstream. The narrower the contract, the narrower the blast radius of a redesign.
Detect changes. Looking only at whether parsing threw an exception misses silent failures. Therefore, monitor the shape of the results themselves. Check whether record counts have changed sharply from normal levels, whether missing-value rates for core fields have suddenly increased, and whether value distributions are outside normal ranges (for example, a price of 0 or an abnormally large price). If even one condition is triggered, notify the data team channel. The goal is to detect redesigns through changes in the shape of the data before a person discovers them.
This is how we work with data teams on top of this buffer. We define schemas together with the data team from the beginning. We align on which fields will be core contract fields and on each field's type, nullability, and meaning, then make downstream dbt and dashboard systems reference that contract. Even when changes are necessary, we do not unilaterally swap out schemas. We announce changes in advance, provide a deprecation period, and serve old and new versions in parallel during that period so downstream systems have time to migrate before the old version is retired. This allows downstream systems to move within an announced window even when a site changes, rather than having a redesign immediately create a dashboard gap.
Designing Accountability on the Assumption of Breakage
Technical buffering alone is not enough. What truly disrupts operations is a situation where the contract does not specify who will fix something when it breaks. Therefore, a data contract must specify the owners of detection, remediation, and recollection alongside the schema.
| Ownership | What to Specify in the Contract |
|---|---|
| Detection | Who operates anomaly signals, which channel receives notifications, and what the recovery target time is. The detecting party should be the collection operator, not the consumer |
| Remediation | Who is responsible for repairing rules when a parser breaks due to a site redesign. If outsourced, the vendor |
| Recollection(backfill) | Whether there is a standard procedure for filling gaps during the broken period |
Assigning remediation responsibility to the vendor is the core value of outsourcing. It keeps continuous maintenance tasks—such as handling blocks, tracking site changes, and repairing parsers—from falling on the people who operate the data team's DAGs, allowing the data team to focus on loading, modeling, and usage.
Recollection should be treated as a standard procedure, not an exception. Fill gaps by specifying the period and conditions, while assuming idempotent loading so repeated loads using the same key do not accumulate duplicates. Use watermarks and incremental cursors to fill only the missing interval accurately. Along with this, deliver metadata that can immediately be used for post-load validation (DQ), auditing, and lineage. Collection timestamps and collection conditions (region and login status) provide the basis for interpreting the meaning of values; source URLs and collection cadence are used for catalog and lineage registration; and record counts plus missing/anomaly flags feed into automated validation gates. Also leave a completion marker(manifest/_SUCCESS) so downstream systems can determine that loading has fully completed, preventing orchestration from running transformations on partially loaded data. Only then can you answer the question, "Where did this number come from?" during an audit.
In short, an external web data contract is not a promise that "nothing will break." It is a promise of who takes responsibility for detection, remediation, and recollection when something breaks, and by when recovery will be completed. Only when this ownership is written into the contract can external web sources be managed internally like well-governed source systems.
Summary
The fundamental reason external web data keeps breaking is that, unlike internal sources, the source is not a cooperative contract party. Unannounced redesigns, A/B tests, and condition-specific pages undermine the premise of schema stability. Therefore, an external web data contract should not say "it will not break"; it must explicitly define the owners of detection, remediation, and recollection on the assumption that it will.
Hashscraper works with data teams in this way. We define schemas together, handle changes through advance notification and parallel versions, process recollection through standard procedures based on idempotent loading and watermarks, and provide metadata and completion markers that can immediately be used for catalogs, lineage, and DQ. Development, maintenance, block handling, and collection monitoring are included in a monthly outsourcing fee, so DAG operations staff are not worn down by maintaining external collection systems. At scale as well, we have experience operating large-scale online monitoring for a regulatory agency as an outsourced regular pipeline.
Outsource external web collection and maintenance so it can be handled like a well-managed source system, while the data team focuses on loading, modeling, and usage — this is the practical path to reliably bringing external data into dashboards in an era of constant site redesigns.
Related Articles
- From Crawled Data to Decision-Making
- Data Received in Excel vs Data Viewed in a Dashboard — What Delivery Format Changes
- Companies Where Each Department Crawls Separately — Enterprise Data Collection Governance to End Duplicate Investment
Get Started Now
We will review what contract your external web sources connected to current pipelines need so that a single redesign does not empty your dashboards. From schema definition to change notifications, recollection ownership, and delivery specifications, we design it to fit your data team's pipeline.
