Company that buys the same competitor data three times
It is common within a company. The marketing team outsources competitor price collection to an external company, the sales team scrapes the same site with an intern using Python, and the product team subscribes to market research SaaS. The three departments are essentially procuring the same data three times without knowing each other.
From the perspective of each department, it is a rational choice. They each found their own way because they needed the data. However, from the perspective of the entire company, it incurs duplicate costs, the quality varies, and most importantly, unmanaged crawlers quietly running become a problem. This article diagnoses this issue from an organizational perspective and discusses how to design governance to consolidate data collection.
Three costs of scattered collection
1. Duplicate procurement costs
When multiple departments separately purchase overlapping data, negotiation power is dispersed, and total spending increases unnecessarily. Each contract may be small individually, but when combined, they often amount to a significant sum.
2. Shadow crawler risk
Scripts created by individuals, tools introduced without review—data collection that the company is unaware of continues. It may not have gone through legal review, and if the person in charge leaves, it becomes a black box that no one can touch. The most dangerous situation is when a problem arises, and the company claims they had no knowledge of such data collection.
3. Unreliable data
If each department has different collection methods, criteria, and frequencies, even though it's the same 'competitor price,' the numbers may differ. If departmental data does not match during a management meeting, discussions based on data turn into arguments of "whose numbers are correct."
Cause: Delegating collection to 'department tasks'
The root of this situation lies in delegating data collection to each department to handle on their own. Internal system data is managed according to data organization standards, but external web data is left unattended outside of that governance. Demand is scattered across multiple departments without a central standardization authority, leading to individualism.
The direction of the solution is clear. Collect demand and standardize supply.
4 steps to design enterprise collection governance
1. Diagnosis — Who is collecting what now
First, identify scattered collections. Create a list of which department is procuring what data, in what way (outsourcing, in-house, SaaS), and at what cost. In most cases, duplicates and shadow crawlers are first revealed at this stage.
2. Integration — Consolidating demand into one channel
Collect departmental demands, differentiate between common and individual demands. Collect overlapping data once to be shared by multiple departments, and add department-specific data on the same standard.
3. Standardization — Unifying the supply method
Establish standards for collection methods, quality criteria, delivery formats, and update cycles. When data comes in according to one standard, departmental numbers match, and sources and cycles are recorded in the catalog for audit purposes.
4. Delegation — One entity takes responsibility for operations
Operate standardized collection internally by the data organization or outsource it to manage it through one channel. The key is to make it a "collection that the company understands and manages." Shadow crawlers disappear, and responsibilities become clear even when issues arise.
Benefits of integration: Managing costs and risks simultaneously
Consolidating collection into one channel resolves three issues at once. Duplicate procurement costs decrease, untracked collections come under management reducing legal and compliance risks, and data coming in under one standard ensures departmental numbers match.
Utilizing outsourcing speeds up integration. Combining different methods from each department internally may take time, but adding multiple department demands on a standardized supply pipeline naturally centralizes the channel. Hashscraper collects, refines, and delivers multiple targets and items as one subscription, allowing for delivery in different formats (Excel, API, DB, dashboard) by department, making it suitable for the role of an enterprise collection channel. Like cases where regulatory agencies operate large-scale online monitoring through one pipeline, large-scale multi-department demands can also be consolidated into one channel.
Frequently Asked Questions
Q. Can various departmental requirements be covered by one channel even if they are different?
Even if the target sites and items are different, the supply structure of collection, refinement, and delivery is common. Therefore, departmental requirements can be layered on one pipeline. Different delivery formats (one in Excel, one in API) can also be accommodated.
Q. Should we switch everything at once even though each department has existing contracts?
No. Typically, duplicates are removed during diagnosis, and integration is done sequentially according to contract renewal timing. Just by collecting overlapping data, costs and risks are reduced.
Q. Will integration make us dependent on a specific vendor?
If you secure data ownership and standard delivery formats (transferable formats such as Excel, API) in the contract, the dependency risk decreases. The goal of governance is to establish 'standards managed by the company,' not a specific vendor.
Recommended Readings
- Buying External Data, Building In-House, or Outsourcing — Purchase, Build, Outsource Decision Framework
- Crawling Subscription vs. Individual Billing — You'll incur losses if you don't compare the Total Cost of Ownership (TCO) for a year
- From Crawled Data to Decision Making
- Why do large companies give up crawling data directly?
Start Now
Understanding the scattered collection in each department is the first step. If you let us know what data you are currently procuring and how, we will diagnose potential savings and standardization options during integration.
