"I want to receive a daily summary at 8 AM containing only competitors' new products added yesterday and products that went out of stock."
That is the entire requirement in one sentence. But when you request a quote, the conversation keeps drifting toward, "Can you collect data from that site?"
Collection is not the hard part of this request.
Competitor product lists are already all visible on the screen. Open the page, and you can see them.
There is only one thing you cannot see: which of them did not exist yesterday.
Prices are written on today's screen. New products and out-of-stock items exist only if you have yesterday's list.
And this difference directly translates into revenue. If you learn about a competitor's new product even one day earlier, you gain response lead time. If you catch the moment a competitor's core product goes out of stock, you can pursue the spillover benefit of that demand shifting to you. Conversely, if you miss the restocking of a product competing with your bestseller, that window quietly closes. New product and out-of-stock monitoring is not about "understanding the current situation"—it is about buying response time.
3-Line Summary (TL;DR)
- New products and out-of-stock items are not values read from a page; they are the difference (diff) between yesterday's list and today's list. Therefore, the first design target is not the crawler but a baseline snapshot recorded daily using consistent criteria.
- The second challenge is that out-of-stock labels are not standardized. Out of stock, temporarily out of stock, restock notifications, and option-level stockouts are expressed differently by each site. Without defining what counts as out of stock, you accumulate figures measured with a different ruler every day.
- "Every morning" is not the collection time; it is the delivery time. If you want delivery at 8 AM, you must work backward to account for validation and retry time. If you send everything without thresholds, no one will open it from the third week onward.
Table of Contents
- What New Product Monitoring and Out-of-Stock Monitoring Are
- If You Discard Yesterday's List, You Will Never See Today's New Products
- Out of Stock Is Expressed Differently on Every Site
- Every Morning Is Not the Collection Time but the Delivery Time
- No One Looks at a Report With 200 Items a Day After the Third Week
- Comparison by Approach: Manual Checks vs. No-Code Tools vs. Managed Services
- 5 Questions for Self-Assessment Before You Start
- 5 Implementation Steps
- Frequently Asked Questions
- Conclusion
What New Product Monitoring and Out-of-Stock Monitoring Are
New product monitoring is the practice of collecting product lists from competitors and sales channels at set intervals, then automatically identifying products that newly appear but did not exist at the reference point in time.
Out-of-stock monitoring is the practice of periodically collecting the availability status of the same products and detecting the moment their status changes, such as available → out of stock or out of stock → restocked.
Both definitions contain the same words: did not exist, and changed.
This is where they diverge from price monitoring.
For prices, one page today gives you the answer. The number is displayed on the screen.
New products and out-of-stock items cannot be identified from today's page alone. You need yesterday's data for comparison.
That is why this work is closer to comparison than collection. The design criteria for price monitoring are separately summarized in How to Choose a Data Collection Service for E-Commerce Price Comparison and Monitoring.
If You Discard Yesterday's List, You Will Never See Today's New Products
A baseline snapshot is data that stores an entire product list and its statuses at a specific point in time, serving as the reference for comparison with the next cycle.
For comparison to work, three things must remain fixed.
- Identifier — What will be considered the same product? (Product ID, URL, product name + option)
- Scope — Is yesterday's collection scope the same as today's?
- Time — Is it captured at the same time every day?
The second point is where most issues arise.
If you collected three list pages yesterday and five pages today, the report will show dozens of new products—even though none of them are actually new.
An even more difficult issue remains. A product seen for the first time is not necessarily a new product.
There are at least four reasons why a product may appear in a list for the first time.
- Genuinely new product — A newly launched product
- Renewal re-listing — A product page newly created because the size or packaging changed
- Option added — An existing product that appears separate after colors or sizes are added
- Restock return — A product that disappeared from the list when sold out and later returned
The fourth is an especially subtle false positive. Many sites remove sold-out products from their lists entirely, causing restocks to be identified as new products every time.
The distinction is made not by technology, but by rules. Was a new product ID issued? Should products whose name similarity exceeds a threshold be classified as renewal candidates? If a product was in the list within the past 30 days, should it be considered a reappearance?
Having supported data collection for more than 500 companies, the first failure in new product reports was usually not blocking. It was the absence of yesterday's saved list.
Who writes these rules, and who maintains them even when the site is redesigned? This project is essentially about that one question.
Out of Stock Is Expressed Differently on Every Site
Out of stock is not a single value either. Each site displays the same condition—"you cannot buy it right now"—differently.
- Out of stock / temporarily out of stock / restock expected / sales discontinued
- A state where the purchase button disappears and only "Apply for Restock Notification" remains
- The button remains, but every option is unavailable when opened
- Only 3 out of 12 options are out of stock (the product itself is still for sale)
- A state where the product disappears entirely from search results
One operational decision is required here.
"If 3 out of 12 options are out of stock, is the product out of stock or not?"
If the team cannot answer this question in one sentence, the number of out-of-stock items arriving every day is a figure counted using different criteria each day.
A practical approach is to record it in two layers.
- Product level — Can it be purchased right now? (Available / unavailable)
- Option level — How many of the total options are unavailable?
Also retain the original label displayed on the screen. This lets you reclassify historical data retrospectively even when a site changes its wording.
Some sites expose inventory quantities, such as "3 left." Unlike new products and stockouts, this number is a 'value' that can be read directly from today's page, but it is worth recording as a leading signal of stockouts. If you detect the moment inventory drops below a threshold, you can respond not after the item is sold out, but before it happens. However, inventory quantity displays also vary by site in accuracy and presentation, so it is safest to use them only as supplementary signals on sites where they are available, and judge other sites solely by available/unavailable status.
The last item—disappearing from search results—must be handled separately. Is it out of stock, discontinued, or has our collection scope changed? All three look identical on screen.
Out of stock is not a status; it is a definition. Without documenting the definition, you accumulate figures measured with a different ruler every day.
Every Morning Is Not the Collection Time but the Delivery Time
The requirement is an "8 AM email," but design usually stops at "running the crawler at dawn." The order needs to be reversed.
Count backward from the delivery time. It varies by target volume and site difficulty, but the framework looks like this.
| Time | Task |
|---|---|
| 08:00 | Report arrives in the recipient's email inbox or Slack (this is the requirement) |
| 07:40 | Generate and send the report |
| 07:00 | Validate data consistency + compare against yesterday's snapshot |
| 05:30 | Start collection — including time for retries of failures |
| Previous day 05:30 | Baseline snapshot used for comparison |
Two slots are frequently left empty: validation and retries.
If you create the schedule assuming collection succeeds on the first attempt, the report will be empty or late on days when failures occur. A pipeline with no buffer is quiet on the day it fails.
Maintaining the same snapshot time every day is important for the same reason. If yesterday's snapshot was taken at 3 AM and today's at 9 AM, products added in between may be detected across two days or missed entirely.
The hard part is not "morning," but "every day." Building it once takes a day; ensuring it arrives every day at the same time under the same criteria is operations.
No One Looks at a Report With 200 Items a Day After the Third Week
If you monitor all categories across three competitors, you can get hundreds of new product and stockout changes per day.
In the first week, everyone reads everything. In the second week, they skim it. In the third week, they do not open it.
That is why thresholds account for half of report design.
- Scope — Start not with all categories, but with categories, price ranges, and brands you actually respond to
- Importance — Pin new products in core categories and stockouts of products competing with your bestsellers at the top
- Grouping — Send one daily summary instead of individual alerts. Top 10 items + the full list as an attachment
- Recipients — New products go to product planning; stockouts go to sales and MD teams. No one should receive alerts that belong to someone else
The purpose of a report is not to show everything. It is to leave only what needs to be seen today.
The design of a loop connecting detection to response is explained in Monitoring Is Not Collection but Notification — Building a Response Loop.
Comparison by Approach: Manual Checks vs. No-Code Tools vs. Managed Services
A managed data collection service is a subscription service in which a provider operates crawler development, blocking response, repairs after site changes, anomaly detection, and on-time delivery on behalf of a company, while the company receives only the resulting reports.
When comparing the three approaches on the same criteria, the differences look like this.
| Category | Daily Manual Checks | No-Code Tool Scheduler | Managed Data Collection Service |
|---|---|---|---|
| Representative tools | Browser bookmarks | Octoparse, Thunderbit, etc. | Hashscraper, etc. |
| New product identification (difference from yesterday) | Memory and visual estimation | Results are available by cycle — comparison is generally the user's responsibility | Snapshot retention + identification rules designed as requirements |
| Handling out-of-stock labels | Humans review and decide | Collect the specified elements as-is | Build and maintain a site-specific label dictionary |
| Daily on-time delivery | Depends on the staff member's arrival time | Scheduled cloud execution (according to official websites) | Designed and operated by working backward from delivery time |
| Anomaly detection (empty values, sudden count changes) | When something feels wrong | User checks directly | Includes count trends and required-value validation (99.7% accuracy) |
| When a site is redesigned | Requires more manual effort | User repairs the rules | Provider detects and repairs it (included in subscription) |
| Best suited for | 10 or fewer targets, once a week | Few targets, stable structure, direct operation possible | Daily morning reports are tied to business operations |
The rows to focus on in the table are the two above: new product identification and out-of-stock label handling. Everything else follows from those decisions.
It is not because no-code tools are inadequate. Both rows are matters of definition, not tool functionality. Octoparse and Thunderbit will reliably execute scheduled runs once you create the rules (according to their official websites). However, the decision to "compare with yesterday and retain only newly added rows" still remains the responsibility of people.
If you have a development team, there is a fourth option. You can purchase the collection layer through developer-focused scraping APIs such as Zyte and Firecrawl, and build snapshot retention, comparison, and report generation yourself (according to each official website). This provides the greatest flexibility.
However, the three topics covered in this article—snapshot management, an out-of-stock label dictionary, and on-time delivery—are not defined by any API. They remain entirely with the party building the system.
Hashscraper provides this managed approach based on experience collecting data from more than 5,000 domestic sites and proxies in 195 countries, and is used by more than 500 companies.
5 Questions for Self-Assessment Before You Start
Check these. These five lines are faster than three quotes.
- [ ] Does the product list collected yesterday still exist now — in a form that can be compared line by line with today's?
- [ ] Do you have documented rules for separating a first-seen product into new product, renewal, added option, and restock?
- [ ] Can the team answer in one sentence whether a product with only some options out of stock should count as out of stock?
- [ ] On a day when collection fails, can the recipient tell whether the report arrived empty or did not arrive at all?
- [ ] Did last week's report lead to any actual action?
If No. 1 is "No," you are not yet at the stage of comparing tools. The first step is to start saving lists today.
If No. 5 is "No," it is a threshold issue, not a collection issue. Narrowing the target scope will be much more effective, much faster.
5 Implementation Steps
Step 1. Narrow down the monitoring targets — Start not with "all competitors," but with "competing products in the categories we respond to." The broader the scope, the broader the false positives.
Step 2. Write the identification rules and out-of-stock definition on one page — What is the identifier? How will renewals and new products be distinguished? How will option stockouts be counted? This one page is the requirements document.
Step 3. Create a schedule by working backward from the delivery time — Place collection, retries, validation, and delivery in reverse order. The key is not leaving validation and retry slots empty.
Step 4. Define the delivery format and thresholds — Send it through the place where responsible staff already work: Excel, email, Slack, or API. For format-specific decision criteria, see Data Received in Excel vs. Data Viewed on a Dashboard.
Step 5. Run a two-week pilot to eliminate false positives, then scale — For the first two weeks, manually compare the new products that appear in reports. This is when every source of false positives becomes visible, such as counting renewals or restocks as new products.
What you need to do today is not select a provider. It is Step 2—write one sentence defining what "new product" and "out of stock" each mean for your team.
Frequently Asked Questions
Q. I want to automatically receive daily morning data on competitors' new products and out-of-stock status. How should I do it?
A. There are three steps. ① Narrow the target list (which categories on which sites), ② define identification rules (what counts as a new product and what counts as out of stock), and ③ create a collection, validation, and delivery schedule by working backward from the delivery time. Then choose the approach. If the number of targets is small and someone internally can fix rules when they break, you can start with scheduled runs using no-code tools like Octoparse or Thunderbit (according to their official websites). If targets increase or business operations depend on the report every morning, a managed data collection service is more practical. Hashscraper designs and operates requirements including snapshot retention, identification rules, and on-time delivery.
Q. How do you distinguish a new product from a renewal?
A. They cannot be separated automatically with 100% accuracy. Narrow the classification with rules—whether a new product ID was created, whether product-name similarity exceeds a threshold, and whether it appeared in the list within the past N days. Combine these three signals to classify products into "new product / renewal candidate / reappearance," then have people review only the candidate group. This is the most stable approach in practice.
Q. Out-of-stock labels vary by site. Can they be standardized?
A. Create a site-specific label dictionary and normalize them into a single status value. It is important to retain the original label displayed on screen as well. That way, historical data can be reclassified even if a site changes its wording.
Q. Can I receive it through Slack or an API instead of Excel?
A. Yes. Based on Hashscraper's offering, it supports Excel files, automated email delivery, API integration, and direct database loading. There is one principle: send it where the responsible person is already working. A structure where you need to go check differs in response speed from one where the information arrives to you.
Q. How often should out-of-stock data be collected?
A. Match it to your response cycle. If decisions are made in a daily morning meeting, once a day is sufficient. For product categories where inventory is depleted within a day, design a shorter interval—but reducing the interval increases blocking risk and cost together. The standard is not a technical limit, but response speed.
Conclusion
The design of a "daily morning new product and out-of-stock report" can be reduced to this:
- Keep yesterday's list — Without a baseline snapshot, neither new products nor stockouts can be calculated
- Define what counts as a new product and out of stock — Rules that distinguish renewals, added options, and restocks
- Work backward from the delivery time — Do not leave validation and retry slots empty
- Leave only what needs to be seen today — A 200-item report is the same as a zero-item report
Prices are read; new products and stockouts are counted.
Counting requires a ruler. The same ruler as yesterday.
What you need is not today's product list. It is the difference between yesterday and today.
Collection that does not save yesterday will miss today's new products forever.
Get Started Now
Tell us the competitors and categories you want to monitor, along with your desired delivery time, and we will provide a free assessment of collection feasibility and how to define new product and out-of-stock identification rules. New sign-ups receive 50,000 credits, so you can verify result quality first.
.jpg)
