When everything is visible on the screen, but you feel stuck moving it to Excel
Competitor prices, product reviews, company lists, news articles - all the necessary information is visible on the web screen, but it can be daunting to move it to Excel for analysis. If it's just one or two pages, copying and pasting will do, but if it's hundreds of pages, it's a different story.
Transferring publicly available information on websites into tabular data is called web data extraction (web crawling, web scraping). This article compares and summarizes four starting methods for those considering data extraction for the first time. It is written in a way that can be understood even without coding knowledge.
Method 1. Manual Copying - Quick start, quick limitations
This method involves simply copying the content on the screen and pasting it into Excel. It doesn't require any tools or costs, so if the target is just one or two pages and it's a one-time task, this is the answer.
The limitations are clear. As the quantity increases even slightly, the time becomes unmanageable, and mistakes such as omissions or incorrect transfers by humans can occur. Above all, it becomes impossible to maintain when repetition such as "checking again every week" is required.
Method 2. Extraction Tools - Without coding, even for simple sites
This method involves using browser extensions or no-code data extraction tools. By clicking on the desired items on the screen, it creates a table, allowing you to collect hundreds of items without coding.
However, there are limitations to what the tool can handle.
- It often struggles with dynamic pages where new content appears when scrolling or pages requiring login.
- Large sites have mechanisms that block automated access, making it easy to block tool-level access.
- When the site layout changes, you have to set it up from scratch again.
It is useful for occasionally collecting data from standardized simple sites, but for more than that, the next two methods are more suitable.
Method 3. Custom Development - Highest flexibility, ongoing maintenance
This method involves developers creating crawlers using Python or similar languages. It offers the highest level of flexibility to process any site or item into the desired format.
However, it requires someone who can create it, and the real work begins after creation. You need to modify the crawler whenever the site changes, set up separate infrastructure (proxies, etc.) to bypass strong blocking sites, and monitoring the collection process is also part of the job. In fact, even large companies with data teams sometimes give up direct collection due to the maintenance burden.
If it's difficult to keep development resources solely for this purpose, it's a good idea to calculate the operational costs before starting.
Method 4. Data Collection Service - Provide requirements and receive data only
This method involves outsourcing data collection to specialized companies. You only need to provide requirements like "I want to receive this item from this site at this frequency," and the company takes care of everything from developing the crawler to handling blocks, maintaining the site changes, and monitoring the collection, providing you with the resulting data.
- No coding knowledge is required - requirements can be communicated in everyday language.
- It excels in handling heavily blocked large sites, repetitive collections, and large-scale collections.
- You can choose the format you receive, such as Excel, email, API, or database integration.
Of course, it comes with a cost. Therefore, if it's a small one-time collection, the previous methods are better, and as you need to receive data continuously or in large quantities, the service becomes more advantageous.
Comparison at a glance
| Category | Manual Copying | Extraction Tools | Custom Development | Data Collection Service |
|---|---|---|---|---|
| Coding Knowledge | Not required | Not required | Required | Not required |
| Manageable Quantity | Dozens | Hundreds | Unlimited | Unlimited |
| Repetitive (Regular) Collection | Practically impossible | Limited | Possible | Possible |
| Strongly Blocked Sites | - | Difficult | Infrastructure setup required | Possible |
| Site Change Response | - | Requires reconfiguration | Direct repair | Managed by the company |
| Cost | Time | Tool usage fee | Labor + Infrastructure | Service usage fee |
Criteria for choosing where to start
Just consider these three things, and you'll generally find the answer.
- Quantity - Manual for dozens, tools for hundreds, development or service for more.
- Frequency - Light methods for one-time, automation (development/service) for weekly/daily repetition.
- Site Difficulty - For heavily blocked sites like large e-commerce or social media, tools are challenging, and development requires significant infrastructure, making it closer to the service area.
In summary: If it's a light one-time task, try manual or tool first, and if it's data you need to continuously receive, decide based on whether you can secure operational resources between development and data collection services.
Things to know before starting: Not all data can be collected
- You cannot extract information that is not on the screen. Web data extraction involves transferring information from publicly available screens, not extracting private data.
- There are areas that require legal review. Using public data for internal analysis and collecting personal information or redistributing content as it is have different characteristics. Depending on the purpose and usage method, judgments may differ, so it's safer to confirm before collecting if it's ambiguous.
Frequently Asked Questions
Q. Can I start data extraction without knowing any coding?
Yes. For small amounts, you can try using extraction tools directly, and for larger scales, you can communicate requirements in everyday language to data collection services. Simply summarizing "which information from which site and how often" is sufficient.
Q. How much does data collection service cost?
It varies depending on the site's difficulty and the amount/frequency of collection. By getting a quote, you can check the cost structure (development cost, operating cost, maintenance), and a guide on how to compare is provided in the estimation guide in the article 'Good Reads Together.'
Q. Can any site be fully collected?
Most publicly available screen information is collectible, but the difficulty varies by site. It's best to review collectibility first, especially for heavily blocked sites, and then start.
Good Reads Together
- Why do large companies' data teams give up direct crawling?
- Crawling outsourcing process - From inquiry to first data, organized step by step.
- How to choose a crawling company - 7 things to check before outsourcing data collection.
- Why do crawling estimates vary by company? - Cost structure and hidden costs.
Start Now
If you're unsure which method is right, let us know the site and items you want to collect. We'll diagnose the collectibility and recommend the appropriate method for free.
