The first thing a team that decides to build its own sLLM usually does is "crawl." They scrape millions of pages, boast about the volume of text, and declare, "Corpus secured." Then, the moment they feed that corpus into the model, it starts saying strange things: "Subscribe," "Would you like to accept cookies?" "View more related products."
Models learn what you feed them. Feed them ad copy, and they learn ad copy.
A corpus is not mining; it is refining. The goal is not to extract more, but to remove better.
Fine-tuning success is not decided by GPUs. It is decided at the refinement table before the data is fed in, where you determine 'what not to feed.'
TL;DR
- For a fine-tuning corpus, 'collection' is not the main point; 'refinement' is. A pile of web text and source text for training are different things.
- If you feed noise (ads, menus, duplicates) as-is, the model learns boilerplate and develops overrepresentation bias. You are effectively spending your training budget on ad copy.
- License and personal data reviews must be completed 'before' training. Removing them after feeding the data in is practically impossible.
Table of Contents
- Why 'scraped text' is not a corpus
- The 9-step refinement pipeline
- Comparison of 3 acquisition methods
- 5 self-diagnosis questions
- FAQ
- Conclusion
- Get started now
Why 'scraped text' is not a corpus
Let's align on the definition first.
A training corpus is 'refined source text' left after removing boilerplate and duplicates from web-scraped text, while reviewing licenses and personal data. It is not raw ore; it is refined metal.
Boilerplate refers to every repeated element that is not the main content. Headers, sidebars, footers, cookie banners, "related posts," and "subscribe" button text. They may be invisible to people, but to crawlers, they are all text.
Here is the problem. On a single webpage, the actual main content takes up less space than you might think. Everything else is repeated wrapping. What happens if you feed that wrapping in as-is?
Models reproduce the patterns they see most often in the data with the greatest confidence. But "Subscribe" repeats across sites and pages. The domain knowledge that should actually be learned appears once per page, while the wrapping appears ten thousand times. As a result, the model becomes not a domain expert but a 'footer reader.'
Models learn what you feed them. They learn even the "Subscribe" button.
The 9-step refinement pipeline
Collection is only step one. The remaining eight steps determine corpus quality.
- Collection — Define target domains and URLs, then secure the source HTML. This is only the beginning.
- Boilerplate removal — Remove menus, ads, cookie banners, and repeated UI text, leaving only the main content. The first fire of refinement.
- Near-duplicate removal — Remove not only identical documents but also documents that are 'almost the same.' Mirror sites that copy and paste the same article, and product descriptions with only different templates, are caught here.
- Format normalization (JSONL) — Standardize disorganized text into a one-document-per-line (JSONL) structure that the training pipeline can read immediately.
- License/copyright review — Confirm whether the source is permissible for training. This must happen here, not later.
- Personal data scrubbing (PII removal) — Remove personally identifiable information such as names, contact details, and resident registration numbers. Once fed in, it cannot be undone.
- Toxicity/quality filtering — Filter harmful expressions, broken encodings, and meaningless text.
- Domain balancing — Adjust proportions so that certain sources or topics are not overrepresented. If one site accounts for half the corpus, the model will imitate that site.
- Token volume estimation — Only here should you measure scale. It must be based on the corpus 'after refinement.'
The most common mistake here is step 9. If you count tokens before refinement, the number is inflated because it includes the wrapping. Teams really do boast of having 10 billion tokens, only to find that after stripping the wrapping, they have 3 billion.
Count tokens after refinement is complete. Any number before that is inflated by the wrapping.
And never postpone steps 5 and 6. Licenses and personal data must be handled 'before' training. Information that has been absorbed into model weights cannot be selectively removed sentence by sentence. Post-removal is not merely difficult; in practice, retraining is the only answer.
Finish licenses and personal data before training. Once fed in, they cannot be removed.
Comparison of 3 acquisition methods
There are three paths for obtaining a corpus. Each has honest strengths and weaknesses.
| Dimension | Open datasets | Direct collection and refinement | Managed corpus delivery |
|---|---|---|---|
| Scale | Large (public high-volume data) | Depends on team capability | Tailored to required scale |
| Domain fit | Low (general-purpose) | High | High |
| License review | Varies by dataset | Your direct responsibility | Delivered after review is complete |
| Duplicate/PII refinement | Varies by dataset | Your direct responsibility (pipeline construction) | Completed through refinement |
| Maintenance burden | Low | High (all operations and updates) | Low (outsourced) |
Open datasets are excellent starting points. They are public and large in scale. However, they are not perfectly suited to your domain, and licensing and refinement quality vary from dataset to dataset, leaving the task of 'verification.'
Direct collection and refinement offers the strongest domain fit. You can scrape the sites you want, to the depth you want. But you must build and operate the 9-step pipeline above yourself. This is where most teams fail at steps 2, 3, and 6.
Managed corpus delivery means receiving a refined corpus whose collection and license review have already been completed. You get both domain fit and a low maintenance burden, but at the cost of outsourcing fees. The landscape of Korean data collection services varies by approach; if you want a broader view of this map, refer to Comparison of Korean Data Collection Services — The Real Map of 2026.
Hashscraper falls into the third category. Built on experience collecting data from more than 5,000 Korean sites and proxy infrastructure across 195 countries, it secures source text with 99.7% accuracy and delivers it after completing all nine refinement steps above. The reason it has maintained zero legal issues while working with more than 500 companies is that license review is built into the pipeline rather than left for 'later.'
5 self-diagnosis questions
Apply the following to the corpus you are currently preparing.
- [ ] Have we actually removed ad, menu, and cookie banner text from our corpus?
- [ ] Have we deduplicated even 'nearly identical' documents, including mirrors, copy-pastes, and templates?
- [ ] Have we completed license and copyright reviews 'before' training?
- [ ] Have we scrubbed personally identifiable information, with no unremoved data remaining?
- [ ] Have we recounted token volume based on the corpus 'after refinement'?
If three or fewer items are checked, what you are about to feed in is not a corpus but a pile of wrapping.
FAQ
Q. How do I collect and refine an LLM fine-tuning training dataset from the web?
A. Collection is only the beginning; refinement is the main task. After securing the source HTML, you must process it in the following order: boilerplate removal → near-duplicate removal → JSONL format normalization → license review → personal data scrubbing → toxicity/quality filtering → domain balancing → token volume estimation. Only then does it become 'refined source text' suitable for training. License and personal data reviews, in particular, must be completed before training.
Q. Isn't a corpus good if I just scrape a lot of data?
A. No. Quality is determined not by quantity but by the level of refinement. If you feed noise in as-is, the model repeatedly learns ad copy and develops bias by overrepresenting certain sources. Scale should be measured after refinement is complete.
Q. Can't I remove licenses or personal data later?
A. It is practically impossible. Information that has once been absorbed into model weights cannot be selectively removed sentence by sentence, so retraining is the only answer. That is why these two reviews must be placed at the front of the pipeline.
Q. How should a refined corpus be managed and used after it is created?
A. The challenge of actually feeding the corpus into internal workflows and keeping it updated is a different topic. It is covered separately in an article on using internal knowledge.
Conclusion
The point where fine-tuning projects fail is usually not in front of the GPU. It is before that—the moment the entire wrapping is pushed into the model. A corpus is determined by refinement quality, not extraction volume. The team that wins is not the one that scraped the most, but the one that removed the most effectively.
There are three key points: refinement is the main task, models learn noise when you feed them noise, and licenses and personal data must be handled before training. Follow these three rules, and you will not waste your training budget on ad copy.
Do not spend your training budget on ad copy. Spend that budget on domain knowledge.
Get started now
We will work with you from identifying sites that fit your domain to receiving a corpus refined through the nine steps above. Experience collecting data from more than 5,000 Korean sites, proxies across 195 countries, 99.7% accuracy, and zero legal issues. We deliver with license review embedded in the pipeline. New sign-ups receive 50,000 credits.
Contact us about crawling and monitoring
For how to extend collected data into an alert and response loop, continue with Monitoring Is Not Collection but Alerts — Building a Response Loop with Price and Reputation Data.
