Skip to main content

Command menu

Jump to a page or search the catalog

Labeled Datasets

Tech Link Quality Corpus — 215,061 verified web page records

215,061 quote-verified URL metadata records from arXiv, GitHub, Wikipedia and Stack Overflow — with AI quality verdicts, keywords and documented methodology. JSONL.

$39.00

What you get: 1 file · 51.49 MB · ZIP

See purchase options

Product description

# Tech Link Quality Corpus — 215,061 verified web page records

Delivery: ZIP package with the JSONL data file (zstd-compressed), LICENSE, buyer guide (README) and the judge methodology (METHODOLOGY.md). Instant download.

## What you get

Metadata for 215,061 English-language technical web pages — the quote-verified layer of a 225,852-record corpus. Every record: URL, domain, source type, an LLM-judge verdict — "substantive page vs stub" — with a short written rationale, and 10–15 keywords.

Records also carry page_topic_hint — a rule-extracted page topic (arXiv categories like "arxiv: Computer Science > Machine Learning"). It is filled for ~22% of records; the rest carry a technical placeholder. For topical filtering use keywords, present on every record.

One JSONL record (trimmed to key fields; full records have 16 fields: pid, slug, theme, url, domain, tap, source, track, worthy, reason, keywords, phrases, entities, brands_products, quotes_verified, page_topic_hint):

```json

{

"pid": "kv_cache_llm_05509c8977",

"url": "https://arxiv.org/abs/2607.20495",

"domain": "arxiv.org",

"tap": "arxiv",

"worthy": true,

"reason": "A substantive arXiv paper page with a full abstract on workload-aware caching...",

"keywords": ["KV cache", "LLM inference", "eviction policy"],

"page_topic_hint": "arxiv: Computer Science > Machine Learning",

"quotes_verified": true

}

```

## What it is good for

- Training a page-quality classifier: 215k labeled examples (substantive vs stub/boilerplate/404/ad) with per-record rationale

- Seed list for RAG / AI-search crawling: live, substantive technical URLs, pre-filtered

- URL prospecting filters for SEO tooling

## Sources and composition

- 1,899 distinct domains in the dataset; top sources by records: github.com 28.9%, arxiv.org 28.5%, en.wikipedia.org 21.3%, stackoverflow.com 17.7%, datatracker.ietf.org 2.3%, others ~1.5% (EUR-Lex, ICLR, Ethresear.ch and misc tech blogs)

- 100% https, 99.98% unique URLs, snapshot: November 2026

- Quality checks: every record cross-verified against the page text quotes_verified: true in all 215,061); records with broken judge–page binding excluded; live spot-check of 25 random URLs: 21/25 alive, all substantive

## What is NOT included (by design)

- Page texts (metadata only)

- Verbatim quotes from pages (copyright hygiene)

- The original harvest-query field — it is a search seed, not a classification. For topical filtering use keywords (on every record); page_topic_hint adds arXiv categories where available (~22% of records)

## Honest limitations

- Verdicts are machine-generated (GLM-5.3-flash, temperature 0.2, documented prompt); the prompt targets a technical corpus, but the judge accepted some substantive non-technical pages (~15–20% of worthy=true) — filterable by reason text, and the methodology to re-classify is included

- 12,000-character input cap per page: annotations describe the head of each page

- page_topic_hint is filled for ~22% of records (mostly arXiv categories); the rest carry a technical placeholder

- Links age; snapshot dated November 2026

## Format

- clean-verified.jsonl.zst: 215,061 quote-verified records — the data file in this package (51.5 MB compressed); the package also includes LICENSE, buyer guide (README) and METHODOLOGY.md

- The full layer (225,852 records, each with the quotes_verified flag) can be provided on request — ask in the product chat

- JSONL, one record per line, zstd frame; opens with any language's zstd binding

- Free sample of 4,000 records on Hugging Face (link in the product card)

What you get

1 file · 51.49 MB

  • getly-package.zipZIP · 51.49 MB
Try it free

Free trial from huggingface.co

LicensePersonal use only
PublishedNov 15, 2026
File size51.49 MB
File formatZIP
Versionv1.0

Tags

dataseturl metadataragquality filteringllm judgeweb crawldata curationarxivgithubsearch
D
Defmass Studio

Frequently asked questions

Do I get access instantly?

Yes. This is a digital product - you can download it right after checkout, straight from your library.

Can I use it for commercial projects?

Personal use only. Use it for yourself - not in work you sell or hand to a client - and do not resell or redistribute the file.

What's your refund policy?

You can request a refund within 30 days. If something isn't right, request it from your order and the Getly team reviews it.

What file formats and sizes will I get?

You'll download 1 file - ZIP - 51.49 MB in total.

Do I get free updates?

Yes - when the seller ships a new version, you can re-download the latest files from your library at no extra cost. Check the changelog for recent updates.

Price

$39.00