For years, using a language model across an entire dataset sounded like an expensive research project.
That assumption is now outdated.
Models have become capable enough to handle useful classification, extraction, and generation tasks while Flash-class inference has become cheap enough to run across thousands of records. Combined with high concurrency and a small amount of data engineering, an LLM can now behave like another transformation step in a pipeline.
We recently used this pattern on more than 10,000 items. We processed them with Gemini 2.5 Flash through OpenRouter, using up to 300 concurrent workers and reaching close to 30 requests per second at peak. The full run finished in minutes.
The interesting part was not the model or the speed on its own. It was the problem this unlocked.
We had the outputs we wanted, but we did not have the prompts that could have produced them. To prepare the data for fine-tuning, we needed to reverse-engineer a plausible instruction for every output and add structured labels such as intent, tone, topic, audience, and constraints.
Doing that manually for more than 10,000 records would have been absurd. Writing deterministic rules would have produced shallow labels and brittle prompts. Running a capable, inexpensive model across every row made the task practical.
I will cover the fine-tuning side in a future post. This one is about the more general tool: using large-scale inference to turn unstructured or incomplete data into something structured and useful.
Think of the Model as a Data Transformation
The mental model is simple:
For each item, we sent the model:
- The original output
- A small amount of available metadata
- A precise definition of the fields we wanted
- Rules about what it could and could not infer
- A strict JSON response schema
The model returned a record shaped roughly like this:
{
"instruction": "A natural request that could produce the known output",
"intent": "educate",
"tone": "direct",
"topic": "data engineering",
"audience": "technical practitioners",
"angle": "a practical lesson from a real implementation",
"constraints": ["concise", "no invented facts"],
"language": "en"
}
The known output remained unchanged. The model's job was to reconstruct the missing context and attach useful metadata, not rewrite the source.
This distinction matters. We were not asking an LLM to create a synthetic dataset from nothing. We were using it to complete a partially observed one.
That pattern appears everywhere. Companies often have the result of a process but not the labels, structure, or context needed to analyze it or use it for machine learning.
Why OpenRouter Was Useful
OpenRouter gave us a common API across model providers. That mattered more than expected.
Our first model worked at modest concurrency but started returning empty content during the high-concurrency run. Because the provider interface stayed the same, we switched to Gemini 2.5 Flash and resumed the job without redesigning the pipeline.
This is one of the practical benefits of routing inference through a model gateway. Model choice becomes configuration instead of architecture.
You can test a few models on the same evaluation sample, compare quality and cost, then route the large batch to the best fit. If a provider has reliability issues, you can move the remaining work elsewhere. Stronger and more expensive models can be reserved for failed or ambiguous records.
The best model for a benchmark is not necessarily the best model for a 10,000-row job. For batch enrichment, the real criteria are:
- Output quality on your specific examples
- Consistent structured responses
- Throughput under concurrency
- Rate limits and provider reliability
- Input and output token cost
- Recovery behavior when requests fail
Gemini 2.5 Flash gave us the combination we needed for this run.
Concurrency Changes the Economics of the Workflow
A fast model called sequentially is still a slow data pipeline.
At one request per second, 10,000 items take almost three hours. At close to 30 requests per second, the same number of requests can complete in under six minutes before accounting for retries and other overhead.
Our implementation was intentionally simple. A thread pool submitted independent records to OpenRouter. Each successful response was validated and immediately appended to the output dataset. Each failure was written to a separate error file.
The important part was not the exact concurrency number. It was that every item was independent and the pipeline was designed to use that fact.
But setting concurrency to 300 is not a universal recommendation. The right number depends on provider limits, model capacity, request size, timeout behavior, and your own network. Start with a small sample, increase gradually, and measure successful throughput rather than submitted requests.
More workers do not help if they only create more rate limits, timeouts, or malformed responses.
The Unexciting Engineering Made It Work
Calling the model was the easy part. Making 10,000 calls safely required standard data pipeline practices.
1. Ask for structured output
We requested JSON and defined the exact expected fields. Free-form prose would have made the next stage unnecessarily fragile.
2. Validate every response
A successful HTTP response is not necessarily a valid record. We checked that required keys existed, that the instruction was not empty, and that list fields were actually lists.
3. Retry transient failures
Rate limits, timeouts, server errors, and invalid JSON are normal at this volume. The client retried transient HTTP errors with backoff. Invalid JSON triggered a repair request at temperature zero.
4. Fail one item, not the batch
One bad response should not stop 9,999 good ones. Errors went to their own JSONL file with the item identifier, model, timestamp, and error message.
5. Make the job resumable
Successful identifiers were persisted as the job ran. Restarting with a resume flag skipped completed items and processed only the remainder. This also let us switch models after the first provider became unreliable.
6. Track usage and cost
We accumulated input tokens, output tokens, estimated cost, elapsed time, successes, and failures. Cheap inference can still become expensive when prompts are unnecessarily large or retries get out of control.
7. Preserve lineage
Every enriched row kept its source identifier, original output, generated metadata, model name, and enrichment timestamp. If the prompt or model changes, we can compare versions or rebuild the derived dataset.
None of these ideas are specific to AI. They are the same properties we want from any reliable data pipeline: idempotence, observability, validation, error isolation, and recoverability.
Where This Pattern Is Useful
The prompt-generation problem is specific, but the pattern is not. Large-scale inference is useful whenever the input contains meaning that would be difficult to capture with SQL, regular expressions, or a long list of rules.
Customer support
Turn historical tickets into structured records with intent, product area, urgency, sentiment, resolution type, and a concise summary. Use those labels to find recurring problems, improve routing, or prepare evaluation data for a support agent.
Ecommerce and marketplaces
Normalize messy supplier catalogs. Extract product attributes from descriptions, map inconsistent category names into a taxonomy, detect likely duplicates, and flag records that need human review.
Finance and operations
Classify transaction memos, invoice descriptions, reimbursement notes, or reconciliation exceptions. Generate consistent categories and explanations while keeping deterministic controls around any financial decision.
Manufacturing and field service
Convert technician notes and maintenance logs into equipment, symptom, failure mode, action taken, and parts-used fields. Years of free text can become searchable operational history.
Real estate
Extract amenities, property condition, renovation details, and location signals from inconsistent listing descriptions. Normalize the results into fields that can be filtered and analyzed.
Healthcare administration
Structure non-diagnostic administrative notes, referral reasons, and scheduling messages to improve routing and reporting. Sensitive data requires appropriate privacy controls, vendor agreements, and human oversight.
Legal and compliance
Build a first-pass inventory of clauses, obligations, renewal terms, or policy topics across document collections. Use the output to prioritize expert review, not to replace it.
Research and knowledge management
Tag documents by topic, method, entity, claim, and evidence type. Generate summaries or candidate links between records so a large archive becomes easier to navigate.
The common shape is:
- You already have a large collection of records.
- Important information is trapped in text or inconsistent fields.
- The desired output can be represented as a schema.
- A human can judge good and bad examples.
- Some error rate is acceptable or can be routed to review.
If all five are true, batch inference is worth considering.
Start With an Evaluation Set, Not 300 Workers
The most dangerous part of cheap inference is that it makes producing bad data cheap too.
Before scaling, take 50 to 100 representative records and label enough of them manually to judge the model. Include easy cases, long-tail cases, multilingual content, missing context, and records where the correct answer is "unknown."
Then test:
- Does the schema capture what the business actually needs?
- Does the model invent facts when context is missing?
- Are labels consistent across similar records?
- Can deterministic validation catch obvious failures?
- Which cases should go to a person or stronger model?
Only after the output is useful should you increase concurrency.
LLM enrichment also should not replace deterministic transformations where deterministic logic works. Dates, identifiers, arithmetic, known mappings, and regulatory rules belong in code. Use the model for semantic ambiguity, then surround it with conventional software.
A New Tool for the Data Engineer's Belt
Data teams have traditionally reached for parsers, lookup tables, fuzzy matching, manual labeling, and custom machine learning models when data was messy.
Those tools still matter. But there is now another option between brittle rules and a full ML project: send each record through a general-purpose model, request a typed result, validate it, and scale the process horizontally.
The code can be small. The model can be inexpensive. The run can finish in minutes.
The real skill is designing the transformation: deciding what context to provide, what the model is allowed to infer, what the schema should contain, how quality will be measured, and how failures will be recovered.
As models continue to get cheaper and better, inference at scale will stop feeling like an AI experiment. It will become a normal part of data engineering.
The next time you face 10,000 dirty or incomplete records, do not assume the only choices are manual cleanup or months of custom modeling. A well-designed batch inference pipeline might be the fastest path from unstructured data to a useful dataset.