Inference.net
Inference infrastructure for engineering teams running large language models in production. One gateway reaches 59 models; trace agent runs, evaluate candidates on live traffic, then fine-tune and deploy task-specific models claimed to cost up to 90% less.
What is Inference.net?
Inference.net is the platform published by Inference R&D, Inc., a San Francisco company, and it presents itself as inference infrastructure for AI-native teams. Its starting argument is economic rather than technical. The company page contends there is less of a GPU shortage than generally believed: data centres hold unused capacity, but in fragments — a few minutes here, a few hours there — that perish if nobody consumes them, like electricity that cannot be stored. Inference.net built scheduling and orchestration software to aggregate those fragments across data centres, casts itself as the buyer of last resort for them, and passes the resulting discounts on to customers.
Around that thesis sits a six-part product line. Deploy hosts models on dedicated GPUs, advertising 99.99% uptime and per-hour pricing such as $9.98 an hour on a B200 with 180 GiB of VRAM. Observe and Trace capture production traffic and complete agent runs. Train fine-tunes task-specific models, Evaluate scores them against real traffic, and HALO, open source under MIT, analyses agent behaviour. The documentation condenses all of this into two entry points, Tracing and Gateway, usable separately or together.
The Gateway is the practical centre of gravity. One API key reaches a catalogue of 59 models spanning the Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Grok, Llama, Qwen, Gemma and Nemotron families, and it forwards requests to OpenAI, Anthropic, Vertex AI, Amazon Bedrock, OpenRouter, Groq, Cerebras or any OpenAI-compatible endpoint. Halo, described as a recursive language model, reads an agent's entire body of runs instead of the truncated slice a normal context window allows, and returns ranked failure modes, each backed by the trace IDs that show it happening.
The promise is that these pieces close a loop — trace, evaluate, build a dataset, fine-tune, deploy — on one platform. The vendor claims custom models up to 100 times smaller than a GPT-5-class system, two to three times faster and up to 90% cheaper, and publishes case studies to support it: Cal AI cut latency by 66%, Wynd Labs processes video at 95% lower cost, and Gravity Ads went from 900 ms to 240 ms at the 90th percentile. Two in-house open-source models, Cliptagger and Schematron, round out the catalogue.
What it does
- Serve open-source, proprietary and fine-tuned models behind a single API key
- Route existing LLM traffic through one gateway by changing a single base URL
- Capture full agent traces covering LLM calls, tool calls and framework steps
- Analyse those traces with Halo and get failure modes ranked, each citing the trace IDs that prove it
- Replay real production traffic against candidate models with AutoEvals and get a recommended winner
- Turn traces into datasets in one click and fine-tune a task-specific model on them
- Deploy that model on dedicated GPUs and track latency, cost, error rate and drift
When to use Inference.net / When not to
A quick filter to help you decide if Inference.net is the right fit.
When to use Inference.net
- Engineering teams running high-volume LLM workloads: the site explicitly addresses companies spending more than $50,000 a month with closed-source providers.
- MLOps and platform engineers who want one gateway and one API key across OpenAI, Anthropic, Gemini, Vertex AI, Bedrock, Groq and Cerebras, with unified latency and cost metrics.
- Teams building multi-step agents that need full run traces plus ranked, evidence-backed failure modes rather than another dashboard to scroll through.
- Product teams looking to replace a general-purpose model with a smaller task-specific one on a repetitive job such as document extraction, image captioning or content classification.
- Open-source AI projects, which can apply for up to $10,000 of free compute through the grants programme, awarded on a rolling basis.
When not to use Inference.net
- Non-technical users: everything begins with instrumented code or a rerouted base URL, and there is no no-code interface anywhere in the product.
- Anyone expecting a mobile or desktop application: there is no iOS app, no Android app and no browser extension.
- Teams whose main need is prompt management, which the vendor openly admits it does not offer yet on its own comparison page.
- Organisations requiring a fully self-hostable platform: only the Halo analysis engine is open source under MIT, not the platform itself.
- European buyers who need a documented GDPR posture: the site never mentions the GDPR, publishes no DPA and no subprocessor list, and personal data is processed in the United States.
How to use Inference.net
A typical end-to-end flow, from setup to results.
- Create an account at inference.net/register; the free Pay as you go plan opens immediately
- Choose an entry point: Gateway to reroute existing traffic, or Tracing to instrument agent code — most teams start with one and add the other later
- For Gateway, change the base URL of your existing SDK and pass your Inference project key in the Authorization header
- Pass your own provider key in the x-inference-provider-api-key header; the gateway forwards it downstream, so you keep your provider accounts
- Optionally tag traffic with x-inference-environment and x-inference-task-id to group requests for filtering and analytics
- For Tracing, install the @inference/tracing (TypeScript) or inference-tracing (Python) SDK and call setup() once to instrument your providers and frameworks
- Watch traces, latency percentiles, cost attribution and error rates land in the dashboard, then run Halo to get a ranked list of what to fix
- Turn traces or filtered subsets into datasets in one click, and run AutoEvals to replay that traffic against candidate models
- Fine-tune a task-specific model on the dataset and compare it with the baseline before promoting it to production
- Deploy the model to a dedicated GPU and serve it behind the same API key; book a call with the research team if you want help sizing the work
Pros & Cons
Pros
- The whole loop lives on one platform — trace, evaluate, build datasets, fine-tune, deploy — instead of stitching four vendors together
- Adoption cost is genuinely low: one base URL change for the gateway, one setup() call for tracing, and you keep your existing provider keys
- Published customer results are specific rather than vague: 66% less latency at Cal AI, 95% lower cost at Wynd Labs, 900 ms down to 240 ms at Gravity Ads
- The free plan is substantial, with 1 million gateway requests and 1 million tracing spans a month, enough for a real pilot
- No per-seat pricing: the Growth plan carries unlimited seats, so the whole team can be invited without a budget conversation
- Halo is MIT-licensed and runs locally on exported traces, which limits lock-in on the analysis layer specifically
- Trained weights are described as portable, and the security posture is stated plainly: SOC 2 Type II, encryption in transit and at rest, secrets excluded from traces and logs
Cons
- No GDPR posture whatsoever: the term never appears anywhere on the site, and there is no Data Processing Agreement and no subprocessor list
- The Trust Center that would carry the SOC 2 report returned HTTP 429 on every attempt, so the certification could not be verified at its source
- Personal data is processed in the United States, with no European hosting option advertised
- The licence the terms take over User Content and Output is very broad — perpetual, irrevocable, sublicensable, worldwide — which sits awkwardly beside the marketing promise that your data is your moat
- Fees are non-refundable, subscriptions renew automatically, and cancellation requires three days' notice before the next term begins
- The step from free to paid is abrupt: $0, then $250 a month, with nothing in between
- Prompt management and full platform self-hosting are both missing, as the vendor itself concedes on its comparison page
Pricing & Plans
There is a permanent free plan, not a time-limited trial. Pay as you go costs $0 plus usage and includes 1 million gateway requests and 1 million tracing spans per month. The lowest paid entry point is the Growth plan at USD 250.00 per month. Enterprise pricing is quoted on request. Two charges sit outside the plan ladder and should be budgeted separately: dedicated deployments are billed by the hour, from $9.98 an hour on a B200 instance, and catalogue models are billed per million tokens, so the final invoice tracks actual usage.
- 1M included gateway requests
- 1M tracing spans per month
- 14-day data retention
- 1 seat
- 30 requests per minute
- $50 one-time opening credit
- 50M gateway requests per month
- 50M tracing spans per month
- unlimited data retention
- unlimited seats
- 250 requests per minute
- custom contracts and committed-use pricing
- dedicated infrastructure and deployment limits
- a direct support channel
- and custom models trained for the workload
- Dedicated model deployment — billed hourly on top of a plan
- from $9.98 an hour on a B200 instance with 180 GiB of VRAM
Data, GDPR & hosting
A consolidated view of how Inference.net handles your data.
GDPR overview
There is no GDPR implementation to report, and that absence is the finding. Across every page collected, including the privacy policy and the terms of service, the words GDPR, EEA and Article 27 never appear once; no EU representative is designated, no Data Processing Agreement is published, and no subprocessor list exists. Rights are framed in United States terms instead: personal data can be erased through a dedicated deletion request form, and corrections can be requested by writing to help@inference.net. The terms have the user acknowledge that personal data may be transferred to, and processed in, the United States. On age, the privacy policy disclaims knowing collection below 13 and promises deletion for a child under 16, while the terms bar anyone under 18. Both documents were last updated on 21 January 2026.
Who owns the data?
The terms are unusually direct: Inference R&D, Inc. claims no ownership over User Content, which remains the customer's, and the Deploy page adds that trained model weights stay yours and stay portable. In exchange, however, users grant the company a paid-up, royalty-free, transferable, perpetual, irrevocable, non-exclusive and worldwide licence, with the right to sublicense, to use, copy, modify, publish and make derivative works of that content and any Output, both to run the service and for the wider business of Inference.net and its affiliates. Sharing content with other users is off by default and must be switched on deliberately. The privacy policy adds that data is neither sold nor shared.
Reuse rights
Customers keep their own content and their trained weights, so both can be reused freely, including moving weights off the platform entirely. On the vendor's side, the privacy policy confines use of customer data to delivering the inference service and debugging it: the data is not sold, not shared, and explicitly not used to train AI models. Account, payment, contact and usage data are nonetheless collected, and cookies and tracking technologies feed analytics and interest-based advertising through third parties including Google Analytics, ad networks and social platforms. Stripe processes payments and never passes full card details to the vendor, while Google and Okta sign-in can supply identity data. Content sharing between users stays disabled unless the customer turns it on.
Data retention & training
Hosting summary
The terms state that personal data may be transferred to, and processed in, the United States, and no other hosting country or region is named anywhere on the site. Jurisdiction follows the same line: the terms are governed by Delaware law, with Delaware as the exclusive forum. For the models themselves, the Deploy page offers public cloud, private cloud or hybrid deployment so that customers can place workloads where their compliance and data-residency requirements demand, though no specific regions are listed. The underlying business model rests on aggregating spare capacity across multiple data centres, none of which are named or located publicly. Security is described in general terms: SOC 2 Type II, encryption in transit and at rest, and automatic exclusion of API keys, tokens and credentials from all traces and logs. No subprocessor list is published, and the Trust Center that would normally carry one was unreachable during this review. The website itself sits behind Cloudflare on an anycast address, which says nothing about where customer data actually lives.
Things to keep in mind
Risks and trade-offs to weigh before adopting Inference.net.
- The licence the terms take over User Content and Output is perpetual, irrevocable, sublicensable and worldwide; read it before routing sensitive customer prompts through the gateway
- No GDPR documentation, no DPA and no subprocessor list, while personal data is processed in the United States: a genuine obstacle for regulated European use
- The SOC 2 Type II claim is repeated across the site, but the Trust Center backing it was unreachable, so for now it rests on the vendor's own word
- Subscriptions renew automatically and fees are non-refundable, including for time never used; cancelling requires three days' notice before the next term
- Binding individual arbitration and a class-action waiver apply unless you opt out within thirty days by emailing legal@inference.net
- Routing every model call through one gateway concentrates a dependency: an outage, a price change or a policy change there reaches all your providers at once
- Leaning on an AI-generated ranking of your own failure modes can quietly dull a team's diagnostic instinct; Halo cites trace IDs precisely so its findings can be checked, and they should be
Setup & Integrations
Technical difficulty
Moderate, and firmly developer territory. The lightest path is the gateway: change one base URL and set two headers, with no SDK swap and no application rewrite. Tracing needs a package install and a single setup() call in TypeScript or Python. The vendor advertises five minutes to first traces and five minutes to a deployed model, which is plausible for that first step but not for the full loop. What is genuinely required is comfort with application code and API key management. There is no no-code path: the dashboard Playground complements the SDKs rather than replacing them.
Deployment
Integrations
Behind Inference.net
Fundraising
Resources
All the official URLs gathered for verification and reference.
Alternatives
Tools that compete with or complement Inference.net.
Frequently asked questions
What does Inference.net actually do?
Is there a free plan?
What does the first paid tier cost?
Will my data be used to train AI models?
Who owns the content and the models I create?
Is there an API, and how many models can I reach?
Where is my data hosted?
Does Inference.net comply with the GDPR?
What is the minimum age, and how do I cancel?
Should you pick Inference.net?
Inference.net is a coherent and unusually complete piece of engineering infrastructure. Where most vendors sell a single layer — a gateway, a tracing tool, a fine-tuning service — this one sells the loop that connects them, and the argument holds together: capture production traffic, find what is failing, turn it into a dataset, train a smaller model on it, and serve that model behind the same API key. The economic thesis underneath, buying perishable data-centre capacity that would otherwise expire unused, explains the pricing rather than merely decorating it.
For teams already spending heavily on frontier models, the case is easy to test. The free plan is generous enough for a genuine pilot, adoption costs little more than a base URL change, and the published customer results are specific enough to argue with. The absence of per-seat pricing and the MIT licence on Halo both lower the cost of changing your mind later.
The reservations are not about the product. They are about what the site does not say. Nowhere across the eighteen pages reviewed does the word GDPR appear; there is no Data Processing Agreement, no subprocessor list, and personal data is processed in the United States. The Trust Center that would answer these questions was unreachable throughout. Any European buyer should treat that as an open question to put to the vendor directly rather than a detail. The licence the terms take over user content is also broader than the marketing implies, and worth reading before routing anything sensitive through the gateway.
This is a tool for engineering teams, not end users: without instrumented code, nothing happens. Within that audience, and outside regulated European contexts where the compliance gap matters, it is a serious and well-documented option.
- Choosing a selection results in a full page refresh.
- Opens in a new window.