Role in the AI Retrieval Workflow

Inference is what turns a retrieved candidate into a ranked result. It's the step where a model actually scores relevance, rather than just narrowing the field. Vespa’s approach to running that inference on the same node as the data is what keeps it fast at scale.

Why On-Node Machine Learning Inference Matters

Machine learning inference usually means a round trip: retrieve candidates from one system, ship them to a separate model-serving layer, wait for a response, and stitch the result back together. Every hop adds latency, cost, and one more place for the model and the data to drift out of sync.

Vespa collapses that into one system. Models are deployed alongside your data and evaluated at serving time on the same nodes that already hold the documents being scored. There's no separate inference cluster to provision, monitor, or keep in lockstep with your index.

This pattern has a name: architecture inversion. 

Instead of the traditional two-tier setup (retrieve from a search engine or database, then ship candidates to a separate stateless compute tier), the compute moves into the data layer itself. It's the same approach the largest recommendation and ranking systems in the world converged on once brute-force scoring stopped being optional.

No round trip. No second system. No drift between the model and the data.

That architecture is what makes it possible to:

  • Cut inference latency, not just serving latency. Because the model runs on the same node as the data, there's no network call to a separate prediction service. The cost that usually dominates real-time ML serving simply isn't there.
  • Use the framework your team already trained in. ONNX, TensorFlow, XGBoost, LightGBM, and Vespa's native tensor format are all first-class. Export what you have instead of rebuilding it for a proprietary format.
  • Scale inference the same way you scale search. Add content nodes and inference capacity grows with it. There’s no separate autoscaling policy for a model-serving fleet that has to keep pace with your index.
  • Go from embeddings to generation on one platform. Embedders, cross-encoders, classifiers, and LLMs can all be part of the same application package. One deploy, one place to version, one system to operate.

How Machine Learning Inference Works in Vespa

Models can be evaluated two ways. Most inference happens inline as part of a ranking expression (this is real-time model inference), using the same phased approach that keeps expensive models scoped to a small candidate set. 

But Vespa also supports stateless model evaluation: any model in the application package can be exposed as a REST or Java API and called on demand, independent of the ranking path, with the same versioning and safe-deployment guarantees as everything else in the application. 

The practical effect: your team ships one application, versions one thing, and debugs one system, instead of coordinating deployments across search infrastructure and a separate model-serving layer.

Machine Learning Inference flow

Teams that don’t use inline inference typically run a separate model-serving layer like a prediction API, a feature store, or a vector database sitting apart from search. That means provisioning it, monitoring it, and keeping its model versions in lockstep with the index by hand. 

Vespa removes that seam entirely: the model lives next to the data it scores, versioned and deployed as part of the same application package.

Configuration

Machine learning inference in Vespa is built into the same schema and deployment model as everything else on the platform. Here's what you control:

  • Framework support: ONNX, TensorFlow, XGBoost, LightGBM, and Vespa's native tensor format, so you're not locked into one training stack or forced to retrain just to deploy.
  • Ranking-phase placement: Run models in first-, second-, or global-phase ranking, matching model cost to how many candidates it needs to score, so accuracy doesn't come at the price of latency.
  • Embedders: Generate embeddings for queries and documents inside Vespa, in the same call that runs the rest of the query, cutting out the network cost of a separate embedding service.
  • Stateless model evaluation: Expose any model as a REST or Java API for one-off inference outside the ranking path, without standing up dedicated model-serving infrastructure.
  • GPU acceleration: Available for embedding and ranking inference, and automatic on Vespa Cloud wherever GPU-provisioned nodes are available, so bigger models don't mean a manual infrastructure project.
  • LLMs: Call local models or external, OpenAI-compatible APIs directly from a Vespa application for generation, RAG, or document enrichment, keeping generation in the same place as retrieval.

By moving computation to where the data already lives, Vespa turns inference from a bottleneck into just another part of the request as fast, as scalable, and as easy to operate as the rest of your search and retrieval pipeline. Whether you're scoring with a cross-encoder, generating embeddings, or calling an LLM, the model runs where the work already happens.

Learn with Vespa

Learn how to build search, recommendation, and RAG applications with Vespa through a free, self-paced course that combines hands-on exercises with links to the documentation.

Start the free course

Frequently Asked Questions

Need more than a quick answer?

If these FAQs don't answer your question, there are several ways to continue:

Learn the fundamentals with our free online training at learn.vespa.ai.

Experience Vespa yourself with a free Vespa Cloud trial.

Watch the Getting Started with Vespa AI Search YouTube video

Contact our team to discuss your application or migration project.
What's the difference between inline inference and on-node inference?
Inline inference describes when inference happens in the data or query pipeline, whereas on-node inference describes where the model is executed. In Vespa, inline inference is executed as part of a query or indexing pipeline, without requiring an external application to orchestrate it. With Vespa's on-node inference, the model runs directly on the Vespa node that is processing the request or document.
What is on-node machine learning inference in Vespa?
On-node machine learning inference is the ability to run trained machine learning models directly on the Vespa nodes that serve search, recommendation, and AI workloads. Also known as architecture inversion, this eliminates the need to send requests to a separate model serving system, reducing latency and simplifying deployment. Models can be used for tasks such as ranking, classification, embedding generation, and other inference workloads as part of query processing or document indexing.
Which model formats does Vespa support?
Vespa supports ONNX, TensorFlow, XTBoost, LightGBM, and its own native tensor-based models. ONNX is commonly used as an interchange format for models exported from PyTorch or Hugging Face.
Does running models slow down search?
Not by design; Vespa uses phased ranking to control where expensive models run. First-phase ranking stays cheap and covers every match, while models like cross-encoders are reserved for second- or global-phase ranking, where the candidate set is already small.
What's the difference between ranking-time inference and stateless model evaluation?
Ranking-time inference scores results as part of the query path; stateless model evaluation runs independently of it. Stateless model evaluation exposes any model in the application package as a standalone REST or Java API, for use cases like classification or scoring that aren't tied to a search request.
Does Vespa support GPU acceleration?
Yes, Vespa supports GPU acceleration for embedding and ranking model inference. On Vespa Cloud, GPU acceleration is used automatically on nodes provisioned with a GPU; on self-managed Vespa, it's configured through the embedder or ONNX model settings.
Which LLMs does Vespa support?
Vespa supports a wide range of large language models. You can deploy supported open-source models through Vespa Model Hub, use your own self-hosted models, or connect to external OpenAI-compatible APIs. LLMs can be invoked directly from Vespa applications for inference, retrieval-augmented generation (RAG), or document enrichment during indexing.