Step 0: Setup

Truss

The simplest way to serve AI/ML models in production

Welcome to Truss

Create, deploy, and invoke an ML model server in less than 5 minutes

Quickstart

Get up and running quickly with Truss' developer workflow

User guide

A guide to using secrets securely in your ML models

How to use secrets

A guide to configuring a base image for your truss

How to use base images

Load model weights without Hugging Face or S3

How to load model weights

A guide on configuring your truss to use external packages

How to use external packages

Accelerate cold starts by caching your weights

How to cache model weights

A guide to setting concurrency for your model

How to configure concurrency

High performance model serving with TGI

Getting Started

LLM with Streaming

Text-to-image

Deploy a language model, with the model weights cached at build time

Fast Cold Starts with Cached Weights

High Performance LLM with TGI

High Performance LLM with vLLM

Load a model that requires authentication with Hugging Face

Private Hugging Face Model

Deploy a model with both Python and system dependencies

Model with system packages

Deploy a Truss on Baseten

Set your model resources, dependencies, and more

Config options

The simplest way to serve models in production

truss

truss init

truss push

truss watch

truss predict

Learn model deployment

Set up your development environment and create a Truss

Step 0: Create a Truss

Configure a remote host and push the base Truss

Step 1: Spin up a model server

Step 2: Call the model server

Patch the model server with local changes

Step 3: Enable live reload

Add required Python packages to the model server.

Step 4: Set Python requirements

Step 5: Implement model load

Add model inference and invoke the model server

Step 6: Implement model inference

Push your Truss to production-ready infrastructure

Step 7: Publish your Truss

Truss is MIT licensed and welcomes contributions

Learn

Truss 101: Model serving