Model Delivery

Guide · Open source · Apache 2.0

Keep model files in S3 or Cloudflare R2

Model Delivery is an open-source, self-hosted registry for AI model files that Hugging Face clients download from by setting HF_ENDPOINT. This page shows how to keep the model files in an S3-compatible bucket instead of on the server's disk, how to let clients download straight from the bucket, and how the registry backs up its metadata into the same bucket.

Why put the files in a bucket

Model files are large and there are many of them. On local disk, which is the default, the server needs a volume big enough for all of them, and that volume is the thing to protect. With S3 storage the files live in a bucket, and the server's disk holds a small database and scratch space.

Storage works the same way in both cases: each file is stored once, under the SHA-256 of its content, however many revisions or repositories contain it.

The settings

export REGISTRY_STORAGE=s3
export REGISTRY_S3_BUCKET=my-models
export REGISTRY_S3_ENDPOINT=https://<account id>.r2.cloudflarestorage.com   # Cloudflare R2
export REGISTRY_S3_REGION=auto                                              # R2's region is "auto"
export REGISTRY_S3_ACCESS_KEY=...
export REGISTRY_S3_SECRET_KEY=...
modelctl serve

For another service, set the endpoint to its URL and the region to the one it needs; with no endpoint set, the default is AWS. The bucket must already exist: the registry does not create it. It checks the bucket at start and exits with a clear message if it cannot reach it.

Everything but the two keys can also be given as a flag (--storage s3 --s3-bucket my-models and so on). The keys are read from the environment only. If they are unset, the registry uses AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and failing those the role of the instance or pod it runs on. REGISTRY_S3_PREFIX puts every object under a key prefix, to share a bucket. The operations guide lists the bucket permissions the registry needs.

Which services it has been tested with

Two: Cloudflare R2 and SeaweedFS. The project's automated tests run against SeaweedFS on every change.

The registry uses the standard S3 API, so AWS S3, Backblaze B2, Ceph and MinIO are expected to work, but they have not been tested.

What still lives on the server

Redirected downloads

By default the registry streams each file from the bucket to the client, so all file traffic passes through it. One setting changes that:

export REGISTRY_S3_REDIRECT=true     # or pass --s3-redirect

The registry then answers a download with a redirect to a signed bucket URL, valid for one hour. The client downloads directly from the object store and the registry carries no file traffic. It still checks the caller's access before it signs a URL. huggingface_hub is tested in this mode.

Turn it on only when both of these hold:

Metadata backups in the same bucket

The model files are in the bucket, but the database that names them is on the server. The registry backs that database up itself, on a schedule, and with S3 storage it stores the backups in the same bucket, under backups/. That puts them off the server.

export REGISTRY_BACKUP_INTERVAL=1h       # off unless set
export REGISTRY_BACKUP_KEEP=48           # how many to keep; 48 is the default
export REGISTRY_RESTORE_IF_EMPTY=true    # at start, with no database, restore the newest backup

Every interval the server takes a consistent snapshot while it keeps serving, runs SQLite's integrity check on it, and stores it. The model files are not copied: they are already in the bucket under their digests and never change.

With REGISTRY_RESTORE_IF_EMPTY, a server that starts with no database restores the newest stored backup before serving, so a server whose disk was replaced comes back with its catalogue. With no backup either, it starts as a new registry. A data directory that already has a database is never touched. To restore by hand, stop the registry and run modelctl restore --latest with the same storage settings as the server, so that it finds the stored backups.

Three things to know about a restore:

The operations guide has the detail on backup and restore. GET /metrics, a Prometheus endpoint, reports the age of the newest backup, and the guide's monitoring section has an alert for one that is too old.

What it does not do

Reference: S3 storage in the operations guide, the settings table in the README, and how files are stored.

Related guides