Guide · Open source · Apache 2.0
Keep model files in S3 or Cloudflare R2
Model Delivery is an open-source, self-hosted registry for AI model files that Hugging Face clients download from by setting HF_ENDPOINT. This page shows how to keep the model files in an S3-compatible bucket instead of on the server's disk, how to let clients download straight from the bucket, and how the registry backs up its metadata into the same bucket.
Why put the files in a bucket
Model files are large and there are many of them. On local disk, which is the default, the server needs a volume big enough for all of them, and that volume is the thing to protect. With S3 storage the files live in a bucket, and the server's disk holds a small database and scratch space.
Storage works the same way in both cases: each file is stored once, under the SHA-256 of its content, however many revisions or repositories contain it.
The settings
export REGISTRY_STORAGE=s3
export REGISTRY_S3_BUCKET=my-models
export REGISTRY_S3_ENDPOINT=https://<account id>.r2.cloudflarestorage.com # Cloudflare R2
export REGISTRY_S3_REGION=auto # R2's region is "auto"
export REGISTRY_S3_ACCESS_KEY=...
export REGISTRY_S3_SECRET_KEY=...
modelctl serve
For another service, set the endpoint to its URL and the region to the one it needs; with no endpoint set, the default is AWS. The bucket must already exist: the registry does not create it. It checks the bucket at start and exits with a clear message if it cannot reach it.
Everything but the two keys can also be given as a flag (--storage s3 --s3-bucket my-models and so on). The keys are read from the environment only. If they are unset, the registry uses AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and failing those the role of the instance or pod it runs on. REGISTRY_S3_PREFIX puts every object under a key prefix, to share a bucket. The operations guide lists the bucket permissions the registry needs.
Which services it has been tested with
Two: Cloudflare R2 and SeaweedFS. The project's automated tests run against SeaweedFS on every change.
The registry uses the standard S3 API, so AWS S3, Backblaze B2, Ceph and MinIO are expected to work, but they have not been tested.
What still lives on the server
registry.db, the metadata. It is a small SQLite file, and it is the only record of which objects make up which revision. A bucket without its database is a pile of unnamed files. Keep the data directory on a persistent, local disk; SQLite over a network filesystem such as NFS is unreliable.tmp/, scratch space. Each uploaded or mirrored file is received and verified on local disk before it is sent to the bucket. You need free space for the largest file being received, times the number received at once; a mirror fetches four files in parallel.
Redirected downloads
By default the registry streams each file from the bucket to the client, so all file traffic passes through it. One setting changes that:
export REGISTRY_S3_REDIRECT=true # or pass --s3-redirect
The registry then answers a download with a redirect to a signed bucket URL, valid for one hour. The client downloads directly from the object store and the registry carries no file traffic. It still checks the caller's access before it signs a URL. huggingface_hub is tested in this mode.
Turn it on only when both of these hold:
- Clients can reach the bucket endpoint. If the registry reaches the bucket by an internal address that clients cannot use, leave redirects off.
- A signed URL is an acceptable grant. Anyone holding one can download that one file until it expires, without a registry token.
Metadata backups in the same bucket
The model files are in the bucket, but the database that names them is on the server. The registry backs that database up itself, on a schedule, and with S3 storage it stores the backups in the same bucket, under backups/. That puts them off the server.
export REGISTRY_BACKUP_INTERVAL=1h # off unless set
export REGISTRY_BACKUP_KEEP=48 # how many to keep; 48 is the default
export REGISTRY_RESTORE_IF_EMPTY=true # at start, with no database, restore the newest backup
Every interval the server takes a consistent snapshot while it keeps serving, runs SQLite's integrity check on it, and stores it. The model files are not copied: they are already in the bucket under their digests and never change.
With REGISTRY_RESTORE_IF_EMPTY, a server that starts with no database restores the newest stored backup before serving, so a server whose disk was replaced comes back with its catalogue. With no backup either, it starts as a new registry. A data directory that already has a database is never touched. To restore by hand, stop the registry and run modelctl restore --latest with the same storage settings as the server, so that it finds the stored backups.
Three things to know about a restore:
- A backup is a point in time. Revisions, tokens and deletions recorded after it are not in it: at most one interval of changes with scheduled backups. The files of lost revisions are still in the bucket, but the registry no longer knows them; push or mirror those revisions again.
- Revoked tokens come back. A token you revoked after the backup was taken works again after a restore, the automatic one included. Run
modelctl token listafterwards and revoke again what should not be there. - The bucket holds the catalogue. A stored backup contains token hashes, not the tokens, and the full list of what the registry holds. Anyone who can read the bucket can read it.
The operations guide has the detail on backup and restore. GET /metrics, a Prometheus endpoint, reports the age of the newest backup, and the guide's monitoring section has an alert for one that is too old.
What it does not do
- It is still one server. The files are in the bucket, but the metadata is one SQLite file, so you run a single instance. There is no replication or high availability. Do not point two registries at one bucket and prefix: each would remove files the other still refers to.
- It does not migrate existing data. There is no command that moves a registry from local disk into a bucket. The operations guide describes copying the files by hand.
- It does not want a lifecycle rule that expires objects. The registry decides which files are unreferenced, with
modelctl gc. A rule that aborts incomplete multipart uploads after a day or two is worth having. - Checking stored files reads all of them.
modelctl verifydownloads every object through the registry, which your storage provider may charge for. - Backups cover the database, not the model files. The registry does not copy model files anywhere else; protecting the bucket is up to you.
Reference: S3 storage in the operations guide, the settings table in the README, and how files are stored.
Related guides
- Run your own Hugging Face mirrorCopy models from Hugging Face and serve them with
HF_ENDPOINT. - Pull-through cacheLet the registry fetch each file the first time it is asked for.
- Offline and air-gapped networksServe models where there is no route to huggingface.co.
- Model DeliveryWhat the registry is, and everything else it does.