Model Delivery

Guide · Open source · Apache 2.0

A pull-through cache for Hugging Face models, for CI and clusters

Model Delivery is an open-source, self-hosted registry for AI model files: one program, modelctl, that you run on a machine of your own. This page shows how to run it as a caching proxy in front of Hugging Face, so that build jobs and cluster nodes fetch each model file from Hugging Face once and from your own server after that.

When a cache fits better than a mirror

Continuous integration jobs and cluster nodes start from an empty disk and download the same models again and again. Each download depends on the route to huggingface.co, and a rate limit or an outage there fails a build that has nothing else wrong with it.

You can mirror the models ahead of time, which copies whole revisions. A pull-through cache needs no list of models in advance: clients point at the registry, and it fetches from Hugging Face what it does not hold, one requested file at a time. A job that wants one file of a large repository costs you one file.

Turn it on

Pull-through is off by default. One setting on the server turns it on, and clients need only HF_ENDPOINT.

# On the server
REGISTRY_PULL_THROUGH=true modelctl serve

# On each client, job or node
export HF_ENDPOINT=http://localhost:8080
hf download hf-internal-testing/tiny-random-bert config.json    # fetches that one file

It works at two levels. The first request for a revision the registry has never seen makes it ask Hugging Face for that revision's file listing: every path, size and digest. The first download of a file makes it fetch that file, check it against the digest from the listing, and store it. Later downloads of the file never reach Hugging Face.

Datasets work the same way: ask for one with repo_type="dataset" or --repo-type dataset, and the registry fetches its listing and files on demand.

Downloads of the same file share one fetch, so workers that start together and ask for the same file cause one download from Hugging Face. The web interface and modelctl inspect show which files of a revision have been fetched.

The first download of a file is sent while it is fetched

This is the part to know before you rely on it. The registry sends a file to a client while it is still fetching it from Hugging Face, so the first download starts at once and takes about as long as the fetch does. Clients that ask while the fetch is running get the file from the same fetch, from its beginning, and clients that ask afterwards are served from storage. If Hugging Face itself sends nothing for longer than the client's timeout (ten seconds between bytes for huggingface_hub), that request times out; the fetch carries on, so download again.

Bytes go out before the check, because the check needs the whole file. The end is held back: the last mebibyte is not sent until the whole file has arrived and matched its digest and size, so a client never receives a complete file that failed the check. If the check fails, the registry cuts the download short, which clients report as an error, and nothing is stored. Files of a mebibyte or less, which covers most configuration and tokenizer files, are never sent before they are verified.

Two cases are not covered. A download tool that keeps a cut-off file and appends to it later, instead of replacing it as huggingface_hub and modelctl pull do, can end with a damaged file. A restart of the registry between the cut and the client's retry has the same effect. If either matters to you more than the speed of the first download, turn streaming off:

REGISTRY_PULL_THROUGH_STREAM=false    # or --pull-through-stream=false

The registry then sends nothing until the file is stored and verified, and the first request for a large file can time out. The fetch carries on, and a retry once it has finished is served at once. To have a whole model on hand before anyone asks, or to have it available offline, mirror it:

modelctl mirror meta-llama/Llama-3.2-1B

modelctl mirror fetches everything still missing from a revision and reuses what pull-through already stored. The operations guide has the full account of the first download of a file.

Limit what can be fetched, and by whom

Anyone who is allowed to download can trigger a fetch. On a private network that is what you want. Anywhere else, an open registry with pull-through lets any visitor make it download and store any public file, which fills your disk and uses your bandwidth. The server logs a warning at start if pull-through is on with neither of the first two controls below.

# Only token holders can read, and therefore fetch
export REGISTRY_REQUIRE_AUTH=true

# Only these repositories are fetched; others are served only if already stored
export REGISTRY_PULL_THROUGH_ALLOW='meta-llama/*,google/*,datasets/acme/*'

# Largest file fetched on demand, in bytes (here 20 GiB); unset or 0 is no limit
export REGISTRY_PULL_THROUGH_MAX_FILE_BYTES=21474836480

REGISTRY_PULL_THROUGH=true modelctl serve

Each setting is also a flag: --pull-through, --pull-through-allow, --pull-through-max-file-bytes, --pull-through-stream and --require-auth.

To see what the cache is doing, GET /metrics is a Prometheus endpoint with the cache hit ratio and upstream failures; no metric names a repository or a file. The operations guide has monitoring, with alerts for a failing upstream and a full fetch queue.

Private models, gated models and your own

Private and gated models are fetched with the server's REGISTRY_UPSTREAM_TOKEN, a Hugging Face token that has access to them. Without one, such a request fails with an error saying so. Once fetched, who can read the model is decided by your registry's settings; honour the model's licence and access terms.

A repository you created in the registry, by pushing or uploading, is never added to from Hugging Face. For a name the registry does not hold yet, the allow list is the control: without one, the first client to ask for my-team/model before you have pushed it gets whatever Hugging Face has under that name. If you push your own models to a registry that has pull-through on, set the allow list to the Hugging Face namespaces you use and leave your own out.

What the cache does not do

The operations guide has the full account of pull-through caching, and the architecture page says how it works inside.

Related guides