Guide · Open source · Apache 2.0
A pull-through cache for Hugging Face models, for CI and clusters
Model Delivery is an open-source, self-hosted registry for AI model files: one program, modelctl, that you run on a machine of your own. This page shows how to run it as a caching proxy in front of Hugging Face, so that build jobs and cluster nodes fetch each model file from Hugging Face once and from your own server after that.
When a cache fits better than a mirror
Continuous integration jobs and cluster nodes start from an empty disk and download the same models again and again. Each download depends on the route to huggingface.co, and a rate limit or an outage there fails a build that has nothing else wrong with it.
You can mirror the models ahead of time, which copies whole revisions. A pull-through cache needs no list of models in advance: clients point at the registry, and it fetches from Hugging Face what it does not hold, one requested file at a time. A job that wants one file of a large repository costs you one file.
Turn it on
Pull-through is off by default. One setting on the server turns it on, and clients need only HF_ENDPOINT.
# On the server
REGISTRY_PULL_THROUGH=true modelctl serve
# On each client, job or node
export HF_ENDPOINT=http://localhost:8080
hf download hf-internal-testing/tiny-random-bert config.json # fetches that one file
It works at two levels. The first request for a revision the registry has never seen makes it ask Hugging Face for that revision's file listing: every path, size and digest. The first download of a file makes it fetch that file, check it against the digest from the listing, and store it. Later downloads of the file never reach Hugging Face.
Datasets work the same way: ask for one with repo_type="dataset" or --repo-type dataset, and the registry fetches its listing and files on demand.
Downloads of the same file share one fetch, so workers that start together and ask for the same file cause one download from Hugging Face. The web interface and modelctl inspect show which files of a revision have been fetched.
The first download of a file is sent while it is fetched
This is the part to know before you rely on it. The registry sends a file to a client while it is still fetching it from Hugging Face, so the first download starts at once and takes about as long as the fetch does. Clients that ask while the fetch is running get the file from the same fetch, from its beginning, and clients that ask afterwards are served from storage. If Hugging Face itself sends nothing for longer than the client's timeout (ten seconds between bytes for huggingface_hub), that request times out; the fetch carries on, so download again.
Bytes go out before the check, because the check needs the whole file. The end is held back: the last mebibyte is not sent until the whole file has arrived and matched its digest and size, so a client never receives a complete file that failed the check. If the check fails, the registry cuts the download short, which clients report as an error, and nothing is stored. Files of a mebibyte or less, which covers most configuration and tokenizer files, are never sent before they are verified.
Two cases are not covered. A download tool that keeps a cut-off file and appends to it later, instead of replacing it as huggingface_hub and modelctl pull do, can end with a damaged file. A restart of the registry between the cut and the client's retry has the same effect. If either matters to you more than the speed of the first download, turn streaming off:
REGISTRY_PULL_THROUGH_STREAM=false # or --pull-through-stream=false
The registry then sends nothing until the file is stored and verified, and the first request for a large file can time out. The fetch carries on, and a retry once it has finished is served at once. To have a whole model on hand before anyone asks, or to have it available offline, mirror it:
modelctl mirror meta-llama/Llama-3.2-1B
modelctl mirror fetches everything still missing from a revision and reuses what pull-through already stored. The operations guide has the full account of the first download of a file.
Limit what can be fetched, and by whom
Anyone who is allowed to download can trigger a fetch. On a private network that is what you want. Anywhere else, an open registry with pull-through lets any visitor make it download and store any public file, which fills your disk and uses your bandwidth. The server logs a warning at start if pull-through is on with neither of the first two controls below.
# Only token holders can read, and therefore fetch
export REGISTRY_REQUIRE_AUTH=true
# Only these repositories are fetched; others are served only if already stored
export REGISTRY_PULL_THROUGH_ALLOW='meta-llama/*,google/*,datasets/acme/*'
# Largest file fetched on demand, in bytes (here 20 GiB); unset or 0 is no limit
export REGISTRY_PULL_THROUGH_MAX_FILE_BYTES=21474836480
REGISTRY_PULL_THROUGH=true modelctl serve
- The allow list is a comma-separated list of patterns over the repository id. It limits which repositories can be fetched, not how much of them. A dataset is matched as
datasets/namespace/name, sodatasets/acme/*names datasets andacme/*models only. - The size limit refuses a download of a larger file with an error that names the setting and the
modelctl mirrorcommand that fetches the file deliberately. Listings and file metadata for such a file still answer. - The queue is fixed: at most four fetches run at a time and at most 64 more wait their turn. When it is full, a request that would start another fetch is answered
503withRetry-After. Requests for files already stored are not affected. - With
REGISTRY_REQUIRE_AUTH, clients send a registry token inHF_TOKEN. Create one withmodelctl token create ci --role read.
Each setting is also a flag: --pull-through, --pull-through-allow, --pull-through-max-file-bytes, --pull-through-stream and --require-auth.
To see what the cache is doing, GET /metrics is a Prometheus endpoint with the cache hit ratio and upstream failures; no metric names a repository or a file. The operations guide has monitoring, with alerts for a failing upstream and a full fetch queue.
Private models, gated models and your own
Private and gated models are fetched with the server's REGISTRY_UPSTREAM_TOKEN, a Hugging Face token that has access to them. Without one, such a request fails with an error saying so. Once fetched, who can read the model is decided by your registry's settings; honour the model's licence and access terms.
A repository you created in the registry, by pushing or uploading, is never added to from Hugging Face. For a name the registry does not hold yet, the allow list is the control: without one, the first client to ask for my-team/model before you have pushed it gets whatever Hugging Face has under that name. If you push your own models to a registry that has pull-through on, set the allow list to the Hugging Face namespaces you use and leave your own out.
What the cache does not do
- It does not refresh. It fills gaps. A branch the registry already holds is served as recorded, even if it has moved on Hugging Face. Clients that pin a revision id always get exactly that revision. To pick up a newer
main, runmodelctl mirror. - It does not make a model available offline. A file that was never downloaded still depends on Hugging Face: if it is unreachable, files already fetched are served and the others fail with an error saying so. For a network with no route out, see offline and air-gapped networks.
- The first download is sent before it is fully verified. The end of the file is held back until the whole file has matched its digest, and a failed check cuts the download short. See above;
--pull-through-stream=falsesends nothing until the file is verified. - There is no total quota. The size limit bounds one file, not the disk. Watch free space.
- There is no built-in rate limiting. Use a reverse proxy for that if the registry is reachable by people you do not know.
- No Spaces, one server. Models and datasets are cached; Spaces are not supported. There is no replication or high availability.
The operations guide has the full account of pull-through caching, and the architecture page says how it works inside.
Related guides
- Run your own Hugging Face mirrorCopy whole models ahead of time, so the first download needs no fetch.
- Offline and air-gapped networksServe models where there is no route to huggingface.co.
- S3 and Cloudflare R2 storageKeep the cached files in a bucket instead of on the server's disk.
- Model DeliveryWhat the registry is, and everything else it does.