Guide · Open source · Apache 2.0
Keep your Hugging Face mirror up to date
Model Delivery is an open-source, self-hosted registry for AI model files: one program, modelctl, that you run on a machine of your own. This page shows how to make a mirror follow its upstream, so that when a branch on Hugging Face moves, your registry mirrors the new revision on a schedule or when your CI tells it to.
A mirror goes stale
A mirror is a copy of the revisions it fetched. When the model's main branch moves on Hugging Face, your copy of main stays where it was. You can run modelctl mirror again by hand, or from a cron job, but then you own the schedule, the exit codes and the question of what happened to the run nobody watched.
The registry can do that for you. Following is opt-in, per repository: a mirror stays as it was unless you follow it.
Follow a repository, and set a schedule
Two things turn it on: following the repository, and telling the server how often to check.
# Mirror main and follow it
modelctl mirror openai-community/gpt2 --follow
# Or mirror every branch and tag, and follow all of them
modelctl mirror openai-community/gpt2 --all-refs --follow
# On the server: check every followed repository this often
REGISTRY_SYNC_INTERVAL=1h modelctl serve # or --sync-interval 1h
The schedule is off unless REGISTRY_SYNC_INTERVAL is set, and the shortest interval is five minutes. Each followed repository has its own place within the interval, so repositories followed together are not all checked at once, and they are checked one at a time. A check asks the upstream for the repository's branches and tags, one small request. It leaves a ref alone when the registry already has it at the same revision. When a followed branch has moved, the new revision is mirrored in full, every file verified, and only then does the branch move. Files the registry already stores are not downloaded again, so only what changed comes across.
A followed repository follows one branch (by default the one it has, or main) or every branch and tag. modelctl mirror --all-refs on its own mirrors every branch and tag once, without following. Following needs a token with the write role. The list of followed repositories is kept in the metadata database, so it survives a restart and is in every backup.
Only a repository that came from the upstream can be followed, by mirroring or by pull-through. One you created here by a push or an upload is refused: the registry never adds upstream content to a repository it did not create from the upstream.
What a sync does, and what it never does
A sync is a mirror, not a listing: after it, the new revision is available offline. It only adds.
- It never removes or changes an earlier revision. A revision id that was valid stays valid and serves the same files, so anything pinned to a revision id is unaffected by a sync.
- It never deletes anything. A branch or tag deleted upstream stays in the registry at the revision it had. The check reports it as
deleted_upstream. Delete the repository if you want it gone. - It never moves a tag. Upstream tags are recorded as tags, and a tag that the upstream moves to another revision is reported as
tag_moved, with a warning in the log, while your registry's tag stays where it was. Follow a branch to track something that moves.
Private and gated repositories are checked with the server's REGISTRY_UPSTREAM_TOKEN. A token passed to one mirror command is not stored, so a scheduled sync cannot use it.
Sync now, from CI
When you know the upstream has moved, for example in a job that runs after you publish a model, ask for a check instead of waiting for the schedule.
modelctl sync openai-community/gpt2 # waits; exits non-zero if the check failed
modelctl sync --all # starts a check of every followed repository, returns at once
The first form answers when the check is over and exits non-zero if it failed, so a CI job fails with it. It needs a token with the write role, set in MODELCTL_TOKEN like the other client commands. The REST API has the same checks over HTTP, for a webhook.
What failures look like
modelctl follow list # what is followed, and how it went
modelctl follow stop openai-community/gpt2
A check that fails is recorded against the repository. The follow list shows the last error, when it happened and how many checks in a row have failed, and the log has a sync failed line. A failed check records no revision and moves no branch: a file that does not verify leaves the branch where it was. When the upstream caused the failure, the reason is shown; a failure of the registry's own, such as a full disk, is shown as no more than that and logged in full.
A repository whose checks keep failing is checked less often, at twice the wait after each failure and up to a day, while the others go on as usual. The first check that succeeds puts it back on the interval.
GET /metrics has modelregistry_sync_failures_total and modelregistry_sync_oldest_success_age_seconds, which are what to alert on. The operations guide's monitoring section has an alert, RegistryMirrorNotSyncing, for a followed repository that has not synced for three intervals.
Datasets
It works for dataset repositories too. Give the modelctl commands --repo-type dataset.
modelctl mirror --repo-type dataset stanfordnlp/imdb --follow
modelctl sync --repo-type dataset stanfordnlp/imdb
What it does not do
- It is opt-in and off by default. A repository is followed only when you say so, and nothing is checked on a schedule until
REGISTRY_SYNC_INTERVALis set. Without it you can still ask for a sync. - It only adds. A branch or tag deleted upstream stays in the registry, and a tag moved upstream is reported, not moved.
- Pull-through alone never refreshes a branch. A request for a branch the registry holds is answered with what it holds. A followed branch is refreshed by its sync and by nothing else.
- Only repositories that came from upstream. A repository you pushed cannot be followed.
- One upstream. Every mirror, sync and pull-through fetch goes to the one hub the registry is set to use.
- One server. There is no replication or high availability.
The full account is keeping mirrors up to date in the operations guide.
Related guides
- Run your own Hugging Face mirrorThe mirror commands in full, with private and gated models.
- Pull-through cacheLet the registry fetch each file the first time it is asked for.
- Offline and air-gapped networksMirror on a connected machine, serve where there is no route out.
- S3 and Cloudflare R2 storageKeep the model files in a bucket instead of on the server's disk.
- Model DeliveryWhat the registry is, and everything else it does.