Model Delivery

Guide · Open source · Apache 2.0

Keep your Hugging Face mirror up to date

Model Delivery is an open-source, self-hosted registry for AI model files: one program, modelctl, that you run on a machine of your own. This page shows how to make a mirror follow its upstream, so that when a branch on Hugging Face moves, your registry mirrors the new revision on a schedule or when your CI tells it to.

A mirror goes stale

A mirror is a copy of the revisions it fetched. When the model's main branch moves on Hugging Face, your copy of main stays where it was. You can run modelctl mirror again by hand, or from a cron job, but then you own the schedule, the exit codes and the question of what happened to the run nobody watched.

The registry can do that for you. Following is opt-in, per repository: a mirror stays as it was unless you follow it.

Follow a repository, and set a schedule

Two things turn it on: following the repository, and telling the server how often to check.

# Mirror main and follow it
modelctl mirror openai-community/gpt2 --follow

# Or mirror every branch and tag, and follow all of them
modelctl mirror openai-community/gpt2 --all-refs --follow

# On the server: check every followed repository this often
REGISTRY_SYNC_INTERVAL=1h modelctl serve    # or --sync-interval 1h

The schedule is off unless REGISTRY_SYNC_INTERVAL is set, and the shortest interval is five minutes. Each followed repository has its own place within the interval, so repositories followed together are not all checked at once, and they are checked one at a time. A check asks the upstream for the repository's branches and tags, one small request. It leaves a ref alone when the registry already has it at the same revision. When a followed branch has moved, the new revision is mirrored in full, every file verified, and only then does the branch move. Files the registry already stores are not downloaded again, so only what changed comes across.

A followed repository follows one branch (by default the one it has, or main) or every branch and tag. modelctl mirror --all-refs on its own mirrors every branch and tag once, without following. Following needs a token with the write role. The list of followed repositories is kept in the metadata database, so it survives a restart and is in every backup.

Only a repository that came from the upstream can be followed, by mirroring or by pull-through. One you created here by a push or an upload is refused: the registry never adds upstream content to a repository it did not create from the upstream.

What a sync does, and what it never does

A sync is a mirror, not a listing: after it, the new revision is available offline. It only adds.

Private and gated repositories are checked with the server's REGISTRY_UPSTREAM_TOKEN. A token passed to one mirror command is not stored, so a scheduled sync cannot use it.

Sync now, from CI

When you know the upstream has moved, for example in a job that runs after you publish a model, ask for a check instead of waiting for the schedule.

modelctl sync openai-community/gpt2    # waits; exits non-zero if the check failed
modelctl sync --all                    # starts a check of every followed repository, returns at once

The first form answers when the check is over and exits non-zero if it failed, so a CI job fails with it. It needs a token with the write role, set in MODELCTL_TOKEN like the other client commands. The REST API has the same checks over HTTP, for a webhook.

What failures look like

modelctl follow list     # what is followed, and how it went
modelctl follow stop openai-community/gpt2

A check that fails is recorded against the repository. The follow list shows the last error, when it happened and how many checks in a row have failed, and the log has a sync failed line. A failed check records no revision and moves no branch: a file that does not verify leaves the branch where it was. When the upstream caused the failure, the reason is shown; a failure of the registry's own, such as a full disk, is shown as no more than that and logged in full.

A repository whose checks keep failing is checked less often, at twice the wait after each failure and up to a day, while the others go on as usual. The first check that succeeds puts it back on the interval.

GET /metrics has modelregistry_sync_failures_total and modelregistry_sync_oldest_success_age_seconds, which are what to alert on. The operations guide's monitoring section has an alert, RegistryMirrorNotSyncing, for a followed repository that has not synced for three intervals.

Datasets

It works for dataset repositories too. Give the modelctl commands --repo-type dataset.

modelctl mirror --repo-type dataset stanfordnlp/imdb --follow
modelctl sync --repo-type dataset stanfordnlp/imdb

What it does not do

The full account is keeping mirrors up to date in the operations guide.

Related guides