Contributing
Use this guide when changing mirror.py: set up the development environment, follow the Python conventions below, preserve the master-worker and persistence contracts, and verify the behavior affected by your change. These conventions apply to new and modified code; keep unrelated code unchanged.
Start with the contribution guidelines for the required issue discussion and PR workflow. Begin implementation once the issue discussion has settled the problem, scope, and development direction. Use this page for the technical development conventions and checks.
Development setup
mirror.py requires Python 3.10 or later and runs on Linux only. The project uses uv for
package management.
Install the package in editable mode:
uv sync --group docs
The default dev dependency group includes pytest and pytest-docker. It replaces
the former .[dev] extra; use uv sync for development without documentation
dependencies. Integration images install the local wheel and its apt-mirror2
extra, so Python tool versions follow project metadata rather than separate pins.
Dependency updates retain Python 3.10 and Node 22.12 compatibility. Bandersnatch
uses environment markers: 6.5.0 on Python 3.10, 6.6.0 on Python 3.11, and 8.0.0
on Python 3.12 and later. Sphinx 8.1.3 and MyST Parser 4.0.1 remain pinned for
Python 3.10 support. Resolve Python and npm lockfiles when updating dependencies,
then run the unit, integration, editor, and documentation checks below.
Runtime setuptools is limited to 81.x only on Python 3.10 because bandersnatch
6.5 imports pkg_resources, which setuptools 82 removed. Isolated package builds
use the current setuptools build backend independently.
Run the unit test suite:
uv run pytest
Run the integration tests (requires Docker):
uv run pytest -m integration
Run the default and integration suites sequentially because they can share
temporary resources. The default run excludes tests marked integration.
Change scope and design
Read the relevant implementation, callers, and tests before editing. Define the expected result and the smallest check that can prove it. For multi-step changes, use a short plan with a verification step for each outcome.
Modify only files and lines needed for the requested behavior. Match the surrounding style without reformatting or refactoring adjacent code.
Prefer the existing responsible module, helper, type, or dependency. Add a file or abstraction only when the behavior or architecture requires it.
Avoid speculative options, extension points, and handling for impossible cases. Refactor when ownership is unclear, not merely to reduce line count.
Remove imports, variables, and functions made unused by your change. Report unrelated dead code rather than removing it.
Resolve ambiguity before changing public behavior, security, persisted data, or another outcome that is difficult to reverse. State reasonable assumptions for routine implementation choices.
Prefer the standard library and installed dependencies. Obtain agreement before introducing a new third-party package.
Python coding style
Functions and module structure
Write reusable functions with one responsibility. Name helpers with an action
and subject, such as parse_date_string or build_sync_command. Pass inputs
explicitly, return results explicitly, and keep side effects at the layer that
owns them. Extract meaningful logic into named functions that can be used
independently without hidden external state.
Keep worker functions short. A coordinator may remain longer when that keeps the execution sequence clear; do not split it into artificial helpers. Use comments to explain meaningful stages where needed. Prefer flat modules and functions when classes do not provide necessary state or behavior. Preserve the project’s existing dataclasses and module ownership.
Avoid new global dependencies. If a function uses global state, declare the global at the beginning of the function and keep its ownership explicit.
Formatting and annotations
Follow PEP 8 except for its line-length guidance. Keep every
defsignature on one line, regardless of length, including its type annotations.Annotate every function’s parameters and return value. Use
-> Nonefor functions that do not return a value, and keep syntax compatible with Python 3.10.Use descriptive English identifiers. Prefer explicit steps over nested conditional expressions or dense comprehensions.
Write comments in English. Favor self-explanatory code; when explanation is needed, put a block comment above the relevant code and explain the constraint or reason. Avoid decorative headings and comments that repeat the code.
Keep imports consistent with the module’s neighbors. Sync modules start with imports rather than a module docstring and place
mirrorimports before standard-library imports.Do not use emojis in identifiers, comments, logs, error messages, CLI or UI output, or commit messages.
Docstrings
Every function used outside its module needs a full English docstring with a
one-line description, every argument, and the return value. Types belong in the
annotated signature. Internal helpers may use a one-line docstring. Use Args:
and Returns: sections consistently across the project, including sync modules.
For example, a boundary validator can be documented as follows:
def validate_retry_count(value: int) -> int:
"""Validate the configured retry count.
Args:
value: Requested number of retries.
Returns:
The validated retry count.
Raises:
ValueError: If the value is not an integer or is negative.
"""
if type(value) is not int or value < 0:
raise ValueError("Retry count must be a non-negative integer")
return value
Validation and error handling
Validate untrusted input at system boundaries: configuration loading, socket requests, remote metadata, and external-tool command construction. Preserve existing authentication, authorization, path checks, signature verification, metadata size and time limits, and concurrency protections.
Raise specific exceptions with clear messages. Never swallow exceptions silently. Catch failures at the layer that can report or recover from them, and preserve the existing sync completion and cleanup behavior.
Pass external commands as argument lists rather than interpolated shell strings. Tools that interpret their own command or configuration language still need validation and escaping for that language. Keep credentials out of logs and error messages.
Logging and terminal output
Use the existing mirror.logger facilities and per-package loggers. Preserve
their formatting, rotation, and compression rather than introducing separate
handlers or bare print() calls. Use the existing Click or prompt_toolkit
output path for CLI interactions, with a plain-text fallback for styled output.
Use uv for Python package management. When selecting Python explicitly,
specify a version, not an interpreter path. Leave other uv settings at their
defaults, including its cache location.
Dependencies and separate services
Prefer the standard library and existing dependencies. Do not introduce dependencies that require separately deployed or managed services, such as Redis or MySQL, unless the requirement and adoption have been explicitly agreed in the issue before implementation. This applies both to direct service integrations and to libraries that require such services.
Use the existing filesystem, JSON state, and Unix sockets when they meet the requirement. Evaluate dependencies by the services they require and the operational burden they introduce, rather than by the number of imports.
Architecture and state ownership
The master schedules syncs and owns package state. The worker owns sync subprocesses, allowing them to continue when the master restarts. Read the architecture overview before changing this flow.
Area |
Responsible module |
|---|---|
CLI dispatch and command implementations |
|
Configuration loading and state persistence |
|
Unix socket RPC and completion notifications |
|
Sync preparation and lifecycle |
|
Subprocess creation, tracking, and pruning |
|
Events, logging, and plugin registration |
|
/etc/mirror/config.json is operator-managed and read-only during daemon and
worker runtime. Only initial provisioning through mirror setup writes it.
Runtime status, error counts, log paths, and timestamps belong in stat.json;
there is intentionally no Config.save().
When adding a persisted status field, extend Package.StatusInfo or the
appropriate state dataclass, preserve loading through from_dict(), and expose
the field through Package.to_dict() so save_stat_data() persists it. If web
clients need it, also update generate_and_save_web_status(). Preserve atomic
state writes and locking. See state files.
Developing sync methods
Read mirror/sync/rsync.py, ftpsync.py, and lftp.py before changing a sync
method. Keep method-specific discovery and helpers in mirror/sync/<method>.py
unless the architecture requires sharing them. Place constants and data types
near the top, keep setup and execute easy to find, and group related helpers.
Separate command building from execution. A builder returns arguments and environment settings without network, filesystem, or worker side effects.
Prepare jobs through
execute(package, logger, trigger="auto")and delegate long-running subprocesses tomirror.socket.worker.execute_command(). Preserve the package job ID, configured UID/GID, environment, and log path. Discovery requiring the worker’s identity or environment belongs there too.Report failures before delegation through
mirror.sync.on_sync_done(). After delegation, let worker completion drive the normal result lifecycle.Give generated tool configurations an explicit temporary-directory owner. Keep them available and accessible to the worker until execution finishes, then clean up on success, failure, and handled termination.
Expose the method through
plugin()usingmirror.plugin.sync_pluginand register built-in methods in themirror.syncentry points inpyproject.toml. See the plugin author guide.
Verification and review
Start with the smallest test that exercises the changed behavior, for example
uv run pytest tests/test_sync_worker_delegation.py for worker delegation.
Broaden testing when the affected scope, a failure, or an unresolved risk
requires it. Add tests for behavior changes, regressions, trust boundaries, and
important edge cases. Do not add tests that merely restate a refactor or a
reversible, low-impact edit.
For sync behavior changes, cover command building and worker delegation with unit tests and real synchronization with the relevant integration tests. For readability-only code changes, use existing tests to demonstrate that behavior is preserved. For prose-only documentation changes, build the documentation.
Before submitting a change, inspect the diff for scope, unused code introduced
by the change, credential exposure, and missing documentation. Describe the
resulting behavior, the reason for the change, checks run, and any remaining
limitations. Use Conventional Commit subjects, for example
fix: preserve worker job state after master restart.
Building documentation
The configuration editor requires Node.js 22.12 or later and npm. From a clean checkout, install its locked dependencies, build its local assets, and build the Sphinx site with one command:
npm --prefix docs/editor run docs:build
The generated editor assets are ignored by Git. An HTML build fails with a clear
message if they are absent. After the initial build, rerun npm --prefix docs/editor run build when editing frontend sources; prose-only changes can
use the normal uv run --group docs sphinx-build -b html -W --keep-going docs docs/_build/html command.
Serve the site locally instead of opening file:// URLs, because Monaco uses
Web Workers:
python -m http.server 8000 --directory docs/_build/html
Open http://localhost:8000/guide/config-editor.html. Editor assets and workers
are bundled with the site; no CDN or external schema service is used.
Run editor unit and browser tests:
npm --prefix docs/editor test
uv run pytest tests/test_editor_contract.py
cd docs/editor
npx playwright install chromium firefox
npm run test:browser
The editor schema is shared by the forms and Monaco. Python loaders and sync validation remain authoritative. When changing configuration fields, update the editor schema and its contract fixtures as well as the reference docs.
Deploying documentation
Configure Cloudflare Pages with the repository root as the root directory:
Setting |
Value |
|---|---|
Build command |
|
Build output directory |
|
Build variable |
|
Build variable |
|
The build command installs uv and the editor’s locked npm dependencies, then
builds the editor and Sphinx HTML. Cloudflare publishes the output directory.
Keeping sync-method docs in sync
The option pages under docs/sync-methods/ and config.md document the configuration
fields accepted by each sync module. When you change a sync module’s options — adding, removing,
or renaming a field — you must update both the relevant sync-methods/ page and config.md
in the same pull request.
Integration testing
Integration Tests
End-to-end tests for mirror.py running against real rsync, ftpsync (archvsync), lftp, debmirror, and apt-mirror2 upstream fixtures in Docker. Verifies that the master-worker daemon pair correctly performs syncs, recovers from process restarts, and persists state — none of which is provable with the in-process unit suite under tests/.
These tests are deselected by default (pyproject.toml sets addopts = -m 'not integration'). Run them explicitly:
uv sync
uv run pytest -m integration -v
The mirror image installs a wheel built from the current source tree by a
session-scoped fixture (built_wheel in conftest.py). This means tests
exercise in-progress changes immediately — no PyPI round-trip needed. The
wheel is placed in docker/mirror/dist/ (gitignored) and rebuilt only when
the source SHA changes.
First run builds the wheel and five Docker images. Subsequent runs reuse both the wheel and cached images.
Container topology
Five containers share the default bridge network defined in docker-compose.yml:
mirror, rsync-fixture, ftpsync-fixture, lftp-fixture, and apt-fixture.
The diagram below shows the original rsync/ftpsync paths; the shared APT fixture
serves both debmirror and apt-mirror2 over HTTP port 8000 and FTP port 2121.
┌────────────────────┐ ┌────────────────────┐
│ rsync-fixture │ │ ftpsync-fixture │
│ ubuntu + rsyncd │ │ ubuntu + rsyncd │
│ module [data] │ │ module [debian] │
└─────────▲──────────┘ └──────────▲─────────┘
│ rsync:// │ rsync://
└────────────┬─────────────┘
│
┌─────────────┴───────────────────────────┐
│ mirror (ubuntu:22.04, python3.10) │
│ systemd (PID 1) │
│ ├─ mirror-worker.service │
│ └─ mirror.service │
│ /var/run/mirror/{master,worker}.sock │
│ │
│ bind-mounts to host: │
│ /srv/publish ─► ${TMP}/publish │
│ /var/lib/mirror ─► ${TMP}/state │
│ /var/log/mirror ─► ${TMP}/log │
└─────────────────────────────────────────┘
${TMP} is a per-session host temp dir (set as the INTEGRATION_TMP env var by conftest.py). pytest reads stat.json, package logs, and the published mirror tree directly from these bind-mounted paths.
Container names are pinned (container_name: mirror|rsync-fixture|ftpsync-fixture) so docker exec mirror … works regardless of the compose project name.
Why master and worker are split
Worker spawns the actual sync subprocesses (rsync, ftpsync) as its own children. Master only schedules and tracks state. When master restarts:
Worker keeps running.
Already-running rsync/ftpsync subprocesses keep running (parent is worker, not master).
New master reconnects to worker via
worker.sockand resumes receivingjob_finishednotifications.
This is what test_master_restart.py proves: capture worker PID, restart master mid-sync, assert worker PID is unchanged and the sync still reaches ACTIVE.
How tests interact with the stack
All test interactions go through the mirror_stack fixture (defined in conftest.py, implemented in helpers.py). No test imports mirror.* directly — the test process talks to the containerized version exclusively.
Action |
Mechanism |
|---|---|
Trigger sync |
|
Wait for status |
|
Restart process |
|
Inspect publish tree |
Read |
Swap upstream content |
|
Clone failure injection |
Temporary git wrapper inside the mirror container; fixture networking remains available |
Package source
The mirror image installs a locally-built wheel from docker/mirror/dist/:
COPY dist/mirror_py-*.whl /tmp/
RUN for wheel in /tmp/mirror_py-*.whl; do python3 -m pip install --no-cache-dir "$wheel[apt-mirror2]"; done
conftest.py:built_wheel (session-scoped) runs uv build --wheel against the
current source tree and places the artifact in dist/. The build is gated by
a SHA of mirror/** plus pyproject.toml, so unchanged source skips rebuild.
Public releases are independent: a v* tag push triggers
.github/workflows/pypi.yaml, which uses PyPI Trusted Publisher (OIDC) to
upload the artifact. The integration suite does NOT pull from PyPI — local
testing always uses the working-tree build.
Per-test isolation
Each test starts with a fresh mirror_stack:
Stop both master and worker services, including active sync subprocesses.
Clear publish/state and package-log contents inside the container, preserving bind-mount directories.
Start worker, wait for its socket, then start master and wait for readiness.
Wait for fresh state and for initial automatic syncs to settle before yielding.
Tests that change configuration, fixture content, or executable wrappers restore
those changes in finally blocks. Run suites sequentially: container names and
fixture resources are shared.
Test scenarios
File |
Scenario |
|---|---|
|
Runs archvsync’s |
|
Basic rsync; FFTS short-circuit when upstream unchanged; full sync when FFTS file changed |
|
Signed HTTP option discovery, explicit subsets, updates and cleanup, and failure recovery |
|
Multiple signed flat repositories, source indexes, automatic cleanup, hash and key failures, and FTP discovery |
|
Basic ftpsync; offline fallback exercises the embedded base64 archvsync ( |
|
Master restart during a 200MB sync does not kill worker subprocess (PID stable); master reconnects and sync completes |
|
Worker restart recovery; master gracefully handles worker unavailability |
|
Restart, CLI, and SIGHUP reload; validation, concurrent requests, and removal of idle or actively syncing packages |
|
Failed package retries after |
|
|
|
Per-package log file is gzip-compressed after sync completes |
Fixtures
apt-fixture (docker/apt-fixture/)
One Python HTTP/FTP server exposes a shared signed Debian archive at
http://apt-fixture:8000/debian and ftp://apt-fixture:2121/debian.
Static v1/debian/ and v2/debian/ trees contain
multiple distributions, components, and architectures, plus binary and source
indexes and signed InRelease and Release metadata. The allonly
distribution intentionally has no InRelease, exercising the signed
Release.gpg fallback. The stable alias duplicates bookworm metadata.
Only public keyrings are stored; private keys are not included.
Compose mounts the Debian keyring read-only at /etc/mirror/debmirror-test.gpg.
The flat repositories retain their separate keys, mounted at
/etc/mirror/apt-mirror2-ubuntu2404.gpg and
/etc/mirror/apt-mirror2-ubuntu2604.gpg. Both tools use the same Debian tree;
the mirror image installs debmirror and apt-mirror2 for real subprocess tests.
test_e2e_debmirror.py exercises master socket requests, worker subprocess
execution, completion notifications, persisted status and package logs:
Omitted
dist,section, andarchdiscover all binary targets, deduplicate the suite alias, and leave source packages disabled by default.Updating to v2 discovers a new distribution, component, and architecture, downloads their payloads, preserves a pool file still referenced by another distribution, and removes the obsolete v1 payload.
Explicit list selections restrict the mirror and
source: truedownloads source payloads.Tampered signed metadata and denied or incomplete directory listings fail without deleting the existing mirror. An explicit all-only distribution bypasses listing and exercises debmirror’s
--arch nonebehavior.
Only /debian and its listing controls are restored after each debmirror test,
including failures. Flat repository contents are left intact.
The small debmirror-test package also auto-syncs when the shared stack resets
for other integration tests. Its hourly schedule avoids background updates
during the explicitly triggered debmirror scenarios.
Run only these scenarios with:
uv run pytest -m integration tests/integration/test_e2e_debmirror.py -v
They are also included in the full uv run pytest -m integration -v suite.
The same fixture also serves /ubuntu2404/, /ubuntu2604/, and /sourceonly/.
The two CUDA-shaped flat repositories use different keys. Their static v1 and
v2 trees cover updates, retained files, automatic cleanup, source indexes, and
a source-only repository. apt-mirror2 tests reset only these three directories;
they never replace the shared data root or Debian archive. Its FTP test uses
the shared Debian tree, including allonly for signed Release.gpg fallback.
test_e2e_apt_mirror2.py runs the real apt-mirror2 v16 process through the
master and worker. It also verifies repository-specific key isolation, package
hash failures, cleanup suppression after errors, and recovery.
rsync-fixture (docker/rsync-fixture/data/)
Minimal tree exposed via rsyncd’s [data] module:
fullfiletimelist-test— FFTS metadata. Stable across reads, so re-syncs short-circuit.README— single-line markerdists/test/Release— Debian-style release filepool/main/p/pkg/pkg_1.0.deb— zero-byte placeholder
ftpsync-fixture (docker/ftpsync-fixture/data/)
Minimal Debian-style archive exposed via rsyncd’s [debian] module:
Project/trace/master— required by archvsync stage 1dists/test/Release+ zero-byteRelease.gpgpool/main/p/pkg/pkg_1.0.deb— zero-bytels-lR.gz— zero-byte
fixtures/tree_v2/
Alternate content used by test_ffts_changed_triggers_full_sync:
Modified
fullfiletimelist-testaddsNEW_FILEentry → FFTS dry-run reports change → full sync.NEW_FILEitself appears in publish tree after sync.
The test restores tree v1 in a finally block to keep later tests deterministic.
Configuration (docker/mirror/config.json)
Six packages baked into the image:
Package |
synctype |
src |
syncrate |
Purpose |
|---|---|---|---|---|
|
rsync |
|
|
Auto-syncing rsync target with FFTS enabled |
|
ftpsync |
|
|
Manually triggered ftpsync target |
|
lftp |
|
|
FTP target |
|
debmirror |
|
|
Signed Debian archive target |
|
apt-mirror2 |
two flat HTTP repositories |
|
Multi-source signed flat archive target |
|
rsync |
|
|
Always fails to verify error-retry behavior |
errorcontinuetime is set to 10 seconds for fast retry observation.
Known caveats
syncrate: "PUSH"disables automatic scheduling. Tests trigger manual-only packages explicitly.Preflight is an independent tool-level check; no pytest dependency-ordering plugin is required.
The offline-fallback test restores its temporary git wrapper even when assertions fail.
rsyncd in fixture containers runs as
uid = rootfor simplicity; this is acceptable for a sealed test container but is not a production pattern.
Layout reference
tests/integration/
├── conftest.py # session/per-test fixtures, INTEGRATION_TMP
├── helpers.py # MirrorStack class — all docker/host interactions
├── docker-compose.yml # 5 services with pinned container_name
├── docker/
│ ├── rsync-fixture/ # Dockerfile + rsyncd.conf + data/
│ ├── ftpsync-fixture/ # Dockerfile + rsyncd.conf + data/
│ ├── apt-fixture/ # Shared HTTP/FTP server + signed flat and Debian repositories
│ └── mirror/ # Dockerfile + config.json + dist/ (gitignored); systemd services from mirror setup
├── fixtures/
│ └── tree_v2/ # Alternate rsync content for FFTS-changed test
└── test_*.py # Integration scenarios