VictoriaLogs does not de-duplicate

VictoriaLogs is a log store. That sounds like a description, but it is a design constraint with consequences, and the fleet’s own operations notes record the exact one: VictoriaLogs does not de-duplicate (the upstream item is VictoriaMetrics#5080, cited as open). So anything that hands it the same line twice gets two lines back.

That is not a bug report. It is the behaviour a store should have. The point is that two ordinary design facts become duplicated history if you do not account for them.

A retrying agent re-sends a batch

The file collector here is vlagent, configured in modules/monitoring/vlagent.nix. It writes to a disk buffer and keeps a checkpoint:

1
2
-remoteWrite.tmpDataPath=/var/lib/vlagent
-fileCollector.checkpointsPath=/var/lib/vlagent/file-checkpoints.json

That is a correct, durable design — the unit even restarts on failure with Restart = "on-failure" and RestartSec = "5s". But a batch that was written to VictoriaLogs and then not acknowledged — the connection dropped, the process restarted mid-flight — is re-sent from the checkpoint. Because the store does not de-duplicate, that is not an idempotent replay. It is a second copy of the same lines.

A restart loop multiplies every startup line

Every unit that logs on start logs again on every restart. A service that crash-loops therefore multiplies its own startup output, and the fleet’s own notes record the shape of it: a preStart that touches a spool file without the directory existing loops on “Permission denied”, and the unfixed loop floods VictoriaLogs with thousands of “Failed to start” entries, dominating error-level log counts. The store is faithfully recording a system that is genuinely failing to start thousands of times. The store is not the thing that is wrong.

The shutdown window

The third fact is subtler: a service can log so heavily while stopping that it dominates a time window, and a stop that is held open by a stalled log upload keeps the process alive to log more. This fleet hit exactly that with systemd-journal-upload, whose POST had no timeout and could block the unit for the whole 90 s stop cap. Bounding the transfer changed the shape of the shutdown: the notes record the measured improvement on the giga host — 90 s → 11 s (stop cap only) → 6 s once NetworkTimeoutSec was set, with the shutdown window’s tail complete in the store (847 on-disk lines, at least 2000 in VictoriaLogs).

The same note records the cost of not bounding it: because the store does not de-duplicate, a SIGKILLed stop can re-ingest up to one state-save interval, measured at roughly one duplicate line. Small — and exactly the kind of number that stays invisible until it is not.

Rules

• “Is this line cheap to store?” belongs in the logging decision. A line written once per loop iteration is not the same as a line written once per state change.
• Prefer one line per state change over one per iteration. Log the transition, not the tick.
• Remember the replay is not free. A disk-buffered retry is durable; it is not idempotent.
• When something looks slow, check ingest rate and per-stream volume. A restart loop and a replayed batch both show up as volume, not as an error message.
• Ask what you are actually shipping. The question is not whether the store is correct — it is whether the volume arriving is the volume you intended to ship.

What you can check

• modules/monitoring/vlagent.nix in the nix-config repository for the tmpDataPath buffer, the checkpoint path and the Restart policy.
• VictoriaMetrics#5080, the de-duplication item the fleet’s notes cite as open.
• The measured shutdown figures and the one-duplicate-line cost, in the victorialogs-vmagent operations notes (the systemd-journal-upload section), alongside the systemd-journal-upload.service: State 'stop-sigterm' timed out. Killing. line that starts the diagnosis.

This is the same argument as the logging posture page: what you ship decides what is in the store, and the store keeps what it is given.