agent.yaml
Four keys to reach the central, and the reporters you add yourself. It lives at /etc/silentfailure/agent.yaml and the installer writes it with mode 0600.
The keys
One key per line, with # for comments. An unknown key is rejected at startup rather than ignored, so a typo stops the agent instead of quietly costing you a setting.
| KEY | NEEDED | WHAT IT DOES |
|---|---|---|
central_url | Always | Where the central is. The only key with no default. |
enroll_token | Until enrolled | The join token from Settings → Hosts. Read only while enrolling: once the host is approved it has a real token of its own, and this key is never read again. You may delete it afterwards, and a -enroll-token flag beats the file. |
central_ip | If DNS cannot resolve | Skips name resolution. Worth knowing why it exists: in our own fleet the name is resolved by the Pi-hole hosts the system itself watches, so a Pi-hole that is down cannot report that it is down. |
host_id | Optional | A name you would like. Only a request: the central may assign its own at enrollment, and that one wins — it is written to the state file and identifies every batch afterwards. Left out, the hostname is used. |
Which ones you need when
With four keys the question is never which exist, but which are required right now.
| SITUATION | THE FILE NEEDS |
|---|---|
| A fresh machine, before enrollment | central_url + enroll_token |
| The same, where DNS cannot resolve the central | plus central_ip |
| After enrollment | only central_url. The real token lives in the state file, not here |
| Upgrading the agent | nothing. install.sh keeps an existing file and only rewrites the keys you pass it |
Two files, and only one of them is yours
Both hold configuration, so they get mixed up. The difference matters when something goes wrong.
| FILE | WHAT IS IN IT | WHO WRITES IT |
|---|---|---|
| /etc/silentfailure/agent.yaml | The keys on this page. Mode 0600, because the join token is a secret. | You, or install.sh |
| /var/lib/silentfailure/state.json | The real token, the assigned host id, log offsets and the send buffer. Mode 0600, directory 0700. | The agent itself. Never edit it |
What is deliberately not in the file
This is the most common question about agent.yaml, and the answer is that the file is short on purpose.
Collection intervals, log patterns, which containers to read, rate limits and the file watchlist all come from the central, versioned, so they can be tuned in the dashboard for a whole fleet rather than edited on every machine. If you are looking for your collection interval here, it is under the host in the app — not in this file.
The one exception is reporters, below. They are the things the central must never be able to set.
Reporters: what you want watched
Everything else the agent collects is something we decided to measure. A reporter is something you decided matters on this machine.
# A systemd unit: did it run, and did it go well
reporter.srv01-backup.unit: srv01-backup.service
reporter.srv01-backup.expect_within: 26h
# A file: is the newest match fresh enough
reporter.forgejo-dump.file: /data/backups/forgejo/*.sql.gz
reporter.forgejo-dump.expect_within: 26h| FIELD | WHAT IT MEANS |
|---|---|
reporter.<name>.unit | A systemd unit. Its result and when it last ran. Needs expect_within. |
reporter.<name>.file | A glob. The newest file it matches. Needs expect_within. |
reporter.<name>.command | An absolute path the agent runs. No arguments — write a wrapper script if you need them. Needs interval. |
reporter.<name>.expect_within | How old the answer may be: 26h, 2d, 90m. Required for unit and file, and not allowed with command. |
reporter.<name>.interval | How often to run a command. Required for command, with a floor of one minute. There is no default, because the agent cannot know how expensive your script is. |
reporter.<name>.timeout | The ceiling on one run of a command. Optional, and 20s if you leave it out. A command that hangs is reported as blind, not failed: we did not get an answer. |
reporter.<name>.extract | Named expressions against the newest matching file: extract.<name>. Each needs exactly one capture group. Only the last megabyte is read, because a script writes its summary at the end. |
The name may be letters, digits, dash and underscore — not a dot, which separates the name from the field. Each reporter needs exactly one of unit, file and command.
Reading numbers out of a log
The job already writes what you want to know. You should not have to change it to be able to see it.
# The nightly backup already logs this line:
# enheder: 20 (ok 19) · filer overfoert: 10831 · slettet: 8299
reporter.backup-log.file: /home/hurup/backup/log/srv01-to-nas-*.log
reporter.backup-log.expect_within: 26h
reporter.backup-log.extract.transferred: filer overfoert: (\d+)
reporter.backup-log.extract.deleted: slettet: (\d+)An expression that finds nothing makes the reporter blind, not ok. If someone changes the script output, the expression stops matching — and a reporter that then reported green would be the worst thing a check can do. Blind rather than failed, because it is not a statement about the job: we could not read the answer.
Running a command
The universal key. Anything you can write as a script can be watched — and the finished Nagios plugins work as they are.
reporter.cert-check.command: /usr/local/bin/check-cert.sh
reporter.cert-check.interval: 6h
reporter.cert-check.timeout: 20s| THE SCRIPT SAYS | AND THE AGENT REPORTS |
|---|---|
| exit 0 | ok, with the first line of stdout as the message |
| exit 1 or 2 | failed. Both, so a Nagios WARNING and CRITICAL are not told apart by the state — the exit code travels with the alert, and the severity is set on the rule |
| exit 3 | blind. Nagios calls it UNKNOWN, and that is exactly the fourth state: the check could not determine anything |
|days=61;30;7 | Numbers. Thresholds and units after the value are the plugin language for limits, and the central has its own, so only the first part is kept |
{"message":"…","days":61} | JSON works too, for a script written for the purpose rather than borrowed |
| nothing, and it hangs | blind at the timeout. We did not get an answer, and that is not a statement about the job |
The command inherits the agents hardening — an empty capability set, a read-only filesystem, a private /dev and a system call filter. A plugin that pings will therefore fail, because ICMP needs a raw socket. Its stderr is sent along so that looks like what it is rather than like a broken script.
Why expect_within is required
Because systemd says success when nothing happened at all.
$ systemctl show -p LoadState,Result,ExecMainExitTimestamp fstrim.service
Result=success
ExecMainExitTimestamp= # empty: it has never run
$ systemctl show -p LoadState,Result no-such.service
Result=success
LoadState=not-found # it does not exist at allA unit that has never run reports success. A unit that does not exist reports success too. Without a deadline, it went well, it never ran and you misspelled the name all look the same — and the one thing a reporter must never do is report green for a line you typed wrong.
What the central is told
Four states, and the three that are not green are three different messages.
| STATE | MEANS | RULE |
|---|---|---|
| ok | Ran within the deadline | — |
| failed | Ran and ended badly — or the target does not exist | job.failed, warning |
| missed | It did not happen: never ran, or older than the deadline | job.missed, critical |
| blind | We could not look: no systemd, a bad glob, an unreadable file | job.blind, warning |
missed is the critical one because it is the one that costs data. A run that failed shouted about it; one that stopped happening is silent until the day you need the result.
Reporters can only be set on the host
The central can show you a reporter. It cannot define one.
The central already pushes configuration down to the agent — intervals, log patterns, the watchlist. If reporters could arrive that way, a reporter that ran a command would turn a monitoring agent into a way to run commands as root on every machine in every organisation, from one place. So the file is the only source, and a reporter you add takes effect on the next collection round without restarting the agent.
If the file becomes invalid, the agent keeps running and tells the central why. It does not close the alarms for reporters it can no longer read: nobody reported is not the same as nothing is wrong.