Everything we watch
83 rules and 27 measurements. This page is generated from the same catalogue the product runs on, so it cannot say something the software does not do.
Measurements
The agent reads these every 30 seconds. Anything marked tunable can carry a threshold rule of your own — pick a measurement, a direction, a number and how long it has to hold.
| MEASUREMENT | UNIT | PER | YOUR OWN RULES |
|---|
cpu.usage_percent | % | host | yes |
load.1 | — | host | yes |
load.5 | — | host | yes |
load.15 | — | host | yes |
memory.used_percent | % | host | yes |
memory.available_bytes | bytes | host | yes |
memory.total_bytes | bytes | host | yes |
swap.used_percent | % | host | yes |
swap.used_bytes | bytes | host | yes |
swap.total_bytes | bytes | host | yes |
disk.used_percent | % | mountpoint | yes |
disk.inodes_used_percent | % | mountpoint | yes |
disk.free_bytes | bytes | mountpoint | yes |
disk.total_bytes | bytes | mountpoint | yes |
diskio.read_bytes | bytes | device | no — counter since boot |
diskio.write_bytes | bytes | device | no — counter since boot |
net.rx_bytes_per_second | bytes per second | device | yes |
net.tx_bytes_per_second | bytes per second | device | yes |
net.rx_bytes | bytes | device | retired |
net.tx_bytes | bytes | device | retired |
net.conntrack_count | connections | host | yes |
net.conntrack_max | connections | host | yes |
tcp.established | connections | host | yes |
tcp.retrans_percent | % | host | yes |
tcp.listen_overflows_per_second | per second | host | yes |
tcp.listen_drops_per_second | per second | host | yes |
udp.rcvbuf_errors_per_second | per second | host | yes |
Rules
Each one can be left alone, forced to a severity, or switched off — per organisation and per host. Thresholds shown are the defaults.
host
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Host is not reportinghost.offline | No heartbeat for 45 seconds | critical | — |
| Clock out of stephost.clock_skew | — | — | — |
| Host rebootedhost.rebooted | — | — | — |
collect
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Collection failingcollect.failing | A part of the collection keeps failing: the package list, the image list, the port scan, docker or a metric | warning | — |
| Journal cannot be readcollect.journal | The agent cannot read the journal, so the two auth rules see nothing | critical | — |
disk
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Disk fulldisk.full | Mountpoint above 85% / 95% | warning / critical | Warn at (85%), Critical at (95%) |
| Disk space filling updisk.filling | — | — | — |
inodes
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Inodes filling upinodes.filling | — | — | — |
storage
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Storage pool degradedstorage.degraded | — | — | — |
| Storage pool state unknownstorage.unknown | — | — | — |
| Storage pool nearly fullstorage.nearly_full | — | — | — |
| Storage pool scrub overduestorage.scrub_stale | — | — | — |
| Storage pool reporting errorsstorage.errors | — | — | — |
| Thin pool metadata nearly fullstorage.metadata_full | — | — | — |
unit
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Service failedunit.failed | — | — | — |
| Service stoppedunit.stopped | — | — | — |
container
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Container not runningcontainer.down | A container marked as expected running is not running, or is gone from the host | critical | — |
| Container restart loopcontainer.restart_loop | A container has restarted 3 times within 60 minutes — critical if it is expected running | warning / critical | — |
| Privileged containercontainer.privileged | A container runs privileged: every capability, every device | critical | — |
| Docker socket mounted into a containercontainer.docker_socket | A container has the docker socket mounted, which is root on the host | critical | — |
| Container limits not enforcedcontainer.limit_ineffective | — | — | — |
docker
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Docker daemon gonedocker.gone | Docker is gone from a host that still has containers on record | critical | — |
port
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| New listening portport.new | — | — | — |
| Port reachable from anywhereport.exposed | — | — | — |
| Expected port is goneport.gone | — | — | — |
package
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Packages changedpackage.changed | — | — | — |
authkeys
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| authorized_keys changedauthkeys.changed | — | — | — |
config
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Watched file changedconfig.changed | — | — | — |
ldpreload
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| ld.so.preload is presentldpreload.present | — | — | — |
auth
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Repeated failed loginsauth.bruteforce | 10 failed SSH logins from one address in 15 minutes | warning | Alert at (10 attempts) |
| A login succeeded after failuresauth.breach | A login accepted from an address that just failed 5 times | critical | Alert at (5 attempts) |
| Repeated denied sudoauth.sudo_denied | — | — | — |
vuln
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Critical vulnerabilityvuln.critical | — | — | — |
| Known exploited vulnerabilityvuln.exploited | — | — | — |
| Secret found in an imagevuln.secret | A secret is baked into an image that is running | critical | — |
updates
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Security updates waitingupdates.security_pending | A security update has been pending for 7 days | warning | Allow (7 days) |
| Reboot requiredupdates.reboot_pending | The host has needed a reboot for 14 days | warning | Allow (14 days) |
| Image has no vulnerability dataupdates.image_unscannable | — | — | — |
| Running an older kernel than installedupdates.kernel_stale | The running kernel is older than the newest installed one | warning | — |
| Release is end of lifeupdates.distro_eol | The distribution stops getting security updates within 90 days / already has | warning / critical | — |
| Services still running old librariesupdates.restart_pending | — | — | — |
| Firmware update waitingupdates.firmware_pending | — | — | — |
| Firmware cannot be readupdates.firmware_blind | fwupd is missing, or its LVFS metadata is older than 30 days | warning | Metadata stale after (30 days) |
kernel
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Filesystem went read-onlykernel.fs_readonly | The kernel logged it within the last 7 days | critical | — |
| Filesystem errorkernel.fs_error | The kernel logged it within the last 7 days | warning | — |
| I/O errorkernel.io_error | The kernel logged it within the last 7 days | warning | — |
| Storage link resetkernel.link_reset | The kernel logged it within the last 7 days | warning | — |
| Machine checkkernel.hardware_error | The kernel logged it within the last 7 days | warning | — |
| Corrected memory errorkernel.memory_error | The kernel logged it within the last 7 days | warning | — |
| Blocked by kernel lockdownkernel.lockdown | The kernel logged it within the last 7 days | warning | — |
| Connection tracking table filled upkernel.conntrack_full | The kernel logged it within the last 7 days | critical | — |
| Accept queue overflowedkernel.syn_backlog | The kernel logged it within the last 7 days | warning | — |
| Killed to free memorykernel.oom_kill | The kernel logged it within the last 7 days | warning | — |
| Thin pool cannot writekernel.thin_exhausted | The kernel logged it within the last 7 days | critical | — |
| Thin pool nearly fullkernel.thin_low_water | The kernel logged it within the last 7 days | warning | — |
ssh
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| sshd config is invalidssh.config_invalid | — | — | — |
| Root may log in over SSHssh.root_login | — | — | — |
| Passwords accepted over SSHssh.password_auth | — | — | — |
| sshd config changedssh.config_changed | — | — | — |
account
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| A second root accountaccount.uid0 | — | — | — |
| An account with no passwordaccount.empty_password | — | — | — |
| New accountaccount.new | — | — | — |
| Account changedaccount.changed | — | — | — |
executable
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Executable in a world-writable directoryexecutable.in_writable_dir | — | — | — |
secret
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Secret readable by every accountsecret.world_readable | — | — | — |
suid
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| New setuid file no package ownssuid.new | — | — | — |
firewall
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Firewall state unknownfirewall.unknown | — | — | — |
egress
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| New outbound destinationegress.new | — | — | — |
net
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Link slower than expectednet.link_degraded | — | — | — |
| Link is downnet.link_down | — | — | — |
| Link is flappingnet.link_flapping | The carrier has changed 3 times or more since the last batch | warning | Flapping at (3 changes) |
| Interface is gonenet.gone | — | — | — |
| Link speed unknownnet.unknown | — | — | — |
| Connection tracking table nearly fullnet.conntrack_full | The table is above 90% / 98% of its limit | warning / critical | Warn at (90%), Critical at (98%) |
| Connections dropped at a listening socketnet.listen_overflow | More than 1 per second, averaged over the last reading | warning | Alert at (1 per second) |
| UDP datagrams dropped, receive buffer fullnet.udp_drops | More than 1 per second, averaged over the last reading | warning | Alert at (1 per second) |
security
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Firmware security droppedsecurity.posture_regressed | — | — | — |
check
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Check is downcheck.down | — | — | — |
| Certificate expiringcheck.cert_expiring | — | — | — |
| Certificate invalidcheck.cert_invalid | — | — | — |
| Weak TLScheck.tls_weak | A checked URL negotiates TLS 1.1 or older, or a cipher below 128 bits | warning | — |
job
| RULE | WHEN IT FIRES | SEVERITY | TUNABLE |
|---|
| Job did not reportjob.missed | — | — | — |
| Job failedjob.failed | — | — | — |
What is not on this list
Anything the agent cannot see
A host that stops reporting is its own alert. A measurement that is missing is shown as missing, never as zero — a conntrack table that does not exist is not an empty one.
Thresholds we have not measured
Where we do not know what a healthy machine looks like, there is no rule. The measurement is collected anyway, so you can set the number yourself once you know yours.
The sources behind these numbers, and the ones it deliberately does not touch.