Camera Health Monitoring: SNMP, ONVIF Events and Video Loss Alarms

Ping is green, no recording: build layered camera health monitoring with the PoE port, RTSP stream, ONVIF events and recording gap scans; threshold and alarm recommendations.
Technical summary
Technology
SNMPv3, IF-MIB, POWER-ETHERNET-MIB (RFC 3621), ONVIF Event Service, RTSP, VMS API
Topic
Automatic monitoring of cameras, streams, recording and image quality
Level
Intermediate–advanced
Prerequisites
  • Basic SNMP and network monitoring knowledge
  • Access to the management interface of the VMS/NVR in use
In this article8 sections

The camera responds to ping and the monitoring screen is green, but when the recording is checked after an incident there are three days of black screen. The lens may have been sprayed over, the focus may have drifted, the IR may have burnt out or the VMS may never have received the stream; ping sees none of this. Below we describe a layered health monitoring model for camera systems that runs from the switch port to image quality, where each metric is read from and how alarm thresholds are set up on site.

Ping isn’t enough: a layered monitoring model

If you reduce camera health to a single “up/down” status, you only find out there’s no recording when reviewing retrospectively after an incident. The right approach is to monitor the camera in five layers. Each layer assumes the one below is sound; alarms are suppressed in the same order. If the switch port is down, no separate “no stream” alarm is raised.

  1. Physical and power layer: Is the switch port connected, is PoE power being delivered, are port error counters rising?
  2. Network layer: Is the camera IP reachable, are latency and packet loss acceptable?
  3. Service layer: Do the RTSP and ONVIF services respond, does authentication succeed, is the clock in sync?
  4. Stream and recording layer: Is the VMS/NVR receiving the stream, is recording written to disk, are frame rate and bitrate in the expected range?
  5. Image layer: Does the scene still show the planned area; is there covering, blur, excessive darkness or an angle change?

The first two layers come from the network monitoring system (Zabbix, LibreNMS, PRTG, etc.), the third and fourth from the VMS’s health module or API, and the fifth from the camera’s own analytics or the VMS’s image quality checks. A common mistake is these sources sitting on separate screens under separate names: the switch says “Gi1/0/14 down”, the VMS says “Channel 37 no connection”, and nobody knows they’re the same camera. Every camera should map to a single ID in the inventory, and alarms should be gathered in a single stream.

Table of metrics to monitor

The table shows recommended metrics and typical sources for each layer. Not every metric exists in every product. Before setting up, test what your camera, switch and VMS models really offer; what the datasheet says and what the API returns aren’t always the same.

Layer Metric Source Control frequency
Strength PoE port status, power drawn Switch SNMP (POWER-ETHERNET-MIB) 1–5 min
Physical ifOperStatus, CRC/input errors Switch SNMP (IF-MIB) 1–5 min
Network ICMP latency, packet loss Monitoring server 1 min
Service RTSP/ONVIF response, NTP offset Synthetic check, ONVIF 5–15 min
Stream Frame rate, bitrate, connection status VMS API / health module 1–5 min
Record Last recording write time, gaps VMS API, NVR events 5–15 min
Image Tamper, loss of focus, scene change ONVIF events, VMS analytics Event-based
Hardware Internal camera temperature, SD card status Camera SNMP / manufacturer API 15 min

SNMP, ONVIF events and the VMS API

SNMP on the switch side

The most reliable first indicator of camera health isn’t the camera itself but the switch port it’s connected to. A camera that loses power can’t report anything. Port status and error counters are read from the standard IF-MIB, and PoE detection status from POWER-ETHERNET-MIB, defined in RFC 3621. On a PoE-powered port, when the pethPsePortDetectionStatus value moves away from deliveringPower(3) (for example to searching(2), fault(4) or otherFault(6)), the camera has lost power. Reading with SNMPv3 authPriv:

# PoE detection status (POWER-ETHERNET-MIB::pethPsePortDetectionStatus)
snmpwalk -v3 -l authPriv -u monitor -a SHA -A 'AuthPassword' -x AES -X 'PrivPassword' \
  10.20.0.2 1.3.6.1.2.1.105.1.1.1.6

# Port operational status (IF-MIB::ifOperStatus) and input errors (IF-MIB::ifInErrors)
snmpwalk -v3 -l authPriv -u monitor -a SHA -A 'AuthPassword' -x AES -X 'PrivPassword' \
  10.20.0.2 1.3.6.1.2.1.2.2.1.8
snmpwalk -v3 -l authPriv -u monitor -a SHA -A 'AuthPassword' -x AES -X 'PrivPassword' \
  10.20.0.2 1.3.6.1.2.1.2.2.1.14

Power drawn per port is mostly in vendor-specific MIBs (such as CISCO-POWER-ETHERNET-EXT-MIB on Cisco). This is also the first place to look in the console: if a camera reboots periodically and the power it draws is close to its class limit, the budget may be exceeded the moment IR kicks in at night. On switches in the camera VLAN, SNMP should be v3 only and open only to the monitoring server; switches with a v2c community still left at “public” are still common. For details, see the SNMPv3 authPriv configuration guide.

ONVIF events on the camera side

ONVIF-compliant cameras publish events via the Event service. The monitoring platform subscribes with CreatePullPointSubscription and pulls events with PullMessages. The topics a camera supports are queried with GetEventProperties; those related to image quality are usually under the tns1:VideoSource tree (scene change, blurred image, excessive darkness and so on). Topic names vary with model and firmware version. After a firmware update, the event subscription can silently drop; add “are events coming through?” to the post-update checklist. Which profile offers event and analytics support is detailed in the ONVIF Profile T, G and M comparison; the specifications are published on ONVIF’s official site.

VMS API

Most enterprise VMS products provide camera connection status, recording status and last recording time via a REST API or SDK. This is the most valuable source for the stream and recording layer, because only the system doing the recording knows about the “camera reachable but no recording” state. Create a separate, read-only service account for the API; this account shouldn’t have permission to view live or recorded video. The monitoring account can’t see video.

Synthetic stream check

If the VMS API is limited, a check that tests the RTSP stream directly is added. This check opens an extra session on the camera; older models have a low concurrent session limit, and a test connecting to the main stream can drop the VMS’s connection. So run it on the substream and at sparse intervals:

ffprobe -v error -rtsp_transport tcp -select_streams v:0 \
  -show_entries stream=codec_name,width,height,avg_frame_rate -of csv=p=0 \
  "rtsp://monitor:[email protected]:554/stream2"
# Example of expected output: h264,640,360,15/1

If the resolution or frame rate differs from the value in the inventory, someone may have changed the camera settings. The RTSP path (/stream2 etc.) varies by manufacturer; don’t guess the right path, get it with ONVIF GetStreamUri.

Image quality checks: tamper, loss of focus

The image layer is the most skipped, and the one that hurts most at the moment of an incident. A camera can record useless footage while everything is green on the network side. Situations to monitor:

  • Occlusion: the lens being covered, painted or having an object placed in front of it.
  • Field of view change: the camera turning due to an impact or loosening; the image is clean but shows the wrong area.
  • Loss of focus: blurring on motorized-lens cameras due to temperature changes or condensation inside the dome.
  • Excessive darkness: an IR LED failure or the day/night transition getting stuck.
  • Frozen image: the stream arrives but frames don’t change; seen with some encoder faults, and the bitrate graph turns into a flat line.

Tamper detection on the camera reacts quickly, but if its sensitivity is badly set, it raises an alarm at every headlight. After a few weeks, the team stops reading this alarm. A complementary method is for the VMS or a small script to take a snapshot from each camera a few times a day (for example at 10:00 and 22:00) and compare it with a reference image. Save the reference image during commissioning, with the framing approved in the acceptance test. Without a reference, there’s no measure for saying “it changed”.

Monitoring recording continuity

The question at this layer isn’t “is the camera recording right now?” but “is all the expected recording from the last 24 hours on disk?”. Three checks are used together:

  1. Last recording time: on a continuously recording camera, an alarm if the last written block is older than a few minutes.
  2. Recording gap scan: each camera’s timeline is scanned daily, and gaps longer than the set duration are reported. Gaps are normal for event-based recording; those cameras are assessed with a separate rule.
  3. Retention check: the date of each camera’s oldest recording is compared with the target retention period. Old recordings being deleted earlier than planned when the disk fills up can only be caught this way.

A common picture on site: a camera’s bitrate profile was raised, the disk calculation wasn’t updated, and the NVR quietly started overwriting old recordings. On paper the retention target is 30 days; in reality much less stays on disk, and nobody noticed. The NVR’s disk and RAID status is also part of this layer; read errors or a degraded array are among the most common sources of recording gaps. Steps at the moment of failure are described in the NVR disk failure and RAID rebuild guide.

Alarm thresholds and escalation

Badly set thresholds produce either alarm fatigue or silent failure. The values below are a starting point; they should be adjusted after two to four weeks of observation on site.

Status Recommended trigger Priority
No stream on a critical camera 3 consecutive failed checks High
Recording isn’t being written to disk Last recording > 5 min (continuous recording) High
PoE fault / port down At the moment of the state change Depending on camera importance
Port CRC errors increasing Counter increase continuing for 15 min Intermediate
Tamper / scene change Event + snapshot verification Medium–high
Frame rate drop Well below the profile value, for 10 min Low
Falling below the retention period Detected in the daily check Intermediate

For escalation, divide cameras into criticality classes: points with high evidential value such as checkouts, entrance gates and shipping docks are first class; general area cameras are second class. When ten cameras on the same switch go down at once, a single “switch/uplink” alarm should arrive, not ten separate alarms. For this, the switch–port–camera dependency is defined in the monitoring system. If this relationship isn’t in the inventory, the camera inventory must be fixed first; alarm suppression can’t be set up without a dependency map.

Periodic maintenance windows are also part of the alarm design. If the relevant cameras aren’t put into planned maintenance mode during a firmware update or switch replacement, the team gets used to the noise. A team used to noise misses the real alarm.

Reporting and weekly review

Real-time alarms alone aren’t enough; small recurring problems only show up in the trend. In a half-hour weekly review, look at:

  • The cameras producing the most alarms and their root causes (cable, connector, PoE budget, firmware).
  • Total recording gap time and the critical cameras affected.
  • Open tamper and field of view findings; whether they were fixed on site.
  • Recorders falling below the retention period and the disk usage trend.
  • False alarms caused by thresholds and the setting changes made.

If the same camera is on the list every week, the problem isn’t the alarm; it’s on site. It usually turns out to be an oxidized RJ45 or a junction box that has taken on moisture outdoors. Automatic monitoring doesn’t replace manual checks on site; the periodic maintenance checklist continues to be used for lens cleaning, mounting tightness and cabinet checks. How to tie monitoring data to the availability and recording continuity metrics in a service contract is the subject of the SLA design for managed camera services article.

To combine layered monitoring with your existing network monitoring infrastructure, take a look at the scope of our camera network monitoring and fault alarms service.

Frequently asked questions

Why isn’t a camera responding to ping enough?

Ping only shows that the camera’s network interface is working. Whether the stream reaches the VMS, whether recording is written to disk and whether the image has been covered, blurred or turned the wrong way can’t be told from ping.

Should PoE faults be monitored from the camera side or the switch side?

From the switch side. A camera that loses power can’t send any data; the switch, however, keeps reporting port detection status and fault status via POWER-ETHERNET-MIB.

Why does the tamper alarm produce so many false alarms?

Tamper detection on the camera usually reacts to sudden brightness or scene changes; headlights, day–night transitions and lighting changes trigger it. Tune sensitivity to the site and verify the alarm with a snapshot compared against a reference image.

How often should recording gaps be checked?

For critical cameras recording continuously, the last recording time should be checked at intervals of a few minutes, and a timeline gap scan done daily for all cameras. The date of the oldest recording should also be compared regularly with the retention target.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek Video Surveillance

IP cameras, recording and central management infrastructure. We plan blind spots, retention periods and the camera network around the risks of each site.

Knowledge Center

Other articles