NVR Disk Failure: Symptoms, RAID Rebuild and Preventing Loss

Early symptoms of NVR disk failure, interpreting SMART counters, the right order of intervention, RAID rebuild risks and detecting recording gaps, step by step.
Technical summary
Technology
NVR, RAID 5, RAID 6, SMART, smartctl, mdadm, hot spare
Topic
Responding to disk failure and rebuilding on a running recording system
Level
Intermediate–advanced
Prerequisites
  • Admin access to the recorder
  • RAID concepts
  • Linux command line (for server-based recording)
In this article8 sections

The most worked part of a recorder is the disk. It writes around the clock, often sits in a hot cabinet, and its failure may go unnoticed until someone asks for footage of an incident. If there’s no footage that day, there’s nothing left to do. This guide explains the early symptoms of disk failure on a running NVR or recording server, the right order of intervention at the moment of failure, the risks of the RAID rebuild process and how recording loss is detected, in the order we work on site. How to choose the RAID level at the design stage is the subject of the choosing a RAID level for video recording article.

Early symptoms of disk failure

Most disks don’t die all at once; they degrade gradually. The symptoms below are usually seen before failure; most are missed because nobody is looking:

  • Recording gaps: short, irregular interruptions on the timeline. If they appear on all channels at the same time, the disk or storage subsystem is suspect, not the camera.
  • Disk errors in the device log: entries on NVRs such as “HDD error”, “record exception” or the disk switching to an “abnormal” state.
  • Slow playback and search: if opening past recordings takes noticeably longer, there may be read retries on the disk.
  • A “degraded” warning in RAID status: the array keeps running but redundancy is gone. This isn’t a symptom; it’s an emergency.
  • Shorter retention than expected: a disk dropping out of the array, or bad sections being disabled on some NVRs, can reduce capacity.
  • Physical symptoms: clicking sounds, repeated spin-up sounds, the disk overheating.

How to interpret SMART values

SMART is the health counters a disk keeps itself. On embedded NVRs, this information usually appears in the disk management screen under “HDD detection” or a similar menu; on Linux or Windows-based recording servers, it’s read with smartmontools. These attributes are the most useful on recording disks:

ID Attribute Meaning Interpretation
5 Reallocated_Sector_Ct Sectors moved to the spare area An upward trend is critical
187 Reported_Uncorrect Uncorrectable read errors Any non-zero value is serious
188 Command_Timeout Commands that timed out A disk or power problem
197 Current_Pending_Sector Unreadable sectors waiting to be remapped The most important warning before a rebuild
198 Offline_Uncorrectable Sectors that couldn’t be corrected in an offline scan Any non-zero value is serious
199 UDMA_CRC_Error_Count Interface transmission errors Usually the cable or backplane

Attribute names and some IDs can differ by manufacturer. Look at the trend more than a single value: if the Reallocated or Pending counters rise week on week, the disk is added to the replacement plan even if the overall health result shows “PASSED”. “PASSED” only says the threshold hasn’t been exceeded yet. A UDMA CRC error, on the other hand, usually points not to the disk but to the data cable or backplane connection; on site, it’s common for a healthy disk to be replaced for nothing when the real problem turned out to be a loose SATA cable. Check the connection first.

# Overall health and attributes
smartctl -H /dev/sda
smartctl -A /dev/sda

# Start a long self-test and read the result
smartctl -t long /dev/sda
smartctl -l selftest /dev/sda

# Disk 0 behind an LSI/Broadcom MegaRAID controller
smartctl -a -d megaraid,0 /dev/sda

The long test takes hours depending on disk capacity, and longer under recording load. For command details, see the smartmontools project page.

First steps at the moment of failure

In corporate security camera systems, when a disk drops out of the array or the NVR reports a disk error, hasty action turns a single disk failure into data loss. Pulling a disk in a panic is the most expensive mistake. Recommended order:

  1. Record the situation: take screenshots of the RAID screen, the disk list and the error log. Note which serial-numbered disk in which slot has failed.
  2. Check the other disks: look at the SMART values of the other disks in the array. If another disk has Pending or Uncorrectable sectors, the rebuild risk is high; consider exporting critical recordings first.
  3. Secure critical recordings: if there’s an ongoing incident review or a recording that must be kept, export it before the rebuild starts.
  4. Avoid power-cycling the device: needlessly restarting a degraded array can cause a disk running at the edge to drop out too.
  5. Prepare the right replacement disk: use a disk with at least the capacity of the failed disk, preferably the same model or a surveillance-class disk on the manufacturer’s compatibility list.

Disk replacement and the rebuild process

On embedded NVRs and hardware RAID controllers, the process usually proceeds with the disk being replaced and the array starting the rebuild automatically or from the menu. On NVRs that don’t support hot-swap, the device is shut down to replace the disk; in that case the recording outage is planned in advance and the relevant teams are notified. On recording servers using Linux software RAID (mdadm), the steps are:

# Array status
cat /proc/mdstat
mdadm --detail /dev/md0

# Mark the failed disk as failed and remove it from the array
mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1

# Copy the GPT partition table of the healthy member (sda) to the new disk (sdb),
# then assign a new random GUID to the new disk
sgdisk -R=/dev/sdb /dev/sda
sgdisk -G /dev/sdb

# Add the new partition to the array
mdadm /dev/md0 --add /dev/sdb1

# Monitor rebuild progress
watch cat /proc/mdstat

The partition layout is copied from a healthy member of the array, not from the failed disk. Mixing up target and source in the sgdisk -R command corrupts the healthy disk’s partition table; this is the line in the console that needs the most care. After replacement, the new disk’s device name may differ. Before running the commands, verify the serial number with the output of ls -l /dev/disk/by-id/.

Hot spare

A hot spare is a disk waiting idle in the array that automatically starts the rebuild when a disk drops out. It keeps the degraded period from stretching until someone sees the warning and brings a disk. It’s especially valuable on recorders at remote sites and where night or weekend response may be delayed. Even with a hot spare, a failure notification should arrive and the used spare should be replaced with a new one. Otherwise, there’s no spare at the next failure.

Risks during a rebuild

A rebuild means reading all remaining disks in the array from start to finish. During this time:

  • Losing a second disk: RAID 5 withstands the loss of one disk; if a second disk drops out during the rebuild, the array is lost. RAID 6 tolerates the loss of two disks. The tendency of disks bought from the same batch and run for the same time to fail at similar times increases this risk.
  • Unreadable sectors: if one of the remaining disks has an unreadable sector, the data in that area can’t be rebuilt. Depending on the controller, the rebuild may stop or that section stays corrupted.
  • Performance drop: the rebuild load and continuous recording writes share the same disks; frame loss or recording queue overflow may occur. On some controllers, rebuild priority can be adjusted; if recording continuity has priority, rebuild speed is lowered and it takes longer.
  • Heat: disks run at full load throughout the rebuild; check cabinet ventilation.

Rebuild time depends on capacity, effective speed, recording load and storage architecture; the longer it takes, the longer the risk window. RAID doesn’t replace an independent recording backup; a separate copy of critical recordings is planned separately. Monitor the SMART values of the other disks during the rebuild and do no other maintenance on the array until it finishes. Firmware updates included.

Detecting recording gaps

After a failure, the question “what did we lose?” must be answered. Method:

  • On the recording platform’s timeline or recording search screen, check the recording continuity of all channels from the start of the failure to the end of the rebuild.
  • Find in the device log when the disk error was first seen. The problem has often started before the alarm was noticed.
  • Enter the lost intervals in a table with channel, start and end times. If an incident review comes in for that interval, this table is the evidence of why footage couldn’t be provided.
  • If recordings falling in the corrupted area can’t be played back, record that too.

Preventive checks

  • Test that disk and RAID status warnings actually reach someone via email, SNMP trap or the monitoring system. A warning going to a mailbox that’s configured but read by nobody doesn’t count as a warning.
  • Record SMART values monthly; look at the upward trend more than absolute values.
  • Prefer disks designed for surveillance use; desktop disks aren’t suited to continuous write loads (see choosing surveillance disks).
  • Keep at least one compatible spare disk on the shelf.
  • Monitor cabinet temperature and UPS status; sudden power cuts harm disk and array consistency.
  • Verify during periodic maintenance that retention matches the calculated value; for a detailed routine, see the camera periodic maintenance checklist.

For fault response and capacity upgrades, see our NVR disk failure and capacity upgrades page, and for regular checks our periodic maintenance and support service.

Frequently asked questions

Are recordings lost when an NVR disk fails?

If the disk is in a RAID array providing redundancy, recordings are preserved on a single disk failure; the array keeps running in a degraded state. On NVRs without RAID, recordings on the failed disk are usually lost and the channels written to that disk go unrecorded.

If the SMART result is PASSED, is the disk healthy?

Not always. The overall health result fails only when thresholds are exceeded. If the Reallocated, Current Pending and Offline Uncorrectable counters are rising, the disk is added to the replacement plan even if it shows PASSED.

Does recording continue during a rebuild?

On most NVRs and RAID controllers it does; however, since the rebuild load shares the disks, frame loss or recording delays may occur. If rebuild priority can be adjusted, a balance is struck between recording continuity and rebuild time.

Can a different disk model be fitted in place of the failed disk?

A surveillance-class disk with at least the capacity of the failed disk, on the recorder manufacturer’s compatibility list, can be used. The same model isn’t mandatory, but different performance characteristics can affect the array’s overall performance.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek Video Surveillance

IP cameras, recording and central management infrastructure. We plan blind spots, retention periods and the camera network around the risks of each site.

Knowledge Center

Other articles