When a recording server fails, recording stops for all cameras connected to it. The time until an operator notices a black screen in live view is lost footage, and it doesn’t come back. High availability design doesn’t treat this risk as a single item; it addresses recording server failover, the management server and database, storage and network layers separately. Below we explain which failover model to choose when, how to calculate and close the recording gap during takeover and how to test the design on site. Choosing the platform itself is the subject of the VMS or NVR article.
Where are the single points of failure
The work starts with an inventory. For each component in the chain, from the camera to the viewing client, one question is asked: if this goes, what stops? In a typical enterprise VMS, the table comes out roughly like this:
| Component | Impact of failure | Redundancy method |
|---|---|---|
| Recording server | Recording stops for connected cameras | Failover server, dual recording |
| Management server | Login, configuration, some alarms | Cluster or manufacturer redundancy |
| Database (SQL) | Management layer doesn’t work | Cluster, Always On, regular backups |
| Storage array/controller | No recordings can be written | Dual controllers, RAID, a second target |
| Camera access switch | Connected cameras deliver no video | Edge recording; switch redundancy limited |
| Core/server switch | Servers and cameras are cut off | Stack/MLAG, dual uplinks |
| Power (UPS, PDU) | The whole cabinet stops | Dual PSUs, A/B feeds |
The weakest row in the table is usually between the camera and the access switch. The camera connects to a single port; physical redundancy is practically never built at this layer. Instead, the camera records to its own SD card, and when the connection returns the gap is backfilled into the VMS. This is also the common gap: the server side has been doubled, and the camera side never considered.
Recording server failover models
Enterprise VMSs offer recording server redundancy under different names but with similar logic. Higher editions of Milestone XProtect have cold and hot standby failover recording servers, and Genetec Security Center has failover archiver lists and simultaneous redundant archiving options. Verify from the manufacturer’s current documentation which option is enabled at which license level; a license item skipped at the quotation stage changes the design later.
| Model | How it works | Recording gap | Cost |
|---|---|---|---|
| Cold standby (N+1) | The standby server connects to the relevant cameras after a failure | Detection + takeover time | Low |
| Hot standby (1+1) | The standby waits ready with a single server’s configuration | Shorter | Intermediate |
| Simultaneous dual recording | Two servers continuously record the same cameras | None | High (2× storage) |
| Virtualization HA | The recording server VM moves to another host and restarts | VM boot time | Depends on the infrastructure |
Cold standby and N+1
In N+1, several recording servers share a single standby. The standby doesn’t normally record; when no health signal arrives from a primary server, it connects to that server’s cameras and starts recording. When the primary returns, failback happens; some VMSs move recordings written to the standby during failover to the primary’s storage and create a single timeline, while others leave two separate parts. The model’s limit is clear: if two primaries fail at once, the standby takes only one. A power loss in the same cabinet is exactly this scenario. That’s why the standby server shouldn’t share the same cabinet and the same UPS as the primaries.
If the standby server is in the same cabinet, N+1 exists only on paper.
Hot standby
In hot standby, the standby server is assigned to a single primary and keeps its configuration preloaded. The time to connect to cameras and start streams shortens, and the gap narrows. It’s the right model for servers hosting cameras where a recording interruption isn’t acceptable, such as entry points, cash rooms and production lines. The cost is separate hardware and licensing for each primary.
Simultaneous dual recording
Two independent servers record the same camera at the same time. The camera either produces two separate unicast streams, or multicast is used. A second unicast stream puts extra load on the camera’s processor and uplink; on older or entry-level cameras this load comes back as dropped frames. Because storage doubles, dual recording is usually applied to a critical camera group, not the whole system. To calculate network load in advance, use the method in the camera network bandwidth calculation article.
HA at the virtualization layer
If the recording server runs in a virtual machine, the hypervisor’s HA feature (Hyper-V Failover Cluster, vSphere HA, Proxmox HA) restarts the VM on another host when a host fails. This doesn’t replace the VMS’s own failover. There’s no recording while the VM boots and the recording service comes up. By default, hypervisor HA reacts only to host failure; it doesn’t see the recording service hanging inside the VM unless VM/application monitoring is configured separately. Recording may have stopped for minutes while everything is green in the console. Also, continuous high write load and shared storage must be inputs to the cluster design from the start; recording VMs added later affect the cluster’s other workloads too. For general rules of cluster planning, see the Hyper-V Failover Cluster planning article.
Management server and database redundancy
The management server handles user login, authorization, camera configuration and the rules engine. In most enterprise VMSs, recording continues when the management server stops, because recording servers keep the configuration in a local cache. But operators may not be able to log in again, and alarm workflows and central event rules may not work. An operator who restarts their client on the night shift can’t get back into the system. This behavior varies from manufacturer to manufacturer; test it on your own system at the design stage and write down the result.
Redundancy options fall into three groups:
- Operating system cluster: The management service runs as a role on a Windows Server Failover Cluster. Verify in which version and license the manufacturer supports this setup.
- The manufacturer’s built-in redundancy: Some platforms offer their own failover list for the management/directory role and don’t require cluster infrastructure.
- Database redundancy: The configuration and event database is usually on Microsoft SQL Server. In critical installations, SQL Server Always On or a Failover Cluster Instance can be used; the minimum requirement is regular full backups and a tested restore. The Microsoft documentation is the starting point for Always On Availability Groups.
A database being small doesn’t make it unimportant. The camera list, user roles, recording rules and retention settings are there. If this database is lost, rebuilding the system takes days even if the video files are still on disk. A situation common on site: a management server installed with SQL Express, never backed up, whose service stops when the disk fills.
The recording gap during failover
In cold or hot standby, a gap is unavoidable; the question is its size. The gap consists of four parts: the health check interval and the number of consecutive failures needed for failover, the standby server loading the configuration, establishing RTSP sessions with the cameras, and the arrival of the first I-frame. The last is forgotten in most calculations. A camera stream can’t be decoded until an I-frame arrives; a long GOP setting extends the gap by a few seconds.
Worked example: If a health check runs every 10 seconds and failover is triggered after three consecutive failures, detection takes up to 30 seconds. If the standby server takes 20 seconds to connect to 60 cameras and the GOP length is 4 seconds, the total gap approaches a minute. More frequent health checks shorten the time; but an overly aggressive setting triggers unnecessary failovers on short network fluctuations, and each unnecessary takeover produces its own gap. The balance is found by testing.
There are two ways to close the gap:
- Simultaneous dual recording: No gap forms at all; the cost is in storage and network.
- Backfill with edge recording: The camera records to its SD card continuously or when the connection drops; when the connection returns, the VMS pulls the missing range from the camera. Both the VMS and the camera must support this; details are in the edge recording and ANR article.
Storage and network redundancy
If storage is a single point, recording server failover loses its meaning. Where the standby server writes is at the heart of the design. In the cold standby model, the standby server mostly writes to its own local storage; its capacity must then cover the data the largest primary it takes over will produce during the longest expected failover period. If the primaries use a shared SAN, the SAN must have dual controllers and be connected to the servers with multipath.
On the network side, recording servers are connected to the camera VLAN and the storage network with at least two physical links, preferably to two separate switches. Core switches in a stack or MLAG setup preserve server access when a single switch fails. Plugging two NICs into the same switch only protects against a cable failure. Storage connection types and multipath design are explained in detail in the iSCSI, NAS and DAS comparison article.
The same logic applies to power. Each PSU of a dual-PSU server should be connected to a separate PDU, preferably a separate UPS. If both PSUs are on the same PDU, you’re protected only against a PSU failure. When the feed goes, both go together.
Failover test procedure
In enterprise camera systems, high availability is an assumption until it’s tested. The following sequence can be applied in a maintenance window on a live system that is recording:
- Record the starting state: Document which camera records on which server, that server clocks are synchronized via NTP and that all cameras are recording.
- Place a visible time reference: Put a screen or clock showing seconds in the test camera’s field of view. Measuring the gap from the image is more reliable than log timestamps.
- Create the failure realistically: Stopping the recording service, pulling the server’s network cable and cutting power trigger different detection paths. Try at least two; stopping the service alone tests the easiest scenario.
- Measure the takeover: Note the moment failover is triggered, the moment the first frame is recorded on the standby server and the moment live view returns.
- Try functions during failover: Run live view, playback, alarms and export while on the standby server.
- Test failback: Bring the primary back; check whether recordings are merged and whether there’s an overlap or a second gap in the timeline. The failback gap is the most often skipped item.
- Verify backfill: If edge recording is used, confirm by playback that the gap was filled from the camera.
- Document the result: Compare the measured gap with the expected value and add it to the test record.
The same steps can be used in project acceptance; the general framework of the acceptance test is in the camera project acceptance test article.
Design checklist
- The acceptable recording gap duration has been defined in writing for each camera group.
- A decision on dual recording or backfill with edge recording has been made for critical cameras.
- The N+1 standby server doesn’t share the same cabinet and UPS as the primaries.
- The standby server’s storage capacity has been calculated for the longest expected failover period.
- It has been tested that recording continues during a management server outage, and the affected functions have been documented.
- The backup and restore procedure for the VMS database has actually been tried.
- Storage has dual controllers and is connected with multipath, or the standby server writes to independent storage.
- Servers are connected to two switches and two separate power feeds.
- Failover and failback tests were done at acceptance, and an annual repeat is on the calendar.
- Failover events arrive in the monitoring system as alarms; a silent failover doesn’t go unnoticed for months.
The last item looks small, but it isn’t. An unnoticed failover means being left without a standby at the second failure. For the design and testing of recording and management layer redundancy, you can find the scope on the NVR/VMS redundancy and high availability service page.
Frequently asked questions
Is recording lost while a failover recording server takes over?
In cold and hot standby models, yes. A gap remains between detection of the failure and the standby server connecting to the cameras; this window can stretch from a few seconds to a few minutes. There are two ways to close the gap: simultaneous dual recording, or backfilling the recording on the camera’s SD card (edge recording) into the VMS.
What is the difference between N+1 and 1+1 failover?
In the 1+1 model, each recording server has its own standby. In the N+1 model, several recording servers share a single standby. N+1 needs less hardware, but if two servers fail at once, only one is taken over.
Does recording stop when the management server stops?
In most enterprise VMSs it doesn’t; recording servers keep the configuration in a local cache and continue recording. What’s affected is client login, configuration changes and some alarm functions. Behavior varies by manufacturer; test and document it on your own system.
How often should failover testing be done?
Once during installation as part of the acceptance test, then at least once a year. It should also be repeated after a major VMS version upgrade, a server hardware change or a network topology change. In every test, write the measured recording gap duration in the test record.