In video analytics projects, the longest arguments start when the question “is the system working well?” has no measurable answer. The operator complains about false alarms, the contractor says the installation is correct, and the client doesn’t know what they accepted. Everyone is right, because nobody kept count. Below we give, step by step, how precision, recall and the false alarm rate are calculated for a video analytics test, how a representative test set is built and how the acceptance criterion is written.
Why a measurable acceptance criterion
Analytics is a probabilistic system: in each frame it answers the question “is this a person?” with a confidence score and turns outputs above the threshold into alarms. Lower the threshold and fewer events are missed, but false alarms rise; raise it and the opposite happens. That’s why a target of “100% accurate” can’t be written. What can be written is how much tolerance there is for which type of error.
A threshold is always a trade-off.
A measurable criterion has three concrete benefits on site:
- Contractual clarity: Instead of “high accuracy,” the specification contains a verifiable statement such as “virtual fence intrusion recall of at least X in night conditions.”
- Justification for tuning decisions: When a threshold, zone or duration parameter changes, its effect is measured on the same data; the decision isn’t made on impressions.
- Comparison over time: Whether performance has dropped after a firmware or model update is seen with the same test set.
The subject here is measuring analytics accuracy. For the general acceptance process covering image, recording, network and access tests, see the camera project acceptance test guide.
Precision, recall and other metrics
Every test event and every system alarm falls into one of four classes. In event-level assessment, an alarm counts as matched if it occurred on the same camera as the real event and within a predefined time window. If this window isn’t written down before the test starts, the argument over results never ends either.
| Class | Meaning |
|---|---|
| True positive (TP) | There’s a real event, and the system raised an alarm |
| False positive (FP) | There’s no real event, and the system raised an alarm |
| False negative (FN) | There’s a real event, and the system raised no alarm |
| True negative (TN) | No event, no alarm |
The basic metrics derived from these classes:
- Precision = TP / (TP + FP): How many of the alarms raised are real. This determines the operator’s trust in alarms.
- Recall = TP / (TP + FN): How many of the real events are caught. The miss rate is 1 − recall. Missed events are measured against the number of known (staged or labeled) site scenarios.
- F1 score: The harmonic mean of precision and recall. Useful for comparison with a single number; in an acceptance criterion, however, writing the two metrics separately is clearer.
- Notification latency: The time from the event starting in the scene to the notification reaching the recipient. It’s measured and reported separately from precision and recall.
- False alarm rate: Given per camera together with the monitoring time (for example alarms/camera-hour); a figure given without the time can’t be compared.
- False alarm rate: The number of FPs divided by the number of camera-hours or camera-days monitored. In video surveillance, the number of TNs is undefined (“events that didn’t happen” can’t be counted); that’s why this time-normalized metric is used instead of the classic false positive rate.
In frame-level object detection assessment, matching is done by the intersection over union (IoU) of the detection box and the reference box. In site acceptance, event-level measurement is more meaningful, because the operator sees the alarm, not the frame. For general definitions of the metrics, the scikit-learn model evaluation documentation is a solid reference.
Worked example
The values below are a constructed worked example to illustrate the method. In a 16-camera perimeter security project, let a total of 200 intrusion events, staged and naturally occurring, be labeled on a 72-hour test recording. Let the system have produced 230 alarms in the same period, of which 180 matched real events.
- TP = 180, FP = 230 − 180 = 50, FN = 200 − 180 = 20
- Precision = 180 / 230 ≈ 0.78
- Recall = 180 / 200 = 0.90 (miss rate 0.10)
- F1 ≈ 0,84
- Monitored time = 16 × 72 = 1,152 camera-hours; false alarm rate = 50 / 1,152 ≈ 0.043 alarms/camera-hour, or about 1 false alarm per camera per day.
The number of events determines how reliable the measured value is. The 95% confidence interval calculated with the Wilson method is about 0.85–0.93 for recall in this example. Had the same rate been measured on 20 events (18/20), the interval would be about 0.70–0.97. Nobody signs acceptance with an interval that wide. You can repeat the calculation with this short script:
import math
def wilson(k, n, z=1.96):
p = k / n
d = 1 + z**2 / n
c = (p + z**2 / (2 * n)) / d
h = z * math.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / d
return round(c - h, 3), round(c + h, 3)
tp, fp, fn = 180, 50, 20
camera_hours = 16 * 72
print("precision", round(tp / (tp + fp), 3))
print("recall", round(tp / (tp + fn), 3), "CI", wilson(tp, tp + fn))
print("FP/camera-hour", round(fp / camera_hours, 3))
A representative test set
A measurement is only as valuable as how well the test set represents the site. Analytics measured only on recordings shot in daylight and clear weather behaves completely differently on the first rainy night. Night or rain performance can’t be inferred from a daytime pilot: if acceptance is done on a sunny afternoon, night, fog and rain conditions stay unmeasured. Divide the test set into condition groups and calculate separate metrics for each group:
- Light: day, dusk, IR night mode, backlight, vehicle headlights.
- Weather: rain, fog, snow, strong wind (swaying vegetation and flags); can be added to the test plan later, depending on the season.
- Density: empty scene, sparse traffic, crowds and occlusion.
- Target behavior: normal walking, running, crouching or crawling, dark camouflage-like clothing, waiting at the zone boundary.
- Negative scenarios: animals, shadows, reflections, insects and spider webs, IR reflecting off raindrops. False alarm measurement can’t be done without these.
Typical sources of false alarms on site and methods to reduce them are in the how to reduce false alarms in video analytics article; you can derive the negative scenarios in the test set directly from that list.
Test protocol
- Write down the scope and the event definition: For each analytics rule, the definition of an “event” must be clear. For example, in a virtual fence an event is a person entering the zone for longer than a set time; zone and duration parameters are fixed in advance using the approach in the virtual fence and area intrusion rule design article.
- Freeze the configuration: Before the test starts, the firmware version, model version, threshold and zone settings are recorded. Nobody touches them during the test.
- Define the matching rule: The time window accepted for an alarm to match a real event (for example a few seconds after the event starts) and how repeated alarms for the same event are counted are written down.
- Produce and record events: Staged scenarios are run at planned times under defined conditions. Continuous recording with natural traffic is also added to the set.
- Label the reference independently: The list of real events is prepared by watching the recording, without seeing the system alarms. If possible, two people label, and disagreements are resolved separately.
- Match and calculate: The alarm log is exported from the VMS, matched against the reference list, and metrics are calculated per condition group.
- Report: The report includes the configuration record, event counts, metrics and confidence intervals, and a short analysis of examples of missed events and false alarms.
In a staged test, the team unknowingly drifts toward “easy” scenarios: walking straight, through the middle, directly in front of the camera. Deliberately put zone edges, long distances and crossing perpendicular to the camera on the scenario list. No hard scenarios, no test.
Acceptance criteria table
Acceptance thresholds are set with the project team according to the scenario’s risk profile. The table shows how the criterion should be written; you fill in the values for your project.
| Analytics | Primary metric | Secondary metric | Condition breakdown |
|---|---|---|---|
| Virtual fence / intrusion | Recall ≥ target | FP/camera-day ≤ target | Day, night, rain |
| Line crossing | Recall and direction accuracy | Precision | Sparse, crowded |
| People counting | Entry and exit counting error | Total hourly deviation | Busy hours, quiet hours |
| PPE detection | Precision | Recall | Distance, light |
| Vehicle–pedestrian proximity | Recall | FP/shift | Aisle, intersection |
For statistical analytics such as people counting, counting deviation is used instead of event-level metrics; we explained the method in detail in the measuring accuracy in people counting analytics article. In the acceptance record, the measured value, confidence interval and “pass/fail” result for each criterion should sit side by side.
Continuous monitoring
Performance on acceptance day isn’t permanent. A new rack is added to the site, trees grow, the camera shifts a few degrees from vibration, a firmware update arrives. The metrics change silently. After acceptance, set up these mechanisms:
- Operator feedback: Each alarm is marked by the operator as “real / false.” These marks give weekly precision and false alarm rate per camera.
- Periodic miss test: Recall can’t be measured from alarm logs alone; a missed event doesn’t appear in the log. At regular intervals, run a staged test walk or review a random sample of recordings.
- Regression test: The acceptance test set is archived; when the model, firmware or camera settings change, the same set is run again.
- Threshold change log: Which parameter was changed on which camera, by whom and based on which measurement is recorded.
Test plan, measurement and acceptance record processes can be carried out under camera project management and acceptance testing, and setting up and tuning analytics rules under AI video analytics deployment.
Frequently asked questions
Why doesn’t the accuracy figure given by the manufacturer hold on site?
Because that figure is measured on the manufacturer’s own test data under its own conditions. Camera angle, the target’s pixel size, lighting, weather and misleading objects in the scene differ on every site. Make the acceptance decision with a test set built from the project’s own cameras and conditions.
How many events should the test set contain?
There’s no fixed number; the number of events determines the confidence interval of the measurement. For example, 18 correct detections out of 20 events gives a recall in a range of about 0.70–0.97, while 180 correct detections out of 200 events narrows this range to about 0.85–0.93. If a condition group doesn’t have a meaningful number of events, no acceptance decision can be made for that group.
Which matters more, precision or recall?
It depends on the scenario. In scenarios where the cost of a missed event is high, such as perimeter security, recall takes priority; false alarms are limited by operator workload. For people counting or statistical analytics, the balance of errors in both directions and the total counting error are more meaningful.
Should the test be repeated after acceptance?
Yes. Firmware or model updates, camera angle drift, new objects added to the site and seasonal change alter performance. Archive the acceptance test set and run a regression test on the same set after these changes.