Smart Video Search with Metadata: Attribute-Based Incident Review

How does smart video search work on site? Attribute generation, metadata storage, night and angle errors that make you miss results, the incident review flow and privacy limits.
Technical summary
Technology
VMS, video analytics, metadata indexing, ONVIF Profile M, NTP
Topic
Incident review in recordings with attribute-based search
Level
Intermediate
Prerequisites
  • Recording and export permission on the VMS/NVR
  • Camera layout plan
  • Recording retention policy
In this article7 sections

The description that comes in after an incident is always equally vague: “someone in a red jacket who left by the dock in the afternoon.” Watching hours of recordings from dozens of cameras from the start with that description is the job on which an investigation team loses the most time. Metadata-based smart video search narrows the range to watch down to minutes by querying object and attribute information generated during recording. But it only finds what the analytics saw; below we explain how search works, where metadata is generated and stored, why results mislead and an investigation flow that works on site.

How metadata-based search works

Classic motion search lists moments with pixel changes in the selected area. A tree swaying in the wind, cloud shadows and vehicle headlights are pixel changes too; that’s why the list comes back crowded. Metadata search doesn’t query the video at all. During recording, the analytics engine processes every frame (or frames sampled at intervals) and writes these to a separate record:

  • Object class: classes the model recognizes, such as person, vehicle, bicycle, forklift.
  • Position: the object’s bounding box and timestamp in each frame.
  • Track ID: a number linking the same object’s positions across consecutive frames, meaningful only within that camera.
  • Attributes: classifier outputs and confidence scores such as color, vehicle type, direction of movement, estimated speed and the presence of a hat or bag.

When a search is run, the VMS queries this index: the query “Cameras 12–18, 14:00–16:00, class=person, upper clothing=red, direction=exit” returns a list of tracks in a few seconds. The operator sees a thumbnail and short clip for each track, opens the relevant one and watches the actual video. There’s a fact here that mustn’t be skipped: search queries not the video, but the summary the analytics wrote about the video. If the analytics missed the object or labeled the attribute wrongly, that recording never appears in the results list.

If it isn’t in the index, it isn’t in the search.

Some platforms offer similarity search alongside attribute search: the operator selects an image of a person or vehicle, and the system compares appearance feature vectors to rank similar images on other cameras. It saves serious time in cross-camera tracking. On the other hand, these vectors are more sensitive personal data than attribute labels; we come back to this in the privacy section.

Attributes generated

Which attributes are generated varies by manufacturer and model. The table shows common attribute groups and the typical condition on site that undermines each one’s reliability. When choosing a product, ask one question for each row: will we really use this attribute?

Attribute Typical values Condition limiting reliability
Object class Person, vehicle, bicycle Small targets, heavy occlusion
Clothing color Upper/lower clothing, color family IR mode, white balance, colored light
Accessories Hat, bag, backpack Target’s pixel size, angle
Vehicle type Car, van, truck, bus Partial visibility, unusual body shapes
Vehicle color Color family Reflections, headlights, night
Direction and speed of movement Direction vector, relative speed Perspective, lack of calibration
Zone/line event Zone entry, line crossing Track breaks, accuracy of zone drawing

Color is a classifier output, not a measurement. The system can label a navy jacket “black” and a gray car at sunset “orange.” That’s why well-designed search screens allow multiple color selections and lowering the confidence threshold. An operator who doesn’t know these two settings thinks a search that has results is “empty.”

Generating and storing metadata

Metadata can be generated in three places. The choice determines both cost and whether retrospective search is possible:

  1. On the camera (edge): An analytics-capable camera publishes a metadata stream alongside the video stream. On the ONVIF side, analytics metadata and event interfaces are defined by ONVIF Profile M. However, the VMS indexing this stream and making it searchable is a separate capability; don’t accept it without testing on the camera–VMS pair.
  2. On the recording or analytics server: An AI video analytics service connected to the VMS decodes live streams, runs the model and writes the index directly to the VMS database. It’s independent of camera brand; in return, it needs GPU and network capacity.
  3. Retrospective processing: After an incident, only the recording from the selected cameras and time range is analyzed. If there’s no index in advance, this is the only option; the time depends on the amount of video to process and the hardware.

This decision isn’t made separately from the overall placement of analytics; we compared the two approaches in the edge analytics or server-based analytics article.

The points most often skipped on the storage side:

  • Volume: Metadata is small compared with video, but in busy scenes track and per-frame box records grow quickly. Measure daily index size on pilot cameras and size the database disk accordingly.
  • Link to video: The timestamp in the index points to the video. If camera clocks aren’t synchronized via NTP, metadata and video drift apart and tracking across cameras gets harder. The first place to check is NTP.
  • Retention period: Metadata left after the video is deleted turns into an unverifiable summary. Define the retention period together with video, and apply automatic deletion to both.
  • Backup and high availability: The VMS failover setup must also cover the index database. If it doesn’t, after switching to the standby server the video is there but can’t be searched.

What limits search accuracy

Everyone expects smart search to “find everything.” In reality, search finds as much as the analytics saw. Common limiting factors:

Pixels and angle

Attribute classifiers need more pixels than object detection. A camera installed with a wide angle for general coverage detects a person but may not reliably distinguish a backpack or clothing color. Cameras that show the target large at passage points such as entrances, turnstiles and doors are the most efficient source for search. We explained the effect of camera geometry with numerical examples in the effect of camera angle and resolution on analytics accuracy article.

Light and color

In IR-lit night mode, the image is monochrome; the color attribute is either not generated at all or generated meaninglessly. Sodium vapor lamps or colored LED lighting also shift color perception. Searching night recordings with a color filter is the most common reason for missing results.

Track breaks and occlusion

In a crowd or when passing behind a column, the track breaks and the same person is recorded as two separate tracks. The number of results rises, but the real problem is something else: the attribute may have been calculated correctly on only one track, and the other track may not have matched the filter. The person appears once in the list, and their second pass is lost.

Analysis frame rate and version

To save resources, analytics may run at a few frames per second. In that case, a vehicle passing quickly appears in one or two frames and its attributes are poorly estimated. After a model update, the generated attributes may also not have the same distribution as earlier records. In a search spanning months, this difference distorts the result.

Incident review workflow

In enterprise camera systems, smart search produces value not on its own but within a defined review process. The flow that works on site:

  1. Open the incident record: The requester, the description of the incident, estimated time and place, the purpose of the review and the authority basis are recorded. Search permission is tied to this record.
  2. Define the scope: Cameras seeing the incident location and cameras at entry and exit points are selected. The time range is kept wide enough to cover before and after the estimated time of the incident; the time given by the person describing it is often wrong.
  3. Start with a broad filter: Search first with reliable attributes such as class, zone and direction; add color and accessory filters in the second round.
  4. Verify the result from video: Each candidate track is assessed by watching the recording itself, not from the thumbnail. Verified moments are bookmarked.
  5. Build the chain across cameras: From the verified image, move to neighboring cameras using time and direction information; similarity search, if available, is used here.
  6. Protect and export the evidence: The relevant clips are exported in a form that allows integrity verification, and an action log is kept. Details of this step are in the camera recording incident investigation and chain of custody article.
  7. Write a closing note: The filters used, the cameras reviewed and the result are added to the incident record in a short note. This is the first document to look at when the same incident is reopened or in an audit.

A fallback method should be defined for situations search can’t find: motion search or accelerated playback on critical cameras. The team should know this distinction: “the search returned nothing” and “the incident didn’t happen” aren’t the same thing.

For platform selection, indexing architecture and the setup of review screens, you can find the scope on the smart video search and incident review service page.

Metadata and privacy

Metadata can point to a person even without video. A record of “the person in a black jacket with a backpack who entered through door A at 08:02” can identify a single person among the limited number of employees using that door that day. Treat metadata within the same data protection framework as video.

  • Access permission: The search screen is a more powerful tool than live viewing. Open it only to the review role; have every query written to the audit trail with the user, time and criteria.
  • Purpose limit: Use process and permission design to prevent infrastructure set up for security incident review from drifting into tracking employee behavior.
  • Attribute selection: Turn off attributes you don’t use. Attributes such as age or gender estimation contribute little to incident review but increase privacy risk.
  • Similarity and face search: Appearance vectors, and especially face templates, open the biometric data discussion; we covered the technical limits separately in the KVKK limits in video analytics article.
  • Masking: Verify per product whether analytics runs in areas with a privacy mask applied. If attributes are generated in a masked area, there’s a data flow nobody expects.

Frequently asked questions

Does metadata search also work on night recordings?

Partly. Object class, direction and zone information are generated at night too. The moment the camera switches to black-and-white mode with IR, the color attribute becomes unreliable and the color filter misses results. In night searches, work with class, direction, time and zone filters instead of color.

Can metadata be kept longer than the video recording?

Technically yes, but metadata is also a record that can be linked to a person. Its retention period must be justified by the purpose and defined in the policy together with video. If you can’t explain why you keep metadata after the video is deleted, deleting both together is the more defensible route.

Can smart video search be done with existing cameras?

If the cameras don’t produce analytics metadata, attribute extraction is done on the server side, from the live stream or recorded video. Older recordings need retrospective processing, and GPU capacity is planned accordingly. The camera still determines the quality of the result: if it doesn’t see the target with enough pixels and from the right angle, search stays weak too.

Does a search result count as evidence on its own?

No. A search result only shows candidate recordings to watch. The event is established when an operator watches and verifies the relevant video; what’s used as evidence isn’t the metadata but the video recording exported with its integrity preserved.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek Video Surveillance

IP cameras, recording and central management infrastructure. We plan blind spots, retention periods and the camera network around the risks of each site.

Knowledge Center

Other articles