introduction to tracking metrics -凯发k8网页登录

assignment and error metrics

the general workflow for evaluating a multi-object tracker in the presence of ground truth can be divided into 2 main steps. first, tracks are assigned to truths using an assignment algorithm. second, using the computed assignment as an input, state-estimation accuracy for each track is computed. in each step, some key metrics for tracker's performance can be obtained. in sensor fusion and tracking toolbox™, the metrics corresponding to these two steps are termed as assignment metrics and error metrics respectively.

the assignment metrics assesses the characteristics of a tracker resulting due to incorrect assignment of sensor data to tracks due to ambiguity, missed targets or presence of false alarms. incorrect assignment of sensor data can result in events like swapping of tracks between targets, creation of redundant tracks for a target, dropping of a true target track and creation of a false track. on the other hand, error metrics provide an assessment on state-estimation or object-level accuracy of the tracker. even when a tracker does a good job at assignment, inaccuracies at track level can happen due to noise in the measurements as well as inaccuracies in modeling the target's motion model and sensor measurement model.

tracks and truths definition

the assignment metrics use a gated greedy nearest-neighbor approach which allows many-to-one track assignments, which means that multiple tracks can be assigned to the same truth. at each step, the metrics aims to assign tracks to truths while accounting for their assignment in the previous time step. this helps the assignment metrics classify tracks into certain categories based on the current as well as previous assignments. the categories used by the assignment metrics for tracks are described in the images below.

the assignments metrics also represents truths as different categories of "targets" from the tracker's perspective. this helps the assignment metrics to record events such as track establishment and target break. the establishment and break events are shown in the image below.

by classifying tracks and truths into multiple categories, the assignment metrics output a cumulative analysis of the tracker's performance. the metrics also provide detailed information about assignment information for each truth and track. furthermore, the assignment results at each step from the assignment metrics can also be used to evaluate the error metrics. the error metrics output a cumulative analysis of a tracker's accuracy in state-estimation. similar to assignment metrics, error metrics also provide detailed information for each track and truth.

next, you will learn how to use the assignment and error metrics in matlab.

compute and analyze metrics

to use the assignment metrics, you create the trackassignmentmetric system object™. you also specify properties of the object using name-value pairs to configure the metric for different applications. in this example, you specify the assignment and divergence distance as absolute error in position between track and truth. you also specify the distance thresholds for assignment and divergence of a track.

assignmentmetrics = trackassignmentmetrics(...
    'assignmentdistance','posabserr',...
    'divergencedistance','posabserr',...
    'divergencethreshold',1,...
    'assignmentthreshold',0.85)

assignmentmetrics = 
  trackassignmentmetrics with properties:
    distancefunctionformat: 'built-in'
               motionmodel: 'constvel'
       assignmentthreshold: 0.8500
       divergencethreshold: 1
        assignmentdistance: 'posabserr'
        divergencedistance: 'posabserr'

to use the error metrics, you create the trackerrormetrics system object. since no name-value pairs are provided, the default values are used. in this example, the targets are assumed to be tracked using a constant-velocity model.

errormetrics = trackerrormetrics

errormetrics = 
  trackerrormetrics with properties:
    errorfunctionformat: 'built-in'
            motionmodel: 'constvel'

the assignment and error metrics are driven by simulated tracks and truths by using a helper class, helpermetricevaluationscenarios. an object of this class outputs a list of tracks and truths at each time step.

tracktruthsimulator = helpermetricevaluationscenarios;

you also visualize the results of assignment and error metrics using a helper class named helpermetricevaluationdisplay. these helper classes are included with the example in the same working folder as this script.

display = helpermetricevaluationdisplay(...
    'plottrackclassification',true,...
    'plottruthclassification',true,...
    'plottrackerrors',false,...
    'plottrutherrors',false,...
    'plotassignments',true,...
    'recordgif',true,...
    'errortoplot','posrms');

you generate the list of tracks and truths at each time and run the assignment and error metrics using the following workflow.

% time stamps at which the metrics are updated
timestamps = 0:0.1:10;
n = numel(timestamps);
% initialization of recorded variables
posrmse = zeros(n,1);
velrmse = zeros(n,1);
posanees = zeros(n,1);
velanees = zeros(n,1);
trutherror = cell(n,1);
trackerror = cell(n,1);
% loop over time stamps
for i = 1 : n
    % current time
    time = timestamps(i);
    
    % generate tracks and truths using simulation
    [tracks, truths] = tracktruthsimulator(time);
    
    % you provide the tracks and truths as input to the assignment metrics.
    % this outputs a cumulative summary from all the tracks and the truths.
    [trackassignmentsummary(i), truthassignmentsummary(i)] = assignmentmetrics(tracks, truths); %#ok 
    
    % for detailed assignment information about each track, you use the
    % trackmetricstable method.
    tracktable = trackmetricstable(assignmentmetrics);
    
    % similarly, for detailed assignment information about each truth, you
    % use the truthmetricstable method.
    truthtable = truthmetricstable(assignmentmetrics);
    
    % for running error metrics, you obtain the current assignment of
    % tracks to truths using the currentassignment method.
    [assignedtrackids, assignedtruthids] = currentassignment(assignmentmetrics);
    
    % the current assignment information as well as the list of tracks and
    % truths at the current time can then be passed as an input to the
    % error metrics.
    [posrmse(i),velrmse(i),posanees(i),velanees(i)] = errormetrics(tracks, assignedtrackids, truths, assignedtruthids);
    
    % for detailed error information about each record truth and track, you
    % use the cumulative metric methods on the error metrics.
    trackerror{i} = cumulativetrackmetrics(errormetrics);
    trutherror{i} = cumulativetruthmetrics(errormetrics);
    
    display(tracks, truths, assignedtrackids, assignedtruthids, tracktable, truthtable);
end

analyze assignment metrics

the animation below shows a visual representation of the assignment metrics. in the beginning, both p1 and p2 did not have any associated tracks due to establishment delay. therefore, they were both categorized as missed targets. around x = 5 meters, p1 was not tracked for a few time steps. notice that the assignment metrics outputs p2 as a dropped target during this time. when the tracks swapped between p1 and p2, they were initially declared divergent as they moved out of divergence thresholds. after a few time steps, the tracks reached assignment gate of new truths and became associated to the new truth.

the visualization of the metric for each track and truth can help in easier understanding of different events. however, it can become overwhelming for large number of tracks and truths. an alternate approach is to use the summarized output of the assignment metrics. these outputs provide an overall summary of the assignment metrics up to the current time step. the fields and values of the assignment metrics summary at end of the simulation is shown below.

disp(trackassignmentsummary(end));

           totalnumtracks: 5
           numfalsetracks: 1
             maxswapcount: 1
           totalswapcount: 2
       maxdivergencecount: 1
     totaldivergencecount: 3
      maxdivergencelength: 7
    totaldivergencelength: 17
       maxredundancycount: 1
     totalredundancycount: 1
      maxredundancylength: 34
    totalredundancylength: 34

disp(truthassignmentsummary(end));

              totalnumtruths: 2
            nummissingtruths: 0
      maxestablishmentlength: 5
    totalestablishmentlength: 10
               maxbreakcount: 2
             totalbreakcount: 4
              maxbreaklength: 9
            totalbreaklength: 9

by capturing the assignment summary at each step, you can also plot the cumulative analysis. the following plots show a subset of the fields for both track and truth assignment summary.

helperplotstructarray(display,trackassignmentsummary);

helperplotstructarray(display,truthassignmentsummary);

analyze error metrics

similar to assignment metrics, the error metrics can also be computed and visualized per track and truth. the fields of the estimation error for each track and truth at the end of simulation are shown below. each of these fields represent the cumulative error over each track and truth.

estimation error for each track

disp(trackerror{end});

    trackid    posrms      velrms     posanees    velanees 
    _______    _______    ________    ________    _________
       1       0.72278     0.87884     0.52241      0.77236
       2       0.16032    0.083823    0.025702    0.0070263
       3       0.52898      1.3192     0.27982       1.7403
       4       0.74411       2.112     0.55369       4.4604

estimation error for each truth

disp(trutherror{end});

    truthid    posrms     velrms    posanees    velanees
    _______    _______    ______    ________    ________
       1       0.52259    1.3918     0.2731      1.9372 
       2       0.58988     1.259    0.34795      1.5851 

by recording the errors at each time-step, you can also plot error vs time for each track and truth. the following plots show the rms error in position for each track and truth. after a track is deleted, the error in its estimate remains constant. notice that in the first-half of the simulation, the error in estimate of t3 is higher than that of t4 because the trajectory of t3 did not match the trajectory of p2 perfectly. also notice the drop in error for estimate of t3 in the first-half as it slowly converged to the true position. during the track-swap event. the error in estimate of both t3 and t4 increased till their assignments switched.

% a utility to plot the table array. 'posrms' is the field to plot
helperplottablearray(display,trackerror,'posrms');

the error in estimate of each truth is simply the cumulation of estimation error of all tracks assigned to it. when p2 was assigned a redundant track around 8th time-step of the simulation, its estimation error jumped. as the redundant track approached p2, the error decreases, but again increases when its associated track diverged.

helperplottablearray(display,trutherror,'posrms');

for a large number of tracks and truths, you can also use cumulative error metrics for all assigned tracks during their life-cycle. notice the increase in error when the tracks swapped (near the 80th step). also notice the smaller peak around the 25th time step when t1 diverted from its assigned truth p2 and was overtaken by the redundant track t3.

f = figure('units','normalized','position',[0.1 0.1 0.6 0.6]);
ax = axes(f);
subplot(2,2,1)
plot(posrmse,'linewidth',2);
xlabel('time step');
ylabel('rms error in position (m)');
grid('on');
subplot(2,2,2)
plot(velrmse,'linewidth',2);
xlabel('time step');
ylabel('rms error in velocity (m/s)');
grid('on');
subplot(2,2,3)
plot(posanees,'linewidth',2);
xlabel('time step');
ylabel('average normalized error in position');
grid('on');
subplot(2,2,4)
plot(velanees,'linewidth',2);
xlabel('time step');
ylabel('average normalized error in velocity');
grid('on');
sgtitle('cumulative errors for all tracks','fontweight','bold');

summarizing metrics as scores

a common use-case for evaluating the performance of a multi-object tracker is to tune its parameters. this is typically done by combining the metrics into a single cost value that can serve as a function for an optimization algorithm. the assignment and error metrics provide several different measures of effectiveness (moes) and can be combined by first choosing the most relevant metrics and then performing a weighted combination depending on the application. the selection of the correct moes as well as their combination into a single score can be challenging. as an alternative to this approach, you can use the optimal subpattern assignment (ospa) [1] metric and generalized optimal subpattern assignment (gospa) [2] metric. both ospa and gospa metric assess the performance of a multi-object tracker by combining both assignment as well as state-estimation accuracy into a single cost value. next, you will learn about ospa and gospa metric and the workflow to compute these metrics in matlab.

ospa metric

the ospa metric can be considered as a statistical distance between multiple tracks and truths. to compute the ospa metric, the algorithm first assigns existing tracks and truths to each other using a global nearest neighbor (gnn) algorithm. once the assignment is computed, the metric divides the overall distance into two sub-components - localization and cardinality-mismatch. the localization component captures the errors resulting from state-estimation accuracy, while the cardinality-mismatch component captures the effect of redundant tracks, false tracks and missed truths. the traditional ospa metric does not take into account the temporal history of tracks i.e. the assignments from previous step do not affect the metric at the current step. therefore, effects like track-switches are not captured in the traditional ospa metric. to circumvent this, a new sub-component was introduced for ospa called the "labeling" component [3]. combination of traditional ospa with "labeling" component is sometimes referred to as "ospa for tracks" (ospa-t) or labelled-ospa (lospa) [4].

to use the ospa metric in matlab, you use the trackospametric system object. you can switch from ospa to labelled-ospa by providing a non-zero valued labelingerror property. to understand how each sub-component is calculated, refer to the "algorithms" section of .

ospametric = trackospametric('distance','posabserr',...
    'cutoffdistance',1,...
    'labelingerror',0.25);

next you run the same scenario and compute the ospa metric at each time step.

timestamps = 0:0.1:10;
n = numel(timestamps);
% scene simulation
tracktruthsimulator = helpermetricevaluationscenarios;
% initialize variables
ospa = zeros(n,1);
loccomponent = zeros(n,1);
cardcomponent = zeros(n,1);
labelingcomponent = zeros(n,1);
% loop over time stamps
for i = 1:numel(timestamps)
    time = timestamps(i);
    
    % track and truth
    [tracks, truths] = tracktruthsimulator(time);
    
    % call the system object as a function and record ospa and all 3 components
    [ospa(i),loccomponent(i),cardcomponent(i),labelingcomponent(i)] = ospametric(tracks, truths);
end

analyze ospa metric

figure('units','normalized','position',[0.1 0.1 0.6 0.6]);
subplot(2,1,1);
plot(ospa,'linewidth',2);
xlabel('time step');
ylabel('cost');
title('ospa');
grid('on');
subplot(2,1,2);
plot([loccomponent cardcomponent labelingcomponent],'linewidth',2);
xlabel('time step');
ylabel('component cost');
title('ospa components');
legend('localization','cardinality-mismatch','labeling');
grid('on');

notice the correlation between ospa metric and different events in the scenario. the ospa is high initially because of establishment delay. after establishment, the ospa stays still relatively high due to the presence of a redundant track. after the redundant track was deleted, the ospa dropped to a lower value. around 50th time step the ospa gained value as truth p1 was dropped. notice that while the ospa metric captures all these events correctly by providing a higher value, it does not provide finer details about each truth and track and the accuracy in their estimate. the only information available from ospa is via its components. a higher localization component indicates that the assigned tracks do not estimate the state of the truths correctly. this localization component is computed using the same type of distance as the assignment. a higher cardinality component indicates the presence of missed targets and false or redundant tracks. a higher labeling error indicates that the tracks are not labelled correctly, which indicates that the tracks are associated to their closest available truths.

gospa metric

the approach used to compute gospa is similar to ospa metric. using a slightly different mathematical formulation, the gospa metric additionally computes subcomponents such as "missed targets component" and "false tracks component". similar to traditional ospa, the gospa also does not take into account temporal history of tracks. however, a metric similar to labelled-ospa can be achieved by adding a switching component [5]. the switching component captures the effect of switching assignments between truths. each assignment change for the truth is penalized after being categorized as half-switch or full-switch. a half-switch refers to the event when a truth switches assignment from a track to being unassigned or vice-versa. a full-switch refers to the event when a truth switches assignment from one track to another.

to use the gospa metric in matlab, you create the system object. to account for track-switching, you provide a positive value for switchingpenalty.

gospametric = trackgospametric('distance','posabserr',...
    'cutoffdistance',1,...
    'switchingpenalty',0.25);

next you run the same scenario and compute the gospa metric at each time step.

timestamps = 0:0.1:10;
n = numel(timestamps);
tracktruthsimulator = helpermetricevaluationscenarios;
% initialize variables
labeledgospa = zeros(n,1);
traditionalgospa = zeros(n,1);
loccomponent = zeros(n,1);
missedtargetcomponent = zeros(n,1);
falsetrackcomponent = zeros(n,1);
switchingcomponent = zeros(n,1);
% loop over time stamps
for i = 1:numel(timestamps)
    time = timestamps(i);
    
    % track and truth
    [tracks, truths] = tracktruthsimulator(time);
    
    % call the system object as a function and get all 4 components
    [labeledgospa(i),traditionalgospa(i),switchingcomponent(i),loccomponent(i),missedtargetcomponent(i),falsetrackcomponent(i)] = gospametric(tracks, truths);
end

analyze results

figure('units','normalized','position',[0.1 0.1 0.6 0.6]);
subplot(2,1,1);
plot(labeledgospa,'linewidth',2);
xlabel('time step');
ylabel('cost');
title('gospa');
grid('on');
subplot(2,1,2);
plot([loccomponent missedtargetcomponent falsetrackcomponent switchingcomponent],'linewidth',2);
xlabel('time step');
ylabel('component cost');
title('gospa components');
legend('localization','missed target','false track','switching');
grid('on');

notice that the gospa metric also captures the effect of different events during the scenario similar to the ospa metric. in contrast to ospa, it also provides information if the metric is higher due to false tracks or missed targets. notice the peak is missed target component around 50th time step. this peak denotes the event when p1 was missed for a few steps. the peak around 80th time-step in missed target component denotes the event when the tracks swapped. the delay between divergence and reassignment resulted in missed targets as well as a false track components.

the peaks in the track switching component denote different events. the first peak accounts for truths switching from unassigned to assigned. the second peak accounts for the switching of tracks on p2. the third and fourth peak captures that truth p1 was unassigned and then assigned to another track respectively. the last two peaks account for truth unassignment and then reassignment.

similar to ospa, gospa also does not provide detailed information about each track and truth. the information from gospa is available via its components. as the name states, a higher missed target component denotes that targets are not being tracked and a higher false track component denotes the presence of false tracks. a higher switching penalty denotes events like establishment, track swaps and dropped tracks. the subdivision of components to include missed targets and false tracks assists in modifying the correct parameter of the tracker. for example, if false tracks are being created, a typical solution is to try and increase the threshold for track-confirmation.

summary

in this example you learned about three different tools to analyze the performance of a multi-object tracking system. you also learned about the workflow in matlab to use these metrics. you also leaned about the type of information they provide and how it can be analyzed. the tools described in this example are flexible and can be customized for different applications and use-cases. there are "built-in" options available on all the tools to specify a different types of distances to be used for the metric. for example, absolute error in position or velocity or normalized error in position or velocity. you can also specify three different motion models for the tracks: constant velocity, constant turn-rate and constant acceleration. these "built-in" functionalities support tracks in form of objecttrack and truths generated by scenario simulations using trackingscenario. if the format of tracks and truths for your application is different, you can pre-process them to convert them. this allows you to use all the "built-in" functionality of the metrics. alternatively, you can specify a custom distance function between a track and a truth. this allows you to control the distance as well as the format of each track and truth. to switch between custom and built-in functionality for ospa and gospa metric, specify the distance as 'custom'. to switch between custom and built-in functionality for assignment and error metrics, change the distancefunctionformat and errorfunctionformat to 'custom' for assignment and error metrics respectively.

references

[1] schuhmacher, dominic, ba-tuong vo, and ba-ngu vo. "a consistent metric for performance evaluation of multi-object filters." ieee transactions on signal processing 56.8 (2008): 3447-3457.

[2] rahmathullah, abu sajana, ángel f. garcía-fernández, and lennart svensson. "generalized optimal sub-pattern assignment metric." 2017 20th international conference on information fusion (fusion). ieee, 2017.

[3] ristic, branko, et al. "a metric for performance evaluation of multi-target tracking algorithms." ieee transactions on signal processing 59.7 (2011): 3452-3457.

[4] mahler, ronald ps. advances in statistical multisource-multitarget information fusion. artech house, 2014.

[5] garcia-fernandez, angel f., et al. “a metric on the space of finite sets of trajectories for evaluation of multi-target tracking algorithms.” ieee transactions on signal processing, vol. 68, 2020, pp. 3917–28.