1 Introduction↩︎

With the development of mobile communications, video has become a powerful medium to record life and transform information. As a result, video understanding technologies have aroused increasing research interests. Among these technologies, temporal action detection [2]–[4] can discover action instances from untrimmed videos and extract valuable information. Well-performed action detection algorithms can benefit smart surveillance [5], anomaly detection [6] In recent years, along with action detection technologies becoming mature, a more challenging but more practical task, namely online action detection, has been proposed [7]. The online action detection algorithm tackles a streaming video, reports the occurrence of an action instance, and keeps alarming until the action ends [7]. In inference, the algorithm only employs historical frames that have been observed, but has no access to future frames.

As an early exploration, Geest [7] discovered the importance of modeling long-term dependencies. Later, Xu [8] revealed the value of anticipating future status to enhance the long-term dependencies modeling. OadTR [1] recently utilized the multi-head self-attention module to jointly model historical dependencies and anticipate the future, which achieved promising online action detection results.

As an under-explored domain, there are three core challenges for online action detection: How to model long-term dependencies? How to associate a frame with representative frames from the same category? How to conduct detection efficiently? Existing works [1], [7], [8] primarily focus on the long-term dependencies modeling, but ignore the other two challenges. However, as shown in Figure 1 (a), both analyzing historical frames and anticipating future status only model relationships within a video segment, leaving the category-level modeling under-explored. Because an action category contains multiple instances and each instance exhibits special appearance and motion characteristic, the guidance of exemplary frames can make the online detection algorithm more robust to resist noises within a video segment. In addition, a practical online action detection algorithm should always consider the computational efficiency, including both the efficiency to perform online detection and the efficiency to extract video features.

This paper develops an exemplar-consultation mechanism to tackle above three challenges in a unified framework. The exemplar-consultation mechanism first jointly transforms a frame and its exemplary frames to the key space and value space. Then, it measures the similarity in the key space and employs the similarity to aggregate information in the value space. As both feature transformation and similarity measurement require limited computations, the proposed exemplar-consultation mechanism is efficient. Considering a video segment, we can effectively model long-term dependencies by using historical frames as exemplars based on the exemplar-consultation mechanism. As we only compare one frame with its historical frames, rather than performing self-attention on all frames, the computational burden is alleviated. Similarly, we can also regard representative frames of each category as exemplars and conduct category-level modeling based on the exemplar-consultation mechanism. Compared with a video segment, category exemplars can provide complementary guidances and make the algorithm more robust.

By consulting exemplars, we build a unified framework, namely Colar, to perform online action detection, as shown in Figure 2. Colar maintains the dynamic branch and the static branch in parallel, where the former models long-term dependencies within a video segment and the latter models category-level characteristics. In the dynamic branch, Colar consults previous frames and aggregates historical features. In the static branch, Colar first obtains category exemplars via clustering, then consults exemplars and aggregate category features. Finally, two classification scores are fused to detect actions. Moreover, we analyze the running time bottleneck of existing works and discover the expensive costs to extract flow features. Thus, we employ a spatio-temporal network to only dispose of video frames and perform end-to-end online action detection, which only takes 9.8 seconds to tackle a one-minute video. To sum up, this paper makes the following contributions:

• We make an early attempt to conduct category-level modeling for the online action detection task, which provides holistic guidance and makes the detection algorithm more robust.

• We propose the exemplar-consultation mechanism to compare similarities and aggregate information, which can efficiently model long-term dependencies and category particularities.

• Due to the effectiveness of the exemplar-consultation mechanism and the complimentary guidance from category-level modeling, our method employs a lightweight architecture. Still, it achieves superior performance and builds new state-of-the-art performance on three benchmarks.

2 Related work↩︎

Modeling temporal dependencies. Different from image-based task, detection [9]–[12], localization[13]–[15] and segmentation[16], it is crucial to model temporal dependencies for online action detection. Existing works rely on recurrent networks, including both LSTM-based methods [8], [17], [18] and GRU-based methods [19]. Specifically, Geest [17] proposed a two-stream LSTM [20] network. Similarly, TRN [8] employed LSTM blocks to model historical temporal dependencies. Recently, OadTR [1] drove the recurrent-network paradigm into a transformer-based paradigm and effectively captured the long-term relationship via self-attention. Although OadTR [1] effectively models long-term dependencies, the self-attention process for all frames leads to the computational burden problem. This work regards historical frames as exemplars and utilizes the exemplar-consultation mechanism to model long-term dependencies.

Anticipating future. Although online action detection algorithms cannot access future frames, anticipating future features can assist the decision of current frame. In RED [21], Gao estimated features for future frames and calculated the classification loss and the feature regression loss to improve the anticipation quality, which is further developed by TRN [8] and OadTR [1]. In this paper, the static branch employs the exemplar-consultation mechanism to compare a frame with representative exemplars of each category and brings complementary information to the dynamic branch.

Offline action detection. The offline action detection algorithm aims to discover action instances from untrimmed videos [22]–[24], where all video frames can be utilized. Some algorithms [2], [3], [25], [26] tackled video frames to perform localization. In addition, a majority of works [4], [27] first extracted video features from powerful backbone networks[28]–[31], then performed action localization based on video features. From the view of anchor mechanism, the representative works include anchor-based methods [4], [27] and anchor-free methods [32]–[34]. Besides, multiple effective modules have been proposed, graph convolutional module [35], [36]. Moreover, action detection under the weakly supervised setting [37]–[40] was also well explored.

The primary difference between online action detection and offline action detection algorithms lies in whether future frames can be accessed. In offline algorithms, Xu [25] performed data augmentation via playing the video in reverse order, while Zhu [4] modeled the relationship among multiple proposals within a video. However, these procedures are unsuitable for the studied online action detection task.

Space-time memory network. Oh [41] proposed the space-time memory network to efficiently connect a frame and its previous frames via space-time memory read. It has verified effective performance on modeling temporal information, and has been extended to multiple tasks, video object detection [42], video object segmentation [43], [44], tracking [45]. In contrast to the space-time memory network, our static branch employs the exemplar-consultation mechanism to model the intra-category relationship. Specifically, it first aggregates particular features from each category, and then combines multiple features to obtain the category feature.

3 Method↩︎

Given a video stream, the online action detection algorithm should report the occurrence once the action starts and keep alarming until the action ends. The learning process is guided by the frame-level classification label \(\mathbf{y}=[y_{0}, y_{1}, ..., y_{C}]\), where \(y_{c} \in \{0, 1\}\) indicates whether frame \(\mathbf{f}_{0}\) belongs to the \(c^{th}\) category. As shown in Figure 2, we first employ a backbone network to extract video features. Then, we propose the dynamic branch to model long-term dependencies within a segment and propose the static branch to capture the holistic particularity for each category. Finally, two detection results are fused to perform the online action detection task.

3.1 Dynamic branch↩︎

As neighboring frames can provide rich contextual cues to determine the category label of the current frame, the core idea of the dynamic branch is to model local evolution by comparing a frame with its previous historical frames and dynamically aggregating the local features. The upper part of Figure 2 exhibits the detailed operations in the dynamic branch. Compared with the standard multi-head self-attention mechanism of OadTR[1], our proposed dynamic branch makes two reasonable designs, which sufficiently benefit the online action detection task. First, we use temporal convolution with kernel size 3 to model local cues among historical frames, which is complementary to the global modeling of self-attention. Second, we make two simplifications over OadTR, removing the class token and replacing multi-head self-attention with one-head attention on the current frame. The simplifications reduce learning difficulty and benefit the performance when training data is not rich enough.

Given a video feature sequence, we first transform a feature \(\mathbf{f}_{t}\) to the key space and the value space, where the former is responsible for comparing similarity, and the latter can be used for feature aggregation. \[\mathbf{f}_{t}^{k} = \Phi^{k}(\mathbf{f}_{t}), \;\;\mathbf{f}_{t}^{v} = \Phi^{v}(\mathbf{f}_{t}),\] where \(\Phi^{k}\) and \(\Phi^{v}\) indicate two convolutional layers in the dynamic branch. Then, we measure the pair-wise affinity between \(\mathbf{f}_{0}^{k}\) and other key features (\(\mathbf{f}_{t}^{k}\)) via calculating the cosine similarity: \[\mu_{t} = cos(\mathbf{f}_{0}^{k}, \mathbf{f}_{t}^{k}) = \frac{\mathbf{f}_{0}^{k} \cdot \mathbf{f}_{t}^{k}}{||\mathbf{f}_{0}^{k}|| \cdot ||\mathbf{f}_{t}^{k}||}.\]

Given a series of affinity values \([\mu_{-T}, ..., \mu_{-1}, \mu_{0}]\), we perform softmax normalization and obtain the attention mask \([\hat{\mu}_{-T}, ..., \hat{\mu}_{-1}, \hat{\mu}_{0}]\). As each element \(\mu_{t}\) indicates the similarity between the previous \(t^{th}\) frame and the current frame, we can aggregate value features among previous frames and obtain the historical feature \(\mathbf{f}\): \[\mathbf{f} = \sum_{t=-T}^{0} \hat{\mu}_{t} \cdot \mathbf{f}_{t}^{v}.\] In the end, the dynamic branch jointly considers value feature \(\mathbf{f}_{0}^{v}\) and the historical feature \(\mathbf{f}\) (via summation) and conduct online action detection: \[\mathbf{s}^{\rm d} = \Omega^{\rm d}(\mathbf{f}_{0}^{v}, \mathbf{f} | \Theta^{\rm d}),\] where \(\Omega^{\rm d}\) is the classifier in the dynamic branch with parameter \(\Theta^{\rm d}\), and \(\mathbf{s}^{\rm d} \in \mathbb{R}^{C+1}\) is the classification score from the dynamic branch.

3.2 Static branch↩︎

Considering action instances from the same category, some instances with distinctive appearance characteristics and clear motion patterns can be selected as exemplars to represent this category. We employ the K-means clustering algorithm for each category, carry out clustering, and obtain \(M\) exemplary features. On this basis, the online action detection task can be formulated as comparing a frame with representative exemplars of each category. As a result, the static branch can provide complementary cues to the dynamic branch and makes the online detection algorithm robust to noises within the local video segment.

Before stepping to detailed operations in the static branch, it is necessary to analyze its efficiency. First, using another branch increases a certain computation. However, compared with OadTR[1], we not only simplify the attention computation but also remove decoder layers. Thus, our holistic computation is smaller than OadTR[1], and we require less memory as well (see experiments in Sec.4.2). In addition, even given a dataset with millions of samples and thousands of categories, the modern implementation[46] of the K-Means algorithm can still efficiently generate exemplars, as verified by DeepCluster[47].

As shown in the bottom part of Figure 2, the static branch operates with the category exemplars \(\{\mathcal{E}_{c} = [\mathbf{e}_{c, 1}, \mathbf{e}_{c, 2}, ..., \mathbf{e}_{c, M}]\}_{c=0}^{C}\) to classify feature \(\mathbf{f}_{0}\), where each category contains \(M\) representative exemplars. At first, we convert each exemplar \(\mathbf{e}_{c,i}\) to the key space and the value space: \[\mathbf{e}_{c,i}^{k} = \Psi^{k}(\mathbf{e}_{c,i}), \;\;\mathbf{e}_{c,i}^{v} = \Psi^{v}(\mathbf{e}_{c,i}),\] and convert the frame feature \(\mathbf{f}_{0}\) to the key space and value space as well: \[\mathbf{e}_{0}^{k} = \Gamma^{k}(\mathbf{f}_{0}), \;\;\mathbf{e}_{0}^{v} = \Gamma^{v}(\mathbf{f}_{0}),\] where \(\Psi^{k}\), \(\Psi^{v}\), \(\Gamma^{k}\) and \(\Gamma^{v}\) indicate convolutional layers. In the key space, we can measure the similarity between feature \(\mathbf{f}_{0}\) and exemplar \(\mathcal{E}_{c}\) from the \(c^{th}\) category: \[\nu_{c,i} = cos(\mathbf{e}_{0}^{k}, \mathbf{e}_{c,i}^{k}) = \frac{\mathbf{e}_{0}^{k} \cdot \mathbf{e}_{c,i}^{k}}{\|\mathbf{e}_{0}^{k}\| \cdot \|\mathbf{e}_{c,i}^{k}\|}.\] Based on the pair-wise similarity between \(\mathbf{e}_{0}^{k}\) and all exemplars \([\mathbf{e}_{c, 1}, \mathbf{e}_{c, 2}, ..., \mathbf{e}_{c, M}]\) from the \(c^{th}\) category, we can first calculate the attention mask \([\hat{\nu}_{c, 1}, ..., \hat{\nu}_{c, M}]\) via softmax normalization, and then aggregate all exemplars to represent the current frame from the perspective of the \(c^{th}\) category: \[\mathbf{e}_{c} = \sum_{i=1}^{M} \hat{\nu}_{c, i} \cdot \mathbf{e}_{c,i}^{v}.\]

After comparing the current frame with representative exemplars of all categories, we obtain category-specific features \([\mathbf{e}_{0}, \mathbf{e}_{1}, ..., \mathbf{e}_{C}]\). Considering a feature from the \(c^{th}\) category, it would be similar to exemplars from the \(c^{th}\) category while be different from other exemplars. Thus, we use a convolutional layer to estimate the attention weight \(\mathbf{a} \in \mathbb{R}^{C+1}\) and aggregate category feature \(\mathbf{e}\): \[\mathbf{e} = \sum_{c=0}^{C} a_{c} \cdot \mathbf{e}_{c}.\]

The exemplary feature \(\mathbf{e}\) is generated from all exemplars and can reveal the category characteristics. In the end, the static branch employs both value feature \(\mathbf{e}_{0}^{v}\) and category feature \(\mathbf{e}\) to predict the classification score \(\mathbf{s}^{\rm s}\): \[\mathbf{s}^{\rm s} = \Omega^{\rm s}(\mathbf{e}_{0}^{v}, \mathbf{e} | \Theta^{\rm s}),\] where \(\Omega^{\rm s}\) is the classifier with parameter \(\Theta^{\rm s}\).

3.3 Efficient online action detection↩︎

Given a series of pre-extracted video features, the dynamic branch connects a frame with its historical neighbors and models local evolution, while the static branch compares a frame with representative exemplars and models category particularity. It is convenient to fuse the predictions of two branches and detect actions online. However, the feature extraction process, especially calculating optical flows, requires heavy computations, which prevents us from conducting online action detection in practical scenarios.

To alleviate the computational burdens, we can employ a spatio-temporal network to tackle video frames and provide representative features for the dynamic and the static branch. Considering the video recognition performance and calculation efficiency, we utilize the ResNet-I3D network [48], discard the last classification layer, and construct our feature extraction backbone. Given a video sequence with \(T\) frames, the output of the backbone network is \(\mathbf{x} \in \mathbb{R}^{D \times T/8}\), where \(D\) indicates the feature dimension. In practice, as the benchmark datasets contain limited training videos, we find frozen the first three blocks can produce more accurate detection results.

3.4 Training and inference↩︎

Given a frame, the dynamic branch and the static branch predict its classification score \(\mathbf{s}^{\rm d}\) and \(\mathbf{s}^{\rm s}\), respectively. We calculate the cross-entropy loss to guide the learning process: \[\mathcal{L}_{cls}^{\rm d} = -\sum_{c=0}^{C} \mathbf{y}_{c} {\rm log}(\hat{\mathbf{s}_{c}}^{\rm d}), \;\;\mathcal{L}_{cls}^{\rm s} = -\sum_{c=0}^{C} \mathbf{y}_{c} {\rm log}(\hat{\mathbf{s}_{c}}^{\rm s}),\] where \(\hat{\mathbf{s}_{c}}^{\rm d}\) and \(\hat{\mathbf{s}_{c}}^{\rm s}\) indicate scores after softmax normalization. Besides, as the dynamic and static branches tackle the same frame, two classification scores should be consistent. Thus, we introduce the consistency loss \(\mathcal{L}_{cons}\) to enable mutual guidance among two branches: \[\mathcal{L}_{cons} = \mathcal{L}_{KL}(\hat{\mathbf{s}}^{\rm d} \| \hat{\mathbf{s}}^{\rm s}) + \mathcal{L}_{KL}(\hat{\mathbf{s}}^{\rm s} \| \hat{\mathbf{s}}^{\rm d}),\] where \(\mathcal{L}_{KL}\) indicates the KL-divergence loss. As verified by Zhang [49], the consistency loss can lead to a robust model with better generalization. To sum up, the training process is guided by the following loss: \[\mathcal{L} = \mathcal{L}_{cls}^{\rm d} + \mathcal{L}_{cls}^{\rm s} + \lambda \mathcal{L}_{cons},\] where \(\lambda\) is a trade-off parameter.

In inference, the dynamic classification score \(\mathbf{s}^{\rm d}\) and the static classification score \(\mathbf{s}^{\rm s}\) are fused via a balance coefficient \(\beta\) to perform online action detection: \[\mathbf{s} = \beta\;\hat{\mathbf{s}}^{\rm s} + (1 - \beta)\;\hat{\mathbf{s}}^{\rm d}. \label{eqFusion}\tag{1}\]

4 Experiments↩︎

4.1 Setups↩︎

Dataset. We carry out experiments on three widely used benchmarks, THUMOS14 [55], TVSeries [7] and HDD [56]. THUMOS14 [55] includes sports videos from 20 action categories, where the validation set and test set contain 200 and 213 videos, respectively. On THUMOS14, challenges for online action detection include drastic intra-category varieties, motion blur, short action instances We follow previous works [1], [8], [17], [19], train the model on the validation set, and evaluate performance on the test set.

TVSeries [7] collects about 16 hours of videos from 6 popular TV series. The dataset contains 30 daily actions, where the total instance number is 6231. The TVSeries dataset exhibits some challenging characteristics, temporal overlapping action instances, a large proportion of background frames and unconstrained perspectives.

HDD [56] contains 104 hours of human driving video, belonging to 11 action categories. The videos were collected from 137 driving sessions using an instrumented vehicle equipped with different sensors. Following existing works [1], [8], [17], we use 100 sessions for training and 37 sessions for testing.

Metric. We adopt mean average precision (mAP) and calibrated mean average precision (cmAP) to measure the performance of online action detection algorithms. As for mAP, we first collect classification scores for all frames and then calculate precision and recall based on sorted results. Afterward, we calculate interpolated average precision to obtain AP scores for a category and finally regard the mean value of AP scores among all categories as mAP. Considering the drastically imbalanced frame numbers of different categories, Geest [7] proposed to calibrate the mAP score. In particular, we first calculate the ratio \(w\) between background frames and action frames and then calculate the calibrated precision as: \[cPre(i) = \frac{w \cdot TP(i)}{w \cdot TP(i) + FP(i)}.\] Afterward, the calibrated average precision \(cAP\) for a category can be calculated as: \[cAP = \frac{\sum_{i} cPre(i) \cdot \mathbf{1}(i)}{\sum_{i} \mathbf{1}(i)},\] where \(\mathbf{1}(\cdot)\) indicates whether the \(i^{th}\) frame belongs to the considered action category. Finally, cmAP can be obtained via calculating the mean value among all cAPs.

8pt

1.5pt

3.5pt

Implementation details. Following previous works [1], [8], [17], [19], we first conduct experiments with pre-extracted features. The feature extractor uses the two-stream network [57], whose spatial stream adopts ResNet-200 [58] and temporal stream adopts BN-Inception [59]. We report two experiments where the two-stream network [28], [57] is trained on the ActivityNet v1.3 dataset [60] or the Kinetics-400 [29] dataset to verify the generalization of the proposed Colar method. As for end-to-end online action detection, our backbone network is based on the ResNet50-I3D architecture [48], where the last average pooling layer and classification layer are removed. The ResNet50-I3D network is pretrained on Kinetics-400 [29] dataset, and we use the weight file provided by MMAction2[61]. In training, we freeze the first three blocks of the backbone network. Video frames are extracted with a frame rate of 25fps, where the spatial size is set as 224\(\times\)​224. We use the Adam [62] algorithm to optimize the whole network and set the batchsize as 16. The initial learning rate is \(3 \times 10^{-4}\) and decays every five epochs.

4.2 Comparison experiments↩︎

Quantitative comparisons. We make a comparison with current state-of-the-art methods [1], [8], [17], [19] and consistently build new high performance on THUMOS14 [55], TVSeries [7], and HDD [56] benchmarks. As shown in Table 1, based on TSN-ActivityNet features, our Colar brings an mAP gain of 1.1% over OadTR [1], and the improvements would be an mAP of 1.7% if the comparison is based on TSN-Kinetics features. The consistent improvements over current state-of-the-art methods verify the efficacy of our proposed exemplar-consultation mechanism. In addition, the proposed Colar can directly tackle video frames and perform online action detections, which achieves 58.6% mAP. In addition to THUMOS14, experiments on TVSeries [7] and HDD [56] benchmarks also verify the superiority of our method, as shown in Table 2 and Table 3.

Effectiveness and efficiency. Table 4 analyzes the performance and running time under different setups. When using pre-extracted features, Colar performs superior to existing methods [1], [19]. It is worth noting that extracting RGB and flow features takes 46.5 seconds, of which calculating optical flow costs the majority of the time. When only flow features or RGB features are available, the feature extraction cost is reduced, but both OadTR [34] and our Colar observe performance drop. In particular, it costs 44.2s to extract flow features, where OadTR [34] and Colar observe 7.4% and 7.3% performance drops, respectively. The cost to extract RGB features is small, but the online detection performance decreases a lot.

Given the ResNet50-I3D network [48], we first extract features from video frames and then train the proposed Colar method, which gets 53.4%. In contrast, the proposed end-to-end learning paradigm achieves 58.8%. To sum up, our proposed Colar method achieves a good balance between effectiveness and efficiency. Given pre-extracted features, Colar makes accurate detection results. Given only video frames, Colar costs 9.8 seconds to tackle a one-minute video and achieves comparable performance. In addition, we measure the memory cost under identical setups, where Colar requires 2235M memory and OadTR[1] requires 4375M memory.

2.5pt

Performance under different action portions. Table 5 elaborately studies the online action detection performance when different action portions are observed. The proposed Colar achieves promising accuracy when using the TSN-ActivityNet feature, the TSN-Kinetics feature, and only video frames. In particular, considering the most severe cases that only the first 10% portion of actions are observed, the previous state-of-the-art method OadTR [1] shows inferior performance to IDU [19]. However, the proposed Colar consistently exceeds OadTR, due to that the static branch effectively connects a frame with representative exemplars of each category and provides complimentary guidance.

4.3 Ablation experiments↩︎

Efficacy of each component. The proposed Colar method consists of the dynamic branch and the static branch, as well as a consistency loss \(\mathcal{L}_{cons}\) to enable mutual guidance between two branches. Table 6 studies the efficacy of each component on all three benchmark datasets. Firstly, the dynamic branch performs superior to the static branch, demonstrating the necessity of carefully modeling temporal dependencies. Besides, without the consistency loss, directly fusing prediction scores of two branches (using Eq. (1 )) only observes limited improvements, while \(\mathcal{L}_{cons}\) can further improve the detection performance.

Ablations about the dynamic branch. As shown in Figure 3 (a), we first study the influence of temporal scope \(T\) in modeling temporal dependencies and find 64 is a proper choice for the dynamic branch. The too-short temporal scope is insufficient to perceive evolvement within a video segment, while too long temporal scope would bring noises. In addition, we vary the number of convolutional layers and choose two layers, as shown in Figure 3 (b).

Ablations about the static branch. Based on K-Means clustering, the number of exemplars is an influential parameter for the static branch. As shown in Figure 3 (c), the ability of limited exemplars is insufficient and overwhelming exemplars would damage the performance as well.

Ablations about the complete method. Given the complete method, Figure 3 (d) studies the performance under different feature channels and verifies 1024 is a proper choice. Figure 3 (e) studies the coefficient \(\lambda\) for the consistency loss in the training phase, while Figure 3 (f) verifies the influence of coefficient \(\beta\) in the inference phase. We find \(\lambda=1\) and \(\beta=0.3\) are proper choices.

4.4 Qualitative analysis↩︎

Figure 4 qualitatively analyzes the proposed Colar method. Figure 4 (a) exhibits the static and dynamic scores within a video segment. Because the Volleyball Spiking instance shows dramatic viewpoint changes, the dynamic branch predicts low confident scores for some unique frames (shown in the yellow dotted box). In contrast, the static branch consults representative exemplars from the Volleyball Spiking category and consistently predicts high scores for these unique frames. Figure 4 (b) presents a video segment containing multiple action instances, exhibits the similarity between the current frame (the last one) and its historical frames. The similarity weights clearly highlight historical action frames and suppress background frames, which contributes to aggregating temporal features.

5 Conclusion↩︎

This paper proposes Colar, based on the exemplar-consultation mechanism, to conduct category-level modeling for each frame and capture long-term dependencies within a video segment. Colar compares a frame with exemplar frames, aggregates exemplar features, and carries out online action detection. In the dynamic branch, Colar regards historical frames as exemplars and models long-term dependency with a lightweight network structure. In the static branch, Colar employs representative exemplars of each category and captures the category particularity. The prominent efficacy of Colar would inspire future works to pay attention to category-level modeling. In addition, as Colar has made a good trade-off between effectiveness and efficiency, it is a promising direction to conduct online action detection directly from streaming video data, which can benefit practical usage.

Limitations. Because Colar is only verified on the benchmark datasets, it may observe performance drop in practical scene due to new challenges, long-tail distribution, open-set action categories. Besides, the unintended usage of Colar for surveillance may violate individual privacy.

Acknowledgments. This work was supported in part by the Key-Area Research and Development Program of Guangdong Province (No.2019B010110001) and the National Natural Science Foundation of China under Grant U21B2048, 62036011, and the Open Research Projects of Zhejiang Lab (No.2019KD0AD01/010).

References↩︎