Literature
Display
← All papers

What Can Help Pedestrian Detection?

Jiayuan Mao · Tete Xiao · Yuning Jiang · Zhimin Cao

· CVPR 2017 ↗

Computer Vision6 min text · CondensedOriginal paper ↗
Contents

At a glance

What information helps a convolutional pedestrian detector beyond the image itself? Controlled experiments compare semantic segmentation, edges, optical flow, disparity, and hand-crafted channels. Semantic context is most useful for small pedestrians; detailed boundaries help localization when images are enlarged.

HyperLearner then uses auxiliary feature maps as training supervision rather than extra inference inputs. A shared representation learns feature prediction and pedestrian detection jointly. Results improve over the tested baselines on KITTI, Cityscapes, and Caltech, with gains depending on image scale, task labels, and evaluation settings. The claims concern these 2017 systems and benchmarks, not current detector rankings.

1 Introduction

Small pedestrians can resemble signs, posts, and other background structures, making context valuable. Crowded scenes create a second problem: convolution and pooling blur boundaries between nearby people, so a detector may classify the scene correctly while placing boxes badly. Figure 1 illustrates both cases.

The study first supplies different channel features directly to a detector to isolate their value. It then trains a shared network to predict useful channels alongside detection, avoiding a separate feature-producing network or extra sensory input at deployment. Figure 2 distinguishes appearance, semantic, temporal, and depth cues.

Refer to caption
(a)
Refer to caption
(b)
Figure 1: (a) Examples of true pedestrians and hard negative samples of low resolution. Without extra semantic contexts, it is difficult to discriminate between them, even for human eyes. (b) Example of pedestrians in crowded scenes, where CNN-based detectors fail to locate each individual without low-level apparent features.
Refer to caption
Figure 2: A demonstration of various channel features. Includes: apparent-to-semantic features, temporal features, depth features.

2 Related work

Channel features had already helped boosted-forest pedestrian detectors, including gradient, optical-flow, and semantic representations. CNN detectors instead learn their own features and often use region proposals. This paper tests whether the earlier extra-channel advantage persists in a CNN pipeline, then connects feature aggregation with auxiliary-task supervision. [10] [23] [24] [33]

3 Channel features for pedestrian detection

The first experiments feed additional channels into a detector to compare the information they contribute.

3.1 Preliminaries

KITTI supplies pedestrian labels, adjacent video frames, and stereo images, enabling all three groups of cues to be tested. The authors use an established split of its 7,481 labeled images for training and validation. Easy, moderate, and hard settings differ in object scale and visibility; moderate is the principal metric. [26]

The baseline is Faster R-CNN with an ImageNet-pretrained VGG-16 backbone. Removing the conv5 layers preserves spatial resolution, and five anchor scales with seven aspect ratios target small objects. These modifications distinguish the baseline from the separately listed reproduction of standard Faster R-CNN. [24]

3.2 Introduction to channel features

Appearance-to-semantic cues include hand-crafted integral channels, learned HED edges, semantic segmentation, and a blurred segmentation heatmap. Their comparison separates fine boundaries from category context: blurring keeps semantics while removing localization detail. Segmentation is learned from MS-COCO; same-class neighboring people need not have distinct boundaries in that representation. [18] [31]

Optical flow represents temporal information from adjacent frames. DispNet estimates disparity from stereo images as the depth channel. These are auxiliary inputs in the initial analysis, unlike the later HyperLearner's training-only supervision. [21]

Refer to caption
Figure 3: As described in Section 3.3, our Faster R-CNN for channel feature integration. The side branch takes channel features as input and generates channel feature representations before concatenated with conv4_3.

3.3 Integration techniques

A shallow side branch converts each extra channel into a 128-channel map at one-eighth input resolution, concatenated with conv4_3 before proposal generation and detection. Two randomly initialized convolution layers work better than one layer or an independently pretrained side branch in the tested configurations. The authors hypothesize that pretraining makes the detector rely too heavily on the side branch; this is an interpretation of observed activations, not a general theorem. Table 1 retains the exact comparison.

Model Pedestrian
#Convs Init. W. Mod Easy Hard
O N/A N/A 68.96 73.33 60.43
A 2 random 70.80 78.15 62.16
B 1 random 70.40 75.17 61.92
C 2 pretrained 69.92 77.33 61.65
Table 1: Detection improvement by integrating channel features on KITTI validation set. Model “O” is our baseline detector. “#Convs” means the number of convolution layers in the side branch. “Init. W.” denotes initial weights for the side branch. The input images are not enlarged.
Model Recall
(0,80]\left(0,80\right] (80,160]\left(80,160\right] (160,inf]\left(160,\inf\right] all scales
Baseline 21.3% 87.6% 96.8% 70.0%
+Segmentation 35.6% 88.2% 96.8% 74.0%
Table 2: Recall comparison at 70% precision between baseline and segmentation channel at different pedestrian heights. The results are based on 1x scale.

3.4 Comparison and analysis

At original image scale, segmentation and blurred heatmaps give the largest gains; extra raw-image input gives little consistent improvement. At 70% precision, segmentation raises overall recall from 70.0% to 74.0%, with the largest increase for pedestrians up to 80 pixels tall: 21.3% to 35.6%. This supports a role for semantic context in small-object discrimination.

At double scale, edge and segmentation cues remain useful while the boundary-free heatmap contributes less. An error analysis on about 800 validation images and the top 200 false positives links edges to improved localization. The reported 9-point localization-error reduction is relative to the baseline's error composition, not an overall detection-accuracy gain. Optical flow mainly helps at original scale; disparity behaves more like semantic context than a uniformly useful depth measurement. All values and difficulty-specific differences remain in Tables 2–3 and Figure 4.

Model Pedestrian 1x Input Improvement Pedestrian 2x Input Improvement
Mod Easy Hard Mod Easy Hard Avg Mod Easy Hard Mod Easy Hard Avg
Fr-RCNN* [24] 59.29 64.53 53.01 - - - - 71.05 76.00 62.08 - - - -
MS-CNN [2] 68.37 73.70 60.72 - - - - 72.26 76.38 64.08 - - - -
Our Baseline 68.96 73.33 60.43 - - - - 71.21 77.73 62.19 - - - -
+ Original img 68.63 76.61 60.45 -0.33 +3.28 +0.02 +0.99 71.33 76.72 62.17 +0.12 -1.01 -0.02 -0.30
+ ICF 68.40 73.56 60.20 -0.56 +0.23 -0.23 -0.19 71.80 77.40 62.79 +0.59 -0.33 +0.60 +0.29
+ Edge 69.49 76.28 60.89 +0.53 +2.95 +0.46 +1.31 72.34 78.32 63.28 +1.13 +0.59 +1.09 +0.94
+ Segmentation 70.80 78.15 62.16 +1.84 +4.82 +1.73 +2.80 72.54 78.49 63.61 +1.33 +0.76 +1.42 +1.17
+ Heatmap 70.33 78.03 61.75 +1.37 +4.70 +1.32 +2.46 71.39 77.64 62.34 +0.18 -0.09 +0.15 +0.08
+ Disparity 70.03 77.74 61.48 +1.07 +4.41 +1.05 +2.18 71.72 77.52 62.47 +0.51 -0.21 +0.28 +0.19
+ Optical Flow 69.39 77.07 60.79 +0.43 +3.74 +0.36 +1.51 71.13 76.85 62.24 -0.08 -0.88 +0.05 -0.25
Table 3: Channel features comparison on KITTI validation set. We list improvement across all three KTTTI metrics as well as the average. *: Our reproduced Faster R-CNN with same parametrs as in [24]. The baseline is a re-implementation of Faster RCNN pipeline, consisting of slight differences with the basic Faster RCNN (See Section 3.1).
Original figure
(a) False positive sources at 70%70\% recall rate
Original figure
(b) Top-200 false positives sources
Figure 4: False positive analysis for baseline, edge channel and heatmap channel at 2x scale. All false positives are categorized into four types: localization error, background classification error, cyclist classification error, and annotation error. Localization error is defined as non-matched detection bounding boxes which overlap with a groundtruth but iou <0.5<0.5, while background error has no overlap with any groundtruth box. Cyclist error happens when a bounding box match cyclist groundtruth. Annotation error occurs when detection “matches” a de facto groundtruth which, however, is not annotated.
Refer to caption
Figure 5: The proposed HyperLearner, which consists of 4 components: body network, channel feature network (CFN), region proposal network (RPN) and Fast R-CNN (FRCNN). HyperLearner learns representations of channel features while requiring no extra input in inference. Refer to Section 4.1 for details.

4 Jointly learn the channel features

Computing auxiliary maps at inference can be expensive. HyperLearner instead learns to predict the useful maps as an auxiliary task, allowing their supervision to shape the features used by the detector.

4.1 HyperLearner

HyperLearner has a shared body, a channel-feature network, a region-proposal network, and a Fast R-CNN head. Features from VGG layers conv1_2, conv2_2, conv3_3, and conv4_3 each pass through two convolutions to produce 32 channels. Upsampling and concatenation create a common-resolution aggregated map.

The channel-feature network predicts the auxiliary map from that representation. Proposal generation and detection use both the aggregated map and conv4_3, so auxiliary supervision changes their shared features. At inference, only the original image is required; the external network that produced training targets is unnecessary. Figure 5 preserves the full architecture.

4.2 Training Details

Training combines pixel-level feature prediction with the usual proposal and detection classification/regression losses. Binary edge maps use class-balanced cross-entropy; multiclass segmentation uses cross-entropy, and other auxiliary tasks use mean-squared error. The target map is S, the predicted map C, and the pixel loss is averaged over its H by W locations. Positive pixels have targets above 0.5: their weight is one minus the positive fraction, while negative pixels receive the positive fraction. All four detector-loss weights are one.

Training proceeds in four stages: train the channel-feature network with pretrained body layers fixed; train the proposal network with the body and auxiliary network fixed; train the detection head while fixing earlier components; then optimize all layers jointly. Proposal coordinates are treated as fixed when differentiating the detection-head loss.

Equations1H×W​∑(x,y)ℓ⁡(Sx,y,Cx,y),\displaystyle\frac{1}{H\times W}\sum_{(x,y)}\ell(S_{x,y},C_{x,y}),ℓ⁡(p,q)=βx,y​(−p​log⁡q−(1−p)​log⁡(1−q)),\ell(p,q)=\beta_{x,y}{\big(}-p\log q-(1-p)\log(1-q){\big)},ℒ=ℒCFN+λ1​ℒRPNcls+λ2​ℒRPNbbox+λ3​ℒFRCNNcls+λ4​ℒFRCNNbbox\mathcal{L}=\mathcal{L}_{\mathrm{CFN}}+\lambda_{1}\mathcal{L}_{\mathrm{RPNcls}}+\lambda_{2}\mathcal{L}_{\mathrm{RPNbbox}}+\lambda_{3}\mathcal{L}_{\mathrm{FRCNNcls}}+\lambda_{4}\mathcal{L}_{\mathrm{FRCNNbbox}}

5 Experiments and results

KITTI, Cityscapes, and Caltech test the approach under different image scales and supervision. A HyperNet control keeps the multilevel aggregation but removes auxiliary feature supervision, separating the effect of architecture from the effect of the extra training task.

5.1 KITTI Dataset

At original scale, KITTI moderate performance rises from 69.80 for the baseline to 71.15 with segmentation supervision and 71.25 with edge supervision; aggregation alone reaches 69.72. At double scale, the corresponding baseline and edge-supervised results are 71.73 and 72.51, with HyperNet at 72.23. Gains are therefore larger at original resolution, and architecture alone explains some high-resolution improvement.

Results differ across easy, moderate, and hard settings: Table 4 does not show that either auxiliary task uniformly wins every metric. These are validation-set comparisons with the stated modified detector, not results on KITTI's held-out official test set.

Model 1x input 2x input
Mod Easy Hard Mod Easy Hard
Fr-RCNN* [24] 59.29 64.53 53.01 71.05 76.00 62.08
MS-CNN [2] 68.37 73.70 60.72 72.26 76.38 64.08
Baseline 69.80 74.37 61.20 71.73 77.84 62.30
HyperNet 69.72 76.91 61.10 72.23 77.96 63.43
+Segmentation 71.15 79.43 62.34 72.35 79.17 62.34
+Edge 71.25 78.43 62.15 72.51 78.51 63.24
Table 4: Results on KITTI validation set, the model HyperNet refers to the HyperLearner without CFN. Evaluation follows moderate metric in KITTI.
*: Fr-RCNN follows setting as [24] while baseline model is Faster-RCNN with slightly different parameters. See also Table 3.

5.2 Cityscapes dataset

Cityscapes supplies human segmentation labels directly, so the auxiliary task uses person pixels rather than predictions from a separately trained segmentation model. Experiments use the finely annotated training and validation images.

At 540p, average precision increases from 74.97 to 77.22 while measured per-image latency rises from 130 to 140 ms on one Titan X GPU. At 720p, precision rises from 86.89 to 87.67 and latency from 240 to 250 ms. The detection evaluation derives boxes from person polygons and excludes small, heavily occluded, cyclist, and person-group cases as detailed in Table 5. These restrictions bound the result's scope.

5.3 Caltech dataset

Caltech evaluation uses the refined annotations for both training and testing and evaluates 4,024 test frames. Figure 7 compares miss-rate curves on the reasonable subset using the updated labels. The authors report the strongest results among the methods in that historical comparison; the curve and protocol should accompany the ranking claim. [11] [34]

6 Summary

The experiments support two complementary roles for auxiliary information: semantic context helps disambiguate small pedestrians, while boundaries help localize people in higher-resolution images. HyperLearner transfers those cues through a shared representation during training, avoiding extra inference inputs.

The evidence comes from specific backbones, tasks, scales, labels, and benchmark subsets. Auxiliary supervision and multilevel aggregation both matter, and the tables show nonuniform gains. Generalization to other detector families or unrestricted crowded-scene deployment is not established by these experiments.

Model 540p input 720p input Improvement
Speed AP Speed AP 540p 720p
Baseline 130ms 74.97 240ms 86.89 - -
HyperNet 140ms 74.30 250ms 86.67 -0.53 -0.22
Jointsegmap 140ms 77.22 250ms 87.67 +2.25 +0.78
Table 5: Results on Cityspcaes validation set. The speed column shows the time each model needed to perform detection on a single image. The speed is tested on single NVIDIA TITAN-X GPU. We use all segmentation polygons labeled “person” to generate bounding boxes for the pedestrian detection task. Following the standard in Caltech dataset [11], all persons with (pixel-level) occlusion greater than 0.5 or of height less than 5050 pixels are ignored. Furthermore, all polygons labeled “cyclist” or “person group” are also ignored.
Complete original figure with every panel, label and legend.
Figure 6: Results of HyperLearner on Cityscapes validation set. The left column shows our detection result, while the right column demonstrate CFN’s output learned from segmentation labeling.
Original figure
Figure 7: Detection quality on Caltech test set (reasonable, 𝑀𝑅−2N​(𝑀𝑅−4N)\mathit{MR}_{-2}^{N}(\mathit{MR}_{-4}^{N})), evaluated on the new annotations [34]. We achieve state-of-the-art results on both evaluation metrics.
Concept demonstration

Open image ↗