Resource-Efficient Multiview Perception: Integrating Semantic Masking with Masked Autoencoders

Abstract

Multiview systems have become a key technology in modern computer vision, offering advanced capabilities in scene understanding and analysis. However, these systems face critical challenges in bandwidth limitations and computational constraints, particularly for resource-limited camera nodes. This paper presents a novel approach for communication-efficient distributed multiview detection and tracking using masked autoencoders (MAEs). We introduce a semantic-guided masking strategy that leverages pre-trained segmentation models and a tunable power function to prioritize informative image regions. This approach, combined with an MAE, reduces communication overhead while preserving essential visual information. We evaluate our method on both virtual and real-world multiview datasets, demonstrating comparable performance in terms of detection and tracking performance metrics compared to state-of-the-art techniques, even at high masking ratios. Our selective masking algorithm outperforms random masking, maintaining higher accuracy and precision as the masking ratio increases. Furthermore, our approach achieves a significant reduction in transmission data volume compared to baseline methods, thereby balancing multiview tracking performance with communication efficiency.

Multiview detection and tracking for sports analytics.

Supplementary Materials

Figure 1: Kappa abalation study figure.

This figure presents an ablation study conducted to determine the optimal value for our power function parameter κ in the semantic-guided masking algorithm. We evaluated five different κ values (0.00, 0.05, 0.10, 0.15, and 0.20) against the Multiple Object Detection Accuracy (MODA) metric on the Wildtrack dataset across various masking ratios. The results demonstrate that κ=0.15 provides the best balance between random and deterministic masking behaviors, maintaining high detection accuracy even as the masking ratio increases. At κ=0.00, which approximates random masking, we observe a rapid decline in performance as the masking ratio increases. In contrast, κ=0.15 maintains robust performance until approximately 0.80 masking ratio, after which performance decreases more rapidly. This validates our choice of κ=0.15 for the main experiments, as it effectively prioritizes informative image regions while preserving sufficient global context for accurate multiview detection. Communication cost.

Figure 2: Communication cost vs. masking ratio.

This figure illustrates the significant communication efficiency gained through our proposed MAE-based approach compared to traditional multiview methods. The graph plots communication volume (in megabits) against masking ratio, clearly demonstrating the inverse relationship between masking ratio and data transmission requirements. The baseline approach (original method) shown by the dashed red line maintains a constant high communication cost of approximately 154.83 megabits regardless of masking configuration. In contrast, our proposed method (blue line) shows a linear decrease in communication volume as the masking ratio increases, approaching near-zero transmission costs at masking ratios close to 1.0. At our recommended operating point (70% masking ratio), the communication volume is reduced to approximately 11.7 megabits, representing a 13.33-fold reduction compared to conventional approaches. This dramatic decrease in communication requirements demonstrates the practical applicability of our method in bandwidth-constrained environments while maintaining competitive detection and tracking performance.

BibTeX

@inproceedings{dakic2025MAE,
      title = {Resource-Efficient Multiview Perception: Integrating Semantic Masking with Masked Autoencoders}, 
      author = {Kosta Dakic and Kanchana Thilakarathna and Rodrigo N. Calheiros and Teng Joon Lim},
      year = {2025},
      booktitle ={2025 IEEE International Conference on Pervasive Computing and Communications (PerCom)}
}