Skip to main content

A survey and taxonomy on intelligent surveillance from a system perspective

  • Radu-Casian Mihailescu (a1), Paul Davidsson (a1), Ulrik Eklund (a1) and Jan A. Persson (a1)

Recent proliferation of surveillance systems is mostly attributed to advances in both image-processing techniques and hardware enhancement of smart cameras, as well as the ubiquity of sensor-driven architectures. Owing to these capabilities, new aspects are coming to the forefront. This paper addresses the current state-of-the-art and provides researchers with an overview of existing surveillance solutions, analyzing their properties as a system and drawing attention to relevant challenges when developing, deploying and managing them. Also, some of the more prominent application domains are highlighted here. In an effort to understand the development of the advanced solutions, based on their most distinctive characteristics, we propose a taxonomy for surveillance systems to help classify them and reveal gaps in existing research. We conclude by identifying promising future research lines.

Hide All
Aguilar-Ponce, R., Kumar, A., Tecpanecatl-Xihuitl, J. L. & Bayoumi, M. 2007. A network of sensor-based framework for automated visual surveillance. Journal of Network and Computer Applications 30(3), 12441271.
Ahmadi, A., Mitchell, E., Richter, C., Destelle, F., Gowing, M., O’Connor, N. & Moran, K. 2015. Toward automatic activity classification and movement assessment during a sports training session. Internet of Things Journal, IEEE 2(1), 2332.
Bellifemine, F., Caire, G., Poggi, A. & Rimassa, G. 2008. JADE: a software framework for developing multi-agent applications. Lessons learned. Information and Software Technology 50(12), 1021.
Bicocchi, N., Fontana, D. & Zambonelli, F. 2014. A self-aware, reconfigurable architecture for context awareness. In IEEE Symposium on Computers and Communication (ISCC), 1–7.
Biswas, P. K., Qi, H. & Xu, Y. 2008. Mobile-agent-based collaborative sensor fusion. Information Fusion 9(3), 399411.
Braubach, L., Lamersdorf, W. & Pokahr, A. 2003. Jadex: implementing a BDI-infrastructure for JADE agents. Search of Innovation (Special Issue on JADE) 3, 7685.
Brown, B., Wei, W., Ozburn, R., Kumar, M. & Cohen, K. 2015. Surveillance for intelligent emergency response robotic aircraft (SIERRA)- VTOL aircraft for emergency response. In AIAA Infotech. American Institute of Aeronautics and Astronautics.
Brutzer, S., Hoferlin, B. & Heidemann, G. 2011. Evaluation of background subtraction techniques for video surveillance. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1937–1944.
Bustamante, A. L., Molina, J. M. & Patricio, M. A. 2014. A practical approach for active camera coordination based on a fusion-driven multi-agent system. International Journal of Systems Science 45(4), 741755.
Byun, J. S. & Park, S. 2011. Development of a self-adapting intelligent system for building energy saving and context-aware smart services. IEEE Transactions on Consumer Electronics 57(1), 9098.
Camplani, M. & Salgado, L. 2011. Scalable software architecture for on-line multi-camera video processing. In Proceedings of SPIE, 7871, 6–21.
Carchiolo, V., Longheu, A., Malgeri, M., Mangioni, G. & Torrisi, D. 2010. Pervasive home security: an intelligent domotics application. In Intelligent Distributed Computing IV, Studies in Computational Intelligence 315, M. Essaaidi, M. Malgeri & C. Badica (eds). Springer Berlin Heidelberg, 155164.
Castanedo, F., Garcia, J., Patricio, M. & Molina, J. 2008. A multi-agent architecture to support active fusion in a visual sensor network. In Second ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC), 1–8.
Chen, S.-L., Chang, S.-K. & Chen, Y.-Y. 2015. Development of a multisensor embedded intelligent home environment monitoring system based on digital signal processor and Wi-Fi. International Journal of Distributed Sensor Networks 2015, 11.
Cho, Y., Lim, S. O. & Yang, H. S. 2010. Collaborative occupancy reasoning in visual sensor network for scalable smart video surveillance. IEEE Transactions on Consumer Electronics 56(3), 19972003.
Christensen, E., Curbera, F., Meredith, G. & Weerawarana, S. 2001. Web Service Definition Language (WSDL). Technical report,
Collins, R., Lipton, A., Fujiyoshi, H. & Kanade, T. 2001. Algorithms for cooperative multisensor surveillance. Proceedings of the IEEE 89(10), 14561477.
Compton, M., Barnaghi, P., Bermudez, L., Garca-Castro, R., Corcho, O., Cox, S., Graybeal, J., Hauswirth, M., Henson, C., Herzog, A., Huang, V., Janowicz, K., Kelsey, W. D., Phuoc, D. L., Lefort, L., Leggieri, M., Neuhaus, H., Nikolov, A., Page, K., Passant, A., Sheth, A. & Taylor, K. 2012. The SSN ontology of the W3C semantic sensor network incubator group. Web Semantics: Science, Services and Agents on the World Wide Web 17, 2532.
Davidsson, P. & Boman, M. 2000. Saving energy and providing value added services in intelligent buildings: a MAS approach. In Agent Systems, Mobile Agents, and Applications, Lecture Notes in Computer Science 1882, D. Kotz & F. Mattern (eds). Springer Berlin Heidelberg, 166177.
Davidsson, P. & Boman, M. 2005. Distributed monitoring and control of office buildings by embedded agents. Information Sciences 171(4), 293307.
Fernández-De-Alba, J. M., Fuentes-Fernández, R. & Pavón, J. 2015. Architecture for management and fusion of context information. Information Fusion 21, 100113.
Findeisen, M., Meinel, L., Richter, J. & Hirtz, G. 2015. An omnidirectional stereo sensor for human behavior analysis in complex indoor environments. In 2015 IEEE International Conference on Consumer Electronics (ICCE), 17–19.
Gascuena, J. M., Castillo, J. C., Navarro, E. & Fernandez-Caballero, A. 2014. Engineering the development of systems for multisensory monitoring and activity interpretation. International Journal of Systems Science 45(4), 728740.
Hachem, S., Teixeira, T. & Issarny, V. 2011. Ontologies for the internet of things. In Proceedings of the 8th Middleware Doctoral Symposium MDS ‘11, 3:1–3:6. ACM.
Haesevoets, R., Van Eylen, B., Weyns, D., Helleboogh, A., Holvoet, T. & Joosen, W. 2007. Context-driven dynamic organizations applied to coordinated monitoring of traffic jams. In Proceedings of Workshop Engineering Environment-Mediated Multiagent Systems, 126143.
Hoffmann, M., Wittke, M., Hahner, J. & Muller-Schloer, C. 2008. Spatial partitioning in self-organizing smart camera systems. IEEE Journal of Selected Topics in Signal Processing 2(4), 480492.
Hu, W., Tan, T., Wang, L. & Maybank, S. 2004. A survey on visual surveillance of object motion and behaviors. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 34(3), 334352.
Hull, R., Kumar, B., Lieuwen, D., Patel-Schneider, P. F., Sahuguet, A., Varadarajan, S. & Vyas, A. 2005. Improving user experience through rule-based service customization. International Journal of Cooperative Information Systems 14(4), 469502.
Jovanovic, M. & Rinner, B. 2007. Middleware for dynamic reconfiguration in distributed camera systems. In Fifth Workshop on Intelligent Solutions in Embedded Systems, 139–150.
Khajenasiri, I., Patti, E., Jahn, M., Acquaviva, A., Verhelst, M., Macii, E. & Gielen, G. 2014. Design and implementation of a multi-standard event-driven energy management system for smart buildings. In IEEE 3rd Global Conference on Consumer Electronics (GCCE), 20–21.
Ko, T. 2008. A survey on behavior analysis in video surveillance for homeland security applications. In 37th IEEE Applied Imagery Pattern Recognition Workshop, 1–8.
Krahnstoever, N., Yu, T., nam Lim, S., Patwardhan, K. & Tu, P. 2008. Collaborative real-time control of active cameras in large scale surveillance systems. In Proceedings Workshop on Multicamera and Multi-modal Sensor Fusion Algorithms and Applications, 26, 1–12.
Lassila, O. & Swick, R. R. 1999. Resource Description Framework (RDF) Model and Syntax Specification. Technical report, W3C. ttp://
Li, X., Lu, R., Liang, X., Shen, X., Chen, J. & Lin, X. 2011. Smart community: an internet of things application. IEEE Communications Magazine 49(11), 6875.
Longheu, A., Carchiolo, V., Malgeri, M., Mangioni, G., Longheu, R., Carchiolo, V. & Malgeri, M., I, V. A. D. 2012. An intelligent and pervasive surveillance system for home security. International Journal Computer Communications 7(2), 312324.
Meinel, L., Findeisen, M., Hes, M., Apitzsch, A. & Hirtz, G. 2014. Automated real-time surveillance for ambient assisted living using an omnidirectional camera. In IEEE International Conference on Consumer Electronics (ICCE), 396–399.
Meisels, A. 2007. Distributed Search by Constrained Agents: Algorithms, Performance, Communication. Advanced Information and Knowledge Processing. Springer.
Mihailescu, R., Persson, J. A., Davidsson, P. & Eklund, U. 2016. Towards collaborative sensing using dynamic intelligent virtual sensors. In Intelligent Distributed Computing X – Proceedings of the 10th International Symposium on Intelligent Distributed Computing – IDC 2016, 217–226. 10–12 October.
Monari, E., Voth, S. & Kroschel, K. 2008. An object- and task-oriented architecture for automated video surveillance in distributed sensor networks. In IEEE Fifth International Conference on Advanced Video and Signal Based Surveillance (AVSS ‘08), 339–346.
Nilsson, F. 2008. Intelligent Network Video: Understanding Modern Video Surveillance Systems. CRC Press.
Nguyen, N. T., Venkatesh, S., West, G. & Bui, H. H. 2003. Multiple camera coordination in a surveillance system. ACTA Automatica Sinica 29, 408422.
Nguyen, T. A., Raspitzu, A. & Aiello, M. 2014. Ontology-based office activity recognition with applications for energy savings. Journal of Ambient Intelligence and Humanized Computing 5(5), 667681.
Oliver, N., Horvitz, E. & Garg, A. 2002. Layered representations for human activity recognition. In Proceedings Fourth IEEE International Conference on Multimodal Interfaces, 3–8.
Oliver, N., Rosario, B. & Pentland, A. 2000. A Bayesian computer vision system for modeling human interactions. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8), 831843.
Park, H. D., Min, O.-G. & Lee, Y.-J. 2017. Scalable architecture for an automated surveillance system using edge computing. Journal of Supercomputing 73(3), 926939.
Patti, E., Acquaviva, A., Jahn, M., Pramudianto, F., Tomasi, R., Rabourdin, D., Virgone, J. & Macii, E. 2014. Event-driven user-centric middleware for energy-efficient buildings and public spaces. IEEE Systems Journal 10(3), 11371146.
Pavon, J., Gomez-Sanz, J., Fernndez-Caballero, A. & Valencia-Jimenez, J. J. 2007. Development of intelligent multisensor surveillance systems with agents. Robotics and Autonomous Systems 55(12), 892903.
Piette, F., Dinont, C., Seghrouchni, A. E. F. & Taillibert, P. 2015. Deployment and configuration of applications for ambient systems. In Proceedings of the 6th International Conference on Ambient Systems, Networks and Technologies, ANT-2015, Procedia Computer Science, 27, 373–380.
Power, P. W. & Schoonees, J. A. 2002. Understanding background mixture models for foreground segmentation. In Proceedings Image and Vision Computing New Zealand, 267–271.
Prud’hommeaux, E. & Seaborne, A. 2006. SPARQL Query Language for RDF, Technical report.
Qian, H., Wu, X. & Xu, Y. 2011. Intelligent Surveillance Systems, 51. Springer Science & Business Media.
Quaritsch, M., Kreuzthaler, M., Rinner, B., Bischof, H. & Strobl, B. 2007. Autonomous multicamera tracking on embedded smart cameras. EURASIP Journal on Embedded Systems 2007(1), 3545.
Rao, A. S. & Georgeff, M. P. 1995. BDI agents: from theory to practice. in Proceedings of the First International Conference on Multi-Agent Systems (ICMAS), 312–319. MIT Press.
Rothkrantz, L. 2013. Crisis management using multiple camera surveillance systems. In Proceedings of the 10th International Conference on Information Systems for Crisis Response and Management, 617–626.
Ryoo, M. S. & Aggarwal, J. K. 2008. Recognition of high-level group activities based on activities of individual members. In Proceedings of the 2008 IEEE Workshop on Motion and Video Computing, 1–8.
Silva, R., Arakaki, J., Junqueira, F., Filho, D. S. & Miyagi, P. 2012. Modeling of active holonic control systems for intelligent buildings. Automation in Construction 25, 2033.
Snidaro, L., García, J. & Llinas, J. 2015. Context-based information fusion. Information Fusion 25(C), 1631.
Sobhani, F., Kahar, N. F. & Zhang, Q. 2015. An ontology framework for automated visual surveillance system. In Proceeding of the 13th International Workshop on Content-Based Multimedia Indexing (CBMI), 1–7.
Soldatos, J., Pandis, I., Stamatis, K., Polymenakos, L. & Crowley, J. L. 2007. Agent based middleware infrastructure for autonomous context-aware ubiquitous computing services. Computer Communications 30(3), 577591.
Stavropoulos, T. G., Vrakas, D., Vlachava, D. & Bassiliades, N. 2012. BOnSAI: a smart building ontology for ambient intelligence. In Proceedings of the Second International Conference on Web Intelligence, Mining and Semantics, WIMS ‘12, 30:1–30:12. ACM.
Town, C. 2007. Multi-sensory and multi-modal fusion for sentient computing. International Journal of Computer Vision 71(2), 235253.
Valera, M. & Velastin, S. 2005. Intelligent distributed surveillance systems: a review. IEEE Vision, Image and Signal Processing 152(2), 192204.
Veeraraghavan, A., Chellappa, R. & Roy-Chowdhury, A. 2006. The function space of an activity. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1, 959–968.
Velipasalar, S., Schlessman, J., Chen, C.-Y., Wolf, W. & Singh, J. 2006. SCCS: a scalable clustered camera system for multiple object tracking communicating via message passing interface. In IEEE International Conference on Multimedia and Expo, 28, 277–280.
Vicaire, P., He, T., Cao, Q., Yan, T., Zhou, G., Gu, L., Luo, L., Stoleru, R., Stankovic, J. A. & Abdelzaher, T. F. 2009. Achieving long-term surveillance in vigilnet. ACM Transactions on Sensor Networks 5(1), 9:19:39.
Vogler, C. & Metaxas, D. 1999. Parallel hidden Markov models for American sign language recognition. In The Proceedings of the Seventh IEEE International Conference on Computer Vision, 1999, 1, 116–122.
W3C OWL Working Group 2009. OWL 2 Web Ontology Language Document Overview, Technical report, W3C.
Weinland, D., Ronfard, R. & Boyer, E. 2011. A survey of vision-based methods for action representation, segmentation and recognition. Computer Vision and Image Understanding 115(2), 224241.
Winkler, T. & Rinner, B. 2014. Security and privacy protection in visual sensor networks: a survey. ACM Computing Surveys 47(1), 2:12:42.
Woolridge, M. 2009. Introduction to Multiagent Systems, 2nd edition. John Wiley & Sons Inc.
Yan, L., Chao, L., Ke Wei, P. & Tao, S. 2015. Crowd counting using wireless infrared distance sensors for indoor environments. In Informatics and Communication Technologies for Societal Development, E. B. Rajsingh, A. Bhojan & J. D. Peter (eds). Springer, 18.
Yi, P., Iwayemi, A. & Zhou, C. 2011. Developing ZigBee deployment guideline under WiFi interference for smart grid applications. IEEE Transactions on Smart Grid 2(1), 110120.
Yilmaz, A., Javed, O. & Shah, M. 2006. Object tracking: a survey. ACM Computing Surveys 38(4), 145.
Yong, C., Qiao, B., Wilson, D., Wu, M., Clements-Croome, D., Liu, K., Egan, R. & Guy, C. 2007. Coordinated management of intelligent pervasive spaces. In Proceedings of the 5th IEEE.
Yu, T., Sekar, V., Seshan, S., Agarwal, Y. & Xu, C. 2015. Handling a trillion (unfixable) aws on a billion devices: rethinking network security for the internet-of-things. In Proceedings of the 14th ACM Workshop on Hot Topics in Networks, HotNets-XIV, 5:1–5:7. ACM.
Zhang, D., Gatica-Perez, D., Bengio, S. & McCowan, I. 2006. Modeling individual and group actions in meetings with layered HMMs. IEEE Transactions on Multimedia 8(3), 509520.
Zhu, F., Mutka, M. W. & Ni, L. M. 2005. Service discovery in pervasive computing environments. IEEE Pervasive Computing 4(4), 8190.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

The Knowledge Engineering Review
  • ISSN: 0269-8889
  • EISSN: 1469-8005
  • URL: /core/journals/knowledge-engineering-review
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed