On Assisted Living of Paralyzed Persons through Real-Time Eye Features Tracking and Classification using Support Vector Machines
DOI:
https://doi.org/10.26415/2572-004X-vol3iss1p316-333Keywords:
Assisted living; Rehabilitation; Paralyzed persons; Eye-blink detection; Eyeball detection; Biomedical engineering; SVM; Machine learning; Image processingAbstract
Background: The eye features like eye-blink and eyeball movements can be used as a module in assisted living systems that allow a class of physically challenged people speaks – using their eyes. The objective of this work is to design a real-time customized keyboard to be used by a physically challenged person to speak to the outside world, for example, to enable a computer to read a story or a document, do gaming and exercise of nerves, etc., through eye features tracking
Method: In a paralyzed person environment, the right-left, up-down eyeball movements act like a scroll and eye blink as a nod. The eye features are tracked using Support Vector Machines (SVMs).
Results: A prototype keyboard is custom-designed to work with eye-blink detection and eyeball-movement tracking using Support Vector Machines (SVMs) and tested in a typical paralyzed person-environment under varied lighting conditions. Tests performed on male and female subjects of different ages showed results with a success rate of 92%.
Conclusions: Since the system needs about 2 seconds to process one command, real-time use is not required. The efficiency can be improved through the use of a depth sensor camera, faster processor environment, or motion estimation.
Downloads
Download data is not yet available.
References
[1] Estrela, V. V., A. C. B. Monteiro, R. P. França, Y. Iano, A. Khelassi, and N. Razmjooy. Health 4.0: Applications, Management, Technologies and Review. Medical Technologies Journal, Vol. 2, no. 4, Jan. 2019, pp. 262-76, https://doi.org/10.26415/2572-004X-vol2iss4p269-276
[2] Light, J., McNaughton, D. (2014). Communicative competence for individuals who require augmentative and alternative communication: A new definition for a new era of communication? Augmentative and Alternative Communication. 30 (1): 1–18. doi:10.3109/07434618.2014.885080
[3] Miraz, M., Ali, M., (2015). A Review on Internet of Things, Internet of Everything and Internet of Nano Things, In Proc. IEEE Int'l Conf. Internet Technologies and Applications, pp. 219-224, Wrexham.
[4] Skiba, D. (2016). Emerging Technologies: The Internet of Things (IoT), Nursing Education Perspectives, Vol. 34, No. 1, pp. 63–64.https://doi.org/10.5480/1536-5026-34.1.63
[5] Fernendez, F., Pallis, G., (2014). Opportunities and Challenges of the Internet of Things for Healthcare, Proceedings of Int'l Conf. on Wireless Mobile Communication and Healthcare, pp. 263-266, Athens. https://doi.org/10.4108/icst.mobihealth.2014.257276
[6] Sotiriadis, S., Bessis, N., Asimakopoulou, E., Mustafee, N., (2014). Towards Simulating the Internet of Things, In Proc. of IEEE 28th Int'l Conf. on Advanced Inf. Networking and Application Workshops, pp. 444-448, Victoria.
[7] Catarinucci, L., Donno, D., Mainetti, L., Palano, L., Patrono, L., Stefanizzi, M.., Tarricone, L. (2015). An IoT-Aware Architecture for Smart Healthcare Systems, IEEE Internet of Things Journal, Vol., 2, No., 6, pp. 515-526.
[8] Ge, S., Chun, S., Kim, H., Park, J. (2016). Design and Implementation of Interoperable IoT Healthcare System Based on International Standards, In Proc. of 13th IEEE Annual Cons. Comm. & Net. Conf., pp. 119-124.
[9] Gupta, M., Patchava, V., Menezes, V., (2015). "Healthcare based on loT using Raspberry Pi," Int'l Conf. on Green Computing and Internet of Things, pp. 796–799, Noida.
[10] Alkeem, E., Yeun, C., Zemerly, M., (2015). Security and Privacy Framework for Ubiquitous Healthcare IoT Devices, In Proc. 10th Int'l Conf. for Internet Techn. Sec.Trans., pp. 70-75, London.
[11] He, D., and Zeadally, S., (2015). An Analysis of RFID Authentication Schemes for Internet of Things in Healthcare Environment Using Elliptic Curve Cryptography, IEEE Internet of Things J., Vol., 2, No., 1, pp.72-83.
[12] Xu, B., Xu, L., Cai, H., Xie, C., Hu, J., Bu, F. (2014). Ubiquitous Data Accessing Method in IoT-Based Information System for Emergency Medical Services, IEEE Trans. Industrial Inf., Vol., 10, No., 2, pp. 1578-1586.
[13] Memon, Q., Khoja, S. (2012). RFID–based Patient Tracking for Regional Collaborative Healthcare, International Journal of Computer Applications in Technology, 45 (4), 231-244.
https://doi.org/10.1504/IJCAT.2012.051123
[14] Laplante, P., Laplante, N., (2016). The Internet of Things in Healthcare-Potential Applications and Challenges, IT Professional, pp. 2-4, Vol. 18, May-June.
[15] Istepanian, R., Sungoor, A., Faisal, A., Philip, N. (2011). Internet of m-Health Things- m-IOT, IET Seminar on Assisted Living, pp. 1-3, London.
[16] Jiménez, F., Torres, R., (2015). Building an IoT-aware Healthcare Monitoring System, In Proc. 34th Int'l Conf. Chilean Computer Science Society, pp. 1-4, Santiago. https://doi.org/10.1109/SCCC.2015.7416592
[17] Mohamed, J. (2014). Internet of Things: Remote Patient Monitoring Using Web Services and Cloud Computing, In Proc.IEEE International Conference on Internet of Things, pp. 256–263, Taipei. https://doi.org/10.1109/iThings.2014.45
[18] Pasluosta, C., Gassner, H., Winkler, J., Klucken, J., Eskofier, B. (2015). An Emerging Era in the Management of Parkinson's disease: Wearable Technologies and the Internet of Things, IEEE J. Biomed Health Inf Vol., 19, No. 6, pp. 1873–1881. https://doi.org/10.1109/JBHI.2015.2461555 PMid:26241979
[19] Amendola, S., Lodato, R., Manzari, S., Occhiuzzi, C., Marrocco, G., (2014). RFID Technology for IoT-Based Personal Healthcare in Smart Spaces, IEEE Internet of Things Journal, Vol., 1, No., 2, pp. 144-152.
[20] Gope, P., Hwang, T., (2016). BSN-Care: A Secure IoT-Based Modern Healthcare System Using Body Sensor Network, IEEE Sensors Journal, Vol.16, No. 5, pp. 1368-1376. https://doi.org/10.1109/JSEN.2015.2502401
[21] Jara, A., Zamora, M., and Skarmeta, A., (2010). An Architecture based on Internet of Things to Support Mobility and Security in Medical Environments, In Proc 7th IEEE Consumer Comm. Net. Conf., pp. 1 – 5, Las Vegas.
[22] Bayasut, B., Ananta, G., Muda, A. (2011). Intelligent Biometric Detection System for Disabled People, 11th International Conference on Hybrid Intelligent Systems, pp. 346–350, Melacca. https://doi.org/10.1109/HIS.2011.6122130
[23] Lopes, N., Pinto, F., Furtado, P., Silva, J., (2014). IoT Architecture Proposal for Disabled People, In Proc. 10th IEEE Int'l Conf. Wireless and Mobile Comp., Net. Communications, pp. 152 – 158, Larnaca.
[24] Yang, L., Ge, Y., Li, W., Rao, W., Shen, W., (2014). A Home Mobile Healthcare System for Wheelchair Users, In Proc. of the 18th IEEE Int'l Conf. on Computer Supported Cooperative Work in Design, pp. 609–614, Hsinchu. https://doi.org/10.1109/CSCWD.2014.6846914
[25] Blackstone, S., Beukelman, D., and Yorkston, K. (2015). Patient-Provider Communication: Roles for Speech-Language Pathologists and Other Healthcare Professionals, Plural Publishers.
[26] Ghosh, S., Nandy, T., & Manna, N. (2015). Real time eye detection and tracking method for driver assistance system. In Advancements of Medical Electronics (pp. 13-25). Springer, New Delhi. https://doi.org/10.1007/978-81-322-2256-9_2
[27] Galarza, E. E., Egas, F. D., Silva, F. M., Velasco, P. M., & Galarza, E. D. (2018). Real Time Driver Drowsiness Detection Based on Driver's Face Image Behavior Using a System of Human Computer Interaction Implemented in a Smartphone. In Proc. Int'l Conf. on Information Theoretic Security, pp. 563-572. Springer, Cham. https://doi.org/10.1007/978-3-319-73450-7_53
[28] Menezes, P., Francisco, J., & Patrão, B. (2018). The Importance of Eye-Tracking Analysis in Immersive Learning-A Low Cost Solution. In Online Engineering & Internet of Things (pp. 689-697). Springer, Cham. https://doi.org/10.1007/978-3-319-64352-6_65
[29] Liu, S. S., Rawicz, A., Ma, T., Zhang, C., Lin, K., Rezaei, S., & Wu, E. (2018). An eye-gaze tracking and human computer interface system for people with ALS and other locked-in diseases. CMBES Proceedings, 33(1).
[30] Galdi, C., & Nappi, M. (2019). Eye Movement Analysis in Biometrics. In Biometrics under Biomedical Considerations, pp. 171-183. Springer, Singapore. https://doi.org/10.1007/978-981-13-1144-4_8
[31] Razmjooy, N., Mousavi, B. S., and Soleymani, F. (2013). A hybrid neural network Imperialist Competitive Algorithm for skin color segmentation. Mathematical and Computer Modelling, 57(3-4), 848-856. https://doi.org/10.1016/j.mcm.2012.09.013
[32] Petrisor, D.; Fosalau, C.; Avila, M.; Mariut, F., (2011). Algorithm for face and eye detection using colour segmentation and invariant features, In Proc. 34th Int'l Conf. on Tel. and Signal Proc., vol., no., pp. 564-569, 18-20.
[33] Praglin, M., & Tan, B., (2014). Eye Detection and Gaze Estimation. Eye, pp. 1-5
[34] Lin L., Huang C., Ni X., Wang J., Zhang H., Li X., Qian Z. (2015). Driver fatigue detection based on eye state, Technology and Health Care: European Society for Engineering and Medicine, 23(s2): S453-S463 https://doi.org/10.3233/THC-150982 PMid:26410512
[35] Zia, M. A., Ansari, U., Jamil, M., Gillani, O., & Ayaz, Y. (2014). Face and Eye Detection in Images using Skin Color Segmentation and Circular Hough Transform, In Proc. IEEE Int'l Conf. Rob. Em. A Tec. Eng., pp. 211-213.
[36] Schölkopf B., Platt J., Shawe-Taylor J., Smola A., Williamson R., (2001). Estimating the Support of a High-dimensional Distribution, J. of Neural Computation, Vol. 13, No. 7, pp. 1443-1471 https://doi.org/10.1162/089976601750264965 PMid:11440593
[37] Divjak, M., Bischof, H., (2009). Eye-blink based fatigue detection for prevention of Computer Vision Syndrome, In Proc. Conf. on Machine Vision Applications (MVA), pp. 350–353
[38] Heishman, R., Duric, Z., (2007). Using Image Flow to Detect Eye-blinks in Color Videos, IEEE Workshop on Applications of Computer Vision, p. 52 https://doi.org/10.1109/WACV.2007.61
[39] Holland, E. (2008). Marquardt's Phi mask: Pitfalls of Relying on Fashion Models and the Golden ratio to describe a Beautiful Face, Aesthetic Plastic Surgery, Vo. 32, No. 2, pp. 200-208 https://doi.org/10.1007/s00266-007-9080-z PMid:18175168
[40] Nguyen, M.H., Perez J,. and De la Torre, F. (2008). Facial Feature Detection with Optimal Pixel Reduction SVM, In Proc. 8th IEEE Int'l Conf. Aut. Face & G. Recog., Amsterdam, pp. 1-6, doi: 10.1109/AFGR.2008.4813372
[41] Dhruw, K. K., (2009). A. Eye Detection Using Variants of Hough Transform B. Off-Line Signature Verification, Doctoral dissertation, National Institute of Technology Rourkela.
[42] Krolak, A., and Strumiłło, P. Eye-blink detection system for human–computer interaction, Univ Access Inf Soc, 11:409–419DOI 10.1007/s10209-011-0256-6.
[43] Coelho, A.M. and Estrela, V.V. (2012). A Study on the Effect of Regularization Matrices in Motion Estimation, IJCA (0975 – 8887), Vol 51, 19.
[44] Marins, H. R. and Estrela, V.V. (2016). On the Use of Motion Vectors for 2D and 3D Error Concealment in H.264/AVC Video. Feature Detectors and Motion Detection in Video Processing. IGI Global, 2017. 164-186. doi:10.4018/978-1-5225-1025-3.ch008
[45] Bloehdorn S. et al. (2005). Semantic Annotation of Images and Videos for Multimedia Analysis. In: Gómez-Pérez A., Euzenat J. (eds) The Semantic Web: Research and Applications. ESWC 2005. Lecture Notes in Computer Science, vol 3532. Springer, Berlin, Heidelberg https://doi.org/10.1007/11431053_40
[46] Memon, Q., Smarter Healthcare Collaborative Network, Building Next-Generation Converged Networks: Theory, 2013
[2] Light, J., McNaughton, D. (2014). Communicative competence for individuals who require augmentative and alternative communication: A new definition for a new era of communication? Augmentative and Alternative Communication. 30 (1): 1–18. doi:10.3109/07434618.2014.885080
[3] Miraz, M., Ali, M., (2015). A Review on Internet of Things, Internet of Everything and Internet of Nano Things, In Proc. IEEE Int'l Conf. Internet Technologies and Applications, pp. 219-224, Wrexham.
[4] Skiba, D. (2016). Emerging Technologies: The Internet of Things (IoT), Nursing Education Perspectives, Vol. 34, No. 1, pp. 63–64.https://doi.org/10.5480/1536-5026-34.1.63
[5] Fernendez, F., Pallis, G., (2014). Opportunities and Challenges of the Internet of Things for Healthcare, Proceedings of Int'l Conf. on Wireless Mobile Communication and Healthcare, pp. 263-266, Athens. https://doi.org/10.4108/icst.mobihealth.2014.257276
[6] Sotiriadis, S., Bessis, N., Asimakopoulou, E., Mustafee, N., (2014). Towards Simulating the Internet of Things, In Proc. of IEEE 28th Int'l Conf. on Advanced Inf. Networking and Application Workshops, pp. 444-448, Victoria.
[7] Catarinucci, L., Donno, D., Mainetti, L., Palano, L., Patrono, L., Stefanizzi, M.., Tarricone, L. (2015). An IoT-Aware Architecture for Smart Healthcare Systems, IEEE Internet of Things Journal, Vol., 2, No., 6, pp. 515-526.
[8] Ge, S., Chun, S., Kim, H., Park, J. (2016). Design and Implementation of Interoperable IoT Healthcare System Based on International Standards, In Proc. of 13th IEEE Annual Cons. Comm. & Net. Conf., pp. 119-124.
[9] Gupta, M., Patchava, V., Menezes, V., (2015). "Healthcare based on loT using Raspberry Pi," Int'l Conf. on Green Computing and Internet of Things, pp. 796–799, Noida.
[10] Alkeem, E., Yeun, C., Zemerly, M., (2015). Security and Privacy Framework for Ubiquitous Healthcare IoT Devices, In Proc. 10th Int'l Conf. for Internet Techn. Sec.Trans., pp. 70-75, London.
[11] He, D., and Zeadally, S., (2015). An Analysis of RFID Authentication Schemes for Internet of Things in Healthcare Environment Using Elliptic Curve Cryptography, IEEE Internet of Things J., Vol., 2, No., 1, pp.72-83.
[12] Xu, B., Xu, L., Cai, H., Xie, C., Hu, J., Bu, F. (2014). Ubiquitous Data Accessing Method in IoT-Based Information System for Emergency Medical Services, IEEE Trans. Industrial Inf., Vol., 10, No., 2, pp. 1578-1586.
[13] Memon, Q., Khoja, S. (2012). RFID–based Patient Tracking for Regional Collaborative Healthcare, International Journal of Computer Applications in Technology, 45 (4), 231-244.
https://doi.org/10.1504/IJCAT.2012.051123
[14] Laplante, P., Laplante, N., (2016). The Internet of Things in Healthcare-Potential Applications and Challenges, IT Professional, pp. 2-4, Vol. 18, May-June.
[15] Istepanian, R., Sungoor, A., Faisal, A., Philip, N. (2011). Internet of m-Health Things- m-IOT, IET Seminar on Assisted Living, pp. 1-3, London.
[16] Jiménez, F., Torres, R., (2015). Building an IoT-aware Healthcare Monitoring System, In Proc. 34th Int'l Conf. Chilean Computer Science Society, pp. 1-4, Santiago. https://doi.org/10.1109/SCCC.2015.7416592
[17] Mohamed, J. (2014). Internet of Things: Remote Patient Monitoring Using Web Services and Cloud Computing, In Proc.IEEE International Conference on Internet of Things, pp. 256–263, Taipei. https://doi.org/10.1109/iThings.2014.45
[18] Pasluosta, C., Gassner, H., Winkler, J., Klucken, J., Eskofier, B. (2015). An Emerging Era in the Management of Parkinson's disease: Wearable Technologies and the Internet of Things, IEEE J. Biomed Health Inf Vol., 19, No. 6, pp. 1873–1881. https://doi.org/10.1109/JBHI.2015.2461555 PMid:26241979
[19] Amendola, S., Lodato, R., Manzari, S., Occhiuzzi, C., Marrocco, G., (2014). RFID Technology for IoT-Based Personal Healthcare in Smart Spaces, IEEE Internet of Things Journal, Vol., 1, No., 2, pp. 144-152.
[20] Gope, P., Hwang, T., (2016). BSN-Care: A Secure IoT-Based Modern Healthcare System Using Body Sensor Network, IEEE Sensors Journal, Vol.16, No. 5, pp. 1368-1376. https://doi.org/10.1109/JSEN.2015.2502401
[21] Jara, A., Zamora, M., and Skarmeta, A., (2010). An Architecture based on Internet of Things to Support Mobility and Security in Medical Environments, In Proc 7th IEEE Consumer Comm. Net. Conf., pp. 1 – 5, Las Vegas.
[22] Bayasut, B., Ananta, G., Muda, A. (2011). Intelligent Biometric Detection System for Disabled People, 11th International Conference on Hybrid Intelligent Systems, pp. 346–350, Melacca. https://doi.org/10.1109/HIS.2011.6122130
[23] Lopes, N., Pinto, F., Furtado, P., Silva, J., (2014). IoT Architecture Proposal for Disabled People, In Proc. 10th IEEE Int'l Conf. Wireless and Mobile Comp., Net. Communications, pp. 152 – 158, Larnaca.
[24] Yang, L., Ge, Y., Li, W., Rao, W., Shen, W., (2014). A Home Mobile Healthcare System for Wheelchair Users, In Proc. of the 18th IEEE Int'l Conf. on Computer Supported Cooperative Work in Design, pp. 609–614, Hsinchu. https://doi.org/10.1109/CSCWD.2014.6846914
[25] Blackstone, S., Beukelman, D., and Yorkston, K. (2015). Patient-Provider Communication: Roles for Speech-Language Pathologists and Other Healthcare Professionals, Plural Publishers.
[26] Ghosh, S., Nandy, T., & Manna, N. (2015). Real time eye detection and tracking method for driver assistance system. In Advancements of Medical Electronics (pp. 13-25). Springer, New Delhi. https://doi.org/10.1007/978-81-322-2256-9_2
[27] Galarza, E. E., Egas, F. D., Silva, F. M., Velasco, P. M., & Galarza, E. D. (2018). Real Time Driver Drowsiness Detection Based on Driver's Face Image Behavior Using a System of Human Computer Interaction Implemented in a Smartphone. In Proc. Int'l Conf. on Information Theoretic Security, pp. 563-572. Springer, Cham. https://doi.org/10.1007/978-3-319-73450-7_53
[28] Menezes, P., Francisco, J., & Patrão, B. (2018). The Importance of Eye-Tracking Analysis in Immersive Learning-A Low Cost Solution. In Online Engineering & Internet of Things (pp. 689-697). Springer, Cham. https://doi.org/10.1007/978-3-319-64352-6_65
[29] Liu, S. S., Rawicz, A., Ma, T., Zhang, C., Lin, K., Rezaei, S., & Wu, E. (2018). An eye-gaze tracking and human computer interface system for people with ALS and other locked-in diseases. CMBES Proceedings, 33(1).
[30] Galdi, C., & Nappi, M. (2019). Eye Movement Analysis in Biometrics. In Biometrics under Biomedical Considerations, pp. 171-183. Springer, Singapore. https://doi.org/10.1007/978-981-13-1144-4_8
[31] Razmjooy, N., Mousavi, B. S., and Soleymani, F. (2013). A hybrid neural network Imperialist Competitive Algorithm for skin color segmentation. Mathematical and Computer Modelling, 57(3-4), 848-856. https://doi.org/10.1016/j.mcm.2012.09.013
[32] Petrisor, D.; Fosalau, C.; Avila, M.; Mariut, F., (2011). Algorithm for face and eye detection using colour segmentation and invariant features, In Proc. 34th Int'l Conf. on Tel. and Signal Proc., vol., no., pp. 564-569, 18-20.
[33] Praglin, M., & Tan, B., (2014). Eye Detection and Gaze Estimation. Eye, pp. 1-5
[34] Lin L., Huang C., Ni X., Wang J., Zhang H., Li X., Qian Z. (2015). Driver fatigue detection based on eye state, Technology and Health Care: European Society for Engineering and Medicine, 23(s2): S453-S463 https://doi.org/10.3233/THC-150982 PMid:26410512
[35] Zia, M. A., Ansari, U., Jamil, M., Gillani, O., & Ayaz, Y. (2014). Face and Eye Detection in Images using Skin Color Segmentation and Circular Hough Transform, In Proc. IEEE Int'l Conf. Rob. Em. A Tec. Eng., pp. 211-213.
[36] Schölkopf B., Platt J., Shawe-Taylor J., Smola A., Williamson R., (2001). Estimating the Support of a High-dimensional Distribution, J. of Neural Computation, Vol. 13, No. 7, pp. 1443-1471 https://doi.org/10.1162/089976601750264965 PMid:11440593
[37] Divjak, M., Bischof, H., (2009). Eye-blink based fatigue detection for prevention of Computer Vision Syndrome, In Proc. Conf. on Machine Vision Applications (MVA), pp. 350–353
[38] Heishman, R., Duric, Z., (2007). Using Image Flow to Detect Eye-blinks in Color Videos, IEEE Workshop on Applications of Computer Vision, p. 52 https://doi.org/10.1109/WACV.2007.61
[39] Holland, E. (2008). Marquardt's Phi mask: Pitfalls of Relying on Fashion Models and the Golden ratio to describe a Beautiful Face, Aesthetic Plastic Surgery, Vo. 32, No. 2, pp. 200-208 https://doi.org/10.1007/s00266-007-9080-z PMid:18175168
[40] Nguyen, M.H., Perez J,. and De la Torre, F. (2008). Facial Feature Detection with Optimal Pixel Reduction SVM, In Proc. 8th IEEE Int'l Conf. Aut. Face & G. Recog., Amsterdam, pp. 1-6, doi: 10.1109/AFGR.2008.4813372
[41] Dhruw, K. K., (2009). A. Eye Detection Using Variants of Hough Transform B. Off-Line Signature Verification, Doctoral dissertation, National Institute of Technology Rourkela.
[42] Krolak, A., and Strumiłło, P. Eye-blink detection system for human–computer interaction, Univ Access Inf Soc, 11:409–419DOI 10.1007/s10209-011-0256-6.
[43] Coelho, A.M. and Estrela, V.V. (2012). A Study on the Effect of Regularization Matrices in Motion Estimation, IJCA (0975 – 8887), Vol 51, 19.
[44] Marins, H. R. and Estrela, V.V. (2016). On the Use of Motion Vectors for 2D and 3D Error Concealment in H.264/AVC Video. Feature Detectors and Motion Detection in Video Processing. IGI Global, 2017. 164-186. doi:10.4018/978-1-5225-1025-3.ch008
[45] Bloehdorn S. et al. (2005). Semantic Annotation of Images and Videos for Multimedia Analysis. In: Gómez-Pérez A., Euzenat J. (eds) The Semantic Web: Research and Applications. ESWC 2005. Lecture Notes in Computer Science, vol 3532. Springer, Berlin, Heidelberg https://doi.org/10.1007/11431053_40
[46] Memon, Q., Smarter Healthcare Collaborative Network, Building Next-Generation Converged Networks: Theory, 2013
Published
2019-04-01
Issue
Section
Medical technologies
