Robotic Assistant for Object Recognition Using Convolutional Neural Network
Article Sidebar
Published:
Feb 12, 2024
Keywords:
Convolutional Neural Network, Robotics, Object Recognition, Computer Vision, YOLO
Main Article Content
Abstract
Visually impaired persons encounter certain challenges, which include access to information, environmental navigation, and obstacle detection. Navigating daily life becomes a big task with challenges relating to the search for misplaced personal items and being aware of objects in their environment to avoid collision. This necessitates the need for automated solutions to facilitate object recognition. While traditional methods like guide dogs, white canes, and Braille have offered valuable solutions, recent technological solutions, including smartphone-based recognition systems and portable cameras, have encountered limitations such as constraints relating to cultural-specific, device-specific, and lack of system autonomy. This study addressed and provided solutions to the limitations offered by recent solutions by introducing a Convolutional Neural Network (CNN) object recognition system integrated into a mobile robot designed to function as a robotic assistant for visually impaired persons. The robotic assistant is capable of moving around in a confined environment. It incorporates a Raspberry Pi with a camera programmed to recognize three objects: mobile phones, mice, and chairs. A Convolutional Neural Network model was trained for object recognition, with 30% of the images used for testing. The training was conducted using the Yolov3 model in Google Colab. Qualitative evaluation of the recognition system yielded a precision of 79%, recall of 96%, and accuracy of 80% for the Robotic Assistant. It also includes a Graphical User Interface where users can easily control the movement and speed of the robotic assistant. The developed robotic assistant significantly enhances autonomy and object recognition, promising substantial benefits in the daily navigation of visually impaired individuals.
Article Details
How to Cite
Oluyele, S., Adeyanju, I., & Sobowale, A. (2024). Robotic Assistant for Object Recognition Using Convolutional Neural Network. ABUAD Journal of Engineering Research and Development, 7(1), 1-13. https://doi.org/10.53982/ajerd.2024.0701.01-j
Issue
Section
Articles
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
References
[1] Ester, M., & Angel, d. P. (2017). Object Detection and Recognition for Assistive Robots. IEEE Robotics & Automation Magazine, 1(1), 1-12.
[2] Anderson, J., & Rainie, L. (2018). Improvements ahead: How humans and AI might evolve together in the next decade. Pew Research Centers: Artificial Intelligence and the future of humans, 4.
[3] Alisha, P., Kanika, M., & Leah, F. (2018). Accessibility Came by Accident": Use of Voice-Controlled Intelligent Personal Assistants by People with Disabilities. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 18, 1-13.
[4] Wisskirchen, G., Biacabe, B. T., Bormann, U., Muntz, A., Niehaus, G., Soler, G. J., & Brauchitsch, B. v. (2017). Artificial Intelligence and Robotics and Their Impact on the Workplace. IBA Global Employment Institute, 1, 1-120.
[5] Stevelink, S., Malcolm, E., & Fear, N. (2015). Visual impairment, coping strategies and impact on daily life: a qualitative study among working-age UK ex-service personnel. BMC Public Health, 1, 1-15.
[6] Yi, C., Flore, R. W., R. C., & Tian, Y. (2013). Finding Objects for Assisting Blind People. Network Modeling Analysis in Health Informatics and Bioinformatics, 2, 71-79.
[7] Singh, G., Kandale, O., Takhtani, K., & Dadhwal, N. (2020). A Smart Personal AI Assistant for Visually Impaired People. International Research Journal of Engineering and Technology (IRJET), 7(6), 1450-1454.
[8] Rickly, J., Halpern, N., Hansen, M., & Welsman, J. (2021). Travelling with a Guide Dog: Experiences of People with Vision Impairment. Transport Inequalities, Transport Poverty and Sustainability, 13(5), 2840.
[9] Attia, I., & Asamoah, D. (2020). The White Cane. Its Effectiveness, Challenges and Suggestions for Effective Use: The Case of Akropong School for the Blind. Journal of Education, Society and Behavioural Science, 33(3), 47-55.
[10] Encalada, E. G., Jordán, C. d., Chicaiza, V. E., & Pazmiño, S. J. (2022). Enhancing Reading Competence through the Braille System for Visually Impaired People: A Preliminary Study. International Journal of Teaching and Learning, 1(1), 65-77.
[11] Shaikh, S. (2020). Assistive Object Recognition System for Visually Impaired. International Journal of Engineering Research & Technology (IJERT), 9(9), 736-740.
[12] Mohane, V., & Gode, C. (2016). Object recognition for blind people using portable camera. 2016 World Conference on Futuristic Trends in Research and Innovation for Social Welfare (Startup Conclave), 1-4.
[13] Alzahrani, N., & Al-Baity, H. (2023). Object Recognition System for the Visually Impaired: A Deep Learning Approach using Arabic Annotation. Applications of Neural Networks for Speech and Language Processing, 12(3), 541.
[14] Wiley, V., & Lucas, T. (2018). Computer Vision and Image Processing: A Paper Review. International Journal of Artificial Intelegence Research, 2(1), 28-36.
[15] Kakkar, S., & Monga, V. (2017). A STUDY ON ARTIFICIAL INTELLIGENCE IN E-COMMERCE. International Journal of Advances in Engineering & Scientific Research, 4(4), 62-68.
[16] Aswinvenu. (2019). Element14 Community. Retrieved 2023, from https://community.element14.com/products/roadtest/rv/roadtest_reviews/648/raspberry_pi_3_model_5
[17] Secinaro, S., Calandra, D., Secinaro, A., Muthurangu, V., & Biancone, P. (2021). The role of artificial intelligence in healthcare: a structured literature review. BMC Medical Informatics and Decision Making, 21(1), 1-23.
[18] Biswal, A. (2021). Top 10 Artificial Intelligence Applications. Simplilearn Journal on Artificial Intelligence, 1, 50-62.
[19] Khensous, G., Labed, K., & Labed, Z. (2023). Exploring the evolution and applications of natural language processing in education. Revista Română de Informatică și Automatică, 33(2), 61-74.
[20] Pedro, R., & Garcia, F. (2015). Object recognition for a service robot. Master's thesis, University of Aveiro,, 1(1), 1-80.
[21] Wei, L., Dragomir, A., Dumitru, E., Christian, S., Scott, R., Cheng-Yang, F., & Alexander, B. (2015). SSD: Single Shot MultiBox Detector. Journal for Computer Vision and Pattern Recognition, 5, 21-37.
[22] David, C. C., Fiorella, S., & Marina, I. (2022). A Framework for Safe and Intuitive Human-Robot Interaction for Assistant Robotics. In Proceedings of the 2022 IEEE 27th International Conference on Emerging Technologies and Factory Automation, 1-4.
[23] Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2016). Region-Based Convolutional Networks for Accurate Object Detection and Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(1), 142-158.
[24] Anand, J., & Divyakant., M. (2020). A Comparative Study of Various Object Detection Algorithms and Performance Analysis. International Journal of Computer Sciences and Engineering, 8, 158-163.
[25] Zientara, P. A., Lee, S., Smith, G. H., Brenner, R., Itti, L., Rosson, M. B. & Narayanan, V. (2017). Third Eye: A Shopping Assistant for the Visually Impaired. IEEE Computer Society Journal, 50(2), 16-24.
[26] Elmannai, W., & Elleithy, K. (2017). Sensor-Based Assistive Devices for Visually-Impaired People: Current Status. Sensors for Globalized Healthy Living and Wellbeing, 17(3), 565.
[27] Chaudhari, G., Deshpande, A., & Liu, K. (2017). Smart Robotic Assistant for Visually Impaired. National Science Foundation, 12(7), 345-352.
[28] Nguyen, H., Nguyen, M., Nguyen, Q., Yang, S., & Le, H. (2020). Web-based object detection and sound feedback system for visually impaired people. International Conference on Multimedia Analysis and Pattern Recognition, 1-6.
[29] Potdar, K., Pai, C. D., & Akolkar, S. (2018). A Convolutional Neural Network based Live Object Recognition System as Blind Aid. Computer Vision and Pattern Recognition, 1(1).
[30] Bhandari, A., Prasad, P., Alsadoon, A., & Maag, A. (2019). Object detection and recognition: using deep learning to assist the visually impaired. Disability and Rehabilitation: Assistive Technology, 16(2), 1-9.
[31] Parikh, N., Shah, I., & Vahora, S. (2018). Android Smartphone Based Visual Object Recognition for Visually Impaired Using Deep Learning. International Conference on Cryptography, Security and Privacy, 420-425.
[32] Shirley, Rane, K., Rao, K. H., B, B. B., Agrawal, P., & Rawat, N. (2023). Machine learning and Sensor-Based Multi-Robot System with Voice Recognition for Assisting the Visually Impaired. Journal of Machine and Computing, 3(3), 206-215.
[33] Najm, H., Elferjani, K., & Alariyibi, A. (2022). Assisting Blind People Using Object Detection with Vocal Feedback. 2022 IEEE 2nd International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering (MI-STA), 48-52.
[34] Adeyanju, I. A., Azeez, M. A., Bello, O. O., & Badmus, T. A. (2022). Development of a Convolutional Neural Network-Based Object Recognition System for Uncovered Gutters and Bollards. ABUAD Journal of Engineering Research and Development (AJERD), 5(1), 147-154.
[35] Gautam, S., Sivaraman, K., Muralidharan, H., & Baskar, A. (2015). Vision System with Audio Feedback to Assist Visually Impaired to Grasp Objects. Procedia Computer Science, 58, 387-394.
[36] Breve, F. A., & Fischer, C. (2020). Visually Impaired Aid using Convolutional Neural Networks, Transfer Learning, and Particle Competition and Cooperation. IEEE International Joint Conference on Neural Network, 1-8.
[37] Caballero, A., Catli, K. E., & Babierra, A. G. (2020). Object Recognition and Hearing Assistive Technology Mobile Application using Convolutional Neural Network. 2020 International Conference on Wireless Communication and Sensor Networks, 41-48.
[38] Lailla, B., Yandre, C., & Linnyer, A. (2018). Automata Classification with Convolutional Neural Networks for Use in Assistive Technologies for the Visually Impaired. 11th PErvasive Technologies Related to Assistive Environments Conference, 157-168.
[39] Shaikh, S., Karale, V., & Tawde, G. (2020). Assistive Object Recognition System for Visually Impaired. The International Journal of Engineering Research & Technology (IJERT), 9(9), 736-740.
[40] Jessica, A., Veena, S. H., Srivarshini, S., Krishna, R. G., & Mounica, M. (2022). Convolutional Neural Networks based Voice Assistive System for Blind People. International Conference on Communication and Electronics Systems (ICCES), 7, 1608-1613.
[41] Kinra, A., Walia, W., & Sharanya, S. (2023). A Comprehensive and Systematic Review of Deep Learning Based Object Recognition Techniques for the Visually Impaired. 2023 2nd International Conference on Computational Systems and Communication (ICCSC), 2, 1-6.
[42] Nikhil, A. S., Swarnalatha, & Lav, M. (2019). Guiding and Navigation for the Blind using Deep Convolutional Neural Network Based Predictive Object Tracking. International Journal of Engineering and Advanced Technology, 9(3), 306-313.
[43] Dahiya, D., Gupta, H., & Dutta, M. K. (2020). A Deep Learning based Real Time Assistive Framework for Visually Impaired. 2020 International Conference on Contemporary Computing and Applications (IC3A), 106-109.
[44] Hsieh, Y.-Z., Lin, S.-S., & Xu, F.-X. (2020). Development of a wearable guide device based on convolutional neural network for blind or visually impaired persons. Multimedia Tools and Applications, 29473–29491.
[45] Yi, C., Flores, R. W., Chincha, R., & Tian, Y. (2013). Finding Objects for Assisting Blind People. Netw Model Anal Health Inform Bioinform, 2(2), 71-79.
[46] Neto, L. B., Grijalva, F., Maike, V. R., Martini, L. C., & Florencio, D. (2017). A Kinect-Based Wearable Face Recognition System to Aid Visually Impaired Users. IEEE, 47(1), 52-64.
[2] Anderson, J., & Rainie, L. (2018). Improvements ahead: How humans and AI might evolve together in the next decade. Pew Research Centers: Artificial Intelligence and the future of humans, 4.
[3] Alisha, P., Kanika, M., & Leah, F. (2018). Accessibility Came by Accident": Use of Voice-Controlled Intelligent Personal Assistants by People with Disabilities. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 18, 1-13.
[4] Wisskirchen, G., Biacabe, B. T., Bormann, U., Muntz, A., Niehaus, G., Soler, G. J., & Brauchitsch, B. v. (2017). Artificial Intelligence and Robotics and Their Impact on the Workplace. IBA Global Employment Institute, 1, 1-120.
[5] Stevelink, S., Malcolm, E., & Fear, N. (2015). Visual impairment, coping strategies and impact on daily life: a qualitative study among working-age UK ex-service personnel. BMC Public Health, 1, 1-15.
[6] Yi, C., Flore, R. W., R. C., & Tian, Y. (2013). Finding Objects for Assisting Blind People. Network Modeling Analysis in Health Informatics and Bioinformatics, 2, 71-79.
[7] Singh, G., Kandale, O., Takhtani, K., & Dadhwal, N. (2020). A Smart Personal AI Assistant for Visually Impaired People. International Research Journal of Engineering and Technology (IRJET), 7(6), 1450-1454.
[8] Rickly, J., Halpern, N., Hansen, M., & Welsman, J. (2021). Travelling with a Guide Dog: Experiences of People with Vision Impairment. Transport Inequalities, Transport Poverty and Sustainability, 13(5), 2840.
[9] Attia, I., & Asamoah, D. (2020). The White Cane. Its Effectiveness, Challenges and Suggestions for Effective Use: The Case of Akropong School for the Blind. Journal of Education, Society and Behavioural Science, 33(3), 47-55.
[10] Encalada, E. G., Jordán, C. d., Chicaiza, V. E., & Pazmiño, S. J. (2022). Enhancing Reading Competence through the Braille System for Visually Impaired People: A Preliminary Study. International Journal of Teaching and Learning, 1(1), 65-77.
[11] Shaikh, S. (2020). Assistive Object Recognition System for Visually Impaired. International Journal of Engineering Research & Technology (IJERT), 9(9), 736-740.
[12] Mohane, V., & Gode, C. (2016). Object recognition for blind people using portable camera. 2016 World Conference on Futuristic Trends in Research and Innovation for Social Welfare (Startup Conclave), 1-4.
[13] Alzahrani, N., & Al-Baity, H. (2023). Object Recognition System for the Visually Impaired: A Deep Learning Approach using Arabic Annotation. Applications of Neural Networks for Speech and Language Processing, 12(3), 541.
[14] Wiley, V., & Lucas, T. (2018). Computer Vision and Image Processing: A Paper Review. International Journal of Artificial Intelegence Research, 2(1), 28-36.
[15] Kakkar, S., & Monga, V. (2017). A STUDY ON ARTIFICIAL INTELLIGENCE IN E-COMMERCE. International Journal of Advances in Engineering & Scientific Research, 4(4), 62-68.
[16] Aswinvenu. (2019). Element14 Community. Retrieved 2023, from https://community.element14.com/products/roadtest/rv/roadtest_reviews/648/raspberry_pi_3_model_5
[17] Secinaro, S., Calandra, D., Secinaro, A., Muthurangu, V., & Biancone, P. (2021). The role of artificial intelligence in healthcare: a structured literature review. BMC Medical Informatics and Decision Making, 21(1), 1-23.
[18] Biswal, A. (2021). Top 10 Artificial Intelligence Applications. Simplilearn Journal on Artificial Intelligence, 1, 50-62.
[19] Khensous, G., Labed, K., & Labed, Z. (2023). Exploring the evolution and applications of natural language processing in education. Revista Română de Informatică și Automatică, 33(2), 61-74.
[20] Pedro, R., & Garcia, F. (2015). Object recognition for a service robot. Master's thesis, University of Aveiro,, 1(1), 1-80.
[21] Wei, L., Dragomir, A., Dumitru, E., Christian, S., Scott, R., Cheng-Yang, F., & Alexander, B. (2015). SSD: Single Shot MultiBox Detector. Journal for Computer Vision and Pattern Recognition, 5, 21-37.
[22] David, C. C., Fiorella, S., & Marina, I. (2022). A Framework for Safe and Intuitive Human-Robot Interaction for Assistant Robotics. In Proceedings of the 2022 IEEE 27th International Conference on Emerging Technologies and Factory Automation, 1-4.
[23] Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2016). Region-Based Convolutional Networks for Accurate Object Detection and Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(1), 142-158.
[24] Anand, J., & Divyakant., M. (2020). A Comparative Study of Various Object Detection Algorithms and Performance Analysis. International Journal of Computer Sciences and Engineering, 8, 158-163.
[25] Zientara, P. A., Lee, S., Smith, G. H., Brenner, R., Itti, L., Rosson, M. B. & Narayanan, V. (2017). Third Eye: A Shopping Assistant for the Visually Impaired. IEEE Computer Society Journal, 50(2), 16-24.
[26] Elmannai, W., & Elleithy, K. (2017). Sensor-Based Assistive Devices for Visually-Impaired People: Current Status. Sensors for Globalized Healthy Living and Wellbeing, 17(3), 565.
[27] Chaudhari, G., Deshpande, A., & Liu, K. (2017). Smart Robotic Assistant for Visually Impaired. National Science Foundation, 12(7), 345-352.
[28] Nguyen, H., Nguyen, M., Nguyen, Q., Yang, S., & Le, H. (2020). Web-based object detection and sound feedback system for visually impaired people. International Conference on Multimedia Analysis and Pattern Recognition, 1-6.
[29] Potdar, K., Pai, C. D., & Akolkar, S. (2018). A Convolutional Neural Network based Live Object Recognition System as Blind Aid. Computer Vision and Pattern Recognition, 1(1).
[30] Bhandari, A., Prasad, P., Alsadoon, A., & Maag, A. (2019). Object detection and recognition: using deep learning to assist the visually impaired. Disability and Rehabilitation: Assistive Technology, 16(2), 1-9.
[31] Parikh, N., Shah, I., & Vahora, S. (2018). Android Smartphone Based Visual Object Recognition for Visually Impaired Using Deep Learning. International Conference on Cryptography, Security and Privacy, 420-425.
[32] Shirley, Rane, K., Rao, K. H., B, B. B., Agrawal, P., & Rawat, N. (2023). Machine learning and Sensor-Based Multi-Robot System with Voice Recognition for Assisting the Visually Impaired. Journal of Machine and Computing, 3(3), 206-215.
[33] Najm, H., Elferjani, K., & Alariyibi, A. (2022). Assisting Blind People Using Object Detection with Vocal Feedback. 2022 IEEE 2nd International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering (MI-STA), 48-52.
[34] Adeyanju, I. A., Azeez, M. A., Bello, O. O., & Badmus, T. A. (2022). Development of a Convolutional Neural Network-Based Object Recognition System for Uncovered Gutters and Bollards. ABUAD Journal of Engineering Research and Development (AJERD), 5(1), 147-154.
[35] Gautam, S., Sivaraman, K., Muralidharan, H., & Baskar, A. (2015). Vision System with Audio Feedback to Assist Visually Impaired to Grasp Objects. Procedia Computer Science, 58, 387-394.
[36] Breve, F. A., & Fischer, C. (2020). Visually Impaired Aid using Convolutional Neural Networks, Transfer Learning, and Particle Competition and Cooperation. IEEE International Joint Conference on Neural Network, 1-8.
[37] Caballero, A., Catli, K. E., & Babierra, A. G. (2020). Object Recognition and Hearing Assistive Technology Mobile Application using Convolutional Neural Network. 2020 International Conference on Wireless Communication and Sensor Networks, 41-48.
[38] Lailla, B., Yandre, C., & Linnyer, A. (2018). Automata Classification with Convolutional Neural Networks for Use in Assistive Technologies for the Visually Impaired. 11th PErvasive Technologies Related to Assistive Environments Conference, 157-168.
[39] Shaikh, S., Karale, V., & Tawde, G. (2020). Assistive Object Recognition System for Visually Impaired. The International Journal of Engineering Research & Technology (IJERT), 9(9), 736-740.
[40] Jessica, A., Veena, S. H., Srivarshini, S., Krishna, R. G., & Mounica, M. (2022). Convolutional Neural Networks based Voice Assistive System for Blind People. International Conference on Communication and Electronics Systems (ICCES), 7, 1608-1613.
[41] Kinra, A., Walia, W., & Sharanya, S. (2023). A Comprehensive and Systematic Review of Deep Learning Based Object Recognition Techniques for the Visually Impaired. 2023 2nd International Conference on Computational Systems and Communication (ICCSC), 2, 1-6.
[42] Nikhil, A. S., Swarnalatha, & Lav, M. (2019). Guiding and Navigation for the Blind using Deep Convolutional Neural Network Based Predictive Object Tracking. International Journal of Engineering and Advanced Technology, 9(3), 306-313.
[43] Dahiya, D., Gupta, H., & Dutta, M. K. (2020). A Deep Learning based Real Time Assistive Framework for Visually Impaired. 2020 International Conference on Contemporary Computing and Applications (IC3A), 106-109.
[44] Hsieh, Y.-Z., Lin, S.-S., & Xu, F.-X. (2020). Development of a wearable guide device based on convolutional neural network for blind or visually impaired persons. Multimedia Tools and Applications, 29473–29491.
[45] Yi, C., Flores, R. W., Chincha, R., & Tian, Y. (2013). Finding Objects for Assisting Blind People. Netw Model Anal Health Inform Bioinform, 2(2), 71-79.
[46] Neto, L. B., Grijalva, F., Maike, V. R., Martini, L. C., & Florencio, D. (2017). A Kinect-Based Wearable Face Recognition System to Aid Visually Impaired Users. IEEE, 47(1), 52-64.