Robotic Assistant for Object Recognition Using Convolutional Neural Network
Article Sidebar
Main Article Content
Abstract
Visually impaired persons encounter certain challenges, which include access to information, environmental navigation, and obstacle detection. Navigating daily life becomes a big task with challenges relating to the search for misplaced personal items and being aware of objects in their environment to avoid collision. This necessitates the need for automated solutions to facilitate object recognition. While traditional methods like guide dogs, white canes, and Braille have offered valuable solutions, recent technological solutions, including smartphone-based recognition systems and portable cameras, have encountered limitations such as constraints relating to cultural-specific, device-specific, and lack of system autonomy. This study addressed and provided solutions to the limitations offered by recent solutions by introducing a Convolutional Neural Network (CNN) object recognition system integrated into a mobile robot designed to function as a robotic assistant for visually impaired persons. The robotic assistant is capable of moving around in a confined environment. It incorporates a Raspberry Pi with a camera programmed to recognize three objects: mobile phones, mice, and chairs. A Convolutional Neural Network model was trained for object recognition, with 30% of the images used for testing. The training was conducted using the Yolov3 model in Google Colab. Qualitative evaluation of the recognition system yielded a precision of 79%, recall of 96%, and accuracy of 80% for the Robotic Assistant. It also includes a Graphical User Interface where users can easily control the movement and speed of the robotic assistant. The developed robotic assistant significantly enhances autonomy and object recognition, promising substantial benefits in the daily navigation of visually impaired individuals.
Article Details
References
Ester, M., & Angel, d. P. (2017). Object Detection and Recognition for Assistive Robots. IEEE Robotics & Automation Magazine, 1(1), 1-12.
Anderson, J., & Rainie, L. (2018). Improvements ahead: How humans and AI might evolve together in the next decade. Pew Research Centers: Artificial Intelligence and the future of humans, 4.
Alisha, P., Kanika, M., & Leah, F. (2018). Accessibility Came by Accident": Use of Voice-Controlled Intelligent Personal Assistants by People with Disabilities. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 18, 1-13.
Wisskirchen, G., Biacabe, B. T., Bormann, U., Muntz, A., Niehaus, G., Soler, G. J., & Brauchitsch, B. v. (2017). Artificial Intelligence and Robotics and Their Impact on the Workplace. IBA Global Employment Institute, 1, 1-120.
Stevelink, S., Malcolm, E., & Fear, N. (2015). Visual impairment, coping strategies and impact on daily life: a qualitative study among working-age UK ex-service personnel. BMC Public Health, 1, 1-15. DOI: https://doi.org/10.1186/s12889-015-2455-1
Yi, C., Flore, R. W., R. C., & Tian, Y. (2013). Finding Objects for Assisting Blind People. Network Modeling Analysis in Health Informatics and Bioinformatics, 2, 71-79.
Singh, G., Kandale, O., Takhtani, K., & Dadhwal, N. (2020). A Smart Personal AI Assistant for Visually Impaired People. International Research Journal of Engineering and Technology (IRJET), 7(6), 1450-1454.
Rickly, J., Halpern, N., Hansen, M., & Welsman, J. (2021). Travelling with a Guide Dog: Experiences of People with Vision Impairment. Transport Inequalities, Transport Poverty and Sustainability, 13(5), 2840. DOI: https://doi.org/10.3390/su13052840
Attia, I., & Asamoah, D. (2020). The White Cane. Its Effectiveness, Challenges and Suggestions for Effective Use: The Case of Akropong School for the Blind. Journal of Education, Society and Behavioural Science, 33(3), 47-55. DOI: https://doi.org/10.9734/jesbs/2020/v33i330211
Encalada, E. G., Jordán, C. d., Chicaiza, V. E., & Pazmiño, S. J. (2022). Enhancing Reading Competence through the Braille System for Visually Impaired People: A Preliminary Study. International Journal of Teaching and Learning, 1(1), 65-77. DOI: https://doi.org/10.17501/26827034.2021.1106
Shaikh, S. (2020). Assistive Object Recognition System for Visually Impaired. International Journal of Engineering Research & Technology (IJERT), 9(9), 736-740.
Mohane, V., & Gode, C. (2016). Object recognition for blind people using portable camera. 2016 World Conference on Futuristic Trends in Research and Innovation for Social Welfare (Startup Conclave), 1-4. DOI: https://doi.org/10.1109/STARTUP.2016.7583932
Alzahrani, N., & Al-Baity, H. (2023). Object Recognition System for the Visually Impaired: A Deep Learning Approach using Arabic Annotation. Applications of Neural Networks for Speech and Language Processing, 12(3), 541. DOI: https://doi.org/10.3390/electronics12030541
Wiley, V., & Lucas, T. (2018). Computer Vision and Image Processing: A Paper Review. International Journal of Artificial Intelegence Research, 2(1), 28-36. DOI: https://doi.org/10.29099/ijair.v2i1.42
Kakkar, S., & Monga, V. (2017). A STUDY ON ARTIFICIAL INTELLIGENCE IN E-COMMERCE. International Journal of Advances in Engineering & Scientific Research, 4(4), 62-68. DOI: https://doi.org/10.1016/j.engappai.2017.03.010
Aswinvenu. (2019). Element14 Community. Retrieved 2023, from https://community.element14.com/products/roadtest/rv/roadtest_reviews/648/raspberry_pi_3_model_5
Secinaro, S., Calandra, D., Secinaro, A., Muthurangu, V., & Biancone, P. (2021). The role of artificial intelligence in healthcare: a structured literature review. BMC Medical Informatics and Decision Making, 21(1), 1-23. DOI: https://doi.org/10.1186/s12911-021-01488-9
Biswal, A. (2021). Top 10 Artificial Intelligence Applications. Simplilearn Journal on Artificial Intelligence, 1, 50-62.
Khensous, G., Labed, K., & Labed, Z. (2023). Exploring the evolution and applications of natural language processing in education. Revista Română de Informatică și Automatică, 33(2), 61-74. DOI: https://doi.org/10.33436/v33i2y202305
Pedro, R., & Garcia, F. (2015). Object recognition for a service robot. Master's thesis, University of Aveiro,, 1(1), 1-80.
Wei, L., Dragomir, A., Dumitru, E., Christian, S., Scott, R., Cheng-Yang, F., & Alexander, B. (2015). SSD: Single Shot MultiBox Detector. Journal for Computer Vision and Pattern Recognition, 5, 21-37.
David, C. C., Fiorella, S., & Marina, I. (2022). A Framework for Safe and Intuitive Human-Robot Interaction for Assistant Robotics. In Proceedings of the 2022 IEEE 27th International Conference on Emerging Technologies and Factory Automation, 1-4. DOI: https://doi.org/10.1109/ETFA52439.2022.9921569
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2016). Region-Based Convolutional Networks for Accurate Object Detection and Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(1), 142-158. DOI: https://doi.org/10.1109/TPAMI.2015.2437384
Anand, J., & Divyakant., M. (2020). A Comparative Study of Various Object Detection Algorithms and Performance Analysis. International Journal of Computer Sciences and Engineering, 8, 158-163.
Zientara, P. A., Lee, S., Smith, G. H., Brenner, R., Itti, L., Rosson, M. B. & Narayanan, V. (2017). Third Eye: A Shopping Assistant for the Visually Impaired. IEEE Computer Society Journal, 50(2), 16-24. DOI: https://doi.org/10.1109/MC.2017.36
Elmannai, W., & Elleithy, K. (2017). Sensor-Based Assistive Devices for Visually-Impaired People: Current Status. Sensors for Globalized Healthy Living and Wellbeing, 17(3), 565. DOI: https://doi.org/10.3390/s17030565
Chaudhari, G., Deshpande, A., & Liu, K. (2017). Smart Robotic Assistant for Visually Impaired. National Science Foundation, 12(7), 345-352.
Nguyen, H., Nguyen, M., Nguyen, Q., Yang, S., & Le, H. (2020). Web-based object detection and sound feedback system for visually impaired people. International Conference on Multimedia Analysis and Pattern Recognition, 1-6. DOI: https://doi.org/10.1109/MAPR49794.2020.9237770
Potdar, K., Pai, C. D., & Akolkar, S. (2018). A Convolutional Neural Network based Live Object Recognition System as Blind Aid. Computer Vision and Pattern Recognition, 1(1).
Bhandari, A., Prasad, P., Alsadoon, A., & Maag, A. (2019). Object detection and recognition: using deep learning to assist the visually impaired. Disability and Rehabilitation: Assistive Technology, 16(2), 1-9. DOI: https://doi.org/10.1080/17483107.2019.1673834
Parikh, N., Shah, I., & Vahora, S. (2018). Android Smartphone Based Visual Object Recognition for Visually Impaired Using Deep Learning. International Conference on Cryptography, Security and Privacy, 420-425. DOI: https://doi.org/10.1109/ICCSP.2018.8524493
Shirley, Rane, K., Rao, K. H., B, B. B., Agrawal, P., & Rawat, N. (2023). Machine learning and Sensor-Based Multi-Robot System with Voice Recognition for Assisting the Visually Impaired. Journal of Machine and Computing, 3(3), 206-215. DOI: https://doi.org/10.53759/7669/jmc202303019
Najm, H., Elferjani, K., & Alariyibi, A. (2022). Assisting Blind People Using Object Detection with Vocal Feedback. 2022 IEEE 2nd International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering (MI-STA), 48-52. DOI: https://doi.org/10.1109/MI-STA54861.2022.9837737
Adeyanju, I. A., Azeez, M. A., Bello, O. O., & Badmus, T. A. (2022). Development of a Convolutional Neural Network-Based Object Recognition System for Uncovered Gutters and Bollards. ABUAD Journal of Engineering Research and Development (AJERD), 5(1), 147-154.
Gautam, S., Sivaraman, K., Muralidharan, H., & Baskar, A. (2015). Vision System with Audio Feedback to Assist Visually Impaired to Grasp Objects. Procedia Computer Science, 58, 387-394. DOI: https://doi.org/10.1016/j.procs.2015.08.041
Breve, F. A., & Fischer, C. (2020). Visually Impaired Aid using Convolutional Neural Networks, Transfer Learning, and Particle Competition and Cooperation. IEEE International Joint Conference on Neural Network, 1-8. DOI: https://doi.org/10.1109/IJCNN48605.2020.9207606
Caballero, A., Catli, K. E., & Babierra, A. G. (2020). Object Recognition and Hearing Assistive Technology Mobile Application using Convolutional Neural Network. 2020 International Conference on Wireless Communication and Sensor Networks, 41-48. DOI: https://doi.org/10.1145/3411201.3411208
Lailla, B., Yandre, C., & Linnyer, A. (2018). Automata Classification with Convolutional Neural Networks for Use in Assistive Technologies for the Visually Impaired. 11th PErvasive Technologies Related to Assistive Environments Conference, 157-168. DOI: https://doi.org/10.1145/3197768.3201529
Shaikh, S., Karale, V., & Tawde, G. (2020). Assistive Object Recognition System for Visually Impaired. The International Journal of Engineering Research & Technology (IJERT), 9(9), 736-740. DOI: https://doi.org/10.17577/IJERTV9IS090382
Jessica, A., Veena, S. H., Srivarshini, S., Krishna, R. G., & Mounica, M. (2022). Convolutional Neural Networks based Voice Assistive System for Blind People. International Conference on Communication and Electronics Systems (ICCES), 7, 1608-1613. DOI: https://doi.org/10.1109/ICCES54183.2022.9835747
Kinra, A., Walia, W., & Sharanya, S. (2023). A Comprehensive and Systematic Review of Deep Learning Based Object Recognition Techniques for the Visually Impaired. 2023 2nd International Conference on Computational Systems and Communication (ICCSC), 2, 1-6. DOI: https://doi.org/10.1109/ICCSC56913.2023.10143007
Nikhil, A. S., Swarnalatha, & Lav, M. (2019). Guiding and Navigation for the Blind using Deep Convolutional Neural Network Based Predictive Object Tracking. International Journal of Engineering and Advanced Technology, 9(3), 306-313. DOI: https://doi.org/10.35940/ijeat.A1058.1291S319
Dahiya, D., Gupta, H., & Dutta, M. K. (2020). A Deep Learning based Real Time Assistive Framework for Visually Impaired. 2020 International Conference on Contemporary Computing and Applications (IC3A), 106-109. DOI: https://doi.org/10.1109/IC3A48958.2020.233280
Hsieh, Y.-Z., Lin, S.-S., & Xu, F.-X. (2020). Development of a wearable guide device based on convolutional neural network for blind or visually impaired persons. Multimedia Tools and Applications, 29473–29491. DOI: https://doi.org/10.1007/s11042-020-09464-7
Yi, C., Flores, R. W., Chincha, R., & Tian, Y. (2013). Finding Objects for Assisting Blind People. Netw Model Anal Health Inform Bioinform, 2(2), 71-79. DOI: https://doi.org/10.1007/s13721-013-0026-x
Neto, L. B., Grijalva, F., Maike, V. R., Martini, L. C., & Florencio, D. (2017). A Kinect-Based Wearable Face Recognition System to Aid Visually Impaired Users. IEEE, 47(1), 52-64. DOI: https://doi.org/10.1109/THMS.2016.2604367