Un sistema intuitivo de conversión de números mediante reconocimiento gestual para estudiantes con necesidades educativas especiales en Ciencias de la Computación
An intuitive number conversion system employing gestural recognition for students with special educational needs in Computer Science
Renjith P.R
Department of General Education - Higher Secondary, Kerala, India
https://orcid.org/0009-0001-8107-8744
Jeeva Jose
Department of Computer Applications, Baselios Paulose II Catholicos College, Piravom, Kerala, India
https://orcid.org/0000-0002-6919-1381
RESUMEN
La función primordial de un docente es lograr una participación activa y significativa de todos los alumnos en el proceso educativo. Cada clase está constituida por una diversidad de estudiantes que varían en inteligencia, comunicación, niveles de comprensión, habilidades para resolver problemas, aprendizaje y habilidades matemáticas. Es una necesidad urgente hacer que los estudiantes con discapacidades de aprendizaje o intelectuales puedan competir con los otros estudiantes para evitar una división social en el aula. El currículo de ciencias de la computación tiene muchos temas/áreas que plantean dificultades de aprendizaje para las personas con capacidades diferentes. Desafortunadamente, los problemas que enfrentan estos estudiantes no han sido abordados adecuadamente, lo que provoca que se queden atrás en el proceso de aprendizaje. El artículo propone un método de gestos manuales, implementado mediante inteligencia artificial (IA) para conversiones numéricas, que puede ser utilizado tanto por docentes como estudiantes con capacidades diferentes. La eficiencia de la herramienta de gestos manuales para interpretar gestos se probó con varios clasificadores, y se obtuvo una precisión media del 96,97 % en varios sistemas numéricos. El método basado en gestos manuales se evaluó entre los estudiantes y demostró ser eficiente en la retención y recuperación de conceptos. Los estudiantes mostraron una mayor mejora desde las puntuaciones del pre-test hasta el post-test en todas las categorías. Este método resultó muy útil para involucrar a los estudiantes, mejorar la comprensión, hacer el aprendizaje más agradable, con una claridad notable, efectividad y facilidad de uso, lo que lo convierte en una valiosa herramienta educativa.
PALABRAS CLAVE
Inteligencia Artificial; personas con discapacidades; reconocimiento de gestos manuales; aprendizaje inclusivo; conversión numérica.
ABSTRACT
Duty of a teacher is to effectively engage the different types of students in the classroom into the teaching-learning environment. Each class has a variety of students varying in intelligence, communication, understanding levels, problem solving skills, learning and mathematical abilities. It is the need of the hour to make the differently abled learners with learning or intellectual disabilities to compete with regular students to avoid a social divide in class. Computer science curriculum has many topics/areas that pose learning difficulty for the differently abled. The problems faced by such learners were not taken care of sufficiently, which leaves them behind in the learning process. The paper proposes a hand gesture method, implemented using AI for number conversions, which can be used by teachers and differently abled learners. The efficiency of the hand gesture tool to understand hand gestures is tested for various classifiers and an average accuracy of 96.97 % is obtained across various number systems. The hand gesture based method is evaluated among learners and proved efficient in retention and recovery of concepts. Learners showed greater improvement from pre-test to post-test scores across all categories. This method proved very helpful in engaging students, enhancing understanding, enjoyable learning, notable clarity, effectiveness, and ease of use, making it a valuable educational tool.
KEYWORDS
Artificial Intelligence; differently abled; hand gesture recognition; inclusive learning; number conversion.
1. INTRODUCTION
Differently abled learners in classrooms face many challenges and restrictions that hinder them from moving forward academically and socially. Learning becomes a tedious task for them and hence they restrain themselves from it. The teaching materials, teaching-learning process and classroom atmosphere are not sufficient to cater their needs. Computer science is a subject that needs mathematical abilities, which is tough for the differently abled learners with learning and intellectual disabilities. Moreover, there are a lot of factors like accessibility of learning materials, limited assisted learning technologies, and communication barriers, poor representation in learning materials, lack of inclusive pedagogical approaches in curriculum, resource constraints and shortage of special educators in each domain hinder their academic experience. It is highly essential to create an inclusive learning environment for them. Several reasons that highlight this importance are enhanced learning outcomes, improved social skills, address individual learning styles etc.
Number conversions in computer science spans across many areas including data representation and storage, architecture and assembly language, networking and communication protocols, cryptography and security, digital signal processing, embedded systems programming, algorithm design and optimization, digital logic and circuit design.
1.1 Background
Inclusive education is fundamental for fostering equitable learning environments where all students, regardless of their backgrounds or abilities, can access the same educational opportunities and support. By embracing diversity, inclusive classrooms cultivate a rich tapestry of perspectives and experiences, enhancing critical thinking, creativity, and problem-solving skills. Moreover, they prepare students for success in a globalized world by promoting cultural competence and social skills essential for academic and professional settings. Inclusive practices not only address individual needs but also challenge stereotypes, reduce stigma, and promote positive mental health. This study holds profound significance in addressing the educational barriers faced by differently-abled computer science learners with intellectual or learning disabilities. By developing a hand gesture-based interactive AI tool tailored to their needs, the research fosters more inclusive learning environments within computer science education. By exploring innovative instructional methods, the study not only advances approaches to teaching complex computer science concepts but also advocates for increased accessibility and fairness in education. Moreover, the insights gained have the potential to transcend disciplinary boundaries, offering valuable contributions to inclusive education practices across diverse academic domains.
1.2 Research Aim
The aim of this research is to develop a hand gesture-based game like method and an interactive AI tool specifically designed to facilitate the teaching and learning of number conversions for differently-abled computer science learners with intellectual disability (ID) and/or specific learning disabilities (SLD). The game like method can be used in classroom to educate them the concept of number conversions using had gestures. Once they grasped the method the tool can be introduced to them to practice and recheck whether they understood the concepts correctly. The tool provides high degree of interaction, better engagement, ease of use, more confidence, conceptual clarity and moreover a sense of inclusiveness among the differently abled learners.
1.3 Research Objectives
The research involves 2 main objectives –
Objective1: To assess the impact of incorporating hand gesture-based interactions in teaching number conversion concepts to computer science learners with ID and/or SLD, focusing on their comprehension and retention levels
Objective2: To evaluate the usability, accessibility, and adaptability of the tool for teaching number conversions to accommodate diverse learning styles and cater to the unique requirements of differently abled learners in computer science education.
1.4 Significance of the Study
This study holds significant importance in addressing the educational needs and barriers encountered by differently-abled computer science learners with intellectual or learning disabilities. Through the development of a hand gesture-based interactive AI tool designed specifically to facilitate the teaching and learning of number conversions, the research aims to foster more inclusive learning environments within higher secondary education. By delving into innovative instructional methods tailored to the unique requirements of this student demographic, the study not only advances approaches to teaching complex computer science concepts but also advocates for increased accessibility and fairness in education. Furthermore, the insights gleaned from this research have the potential to extend beyond the realm of computer science, offering valuable contributions to inclusive education practices across various academic disciplines. Ultimately, by enriching the educational experiences of differently-abled students in computer science, this study endeavors to equip them with the competencies and expertise needed to thrive in an ever-evolving digital society.
1.5 Scope and Limitations of the Research
The scope of the research is within the differently abled community with intellectual disability (ID) and/or specific learning disability (SLD) in computer science.
Ensuring accessibility for differently-abled learners posed a significant challenge, particularly in designing and implementing hand gesture-based learning materials that catered to diverse disabilities. Moreover, developing the hand gesture recognition system and integrating it with the learning platform required advanced technical expertise and resources, adding complexity to the project. Participant recruitment proved challenging as well, as it involved identifying and recruiting individuals with intellectual or learning disabilities, necessitating collaboration with specialized educational institutions and coordination with caregivers or guardians. Adhering to ethical guidelines and obtaining informed consent, especially from participants with disabilities, required careful planning and sensitivity. Providing adequate training and support for both participants and educators in using the hand gesture-based learning system posed logistical challenges, particularly in ensuring consistent implementation across different settings. Collecting and analyzing data from differently-abled learners presented unique challenges, including variability in response times, communication styles, and cognitive abilities, necessitating careful consideration in data interpretation. Nonetheless, the research yielded valuable insights into the effectiveness and feasibility of using hand gestures as a method of learning number conversions for differently-abled (ID and SLD) computer science learners.
2. Literature review
2.1 Effectiveness of Gestures in Learning
Assistive technology should be designed with the user in mind, like any product. This empowers people with disabilities to customize it for their needs, making technology more accessible and helpful in everyday life (Ladner, 2011).
Preschoolers learn counting better when they use gestures like pointing at objects. This helps them keep track of how many things they are counting and associate the number words with the correct amount (Alibali & DiRusso, 1999).. It is found that adding gestures to speech can improve memory, especially if you are told to use them. This is because gestures seem to work like actions to help us remember things better (Cook et. al., 2010).
The way teachers and students gesture while doing math suggests that math understanding is tied to our bodies, senses, and the physical world around us (Alibali & Natham, 2012). Researchers investigated whether combining physical interactions with realistic learning situations improves student motivation and performance. They found high levels of self-directed motivation and observed unique student interactions within a learning environment designed to be close to real-world scenarios (Lee et. al. 2012).
2.2 Technology Assisted Gesture Based Learning
A new study in Seville University higher education centers is looking at how university leaders view their professors’ knowledge of using technology to help students with disabilities. Early signs show these leaders are concerned about a lack of awareness, limited time for professors to learn, and inadequate training in these helpful technologies (Román-Graván & Fernández-Cerero, 2022). Simulations in tech class can be a powerful tool for teaching and learning. They help students understand real-world processes by recreating them, making learning more engaging and interesting. This approach, which combines problem-solving and simulations, encourages students to explore science and technology through hands-on activities, ultimately deepening their understanding and sparking their curiosity (Micó-Amigo and Bernal Bravo, 2020). Animations have been shown to help people remember and perform human movements. This study wondered if the same could be true for non-human movements. Instead of having people physically copy the movements, they watched an animation with a hand mimicking them. Interestingly, this method led to better memory and performance, suggesting it could be a useful tool for learning about all kinds of movement, not just human (Koning et. al., 2013).
The efficacy of a gesture recognition Kinect-based task prompting system for training individuals with cognitive impairments, employing an ABAB sequence and observing significant improvements in vocational job skills during intervention phases, with implications discussed (Chang et. al., 2011). A primary education physics course investigated the effects of touch and gestural interaction with tablets and robots on learning about friction (Merkouris et. al., 2019). The study found that physical conditions significantly enhanced learning outcomes compared to virtual methods, particularly for students with misconceptions or limited initial knowledge. Additionally, students familiar with touch screen interfaces achieved similar learning gains and reported higher usability compared to those using hand-tilt interfaces. To enhance human-computer interactions for the speech-impaired community, a hand gesture recognition system has been developed. This system utilizes color space segmentation combined with a Convolutional Neural Network (CNN) model, resulting in approximately 10 percent higher accuracy compared to systems that do not employ image segmentation (Chang et. al., 2023). Investigations on embodied interactions and learning within a computer-mediated immersive multimedia environment allowed learners to engage with learning interventions using body movements and gestures (Johnson-Glenberg et al., 2014).
In an experimental study by Xu and Ke (2021), they investigated the effectiveness of Unity3D and Kinect V2 in facilitating embodied interactions for teaching binary and decimal number conversions to novice adult learners. While the experimental group utilized body movements for learning, the control group relied on conventional mouse and keyboard interactions. Results indicated that although embodied interactions improved conceptual learning, they did not significantly enhance knowledge retention and application compared to traditional methods. This underscores the importance of carefully considering implementation and intensity when designing learning systems with novel technologies. An article (Moral-Sánchez et. al., 2023) proposes implementing chat bots powered by artificial intelligence (AI) in the didactics of geometry subject for Primary Education degree students at the University of Málaga, involving 120 students in two groups. The aim is to analyze these AI-driven chat bots and evaluate student satisfaction, showcasing the enhancement of digital competence and noting high satisfaction levels with their AI creations, thereby suggesting the potential transferability of this experience to other educational contexts.
ATCLD (Assistive Technology for Children with Learning Disabilities), a computer software program developed for upper primary education learners at Eros Girls School (EGS) in Windhoek, Namibia, improves academic performance for learners with dyslexia and other learning deficiencies (Hashiyana et. al.,2022). By addressing issues related to English language and mathematics, ATCLD enhances learners’ capabilities through repetition and is planned for implementation in various needy schools across Namibia.
2.3 Number Systems and Number Conversions
2.3.1 Number Systems
Number systems in computer science are fundamental to representing and manipulating data. The most used number systems include:
Binary (base-2): Binary is the foundation of computer systems, representing data using only two symbols, 0 and 1. Each digit in a binary number represents a power of 2, making it ideal for digital computation and storage.
Decimal (base-10): Decimal is the number system humans are most familiar with, using 10 digits from 0 to 9. In computer science, decimal numbers are often used for human-readable output and input, but conversions are required for internal processing.
Octal (base-8): Octal uses eight symbols from 0 to 7. It was more commonly used in older computer systems for its ease of conversion to binary, as each octal digit corresponds to three binary digits.
Hexadecimal (base-16): Hexadecimal uses sixteen symbols, consisting of digits 0-9 and letters A-F, where A to F represents 10 to 15 respectively. Hexadecimal is widely used in computer science for representing binary data more compactly and conveniently, as each hexadecimal digit corresponds to four binary digits.
These number systems are crucial for various operations in computer science, including arithmetic calculations, data representation, and programming. Converting between different number systems is essential for understanding and working with data at the hardware and software levels in computer systems.
2.3.2 Number Conversions
Traditional teaching methods for number conversions (we focus on whole number conversions only) often involve step-by-step procedures and examples to help students understand the process. Here are the approaches used in higher secondary classroom for number conversions from one to another.
2.3.2.1 Binary to Decimal Conversion
Converting from binary to decimal involves multiplying each binary digit by its corresponding power of 2 and summing the results. Here’s how to do it in steps:
Step1. Write down the binary number.
Step2. Starting from the rightmost digit, assign each digit a power of 2, increasing by 1 as you move from right to left.
Step3. Multiply each binary digit by its corresponding power of 2.
Step4. Sum the results to get the decimal equivalent.
2.3.2.2 Decimal to Binary Conversion
Converting from decimal to binary involves repeatedly dividing the decimal number by 2 and keeping track of the remainders. Stepwise method to do the same is listed below:
Step1. Divide the decimal number by 2.
Step2. Record the remainder (either 0 or 1).
Step3. Repeat the process with the quotient obtained in the previous step until the quotient is 0.
Step4. Write down the remainders in reverse order to get the binary equivalent.
2.3.2.3 Octal to Binary Conversion
To convert from octal to binary, you can convert each octal digit separately into its 3-bit binary representation. Follow the steps to perform this conversion:
Step1. Write down the octal number.
Step2. Replace each octal digit with its 3-bit binary equivalent.
Step3. Combine the binary representations of each octal digit to get the final binary number.
2.3.2.4 Binary to Octal Conversion
Converting from binary to octal involves grouping binary digits into sets of three, starting from the rightmost digit, and then converting each group into its octal equivalent.
Step1. If the number of binary digits is not a multiple of three, add leading zeros to the left to make it a multiple of three.
Step2. Group the binary digits into sets of three, starting from the right.
Step3. Convert each group of three binary digits into its octal equivalent.
Step4. Write down the octal digits to get the final octal number.
2.3.2.5 Hexadecimal to Binary Conversion
To convert from hexadecimal to binary, you can convert each hexadecimal digit separately into its 4-bit binary representation.
Step1. Write down the hexadecimal number.
Step2. Replace each hexadecimal digit with its 4-bit binary equivalent.
Step3. Combine the binary representations of each hexadecimal digit to get the final binary number.
2.3.2.6 Binary to Hexadecimal Conversion
Converting from binary to hexadecimal involves grouping binary digits into sets of four, starting from the rightmost digit, and then converting each group into its hexadecimal equivalent.
Step1. If the number of binary digits is not a multiple of four, add leading zeros to the left to make it a multiple of four.
Step2. Group the binary digits into sets of four, starting from the right.
Step3. Convert each group of four binary digits into its hexadecimal equivalent.
Step4. Write down the hexadecimal digits to get the final hexadecimal number.
Previous research indicates that embodied interactions can yield positive outcomes in learning activities. Body movements play a crucial role in enhancing cognitive processes, reinforcing content in working memory, and aiding in the encoding of perception and observation. Early investigations into learning STEM-related topics with body movements primarily focused on face-to-face classroom settings and were predominantly geared towards child learners. Recent studies have increasingly utilized computer-mediated environments and emerging body tracking technology to explore the relationship between body movements and learning across various. However, a common limitation among these studies is that the content knowledge primarily focuses on concept retention and understanding.
In this research, our goal is to delve deeper into the efficacy of embodied interactions, particularly focusing on gestures, within the context of informal computer science education of differently abled learners at the higher secondary level. Our aim is to explore how learners not only grasp concepts and principles but also apply them to perform calculations, specifically in the number conversions involving decimal, binary, octal and hexadecimal systems.
Arithmetic operations indeed play a crucial role in number conversions, especially when transitioning between various number bases. Converting from one number system to another involves different arithmetic operations like addition, subtraction, multiplication, division, and exponentiation. However, for differently-abled learners in computer science, this process can pose challenges as it requires a high level of concentration and understanding of the underlying concepts. The complexities of performing arithmetic operations in different number bases may make the learning experience tougher and more demanding for these students. Therefore, tailored instructional approaches and supportive learning environments are essential to assist differently-abled learners in mastering arithmetic operations and number conversions effectively.
3. METHODS
We focused on examining whether hand gestures method could lead to greater knowledge acquisition and application in math calculations about numeric systems, when compared to traditional methods, in a virtual learning environment for differently abled computer science learners in higher secondary.
The research aims to assess the effectiveness of using hand gestures in teaching number conversions to differently-abled computer science learners by reviewing existing literature, identifying gaps, and defining research questions. Clear research questions are developed to address the impact of hand gestures on teaching number conversions, their effects on differently-abled learners, and potential challenges and benefits. Participants are selected based on characteristics such as age, computer science knowledge, and disabilities, with informed consent obtained. An experimental intervention is designed, incorporating hand gestures into a learning program tailored to the needs of differently-abled learners, while a control group receives standard instruction. Participants receive instruction on foundational concepts of number conversions, such as comprehending the base of a number system and the significance of digits within each system. This knowledge forms the basis for conducting diverse arithmetic operations and executing conversions between different number bases. By mastering these fundamentals, participants develop a robust understanding of number representation and manipulation across various systems, providing a strong foundation for advancing their exploration and proficiency in number conversion techniques.
3.1 Hand Gesture Method
This method employs hand gestures to facilitate 6 conversions between decimal, binary, octal and hexadecimal numbers directly and 2 conversions indirectly (octal to hexadecimal and vice versa). Using the left hand with the palm facing towards you, each finger from the little finger to the thumb represents a bit. For decimal to binary conversion, the fingers represent the bits of a binary number as shown in Figure 1. Conversely, for binary to decimal conversion, the fingers signify weight in ascending powers of 2 as shown in Figure 2. For octal to binary and binary to octal conversions, only 3 fingers are needed because the largest octal digit 7 can be represented in binary using 3 bits. For hexadecimal to binary and binary to hexadecimal conversions, only 4 fingers are needed because the largest hexadecimal digit F(15) can be represented in binary using 4 bits.
Figure 1. Bit values associated with fingers (raised finger denotes 1 and closed finger denotes 1)
Figure 2. Weights associated with fingers (each finger from right to left has a weight equal to the ascending powers of 2)
3.1.1 Binary to Decimal Conversion:
Step 1 (Hand Positioning). Extend your left hand with the palm facing towards you.
Step 2 (Finger Representation). Each finger represents a bit, starting from the little finger (least significant bit) to the thumb (most significant bit).
Step 3 (Weight Assignment) Assign weights to the fingers corresponding to ascending powers of 2, from 1 for the little finger to 16 for the thumb.
Step 4 (Calculate sum from Raised Fingers). Count the number of raised fingers and sum up their corresponding weights.
Step 5 (Converted value). The total weight of the raised fingers represents the decimal equivalent of the binary number.
3.1.2 Decimal to Binary Conversion:
Step 1 (Hand Positioning). Extend your left hand with the palm facing towards you.
Step 2 (Assign Decimal Value). Choose a decimal number below 32 to convert to binary.
Step 3 (Representing Decimal with Fingers). Use the fingers to represent the bits of the chosen decimal number, with the thumb representing the largest bit (16) and the little finger representing the smallest bit (1).
Step 4 (Raising Relevant Fingers). Raise the fingers corresponding to the bits required to represent the chosen decimal number in binary.
Step 5 (Converted value). The raised fingers signify the binary equivalent of the chosen decimal number.
3.1.3 Octal to Binary Conversion
Step 0 (Preparation). Utilize 3 fingers for the conversion since the largest octal digit 7 can be represented using 3 fingers.
Step 1 (Conversion) Convert Each Octal Digit to Binary Represent each octal digit as a sum of values using the 3 fingers.
Step 2 (Binary Representation). Raised finger indicates ‘1’. Closed finger indicates ‘0’.
3.1.4 Binary to Octal Conversion
Step 1 (Grouping Bits). Group 3 bits from right to left in the binary number. Add necessary zeroes if the last group to the left does not have 3 bits.
Step 2 (Representation). Represent Each Digit Using 3 Fingers. Perform the operation for all groups (Utilize 3 fingers to represent each octal digit). Raised finger indicates ‘1’ and closed finger indicates ‘0’.
Step 3 (Octal Generation). Sum up the numbers represented by raised fingers in each group.
3.1.5 Hexadecimal to Binary Conversion
Step 0 (Preparation). Utilize 4 fingers for the conversion since the largest hexadecimal digit (F) can be represented using 4 fingers.
Step 1 (Conversion). Convert Each Hexadecimal Digit to Binary. Represent each hexadecimal digit as a sum of values using the 4 fingers.
Step 2 (Binary Representation). Raised finger indicates ‘1’. Closed finger indicates ‘0’.
3.1.6 Binary to Hexadecimal Conversion
Step 1 (Grouping Bits). Group 4 bits from right to left in the binary number. Add necessary zeroes if the last group to the left does not have 4 bits.
Step 2 (Representation). Represent Each Digit Using 4 Fingers. Perform the operation for all groups (Utilize 4 fingers to represent each hexadecimal digit). Raised finger indicates ‘1’ and closed finger indicates ‘0’.
Step 3 (Hexadecimal Generation). Sum up the numbers represented by raised fingers in each group.
3.2 A Hand Gesture Based Interactive AI Tool
The method described above is implemented by using AI technology, utilizing a computer’s webcam to capture images for training purposes. These images serve as training data to teach the machine to recognize various gestures. By reproducing these gestures in front of the camera, users can obtain converted values corresponding to the gestures made. This innovative tool serves a dual purpose: it can be employed by special educators to facilitate the teaching of number conversion, while also catering to differently abled students, enabling them to independently grasp and comprehend fundamental concepts. In essence, this technology not only aids in educational settings but also promotes inclusivity by empowering individuals with diverse learning needs to engage with and understand numerical concepts more effectively. The working of the tool can be explained in 4 steps as shown in Figure 3.
Figure 3. Steps involved in Hand Gesture Based AI Tool.
Step 1 involved utilizing the webcam of a laptop to gather image data for various hand gestures representing different number systems. By moving hands back and forth in front of the camera, ‘n’ images were captured for each gesture to ensure variability and improve accuracy. The captured images were then organized into folders corresponding to specific number systems. For instance, considering the scenario of representing numbers in different systems such as binary, octal, decimal, and hexadecimal, we created folders labeled with the respective representations (e.g., “Binary”, “Octal”, “Decimal”, “Hexadecimal”). Within each system folder, sub folders were created to represent individual numbers within that system (e.g., “0” to “7” for octal, “0” to “9” for decimal, and “0” to “F” for hexadecimal). For a 4-bit binary system, there would be 16 folders labeled with the corresponding binary numbers from 0000 to 1111. Similarly, for other systems, appropriate folders were created to cover the range of possible values. Each folder contained a set number of images captured for each respective number representation. This meticulous organization ensured the creation of a diverse and comprehensive data set, laying a solid foundation for subsequent phases of the project aimed at training the machine learning model to recognize and interpret hand gestures representing various number systems.
In Step 2, our primary objective revolved around extracting intricate spatial patterns and geometrical attributes from hand and finger movements to construct a dataset tailored for gesture recognition. We narrowed our focus to the left hand and its four fingers, aligning with the context of representing a 4-bit binary number. To achieve this, we delved into a thorough analysis of the spatial configuration and geometric properties exhibited by the hand and its constituent fingers during various gestures. By scrutinizing these aspects, we made a dataset meticulously crafted to encapsulate the fundamental characteristics pivotal for precise gesture recognition. This dataset not only encompassed raw image data but also incorporated rich metadata detailing spatial coordinates, contour shapes, finger positions, and other relevant geometrical attributes. Each entry in the dataset thus encapsulated a comprehensive representation of the hand gesture, encapsulating both visual and spatial information. Following processing, the dataset was serialized and stored as a pickle file. This serialization method ensured seamless accessibility and efficient utilization of the dataset in subsequent steps of the project. By employing this approach, we laid a robust foundation for training our AI model, facilitating its ability to discern and interpret hand gestures accurately, thereby advancing the capabilities of our gesture recognition system.
In Step 3, we leveraged the dataset from previous step to train a classifier tailored for gesture recognition. This involved the input of the spatial patterns and geometrical features extracted from hand gestures into the classifier, thereby enabling it to discern and classify different gestures accurately. During the training process, the classifier assimilated the intricacies of the dataset, learning to distinguish between various hand gestures based on the extracted features. Through iterative refinement, the classifier honed its ability to accurately categorize gestures, ultimately culminating in the creation of a robust model for gesture recognition. Upon completion of the training phase, rigorous evaluation of the classifier ensued. Remarkably, the evaluation revealed that the classifier (Random Forest) achieved an astounding accuracy score of 100 %. This exceptional performance underscored the classifier’s proficiency in correctly classifying all samples within the dataset, reaffirming its effectiveness in identifying hand gestures with unparalleled precision. Furthermore, the trained classifier bestowed the capability to extrapolate its learned knowledge to new input data, thus enabling real-time gesture recognition applications. This pivotal functionality opens avenues for a myriad of practical applications, ranging from interactive user interfaces to assistive technologies catering to individuals with diverse needs. In essence, the successful culmination of Step 3 signifies a significant milestone, showcasing the efficacy of the random forest classifier in accurately discerning and categorizing hand gestures based on the extracted spatial patterns and geometrical features.
In Step 4, the trained model underwent rigorous testing, where sample hand gesture inputs were provided to evaluate its performance. Through this testing phase, the model adeptly recognized and classified the presented hand gestures, thereby yielding the correct binary value corresponding to each gesture. However, it’s imperative to highlight that the accuracy of the classification results hinges significantly upon the quality and correctness of the images provided during Step 1. The robustness and effectiveness of the model during testing are inherently linked to the integrity of the dataset used for training. High-quality images with accurate labeling not only facilitate better training but also contribute to enhanced performance during testing. Therefore, to ensure the reliability and accuracy of the classification results, it is paramount to maintain consistency and precision in the processes of collecting and labeling image data right from the inception of the project. This entails adhering to stringent standards for image quality, ensuring uniformity in gesture representations, and meticulously annotating each image with the correct label. By upholding these standards throughout the project lifecycle, we can bolster the model’s ability to generalize well to unseen data and mitigate the risk of erroneous classifications. This, in turn, fortifies the credibility and utility of the gesture recognition system, paving the way for its seamless integration into various applications and domains.
The implementation of the gesture recognition model was conducted as a project using Python and its associated libraries. This model was demonstrated in a classroom setting, allowing instructors to effectively teach gesture recognition concepts. Additionally, students were able to use their laptop webcams to reinforce their understanding by practicing gestures in front of the camera. The system processes these gestures in real-time and displays the corresponding numerical values, providing immediate feedback to facilitate learning and practice.
4. RESULTS
4.1 Analysis of different classifiers for the Hand Gesture Tool
We have trained the model with different classifiers other than Random Forest Classifier such as Gradient Boosting Classifier, Ada Boost Classifier, Extra Trees Classifier , XGB Classifier, Cat Boost Classifier, KNN, SVM, LGBM and CNN. We have done this for all 4 number systems namely Decimal, Hexadecimal, Binary and Octal and the accuracy is observed as shown in Table 1.
Table 1. Accuracy of classifiers for different number systems (data set size 100).
|
Classifier |
Octal |
Hexadecimal |
Decimal |
Binary |
|
RandomForestClassifier |
100 |
98.2 |
100 |
93.2 |
|
GradientBoostingClassifier |
99.15 |
96.35 |
99.05 |
91.35 |
|
AdaBoostClassifier |
25 |
28 |
25.07 |
25 |
|
ExtraTreesClassifier |
100 |
99.5 |
100 |
95.5 |
|
XGBClassifier |
99.57 |
94.2 |
99.37 |
94.2 |
|
CatBoostClassifier |
99.15 |
98.32 |
99.68 |
94.32 |
|
KNN |
97.88 |
91.12 |
98 |
90.12 |
|
SVM |
98.94 |
98.23 |
95.67 |
94.23 |
Figure 4. Accuracy of classifiers for different number systems.
4.2 Discussion
The study compared the effectiveness of the Hand Gesture Method with the Conventional Method across various parameters among 1536 participants. Participants are divided into two groups – intervention group, which used the hand gesture method and control group, which used the conventional method. In the intervention group, which utilized the Hand Gesture Method, there were 487 male and 355 female participants, with an average age of 16.3 years.
Additionally, the majority of participants in this group were pursuing Computer Science (502) and Computer Application – Commerce (340) streams. Contrastingly, in the control group employing the Conventional Method, there were 389 male and 305 female participants, with a slightly higher average age of 16.5 years. Similarly, most participants in this group were also in their first year of education, with 438 pursuing Computer Science and 256 studying Computer Application (Commerce). Regarding disability types, the intervention group had 498 participants with Intellectual Disability and 356 with Specific Learning Disability. In comparison, the control group consisted of 391 participants with Intellectual Disability and 303 with Specific Learning Disability.
Basics of number systems and problems involving number conversions were taught to both groups before they underwent a pre-test. The pre-test has 10 standardized questions to check the conceptual clarity and problem solving abilities of participants. Scores were recorded based on their conceptual understanding and problem-solving abilities. Followed by an instruction about the hand gesture method, the intervention group underwent a post-test, while the control group, having revised the conventional method, completed their post-test within the subsequent week. The study revealed that implementing the hand gestures method for number conversions proved remarkably effective among learners with cognitive and sensory disabilities when compared to conventional methods. Employing hand gestures as a teaching technique for number conversions among differently abled computer science learners significantly enhanced conceptual learning. Moreover, the intervention group displayed notably higher retention rates of the learning method introduced during the intervention period compared to the control group. This suggests that hand gestures effectively promoted knowledge retention and application as shown in Table 2.
Table 2. Mean scores of intervention and control groups for pre-test and post-test.
|
Mean score |
Intervention Group |
Control Group |
||
|
Pre-test |
Post-test |
Pre-test |
Post-test |
|
|
All 10 questions (10 marks) |
4.3 |
8.9 |
5.1 |
6.5 |
|
Concept questions (5 marks) |
1.2 |
4.8 |
1.5 |
2.3 |
|
Problem questions (5 marks) |
3.5 |
5.0 |
2.3 |
3.5 |
The feedback is received from participants regarding the hand gesture method was overwhelmingly positive across various parameters like engagement, understanding, enjoyment, effectiveness, clarity, comfort, confidence, interest and ease of use. The feedback as shown in Table 3 indicates that the hand gesture method was highly effective in engaging students, enhancing their understanding, and making the learning experience enjoyable. Its clarity, effectiveness, and ease of use were particularly pointed, suggesting that it could be a valuable tool for educational purposes.
Table 3. Participants’ feedback on the AI tool
|
Feedback parameter |
Number of responses |
Positive response (%) |
|
1.Engagement |
1516 |
80 |
|
2.Understanding |
1516 |
76 |
|
3.Enjoyment |
1438 |
75 |
|
4.Effectiveness |
1516 |
89 |
|
5.Clarity |
1516 |
93 |
|
6.Comfort |
1438 |
81 |
|
7.Confidence |
1438 |
84 |
|
8.Interest |
1516 |
83 |
|
9.Ease of Use |
1516 |
96 |
|
10.Overall |
1516 |
91 |
The research opens doors to various opportunities for differently abled learners in learning concepts in computer science. The method can be tailored to fit different types and degrees of disabilities. These methods can be further improved to different learner profiles, thereby optimizing the learning and engagement of learner to the fullest. The retention and recalling capabilities of learners with LD or SLD for long term can also be tested to check whether the method is effective over time.
Sufficient training and support is needed to special educators to implement this effectively to reach the community. Proper inclusions need to be done in the curriculum to redesign the learning materials that support such systems. Methods like this can also be implemented in different areas of computer science and across other disciplines that directly or indirectly adapt technology that make use of AI.
4.3 Challenges
The scope of the research is within the differently abled community with intellectual disability (ID) and/or specific learning disability (SLD) in computer science stream. First, ensuring accessibility for learners with disabilities, particularly in designing hand gesture-based learning materials, was complex due to the diverse range of disabilities to consider. Second, developing and integrating the hand gesture recognition system into the learning platform required advanced technical skills and resources. Third, recruiting participants with intellectual or learning disabilities meeting study criteria was challenging and involved collaboration with specialized educational institutions. Fourth, ethical considerations were paramount, especially in obtaining informed consent and respecting the rights and privacy of participants, particularly those with disabilities. Fifth, providing adequate training and support for participants and special educators in using the hand gesture-based learning system posed logistical challenges. Sixth, collecting and analyzing data from learners with disabilities presented unique challenges due to variations in response times, communication styles, and cognitive abilities. Finally, time and resource constraints impacted the research’s scope and scale, affecting factors such as sample size and data collection breadth. Despite these challenges, the research provided valuable insights into using hand gestures for learning number conversions for learners with intellectual and specific learning disabilities in computer science. Overcoming these hurdles required interdisciplinary collaboration, innovative problem-solving, and a strong commitment to inclusive research practices.
5. CONCLUSION
The study revealed that employing hand gestures as a teaching method for number conversions significantly enhanced conceptual learning among differently-abled computer science learners. Participants exhibited heightened engagement and enthusiasm when utilizing hand gestures, resulting in a more positive response to this instructional approach. This multisensory learning experience facilitated the integration of physical movement with abstract concepts, leading to a deeper understanding of the material. Moreover, the intervention group demonstrated notably higher retention rates compared to the control group, indicating the effectiveness of hand gestures in fostering knowledge retention and application. However, challenges such as implementation and intensity were identified, highlighting the importance of careful design and planning. The findings suggest that the hand gestures method and its implementation using AI as an educational tool hold promise as an effective and engaging instructional strategy for differently-abled computer science learners, potentially yielding improved learning outcomes and retention.
FUNDING
This research is funded by Academic Special Cell, Directorate of General Education -Higher Secondary under Government of Kerala, India. This work comes under Enhancement of Academic Program Mini Research Project for Higher Secondary School Teachers of Kerala, India as per order no. HSE/6904/2023 AcdSpl2 dated 3.11.2023.
REFERENCE
, & (1999). The function of gesture in learning to count: More than keeping track. Cognitive Development, 14(1), 37–56. https://doi.org/10.1016/S0885-2014(99)80017-3
, & (2012). Embodiment in mathematics teaching and learning: Evidence from learners’ and teachers’ gestures. Journal of Learning Science, 21(2), 247–286. https://doi.org/10.1080/10508406.2011.611446
, , , . An Exploration into Human-Computer Interaction: Hand Gesture Recognition Management in a Challenging Environment. SN Comput Sci. 2023, 4(5), 441. https://doi.org/10.1007/s42979-023-01751-y
, , & (2011). A gesture recognition system to transition autonomously through vocational tasks for individuals with cognitive impairments. Research in Developmental Disabilities, 32(6), 2064–2068. https://doi.org/10.1016/j.ridd.2011.08.010
, , & (2010). Gesturing makes memories that last. Journal of Memory and Language, 63(4), 465–475. https://doi.org/10.1016/j.jml.2010.07.002
, & (2013). Gestures in instructional animations: A helping hand to understanding non-human movements? Applied Cognitive Psychology, 27(5), 683–689. https://doi.org/10.1002/acp.2937
, , , & (2022). Assistive Technology for Children with Learning Disabilities as an Artificial Intelligence Application. In Assistive Technologies for Differently Abled Students, 1st ed. (113-125). IGI Global. https://doi.org/10.4018/978-1-7998-4736-6.ch007
, & (2017). Embodied science and mixed reality: How gesture and motion capture affect physics education. Cognitive Research: Principles and Implications, 2(1), 24. https://doi.org/10.1186/s41235-017-0060-9
(2011). Accessible Technology and Models of Disability. In: Oishi, M., Mitchell, I., Van der Loos, H. (eds) Design and Use of Assistive Technology. Springer, New York, NY. https://doi.org/10.1007/978-1-4419-7031-2_3
, , , , & (2012). The effects of using embodied interactions to improve learning performance. Advanced Learning Technologies (ICALT), 2012 IEEE 12th International Conference, 4–6 July 2012, 557–559. https://doi.org/10.1109/ICALT.2012.104
, , , & (2019). Understanding the Notion of Friction Through Gestural Interaction with a Remotely Controlled Robot. Journal of Science Education and Technology, 28(3), 209–221. https://doi.org/10.1007/s10956-018-9760-2
, & (2020). Evaluative research on teaching innovation with simulators in the area of Technology in Compulsory Secondary Education. IJERI: International Journal of Educational Research and Innovation, (14), 134–146. https://doi.org/10.46661/ijeri.4855.
, , & (2023). Analysis of artificial intelligence chatbots and satisfaction for learning in mathematics education. IJERI: International Journal of Educational Research and Innovation, (20), 1–14. https://doi.org/10.46661/ijeri.8196.
, & (2022). Training of university teachers in ICT and disability. The case of the University of Seville. IJERI: International Journal of Educational Research and Innovation, (17), 135–151. https://doi.org/10.46661/ijeri.6168.
, & (2021). Learning Number Conversions through Embodied Interactions, Springer Journal of Technology, Knowledge and Learning 28(1), 1-5. https://doi.org/10.1007/s10758-021-09557-8