The use of computer vision technologies for an objective assessment of indicators of concentration of attention in newborns and infants with visual stimulation for the purpose of developmental care

Abstract

Assessment of visual function in the first few days after birth is mainly limited to the study of eye movements, the ability to fix a gaze and follow an object. In order to determine how the child's gaze moves when examining an object, how long it is fixed on the object, techniques are needed that allow recording the movements of the eyeballs and determining the trajectory of the gaze.

The aim of the study was to develop a method based on machine learning technology and computer vision for automated analysis of eye movement and fixation of the gaze of a newborn and an infant during visual stimulation.

Material and methods. The proposed method includes video filming of newborns and children in the first year of life in a hospital and outpatient. Video recordings from mobile phone cameras with a length of 15 seconds to 3 minutes were used as the initial data. 73 video files were selected from 150 videos of newborns, of which 61 recordings of sufficient quality were selected, in which the child was recognized in at least 30% of frames. For each of the 61 recordings, a neonatal visual stimulation track was recorded. Facial recognition was implemented using a widely used pre-trained model based on machine learning and ultra-precise networks. The eye movement research algorithm includes face search, determination, head position by the location of the eyes, nose, lips, eye area search, pupil search, relative gaze determination and absolute gaze direction determination, and blink tracking.

Results and discussion. A neural network was developed and trained for recognizing facial images of newborns and infants and for locating the eyes on a child's face. The method made it possible to receive data on the direction of the child's gaze in real time using the camera of an ordinary smartphone or a simple web camera. Depending on the size of the displayed image and the distance to it, the system calculates the total time of concentration on the image, and also detects moments when the child is not interested in the image.

Conclusion. The proposed method can be used to analyze the effectiveness of early visual stimulation in children, in the context of long-term effects on psychomotor and cognitive development, as well as to objectify data from various programs for early assessment of visual function in newborns and infants.

Keywords:newborn, preterm, prematurity, visual system, vision, neurosensitive development, psychoneurological development, developmental care, newborn nursing

Funding. The study was carried out as part of the research work on the topic of the state assignment "Improving approaches to intensive care, feeding and nursing premature babies with perinatal pathology" (head of the state assignment topic, deputy director for scientific work - Dmitriy N. Degtyarev, MD, PhD, DSc, Professor).

Conflict of interest. The authors declare no conflicts of interest.

Additional information. A certificate of state registration of a computer program was obtained for the described computer program No. 2020662875 "Software implementation of the newborn gaze direction tracking algorithm”. Copyright holder: Kulakov Obstetrics, Gynecology and Perinatology National Medical Research Center of Ministry of Healthcare of the Russian Federation.

Contribution. General supervision, writing the manuscript - Ryumina I.I.; writing and formatting the manuscript, data collection -Kukhartseva M.V.; scientific editing of the manuscript - Narogan M.V.; scientific consulting, software development management -Borovikov P.I.; scientific consulting on IT technologies - Lagutin V.V.; contributing to the concept and design of the study -Whiteley I.

For citation: Ryumina I.I., Kukhartseva M.V., Narogan M.V., Borovikov P.I., Lagutin V.V., Whiteley I. The use of computer vision technologies for an objective assessment of indicators of concentration of attention in newborns and infants with visual stimulation for the purpose of developmental care. Neonatologiya: novosti, mneniya, obuchenie [Neonatology: News, Opinions, Training]. 2021; 9 (1): 30-41. DOI: https://doi.org/10.33029/2308-2402-2021-9-1-30-41 (in Russian)

References

1. Valenza E., et al. Face preference at birth. J Exp Psychol Hum Percept Perform. 1996; 22 (4): 892-903.

2. Dannemiller J., Stephens B. A critical test of infant pattern preference models. Child Dev. 1988; 2 (1): 210-6.

3. Hunnius S., et al. Effects of preterm experience on the developing visual system: a longitudinal study of shifts of attention and gaze in early infancy. Dev Neuropsychol. 2008; 33 (4): 521-35.

4. Kunnikova K.I., Kotyusov A.I., Valieva E.R. Methods for assessing visual attention in premature infants in social and non-social contexts. In: Scientific forum: Pedagogy and psychology: a collection of articles based on the materials of the IV International Scientific and Practical Conference. Moscow; 2017; 119-25. (in Russian)

5. Imafuku M., et al. Preference for dynamic human images and gazefollowing abilities in preterm infants at 6 and 12 months of age: an eyetracking study. Infancy. 2017; 22 (2): 223-39.

6. Whiteley I., Whiteley G., Fisher R. Earth Designs: Black and White Book for a Newborn Baby and the Whole Family: Volume 1. Bath: Cosmic Baby Books, 2016: 36 p.

7. Whiteley I., Whiteley G., Fisher R. Earth Designs: Black and White Book for a Newborn Baby and the Whole Family: Volume 1. Bath: Cosmic Baby Books, 2016: 36 p.

8. Imafuku M., et al. Preference for dynamic human images and gazefollowing abilities in preterm infants at 6 and 12 months of age: an eyetracking study. Infancy. 2017; 22 (2): 223-39.

9. Kushsairy K., et al. A comparative study between LBP and Haar-like features for Face Detection using OpenCV. In: 4th International Conference on Engineering Technology and Technopreneuship (ICE2T). 2014.

10. Morimoto C.H., Mimica M.R.M. Eye gaze tracking techniques for interactive applications. Comput Vis Image Underst. 2005; 98 (1): 4-24.

11. Hansen D.W., Ji Q. In the eye of the beholder: a survey of models for eyes and gaze. IEEE Trans Pattern Anal Mach Intell. 2010; 32 (3): 478-500.

12. Rowley H.A., Baluja S., Kanade T. Neural network-based face detection. IEEE Trans Pattern Anal Mach Intell. 1998; 20 (1): 23-38.

13. Viola P, Jones M. Managing work role performance: challenges for twenty-first century organizations and their employees. In: Rapid Object Detection using a Boosted Cascade of Simple Features. 2001: 511-8.

14. Blanz V., Vetter T. Face recognition based on fitting a 3D morphable model (initial idea for SfT + NRSfM combination). IEEE Trans Pattern Anal Mach Intell. 2003; 25 (9): 1063-74.

15. Timm F., Barth E. Accurate eye centre localisation by means of gradients. In: VISAPP 2011. Proceedings of the International Conference on Computer Theory and Applications. 2011: 125-30.

16. Baltrusaitis T., et al. OpenFace 2.0: Facial behavior analysis toolkit. In: Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018). 2018: 59-66.

17. Fanelli G., Gall J., Van Gool L. Real time head pose estimation with random regression forests. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. 2011: 617-24.

18. Newman R., et al. Real-time stereo tracking for head pose and gaze estimation. In: Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2000). 2000: 122-8.

19. Chen J., Ji Q. 3D gaze estimation with a single camera without IR illumination. In: Proceedings of the 19th International Conference on Pattern Recognition. 2008: 1-4.

20. Sprague J.M., Meikle T.H. Jr. The role of the superior colliculus in visually guided behavior. Exp Neurol. 1965; 11: 115-46.

21. Bronson G. The postnatal growth of visual capacity. Child Dev. 1974; 45: 873-90.

22. Dubowitz L.M.S., et al. Visual function in the newborn infant: is it cortically mediated? Lancet. 1986; 327 (8490): 1139-41.

23. Atkinson J. The developing visual brain. In: The Developing Visual Brain. Oxford: Oxford University Press, 2008: 65-90.

24. Allen D., Tyler C.W., Norcia A.M. Development of grating acuity and contrast sensitivity in the central and peripheral visual field of the human infant. Vision Res. 1996; 36 (13): 1945-53.

25. Dubowitz L.M.S., et al. The maturation of visual acuity in neurologically normal and abnormal newborn infants. Behav Brain Res. 1983; 10 (1): 39-45.

26. Tinelli F., et al. The assessment of visual acuity in children with periventricular damage: A comparison of behavioural and electrophysiological techniques. Vision Res. 2008; 48 (10): 1233-41.

27. Atkinson J., et al. Changes in infants’ ability to switch visual attention in the first three months of life. Perception. 1992; 21 (5): 643-53.

28. Cioni G., et al. Cerebral visual impairment in preterm infants with periventricular leukomalacia. Pediatr Neurol. 1997; 17 (4): 331-8.

29. Cioni G., et al. Visual information processing in infants with focal brain lesions. Exp Brain Res. 1998; 123 (1-2): 95-101.

30. De Vries L.S., et al. Neurological, electrophysiological and MRI abnormalities in infants with extensive cystic leukomalacia. Neuropediatrics. 1987; 18 (2): 61-6.

31. Mercuri E., et al. Basal ganglia damage and impaired visual function in the newborn infant. Arch Dis Child Fetal Neonatal Ed. 1997; 77 (2): F111-4.

32. Cajal S.R. Histologie du systeme nerveux de l’homme & des vertebres. Paris: Maloine, 1909.

33. Zeki S. The distribution of wavelength and orientation selective cells in different areas of monkey visual cortex. Proc R Soc Lond B Biol Sci. 1983; 217 (1209): 449-70.

34. Ricci D., et al. Early assessment of visual function in full term newborns. Early Hum Dev. 2008; 84 (2): 107-13.

35. Mercuri E., et al. Visual function and perinatal focal cerebral infarction. Arch Dis Child Fetal Neonatal Ed. 1996; 75 (2): 76-82.

36. Mercuri E., et al. Visual outcome in children with congenital hemiplegia: correlation with MRI findings. Neuropediatrics. 1996; 27 (4): 184-8.

37. Hof-van-Duin J., et al. Visual field and grating acuity development in low-risk preterm infants during the first 2,5 years after term. Behav Brain Res. 1992; 49 (1): 115-22.

All articles in our journal are distributed under the Creative Commons Attribution 4.0 International License (CC BY 4.0 license)

CHIEF EDITOR
CHIEF EDITOR
Degtyarev Dmitriy Nikolaevich
Doctor of Medical Sciences, Professor, Deputy Director for Scientific Research of the V.I. Kulakov Obstetrics, Gynecology and Perinatology National Medical Research Center of Ministry of Healthсаre of the Russian Federation, Head of the Chair of Neonatology at the Clinical Institute of Children's Health named after N.F. Filatov, I.M. Sechenov First Moscow State Medical University, Chairman of the Ethics Committee of the Russian Society of Neonatologists, Moscow, Russian Federation

ORCID iD 0000-0001-8975-2425

Journals of «GEOTAR-Media»