Intensive assessment of executive functions derived from performance in cognitive training games

Researcher Antonio Javier Sutil Jiménez presents in this article the most relevant information from the study “Intensive assessment of executive functions derived from performance in cognitive training games”, which included our cognitive stimulation platform, NeuronUP.

Authors and universities involved

This study arose from the PENSA Study whose aim was to investigate the effect on the progression of cognitive decline through a personalized intervention promoting a healthy lifestyle, complemented with a natural compound from green tea called epigallocatechin gallate. The PENSA Study was led by the Barcelonaβeta Brain Research Center (BBRC) and the Hospital del Mar Medical Research Institute (IMIM), and in this work they collaborated with other Spanish universities and NeuronUP to study executive functions in an innovative way through remote monitoring.

Highlighting some of its authors, the conceptualization of this study was carried out by Natalia Soldevila-Domenech, Ilario de Toma and Rafael de la Torre from IMIM. Researchers from Pompeu Fabra University of Barcelona, the Carlos III Health Institute, the Networked Biomedical Research Center for Frailty and Healthy Aging (CIBERFES) in Madrid and Monash University of Melbourne also participated for data analysis, interpretation or manuscript writing. In addition, the participation of Iñigo Fernández de Piérola and Carolina Sastre from NeuronUP was relevant for data collection, aiming to avoid a conflict of interest regarding the results obtained in the study.

Target population

The target population of this study consisted of people without cognitive impairment, aged between 60 and 80 years. Specifically, a subset of 56 people from the PENSA Study was taken, 23 men and 33 women in the mentioned age range. Participants entered the study progressively in groups of between 9 and 14 people.

First, it was verified that they were a population at increased risk of Alzheimer’s disease because they were carriers of the APOE-ε4 gene. Subsequently, a standardized neuropsychological assessment was carried out to confirm that the participants did not have impaired cognition and that their scores were normal. In addition, participants had to meet the criteria for subjective cognitive decline (SCD).

Subjective cognitive decline (SCD)

Subjective cognitive decline (SCD) is the persistent perception a person has of experiencing a decline in their cognitive abilities compared to their normal state. This term was coined approximately a decade ago in order to identify the higher risk of developing mild cognitive impairment or Alzheimer’s disease in those who have this subjective perception.

This concept is very topical and since its creation the number of studies linking SCD and cognitive impairment has continued to increase. For more information, one can refer to the study published in Lancet Neurology* in which José Luis Molinuevo participates, who is also an author of the study reviewed here.

Assessment – Treatment

Cognitive training program

The cognitive training program was created with 36 neurorehabilitation activities from the NeuronUP platform that the participants performed remotely on their computers or tablets.

Initially, participants were instructed in person on how to use the platform and two short sessions were held to familiarize them with the activities. After this in-person workshop, a training plan designed by neuropsychologists was implemented that included 8 executive function activities, 6 memory activities, 6 language activities, 6 attention activities, 7 visuospatial skills activities and 3 orientation activities. These activities were selected to cover diverse cognitive domains related to Alzheimer’s disease, and they were implemented once a month over 12 sessions. These sessions were carried out at a pace of 2 or 3 sessions per week, with each session consisting of 3 activities lasting approximately 10 minutes each.

The full program lasted 12 months. The first two months were dedicated to familiarizing participants with the operation of the games to control learning effects, so they were not analyzed. Performance in the third month was used as a measure of baseline ability assuming that by then participants would have reached their optimal level in each game. Thus, the third month was considered the initial reference. Performance from months 4 to 12 was used to monitor cognitive changes over time.

A scoring system was designed to evaluate monthly performance for each game. The NeuronUP score consisted of an integer that referred to the difficulty level most frequently played, and a decimal that reflected correct and incorrect exercises and the time spent. Using the most frequently played difficulty level as a criterion was considered conservative for assessing cognitive change, and the inclusion of a decimal increases score variability within the same difficulty level.

How the activities work

There were two types of activities that showed between 9 and 12 different difficulty levels. These two types of activities were (1) worksheets that do not change difficulty level automatically and (2) games, whereas the games did change automatically. In the games the initial difficulty level was 2–3, and this varied depending on the participant’s performance. To advance a level it was necessary to complete 5 activities correctly and to decrease the level to fail in 3. Although the same number of activities was administered each month, the difficulty could vary since the starting level was the maximum difficulty level reached in the previous month.

Example activities

1. Balance the Bags (Balance the Bags): A person appears in the supermarket checkout and must place all the products into bags. The activity consists of calculating the weight of each product so that each arm carries the same weight.
2. Home Delivery (Home Delivery): Several buildings light up in turns. The activity consists of remembering the order in which they lit up and reproducing it in reverse order.

Traditional tests and comparison with NeuronUP versions

Following the neuropsychological assessment carried out in the PENSA Study, a comprehensive assessment including a global cognition measure was performed, which was the primary outcome of the PENSA Study: the Alzheimer Disease Cooperative Study Preclinical Alzheimer Cognitive Composite (ADCS-PACC) and a modified version (ADCS-PACC-plus-exe) that includes the Stroop Interference score and FDT Flexibility. The Mini-Mental State Exam (MMSE) and the Montreal Cognitive Assessment (MoCA) were also used to evaluate global cognition, as well as the Semantic Verbal Fluency Test for Animals and the Boston Naming Test.

To assess executive functions the following tests were included: WAIS Digit Symbol Substitution Test, WAIS Visual Puzzle Test, WAIS Digit Span Test (digit span backwards score), the Five Digits Test (FDT) for flexibility and the word–color score of the Stroop Colour and Word Test. Memory was assessed with the Free and Cued Selective Reminding Test (FCSRT) in the immediate free recall (IFR) and delayed free recall (DFR) modalities and the Logical Memory (LM) subtest of the WMS in immediate recall (IR), delayed recall (DR) and recognition.

For the analysis and interpretation of these data, composite scores for memory (NPS-MEM) and executive functions (NPS-EXE) were created by averaging the standardized Z scores.

Regarding the comparison of traditional tests with remote assessment via NeuronUP, it is necessary to clarify that the selection of executive function games was based on traditional neuropsychological measures of executive functions. These traditional neuropsychological measures were also administered in the PENSA study, which allowed comparison between both measures of executive functions. In contrast, NeuronUP memory games were not analyzed because they focus on visual memory, while the traditional memory tests available in the PENSA study measure verbal episodic memory, and they are not comparable as measures of memory.

Results obtained

Model validity and reliability

To study validity a model representing the overall score of the executive function tests was created. The validity of this global model was tested using confirmatory factor analysis.

The process to study validity began by creating a factorial structure based on the 8 NeuronUP executive function tests (NeuronUP EXE, the created model can be seen in figure 1 of the original article). Two of these tests were excluded from the model for showing low correlations with the other scores. The model was then calibrated and tested, yielding a good fit. The hypothetical model obtained a fit in the calibration sample of: χ2(9) = 10, p = 0.383; comparative fit index [CFI] = 0.99; robust mean square error of approximation [RMSEA] = 0.035; and standardized root-mean-square residual [SRMR] = 0.059.

In addition, the factor loadings for each game were significant and above 0.5. The average extracted variance was moderate (0.41) and reliability measured with Cronbach’s Alpha was adequate (α = 0.81). The model was also tested at 6 months and 12 months confirming the validity of the model; reliability scores improved slightly, and these results can be reviewed in the original article.

Convergent and discriminant validity of the model

To study the convergent and discriminant validity of the model the global executive function and memory scores were used using the neuropsychological tests and the NeuronUP platform.

	6 months	12 months
NeuronUP EXE vs Neuropsychology EXE	r = 0.66	r = 0.57
NeuronUP EXE vs Neuropsychology MEM	r = 0.45	r = 0.37

In addition, the NeuronUP EXE measure showed a strong correlation with global cognition measures, including the ADCS-PACC (r = 0.65), the ADCS-PACC-plus-exe (r = 0.63), and the MoCA (r = 0.47), and a moderate correlation with the MMSE (r = 0.42). Finally, a table of correlations of the NeuronUP EXE measure with some of the traditional measures is shown.

Domain	Test	Baseline measure	6‑month measure	12‑month measure
Executive functions	FDT Flexibility	r = 0.10	r = 0.40	r = 0.41
	Digit span backwards	r = 0.34	r = 0.46	r = 0.35
Memory	FCSRT IFR	r = 0.34	r = 0.40	r = 0.15
	LM recognition	r = 0.27	r = 0.20	r = 0.35
Other measure	Boston Naming Test	r = 0.66	r = 0.66	r = 0.66

Adherence to the platform

Average adherence to the cognitive training intervention was 73.8%. Almost all participants (94.6%) completed at least half of the training sessions. In summary, the following table shows the evolution of adherence over the 12 months of training.

Month	1	2	4	7	11	12
Mean adherence (%)	44.2	73.2	77.1	82	78.9	68

In addition, most participants (48 of 56) completed an online survey about the intervention. In this survey almost all indicated that they had the necessary knowledge to use the platform, with over 80% stating that it was easy to use. Up to 70% of them considered that their motivation remained high throughout the study and that the frequency was appropriate.

Conclusion

From the PENSA study a cohort of SCD patients was selected to prevent a possible progression to a neurodegenerative disease. This study validated the creation of a measure of executive functions based on cognitive performance in games carried out on the NeuronUP digital platform.

The study results support the validity of carrying out these measurements through monthly remote monitoring that allows detection of changes in executive functions while minimizing learning effects. Minimization of learning effects is achieved by automatically adjusting difficulty to individual ability to obtain reliable measures of each participant’s optimal performance after repeated testing. This suggests that this measure could be useful to overcome limitations of traditional neuropsychological assessment methods. In addition, adherence to the cognitive training intervention was high and participants rated the usefulness of the digital platform positively.

Bibliography

• Natalia Soldevila-Domenech, Ilario De Toma, Laura Forcano, Patrícia Diaz-Pellicer, Aida Cuenca-Royo, Beatriz Fagundo, Thais Lorenzo, Maria Gomis-Gonzalez, Gonzalo Sánchez-Benavides, Karine Fauria, Carolina Sastre, Íñigo Fernandez De Piérola, José Luis Molinuevo, Antonio Verdejo-Garcia, Rafael de la TorreIntensive assessment of executive functions derived from performance in cognitive training games, iScience, Volume 26, Issue 6, 2023, 106886,ISSN 2589-0042,https://doi.org/10.1016/j.isci.2023.106886.
• * The characterisation of subjective cognitive decline. Jessen, Frank et al. The Lancet Neurology, Volume 19, Issue 3, 271 – 278

If you enjoyed this blog post about the intensive assessment of executive functions derived from performance in cognitive training games, you will likely be interested in these NeuronUP articles:

“This article has been translated. Link to the original article in Spanish:”
Evaluación intensiva de las funciones ejecutivas derivada del rendimiento en juegos de entrenamiento cognitivo