Validation of a Subjective Test for Assessing Cognitive Load in Emergency Coordination Center ProFessionals Under High-Demand Situations Volume 63- Issue 4

Patricia Blanco Hermo^1,2*, Angel Vicario Merino³, Alvaro Gutierrez Martin⁴, Marta Alvarez Calderon¹ and Blanca Larraga Garcia⁴

• ¹Emergency Medical Service of the Community of Madrid (SERMAS, SUMMA 112), Spain
• ²Camilo José Cela University, Madrid, Spain
• ³Universidad Internacional de la Rioja UNIR, Spain
• ⁴Polytechnic University of Madrid, Spain

Received: October 01, 2025; Published: October 15, 2025

*Corresponding author: Patricia Blanco Hermo, Emergency Medical Service of the Community of Madrid (SERMAS, SUMMA 112), Camilo José Cela University, Madrid, Spain

DOI: 10.26717/BJSTR.2025.63.009912

Abstract PDF

Background: No studies have been developed and validated to assess the perceived cognitive load of healthcare professionals at an emergency coordination center during simulated high-demand situations.
Method: We designed a 22-item questionnaire (19 Likert-scale, 3 open-ended) validated via expert judgment (n=7) and pilot-tested (n=10) in simulated high-demand environments. The objective is to provide a practical tool for real-time identification of cognitive overload and its potential mitigation through training.
Results: Expert validation showed high content agreement (CVI > 0.8). Cronbach’s alpha for internal consistency was 0.861. Participants reported perceived cognitive load levels varying with task type and professional profile.
Conclusions: This instrument is reliable and adaptable for identifying perceived cognitive overload in highstress simulations. It can improve training design and safety procedures, avoiding failures due to overload.

Keywords: Cognitive Load; Expert Review; Validation Study; Simulated Training

Abbreviations: CVI: Content Validity Index; NASA-TLX: Task Load Index; SWAT: Subjective Workload Assessment Technique; WP: Workload Profile

The authors of Cognitive Load Theory themselves assert that it was designed to provide guidelines that promote activities optimizing intellectual performance—that is, to be identified as a determining factor in human performance, especially in high-pressure contexts [1,2]. The theory assumes the limited capacity of working memory, so the key is to reduce this “load” and promote the construction of mental schemas. Cognitive overload can lead to errors, decreased performance, and mental fatigue [3]. Therefore, precise, even subjective, evaluation of cognitive load is crucial in preventing adverse events [4]. Besides its role in working memory, cognitive load is crucial in various aspects of learning and instruction. Some key points where cognitive load plays an important role include [5,6].

1. Instructional design: Cognitive load influences how learning materials should be structured to avoid mental overload and facilitate understanding.

2. Schema construction: Essential for forming and automating mental schemas, which enable organ-izing and applying knowledge efficiently.

3. Attention and concentration: An adequate cognitive load helps maintain focus on complex tasks, avoiding distractions and improving performance.

4. Knowledge transfer: Facilitates applying learned material in new contexts by allowing better inte-gration and adaptation of information.

5. Extraneous load reduction: Identifying and minimizing unnecessary elements in information presentation helps reduce extrinsic cognitive load, improving learning efficiency.

6. Development of metacognitive skills: Proper management of cognitive load promotes reflection on the learning process, strengthening metacognitive abilities.

7. Adaptation to expertise level: Cognitive load should be adjusted according to the learner’s prior knowledge, as overload can harm novices while insufficient load may not challenge advanced learners.

8. Optimization of multimedia learning: In environments combining text, images, and audio, managing cognitive load is vital to avoid sensory overload and improve information retention. 9. Mental evaluation effort: Subjective cognitive load measures, such as questionnaires and self-assessments, estimate the mental effort perceived by learners during tasks.

10. Design of effective assessments: Considering cognitive load when creating tests ensures they assess real knowledge without adding unnecessary complexity.

Working memory is a set of processes that can be defined as a mental workspace. Humans are only aware of what is in working memory. All other cognitive activity is hidden from view unless brought into working memory. Therefore, the working memory is used to organize, contrast, and compare information. Interaction between elements in working memory consumes its capacity, reducing the number of items that can be managed simultaneously [7,8]. Baddeley’s theory [9] defines working memory as an active memory system that allows temporary retention and manipulation of information to perform complex cognitive tasks. It divides working memory into a visuospatial sketchpad for processing visual information and a phonological loop for auditory, mainly spoken, information. These two systems are coordinated by a central executive system. From this division comes the idea of Dual Coding, ie, the effectiveness of presenting information visually and auditorily. Any activity design ignoring working memory limitations is inevitably flawed. Although several dimensions have been proposed, all authors agree that cognitive load—especially subjective load—fits into three broad areas. The first includes time pressure aspects of the task (available time, needed time). The second refers to variables related to processing resource demands of the task (mental, sensory, task type). The third relates to emotional aspects (fatigue, frustration, stress level) [10].

To complement these dimensions, various techniques for predicting and assessing mental load have been identified, and their usefulness depends on how well they meet the following criteria: sensitivity, di-agnostic power, selectivity/validity, intrusiveness, reliability, implementation requirements, and operator acceptance [4,11]. Most methods used to evaluate mental load can be classified into three general cate-gories [12]:

1. Performance-based procedures: any increase in task difficulty raises demands, manifested by re-duced performance. The main advantage of these measures is their high diagnostic power. 2. Physiological measures: mental load can be measured through physiological activation levels. Their drawbacks include high implementation requirements, poor acceptance by participants, and questions about their validity as mental workload indices. Examples include P300 evoked potential, pupil diameter, and heart rate measurements.

3. Subjective procedures: greater capacity expenditure is associated with subjective feelings of effort, which individuals can adequately evaluate. Manyvalidated subjective methods exist for assessing mental load, notably the Cooper-Harper Scale, Bedford Scale, SWAT (Subjective Workload Assessment Technique), NASA- TLX (Task Load Index), and WP (Workload Profile) [13]. The wide variety of sub-jective techniques has led authors to study their characteristics to establish methodology reflecting properties to consider when choosing among techniques, depending on the research objective and con-text. Due to their particular characteristics (minimal implementation requirements, high acceptance, good validity and reliability, etc.), subjective instruments are most frequently used in applied contexts.

General Objective

To develop and validate a specific subjective instrument to assess perceived cognitive load in healthcare professionals at an emergency coordination center during high-demand simulated situations, specifically adapted to the characteristics and demands of their work contexts.

Specific Objectives

1. Design an initial cognitive load assessment questionnaire based on literature review and adapted to the work environments of emergency physicians and nurses in an emergency call center.
2. Subject the questionnaire to expert judgment validation to assess item relevance, clarity, and per-tinence.
3. Analyze the instrument’s adequacy to differentiate perceived cognitive load levels according to task type, professional profile, or participant experience.

Cognitive load is a determining factor in performance and decision- making in mentally demanding contexts, such as coordination of healthcare emergencies at coordination centers. Accurate evaluation of this load allows detection of overload situations, error prevention, and design of strategies to optimize performance and safety. However, most of the validated instruments available for assessing cognitive load have significant limitations when applied to these specific contexts: they include irrelevant items, are not adapted to the language or tasks of the professionals involved or have not been validated in similar populations. Therefore, there is a need to develop and validate a subjective instrument conditions based on individuals’ self-perception, specifically tailored to the working conditions of staff that work under of great cognitive pressure such as emergency physicians and nurses participating in high-load emergency call simulations. The development and validation of this instrument through expert judgment and pilot testing in simulated situations will provide a reliable, valid, and useful tool for researching and managing cognitive load in these environments. This will not only provide scientific evidence in a scarcely explored area within these professions but also have practical applications for training, simulation design, and operational safety improvement.

Study Design

A quasi-experimental instrumental study with quantitative and qualitative approaches, aiming to design and validate a subjective questionnaire to evaluate perceived cognitive load in professionals under high cognitive demand during simulated situations. Prediction models are used in various healthcare settings to estimate the value of an outcome or risk. Most models estimate the probability of a specific medical condition or whether a specific outcome will occur in the future. Examples of commonly used prediction models include EuroSCORE II (cardiac surgery) [14], the Gail model (breast cancer) [15], the Framingham Risk Score (cardiovascular disease) [16], IMPACT (head injury) [17], and FRAX (osteoporotic and hip fractures) [18]. Poor information from a model could mask flaws in the design, data collection, or conduct of a study that may cause harm. Better information can build greater trust and influence acceptance of the use of prediction models in healthcare by patients and the public. In this case, we need a subjective scale adapted to emergency coordination work. Other validated scales exist, but they are not adapted to this specific task, as mentioned in the introduction, because this work tends to be increasingly demanding for workers. The economic crisis and technological advances have led to an increase in the number of tasks and in their perceptual-cognitive demands, giving rise to more complex work situations in which task accumulation is frequent.

The direct consequence of these factors is an increase in mental workload. Numerous studies have been conducted using validated subjective tests [19-21], although, as we have mentioned, they are not adapted to these specific jobs. To give more consistency to the study and to be able to objectively see its validity, we have followed the TRIPOD+AI guide [22]. The TRIPOD (Transparent Reporting of a Multivariate Prediction Model for Individual Prognosis or Diagnosis) statement was published in 2015 to provide minimum reporting recommendations for studies developing or evaluating the performance of a prediction model [22]. TRIPOD+AI aims to promote comprehensive, accurate, and transparent reporting of studies by developing a prediction model or evaluating its performance. Comprehensive reporting will facilitate study appraisal, model evaluation, and implementation. TRIPOD 2015 (Appendix A & B) Comprises a 37-item checklist, including 25 items for reporting in both development and validation studies, and six additional items for model development studies and six items for validation studies. TRIPOD 2015 focused primarily on models developed using regression models, which was the predominant approach at the time. Since then, additional guidance has been created, such as for studies developing or validating prediction models using clustered data (TRI-POD-Cluster19 20) (https:// www.tripod-statement.org/).

Appendix A: Tripod AI Checklist.

Appendix B: Form.

Depending on the type of study being conducted (development, validation, or both), each checklist item must be addressed somewhere in the report. If a particular checklist item cannot be addressed, the in-formation should be indicated as unknown or irrelevant. Many items follow a natural order and sequence in a report, but others do not.

Participants

The target population includes physicians, nurses, and emergency technicians engaged in divided attention tasks in a lab setting, performing a primary task whose complexity increases by adding progressively difficult secondary tasks. The sample was selected non-probabilistically by convenience, including participants available during the study period, with n=10 for initial validation.

Instruments

The developed questionnaire consists of 22 items (see Table 1, initial question column):

• 19 items with a 5-point Likert scale (from “strongly disagree” to “strongly agree”)

• 3 open-ended questions exploring aspects not captured by closed scales.

Table 1: Proposal for change by experts and adaptation of questions.

Study Phases

To carry out this study, the following development stages have been carried out: • Literature review

and preliminary questionnaire design: various existing instruments (NASA-TLX, SWAT, WP) were analyzed, and relevant items were selected and adapted to the study context (Appendix B).

• Expert judgment validation: the initial questionnaire was presented to a panel of 7 experts in psy-chology, emergency medicine and nursing, technicians, and air traffic controllers. The Content Validity Index (CVI) proposed by Lawshe [23] and adapted by Tristán [24] was used to assess item relevance, clarity, and representativeness [25,26] (Appendix C). After the first expert evaluation, the questionnaire was revised incorporating feedback. The revised version was sent again to experts to confirm correct integration of comments. The questionnaire was approved after this second review (Appendix D).

• Pilot application in simulated situations: the questionnaire was administered as an exploratory study following controlled simulation sessions in laboratory tests with divided attention loads and physiological constants monitoring, as shown in the Figures 1-4 to 10 participants with profiles similar to the target population, collecting quantitative and qualitative data. The aim was to identify comprehension difficulties and estimate response times. All participants rated the questionnaire positively without identifying irrelevant or incomprehensible items. The final validation will be carried out on a larger sample of professionals from an emergency coordination center (n=30) for its final validation.

Appendix C: Questionnaire.

Appendix D: Expert validation table.

Figure 1

Figure 2

Figure 3

Figure 4

The tool was applied in these mentioned simulation contexts, followed by the subsequent analysis:

• to. Quantitative: reliability analysis (Cronbach’s alpha), exploratory factor analysis (in the next phase of the study with a higher n), and descriptive analyses.

• b. Qualitative: thematic analysis of open-ended questions to identify emerging categories related to the cognitive load experience. Based on the results, items may be adjusted or removed to optimize the validity and usefulness of the instrument. The Content Validity Index (CVI) was used for phase 2, the expert validation. Statistical analyzes were performed using Jamovi:

The questionnaire was administered to 7 experts in emergency healthcare and emergency coordination center. Of the participants, 57% were male and 43% female, achieving a balanced gender distribution. The mean age was 49.71 years, with a standard deviation of 4.33. The participants’ professional back-grounds were 57.14% physicians, 28.57% nurses, and 14.49% emergency medical technicians, all working in environments similar to those of the study’s target participants (high cognitive load environments). The Content Validity Index (CVI) was calculated for both adequacy and relevance by averaging all evaluators’ scores for each question, combining the adequacy and relevance ratings. In all cases, for both categories, the scores were above 4, as shown in Table 2, which validates the question.

Table 2: CVI Scoreboard.

The following results were obtained: Based on these evaluations and the comments from the expert panel, the questionnaire was modified according to the experts’ corrections. The questionnaire has been validated in Spanish. In Table 1, the initial and final questions have been translated into English for better understanding. Phase 3, or pre-pilot phase, the test was administered to 10 volunteer participants following expert approval.

Jamovi was again used to conduct descriptive analyses: 40% of the participants were male and 60% female, achieving a balanced gender distribution, with a mean age of 45.4 years and a standard deviation of 9.69. The participants’ professional backgrounds were 10% physicians, 50% nurses, and 40% emergency medical technicians. The average time dedicated to emergency work was 21.2 years, with a median of 25 years. At the time of the study, 40% were single, and 60% were in a relationship. Twenty percent reported alcohol consumption, 50% had visual acuity impairments, and 10% had color blindness. Regarding other health conditions, only 10% had hypertension (HTN). All tests were conducted in the morning, between 9:00 AM and 2:15 PM. The test was administered to all participants immediately after completing the task, with 70% receiving a low workload and 30% a medium workload; none reported a high workload. Performance during the test was above 80% in 80% of the participants, between 70-80% in 10%, and below 70% in only 10% as shown in Figure 5.

Figure 5

Internal consistency of the test was evaluated using Cronbach’s alpha coefficient, obtaining a value of 0.861. This result indicates a high degree of internal consistency among the questionnaire items, sug-gesting that participant responses were homogeneous and that the items coherently measure the construct of perceived cognitive load. This level of reliability is considered adequate for exploratory and initial validation studies.

The development and initial validation of a subjective instrument for evaluating perceived cognitive load in high-demand contexts represents a significant advance both in research and professional practice. The results support the relevance of this proposal in environments where decisions must be made under pressure, with limited resources, and under high mental demand. The findings indicate a high internal reliability (α = 0.861), consistent with other validated subjective instruments for cognitive load assessment, such as NASA- TLX [27] and WP [4]. This level of internal consistency suggests the items measure a coherent underlying dimension—in this case, perceived cognitive load—and supports the instrument’s utility in future applications and research as well as improving the training of these professionals to improve their performance and increase their confidence and reaction speed. The expert validation process ensured clarity, relevance, and appropriateness of the items to the specific study context. This phase was key to guarantee content validity, an essential dimension in constructing measurement tools, particularly in contexts with highly specific professional tasks. The consensus achieved among professionals from diverse disciplines (medicine, nursing, and psychology) strengthens the instrument’s applicability in interprofessional high cognitive demand settings. The pre-pilot phase not only confirmed participant comprehension of the questionnaire but also explored the instrument’s ability to detect differences in perceived load by professional profiles and individual characteristics.

Although the sample was small (n=10), preliminary results showed a reasonable distribution of responses, predominantly low to medium load perceptions, consistent with the controlled de-sign of the simulations. Additionally, descriptive analyzes provided a detailed characterization of participants, which will be useful in future study phases to evaluate the instrument’s sensitivity to demographic, clinical, or con-textual variations. Factors such as professional experience, visual acuity, or alcohol consumption might influence perceived cognitive load and should be considered in subsequent studies with larger samples and multivariate analyses. A noteworthy aspect is the inclusion of open-ended questions, which captured qualitative facets of the cognitive experience not reflected in closed items. This mixed-methods approach contributes to a richer, more contextualized understanding of the phenomenon studied and could be enhanced with more systematic thematic analyzes in future applications. However, some limitations must be acknowledged. The small sample size in the pre-pilot phase limits robust factorial analyzes and generalization of findings. Furthermore, while simulated designs allow some experimental control, they do not fully replicate real working conditions, where contextual and emotional variables may more significantly affect cognitive load. Replication in real environments or with high-er-fidelity simulations will be necessary to confirm the instrument’s ecological validity.

Another limitation relates to potential social desirability bias inherent in self-report questionnaires, which could lead to underestimating perceived load, especially in contexts valuing resilience or stress tolerance. Triangulation with objective measures (such as physiological or performance indicators) could help mitigate this bias in future research. This study constitutes a solid initial step toward building a useful, specific, and valid tool for subjective cognitive load assessment in high-demand professional contexts. Its implementation could contribute not only to early diagnosis of overload situations but also to the design of training, organizational, and technological interventions aimed at optimizing human performance and safety in these critical environments.

An initial validation of a subjective instrument was developed and conducted to assess perceived cog-nitive load among healthcare professionals at an emergency coordinating center during simulated high-demand situations. This instrument was specifically adapted and developed to the characteristics and demands of their work environments and demonstrated an adequate level of reliability. The design was based on a literature review and adapted to the work environments of emergency physicians and nurses. The questionnaire was validated through expert judgment, assessing the relevance, clarity, and adequacy of the items. The instrument is suitable for differentiating levels of perceived cognitive load according to the type of task, professional profile, or experience of the participant.

All authors contributed to the study conception, investigation and design, methodology, PB, BL and MA; software, PB, AG and BL; validation, PB and MA; formal analysis, PB and AV; resources, PB and BL; data curation, PB, AG and BL; writing—original draft preparation, PB and AV; writing—review and editing, PP, MA, AG and AV; visualization, AV, AG and PB.; supervision, PB; project administration, PB All authors have read and agreed to the published version of the manuscript.

The authors thank the anonymous reviewers for their valuable comments and suggestions. They also acknowledge the participation of all the volunteers who performed the high-load cognitive tests and evaluated the cognitive test for its use.

This research has been funded by FIIBAP (Primary Care Biosanitary Research and Innovation Foundation).

The authors declare no conflicts of interest related to this work.

This test needs to be tested by a larger number of participants, so the data will be available for review at the end of the project.

The study was conducted in accordance with the Decla-ration of Helsinki and approved by the Ethics Committee of HOSPITAL UNIVERSITARIO 12 DE OCTUBRE (protocol code 25/092 and date May 13, 2025).

Informed consent was obtained from all subjects involved in the study.

1. Wickens CD (2002) Multiple resources and performance prediction. Theor Issues Ergon Sci 3(2): 159-177.
2. Sweller J, Van Merrienboer JJG, Paas FGWC (1998) Cognitive architecture and instructional design. Educ Psychol Rev 10(3): 251-296.
3. Young MS, Brookhuis KA, Wickens CD, Hancock PA (2015) State of science: Mental workload in ergonomics. Ergonomics 58(1): 1-17.
4. Blond-Valdehita S, Díaz-Ramiro EM, López-Higes R, Martín-García J (2012) Effects of task load and cogni-tive abilities on performance and subjective mental load in a tracking task. An Psicol 28(3): 986-995.
5. Chandler P, Sweller J (1991) Cognitive Load Theory and the Format of Instruction. Cogn Instr 8(4): 293-332.
6. Leppink J, Paas F, Van Der Vleuten CPM, Van Gog T, Van Merriënboer JJG (2013) Development of an instrument for measuring different types of cognitive load. Behav Res Methods 45(4): 1058-1072.
7. Dietrich A (2024) Neurocognitive mechanisms underlying the experience of flow. Conscious Cogn 13(4): 746-761.
8. Johannsen L, Li KZH, Chechlacz M, Bibi A, Kourtzi Z, et al. (2013) Functional neuroimaging of the interference between working memory and the control of periodic ankle movement timing. Neuropsychology 51(11): 2142-2153.
9. Baddeley AD, Eysenck MW, Anderson MC (2020) Memory. (3^rd)., London New York: Routledge, Taylor & Francis Group, pp. 611.
10. Cárdenas-Vélez D, Perales JC, Chirosa LJ, Conde-González J, Aguilar-Martínez D, et al. (2013) The effect of mental workload on the intensity and emotional dynamics of perceived effort. An Psychol 29(3): 662-673.
11. We give DL (2020) Multiple-task performance. (1^st)., Boca Raton: CRC Press.
12. Meshkati N, Hancock PA, Rahimi M, Dawes SM (1995) Techniques in mental workload assessment. In: Eval-uation of human work: a practical ergonomics methodology. University of Missouri. Columbia, pp. 749-782.
13. Blond Valdehita S, Díaz Ramiro EM, Martín García J (2001) Methodological aspects of the subjective assessment of mental workload. Archives of Occupational Risk Prevention 4(4): 160-168.
14. García-Valentín A, Bernabeu E, Pereda D, Josa M, Cortina JM, et al. (2014) Validation of Eu-roSCORE II in Spain. Cir Cardiovasc 21(4): 246-251.
15. Kumar N, Singh V, Mehta G (2020) Assessment of common risk factors and validation of the Gail model for breast cancer: A hospital-based study from Western India. Tzu Chi Med J 32(4): 362-366.
16. Kasim SS, Ibrahim N, Malek S, Ibrahim KS, Aziz MF, et al. (2023) Validation of the general Framing-ham Risk Score (FRS), SCORE2, revised PCE and WHO CVD risk scores in an Asian population. Lancet Reg Health - West Pac 35: 100742.
17. Egea-Guerrero JJ, Rodríguez-Rodríguez A, Gordillo-Escobar E, Fernández-Delgado E, Martínez-Roldán Á, et al. (2018) Impact score for Traumatic Brain Injury: Validation of the Prognostic Tool in a Spanish Cohort. J Head Trauma Rehabil 33(1): 46-52.
18. Baleanu F, Iconaru L, Charles A, Kinnard V, Fils J, et al. (2021) Independent External Validation of FRAX and Garvan Fracture Risk Calculators: A Sub‐Study of the FRISBEE Cohort. JBMR Plus 5(9): e10532.
19. Acosta Prieto JL, Cuello Cuello Y, García Dihigo J, Almeda Barrios Y (2023) Models for assessing mental workload: A systematic review. Rev San Gregor 1(55): 158-180.
20. García Bustos JM, Fouce Fernández JG, García Villameriel LI (2019) Psychosocial Occupational Risk Map in the Community of Madrid. Madrid: UGT-Madrid, pp. 271.
21. García-Izquierdo AL (2017) Ergonomics and psychosociology applied to occupational risk prevention. Oviedo (Asturias): University of Oviedo Press, pp. 727.
22. Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, et al. (2024) TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ 2024: e078378.
23. Lawshe CH (1975) A QUANTITATIVE APPROACH TO CONTENT VALIDITY1. Pers Psychol 28(4): 563-575.
24. Tristán-López A (2008) Modification to the model of Lawshe for the quantitative assessment of the content validity of an objective instrument. Advances in Measurement 6: 35-44.
25. Spielberger CD, Vagg PR (2010) JSS: Work Stress Questionnaire: Manual. Madrid: TEA.
26. Arrogant O (2022) Sampling techniques and sample size calculation: How and how many participants should I select for my research. Intensive Care January 33(1): 44-47.
27. Hart S, Staveland L (1988) Development of NASA-TLX (Tasl Load Index): results of empirical and theoretical research. Human Mental Workload 52: 139-183.