Assessment of Cognitive Impairment in Older Research Participants

In any clinical trial or observational study, a certain level of cognitive function is required in order to ensure participant safety and ability to comply with the study protocol. Participants should adequately understand what is involved in their participation, as well as any potential risks or benefits of participation. The costs of using a specific screening method, however, need to be weighed against the information obtained. This prevents the inefficient use of available resources, such as avoiding assessments that require an extremely long administration time and unnecessarily increase participant burden. Due to the inherent challenges in obtaining a valid assessment of cognitive function in a time efficient manner, all potential assessment measures should be evaluated with a “best fit” approach. Keeping this “best fit” approach in mind, we reviewed and evaluated a number of screening tests for assessing cognitive impairment based on recommendations provided by experts in the field. Based on our review, we provide recommendations about existing methods for screening cognitive function in older research participants. Using the “best fit” approach, we would recommend the Mini-Mental State Examination (MMSE) to screen for cognitive impairment in older research participants. Other measures, such as the Cognitive Abilities Screening Instrument (CASI) and Modified Mini-Mental State (3MS), are indeed promising, but the MMSE can be administered in a time efficient manner and has the highest amount of supporting literature.


Introduction
In any clinical trial or observational study, a certain level of cognitive function is required in order to ensure participant safety and ability to comply with the study protocol. Participants should adequately understand what is involved in their participation, as well as any potential risks or benefits of participation. Although studies involving simple surveys or questionnaires may not require as careful screening as potentially higher risk studies(e.g., pharmacological interventions), [1] it is always necessary to ensure the participant fully understands the potential risks involved before agreeing to participate in the study.
Age has been found to be significantly associated with cognitive decline [2,3] thus cognitive function and understanding of study risks and procedures should be carefully assessed in studies involving adults who are 60 years and older [4]. Careful assessment of cognitive function in older research participants, as well as screening for mild cognitive impairment, could help to prevent incidents such as an accidental overdose resulting from a previously forgotten dose being taken. Ensuring that the participant is capable of adhering to study protocols can prevent inappropriate requirements from being placed on the participant. Additionally, screening for cognitive impairment benefits the scientific community at large; ensuring that participants fully understand what they are agreeing to is vital for ethically sound science.
In a research setting, a number of considerations determine what information an investigator will need to obtain from screening assessments. At the same time, participant burden should be reduced as much as is reasonably possible. Therefore, being able to effectively assess cognitive function in a time efficient manner is an important consideration in the evaluation of potential screening tools. More specifically, the costs of using a specific screening method should be weighed against the information obtained. This prevents the inefficient use of available resources, such as avoiding assessments that require an extremely long time to administer and unnecessarily increase participant burden.
The challenge of cognitive assessment, particularly in older adults, is to strike a balance between comprehensiveness and short administrative time, aimed at reducing subject burden and time constraints−more succinctly, administration time vs. accuracy. Here we attempt to outline the most valid but brief instruments that can be used to ensure the safety of older research participants through appropriate cognitive screening. Due to the inherent challenges in obtaining a valid assessment of cognitive function in a time efficient manner, all potential assessment measures should be evaluated with a "best fit" approach. Keeping this "best fit" approach in mind, we reviewed and evaluated screening tests for assessing cognitive impairment proposed in "A Review of Screening Tests for Cognitive Impairment" [5]. Based on evaluation of empirical findings, we provide recommendations about existing methods for screening cognitive function in older research participants.
The review by Cullen et al. [5] evaluated screening tests for their overall effectiveness in three areas: brief assessments in a physician's office, large-scale screening in the community, and guiding differential diagnoses with more specific needs. With respect to screening for cognitive function in older research participants, evaluations for brief assessments in a physician's office and large scale screening in the community share relevant features and similar requirements, including the need to minimize time involved yet obtain accurate (valid) and consistent (reliable) results. Thus, we selected those screening tools that were reviewed favorably in these two categories, and discuss their potential for being used to evaluate cognitive function in older research participants. Specifically, we review the Modified Mini-Mental State (3MS), Mini-Mental State Examination (MMSE), Cognitive Abilities Screening Instrument (CASI), Short Test of Mental Status (STMS), Short and Sweet Screening Instrument (SASSI), and Addenbrooke's Cognitive Examination-Revised (ACE-R). These screening tools showed the most promise in a recent review with respect to brief assessments in physician's offices and screening for clinical trials, including studies conducted in the community [5].
The 3MS, a modified version of the MMSE, has an administration time of about 10-15 minutes [5] and covers the six key cognitive domains of verbal fluency, reasoning/judgment, expressive language, visual construction, immediate and delayed free verbal recall, and cued verbal recall. This makes the 3MS and the CASI the most comprehensive (i.e., covering 6 cognitive domains). The 3MS is modified for increased specificity, as it has a score range of 1-100 as opposed to the MMSE's 30-point range. While among the most complete in terms of its scope, the 3MS was reported to exhibit wide variation in scores without an accompanying clinical change [5,6].
For example, test-retest (one-month period) differed by as much as 20 points [6]. The observed confidence interval was -16 to 16, which suggests the possibility of wide variability in scores among individuals with no expected change in cognitive status. Another study corroborated these findings, stating that, "…testretest correlations and evidence from regression analyses showed that considerable variability existed among the participants" [7]. The 3MS is attractive in its coverage of the six key domains of cognitive function, [5] but the variance in score merits further study before it can be widely recommended for use in a research setting. Despite these concerns related to score variance, the rate of false negatives for cognitive impairment was found to be quite low (1.4%; CSHA) [8].
Another possible tool to screen for cognitive impairment in older research participants is the MMSE. The MMSE is primarily a written examination, which is scored on a scale of 1-30. With an administration time of 8-13 minutes and coverage of four of the six cognitive domains assessed by the 3MS, [5] the MMSE has an administration time and level of specificity that lies near the middle of the potential screening tools discussed in this paper. Due to its long history as a cognitive screener (introduced in 1975), the MMSE is perhaps the best studied in terms of reliability and validity. For example, a 2008 study evaluated the validity of the MMSE, administered by trained general practitioners, in a public health setting [9]. The study team referred patients who were determined to be cognitively impaired to Alzheimer's Evaluation Units for diagnosis. With a calculated inter-rater agreement of 0.86 between the MMSE scores of the general practitioner and that of the Alzheimer's Evaluation Units, the MMSE as utilized by the general practitioners was determined to be sufficiently accurate to detect patients with cognitive impairment as determined by the Alzheimer's Evaluation Units. This is especially relevant, as the documented implementation of the MMSE by non-specialized personnel provides actual evidence to evaluate its potential usefulness with respect to screening for cognitive impairment in older research participants.
The MMSE has also been found to be a useful tool for investigating mild cognitive impairment (MCI). For example, MMSE scores falling between 23 and 26 (the range thought to represent MCI) indicated MCI in terms of the prediction of future dementia as well as more detailed definitions/methods (Mayo Clinic-defined amnestic, nonamnestic, multiple, and revised MCI) [10]. There are, however, some common critiques of the MMSE. First, the test has been shown to exhibit floor and ceiling effects in some populations (poorly and highly educated), [11] due to its narrow score range of 30 points [12]. The MMSE has also been copyrighted since 2001, which makes it more expensive to implement than some other screening methods.
The CASI is the only other screening test, in addition to the 3MS, that assessed all six cognitive areas discussed; however, it also has the longest administration time of approximately 15-20 minutes [5]. The CASI was listed as a possible compromise between tools with broader coverage and tools used more as a brief assessment, [5] which may be related to its larger score range of 0-100. In its original publication, the CASI was cited as having considerable cross-cultural applicability, which may appeal to investigators who anticipate a more culturally diverse study population and wish to minimize potential biases [13]. Also discussed in the Cullen et al. review, [5] however, was an example of low specificity in some random samples [14]. While there is currently not a significant amount of literature devoted solely to its evaluation relative to the other screening tools discussed, [5,[13][14][15][16] those investigators anticipating the need for cross-cultural applicability may wish to consider utilizing the CASI. It is worth noting that the CASI incorporates elements of the MMSE and the 3MS, thus the convergent validity between these measures can be quite high. For example, Graves et al. [17] reported a correlation coefficient of .92 between the CASI and MMSE in a sample of 57 probable Alzheimer's disease cases.
The STMS has the smallest time requirement (approximately 5 minutes) of the screening tools evaluated here [5]. Decreasing the time required to screen for cognitive function can serve to lessen the length of the overall study visit. Unfortunately, there is limited supporting literature available on the STMS [18][19][20]. Looked for correlations with standardized psychometric testing, and the 2003 study done by Tang-Wai et al. [20] compared the STMS with the MMSE in the detection of mild cognitive impairment. Despite the STMS being developed to outperform the MMSE, Tang-Wai et al. [20] reported only modest evidence of increased sensitivity. The STMS also only covers four of the cognitive domains evaluated (reasoning/judgment, visual construction, and immediate and delayed free verbal recall). Any advantage over the STMS is likely to be observed in the verbal recall domain, which has been described as more complex and nuanced than those found in the MMSE Mansbach et al. [21]. As such, the STMS should only appeal to investigators conducting studies for which time is the primary concern.
The SASSI, with an average administration time of about 10-15 minutes, has a similar time requirement to that of the previously discussed 3MS. In the Cullen et al. [5] review, the SASSI was evaluated as a potential tool for screening cognitive function in the community. It covered all of the key cognitive domains aside from reasoning/judgment and cued verbal recall, and was listed as one of the most promising candidates for brief assessments of cognitive function. However, there is a dearth of literature on the SASSI [22]. While preliminary evaluations do indicate that the SASSI shows promise [22] (compared to a full cognitive battery that took over 30 minutes to administer, it was 4% more sensitive and only 1% less specific), the lack of additional supporting literature makes it difficult to recommend it for use in any given study over another potential screening tool that has far more supporting literature.
The last potential method for screening cognitive function in older research participants reviewed is the ACE-R [5]. With an administration time of about 16 minutes, the ACE-R was listed as covering the cognitive domains of verbal fluency, expressive language, visual construction, and immediate and delayed free verbal recall. Although the ACE-R was listed as a promising test [23][24][25] Larner & Mitchell reported a sensitivity of 95.7% and a specificity of 87.5%), [24] it has not yet been validated in community samples. It was described by Cullen et al. [5] as a tool more orientated towards differential diagnoses for clinicians in secondary and tertiary practice settings. While screening for cognitive function in a research setting does share requirements with brief assessments in physicians' offices or large scale community evaluations (minimizing time requirements, the need for accurate and generalizable results, etc.), it is difficult to recommend a specialized tool such as the ACE-R for a broad assessment of cognitive function. Similar to the previous discussion of the SASSI, the ACE-R may be an effective tool to screen for cognitive function on a smaller scale, but it would not be recommended over other possible screening methods that have a much larger body of supporting literature.
Given the currently available measures to screen cognitive function in older research participants, and based on the available evidence, we believe the MMSE is a useful, efficient, and balanced screening measure of cognitive function for most studies. It is important to note that other tools such as the 3MS and CASI also show promise, but the MMSE currently provides the investigator with the largest body of supporting literature. Further confidence for the selection of these three screeners, in contrast to the other measures reviewed, comes from exposure to additional psychometric scrutiny relating to item bias. The CASI, 3MS, and the MMSE have been previously examined with item response theory methodology, with the aim of improving validity. Bias is a serious problem in psychometric tests. Item bias or differential item functioning (DIF) is said to be present when examinees from different groups have differing probabilities of success on an item, after controlling for overall ability, or total score [26]. That is, group membership may influence the use of, and familiarity with, certain phrases, thus affecting the interpretation of and response to items, and ultimately, test scores. Common areas investigated for item bias or DIF include education, age, race, etc. If an item is free of bias, responses to that item will be related only to the level of the underlying trait (e.g., cognitive ability) that the item is trying to measure. The presence of large numbers of items with DIF is a severe threat to the construct validity of tests and the conclusions based on test scores derived from such items [27].
Ramirez et al. [28] reported that most MMSE items have been found to perform differently for individuals of different educational levels and/or for members of different racial/ethnic groups. However, the authors acknowledge that an examination of the magnitude of item bias has rarely been considered. This point should not be understated, as the impact of individual item bias on the scale's total score may be trivial often, some items favor one group while others favor the other, and the effect on the total score is canceled out. Jones & Gallo [29] also found evidence of item bias by education on the MMSE: Significant sex and education group DIF was detected. Those with low education were more likely to err on the first serial subtraction, spell world backwards, and repeat phrase, write, name season, and copy design tasks. However, Jones and Gallo also reported magnitude, and found that it was small and not a threat to the validity of total scores. More generally, Crane et al. [30] reported that the MMSE, as well as the 3MS and CASI, did present with poor precision for high scores, which is more relevant for change score assessment and not as a screener of cognitive impairment. However, relatively low score on the MMSE (i.e., <15) will indicate that some impairment is most likely present [31].
An examination of item bias in the CASI revealed that gender and race had little impact on CASI scores, but that items were biased to some degree by age, and more so by education level [27]. In a separate investigation into the CASI and DIF, it was also demonstrated that the CASI was not biased by gender, and just a few items related age. As with Krane et al. [32] DIF related to education was the most problematic. However, further analysis into the impact of education-related DIF was minimal; only 1.5% of subjects (sample 1) and .2% subjects in sample 2 presented with salient DIF related to education. The 3MS has also been shown to present with DIF due to education. Mukherjee et al. [33] examined DIF in the 3MS and reported that some of the differences in total scores across education groups were due to DIF. However, rates of cognitive decline were negligible when accounting for or ignoring DIF in education. Similarly, Wiest et al. [34] reported that 40 of the 46 3MS items had education DIF. Despite the widespread presence of DIF in this study, the authors did not examine the magnitude of such DIF, i.e., the impact of item level bias on the 3MS total score.

Conclusion
While no screening method is without its limitations, the considerable amount of empirical support for the MMSE allows the investigator to be aware of potential drawbacks pertaining to their particular study and account for them in the appropriate manner. As noted above, the MMSE can be administered in 8-13 minutes and covers four of the six key cognitive domains identified (expressive language, visual construction, immediate free verbal recall, and delayed free verbal recall). Overall, the MMSE has a substantial body of supporting literature, and a relatively minor weakness, which suggests it would be an appropriate choice for most studies.
Investigators, however, should always remain cognizant of newly presented findings, and adjust their screening methods accordingly. For example, item bias in the MMSE seems most pronounced when examining education status. More specifically, persons with low education can potentially be incorrectly identified as at risk for dementia [35]. Potential solutions for obtaining more accurate MMSE assessment in participants with low education include the use of different cut-scores, adjusting scores for education, or the use of different domains of cognition within the MMSE. We recommend the latter, that is, for subjects who present with low education (at or below the standard total score cut-point), that physicians should further examine the memory sub domain of the MMSE [36] The memory domain of the MMSE has been shown to less biased by education, in contrast to total score [35].
Based on our review of several screening methods, we would recommend the MMSE to screen for cognitive impairment in older research participants. Other measures, such as the CASI and 3MS, are indeed promising, but the MMSE can be administered in a time efficient manner and has the highest amount of supporting literature. The extensive study and intimate knowledge of the MMSE's shortcomings allow the investigator to understand where potential issues may arise and account for them accordingly. We believe that the MMSE provides the "best fit" approach to screening for cognitive impairment in older research participants in a time efficient manner, and that a large body of literature allows investigators to benefit from the assessment's strengths and understand its potential weaknesses.