Background: The application of multi-criteria decision analysis in healthcare, has recently increased as decision makers seek robust data to inform policy decisions. The present study aimed at evaluating the benefits of three licensed drug (Ranibizumab, Dexamethasone and Aflibercept), for treating diabetic macular edema, within 4 Italian regional settings, according to a multi-criteria decision analysis based on the Core Model domains.
Methods: After the assessment of the EUnetHTA Core Model domains, an appraisal was then undertaken by applying the multi-criteria decision analysis methodology, according to 11 criteria. An initial domain-prioritization phase was followed by the technology evaluation across domains. The four appraisals were compared to identify how closely they matched.
Results: Despite the differing professional roles of the experts from four separate Regional healthcare systems, the only statistically significant differences found in the domain-prioritization phase, was the heterogeneity of weight given to the equity dimension (p=0.013). Similarly, the scores attributed to each domain for a particular technology showed minimal inter-evaluator variability, demonstrating the robustness of this multi-criteria decision analysis. The only significant difference found, was that Dexamethasone received higher scores in the safety and efficacy dimensions (p=0.002). The appraisals in all 4 regions demonstrated that Dexamethasone implant received a higher overall score than the comparator technologies.
Conclusions: The evaluation demonstrates that multi-criteria decision analysis can enhance the technology assessment process, providing decision makers with a robust method of assessing the preferable treatment options, for patients suffering from diabetic macular edema.
Health Technology Assessment (HTA) is a widely implemented
technical tool used to inform the decision-making process by
providing policymakers with objective and relevant data, thus
improving the efficiency in the allocation of resources . The main
feature of HTA is the possibility of including different perspectives,
with an evidence-based approach, in the comparison of different
technologies , thus investigating their economic, social, clinical,
ethical, and organisational implications . HTA Agencies and
practitioners worldwide have acknowledged HTA as being “the
gold standard” (i.e. reference practice), for identifying and selecting
the most promising innovations in healthcare, avoiding waste on
innovations that do not generate “enough value”, in relation to
their costs . It should be noted that the definition of healthcare
technologies ranges from biopharmaceuticals and diagnostics to
medical devices, thus also including clinical procedures and clinical
pathways , requiring the implementation of HTA practices for
all the different categories of healthcare technology, and ensuring
a quantitative evaluation, both in the assessment, and in the
appraisal phase. Although HTA can be adopted for the assessment
of any healthcare technology, the formal requirement of a multidimensional
and multi-disciplinary approach as a necessary prerequisite
for the systematic adoption of the technology varies
across healthcare systems. Italian law  for example requires
the implementation of an HTA approach for medical devices.
Given the significance of correctly assessing the impact of the
adoption of a healthcare technology, it is important to carry out
a complete HTA evaluation that does not focus solely on cost and
effectiveness parameters due to scarce economic resources or time
constraints. On the one hand, the literature  has reported how
the use of a mini-HTA  could generate, in many cases, insufficient
information. A mini-HTA generally includes a description of the
assessed technology and the comparator, but evidence regarding
the selection and interpretation of the clinical literature and
other data is often missing. On the other hand, the analysis of all
the implications derived from an innovative technology, with the
assessment of all the dimensions required by the EUnetHTA Core
Model , often takes a significant amount of time, absorbing a
substantial quantity of human and economic resources .
Whilst HTA’s are often based, explicitly or otherwise, on the EUnetHTA Core Model, the appraisal can be further supported  through the use of a Multi-Criteria Decision Analysis (MCDA) approach [10-11]. The combination of both approaches allows for the achievement of a final quantitative comparison of the technologies evaluated, taking into account multiple criteria, in an explicit manner . This provides a quantitative report for payers and decision makers developed in a transparent manner, aimed at supporting the decision-making process [10-11]. Despite the use of MCDA in healthcare, as an alternative methodological approach for assessing the value of healthcare interventions, ranging from licensing decisions at the marketing authorization stage [13-14], to coverage decisions at the HTA stage [10,15], or to treatment selection decisions at prescribing level [16-17], there is little documented about the integration of HTA and MCDA . Focusing on the comparison between healthcare technologies available for the treatment of a specific pathology, the setting of the diabetic macular edema (DME) represents an excellent opportunity for a case study of the integration between HTA and MCDA. One of the most significant healthcare expenditure items, in the ophthalmological setting, is that devoted to intravitreal drugs used for the treatment of maculopathies. DME is a disabling and prevalent maculopathy, affecting on average 6.81% of the diabetic population, thus generating significant social and economic burden. Given the scale of this disease and, above all, its growing potential impact in the future, due to the increasing incidence of diabetes and to the ageing population, it is becoming ever-more necessary to improve patient access to DME treatment strategies and, at the same time, to understand the opportunity for optimizing the cost of such strategies.
The present study aimed at evaluating the implementation of an HTA multi-dimensional approach, integrated with a multicriteria decision analysis in the appraisal phase, in the context of the therapeutic technologies licensed for DME treatment (Ranibizumab, Dexamethasone and Aflibercept). The study was developed within four Italian regions: Lombardy, Liguria, Tuscany and Veneto, thus testing the robustness of the combined approach, in four different regional settings.
In order to achieve the above-mentioned objective, a multidimensional comparison of principal treatments available was carried out, using the EUnetHTA Core Model  and MCDA [10-11]. In particular, the investigated treatments were Ranibizumab (reimbursed for the treatment of DME since December 2012), Aflibercept (reimbursed since March 2015), and Dexamethasone implant (initially reimbursed for second-line “off-label” use in DME from July 2014, then fully reimbursed from August 2015), all utilized within the Lombardy, Liguria, Tuscany and Veneto regions. It should be noted here, that the drug bevacizumab, despite its use in some countries [18-20], was not included in the present evaluation in DME patients, since its use is only foreseen by the Italian NHS for patients with “wet age-related” macular degeneration.
The following study design, consisting of three distinct phases, was implemented.
A. Production of four complete HTA reports, by means of literature research, qualitative and quantitative HTA questionnaires, and interviews administered to 21 healthcare professionals, considering the real-life clinical practice within the four different regions and the use of Ranibizumab, Dexamethasone and Aflibercept treatment options in routine patient management. The reports were structured, with reference to the dimensions derived from the EUnetHTA Core Model.
i. general relevance
iv. economic and financial impact
v. equity aspects
vi. social, ethical and legal dimensions; and
vii. organizational impact (assessed from both a qualitative and quantitative perspective), taking into account the particular characteristics of the technologies offered for the aforementioned assessment.
B. Development of a final appraisal for each report, including a quantitative summary of the results derived from the judgments attributed by healthcare professionals with various roles, and within different organizational settings. The chosen MCDA approach included a step in which the dimensions were prioritised by 21 healthcare professionals, followed by a final evaluation carried out by 12 HTA experts. The latter stage involved attributing a score for each technology, in each dimension, via a 3-level rating score (from “1” low performing to “3” high performing). Further details with regard the methods used for the enhancement of each dimension, as well as the criteria evaluated for MCDA are listed in Table 1. Once having collected all the information, inferential analyses were conducted. In particular, the investigation statistically significant differences, was performed by means of “one-way” Anova tests between the attributed scores of the evaluators, comparing the four appraisals with each other, in order to identify how closely they matched. All statistical analyses were conducted using the Statistical Package for Social Science (SPSS version 22)
The Sample Under Assessment
Nine hospitals, within the four regions, were involved in the study, where real-life data, healthcare professionals’ perceptions and experts’ opinions were collected. The assessment was mainly conducted in medium sized (56%) and public (67%) hospitals. In the Italian setting, a medium size hospital has between 401 to 599 beds available for patients; furthermore, hospitals working within the NHS, may have a private or a public ownership, and thus be classified as a Hospital Authority, Local Health Authority or private enterprise. Twenty-one multi-disciplinary experts in DME were involved in the prioritization phase, where the HTA dimensions were prioritized, in order to show their relative importance through a numerical value, following a rating scale from 1 to 8, thus defining which is the most important dimension. Due to the multidisciplinary nature of HTA, the healthcare professionals involved had different roles: in particular, 57% clinicians and 43% hospital pharmacists. As mentioned previously, following the prioritization of the dimensions, 12 HTA experts (3 per region) reviewed each of the regional reports and assessed the quality of information related to each treatment option, giving a numerical value for each dimension using a 3-level rating score (from “0” low performing to “3” high performing), for each sub-domain.
Results from the HTA
The four HTA reports showed similar overall results across the four regional settings [21-24]. Ranibizumab, Aflibercept and Dexamethasone present similar clinical efficacy, whilst they differ with regards to the number of injections required, and adverse events (AEs) occurring [25-27]. Consequently, the reports all found that the preferable treatment option within the DME target population would be the Dexamethasone implant [22-23]. Furthermore, it was evident that DME could be considered an urgent priority, both from the healthcare professionals and the policy-makers point of view (making up on average 24%, 17%, 15% and 23% of the total maculopathies occurring in Lombardy, Veneto, Liguria and Tuscany respectively). Thus, it was demonstrated, that within the four regions [22-23], a greater utilization of therapies with a low injection burden over a time horizon of 36 months would result in the following considerations.
a) Significant economic savings for the four Regional
Healthcare Services, depending on the market shares gained by
b) An optimisation of the operating theatre time, with less time spent delivering a complete treatment cycle.
c) An overall increase in patient safety, since both ranibizumab and aflibercept are associated with the development of long-term adverse events, with a consequent economic resources’ absorption for their management.
d) Incremental equity of access, positively impacting on the hospital waiting lists (with a consequent improvement in the access to care) and on the retention in care of patients suffering from DME (Figure 1); and
e) a reduction of the social cost, in terms of “out-of-pocket” expenditure.
With regard to the patient’s perspective, administering fewer intravitreal injections allows for a significant decrease in the productivity loss (which has an impact both on the patient and on their care-givers): a longer gap between administrations helps to lower the frequency of visits to the hospital for treatment and check-ups. Focusing on the economic aspects [21-24], a budget impact analysis was implemented assuming a Regional Healthcare Service perspective, comparing a baseline scenario consisting of Ranibizumab intravitreal injections as initial therapy and Dexamethasone implants for the treatment of non-responsive patients, with an innovative scenario characterized by the use of all 3 of the licensed DME treatment options as first-line therapy, on the basis of the market shares gained by each therapeutic option, in reallife practice. For both scenarios, the average number of injections of each treatment option administered per patient per year, was derived from real clinical practice in the participating centres. The same scenarios were also compared from an organizational point of view, in order to understand the potential to free up the operating room (OR), in terms of occupancy hours, since the Dexamethasone implant requires fewer intravitreal injections per patient treated. Table 2 shows detailed information with reference to both the economic and the organizational advantages.
The utilization of the Dexamethasone implant was found to lead to significant economic savings for the four Regional Healthcare Services: considering the real-life implementation of the alternatives (e.g. market shares and injection rates), the economic benefit ranged from a minimum of -2.92% in Liguria to a maximum of -8.04% in Lombardy. From an organizational point of view, the implementation of the innovative scenario led to a reduction of operating room time spent treating the current DME patients ranging from -5.65% in Liguria to -25.36% in Lombardy, potentially providing extra capacity to enhance patient access to care. The assessment of the EUnetHTA dimensions, revealed that, in a clinical setting, characterized by the potential administration of all the three options licensed for the treatment of DME currently available, the treatment of a larger proportion of eligible patients, with the Dexamethasone implant would result in a significant reduction in healthcare expenditure and/or freeing up of resources, that could be used to treat more patients and/or increase injection rates for better clinical results.
Results from the MCDA
As a first step, the dimensions of the EUnetHTA Core model were prioritized , to assess the weight of each of these aspects within the pathology being analysed, thus ordering the variables examined from the most important and highly prioritized (a score of 1) to the least relevant and least prioritized (a score of 7), as shown in Table 3. No statistically significant “between-region” differences emerged regarding the prioritization of the dimensions, except for the equity domain (p-value=0.013), that showed significant inter-regional variability. Twelve HTA experts were enrolled as external evaluators to review the HTA reports and evaluate each technology within each dimension and sub-dimensions, in accordance with a 3-level evaluation scale Table 4. The results showed that the evaluators, albeit having different professional roles and working in different Regional settings, attributed similar performance scores for a particular technology within each of the HTA dimension during the appraisal phase, even despite being conducted in different periods, throughout 2015 and 2016. This demonstrates an important consistency of the appraisal of each technology across Regions. More specifically, there were no statistically significant differences between the evaluators’ appraisal of the technologies, except for Dexamethasone’s safety and efficacy dimensions (p-value=0.002) where higher scores were attributed by the Tuscany, Veneto and Liguria evaluators compared with the Lombardy evaluators. This may be because the Lombardy HTA was conducted earlier in the life cycle of the technology, whereas the HTAs in the other 3 Regions were conducted at a later stage when there was more experience of using the technology.
Once the external evaluators had given their comparative judgements on the completeness and the quality of each HTA report, in terms of sub-dimensions, for each of the alternative treatment option Table 5, the incidence rate of each sub-dimension was calculated (ratio between the numerical value attributed by the HTA experts, and the maximum achievable possible score for the specific dimension). The normalized score was then determined, by multiplying the average score for the three comparators (Ranibizumab, Dexamethasone and Aflibercept), with the percentage of incidence previously calculated. The overall score was obtained by multiplying the normalized score calculated for each dimension, with the normalized value of priority for that dimension, and then summing the scores together for each technology, as shown in Table 5. With regard to the normalized summary scores attributed to each dimension, for each of the three therapeutic option taken singly, no statistically significant differences emerged between the Regions (except for the equity dimension), there being high homogeneity reported in both the prioritization, and in the evaluation phases, thus demonstrating consistent behavior among different professionals, within the appraisal phase. Furthermore, when comparing technologies, the Dexamethasone implant was consistently attributed a higher overall score than the comparators, relating to the final normalized weighted score derived from the MCDA approach (p-value = 0.000) as seen in Chart 1, driven by the significant advantages seen in the economic, organizational and equity domains. These results were robust and consistent among the evaluated technological options, despite the evaluations being made within different organizational settings and by different evaluators.
The present study demonstrates that the MCDA approach can be employed to enhance the HTA process, synthesizing the technologies’ “real value”, through a numerical quantification, thus facilitating the decision-making process as well as giving information with regard to stakeholder preferences in a transparent way. The integration of HTA and MCDA, would also provide decision makers with a robust rationale, for the assignment and implementation of a preferred treatment option. In the general context of limited economic resources, it is increasingly important to be able justify treatment choices in terms of value to the system as well as to the individual patient, thus being consistent with the literature defining the importance of using MCDA in hospitals as an acknowledged method to support transparent and evidencebased decision-making [29-30]. Despite the application of a multidimensional approach to technology assessment being mandatory in the Italian setting only for medical devices , the analysis of the data suggests that HTA and MCDA could be implemented together for the assessment of other types of healthcare technologies such as drug treatments. The present study, with the production of four HTA reports following the EUnetHTA Core Model approach, also provides decision makers with a complete picture of opportunities and threats regarding the technological choices in DME. As reported in the literature , the methodology used in this analysis, is replicable in every setting of interest for being complete and simple to undertake, underlying the relevance to produce a full HTA report, without focusing the attention only on specific domains.
Furthermore, despite inter-regional variability in the relative
weighting of each domain, and in the absolute scores attributed to
the technologies for each domain and sub-domain, the results from
the appraisal phase suggest that the Dexamethasone implant may
be considered the more advantageous technological alternative
to adopt, within the target population, having acquired higher
scores than the comparators utilizing a robust and comprehensive
MCDA approach. The results derived from the MCDA do not show
significant variances between hospitals or between regions and
are also robust across the various professional profiles of the
evaluators involved in the process. This not only suggests that the
approach is robust, but also that when a technology presents an
advantage in the HTA phase, that this is confirmed in the appraisal
phase. This consideration suggests that in the absence of significant
new clinical evidence, the application of the results to other
geographies could be implemented in a streamlined fashion by
carrying out the dimension prioritization and the appraisal phase,
for the selected geography, without necessarily needing to repeat
the evidence-based HTA approach, thus confirming the philosophy
of the EUnetHTA Core Model. The results of the present study
reported a high degree of homogeneity in both the prioritization
and the evaluation phases. Even if other professionals, such as
regional healthcare policy-makers or hospital managers, had been
included the appraisal, we do not believe that the results would
have changed; on the contrary, the level of priority given by these
stakeholders would likely have further emphasized the importance
of the organizational and economic aspects that would have further
increased the advantage for the Dexamethasone implant.
In conclusion, the MCDA approach suggests that the use of Dexamethasone should be considered a preferred treatment to adopt within the target population, having acquired statistically significantly higher scores than the comparators in all the four regions under assessment. Savings resulting from the adoption of the Dexamethasone implant could then be reinvested within the same therapeutic area, thus
i) improving patient access and adherence to therapy,
ii) treating a wider population, and
iii) reducing waiting lists.
The Authors would like to thank all the professionals of the Hospitals involved in the analysis for their assistance in the data collection that significantly improved the quality of results presented.
- Angelis A, Kanavos P (2017) Multiple Criteria Decision Analysis (MCDA) for evaluating new medicines in Health Technology Assessment and beyond: The Advance Value Framework. Soc Sci Med 188: 137-156.
- Turchetti G, Spadoni E, Geisler E (2010) Health technology assessment. Engineering in Medicine and Biology Magazine. IEEE Eng Med Biol Mag 29(3): 70-76.
- Drummond MF, Schwartz JS, Jönsson B, Luce BR, Neumann PJ, (2008) Key principles for the improved conduct of health technology assessments for resource allocation decisions. Int J Technol Assess Health Care 24(3): 244-258.
- (2016) EUNetHTA The HTA Core Model, Version3.0, April 2016.
- Jonsson E, Banta HD (1999) Management of health technologies: an international view. BMJ 319: 1293-NIL_42.
- Kidholm K, Ehlers L, Korsbek L, Kjaerby R, Beck M, et al. (2009) Assessment of the quality of mini-HTA. Int J Technol Assess Health Care 25(1): 42-48.
- (2005) The Danish Centre for Evaluation and Health Technology. Introduction to Mini-HTA - - a management and decision support tool for the hospital service. 2005.
- Foglia E, Lettieri E, Ferrario L, Emanuele Porazzi, Elisabetta Garagiola, et al. (2017) Technology assessment in hospitals: lessons learned from an empirical experiment. Int J Technol Assess Health Care. 2017 Jun 5: 1-9.
- Thokala P, Duenas A (2012) Multicriteria Decision Analysis for Health Technology Assessment. Value Health 15(8): 1172-1181.
- Thokala P, Devlin N, Marsh K, RobBaltussen, MeindertBoysen, et al. (2016) Multiple Criteria Decision Analysis for Health Care Decision Making - An Introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health 19: 1-13.
- Belton V, Stewart TJ (2002) Multiple Criteria Decision Analysis: An Integrated Approach. Kluwer Academic Publishers, 2002.
- Mussen F, Salek S, Walker S (2007) A quantitative approach to benefit-risk assessment of medicines - part 1: the development of a new model using multi-criteria decision analysis. Pharmacoepidemiol Drug Saf. 16 Suppl 1: S2-S15.
- Phillips L, Fasolo B, Zafiropoulos N, Beyer A (2011) Is quantitative benefit-risk modelling of drugs desirable or possible? Drug Discov. Today. Technol 8: e1-e10.
- Goetghebeur MM, Wagner M, Khoury H, Randy J Levitt, Lonny J Erickson, et al. (2012) Bridging health technology assessment (HTA) and efficient health care decision making with multicriteria decision analysis (MCDA): applying the EVIDEM framework to medicines appraisal. Med. Decis. Making 32 (2): 376-388.
- Felli, Rebecca A Noel, Patrizia A Cavazzoni (2009) A multiattribute model for evaluating thebenefit-risk profiles of treatment alternatives. Med. Decis. Making 29 (1): 104-115.
- Tervonen, T, Naci H, van Valkenhoef G, Aris Angelis, Hans L. Hillege, et al. (2015) Applying multiple criteria decision analysis to comparative benefit-risk assessment: choosing among statins in primary prevention. Int. J Soc Med Decis Mak 35(7): 859-871.
- McCarthy M (2016) Higher priced drugs are not cost effective for diabetic macular edema. US study finds. BMJ 9: 353.
- Ross EL, Hutton DW, Stein JD (2016) Diabetic Retinopathy Clinical Research Network. Cost-effectiveness of aflibercept, bevacizumab, and ranibizumab for diabetic macular edema treatment analysis from the Diabetic Retinopathy Clinical Research Network comparative effectiveness Trial. JAMA Ophthalmol 134(8): 888-896.
- Wells JA, Glassman AR, Ayala AR (2015) Aflibercept, bevacizumab, or ranibizumab for diabetic macular edema. N Engl J Med 372: 1193-1203.
- Ferrario L, Foglia E, Bandello F, Staurenghi G, Tadini P, et al. (2015) Treatment of macular edema: what’s new? Evidence from an hta study comparing ranibizumab and dexamethasone implant. Value Health 18(7): A428-429.
- Ferrario L, Foglia E, Bandello F, Tobaldi RF, Nicolò M, et al. (2016) A Multi-Criteria Decision Analysis Approach within The Diabetic Macular Edema Field: The Experience of 3 Italian Regions. Value Health 19(7): A573.
- Ferrario L, Foglia E, Bandello F, Nuti E, Fusco F, et al. (2017) HTA & MCDA In the Treatment of Macular Edema. Value Health 20(9): A810.
- Foglia E, Ferrario L, Bandello F, Camilla Ferri, Innocente Figini , et al. (2017) Diabetic macular edema, innovative technologies and economic impact: New opportunities for the Lombardy Region healthcare system? Acta Ophalmologica. 96(4): e468-e474.
- Boyer D, Yonn Y, Belfort R, FrancescoBandello, Raj K Maturi, et al. (2014) Three-Year, Randomized, Sham-Controlled Trial of Dexamethasone Intravitreal Implant in Patients with Diabetic Macular Edema. Ophthalmology 121(10): 1904 -1914.
- Mitchell P, Bandello F, Schmidt-Erfurth U, Gabriele E Lang, Pascale Massin, et al. (2011) The RESTORE Study Ranibizumab Monotherapy or Combined with Laser versus Laser Monotherapy for Diabetic Macular Edema. Ophthalmology 118(4): 615-625.
- Korobelnik J, Do D, Erfurth U, Ursula Schmidt-Erfurth, Oliver Zeitz, et al. (2014) Intravitreal Aflibercept for Diabetic Macular Edema. Ophthalmology 121(11): 2247-2254.
- Radaelli G, Lettieri E, Masella C, Merlino L, Strada A, et al. (2014) Implementation of Eunethta core model® in Lombardy: The VTS framework. International journal of technology assessment in health care, 30(01): 105-112.
- Martelli N, Billaux M, Borget I, Pineau J, Prognon P, et al. (2015) Introduction of innovative medical devices at French university hospitals: an overview of hospital-based health technology assessment initiatives. Int. J. Technol. Assess. Health Care 31 (1-2): 12-18.
- Diaby V, Goeree R (2014) How to use multi-criteria decision analysis methods for reimbursement decision-making in healthcare: a step-by-step guide, Exp. Rev. Pharmacoeconom. Outcomes Res 14 (1): 81-99.