Skip to main content

Reduced risks of influenza-associated hospitalization and complications following vaccination among over 2 million older individuals: a nationwide study using target trial emulation framework

Abstract

Background

Current evidence on influenza vaccine effectiveness (VE), which is predominately derived from small high-risk older populations and focuses on specific influenza-related complications, might not be generalizable to real-world older populations with diverse characteristics in Taiwan. Therefore, this observational study with a target trial emulation framework aimed to evaluate the clinical effectiveness of an influenza vaccine on influenza infection, complications, and associated healthcare utilization and costs.

Methods

1,214,392 propensity-score-matched pairs of vaccinated and unvaccinated older populations from the influenza season of 2018/2019 were identified from Taiwan’s National Health Insurance Research Database. VE (estimated as 1 minus hazard ratio [HR]*100%) and the HRs were used for influenza events and associated complications, respectively.

Results

Primary analyses show 14% (10–18%) of VE against influenza-associated hospitalization, irrespective of age, frailty status, and underlying influenza risk. Notably, a decline in VE for influenza-associated hospitalization was observed when the observational period following vaccination was extended (25% [19–30%], 23% [18–28%], and 14% [10–18%] for the intervals October 2018–March 2019, October 2018–May 2019, and October 2018–September 2019, respectively). Compared with non-vaccination, having an influenza vaccination significantly reduced risks of influenza-associated death by 30%, various respiratory by 12–26%, cardiovascular complications by 39–47%, and acute kidney injury by 23%. Approximately savings of USD 3,000,000 in total from averting influenza-associated hospitalization following vaccination were found. The non-significant effects of the influenza vaccine on negative control outcomes support the validity of the study procedures.

Conclusions

VE for severe influenza events (i.e., those requiring hospitalization) and related complications among the real-world older population was corroborated. To avoid severe influenza episodes and complications and minimize associated economic consequences, continuous influenza vaccine uptake over different influenza seasons is recommended for this population.

Peer Review reports

Background

People aged ≥ 65 years are at an increased risk of influenza infection and development of serious influenza-related complications (e.g., cardiovascular [CV], severe pulmonary, and kidney diseases, and death [1]) due to the immunosenescence, multiple chronic health problems, and frailty in this population [1]. In Taiwan, an influenza vaccine is therefore government-funded and strongly recommended to adults aged 65 years or above for reducing influenza transmission as well as averting influenza-associated morbidities and mortalities [2]. Despite these efforts, the uptake rate of the influenza vaccine among the older population in routine clinical practice settings remains suboptimal (e.g., around 50–60% [2, 3]).

Influenza vaccine effectiveness (VE) is commonly reported to inform personal decision-making on vaccination [4,5]. However, there are several caveats regarding current evidence for older populations. First, despite substantial evidence supporting VE in the general older population worldwide from annual epidemiological reports of sentinel surveillance systems [6] and the potential vaccine benefits on avoiding influenza-related complications [7,8,9,10], only the subsets (i.e., those with gout [9], disability [11], or breast cancer [12]) or limited numbers (i.e., thousands [7]) of general older populations in Taiwan were analyzed. Therefore, current VE evidence might not be generalizable to general older populations in Taiwan to support decision-making. Second, the concern of residual confounding by indication and healthy vaccinee bias might remain in previous observational studies, with case–control or cohort [5] designs used, leading to imprecise VE and thereby undermining the study validity and confidence to support clinical decision-making [13]. Recently, the target trial emulation framework has been extensively employed in the field of vaccine studies (e.g., COVID-19 vaccine [14]), and this approach comprises an explicit study design framework along with rigorous methodologies which could minimize the possibility of confounding effects and biases commonly seen in the observational studies of real-world data [15]. Lastly, the impact of the uptake of the influenza vaccine on healthcare utilization (e.g., medical costs and length of hospital stay) in Taiwan is also unclear.

Against this background, the present study sought to determine the effectiveness of a standard-dose influenza vaccine on influenza infection, a series of influenza-related complications (i.e., CV, respiratory, and kidney diseases and death), and healthcare utilization and costs in adults aged ≥ 65 years using a target trial emulation approach. Empirical evidence from this study in particular vaccine effectiveness beyond influenza infection (i.e., associated complications and economic consequences) is important for facilitating personal decision-making on vaccination, improving the vaccine uptake rate, and maximizing the value of an influenza vaccine in real-world practice.

Methods

Emulation of hypothetical trial using nationwide claims data

This was a retrospective cohort study using a target trial emulation framework. Our study design for the target trial was adapted according to a published randomized, double-blind, clinical trial [16]. The specifications of the target trial are detailed in Additional file 1: Table 1.

Taiwan’s National Health Insurance Research Database (NHIRD) for 2017–2019 was utilized. The National Health Insurance program in Taiwan covers healthcare services (e.g., outpatient, emergency room [ER] visits, hospitalization, and medication prescriptions) for over 99% of Taiwan’s population. Health-related information is therefore longitudinally collected and recorded in the NHIRD. The health records in the NHIRD are individual-level and de-identified. Details of the NHIRD are available elsewhere [17]. This study was approved by the Institutional Review Board of National Cheng Kung University (111–458-2).

Study population

All individuals in Taiwan aged ≥ 65 years have been eligible for a government-funded influenza vaccine since 2001 [18]. The government-funded vaccine is available each year from October until it runs out or the end of September the following year. Annually, 40% to 60% of older adults received the vaccination; of those vaccinated, > 90% of the vaccinations were administered in October through December. The present study identified all subjects aged ≥ 65 years in the NHIRD for 2018 as the study cohort. Subjects who were first vaccinated in October, November, or December 2018 were placed into the vaccinated group [11, 12]. Those vaccinated in other months (i.e., January to September 2019) or not vaccinated were placed in the unvaccinated group. To emulate the target trial, the eligibility for vaccination was assessed for each older individual (Additional file 1: Table 1). A sequential trial approach [19] (Additional file 1: Fig. S1) with a two-step propensity score (PS) matching was performed to enhance the comparability between the vaccinated and unvaccinated groups regarding baseline characteristics. In the first step, vaccinated subjects were matched with unvaccinated subjects based on age, gender, and city/county in a 1:n ratio. A vaccinated subject was matched with multiple unvaccinated subjects (as much as possible) to ensure similar accessibility to healthcare within matched individuals. In the second step, which was performed on the matched pairs in each month as a stratum (i.e., October, November, and December 2018), 1:1 PS-matched pairs of vaccinated and unvaccinated subjects were obtained using 8-to-1-digit greedy matching. Of note, the index date for a vaccinated subject was the date of receiving the influenza vaccine, whereas that for an unvaccinated subject was set to October 15, November 15, or December 15 based on their matched month stratum to minimize the immortal time bias.

Details of the inclusion and exclusion criteria and matching procedure [20] and the operational definitions [21, 22] of the baseline characteristics are given in Additional file 1: Fig. S1 and Table 2. Each study subject was followed from the index date until the occurrence of an influenza event, reception of the influenza vaccine (i.e., the second vaccination for the vaccinated group and the first vaccination for the unvaccinated group), death, or the end of influenza season (i.e., September 30, 2019), whichever came first (i.e., observational analog of the per-protocol scenario).

Measurements of vaccination status and influenza outcomes

Vaccination status was ascertained from one of the following records in the NHIRD: (1) reception of the influenza vaccine, (2) reimbursement for vaccine administration, or (3) outpatient visit for influenza vaccination, indicating that the influenza vaccination was reimbursed by the Taiwan Center for Disease Control. The main study outcomes were (1) influenza-associated hospitalization, which was defined based on influenza diagnosis codes (the International Classification of Diseases, Tenth Revision, Clinical Modification: J09-J11); (2) influenza-associated outpatient visits, which were defined based on influenza diagnosis codes (J09-J11) with antiviral drug use; and (3) influenza-associated ER visits, which were defined based on influenza diagnosis codes (J09-J11) with antiviral drug use. Because systematic inflammatory responses triggered by influenza events [23] could increase the risks of severe pulmonary [24], CV [25], and kidney diseases [26], these clinical conditions or complications are likely to occur following influenza infection among the older population [27, 28]. The prevention of influenza infection episodes through vaccination is, therefore, crucial to avoid the occurrence of these complications [27, 28]. In this regard, we included influenza-associated complications that occurred during influenza-associated hospitalization as study outcomes of interest. These complications included pneumonia, acute respiratory distress syndrome (ARDS) with ventilator use, sepsis, acute myocardial infarction (AMI), stroke, acute kidney injury, and death, which was ascertained from the Cause of Death files in the NHIRD. Of note, given that the protective effect of a vaccine generally starts 14 days after vaccination [29], study events that occurred ≥ 14 days after the index date were measured. Details of the operational definitions of the study outcomes are available in Additional file 1: Table 3.

Statistical analysis

The standard mean difference (SMD) was utilized to assess the between-group comparability in patient baseline characteristics. An absolute value of SMD ≥ 0.1 was considered to indicate a statistically significant between-group imbalance. The event rate of a study outcome was estimated as the number of events divided by 1000 person-years. The healthcare costs associated with influenza-associated healthcare utilization, including hospital admissions (and corresponding length of stay, measured in days) and outpatient and ER visits, were estimated from the perspective of the healthcare sector. Costs were standardized into 2022 values using the medical component of the consumer price index in Taiwan and are presented in United States dollars (USD). Student’s t-test was used to evaluate the between-group difference in healthcare costs. The Cox proportional hazard model was employed to assess the risk of the study outcome (e.g., influenza infection) with vaccination status. Any unbalanced variables between vaccinated and unvaccinated groups were further treated as covariates and adjusted in the Cox model analysis. The results are presented as hazard ratios (HRs) and associated 95% confidence intervals (CIs). The VE was then calculated as (1 − HR)*100%.

A series of sensitivity analyses were performed to test the robustness of the study findings in the primary analyses. First, to enhance the validity of the measurement of the study outcomes, influenza-associated hospitalization was determined according to the principal diagnosis codes for influenza, and influenza-associated outpatient and ER visits were defined based on the principal diagnosis codes for influenza with the prescription of an antiviral drug. Second, considering influenza occurs throughout the year in Taiwan, the analyses focused on the peak season (month) of influenza, where the end of the follow-up period was restricted to March, April, or May 2019, respectively, to assess whether the VE waned following influenza vaccination [30]. Third, considering the potential of unmeasured confounders (i.e., health awareness), negative control analyses were conducted [31], where influenza events that occurred 7 or 13 days following vaccination, influenza events in the sampled cohort with 50,000 subjects obtained from a vaccine-mismatched season (i.e., influenza season of 2014 or 2015 [32]) using the same patient selection procedures (i.e., the selection of patient cohort followed the target trial emulation framework), and traffic-accident-related hospitalization [30] were treated as study outcomes. No significant risks of these control events with vaccination status were expected (i.e., 95% CI of HR overlapping 1), ensuring the validity of our study materials and procedures. Lastly, given the possibility of the presence of immunosenescence in geriatric populations [33], the modification effect of patients’ baseline characteristics (e.g., age) on vaccination outcomes (i.e., VE) cannot be ruled out. Therefore, a series of interaction tests were carried out for various patient characteristics, including gender (i.e., female or male), age (i.e., ≥ or < 75 years), frailty (i.e., fit or frail), and influenza risk (i.e., presence or absence of high-risk disease conditions such as infectious diseases, blood disorders, and endocrinologic disorders, as specified by Taiwan’s Center for Disease Control), and joint subgroups (i.e., aged ≥ or < 75 years and fit or frail, and aged ≥ or < 75 years and with or without high risk of influenza infection). A two-tailed p-value of < 0.05 was considered to indicate a statistically significant difference. All analyses mentioned above were conducted using SAS software version 9.4.

Results

We identified 3,394,238 subjects aged ≥ 65 years in the NHIRD for 2018 and 2019. After the study eligibility and matching procedures were applied, 1,214,392 pairs of vaccinated and unvaccinated subjects were obtained for the analysis (Additional file 1: Fig. 2). The baseline characteristics of the study cohort before PS matching are provided in Additional file 1: Tables 4–6. Table 1 shows satisfactory between-group comparability in the baseline characteristics (SMD less than 0.1), except for the history of government-funded health examination in the year prior to the index date. In general, the study population had a mean age of 74.3 years and was 46.1% male.

Table 1 Baseline characteristics of the overall study population after two-step propensity score matching

In Fig. 1, the primary analyses show that the event rate of influenza-associated hospitalization was 3.12 and 3.67 per 1000 person-years for vaccinated and unvaccinated subjects, respectively, resulting in a VE of 14% on influenza-associated hospitalization (i.e., adjusted HR [aHR]: 0.86, 95% CI: 0.82–0.90). The event rates of influenza-associated outpatient/ER visits were 0.52/0.20 and 0.56/0.22 for vaccinated and unvaccinated subjects, respectively, but the VE values estimated from these events were insignificant (i.e., VE: 10%/10%, aHRs: 0.90/0.90, 95% CIs: 0.82–1.03/0.75–1.08). The results of sensitivity analyses that assessed influenza events based on principal diagnostic codes and restricted influenza seasons (i.e., October 2018 to the end of March, April, or May 2019) were consistent with the primary analysis findings, showing significant VE for influenza-associated hospitalization (i.e., VE/aHRs [95% CIs]: 13%/0.87 [0.83–0.93], 25%/0.75 [0.70–0.81], 23%/0.77 [0.72–0.82], and 21%/0.79 [0.74–0.84], respectively). No significant effect of influenza vaccination on negative control outcomes was observed (i.e., VE: − 1% [− 42%, 28%] for influenza events that occurred within 7 days following vaccination, 0% [− 29%, 23%] for influenza events within 13 days following vaccination, 11% [− 3%, 22%] for influenza events in the sampled cohort from a mismatched season, and aHR: 1.01 [0.97, 1.05] for traffic accident-related hospitalization). Details of the event rates are given in Additional file 1: Table 7.

Fig. 1
figure 1

Forest plot of adjusted hazard ratios and associated vaccine effectiveness in primary, sensitivity, and negative control outcome analyses. Abbreviations: HR, hazard ratio; CI, confidence interval; ER, emergency room. *The receipt of government-funded health examinations was unbalanced between groups after a two-step propensity score matching and thus was further adjusted in the Cox model. Bold HRs and 95% CIs indicate statistically significant vaccine effectiveness. †Vaccine effectiveness (VE) = (1-HR)*100%. Bold VE values and 95% CIs indicate a statistically significant vaccine protection effect. ‡Influenza-associated outpatient or ER visits were defined as having any diagnosis codes of influenza infection and using antiviral drugs. §500,000 subjects were sampled from the entire elderly population in the influenza season of 2014/2015 (which was a vaccine mismatch season), and the same analytic procedures (e.g., the selection of patient cohort based on a target trial emulation framework) were redone in this cohort to estimate the VE

Figure 2 shows significant interactions of VE with age and multimorbidity frailty. That is, the VE (95% CIs) for influenza-associated hospitalization was 23% (18–29%) and 11% (5–16%) for subjects aged < and ≥ 75 years, respectively (p-value for interaction < 0.0001), and 27% (19–33%) and 11% (7–16%) for fit and frail subjects, respectively (p-value for interaction = 0.001). The joint subgroup analyses also indicate significant interaction of vaccine status with multimorbidity frailty in subjects aged < 75 years; i.e., the VE for influenza-associated hospitalization was 35% (25–43%) and 19% (12–26%) for fit and frail subjects, respectively (p-value for interaction = 0.011). Details of the event rates and associated aHRs and 95% CIs are given in Additional file 1: Table 8.

Fig. 2
figure 2

Vaccine effectiveness for influenza hospitalization across subgroup and joint subgroup analyses. Abbreviations: CI, confidence interval. *Vaccine effectiveness (VE) = (1 − HR)*100%. Bold VE values and 95% CIs indicate a statistically significant vaccine protection effect. Details of event rates and hazard ratios of vaccinated versus unvaccinated subjects for influenza hospitalization are available in Additional file 1: Table 7. †According to Taiwan’s Center for Disease Control, individuals with one of the following disease histories determined within one year before the index date were defined as subjects at high risk of influenza infection: infectious diseases, blood disorders, endocrinologic disorders, neurological disorders, cardiovascular diseases, respiratory diseases, digestive system diseases, musculoskeletal system, and connective tissue diseases, urogenital system disease, congenital malformations, and chromosomal abnormalities

Table 2 shows the detailed event rates and associated aHRs (95% CIs) for clinical complications that occurred during influenza hospitalization. Compared with non-vaccination, reception of the influenza vaccine was associated with significantly reduced risks of influenza-associated death (aHR: 0.70 [95% CI: 0.56–0.88]), infectious/pulmonary diseases (i.e., 0.88 [0.81–0.96], 0.85 [0.74–0.98], and 0.34 [0.19–0.61]) for pneumonia, sepsis, and ARDS with ventilator use, respectively), CV diseases (i.e., 0.61 [0.39–0.95] and 0.53 [0.38–0.75] for AMI and stroke, respectively), and kidney diseases (i.e., 0.77 [0.61, 0.96] for acute kidney injury).

Table 2 Event rates and associated hazard ratios of vaccination versus non-vaccination for clinical complications that occurred during influenza hospitalization

Figure 3 shows significantly lower influenza-associated hospitalization costs (per admission) (i.e., $1866 versus $2377, p-value < 0.0001) and marginally significantly higher influenza-associated outpatient ($18 versus $16, p-value = 0.0215) and ER ($132 versus $112, p-value = 0.0789) costs (per visit) in vaccinated subjects compared with those for unvaccinated subjects. The results of influenza-associated healthcare costs and length of hospital stay are detailed in Additional file 1: Table 9.

Fig. 3
figure 3

Histogram of influenza-associated healthcare costs per event stratified by vaccine status. *p-values indicate statistical between-group difference in healthcare costs

Discussion

This study of over 2 million older individuals showed the beneficial effect of influenza vaccination on influenza-associated hospitalization and a wide range of related complications, including infectious/pulmonary, CV, and kidney diseases and death. These findings extend the current evidence, which mostly focuses on high-risk older populations, to the general older population in real-world settings. We adopted a target trial emulation framework and a series of sensitivity analyses, which not only enhanced the transparency of study design and procedures but also guaranteed high-quality study results. This strengthens confidence in the reported VE and associated clinical/economic benefits, encouraging individuals to receive an influenza vaccination.

Comparison of vaccine effectiveness between previous studies and this study

In the present study, for the older population in the influenza season of 2018–2019, the estimated VE values (95% CIs) for influenza-associated hospitalization, outpatient visits, and ER visits were 14% (10%, 18%), 10% (− 3%, 18%), and 10% (− 8%, 25%), respectively, which fall in the range of VE values reported in previous studies (i.e., 12% [− 31%, 40%] [4] to 26% [20%, 31%] [5]). However, caution should be taken when comparing findings across studies due to differences in the influenza viruses circulating worldwide, which are affected by antigenic drift in local or regional geographic areas, operational definitions of influenza events (i.e., laboratory-confirmed influenza [4, 5] versus clinical diagnoses in the present study), and study procedures (i.e., previous cohort [5] or test-negative case–control studies [4] versus the present study using a target trial emulation design) across studies.

In this study, a substantial effort was made (in terms of methodology) to minimize the confounding effects and biases that are commonly seen in studies [4, 5]. First, a series of sensitivity analyses that restricted influenza events to those confirmed by the principal diagnosis were carried out. The results of these analyses were consistent with the primary findings, strengthening confidence in VE in real-world settings and providing clinical insights to facilitate real-world decision-making. Specifically, in these sensitivity analyses, statistically significant protection by influenza vaccination was only shown for influenza-associated hospitalization (i.e., VE [95% CI]: 13% [7%, 17%]), but not for influenza-associated outpatient and ER visits (9% [− 3%, 20%] and 12% [− 12%, 31%], respectively). These findings imply that influenza vaccination may be effective in alleviating the severity of influenza infection (e.g., avert severe cases that require hospitalization), but it does not decrease infection episodes or mild cases (e.g., influenza-associated outpatient and ER visits) [34,35,36]. Second, we performed several sensitivity analyses using negative control outcomes. The non-significant results supported the success of implementing a target trial emulation design with two-step PS matching in eliminating the concern of unmeasured confounders.

Lastly, considering the possibility of VE attenuation over time following vaccination in older populations [37], sensitivity analyses were conducted to restrict the study follow-up period to different lengths of the peak influenza months in winter (October 2018 to March 2019, October 2018 to April 2019, October 2018 to May 2019, October 2018 to September 2019). It was found that the VE for influenza-associated hospitalization decreased as the time interval increased (i.e., 25% [19%, 30%], 23% [18%, 28%], 21% [16%, 26%], and 14% [10%, 18%] for the considered influenza season lengths, respectively). These results suggest the importance of continuous influenza vaccine uptake over influenza seasons for enhancing immunogenicity against influenza infection in older populations. Also, the provision of high-dose or adjuvant influenza vaccines [38] and the development of a new vaccine platform (e.g., mRNA) [39] are suggested to achieve optimal protection against influenza infection among older individuals.

Variation of VE for influenza hospitalization by age and multimorbidity frailty in older patients

In this study, aging (which is typically associated with immunosenescence) and frailty were found to be strong effect modifiers. There was a large disparity in the VE values across the subgroups stratified by these two variables. That is, subjects who were younger (i.e., 65 ≤ age < 75 years) or fit in the multimorbidity frailty category had higher VE values compared with those of their counterparties (i.e., aged ≥ 75 years and frail), as supported by significant interaction results (p for interaction < 0.05, Fig. 1). These findings were confirmed by interaction tests in the joint subgroup analysis for subjects aged < 75 years and fit or frail. Therefore, these results indicate that the underlying health status (e.g., old age, frailty) may affect VE even in a season with a good match between the vaccine and circulating strains [40, 41]. Young and healthy subjects may have greater VE than those old and frail subjects. Nevertheless, we found that reception of the influenza vaccine was statistically significantly associated with a reduced risk of influenza-associated hospitalization, irrespective of age, frailty status, and with and without high risk of influenza infection; e.g., although low VE values (95% CIs) were obtained for subjects aged 75 years (i.e., 6.6% [1.0%, 11.9%]), frail individuals (11.8% [7.1%, 16.3%]), and high-risk older individuals (13.1% [8.1%, 17.8%]), all estimates were statistically significant. Such specific subgroups in older populations should be prioritized for receiving high-dose, or adjuvanted influenza vaccines whenever those are available in Taiwan. Moreover, the caregivers of these subgroups are recommended to be vaccinated to optimize protection from influenza infection for older individuals through the cocoon strategy [42]. Also, the number needed to treat (NNT) results of influenza vaccination for influenza infection and associated complications can provide explicit insights for clinical decision-making. For example, 47 patients would need to be administered an influenza vaccine relative to non-vaccination for a mean follow-up of 0.8–0.9 years to avert one case having influenza-related hospitalizations (Additional file 1: Table 7). Also, per recommendations from the World Health Organization, a 75% influenza vaccination coverage rate shall be achieved to avoid the annual epidemics, irrespective of age, country, healthcare systems, and race/ethnicity [43]. With consideration of the suboptimal uptake rate of the influenza vaccine in current practice (e.g., 50–60% [2, 3]), promoting the additional benefit of influenza vaccination on the reduction of the CV, pulmonary, and kidney risks to society, adopting high-dose influenza vaccines to the annual vaccination program, and improving vaccine accessibility in rural areas may improve influenza vaccination coverage rates for the general older population, thereby diminishing the disease burden attributable to influenza [40].

Vaccine effectiveness beyond influenza infection in older patients

In addition to a reduced risk of influenza events, the protective effect of influenza vaccination on a wide spectrum of influenza-related complications (i.e., death and infectious/pulmonary, CV, and kidney diseases) was shown in this study. Previous studies only analyzed the additional benefits of the influenza vaccine among specific subgroups (e.g., patients with gout, CV diseases, or respiratory diseases) of older populations [7,8,9,10]. Empirical evidence derived from large-scale general older populations is limited. This study bridges this knowledge gap. It found that influenza vaccination reduces the risks of influenza-related events (i.e., influenza infection, pulmonary disease, and death) by 12–66%, CV diseases by 39–47%, and acute kidney injury by 23%. Mechanisms to support the reduced risks of complications associated with influenza vaccination may be derived from its prevention on influenza infection, which could increase systematic pro-inflammatory cytokines and directly act on vasculature and myocardium, resulting in plaque destabilization and MI or stroke development [44]. Beyond these clinical benefits, the savings from influenza-associated hospitalization following influenza vaccination were also remarkable (i.e., approximately $3,000,000 in total, Additional file 1: Table 9). It is expected that this economic benefit will increase as the uptake rate of the influenza vaccine increases. The savings could be re-allocated to support the universal coverage of the influenza vaccine for the older population and to maintain the national health insurance program.

Study limitations

First, given the implementation of exclusion criteria according to the study target trial setting, our results may not be generalized to the patients excluded from this study. In particular, a certain number of individuals with dementia (~ 6% of the general older population) were excluded. These individuals are vulnerable to and usually have multiple comorbidities [45], and are thus at high risk for influenza infection and associated complications. Influenza events are likely to be under-recognized due to the decline in cognitive function in these patients, affecting the validity of the VE estimates presented in this study. Hence, to understand the VE in patients with dementia, a prospective pragmatic trial design [46] that relaxes the strict trial patient inclusion criteria to accommodate disadvantaged patients in routine care settings could be adopted in the future. Second, despite a large amount (i.e., over two million) of older subjects included in the current study, a certain proportion of the older population was lost in the process of PS matching. Therefore, in future research, other matching methods (e.g., inverse probability of treatment weighting) can be utilized to strengthen the robustness and generalizability of the current study findings. Third, the present study did not include self-paid influenza vaccination because such data are unavailable. However, given the universal coverage of influenza vaccination for individuals aged ≥ 65 years in Taiwan, this concern might be negligible. Also, potential unmeasured confounders (e.g., health awareness) might exist, which were likely to increase the effect size and thereby lead to underestimated VE. To minimize such a concern, we measured several surrogate indicators (e.g., the receipt of health examination and cancer screenings within 1 year prior to the index date) and adjusted them in analysis (e.g., matching procedures). Fourth, the clinical complications that occurred during influenza hospitalizations (Table 2) were likely affected by the timely receipt of antiviral prescriptions, treatment with antibiotics, and receipt of pneumococcal vaccination, which were not measured and adjusted in our analyses. Fifth, there is a lack of laboratory confirmation (e.g., polymerase chain reaction testing) to identify influenza cases. Lastly, the present study was conducted from the perspective of the healthcare sector and, therefore, did not consider non-medical benefits (e.g., the loss of productivity of family members due to having to care for influenza patients) of influenza vaccination. These additional outcomes following vaccination deserve to be included in a future analysis of the overall benefit of influenza vaccination.

Conclusions

This empirical study with a large-scale general older population adds supporting evidence regarding the effects of influenza vaccination on severe influenza events (i.e., those requiring hospitalization), influenza-related complications (i.e., infectious/pulmonary, CV, and kidney diseases and death), and potential health care savings. Beneficial effects were found irrespective of individual age, frailty status, and underlying high risk for influenza infection, thereby promoting a wide adoption of the influenza vaccine in this population. To avert severe infection episodes, undesirable complications, and associated economic consequences while maintaining immunogenicity against influenza, the uptake of annual influenza vaccination is recommended for older populations.

Data availability

Raw data were generated at Taiwan’s National Health Insurance Research Database. Derived data supporting the findings of this study are available from the corresponding author (Dr. Huang-Tz Ou) on request.

Abbreviations

ARDS:

Acute respiratory distress syndrome

AMI:

Acute myocardial infarction

CI:

Confidence interval

CV:

Cardiovascular

ER:

Emergency room

HR:

Hazard ratio

NHIRD:

National Health Insurance Research Database

NNT:

Number needed to treat

PS:

Propensity score

SMD:

Standard mean difference

USD:

United States dollar

VE:

Vaccine effectiveness

References

  1. Langer J, Welch VL, Moran MM, Cane A, Lopez SMC, Srivastava A, Enstone AL, Sears A, Markus KJ, Heuser M, Kewley RM, Whittle IJ. High Clinical Burden of Influenza Disease in Adults Aged ≥ 65 Years: Can We Do Better? A Systematic Literature Review Adv Ther. 2023;40(4):1601–27. https://doiorg.publicaciones.saludcastillayleon.es/10.1007/s12325-023-02432-1.

    Article  PubMed  Google Scholar 

  2. Su CP, Tsou TP, Chen CH, Lin TY, Chang SC. Influenza Control Group; Infectious Disease Control Advisory Committee. Seasonal influenza prevention and control in Taiwan-Strategies revisited. J Formos Med Assoc. 2019;118(3):657–63. https://doiorg.publicaciones.saludcastillayleon.es/10.1016/j.jfma.2018.12.022.

    Article  PubMed  Google Scholar 

  3. Flu Vaccination Coverage, United States, 2020–21 Influenza Season. Centers for Disease Control and Prevention; 2021. Available from: https://www.cdc.gov/flu/fluvaxview/coverage-2021estimates.htm. Accessed 19 Feb 2024.

  4. Flannery B, Kondor RJG, Chung JR, Gaglani M, Reis M, Zimmerman RK, Nowalk MP, Jackson ML, Jackson LA, Monto AS, Martin ET, Belongia EA, McLean HQ, Kim SS, Blanton L, Kniss K, Budd AP, Brammer L, Stark TJ, Barnes JR, Wentworth DE, Fry AM, Patel M. Spread of Antigenically Drifted Influenza A(H3N2) Viruses and Vaccine Effectiveness in the United States During the 2018–2019 Season. J Infect Dis. 2020;221(1):8–15. https://doiorg.publicaciones.saludcastillayleon.es/10.1093/infdis/jiz543.

    Article  CAS  PubMed  Google Scholar 

  5. Baum U, Kulathinal S, Auranen K. Spotlight influenza: Estimation of influenza vaccine effectiveness in elderly people with assessment of residual confounding by negative control outcomes, Finland, 2012/13 to 2019/20. Euro Surveill. 2021;26(36):2100054. https://doiorg.publicaciones.saludcastillayleon.es/10.2807/1560-7917.ES.2021.26.36.2100054.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Annual epidemiological reports. European Centre for Disease Prevention and Control; 2024. Available from: https://www.ecdc.europa.eu/en/publications-data/monitoring/all-annual-epidemiological-reports. Assessed 25 Nov 2024.

  7. Cheng Y, Cao X, Cao Z, Xu C, Sun L, Gao Y, Wang Y, Li S, Wu C, Li X, Wang Y, Leng SX. Effects of influenza vaccination on the risk of cardiovascular and respiratory diseases and all-cause mortality. Ageing Res Rev. 2020;62:101124. https://doiorg.publicaciones.saludcastillayleon.es/10.1016/j.arr.2020.101124.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Yedlapati SH, Khan SU, Talluri S, Lone AN, Khan MZ, Khan MS, Navar AM, Gulati M, Johnson H, Baum S, Michos ED. Effects of Influenza Vaccine on Mortality and Cardiovascular Outcomes in Patients With Cardiovascular Disease: A Systematic Review and Meta-Analysis. J Am Heart Assoc. 2021;10(6):e019636. https://doiorg.publicaciones.saludcastillayleon.es/10.1161/JAHA.120.019636.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Chen CC, Chou KT, Liu JC, Chiu CC, Yang TY, Lin CH, Fang YA, Jian W, Lei MH, Yeh HT, Hsu MH, Hao WR. Association between Stroke Risk and Influenza Vaccination in Patients with Gout: A Nationwide Population-Based Study. Vaccines (Basel). 2022;10(8):1278. https://doiorg.publicaciones.saludcastillayleon.es/10.3390/vaccines10081278.

    Article  CAS  PubMed  Google Scholar 

  10. Pang Y, Wang Q, Lv M, Yu M, Lu M, Huang Y, Wu J, Xie Z. Influenza Vaccination and Hospitalization Outcomes Among Older Patients With Cardiovascular or Respiratory Diseases. J Infect Dis. 2021;223(7):1196–204. https://doiorg.publicaciones.saludcastillayleon.es/10.1093/infdis/jiaa493.

    Article  CAS  PubMed  Google Scholar 

  11. Chang YC, Yu-Tung H, Chen LS, Tung HJ, Huang KH, Ernawaty E, Wu SY. Protective Effect of Seasonal Influenza Vaccination in Elderly Individuals with Disability in Taiwan: A Propensity Score-Matched, Nationwide, Population-Based Cohort Study. Vaccines (Basel). 2020;8(1):140. https://doiorg.publicaciones.saludcastillayleon.es/10.3390/vaccines8010140.

    Article  PubMed  Google Scholar 

  12. Wu SY, Tung HJ, Huang KH, Lee CB, Tsai TH, Chang YC. The Protective Effects of Influenza Vaccination in Elderly Patients with Breast Cancer in Taiwan: A Real-World Evidence-Based Study. Vaccines (Basel). 2022;10(7):1144. https://doiorg.publicaciones.saludcastillayleon.es/10.3390/vaccines10071144.

    Article  PubMed  Google Scholar 

  13. Remschmidt C, Wichmann O, Harder T. Frequency and impact of confounding by indication and healthy vaccinee bias in observational studies assessing influenza vaccine effectiveness: a systematic review. BMC Infect Dis. 2015;15:429. https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12879-015-1154-y.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Hulme WJ, Williamson E, Horne EMF, Green A, McDonald HI, Walker AJ, Curtis HJ, Morton CE, MacKenna B, Croker R, Mehrkar A, Bacon S, Evans D, Inglesby P, Davy S, Bhaskaran K, Schultze A, Rentsch CT, Tomlinson L, Douglas IJ, Evans SJW, Smeeth L, Palmer T, Goldacre B, Hernán MA, Sterne JAC. Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data. Ann Intern Med. 2023;176(5):685–693. https://doiorg.publicaciones.saludcastillayleon.es/10.7326/M21-4269. Epub 2023 May 2. Erratum in: Ann Intern Med. 2023;176(7):1016.

  15. Hernán MA, Wang W, Leaf DE. Target Trial Emulation: A Framework for Causal Inference From Observational Data. JAMA. 2022;328(24):2446–7. https://doiorg.publicaciones.saludcastillayleon.es/10.1001/jama.2022.21383.

    Article  PubMed  Google Scholar 

  16. DiazGranados CA, Dunning AJ, Kimmel M, Kirby D, Treanor J, Collins A, Pollak R, Christoff J, Earl J, Landolfi V, Martin E, Gurunathan S, Nathan R, Greenberg DP, Tornieporth NG, Decker MD, Talbot HK. Efficacy of high-dose versus standard-dose influenza vaccine in older adults. N Engl J Med. 2014;371(7):635–45. https://doiorg.publicaciones.saludcastillayleon.es/10.1056/NEJMoa1315727.

    Article  PubMed  Google Scholar 

  17. Hsieh CY, Su CC, Shao SC, Sung SF, Lin SJ, Kao Yang YH, Lai EC. Taiwan’s National Health Insurance Research Database: past and future. Clin Epidemiol. 2019;11:349–58. https://doiorg.publicaciones.saludcastillayleon.es/10.2147/CLEP.S196293.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Meyer D, Shearer MP, Chih YC, Hsu YC, Lin YC, Nuzzo JB. Taiwan’s Annual Seasonal Influenza Mass Vaccination Program-Lessons for Pandemic Planning. Am J Public Health. 2018;108(S3):S188–93. https://doiorg.publicaciones.saludcastillayleon.es/10.2105/AJPH.2018.304527.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Hernán MA, Alonso A, Logan R, Grodstein F, Michels KB, Willett WC, Manson JE, Robins JM. Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology. 2008;19(6):766–79. https://doiorg.publicaciones.saludcastillayleon.es/10.1097/EDE.0b013e3181875e61.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Parsons L. Reducing Bias in a Propensity Score Matched-Pair Sample Using Greedy Matching Techniques. SAS Institute Inc.; Available from: https://support.sas.com/resources/papers/proceedings/proceedings/sugi26/p214-26.pdf. Accessed 10 Dec 2022.

  21. Sundararajan V, Henderson T, Perry C, Muggivan A, Quan H, Ghali WA. New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality. J Clin Epidemiol. 2004;57(12):1288–94. https://doiorg.publicaciones.saludcastillayleon.es/10.1016/j.jclinepi.2004.03.012.

    Article  PubMed  Google Scholar 

  22. Lai HY, Huang ST, Chen LK, Hsiao FY. Development of frailty index using ICD-10 codes to predict mortality and rehospitalization of older adults: An update of the multimorbidity frailty index. Arch Gerontol Geriatr. 2022;100:104646. https://doiorg.publicaciones.saludcastillayleon.es/10.1016/j.archger.2022.104646.

    Article  PubMed  Google Scholar 

  23. Tavares LP, Teixeira MM, Garcia CC. The inflammatory response triggered by Influenza virus: a two edged sword. Inflamm Res. 2017;66(4):283–302. https://doiorg.publicaciones.saludcastillayleon.es/10.1007/s00011-016-0996-0.

    Article  CAS  PubMed  Google Scholar 

  24. Herold S, Becker C, Ridge KM, Budinger GR. Influenza virus-induced lung injury: pathogenesis and implications for treatment. Eur Respir J. 2015;45(5):1463–78. https://doiorg.publicaciones.saludcastillayleon.es/10.1183/09031936.00186214.

    Article  CAS  PubMed  Google Scholar 

  25. Madjid M, Naghavi M, Litovsky S, Casscells SW. Influenza and cardiovascular disease: a new opportunity for prevention and the need for further studies. Circulation. 2003;108(22):2730–6. https://doiorg.publicaciones.saludcastillayleon.es/10.1161/01.CIR.0000102380.47012.92.

    Article  PubMed  Google Scholar 

  26. Watanabe T. Renal complications of seasonal and pandemic influenza A virus infections. Eur J Pediatr. 2013;172(1):15–22. https://doiorg.publicaciones.saludcastillayleon.es/10.1007/s00431-012-1854-x.

    Article  PubMed  Google Scholar 

  27. Trombetta CM, Kistner O, Montomoli E, Viviani S, Marchi S. Influenza Viruses and Vaccines: The Role of Vaccine Effectiveness Studies for Evaluation of the Benefits of Influenza Vaccines. Vaccines (Basel). 2022;10(5):714. https://doiorg.publicaciones.saludcastillayleon.es/10.3390/vaccines10050714.

    Article  CAS  PubMed  Google Scholar 

  28. Uyeki TM, Hui DS, Zambon M, Wentworth DE, Monto AS. Influenza Lancet. 2022;400(10353):693–706. https://doiorg.publicaciones.saludcastillayleon.es/10.1016/S0140-6736(22)00982-5.

    Article  CAS  PubMed  Google Scholar 

  29. Hu W, Sjoberg PA, Fries AC, DeMarcus LS, Robbins AS. Waning Vaccine Protection against Influenza among Department of Defense Adult Beneficiaries in the United States, 2016–2017 through 2019–2020 Influenza Seasons. Vaccines (Basel). 2022;10(6):888. https://doiorg.publicaciones.saludcastillayleon.es/10.3390/vaccines10060888.

    Article  PubMed  Google Scholar 

  30. Ray GT, Lewis N, Klein NP, Daley MF, Lipsitch M, Fireman B. Depletion-of-susceptibles Bias in Analyses of Intra-season Waning of Influenza Vaccine Effectiveness. Clin Infect Dis. 2020;70(7):1484–6. https://doiorg.publicaciones.saludcastillayleon.es/10.1093/cid/ciz706.

    Article  CAS  PubMed  Google Scholar 

  31. Lipsitch M, TchetgenTchetgen E, Cohen T. Negative controls: a tool for detecting confounding and bias in observational studies. Epidemiology. 2010;21(3):383–8. https://doiorg.publicaciones.saludcastillayleon.es/10.1097/EDE.0b013e3181d61eeb. Erratum.In:Epidemiology. 2010;21(4):589.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Russell CA, Fouchier RAM, Ghaswalla P, Park Y, Vicic N, Ananworanich J, Nachbagauer R, Rudin D. Seasonal influenza vaccine performance and the potential benefits of mRNA vaccines. Hum Vaccin Immunother. 2024;20(1):2336357. https://doiorg.publicaciones.saludcastillayleon.es/10.1080/21645515.2024.2336357.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Haq K, McElhaney JE. Immunosenescence: Influenza vaccination and the elderly. Curr Opin Immunol. 2014;29:38–42. https://doiorg.publicaciones.saludcastillayleon.es/10.1016/j.coi.2014.03.008.

    Article  CAS  PubMed  Google Scholar 

  34. Godoy P, Romero A, Soldevila N, Torner N, Jané M, Martínez A, Caylà JA, Rius C, Domínguez A. Working Group on Surveillance of Severe Influenza Hospitalized Cases in Catalonia Influenza vaccine effectiveness in reducing severe outcomes over six influenza seasons, a case-case analysis, Spain, 2010/11 to 2015/16. Euro Surveill. 2018;23(43):1700732. https://doiorg.publicaciones.saludcastillayleon.es/10.2807/1560-7917.ES.2018.23.43.1700732.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Ferdinands JM, Thompson MG, Blanton L, Spencer S, Grant L, Fry AM. Does influenza vaccination attenuate the severity of breakthrough infections? A narrative review and recommendations for further research. Vaccine. 2021;39(28):3678–95. https://doiorg.publicaciones.saludcastillayleon.es/10.1016/j.vaccine.2021.05.011.

    Article  PubMed  Google Scholar 

  36. Thompson MG, Pierse N, Sue Huang Q, Prasad N, Duque J, Claire Newbern E, Baker MG, Turner N, McArthur C, SHIVERS investigation team. Influenza vaccine effectiveness in preventing influenza-associated intensive care admissions and attenuating severe disease among adults in New Zealand 2012–2015. Vaccine. 2018;36(39):5916–25. https://doiorg.publicaciones.saludcastillayleon.es/10.1016/j.vaccine.2018.07.028.

    Article  PubMed  Google Scholar 

  37. Uemura K, Ono S, Michihata N, Yamana H, Yasunaga H. Duration of influenza vaccine effectiveness in the elderly in Japan: A retrospective cohort study using large-scale population-based registry data. Vaccine. 2023;41(19):3092–8. https://doiorg.publicaciones.saludcastillayleon.es/10.1016/j.vaccine.2023.03.066.

    Article  PubMed  Google Scholar 

  38. Evidence to Recommendations (EtR) Framework: Higher Dose and Adjuvanted Influenza Vaccines for Persons Aged ≥ 65 Years. Advisory Committee on Immunization Practices (ACIP); Available from: https://www.cdc.gov/vaccines/acip/recs/grade/influenza-older-adults-etr.html. Accessed 25 Mar 2024.

  39. Cadar AN, Martin DE, Bartley JM. Targeting the hallmarks of aging to improve influenza vaccine responses in older adults. Immun Ageing. 2023;20(1):23. https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12979-023-00348-6.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Castrucci MR. Factors affecting immune responses to the influenza vaccine. Hum Vaccin Immunother. 2018;14(3):637–46. https://doiorg.publicaciones.saludcastillayleon.es/10.1080/21645515.2017.

    Article  PubMed  Google Scholar 

  41. Gomez Lorenzo MM, Fenton MJ. Immunobiology of influenza vaccines. Chest. 2013;143(2):502–10. https://doiorg.publicaciones.saludcastillayleon.es/10.1378/chest.12-1711.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. Kluczna A, Orzeł R, Bardowska A, Dzierżanowski T. The Degree of Acceptance of Cocoon Strategy of Vaccination against Influenza and COVID-19 in Palliative Home Care Professionals and Caregivers. Vaccines (Basel). 2023;11(7):1235. https://doiorg.publicaciones.saludcastillayleon.es/10.3390/vaccines11071235.

    Article  PubMed  Google Scholar 

  43. World Health Organization. Managing seasonal vaccination policies and coverage in the European Region. 2023 Available from: https://www.who.int/europe/activities/managing-seasonal-vaccination-policies-and-coverage-in-the-european-region. Assessed 25 Nov 2024.

  44. Eur Heart J Suppl. 2023;25(Suppl A):A5-A11. https://doiorg.publicaciones.saludcastillayleon.es/10.1093/eurheartjsupp/suac117.

  45. Chen TB, Yiao SY, Sun Y, Lee HJ, Yang SC, Chiu MJ, Chen TF, Lin KN, Tang LY, Lin CC, Wang PN. Comorbidity and dementia: A nationwide survey in Taiwan. PLoS ONE. 2017;12(4):e0175475. https://doiorg.publicaciones.saludcastillayleon.es/10.1371/journal.pone.0175475.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. Ford I, Norrie J. Pragmatic Trials. N Engl J Med. 2016;375(5):454–63. https://doiorg.publicaciones.saludcastillayleon.es/10.1056/NEJMra1510059.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We are grateful to the Health Data Science Center, National Cheng Kung University Hospital, for providing administrative and technical support.

Funding

This study was supported by a grant from the National Science and Technology Council of Taiwan (grant number NSTC 112–2628-B-006–008-MY3) (recipient: Huang-Tz Ou). The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Author information

Authors and Affiliations

Authors

Contributions

Z.Y.P. designed the study, analyzed and interpreted the data, and wrote the manuscript. Y.T.H. designed the study and analyzed and interpreted the data. W.T.H. designed the study, interpreted the data, and edited the manuscript. J.S.W. designed the study, interpreted the data, and reviewed the manuscript. H.T.O. provided study materials, designed the study, interpreted the data, and wrote the manuscript. All authors approved the final manuscript.

Corresponding author

Correspondence to Huang-Tz Ou.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Institutional Review Board of National Cheng Kung University (111–458-2). The informed consent was waived because this study didn’t involve individual patients.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

12916_2025_3955_MOESM1_ESM.docx

Additional file 1: Table 1. Target trial emulation framework. Table 2. Operational definitions of exclusion criteria, baseline characteristics, government-funded medical examinations, and exposure to cardiovascular/pulmonary medications. Table 3. Operational definitions of clinical outcomes. Table 4. Baseline characteristics of the study population identified from October 2018 before the two-step propensity score matching. Table 5. Baseline characteristics of the study population identified from November 2018 before the two-step propensity score matching. Table 6. Baseline characteristics of the study population identified from December 2018 before the two-step propensity score matching. Table 7. Influenza event rates and vaccine effectiveness in primary, sensitivity, and negative control outcome analyses. Table 8. Event rates and hazard ratios of vaccination versus non-vaccination for influenza hospitalization in subgroup and joint subgroup analyses. Table 9. Descriptive results of influenza-associated healthcare resource utilization stratified by status of vaccination. Fig. S1. Study scheme of sequential trial approach. Fig. S2. Flowchart of cohort selection.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Peng, ZY., Hua, YT., Huang, WT. et al. Reduced risks of influenza-associated hospitalization and complications following vaccination among over 2 million older individuals: a nationwide study using target trial emulation framework. BMC Med 23, 157 (2025). https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12916-025-03955-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12916-025-03955-w

Keywords