Journal of Preventive Medicine And Care

Journal Of Preventive Medicine And Care

Journal of Preventive Medicine and Care

Current Issue Volume No: 1 Issue No: 2

Research Article Open Access Available online freely Peer Reviewed Citation

Using The RE-AIM Framework to Evaluate A Community-Based Smoke Alarm Installation Program

Shelli Stephens-Stidham MPA 1,   Mary A. McCoy MPH 1,   Carey Roper 2,   Emily Campa 2,   Jeffrey J. Barnard MD 3,   Gregory R. Istre MD 1  

1Injury Prevention Center of Greater Dallas (SSS, MAM, GRI)

2Dallas Fire Rescue Department (CR, EC)

3Southwestern Institute of Forensic Sciences, Office of the Medical Examiner (JJB)

Abstract

We used the five dimensions of the RE-AIM model to evaluate a smoke alarm (SA) installation program called Operation Installation (OI), which was implemented in 36 high risk census tracts in Dallas, TX, from 2001-2011. More than 20,000 SAs were installed in 8,134 houses through OI. The RE-AIM model showed that the program had a modest reach into the target population (26.5%), and a relatively high effectiveness (63%) at preventing deaths and injuries in program houses. Adoption and implementation remained high throughout the time period. Individual level maintenance of SAs was high initially (91.8%), but rapidly declined and was only 20% after 10 years.

Application of the RE-AIM model to evaluate this long-term SA installation program highlighted areas that warranted improvement, especially for the dimensions of Reach and individual-level Maintenance. The model may be useful for evaluating the impact of other injury prevention programs.

Author Contributions
Received 20 May 2016; Accepted 25 Jul 2016; Published 27 Jul 2016;

Academic Editor: Prachi Arora, University of Wisconsin - Madison

Checked for plagiarism: Yes

Review by: Single-blind

Copyright ©  2016 Shelli Stephens-Stidham, et al.

License
Creative Commons License     This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Competing interests

The authors have declared that no competing interests exist.

Citation:

Shelli Stephens-Stidham MPA, Mary A. McCoy MPH, Carey Roper, Emily Campa, Jeffrey J. Barnard MD et al. (2016) Using The RE-AIM Framework to Evaluate A Community-Based Smoke Alarm Installation Program. Journal Of Preventive Medicine And Care - 1(2):1-7. https://doi.org/10.14302/issn.2474-3585.jpmc-16-1118

Download as RIS, BibTeX, Text (Include abstract )

DOI 10.14302/issn.2474-3585.jpmc-16-1118

Introduction

Background

Deaths from residential fires and burns remain an important health problem, accounting for 2,520 fire deaths in the U.S. in 2011.1 A smoke alarm (SA) increases the chance of surviving a house fire by 2- to 3-fold.2, 3 While reported SA prevalence is as high as 97% in the U.S. as a whole, houses at highest risk of death and injury from fires continue to have the lowest SA use.4, 5, 6, 7

Based on local data for Dallas, TX,2 we targeted census tracts with the highest per capita rates of house fire-related deaths and injuries and those with lowest median income, for SA installations through a program entitled Operation Installation (OI)2.OI is a joint program of the Injury Prevention Center of Greater Dallas (IPC) and the Dallas Fire Rescue Department (DFRD), and involves door-to-door canvassing and SA installation. A 10-year outcome evaluation of OI from 2001-2011 showed that the adjusted case rate in program houses was 63% lower than non-program houses in the same census tracts.8

However, factors beyond effectiveness can affect program impact.9, 10, 11, 12, 13 The Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework developed by Glasgow, et al provides a comprehensive approach to evaluating an intervention’s overall public health impact.10, 12, 14 In this model, Reach addresses the percentage and representativeness of participants, Effectiveness addresses the impact of the intervention on the targeted outcome, Adoption addresses the proportion and representativeness of settings that deliver the program, Implementation addresses the extent to which the program is delivered as intended, and Maintenance addresses both the individual and organizational (setting level) program delivery over time.10 Of these RE-AIM dimensions, Reach and Effectiveness are individual-level dimensions, Adoption and Implementation are Organizational/setting-level dimensions, and Maintenance has both individual-level and organizational/setting-level dimensions.10

We used the RE-AIM components to measure the impact of OI. Although the RE-AIM model has been applied to a limited number of injury prevention programs, including a Tai Chi program to prevent older adult falls13and sports injury programs,15, 16 to our knowledge, it has never been applied to a long-term SA installation program.

Methods

Operation Installationbegan in 1999 in target census tracts in Dallas that had high rates of house fire-related deaths and injuries. We began systematic data collection for the program in April 2001, and included data through April 2011. Details of the program can be found in a previous paper.8

The five RE-AIM dimensions include Reach, Effectiveness, Adoption, Implementation, and Maintenance, both at the individual level and at the setting level.10, 12 For each RE-AIM dimension, we calculated cumulative results at 2 years and 10 years after initiation.

Reach was calculated by dividing the total number of persons residing in program homes by the total population of the OI census tracts, as previously described.8 Program homes were defined as homes that had received a SA through OI between 2001 and 2011. Non-program homes were all other houses in the same census tracts that did not receive a SA through OI during the same time period.

The representativeness of the OI census tracts was determined by comparing demographic characteristics of the program homes with non-program homes.8

Effectiveness was determined by comparing rates of house fire-related deaths and injuries for program and non-program populations. Details of the effectiveness calculations can be found in a previous paper.8

Adoption was defined as the number of fire departments recruited to participate in the SA distribution.

Implementation was defined as the extent to which the Dallas Fire-Rescue and IPC implemented OI, including the average number of SA’s installed per house. For cost calculations, detailed information regarding time and costs (including materials, salaries, and volunteer hours) were collected during the time period when funding from the Centers for Disease Control and Prevention (CDC) was available – October 2006 through September 2012. We assigned a volunteer hour cost to be $21.79 per hour, which is a standard volunteer hour cost assigned by a Texas government agency. Based on all costs incurred during that time period (2006-2012), we extrapolated the cost of the project for the study period (2001-2011) in 2013 US dollars, the cost per house visited, and cost per SA installed.

Maintenance at the individual level was measured by re-visiting a random sample of homes that had received SAs through OI 2, 4, 6, 8 and 10 years previously, and determining the proportion of SA’s that were still functional. Details of this study have been published by McCoy, et al.17Maintenance at the setting level was determined by the number of canvassing sessions and number of census tracts canvassed during the 10-year period from 2001 through 2011.

Statistical comparison of rates in program and non-program homes for the calculation of effectiveness was done through conditional maximum likelihood estimates of the Rate Ratio, and expressed as Rate Ratio (RR) with corresponding 95% confidence interval (CI) and P-value.

Results

Reach

In the 36 census tracts where OI was conducted, 20,127 SAs were installed in 8,134 homes (28,570 persons reside in those homes). Those 36 census tracts contained 32,480 homes (107,705 persons reside in those homes). The cumulative population reach after two years was 5.7% (6125/107,705) of the total population in the 36 census tracts, and for the 10-year time period the cumulative population reach was 26.5% (28,570/107,705).

In terms of representativeness, individuals in program homes were somewhat more likely to own their home (71.1% vs. 63.7%), have a Hispanic head of household (43.5% vs. 35.5%), and have a head of household over 64 years of age than non-program houses (31.3% vs. 21%). Each of these differences was statistically significant (p < 0.05 by chi square).

Effectiveness

For the first 2 years after SA installation, the rate of deaths and injuries in program houses was 83% lower than non-program houses (1.9 vs. 11.5 deaths and injuries per 100,000 population, respectively; RR, 0.16; 95% CI, 0.01 to 0.89; p< 0.05). For the entire 10-year period, the rate was 63% lower in program houses compared to non-program houses (3.5 vs. 9.5 deaths and injuries per 100,000 population, respectively; RR, 0.32; 95% CI, 0.10 to 0.84, p<.02).8

Adoption/Implementation

Only one fire department (DFRD) was recruited to participate in the program. The intervention was implemented as designed throughout the 10-year period. The average number of SAs installed per house increased from 1.4 during the first two years, to 2.5 for the entire ten years.

The total project cost in 2013 US dollars was $1.14 million; the cost per program home was $169.41, and the cost per SA installed was $56.71.

Maintenance

For the first two years, the program canvassed an average of 5.0 census tracts per year in 8.5 sessions per year. For the entire 10-year period, the program canvassed and average of 3.6 census tracts in 6.2 sessions per year. At the individual level, 91.8% of houses had at least one working SA 2 years following the original installation compared to only 19.9% after 10 years.17The results of the various RE-AIM parameters are summarized in the Table 1.

Table 1. RE-AIM Parameters at 2 years and 10 years, Operation Installation, Dallas, TX 2001-2011
Parameter 2-year 10-year
Reach    
Participation rate: Population in program homes/total population 6,125/107,705 28,570/107,705
(Percent Reached) -0.057 -0.265
Effectiveness    
Death & Injury Rate for Program vs. Non-Program homes* 1.9* vs. 11.5* 3.5* vs. 9.5*
(Percent Effectiveness) -0.83 -0.63
Adoption    
Participation rate at Fire Department 1 of 1 1 of 1
Implementation    
Total no. SA’s installed 2,456 SA’s installed 20,127 SA’s installed
Average no. SA’s installed per house 1.4 SA’s per house 2.5 SA’s per house
Costs of O.I.: N/A Total Cost: $1.141 million
    Cost per home:………….$169.41
    Cost per SA installed: $ 56.71
Maintenance    
Project level    
-Ave. no. census tracts canvassed/year; 5.0 census tracts per year; 3.6 census tracts per year;
-Ave. no. of sessions/ year 8.5 sessions per year 6.2 sessions per year
Individual Level    
- houses with SA’s working 91.8 % of SA working @ 2 years post installation 19.9% of SA working @ 10 years post installation

Discussion

The RE-AIM model has been used to evaluate dimensions of community-based public health programs to determine whether programs are worth sustained investment.9, 10, 11, 12, 13, 18 To our knowledge, it has not been applied to a long-term SA installation program.

Our analysis of OI used the RE-AIM dimensions and found several strengths in the program, including its effectiveness, and its adoption and implementation by the DFRD throughout the 10-year period. The DFRD has now operationalized OI to the point that the program is part of its regular operations. In addition, the cost of the program per program home and per smoke alarm installed, was somewhat lower than previously described.19 This improved efficiency may be related to the large number of SAs installed through OI. Such economies of scale may be important to the cost effectiveness of such programs.19

Our analysis also identified areas for improvement for OI. Although its effectiveness for preventing house fire-related deaths and injuries was high (83% for the first 2 years), it clearly declined after 6 years post installation. This decline in effectiveness may have been due to a decrease in SA function over time. Only approximately 20% of houses still had at least one working SA 10 years following installation; this is somewhat lower than the 33% found by Jackson, et al from a multi-site evaluation.20 Based on the RE-AIM evaluation, DFRD is now installing tamper-resistant smoke alarms and re-canvassing previous project neighborhoods and homes to replace smoke alarms that have been removed or no longer function.

In addition, the project reached only a quarter of the target population. To increase the reach, postcards are being mailed to homes in the project neighborhoods to alert residents when DFRD will be canvassing the neighborhoods and installing smoke alarms. Yard signs have also been used to notify residents in advance of scheduled smoke alarm canvassing sessions.

There are several limitations to this study. Because only one fire department was involved, it is difficult to generalize about the adoption component of the RE-AIM evaluation; other fire departments may not be able to integrate a program such as OI into its ongoing functions. In addition, implementation of OI was most active during the years that DFRD was receiving grant funding from CDC; it is more challenging to implement OI without ongoing support. In addition, we had no household data from non-participating households; we utilized census data to calculate demographic data for this group, which may not be at the same level of accuracy as the data collected from participating households.8

As suggested by Hanson, it is useful to assess the broader population impact of a public health program, using dimensions such as implementation and maintenance, in a real world setting.21 The five RE-AIM dimensions provided insight into areas to focus our future activities for improvement of OI. The RE-AIM model may be a beneficial tool in evaluating other injury prevention programs as well.

Contributors:

SSS, MM, CR and GRI contributed to the concept and design of the project. MM, CR, JJB and EC contributed to the acquisition of data. SSS, MM, JJB and GRI contributed to the analyses and interpretation of data. SSS, MM and GRI had access to all data and are responsible for the overall content. All authors contributed to drafting and revising the article and approved the final version.

Acknowledgements

We would like to thank the following people for their assistance in the project: Kevin Sipes (retired) and Debra Carlin (retired) with the City of Dallas Fire-Rescue Department for their oversight of the project, and members of the Inspection and Life Safety Education Division for the coordination and collection of data; Merissa Yellman for assistance in cost calculations; and David Beveridge, Injury Prevention Center Administrative Assistant, for assistance in formatting various parts of this article.

References

  1. 1.Karter M J.Fire loss in the United States during 2011,Quincy (MA):National Fire Protection Association, Fire Analysis and Research Division;2012.
  1. 2.Istre G, McCoy M A, Osborn L. (2001) Deaths and injuries from house fires. , N Engl J Med 344, 1911-1916.
  1. 3.Mallonee S, Istre G R, Rosenberg M. (1996) Surveillance and prevention of residential fire-related injuries. , N Engl J Med 335, 27-31.
  1. 4.Ballesteros M F, Kresnow M J.Prevalence of Residential Smoke Alarms and Fire Escape Plans. in the U.S.: Results from the Second Injury Control and Risk Survey (ICARIS-2). Public Health Rep.2007Mar-Apr; 122(2), 224-231.
  1. 5.Ahrens M.Smoke alarms in U.S. home fires. Fire analysis and research division. National Fire Protection Association. , Quincy, MA.September2011
  1. 6.Liu Y, Holland A E, Mack K, Diekman S. (2011) Disparities in the prevalence of smoke alarms in U.S. households: conclusions drawn from published case studies. , J Safety Res 42, 409-413.
  1. 7.Sidman E A, Grossman D C, Mueller B A. (2011) Comprehensive smoke alarm coverage in lower economic status homes: alarm presence, functionality, and placement. , J Community Health 36, 525-533.
  1. 8.Istre G R, McCoy M, Moore B J, Roper C, Stephens-Stidham S et al.Preventing Deaths and Injuries from House Fires: An Outcome Evaluation of a Community-Based Smoke Alarm Installation Program.doi: 10.1136/injuryprev-2013-040823.Inj Prev Published Online First:[23July2013].
  1. 9.Farris R P, Will J C, Khavjou O, Finkelstein E A. (2007) Beyond effectiveness: evaluating the public health impact of the WISEWOMAN program. , Am J Public Health 97, 641-647.
  1. 10.Glasgow R E, Vogt T M, Boles S M. (1999) Evaluating the public health impact of health promotion interventions: the RE-AIM framework. , Am J Public Health 89, 1322-1327.
  1. 11.Abrams D B, Orleans C T, Niraura R S, Goldstein M G, Prochaska J O et al. (1996) Integrating individual and public health perspectives for treatment of tobacco dependence under managed health care: a combined stepped-care and matching model. , Ann Behav Med 18, 290-304.
  1. 12.Glasgow R E, Klesges L M, Dzewaltowski D A, Estabrooks P A, Vogt T M. (2006) Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. , Health Education Research 21, 688-694.
  1. 13.Fuzhong L, Harmer P, Glasgow R, Mack K A, Sleet D et al. (2008) Translation of an Effective Tai-Chi Intervention Into a Community-Based Falls-Prevention Program. , Am J Public Health 98, 1195-1198.
  1. 14.Kim A E, Towers A, Renaud J, Zhu J, Shea J A et al. (2012) Application of the RE-AIM Framework to Evaluate the Impact of a Worksite-Based Financial Incentive Intervention for Smoking Cessation. , JOEM 54, 610-614.
  1. 15.Finch D F, Donaldson A. (2010) A sports setting matrix for understanding the implementation context for community sport. , Br J Sports Med 44, 973-8.
  1. 16.Saunders N, Otago L, Romiti M, Donaldson A, White P et al. (2010) Coaches’ perspectives on implementing an evidence-informed injury prevention program in junior community netball. , Br. J Sports Med 44, 1128-32.
  1. 17.McCoy M, Roper C, Campa E, Stephens-Stidham S, Carlin D et al.How Long do Smoke Alarms Function? A Cross-Sectional Follow-Up of a Smoke Alarm Installation Program. doi: 10.1136/injuryprev-2013-040824. Inj Prev Published Online First:[23July2013].
  1. 18.Gaglio B, Shoup J A, Glasgow R E. (2013) The RE-AIM Framework: A Systematic Review of Use Over Time. doi: 10.2105/AJPH.2013.301299. , Am J Public Health 103, 38-346.
  1. 19.Parmer J E, Corso P S, Ballesteros M F. (2006) A cost analysis of a smoke alarm installation and fire safety education program. , J Safety Research 37, 367-373.
  1. 20.Jackson M, Wilson J, Akoto J. (2010) Evaluation of fire-safety programs that use 10-year smoke alarms. , J Community Health 35, 543-8.
  1. 21.Hanson D W, Finch C F, Allegrante J P, Sleet D.Closing the Gap Between Injury Prevention Research and Community Safety Promotion Practice: Revisiting the Public Health Model. Public Health Rep.2012Mar-Apr;. 127(2), 147-155.

Cited by (2)

  1. 1.Yellman Merissa A, Peterson Cora, McCoy Mary A, Stephens-Stidham Shelli, Caton Emily, et al, 2018, Preventing deaths and injuries from house fires: a cost–benefit analysis of a community-based smoke alarm installation programme, Injury Prevention, 24(1), 12, 10.1136/injuryprev-2016-042247
  1. 2.Stephens-Stidham Shelli, Miller Beverly, 2023, , , (), 15, 10.1007/978-3-031-20357-2_2