Mobile Logo in White
Search/Quicklinks

Organizational Readiness to Change Assessment (ORCA)

Instrument Author:

Helfrich, C. D., Li, Y. F., Sharp, N. D

Source:

Helfrich, C. D., Li, Y. F., Sharp, N. D., & Sales, A. E. (2009). Organizational readiness to change assessment (ORCA): Development of an instrument based on the promoting action on research in health services (PARIHS) framework. Implementation Science, 4(1), 38-51.

Source Author:

Helfrich, C. D., Li, Y. F., Sharp, N. D., Sales, A. E.

Journal Name:

Implementation Science

Volume:

4

Issue/Part:

1

Year:

2009

Pagination:

38-51

Abstract:

Background: The Promoting Action on Research Implementation in Health Services, or PARIHS, framework is a theoretical framework widely promoted as a guide to implement evidence-based clinical practices. However, it has as yet no pool of validated measurement instruments that operationalize the constructs defined in the framework. The present article introduces an Organizational Readiness to Change Assessment instrument (ORCA), organized according to the core elements and sub-elements of the PARIHS framework, and reports on initial validation. Methods: We conducted scale reliability and factor analyses on cross-sectional, secondary data from three quality improvement projects (n = 80) conducted in the Veterans Health Administration. In each project, identical 77-item ORCA instruments were administered to one or more staff from each facility involved in quality improvement projects. Items were organized into 19 subscales and three primary scales corresponding to the core elements of the PARIHS framework: (1) Strength and extent of evidence for the clinical practice changes represented by the QI program, assessed with four subscales, (2) Quality of the organizational context for the QI program, assessed with six subscales, and (3) Capacity for internal facilitation of the QI program, assessed with nine subscales. Results: Cronbach’s alpha for scale reliability were 0.74, 0.85 and 0.95 for the evidence, context and facilitation scales, respectively. The evidence scale and its three constituent subscales failed to meet the conventional threshold of 0.80 for reliability, and three individual items were eliminated from evidence subscales following reliability testing. In exploratory factor analysis, three factors were retained. Seven of the nine facilitation subscales loaded onto the first factor; five of the six context subscales loaded onto the second factor; and the three evidence subscales loaded on the third factor. Two subscales failed to load significantly on any factor. One measured resources in general (from the context scale), and one clinical champion role (from the facilitation scale). Conclusion: We find general support for the reliability and factor structure of the ORCA. However, there was poor reliability among measures of evidence, and factor analysis results for measures of general resources and clinical champion role did not conform to the PARIHS framework. Additional validation is needed, including criterion validation. © 2009 Helfrich et al; licensee BioMed Central Ltd.

Descriptors:

n/a

Number of questions:

77 items with 5-point Likert response formats, taking about 15 minutes to complete as a web-based version

Response Options:

n/a

Validity:

n/a

Reliability:

Cronbach’s alpha for scale reliability ranted from 0.74 to 0.95, with authors describing reliability as being poor for 3 evidence scales. Article describes Convergent Validity in detail.

Researchers should review article for full details of psychometric properties estimates.

Subscale/Factors:

Evidence, Context, Faciliation

Sample Descriptors:

Adult

Quality Improvement project staff

Sample Items:

For each of the following statements, please rate the strength of your agreement with the statement, from 1 (strongly disagree) to 5 (strongly agree)

(Research) The proposed practice changes or guideline implementation: strongly disagree disagree neither agree nor disagree agree strongly agree

Don’t know/
a) are(is) supported by RCTs or other scientific evidence from the VA 1 2 3 4 5
b) are(is) supported by RCTs or other scientific evidence from other health care systems 1 2 3 4 5
c) should be effective, based on current scientific knowledge

Measure Descriptors:

Organizational readiness.

References:

  1. Kitson, A., Harvey, G., McCormack, B., Enabling the implementation of evidence based practice: A conceptual framework (1998) Quality in Health Care, 7, pp. 149-158. , 10185141 10.1136/qshc.7.3.149.
  2. Harvey, G., Loftus-Hills, A., Rycroft-Malone, J., Titchen, A., Kitson, A., McCormack, B., Seers, K., Getting evidence into practice: The role and function of facilitation (2002) Journal of Advanced Nursing, 37, pp. 577-588. , 10.1046/j.1365-2648.2002.02126.x 11879422.
  3. McCormack, B., Kitson, A., Harvey, G., Rycroft-Malone, J., Titchen, A., Seers, K., Getting evidence into practice: The meaning of ‘context’ (2002) J Adv Nurs, 38, pp. 94-104. , 10.1046/j.1365-2648.2002.02150.x 11895535.
  4. Rycroft-Malone, J., Seers, K., Titchen, A., Harvey, G., Kitson, A., McCormack, B., What counts as evidence in evidence-based practice? (2004) Journal of Advanced Nursing, 47 (1), pp. 81-90. , DOI 10.1111/j.1365-2648.2004.03068.x.
  5. Kitson, A., Rycroft-Malone, J., Harvey, G., McCormack, B., Seers, K., Titchen, A., Evaluating the successful implementation of evidence into practice using the PARiHS framework: Theoretical and practical challenges (2008) Implementation Science, 3, p. 1.,18179688 10.1186/1748-5908-3-1.
  6. Rycroft-Malone, J., Kitson, A., Harvey, G., McCormack, B., Seers, K., Titchen, A., Estabrooks, C., Ingredients for change: Revisiting a conceptual framework (2002) Quality & Safety in Health Care, 11, pp. 174-180., 12448812 10.1136/qhc.11.2.174. Rycroft-Malone, J., The PARIHS framework A framework for guiding the implementation of evidence-based practice (2004) J Nurs Care Qual, 19, pp. 297-304. , 15535533. Rycroft-Malone, J., Harvey, G., Seers, K., Kitson, A., McCormack, B., Titchen, A., An exploration of the factors that influence the implementation of evidence into practice (2004) Journal of Clinical Nursing, 13 (8), pp. 913-924. , DOI 10.1111/j.1365-2702.2004.01007.x.
  7. Brown, D., McCormack, B., Developing postoperative pain management: Utilising the promoting action on research implementation in health services (PARIHS) framework (2005) Worldviews on Evidence-Based Nursing, 2 (3), pp. 131-141. , DOI 10.1111/j.1741-6787.2005.00024.x.
  8. Stetler, C., Legro, M., Rycroft-Malone, J., Bowman, C., Curran, G., Guihan, M., Hagedorn, H., Wallace, C., Role of “external facilitation” in implementation of research findings: A qualitative evaluation of facilitation experiences in the Veterans Health Administration (2006) Implementation Science, 1, p. 23. , 17049080 10.1186/1748-5908-1-23; Cummings, G.G., Estabrooks, C.A., Midodzi, W.K., Wallin, L., Hayduk, L., Influence of organizational characteristics and context on research utilization (2007) Nurs Res, 56, pp. 24-39. , 10.1097/01.NNR.0000280629.63654.95 17625471; Estabrooks, C.A., Midodzi, W.K., Cummings, G.G., Wallin, L., Predicting research use in nursing organizations: A multilevel analysis (2007) Nurs Res, 56, pp. 7-23. , 10.1097/01.NNR.0000280647.18806.98 17625477.
  9. Bahtsevani, C., Willman, A., Khalaf, A., Stman, M., Developing an instrument for evaluating implementation of clinical practice guidelines: A test-retest study (2008) Journal of Evaluation in Clinical Practice, 14, pp. 839-846. 18331325.
  10. Eccles, M., Grimshaw, J., Walker, A., Johnston, M., Pitts, N., Changing the behavior of healthcare professionals: The use of theory in promoting the uptake of research findings (2005) Journal of Clinical Epidemiology, 58 (2), pp. 107-112. , DOI 0.1016/j.jclinepi.2004.09.002, PII S0895435604002562.
  11. Designing theoretically-informed implementation interventions (2006) Implementation Science, 1, p. 4. (ICEBeRG) The Improved Clinical Effectiveness through Behavioural Research Group 16722571 10.1186/1748-5908-1-4.
  12. Rptm, G., Bosch, M.C., Mejl, H., Eccles, M.P., Wensing, M., Planning and Studying Improvement in Patient Care: The Use of Theoretical Perspectives (2007) The Milbank Quarterly, 85, pp. 93-138. , 10.1111/j.1468-0009.2007.00478.x 17319808.
  13. McCormack, B., McCarthy, G., Wright, J., Coffey, A., Development and Testing of the Context Assessment Index (CAI) (2009) Worldviews on Evidence-Based Nursing, 6, pp. 27-35. , 10.1111/j.1741-6787.2008.00130.x 19207560.
  14. Every, N.R., Fihn, S.D., Sales, A.E.B., Keane, A., Ritchie, J.R., Quality Enhancement Research Initiative in Ischemic heart Disease: A Quality Initiative from the Department of Veterans Affairs (2000) Medical Care, 38, pp. I49-I59. , 10.1097/00005650-200006001-00006.
  15. Sharp, N.D., Pineros, S.L., Hsu, C., Starks, H., Sales, A.E., A Qualitative Study to Identify Barriers and Facilitators to Implementation of Pilot Interventions in the Veterans Health Administration (VHA) Northwest Network (2004) Worldviews Evid Based Nurs, 1, pp. 129-139. , 10.1111/j.1741-6787.2004.04023.x 17129326.
  16. Pineros, S.L., Sales, A.E., Li, Y.F., Sharp, N.D., Improving care to patients with ischemic heart disease: Experiences in a single network of the veterans health administration (2004) Worldviews Evid Based Nurs, 1 (SUPPL 1), pp. 33-40. , 10.1111/j.1524-475X.2004.04042.x 17129333.
  17. Shortell, S.M., O’Brien, J.L., Carman, J.M., Foster, R.W., Hughes, E.F.X., Boerstler, H., O’Connor, E.J., Assessing the impact of continuous quality improvement/total quality management: Concept versus implementation (1995) Health Services Research, 30, pp. 377-401. , 7782222.
  18. Shortell, S.M., Jones, R.H., Rademaker, A.W., Gillies, R.R., Dranove, D.S., Hughes, E.F.X., Budetti, P.P., Huang, C.-F., Assessing the Impact of Total Quality Management and Organizational Culture on Multiple Outcomes of Care for Coronary Artery Bypass Graft Surgery Patients (2000) Medical Care, 38, pp. 207-217. , 10.1097/00005650-200002000-00010 10659694.
  19. Young, G.J., Charns, M.P., Heeren, T.C., Product-Line Management in Professional Organizations: An Empirical Test of Competing Theoretical Perspectives (2004) Academy of Management Journal, 47, p. 723
  20. Holt, D.T., Armenakis, A.A., Feild, H.S., Harris, S.G., Readiness for Organizational Change: The Systematic Development of a Scale (2007) Journal of Applied Behavioral Science, 43, pp. 232-255. , 10.1177/0021886306295295.
  21. Bourgeois, L.J., On the measurement of organizational slack (1981) Academy of Management Review, 6, pp. 29-39. , 10.2307/257138.
  22. Sales, A., Helfrich, C., Ho, P.M., Hedeen, A., Plomondon, M.E., Li, Y.-F., Connors, A., Rumsfeld, J.S., Implementing Electronic Clinical Reminders for Lipid Management in Patients with Ischemic Heart Disease in the Veterans Health Administration (2008) Implementation Science, 3, p. 28. , 18510748 10.1186/1748-5908-3-28.
  23. Bernard, H.R., (2000) Social Research Methods: Qualitative and Quantitative Approaches, , Thousand Oaks, CA: Sage.
  24. Nunnally, J.C., Bernstein, I.H., (1994) Psychometric Theory, , New York, NY: McGraw-Hill Inc 3
  25. Bollen, K.A., (1989) Structural Equations with Latent Variables, , New York: Wiley. Jöreskog, K.G., Sörbom, D., (1995) LISREL 8: Structural Equation Modeling with the SIMPLIS Command Language, , Chicago, Ill.; Hillsdale, N.J.: Scientific Software International; distributed by L. Erlbaum Associates.
  26. Floyd, F., Widaman, K., Factor Analysis in the Development and Refinement of Clinical Assessment Instruments (1995) Psychological Assessment, 7, pp. 286-299. , 10.1037/1040-3590.7.3.286.
  27. Sanchez-Menegay, C., Stalder, H., Do physicians take into account patients’ expectations? (1994) J Gen Intern Med, 9, pp. 404-406. , 10.1007/BF02629524 7931752.
  28. Montgomery, A.A., Fahey, T., How do patients’ treatment preferences compare with those of clinicians? (2001) Qual Health Care., 101, pp. 39-i43. , 11533437.
  29. Davis, R.E., Dolan, G., Thomas, S., Atwell, C., Mead, D., Nehammer, S., Moseley, L., Elwyn, G., Exploring doctor and patient views about risk communication and shared decision-making in the consultation (2003) Health Expectations, 6 (3), pp. 198-207. , DOI 10.1046/j.1369-6513.2003.00235.x.
  30. Shevlin, M., Hunt, N., Robbins, I., A confirmatory factor analysis of the Impact of Event Scale using a sample of World War II and Korean War veterans (2000) Psychol Assess, 12, pp. 414-417. , 10.1037/1040-3590.12.4.414 11147109.
  31. Getzan, C., VA Funding Fails to Meet Increased Demand for Services, Groups Say; As Congress and the President haggle over future Veterans Administration funding, a New England Journal of Medicine study shows an increased risk of mental health disorders among Middle East veterans (2004) The New Standard. Syracuse, NY.
  32. Mays, G.P., Claxton, G., White, J., Managed care rebound? Recent changes in health plans’ cost containment strategies (2004) Health Aff (Millwood), (SUPPL WEB EXCLUSIVES). , 15451964.
  33. Howell, R.D., Breivik, E., Wilcox, J.B., Reconsidering formative measurement (2007) Psychological Methods, 12, pp. 205-218. , 10.1037/1082-989X.12.2.205 17563173. Edwards, J.R., Bagozzi, R.P., On the nature and direction of relationships between constructs and measures (2000) Psychological Methods, 5, pp. 155-174., 10.1037/1082-989X.5.2.155 10937327.
  34. Weiner, B.J., Amick, H., Lee, S.-Y.D., Review: Conceptualization and measurement of organizational readiness for change. A review of the literature in health services research and other fields (2008) Medical Care Research and Review, 65 (4), pp. 379-436. , DOI 10.1177/1077558708317802.
  35. Streiner, D.L., Norman, G.R., (2003) Health Measurement Scales: A Practical Guide to Their Development and Use, Oxford; New York: Oxford University Press Third

Instrument Availability:

Contact author: Christian D Helfrich – christian.helfrich@va.gov

Article Availability:

http://www.scopus.com/inward/record.url?eid=2-s2.0-68749087418&partnerID…

Acronym:

ORCA

DOI:

10.1186/1748-5908-4-38

ISBN:

n/a

ISSN:

17485908 (ISSN)

Download PDF