OBLIKOVANJE DOGOVORNE DEFINICIJE ZA POJEM ZNANOST O UVAJANJU IZBOLJŠAV V ZDRAVSTVU (HIS) V SEDMIH EVROPSKIH DRŽAVAH: PRISTOP Z METODAMI KONSENZA

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Abstract

Introduction

There is a limited body of research in the field of healthcare improvement science (HIS). Quality improvement and ‘change making’ should become an intrinsic part of everyone’s job, every day in all parts of the healthcare system. The lack of theoretical grounding may partly explain the minimal transfer of health research into health policy.

Methods

This article seeks to present the development of the definition for healthcare improvement science. A consensus method approach was adopted with a two-stage Delphi process, expert panel and consensus group techniques. A total of 18 participants were involved in the expert panel and consensus group, and 153 answers were analysed as a part of the Delphi survey. Participants were researchers, educators and healthcare professionals from Scotland, Slovenia, Spain, Italy, England, Poland, and Romania.

Results

A high level of consensus was achieved for the broad definition in the 2nd Delphi iteration (86%). The final definition was agreed on by the consensus group: ‘Healthcare improvement science is the generation of knowledge to cultivate change and deliver person-centred care that is safe, effective, efficient, equitable and timely. It improves patient outcomes, health system performance and population health.’

Conclusions

The process of developing a consensus definition revealed different understandings of healthcare improvement science between the participants. Having a shared consensus definition of healthcare improvement science is an important step forward, bringing about a common understanding in order to advance the professional education and practice of healthcare improvement science.

Izvleček

Uvod

Skupno število raziskav na področju znanosti o uvajanju izboljšav vzdravstvu (Healthcare Improvement Science - HIS) je majhno. Vendar pa bi morala izboljševanje kakovosti in vpeljava sprememb postati neločljivo povezana z vsakodnevnim delom zdravstvenih delavcev na vseh področjih zdravstvenega sistema. Pomanjkanje teoreticnih izhodišč bi lahko delno pojasnilo nizko stopnjo prenosa rezultatov raziskav v zdravstveno politiko.

Metode

Namen članka je predstaviti postopek oblikovanja definicije za pojem znanost o uvajanju izboljšav v zdravstvu. Pri pristopu z metodami konsenza smo uporabili metodo Delfi, izvedeno v dveh krogih, panelne skupine strokovnjakov in tehnike konsenzne skupine. V skupini strokovnjakov in konsenzni skupini je sodelovalo 18 oseb, v okviru metode Delfi smo analizirali 153 odgovorov. Sodelujoče osebe so bile raziskovalci, izobraževalci in zdravstveni delavci iz Škotske, Slovenije, Španije, Italije, Anglije, Poljske in Romunije.

Rezultati

V drugem krogu metode Delfi je bila dosežena visoka stopnja konsenza (86-odstotna) za široko definicijo. Konsenzna skupina je oblikovala naslednjo končno definicijo: »Znanost o uvajanju izboljšav omogoca ustvarjanje znanj za izvajanje sprememb ter zagotavljanje v pacienta usmerjene zdravstvene obravnave, ki je varna, zmogljiva, učinkovita, pravična in pravočasna. Izboljšuje pacientove izide zdravstvene obravnave, sistem nudenja zdravstvenih storitev in zdravje populacije.«

Razprava

Postopek oblikovanja dogovorne definicije je razkril različno razumevanje pojma znanost o uvajanju izboljšav v zdravstvu med sodelujočimi. Oblikovanje skupne dogovorne definicije je pomemben korak naprej, saj pripomore k splošnemu razumevanju tega pojma in s tem tudi k spodbujanju strokovnega izobraževanja in izvajanju znanosti o uvajanju izboljšav v praksi.

1. Introduction

Improving healthcare quality has become a priority over the past years (1). While much of quality improvement work is unscientific (2), the adoption of a more scientific approach to improvement could enhance the ability of health systems to provide high-quality care and use their resources optimally (3). Healthcare improvement science (HIS) represents ‘the combined and unceasing efforts of everyone – healthcare professionals, patients and their families, researchers, payers, planners and educators – to make the changes that will lead to better patient outcomes (health), better system performance (care) and better professional development (learning)’ (4, p. 2). In this respect, quality improvement and change making should become an essential part of healthcare professionals’ work at all levels of the healthcare system (5). Implementation science and quality improvement efforts share the ultimate goal of improving healthcare quality. However, there are differences. Quality begins with a specific problem in a given healthcare system, while implementation science begins with evidencebased practice (EBP) that is not used effectively, then identifies and addresses quality issues at the level of providers, clinics, or the healthcare system (6). Topic reviewers reported a diversity of outcomes used for quality improvement evaluations, flaws in study design, and incomplete reporting (7).

The Health Foundation (8) developed an explanation of what improvement science should be. They defined it as ‘building a knowledge base for improving health services and translating this knowledge into practice to deliver the best possible patient care’ (8, p. 7). Moreover, it has been stressed that the implementation science community should provide guidelines for reporting the use of theoretical frameworks within implementation studies and their efficacy (9). This is important because few hospital-based interventions are essentially theoretical (10). The Health Foundation definition (8, p. 7) states clearly that improvement science ‘requires a systematic, scientifically rigorous approach to close the gap between current and best practice.’

In the case of quality improvement, theory needs development, which draws from a better understanding of practice (11). Many quality improvement practitioners believe that quality improvement is implemented in complex adaptive systems. They sometimes feel limited by research findings that fail to recognize the complexity and the need for adaptation. Quality improvement practitioners could help researchers to see the larger patterns specific to complex systems and draw upon possible solutions (12). Organizational learning framework could be used to explore the factors influencing improvement feasibility (13). Organizations worldwide have adopted interprofessional education and interprofessional collaborative practice in an effort to improve healthcare delivery systems (14), which fits well with the Health Foundation definition (8, p. 7) that ‘improvement science embraces disciplines across health services research, from sociology to statistics, psychology to health economics.’ The next generation of healthcare professionals will require new skills to make sure that quality improvement in health care is successful (15). For this reason, it is important how different healthcare professionals understand HIS.

1.1. The Aim

The article describes a part of the project ‘Improvement Science Training for European Healthcare Workers (ISTEW),’ which aimed to develop shared academic and practice-based programmes that would enable European institutions to build improvement capacity within their healthcare workforce. This article discusses a keystone component of the project, namely the development of a consensus definition for HIS, based on the interprofessional and multicultural consensus of healthcare professionals considered as experts from seven European countries.

2. Methods

2.1. Research Design and Data Collection

Research design comprised of consensus methods, including the nominal group technique, the Delphi process, and the consensus group technique. The aim of consensus methods is to determine the extent to which experts or lay people agree about a given issue. The term ‘agreement’ takes two forms, which need to be distinguished: firstly, the extent to which each respondent agrees with the issue under consideration (typically rated on a numerical or categorical scale) and, secondly, the extent to which respondents agree with each other (16).

2.1.1. The Nominal Group Technique

The nominal group technique uses a highly structured meeting to gather information from relevant experts (usually 9-12 in number) about a given issue. It consists of two rounds, in which panellists rate, discuss, and then rerate a series of items or questions (16). The nominal group technique was used twice. First, we implemented two structured Skype meetings with research group members from seven European countries. The goal was to develop a working definition of HIS, to increase consensus amongst participants about the meaning and content of HIS, and to establish the level of development and understanding of HIS in each country. Expert-based knowledge, literature and comprehension across a core group of seven experts was used. After two meetings, a working definition was developed and tested in the Delphi process. Second, the nominal group technique was used on the same team in order to develop an extended questionnaire for the 2nd Delphi iteration.

2.1.2. The Delphi Process

A Delphi study is a robust method that draws on expert opinions, and compares them with the combined opinions of other participating experts over several rounds, until a consensus on specified criteria is reached (16). A two-iteration Delphi technique was developed with participants from seven European countries. The Delphi technique enabled us to collect data from several domains related to healthcare in education, research, management and practice. Moreover, two questionnaires were designed for each iteration with quantitative and qualitative elements allowing us to achieve a merging of opinions from 87 respondents. The respondents were consulted in two anonymous Delphi rounds. A 5-point Likert scale (1–strongly disagree, 2–disagree, 3–neither agree nor disagree, 4–agree, 5–strongly agree) was used in both questionnaires to rate the importance of each HIS definition element and of the definition as a whole. Respondents could also offer suggestions for improving separate parts of the definition by replying to open-ended questions and providing HIS descriptions used in their countries. Furthermore, we were interested in obtaining some demographic data (the area of work, respondent connection with HIS and involvement in HIS) and data about the description of HIS in their countries. Reliability test results were good. Cronbach’s Alpha for statements (N=6) for 1st Delphi iteration was 0.867 and for statements (N=7) for 2nd Delphi iteration 0.843.

Over several rounds, a specific definition element was included in the next version of the definition if >80% of respondents judged it as important or very important, and excluded if >50% judged it as not important or moderately important (17). The suggested reformulations of the second HIS definition were analysed with the nominal group technique by using qualitative and quantitative data from the first Delphi iteration. These reformulated items were presented in the second Delphi iteration.

2.1.3. The Consensus Group Technique

Following two Delphi iterations and two nominal group techniques, a consensus group technique (a combination of a focus group and a public meeting) was employed. The main difference between consensus groups and focus groups is that in the latter, the moderator or analyst decides, whereas in consensus groups, the participants themselves negotiate and decide the findings (18). We used this technique at a three-day meeting of all ISTEW project members held in Bled (June 2014), Slovenia. The consensus group included 18 participants from seven European countries. We wanted to provide everyone in the group the opportunity to work collaboratively, to be involved in the decision-making processes, to be valued as experts, and for the group to become a forum for change (19). Consensus group members were discussing quantitative and qualitative data of the 2nd Delphi iteration and the suitability of the developed HIS definition. The aim was to arrive at a final definition of HIS.

2.2. The Sample

A non-random convenience sample was used for the Delphi process. Each project team invited 10–15 people they knew to be involved in HIS in their respective settings. ‘Participants’ came from seven European countries (England, Italy, Poland, Romania, Scotland, Slovenia, and Spain) and from different healthcare areas. The total number of respondents in the 1st iteration was 87; of these, 66 chose to participate in the 2nd iteration. HIS fields in which the respondents were involved are shown in Table 1 .

Table 1

Participants’ fields of work in the Delphi study.

Ten members of the research group participated in the nominal group technique, and 18 members of the ISTEW project group participated in the consensus group technique. Members of both groups were experts in HIS from different professions.

2.3. Research Conduction and Ethical Approval

The research was conducted from November 2013 to September 2014. We started with the nominal group technique, first conducted in November 2013 and then in March 2014, followed by the Delphi process (the first and second iterations in December 2013 and April 2014, respectively), and finally, the consensus group technique in June 2014.The research protocol was approved by the scientific ethical committee of the research coordinator, the Faculty of Health Care Jesenice (Slovenia). Partners also adhered to their ethical approval processes for research. Participants were assured about the anonymity and confidentiality of their collaboration. The questionnaires were available either in English or the language of the expert. Partners were responsible for translating the questionnaires, as required.

2.4. Data Analysis

Quantitative data analysis was carried out with descriptive statistics (Mean, Standard Deviation, Frequency, Percentage) and bivariate statistics (Chi-Square (p) – Kruskal-Wallis H nonparametric test, Independent samples t-test, Paired samples t-test, Pearson correlation, Spearman correlation) in the statistical software programme SPSS, v. 20.0. The level of statistical significance was set at p

Qualitative data was obtained from the open-ended questions of the first and second Delphi iterations. First, each partner team extracted the themes from the qualitative data from both Delphi iterations and exposed these themes in the nominal group technique and the consensus group technique. The software programme NVivo, v. 10.0 and content analysis were used.

3. Results

3.1. Characteristics of Respondents in the Delphi Survey

Table 2

Respondents’ involvement in HIS and differences between countries.

X 2 (p) is answer ‘Yes’ between countries.

3.2. Results of the 1st Delphi Iteration

In the 1st iteration, we tested the definition of HIS which was developed with the nominal group technique. The definition is shown in Table 3 . Results reveal a high level of agreement with the definition as a whole on a 5-point scale. Respondents from Scotland expressed a significantly lower agreement with the definition as a whole, but with the highest level of standard deviation of their answers (M=2.83, SD=1.27, p<0.001) and also with its separate parts (a) (M=2.25, SD=1.22, p<0.001), (c) (M=2.67, SD=0.99, p<0.001), (d) (M=2.57, SD=1.78, p=0.029), except for parts (b) and (e).

Table 3

Results for the level of agreement with the definition as a whole and with its separate parts (a - e), and differences between countries in the 1st Delphi iteration (n=87).

Legend: M - central value of a discrete set of numbers calculated by the sum of the values divided by the number of values; SD - Standard deviation; Chi-Square (p) - Kruskal-Wallis H nonparametric test for several independent samples; a, b, c, d, e - parts of the whole definition.

Comment: X 2 (p) is mean value between countries.

The agreement with the definition increased with the level of respondents’ expertise. Correlation analysis showed us that respondents who identified themselves as ‘experts’ were in positive correlation with the agreement with the definition as a whole (r=0.264, p=0.023). Respondents’ fields of work were not in correlation with the level of agreement with the definition as a whole.

Qualitative responses of respondents (n=21) who rated themselves as experts were analysed. A review of responses to the question ‘Which areas are missing from the proposed definition?’ yielded 120 codes and 8 main categories, namely the content of the definition, the purpose of the definition, multidisciplinary approach, definition structure, healthcare professionals, negative consequences, innovative emergent field, and references. Word frequency of codes revealed that, in the opinions of HIS experts, the most frequent words to be included in the proposed definition are the following: Improvement (n=19), social (n=14), science (n=13), health (n=10), methods (n=10), healthcare (n=9), care (n=8), patient (n=8) and work (n=8).

3.3. Results of the 2nd Delphi Iteration

In the second iteration, we tested the definition of HIS developed with the nominal group technique, where the results from the first Delphi iteration were used. The definition is worded in Table 4 . The average level of agreement with the second proposed definition of HIS was higher (M=4.30) compared to the first iteration (M=3.93) (t=-3.006, p=0.004). The average portion of agreement in the first iteration was 78.5% and in the second 86%. For separate parts of the definition, the greatest agreement was expressed with the part (b).

Table 4

Results for the level of agreement with the definition as a whole and with its separate parts (a - f), and differences between countries in the 2nd Delphi iteration (N= 66).

Legend: M – central value of a discrete set of numbers calculated by the sum of the values divided by the number of values; SD – Standard deviation; Chi-Square (p) – Kruskal-Wallis H nonparametric test for several independent samples; a, b, c, d, e – parts of the whole definition.

Comment: X 2 (p) is mean value between countries.

Table 5

Results of countries look for the level of agreement with the definition as a whole and with its separate parts (a – f) in the 2nd Delphi iteration.

3.4. Results of the Consensus Group Technique

The consensus group technique started with a discussion of 18 experts from institutions involved in the ISTEW project. It was moderated by the principal investigator and research fellow for the project, with notes being taken. First, the aim was to increase the understanding of the first and second Delphi iteration results and to present partners’ qualitative analysis results. The consensus group discussion yielded 14 key themes of the HIS definition, which were merged into 3 content categories, namely ‘country differences knowledge’, ‘science and HIS,’ and ‘new dimension of HIS for practice’. The strongest extracted elements were knowledge, change, outcome, population, safety, efficiency, person-centred.

All partners agreed that the HIS definition had to be short and concise. The emphasis of each definition part and the theme from qualitative data was discussed until a consensus was reached. Based on this, the final consensus on the definition of Healthcare Improvement Science is as follows:

‘Healthcare improvement science is the generation of knowledge to cultivate change and deliver personcentred care that is safe, effective, efficient, equitable and timely. It improves patient outcomes, health system performance and population health.’

4. Discussion

HIS is incorporated into every phase of healthcare proceedings and is, consequently, a complex concept to define. Coining a definition that encompasses the multifaceted nature of HIS with all of its variables is very challenging, particularly because the use of the term HIS and its synonyms in the existing literature and other educational documents (grey literature) varies depending on the country and language.

By applying consensus methods, the definition was developed gradually. According to the results of two Delphi iterations, a high consensus among participants was achieved. Consensus increased from the first iteration to the second, until 86% agreement was reached. It is very important that the definition was developed through interprofessional interactions. Only a third of respondents in the Delphi survey perceived themselves as experts in HIS, although the Delphi study coordinators identified them as such. This, we suggest, reflects the extent to which HIS is developed in the partner countries involved, as we found that HIS was not equally developed in all partner countries. Indeed, we found that the term itself is not used at all in Spain, and that the concept is not well developed in Romania, where the term implementation science with its different connotations tends to be more common. Respondents coming from countries where HIS was not broadly developed may have found it challenging to respond in an informed way. Respondents from Scotland, where HIS is a political priority, expressed a significantly lower level of consensus with the entire definition compared to non-English speaking respondents, but the level of agreement was still high at 70%. These differences may be related to a number of issues. Firstly, most respondents from Scotland defined themselves as experts in HIS. Secondly, in Scotland, HIS is a more crucial driver of healthcare institution policies compared to the other countries. Finally, research and accessibility of scientific and professional HIS literature is greater in the United Kingdom compared to other non-English native speaking countries (20).

The definition formed in two Delphi iterations was a broadened one. The consensus group technique greatly contributed to reaching a compromise between experts, so that a short and concise version of the definition was developed. The definition includes a wide range of key themes, such as knowledge for cultivating change, personcentred care, improvement of patient outcomes, health system performance and population health. These themes are also included in definitions proposed by other authors (4, 8). The employed method for developing a definition leads to a better understanding of practice (11).

Only half of respondents agreed that HIS-related national bodies play an important role in health and social care policies in their own country. However, a closer look at the data reveals that the situation is not comparable across countries. Only one third of respondents felt that HIS has an appropriate status in their national healthcare system. As a result, HIS indicators are not developed well and HIS is an uncommon term in some countries. This is a major issue because, as Booth et al. (21) stated, changing population demographics, disease patterns and new research findings demand change in healthcare and require new ways for ensuring continuous improvement. Robson (22) believes that in making healthcare safer, one should join all levels of the profession in an effort to clearly work with their working environment and patient safety.

For a continuous improvement, an understanding and recognition of the need for HIS are crucial. Our definition underpins the need for developing interprofessional study programmes and other materials. Hall and Zierler (23) stressed that, with the growth of interprofessional education and practice in healthcare schools, faculty members have to assume new roles in designing and delivering interprofessional curricula. Senior leadership support is also vital in ensuring sustainable change, and it will be a key requirement in healthcare organisations and universities to make healthcare improvement a priority.

5. Limitations

The sample was not equally weighted for the nationalities of the HIS experts, and this may have impacted the Delphi process. Knowledge of the English language and familiarity with it may have also played a role in providing answers, in questionnaire translation, and in discussions within the nominal and consensus groups. In countries where English is not spoken as the official language, most partners had to first translate the questionnaire, then obtain answers from experts in their languages, and translate the responses back into English. This may have resulted in the loss of understanding and different interpretations. This study has only included seven European countries; consequently, it would be interesting to explore the ways in which HIS is understood, practised and taught across Europe as a whole.

6. Conclusion

A new consensus definition of HIS was developed in a European context. The consensus method approach was pertinent to unify the new concept from multidisciplinary, interprofessional and multicultural perspectives. Having a shared definition of HIS is important to boost healthcare culture and professional training processes. The definition enables higher education institutions in health and social care to address this gap in their curricula, and ensure that improvement science becomes a core competency for all healthcare graduates.

Acknowledgements

This article was written on behalf of the Improvement Science Training for European Healthcare Workers (ISTEW) project team. We would like to acknowledge the contributions of Professor Julita Sansoni, Dr Elisabetta Corvo, Dr Alan Taylor, Dr Katrina Ritters, Magdalena Glowacka, Dr Traian Mihaescu, Dr Radu Crisan, Dr Maria Josefa Cabanero Martinez, Barbara O’Donnell, Dr Joca Zurc and Miram Sanchez San Segundo.

Footnotes

Conflicts of interest:The authors report no conflicts of interest.

Funding

The research was financed by the European Union funded ERASMUS Lifelong Learning Project, ISTEW: Improvement Science Training for European Healthcare Workers (Project No. 539194-LLP-1-2013-1-UK-ERASMUS-EQR)

Ethical Approval

The research protocol was approved by the Senate Committee for Science, Research and Development at the Faculty of Health Care Jesenice (FHCJ) in December 2013. The serial number of Ethics Approval is 12/5 in the academic year 2013/2014. Partners also adhered to their ethical approval processes for research. Participants received written information about different aspects of the study; explained to them were their rights on voluntary participation and withdrawal from the study at any time, as well as their privacy and confidentiality rights.

References

1. Wandersman A, Alia KA, Cook B, Ramaswamy R.. Integrating empowerment evaluation and quality improvement to achieve healthcare improvement outcomes. BMJ Qual Saf. 2015; 24 :645–52. [PMC free article] [PubMed] [Google Scholar]

2. Auerbach AD, Landefeld CS, Shojania KG. The tension between needing to improve care and knowing how to do it. N Engl J Med. 2007; 357 :608–13. [PubMed] [Google Scholar]

3. Marshall M, Pronovost P, Dixon-Woods M.. Promotion of improvement as a science. Lancet. 2013; 381 :419–21. [PubMed] [Google Scholar]

4. Batalden PB, Davidoff F.. What is ''quality improvement'' and how can it transform healthcare? Qual Saf Health Care. 2007; 16 :2–3. [PMC free article] [PubMed] [Google Scholar]

5. Jones A, Williams A, Carson-Stevens A.. Integrating quality improvement into pre-registration education. Nurs Stand. 2013; 27 :44–8. [PubMed] [Google Scholar]

6. Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An introduction to implementation science for the non-specialist. BMC Psychol. 2015; 16 :32. [PMC free article] [PubMed] [Google Scholar]

7. McDonald KM, Schultz EM, Chang C.. Evaluating the state of quality-improvement science through evidence synthesis: insights from the closing the quality gap series. Perm J. 2013; 17 :52–61. [PMC free article] [PubMed] [Google Scholar]

8. Improvement Science Fellowships. London: The Health Foundation, 2011; 2013. http://www.renal.org/docs/default-source/what-we do/Improvement_Science_Fellowships.pdf?sfvrsn=0 The Health Foundation. Available October 15. [Google Scholar]

9. Helfrich CD, Damschroder LJ, Hagedorn HJ, Daggett GS, Sahay A, Ritchie M. et al. A critical synthesis of literature on the promoting action on research implementation in health services (PARIHS) framework. Implement Sci. 2010; 5 :82–101. [PMC free article] [PubMed] [Google Scholar]

10. Conry MC, Humphries N, Morgan K, McGowan Y, Montgomery A, Vedhara K, Panagopoulou E, Mc Gee H.. A 10 year (2000–2010) systematic review of interventions to improve quality of care in hospitals. BMC Health Serv Res. 2012; 12 :1–16. [PMC free article] [PubMed] [Google Scholar]

11. Øvretveit J, Leviton L, Parry G.. Increasing the generalisability of improvement research with an improvement replication programme. BMJ Qual Saf. 2011; 20 :i87–i91. [PMC free article] [PubMed] [Google Scholar]

12. Leviton L.. Reconciling complexity and classification in quality improvement research. BMJ Qual Saf. 2011; 20 :i28–29. [PMC free article] [PubMed] [Google Scholar]

13. Hovlid E, Bukve O, Haug K, Aslaksen AB, von Plessen C.. Sustainability of healthcare improvement: what can we learn from learning theory? BMC Health Serv Res. 2012; 12 :235. [PMC free article] [PubMed] [Google Scholar]

14. Dominguez DG, Fike DS, MacLaughlin EJ, Zorek JA. A comparison of the validity of two instruments assessing health professional student perceptions of interprofessional education and practice. J Interprof Care. 2015; 29 :144–9. [PubMed] [Google Scholar]

15. Estrada CA, Dolansky MA, Singh MK, Oliver BJ, Callaway-Lane C, Splaine M. et al. Mastering improvement science skills in the new era of quality and safety: the Veterans Affairs National Quality Scholars Program. J Eval Clin Pract. 2012; 18 :508–14. [PubMed] [Google Scholar]

16. Jones J, Hunter D.. Qualitative research: consensus methods for medical and health service research. BMJ. 1995; 311 :376–80. [PMC free article] [PubMed] [Google Scholar]

17. Minkman M, Ahans K, Fabbricotti I, Nabitz U, Huijsman R.. A quality management model for integrated care: results of a Delphi and Concept Mapping study. Int J Qual Health Care. 2009; 21 :66–75. [PubMed] [Google Scholar]

18. List D. The Consensus group technique: a users' manual. Wellington: Original Books; 2005. [Google Scholar]

19. Race KE. Hotch DF. Rehabilitation program evaluation: use of focus groups to empower clients. Eval Rev. 1994; 18 :730–40. [Google Scholar]

20. The healthcare quality strategy for NHS Scotland. Edinburg: The Scottish Government; 2010. The Scottish Government. [Google Scholar]

21. Booth BJ, Zwar N, Harris MF. Healthcare improvement as planned system change or complex responsive processes: a longitudinal case study in general practice. BMC Fam Pract. 2013; 14 :51. [PMC free article] [PubMed] [Google Scholar]

22. Robson W.. Eliminating avoidable harm: time for patient safety to play a bigger part in professional education and practice. Nurse Educ Today. 2014; 34 :e1–2. [PubMed] [Google Scholar]

23. Hall WL, Zierler BK. Interprofessional education and practice guide No. 1: developing faculty to effectively facilitate interprofessional education. J Interprof Care. 2015; 29 :3–7. [PMC free article] [PubMed] [Google Scholar]