Data analysis

Introduction

PRISMA guidelines (1) and the Cochrane collaboration (2) provide systematic reviewers with clear instructions as to how such reviews should be conducted and reported.  However,  while the guidance in relation to search strategies and data extraction are generic, most of the rest of the guidance, particularly in relation to interpretation, are aimed at informing approaches to analysis of quantitative data.  Despite its importance, there has been no such similar universal agreement about how the analysis of qualitative data should be handled.  Researchers have been exploring a number of methods as outlined earlier ranging from realist synthesis to narrative summary and meta-ethnography (3,4,5,6) but no clear consensus has yet been reached on the best approach to adopt (6).  Thus how best to analyse qualitative data that describes processes as opposed to quantitative data focusing on outcomes remains uncertain, although some have tried to produce standards such as the RAMESSES publication standards to improve the quality of reporting of certain types of qualitative reviews (7).   The NPT offers new ways to approach the analysis of qualitative data gathered as part of a systematic review.  First, NPT can be used directly as a coding framework, either coding data within each of the four main propositions: coherence, cognitive participation, collective action and reflexive monitoring; or coding to a greater level of detail including the four main propositions and each of the four constructs relating to the main proposition, such as interactional workability within collective action or individual appraisal within reflexive monitoring.  Alternatively, one could undertake thematic coding and then look at how the newly emerging themes fit within the NPT framework, again coding to the level of the main propositions or to the level of the propositions and accompanying constructs.  NPT provides a consistent framework with which to describe work processes and has the capacity to highlight gaps, that might be missed through, for example, thematic coding alone (8).

Things to consider

  • NPT is not a theory of everything.  If one decides to use NPT as a coding framework, it is inevitable that some data will not fit into the framework.  This does not mean that use of NPT is inappropriate.  However, it will be important to note what kind of data falls outside of the framework and whether such data raises issues that seem likely to be important or not.

  • Using NPT as a coding framework can be a good method of identifying gaps in the literature.  It is possible to code using the framework and then to look at the distribution of such codes pictorially using radar plots.  Such an approach can prove to be a powerful means of highlighting gaps or deficiencies in the existing literature.  This does not imply that using NPT allows quantitative claims to be made but merely that these kinds of methods can serve as a valuable way to illustrate the distribution of comments relating to constructs of NPT and identify those issues that have been addressed to a greater extent than others.

  • We would recommend getting two researchers independently to analyse the content of the included papers using NPT to ensure consistency in approach and thereby to increase the robustness and reliability of coding, in particular to ensure that the approach to coding is clearly defined and transparent.

EXAMPLE 1: e-HEALTH IMPLEMENTATION & INTEGRATION

In this study (9) a review of systematic reviews of implementation of e-Health interventions into health care systems was undertaken.  There were clear systematic search strategies with clear inclusion and exclusion criteria and papers were reviewed independently by two reviewers.  NPT was used as a tool to facilitate data analysis with data extracted from eligible papers being coded by two reviewers using NPT as a coding framework. Any areas of disagreement, which were minimal, were resolved through discussion.  Table 1 shows the coding framework that was developed.

Click for Table 1: Normalization process theory coding framework used for qualitative analysis of review data on e-health implementation

Here we provide a coding example (9).   The statement below illustrated data within the interactional workability component of collective action:

“… an effective clinical decision support system must minimise the effort required by clinicians to receive and act on system recommendations” (10)

The above quotation illustrated how ease of use of new technologies, that is, whether new systems helped or hindered professionals in carrying out tasks, was a major barrier or facilitator of implementation.  So coding was undertaken at the level of the subconstruct of interactional workability.  One could also decide to just code to the level of the principal construct such as Collective Action.

EXAMPLE 2 - Uncovering treatment burden as a key concept for stroke care: self-management of stroke

In this study (11,12) a systematic review of qualitative studies examining the patient experience of stroke management was undertaken.  Once again, explicit search strategies were adopted with predefined inclusion and exclusion criteria.  The data were analysed using NPT as a coding framework (see Table  2) The subject matter is quite different from that of the earlier example, demonstrating that NPT can be used to help conceptualise a wide range of qualitative data.

Table 2: NPT based coding framework for stroke care and self-management

A coding example is provided below to illustrate the coherence or sense making work patients undertake in the context of fragmented health care systems where “mixed messages” could cause comprehension challenges.

“A few low motivation patients described some of the stroke unit professionals as giving out unhelpful “mixed messages.” One patient reported that physiotherapists encouraged her to work at rehabilitation. On returning to the ward, however, she thought the nurses discouraged such effort by putting her to bed. This resulted in confusion regarding the correct way to behave. (13)”

 Importantly these studies interogated the data to look for issues that felt outside the coding framework to ensure that data was not inappropriately “shoe horned” into the framework  and allow interpretive creativity (14).

We have provided two published examples where NPT has been used to facilitate data analysis of qualitative systematic review data.  These are provided for illustrative purposes only and as outlined in the previous sections NPT can be used in different ways to assist analysis of qualitative data.

References

  1. Moher D, Liberati A, Tetzlaff J, Altman DG, for the PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339:b2535. Back to text
  2. Green S, Higgins J (2005) Glossary. Cochrane handbook for systematic reviews of interventions 4.2.5. The Cochrane Collaboration. Available: http://www.cochrane.org/resources/glossary.htm . Accessed 19 May 2009.  Back to text
  3. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton S.  Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy 2005; 10 (1): 45-53. Back to text
  4. Greenhalgh T, Kristjansson E, Robinson V. School feeding programmes Realist review to understand. BMJ 2007;335;858-861. Back to text
  5. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review - a new method of systematic review designed for complex policy interventions.  Journal of Health Services Research Policy 2005;10-21-34. Back to text
  6. Elaine Barnett-Page and James Thomas. Methods for the synthesis of qualitative research: a critical review.  Methods for Research Synthesis Node, Evidence for Policy and Practice Information and Co-ordinating (EPPI-) Centre, Social Science Research Unit, Institute of Education, 18 Wobrun Square, London WC1H ONS. http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=188. ESRC National Centre for Research Methods NCRM Working Paper Series Number (01/09)
  7. Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R.  RAMESES publication standards: meta-narrative reviews.  BMC Medicine 2013;11:20; Gough D.  Meta-narrative and realist reviews: guidance, rules, publication standards and quality appraisal.  BMC Medicine 2013;11:22
  8. Mair, Frances, May, Carl, O'Donnell, Catherine , Finch, Tracy, Sullivan, Frank, and Murray, Elizabeth (2012) Factors that promote or inhibit the implementation of e-health systems: an explanatory systematic review. Bulletin of the World Health Organisation, 90 . pp. 357-364. ISSN 0042-9686 (doi:10.2471/BLT.11.099424 Back to example.
  9. Mair, Frances, May, Carl, O'Donnell, Catherine , Finch, Tracy, Sullivan, Frank, and Murray, Elizabeth (2012) Factors that promote or inhibit the implementation of e-health systems: an explanatory systematic review. Bulletin of the World Health Organisation, 90 . pp. 357-364. ISSN 0042-9686 (doi:10.2471/BLT.11.099424
  10. Kawamoto K. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005;330:765–8. doi:10.1136/bmj.38398.500764.8F PMID:15767266
  11. Gallacher K, Morrison D, Jani B, Macdonald S, May CR, Montori VM, Erwin PJ, Batty DG, Eton DT, Langhorne P, Mair FS. Uncovering Treatment Burden As A Key Concept For Stroke Care: A Systematic Review of Qualitative Research.  PLOS MED. 2013); 10(6).; . Back to example
  12. Gallacher K, Jani B, Morrison D, Macdonald S, Blane D, Erwin P, May CR, Montori VM, Eton DT, Smith F, Batty DG, Mair FS.  Qualitative systematic reviews of treatment burden in stroke, heart failure and diabetes: methodological challenges and solutions.  BMC Medical Research Methodology 2013;13(10). Back to example.
  13. Maclean N, Pound P, Wolfe C, Rudd A (2000) Qualitative analysis of stroke patients' motivation for rehabilitation. BMJ 321: 1051. doi: 10.1136/bmj.321.7268.1051.
  14. Dixon-Woods M (2011) Using framework-based synthesis for conducting reviews of qualitative studies. BMC Med 9: 39 doi:10.1186/1741-7015-9-39.