Data collection tools

Introduction

NPT can be used as a framework for the development of survey instruments, to enable the collection of quantitative data in a standardised format. If you are looking for a 'ready made' instrument for measuring implementation processes based on NPT, then the NoMAD instrument is available for use.

However, the NPT does not in itself prescribe the questions that must used as measures of the theory, and you may prefer to develop your own structured questions. 

 

In this section, we draw on our experiences of developing NPT based measures and outline some possible ways of converting the theoretical constructs into structured questions for use in quantitative research. This involves two key areas which are to be discussed: (i) translating theoretical constructs into questions; and (ii) finding appropriate ways of framing survey items with appropriate response categories/scales.

1. Translating theoretical concepts into survey items

The theoretical constructs in the NPT can be used as a starting point for the construction of survey items. For example, let's consider the mechanism of Coherence, and the three propositions that the NPT offers regarding it:

1.1 Embedding is dependent on work that defines and organizes a practice as a cognitive and behavioural ensemble.

1.2 Embedding work is shaped by factors that promote or inhibit actors' apprehension of a practice as meaningful.

1.3 The production and reproduction of coherence in a practice requires that actors collectively invest meaning in it.

The work of translation might begin either by (a) translating the propositions or construct definitions themselves into plain language statements relevant to your own context of study, or (b) using the propositions or construct definitions in their theoretical form as a basis for developing a set of appropriate questions for your own context of study. It is likely that the development of structured items for your own survey would involve some combination of both of these approaches.

To take approach (a) for example, you might consider the meaning of terms such as 'embedding' and 'embedding work' in the context of your study. If the practice in question is use of a computerised decision support tool, then what do you consider to be important aspects of embedding - is it about how often it gets used? Is it about it becoming easier to use? Is it about being integrated and linked with other existing practices? The point here is, that concepts specified within the NPT can only be translated into meaningful terms in the context of what you are using the NPT to study. Looking again at the propositions for Coherence, what 'cognitive' (psychological/attitudinal) and 'behavioural' (actions) are relevant to making a particular practice that you are studying meaningful?

Translation work of this kind is likely to be necessary and ongoing throughout the course of using the NPT, whether this is for quantitative or other uses of it. An alternative starting point for the development of survey items however, may be to use the propositions or definitions of constructs and/or their components to generate questions that are relevant to your study. Taking the definition offered for the construct of Coherence, some questions that might be asked about your practice of interest might include:

  • Does it make sense?
  • Is it easy to describe or explain?
  • Is it clearly distinct from other ways of doing the job?
  • Does it have a clear purpose?
  • Does it offer benefits that are likely to be valued by those involved?

This kind of translation work is aimed at drawing on the NPT to identify important areas of questioning for inclusion in your study. The questions in the form above provide some focus for the development of items for inclusion in a survey. At present, the questions imply a 'yes/no/maybe' answer. Further development of questions into survey items however involves considering what kind of data you wish to collect in a structured survey. This is considered in the next section, however, on the issue of translation from theory to questions there are some things to consider.

Things to consider:

  • In using the NPT in this way, you may find that you either interpret concepts differently from how someone else may interpret them, or you may place more emphasis on some aspects and less on others. That's okay - what is important is that you are able to translate the concepts used in the theory for the development of questions that are relevant to your study.
  • Translation work generally involves breaking down more encompassing statements (like the propositions for coherence) into multiple component parts. This can be done to various levels of detail - how detailed you wish to do this is up to you and may be dependent on the requirements of your study. Something you may wish to consider though - if you are looking to develop construct-level measures that include combinations of individual questions - is how you might 'build' these back up. For example, if a description of the component: Contextual Integration (Construct: Collective Action) framed in relation to e-health  says '....the extent to which organisational effort is allocated to an e-health in proportion to the work that the system is intended to do' it can be deconstructed into two statements:
    • 'sufficient organisational effort has gone into supporting the system' and
    • 'the rewards of using the system outweigh the effort'

Depending on the context of your own study, you may wish to use and analyse simplified statements separately, or you may with to combine them back together, for example by creating a composite score between two or more component items.

  • Sometimes, it may not be possible to include a large number of detailed questions in a survey. Another way of dealing with the translation of complex theoretical constructs into questions is to preface questions with definitions of terms/scope, and/or explanations that help the person answering the questions to respond in a focused way. For example, a set of coherence questions about whether or not a practice is considered 'meaningful', could be prefaced by an explanation such as 'meaningful in this context refers to easy to understand/has value for individuals/has a clear purpose'.
  • It is possible that you may be using the NPT alongside other theoretical approaches, some of which may have been developed to different degrees for survey research. The kind of translation work described in this section should help you in identifying how the concepts of the NPT do or do not translate into questions that overlap with, or contribute additionally, to other theoretical approaches you may wish to include in your study.

2. Framing survey items and developing response formats

Once you have a set of questions or items that reflect the content that you wish to include in your survey, you need an appropriate way of framing them for data collection. How to do this depends on the kind of information you want to get from your survey.

One way of framing questions based on the NPT is to focus on the impact of a new practice. Drawing on the NPT (for example, Construct: Collective Action; Component: Relational Integration), a potential question may be framed as follows:

How much does [X] affect the distribution of work between individuals?

[ ] Not at all
[ ] A bit
[ ] Moderately
[ ] Quite a lot
[ ] Greatly

This question is intended to generate data about the impact of X, and as such the question is framed with reference to a previous state. That is, the question is asking the respondent to make an assessment of how things are changing/have changed with the introduction of the new practice [X]. Since the NPT is interested in questions about normalisation - processes that involve change of some kind - questions about impact are an appropriate way of framing questions in a study using the NPT. Depending on the response categories used for a question such as that presented above, the question could be used to assess impact in a number a ways - it could be about the degree of impact, it could be about direction (ie. Positive or negative), or a combination of both. For example, a similar approach to questioning about impact in a way that includes assessment of the direction of impact could be:

What is the impact of [X] on the distribution of work between individuals?

[ ] Extremely negative
[ ] Quite negative
[ ] More negative than positive
[ ] Not noticeable
[ ] More positive than negative
[ ] Quite positive
[ ] Extremely positive

Both types of questions can be used to assess the amount of impact that a practice [X] has with respect to mechanisms and components in the NPT. The choice of questioning format depends on the objective of your particular study.

So far, the types of question framing suggested are ways of assessing the impact of a particular practice, although (implicitly) they invite comparison to a previous state in which [X] wasn't in use. Alternative question framings could be developed that make direct comparisons with other possible practices, or (more explicitly) with standard previous practice. For example, you may wish to use the concepts of the NPT to develop questions for assessing a new practice [X] 'compared with' previous practice [Y].

Another approach to developing a frame for questioning is to use the popular 'likert' type format, in which you develop a series of statements which are then rated by individuals on a scale of agreement (often ranging from 'strongly agree' to 'strongly disagree'). The concepts used by the NPT are amenable to translation into such statements for rating in this way, for example:

  • The staff who work here have a shared understanding of what the e-health system is used for (Construct:Coherence)
  • The staff here are committed to making this e-health system work (Construct: Cognitive participation)
  • There are ongoing mechanisms for monitoring and appraising how this e-health system is used (Construct:Reflexive Monitoring)

Things to consider:

  • Choosing an appropriate way of framing your questions depends entirely on the kind of information you want your survey to collect - do you want the respondent to make comparisons? If so, with what (eg. Other practices, or over time)?
  • The point above relates mainly to matching your approach to question framing with the design of your study. For example, if you want to do a detailed study of one particular practice in a specific setting then you could frame your questions as specifically as possible to include named practices/systems, common scenarios of use, particular groups of users, etc. Alternatively, if the objective of your study is to use surveys to make comparisons between (a) different practices within a site, or (b) the same practice but across different sites, then you will probably need to try to frame your questions more generically so that the same questions can be given for the different situations you are assessing in your study. It is also possible to combine elements of both objectives (specific but also generic), for example by using a common set of theory-derived questions across sites/practices but to preface them with detailed information about aspects of context that the respondent should think about when answering the questions. For example, the question set could be preceded by an introductory paragraph that explains the practice of interest - for example, if it's a new technology, some references to the technology/system by name, perhaps how long it has been in use for in the participant's organisation, and the kinds of staff/individuals who are likely to be using it.
  • For sets of questions that have a common response format/scale (ie. the same options for every question), you can consider mixing the framing of the statements/questions in terms of positive or negative emphasis. For example, the positively framed statement 'the e-health system is easy to use' could be framed negatively as 'the e-health system is difficult to use'. Mixed framing in this way is sometimes used to encourage participants to read questions carefully, and/or to identify contradictory responding (where similar questions framed in the opposite direction have been endorsed using a similar point on the response scale).
  • Particular study designs will require response scales that allow for a greater degree of quantification, and that are thus more sensitive to picking up differences in responding (ie. that offer a greater number of levels for the respondent to choose from) than those with a limited (eg <7) range of response categories. Two examples of such designs include longitudinal surveys which collect data at different time points and require measures to be sufficiently sensitive to detect change over time; and scale development work, in which the measures must collect data that can be considered sufficiently continuous (ie on a 'scale') for anlaysis using statistical techniques (eg. factor analysis).