Possibility that studies are not representative of all work done on a particular phenomenon

Indian J Anaesth. 2016 Sep; 60(9): 640–645.

Abstract

The conduct of research requires a systematic approach involving diligent planning and its execution as planned. It comprises various essential predefined components such as aims, population, conduct/technique, outcome and statistical considerations. These need to be objective, reliable and in a repeatable format. Hence, the understanding of the basic aspects of methodology is essential for any researcher. This is a narrative review and focuses on various aspects of the methodology for conduct of a clinical research. The relevant keywords were used for literature search from various databases and from bibliographies of the articles.

Key words: Bias, blinding, confounders, population, randomisation, research, sample size, study design, study tools

INTRODUCTION

Research is a process for acquiring new knowledge in systematic approach involving diligent planning and interventions for discovery or interpretation of the new-gained information.[1,2] The outcome reliability and validity of a study would depend on well-designed study with objective, reliable, repeatable methodology with appropriate conduct, data collection and its analysis with logical interpretation. Inappropriate or faulty methodology would make study unacceptable and may even provide clinicians faulty information. Hence, the understanding the basic aspects of methodology is essential.

METHODS

This is a narrative review based on existing literature search. This review focuses on specific aspects of the methodology for conduct of a research/clinical trial. The relevant keywords for literature search included ‘research’, ‘study design’, ‘study controls’, ‘study population’, ‘inclusion/exclusion criteria’, ‘variables’, ‘sampling’, ‘randomisation’, ‘blinding’, ‘masking’, ‘allocation concealment’, ‘sample size’, ‘bias’, ‘confounders’ alone and in combinations. The search engine included PubMed/MEDLINE, Google Scholar and Cochrane. The bibliographies of the searched articles were specifically searched for missing manuscripts from the search engines and manually from the print journals in the library.

The following text highlights/describes the basic essentials of methodology which needs to be adopted for conducting a good research.

Aims and objectives of study

The aims and objectives of research need to be known thoroughly and should be specified before start of the study based on thorough literature search and inputs from professional experience. Aims and objectives state whether nature of the problem (formulated as research question or research problem) has to be investigated or its solution has to be found by different more appropriate method. The lacunae in existing knowledge would help formulate a research question. These statements have to be objective specific with all required details such as population, intervention, control, outcome variables along with time interventions.[3,4,5] This would help formulate a hypothesis which is a scientifically derived statement about a particular problem in the defined population. The hypothesis generation depends on the type of study as well. Researcher observation related to any aspect initiates hypothesis generation. A cross-sectional survey would generate hypothesis. An observational study establishes associations and supports/rejects the hypothesis. An experiment would finally test the hypothesis.[5,6,7]

STUDY POPULATION AND PATIENT SELECTION, STUDY AREA, STUDY PERIOD

The flow of study in an experimental design has various sequential steps [Figure 1].[1,2,6] Population refers to an aggregate of individuals, things, cases, etc., i.e., observation units that are of interest and remain the focus of investigation. This reference population or target population is the group on which the study outcome would be extrapolated.[6] Once this target population is identified, researcher needs to assess whether it is possible to study all the individuals for an outcome. Usually, all cannot be included, so a study population is sampled. The important attribute of a sample is that every individual should have equal and non-zero chance of getting included in the study. The sample should be made independently, i.e., selection of one does not influence inclusion or exclusion of other. In clinical practice, the sampling is restricted to a particular place (patients attending to clinics or posted for surgery) or includes multiple centres rather than sampling the universe. Hence, the researcher should be cautious in generalising the outcomes. For example, in a tertiary care hospital, patients are referred and may have more risk factors as compared to primary centres where a patient with lesser severity are managed. Hence, researchers must disclose details of the study area. The study period needs to be disclosed as it would make readers understand the population characteristics. Furthermore, study period would tell about relevance of the study with respect to the present period.

Flow of an experimental study

The size of sample has to be pre-determined, analytically approached and sufficiently large to represent the population.[7,8,9] Including a larger sample would lead to wastage of resources, risk that the true treatment effect may be missed due to heterogeneity of large population and would be time-consuming.[6] If a study is too small, it will not provide the suitable answer to research question. The main determinant of the sample size includes clinical hypothesis, primary endpoint, study design, probability of Type I and II error, power, minimum treatment difference of clinical importance.[7] Attrition of patients should be attended during the sample size calculation.[6,9]

SELECTION OF STUDY DESIGN

The appropriate study design is essential for the intervention outcome in terms of its best possible and most reliable estimate. The study design selection is based on parameters such as objectives, therapeutic area, treatment comparison, outcome and phase of the trial.[6] The study design may be broadly classified as:[5,6,7]

1. Quantitative:

   1. Observational

      1. Descriptive: Case report, case series, survey

      2. Analytical: Case-control, cohort, cross-sectional

   2. Experimental: Randomised controlled trial (RCT), quasi-experiment

2. Qualitative.

For studying causality, analytical observational studies would be prudent to avoid posing risk to subjects. For clinical drugs or techniques, experimental study would be more appropriate.[6] The treatments remain concurrent, i.e. the active and control interventions happen at the same period in RCT. It may parallel group design wherein treatment and control groups are allocated to different individuals. This requires comparing a placebo group or a gold standard intervention (control) with newer agent or technique.[6] In matched-design RCT, randomisation is between matched pairs. For cross-over study design, two or more treatments are administered sequentially to the same subject and thus each subject acts as its own control. However, researches should be aware of ‘carryover effect’ of the previous intervention and suitable wash period needs to be ensured. In cohort study design, subjects with disease/symptom or free of study variable are followed for a particular period. The cross-sectional study examines the prevalence of the disease, surveys, validating instruments, tools and questionnaires. The qualitative research is a study design wherein health-related issue in the population is explored with regard to its description, exploration and explanation.[6]

Selection of controls

The control is required because disease may be self-remitting, Hawthorne effect (change in response or behaviours of subjects when included in study), placebo effect (patients feel improvement even with placebo), effect of confounder, co-intervention and regression to the mean phenomenon (for example, white coat hypertension, i.e. patients at recruitment may have higher study parameter but subsequently may get normal).[2,6,7] The control could be a placebo, no treatment, different dose or regimen or intervention or the standard/gold treatment. Avoiding a routine care for placebo is not desirable and unethical. For instance, for studying analgesic regimen, it would be unethical not to administer analgesics in a control group. It is advisable to continue standard of care, i.e. providing routine analgesics even in control group. The use of placebo or no treatment may be considered where no current proven intervention exists or placebo is required to evaluate efficacy or safety of an intervention without serious or irreversible harm.

The comparisons to be made in the study among groups also need to be specified.[6,7,9] These comparisons may prove superiority, non-inferiority or equivalence among groups. The superiority trials demonstrate superiority either to a placebo in a placebo-controlled trial or to an active control treatment. The non-inferiority trials would prove that the efficacy of an intervention is no worse than that of the active comparative treatment. The equivalence trials demonstrate that the outcome of two or more interventions differs by a clinically unimportant margin and either technique or drug may be clinically acceptable.

STUDY TOOLS

The study tools such as measurements scales, questionnaires and scoring systems need to be specified with an objective definition. These tools should be validated before its use and appropriate use by the research staff is mandatory to avoid any bias. These tools should be simple and easily understandable to everyone involved in the study.

Inclusion/exclusion criteria

In clinical research, specific group of relatively homogeneous patient population needs to be selected.[6] Inclusion and exclusion criteria define who can be included or excluded from the study sample. The inclusion criteria identify the study population in a consistent, reliable, uniform and objective manner. The exclusion criteria include factors or characteristics that make the recruited population ineligible for the study. These factors may be confounders for the outcome parameter. For example, patients with liver disease would be excluded if coagulation parameters would impact the outcome. The exclusion criteria are inclusive of inclusion criteria.

VARIABLES: PRIMARY AND SECONDARY

Variables are definite characteristics/parameters that are being studied. Clear, precise and objective definition for measurement of these characteristics needs to be defined.[2] These should be measurable and interpretable, sensitive to the objective of the study and clinically relevant. The most common end-point is related to efficacy, safety and quality of life. The study variables could be primary or secondary.[6] The primary end-point, usually one, provides the most relevant, reliable and convincing evidence related to the aim and objective. It is the characteristic on the basis of which research question/hypothesis has been formulated. It reflects clinically relevant and important treatment benefits. It determines the sample size. Secondary end-points are the other objectives indirectly related to primary objective with regard to its close association or they may be some associated effects/adverse effects related to intervention. The measurement timing of the variables must be defined a priori. These are usually done at screening, baseline and completion of trial.

The study end-point parameter may be clinical or surrogate in nature. A clinical end-point is related directly to clinical implications with regard to beneficial outcome of the intervention. The surrogate end-point is indirectly related to patient clinical benefit and is usually measures laboratory measurement or physical sign as a substitute for a clinically meaningful end-point. Surrogate end-points are more convenient, easily measurable, repeatable and faster.

SAMPLING TECHNIQUES: RANDOMISATION, BLINDING/MASKING AND ALLOCATION CONCEALMENT

Randomisation

Randomisation or random allocation is a method to allocate individuals into one of the groups (arms) of a study.[1,2] It is the basic assumption required for statistical analysis of data. The randomisation would maximise statistical power, especially in subgroup analyses, minimise selection bias and minimise allocation bias (or confounding). This leads to distribution of all the characteristics, measured or non-measured, visible or invisible and known or unknown equally into the groups. Randomisation uses various strategies as per the study design and outcome.

Probability sampling/randomisation

• Simple/unrestricted: Each individual of the population has the same chance of being included in the sample. This is used when population is small, homogenous and the sampling frame is available. For example, lottery method, table of random numbers or computer-generated

• Stratified: It is used in non-homogenous population. Population is divided into homogenous groups (strata), and the sample is drawn for each stratum at random. It keeps the ‘characteristics’ of the participants (for example, age, weight or physical status) as similar as possible across the study groups. The allocation to strata can be by equal or proportional allocation

• Systematic: This is used when complete and up-to-date sampling frame is available. The first unit is selected at random and the rest get selected automatically according to some pre-designed pattern

• Cluster: This applies for large geographical area. Population is divided into a finite numbers of distinct and identifiable units (sampling units/element). A group of such elements is a cluster and sampling of these clusters is done. All units of the selected clusters are included in the study

• Multistage: This applies for large nationwide surveys. Sampling is done in stages using random sampling. Here, sub-sampling within the selected clusters is done. If procedure is repeated in more number of stages, then they termed as multistage sampling

• Multiphase: Here, some data are collected from whole of the units of a sample, and other data are collected from a sub-sample of the units constituting the original sample (two-phase sampling). If three or more phases are used, then they termed as multiphase sampling.

Non-probability sampling/randomisation

This technique does not give equal and non-zero chances to all the individuals in the population to be selected in the sample.

• Convenience: Sampling is done as per the convenience of the investigator, i.e., easily available

• Purposive/judgemental/selective/subjective: The sample is selected as per judgement of investigator

• Quota: It is done as per judgement of the interviewer based on some specified characteristics such as sex and physical status.

ALLOCATION CONCEALMENT

Allocation concealment refers to the process ensuring the person who generates the random assignment remains blind to what arm the person will be allotted.[8,9,10] It is a strategy to avoid ascertainment or selection bias. For example, based on an outcome, researcher may recruit a specific category as lesser sicker patients to a particular group and vice versa to the other group. This selective recruitment would underestimate (if treatment group is sicker) or overestimate (if control group is sicker) the intervention effect.[9] The allocation should be concealed from investigator till the initiation of intervention. Hence, randomisation should be performed by an independent person who is not involved in the conduct of the study or its monitoring. The randomisation list is kept secret. The methods of allocation concealment include:[9,10]

• Central randomisation: Some centrally independent authority performs randomisation and informs the investigators via telephone, E-mail or fax

• Pharmacy controlled: Here, pharmacy provides coded drugs for use

• Sequentially numbered containers: Identical containers equal in weight, similar in appearance and tamper-proof are used

• Sequentially numbered, opaque, sealed envelopes: The randomised numbers are concealed in opaque envelope to be opened just before intervention and are the most common and easy to perform method.

BLINDING/MASKING

Blinding ensures the group to which the study subjects are assigned not known or easily ascertained by those who are ‘masked’, i.e., participants, investigators, evaluators or statistician to limit occurrence of bias.[1,2] It confirms that the intervention and standard or placebo treatment appears the same. Blinding is different from allocation concealment. Allocation concealment is done before, whereas blinding is done at and after initiation of treatment. In situations such as study drugs with different formulations or medical versus surgical interventions, blinding may not be feasible.[8] Sham blocks or needling in subjects may not be ethical. In such situation, the outcome measurement should be made objective to the fullest to avoid bias and whosoever may be masked should be blinded. The research manuscript must mention the details about blinding including who was blinded after assignment to interventions and process or technique used. Blinding could be:[8,9]

• Unblinded: The process cannot conceal randomisation

• Single blind: One of the participants, investigators or evaluators remains masked

• Double-blind: The investigator and participants remained masked

• Triple blind: Not only investigator but also participant maintains a blind data analysis.

BIAS AND CONFOUNDERS

Bias is a systematic deviation of the real, true effect (better or worst outcome) resulting from faulty study design.[1,2] The various steps of study such as randomisation, concealment, blinding, objective measurement and strict protocol adherence would reduce bias.

The various possible and potential biases in a trial can be:[7]

• Investigator bias: An investigator either consciously or subconsciously favours one group than other

• Evaluator bias: The investigator taking end-point variable measurement intentionally or unintentionally favours one group over other. It is more common with subjective or quality of life end-points

• Performance bias: It occurs when participant knows of exposure to intervention or its response, be it inactive or active

• Selection bias: This occurs due to sampling method such as admission bias (selective factors for admission), non-response bias (refusals to participate and the population who refused may be different from who participated) or sample is not representative of the population

• Ascertainment or information bias: It occurs due to measurement error or misclassification of patient. For example, diagnostic bias (more diagnostic procedures performed in cases as compared with controls), recall bias (error of categorisation, investigator aggressively search for exposure variables in cases)

• Allocation bias: Allocation bias occurs when the measured treatment effect differs from the true treatment effect

• Detection bias: It occurs when observations in one group are not as vigilantly sought as in the other

• Attrition bias/loss-to-follow-up bias: It occurs when patient is lost to follow-up preferentially in a particular group.

Confounding occurs when outcome parameters are affected by effects of other factors not directly relevant to the research question.[1,7] For example, if impact of drug on haemodynamics is studied on hypertensive patients, then diabetes mellitus would be confounder as it also effects the hemodynamic response to autonomic disturbances. Hence, it becomes prudent during the designing stage for a study that all potential confounders should be carefully considered. If the confounders are known, then they can be adjusted statistically but with loss of precision (statistical power). Hence, confounding can be controlled either by preventing it or by adjusting for it in the statistical analysis. The confounding can be controlled by restriction by study design (for example, restricted age range as 2-6 years), matching (use of constraints in the selection of the comparison group so that the study and comparison group have similar distribution with regard to potential confounder), stratification in the analysis without matching (involves restriction of the analysis to narrow ranges of the extraneous variable) and mathematical modelling in the analysis (use of advanced statistical methods of analysis such as multiple linear regression and logistic regression). Strategies during data analysis include stratified analysis using the Mantel-Haenszel method to adjust for confounders, using a matched design approach, data restriction and model fitting using regression techniques.

SUMMARY

Basic understanding of the methodology is essential to have reliable, repeatable and clinically acceptable outcome. The study plan including all its components needs to be designed before start of the study, and the study protocol should be strictly adhered during the conduct of study.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

REFERENCES

1. Prasad K. Fundamentals of Evidence Based Medicine. 2nd ed. India: Springer India; 2013. [Google Scholar]

2. Peat J, Barton B. Medical Statistics: A Guide to Data Analysis and Critical Appraisal. 1st ed. India: Blackwell Publishing Ltd; 2005. [Google Scholar]

3. Del Mar C, Hoffmann TC. A guide to performing a peer review of randomised controlled trials. BMC Med. 2015;13:248. [PMC free article] [PubMed] [Google Scholar]

4. Rothwell PM. External validity of randomised controlled trials: To whom do the results of this trial apply? Lancet. 2005;365:82–93. [PubMed] [Google Scholar]

5. Sessler DI, Imrey PB. Clinical research methodology 2: Observational clinical research. Anesth Analg. 2015;121:1043–51. [PubMed] [Google Scholar]

6. Stanley K. Design of randomized controlled trials. Circulation. 2007;115:1164–9. [PubMed] [Google Scholar]

7. Levin KA. Study design VII. Randomised controlled trials. Evid Based Dent. 2007;8:22–3. [PubMed] [Google Scholar]

8. Viera AJ, Bangdiwala SI. Eliminating bias in randomized controlled trials: Importance of allocation concealment and masking. Fam Med. 2007;39:132–7. [PubMed] [Google Scholar]

9. Chan YH. Randomised controlled trials (RCTs) – Sample size: The magic number? Singapore Med J. 2003;44:172–4. [PubMed] [Google Scholar]

10. Schulz KF, Grimes DA. Allocation concealment in randomised trials: Defending against deciphering. Lancet. 2002;359:614–8. [PubMed] [Google Scholar]

Articles from Indian Journal of Anaesthesia are provided here courtesy of Wolters Kluwer -- Medknow Publications