Which of the following threats can impact the internal validity in this experimental design?

Experimental Design

Overall, experiments require manipulating a single variable while controlling all other variables, in order to see if manipulating a variable actually produces any changes in other variables.

Now, we are going to discuss some of the things that can go wrong when doing experiments, and how these problems can be avoided.

I Problems associated with Experimentation

There are two basic problems that often occur when doing experimental research.

A Threats to Internal Validity

Remember, when doing experiments, our goal is to observe a change in a variable that can't be attributed to anything other than the variable we have manipulated. Well, sometimes there are problems that occur during experiments, which don't allow us to rule out other possible factors.

Threats to internal validity, are those problems or factors that can sometimes interfere with out ability to identify causal relationships.

In other words, threats to internal validity refers the possibility that other factors caused the outcome. That is, we cannot rule out the possibility that the change that was observed was actually do to the change in the variable manipulated by the researcher.

Common threats to internal validity that are often encountered during experimental research.

1. History

Things that happen during the course of the study (usually outside of the experimenters control), that influence people's reactions.

2. Maturation

Sometimes people change during the course of the study (become hungry, tired, moody), and this change has nothing to do what the variable that was being manipulated. People simply change over time.

3. Testing

Often times research involves testing people before and after the manipulation of the independent variable.

Unfortunately, this is typically not a very good way to do research, because when people are given the same questionnaire or test, they typically do one of two things:

a. they score much higher on tests involving cognitive abilities

People are very quick learners. If you give the same test to the same person, or a very similar test to the same person, their score almost always goes up the second time around. For example, if you take you SAT or ACT exams, twice, you will almost always do better the second time around. People just simply get better at things as they do them more.

b. they give more socially appropriate answers.

People are also very clever at figuring out what people are trying to find out. So, if someone gives you a questionnaire, most of the time we can figure out what they are trying to measure, even if it is somewhat hidden.

Moreover, once people figure out what people are actually trying to get at, we give more socially appropriate answers.

So, if you ask someone to fill out a questionnaire which tries to determine how prejudicial they are and they somehow figure this out. The next time around, they'll provide more socially appropriate answers.

4. Statistical Regression

This threat to internal validity refers to the idea that extreme scores typically move toward the center of the distribution overtime.

In other words, when you test someone and they do extremely poor or extremely well at something, it is very unlikely that they would score the same the second time around.

Statistical Regression becomes a problem in experimental research, when people are studied because of their extreme score on a given measure.

Example:

Let's say I want to test the effectiveness of putting students into study groups for the next exam. In particular, let's say that I want to put the students who scored at the low end of the scale into a study group, with the hope that it will improve their test scores. So, on the next exam, I notice that they did much better. I might think that the study group really helped them out. However, it probably had very little to do with the study group. Rather the difference was simply due to the fact, that people tend to regress towards the mean. That is, extreme scores, typically don't stay that way. They move closer to the mean.

5. Mortality

Mortality refers to the fact that sometimes people drop out of the experiment. That is, some people quit or don't make it to the end of a study.

Most of the time, mortality is not a problem, unless people are dropping out for a particular reason, or the same type of people are dropping out.

6. Selection

Selecting refers to the fact that there are systematic differences between the control and experimental group. In other words, the two groups are not equal to each other to start out with.

This is typically not a problem when people are assigned to groups using random assignment. However, not all of the time can people be randomly assigned to groups.

Remember, sometimes matching is used, or sometimes naturally occurring groups of people are used in experiments.

When matching is used, it is quite possible that you could end up with two groups that are very different from each other in some unknown way.

In short, if the groups are not equal to begin with it is very difficult to tell if differences observed are due to the manipulation of the independent variable or because the groups were different from the start.

7. Demand Characteristics

When subjects try to "be good subjects"

When doing research people typically act differently than they normally would.

And for the most part, people try to be good subjects. They try to figure out what is going on, and try to provide the results wanted.

B Threats to External Validity

Doesn't deal with whether or not the experiment was done correctly. Rather it deals with whether or not the results generalize to others.

In other words, external validity refers to the fact that what was learned is not applicable to other people and situations.

1. Non-representative samples

Most of the time, experimental researchers do not rely on sample that represents a larger population of people, rather researchers typically rely on convenience samples.

2. Specific Conditions are Required

Sometimes the very nature of the experiment is so contrived that the behaviors being studied probably do not occur very often outside laboratory settings.

The more contrived or unnatural the task or situation that people are placed into, typically reduces the likelihood that the results are applicable to many situations that occur outside of the laboratory setting.

In summary, threats to internal validity refer to things that can prevent a causal relationship from being clearly identified, while threats to external validity refer the inability to apply what is found to other people and situations.

Now, that we have covered some of the problems that can occur. Let's talk about different types of experimental designs and the problems associated with each type of design.

How do you eliminate these problems?

1. Exercise great deal of control

Many of the problems we have talked about can be taken care of to some degree by exercising a great deal of experimental control.

2. Select the proper experimental design

II Types of Experimental Designs

What we are going to talk about now, is the different ways that experiments can be done. There is a great deal of variation in how people actually go about doing experiments. Some of the designs used are better than others.

A Pre-experimental Designs

The first type of design I want to talk about is Pre-Experimental Designs. Pre-experimental designs are often employed because they are easy to use and don't take a lot of time and effort.

Unfortunately, Pre-Experimental Designs do not control for threats to internal validity very well.

To compare the designs with each other. Let's go through an example, illustrating how each design would be used.

X = Manipulated stimulus (Manipulation vs. Control)

O = Observation or measure

R = Random assignment (to experimental or control group)

1. One-shot case study *

Basically in this study one group was used. All of the people were exposed to the same stimulus and then they were measured in light of some variable.

In the corporate world, many studies are done this way. People will be exposed to a workshop, or some type of training session and then they will be measured to see what effect the training sessions or workshop had.

What types of problems are there with this type of design.

Even though it is commonly used, it doesn't reveal much useful information.

Problems:

Cannot make a causal statement because there is no point of comparison.

Let's say you did a workshop on motivation, and then you find out that everyone is very motivated. Well, it is impossible to tell if people's motivational level is due to your workshop or not. It may not have anything to do with it at all. In fact, it is possible that people were even more motivated before they took your workshop. You simply have nothing to compare your results with.

Threats to internal validity:

History

Change (if change occurred at all) might have occurred due to some factor external to the experiment.

Maturation

Change (if change occurred at all) might occur simply due to time passing.

2. One group pretest-posttest design*

This type of design is often used in organizations as well. People are measured, exposed to some kind of treatment, and then they are measured again.

The only advantage to using this design is that you can compare people's responses with each other. That is, you can compare how they measured before the treatment and how they compared afterward.

With that said, however, this design is not much better than the last design.

Threats to internal validity:

History

Change might have occurred due to some factor external to the experiment

Maturation

Change might occur simply due to time passing

Testing

Change might occur due to being measured twice

3. Static Group Comparison *

In this design. Two different groups are being compared with each other. One group is exposed to the manipulation and the other group is not. The two groups scores are compared with each other.

In some sense, this experimental design is much better than the last two. If the study is done over a relatively short period of time, history is not likely to be a threat.

Again, if study is done quickly, maturation is not likely to be a very big problem.

However, there is one serious threat to internal validity.

Selection

Groups might not have been equal to begin with. Any differences noticed, may be due to differences present before experimental treatment.

B True Experimental Designs

1. Pretest-Posttest control group design*

This design represents a true experiment. In this case, people are randomly assigned to either the control of the experimental group.

This design is useful, because it lets you make comparisons between people who were exposed to the manipulation and those who were not.

In particular, this design lets you rule out many threats to internal validity

History

If some event occurs that influences people, well it should effect both groups equally. Therefore, differences between groups, must be due to the manipulation of the independent variable.

Maturation

Should effect both groups the same again. Differences between the groups can't be due to the fact that people change, because people in both groups should be changing.

Testing

Effects can't be due to testing, because both groups scores are being tested twice. Therefore any difference between the two groups the second time around must be due to chance or the manipulation of the independent variable.

Selection

Is not a problem. Random assignment will produce equal groups before the films are shown. Moreover, you can test this by comparing the observations of the two groups before the manipulation of the independent variable. Groups should be equal to begin with, or else random assignment failed.

Overall, this type of design, if done properly controls or eliminates problems with internal validity.

However, this type of design does pose one problem when it comes to External Validity.

Specific Conditions
The results may be due the specific conditions used in the study. That is, the same results might only be obtained when the exact same conditions are carried out and these results may not apply to the real world.

In particular, this type of design suffers from the possibility of a testing-treatment interaction.

In other words, there is an interaction between observing people and then exposing them to the manipulated variable – it may produce very different results than just exposing people to the manipulated variable.

2 Posttest Only control group design*

This type of design also controls for threats to internal validity.

This type of design also controls for threats to external validity -- No testing-treating interaction.

Next Lecture

Back to Lectures Page

What are the threats to the internal validity of experimental designs?

What are the 3 main threats to study validity?

What are the threats of validity in a Experiment?

What is main threat to the internal validity of a time series design?