PPA 696 RESEARCH METHODSEXPERIMENTAL DESIGNS FOR RESEARCHCausalityExperimental Designs Control Group Pre-test/Post-test Design Threats to Internal Validity Threats to External Validity Post-Test only Control Group Design CAUSALITYTo establish whether two variables are causally related, that is, whether a change in the independent variable X results in a change in the dependent variable Y, you must establish: 1) time order--The cause must have occurred before the effect; 2) co-variation (statistical association)-- Changes in the value of the independent variable must be accompanied by changes in the value of the dependent variable; 3) rationale-- There must be a logical and compelling explanation for why these two variables are related; 4) non-spuriousness-- It must be established that the independent variable X, and only X, was the cause of changes in the dependent variable Y; rival explanations must be ruled out. To establish causality, one must use an experimental or quasi-experimental design. Note that it is never possible to prove causality, but only to show to what degree it is probable.EXPERIMENTAL DESIGNSTrue experimental designs include: Pre-test/Post-test control group design Solomon Four-Group design Post-test only control group designPre-test/Post-test control group design Show
R O1 X O2 This diagram can be expanded upon as in the following table:
The difference in the control group's score from the pre-test to the post-test indicates the change in the value of the dependent variable that could be expected to occur without exposure to the treatment (independent) variable X. Control group - control group = control group difference The difference in the experimental group's score from the pre-test to the post-test indicates the change in the value of the dependent variable that could be expected to occur with exposure to the treatment (independent) variable X. Experimental group - experimental group = experimental group difference The difference between the change in the experimental group and the change in the control group is the amount of change in the value of the dependent variable that can be attributed solely to the influence of the independent (treatment) variable X. Control group difference - experimental group difference = difference attributable to X This can be illustrated by the following experiment to see whether participation in small group discussions would improve medical students' ability to respond to emotional needs of patients:
The control group used emotional words an average of .89 times per student (in three case studies) on the pre-test and an average of 1.13 times per student (in three case studies) on the post-test. The difference in the control group's score from the pre-test to the post-test is +.24 times per student. This indicates the change in using emotional words that could be expected to occur with regular course work only. The experimental group used emotional words an average of .68 times per student (in three case studies) on the pre-test and an average of 2.02 times per student (in three case studies) on the post-test. The difference in the experimental group's score from the pre-test to the post-test is +1.34 times per student. The experimental group's score from the pre-test to the post-test indicates the change in using emotional words that could be expected to occur with regular course work plus the small group discussions. The difference between the change in the experimental group (+1.34) and the change in the control group (+.24) is +1.10. This is the amount of change in using emotional words that can be attributed solely to the influence of the small group discussions. The controlled or true experimental design allows the researcher to control for threats to the internal and external validity of the study. Threats to internal validity compromise the researcher's ability to say whether a relationships exists between the independent and dependent variables. Threats to external validity compromise the researcher's ability to say whether this study's findings are applicable to any other groups. Controlling for Threats to Internal Validity1) History: did some other current event effect the change in the dependent variable? No, because both groups experienced the same current events.2) Maturation: were changes in the dependent variable due to normal developmental processes? No, because both groups experienced the same developmental processes. 3) Statistical Regression: did subjects come from low or high performing groups? Differences between the two groups that could influence the dependent variable would be controlled for as subjects were generally equivalent at the beginning of the research. 4) Selection: were the subjects self-selected into experimental and control groups, which could affect the dependent variable? No, the subjects were assigned by strict random selection and all had equal chance of getting the treatment or control condition. 5) Experimental Mortality: did some subjects drop out? did this affect the results? About the same number of students made it through the entire study in both the experimental and control groups, so there appears to be no bias. 6) Testing: Did the pre-test affect the scores on the post-test? Both groups got a pre-test; but a pre-test may have made the experimental group more sensitive to the treatment. 7) Instrumentation: Did the measurement method change during the research? The measurement method and instruments did not change. 8) Design contamination: did the control group find out about the experimental treatment? did either
group have a reason to want to make the research succeed or fail? The researcher must do some qualitative investigation to find out if there was design contamination. Controlling for Threats to External Validity1) Unique program features: There may have been an unusually motivated set of facilitators for the small group discussions.2) Effects of Selection: Probably applicable to other medical students. 3) Effects of Setting: Medical schools have their own cultures; doubtful if this would be applicable to other types of students. 4) Effects of History: No information given 5) Effects of Testing: No information given 6) Reactive effects of experimental arrangements: It would be best to replicate the results in other medical schools. Post-Test Only Control Group DesignThis design follows all the same steps as the classic pre-test/post-test design except that it omits the pre-test. There are many situations where a pre-test is impossible because the participants have already been exposed to the treatment, or it would be too expensive or too time-consuming. For large enough groups, this design can control for most of the same threats to internal and external validity as the classic controlled experimental design. For example, it eliminates the threat to internal validity of pre-testing by eliminating the pre-test. It may also decrease the problem of experimental mortality by shortening the length of the study (no pre-test).For small groups, however, a pre-test is necessary. Also, a pre-test is necessary if the researcher wants to determine the exact amount of change attributable to the independent variable alone. Which of the following statements is true of the Solomon fourWhich of the following statements is true of the Solomon four-group evaluation design used for the evaluation of a company's training program? It controls for most threats to internal and external validity.
Which of the following statements is true of Solomon fourWhich of the following statements is true of Solomon four-group evaluation design? It combines both pre-test/post-test comparison group and post-test-only control group design.
Which design is the most accurate and reliable to evaluate training effectiveness?The Kirkpatrick Taxonomy Model
Kirkpatrick Taxonomy is one of the most widely used methods for evaluating the effectiveness of corporate training programs. Developed and designed by Don Kirkpatrick, the framework offers a comprehensive four-level strategy to evaluate the effectiveness of any training course or program.
Which of the following statements is true about application assignments that are used in training programs?Which of the following statements is true about application assignments that are used in training programs? They help trainees understand the relation between the learned skills and their applications in the real world.
|