Table of Contents
8.0 Research Design
One part of the research process that we have not yet discussed are the methods by which data, or observations, can be collected on the units of analysis in a study population, or sample.
The plan for carrying out this observational step in the research process is known as the research design.
Research Design -- A plan consisting of a set of research methods by which the data required to test a hypothesis will be collected.
8.1 Experiments
One important type of research design is the experiment, also known as experimental design. Experiments are perhaps the most powerful type of research design for testing causal relationships between an independent variable and dependent variable.
Experiment -- Research design where: (a) a researcher is able to manipulate exposure to the independent variable by the units of analysis under study; and (b) there is random assignment of units to groups exposed to different levels of the independent variable.
A classic example of an experimental design is known as the Pretest-Posttest Control Group Design.
Time 1 Time 2 Treatment Group R O X O Control Group R O O Where: O = Observation X = Exposure to independent variable or "treatment" R = Random assignment to each group
With this experimental design, units are randomly assigned to the treatment group or the control group. Scores on the dependent variable are then measured at time 1 for units in both the treatment group and control group.
Units in the treatment group are then exposed to the independent variable. After a specified length of time, scores on the dependent variable are then measured again at time 2 for both the treatment group and the control group.
Let's say the independent variable is hypothesized to have a positive causal effect on the dependent variable. If the hypothesis is true, then scores on the dependent variable for the treatment group should have increased more than scores for the control group between time 1 and time 2, since only the treatment group was exposed to the independent variable.
Experimental designs help rule out a spurious relationship due to the ability to randomly assign units to the treatment group and control group. As we discussed in class, there are other factors that could influence a change in the dependent variable besides exposure to the treatment (i.e. independent variable).
With the random assignment of units to the treatment group and the control group, however, the odds are that these other factors (or independent variables) would be equally represented in both groups. Therefore, any change in the dependent variable resulting from these other variables would also be equally represented in both groups.
As a result, a bigger decrease in scores on the dependent variable for the treatment group compared to the control group can be more validly attributed to the independent variable.
The ability to better test and infer causal relationships is what makes experimental research designs powerful and valuable.
However, one problem with experiments is that with most social science research problems, it is impossible for researchers to manipulate the independent variables. In most cases, social scientists have no control over who is exposed to the independent variable, or how much he/she is exposed.
For example, in studying the relationship between migration rates and crime rates in an urban neighborhood, a researcher has no control over which neighborhoods people move into or out of, nor does he/she have any control over when people do this.
A second problem with experiments is that in order to get control over manipulating independent variables, it is often necessary to use artificial or contrived settings in a laboratory.
If an experiment is done in a contrived setting, it can influence the behaviors of those participating, particularly if they are aware that they are participating in a research study.
8.2 Survey Research
A much more commonly used research design for collecting data in social science research is the use of survey research, which involves the following steps:
1. The researcher constructs a survey questionnaire which consists of a set of questions (i.e., indicators) designed to measure the dependent variable and independent variables of interest.
2. A study population of respondents is then identified and a sample of respondents is selected if appropriate.
Respondents are typically individual persons, but they could also be persons representing organizations or groups if that is the unit of analysis.
3. The respondents selected for the study are then administered the survey questionnaire where they provide responses to the survey questions.
4. The responses to the survey are then processed and statistically analyzed in order to test study hypotheses.
A major strength of survey research is that it allows data on a diverse range of variables to be collected and analyzed.
There are 3 primary methods for administering a survey. 1. Mail survey -- survey is sent by respondents by mail, filled out by respondents, and mailed back to researcher.
2. Personal interviews -- respondents are met in person by an interviewer, the interviewer administers the survey and records the responses to survey questions.
3. Telephone survey -- Respondents are contacted over the telephone by an interviewer, the interviewer administers the survey and records the responses to survey questions.
One way in which the 3 methods vary is by “who” administers the survey questionnaire.
With personal interviewers and telephone surveys, the questionnaire is generally administered to respondents by a trained interviewer.
With mail surveys, the questionnaire is “self-administered” by the respondent, without the presence of an interviewer.
A major advantage of having an interviewer administer the questionnaire is that this allows “feedback” from respondents during the survey.
A mail survey does not allow the advantage of respondent feedback, making it even more critical that the instructions for the survey are clear, and that each survey question is well-constructed and can be clearly understood by respondents.
One major advantage of having a survey that is self-administered by the respondent is that it provides anonymity. Respondents may be more willing to truthfully answer sensitive questions that they would not answer if an interviewer were present.
As a technique for collecting data, a major limitation of surveys is the problem of measurement error. Error in the measurement of study variables can result from a number of sources in surveys.
First, as we previously discussed, is the problem of nonresponse. This occurs in two ways.
First, a number of respondents always refuse to answer the questionnaire. Second, some respondents may answer a questionnaire, but refuse to answer specific questions (e.g. income).
Nonresponse bias can occur if there is a systematic pattern to the nonresponse which somehow influences the study findings.
Also accurate measurement with surveys depends upon: (a) respondents being truthful in their responses; (b) being able to accurately recall events or make accurate calculations (if necessary); and (c) respondents having the same interpretation of the meaning of a survey question as all other respondents.
It is probably safe to say that there is some measurement error involved in all surveys. The question is whether it is extensive enough to make inferences drawn from the data to be inaccurate.
8.3 Comparative Research
Stark and Roberts use the term comparative research to refer to studies based on aggregate units of analysis.
Comparative Research -- Research based on aggregate units of analysis.
Groups and formal organizations can be categorized under the term social units since they are defined by social boundaries.
Geographic areas are also known as areal units
Comparative research thus involves some kind of data analysis or research investigation using either social units or areal units.
In using this type of research design, there a number of important issues that must be considered in order to ensure that the findings from such a study are trustworthy and valid.
Issue 1 -- Can the aggregate units be meaningfully compared with the measures being used?
Unless aggregate units are equal to one another, indicators using absolute numbers cannot be meaningfully compared across units.
E.G. Areal units (states) -- number of property crimes committed. States with more people are going to have a greater number of property crimes. Not because they necessarily have an environment that is more conducive to crime, but, because they simply have more people.
One way of resolving this problem is to use a standardized measure that is comparable across all aggregate units. Common examples of standardized measures are rates or percentages.
E.g. property crime rate = number of property crimes within an areal unit per 100,000 population. = (# property crimes/population) * 100,000
Issue 2 -- Do the data used in measures of aggregate units contain important omissions or biases?
Secondary data sets such as government statistics or surveys done by others may not be ideal for a given research purpose. Sometimes, such data sets omit important groups, or are biased in some way.
E.G. 1 The Uniform Crime Reports used in calculating crime rates in the U.S. and Canada are based on the number of arrests for a specific crime. Since they are based on arrests, they actually under-report the real number of crimes that are committed since many crimes are unsolved and/or no arrests are made.
These data would be especially biased for calculating the incidence of crime in counties or states with understaffed and/or inept police departments that make fewer arrests.
Issue 3 -- Are the measures taken from aggregate data valid indicators of the concepts they are intended to measure?
A key validity concern is the notion of face validity; that is, does the indicator logically seem to measure the concept it was intended to?
When a comparative research design is used, and secondary data are analyzed as part of this process, a limitation is typically placed on the study -- that is, the variables included in the data set influence the nature of the research that is done, rather than vice versa.
Issue 4 -- Is the number of units of analysis sufficient for valid statistical tests of hypotheses?
Some secondary data sets used for comparative research contain a small number of units. For example, the state data set in MicroCase has a total of 50 states or units.
The absolute number of units used in a statistical analysis influences the outcome of hypothesis tests. For example, values on the chi-square, z, and t-tests are all influenced by the number of units in a sample.
As a rule of thumb, there must be at least 30 units in sample in order to compute a valid statistical test. With any number less than this, statistical results may not be meaningful.
There are 2 basic approaches to comparative research. 1. Variable-oriented approach -- tests hypotheses by applying statistical techniques such as correlation and regression to variables based on an appropriate set of aggregate units of analysis.
2. Case-oriented approach -- selects a few units (2-5) and examines them in close detail in order the explain or understand the differences between them.
In sum, in using a comparative research design, it is important to determine which of these two approaches would be most effective in meeting the objectives of the research project.
If a variable-oriented approach is selected where the goal is to test hypotheses for aggregate units of analysis using secondary data, then it is important to consider the 4 issues discussed earlier, and identify any implications this might have for the research.
8.4 Field Research
A fourth type of research design is field research.
This design involves the researcher going out and personally observing the behavior under study in its natural setting.
A primary advantage of field research is that it is particularly well-suited to collecting detailed data on the social processes that produce a particular behavior. This is particularly the case for social processes that involve interpersonal interaction between people.
Let’s say that we conduct a statistical analysis and find a negative correlation between divorce rates and income levels using state level data. This statistical finding does not tell us much about the nature of the relationship between income and divorce.
It does not tell us the details of what it is about living under the conditions of a low income that makes divorce more likely among couples -- that is, the social processes that promote a higher incidence of divorce among low income couples.
This type of detailed information and insight into the substantive nature of a relationship between an independent variable and dependent variable cannot easily be obtained from quantitative analysis.
Quantitative analysis can tell us that there is a statistical relationship in a certain direction between a dependent variable and independent variable within a broader population. However, it does not tell us the details of the process by which this relationship works in the real social world.
This is where field research is valuable, because it provides the detailed insights and information needed to gain a deep understanding of the process by which a relationship between an independent variable and dependent variable actually works.
Studies using experiments, survey research, or comparative research with secondary data, typically yield numerical measures of study variables that can be used in quantitative (statistical) analysis.
E.G. In 1990, an average of 56.6% of households across the 50 states were headed by married couples.
Field research more typically yields observations that are not easily reducible to numbers. This is known as qualitative data.
Qualitative Data -- Data in a text-based and/or narrative format that describes or explains a behavioral phenomenon.
E.G. 1 — Views of Diversity in Missoula, MT “One of the things I like about Missoula is the diversity....you can go to a concert and be sitting next to somebody in a suit, dressed up, and somebody in blue jeans and hiking boots.…
there is a real appreciation for culture and activities that is going on. You can go to the farmer’s market on Saturday morning and see everything from dreadlocks to L.L. Bean...there isn’t any kind of pressure to dress a certain way, or to be a certain way. It’s like everything is kind of respectable. There’s a tolerance for diversity here.”
“There is not much cultural diversity here...It’s more.... that they (Anglos) have adapted to different people’s behaviors.....I’m thinking what they are thinking of as cultural diversity....is the university community, the hippies, and different kinds of people of that type.
Very rarely do you see an African American around here.... You will not see an American Indian in a high level position here......There are some African Americans, Asians, Hmong, Chinese, and American Indians....They are still at the very bottom of the wage level as far as acceptance.”
E.G. 2 Observations about the culture in an inner city neighborhood. There is a set of codes & norms for safely negotiating the streets in an inner city neighborhood known as “street wisdom.”
“When I reached the corner, after walking parallel to the stranger for a block, I waited until he had crossed the next street and had moved on ahead. Then I crossed to his side of the street; I was now about thirty yards behind him, and we were now walking away from each other at right angles.
We moved farther and farther apart. He looked back. Our eyes met. I continued to look over my shoulder until I reached my door, unlocked it, and entered. We both continued to follow certain rules of the street. We did not cross the street simultaneously, which might have caused out paths to cross a second time.
We both continued to “watch our backs” until the other stranger was no longer a threat. “ --- Elijah Anderson, StreetWise (1990:219)
Neither of these observations is expressed in exact numerical quantities. But, rather in a text format that conveys a particular meaning about the phenomenon being studied.
In sum, field research allows social behavior to be studied in its natural setting and provides deeper insights into the substantive nature of social life.
The rich, substantive detail of social life can rarely, if at all, be adequately captured through assigning numbers to units to represent behavior as in measuring variables for statistical analysis.
As a research design, field research involves the researcher going out into the “field” and personally observing the behavior under study.
There are a number of difficult issues involved in this. First, can the researcher get access to the people or objects he/she wants to study?
If the objects of study cannot be observed in “public” spaces where anyone can go, it is highly possible that the researcher will have to ask for permission to have access.
An inability to get access to the required field sites will end the study before it starts.
Getting access involves several additional issues. Should the reseacher identify his/her role as a social scientist conducting a research project?
There are several potential trade offs in making this decision. First, it may be necessary for a social scientist to reveal his/her identity in order to get access.
Second, there is the ethical issue of whether persons being observed should know that they are being observed for a research study.
On other hand, revealing one’s identity as a social scientist conducting a research study can have an effect on those being observed. They may consciously or unconsciously alter their behavior in responsee to knowing that they are being observed. This effect is known as the Hawthorne Effect.
Hawthorne Effect -- Modifications in the behavior of subjects that result from their knowing that they are under observation as part of a research study.
A second important issue is, “Should the social scientist directly participate in the behavior under study?”
Participation can provide a much deeper understanding of the objects under study through first hand sense experience.
On the other hand, it might raise ethical dilemmas for the researcher if the behavior under study is considered extremely deviant.
Further, by directly participating in the behavior, the researcher can affect the behavioral process under study. Thus, the behavioral processes being observed may not reflect the natural course of events due to the researcher’s influence.
Finally, another danger of directly participating in the behavior is that the researcher could “go native” and lose his/her ability to objectively analyze the situation.
Based upon the 2 factors -- whether the identity of the researcher is known and whether or not the researcher directly participates in the behavior under study -- Gold developed a typology of the 4 different roles that may be taken by social scientist engaging in field research:
1. Complete Participant -- The investigator’s identity as a researcher is unknown to the subjects being observed and the investigator fully participates in the behavior under study.
2. Participant as Observer -- The investigator’s identity as a researcher is known to the subjects being studied and the investigator fully participates in the behavior under study.
3. Observer as Participant -- The investigator’s identity as a researcher is known to the subjects being studied and the investigator interacts with the subjects, but does not fully participates in the behavior under study.
4. Complete Observer -- The investigator’s identity as a researcher is unknown to the subjects being studied, the investigator has no interaction with the subjects, and does not participate in the behavior under study.
There are no clear guidelines for determining which role should be used. The specific situation for a project involving field research must be evaluated and a judgment made on which role would work best.
If there are problems in getting access to the site needed to make observations, it may be necessary to reveal one’s identity, if it would help gain access.
One way of getting around this problem could be through the use of informants.
Informants are persons who have knowledge of the object under study, and is willing to share that knowledge with the investigator. This could include members of the group being studied and experts on the particular group or subject, among others.
Informants can provide knowledge and insight about the object of study, and may be able to help gain access to the field site.
|