Onwuegbuzie, A.J.& Leech, N.L. (2005, March 10). A Typology of Errors and Myths Perpetuated in Educational Research Textbooks
Current Issues in Education [On-line], 8(7). Available:
A Typology of Errors and Myths Perpetuated in Educational Research Textbooks
Anthony J. Onwuegbuzie
University of South Florida
Nancy L. Leech
University of Colorado at Denver and Health Sciences Center
This paper identifies major errors and myths perpetuated by educational research textbooks. The most pervasive errors and myths advanced by methodology textbooks at the following eight phases of the educational research process are described: (a) formulating a research problem/objective; (b) reviewing the literature; (c) developing the research purpose/research question(s) and hypotheses; (d) selecting a research design/method; (e) collecting data; (f) analyzing the data; (g) interpreting/validating the data; and (h) communicating the findings. Second, the implications of these errors and myths are provided. It is shown how research textbooks can be reframed to break down the barriers between quantitative and qualitative research.
Table of Contents
Setting the Scene
Virtually all graduate students from the field of education are required to take one or more research methodology courses as part of their degree programs (Mundfrom, Shaw, Thomas, Young, & Moore, 1998; Onwuegbuzie, Slate, & Schwartz, 2001). Typically, an immediate goal of these courses is to prepare students to complete their theses and dissertations. More long-term goals of educational research courses include helping students to possess research consumer skills (i.e., the ability to read, to interpret, to synthesize, and to utilize research) and research production skills (i.e., the ability to design and to implement original research studies) (Onwuegbuzie et al., 2001; Ravid & Leon, 1995). Educational research textbooks play a vital role in achieving all of these goals by attempting to “provide a down-to-earth approach that helps students acquire the skills and knowledge required of a competent consumer and producer of educational research” (Gay & Airasian, 2003, p. iii).
Although many of the leading educational textbooks (e.g., Ary, Jacobs, & Razavieh, 2002; Best & Kahn, 2003; Charles & Mertler, 2002; Creswell, 2005; Fraenkel & Wallen, 2003; Gall, Borg, & Gall, 2003; Gay & Airasian, 2003; Gliner & Morgan, 2000; Gravetter & Forzano, 2003; Johnson & Christensen, 2004; Langenbach, Vaughn, & Aagard, 1994; Leary, 2004; Liebert & Liebert, 1995; McMillan & Schumacher, 2001; Punch, 1999; Rosnow & Rosenthal, 2005; Salkind, 2003; Wallen & Fraenkel, 2001) generally are well written, we contend that each one contains statements that are flawed, thereby permeating myths that lead to poor methodological practices. For example, most research methodology textbooks attempt to introduce educational research within the first two chapters. Unfortunately, these chapters are often filled with inaccuracies. In particular, in these chapters, many textbooks discuss the philosophical underpinnings of quantitative and qualitative research paradigms. However, with the exception of a very few textbooks (e.g., Creswell, 2005), no mention is made of the mixed methods paradigm; giving the impression that only two major research paradigms exist. Yet, mixed methods research represents an established third paradigm with a long tradition (Johnson & Onwuegbuzie, 2004; Tashakkori & Teddlie, 1998, 2003b). Similarly, when discussing the emergence of the quantitative (e.g., positivism) and qualitative (e.g., interpretivist) paradigms, no mention is made of the impact of pragmatism on research methodology, even though pragmatism can be traced back to 1861 with the writings of Charles S. Peirce, the “Father of Pragmatism” (Fisch, 1982); and even though pragmatism had “a profound effect on the conduct of scientific and social scientific research, and it accounts for the ready acceptance of ‘qualitative' methods today” (Maxcy, 2003, p. 75).
With this in mind, the purpose of the present paper is to identify the major errors and myths perpetuated by educational research textbooks. Specifically, the current article presents the most prevalent and pervasive errors and myths advanced by research methodology textbooks at various phases of the educational research process. Second, the implications of these errors and myths are provided. In particular, many of these errors advance the myth that research method as a technique (i.e., research design) and research method as logic of justification (i.e., research paradigm) are synonymous. In fact, research designs are presented in educational research textbooks as if only specific types of data can be collected and analyzed for each research design. For example, in outlining experimental designs, virtually all research texts give the impression that only quantitative data can be collected when these designs are used. Yet, there is no reason why qualitative data (e.g., interview responses) cannot be collected as part of the experimental design. Also, experimental and control groups can be compared with respect to qualitative information. Similarly, quantitative data (e.g., test scores) can be collected in qualitative studies (Onwuegbuzie & Leech, in press-a; Tashakkori & Teddlie, 2003a).
Finally, using Onwuegbuzie and Leech's (in press-a) framework, it is shown how research textbooks can be reframed to break down the barriers between quantitative and qualitative research. As such, instead of presenting research as representing a dichotomy of methodological choices that fall strictly either into the quantitative or qualitative research paradigm, it is contended that research should be presented as a holistic, reflective, and integrative process that helps students become both consumers and producers of educational research.
The current article is unique in at least two ways. First, it appears to be the first published paper that has critiqued all of the leading educational research textbooks simultaneously. Second, this article appears to be the only published work that has identified and documented errors made in both the quantitative and qualitative research sections of these textbooks within the same framework. As such, the present essay appears to represent the most wide-ranging discussion of errors and myths perpetuated in educational research textbooks to date.
Errors Found in Research Method Textbooks
A review of commonly used research method textbooks identified many of the major errors and myths reported and perpetuated. Only textbooks written for graduate education introductory research methods courses were included. We define “educational research” as research in any area (e.g., elementary education, curriculum and instruction, educational psychology) related to education.
The most prevalent and pervasive errors and myths advanced by research methodology textbooks at each of the following eight phases of the educational research process was analyzed, including: (a) formulating a research problem and research objective; (b) reviewing the literature; (c) developing the research purpose, research questions(s), and hypotheses; (d) selecting a research design/method; (e) collecting data; (f) analyzing the data; (g) interpreting/validating the data; and (h) communicating the findings. The myths and errors in each of these areas are outlined below.
Formulating a Research Problem and Research Objective
Many textbook authors discuss the sources of research problems as including personal experiences (e.g., Ary et al., 2002; Creswell, 2005; Gay & Airasian, 2003; McMillan & Schumacher, 2001), casual observations (e.g., McMillan & Schumacher, 2001), theories (e.g., Gay & Airasian, 2003), related literature (e.g., Gay & Airasian, 2003; McMillan & Schumacher, 2001), practical situations, current social and political issues (McMillan & Schumacher, 2001), and replications (e.g., Gay & Airasian, 2003). However, few textbooks discuss the important role of the research objective in the educational research process. Yet, this is a very important step in the process because it leads to the determination of whether the study is exploratory, confirmatory, or some combination of both.
The vast majority of research textbooks gives the impression that quantitative research methods are associated with deductive reasoning and qualitative methods are associated with inductive reasoning (Tahsakkori & Teddlie, 2003a). For instance, Gay and Airasian (2003) state: “An inductive research approach is typically qualitative in nature, while a deductive research approach is typically quantitative in nature” (p. 4). However, this creates a false dichotomy between the quantitative and qualitative research paradigms because both quantitative and qualitative research studies can involve both deductive and inductive reasoning, depending on the research objective.
As noted by Onwuegbuzie and Leech (in press-a), the research objective in both quantitative and qualitative studies can be classified as falling on a continuum from exploratory to confirmatory. Exploratory studies often involve the development of theories surrounding how and why a phenomenon exists as it does. Conversely, confirmatory studies involve testing hypotheses that arise from new or existing theories. Moreover, exploratory research objectives center on theory initiation and theory building, whereas confirmatory research objectives focus on theory testing and theory modification. Therefore, as contended by Newman and Benz (1998), exploration and confirmation are linked by theories. Thus, instead of emphasizing deductive/quantitative and inductive/qualitative combinations, as often occurs in the opening chapter of introductory-level research texts, we recommend that authors make it clear that (a) the reasoning process does not dictate the research paradigm and (b) research objectives drive the research methods used.
Reviewing the Literature
Most research texts include a chapter or a specific section on reviewing the literature. As noted by Gay and Airasian (2003, p. 16), a traditional literature review, which is undertaken in the overwhelming majority of studies, “involves systematically identifying, locating, and analyzing documents containing information related to the research problem. These documents can include articles, abstracts, reviews, monographs, dissertation, books, other research reports, and electronic media.” In other words, “a literature review is a written summary of articles, books, and other documents that describes the past and current state of knowledge about a topic” (Creswell, 2005, p. 79). Although many purposes of literature reviews have been provided, such as to plan for a study, to establish the importance of the research problem, to provide a rationale for the study, to reframe a body of literature, and to identify research strategies and specific data collection techniques that have been found to be effective/ineffective (Ary, et al., 2002; Creswell, 2005; Gall et al., 2003; Fraenkel & Wallen, 2003; Gay & Airasian, 2003; Johnson & Christensen, 2004), the most common purpose of a literature review is to synthesize the body of literature in a topic area of interest. As such, traditional literature reviews are inherently qualitative (Onwuegbuzie & Leech, 2003).
Unfortunately, the discussions of literature reviews contained in introductory-level research methodology textbooks typically contain several errors of commission and/or omission that can lead to misconceptions. In particular, this discussion often gives the impression that qualitative literature reviews can provide a neutral representation of the body of literature. However, as noted by Dellinger (2003), neutral representation is not possible because the “review of the literature is inherently an interpretive and value-driven process…[reflecting] the researcher's own story about what is deemed valid, worthwhile, meaningful and valuable in a set of studies” (p. 4).
In addition, introductory research textbooks fail to inform readers that literature reviews have a number of limitations. Most notably, the appropriateness, comprehensiveness, and representativeness of the literature cited in a review are affected by (a) selective inclusion of studies, often based on the reviewer's own subjective assessment of the quality of the underlying studies; (b) failure to examine the characteristics and attributes of the studies as potential explanations for consistent and contradictory findings across studies; (c) failure to consider the context under which each of the investigations took place; (d) differential subjective weighting of studies in the interpretation of a set of results; (e) misleading interpretations of study results; (f) confirmation bias, (g) over-reliance on findings from either quantitative or qualitative studies, and (h) failure to examine moderating and/or mediating variables in the underlying relationship (Onwuegbuzie & Leech, 2003; Wolf, 1986).
Similarly, some or all of the following nine errors identified by Dunkin (1996) emerge from qualitative literature reviews. We contend they should be mentioned in research text: (a) vague study selectivity, (b) arbitrarily assigning equal weight to each study, (c) double-counting study results, (d) incorrectly summarizing the details of one or more studies, (e) attributing findings to a study that are not justified, (f) failing to identify invalid interpretations and inferences, (g) failing to include contradictory findings, (h) generating misleading inferences in the research synthesis because of the aforementioned errors, and (i) failing to include all of a study's results relevant to the synthesis. Introductory textbooks also do not make it clear to readers that summarizing a subset of findings without a cumulative, systematic, and rigorous analysis of the entire set of results provides the reader with no information as to the confidence that can be placed in conclusion coherence (Onwuegbuzie & Leech, 2003). Also not discussed by textbook authors is how literature reviews often do not place the readers in a position to ascertain whether procedural or context variation affects study findings because the relationship between procedure/context and study results is not examined formally (Cooper, 1989).
The problems associated with literature reviews can culminate in inadequate and invalid syntheses of the extant literature. This, in turn, not only can adversely affect the quality of a single study, but also “can have cumulative effects on a body of literature that may manifest in a lineage of poor conceptualization, design, measurement, methodology, and ultimately, inferences” (Dellinger, 2003, p. 2). Thus, literature reviews contain flaws that affect the validity of ensuing inferences, thereby leading to a failure to represent accurately the accumulated body of knowledge.
Another error of omission that is apparent in virtually every textbook is the lack of encouragement for researchers conducting literature reviews to report indices of practical significance (e.g., effect sizes) corresponding to each statistically significant finding. Such a practice would facilitate the use of meta-analyses, in which the mean effect size across the studies is estimated and the variance in effect sizes across studies is examined as a function of selected design variables. Related to this concern, textbooks also should encourage researchers to cite meta-analysis studies in the literature review sections.
Developing the Research Purpose, Research Questions(s), and Hypotheses
Stemming from the research problem, objective, and, in many cases, the review of the literature, the research purpose, questions, and hypotheses delineate what will be researched. Some research textbooks (e.g., Gall et al., 2003) frame research questions in a way that is compatible only with quantitative research studies. Furthermore, these authors do not include research questions in qualitative sections; instead, the impression that is given to readers is that the nature of qualitative research is so open-ended and exploratory that specific questions are not necessary.
In every textbook, hypotheses are presented as belonging exclusively to quantitative research studies. This gives the impression that hypotheses are never tested in qualitative research studies. Yet, as several methodologists contend (e.g., Onwuegbuzie & Teddlie, 2003; Patton, 1990; Tashakkori & Teddlie, 1998), qualitative data can be used to test hypotheses. Thus, hypotheses also should be discussed within the qualitative section of textbooks.
Selecting a Research Design/Method
Many researchers believe that there is a direct relationship between research paradigm and research technique (Bryman, 1984; Onwuegbuzie & Teddlie, 2003). Yet, the research paradigm used does not dictate what procedures should be used (Onwuegbuzie & Leech, in press-b). This is a myth that is perpetuated by textbooks in multiple ways. First, most authors do not make a distinction between research paradigm and technique; research techniques are described as being either quantitative or qualitative. In fact, research designs are presented in educational research textbooks as if only specific types of data can be collected and analyzed for each research design. For example, all textbooks reviewed give the strong impression that only numeric data are collected in experimental studies and that only non-numeric data are collected in qualitative research studies. In so doing, these textbook writers are confusing the logic of justification with the research methods. Yet, there is no reason why qualitative data (e.g., interview responses) cannot be collected as part of the experimental design. Also, experimental and control groups can be compared with respect to qualitative information. Similarly, quantitative data (e.g., test scores) can be collected in qualitative studies (Onwuegbuzie & Leech, in press-a; Tashakkori & Teddlie, 2003a).
Another misleading aspect of the research design section of virtually all textbooks stems from the use of the term “causal-comparative” to denote quasi-experimental studies in which two or more groups are compared with respect to one or more quantitative outcomes. The writers of this article have found from personal experience that the phrase “causal-comparative research design” leads some students to believe that this design can result in the determination of causal relationships. As such, we recommend that this phrase be removed completely from textbooks.
Exacerbating this confusion is the misleading definitions of causal-comparative research presented in textbooks. For example, Gay and Airasian (2003) stated that “causal-comparative research produces limited cause-effect information” and that “a causal-comparative study…approximates cause-effect results without harming the participants” (p. 12). Both of these statements are extremely confusing because it is not clear what it means to obtain “limited cause-effect information” and to approximate “cause-effect results.” These statements appear to suggest that causal-comparative research nearly facilitates the identification of cause-and-effect relationships. Gay and Airasian also state, “causal-comparative research…attempts to determine reasons, or causes, for the existing condition” and that “the researcher attempts to determine the cause, or reason, for preexisting differences in groups of individuals” (p. 337). Similarly, Best and Kahn (2003) incorrectly define causal-comparative studies as “suggesting causation for their findings” (p. 132). Gall et al. (2003) incorrectly state “causal-comparative research design[s] … allow us to study cause-and-effect relationships” (p. 298). Another confusing statement appears in Charles and Mertler (2002): “Causal-comparative research is done to explore possible cause and effect, though it cannot demonstrate cause and effect as does experimental research” (p. 341). Moreover, such statements give the false impression that causal-comparative research designs enjoy a higher status than do correlational designs. This is not the case because causal-comparative designs are essentially correlational, with the only substantive difference between two types of designs from an analytical standpoint being that in causal-comparative designs the independent variable (e.g., treatment group, gender) always represents the nominal scale of measurement (Johnson, 2001). Indeed, it could be argued that attributes that are categorized (e.g., gender) could be included either in a causal-comparative study (as a major independent variable) or in a correlational study as one independent variable in a set. In this respect, we applaud Ary et al. (2002) for making the following statement: “despite its name, causal-comparative research does not establish a simple causal relationship among the variables of a study” (p. 25). Johnson and Christensen (2004) also should be acknowledged for including the following discussion:
Despite the presence of the term causal included in the term causal-comparative research , keep in mind that causal-comparative research is a nonexperimental research method, which means that there is no manipulation of an independent variable by a researcher. Furthermore, techniques of controlling for extraneous variables are more limited than in experimental research (in which random assignment may be possible). Because of the lack of manipulation and weaker techniques of controlling for extraneous variables, it is difficult to make statements about cause and effect. In short, do not be misled by the word causal in the name of this type of research. (p. 41) [emphasis in original]
There are many myths perpetuated in textbooks in relation to collecting data. Nearly all texts indicate that the research approach determines the type of data collected. Specifically, these authors incorrectly state that quantitative research exclusively involves the collection of numerical data, whereas qualitative research exclusively yields non-numerical data (e.g., words, observations). For example, Gall et al. (2003, p. 25) state that quantitative researchers “generate numerical data to represent the social environment” and that qualitative researchers “generate verbal and pictorial data to represent the social environment.” These statements are misleading on at least two levels. First, while researchers sometimes generate data, this is not always the case. Indeed, in many instances, the data of interest already have been generated, and the researcher's goal is to extract them, organize them, analyze them, and make sense of them. In quantitative research, it is very common for archival data to be used that were not generated by the researcher. With respect to qualitative data, the idea of the researcher generating the data runs completely contrary to the interpretivist belief that phenomena should be studied as they occur naturally (Lincoln & Guba, 1985). Second, although a quantitative researcher might be primarily interested in collecting, analyzing, and interpreting numeric data, non-numeric data also might provide valuable insights. Interestingly, in experimental designs, considered the apex of quantitative research, side effect data (i.e., qualitative data types) are routinely collected, analyzed, and interpreted. Similarly, qualitative research can involve both numeric and non-numeric data. For example, in interviews and focus groups, both data types can be collected. Related to this issue, many research texts give the impression that observation is a research method instead of a data collection technique (Tashakkori & Teddlie, 2003a).
Two other main issues concerning data collection are instrumentation and sampling. In sections involving these two components of the research process, misperceptions abound in textbooks. The major misperceptions are now described.
Sampling. In most texts, sampling is described for quantitative designs as being probability or non-probability (Ary et al., 2002; Gay & Airasian, 2003). This is misleading for readers because there is no reason why both these types of sampling could not be used in qualitative studies. Some authors (e.g., Charles & Mertler, 2002; Gravetter & Forzano, 2003) do not even include a discussion about sampling as it relates to qualitative studies. Yet, sampling is as much of an issue in qualitative research as in quantitative research. Specifically, as noted by Onwuegbuzie and Leech (in press-c, in press-d), in qualitative studies, one or more of the following generalizations nearly always are made: from the sample of words to the voice; from the sample of observations to the truth space; from the words of key informants to the voice of the other sample members; from the words of sample members to those of one or more individuals not selected for the study; and from the observations of sample members to the experience of one or more individuals not selected for the study. Consistent with this assertion that sampling permeates qualitative research, Lincoln and Guba (1985) advocate that the techniques of prolonged engagement and persistent observations be used when collecting data in order to increase the researcher's chances of understanding the underlying phenomena. Both of these techniques clearly represent sampling issues. Further, some qualitative researchers find it difficult to refrain from generalizing their results (e.g., thematic representations) beyond their samples to the underlying population (Onwuegbuzie & Leech, 2005a). As such, not only should purposeful sampling techniques be discussed in relation to qualitative research in textbooks, but authors also should make it clear that random sampling should be seriously considered in qualitative studies that lead to findings that are generalized beyond the study participants.
Describing sampling designs as either representing probability sampling or non-probability sampling is misleading for another reason. Specifically, it might give the impression that sampling takes place at only one stage; yet, some studies involve multi-sampling designs (e.g., two-stage sampling), in which sampling is undertaken at more than one stage. In fact, while some textbooks discuss multi-stage sampling designs very briefly (e.g., Creswell, 2005; Gall et al., 2003; Gay & Airasian, 2003), the majority of textbooks (e.g., Ary et al., 2002; Best & Kahn, 2003; Charles & Mertler, 2002; McMillan & Schumacher, 2001; Punch, 1999) do not discuss it at all. Disturbingly, the relatively few authors who discuss multi-stage sampling invariably give the impression that multi-stage sampling belongs to the domain of random sampling. Yet, as noted by Onwuegbuzie and Leech (in press-d), multi-stage sampling can involve random sampling at every stage, non-random sampling at every stage, or a mixture of random sampling and non-random sampling designs at different stages of the random sampling process (i.e., mixed methods sampling strategies). Indeed, it is usual for mixed method studies to utilize mixed methods sampling strategies (Kemper, Stringfield, & Teddlie, 2003; Onwuegbuzie & Collins, 2004).
With respect to qualitative research studies, Patton (1990) identified the following 15 purposeful sampling strategies: extreme/deviant case sampling, intensity sampling, typical case sampling, maximum variation sampling, stratified purposeful sampling, homogeneous sampling, critical case sampling, snowball/chain sampling, criterion sampling, theory-based/operational construct sampling, confirming/disconfirming case sampling, purposeful random sampling, political important sampling, convenience sampling, and opportunistic sampling. Miles and Huberman (1994) identified 16 purposeful sampling designs, adding a sixteenth sampling design, combination/mixed sampling to Patton's 15 sampling schemes. These lists of purposeful sampling techniques provide a useful framework for qualitative researchers. Yet, some textbooks (e.g., Best & Kahn, 2003; Gravetter & Forzano, 2003) do not even discuss qualitative sampling designs. Of those who do provide a summary of purposeful designs, none of the introductory research textbooks examined discussed all 16 purposeful sampling designs. Yet, at the very least, all of these designs can be summarized in only 16 paragraphs, which would take up less than six pages. For example, to their credit, Gall et al. (2003) presented Patton's (1990) 15 sampling schemes in four pages. It should be noted that qualitative research studies also can involve multi-stage sampling schemes, as can happen when the researcher returns to the site multiple times to select additional data. Thus, multi-stage sampling also should be discussed in the context of qualitative inquiry.
Choice of sample size also is presented in a misleading way in virtually every research methodology textbook. Some textbook authors do not even discuss sample sizes (e.g., Gravetter & Forzano, 2003; Punch, 1999). When providing sample size guidelines in quantitative research, most authors provide arbitrary minimum sample sizes. Typically, for correlational research designs, minimum sample sizes of 30 are recommended (e.g., Charles & Mertler, 2002; Creswell, 2005; Gall et al., 2003; Gay & Airasian, 2003; McMillan & Schumacher, 2001). However, this recommendation is extremely unwise because it represents a statistical power of only .51 for one-tailed tests for detecting a moderate relationship (i.e., r = .30) between two variables at the 5% level of statistical significance (Erdfelder, Faul, & Buchner, 1996; Onwuegbuzie & Leech, 2004). This level of power does not represent any significant improvement over flipping an unbiased coin, suggesting that, if this guideline was followed, approximately one-half of all statistical tests in correlational designs involving one-tailed tests of bivariate relationships would be statistically non-significant. For two-tailed tests of bivariate tests, the guideline of 30 participants has even more dire consequences, yielding a statistical power of .38. Onwuegbuzie, Jiao, and Bostick (2004) suggest a minimum sample size of 64 for one-tailed tests and 82 for two-tailed tests to attain .80 power for a moderate correlation.
For causal-comparative designs, a few textbooks (e.g., Gay & Airasian, 2003) suggest minimum group sizes of 30. Unfortunately, this represents a power of only .61 for one-tailed tests (i.e., independent samples t -tests) for detecting a moderate difference (i.e., d = .50) between two groups at the 5% level of statistical significance (Erdfelder et al., 1996) and .48 for two-tailed t -tests. Even more disturbingly, many textbook authors (e.g., Charles & Mertler, 2002; Creswell, 2005; Gall et al., 2003; McMillan & Schumacher, 2001) recommend minimum sample sizes of 15 for both causal-comparative and experimental designs. With respect to the former designs, this represents a power of only .38 for one-tailed tests and .26 for two-tailed tests. (Onwuegbuzie et al., 2004 calculated group sizes of 51 for one-tailed tests and 64 for two-tailed tests.) With such sample size recommendations in textbooks, it is no wonder that the “the average [hypothesized] power of null hypothesis significance tests in typical studies and research literature is in the .40 to .60 range (Cohen, 1962, 1965, 1988, 1992; Schmidt, 1996; Schmidt, Hunter, & Urry, 1976; Sedlmeier & Gigerenzer, 1989)…[with] .50 as a rough average” (p. 40). The current trend of only a few authors (e.g., Ary et al., 2002) discussing the role of power in quantitative research clearly needs to be reversed.
With respect to qualitative studies, rarely is there any discussion of sample size in introductory research methodology textbooks (Onwuegbuzie & Leech, in press-a). Yet, such guidelines have been provided in qualitative research textbooks (e.g., Creswell, 1998). For example, Creswell (1998) recommends that 20-30 persons should be interviewed in grounded theory designs in order to “achieve detail in the theory” (p. 113). For phenomenological designs, Creswell (1998) recommend interviews with up to 10 participants. Onwuegbuzie and Leech (in press-d) provide several rationales for systematically selecting sample sizes in qualitative research studies.
Instrumentation. Instrumentation is the other main issue that arises in textbooks when discussing data collection. Across the board, with a very few exceptions (e.g., Creswell, 2005), textbooks we reviewed only included discussion of instrumentation in the quantitative research chapters. Furthermore, most of the texts described tests as being reliable and valid. That is, they assign properties of reliability to quantitative measuring instruments, even though reliability is a function of scores and not of instruments (Onwuegbuzie & Daniel, 2002a, 2002b, 2003, 2004; Thompson & Vacha-Haase, 2000; Wilkinson & the Task Force on Statistical Inference, 1999). For example, Ary et al. (2002) refer to "reliability of a measuring instruments" (p, 249); Gay and Airasian (2003) include sections they call “Validity of Measuring Instruments” (p. 135) and “Reliability of Measuring Instruments” (p. 141); Gall et al. (2003) use the phrase “Test Validity” (p. 191) and “Test Reliability” (p. 195) ; and McMillan and Schumacher (2001) describe “Components of Test Validity” (p. 243). Such statements can give the impression that because an instrument's developer reports a large reliability coefficient for scores yielded by the normative sample, it can then be assumed that the instrument will continue to yield large reliability estimates in all subsequent studies, regardless of the sample selected. This issue likely explains why the vast majority of quantitative researchers do not report reliability coefficients for data from their samples (Onwuegbuzie, 2002b; Onwuegbuzie & Daniel, 2002a, 2002b, 2003, 2004; Thompson & Snyder, 1998; Vacha-Haase, Ness, Nilsson, & Reetz, 1999; Willson, 1980), even though it has been recommended to report reliability coefficients by authoritative and influential sources (e.g., American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 1999; Wilkinson & the Task Force on Statistical Inference, 1999).
Indeed, only a few textbook authors (e.g., Johnson & Christensen, 2004) use phrases such as “Validity of Test Scores” (pp. 132-133) and “Reliability of Test Scores” (pp. 132-133). However, even these statements can be misleading because they suggest that score reliability is only an issue for tests, and is thus not an issue for other types of quantitative measures (e.g., affective scales). Therefore, we recommend that words such as “instruments” replace the word “test” when the latter is being used in the generic sense.
Further, in none of the textbooks examined was reliability discussed in relation to qualitative studies, even though reliability is important to consider with qualitative data (Daniel & Onwuegbuzie, 2002; Madill , Jordan , & Shirley, 2000). At best, some components of reliability that are pertinent to qualitative research, such as inter-rater reliability and intra-reliability, are discussed only in the quantitative sections of textbooks (e.g., Creswell, 2005; Gay & Airasian, 2003). Similarly, few textbooks (e.g., McMillan & Schumacher, 2001) mention validity issues in qualitative research, even though as many as 50 components of validity in qualitative research have been identified (Onwuegbuzie & Leech, in press-e).
Another important misconception promoted by research textbooks is that quantitative data methods are objective. Yet, as contended by Onwuegbuzie (2002a), in quantitative studies, many research decisions are made throughout the research process that precedes objective verification decisions made. For instance, in developing instruments that yield empirical data, subjective decisions are made as to what items and how many items adequately represent the content domain. These subjective decisions prevent the measures from being objective. The only objective aspect of quantitative measures is the scoring; however, the ensuing scores are not objective. Moreover, when constructing quantitative instruments, items are selected that are believed to represent the underlying the behaviors. Yet, these behaviors stem from observations, which, in turn, stem from qualitative data collection techniques (i.e., observations, interviews). Thus, the role of qualitative research in the development of quantitative measures should be acknowledged. Unfortunately, this is not the case in research methodology textbooks.
The lack of discussion of instrumentation in qualitative research by the majority of textbook authors can give the impression that qualitative researchers do not use instruments to collect data. Yet, many instruments are used by interpretivists to collect data, including the following: observational schedules, interviews schedules, documents (newspapers, personal journals and diaries, official memoranda, letters, jottings individuals write to themselves, e-mail messages, minutes of meetings, records in the public domain, archival material stored in libraries), and audio-visual materials (e.g., photographs, digital images, videotapes, pictures, paintings, physical traces of images) (for a discussion of each of these instruments, see for example Onwuegbuzie et al., 2004).
Another misconception promoted by textbooks is that every instrument produces only one data type, with quantitative instruments, for example, only generating numerical data. However, this line of thinking is contradicted by the proliferation of instruments that contain closed-ended and open-ended items, which generate quantitative and qualitative data. Such instruments that can incorporate a mixture of closed-ended and open-ended items include questionnaires, interview schedules, standardized tests, and documents (cf. Johnson & Turner, 2003). Another source of misunderstanding perpetuated by some textbook authors stems from their treatment of observations as a research design or method rather than as a data collection technique that can be used in both quantitative and qualitative research (Onwuegbuzie & Leech, in press-a; Tashakkori & Teddlie, 2003a). For example, Gay and Airasian (2003, p. 293) define “observational research” as a form of descriptive research in which “the current status of a phenomenon is determined not by asking, as with a survey or through an interview, but by observing.” Yet, observations not only can be collected in descriptive research designs but also in correlational, causal-comparative, and experimental designs, as well as in all qualitative research designs (e.g., case studies, phenomenological research, grounded theory, ethnographic research). Similarly, some authors refer to content analysis as a research method or research design instead of quantitative and qualitative data analysis techniques (Tashakkori & Teddlie, 2003a).
Analyzing the Data
All introductory research methodology textbooks give the impression that there is a one-to-one correspondence between research approach and data analysis technique. Specifically, statistical analyses are associated exclusively with quantitative research, whereas qualitative analyses (e.g., thematic analyses) are associated exclusively with qualitative research. For example, the chapter on descriptive statistics usually follows the chapters that discuss quantitative research. This maintains the myth that statistics should only be used with quantitative designs and not in qualitative designs. Similarly, when a discussion of qualitative analysis is included in the book, it invariable is placed near the qualitative chapters; thus, giving the impression that qualitative analysis should be used only with designs that fall under the qualitative paradigm. Although quantitative data are more likely to necessitate statistical analyses and qualitative data are more apt to justify qualitative data analyses, yet, both statistical analyses and qualitative data analyses can be used to explore and to confirm phenomena (Onwuegbuzie & Leech, in press-a). Moreover, Onwuegbuzie and Teddlie (2003) have demonstrated how quantitative and qualitative data analyses can be integrated within the same analytic framework (i.e., mixed methods data analyses) either sequentially or concurrently. These methodologists discussed how quantitative and qualitative data could be transformed to the other data type. In particular, Tashakkori and Teddlie (1998) used the term “quantitizing” to refer to the conversion of qualitative data (e.g., focus group responses) to quantitative data (e.g., frequencies) and the term “qualitizing” to denote the conversion of quantitative data (e.g., attitude scores) to qualitative data (e.g., profiles). Onwuegbuzie and Teddlie also described how quantitative and qualitative data could be correlated, consolidated, compared, and integrated.
When discussing inferential statistical techniques, textbook authors tend to discuss the role of statistical significance. However, many writers (e.g., Charles & Mertler, 2002) fail to mention the other three types of significance that have been identified, namely, practical significance, clinical significance, and, most recently, economic significance (Leech & Onwuegbuzie, in press). Practical significance indices provide information about the size of observed difference or relationship (e.g., effect size). Clinical significance measures provide data regarding the extent to which the intervention makes a real difference to the quality of life of the participants or to those with whom they interact. Economic significance indices represent the economic value of the effect of an intervention. These errors of omission occur despite the problems associated with null hypothesis significance testing, which include (a) its over-reliance on sample size, (b) the fact that it does not indicate whether a result is true for a population, and (c) the fact that it does not indicate the strength or size of an effect (i.e., the belief that a p < .05 is less important or significant than a p < .001) (Fan, 2001; Kirk, 1996; Thompson, 1993). Although an increasing number of textbooks discuss practical significance (e.g., Ary et al., 2002; Creswell, 2005; Gay & Airasian, Johnson & Christensen, 2004; McMillan & Schumacher, 2001), few textbooks, if any, discuss clinical or economic significance. Yet, these concepts are not too advanced for beginning researchers.
In addition, when discussing inferential statistics, textbook writers do not discuss them within the framework of the General Linear Model. This might give the impression that each statistical analysis represents an independent technique rather than a procedure that subsumes or is subsumed by other inferential procedures. Errors of omission also occur in qualitative data analysis sections of textbooks. In particular, authors do not discuss how obtaining counts of observations can provide more meaning to narrative descriptions (Onwuegbuzie, 2003b; Onwuegbuzie & Teddlie, 2003). Onwuegbuzie (2003b) refers to these counts as effect sizes.
Further, as noted by Onwuegbuzie and Leech (2005b), virtually all research methodology textbook authors (e.g., Ary, Jacobs, & Razavieh, 2002; Creswell, 2005; Fraenkel & Wallen, 2003; Gall, Borg, & Gall, 2003; Gay & Airasian, 2003; Johnson & Christensen, 2004; McMillan & Schumacher, 2001; Wallen & Fraenkel, 2001) present their discussions of statistical analyses in separate chapters from their discussions of research design and other stages of the research process. As such, in these statistics analysis chapters, there tends to be little or no reference to research questions, research design, or the like, giving the impression that statistical analyses occur in a vacuum.
The quality of result interpretations (i.e., inference quality) is a function of how valid the underlying findings are. Thus, validity represents the most important stage of the research process for both quantitative and qualitative research studies. Thus, it is not surprising that all research textbooks examined contain a discussion of validity in relation to quantitative research, including information about Campbell and Stanley 's (Campbell, 1957; Campbell & Stanley, 1963) threats to internal and external validity. However, as noted by Onwuegbuzie (2003), these threats are only discussed in relation to experimental and quasi-experimental designs, giving the impression that validity is not an issue for other types of quantitative designs (e.g., descriptive, correlational). Yet, validity is as relevant for these designs (Huck & Sandler, 1979; McMillan, 2000; Onwuegbuzie, 2003). Moreover, the seminal framework provided by Campbell and Stanley for experimental research is too restricting for non-experimental quantitative designs. Consequently, Huck and Sandler (1979), McMillan (2000), and Onwuegbuzie (2003) have expanded Campbell and Stanley's framework, with Onwuegbuzie's (2003) presentation being the most comprehensive to date.
Disturbingly, only a few textbook writers (e.g., Ary et al., 2002; Gall et al., 2003; Gay & Airasian, 2003; McMillan & Schumacher, 2001) discuss validity as it pertains to qualitative research, even though much literature exists in this area. With 50 components of validity or legitimation identified in the literature (Onwuegbuzie & Leech, in press-e), at least some of these deserve mention. For example, Maxwell's (1992) conceptualization of validity could be summarized in introductory research methodology textbooks. Specifically, Maxwell identified five types of validity in qualitative research: descriptive validity (i.e., factual accuracy of the account as documented by the researcher), interpretive validity (i.e., the extent to which a researcher's interpretation of an account represents an understanding of the perspective of the group under study and the meanings attached to their words and actions), theoretical validity (i.e., the degree to which a theoretical explanation developed from research findings fits the data), evaluative validity (i.e., the extent to which an evaluation framework can be applied to the objects of study, rather than a descriptive, interpretive, or explanatory one), and generalizability (i.e., the extent to which a researcher can generalize the account of a particular situation or population to other individuals, times, settings, or context). With respect to the latter, Maxwell differentiated internal generalizability from external generalizability; with internal generalizability pertaining to the generalizability of an inference within the setting or group studied, whereas, the latter refers to generalizability beyond the group, setting, context, or time. According to Maxwell, internal generalizability is typically more important to qualitative researchers than is external generalizability.
Another misconception advanced in some textbooks (e.g., Best & Kahn, 2003) is that aspects of the research design or data collection procedure can be assessed for validity. For example, Best and Kahn (2003) refer to “Validity and Reliability of the Interview” (p. 324). Yet, it is not the interview that should be assessed for validity and/or reliability. Rather, it is data that emerge from the interview(s) or the resultant inferences made that should be examined for legitimation.
Communicating the Findings
The final step of the research process is to communicate the research findings. With respect to quantitative reports, perhaps the biggest misconception promoted by textbook is that interpretations of results are value free. This can lead to the impression that it is not essential for the context to be provided when reporting a relationship. That is, explanations of phenomena should be undertaken at a high level of abstraction such that the role of culture is ignored, thereby allowing “universal generalizations that encompass the broadest domain of the phenomenon in question” (Moghaddam, Walker, & HarrJ , 2003, p. 123). Indeed, as stated by Tashakkori and Teddlie (2003a): “We are struck by how much some of the ‘quantitative' papers are void of any reference to the cultural context of the behaviors/phenomenon under study” (p. 19). Yet, ignoring the role of culture likely leads to non-interaction-seeking bias, in which the nature of reality of interest is not honored in the optimal sense (Onwuegbuzie, 2003). Thus, all quantitative findings should be contextualized.
With respect to writing qualitative reports, a common weakness of this section in textbooks is a failure to emphasize that qualitative findings should not be generalized beyond the sample unless the sample size is adequately large (Onwuegbuzie & Leech, 2005a). Not making this recommendation explicit likely explains why some qualitative researchers find it difficult to resist the temptation to generalize their results to the underlying population. Instead, they should focus on attempting to obtain insights into particular educational, social, and familial processes and practices that existed within a specific location and context (Connolly, 1998).
Implications of Errors
The implications of the textbook errors outlined above are immeasurable. On the simplest level, the fact that research method textbooks continue to relate myths and errors about the process of research to students keeps the myths alive; many researchers do not understand that what they learned from the textbook is incorrect. Thus, researchers continue to perpetuate errors within their own research projects, presentations of their research, and published articles.
More specifically, many of these errors advance the myth that research method as a technique (i.e., research design) and research method as a logic of justification (i.e., research paradigm) are synonymous. Moreover, many of these myths tend to promote the Incompatibility Thesis, which suggests that quantitative and qualitative paradigms cannot coexist, and cannot and should not be mixed in any way (Howe, 1988). Indeed, by separating quantitative and qualitative approaches, research methodology textbooks read as if they contain two books in one. Additionally, as noted by Onwuegbuzie and Leech (in press-a), separating discussion of quantitative and qualitative approaches in research textbooks, “students may form the impression that research represents a dichotomy of choices rather than an integrative, interactive, and systematic process for the purpose of generating new knowledge or validating or refuting existing knowledge” (p. 15). Such a narrow view of research might dissuade them from learning to conduct “bilingual research” (Tashakkori & Teddlie, 2003a, p. 64), believing that they have to choose one paradigm over the other. Yet, in recent years, the Incompatibility Thesis has been usurped by the Compatibility Thesis, whereby the relationship between the quantitative and qualitative paradigms consisted of isolated events lying on a continuum of scientific research (Howe, 1988; Tashakkori & Teddlie, 1998). Unfortunately, the majority of research textbooks do not presently reflect this trend.
Using Onwuegbuzie and Leech's (in press-a) framework
The framework suggested by Onwuegbuzie and Leech (in press-a) demonstrates how research textbooks can be reframed to break down the barriers between quantitative and qualitative research. In essence, this framework calls for the integration of qualitative and quantitative approaches at each stage of the research process. At the stage at which the research problem and research objective are formulated, rather than linking deductive reasoning with quantitative research and inductive reasoning with qualitative research, textbook authors would describe research objectives as lying on an exploratory-confirmatory continuum. The extent to which the research objective is exploratory or confirmatory would then determine the reasoning process. Once the research objective has been established, the purpose of study, research question(s), and hypotheses (if the research objective is confirmatory) are determined. However, rather than presenting these elements as falling under the quantitative/qualitative divide, they would be discussed in the context of the exploratory-confirmatory research objective continuum. The next major section of the book would describe a typology of quantitative, qualitative, and mixed methods research designs. However, it will be emphasized that the design does not dictate the data type collected. The section on data collection would provide information about sampling and choice of instruments. In the discussion on sampling, instead of presenting the sampling schemes as belonging to either the quantitative research and qualitative fields, all 24 sampling methods outlined by Onwuegbuzie and Leech (in press-a) would be described and classified according to whether they represent probability or non-probability sampling. In addition, to these 24 sampling techniques, the following additional 4 mixed methods sampling strategies would be included: (a) multi-stage mixed methods random sampling (in which all phases of the sampling are random), (b) multi-stage mixed methods non-random sampling (in which all phases of the sampling are non-random); (c) multi-stage mixed methods random-purposive sampling (in which the first phase of the sampling is random and at least one of the subsequent phases is purposive); and (d) multi-stage mixed methods purposive-random sampling (in which the first phase of the sampling is purposive and at least one of the subsequent phases is random). These mixed methods sampling schemes would provide researchers with at least 28 sampling methods at their disposal, regardless of paradigmatic orientation.
With respect to the instrument section of the textbook, Burke and Turner's (2003) mixed methods data collection framework would be used. Here, it would be made clear that there is not a one-to-one correspondence between paradigm and data collection method.
The section on data analysis could incorporate the framework of Onwuegbuzie and Teddlie (2003). Here, it would be emphasized that the analyses selected would be a function of the research objective and purpose. In particular, the data analysis tools discussed would be classified under exploratory and confirmatory techniques. In the section on interpreting and validating data, information about threats to internal validity, external validity, and legitimation would be provided under the headings of inference quality and data quality, as recommended by Tashakkori and Teddlie (2003b). Finally, discussion about quantitative, qualitative, and mixed methods research reports would take place within the same chapter.
Recommendations for Authors, Instructors, and Students
Based on the analysis of textbooks, the following are recommendations for authors of textbooks, and instructors and students of research methodology courses. For authors of textbooks, it is important to update and revise the texts regularly to keep up with recent trends and changes. Because research and analysis techniques are constantly transforming and emerging, keeping up with these changes is important. One avenue to assist with this is for textbook authors to attend national and regional conferences, and to read recent literature in the area of research and analysis techniques. Furthermore, it would be beneficial to have textbooks rigorously reviewed by leading methodologists in the field.
Instructors of research method courses using these texts should keep up with recent trends and changes in research and analysis techniques. Using a critical eye in reading and choosing texts would increase the chances of using texts that contain fewer errors. Finally, instructors should not make the textbook the sole method of information; instructors should supplement the assigned readings with recent articles which outline controversies and the latest developments. For readers of these textbooks, it is imperative to read critically and point out discrepancies.
Evidence has been provided that a significant proportion of published studies, theses, and dissertations are seriously flawed, containing analytical and interpretational errors (Daniel, 1998; Keselman et al., 1998; Onwuegbuzie, 2002b; Thompson, 1998; Vockell & Asher, 1974; Ward, Hall, & Schramm, 1975; Witta & Daniel, 1998). Onwuegbuzie and Daniel (2003) presented the following five reasons for this trend: (a) graduate-level instruction in which research methodology is taught as a series of linear steps, rather than as a holistic, interactive, and reflective process; (b) graduate-level curricula that severely limit students' exposure to quantitative and qualitative research approaches; (c) the endorsement of various misperceptions about the nature of research; (d) research methodology instructors teaching out of their specialty areas; and (e) failure, reluctance, or even refusal to recognize that methodological techniques that were popular in previous decades no longer represent best practices and may now be considered inappropriate, invalid, untimely, or obsolete.
However, the present essay has demonstrated that an additional reason likely prevails for the vast number of published studies, theses, and dissertations that are seriously flawed. Specifically, it is likely that the errors and myths perpetuated by textbooks also play an important role in the proliferation of research inquiries that involve questionable practices and, consequently, invalid findings and inferences. Thus, it is imperative that research methodology textbook authors exercise much more caution in what they write.
Anthony J. Onwuegbuzie
Department of Educational Measurement and Research, College of Education , University of South Florida , 4202 East Fowler Avenue , EDU 162, Tampa , FL , 33620-7750 , or E-Mail: ( email@example.com )
Nancy L. Leech
University of Colorado at Denver and Health Sciences Center
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing (rev. ed.). Washington : American Educational Research Association.
Ary, D., Jacobs, L. C., & Razavieh, A. (2002). Introduction to research in education (6th ed.). Belmont, CA : Wadsworth/Thomson Learning.
Best, J. W., & Kahn, J. V. (2003). Research in education (9th ed.). Needham Heights, MA: Allyn & Bacon.
Bryman, A. (1984). The debate about quantitative and qualitative research: A question of method or epistemology? British Journal of Sociology, 35, 78-92.
Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54, 297-312.
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research . Chicago: Rand McNally.
Charles, C. M., & Mertler, C. A. (2002). Introduction to educational research (4th ed.). Boston, MA: Allyn & Bacon.
Cohen, J. (1962). The statistical power of abnormal social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145-153.
Cohen, J. (1965). Some statistical issues in psychological research. In B. B. Wolman (Ed.), Handbook of clinical psychology (pp. 95-121). New York: McGraw-Hill.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, N.J. : Erlbaum.
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.
Connolly, P. (1998). ‘Dancing to the wrong tune': Ethnography generalization and research on racism in schools. In P. Connolly & B. Troyna (Eds.), Researching racism in education: Politics, theory, and practice . Buckingham, UK: Open University Press.
Cooper, H. M. (1989). Integrating research: A guide for literature reviews. Newbury Park , CA : Sage.
Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five traditions . Thousand Oaks, CA: Sage.
Creswell, J. W. (2005). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (2nd ed.). Upper Saddle River , NJ: Pearson Education.
Daniel, L. G. (1998, December). Use of statistical significance testing in current “general” educational journals: A review of articles with comments for improved practice . Paper presented at the annual meeting of the Association for the Advancement of Educational Research, Ponte Vedra, FL.
Daniel, L.G., & Onwuegbuzie, A. J. (2002, November). Reliability and qualitative data: Are psychometric concepts relevant within an interpretivist research paradigm ? Paper presented at the annual meeting of the Mid-South Educational Research Association, Chattanooga, TN.
Dellinger, A. B. (2003, February). Exploring the relationship between validity and the review of the literature . Paper presented at the annual meeting of the Southwestern Educational Research Association, San Antonio, TX .
Dunkin, M. J. (1996). Types of errors in synthesizing research in education. Review of Educational Research, 66(2), 87-97.
Erdfelder, E., Faul, F., & Buchner, A. (1996). GPOWER: A general power analysis program. Behavior Research Methods, Instruments, & Computers, 28, 1-11.
Fan, X. (2001). Statistical significance and effect size in education research: Two sides of a coin. Journal of Educational Research, 94, 275-282.
Fisch, M. H. (Ed.) (1982). Writings of Charles S. Peirce: A chronological edition: Vol. 1. 1857-1866. Bloomington: Indiana University Press.
Fraenkel, J. R., & Wallen, N. E. (2003). How to design and evaluate research in education (5th ed.). Boston, MA: McGraw-Hill.
Gall, M. D., Borg, W. R., & Gall, J. P. (2003). Educational research: An introduction (7th ed.). White Plains, NY: Longman.
Gay, L. R., & Airasian, P. (2003). Educational research: Competencies for analysis and application (7th ed.). Upper Saddle River, NJ: Pearson Education.
Gliner, J. A., & Morgan, G. A. (2000). Research methods in applied settings: An integrated approach to design and analysis . Mahwah, NJ: Lawrence Erlbaum Associates.
Gravetter, F. J., & Forzano, L. B. (2003). Research methods for the behavioral sciences . Belmont , CA : Wadsworth/Thomson Learning.
Howe, K. R. (1988). Against the quantitative-qualitative incompatibility thesis or dogmas die hard. Educational Researcher, 17 , 10-16.
Huck, S. W., & Sandler, H. M. (1979). Rival hypotheses: Alternative interpretations of data based conclusions . New York : Harper Collins.
Johnson, R. B. (2001). Toward a new classification of nonexperimental quantitative research. Educational Researcher, 30 (2), 3-13.
Johnson, R. B., & Christensen, L. B. (2004). Educational research: Quantitative, qualitative, and mixed ap proaches (3rd ed.). Boston , MA: Allyn and Bacon.
Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26.
Johnson, B., & Turner, L. A. (2003). Data collection strategies in mixed methods
research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 297-319). Thousand Oaks, CA: Sage.
Kemper, E. A., Stringfield, S., & Teddlie, C. (2003). Mixed methods sampling strategies in social science research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 273-296). Thousand Oaks, CA: Sage.
Keselman, H. J., Huberty, C. J., Lix, L. M., Olejnik, S., Cribbie, R. A., Donahue, B., Kowalchuk, R. K., Lowman, L. L., Petoskey, M. D., Keselman, J. C., & Levin, J. R. (1998). Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research, 68, 350-386.
Kirk, R. (1996). Practical significance: A concept whose time has come. Education and Psychological Measurement, 56, 746-759.
Langenbach, M., Vaughn, C., & Aagard, L. (1994). An introduction to educational research . Boston , MA : Allyn & Bacon.
Leary, M. R. (2004). Introduction to behavioral research methods (4th ed.). Pacific Grove, CA: Brooks/Cole.
Leech, N. L., & Onwuegbuzie, A. J. (in press). A proposed fourth measure of significance: The role of economic significance in educational research. Evaluation and Research in Educatio.
Liebert, R. M., & Liebert, L. L. (1995). Science and behavior: An introduction to methods of psychologic al research. Englewood Cliffs, NJ: Prentice-Hall.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.
Madill, A., Jordan, A., & Shirley, C. (2000). Objectivity and reliability in qualitative analysis: Realist, contextualist and radical constructionist epistemologies. The British Journal of Psychology, 91, 1-20.
Maxcy, S. J. (2003). Pragmatic trends in mixed methods research in the social sciences: The search for multiple modes of inquiry and the end of the philosophy of formalism. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 51-89). Thousand Oaks, CA: Sage.
Maxwell, J. A. (1992). Understanding and validity in qualitative research. Harvard Educational Review, 62, 279-299.
McMillan, J. H. (2000, April). Examining categories of rival hypotheses for educational research . Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
McMillan, J. H., & Schumacher, S. (2001). Research in education: A conceptual introduction (5th Ed.). New York, NY: Longman.
Miles, M., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage.
Moghaddam, F. M., Walker, B. R., & Harr J , R. (2003). Cultural distance, levels of abstraction, and the advantages of mixed methods. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 111-134). Thousand Oaks, CA: Sage.
Mundfrom, D. J., Shaw, D. G., Thomas, A., Young, S., & Moore, A. D. (1998, April). Introductory graduate research courses: An examination of the knowledge base. Paper presented at the annual meeting of the American Educational Research Association, San Diego , CA .
Newman, I. , & Benz, C. R. (1998). Qualitative-quantitative research methodology: Exploring the interactive continuum. Carbondale, Illinois: Southern Illinois University Press.
Onwuegbuzie, A. J. (2002a). Positivists, post-positivists, post-structuralists, and post- modernists: Why can't we all get along? Towards a framework for unifying research paradigms. Education, 122 (3), 518-530.
Onwuegbuzie, A. J. (2002b). Common analytical and interpretational errors in educational research: An analysis of the 1998 volume of the British Journal of Educational Psychology. Educational Research Quarterly, 26, 11-22.
Onwuegbuzie, A. J. (2003a). Expanding the framework of internal and external validity in quantitative research. Research in the Schools, 10(1), 71-90.
Onwuegbuzie, A. J. (2003b). Effect sizes in qualitative research: A prolegomenon. Quality & Quantity: International Journal of Methodology, 3, 393-409.
Onwuegbuzie, A. J., & Collins, K. M. T. (2004, November). A framework for developing sampling designs in mixed methods research . Paper to be presented at the Australian Association for Research in Education, Melbourne, Australia .
Onwuegbuzie, A. J., & Daniel, L. G . (2002a). Uses and misuses of the correlation coefficient. Research in the Schools, 9, 73-90.
Onwuegbuzie, A. J., & Daniel, L. G. (2002b). A framework for reporting and interpreting internal consistency reliability estimates. Measurement and Evaluation in Counseling and Development, 35, 89-103.
Onwuegbuzie, A. J., & Daniel, L. G. (2003, February 12). Typology of analytical and interpretational errors in quantitative and qualitative educational research. Current Issues in Education [On-line], 6(2). Available: http://cie.ed.asu.edu/volume6/number2/
Onwuegbuzie, A. J., & Daniel, L. G. (2004). Reliability generalization: The importance of considering sample specificity, confidence intervals, and subgroup differences. Research in the Schools, 11 (1), 61-72.
Onwuegbuzie, A. J., Jiao, Q. G., & Bostick, S. L. (2004). Library anxiety: Theory, research, and applications . Lanham, MD: Scarecrow Press.
Onwuegbuzie, A. J., & Leech, N. L. (2003, February). Meta-analysis research: Cautions and limitation s. Paper presented at the annual meeting of the Eastern Educational Research Association, Hilton Head, SC.
Onwuegbuzie, A. J., & Leech, N. L. (2004). Post-hoc power: A concept whose time has come. Understanding Statistics, 3, 151-180.
Onwuegbuzie, A. J., & Leech, N. L. (2005a, March). Generalization practices in qualitative research: Trends in the literature. Paper presented at the annual meeting of the Eastern Educational Research Association, Sarasota, FL.
Onwuegbuzie, A. J., & Leech, N. L. (2005b, February). Linking research questions to mixed methods data analysis procedures. Paper presented at the annual meeting of the Southwest Educational Research Association, New Orleans, LA.
Onwuegbuzie, A. J., & Leech, N. L. (in press-a). Taking the “Q” out of research: Teaching research methodology courses without the divide between quantitative and qualitative paradigms. Quality & Quantity: International Journal of Methodology .
Onwuegbuzie, A. J., & Leech, N. L. (in press-b). On becoming a pragmatist researcher: The importance of combining quantitative and qualitative research methodologies. International Journal of Social Research Methodology: Theory & Practice .
Onwuegbuzie, A. J., & Leech, N. L. (in press-c). The role of sampling in qualitative research. Academic Exchange Quarterly.
Onwuegbuzie, A. J., & Leech, N. L . (in press-d). Qualitative power analyses. Quality & Quantity: International Journal of Methodology.
Onwuegbuzie, A. J., & Leech, N. L. (in press-e). Validity and qualitative research. An oxymoron? Quality & Quantity: International Journal of Methodology.
Onwuegbuzie, A. J., Slate, J. R., & Sc hwartz, R. A. (2001). Role of study skills in graduate-level educational research courses. Journal of Educational Research, 94, 238-246.
Onwuegbuzie, A. J., & Teddlie, C. (2003). A framework for analyzing data in mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 351-383) . Thousand Oaks, CA: Sage.
Patton, M. Q. (1990). Qualitative research and evaluation methods (2nd ed.). Newbury Park, CA: Sage.
Punch, K. F. (1999). Introduction to social research: Quantitative and qualitative approaches . Thousand Oaks, CA: Sage.
Ravid, R., & Leon, M. R. (1995, April). Students' perceptions of the research component in Master's level teacher education programs . Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA .
Rosnow, R. L., & Rosenthal, R. R. (2005). Beginning behavioral research: A conceptual primer (5th ed.). Englewood Cliffs, NJ: Prentice Hall.
Salkind, N. J. (2003). Exploring research (5th ed). Upper Saddle River, N.J: Prentice Hall.
Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers. Psychological Methods, 1, 115-129.
Schmidt, F. L., Hunter, J. E., & Urry, V. E. (1976). Statistical power in criterion-related validation studies. Journal of Applied Psychology, 61, 473-485.
Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316.
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches . Thousand, CA: Sage.
Tashakkori, A., & Teddlie, C. (2003a). Issues and dilemmas in teaching research methods courses in social and behavioral sciences: A US perspective. International Journal of Social Research Methodology, 6 (1), 61 - 77.
Tashakkori, A., & Teddlie, C. (2003b). Major issues and controversies in the use of mixed methods in the social and behavioral sciences . In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 3-50). Thousand Oaks, CA : Sage.
Thompson, B. (1993). The use of statistical significance research: Bootstrap and other alternatives. The Journal of Experimental Education, 61, 361-377.
Thompson, B., & Snyder, P. A. (1998). Statistical significance and reliability analyses in recent JCD research articles. Journal of Counseling and Development , 76, 436-441.
Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174-195.
Vacha-Haase, T., Ness , C., Nilsson, J., & Reetz, D. (1999). Practices regarding reporting of reliability coefficients. A review of three journals. The Journal of Experimental Education, 67, 335-341.
Vockell, E. L., & Asher, W. (1974). Perceptions of document quality and use by educational decision makers and researchers. American Educational Research Journal, 11, 249-258.
Wallen, N. E., & Fraenkel, J. R. (2001). Educational research: A guide to the process . Mahwah, NJ: Lawrence Erlbaum.
Ward, A. W., Hall, B. W., & Schramm, C. E. (1975). Evaluation of published educational research: A national survey. American Educational Research Journal, 12, 109-128.
Wilkinson, L., & the Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
Willson, V. L. (1980). Research techniques in AERJ articles: 1969 to 1978. Educational Researcher, 9(6), 5-10.
Witta, E. L., & Daniel, L. G. (1998, April). The reliability and validity of test scores: Are editorial policy changes reflected in journal articles? Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA .
Wolf, F. M. (1986). Meta-analysis: Quantitative methods for research synthesis. Newbury Park, CA: Sage.