Wolcott, H. F. (1994). Transforming qualitative data. Thousand Oaks, CA: Sage Publications (excerpt includes pp. 347-356, 371-373).

 

Seeking Validity, or on Not Getting It All Wrong

I have quoted the following words from Clifford Geertz's (1973) essay “Thick Description” so often that I am coming to think of them as my own:

Cultural analysis is intrinsically incomplete. And, worse than that, the more deeply it goes the less complete it is. It is strange science whose most telling assertions are its most tremulously based, in which to get somewhere with the matter at hand is to intensify the suspicion, both your own and that of others, that you are not quite getting it right. (p. 29)

I assume that by “cultural analysis” Geertz refers both to what one is attempting to describe and how one interprets it. Thus we can say of field studies in general that “to get somewhere with the matter at hand is to intensify the suspicion ... that you are not quite getting it right.”

But I also go to considerable pains not to get it all wrong. At least in its broad sense of scientific accuracy or correctness, validity haunts qualitative researchers as a specter, even if it may not be precisely the quality we seek. Others have offered arguments to show why validity can be claimed, either in their work or on behalf of qualitative researchers in general (e.g., Goetz & LeCompte, 1984, p. 221), and I take this opportunity to do the same.

In nine points that I follow, I have described what I do, try to do, or think I do to satisfy the implicit challenge of validity. I hasten to add that I probably do some of these things to keep the question from being raised at all, since it can be one of those accusations not lightly dismissed even if subsequently refuted. There is no particular order to the points as presented, although I more or less follow a progression from early stages of fieldwork, devoted primarily to getting information, through later stages given more fully to analysis and writing.

Talk Little, Listen a Lot

Like any fieldworker, I have some personal qualities that I believe serve me well in research. A seemingly unlikely one (about which others may disagree) is that I regard myself as neither particularly talkative nor particularly gregarious. I am basically a loner in thought and work; I do not like to be lonely, but I enjoy and need solitude when pursuing my academic tasks. It requires great patience under any circumstances for me to “sit and visit.” A rather inevitable consequence of being inquisitive without being a talker is that my conversational queries usually prompt others to do the talking. During fieldwork, I make a conscious effort to be sociable, thus providing opportunities for people to talk to me. My work ethic takes over to help me become not only more social but more attentive and responsive, and out pour the informants' stories and explanations so essential to good fieldwork.

(Parenthetically, I note my suspicion that many fieldworkers talk too much and hear too little. They become their own worst enemy by becoming their own best informant. This is especially serious in school research, where we often presume to “'know” what is supposed to be happening and consequently may never ask the kinds of questions we would ordinarily ask in any other research setting. I have suggested elsewhere (Wolcott, 1984) that educational researchers need to be wary of this “ethnography minus one” approach.)

Since no one ever can say everything about anything, in virtually any conversation, and especially during fieldwork, I find myself pondering what part of the whole story is being told and what part of that I am actually understanding. Like most of us, I think I sense when I am getting a straight story, when I am getting a story straight, and when I am on a detour of my own or another's making. If the latter, I usually try to swing by that way another time. I never confront informants with contradictions, blatant disbelief, or shock, but I do not mind presenting myself as a bit dense, someone who does not catch on too quickly and has to have things repeated or explained—what Kirk and Miller (1986) describe as “willing to look a fool for the sake of science” (p. 49).

Record Accurately

Whenever I engage in fieldwork I try to record as accurately as possible, and in precisely their words, what I judge to be important of what people do and say. I note that this occurs during formal fieldwork because, unlike some ethnographers, I am not a regular diary keeper. Rather, like most of them, I detest notetaking, in part because I tend to be so meticulous about it. I record field notes only when my work ethic demands. When I do take notes, I endeavor to make them as soon as possible after an event, if not at that moment. I prefer to make notes during observations or interviews—including written notes to supplement mechanically recorded ones. (It is not a bad idea to remind people of one's research presence and purposes.) By recording as soon as possible, to capture words and events as observed, I try to minimize the potential influence of some line of interpretation or analysis that might have me remembering and recording too selectively or reinterpreting behavior prior to recording it.

Begin Writing Early

Since I do not work quickly anyway, I make something of a fetish of taking my time at fieldwork. In seeming contrast, however—because I have come to regard writing as an integral part of fieldwork rather than as a separate stage initiated after fieldwork is completed—I often begin preparing a rough draft soon after fieldwork begins. Most recently I have suggested that qualitative researchers consider writing a preliminary draft of a descriptively oriented study before even venturing into the field (Wolcott, 1990). The intent is twofold: to make a record of what one already knows or suspects and to identify obvious gaps where more information will be needed.

In every study in which I have begun writing early, I have been able to share a draft (but not that first one, by any means) with others knowledgeable about the setting. Thus I obtain valuable feedback for myself and sometimes provide welcome feedback for others who may have expressed curiosity or concern about what I was observing and what kind of story I would tell.

To borrow a phrase from linguistics, my accounts move forward by “successive approximations.” It would be nice if that meant. they became successively more accurate; I suspect they only become successively better contextualized (i.e., more complex). But I try to “stick around” long enough, and keep in touch long afterward, so that events observed can be reviewed from the perspective of time-for observer and observed alike. Twenty-seven years after beginning fieldwork, I am still in con tact with Kwakiutl families I met in 1962 (Wolcott, 1989). Twenty years after initiating a study of the principalship in 1966, I sat down with “Ed Bell” to record his reflections on that study and his now completed career as an educational administrator. And in neither of those studies—nor in any of my others—does it seem that I have ever quite gotten it right.

In spite of extending one's fieldwork and one's subsequent reflections over time, I begin making detailed notes immediately upon initiating fieldwork, as I pointed out before. Right or wrong as first impressions may be, I feel they should be carefully recorded as a baseline from which the work proceeds. First impressions also serve as a useful resource in subsequent writing. Through them, researchers can introduce readers to settings the way they themselves first encountered them, rather than in the presumably more discerning way they have come to see them through extended time for observation and reflection.

Let Readers “See” for Themselves

I make a conscious effort to include primary data in my final accounts, not only to give readers an idea of what my data are like but to give access to the data themselves. In striking the delicate balance between providing too much detail and too little, I would rather err on the side of too much; conversely, between overanalyzing and underanalyzing data, I would rather say too little. Accordingly, my accounts are often lengthy; informants are given a forum for presenting their own case to whatever extent possible and reasonable. This poses a dilemma: In reading the descriptive accounts of others, I confess that I often skip over the quoted material in my haste to “get right at it” and see what the researcher made of it all; yet I knowingly risk boring my readers with potentially tedious detail.

More subtly, my growing bias toward letting informants speak for themselves is exactly that—a bias in favor of trying to capture the expressed thoughts of others rather than relying too singularly on what I have observed and interpreted. The extent to which participant observation and interviewing are a natural complement or get at quite different aspects of thought and action has always vexed experienced fieldworkers (see, for example, Bernard, Killworth, Kronenfeld, & Sailer, 1984; Dean & Whyte, 1958; Freeman & Romney, 1988). Terms like triangulation and multi-instrument approach may strike neophytes as ample safeguard against error in qualitative research, but anyone who has done fieldwork knows that if you address a question of any consequence to more than one informant, you may as well prepare for more than one answer. I try to report what I observe and to offer an informed interpretation of those observations, my own or someone else's. Only the most central of issues in one's research warrant the thorough probing implied by triangulation. We are better off reminding readers that our data sources are limited, and that our informants have not necessarily gotten things right either, than implying that we would never dream of reporting an unchecked fact or underverified claim.

Report Fully

I am not disconcerted by data that do not fit the developing account or my interpretation of it. That does not mean I report every discrepant detail, but I do keep such bits and pieces in front of me (often quite literally on 5” x 8” cards) as a way of testing my efforts at making sense of things. Sometimes a comment or observation can be introduced via brackets or footnote to flag an issue that is not as well resolved as the prose implies, or not developed more fully, because my data are “thin” or certain events never occurred during the period of fieldwork.

When I can do so without seeming too obtuse, I also include comments and observations that I do not understand or for which I feel I have no better basis for discerning meanings than might the reader. To illustrate: A Kwakiutl parent commented during a discussion about the kind of teacher best suited to a village assignment, “I think what we need here is a teacher who isn't too smart” (Wolcott, 1967, p. 85). I did not know exactly what he meant, and I still do not (although I have a hunch). I included it without comment—and today it continues to provoke possible interpretations every bit as plausible as my own.

Be Candid

I opt for subjectivity as a strength of qualitative approaches rather than attempt to establish a detached objectivity that l am not sure I want or need. As I am doing here, I have always put myself squarely into the settings or situations being described to whatever extent seemed warranted for the purpose at hand. With some fear and trepidation, I introduced that strategy in my doctoral dissertation. And committee members raised no concern except for the question of excess. I decided that if I could get away with it there, I certainly could be as forthright in the future when writing to satisfy myself.

To the extent that my feelings and personal reactions seem relevant to a case, I try to reveal them: The greater their possible influence, the more attention they receive and the earlier they appear in the account. In writing Teachers Versus Technocrats (1977), I was distressed both by the nature of a project designed to impose greater accountability on public education and by the heavy-handed manner by which it was being implemented. As a result, I began the monograph with a chapter titled “Caution—Bias at Work” to bare my feelings and objections.

How far to go with personal revelation? I see no easy resolution. The issue has become of more immediate concern than I could have imagined, as will become apparent in the course of this essay. Qualitative research has brought researchers self-consciously back into the research setting. That has been healthy for all, including those quantitative types who wanted everyone to believe that they were not part of their own investigations. Yet when someone remarks, often charitably, “I learned as much about you in your study as I did about the people you were studying,” I feel dismay at the likelihood of having taken more light than I have shed.

I try to draw a distinction between revealing my feelings and imposing my judgments, however. If circumstances call for me to draw implications or suggest possible remedies, I try to “change hats” conspicuously. There is simply no way one can get from a descriptive account of what is to a prescriptive account of what should be done about it. Those are value judgments. Granted, such judgments are critical to the work of practicing educators. It is appropriate for them to seek whatever help they can, and for us to be prepared to offer help, but we need to clearly mark the boundaries where research stops and reform begins. A different set of principles must be employed to validate our personal or professional authority to offer pronouncements about what needs improving and how to go about it.

The big value judgments are easy to spot because words like should and ought abound in sentences containing them. There are opportunities for eliminating little judgments as well, simply by careful use of the editing pencil. Little words of judgment creep into all kinds of sentences but can be rounded up and marched right off the page again. Consider the difference between reporting, “Only one villager had ever graduated from high school” versus “One villager had graduated from high school,” or “Few pupils were at task” versus “Five pupils appeared to be engaged in the assignment.”

Seek Feedback

I share my developing manuscripts with informed readers as part of the process of analyzing and writing. Rather than a mass distribution of a manuscript in next-to-last draft, what I have in mind is a continuous process of asking one or more individuals to read the current version. Academic colleagues are usually good for one careful reading at most; there is little point in pressing busy people for more, but no excuse for not asking at all.

Accuracy of reported information is one critical dimension, and readers close to the setting provide yeoman service by checking for correctness and completeness. Further, their reactions sometimes help me recognize where the reporting or the interpretation (or both) seems overblown or underdeveloped. Readers not so closely involved can also be helpful in assessing the suitability of my analytical concepts, my sensitivity to the people involved, or the adequacy and appropriateness: of interpretations made and lessons drawn. Readers who disavow their expertise or their familiarity with protocol in qualitative research may offer valuable suggestions about style and sequence, may question inadequate explanations or definitions, or may express straightforward but intuitive reactions conveyed in such statements as, “I just don't see what you are getting at here.” I also like to circulate working drafts among my graduate students. In terms of providing feedback, they are not necessarily one's severest critics, but even when they are not, they keep me mindful of my audience. In addition, it is valuable for them to have a glimpse of manuscripts in process rather than to have access only to polished final accounts.

Am I straying from validity? I believe not. I am describing a constellation of activities intended not only to help me get things right (or keep me from getting them all wrong) but to convey ideas in such a way that the reader, who is also not quite getting it right, is not getting it all wrong, either. I am willing to admit that some of this activity is image building, intended to create the impression that my accounts are credible. To the extent that is true, I can only add that having gone to great lengths to make such an impression, there is no particular reason not to work to live up to it.

Try to Achieve Balance

At some particular point or points in the writing/ revision process, I lake time either to return to the field setting or, second best, to read entirely through my field notes one more time (why do I find that so onerous a chore?). Then I reread my current draft to assess the extent to which the account I have created squares with the setting and individuals on which it is based. Objectivity is not my criterion as much as what might be termed rigorous subjectivity (or “disciplined subjectivity,” following Erickson, 1973, p. 15). It is I who must be satisfied now, with elusive criteria like balance, fairness, comp1eteness, sensitivity.

Ed Bell raised the question of balance in reacting to my study of him in his role as principal (Wolcott, 1973), but the issue exists in every study I have conducted. He found my reporting sufficiently accurate but nonetheless expressed dismay that the recounting of his problems (e.g., teacher evaluation; parent complaints, disagreements with the central office about his “leadership style”) received a disproportionate amount of attention. Ed literally worked day and night to smooth out problems and to create a positive, constructive atmosphere. I meticulously uncovered and examined every little malfunction to show what a principal must contend with.

Had I conveyed the minute-by-minute routine of Taft School during those 2 years, I think I would have fallen asleep writing about it long before anyone could have fallen asleep reading about it. Somehow I had to communicate the customary “hum” of a smoothly functioning elementary school while also assuring readers that the account would probe beneath the surface. Now, years later, Ed insists that my study helped him because it enabled him to see things that “needed improving.” Even he has forgotten what he once expressed as “real disappointment” at the perceived lack of balance in my account.

Write Accurately

As. a parallel activity to the field check just described, at one or more points during the writing process I read through a manuscript with an eye for what might be called technical accuracy. This stage usually comes rather late, prompted by feelings that the content and interpretation are pretty well in place but that I still need to make-an almost word-by-word assessment of the manuscript. One such check is for coherence or internal consistency. Much as coherence may appear to be a concession to the strictures of validity, I think of it more as an element of style. I accept as a compliment that something I have written appears to have “internal validity,” but frankly I regard consistency (which I think is implied) as much an author's trick as it is revealing of research acumen. That our studies are so free of inner contradiction ought really to set us wondering how they can be describing human behavior. As long as we employ consistency as a criterion, however, we will continue to find it in full measure.

There is another kind of internal review that seems more critical. l have no better term for it than a “word check.” I mean a literal sentence-by-sentence examination to check that the verbs are appropriate, the generalizations have real referents in what I have seen or heard, and the points of conjecture are marked with appropriate tentativeness. Admittedly, some part of this task is also a kind of window-dressing.  At a minimum I strive to make every sentence technically correct. I confess to having written sentences that virtually defy editing because they cannot be changed and still retain their truth element. (I give my own special meaning to “technical writing” —sentences that are correct as written but actually reveal how little, rather than how much, I know.) Still, in attempting to satisfy canons of technical accuracy, I try as well to be forthright (or sometimes just more modest, if that is what the circumstances warrant). The fact that a sentence needs fine honing serves notice that maybe it should not remain in the manuscript.

If such intentional wordsmithing seems unbecoming in scholarly work, it seems better dealt with head-on than by wishing it away from so human an undertaking as the human reporting of human social life. Qualitatively oriented studies of what goes on in laboratory lire, the bastion of true science, do more than hint that comparable efforts at the social construction of facts are not unknown among so-called hard scientists as well (see, for example, Gould, 1981; LaTour & Woolgar, 1986).

Wordsmithing also has its complementary and lighter side in sometimes allowing researchers to convey more, rather than less, information or to keep confidences. Let me illustrate with two sentences from A Kwakiutl Village and School (1967) describing tensions that arose over thefts of teachers' persona1 property in outlying villages:

During the year, five teachers in local day schools suffered losses by theft either of personal property taken from the teacherages or of skiffs and kickers (small outboard motors). Two thefts were privately resolved between teachers and villagers, two were reported to the RCMP, and no action was taken on the fifth. (p. 87)

Over the Christmas holiday, someone literally had “taken the fifth,” a  bottle of liquor hidden among the belongings of an older female teacher concerned as to whether she was breaking the law by bringing liquor onto the reserve in the first place. She dared not report the loss. She also felt some personal awkwardness, recognizing that the theft implicated a particular villager. Those of us who knew about her missing liquor had our chuckle and felt vindicated because all the thefts—with their demoralizing impact on everyone involved—had been reported accurately without exacerbating hard feelings.

Having reviewed some of the procedures I follow that provide the basis on which I would claim validity—were I pressed to claim it—I have come to the end of this part of my paper. These activities are my “answer” to validity, but reviewing them has only confirmed my feelings that the more important issue before us is to examine whether validity is the right question.

References

Becker, H. S. (1958). Problems of inference and proof in participant observation. American Sociological Review, 23, 652-660.

Bernard, H. R. (1988). Research methods in cultural anthropology. Beverly Hills, CA: Sage.

Bernard, H. R., Killworth, P. D., Kronenfeld, D., & Sailer, L. (1984). The problem of informant accuracy: The validity of retrospective data. Annual Review of Anthropology, 13, 495-517.

Brinberg, D., & McGrath, J. E. (1982). A network of validity concepts within the research process. In D. Brinberg & L. H. Kidder (Eds.), Forms of validity in research (pp. 5-21). San Francisco: Jossey-Bass.

Brown, R. (1965). Social psychology. New York: Free Press.

Campbell, D. T. (1975). “Degrees of freedom” and the case study. Comparative Political Studies, 8(2), 178-193.

Campbell, D. T. (1986). Science's social system of validity-enhancing collective belief change and the problems of the social sciences. In D. W. Fiske & R. A. Shweder (Eds.), Metatheory in social science: Pluralisms and subjectivities (pp. 108-135). Chicago: University of Chicago Press.

Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 443-507). Washington, DC: American Council on Education.

Cronbach, L. J. (1986). Social inquiry by and for earthlings. In D. W. Fiske & R. A. Shweder (Eds.), Metatheory in social science: Pluralisms and subjectivities (pp. 83-107). Chicago: University of Chicago Press.

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302.

Dean, J., & Whyte, W. F. (1958). How do you know if the informant is telling the truth? Human Organization, 17(2), 34-38.

Dobbert, M. L (1982). Ethnographic research: Theory and applications for modern schools and societies. New York: Praeger.

Erickson, F. (1973). What makes school ethnography 'ethnographic'? Council on Anthropology and Education Newsletter, 4(2), 10-19. [Revised and reprinted in Anthropology and Education Quarterly, 15(1984), 51-66.)

Freeman, L. C., & Romney, A. K. (1988). Words, deeds and social structure: A preliminary study of the reliability of informants. Human Organization, 46(4), 330-334.

Geertz, C. (1973). Thick description. In C. Geertz (Ed.), The interpretation of cultures (pp. 3-30). New York: Basic Books.

Goetz, J. P., & LeCompte, M. D. (1984). Ethnography and qualitative design in educational research. Orlando, FL: Academic Press.

Gould, S. J. (1981). The mismeasure of man. New York; W. W. Norton.

Guba, E. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries. Education Communication and Technology Journal, 29(2), 75-91.

Kaplan, A. (1964). The conduct of inquiry. San Francisco: Chandler.

Kirk, J., & Miller. M. L (1986). Reliability and validity in qualitative research (Sage University Paper Series on Qualitative methods, Vol. 1). Beverly Hills, CA: Sage.

La Tour, B., & Woolgar, S. (1986). Laboratory life: The construction of scientific facts. Princeton, NJ: Princeton University Press.

McEwen, W. J. (1963). Forms and problems of validation in social anthropology. Current Anthropology, 4, 155-183.

Miles, M. B., & Huberman, A. M. (1984). Drawing valid meaning from qualitative data: Toward a shared craft. Educational Researcher, 13(5), 20-30.

Peacock, J. L. (1986). The anthropological lens: Harsh light, soft focus. New York: Cambridge University Press.

Pelto, P. J., & Pelto, G. H. (1978). Anthropological research: The structure of inquiry (2nd ed.). New York: Cambridge University Press.

Phillips, D. C. (1987). Validity in qualitative research. Education and Urban Society, 20(1), 9-24.

Spindler, G. (Ed.). (1982). Doing the ethnography of schooling. New York: Holt, Rinehart and Winston. (Revised edition, Prospect Heights, IL: Waveland Press, 1987)

Tyler, S. A. (1986). Post-modem ethnography: From document of the occult to occult document. In J. Clifford & G. E. Marcus (Eds.), Writing culture: The poetics and politics of ethnography (pp. 122-140). Berkeley: University of California Press.

Van Maanen, J. (1988). Tales of the field: On writing ethnography. Chicago: University of Chicago Press.

Vendler, Z. (1984). Understanding people. In R. A. Shweder & R. A LeVine (Eds.), Culture theory: Essays on mind, self, and emotion (pp. 200-213). New York: Cambridge University Press.

Vidich, A. J., & Bensman, J. (1954). The validity of field data. Human Organization, 13(1), 20-27.

Weber, R. P. (1985). Basic content analysis (Sage University Paper Series on Quantitative Applications in the Social Sciences). Beverly Hills, CA: Sage.

Wolcott, H. F. (1967). A Kwakiutl village and school. New York: Holt, Rinehart and Winston. (Reissued, Prospect Heights, IL: Waveland Press, 1984, 1989)

Wolcott, H. F. (1973). The man in the principal's office: An ethnography. New York: Holt, Rinehart and Winston. (Reissued, Prospect Heights, IL: Waveland Press, 1984)

Wolcott, H. F. (1974). The African beer gardens of Bulawayo: Integrated drinking in a segregated society. New Brunswick, NJ: Rutgers Center of Alcohol Studies.

Wolcott, H. F. (1977). Teachers versus technocrats: An educational innovation in anthropological perspective. Eugene, OR: Center for Educational Policy and Management, University of Oregon.

Wolcott, H. F. (1982a). The anthropology of learning. Anthropology and Education Quarterly, 13(2), 83-108.

Wolcott, H. F. (1982b). Adequate schools and inadequate education: An anthropological perspective. (Commissioned paper prepared for the School Finance Project, Contract NIE-P-81-0271). Washington. DC: U.S. Department of Education.

Wolcott, H. F. (1983a). Adequate schools and inadequate education: The life history of a sneaky kid. Anthropology and Education Quarterly, 14(1), 3-32.

Wolcott, H. F. (1983b). A Malay village that progress chose: Sungai Lui and the Institute of Cultural Affairs. Human Organization, 42(1), 72-81.

Wolcott, H. F. (1984). Ethnographers sans ethnography: The evaluation compromise. In D. Fetterman (Ed.), Ethnography in educational evaluation (pp. 177-210). Beverly Hills, CA: Sage.

Wolcott, H. F. (1987). Life's not working: Cultural alternatives to career alternatives. In G. W. Noblit & W. T. Pink (Eds.), Schooling in social context: Qualitative studies (pp. 303-325). Norwood, NJ: Ablex Publishing.

Wolcott, H. F. (1988). Ethnographic research in education. In R. M. Jaeger (Ed.), Complementary methods for research in education (pp. 187-249). Washington, DC: American Educational Research Association.

Wolcott, H. F. (1989). A Kwakiutl village and school-25 years later. Afterword to the second printing of A Kwakiutl village and school. Prospect Heights, IL: Waveland Press.

Wolcott, H. F. (1990). Writing up qualitative research (Sage University Paper Series on Qualitative Research). Beverly Hills, CA: Sage.