Evaluating a women's access course
New South Wales Department of Technical and Further Education
The purpose of this paper is to report on some aspects of the research design and methodology developed to evaluate a number of women's access courses provided by TAFE in NSW. The evaluation of the New Opportunities for Women (NOW) program was designed as a two-part project, involving a state wide survey of all participants since 1985 (Part 1) and a case study of one course using a number of qualitative data gathering techniques (Part 2).1 The data gathering stages of both parts of the project have been completed, and analysis of the results has confirmed that the research methodology developed for the evaluation was appropriate and productive (the findings from Part 2 are presented in Richards 1987).
To demonstrate the effectiveness of this two-part approach, and the different techniques used within it, this chapter considers two aspects of the evaluation methodology. Firstly, the discussion will show how the methodological choices made during the process of project design and development were influenced by the kinds of questions being asked of the research and by the circumstances within which some answers could be found, These methodological decisions were made prior to the conduct of the fieldwork phases of both parts of the research. The second aspect of the evaluation methodology considered here concerns the relationships that can develop between a researcher and those who are the focus of the research, in a project where qualitative methods are used over an extended period of time. In practice, these relationships have important implications for the kinds of information that research subjects are prepared to disclose. At the level of theory, the effect of these relationships upon the process of data gathering are part of the issue of scientific objectivity and the related questions of bias and the distortion of data. Although this issue was taken into account during the design stage of the project, its significance in practice was appreciated more fully once the research had begun.
Women's access courses in New South Wales TAFE
Since the mid 1970s, both Federal and State governments have been concerned with developing equal opportunity policies and initiatives, in particular in employment and education. In response to these moves, TAFE Authorities throughout Australia have formulated strategies to make their provision more equitably accessible to all groups in the community.
As part of this recent commitment to equality of opportunity, several of the States have developed programs designed to increase women's access to TAFE and through increased education and qualifications, to those sections of the workforce for which TAFE provides vocational training. The women's access program offered within the NSW TAFE system is the New Opportunities for Women (NOW) program. The NOW course is directed to women over the age of 25 who have limited education or outdated vocational qualifications who would like to undertake further education and training, and/or re-enter employment, and who are uncertain how to go about this" (NOW Syllabus, p.l). The course aims "to broaden and give direction to the personal and educational options of mature age women beyond traditional notions of what is appropriate for women" (NOW Syllabus, p.l), into those areas of technical employment with prospects of secure and satisfying work. The attention given in the course to technical occupations and qualifications follows from the hope that this focus "will contribute to an overall change in labour market opportunities for women at a time of rising under and unemployment, when traditional women's skills are becoming redundant in many industries and occupations" (Fell, 1983, p.l).
Many of the women in the NOW target group have withdrawn from the workforce for long periods to raise children. It has been argued by course developers that these women in particular lack the qualifications, skills and confidence which are necessary for successful competition for jobs in today's tight labour market. To this end the course also provides vocational guidance, support and assistance which will maximise opportunities for re-entry into further training or employment.
NOW courses are offered by TAFE in South Australia, Victoria, ACT, Western Australia and Tasmania. In NSW, the NOW course began in 1982 at three TAFE colleges. By 1986 over 80 courses were offered at both metropolitan and country colleges. The course has been developed in four formats to meet the needs of particular groups of women: NOW Base, NOW for Aboriginal women, women of non-English speaking background and deaf women.
Designing a course evaluation: Some methodological considerations
Several evaluations of NOW courses in NSW and other states had been conducted prior to the development of the project discussed in this chapter. In 1983, the first NOW evaluation in NSW, based on self evaluation and questionnaires administered by course coordinators, found that students were being drawn from the expected target group, and that their responses to the course were generally positive (Fell, 1983). Unfortunately however the evaluation was hampered severely by methodological difficulties such as low response rates, non-comparability across research instruments and too small a final sample to make statistical analysis possible. While many of these difficulties were a consequence of the newness of the course at the time of the research, the result was a very limited evaluation of the course. No interviews were conducted with either students or staff to gain greater understanding of issues such as the effectiveness of the teaching strategies used in the course, the nature of group formation amongst students and its role in the outcomes of the course, the possible sources of the reported increase in students' self confidence and issues related to the processes and dynamics of the course over time. Evaluations had also been completed of the Western Australian, Victorian and South Australian TAFE women's access programs (Pine, 1985; Jenkins, 1984; Jones, 1983 respectively). However, in terms of their relevance to the design of the evaluation discussed here, these previous evaluations were conducted within differing TAFE systems, involved different combinations of access programs, and varied considerably in their research objectives and choice of respondents.
Despite these variations in methodology, focus and research subjects however, all previous evaluations produced findings and conclusions which were relevant to the later NSW project. Each nominated childcare, timetabling, accessibility and income support as the most influential factors in the successful development and maintenance of women's access programs. The draft report by Pocock (1985) supported these findings. This important review of strategies for improving access and equity in education for women within the TAFE system at both the national and State levels provided an understanding of the larger institutional setting within which the various NOW courses operated. However, given its purpose and methodology, this research could not explore how the NOW courses work internally (ie, from the students' point of view). Thus, although all previous evaluations arrived at similar findings in terms of the factors which influence the success or otherwise of women's access programs, none of them reported on how these factors worked from the students' experience of them and none was in a position to draw a broader picture of an established women's access course offered across extensive areas of a State system. Thus the NSW course providers had very little information, either from NSW or from experiences in other States, about those issues considered central to the NOW program, namely,
During the design stages of the project discussed here, it was felt that an evaluation which provided a State wide examination of the target group and course outcomes, and at the same time a detailed, intensive investigation into the day-to-day processes of the course and the women's experiences of these, would provide a greater understanding of the course than had been possible in previous evaluations.
It was decided that the most effective technique for obtaining a state wide picture of the target group and the measureable outcomes of the course would be a series of standardised questionnaires. However a detailed examination of students' responses to the course over its 18 weeks would require a methodology which allowed access to the day-to-day environment and activities of the course. To accommodate these differences in research objectives, the evaluation was designed as a two part project. Part 1 was a survey of all course participants in NSW over the two year period from Semester 2, 1984 to Semester 1, 1986. During that period, questionnaires answered by students at the beginning, the end and six months after completion of the course, provided data needed for
While it was felt that questionnaires administered at particular points during and after the course would yield standardised and statistically meaningful data about all NOW participants in the State, a technique such as this could not uncover the dynamics, interactions and processes which develop throughout the course and which, according to teachers' and students' anecdotal reports, contribute significantly to its outcomes. Course providers wished to learn more about the processes of change which occur amongst students during the course, the interactions amongst students and between students and teachers, the relationships between the women's lives outside the course (both past and present) and their experiences in the course, and the ways in which mature age women learn and define what is relevant to them in learning. These questions required a research methodology which would enable the researcher to become a part of the research setting and to be responsive to those participating in the course and to events as they unfolded. Such a method also needed to be flexible to changes in the research design if they were called for, be capable of being sustained over a long period and be open to a variety of sources of data.
All of these methodological requirements needed to be met if a fuller understanding of the course was to be gained from this evaluation. Since it is possible to satisfy many of these requirements via the techniques of observation, participant observation, open ended interviews and informal conversations, the second stage of the evaluation was designed along these lines. Thus Part 2 was to be an ethnographic study of the NOW course at Gilmour College of TAFE in Sydney, commencing in Semester 1, 1986. This part of the study sought to understand the effectiveness of the course from the perspective of students.2
Unfortunately, over the four and a half months of the course, access to all areas of the lives of the group at Gilmour was not possible. This considerably restricted the scope of interaction with the students and thus the range of data gathering opportunities. In light of these limitations, the evaluation methodology became that of a case study rather than an ethnography, based on a combination of participant observation during part of the course and in-depth interviews with students both during and after the course.
As we can see, the decisions about methodology made during the process of research design were determined partly by logistics. It is almost impossible to interview meaningfully every member of a population of almost two thousand students, while it is possible to ask questions of them through the mechanism of a survey. These decisions were made also in consideration of the kinds of research questions which could he asked and the nature of the data that would answer them best. While it is possible with a large group of very scattered respondents to gather up the widest sweep of information through survey techniques, such techniques can only bring together certain kinds of information. Information about dynamic and interactive processes which require observation on the part of the researcher, as well as verbal description and explication from those involved in these processes, would inevitably fall through the far flung survey net.
The NOW course was developed originally within the framework of a feminist analysis of the effects of the sexual division of labour on women's position in the workforce, the lack of access for mature age women to vocational training within TAFE, and the kind of specialised skills training and guidance needed by mature age women who wished to re-enter employment or education after long absences. Over the last two decades, feminist researchers and theorists have documented the personal and collective experiences of women in almost all aspects of their lives, and used these to elaborate theories about the nature and origins of women's consistently subordinate positions in larger social structures and their experiences of power and domination in personal relationships. At the same time, feminist researchers have developed extensive critiques of some of the dominant paradigms in social science philosophy and method (Bowles & Duelli-Klein, 1983; Stanley & Wise, 1983; Roberts, 1981; Westkott, 1979). Some of these criticisms, which are particularly relevant here, point to the omission of women's experiences, as they speak and feel about them, from theoretical and methodological concerns in mainstream social science, the imposition of abstract, a priori theory upon data which often is predetermined by the method used to obtain it, the fragmentation of reality into discrete, methodologically and theoretically "manageable" units of analysis, and the distinction made between the researcher and the "object" of research in the interest of scientific objectivity. Du Bois (1983) amongst others (Reinharz, 1983; Duelli-Klein, 1983) argues for an alternative methodology which can make good some of the sins and omissions of traditional enquiry. In her view, research about and for (not on) women needs to generate concepts and frameworks that are:
...firmly and richly grounded in the central experiencing of women. And this demands methods of inquiry that open up our seeing and thinking, our conceptual frameworks, to new perceptions that actually derive from women's experience ... Our scientific methods ... require seeing things as they are: whole, entire, complex ... (and) in context ... It is only within its (context) that experience, reality can be known. And this (context) includes the knower (1983, pp.110-11, emphases in original).
This last point, the relationship within the research setting between the knower and the known, has had particular significance for the NOW evaluation, as illustrated in the final section of this chapter. Positivist (and, as Hammersley & Atkinson, 1983, argue, also naturalist) scientific paradigms distinguish between the knowledge and actions of the researcher and those of the individuals who are the focus of the research. This leads to their "joint obsession with eliminating the effects of the researcher on the data" (Hammersley & Atkinson, 1983, p.14). This determination to remove researcher bias and distortion of data also means removing any recognition of the possibility of a relationship between the researcher and the researched. Even at the level of common sense and everyday experience, however, this search for "scientific objectivity through detachment from the research process" is doomed to fail, as we necessarily and inevitably are part of the world we research. It is almost unavoidable that some sort of relationship is established between the researcher and the research subjects and that some effect flows from the researcher's presence in the research setting or from intrusion into it through some medium (eg, by telephone or mail-out questionnaire).
Rather than a source of bias and distortion, the outcomes of relationships between researcher and researched may form part of the data, as Hammersley and Atkinson point out (1983, p.15). For this to happen, "historical truths (should be) grasped not by attempting to eliminate subjectivity but through the intersubjectivity of meaning of subject and object" (Westkott, 1979, p.426). This intersubjectivity does not mean, however, that the researcher and the researched may identify with each other, but that a "dialectical relationship" (Westkott, 1979, p.426) of question and answer (and perhaps confrontation, evasion, counter-question and silence) is established between them. It could be argued further that it is by virtue of this relationship that in many instances we are able to do research at all, and that often we increase our knowledge of social reality because of the relationship between researcher and researched and not, as is commonly held, in spite of it.
... (T)he mythology of "hygienic" research with its accompanying mystification of the researcher and the researched as objective instruments of data production [should] be replaced by the recognition that personal involvement is more than dangerous bias - it is the condition under which people come to know each other and to admit others into their lives (Oakley, 1981, p.58, emphasis added).
During the two and a half months which I spent in the NOW course at Gilmour College, the relationships that I established with the 15 women in the group (who were aware of my research role) enabled me to gain an understanding of what the course meant to its students beyond any that had been possible in previous evaluations. During this time I was aware that, in developing these relationships, what I saw, heard and noted down as data was not only material absorbed by me, but was also in part created by my presence there and by the history and nature of these relationships. Within the group, as a researcher and a fellow student, I was both insider and outsider. Thus at times I knew I was not being told everything and yet, at other times and because of these relationships, I was often "privileged with confession, the otherwise unsaid, the heartfelt and the bitter" (Ball, 1984, p.17).
It has been argued that "idiosyncrasies of person and circumstance are at the heart, not the periphery, of the scientific enterprise" (Johnson in Bell & Newby, 1977, p.9). The centrality of the personal to scientific inquiries into the social is demonstrated in the history of one woman in the NOW course at Gilmour College. The significance of the relationship between the researcher and the researched to that inquiry is also demonstrated in her history, and our joint history in the course. As the following discussion shows it was through coming to know each other through our experiences during the course that I was privileged to gain information that would have remained, under different circumstances of research, the "otherwise unsaid".
"What do you say when you talk to someone about it?"
Julie is a Maori woman in her mid-thirties who arrived in Australia 10 years ago with her two small children. She finished school in New Zealand at 15 and, in the years before her marriage, worked mainly in textile factories as a machinist. Since her divorce five years ago, she has been living on Supporting Parents Benefit, with occasional part-time factory work.
This was the picture of Julie's life given by the information asked of her on the NOW course application form and it was all that I or the teachers knew of her when the course began. Throughout the next few months, she often remarked that she had enrolled in the NOW course to "brush up on my maths and English". However, as I eventually discovered, her unstated agenda in coming to Gilmour was not to enrol in a short refresher course in mathematics and English, but to find some solution to her problems with literacy and numeracy. Julie was almost illiterate and innumerate, yet although she hoped to find ways of changing this in the course, her anxiety about not being able to read and write easily, her well developed skills in concealing this inability and her great unwillingness to broach the subject with anyone, including her close friends, made it almost impossible for her to ask for the help she needed. Throughout her adult life, in situations which looked like exposing her difficulties with words and numbers, she would either bluff her way through them or find legitimate ways of withdrawing from them altogether. With practice, she had become very successful at determining the limits to what anyone would know about her ability to read and write, her formal qualifications and her past educational experiences. When these well rehearsed strategies were brought to a situation like the NOW course, which works within a cooperative, sympathetic and non-threatening teaching and learning environment, it was very easy for Julie to avoid those moments in which the true extent of her illiteracy and innumeracy could become known.
I fooled my way through, like I do a lot of things. When we had to write things in class I'd sit there, you know, and watch the others. They were so good at it. Then (the teacher) would say, "O.K. Have you finished?" and I'd say, "Yeah, I've finished", even though, you know, I didn't write a thing. I had it all in my head, right, like I'd get everything ready in my head so if she's asked me I could say it. Then I'd hope like mad she wouldn't come and look at it.
This determination to avoid exposure as an illiterate adult coexisted uneasily with the often overwhelming need to find out just how bad the situation was and what she could do about it. Her strong desire to "do things with my life", and her confidence in her ability to take on whatever she put her mind to if only she were "educated", were frustrated by this ever present problem.
To me it's a very big problem. Because I don't want this you know. I want to do so much, with my sons and with myself, and I can't ... and when I think about that, that's depressing. It's a problem finding out what level I'm at. I don't know where I am in my learning. If I knew that I'd feel better about myself ... See, I know I've got the confidence in myself to go out and do these things if I had the education. I'd be off, there'd be no stopping me, you know, I'd do a lot, I know I would, but I can't ... And it's really hard to know what to do. What do you say when you talk to someone about it? If I ring on the phone, do I say, "Hello my name's Julie and I can't read"?
Although she gained in many other ways from her experiences in the NOW course, Julie felt she had made no real advance in overcoming a central problem in her life.
The first clue that Julie might be having trouble with words and numbers came very early in the course when we worked together on a project that involved writing. During this exercise I noticed that her handwriting was the large, slow printing of early primary school years and that she was very reluctant to show anyone what she had written. I thought that perhaps she lacked basic literacy and numeracy skills and that, if this were the case, then her educational needs and past experiences would be quite different from those of the other women in the group. Without this original non-verbal clue that my combined role of student and participant observer gave me, I might not have been able eventually to understand the real nature of Julie's experience of the course and to assess the extent to which the course was of benefit to her particular needs. Because of the research techniques used during the case study at Gilmour College, I gained an insight into one person's experience of the course that other methodologies could not have provided. From observation and participation, I discovered firstly that the problem existed and secondly that, given her efforts to hide it, it was likely that only under certain circumstances would she discuss it and its effect on her life.
Qualitative research is often carried out in circumstances within which it is possible to establish a personal relationship with those involved in the research, through shared experiences over a period of time. The importance of the relationships that are possible through this methodology to the kinds of research questions that are being asked is illustrated by the consequences of the relationship Julie and I established during the course. Through observing the energy and skill she invested in camouflaging her literacy and numeracy difficulties, and guessing at the anxiety behind the spirited manner with which she maintained this cover, I sensed that only if and when she felt comfortable with me could I talk about it with her, and piece together the events which had placed her in this position. Without this understanding of the events in her life, and without hearing from her how she perceived herself and her needs, I could not assess the strengths and weaknesses of the course from her point of view. And without a relationship between us based at least on her confidence that she would be understood and accepted, we could not arrive at the point of talking directly about her needs of the course. It was through the experiences we shared during the months of the course that such a relationship grew, until eventually eight months after the course began we were able to discuss at length how it came about that she could barely read and write.
Julie's family had moved constantly around New Zealand while her father looked for seasonal work and battled with the financial difficulties of raising and educating a family of 14 children. For Julie the result was a badly disrupted primary school career, after which she left high school at 15 with no idea of her real educational level, or of her abilities.
Like, we'd move to one school and that'd be great, and then we'd move on, and then by the time high school came the teachers more or less had to send me on because I was too old. And when I got (there),just honestly, I didn't know where I was going so I just didn't bother. I'd try but it didn't make any difference ... I was lost, you know, I really was, I was absolutely lost. I really didn't know what I was learning, it was just too fast for me. I didn't know what level I was at and I really thought there was no hope.
The strategies that she used in the NOW course were echoes of those which she began using at school and of the "bluff' she often described using to get jobs later on. Her unwillingness to talk about her school years seemed to stem from a sense of shame at being the only one she knew of like this and from a great fear of ridicule.
At school it got to the point where I was too embarrassed to say that I didn't know and put my hand up in front of the class and say it ... I was so far behind anyway that I didn't speak out and say anything because I didn't want to be put aside and treated as someone different or a bit slow... [And when the teacher said], "Does everyone know it?"... I'd just agree and say, "Yeah, I know it". What could I do? ... I haven't told anyone. Actually I've told you and another friend of mine ... it's funny, she's the only one of my friends I've ever told. I just felt good talking about it with her because I knew in my heart she wouldn't laugh at me, that she wouldn't talk about it or judge me. I know a lot of people would. They'd find it quite funny actually, you know, make fun of me. That's why I don't tell them.
It is highly unlikely that Julie would have written these words on a questionnaire, or spoken them to an unknown interviewer asking for her opinions of the course. From her account, she only became willing to talk to me because of the degree of trust and understanding that had developed between us. ("I feel like I can talk to you about it, you know. You understand what I'm trying to say, you don't make me feel bad.") The opportunity and circumstances for that trust and understanding to grow, to form the basis of our relationship, were provided by the place in the course that the research role of participant observer gave me. Without that relationship, and the questions and answers that moved to and fro within it, important insight into the workings of the NOW course would have been lost.
This paper began by arguing that the choice of research methods used in the NOW evaluation was governed by the nature of the research questions being asked. The NOW course was developed on the basis of a feminist analysis of the educational needs of socio-economically and educationally disadvantaged women, particularly in relation to curriculum content and teaching and learning processes. The program deliverers, who instigated the evaluation, required different kinds of data to measure the accuracy of that analysis, and to explore the effectiveness of the program which was developed from it. These were, firstly, State-wide information about the composition of the target group, postcourse activities of students and variations between the formats of the course and regions within the State and, secondly, a more detailed and holistic understanding of mature age women students' responses to a re-entry course of this kind.
In the design stage of the evaluation, it was decided that only qualitative research methods could yield the type of data necessary to assess the effectiveness of the course from the students' experience of it. This brief account of the relationship that developed during the course between Julie and me illustrates the usefulness of these techniques, and just how misleading data collected via quantitative methodologies alone might have been in understanding the complex elements of this experience.
Ball, S.J. (1984). Beachside reconsidered: Reflections on a methodological apprenticeship. In R.G. Burgess (Ed.), The research process in educational settings: Ten case studies. Lewes: Falmer Press.
Bell, C. & H. Newby, H. (Eds.) (1977). Doing sociological research. New York: The Free Press.
Bowles, G. & Duelli Klein, R. (Eds.) (1983). Theories of women's studies. London: Routledge and Kegan Paul.
Du Bois, B. (1983). Passionate scholarship: Notes on values, knowing and method in feminist social science. In G. Bowles and R. Duelli Klein (Eds.), Theories of Women's Studies. London: Routledge & Kegan Paul.
Duelli-Klein, R. (1983). How to do what we want to do: Thoughts about feminist methodology. In G. Bowles and R. Duelli Klein (Eds.), Theories of Women's Studies. London: Routledge & Kegan Paul.
Fell, L. (1983). New opportunities for women program: Course evaluation 1983. Sydney: Women's Coordination Unit, NSW Department of TAFE.
Hammersley, M. & Atkinson, P. (1983). Ethnography: Principles and practice. London: Tavistock.
Jenkins, D. (1984). An evaluation of women's access programs conducted by TAFE colleges in the State of Victoria. Canberra: Evaluations and Investigations Program, Commonwealth Tertiary Education Commission.
Jones, G. (1983). NOW Project: An evaluation of the pilot courses. Adelaide: Equal Opportunities Unit, South Australian Department of TAFE.
Women's Coordination Unit. NOW Syllabus. Sydney: Women's Coordination Unit, NSW Department of TAFE.
Oakley, A. (1981). Interviewing women: A contradiction in terms. In H. Roberts (Ed.), Doing feminist research. London: Routledge & Kegan Paul.
Pine, J. (1985). Evaluation of the New Opportunities for Women (NOW) program. Perth: TAFE Women's Interests, Western Australian Department of TAFE.
Pocock, B. (1985). Changing systems: Women, work and TAFE. Prepared for National Working Party of Women's Advisers in TAFE, through Commonwealth Tertiary Education Commission.
Reinharz, S. (1983). Experiential analysis: A contribution to feminist research. In H. Bowles and R. Duelli Klein (Eds.), Theories of women's studies. London: Routledge & Kegan Paul.
Richards, W. (1987). NOW and then what?: Evaluating a TAFE access course for mature age women (Draft report). Sydney: Women's Coordination Unit, NSW Department of TAFE.
Roberts, H. (Ed.) (1981). Doing feminist research. London: Routledge and Kegan Paul.
Stanley, L. & Wise, S. (1983). Breaking out: Feminist consciousness and feminist research. London: Routledge and Kegan Paul.
Westkott, M. (1979). Feminist criticism of the social sciences. Harvard Educational Review, 49(4), 422-430.
Cite as: Richards, W. (1988). Privileged information: Evaluating a women's access course. In C. McBeath, (Ed). (1988). Case studies in TAFE curriculum, p.57-64. Perth, WA: West Australian Social Science Education Consortium, Curtin University of Technology. http://www.clare-mcbeath.id.au/case-studies/chap8.html