The Journey Continues: Ensuring Cross-Culturally Competent Evaluation - Case Study A: Evaluation of a School District's Services

The situation

A hypothetical mental health agency is partnering with a school district to implement a program that provides services to students with behavioral issues. The local funder has asked that the program be evaluated to determine its effectiveness.

The school district is located in a geographic area that is rapidly becoming ethnically and racially diverse. Approximately 45% of the student population is African-American/Black, 33% Latino, 20% White and the remaining 2% “other.” Latinos make up the fastest growing group in the area; the number of Spanish-speaking students has just about doubled in the last two years. The average household income of families in the area is $35,000, below the average for the county in which the school district is located. The students participating in the school district’s mental health program reflect similar diversity.   

The mental health agency and school district contracted with a program evaluator who has worked with them before. The budget for the six-month evaluation is $10,000. Staff responsible for the program’s implementation told the evaluator they wanted focus groups with school counselors, teachers, students in the program and their parents. The staff wanted answers to the following questions:  
  1. Is the program effective in addressing behavior problems? Why or why not?
  2. Which other professionals need to be involved to help address these issues, and provide the necessary support to the students and their parents?
  3. What needs to change in order for everyone (e.g., school staff, mental health agency staff and parents) to work together to help and support the students who are struggling with these issues?

The evaluation

The evaluator worked closely with the school personnel to determine the best ways to recruit participants for the focus groups. With a budget of $10,000, the evaluator estimated that only five focus groups were possible, one with each of the following groups – teachers, school counselors and students – and two with the students’ parents. The evaluator agreed to let the school staff manage the invitations and logistics for the focus groups, so that she could have sufficient funds to develop the focus group protocol, analyze the data and summarize the findings.  

The school staff contacted the teachers and school counselors by email and telephone. To invite the students and their parents, the staff gave the students a letter and a consent form to take home to their parents. The letter explained the purpose of the focus groups, invited the parents to participate and also asked permission for their children to participate. If the parents consented to their children’s participation, they would sign the form and give it to their children to hand in to the evaluator before the focus group began. If the parents themselves chose to participate in the focus groups, they would simply show up at the time and location that was most convenient for them (three options were provided in the letter, and they were asked to pick the most convenient one).

The staff arranged to have the focus groups at the school between 4 and 6 p.m., assuming that it was a familiar location and convenient time for the students and their parents. The staff attempted to recruit 15 people for each focus group (except for the school counselors group because there are only four counselors in the school), in the hopes that at least 12 people would attend each session.

The evaluator was somewhat concerned the procedures for inviting the students and their parents were too passive and impersonal; however, given the budget constraints, she felt that they had no other choice unless they reduced the number of focus groups. She expressed her concern to the school and mental health agency staff, who acknowledged that the situation was less than ideal, but the best that could be done given the circumstances. According to the staff, the resources were limited, and frankly, it didn’t matter when they scheduled the focus groups because some of the parents simply didn’t care enough about their children’s education to attend anything organized by the school.

All five focus groups occurred as scheduled and the response rates were as follows: 10 teachers, 4 school counselors, 10 students and 7 parents (4 in one focus group and 3 in the other). The majority of parents who participated were African-American and White.

  • The evaluator summarized the results and submitted her report to the school district and mental health agency staff. She made the following recommendations in her report:
  • The school and mental health agency might consider programs to help the parents of students with behavioral issues strengthen their parenting skills
  • School counselors should work closer with the teachers of these students to strengthen the support provided in and outside the classroom
  • The mental health agency might consider offering additional counseling support to the students and their parents, independent of the school
  • Provide a peer-support program for the students and their parents.

Commentaries by the experts

The experts discussed the following issues and ways to make the evaluation more cross-culturally competent.

The budget is not sufficient. The experts agreed that the expectations of the mental health agency and school district were unreasonable for a $10,000 budget. Parent engagement in any school-related activity takes a more concerted effort than just one written invitation. The invitation method disregards three types of cultural and language differences among the parents: how they respond to invitations to share information about their families and children, the languages they speak and read, and their past experiences with data collection requests and activities. Consequently, the parents who responded to the invitation were not likely to represent the range of race, ethnicity and cultures of the students in the program; they were more likely to be parents who: could read English, were not shy about participating in any school activity, already had relationships with their children’s teachers and had their own transportation. At a minimum, if nothing else were changed to be more inclusive of parents from different backgrounds, the evaluation report should clearly describe this sampling bias, and how it affects the findings and recommendations.

A feasible design for the budget that still ensures a high-quality and cross-culturally competent evaluation. Cross-cultural competency is equivalent to high quality; any attempt to trade off cultural competency contradicts good evaluation practice, according to the experts who commented on this situation. The American Evaluation Association’s guiding principle on evaluator competence emphasizes the importance of recognizing, accurately interpreting and respecting diversity, as well as establishing a culturally-competent evaluation team.1

If a budget increase was not an option, the design could be modified to reallocate more funds for the parent engagement component, without “trading” cross-cultural competency for another competency. The evaluator needs to convince the school district and mental health agency staff that this modified design would yield better data; otherwise, the findings could potentially be weak and, if used to inform program improvement, even harmful. The American Evaluation Association’s guiding principles on systematic inquiry and integrity/honesty recommend that evaluators explore the strengths and limitations of various approaches with their clients and negotiate the best design possible.1

The modifications could include a web-based survey of the teachers and school counselors, instead of focus groups, because this group of people is more likely to: be familiar with surveys, easily understand the purpose of the study, have no literacy challenges and have access to a computer. Further, the sample size of teachers could be increased to gather more feedback.

The students’ opinions and feedback could also be solicited through a paper-and-pencil survey. The same consent procedures apply; the students would be given an explanation about the survey and a consent form for their parents to read and sign. The form will have to be available in English and Spanish, and easily understood by anyone with sixth grade reading skills.

The students could be given the questionnaire to complete after a program activity, if their parents consented. The students should place and seal the completed questionnaire in an envelope provided, to ensure confidentiality. The program staff would immediately send the completed questionnaires to the evaluator by overnight mail.

The use of a survey for the above groups will save the evaluator money, which could then be used to increase the number of parent focus groups from two to four (including one monolingual Spanish-speaking group), and to implement a more culturally appropriate invitation process that would ensure better representation of the racial, ethnic and culture diversity of the school.

A culturally appropriate invitation process. As suggested in the report, The Importance of Culture in Evaluation, the evaluator should ask herself the following questions before collecting any data:
  • Who can better help me understand the experiences of the parents in this school?
  • Who can help introduce me and the evaluation to the parents? What are past experiences for parents with an evaluator like me and with evaluation?
  • Who can I contact for professional interpretation and translation assistance, since I don’t speak Spanish?

The evaluator first has to frame the focus groups as informal conversations and discussions, because some of the parents may be unfamiliar with focus groups; confidentiality and consent issues must be addressed during the recruitment process, and reiterated before the conversation or discussion begins. Then, the evaluator could identify two to three parents to advise the data collection process. The evaluator could start by asking the program staff which parents interacted most frequently with them and showed interest in their children’s participation in the program. The evaluator could approach these parents, explain the evaluation and ask them about:

  • Any concerns that they might have about the evaluation
  • The best way to invite parents like them to participate
  • Incentives and other support (e.g., child care, transportation, professional interpretation, a convenient location and time) that would make their participation in the focus groups possible and comfortable.
The evaluator could also ask these parents to help reach out to other parents and ask them the same questions, as well as introduce these parents to the evaluation, and alert them about a follow-up telephone call and written invitation from the evaluator.

The above approach would help the evaluator avoid inaccurate assumptions about the parents and alleviate skepticism or discomfort the parents might have about their participation. As a result, the response rate could increase. The evaluator should also hire an interpreter to help reach out to any non-English-speaking parents and ask them the same questions.

The American Evaluation Association has a guiding principle that supports the cultural sensitivity described above: respect for people.1 According to this principle, evaluators are responsible for understanding and respecting the race, ethnicity, culture, religion, age, gender and sexual orientation of participants, and to consider how these characteristics affect the evaluation process, from design to reporting.  

Other considerations to ensure cross-cultural competence. In addition to the invitation process, the evaluator could consider other actions to further ensure the evaluation’s cross-cultural competence. For instance, besides engaging parents who speak little or no English, the evaluator could also be inclusive of parents who are not literate in their own native language. This would require the evaluator to read the consent form over the telephone, and record the conversation as proof of consent.

The evaluator might want to conduct separate focus groups for people from different racial, ethnic and cultural backgrounds, including one for monolingual Spanish speakers. The evaluator could approach this focus group in one of two ways: hire a professional interpreter who is also a group facilitator to conduct the focus group by him/herself, or conduct the focus group herself with the help of a professional interpreter and simultaneous translation equipment.

Back to top