The Journey Continues: Ensuring Cross-Culturally Competent Evaluation - Summary of Themes
Several themes emerged from the experts’ commentaries across the divergent situations that are useful in reframing evaluation-related situations to become more cross-culturally competent. As well, The Importance of Culture in Evaluation offers specific suggestions and steps for building cross-cultural competence.

Evaluators play an important role in promoting social equity

Evaluators are not just responsible for collecting, interpreting and reporting data. On the contrary, evaluators bear a bigger set of responsibilities, from educating other people about the responsible use of evaluation to using evaluation to promote social equity. This finding was reflected in Case Study A, which involved parents from different racial, ethnic and cultural backgrounds, and in Case Study C, which involved a Native American community. This is because evaluators’ work generates information – and information is power. As such, evaluators have to consider the implications of the findings, whether positive or negative, and how they could be used to benefit or harm a particular group of people.

The group’s history is a critical variable that evaluators should consider

In consideration of what outcomes are reasonable to expect and how evaluation findings might be used, evaluators need to pay close attention to the history of the people who are affected by the evaluation (e.g., Case Study C). While history obviously can’t be changed, it can be better understood to explain the outcomes experienced today by the people affected by the evaluation and, therefore, the true impact of the intervention.

Keep current on the dynamic context in which the evaluation is operating

Learning about the group’s history and its impact is not sufficient to fully understand the possible implications of the evaluation findings. Evaluators also need to stay abreast of the current context within which the evaluation is operating. Evaluation findings can be taken out of context to support or object to a policy or proposed legislation, like in Case Study D. Evaluators can gather information about such events by participating on Listserves and social networks that deal with the relevant issue, and by consulting with experts, particularly leaders and organizations, that advocate for the equity of the people involved.

Create a dissemination strategy as part of the evaluation design

Properly disseminating the findings is the responsibility of the evaluator and the funder. Possible scenarios about how the findings will be received by various audiences should be discussed early on in the evaluation process, so that a plan can be developed to address any potential controversy, as described in Case Study D. Evaluators also should avoid speaking freely to journalists and other media people, unless they are clear about the messages that need to be delivered or have been trained in strategic communications.

Evaluators must be aware of the dynamics of multiple social identities and group membership

All the situations described in the case studies allude to the complexity of multiple social identities (or group membership) and how they can affect the interactions between the evaluator and the participants. Case Studies B and C also suggest that racial and ethnic concordance between evaluators and evaluation participants, while helpful, is not sufficient to ensure cross-cultural competence. More important is a strategy to ask professional colleagues who share similar demographic characteristics and those who don’t, or those who have worked extensively with similar types of evaluation participants, to review the findings in order to minimize the evaluator’s potential biases as an “insider” or “outsider.”

Reach out and work with bridge builders or cultural translators

The complexities associated with multiple social identities and group membership are reasons why it is important for evaluators to reach out to people or organizations that already have credibility with the people participating in the evaluation (i.e., bridge builders or cultural translators). While racial, ethnic and cultural concordance between evaluators and the people involved in the evaluation is helpful, it is not enough to account for all the possible layers of diversity within the group of people. Bridge builders or cultural translators can help address this challenge; they can help “translate” concepts, terms and behaviors (e.g., reactions to certain words or processes). This is especially critical when working with communities that have been harmed by research in the past, like in Case Studies C and D. Bridge builders or cultural translators can also help recruit participants for the evaluation, like in Case Study A with the focus groups.

Provide a way for evaluation team members to reflect

Typically, more attention is paid to the potential impact of the evaluation on key informants, focus group participants or survey respondents than to evaluation team members. Participants can sometimes make remarks that offend evaluators (e.g., stereotypical comments about the cultural group that the evaluator belongs to), who may find themselves in an awkward position of having to check their emotions in order to maintain an objective stance. Such interactions, if not discussed, could affect the evaluator’s line of questioning during a subsequent interview or focus group, or interpretation of the data. Evaluation directors might want to conduct regular meetings with team members to reflect on the previous interactions, how the interactions may have affected the data collection and analysis process, and discuss strategies to correct the situation or prevent it in the future.

Evaluators need to consider the use of advisory committees to guide the evaluation

All the situations presented in this document could have been improved with the guidance of a small group of people from the communities involved. Their involvement could be formal (e.g., an advisory committee that meets regularly) or informal (e.g., individuals who are available to review materials and provide advice). Key questions and issues to discuss with such individuals include:
  • What has worked or not worked with regard to data collection in the community?
  • Are there tensions within the community that the evaluator should be aware of?
  • Are there certain individuals with whom the evaluator must engage in order to gain entry into the community?
  • How can the findings best be used to promote justice and equity for the community?

Evaluators need to promote the value of quality, not quantity

Engaging the right people as key informants should not be compromised for the sake of having a large, but perhaps incomplete sample, as suggested in Case Study A. Evaluators have to help funders understand the value of both qualitative and quantitative data. The value of qualitative data is especially important when the situation requires the evaluator to first understand the cultural meaning of the issues being examined. Until the evaluator develops this understanding, it may be inappropriate for him/her to adapt an existing questionnaire.

Plan for translation, interpretation and other language challenges

To engage the right people, evaluators sometimes have to be prepared to provide language translation and interpretation, as well as verbal communication for people with low literacy (like in Case Study A). It is important to use professional translation and interpretation, and not to use family members, social workers or other involved people for translation and interpretation. It is equally important to ensure that evaluation-related documents (e.g., description of the evaluation’s purpose, questionnaires and consent forms) are appropriate for the participants’ reading levels.