The Journey Continues: Ensuring Cross-Culturally Competent Evaluation - Case Study C: Evaluation of a Program Located in a Native American Community

The situation

A grant program, established by a hypothetical national foundation, addresses domestic violence through evidence-based intervention strategies. Eight nonprofit organizations in the southwest region of the United States were selected to receive five-year grants. These organizations ranged from family resource centers to domestic violence shelters. In their proposals to the foundation, the organizations identified the evidence-based interventions they intended to adopt and adapt.

The foundation was interested in learning the following:
  • Which evidence-based interventions did the grantees adopt and adapt?
  • Did the number of women served and assisted by the grantees increase as a result of the adapted interventions?
  • Did domestic violence incidents decrease as a result of the grantees’ work?

The foundation hired an evaluation firm to conduct a process and outcome evaluation of the grant program.

The evaluation

The evaluation team worked closely with the foundation’s program staff to develop a logic model for the grant program, the core set of process and outcomes measures to be monitored by the evaluation team, and the schedule for data collection and reporting.

To manage the eight grantees, the evaluation team director assigned two team members (“the site team”) to each grantee. Site teams were responsible for all communication and evaluation activities for their assigned sites.

Two people – a Native-American woman and an Asian-American woman – were assigned to work with the one grantee located in a Native-American community. To ensure that the site teams approached their responsibilities in a uniform way, the evaluation director developed a set of protocols for all the team members to follow. The protocols included guidelines for when and how the site teams should check-in with representatives from the sites they managed, collect data for the national evaluation, and report the findings back to the site’s representatives.

One year into the evaluation, initial evaluation findings showed that the Native-American grantee had made the least progress, compared to the other seven grantees. The number of women served and assisted by this grantee had not changed since the grant project started; further, the intervention was unclear even though the grantee was supposed to have adopted an evidence-based intervention that focused on strengthening communication among family members.

When the Native-American grantee saw the evaluation report, they were very concerned. The grantee’s project director and an elected Tribal leader contacted the foundation’s program officer for the grant program, and shared their concern that the findings about their site were not accurate. They felt that the evaluation team did not understand their intervention. They added that the evaluation did not highlight a significant accomplishment in the community – the establishment of an intergenerational program where Native-American youth and elders get together to talk about the violence in their lives and community.

Commentaries by the experts

The experts’ responses emphasized the importance of paying attention to the following when conducting evaluations in Native-American communities.

Community history and context. A core set of process and outcome measures is usually necessary for an evaluation that involves multiple sites in order to test the program’s logic model. At the same time, the evaluator has to consider the unique political, economic, social and cultural circumstances of the participating sites, and how these circumstances affect the implementation process and outcomes at each site. The set of protocols, to ensure uniformity in the way site teams approached their sites, should also require the teams to consider and document those factors and conditions that are likely to vary across sites (e.g., community composition, local governance and recent events [such as the shooting of a police officer or a youth’s suicide] that may have affected the community in significant ways).

The evaluation team for the Native-American grantee needs to understand the nature of domestic violence in Native-American communities, in order to know what additional outcomes were important to document besides the core measures (e.g., why the intergenerational program was considered by the site’s representatives as a significant outcome, yet under-emphasized by the evaluation team). As suggested in The Importance of Culture in Evaluation, the evaluation team needs to stop and consider:  
  • What is behind the high incidence of domestic violence in this community?
  • What are some of the root causes?
  • What is known about evaluating domestic violence prevention or intervention programs in Native-American communities?

The concept of historical trauma or historical grief is especially important to understand when working with Native Americans.2, 3 Historical trauma is unresolved grief that is likely to affect the survivors in negative ways; such grief is passed from one generation to the next and, therefore, could have a cumulative impact on communities.3 To address domestic violence in their community, it might be prudent for the program staff on site to first create a culturally-rooted process that will allow community members to talk about their experiences and acknowledge historical trauma as a condition and, thus, initiate the healing. This process takes time.

It is reasonable to assume that this unique circumstance in the Native-American site means that: 1) it may take a relatively longer time to see any increase in the number of women served and assisted in the Native-American grantee site compared to the other sites, and 2) the community healing process and its results (including the number of participants) are critical for the evaluation team to capture and document, even though it may not have been part of the core process and outcome measures.

Use of research and evaluation in Native-American communities. Evaluators who work with Native-American communities must first and foremost recognize the history of how research about Native-Americans has harmed, rather than strengthened, their communities.4, 5 The evaluation team had to follow the evaluation protocols to ensure uniformity in the evaluation’s approach and implementation; however, in doing so, they may not have paid enough attention to building trust and relationships with representatives from the Native-American community. Their actions, therefore, were most likely viewed with suspicion by the group’s representatives.

To build trust and relationship, the evaluation team should first engage the project director and other Tribal leaders in a general discussion about their vision of the domestic violence program, their expectations and concerns, their hope for positive change in the community as a result of the effort, as well as their previous experiences with evaluation. Such a discussion would also inform the evaluation’s choice of core measures and ensure that outcomes critical to this grantee are captured and documented appropriately. Examples of questions that can be used to guide this discussion are:
  • What does evaluation mean to you?
  • What experiences have you had with evaluation? Was it positive or negative, and why?
  • What do you expect to accomplish with this evaluation?
  • Who needs to be part of the planning and implementation process?
  • How have you used evaluation findings in the past?
  • How can the evaluation process align with the Tribal government decision-making processes?

There is a growing body of information about conducting research and evaluation in Native-American communities. For example, the American Indian Law Center has a publication to help guide Native communities develop their own research codes that might be useful for non-Native researchers and evaluators to review (see http://www.ihs.gov/medicalprograms/research/pdf_files/mdl-code.pdf), and the American Indian Quarterly frequently publishes articles about evaluation research in different tribal communities.

Nation building and codes of conduct in tribal communities. Evaluation is a process of inquiry. Through this process, information is uncovered that could be used to benefit communities (e.g., improve policies that perpetuate social inequities). In tribal communities, evaluation can support nation-building by providing feedback about the abilities of tribal institutions to address contemporary issues (e.g., financial management, mental health, crime) and, ultimately, improve the economic and social conditions in these communities.6 Therefore, evaluation in Tribal communities is a tool that can serve the larger goal of decolonization and sovereignty.6

Evaluators working in tribal communities should respect and comply with each tribe’s codes for conducting research on their lands, and consult with the tribe’s Internal Review Board (IRB) processes, if one exists. Evaluators should always check with the community’s representative about the tribe’s approval process for conducting research and evaluation on its land.5

Diversity of the Native-American people. The Native-American communities across the United States share some similarities, but they also vary in their histories, cultural traditions, languages and experiences with non-Native systems and institutions. It is helpful to have a member of the evaluation team be of similar race and ethnicity as the evaluation participants; however, it is not sufficient and should not replace the work that needs to be done by evaluators to become cross-culturally competent. The Importance of Culture in Evaluation discusses how people have several social identities because they tend to belong to two or more groups (e.g., an educated Asian female professional), and how these identities often become the basis for creating an “us versus them” dynamic, even within a cultural group. Evaluators who work with cultures different from their own should remember that there is always more to learn about another culture. This is an important lesson even for evaluators who work with cultures similar to their own. Use of advisory committees that consist of people from diverse backgrounds can help the evaluator look at the evaluation from different angles and through different world views.

Back to top