A report/memo that addresses previous ISRP comments is needed.

A sound scientific proposal should respond to the six questions and related material at the beginning of the regional coordination section.

The ISRP's 2007-09 set of review comments provides qualifications on evaluation of outcomes that need to be addressed in the current proposal. ISRP comments include:

“Overall, the response misses the point and does not address the ISRP’s comments on the need for better self-evaluation and monitoring of CRITFC activities.”

“The sponsors need to take a more proactive approach to learn how to conduct an effectiveness evaluation and to conduct it. At present, effectiveness is asserted rather than documented.”

“Stating, ‘As already agreed to by the ISRP, monitoring of coordination effectiveness is difficult to evaluate quantitatively’ is again missing the point. Although it is difficult, it is both desirable and possible. The point is that careful thought should be given to what effectiveness would look like and how it can be measured, then develop a plan to measure it and evaluate it. Agreeing to ‘document any incidences of overlap or redundancy with CRITFC and individual tribal projects if they occur as a measure of effectiveness’ is not sufficient and does not address the central question of effectiveness.”

The proposal should be re-written to include a better statement of objectives as desired outcomes and separate from tasks. Some text that could serve as the basis for rewritten objectives is already contained in the proposal. The proposal should be more explicit about how adaptive management is conducted within this project, and about how methods of implementation can be measured and evaluated for success. Metrics to measure performance should be identified beyond the general statement in the "objectives" section. A plan should be developed to use these metrics to evaluate performance, including stakeholder evaluation.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

The project has five components. The percentage of project time spent on each is not identified.

Significance to regional programs: Wy-Kan-Ush-Mi Wa-Kish-Wit was publically presented in 1995 in CRITFC journal Wana Chinook Tymoo. To expand understanding of the background and concepts, a set of videos (Chinook Trilogy) presented the plan’s background in tribal culture, the problems that led to the need for a tribal plan, and the basic elements of the plan. The proposal states, "The tribes’ recommendations in Wy-Kan-Ush-Mi Wa-Kish-Wit are designed to: define problems, propose remedial actions, set objectives, and describe means to evaluate the actions.” Wy-Kan-Ush-Mi Wa-Kish-Wit has had considerable impact on fish and wildlife actions in the Columbia Basin. The guidance from Wy-Kan-Ush-Mi Wa-Kish-Wit is relevant to the Fish and Wildlife Program, FCRPS BiOp, Fish Accords, and actions of other sovereigns.

Wy-Kan-Ush-Mi Wa-Kish-Wit lays out the perspectives of the Nez Perce Tribe, the Confederated Tribes of the Warm Springs Reservation of Oregon, the Confederated Tribes of the Umatilla Indian Reservation, and the Confederated Tribes and Bands of the Yakama Nation, the members of The Columbia River Inter-tribal Fish Commission. One of the goals is to revise Wy-Kan-Ush-Mi Wa-Kish-Wit.

Problem statement: a complete description of the role of CRITFC staff in coordinating member tribes to implement the Wy-Kan-Ush-Mi Wa-Kish-Wit and the NPCC Fish and Wildlife Program.

From a regional-coordination, science perspective, addressing three questions more systematically would be helpful. First, how well are the concepts presented in Wy-Kan-Ush-Mi Wa-Kish-Wit understood by tribal and nontribal members in the region? Second, who is the audience for Wy-Kan-Ush-Mi Wa-Kish-Wit? When initially presented CRITFC leaders tried to help all residents of the Northwest understand the proposals in Wy-Kan-Ush-Mi Wa-Kish-Wit. What coordination activities are recommended for each of the target audiences? How are these methods of outreach, education, and information dissemination evaluated? Third, what is needed in the way of revision to Wy-Kan-Ush-Mi Wa-Kish-Wit? Have the members engaged in a coordinated approach to identify themes for revision? Are other Columbia Basin tribes joining the effort? Has science suggested the need for revision? Is revision needed to increase understanding or to add elements to provide a more complete picture? Are there missing elements that need to be incorporated and elaborated?

Objectives: The project has two objectives: 1. Implement and update Wy-Kan-Ush-Mi Wa-Kish-Wit; 2. Provide coordination and outreach to tribes. The proposal objectives of coordination and outreach are worded as tasks rather than as desired outcomes. The deliverables include regional coordination, tribal coordination, outreach and education, “incorporating the principles of the tribal salmon plan, Wy-Kan-Ush-Mi Wa-Kish-Wit, into the Fish and Wildlife Program” and managing and administering the project.

Actual objectives in terms of outcomes statements are contained in the descriptions provided with the objectives. The proposal lacks any plan to observe and measure any to the objectives identified.

Emerging limiting factors: The statement refers to CRITFC projects related to climate change, toxics, water quality, habitats, and invasive species but does not address limiting factors as related to the implementation of this coordination project.

2. History: Accomplishments, Results, and Adaptive Management (ISRP Review of Results)

Financial performance and history: Brief explanations of differences between expenditures and contracted amounts are given.

Performance: Brief description of most reports and deliverables on time.

Major accomplishments: A summary statement of major accomplishments for every year of project funding. The most recent year included a conference, outreach, testimony and comments.

Response to ISRP comments (also see Qualifications): The proposal sponsors did not respond to past ISRP comments asking for observation and measurement of outcomes from their activities. ISRP and Council comments from 2000 and 2007 should be incorporated into the proposal.

It is not clear from the content of this proposal that the Wy-Kan-Ush-Mi Wa-Kish-Wit project is meeting NPCC or tribal needs for regional coordination of watershed activities. Responses to past ISRP and Council comments have been inadequate. In particular, an effectiveness evaluation plan needs to be developed and explained in this proposal in order for the project to meet scientific criteria. Many sections of the proposal need to be re-written to adequately address the requested information (see Qualifications).

Adaptive management: A statement is given about transmitting information to member tribes, generating comments and actions. The proposal is not framed in an adaptive management framework.

ISRP Retrospective Evaluation of Results

The history of significant accomplishments in the proposal runs from 1998 to the present. The financials give history since 2006, which was the first year of Bonneville funding. In 2006, successes with Bonneville were summarized in CRITFC ’s Wana Chinook Tymoo journal. A PCSRF brochure highlighted success stories of cost-sharing with BPA. The most recent effort to communicate the outcomes from Wy-Kan-Ush-Mi Wa-Kish-Wit based approaches was the “Future of Our Salmon: A Vision of Restoration in the Columbia River Basin,” held June 1-2, 2011.

The 2010 “Implement Wy-Kan-Ush-Mi Wa-Kish-Wit” annual report lists activities on revising the plan during FY 2010, but it includes no lessons learned or hypotheses tested and remaining. More investigation and analysis of how coordination activities will result in getting Wy-Kan-Ush-Mi Wa-Kish-Wit principles, objectives, remedial actions, and means to evaluate actions into the Fish and Wildlife Program would be desirable.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (hatchery, RME, tagging)

Project relationships: listed are other CRITFC projects, PSC, PCSRF, and PNAMP

Regional coordination focus: Wy-Kan-Ush-Mi Wa-Kish-Wit is a Northwest regional project that directly affects the success of the Fish and Wildlife Program in the Columbia Basin. Wy-Kan-Ush-Mi Wa-Kish-Wit concepts are applicable to fish and wildlife, endangered species, BiOp, and issues within and outside the Columbia Basin.

The proposed work includes, “Review of technical documents and processes;” “Coordination of projects, programs and funding sources within subbasins;” “Facilitating and participating in focus workgroups on Program issues;” “Information dissemination (technical, policy, and outreach);” and “Project proposal review.” In describing the work, mention is made of workgroups, forums, committees, conferences, outreach, brochure preparation, testifying. These are important inputs for gaining desired outcomes. The proposal, however, does not report any outcomes in the sense that the meetings, brochures, conferences, and other coordination activities had an impact that was intended or unintended. Further, there is no evaluation of which coordination activities worked to achieve specific objectives (see Qualifications).

Value-added: This section describes a fish tagging training session and the Lamprey Technical Working Group recommendations. How did coordination affect the outcomes in these two instances? Did coordination improve the training session in some way? Did meeting improve the lamprey recommendations over other forms of coordination?

The proposal says, “Provided testimony to Environmental Quality Commission on Oregon water quality standards.” What was the outcome? Was the testimony developed through coordination among CRITFC members, others?

4. Deliverables, Work Elements, Metrics, and Methods

Deliverables: The project has five deliverables. Each has a brief description of work to be performed, and each is related to an objective.

In terms of deliverables, when will the update be completed? What are some of the projected changes, improvements, differences? Can the audience be more clearly identified? The proposal says that the purpose is, “advising tribal policy makers and tribal staff on technical, scientific, funding and policy issues, facilitating participation by tribal staff.” Isn’t the audience broader? How will Wy-Kan-Ush-Mi Wa-Kish-Wit be presented to the intended audiences?

The proposal says, says the CRITFC Watershed Department has managed the PCSRF (Pacific Coast Salmon Recovery Funds), implemented 153 projects. What has been the outcome of these projects in terms of achieving Fish and Wildlife Program goals and objectives? Should there be coordination between PCSRF projects and projects in the 2008 Columbia Basin Fish Accords Memorandum of Agreement? Could coordination between these two programs use resources more efficiently and more effectively?

Work Elements: The proposed work elements are listed without added detail: 99. Outreach and Education, 115. Produce Inventory or Assessment, 122. Provide Technical Review, 189. Coordination-Columbia Basinwide, and 191. Watershed Coordination. Only 99 has metrics, but they are more inputs rather than outcomes. The description of work is a list of coordination activities but is without discussion of outcomes, monitoring, measurement, evaluation, or lessons learned.

Can output metrics and methods be identified to go with these work elements? Ideally, the hypothesis(es) developed in the proposal would be measured during the course of the coordination activities and results presented in the report on this project. There are many ideas discussed in the proposal that are amenable to this approach. Selecting a few of the most important questions, concerns, or hypotheses and monitoring them is recommended.

Measures and metrics: Descriptions of work performed are provided in several places in the proposal. There are no metrics for measuring effectiveness, and no mechanism to get stakeholder feedback. There is a reasonable description of value added by the project to the tribes and to the region. No indicators such as the trend in number of projects, total dollars, or partnerships in tribally directed projects are offered.

In the current proposal, consider how will ensuring “implementation of Wy-Kan-Ush-Mi Wa-Kish-Wit principles and objectives in projects” be evaluated? What methods work best at communicating Wy-Kan-Ush-Mi Wa-Kish-Wit principles? How do monitoring, coordinating, updating, participating, commenting inform people about Wy-Kan-Ush-Mi Wa-Kish-Wit principles? What activities work best? What audiences are most important? How well do these audiences understand Wy-Kan-Ush-Mi Wa-Kish-Wit principles?

4a. Specific comments on protocols and methods described in MonitoringMethods.org

The protocols for the work elements are published but do not provide adequate guidance on the methods and metrics. Guidance is available from ISRP (2007-14:2). The project sponsors can strengthen the science in proposals by developing methods and metrics for the most important project objectives.

Modified by Dal Marsters on 4/17/2012 2:37:01 PM.

CRITFC provided helpful answers to many ISRP comments. The response concerning outreach was well done. The list of over 150 completed or ongoing projects is impressive. However, no lists of technical reports or data resulting from these projects could be provided because of the "limited time frame." It is surprising that CRITFC does not routinely have this information available.



Better evaluation and documentation of the effectiveness of previous coordination efforts and project implementation in the form of feedback from the four Tribes and other agencies could help CRITFC to identify those activities that have been most effective and to prioritize future efforts. But overall, the response misses the point and does not address the ISRP's comments on the need for better self-evaluation and monitoring of CRITFC activities.



The statement: "It is impossible to clearly state what the most effective activities are" is disconcerting in a coordination project, and can only true if no attempts to evaluate effectiveness are made. Approval of projects by the CRITFC Commission does not constitute an evaluation. The sponsors need to take a more proactive approach to learn how to conduct an effectiveness evaluation and to conduct it. At present, effectiveness is asserted rather than documented. Responses #12 and 16 address some potential indicators of effectiveness, but these remain assertions rather than demonstrations of effectiveness.



If it is the case (response #12) that "Effectiveness may well be measured by the success of preserving the tribal institutional capacity and leadership to deliver on-the-ground projects, collaboration to make shared decisions with state and federal co-managers on key policy issues, participation in forums that shape future actions by BPA and other federal entities that oversee the operation of the hydrosystem, and education and outreach to build and sustain partnerships," the elements of this statement provide guidance as to the types of indicators that would be appropriate to assess performance.



Response 17 also addresses the effectiveness evaluation issue. Stating, "As already agreed to by the ISRP, monitoring of coordination effectiveness is difficult to evaluate quantitatively" is again missing the point. Although it is difficult, it is both desirable and possible. The point is that careful thought should be given to what effectiveness would look like and how it can be measured, then develop a plan to measure it and evaluate it. Agreeing to "document any incidences of overlap or redundancy with CRITFC and individual tribal projects if they occur as a measure of effectiveness" is not sufficient and does not address the central question of effectiveness.



The response provides no indication of a prioritized approach to planning. Planning is apparently entirely reactive to short-term priorities expressed by CRITFC members. Response 15 describes some of the elements of consideration in coordination but does not explain the process of prioritization.



The recommended qualification to funding is that the sponsors be required to develop an effectiveness evaluation plan.