A sound scientific proposal should respond to the six questions and related material at the beginning of the regional coordination section.

The proposal contains so much detail that it is difficult to review. Future proposals would be improved through more summary and synthesis of relevant information.

The proposal provides extensive insight into a scientific perspective on program coordination. A number of hypotheses are presented about the coordination process and its outcomes. The approach provides narrative findings for the experience gained by CBFWA. The insights provide compelling analysis for developing a sound scientific perspective on program coordination early in the evaluation process.

Proposal strengths:

• The proposal is fully documented; methods and accomplishments are exhaustively described.
• The limiting factors statement addresses large-scale issues that have the potential to limit the effectiveness of the project. This is rare among proposals.
• The proposal provides extensive insight into a scientific perspective on program coordination.
• Performance metrics have been identified and used to evaluate project effectiveness.

Weaknesses:

• So much detail is presented that it's difficult for the reviewer to track proposal content. The project is not only complex in itself it is also undergoing significant structural change.
• It is unclear where sturgeon or anadromous fish fit into CBFWA activities.
• It is sometimes difficult for external reviewers to assess the effectiveness of the project.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

The CBFWA proposal offers a detailed narrative review of the coordination history from 1989 to the present. It analyzes changes in coordination that have occurred and reasons for them. The proposal also raises a number of policy issues to be addressed by Bonneville and the Council.

Problem statement: The problem statement is overly long, but at its end a summary conclusion adequately states the problem the proposal is designed to address.

Objectives: The proposal is focused around seven objectives, but the implicit overarching objective of this proposal is to coordinate disparate regional coordination projects around subject-matter themes.

Each objective has several deliverables that include development and maintenance; communications; coordinating, implementing, and facilitating; collate and summarize; attend and participate.

Emerging limiting factors: The proposal identifies three limiting factors for effective regional coordination: 1) perception of fairness, 2) participation and buy-in, and 3) adequate funding for both facilitation and participation. The proposal aims to address recent changes in these limiting factors.

2. History: Accomplishments, Results, and Adaptive Management (ISRP Review of Results)

Financial performance and history: This section provides an adequate description of a financial history complicated by inconsistencies in reporting and budgeting dates and by a changing project structure.

Performance: An adequate short description is provided.

Major accomplishments: A detailed description of the project's major accomplishments in its former version – the Annual Work Plan. Because it has been coordinating activities since 1989, CBFWA has extensive coordination experience and the proposal lists many insights. Further, the proposal provides detailed discussion on why some members have left CBFWA coordination and facilitation services. The new project configuration will begin with this funding cycle. Metrics of performance are numbers of meeting attendees and qualitative evaluation of outcomes made possible by actions of the CBFWA in various fora. A stakeholder survey was also conducted.

Response to ISRP comments: A complete description of ISRP comments and CBFWA response in terms of developing tools to monitor impact is provided.

Adaptive management: A good description of changes in CBFWA focus and configuration in response to changing circumstances in the region.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (hatchery, RME, tagging)

The geographic interests of CBFWA encompass the entire Columbia Basin for both fish and wildlife. CBFWA's goal has been to include all sovereigns and action agencies in the coordination process. In addition, CBFWA encompass the Willamette/Lower Columbia, Middle Columbia, Upper Columbia, and Snake River Recovery areas.

Project relationships: The statement provides a history of the changing configuration of CBFWA and a long list of coordination, monitoring and other programs throughout the region with which it is coordinated.

Tailored questions, data: The proposal provides a long description of the Status of the Resource website and its function. Information provided in the presentation indicates that the project has added tribal data coordinators to the already participating agency representatives. The narrative analysis of the regional coordination problem is excellent and provides useful insights; more attention to identification of a scientific component to the proposal would help to plan for future success.

More findings like this one would be valuable, "These factors illustrate in high relief the Fish and Wildlife Program’s recognition that coordination efforts and funding should be focused through a set of functional activities that need coordination, and not necessarily on the basis of entities desiring coordination funding." This seems to represent a critical principle for organizing coordination activities. Another important set of coordination hypotheses, "solutions intended to increase coordinated efficiencies and effectiveness. This includes developing coordinated synthesis reports, sharing data and information through scientific papers and science/policy forums, holding regular workshops focused on specific species, methods, or geographic areas, and on several topics, the drafting of basin-wide management plans." In this same section, "CBFWA Members recognized the role the organization can play in delivering useful technical, science-based products associated with protection, mitigation and enhancement of the Columbia Basin’s anadromous and resident fish, and wildlife."

The proposal suggests that “the adaptive management framework for which coordination” be used. “Adaptive management” is mention in 4 of the 7 project objectives, in many of the deliverables, and development of an “adaptive management framework” is frequently mentioned. Can this framework be more explicitly and specifically identified? How have the many lessons learned been built into each adaptive management cycle? What is the typical length of an adaptive management cycle? In the adaptive management section a very interesting process is described that suggests that funding arrangements changed the need and approaches to collaboration. This is a very interesting insight. It does not illuminate the adaptive management framework often discussed, but it does indicate that funding is an important driver as to participation in coordination activities. The identification of factors that may limit the effectiveness regional coordination including perception of fairness, participation and buy-in, and adequate funding for both facilitation and participation, is an insightful and useful hypothesis. Did the conduct of a consumer satisfaction survey in 2010 help in assessing these variables?

4. Deliverables, Work Elements, Metrics, and Methods

Deliverables: The project has 24 deliverables related directly to the seven objectives. Each is described in detail.

Project components: The project has eight components, each described in some detail: Data Management (5%); Monitoring and evaluation (10%); Developing and tracking biological objectives (5%); Review of technical documents and processes (10%); Project proposal review (5%); Coordination of projects, programs, and funding sources within subbasins 20%); Facilitating and participating in focus workgroups on Program Issue (25%); and Information dissemination (20%).

Work elements: CBFWA lists many work elements (11).

Methods and metrics: Methods are described in detail in several different sections. Metrics are also described. Measurement of performance is through numbers at meetings, outcomes of coordination, and a survey of stakeholder satisfaction.

One form of assessing effectiveness is the output of meeting results, documents, and other evidence of outcomes of coordination and facilitation actions. Another way to assess effectiveness is input from the state, federal, and tribal agencies involved in the process, who are well-positioned to assess this effectiveness. Other entities interacting with the program but not formally part of the CBFWA functions are also able to provide input. Some possible approaches to at least showing the degree of success would be, as a minimum, letters from each agency/tribe responding specifically to a series of questions as to how well the CBFWF program is meeting their needs in key areas and how the program might be improved. This request could also be addressed to some outside entities that participate with the workgroups. Some questions should address not only how well the CBFWA is meeting agency and tribal needs, but benefiting the salmon and other basin resources in specific ways that otherwise would not occur. It would also be of interest to know how the program involves entities such as the Oregon and Washington state agencies and the Corps of Engineers, and if more coordination among them and CBFWA entities is possible or can be expected.

4a. Specific comments on protocols and methods described in MonitoringMethods.org

The protocols for the 11 work elements are published but do not provide adequate guidance on the methods and metrics. The project sponsors can strengthen the science in proposals by developing methods and metrics for the most important project objectives. The relative value of “electronic meetings” vs. “face-to-face sessions” would be useful to study. Another worthy topic for review is dimensions of “facilitation.”

What part of “the ISRP for reporting metrics for regional coordination (ISRP 2007-14)” will be implemented? The document suggests (ISRP 2007-14:4), “Metrics of Impact: (e.g., how effective is the project: what is its added value of the coordination project) changes in behavior, value to the members, user evaluation of product utility, lack of redundancy, member assessment of effectiveness and impact, benefits to fish and wildlife of enhanced coordination activities, specific projects or resources benefited by the project, specific effect of coordination on conservation and management.” Where in the proposal are these suggested metrics of impact operationalized? A hypothesis worth testing is whether change in funding has led to decreased regional coordination.

Modified by Dal Marsters on 4/13/2012 12:24:40 PM.

The response includes a detailed description of the types of coordination and facilitation services that CBFWA is or could be providing. It adds information that was missing from the proposal regarding the operational meaning of general coordination terms. The response states that without CBFWA, the BPA, NPCC and the ISRP would find it difficult to staff activities such as holding meetings and providing website services. In addition, the response states that the "Columbia River Basin is dependent on the coordination, administration, and technical services that the CBFWA provides" for two monitoring and evaluation coordination partnerships (PNAMP and CSMEP). CBFWA activities in this regard include subcontracting services, participation in meetings, and website services. In 2005 CBFWA began to further expand its role to data inventory and reporting services. The response further states that the CBFWA role extends beyond coordination of its members to services for non-member entities.



Overall, a better demonstration is needed that CBFWA's services are provided in the most cost-effective manner. The response provides a better description of the association of the $900k budget line to the "annual report", including good detail on the range of products associated with the report. However, questions remain as to whether the costs are reasonable, especially given that a template of the website is already up and running.



The response also provides a description of the withdrawal of the Kalispel and Spokane tribes from membership. It appears that the interests of these two entities were not being addressed at the policy level; however, little explanation is provided as to why this situation exists. Does CBFWA have mechanisms to cope with "under-represented" groups?



The description of performance metrics is useful. As the sponsors indicate, existing performance metrics measure output (e.g. number of meetings, number of participants) but not impact (changes in behavior, value to the members). The table of number of meetings is interesting, particularly the very low number of PNAMP meetings (n=1) relative to other kinds of meeting such as "member meetings." However, evaluating performance on the basis of the number of meetings held, average number of participants, and reports produced is not, as the sponsors acknowledge, sufficient to assess impacts.



As recommended by ISRP, the sponsors conducted a literature review of metrics to assess coordination effectiveness. Review results were not provided but apparently were not considered applicable: "Results from coordination-oriented literature searches provide a broad set of techniques and metrics that are not consistent for coordination efforts, a situation that is comparable to differences that exist among monitoring and evaluation efforts for physical and biological projects."



Regardless of the range of approaches, the ISRP maintains that coordination efforts such as these can be evaluated. The response provides a vigorous defense of the need for the CBFWA, asserting that more coordination will result in better survival and recovery of fish and wildlife populations. However, no quantitative measures are developed for determining the degree to which this is the case. The Status of the Resource Project should provide useful information on key variables such as escapements, but the response does not give much information on project status or data QA/QC. Will Columbia River Basin fish and wildlife agencies rely on the Project for data or will the project duplicate agency data?



The recommended qualification to funding is that the project should develop an approach to monitor its impact in terms of changes in behavior and value to the members. In addition to the PISCES metrics, it would be useful to have CBFWA develop member-feedback instruments to evaluate member assessment of effectiveness and impact. In addition, the new cluster of products included under the Status of the Resource report provides an opportunity for user evaluation of product utility.