This long-running, successful, and adaptive project is integral to several other projects (Restore Salmon Creek Anadromous Fish 199604200, Chief Joseph Hatchery Program 2003023, Okanogan Subbasin Habitat Program 200722400, Okanogan Habitat Acquisition and Restoration 200810200, Land and Water Acquisition 200810400, Upper Columbia Programmatic Habitat 201000100, Upper Columbia Spring Chinook and Steelhead Juvenile and Adult Abundance 201003400) in the upper Columbia River basin and most are conducted by the Confederated Tribes of the Colville Reservation (CTCR) . The project has four broad objectives; three are directed toward monitoring and evaluating annual changes in: 1) listed salmonid fish populations, 2) a suite of habitat variables, and 3) VSP parameters in summer steelhead. The fourth objective is to fill recognized data gaps in the status and trends of habitat variables. The proponents use EDT models specific to the Okanogan and Methow subbasins to address information needs.

One of the major strengths of the project is its data management system and publicly accessible dashboards for understanding status and trends of listed salmonids and habitat conditions in the subbasins of the Okanogan and Methow basins. The project is a major contributor to monitoring and landscape evaluation in the upper Columbia River basin.

While reviewing another upper Columbia project, the ISRP became aware of Yakama Action Effectiveness Monitoring (BPA # 201700300), which is a pilot project through 2022 being conducted by the Yakama Nation (YN). The project description in cbfish.org in part states:

“The contract will cover work involved in the creation of an Upper Columbia Habitat Action Effectiveness Monitoring Plan….The contract will include development of clearly defined goals and objectives for habitat action effectiveness monitoring, the selection of habitat action sites and available control sites, the development of at least one testable monitoring question and hypothesis for each project objective, and the selection of metrics and variable that will be used to measure fish response and test hypotheses.”

If UCHIP is doing habitat restoration in the Okanogan, Methow, Entiat and Wenatchee subbasins and OBMEP is providing monitoring and evaluation, the results of the YN AEM project would be of value and should be explored.

The ISRP’s recommended Conditions are listed below. The proponents need to assist with development of an M&E Matrix during the response loop (September 24 to November 22, 2021) and to provide information to address the other following Conditions in future annual reports and work plans.

1. Data sharing. Describe the collaborative sharing of data with other monitoring projects and the use of independent measurements to evaluate the quality of the monitoring data and identify areas that require improvement? The description of these collaborative efforts should indicate how they coordinate with geographically overlapping projects.
2. SMART implementation objectives. Provide quantitative characteristics and inherent time frames for tasks in the project implementation objectives.
3. M&E matrix – support. One of the challenges for ISRP reviewers is understanding the specific monitoring that is being conducted for multiple implementation projects. As habitat projects and monitoring projects are not presented as part of an integrated proposal or plan, the need for a crosswalk to identify the linkages between implementation and monitoring is extremely important for basins or geographic areas. The ISRP is requesting a response from the Upper Columbia River Programmatic Habitat Project (201000100) to summarize the linkages between implementation and monitoring projects in the Wenatchee, Entiat, Methow, and Okanogan basins. During the response loop, as a key M&E project and partner in the basin, we ask your project to assist them in creating the summary and provide information to them about what, where, and when your monitoring occurs and what is being monitored for and shared with implementation projects in the basin. A map or maps of locations of monitoring actions would be helpful in this regard.

The ISRP suggests that future proposals should be edited to reduce length and redundancy.

Q1: Clearly defined objectives and outcomes

The overarching goals of this long-running monitoring and evaluation project are to continue to monitor the status and trends of habitat and biological parameters in the Okanogan and Methow subbasins and provide that information to other projects (e.g., OSHIP 200722400). Other projects use the monitoring information to prioritize and evaluate habitat restoration actions.

As the ISRP found for other monitoring projects, some of the objectives stated in the proposal tend toward SMART objectives (i.e., 1, 2, & 3; pages 56-57), but lack quantification such as stating how many reaches of each subbasin will be monitored each year. Other objectives (e.g., 4 through 7; pages 57-59) are actions that the proponents will take to meet their objectives. The proposal does not include quantitative objectives for the biological performance of Chinook and steelhead in these basins, which are the objectives of the CTCR, Upper Columbia Salmon Recovery Board, and WDFW. The objectives stated in this project, therefore, are implementation objectives related to obtaining and analyzing the data. As such, the objectives identify the required actions but do not provide quantitative measures related to the implementation of actions. The objective statements could be improved by including the necessary or desired statistical requirements for the analyses (e.g., variance and error requirements, model fitting requirements, model performance) or the operational performance requirements of the detection sites (e.g., intended proportion of the sampling period when detection sites are operating, tagging rates, detection efficiencies for PIT tag arrays and smolt traps), which could be used to evaluate the performance of the project. Several of these performance measures are reported in the Methods section or Progress to Date section and could be incorporated explicitly into the project objectives.

Q2: Methods

The methods are well described and have been refined or changed when needed to meet project objectives. One of the strengths of the project has been the proponents’ practice to overlap new methods with the procedures they are replacing, often for multiple years to ensure that the new procedures meet expectations. From its inception, the project has relied upon the EDT model as an analytical tool to integrate its data, and it has successfully used the model to identify habitat constraints on summer-run steelhead. As the OBMEP project has proceeded, its empirical data have been used to adjust some species-habitat rules of the Okanogan EDT model. The project’s methods are scientifically sound and often innovative. Project information is quickly made available for project review, use by other projects and management agencies, and recovery programs.

The proposal does not indicate how coordination with overlapping projects in the same geographic areas occurs, such as the Upper Columbia Spring Chinook and Steelhead Monitoring Project of WDFW (201003400). Do these projects share independently generated data? Do they incorporate collective data into their modeling and data analyses? Do they compare estimates for the same reaches or sub basins developed by different monitoring entities? Such collaboration and cross comparison will be extremely important for assessing information on salmon and steelhead in the upper Columbia River Basin.

The proponents identify climate change and changing regulatory conditions as potentially confounding factors. The CTCR and their EDT consultants created scenarios of future climate change. It is not clear whether they will consider these and other future climate scenarios in their analyses, and the proposal did not indicate how such projections factor into land management, hatchery operations, and Tribal decisions. Are there major examples where future climate conditions depicted by OBMEP and the EDT analysis have altered and improved the prioritization, design, and implementation of habitat or hatchery projects?

The proponents criticized the CHaMP and ISEMP monitoring programs in several sections of the proposal and indicated that the review by Rosgen et al. (2018) “highlighted numerous concerns including the appropriateness of GRTS random site selection and ‘upscaling’ site level indicators to the watershed level.” In his presentation to the Council, Dave Rosgen noted that the measurements for temperature, large wood, sediment, and riparian vegetation in the CHaMP protocol were scientifically sound. In addition, there are valid statistical justifications for the GRTS random site selection approach for habitat monitoring at the scale of the Columbia River Basin. The ISRP mentions this to clarify the weaknesses and strengths of the CHaMP and ISEMP programs and ensure that the data from those efforts are preserved and used. The ISRP appreciates the experience and participation of the CTCR in developing future regional habitat monitoring.

Q3: Provisions for M&E

The project has a well-developed adaptive management process. The project uses an iterative review process to ensure that the EDT model’s environmental attribute ratings are credible. If they do not, changes are made on how information is gathered. They found, for example, that the field-based methods used to conduct riparian surveys were not obtaining the data needed to parameterize this attribute in their EDT model. A new approach that uses high resolution images and LiDAR plus field verification is now being used to collect this information. The proposal describes how the OBMEP works with its management hierarchy (Fish and Wildlife Director, Anadromous Division Lead, a Senior Research Scientist, and Subdivision Leads). It does not, however, clearly state how decisions are documented for future reference. It is assumed that adaptive evaluation is recorded in annual reports, and the ISRP encourages the proponents to state this portion of the process clearly. We commend the OBMEP for convening an annual adult steelhead project meeting for decision-making. Does it have a similar process for its Chinook salmon measurements and analyses? Given the large number of BPA-funded projects of the CTCR, would it be appropriate to have an annual science meeting of the CTCR projects to present, discuss, and evaluate outcomes and make future management decisions or does the Upper Columbia Science Conference serve this function?

One critical aspect that requires additional information is how the OBMEP monitors specific outcomes for habitat projects (OSHIP and Restore Salmon Creek) and the land acquisition projects (Land & Water Acquisition and Okanogan Habitat Acquisition and Restoration). These other projects point to OBMEP as one of their primary sources of project monitoring information. The ISRP needs a collective description of the specific monitoring information that is provided to habitat restoration projects, land acquisition projects, and hatchery projects. This comprehensive monitoring documentation should provide a matrix of project-specific monitoring functions and crosswalk of monitoring methods across different projects. Because OBMEP is the primary monitoring project of the CTCR, it is appropriate that this project and all other relevant monitoring, habitat, and hatchery projects in the upper Columbia River basin assist such an effort with the UCSRB as lead.

Q4: Results – benefits to fish and wildlife

The proposal provides a comprehensive overview of the project and how outputs from its EDT model have been used to identify priority habitat restoration opportunities. The project’s data on juvenile and adult abundance and distribution patterns along with its measurements of the cumulative effects of the habitat restoration actions that have occurred in the Okanogan Subbasin allow it to track changes in VSP parameters. This type of feedback is critical for fish recovery scientists and habitat restoration practitioners. The proponents’ data management system and publicly accessible database provide excellent analyses of status and trends of listed salmon and steelhead in relationship to habitat conditions. In particular, the ISRP commends the project for its rapid synthesis and public availability of landscape level data.

Qualification: The proponent should incorporate suggestions generated during the upcoming 2011 CHaMP/ISEMP workshop.

The proponents’ response clarifies that the OBMEP Colville Tribes Fish Accord project will continue much as it has since 2005 with some modification for consistency with the CHaMP protocols. A strength of this proposal is the plan to collect data for a period of time using historical protocols and the new CHaMP protocols simultaneously. This procedure will provide a comparison of results using the two protocols and enable modification of the historical data, if required, to make it compatible with data collected using the new methods.

The response made it clear that OBMEP will not be responsible for analyses of the data they collect; their objective is to collect the information and pass it on to the organizations (CHaMP, ISEMP) that will conduct the analyses. Therefore, the ISRP suggestion in the review of the initial proposal that hypotheses to be tested be clearly stated may not be appropriate for the OBEMP project. Rather development of these objectives should be the responsibility of the CHaMP and ISEMP scientists. The ISRP has proposed a workshop be held in 2011 for CHaMP/ISEMP collaborators to clarify objectives and roles. OBEMP should participate in this meeting and adjust their project accordingly.

The response from the proponents clarified the relationship between OBEMP and CHaMP and linkages with other monitoring programs in the basin. The proponents also provided some useful diagrams depicting the complex scheme of the UCR adaptive management program and OBEMP’s role in this scheme. The relationship between OBMEP and CHaMP is still developing, so not all methodological decisions have been made. However, it is clear that OBMEP will serve as one provider of raw data to CHaMP and ISEMP with PNAMP providing data management and analytical tool support. OBMEP data will be linked to CHaMP data through the STEM database (data entry is a two-year process). The OBEMP data also will be used to improve the EDT model. It is envisioned that this consistent, collaborative effort will provide answers to key questions of interest in the region.

A coordinated monitoring program for habitat and fish has been needed in the Columbia Basin for a long time. The proposal for implementing the Okanogan component of this program is headed in the right direction but some additional information is required to provide a thorough technical review. In particular, the following should be addressed: 1. the relationship between OBEMP and CHaMP 2. linkages with other monitoring programs in the basin 3. data analysis techniques and who will be responsible for this task 4. the process to be used for linking historical data and CHaMP data 5. a better description of the adaptive management program, and 6. use of these data to improve analytical tools, like EDT, also should be a key objective of this project 1. Purpose, Significance to Regional Programs, Technical Background, and Objectives A more complete understanding of the current condition of habitat and fish populations and how they respond to restoration measures is critical to the development of an effective salmon recovery program for the Columbia Basin. This project proposes to modify an existing monitoring program in the Okanogan Subbasin to make it compatible with a basinwide effort to coordinate habitat and fish monitoring programs (CHaMP), a very worthwhile goal. More consistency among monitoring programs in the basin should help provide a much more comprehensive picture of the status and trends in habitat and fish populations and accelerate the accumulation of information regarding the effectiveness of various restoration techniques. The current project in the Okanogan Basin has monitored anadromous fish at the population scale over the last five years. This proposal hopes to link these population data to habitat restoration actions. The enhanced monitoring program envisioned by this proposal (OBMEP) would continue to monitor key components of juvenile fish production, habitat condition, water quality, and adult abundance. However, the habitat parameters being monitored would be considerably expanded and methods would be modified to be consistent with those specified by CHaMP. The modification of the methodologies that have been used previously raises concerns about (1) compatibility of data from previous years and new data, and (2) usefulness of new data when using old models (e.g., EDT). These are proper concerns, but consistency of data collected across the basin is also important. The technical background for this project was satisfactory. However, objectives for this project were somewhat incomplete. The overarching objective for this project is to continue to collect data to assess progress towards recovery goals – certainly an appropriate objective. But the more specific objectives were often not very informative. For example, rather than simply stating that an objective is to participate in a basinwide monitoring program, specify the hypotheses or questions that the collected data will be used to address and indicate how answers to these questions will influence management decisions. The ultimate goal of any of the monitoring plans in the basin is to develop more effective restoration methods. Thus, project objectives should always clearly link back to this goal. 2. History: Accomplishments, Results, and Adaptive Management The history, accomplishments and results of the project to date are not presented in the proposal, but links are provided to numerous reports that do detail some aspects of the work that has occurred since 2005. Summaries of some preliminary data also are presented on their web site. In general, the data set is too short at the present time to reach many conclusions, and data analysis has not proceeded very far. The relevance of this work to management of habitat and fish in the Okanogan Subbasin is less clearly presented. Although there was an outline of an adaptive management program provided, much more detail is required in order to review this aspect of the monitoring program. A detailed discussion about adaptive management associated with this project, both historically (i.e., how have results to date been used to inform management decisions) and in the future, as the project integrates with CHaMP should be included in the proposal. This discussion should specifically focus on how changes in the understanding of habitat effects on fish population dynamics will be incorporated into prioritization of restoration projects or decisions of fisheries management. 3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (Hatchery, RME, Tagging) The relationship between this project, ISEMP, and CHaMP is discussed briefly in the proposal, but insufficient detail is provided to judge the degree of coordination among these efforts. The OBEMP program will become a component of the basinwide CHaMP program under this proposal. The role of OBEMP and CHaMP in data analysis, producing reports and communicating results should be clearly defined in the proposal. It appears as if the primary tool that will be used for linking habitat results to fish population response is a new version of EDT. These EDT runs will then be used to update restoration plans. Use of this model has plenty of precedent in the Columbia Basin, and it is certainly a legitimate method for conducting this sort of analysis. However, this project will collect empirical information on both habitat conditions and fish population performance. These data could be used to evaluate some of the assumptions about fish-habitat relationships that are the foundation of the EDT model. In fact, using field data to test and then modify the EDT relationships represents an important adaptive management linkage, if this model is to be used to inform fish and habitat management decisions in the subbasin. It may be more appropriate for this type of evaluation to be done with a more integrated data set (the full CHaMP data or some combination of CHaMP and ISEMP data?). Some discussion of this issue should be incorporated into the proposal. 4. Deliverables, Work Elements, Metrics, and Methods The Work Elements, Metrics, and Methods were primarily addressed by providing lists of the parameters that will be measured coupled with references to documents that detail the actual sampling protocols (e.g., ISEMP sampling methods). Referencing external documents for methods is certainly an efficient manner of presenting this information and the methods cited are generally very appropriate. There were several items, however, that require some clarification. The rationale for number and type of sample locations was not clearly presented. The use of GRTS for sample site selection is a widely accepted method. But it was unclear in the proposal whether all 50 sample sites would be selected by this method or if 25 new sites would be selected each year using GRTS to augment 25 sites that were included in the monitoring initiated in 2005. Has any analysis been done to evaluate whether 50 sites are sufficient to adequately capture trends in habitat condition in the Okanogan Subbasin? What is the rationale behind sampling 25 new sites each year and 25 old sites sampled every 5 years in a rotating panel? Some additional explanation of this aspect of the project design should be included in the proposal. A primary concern with this section of the proposal is a lack of specificity about how past monitoring protocols will be coordinated with the new methods used in CHaMP. The proposal does state that there are three precautions that will be followed in modifying sampling protocols, one of which is “entirely new data or incompatible data is collected in addition to new data being collected.” This statement is unclear but seems to imply that data for certain parameters will be collected using both the new CHaMP protocols and the methods that have been used since 2005. If this interpretation is correct, it implies that data collected using these two protocols will be used to determine how comparable the results are and to develop a method for converting the historical data, if required. The extra effort necessitated by the change in methods is unfortunate, but collecting using both the old and new method is necessary to ensure that data collected under the old sampling regime and that collected under CHaMP are compatible. But no mention is made regarding such a comparison of data collected under the old and new protocols. More detail about the variables that will be impacted by changes in methods and some discussion as to how this issue will be addressed should be included in the proposal. The proposal also would have benefited from some additional information about the location where fish population data will be collected. It was not clear where the smolt trap is located or where redd surveys will be conducted. Inclusion of a map indicating location would have been helpful. There was very little information provided regarding the methods that will be used to analyze the data or who will be responsible for this task. Will OBEMP conduct the analyses or will analytical responsibility fall to CHaMP? Regardless of who conducts the analyses, some description of how this would be accomplished should be in the proposal. Assessing trends in habitat conditions is relatively straightforward. But the methods that will be used to analyze some of the fish data were less clear. For example, it appears as if adult salmon abundance will be assessed using three different methods: redd counts, adult enumeration, and underwater video. How will these three data sources be used in developing an estimate of spawner abundance? Also some discussion of the process by which habitat and fish population data will be related should be included. The proposal does indicate that EDT will be one of the tools used for this purpose, but as indicated above, these data could be used to progressively improve models like EDT. Developing better assessment tools should be one of the key objectives of a program like this.

This continues to be a fine example of a monitoring project, which the ISRP supports wholeheartedly.

There is a good description of the ongoing program, experimental design protocols, etc. The project was begun in 2004 with EMAP site selection, development of protocols, etc. There is a nice report of what was done, faulted only by not giving a summary of results. The proposal clearly places the work in the regional monitoring framework. This program is an important part of implementing the subbasin plan. There are excellent details on other related projects in the area. This project is providing M&E for a number of related BPA projects.

The M&E objectives are clearly explained and methods are clearly outlined and stated, with references to the standard protocols. One technical caution: The proposal claims, "The health of a stream can be determined from the species of macroinvertebrates present." It goes on to say that "Benthic macroinvertebrate samples will be collected annually from each of the EMAP sites." Consideration needs to be given to the time of year when those samples might be taken. Life cycles of many aquatic invertebrates remove them from the stream environment seasonally, and/or render them difficult to sample at other times. There is no discussion of this point and its effects on the sampling protocol.

Only a brief narrative is given on facilities. Personnel are excellent. There are specific information transfer work elements (coordination, outreach). The proposal emphasizes this aspect as a major part of its effort.

This project is a critical link to evaluate the management efforts in the Okanogan basin. Benefits are expected to accrue in time as information gathered accumulates and is interpreted and acted upon. The thorough monitoring system will undoubtedly benefit the focal species in the long run, depending on actions taken to correct any problems.