1. Objectives

The integrated report states that the single objective for both programs was to develop RME methods to assess status and trends, restoration opportunities, and the benefits of habitat restoration actions on salmon and steelhead and their habitat in the tributary environment. The larger spatial framework of these two programs is a critical context for evaluating whether they were scientifically sound. “The goal of ISEMP and CHaMP was to support the development of more realistic, spatially explicit, tributary habitat restoration scenarios and to quantify the degree to which these restoration actions might benefit salmonid populations in the interior Columbia River Basin." In addition, the integrated report explicitly states eight major questions the two programs were designed to answer. The objectives, methods, and analytical techniques are adequately described.

The two projects were terminated in late 2017 and early 2018.

2. Methods

The methods used in the ISEMP and CHaMP programs are thoroughly documented, and the ISRP evaluated the projects’ scientific soundness in multiple review processes from their beginning through 2013.¹ In addition, the ISAB and ISRP evaluated the projects’ contributions toward the Program’s critical uncertainties in 2015-2016 (ISAB/ISRP 2016-1). The ISRP found the projects’ methods generally to be scientifically sound and offered recommendations for improvements. Methods and results are also published in top peer-reviewed journals in fisheries, geomorphology, and ecology, and thus the methods underwent peer review by experts in those fields.

A review of the utility of CHaMP measurements and their protocols was contracted by BPA and completed by Wildlands Hydrology. Neither the ISRP nor ISAB have reviewed Wildland Hydrology’s assessment, which raised concerns, particularly about channel bedform classifications and identification of bankfull channel. In response to questions during their public presentation, leaders of Wildland Hydrology’s assessment team indicated that the CHaMP measurements of several of the major factors in habitat quality in the Columbia River Basin (e.g., riparian vegetation, large wood, temperature, and sediment particle size) are scientifically sound.

One of the fundamental characteristics of the ISEMP and CHaMP programs that differentiates them from AEM is the spatial and temporal scales for which they were designed. These two programs are designed to "assess status and trends, restoration opportunities, and the benefits of habitat restoration actions on salmon and steelhead and their habitat in the tributary environment." They are designed to identify these properties at the scale of entire watersheds or basins. AEM is designed primarily to assess the effectiveness of specific types of restoration actions at the site and project scale rather than entire watersheds or basins. Measurements in these different programs have the potential to be complementary rather than mutually exclusive if the necessary cross walks and data consistency for the individual site data are developed.

3. Results

The ISEMP/CHaMP project broke new ground at the cutting edge of (a) measuring and modeling stream habitat across a wide range of spatial scales, and (b) relating this to salmonid abundance and production. The integrated ISEMP/CHaMP report provided summaries of three intensively monitored watersheds, 10 studies of fish and habitat status and trends, 23 tools for restoration assessment and planning, 6 life-cycle models, and 4 data support projects. CHaMP sampled more than 800 sites in 8 years, which will provide critical information and a foundation for monitoring for years to come. Data/models and source code are readily available through cloud-based data warehouses and GitHub sites. Unlike most monitoring of local reaches, the data and analysis from ISEMP/CHaMP can be extrapolated from study reaches to the Basin. It is not an overstatement that many of the methods and results will be useful for decades to come among researchers and managers in the Columbia River Basin and worldwide.

Of the 46 subprojects summarized in the report, several have contributed substantially to our understanding of status and trends in major basins, design of future monitoring, and decision making in the Fish and Wildlife Program. We highlight several of these to illustrate the value of the results from these two programs:

• ISEMP/CHaMP developed methods to estimate juvenile salmon and steelhead density throughout a river network based on measures of the primary production of benthic algae, the ultimate source of food for invertebrates that feed fish communities. About half of the variation in fish abundance is explained by these algae growing on the streambed. In turn, gross primary production (GPP) can be predicted from models of stream temperature, conductivity, and solar energy, allowing researchers to calculate estimates of fish abundance from maps of these three physical variables.

• ISEMP/CHaMP adapted the Barker Model to analyzing mark-recapture data where fish are marked at one place and time, such as with PIT tags, but then captured (or detected by PIT-tag antennas) at distant locations over a wide range of times. This analytical technique is a key to measuring effects of habitat restoration on migratory fish like salmon and steelhead, which have smolts that emigrate. Without it, analysis of the abundance of fish with these complex life cycles is very difficult. The model will improve estimates of salmonid survival as they migrate through the river network and hydrosystem

• To relate fish abundance to habitat, Quantile Regression Forest models were developed. These models can address the thorny problem that habitat is not always the sole limiting factor for fish abundance and so data are often highly variable. This statistical method can deal with this problem.

• ISEMP/CHaMP created six life cycle models, which represent an extremely important tool for future analyses and decision making throughout the Basin. In recent reviews, the ISAB concluded these life cycle models provide holistic analyses of potential limiting factors over the full life cycle and geographic range of anadromous salmon and steelhead (ISAB 2017-1, ISAB 2018-1). These models will assist landscape-level assessment of status and trends by the Fish and Wildlife Program in the future and will greatly assist decision makers in the region.

• The Intensively Monitored Watersheds revealed several major findings, illustrating the value in this approach. The studies revealed both positive responses to restoration actions in one basin and no response in another basin. The balanced and unbiased analysis of the research data should be a model for other projects, because sometimes the desire or commitment to provide benefits to fish populations and habitats colors the interpretation of project results.

• The status and trends subprojects provided a scientifically strong foundation for analysis of future trends. As examples, the temperature subproject and the survival estimation subprojects identified critical issues about habitat conditions and population status while providing valuable analytical tools. The temperature estimation approach incorporated riparian condition, floodplain conditions, and discharge, which are significant predictors and elements of many restoration actions within the Basin. The visualization techniques for temperature will contribute to future temperature studies in the Northwest.

• ISEMP and CHaMP developed more than 23 restoration tools to assist restoration practitioners and assist in the analysis of ISEMP and CHaMP data. Several of these have been used by other projects and have substantially contributed to the synthesis of ISEMP and CHaMP data. Examples of major contributions are the Quantile Regression Forest models for fish habitat relationships, geomorphic network analysis tool box, riparian condition assessment tool, wood recruitment assessment tool, and river classification and geomorphic condition tool. These analytical tools will continue to assist and inform restoration planning and design.

Many of the ISEMP projects created GIS mapping tools to estimate variables needed to drive other models. For example, abundance and productivity of salmonids are different in valleys of different types (i.e., wide valleys with low gradient where the stream meanders vs. narrow and steep valleys where the stream is mainly cascades). To address this relationship in life-cycle models, they developed a tool to measure valley bottom width from digital maps using GIS.

A key finding from these projects is the need to design monitoring studies of restoration actions at appropriate scales of space and time for expected resource outcomes. Projects were able to detect local improvements but often failed to detect population-wide responses (e.g., Entiat IMW). Not unexpectedly, population metrics suffer from spatial mis-matches (e.g., population spatial scale is much larger than restoration spatial scale), from high variability in responses, from influences by many factors over and above restoration, and a longer-time lag for restoration actions to become effective. Detection of population scale responses to local restoration actions likely will need much longer time scales than are typically included in most monitoring programs.

The tests of the effects of habitat restoration carried out and published by the ISEMP/CHaMP program are among the best conducted to date, worldwide. An example is the research on the effects of Beaver-Dam analogs on steelhead in the Bridge Creek watershed of the John DayRiver (Bouwes et al. 2016²). As a result, this approach that requires less heavy equipment and site disturbance is being used increasingly in other programs throughout the Basin.

The program also broke new ground in determining how to measure habitat across a wide range of spatial scales from local sites to whole sub-watersheds, and where best to focus sampling efforts in such a daunting task. For example, a key result was that measuring more sites at intermediate intervals (e.g., 3 years) is more important than measuring fewer sites every year, because variation in habitat is larger across sites than among years within a given site.

Another key finding was that some of the most striking improvements in habitat and fish numbers have been achieved from lower cost projects that focus on natural-processes, such as small-scale structures and beaver input, not just from larger and more expensive projects.

Key project findings were shared widely in the refereed literature. The list of 53 peer-reviewed publications from the project, many of which appeared just in the last few years, is very impressive.

Lessons Learned and Areas for Improvement
The CHaMP protocol has been used by more than 10 state and Federal agencies and tribes to monitor fish habitat and restoration projects. Researchers and managers will continue to reap benefits from the tools developed and the papers published by the ISEMP/CHaMP program for decades to come. Despite the body of information developed by ISEMP and CHaMP, the lack of effective outreach and information sharing was a major weakness of the programs.

The programs did not provide frequent and timely syntheses of their measurements for use by regional projects and leaders of the Council and BPA. The programs were not successful in showing managers and policy makers why the results they were producing were useful. Examples of practical information for managers and decision makers in clear language free of jargon were needed. Instead, many of the reports were difficult to read and interpret, often because the writing was suitable for statisticians and GIS experts but not the intended audience. Most of the audience is lost by jargon, such as “site level sample inclusion probabilities,” “riparian vegetation departure products,” and “user-defined non-zero probability” in the Summary Report. Great results cannot be used if the audience who is supposed to use them cannot easily understand them.

One area that did not receive much attention in the project was the linkage of upslope/upstream processes and management to conditions in the downstream, valley bottom and fish bearing sections of watersheds. Although the need to provide tools or links for including these considerations was identified in past reviews, it does not appear to be included in the summary reports. Certainly, this would have added complexity to the work, but it is an important component for ultimately understanding, managing and restoring aquatic habitat and fish populations at watershed scale.

A final goal for ISEMP/CHaMP would be to show managers and decision makers results they can really use. The projects have ended, but NOAA Fisheries and collaborators are continuing to analyze, synthesize, and present their findings. It will be beneficial for the Fish and Wildlife Program to track and publicize when and where the findings and data are available into the future.

Design of Future Monitoring Programs
Although the projects are now closed, some observations looking back on the multi-year effort may be useful in designing and implementing future, large scale projects, particularly those that address similar research questions. The ISRP recommends that design of future monitoring should incorporate many of the 54 monitoring protocols and 800 CHaMP monitoring sites to take advantage of the 7-yr database for future trend analysis. The results of ISEMP and CHaMP can be used both for designing M&E projects and as prior long-term data for future monitoring sites. The investment in ISEMP and CHaMP has produced a substantial body of habitat data, methodological advances, analytical tools, and life cycle models, which provide a foundation on which to build future monitoring programs. This legacy should be carefully considered when developing the new tributary habitat RME strategy co-led by BPA, Council, and NOAA.

The analysis of measurement noise and overall variance for 54 different habitat measurements provides important insights for future monitoring. This analysis identifies the influence of different sources of variance for the habitat metrics used in the monitoring program. For example, most of the variance in estimates of coniferous riparian forest cover was related to either watershed or site components of variance, and variance in large wood fish cover was related primarily to site components. Interannual differences in their estimates were small, and they suggested greater information could be obtained by sampling a greater number of sites less frequently. They also found that crew-to-crew differences had little effect on the variance of their metrics. These results of the ISEMP/CHaMP programs should inform future design of monitoring programs in the Basin.

One of the most critical challenges for RME projects is providing information that can be directly applied to management issues throughout the life of the project, not just at the end. This requires agreement at the start of the project between the funders and the investigators about the degree of uncertainty that decision makers/management are willing to accept and the format for reporting. This also would require all users to understand that interim information be updated or revised in the future. More discussion and agreement on this issue between funders and project leaders at the outset of this long-term project would have been useful.

One of the innovative organizational aspects of the integrated final report is the concise and informative summaries of the individual sub-projects. The format provides a diagrammatic road map for the 46 sub-projects, graphical description of the relationships between the projects, concise nontechnical descriptions of the subprojects, concisely stated major findings and uses, contact information, and URL links to products, publications, web pages for tools or computer code, locations of data, and supporting files. The Council and BPA should consider using similar formats for other reports and products in the future.

4. 2017 Research Plan uncertainties validation

The summary report did not explicitly identify linkages to the 2017 Research Plan, but the 2017 Research Plan critical uncertainties database identifies direct or indirect linkages with 18 critical uncertainties for the ISEMP and CHaMP projects. The proponents did not directly crosswalk against the database. A quick review of the uncertainties database showed that most of them were covered by the projects except perhaps (a) resiliency to climate change, (b) additional habitat actions that are needed, and (c) estuary and plume actions and monitoring.

____________________________________

¹ ISRP 2013-2 includes a comprehensive history of ISRP and ISAB reviews of the projects beginning on page 5.

² Bouwes, N, N Weber, CE Jordan, WC Saunders, IA Tattam, C Volk, JM Wheaton and MM Pollock. 2016. Ecosystem experiment reveals benefits of natural and simulated beaver dams to a threatened population of steelhead (Oncorhynchus mykiss). Sci. Rep 6:28581. 10.1038/srep28581. http://www.ncbi.nlm.nih.gov/ pubmed/27373190. https://isemp.egnyte.com/dl/i3zz8KMhML.