The ISRP recommends that the project "Does Not Meet Scientific Review Criteria” and suggests that the project be fully restructured as necessary to raise the scientific foundation of the project to an acceptable level. Based on the original project proposal, revised proposal, and responses provided to the ISRP’s preliminary review, we find that the monitoring program is not scientifically adequate for either evaluating effectiveness of restoration actions or for determining the status and trends of Chinook salmon, steelhead, or bull trout populations. Fundamentally, many details of the monitoring program are unclear, and at times contradictory.

The ISRP concludes that a major proposal revision and resubmittal are necessary for this project to meet scientific review criteria, and the ISRP should review the revised proposal. The revised proposal should address the issues identified in our preliminary review, response request, and the comments that follow.

The ISRP is particularly concerned that the issues raised in this review are essentially the same as questions and conditions/qualifications raised in previous ISRP reviews of the project. In the 2018 Research Status Review (ISRP 2018-8), the ISRP evaluated the project’s progress and concluded that it was underperforming. At that time, the ISRP felt that the project needed to be re-examined because of the extensive changes to the project and the future uncertainty of three basinwide monitoring programs (i.e., CHaMP, AEM, and ISEMP). In that review, the ISRP recommended that the project should redefine and update monitoring questions, quantitative objectives, and realistic timelines. Further, the review recommended that the proponents develop “an explicit project framework that describes the research/monitoring path for future activities.” The ISRP also recommended that “the proponents need to re-group and seek sustained (multi-year) scientific mentorship, both internally and externally,” suggesting that “scientific mentorship perhaps could be provided by CRITFC or independent advisors with experience on similar projects.” Additionally, in the 2010 Review and 2012 ISRP Follow-up, the ISRP asked that the proponents provide a summary of data collected, results, and program modifications after three years, which would have been due in 2015. To the ISRP’s knowledge, this progress report was not submitted.

The ISRP has major concerns about the project’s scientific rigor. Because the monitoring activities conducted by this project are essential to several other projects, the ISRP recommends that this project submit a fully revised proposal, preferably by December 31, 2022. As suggested in the 2018 Research Status Review, the project should obtain substantial assistance, both internally and externally, from either CRITFC or other external advisors with experience in the design of monitoring programs and data analysis.

In our preliminary review, we requested additional information and clarifications on the eight topics listed below. The CTUIR responded to some – but not all – of the eight topics. Two requests for responses — 7. Guidance on Monitoring Methods, Design, and Analysis and 8. Confounding Factors — are not addressed by the proponents in their response or in the revised proposal. Additionally, the proponents’ other responses do not directly address our specific queries and much of the longer explanations are redundant with responses provided earlier in the document. The proponents did, however, make an effort to respond to the ISRP’s request to support the development of an M&E matrix, which is greatly appreciated. Our final comments below are based on responses received from the proponents, as well as an examination of the revised proposal:

1. M&E Matrix – Support. The proponents added a matrix (Table 1) to the revised proposal giving locations of monitoring actions with links to specific restoration projects. They pointed out that they require additional time (beyond the Response Loop deadline) because their project is not limited to the Grande Ronde Basin and they are working with several other groups, and the ISRP understands this time limitation. As the Fish and Wildlife Program develops summaries of monitoring and evaluation for the geographic areas, the ISRP encourages the CTUIR to contribute their information and expertise, which will benefit projects monitored under this program, as well as other projects within those subbasins. See the ISRP’s programmatic comment on habitat restoration M&E in this report.

2. SMART Objectives. While the proponents agree that SMART objectives are needed, they did not actually provide them. Rather, they indicated that they would be forthcoming in the future, and then gave an example of SMART objectives for recruiting college students. The objectives in the revised proposal are marginal improvements, but generally do not meet criteria for SMART objectives. While some components of SMART objectives are included, some specific components of the new “SMART” objectives are insufficient.

Importantly, the objectives do not include quantifiable expected outcomes and do not identify the metrics to be used in measuring outcomes. The proponents also misinterpreted our use of “Time-bound.” We intended them to identify the timeline in which the expected benefits will occur. Also, note that the term “evaluate” should be replaced by “quantify or measure” in the new objectives.

Examples of SMART objectives are provided on page 5 and Figure 2 in the Proposal Form Template. The ISRP also encourages the proponents to contact us to discuss developing their SMART objectives.

3. Results of Past Monitoring. The responses under “Overall Conclusion” are tangential to the ISRP request to describe clear and complete results of past monitoring. We expected the response to include summaries of key findings and the data to support those findings.

4. Benefits for Fish and Wildlife. The ISRP requested a summary of the project’s benefits to fish and wildlife that is scientifically rigorous, explicit, unambiguous, and quantitative. Further, where benefits have not been realized, the ISRP requested that the proponents identify specific adjustments and changes in the program to address problems in achieving desired outcomes. The proponents’ responses describe methods to be employed, what they hope the outcomes for fish and wildlife might be, and a strategy for implementation. In the revised proposal, the proponents should explain how the results of their project will be used to determine the benefits to fish and wildlife, including how their project will quantify benefits to fish and wildlife, what their project has already quantified, and what other projects will use their information to determine benefits.

5. Project Evaluation and Adjustment. The ISRP requested a thorough description of the project’s evaluation and adaptive adjustment process, both within the project and how the proponents communicate with projects receiving their monitoring results (for instance, their input on the monitoring study designs, mechanisms for regular data and information transfer). The ISRP expects a full description that includes information on known decision points, how data are used to support evaluation of this project in meetings its objectives, explicit schedules for evaluation and decision-making, and documentation of decisions and project changes. The CTUIR responses do not address these issues. Further, the description of Adaptive Management does not provide adequate details for evaluation. The proponents did, however, add several useful flow charts (Figures 2, 3 and 4) to the revised proposal showing general programmatic linkages for 1) policy and management structure, 2) adaptive management pathways, and 3) oversight, planning and coordination.

The ISRP notes that the project evaluation and adjustment section appears to provide details for a different CTUIR project. The plan described in the response seems reasonable, but the updated proposal does not align with this plan or provide an indication that the project is actually following these steps. The plan requires quantitative objectives or hypotheses, reporting empirical data, establishing biological targets, and applying decision rules. How is CTUIR implementing these for the Biomonitoring project?

6. Monitoring Design. No response was provided by CTUIR and this section of the revised proposal remains largely unchanged from the original. The ISRP expected a description of the process for developing monitoring actions, including statistical design and analyses, modeling, and field and laboratory methods. As well, the ISRP expected to learn 1) how the project coordinates with projects that are implementing restoration actions as a way of improving monitoring methods, design, and analysis, and 2) how the project obtains comprehensive review and assistance of its monitoring efforts and analyses. The ISRP recommended that a steering committee with specific technical expertise, as well as internal CTUIR and ad hoc external scientific assistance, would greatly strengthen the project, if such technical guidance is not already formally established. The CTUIR did not respond to this important recommendation.

In addition, the ISRP was confused about inconsistencies within the response and between the response and the updated proposal. One example is regarding the number of sites that are in pre-restoration monitoring. At various places, the response states that “a majority,” “all,” “5,” and “half” of the projects are in pre-restoration monitoring, which does not align with the information presented in Table 1 of the updated proposal. Without this basic information, it is impossible for the ISRP to provide a scientific review of the monitoring design.

Going forward, it appears that the project will include a blend of site-scale and reach-scale monitoring. This raises a number of fundamental questions that are not addressed in the proposal or in the response: How will those datasets be related to each other? How will measurements at difference scales be integrated and synthesized? Furthermore, the response frequently indicates that the proponents “anticipate” doing things in “in the future,” but the ISRP needs a definitive monitoring plan so we can review it. When will that be plan be available? And what are the steps the proponents will undertake to complete that plan?

7. Guidance on Monitoring Methods, Design, and Analysis. The proponents did not list this as a response category and did not provide a response to this request.

The ISRP requested that the proponents: “Explain how the project obtains comprehensive review and assistance of its monitoring efforts and analyses. The ISRP recommends that a steering committee with specific technical expertise, as well as internal CTUIR and ad hoc external scientific assistance, would greatly strengthen the project, if such technical guidance is not already formally established.” Perhaps the proponents felt that they responded to this in the previous item.

8. Confounding Factors. No response was provided by the CTUIR. The ISRP asked the proponents to thoroughly describe potential confounding factors that would directly limit the effectiveness of this monitoring activity, and how the project is responding to those factors, rather than factors that generally limit restoration recovery. This does not appear to have been changed in the revised proposal.

An Additional Consideration

In the 2017 Wildlife Project Review (ISRP 2017-7), the ISRP encouraged the CTUIR to use the River Vision and First Foods concept for monitoring and evaluation in addition to traditional monitoring methods from western science, and the ISRP continues to encourage this approach for this project. We acknowledge that the CTUIR project based on the River Vision and First Foods paradigm will have unique characteristics that reflect cultural values, and we anticipate these will strengthen the relevance and scope of the outcomes from monitoring and evaluation. The framework and methods for monitoring and evaluation still need to be explicit regardless of whether the program is based on Tribal and/or western values and experiences.

 

Preliminary ISRP report comments: response requested (Provided for context. The proponents responded to the ISRP’s questions; see response link and final review above.)

Response request comment:

The Biomonitoring project is a central component in the CTUIR’s restoration efforts in five basins. The need for the program is well justified in the Problem Statement, and it is clear that other projects depend on this effort to understand the benefits and limitations of their restoration efforts. However, the project appears to be a somewhat ad hoc collection of monitoring efforts that lack a coherent strategy and statistical robustness. Based on the information provided, the ISRP cannot determine if the monitoring program is scientifically appropriate for either evaluating effectiveness of restoration actions or for determining the status and trends of Chinook salmon, steelhead, or bull trout populations.

In a revised proposal, the proponents need to demonstrate that they have a robust and logical monitoring program, one that directly supports understanding and enhancing the benefits of restoration, which is the project’s broader goal. The proposal is currently deficient in the development of goals and SMART objectives, description of methods, the analysis and presentation of project results and findings, the development of a clear adaptive evaluation process, and the documentation of overall benefits to fish and wildlife. A much more rigorous interpretation of the datasets is needed. These deficiencies raise important questions about functional linkages with the separately BPA-funded habitat projects, which rely on the monitoring results produced by this project. The revised proposal and response should clearly demonstrate that the habitat restoration projects are receiving reliable, relevant, and timely information from this monitoring and evaluation project.

The ISRP requests the proponents to address the following points in a revised proposal and to provide a brief point-by-point response to explain how and where each issue is addressed in the revised proposal. As well, the proposal needs to be reformatted according to the Council’s guidance for proposal preparation.

1. M&E matrix - support. As habitat projects and monitoring projects are not presented as part of an integrated proposal or plan, the need for a crosswalk to identify the linkages between implementation and monitoring is extremely important for basins or geographic areas. The ISRP is requesting a response from the Grande Ronde Model Watershed Project (199202601) to summarize the linkages between implementation and monitoring projects in the Grande Ronde and Imnaha geographic area. We also ask this project to assist other lead projects that are developing M&E summaries of other basins where this project is operating. We ask this project to assist them in creating the summary and provide information to them about what is being monitored by this project and where and when the monitoring occurs. A map or maps of locations of monitoring actions would be helpful in this regard.

2. SMART objectives. Provide a single description of goals and SMART objectives (see proposal instructions). Where objectives are not completely quantifiable with biologically and physically relevant metrics, the proponents should be as explicit as possible about the desired characteristics and outcomes of implementing actions to achieve specific objectives.

3. Results of past monitoring. Describe clear and complete results of past monitoring for each of the three focal species, as well as the actions and/or basin characteristics that have been monitored. The proponents should summarize and explain overall conclusions for the major monitoring components. For instance, a summary could address the fundamental conclusions from the monitoring program as well as the implications for future actions and management directions. If monitoring has demonstrated that restoration actions have not been effective, then explain how the CTUIR will modify the BPA-funded restoration projects and the CTUIR Natural Resources Program.

4. Benefits for fish and wildlife. Provide a clear empirical description of the project’s benefits and its related restoration actions for fish and wildlife. This is a fundamental criterion of projects funded under the Fish and Wildlife Program. The explanation of benefits to fish and wildlife should be scientifically rigorous, explicit, unambiguous, and quantitative. Where benefits have not been realized, the proponents should identify specific adjustments and changes in the program to address the lack of desired outcomes.

5. Project evaluation and adjustment. Provide a thorough description of the project’s evaluation and adaptive adjustment process, both within the project and how the proponents communicate with projects receiving their monitoring results (for instance, their input on the monitoring study designs, mechanisms for regular data and information transfer). The description should include information on known decision points, explicit schedules for evaluation and decision-making, and documentation of decisions and project changes.

6. Monitoring design. Describe the process for developing monitoring projects, including statistical design and analyses, modeling, field methods, and laboratory methods. This description should also include how the project coordinates with projects who are implementing restoration actions.

7. Guidance on monitoring methods, design, and analysis. Explain how the project obtains comprehensive review and assistance of its monitoring efforts and analyses. The ISRP recommends that a steering committee with specific technical expertise, as well as internal CTUIR and ad hoc external scientific assistance, would greatly strengthen the project, if such technical guidance is not already formally established.

8. Confounding factors. Thoroughly describe potential confounding factors that would directly limit the effectiveness of this monitoring activity, and how the project is responding to those factors, rather than factors that generally limit restoration recovery.

Q1: Clearly defined objectives and outcomes

During the 2018 Research Status Review (ISRP 2018-8), the ISRP concluded that this project was underperforming. At that time, given the extensive changes to the project, as well as the general uncertainty regarding funding and the future of three Basin-wide monitoring programs (i.e., CHaMP, AEM, and ISEMP), the ISRP felt that the project needed to be re-examined and re-defined with updated monitoring questions, quantitative objectives, and realistic timelines. Most of these concerns still apply. The current proposal only reinforces the impression that the project is fragmented in many ways. The proposal makes it very difficult for the ISRP to understand where and why data are being collected, how appropriate statistical analyses are being used to support study design and synthesis of monitoring data, how the monitoring data support key findings about restoration actions, and how those key findings are being used to adjust existing habitat restoration projects.

The objectives need to be in a SMART format. Without a SMART format the activities cannot be adequately evaluated during the next review. While a list of 11 qualitative objectives are provided, they do not seem feasible, and the activities defined under the Methods section do not clearly align with those objectives. In addition, based on the text on page 31, it is unclear if the project is still relying on the objectives (and methods?) of CHaMP and AEM, or if they have modified these. If these were modified, explain how the new data are expected to crosswalk with data collected under the prior objectives and methods? The project may be more effective if it focuses instead on fewer objectives and on objectives that are well framed for evaluating the effects of targeted restoration actions.

Further, the project’s goals and objectives are presented in two sections: 1) Progress to Date and 2) Goals and Objectives. The two descriptions of goals and objectives are not consistent. The Progress to Date section lists a single goal and 5 objectives. The Goals and Objectives section lists 3 goals and 11 objectives.

In the “Progress to Date” section, most objectives are general statements of desired actions or outcomes. Only a few objectives are quantitative. Explicit time frames are not provided for the subset of objectives that are quantitative, and the proposal does not explain the biological or ecological basis for any objectives. Further, there is a list of a “Summary of the Physical Benefits of Restoration Monitoring” and list of a “Summary of the Biological Benefits of Restoration Monitoring,” but the lists are actually potential physical or biological outcomes that the project desires rather than benefits quantitatively determined via monitoring. Even in the list provided, the physical or ecological benefits are not relevant for all focal species or for all locations within the river network. Separate outcomes for salmonid abundance and salmonid densities are listed even though the two are intricately related and equivalent in most locations. Some separate types of outcomes are mistakenly combined in a single outcome.

In the “Goals and Objectives” section, separate goals are indicated for 1) biological monitoring of treatments and controls for restoration projects, 2) physical monitoring of treatments and controls for restoration projects, and 3) biological and physical monitoring of reference reaches. The objectives are implementation objectives that call for different types of monitoring activities or analyses, they are not SMART objectives (quantitative or time bound). The proposal repeats the importance of monitoring at appropriate spatial scales but neither monitors nor applies analytical methods for a range of spatial scales. In particular, no approach for basin-scale monitoring and analysis is described in the proposal.

Q2: Methods

The methods indicate that five activities will be conducted:

• PIT tagging of spring Chinook in the upper Umatilla River and at Meacham Creek

• Seasonal snorkel events for juvenile abundance at treatment and control reaches

• Analysis of redd surveys conducted by project partners

• Macroinvertebrate sampling

• Physical habitat monitoring at 1,3,5, and 10 years post project.

Unfortunately, there was no clear overarching logic plan for what is monitored where and why, and the proposal contradicted itself in multiple places. Some examples are noted below.

Basic information about site selection is needed. While the proposal indicates that sites were selected according to the same methods as CHaMP and AEM protocols, the proposal does not describe those methods or the sites that were selected. Furthermore, the Meacham Creek site was selected, despite not having pre-project data, because it embodies one of the largest ongoing restoration efforts for addressing instream and floodplain function, but that justification is not consistent with site selection in CHaMP or AEM. It is also not clear how many sites are being monitored, or where those sites are located. At different places within the proposal, the text indicates that there are 9, 11, or 16 sites.

In addition, descriptions of methods are missing or unclear, particularly related the analysis of data. For instance, the project conducted WUA analysis and concluded that it increased for some of the projects. However, given the many analysis decisions and limitation of WUA-type analyses, more information is needed on how the analysis was conducted. As well, some basic information is missing, such as in Fig. 8. Why are the proponents showing only the highest correlated physical values (pool area), and what are the correlations to? In addition, do the proponents plan to continue the WUA analysis under the next project period?

Interpretation of prior data is also problematic, particularly about whether fish abundances and densities increased as a result of restoration actions (i.e., pages 12, 36, and 40). The text contradicts itself in multiple places, and the findings do not always appear to be based on the data. Furthermore, the data (as presented) do not include evaluation of variability or uncertainty in interpreting the significance of any differences. In other cases, the proponents use vague words like “increase” to reflect differences between a treatment and control, rather than empirical changes at a site over time.

More specific concerns are bulleted below. Taken together, these issues reduce confidence in the findings.

• Figure 3 and 4: Years need to be placed on these plots to help understand the context for identifying and understanding the pre-post years. Were they hot and dry, cold and wet, or moderate years? How are the proponents interpreting data to evaluate causality? Also, is there a time series of data that are condensed somehow into a pre- and post-dataset?

• Page 18: It is not clear how juvenile PIT tags will be used in evaluating restoration effectiveness. Please clarify.

• Pages 19 -21: What did the proponents learn from the drift monitoring studies?

• Page 25: If it is likely that redd abundance will not be sensitive to site-scale restoration, then why are the proponents monitoring redds? Also, the subsequent text suggests that the program may move in the direction of monitoring spawning suitability rather than counting redds. Is this what is being planned for the next project period?

• Page 27: The proposal indicates that coordination with habitat restoration implementers will continue, but it is not clear what kind of coordination is currently happening and what more may be needed.

• Please clarify how this project relates to the PHaMS effort. What complementary monitoring and methods are being referred to and how are the data and personnel coordinated?

• Page 35: The proponents “propose to integrate surveys of reference areas to compare with our treatment and control sites.” Why? If there are indeed multiple years of data, why is this necessary? It seems like critical information will be lost concerning annual variability at a site.

• Page 37: What are the results of the 2019 field test of the rapid instream physical habitat monitoring methods? And against what criteria are those methods being tested?

• Page 41: Benthos are being monitored annually, but previously the text said that this effort had ended. Will this effort be continued in the next project phase? If so, please justify given that the proposal earlier reported them to be of limited value.

• Page 45: What was the process for coordination with experts in the field? Who were the experts and what disciplines did they represent? Does the project receive ongoing guidance from these experts? How was the protocol modified as a result?

• Page 46 (Table 1): It is not clear how this is related to the project since the actions in the table do not reflect the actions undertaken by CTUIR’s habitat restoration project (199608300), and it is not clear how the proposed monitoring actions align with the touchstones in the Table.

Finally, information is needed to explain how the project intends to synthesize the results of monitoring other than before and after comparisons of specific sites. Methods for basin scale monitoring or synthesis need to be described in the revised proposal. While the proposal indicates that CHaMP and AEM protocols are employed, the proponents do not indicate which protocols are used. It does not appear that these methods are documented in MonitoringResources.org, but they should be.

Although a positive aspect of the project is the storage of data in the Tribe’s CDMS and the Sitka Tech’s GeoOptix program, the data management process is not adequately described. This component is critical for data archiving and retrieval. Have the proponents documented the structure and details of the data management system and metadata documentation?

Q3: Provisions for M&E

Basic linkages to the project adjustment process are not readily apparent. The proponents are involved with other projects, but the mechanisms for data and information transfer need to be transparent.

While the proposal mentions that adjustments are “frequently suggested by oversight committees and/or local oversight groups, and annual operating plans,” it is not clear who those oversight groups are or how the proponents consider and respond to those suggestions. The ISRP assumes they are the groups identified in Table 1 (page 51), but it is not clear what information or process is used to evaluate the program. Although the ISRP and Council review process provides an important opportunity for project adjustment and evaluation (page 52), the ISRP is not the appropriate venue for regular project evaluation and adjustment processes because of the multi-year intervals between ISRP reviews and the limited discussion between the project collaborators. The project needs a strategic and structured process for evaluating whether it is meeting its objectives and for steering the project’s vision.

The ISRP agrees that (p. 27) “Expanding the collaboration of monitoring efforts beyond salmon centric projects to include other significant species such as mussels and lamprey is critical to inform effects of restoration on CTUIR First Foods.” Do the proponents have plans and capacity to expand in these directions?

While the proponents believe that the conclusion, discussion, and recommendations from annual reports to BPA are important inputs for adaptive management, they do not complete the cyclic adaptive management process. How are these recommendations communicated to the related projects in a timely manner for them to make project modifications? While the proponents mention that (p. 61) “CTUIR RM&E and CRITFC are also part of the Adaptive Management sub-group and are co-authors on a recent collaborative adaptive management paper,” it is not clear that they are involved in regular adaptive management or structured decision-making processes for this project.

In the present narrative, the proponents do not address the potential compounding factors that would limit the effectiveness of a monitoring project. Issues such as poorly managed livestock and irrigation activities, stream channelization, pollutants, and improper fish passage are mentioned. However, the ISRP feels that factors such as (for instance) up-to-date training, staff turnover rates, vandalism at sites or to equipment, or a mismatch of data collected and the objectives for specific restoration actions would more directly influence project performance.

The project incorporates the River Vision Touchstone Relationships in their monitoring plan. This is a positive integration of cultural knowledge and perspectives, and the ISRP encourages the project to continue to make such linkages even more explicit.

Q4: Results – benefits to fish and wildlife

The goal of the Biomonitoring project is to assess the effectiveness of CTUIR’s habitat restoration actions for spring Chinook salmon, summer steelhead, and bull trout populations. The reporting of the project’s monitoring results is confusing and unclear. Many details of specific findings are lost in overall averages for sites or species.

 The proponents present significant information and analyses on fish abundance and density for a variety of pre-treatment and post-treatment situations. Even though significant improvements in physical habitat conditions are documented, the fish data mostly show no statistically significant difference in average abundance and density for the vast majority of sites. While there may not have been adequate time after restoration actions to reveal positive fish responses or the sampling may have been too limited in spatial scope, the fact remains that the majority of sites are not adequately responding to restoration measures. The results are equivocal in most cases. Why? The proponents do not explore this question. However, it is their responsibility to see that the key information becomes part of the adaptive management process of the individual implementation projects so that it might be used to adjust restoration actions. Unfortunately, that essential link is not clearly evident in the proposal narrative. Another interpretation might be that, since it is not clear that the project is informing or improving restoration actions, there is no evidence of benefit to fish; a significant finding for which the associated habitat projects should be making adjustments.

Specifically,

• Project results are presented for juvenile salmonid density and abundance, but all salmonid species are combined. The results often are not consistent within or between projects, and the proposal does not discuss or explain why they are not consistent.

• Physical monitoring results also are presented collectively for five different projects, but the outcomes for channel and floodplain area and pool area are combined for all sites as overall averages (separate values are graphically illustrated). The proposal reports that wetted usable area for juvenile Chinook and steelhead increase as bankfull area or total wetted area increases, which almost inherently has to be true, and does not document the effectiveness or success of restoration.

• Numbers of PIT-tagged juvenile Chinook captured in screw traps are reported but the data are not analyzed to estimate survival, migration timing, or anything other than the percent of tagged fish observed in the downstream trap (2-3%). If this is correct, the survival rates for out-migrating Chinook juveniles are shockingly low and a major cause for concern.

• Graphs of macroinvertebrates are included for eight restoration sites, but it is not clear whether the results are benthic invertebrates or drift (both terms are used), and the outcomes are not discussed or explained. Few if any sites exhibited differences in these measures of macroinvertebrates before or after restoration, indicating that restoration is not influencing macroinvertebrate communities. Unfortunately, the proposal does not explain the findings or draw conclusions.

• The proposal uses two graphs to illustrate redd densities for Chinook in 2020 and steelhead from 2009-2020. The proposal states that the project has “seen no difference in redds/km when comparing the treatment and control reaches (Figure 14 and 15). With future habitat projects in the mainstem Umatilla, we might see these densities change.” The data for treatment and control sites are not presented, and the hopeful conjecture is not supported by data or explained.

Finally, the proposal states that “responses at the population and watershed level will require a much broader analysis and incorporation of methodology within each basin” but provides no proposed analyses, models, or monitoring approaches that would provide a coherent landscape analysis. The discussion of overall findings for fish abundance and physical habitat responses is confusing, contradictory, and largely provides reasons why the restoration has not been effective, or the monitoring has not been sufficient to detect a response.

Qualification: Currently, the proposal does not provide enough information to allow a complete scientific evaluation. Overall, this proposal is to provide funding to work with a consulting firm (Stillwater Sciences) to develop an experimental design for evaluation of the effectiveness of habitat restoration activities in five subbasins. Given the importance of this work, the ISRP recommends that it review the proposed experimental design when it is completed.

The proponents of this work propose to develop and publish in the peer-reviewed literature a detailed design for evaluating habitat enhancement across five basins managed by the CTUIR. Evaluating the effectiveness of habitat enhancement, especially for anadromous steelhead and Chinook, but also migratory fish like bull trout, is very challenging because: a) effects of factors outside of the basins (like ocean conditions) interact with any effects of habitat enhancement, and b) fish use habitat across multiple scales, even in freshwater, so response variables must be measured across multiple scales as well.

Overlaid on this is a third challenge, which is that habitat enhancement is combined with flow augmentation and hatchery supplementation in various tributaries, making simple treatment-control comparisons difficult. In many cases, these three main actions are combined, so that simple comparisons are often confounded. For example, effects of habitat enhancement measured in a treatment vs. control stream could be caused by another factor that is also different between the two streams (e.g., one has received supplementation whereas the other has not).

1. Purpose, Significance to Regional Programs, Technical Background, and Objectives

The CTUIR is proposing to develop and publish an experimental design and sampling protocol to measure biological results of habitat enhancement actions in a scientifically defensible way. The project is well justified and badly needed in the Umatilla Basin. This project is related to regional intensively monitored watershed (IMW) programs and to MERR, PNAMP, CSMEP, and ISEMP (although it is unclear how it will be integrated with these other programs). It addresses the ISRP’s recommendation for habitat effectiveness monitoring in the Umatilla Basin.

Discussion of the technical background could have been improved if the proponents had presented the conceptual experimental design that they say was completed in 2009. The proponents also say they have summarized RM&E actions in each of the five subbasins on ceded land. It would have been useful, even necessary, to include the summary in the proposal in abbreviated form.

The technical challenge, as described above, will be to develop an experimental design which can:

a) Address the interaction between freshwater habitat (and changes to it by habitat enhancement) and mainstem survival, ocean conditions, and climate change.

b) Address how effects of habitat enhancement can be teased apart from effects of supplementation, flow augmentation, or other factors that affect freshwater survival and growth. It seems that statistical models could be developed across this larger number of basins and tributaries to partition out the effects of these factors, as well as the effects of different basins, and changes through time, perhaps using a “model selection” approach (Burnham and Anderson 2002. Springer).

c) Address what response variables about fish and habitat will be measured at what scale, to test specific hypotheses. For example, hypotheses may range from some at the local scale such as “Growth of juvenile Chinook will be greater in off channel than mainstem habitats” to very broad-scale hypotheses such as “Relative reproductive success of steelhead is greater in tributaries where habitat has been enhanced versus control watersheds without habitat enhancement, after accounting for the effects of density-dependence, ocean conditions, and changes in supplementation.”

2. History: Accomplishments, Results, and Adaptive Management

The proponents have completed a conceptual experimental design but did not include it in the proposal.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (Hatchery, RME, Tagging)

The project appears to be closely related to other projects on ceded lands and to ODFW’s Umatilla Intensively Monitored Watershed program, but these linkages were not entirely clear.

Emerging limiting factors include overarching effects of climate change, lag times from other projects or disturbances, and effects of supplementation on biological responses. How the experimental design will tease apart these effects from those of habitat enhancement will need to be clearly laid out.

4. Deliverables, Work Elements, Metrics, and Methods

Metrics and methods will need to be made clear in the design. At a finer level than the interacting effects described above, rarely is a habitat response (e.g., side channel development) independent of other habitat changes (e.g., flood plain reconnection). Separating a single response from all other habitat responses, say in a reach, could be difficult. The design will also need to clearly lay out how this will be addressed.

Qualification: Currently, the proposal does not provide enough information to allow a complete scientific evaluation. Overall, this proposal is to provide funding to work with a consulting firm (Stillwater Sciences) to develop an experimental design for evaluation of the effectiveness of habitat restoration activities in five subbasins. Given the importance of this work, the ISRP recommends that it review the proposed experimental design when it is completed. The proponents of this work propose to develop and publish in the peer-reviewed literature a detailed design for evaluating habitat enhancement across five basins managed by the CTUIR. Evaluating the effectiveness of habitat enhancement, especially for anadromous steelhead and Chinook, but also migratory fish like bull trout, is very challenging because: a) effects of factors outside of the basins (like ocean conditions) interact with any effects of habitat enhancement, and b) fish use habitat across multiple scales, even in freshwater, so response variables must be measured across multiple scales as well. Overlaid on this is a third challenge, which is that habitat enhancement is combined with flow augmentation and hatchery supplementation in various tributaries, making simple treatment-control comparisons difficult. In many cases, these three main actions are combined, so that simple comparisons are often confounded. For example, effects of habitat enhancement measured in a treatment vs. control stream could be caused by another factor that is also different between the two streams (e.g., one has received supplementation whereas the other has not). 1. Purpose, Significance to Regional Programs, Technical Background, and Objectives The CTUIR is proposing to develop and publish an experimental design and sampling protocol to measure biological results of habitat enhancement actions in a scientifically defensible way. The project is well justified and badly needed in the Umatilla Basin. This project is related to regional intensively monitored watershed (IMW) programs and to MERR, PNAMP, CSMEP, and ISEMP (although it is unclear how it will be integrated with these other programs). It addresses the ISRP’s recommendation for habitat effectiveness monitoring in the Umatilla Basin. Discussion of the technical background could have been improved if the proponents had presented the conceptual experimental design that they say was completed in 2009. The proponents also say they have summarized RM&E actions in each of the five subbasins on ceded land. It would have been useful, even necessary, to include the summary in the proposal in abbreviated form. The technical challenge, as described above, will be to develop an experimental design which can: a) Address the interaction between freshwater habitat (and changes to it by habitat enhancement) and mainstem survival, ocean conditions, and climate change. b) Address how effects of habitat enhancement can be teased apart from effects of supplementation, flow augmentation, or other factors that affect freshwater survival and growth. It seems that statistical models could be developed across this larger number of basins and tributaries to partition out the effects of these factors, as well as the effects of different basins, and changes through time, perhaps using a “model selection” approach (Burnham and Anderson 2002. Springer). c) Address what response variables about fish and habitat will be measured at what scale, to test specific hypotheses. For example, hypotheses may range from some at the local scale such as “Growth of juvenile Chinook will be greater in off channel than mainstem habitats” to very broad-scale hypotheses such as “Relative reproductive success of steelhead is greater in tributaries where habitat has been enhanced versus control watersheds without habitat enhancement, after accounting for the effects of density-dependence, ocean conditions, and changes in supplementation.” 2. History: Accomplishments, Results, and Adaptive Management The proponents have completed a conceptual experimental design but did not include it in the proposal. 3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (Hatchery, RME, Tagging) The project appears to be closely related to other projects on ceded lands and to ODFW’s Umatilla Intensively Monitored Watershed program, but these linkages were not entirely clear. Emerging limiting factors include overarching effects of climate change, lag times from other projects or disturbances, and effects of supplementation on biological responses. How the experimental design will tease apart these effects from those of habitat enhancement will need to be clearly laid out. 4. Deliverables, Work Elements, Metrics, and Methods Metrics and methods will need to be made clear in the design. At a finer level than the interacting effects described above, rarely is a habitat response (e.g., side channel development) independent of other habitat changes (e.g., flood plain reconnection). Separating a single response from all other habitat responses, say in a reach, could be difficult. The design will also need to clearly lay out how this will be addressed.

CTUIR Response

This biomonitoring plan will be implemented through existing CTUIR M&E programs which is consistent with how ODFW is proposing to implement each of the proposed IMW projects (ODFW Umatilla O&M project and Grande Ronde life history project).  The biomonitoring design will utilize monitoring efforts comparable to CHaMP protocol with emphasis on the biological response to habitat restoration.  CTUIR will continue to coordinate and integrate efforts of model watersheds, ODFW, CRITFC, USFS, and BOR efforts that focus on viable salmon population criteria and habitat effectiveness monitoring.  A biomonitoring plan will include standardized monitoring metrics for determining habitat effectiveness at the watershed, stream segment, and stream reach scale.  CTUIR will participate in model development with actions agencies to standardized metrics are used.  The biomonitoring plan will complement and enhance co-manager data collection methods in an effort to avoid overlap and maximize the efficient use of resources to be in compliance with RPA 57.4 and 57.5.

 

CTUIR has operated screw traps in the Umatilla River at approximate RM 54.0, 76.0, and 81.0.  Continuation of a screw trap is certain at RM 81.0 and tentative at RM 54.0 and 76.0.  CTUIR does not operate screw traps in the John Day and upper Grand Ronde rivers.  CTUIR will coordinate those efforts with ODFW to collect the necessary information and if not we will coordinate with ODFW efforts to expand trapping.