Response requested comment:

The ISRP felt that the proposal lacked adequate detail to evaluate the validity or feasibility of the project to meet its objectives. Key elements of the workflow are missing, as articulated below. Equally important, the proposal does not describe how the data will be used to inform restoration prioritization or design, as well as how the program will be evaluated and adaptively managed. Therefore, the proponents are asked to respond to the following issues. The ISRP understands that some of the issues may take more time to fully address than provided in the month-long response loop. In those cases, the response should discuss an approach to address the questions in the future.

1.      Provide quantitative objectives and hypotheses to guide the monitoring, and set realistic timelines. The ISRP does note, however, that many of the deliverables provide for quantitative measurements.

2.      Identify regional monitoring programs with which this program has relevance and relationships. For instance, describe the significance and relationship of this project to the basin-scale AEM project and to other databases in the region.

3.      Provide adequate detail on project relationships and costs. For instance, the proposal identifies several related projects and offers some examples of how this project could support those efforts. If these relationships are to be realized they should be formalized before being presented as project benefits and the work plan should accurately reflect the personnel FTEs required to complete the work.

4.      Provide an adaptive management (AM) plan, one that addresses internal processes as well as external influences.

5.      Provide details on the workflow as it relates to the broad goal of examining project-scale effectiveness at a resolution allowing one to adaptively manage the habitat program (e.g., site selection, site design, prioritization). There are three broad categories of details that are needed.

        a.      What specific activities are planned under this phase of the project? SMART goals with a supporting Gantt chart wouldhelp articulate what will be completed by whom and when.

        b.      The monitoring and sampling plans need further detail. The proponents should consider consulting with experts to design the monitoring plan so that data contribute to informing restoration design by establishing causality, providing consistency across sites, addressing statistical power in detecting effects on fish, etc., as detailed below.

        c.       How will the project-scale effectiveness translate into improvements in the habitat program (prioritization, site selection, site design, etc.)? Several examples of those details are provided below.

6.      Describe how the anticipated results (e.g., doubling of fish) will inform future projects. As well, assist the ISRP in understanding how the use of existing resources (e.g., restoration design and monitoring literature, other monitoring programs) are informing the process of establishing an action effectiveness monitoring project.

7.      Provide details of the project roll out. These were not explained in the proposal or the presentation. How long into the future before the proposed 4-5 projects per year are implemented?

8.      Identify threats to program investments and eventual success. This was not done in the proposal.

It will be advantageous for the proponents to discuss the content of the response with the ISRP before submitting it. Once the final version is received, the ISRP will reevaluate the proposal for scientific validity.

Comment:

1. Objectives, Significance to Regional Programs, and Technical Background

The proposal provides an overarching goal and three clearly stated objectives around tracking the fish response to restoration actions. While these objectives are clearly stated, the proposal does not provide adequate detail to understand if these objectives can be achieved (i.e., the objectives are too general). The proponents need to provide quantitative objectives, and hypotheses to guide monitoring efforts and set realistic timelines. The ISRP does note, however, that many of the Deliverables will be quantitative measurements.

Regarding significance to regional programs, the proposal lists some regional plans to which this project is relevant. However, perhaps more importantly, the proposal does not identify regional monitoring programs with which this program has relevance and relationships. For example, what is the significance and relationship of this project to the basin-scale AEM project or other databases in the region? Is there overlap with other monitoring programs that could result in a greater efficiency and scope with a little additional coordination? Are there opportunities to collaborate on sharing data repositories and dissemination?

The focus on biological sampling in winter (in addition to other seasons) is a positive feature of the proposed activities. Few other projects consider winter conditions, which the ISRP believes are critical in shaping fish population dynamics and habitat characteristics.

2. Results and Adaptive Management

The project currently does not have any results since it was launched only last year. More importantly, the project does not have an adaptive management (AM) process. An AM plan should be developed now, in advance of any issues that may arise. Having such a plan will allow the proponents to proactively identify and respond to issues in a timely manner.

3. Methods: Project Relationships, Work Types, and Deliverables

Is it not clear that the workflow, as presented, will lead to the broad goal of examining project-scale effectiveness at a resolution allowing the proponents to adaptively manage the habitat program (site selection, site design, prioritization). Based on the workflow presented in the proposal and the presentation, the ability to actually inform project prioritization and design is low. Broadly, it is not clear how the results (e.g., doubling of fish) will inform future projects. Data from a case study posted on a database do not lead to better projects. There are several key steps in a workflow needed for the data to actually inform a longitudinal improvement in project design. For example:

·         Establish a site selection and a sampling plan that represents the range of conditions and project types

·         Provide an explicit definition of success and failure, and failure modes

·         Field data collection - specifically, identify characteristics that can be used to evaluate drivers of project success (e.g., volume of large wood below active channel?)

·         Data processing - QA/QC, metrics

·         Since the description of the analysis of the BACI design does not appear to account for the multiple pre- and post-treatment measurements, a more sophisticated model will be needed.

·         Database interfacing and management - The plan needs to be more comprehensive than only posting it to a dashboard.

·         Data analysis and interpretation - In addition to analyses of fish data, identify the habitat data to be collected and articulate how they will be used to examine relationships between project design and fish response.

·         Establish a plan for creating institution experience and memory. Identify how data are used to inform revisions to projects.

Related to this, the ISRP would like to understand how the proponents are using existing resources (e.g., restoration design and monitoring literature, other monitoring programs) to inform their process.

Outside of fish data development, adding a peer-review process for individual projects and data is likely to be very useful in adapting restoration designs. The Upper Columbia Salmon Recovery Board supports several projects in the region. The ISRP suggests that consulting these experts would be helpful, if it hasn't occurred already.

Regarding project relationships, the proposal identifies several related projects and offers some examples of how this project could support those efforts. For instance, the proposal notes that PIT-tag data could be used to estimate overwinter survival rates, which could be applied to the Upper Columbia LCM. However, as written, the proposal does not commit to this or the other identified projects. By using words like "may" or "could," it suggests that this is not really part of the work plan. These relationships should be solidified and articulated before they can be reported as benefits. And if such relationships do become established, the work plan will need to reflect the FTEs required to complete the work. In the LCM example above, who will be analyzing the PIT tag data to produce the overwintering rates? Also, who will be coordinating with developers of the LCM to exchange these data in a format that is needed/useful? These tasks take time, so if this is an expected outcome of this project, personnel needed to complete the work will need to be included and budgeted for in the work plan.

The timeline for rolling out the project indicates a long period before the project will actually benefit restoration designs. For instance, with one project implemented in 2021, this would mean that the project will not inform other projects until 2022 or later, and that there would be only one type of project that is investigated until some unknown future date. When does the project begin monitoring subsequent projects? That is, how long into the future before the proposed 4-5 projects per year is implemented? The details of the roll out were not explained in the proposal or the presentation.

The project would possibly benefit from assistance from the University of Washington group (i.e., J. Skalski), specifically to optimize the design, sampling, and analysis so that the data generated and analyses conducted are of the highest quality and aimed at questions that can be answered with confidence.

For instance, details of how the sampling plan would be designed at individual sites was not provided. Such a plan is needed to ensure that monitoring contributes to restoration design. What spatial design and resolution will be used? How does the temporal frequency of sampling align with key ecological processes? How will control sites be selected to avoid bias (See Bouwes et al. 2016)? Monitoring at sites should be carefully designed to help establish causality across sites in order to advance restoration design.

Use of the Barker model is ideal for estimating survival, but density estimates from snorkel surveys are likely not adequate to achieve an estimate of true density. Other methods should be explored, such as capture-recapture methods, or electrofishing or seining removal methods. The assumption of closure of the population must also be addressed.

Estimates of growth and condition will likely not be able to account for the effects of immigration and emigration. For example, changes in condition may be caused by emigration (fish in better condition emigrate between samples). Growth and condition measured for marked fish that one presumes remained as residents still leaves open the question about the growth and condition of those that emigrated or immigrated.

As described in ISAB 2018-1 Review of Spring Chinook in the Upper Columbia River (Section 4.2.4; p. 129-130), long periods of pre-treatment sampling (five years or more) are needed to have hope of detecting changes such as a doubling or halving of fish abundance. This is due to habitat restoration experiencing the normal levels of annual variation (i.e., process variation). How will the investigators address this issue when the data are analyzed and presented to the funding agencies and managers? Will this level of precision be acceptable? If not, how can the power be increased? Given only two years of pre-treatment data, the precision is likely to be less, requiring even a greater change (e.g., more than doubling of fish abundance) to be detected.

The most comprehensive measure of habitat effects is changes in smolts produced per redd. This requires measurements at the scale of subbasins, some of which is occurring for tributaries like the Twisp River in the Methow basin. How will this project be integrated with these larger efforts to collect data that can provide treatment versus control comparisons of these ultimate metrics of fish performance at the subbasin scale?

Finally, threats to program investments and eventual success are not identified. The proponents need to identify them and evaluate their potential for project disruption.