Contract Description:
The purpose of the Tribal Data Network (TDN) is to assist CRITFC and its member tribes in the timely and accurate capture, storage, processing, and dissemination of data for management of anadromous fish and their habitats. There are four main objectives for the project.
1. Assist CRITFC and member tribes to develop cost effective computer architectures and data management strategies for anadromous fish and habitat data. Develop pilot systems and tools demonstrating cost effective ways to collect, store, summarize, and disseminate fish and habitat data. Convert successful pilot projects into production computer systems as resources allow.
2. Provide data management services to the tribes. These take a variety of forms, but includes:
a. Partial support for tribal data coordinators (in this expanded proposal),
b. Developing shared data capture, management, and reporting tools for CRITFC and member tribes,
c. Expert advice and coordination of efforts, especially through the annual Tribal Data Workshops and
d. Limited support for tribal infrastructure through one-time purchases of hardware and software.
3. Assist member tribes to build internal capacity for improved data management and to achieve the tribal gravel-to-gravel management vision of the Commission. Combine local data from tribal sources with regional and international data on mainstem, estuary and ocean impacts on salmon and salmon management decisions; and
4. Enable tribal participation in regional data management coordination processes, provide tribal input and coordination on regional coordination of data management and sharing best practices, facilitate inter-tribal coordination on the level of monitoring data.
At the end of the first two years of the project, a suite of pilot projects is underway, and some pilots have already displayed demonstrable success. The data management approach and technology used in the current pilots has proven successful, and can be applied elsewhere. The expectation is that the technologies used in the pilots will be used to address priorities identified in the Coordinated Assessment process. TDN system architectures depend on lessons learned during the past 15 years of experience developing data management systems for the mainstem Columbia River.
Lessons Learned:
1. The most cost effective data management systems are those that enter field data one time and one time only. It is always more expensive to re-enter the same data into multiple formats.
2. QA and QC should be pushed out to be as close to the field staff as possible. The person who entered the data is the best at performing QA/QC on the data, for he/she knows the most about the data. QA/QC is far more accurate and cost effective when performed by the field staff.
3. A paper trail and an audit trail should be created. Data in the Columbia Basin is often subjected to scientific, judicial, and policy review; data management systems must be designed to provide defensible accounting from top to bottom. Estimates and other summarized quantities need to be able to be traced back to field data collection, and responsibility for the data collection must be able to be traced back to identifiable field staff.
4. To get buy-in from the field staff, provide tools that actually make their job easier. To get management buy-in, provide aggregated data more accurately and timely.
5. Minimize fish handling, minimize stress on fish. When monitoring ESA listed fish in the Columbia Basin, there are so many concurrent studies; over handling of fish is a recurrent problem.
6. Focus effort on getting the field data into SQL servers that drive dynamic web sites. Once the monitoring data is stored electronically under policy maker control, then it just a matter of what, where, and who to output the data to. Once data is in an SQL server, it is a simple matter to aggregate and summarize the data into any format. SQL servers driving dynamic web sites and web services gives one the most flexible method of outputting monitoring data into whatever formats management needs now and in the future.
7. Build on existing field data collection methods and practices whenever possible to minimize training expense and disruption of current field data collection efforts.
8. All access to monitoring data is controlled by the individual tribe who owns it. CRITFC generated data is controlled by the Fish Commission, individual tribal monitoring data is controlled by the individual tribe that owns it. Where the monitoring data is hosted and stored is up to the individual tribes.
Current Pilot and Production Projects using digital pen technology and web services architecture:
1. Bonneville Adult Fish Facility Data Management
2. Snake River Harvest Estimate
3. Zone 6 Harvest Estimate (Nez Perce Tribe component)
4. Below Bonneville Harvest Estimate
5. Willamette Falls Lamprey Data Management
6. Klickitat Basin Surveys
a. Spawner Patch Survey
b. Habitat Unit Survey
c. Large woody debris survey
d. Stream Bedrock survey
e. Large woody debris jam survey
f. Discharge data
g. Electrofishing survey
h. Sampling events tracking table
i. Stream Survey Tracking table showing reaches for each stream
These projects all use paper forms and a digital pens to collect field data. There are 17 pens deployed to date. The data is transmitted to a web site in the cloud where an electronic copy of each form can be viewed by field staff from any web browser. A slider bar along the bottom of the form allows the field staff to view their own handwriting, and to view what the computer interprets it to be. Validation code in each cell allows for range checking, lookup tables, and other constraints to reduce transcription and other errors. When the field staff has validated the data, pressing an “approve” button sends the data via web services to an SQL server hosted by the appropriate entity, be it CRITFC or a member tribe. Summarization, aggregation, and other data processing takes place on the SQL servers, and the data is almost instantly made available on web sites under tribal management control in whatever form is required. Once the data is hosted on an SQL server, it is a simple programming task to output it into whatever form is needed, on whatever schedule is needed (for example, DETs). A digital copy of each form in the PDF format is also generated by this process, so one can trace any estimate back to the original paper forms, or an electronic scan of the original paper forms.
Other current pilot and production projects:
1. Portable Anadromous Data Structure (PADS) to facilitate lifecycle analysis for legacy and current biological and habitat data in the Grand Ronde basin. This effort is focused on Chinook on the Upper Grand Ronde and Catherine Creek.
2. Web based tag loss application for estimating tag loss.
3. CRITFC Data Center for Tribal Data Network.
Current resources are focused on assisting the tribes to get field data consolidated on SQL servers under the appropriate tribal management control. Current funding supports the ongoing activities of the TDN, and some new development. Additional resources are required to produce the estimates and summarized data currently desired by the Coordinated Assessment from the existing field data. If the field data is consolidated under management control on SQL servers, it is a straightforward task to produce any format of output that the Coordinated Assessment (DETs) or adaptive management may require.
Appendix M of the "Columbia River Basin Collaborative Data Sharing Strategy: Salmon and Steelhead Population Abundance and Productivity Indicators" describes Tier 1 high priority data gaps that need to be filled to enable CRITFC member tribes to share the three initial VSP parameters with the region. The additional funds needed to fill these Tier 1 gaps is $351,000.
$100,000 in funding has been provided, which will enable the Tribal Data Network to address the goals of the Collaborative Data Sharing Strategy, and the Framework for Fish and Wildlife Program Data Management.
The modified statement of work enumerates three deliverables to be produced with this additional funding:
1) A Preliminary Data Inventory for all BPA funded projects for each tribe
2) Progress on a Tribal Data Management Strategy for each tribe
3) Progress on producing Monitoring Data Sharing Agreements for each tribe
The specific statements of work for each of the Tribal Data Stewards are found in attachments. it is hoped that four tribal data stewards will be hired and onboard by June 1st, 2013. These data stewards will focus on producing specific deliverables similar to the general deliverables described here. For specific deliverable variations, see the attachments.