Evaluation Privacy Guidelines

From Earth Science Information Partners (ESIP)

Note: This document is currently being drafted and the text does not necessarily reflect the final policies and procedures of the evaluation process.

Testbed Evaluation Privacy Guidelines

The evaluation process is collaborative; however, there are some privacy expectations for the evaluators and the PIs.

For PIs and other project members, it’s important to note that certain evaluation criteria assume that the codebase or other items are publicly available. However, we understand that this evaluation process may occur during a development cycle or involve a private codebase for other reasons. Please refer to (INSERT Tech Evaluation Thresholds) for the private codebase constraint. For the former, please note when providing access to the codebase (or other documents) if those items are intended to remain private so that the collaboration space can be managed correctly for those items. Evaluators are expected to respect the PIs’ privacy concerns in these matters.

ESIP will continue to improve and assess the evaluation process itself. To do that, staff may access the evaluation documents, such as the TEF spreadsheets, for analysis purposes. The outcomes of any analyses may be made public to the ESIP community or the larger geosciences community through presentations or published articles. The analyses will anonymize the evaluators; PIs, due to the nature of the projects and the process, will not be anonymized.

Shared Collaboration Space

The TEF uses the Open Science Framework collaborative workspace for document sharing, working notes and other collaborative activities. For the most part, the way the space is used is up to the larger evaluation group. (INSERT: note if any part of the OSF space is public-facing.)

To ensure that the evaluation is objective, the collaboration space does contain a space intended to be private to the evaluators. While ESIP will provide an OSF project with a private evaluator space at the start of the evaluation process, these spaces can be modified by the evaluation participants so consider where and how information is posted if you have concerns about access levels.

The collaboration space will be accessible to members of ESIP staff for administration purposes. For reference, the OSF user names of the staff will be provided to the participants. In addition, the evaluators space will be accessed during the report writing phase.

INSERT: policy for archiving the collaboration spaces, contingent on the capabilities of the OSF (can an archive be exported from the hosted platform at all?) and the features of that archive (if an archive can be exported, does it retain the permissions within the space?).

Draft and Final Reports

This section is in progress and the text does not reflect the final policies.

  • Release of final reports MAY depend on the organization requesting the evaluation.
  • We can assume the draft reports are confidential, where the drafts are provided to the PIs for a brief response period.
  • We can also assume that the final reports are confidential, delivered only to the PIs and the funding organization (or whoever is requesting the evaluation).
  • We have discussed analysis of the TEF spreadsheet and outcomes previously so, given the previous statements, what are the limits of the analysis and are those products public (if anonymized)?