P&S Telecon - December 21,2015

Applying TEF for Incubation Testbed Projects

Discussion Question:Testbed Evaluation Framework (TEF) process worked well for AIST evaluations, but was very structured to achieve an evaluation of the AIST project Technology Readiness Levels (TRLs). For other projects coming through the Testbed - e.g., the regular incubation projects that ESIP funds every 6 months - can we leverage the TEF as a space to support feedback and infusion opps for those projects?

First project that this may apply to: the recently funded "". Met with them early today and brainstormed (see this google doc for those notes).

Role of P&S for Incubation projects in the past

• P&S (Testbed Configuration Board) decided which projects are funded and how they are advertised - i.e., posters at twice-yearly meetings, mid-year demo on P&S or IT&I call, etc.
• This discussion comes to how we evaluate a project once it is funded – do we bring a third party as part of the Testbed Incubation process?

Financially Supporting Evaluation

• Cost is an issue: asking awarded projects to do an evaluation on top of their proposed work can cost more – can be funded separately, or can be small effort.
• External evaluators are funded separately.

Deciding what Projects are Candidates for Evaluation

• Some projects may not want to move along the trajectory and be evaluated - that is OK.
• Perhaps there is a Process for Applying for Evaluation?
• Perhaps a goal for P&S next year could be to formalize:
  • What are the requirements that an ESIP project would be evaluated against?
  • We are funding projects more than one time – when we are at funding at say round 3, we can impose an evaluation?
  • Or a project suggests itself to be evaluated?

ESIP “Stamp of Approval”

• Evaluation part of this "approved/evaluated by ESIP" idea? Datasets, information quality
• Do we retain the AIST framework or not?

Transparency

What to do about transparency when the evaluation does not go well? It’s a public process. Criteria needs to be very clear – folks need to know what they are getting into, and evaluation may not always yield good news.

Larger role for P&S members in TEF Process

Important topic here - not sure what this looks like, let's think about this when we are together next month at the Winter Meeting.