Difference between revisions of "ESIP 2011 Winter Meeting Decisions Workshop"

From Earth Science Information Partners (ESIP)
(Created page with "=== Notes from Session === Insight to Impact * Why Evaluate ** To provide credible information and verify that initiative is doing as planned ** Assess impact ** Discover challe...")
 
Line 82: Line 82:
 
* Randomized clinical trials
 
* Randomized clinical trials
 
** Do not always provide better evidence than observational studies – especially for rare adverse effects
 
** Do not always provide better evidence than observational studies – especially for rare adverse effects
o Comparative effectiveness research
+
* Comparative effectiveness research
o Conducting and synthesizing existing research comparing the benefits and harms of different strategies and interventions to monitor condition sin “real world” settings
+
** Conducting and synthesizing existing research comparing the benefits and harms of different strategies and interventions to monitor condition sin “real world” settings
o Strength of Evidence
+
* Strength of Evidence
o Risk of bias  
+
** Risk of bias  
o Consistency
+
** Consistency
o Directness
+
** Directness
o Precision
+
** Precision
Dose-response association – differential exposure/duration of participation
+
*** Dose-response association – differential exposure/duration of participation
Confounding factors – present or absent
+
*** Confounding factors – present or absent
Magnitude of the effect/impact – strong or weak
+
*** Magnitude of the effect/impact – strong or weak
Publication bias – selective publication of studies/ no current studies available
+
*** Publication bias – selective publication of studies/ no current studies available
o Grading strength – high/moderate/low/insufficient : based on availability of evidence and extent that it reflects reality
+
** Grading strength – high/moderate/low/insufficient : based on availability of evidence and extent that it reflects reality
o Establishing metrics – SMART approach
+
* Establishing metrics – SMART approach
o Specific  
+
** Specific  
o Measureable  
+
** Measureable  
o Actionable  
+
** Actionable  
o Relevant  
+
** Relevant  
o Timely
+
** Timely
Analytic too SWOT
+
*** Analytic too SWOT
Strengths
+
**** Strengths
Weaknesses
+
**** Weaknesses
Opportunities
+
**** Opportunities
Threats
+
**** Threats
o Identify how to harness opportunities and strengths in order to tackle weaknesses and threats
+
***** Identify how to harness opportunities and strengths in order to tackle weaknesses and threats
o Not just a list of factors, a list of actions
+
***** Not just a list of factors, a list of actions

Revision as of 17:51, January 13, 2011

Notes from Session

Insight to Impact

  • Why Evaluate
    • To provide credible information and verify that initiative is doing as planned
    • Assess impact
    • Discover challenges early to optimize outcome/impact
    • Prioritize resources and activities, to make changes and insure sustainability
  • Addressing complexity
    • ESIP is a difficult organization to evaluate due to its diverse membership
      • Examine individual objectives instead of just goals
    • Complex systems like ESIP:
      • Connections are essential +simple rules lead to complex responses + Individuals have creative opportunity to respond within rules
      • Requires complex evaluation methods
    • Complex adaptive systems:
      • Input → activity→ output → outcome → IMPACT
      • Output is evaluated and feedback into the system as another input
      • Impact addresses that product was not only used, but that the use had an effect.
    • Traditional approach: past events predict future outcomes
    • Emergence- agents interact in random ways (interpersonal relationships and social networking)
    • Connectivity – systems depend on interconnections and feedback→ dissemination across stakeholders
    • Interdependence - of environment and other human systems.
      • Butterfly effects, small changes have large impacts, cultural sensitivity to the differences between agencies involved in ESIP
    • Rules- systems are governed by simple conventions that are self-organized.
      • Underlying consistencies and patterns may appear random and lead to different outcomes than anticipated
    • Outcomes are optimized in terms of meeting specific thresholds, predictability is not expected except in broad focus.
  • Where to start? Discussion of 1st Key Evaluation Findings
    • Concerns and Recommendations
      • Stakeholder focused - unmet needs, varying expectations, no when or why → Improve communication strategy/ clarify purpose, process, value added and engagement of wider audience/ Establish clear mechanisms for acknowledging contributions
      • Geoss Focused - detrimental effect of voluntary nature, lack of resources→ conduct gap analysis, alternative models, long-term strategy for support and sustainability (membership fees)
      • Many suggestions ambiguous, and not really actionable
  • Managing Data Complexity/Characterizing Programs
    • Plausibility – correct logic
    • Feasibility – sufficient resources
    • Measurability – credible ways to discover results
    • Meaningfulness – stakeholders can see effects
  • Theory of Change
    • Identifies a causal pathway from implementation of the program to the intended outcomes by specifying what is needed for outcomes to be achieved
    • To build one:
      • Identify long-term goals and assumptions behind them
      • Backwards mapping and connect the preconditions or requirements necessary to achieve that goal
      • Identifying the interventions that your initiative will perform
      • Develop indicators to evaluate outcome
      • Writing a narrative to explain the logic
    • Outcome mapping
      • Causal chain between short-term outcome and long-term goals.
    • Looking for impact
      • Identify intermediate outcomes
      • Use near-real-time assessment
  • Approaches to Evaluation
    • Needs assessment – magnitude of need, possible solutions
    • Evaluability assessment – has there been sufficient implementation
    • Conceptualization-focused evaluation – help define the program, target population, possible outcomes
    • Implementation evaluation
    • Process evaluation
    • Developmental evaluation – focus on innovative engagement
    • Outcome evaluation
    • Impact evaluation
    • Cost-effectiveness and cost-benefit analysis – standardizing outcomes in dollar costs and values
    • Meta-analysis – studies impact across studies of a similar magnitude for an overall judgment on an evaluation question
  • Gap analysis
    • Existing status
    • Apirants – condition in comparison to other competing organizations
    • Market – potential to grow given current political economic and demographic conditions
    • Program/product – are there products not being produced that could be?
  • Data collection
    • Outcome based monitoring – don’t collect data for the sake of it, monitor to benefit the outcome and achieve goals
    • Goal driven management – needs to be done for a reason, not because it is the rule
    • Go from best-guess decisions to data-based decision making
    • Cooperate across partners, collaboration is priority over competition/discrimination between different departments or roles
    • Anticipate need instead of reacting
    • Information is disseminated and transparent
  • Measurement Precision
    • Consistency and accuracy – not fixed, variable due to differences in collection procedures and understanding of data
    • Measuring validity of the pipeline of data, not the scientific validity of the content.
  • Validity
  • Balancing data and methods
    • Qualitative v quantitative, contextual v less contextual
    • Attitudes and underlying reasons v pure data
    • Anecdotal data can be mined using qualitative software if there are enough stories and statements
  • Randomized clinical trials
    • Do not always provide better evidence than observational studies – especially for rare adverse effects
  • Comparative effectiveness research
    • Conducting and synthesizing existing research comparing the benefits and harms of different strategies and interventions to monitor condition sin “real world” settings
  • Strength of Evidence
    • Risk of bias
    • Consistency
    • Directness
    • Precision
      • Dose-response association – differential exposure/duration of participation
      • Confounding factors – present or absent
      • Magnitude of the effect/impact – strong or weak
      • Publication bias – selective publication of studies/ no current studies available
    • Grading strength – high/moderate/low/insufficient : based on availability of evidence and extent that it reflects reality
  • Establishing metrics – SMART approach
    • Specific
    • Measureable
    • Actionable
    • Relevant
    • Timely
      • Analytic too SWOT
        • Strengths
        • Weaknesses
        • Opportunities
        • Threats
          • Identify how to harness opportunities and strengths in order to tackle weaknesses and threats
          • Not just a list of factors, a list of actions