ESIP 2011 Winter Meeting Decisions Workshop
Notes from Session
Insight to Impact
- Why Evaluate
- To provide credible information and verify that initiative is doing as planned
- Assess impact
- Discover challenges early to optimize outcome/impact
- Prioritize resources and activities, to make changes and insure sustainability
- Addressing complexity
- ESIP is a difficult organization to evaluate due to its diverse membership
- Examine individual objectives instead of just goals
- Complex systems like ESIP:
- Connections are essential +simple rules lead to complex responses + Individuals have creative opportunity to respond within rules
- Requires complex evaluation methods
- Complex adaptive systems:
- Input → activity→ output → outcome → IMPACT
- Output is evaluated and feedback into the system as another input
- Impact addresses that product was not only used, but that the use had an effect.
- Traditional approach: past events predict future outcomes
- Emergence- agents interact in random ways (interpersonal relationships and social networking)
- Connectivity – systems depend on interconnections and feedback→ dissemination across stakeholders
- Interdependence - of environment and other human systems.
- Butterfly effects, small changes have large impacts, cultural sensitivity to the differences between agencies involved in ESIP
- Rules- systems are governed by simple conventions that are self-organized.
- Underlying consistencies and patterns may appear random and lead to different outcomes than anticipated
- Outcomes are optimized in terms of meeting specific thresholds, predictability is not expected except in broad focus.
- ESIP is a difficult organization to evaluate due to its diverse membership
- Where to start? Discussion of 1st Key Evaluation Findings
- Concerns and Recommendations
- Stakeholder focused - unmet needs, varying expectations, no when or why → Improve communication strategy/ clarify purpose, process, value added and engagement of wider audience/ Establish clear mechanisms for acknowledging contributions
- Geoss Focused - detrimental effect of voluntary nature, lack of resources→ conduct gap analysis, alternative models, long-term strategy for support and sustainability (membership fees)
- Many suggestions ambiguous, and not really actionable
- Concerns and Recommendations
- Managing Data Complexity/Characterizing Programs
- Plausibility – correct logic
- Feasibility – sufficient resources
- Measurability – credible ways to discover results
- Meaningfulness – stakeholders can see effects
- Theory of Change
- Identifies a causal pathway from implementation of the program to the intended outcomes by specifying what is needed for outcomes to be achieved
- To build one:
- Identify long-term goals and assumptions behind them
- Backwards mapping and connect the preconditions or requirements necessary to achieve that goal
- Identifying the interventions that your initiative will perform
- Develop indicators to evaluate outcome
- Writing a narrative to explain the logic
- Outcome mapping
- Causal chain between short-term outcome and long-term goals.
- Looking for impact
- Identify intermediate outcomes
- Use near-real-time assessment
- Approaches to Evaluation
- Needs assessment – magnitude of need, possible solutions
- Evaluability assessment – has there been sufficient implementation
- Conceptualization-focused evaluation – help define the program, target population, possible outcomes
- Implementation evaluation
- Process evaluation
- Developmental evaluation – focus on innovative engagement
- Outcome evaluation
- Impact evaluation
- Cost-effectiveness and cost-benefit analysis – standardizing outcomes in dollar costs and values
- Meta-analysis – studies impact across studies of a similar magnitude for an overall judgment on an evaluation question
- Gap analysis
- Existing status
- Apirants – condition in comparison to other competing organizations
- Market – potential to grow given current political economic and demographic conditions
- Program/product – are there products not being produced that could be?
- Data collection
- Outcome based monitoring – don’t collect data for the sake of it, monitor to benefit the outcome and achieve goals
- Goal driven management – needs to be done for a reason, not because it is the rule
- Go from best-guess decisions to data-based decision making
- Cooperate across partners, collaboration is priority over competition/discrimination between different departments or roles
- Anticipate need instead of reacting
- Information is disseminated and transparent
- Measurement Precision
- Consistency and accuracy – not fixed, variable due to differences in collection procedures and understanding of data
- Measuring validity of the pipeline of data, not the scientific validity of the content.
- Validity
- Balancing data and methods
- Qualitative v quantitative, contextual v less contextual
- Attitudes and underlying reasons v pure data
- Anecdotal data can be mined using qualitative software if there are enough stories and statements
- Randomized clinical trials
- Do not always provide better evidence than observational studies – especially for rare adverse effects
o Comparative effectiveness research o Conducting and synthesizing existing research comparing the benefits and harms of different strategies and interventions to monitor condition sin “real world” settings o Strength of Evidence o Risk of bias o Consistency o Directness o Precision • Dose-response association – differential exposure/duration of participation • Confounding factors – present or absent • Magnitude of the effect/impact – strong or weak • Publication bias – selective publication of studies/ no current studies available o Grading strength – high/moderate/low/insufficient : based on availability of evidence and extent that it reflects reality o Establishing metrics – SMART approach o Specific o Measureable o Actionable o Relevant o Timely • Analytic too SWOT • Strengths • Weaknesses • Opportunities • Threats o Identify how to harness opportunities and strengths in order to tackle weaknesses and threats o Not just a list of factors, a list of actions