201303 Program research methodology Malawi
Africa RISING research co-learning workshop
6 – 8 March 2013
Lilongwe, Malawi[edit | edit source]
Workshop objectives[edit | edit source]
- . Ensure that we have a common, clearly articulated vision for the program
- . Articulate the assumptions and approaches of the research teams and the experimentally
significant scales at which they will be working
- . Identify divergent assumptions, approaches, and scales (the other side of the coin of Obj. 2)
- . More fully utilize broad range of skills and experiences of program participants to strengthen
all research teams (develop clear pathways to ensure continued strengthening)
- . Identify skill gaps (e.g., anthropology, social science, economists) and determine how to fill
them and/or leveraging other projects/experts
- . Ensure that the outcomes, results are written up and communicated
Participants[edit | edit source]
- Tracy Powell, USAID/DC
- Jeroen Groot, Wageningen University
- Lotte Klapwijk, Wageningen University
- Regis Chikowo, Michigan State University
- Mateete Bekunda, IITA/Tanzania
- Asamoah Larbi, IITA/Ghana
- Kathy , IITA/Nigeria
- Beliyou Haile, IFPRI/DC
- Peter Thorne, ILRI/Ethiopia
- Ian Wright, ILRI/Ethiopia
- Irmgard Hoeschle-Zeledon, IITA/Nigeria
- Sieg Snapp, Michigan State University
Agenda[edit | edit source]
Wednesday, 7 March
14:30 Review of January Accra meeting & presentation of agenda – Jerry Glover
- genesis of Lilongwe meeting
- identify note-taking, plan for reporting out, posting on wiki site
15:00 Tanzania strategy and discussion – Jerry and Mateete
16:00 RO1 proposed strategy – Jereon Groot and Lotte Klapwijk
18:00 General discussion
Thursday, 8 March
08:30 RO1 discuss
- 15 mins. Peter Thorne
– Jereon Groot and Lotte Klapwijk
11:00 Research activity simulation--Malawi – Jerry Glover
14:00 Discussion of RO2 & RO3 -- Mateete, Asamoah, Peter
- review hypotheses
Friday, 9 March
08:00 M&E implications -- Beliyou
09:30 Develop protocol and body for program-wide input, consensus, information dissemination
10:00 Additional USAID related news/issues
-- Development of larger Sustainable Intensification program: Africa RISING, CSISA, Sust. Intens. Innovation Lab, IPM Innovation Lab, Small-
scale irrigation program (US university led in cooperation w/ AR and IWMI)
-- Vital Signs – BMGF-funded, Cons. Int’l implemented—data collection, knowledge management contacts (BMGF) Stan Wood and Kate Schneider
- USAID mission relations—updates and improvements?
- Mali situation re: USAID funding seems to be improving—implications for AR?
10:30 Communicating workshop results/decisions—put as much up on the wiki site as possible
11:30 Need to develop a longer term calendar—events we can put on the calendar?
Notes of the Meeting[edit | edit source]
Meeting Notes: 3/6/2013
I. MEETING INTRODUCTION (Jerry Glover)
- A brief history of Africa RISING, which has been embraced as a test case for / component of HumidTropics. In alignment with the CG reorganization, which emphasizes engagement of multiple centers in research partnerships, Africa RISING unites 3 distinct regional projects and an associated M&E project under a single umbrella program.
- At the January Accra meeting, divergent visions of Africa RISING impeded communication between the various national, regional, and M&E research teams – particularly divergent assumptions about what methodological approaches the program would take, and at what experimental scales these approaches would be applied.
- The regional and M&E project leaders agreed that this Lilongwe meeting would provide an opportunity to discuss these divergent assumptions, articulate a common vision for the program, and help understand the different terminologies used to describe various experimental scales across the regional programs (district, village, community, household).
II. RESEARCH SCALES EXERCISE
- At the January Accra meeting, the Mali, Ethiopia and Ghana research teams went through an exercise in which they identified the various experimentally relevant scales at which they work. This exercise highlighted differences in the terminology and assumptions of various research teams when they discuss their activities, and identified potential sources of miscommunication and misunderstanding across the program.
- At this meeting, Mateete went through the research scales exercise on behalf of the Tanzania project. For each regional project that has undergone the exercise thus far, the terms used for each experimentally relevant scale are summarized below:
- 25 communities total
- 10-50 farms/community
- communities stratified by market access, growing period, population
- 3 development domains total
- 5-10 communities/DD
counterfactuals identified at this level
- farms include non-contiguous fields
- 3-10 split-plot treatments per farm
- 4-20 farms/village
- 3 total development domains
- 10-20 action villages/development domain (+ 10-20 counterfactual villages/DD)
- DDs stratified by agroecological zone and market access Cercle
- 1-2 development domains/cercle (Koutiala=1, Bougouni=1-2)
- will stratify HHs by livelihoods typologies identified in Quick Win project
- 30 farms/kebele Woreda
- only 2 woredas total for now (plus 3 counterfactual woredas); plan add more kebeles/woreda later this year
- 2-6 kebeles/woreda
- stratified by market access
- experimental relevance of this scale is unclear, particularly in relation to administrative zones
- ~1-3 woredas/DD (plus 1-3 counterfactual woredas)
- 500-1000 total HHs/village; ~10-100 action HH’s/village
- counterfactual villages will be identified at this scale (within same DD)
10 Development Domains
- stratified by rainfall and elevation
- 1 village/DD (for now; will scale up at this level later on)
- 10 DDs total (5 in Babati, 4 in Kongwa, 1 in Kiteto)
- experimental relevance of this scale is unclear; possibly relevant for R4D platform development
- Discussion: Are “development domains” an experimentally relevant scale?
- Household Typologies versus hierarchical Development Domains: these are two complementary/overlapping rubrics for systematically clustering/stratifying farm types in order to identify key constraints and drivers of agricultural intensification, and facilitate targeted scaling.
- Development domains drove IFPRI’s initial strategy for impact assessment, and will likely remain relevant to their analysis
- Development domains may not be experimentally useful for farm/household level research, but will become relevant when the teams begin looking at RO3 strategies for identifying drivers of intensification and scaling strategies.
III. IITA ADOPTS WAGENINGEN UNIVERSITY’S PROPOSED RO1 STRATEGY
- Jeroen Groot presented details of the Wageningen team’s proposed farming systems analysis (which was initially discussed at the HumidTropics meeting in Nairobi).
- View the presentation
- Read the proposal (current as of March 27, 2013)
- Read background literature on farming systems analysis, modeling, and household typologies:
circulated by the Wageningen team ([http:dx.doi.org/10.1016/j.agee.2006.03.037| Groot et al. 2007], [http:dx.doi.org/10.1016/j.eja.2009.07.002| Groot et al 2010], [http:dx.doi.org/10.1111/j.2041-210X.2011.00114.x| Groot et al. 2011], [http:dx.doi.org/10.1016/j.agsy.2012.03.012| Groot et al. 2012])
circulated by Africa RISING researchers ( Byrnes 1989, [http:dx.doi.org/10.1016/j.agee.2005.04.001| Tittonell et al. 2005a], [http:dx.doi.org/10.1016/j.agee.2005.04.003| Tittonell et al. 2005b], [http:dx.doi.org/10.1016/j.agsy.2009.10.001| Tittonell et al. 2010], [http:dx.doi.org/10.1007/978-90-481-2543-2_76| Tittonell et al. 2011])
- Summary of proposed farming system analysis activities. \
Ideally, the Wageningen approach requires subsampling of 50-100 households from the baseline survey population (for rapid, survey-based characterization of farm and livelihood status), followed by further subsampling of 10-50 households from the rapid characterization population (for more intensive, survey-based farm and livelihood characterization). These data will be used to generate household typologies, model farm systems, explore alternative farm configurations, and identify entry points to sustainable intensification for subsequent RO2 activities.
- Critique: Don’t fall into the “Diagnosis Trap”.
There was general agreement that this type of systems analysis has the potential of falling into an analytical feedback loop, in which researchers perpetually refine their models without ever graduating to the RO2, field-intervention stage. Subsequent discussion at this meeting will address the question of how to link RO1 outputs to RO2 actions in order to escape this “diagnosis trap”.
IITA (Irmgard, Mateete, Asamoah) officially adopted Wageningen’s proposed RO1 plan for the West Africa and East and Southern Africa projects.
the Malawi team__ will conduct complementary modeling research (and some care must be taken to ensure that graduate students working in these respective efforts don’t duplicate efforts geographically or overburden the same farmers). __Ethiopia: Peter Thorne already has plans to work with ILRI’s Mark VanVeuk (sp?) to do 2-3 months of R01-related work on ex ante impact evaluation and modeling, and will pursue those plans before potentially incorporating Wageningen’s approach.
- Next Steps.
To proceed, the Wageningen team is working to recruit one postdoctoral researcher and two graduate students (including Lotte), who will then train work teams in Ghana, Mali, Tanzania, and Malawi to implement the farm characterizations. Training will start in early April in Tanzania.
for survey. Ideally, the baseline survey will be done in advance of Wageningen’s preliminary/rapid characterization. However, given ongoing delays in baseline implementation, these data may not be collected in time for their activities in all areas; the farming systems team should therefore be prepared to start collecting their rapid characterization data separately where necessary.
Need to maintain communication with HumidTropics.
This effort should not proceed in isolation, and should continue to inform the HumidTropics approach. Shared personnel (i.e. the Ethiopia project’s gender specialist, who will be 50% Africa RISING/25% HumidTropics/25% fish & livestock) should help maintain lines of communication, but efforts should be taken to ensure continued cross-pollination.
Meeting Notes: 3/7/2013[edit | edit source]
I. INTRODUCTION TO R01 ACTIVITIES IN ETHIOPIA (Peter Thorne)
- To accompany the Wageningen group’s proposed farming systems research activities, Peter Thorne presented an overview of past ILRI research in farming systems characterization and analysis, which will inform planned RO1 activities to be implemented in the coming weeks. This participatory approach engages local farmers to identify key capital asset indicators, then uses these indicators to stratify farms into different “livelihoods typologies,” which can then be used to identify household types amenable to similar RO2 intensification solutions.
- View the presentation
II. INTRODUCTION TO RO1 ACTIVITIES IN MALAWI (Sieg Snapp)
- To accompany the Ethiopia and Wageningen teams’ presentations on RO1 strategies, Sieg Snapp also presented a brief overview of planned and ongoing farm characterization efforts in Northern Malawi (not Africa RISING) where calories met are used to typoplogize households. In this approach, mother-baby trials provide plot-level data on agronomic performance across different agroecologies, which the Malawi team will integrate into household-level models alongside survey data, participatory systems analysis, and participatory modeling (particularly for climate risk assessment).
- View the presentation
III. DISCUSSION OF IFPRI’S PLANNED R01 ACTIVITIES/BASELINE (Beliyou Haile)
- Beliyou presented an overview of options for how to move forward on the IFPRI baseline survey (rolled into detailed discussion below).
- View the presentation (link to come)
- Read IFPRI’s summary document (link to come)
- Discussion of the status of the baseline survey, and the feasibility of meeting the current timeline.
Currently, regional M&E coordinators have not been hired and survey implementers have not been identified for any of the five Africa RISING countries:
- Ghana: IPA fell through; IFPRI is waiting on ISSER to propose a budget for survey implementation
- Mali: not certain of survey status given political situation
- Tanzania: discussions with potential survey firm stalled due to lack of clarity on the number & location of households to be surveyed; can potentially re-open discussion once greater clarity is attained on the IFPRI baseline SOW.
- Malawi: has not yet been considered (Sieg can potentially facilitate through her network)
- Ethiopia:survey not due until September, which leaves time to explore options for enumerators (Peter suggested ATA)
IV. GENERAL COMMENTS ON AFRICA RISING IMPACT EVALUATION
- Africa RISING research activities are not suitable for evaluation via a “rigorous” impact assessment as initially proposed by IFPRI (where “rigorous” is narrowly defined as a fully randomized RCT).
- despite randomized site selection, Africa RISING’s participatory research model requires researchers to work with volunteer farmers (i.e. nonrandom selection at the farm/household level)
- despite some criticism of the assumptions/data on which IFPRI based their power calculations, the regional project teams generally agreed that Africa RISING could not sample enough households to meet the requirements of IFPRI’s proposed impact evaluation design.
- at a given action site, researchers will evaluate many different technologies and technology combinations, rather than a discrete technology or approach, making it difficult to identify how any net impact was achieved. This limits the utility of IFPRI’s proposed evaluation design.
- as a research program, Africa RISING’s success shouldn’t be measured by the standards of a development program (subsequent discussion of how we should measure success is described below).
- Although the proposed RCT impact evaluation is not feasible, alternative approaches to impact assessment are available and widely accepted by the academic community. IFPRI’s impact assessment should make use of these tools, particularly panel data. (Sieg will circulate examples/references.)
Background literature on sampling design:
- circulated by Sieg Snapp: ([http:dx.doi.org/j.agsy.2009.10.001| Tittonell et al. 2010], WB 2008 [http:dx.doi.org/10.1016/j.worlddev.2011.09.019| Richardson et al. 2012] BOC Survey Sampling Manual)
- circulated by Carlo Azzari: ( list of suggested references)
- Anticipated scale of impacts.
In order to capture development impacts produced by Africa RISING activities, the impact assessment should sample at a relevant scale(s). The regional teams expect their activities to yield development impacts at the household and community scales, at the level of a group of certain farmers (typologies), and possibly at the scale of the development domain. (Any plans to sample at a higher scale, i.e. national, are inappropriate in the context of Africa RISING.) A variety of methods may be employed to evaluate program impacts at the various scales.
- Goal of evaluation (1): The survey tool should collect data that allow farm typologies arising from farm systems research to be extrapolated at the level of the development domain. This will allow researchers to infer potential targets and predict ex ante impacts of scaling.
- Goal of evaluation (2): The regional research teams wish to detect adoption/uptake of technologies, both within and beyond the intervention communities. The baseline and endline surveys should therefore sample both intervention and non-intervention farmers within the action site (to detect adoption within the intervention community, or “spillover”), as well as farmers within the counterfactual community (to detect adoption outside the intervention community, or “contamination”).
V. DISCUSSION OF WHAT THE REGIONAL RESEARCH TEAMS WANT FROM AN IMPACT ASSESSMENT (& DEVELOPMENT OF A SCOPE OF WORK STATEMENT)
Before initiating baseline collection, IFPRI requests greater clarity on the regional teams’ needs and expectations for the impact evaluation. By answering the following questions, the meeting participants will attempt to provide guidance:
- Desired outcomes of an impact assessment:' (Reference p. 12 of the program document) An impact assessment should provide evidence that Africa RISING interventions have improved nutrition/income/productivity at the household level, and provide evidence of successful R4D platforms. It should provide evidence that can be used to design more effective development programs, which would then be executed by others (in contrast, the assessment should not attempt to provide evidence that Africa RISING itself has had broad development impact, as this is not the goal of the research program). Ideally, the impact assessment will facilitate scaling by helping to extrapolate successful intensification strategies to larger scales, i.e.// common agroecological zones across Africa.
Key questions for consideration today:
- As a research-for-development program, what impacts does Africa RISING want to measure?
- What evidence do the regional project teams need to measure these impacts/determine successful
achievement of program objectives?
- Who is responsible for evaluating program impacts at various scales?
Ian’s proposed framework for Africa RISING impact assessment:
To assess program success, an impact evaluation should address the following three questions, at the specified scales.
- What are the processes by which Africa RISING is improving productivity?
Relevant scale: the household/farm scale
Who is responsible for providing evidence: led by the regional project research teams, using a variety of agronomic, systems, and other approaches (possible role for IFPRI?)
- What are the implications of these productivity-enhancing processes for sustainability?
Relevant scale: the household/farm scale, with some NRM implications at the landscape level
Who is responsible? Led by the regional research teams, using a variety of agronomic, systems, and other approaches
(possible role for IFPRI?)
- What are the impacts of these productivity-enhancing processes on development outcomes, at a variety of scales?
Household/farm livelihoods scale:
- led by the regional research teams
- employs a variety of agronomic, systems, and other approaches
Community/kebele scale (e.g. impacts on other value chain actors)
- led by the regional research teams and IFPRI
- employs a variety of agronomic, systems, and other approaches
Development Domain scale:
- led by IFPRI
- assessment based on data from survey tool
- (Here, “development domain” refers to the original IFPRI designation from the site selection process, which takes into account locally relevant market, biophysical, and demographic indicators and is therefore defined in a country-specific context.)
- addresses the broader development agenda and program-wide design, including scaling and adoption
- A draft concept note summarizing the above framework will be sent Thursday (3/7) to Carlo, who will provide a preliminary response. A small working group (led by Ian) will then clarify and expand on the initial concept note by drafting a detailed scope of work for the impact assessment. They will circulate a preliminary draft of the scope of work to the regional research teams by Tuesday, March 12, and a revised draft to Carlo by Friday, March 15. Jerry will schedule a follow-up meeting with Carlo shortly thereafter to discuss how to proceed.
- Link to Ian’s draft proposal for Africa RISING impact evaluation (current as of March 18, 2013)
VI. COMMENTS ON THE SURVEY TOOL
- The current draft of the survey tool looks generally adequate, but should undergo a final round of revision to:
- reflect the newly evolved vision of what the impact assessment should achieve
- ensure the tool supports extrapolation of farm typologies generated by systems analysis
- incorporate input from the regional research teams regarding local norms and contexts
- include indicators that address sustainability and farmer innovation
- address any other final concerns prior to survey implementation
- Beliyou will circulate the current survey draft to Mateete, Asamoah, Peter, Jeroen, Tom (and Sieg?) by Monday, March 11. They will respond with comments by Saturday, March 16.
Meeting Notes: 3/8/2013[edit | edit source]
I. DISCUSSION OF RO1 & RO2
Given the “diagnosis trap,” how to ensure that RO1 actually feeds into RO2 actions?
- Emphasis on iterative nature of RO1 and RO2 activities: Agronomy teams document farmer innovations and perform participatory evaluations, which inform models, which inform research interventions, which refine models, etc.
- In the Malawi context, there’s a nice separation of the mother (demonstration) and baby trials, which leaves room for variations in the on-farm research approach depending on the local context. Sieg will post research design references for different variations on multi-location trials.
- How to integrate on-farm research with modeling/systems research?
- use typologies to identify key constraints & potential technology solutions (i.e. what types of risk do certain household types suffer, and how can we reduce risk through technology selection, such as using climate-resilient varieties or reducing monocrop dependency)
- potentially discuss ways to incorporate/inform technology sequencing or technology combination hypotheses using modeling approach? (these hypotheses are currently flagged for possible deletion in the program document, but may be worth revisiting as research strategies evolve)
- Make systems modeling outputs relevant to farmers! E.g., generate visualization tools from farming systems models, which can then be used in participatory modeling exercises with farmers. Another possibility: use models to create situation-specific game to illustrate intensification strategies/tradeoffs?
How to ensure that farming systems research skills percolate throughout the entire program?
- The Malawi and Wageningen teams have greater depth of experience/knowledge in farming systems modeling than some other groups, and can potentially act as resources.
- To this end, the Malawi team will tentatively organize a modeling meeting in July. Regis will write and circulate a short description as a first step, then the group will narrow down dates and add the event to the Africa RISING online event calendar. Peter Thorne suggested the possibility of making this a more global forum with participation from centers of excellence. This would require additional funding.
II. REGIONAL PROJECT WORK PLANS
- At the January Ghana meeting, the West Africa work plans were frozen on account of being “too conventionally agronomic.” Given the current discussions, do we have a new vision/proposal to advance?
- In Ghana,
Asamoah had originally planned to follow the RCT method laid out by Joseph Rusike (which embraced a somewhat different vision of others in Africa RISING). With the new guidance discussed at this meeting, he will instead pursue an approach that embraces mother-baby trials and farming systems modeling, incorporating the Wageningen team’s analyses. He will guide the Mali team in this new direction.
- In Tanzania,
with the cropping season in full swing, the research teams are focusing on ground-truthing new technologies rather than integrating old ones, in an effort to build partnerships between research implementers that will facilitate later cross-pollination. Though this may look like component technology trials in the initial work plans, it is part of a larger strategy of getting multiple partners to co-locate activities at the same site, with the eventual goal of integrating technologies/efforts.
- The current discussions highlight the need to ensure that regional work plans fit into the larger vision of the Africa RISING program. Moving forward, how can we ensure this?
The regional lead scientists (Asamoah, Mateete, and Peter) will meet – either in person at planned AR events or virtually – and discuss work plans before they go to the Steering Committees for approval. They will ensure that work plans reflect the coherent vision of Africa RISING, and that coherent guidance and messages are communicated on through their respective regional programs.
III. CALENDAR OF EVENTS
- Everyone needs to do a better job of planning events in advance and updating the shared calendar by e-mailing Kathy. The following tentative calendar should be fleshed out/finalized ASAP:
- ESA review and planning meeting in September. 4 days (select dates by end of March)
- Ethiopia review and planning meeting. 2-3 days. September/October, maybe combine with learning event?
- West Africa Review and Planning meeting in January 2014. 3 days. (select dates by end of March)
- Program Coordination Team Meeting. (dates, location TBD)
- Learning Event. Possibly to be held in Addis, possibly to be combined with the Review & Planning meeting. Need to check with Peter Ballantyne – may be redundant with planned modeling workshop in Malawi, in which case consider canceling/postponing.
- M&E Meeting. September? (IFPRI needs to determine whether this is necessary, what the objectives are, etc.) Also possible to combine with a learning event.
- Regional field visits? (midyear in W. Africa?)
IV. USAID UPDATES
- Within USAID’s reorganized research framework, Africa RISING fits into the Food Security Innovation Center’s Program in Sustainable Intensification, which will soon include new programs and potential partners (i.e. a Sustainable Intensification CRSP to replace SANREM; a new IPM CRSP). Africa RISING should consider ways to integrate/leverage activities with these new partners. Co-location? Other suggestions?
- As USAID’s Sustainable Intensification program gets underway, Jerry’s attention will increasingly shift away from Africa RISING. As a result, the program will need to be more effective at coordinating communications internally, as Jerry will be unavailable to serve as a central referee for messages and decision-making.
- VitalSigns, a Gates Foundation project currently headed by Stan Wood, has approached USAID about potential opportunities to partner with Africa RISING. Jerry has an upcoming meeting scheduled, but the project could potentially offer good data management opportunities, with an emphasis on linking data at various research scales. Jerry will update as he learns more.
- The Mali situation is improving: USAID now has formal approval to work in Mali (though not to work directly with the government). This frees up $2 million in USAID mission funds for ICRISAT Mali, though Jerry suggests further consideration by the mission of the best way for ICRISAT to use these funds (through Africa RISING, or outside of it?)
- ILRI has added a new Africa RISING communications specialist, Simret Yasuba (50% program-wide and 50% Ethiopia-specific).
- Mateete will host The 1st International Nitrogen Conference in Kampala on November 18th – 22nd . Jerry will look into providing some additional funding, and whether the event might be combined with a similar event planned by the Nitrogen Initiative.