Editing Moneval3

==Africa RISING Monitoring and Evaluation Annual Meeting<br/>13-14 November 2014<br/>[http://www.tripadvisor.com/Hotel_Review-g297913-d645908-Reviews-Ngurdoto_Mountain_Lodge-Arusha_Arusha_Region.html| Ngurdoto Mountain Lodge], Arusha, Tanzania==

----
<br/>
The annual Africa Research in Sustainable Intensification for the Next Generation (Africa RISING) Monitoring and Evaluation (M&E) Meeting brings together Africa RISING researchers, USAID, and other stakeholders to discuss about monitoring and evaluation of the program. Africa RISING comprises three research-for-development projects to sustainably intensify key African farming systems in West Africa, East and Southern Africa, and the Ethiopian highlands. <br/>

<br/>
It is supported by the United States Agency for International Development (USAID) as part of the U.S. government’s Feed the Future (FTF) initiative. In 2012, the International Institute of Tropical Agriculture (IITA) launched the Sustainable Intensification of Cereal-based Farming Systems in the Guinea Savannah Zone of West Africa and in East and Southern Africa, while International Livestock Research Institute (ILRI) began the Sustainable Intensification of Crop-livestock Systems to improve food security and farm income diversification in the Ethiopian highlands. The International Food Policy Research Institute (IFPRI) is leading the monitoring and evaluation activities of all three projects.
<br/>
<br/>

The 2014 M&E meeting will provide an opportunity for Africa RISING implementing partners and the M&E team to discuss what has and has not worked thus far in Africa RISING and discuss the roadmap. We hope to encourage meeting participants to view this meeting as a learning opportunity and a chance to explore new ideas, encourage participants to discuss their successes and failures openly, and encourage participants to seek out collaborative research opportunities across organizations.
<br/>

<br/>
===Objectives=== 
* <span style="font-family: Arial,Helvetica,sans-serif;">To integrate the goals of the M&E team and the research teams on the ground</span>
* <span style="font-family: Arial,Helvetica,sans-serif;">To share and discuss initial results from the baseline data collection</span>
* <span style="font-family: Arial,Helvetica,sans-serif;">To present an updated version of the Project Mapping and Monitoring Tool (PMMT) and fine-tune its potential usage in the field, based on feedback from implementing teams</span>
* <span style="font-family: Arial,Helvetica,sans-serif;">To define and agree on M&E activities for 2015 and 2016</span>
* <span style="font-family: Arial,Helvetica,sans-serif;">To discuss the results of the external evaluation for West Africa and prioritize activities around key insights/recommendations from the evaluation, as well as how to combine it with the evaluation design of the M&E team.</span>

===Agenda=== 
''All presentations that were featured in the meeting are linked from the notes below, related to the session where they belong.''

<u>Thursday 13 November</u><br/>
09:00 Welcoming remarks (facilitator). Some introductory exercise around M&E objectives and activities. Introducing the meeting objectives and agenda.<br/>
09:45 Chat show with USAID, IFPRI, IITA and ILRI. Setting the scene, reintroducing the project overview and main issues, going back to main components and features of the monitoring and evaluation activities and outputs, as well as outcomes and impact, reflecting back on main lessons and expectations from regional teams about M&E.<br/>
10:30 Break<br/>
11:00 Taking stock of progress since the last M&E meeting<br/>
11:30 Presentation of Africa RISING Program evaluation design, spatial distribution of Africa RISING technologies, farmer typologies<br/>
12:30 Lunch break<br/>
13:30 Presentation of results from Africa RISING Baseline Evaluation Surveys (ARBES)<br/>
15:00 Break<br/>
15:30 Breakout session by megasite<br/>
17:00 Report back to plenary<br/>
17:30 Close<br/>

<u>Friday 14 November</u><br/>
09:00 Ex-ante analysis of crop outcomes (Jawoo Koo and Adam Komarek, IFPRI); looking at CSISA experience (David Spielman, IFPRI)<br/>
10:30 Break<br/>
11:00 Field researchers' successes and failures (evidence from the field), and WA external evaluation<br/>
12:30 Lunch break<br/>
13:30 Continued monitoring with updated tools (PMMT 2.0)<br/>
15:00 Break<br/>
15:30 Tying up the loose ends around data management<br/>
16:00 Discuss M&E plan for 2015 & 2016<br/>
17:00 Closing, next steps, thank you's<br/>
17:15 Closed<br/>


===Presentations===
* [http://www.slideshare.net/africa-rising/ar-me-spielmannov2014| Improving evidence on the impact of agricultural research and extension: Reflections on CSISA’s experience]
* [http://www.slideshare.net/africa-rising/ar-me-koonov2014| Ex-ante modeling analysis of maize yield responses to nutrient management options in Malawi: Action Sites – Dedza & Ntcheu]
* [http://www.slideshare.net/africa-rising/ar-me-komarekinov2014| DAHBSIM: Dynamic Agricultural Household Bio-Economic Simulation Model]
* [http://www.slideshare.net/africa-rising/ar-me-hailenov2014| Africa RISING evaluation design]
* [http://www.slideshare.net/africa-rising/ar-me-groot2nov2014| Typologies for farming systems analysis in Africa RISING]
* [http://www.slideshare.net/africa-rising/ar-me-grootnov2014| Farming systems analysis in Africa RISING]
* [http://www.slideshare.net/africa-rising/ar-me-bacounov2014| Project mapping, monitoring and data management tools for Africa RISING]
* [http://www.slideshare.net/africa-rising/ar-me-azzarrinov2014| Summary of farmer typologies in the Africa RISING baseline survey data]
* [http://www.slideshare.net/africa-rising/ar-me-bekundanov2014| Monitoring and evaluation at the Africa RISING ESA project level]




----
<span style="font-size: 21px;">Meeting notes</span><br/>

===Opening by Carlo Azzarri=== 
'''Objectives of the M&E meeting'''<br/>
* <span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">To integrate the goals of the M&E team and the research teams on the ground</span>
* <span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">To share and discuss initial results from the baseline data collection</span>
* <span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">To present an updated version of the Project Mapping and Monitoring Tool (PMMT) and fine-tune its potential usage in the field, based on feedback from implementing teams</span>
* <span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">To define and agree on M&E activities for 2015 and 2016</span>
* <span style="background-color: #ffffff; font-family: Arial,Helvetica,sans-serif;">To discuss the results of the external evaluation for West Africa and prioritize activities around key insights/recommendations from the evaluation, as well as how to combine it with the evaluation design of the M&E team.</span>

<br/>
Last year we heard about the experiences of CSISA in Addis Ababa. It would be nice to put Africa RISING activities...<br/>


<br/>
===Talk show about program overview and issues=== 

[http://www.slideshare.net/africa-rising/ar-me-bekundanov2014 See presentation Mateete Bekunda - M&E in East and Southern Africa]
<br/>
* Researchers do internal M&E for their projects.
* Our evaluation starts with the proposals that we write.
* For ESA we didn’t develop our own hypothesis so we opted to only develop research objectives that are linked to the program hypotheses.
* In ESA we have a log frame that we use to measure ourselves against
* We have an activity schedule that allows project management to get to the field and monitor our work
* The farmers also evaluate activities that we do on the ground
* We have a template for the scientists reporting – it includes: implemented works and achievements and the deliverables; challenges, success stories, and the FtF indicators
* In ESA our code of conduct where we say if you haven’t written it, then you haven’t done it
* We have also conducted write shops where we listed paper we intend to produce informing the wider scientific community of the work we have done
* We believe for phase one we can be monitored and evaluated based on the outputs of our research activities
* IFPRI has started conducting some impact studies that can also be used.


<br/>

===West Africa M&E progress:===
* The baseline survey was completed in Ghana and Mali under the leadership of IFPRI
* The west Africa team develop a project document it has log frame and some indicators that contribute to the program hypothesis
* We monitior FtF indicators in collaboration with IFPRI
* The baseline survey has been completed and we are expecting a report
* We hope to finalize the project document and log frame-
* The log frame has some indicators which need to be monitored at the local level, however the FtF indicators should be monitored at a higher level. So the question is how are we going to monitor it at the project level-do we have the man power/capacity…..and if IFPRI will do it, what are their plans for doing it?

'''Talk show: Asamoah Larbi (AL), Carlo Azzarri (CA), Jerry glover (JG), Mateete Bekunda (MB), Peter Thorne (PT).'''<br/>

''What do you hope to achieve with the M&E of Africa RISING (asked in regards to their roles in the AR projects)?''<br/>
* <span style="line-height: 1.5;">PT: Evidence is all we need to show our research is contributing to improve people’s lives- we want quantitative evidence as well as qualitative explanations. That gives us a basis to scale and build.</span>
* <span style="line-height: 1.5;">AL: It will help us ensure that project/program outputs are achieved.</span>
* <span style="line-height: 1.5;">JG: M&E will be important for us to improve research effectiveness at lower level , but should also go beyond and contribute to the activities of other development partners along the impact pathway to guide them in their efforts to take our research outputs/technologies or innovations and better target the farmers that they works with so that those technologies are more effectively taken up and we can move along that impact pathway more efficiently and rapidly. We can already see the USAID missions in various African countries now moving the AR efforts into the developments where specific technologies / systems have been identified as effective and are being used to scale up.</span>
* <span style="font-family: Calibri,sans-serif; font-size: 11pt; line-height: 1.5;">MB: M&E helps us as scientists to get out of our “disciplinary boxes” as scientists and enables us to look beyond just the research work we are doing into asking ourselves other questions about impact of our science.</span>

''Within that time cycle, there seem to be contradictions between activities and information. If you want to provide feedback on specific site research activities, do you agree there may be contradictions?''<br/>
* <span style="line-height: 1.5;">PT: You probably need some sort of balance - some messages that you want to deliver isn’t going to be relevant to certain audience. Some you may want to provide narrative and others want numbers. I would be surprised if you have a situation where you only need numbers and not narratives or vice versa. You just have to make sure you collect everything that you likely to need.</span>
* <span style="line-height: 1.5;">AL: I agree with Peter. Since we have different uses we better collect different things for different uses.</span>
* <span style="line-height: 1.5;">CA: Not a single data set can solve all our questions at all levels of AR. For example the scientists could be interested in some other data about the soils while this may not be the aspect interest at another level of the program.</span>
* <span style="line-height: 1.5;">JG: It is useful to consider the historical aspects of Africa RISING. In the early extensive consultations we did with global experts in agricultural research, it was pretty unanimous that we don’t go into the “traditional straight forward approach to solving problems”. Across the board people said we had to go into these ore complex situations working across heterogeneous landscapes, across diverse cultures and address simultaneously not just biophysical constraints but also socio-economic, cultural, nutritional etc. Therefore the different approaches in different regions have to be adopted, the mixing of quantitative and qualitative approaches of data collection and the reality of very different landscape and cultures. It has not been neat, but there are probably some contradictions and confusions. As we are getting into the fourth year we are seeing places that can be standardized and other where we can bring these complexities and contradictions into greater consistency. So in the end I think it’s going to be the benefit of the risk of using different approaches so that it does capture the complexity of the problem overall. Things are more clearer now at least than when we started.</span>
* <span style="line-height: 1.5;">MB: I think the word contradiction isn’t correct. The fact that scientists are still working together despite lots of arguments and discussions is good enough experience. It is also part of a learning process</span>

''How do we reconcile all data uses and methods of collecting them and who is responsible for collecting and coordinating the use of the data''<br/>
* <span style="line-height: 1.5;">MB: We are supposed to be a team and to work on different sections of correcting information, analysing data etc. We should not attribute things to specific institutions. </span>
* <span style="line-height: 1.5;">JG: We will have to reconcile data uses before we can identify what the loose ends are. We are better aligned now and that's the responsibility of Carlo, Peter and Irmgard to identify opportunities to align. But at program level, that's where some serious reconciliation needs to go on to provide standardization, coherence etc. but that will only happen after a certain amount of analysis has taken place... </span>
* <span style="line-height: 1.5;">CA: I don't think we need to reconcile as in using the same data or write similar resaerch outputs but what we need to do is to use our comparative advantage. The M&E team does not have enough experience collecting data that agronomists need but we have other expertise.</span>
* <span style="line-height: 1.5;">AL: We have to work together to collect data. It must be team work. </span>
* <span style="line-height: 1.5;">PT: We need some consensus across all partners. We can also be honest: we've had some very difficult issues to tackle and the fact we're talking together is very encouraging. We work very well together as a team and the issue of data use and consolidation is much easier to reach consensus on than on other issues.</span>
* <span style="line-height: 1.5;">CA: Data management could be prioritized in the sense that we can't collect all data ourselves. Last night (Data management discussion) we agreed to use ILRI CKAN and we now cannot ignore this. We'll be responsible for monitoring data that goes to ILRI CKAN and checking the meta data enforcing requirements etc. but we can't be responsible for the content of the data. </span>


<br/>
''<span style="line-height: 1.5;">It's been said that M&E should be the glue among all the sites - does that imply a project management role and if so is that an IFPRI role, to what extent etc.?</span>''<br/>
* PT: No it doesn't require a project management role. I'm not quite sure what it means to say "M&E is the glue of the project". What M&E can offer is a [...].
* AL: M&E is the responsibility of IFPRI so IFPRI should enable this across all the sites, but not including project management.
* CA: I agree with both. We are involved in different activities than researchers. We are not implementing but monitoring activities on the ground. We should not be included in project management but should be involved in the selection of treatment and control sites and the statistical ways to assess development domains, so in that sense we should be part of project management, but not lead it, just advise it.
* JG: Prior to the consultative process leading to AR, the program was envisioned as three separate projects. Early on people were confident that these three projects should be wrapped into one single coherent program. What was going to be the commonality? What is the connection between them? The IFPRI role is in surface the glue but linking these projects with data, monitoring and evaluation etc. is the role that happened and got clarified in the process. How well this program holds together or not will become increasingly clear. We have to tease out common threads across the program. Household surveys link the three project outcomes and I hope that will come out of the last two years of the program when interpreting/analyzing data etc. When looking at high level data and mapping them with access to markets etc. it's telling me how it's going to hold the project together. The glue is coming.


<br/>
''Peter, you said we should be standardizing data sets. Does that mean that IFPRI is only on the quantitative side or not.''<br/>
* <span style="line-height: 1.5;">PT: I would like us to develop that more. We haven't gone far enough with the integration of qual and quant data. We should do it together with strong inputs. Your comparative advantage is on quantitative data. Our message to the Minister of Agriculture would be: "We've got this, it's really good, here's the evidence". We have both qual and quant data. What are the numbers that we need? We need to get the right data for the audience that we consider. I'm very open to quant data sets also and I hope we work together on this. </span>
* <span style="line-height: 1.5;">JG: I hope we develop a better process. It would be useful if the project level research teams were more involved in the IFPRI work at a much higher level. For the research teams, ultimately it would be essential. Likewise what is essential is that IFPRI has been mostly focusing on that high level and we need some landing into the project level to get a better sense of the specific research projects on the ground and some insights to inform that high level perspective and pulling this together in a more coherent way. </span>
* <span style="line-height: 1.5;">MB: In Babati there was some landing. One of the problems is that IFPRI hasn't had people on the ground. If they had been in place, this would have worked out. The impact study was interesting to all of us. </span><br/>


<br/>
''<span style="line-height: 1.5;">What does it take for project teams to reach the 30.000 ft perspective?</span>''
* <span style="line-height: 1.5;">JG: We have a retreat planned for team leaders to come to DC and that's a perfect example of how that higher level vision can be developed from the details of research on the ground. Perhaps IFPRI is not on the ground enough but perhaps project leaders are not up in the air enough to discuss with IFPRI.</span>
* <span style="line-height: 1.5;">CA: IFPRI has a different structure than other CG centres. We have centres on the ground but not staff administering interventions. We work with other CG centres but using different methods. We found it difficult for IFPRI to advertise the positions on the ground. It is difficult for us to advise on something that we don't know anything about (agronomy). We had good candidates selected for the program but the IFPRI recruitment team selects them based on papers published in economic journals. I was trying to push for someone with their feet more on the ground. We are looking at things in a different perspective. We don't have enough info at this stage of the project to show adoption.</span><br/>


<br/>

''<span style="line-height: 1.5;">PMMT is a tool that helps track progress through different activities. In what ways does the current version of the PMMT suit your needs and what changes would you suggest?</span>''<br/>
* AL: It's the first year that most projects are using it and there was a training to some of the partners in West Africa. I don't think people understood most of the things that it was supposed to do. In the future we need to find ways that some of our national partners can also use it. The feedback we had so far was that the system was not user-friendly.
* PT: I'm a big fan of this kind of tools in principle but there's a number of issues leading to rubbish-in / out. We have to generalize the data; Internet connectivity is a problem so we need to think about offline solutions too; A number of these tools are developed - to what extent could we have built upon what is out there as opposed to building our own tool; They are great tools and external observers are really happy with this type of tools but it's difficult to update these tools adequately.
* MB: Some of our people think it's an 'extra' responsibility to input data into that tool. Our partners are highly involved and they see anything new as an extra piece of work.
* CA: We designed the PMMT to link into existing technologies. Connectivity: Many times a week we have calls to upgrade the speed of the PMMT but there's a lot of data to be uploaded. There are technical limitations that we cannot overcome but we are addressing issues constantly. Next year the PMMT needs to be fully integrated into ILRI CKAN.
* JG: I see 2 problems. There's a great range of skillsets as to who enters data and there's issues with accessing the system to enter data. We'll have to commit resources to ensure that the data is uploaded, properly. Perhaps data facilitators in each research team could address the conditions in which the teams enter data. Somebody working directly with the scientists that have the data.
* PT: That's a really good idea and could be one of the roles played by the IFPRI staff on the ground.
* JG: If IFPRI were ready to take that challenge that yes but it is ILRI and IITA that have to take that responsibility. IFPRI don't have leverage over the scientists on the ground.

'''<span style="line-height: 1.5;">Questions from the audience</span>'''<br/>
* Irmgard: Re: IFPRI's role on project management. To a good extent you are already involved, as IFPRI sits in the project and program committees. What do you understand by project management may affect this but you already have a role. Bridging the gap of ground and 30000 feet, we had a discussion last year with the IFPRI M&E coordinators on the ground and we agreed they would be fully part of the research team. There was no control but there was an impression that they didn't want to be included in the research teams (that was probably a personal thing). They are the link between the DC team and on the ground research teams.<br/>
CA: Yes and no: When our coordinators were on the ground they were consultants and we couldn't enforce their deliverables. They left for various reasons but the best candidates according to IFPRI's usual skillset are hopefully going to stay with us but I fear there is a disconnect between what this person expects in terms of research and what s/he will find on the ground.<br/>
IHZ: I understand there are these institutional restrictions for who you recruit. But if IFPRI doesn't have the person we need, it won't help very much. That person helps the program, not first and foremost for the institution. This is a unique project, it is complex and it requires flexibility from the institutions to find relevant candidates. I thought the reason for recruiting these on-the-ground specialists was to help the program not the personal researchers' agendas.<br/>
CA: Yes and at interviews we make sure that the candidate understands they will be on the ground. This is the same tension between IFPRI and others on the PCT team. When we have to evaluate proposals for the next year, we cannot really assess them because agronomic trials are not our expertise. I cannot decide how to evaluate these complex agronomic trials. The research teams shouldn't also judge what IFPRI is doing.<br/>
Moffat: There will be new opportunities to balance issues out around policy updates. There will be an opportunity to connect others through an open data policy. By contract, all USAID grantees will have to collect and share (qualitative) data.<br/>

===<span style="background-color: #ffffff;">Africa RISING Program evaluation design, spatial distribution of Africa RISING technologies, farmer typologies</span>===
*[http://www.slideshare.net/africa-rising/ar-me-grootnov2014 See presentation of Jeroen Groot on farming system analysis] <br/>
* [http://www.slideshare.net/africa-rising/ar-me-hailenov2014| See presentation of Carlo Azzarri on the program evaluation design]<br/>
* [http://www.slideshare.net/africa-rising/ar-me-azzarrinov2014 See presentation of Carlo Azzarri on farm typology]<br/>
* [http://www.slideshare.net/africa-rising/ar-me-groot2nov2014 See presentation of Jeroen Groot on farm typology]<br/>
Use of typology for targeting, selection, scaling up and scaling out...


<br/>
'''Q&A''':<br/>
Q: Your typology seems to be entirely based on cereal production - how can that be. This typology is meant to be multidisciplinary.
* A: The table with the 6 variables etc. shows an increasing typology. Labour and other variables also increase with the typology. Cereal production is a driving variable. This is suggested by our agronomist colleagues.
Q: How did you select variables? Did you arrive on these important variables
* A:
Q: Is there any plan of looking at other variables which are important for doing a functional typologies apart from the structural ones?
* A: Yes we can change them as much as you want. We are able to go from structural to functional typologies and therefore plan with the scientists on what you want.</span>
Q:In your results are you talking about hiring in or hiring out?
* A: This is hiring in. This is percentage of household for each typology hiring in labour.So the idea here is that the richer you are (better off you are) the more likely you are to hire farm laborers<br/>


<br/>
''See presentation of Carlo Azzarri on ARBES survey in Malawi (coming soon)''
Q: How did you choose the controls?
* A: In the case of Malawi the site was selected because the MSU was working in the sites before. The controls were selected randomly within the same agricultural intervention domain
Q:
* A: The intervention started with the inception meeting in early 2012 and then quick wins were implemented for a year so we can't talk about the influence of Africa RISING. We are still calculating how much of the impact is due to Africa RISING or other interventions... We can control for it but this impact.
Q: Female-headed hhs have better access to markets - is that because they target different markets?
* A: That could be a possible interpretation, but it’s a good question to investigate further.
:- Comment: We're looking at multiple scales and are now proposing four different scales.


<br/>
'''See presentation of Cleo Roberts on ARBES survey in Tanzania (coming soon)'''<br/>
Q: You just selected one representative domain?
* A: Now we just used one indicator for each variable. Here we are standardized the households and the distribution is across the variables to be able to compare scales.
Q: The radar scales seem to indicate effect but there are also endogenous impact factors. Can you remind us what factors are exogenous and what not?
* A: For the moment we are only talking about descriptors. These factors can be exogenous or endogenous. We have to separate these exo/endo variables to see how they match households. If we compare similar households perhaps these results would change.
* Comment Peter: This question is very important but this is very hot off the press.
* Comment Jerry: I would be a bit more cautious because we have identified proper indicators - but we can visualize multiple statistical models within the same graph.


<br/>
'''See presentation of Carlo Azzarri on ARBES survey in Ethiopia'''  ''(coming soon)''<br/>
Q: Your economic variables, are they per household or farm size (per Ha)?
* A: ??? It's longer term.
Q: Did you have control farmers for Ethiopia?
* A: No we didn’t have control farmers for Ethiopia. We are thinking of using secondary information to compare


<br/>
'''See presentation of Cleo Roberts on ARBES survey in Mali (coming soon)'''<br/>
In summary: ARBES uses can be for: evaluation, long term plans, generalizable conclusions, limited monitoring.<br/>
:- Comment Cleo: In Sikasso, there is an overwhelming majority of female-headed households which skews the results
Q: How do you assign individual fields to the household data sets vs. larger compounds?
* A: We ask in the questionnaire who has the responsibility over this plot/crop etc.
:- But I'm talking about certain fields that can easily be assigned to a household, while others should be assigned to a larger scale/unit/compound
:- We asked about labour given to people outside their household and inside it. And we derived the benefits from those fields. Some respondents share their plots etc. so it was sometimes difficult to assess their benefits and contributions...
Q: I see that in many of the sites the Africa RISING intervention seems to have been beneficial but this is a baseline - can you clarify?
* A: We know the results of baseline surveys. Some of this data is from 2012 and we are looking at variables from the past. It's an open question. In the case of Malawi, we are doing more research to unpack the possible biases in the selection of households.
:- Comment Jerry: The folks in Mali mentioned that we never fully took care of looking at that issue of households and larger compounds. Perhaps modeling will have to be evaluated separately. That was a looooong discussion on the first year.
:- The household definition in Mali took us a long time. We spent almost every training day bringing it up. We separated out the groups 'this is a household, this is a household etc.' We tried to match and see how they differ. Some things have not changed due to the project (e.g. assets they had prior to the project, some kind of land etc.)...
:- Comment: In Southern parts of Mali, there are very different farming production systems.
Q: Are you expanding after the project or keeping within it?
* A: To generalize you need a statistical approach to calculate the characteristics of the beneficiaries of AR. We put a benchmark on them. With scaling up we need to look at farmers that have similar characteristics and see if the results can be generalized to similar farmers. We don't know whether that will work or not and it's time consuming.

=Group work: What information would you like the IFPRI team= 
==East & Southern Africa group== 
On the PMMT, it's generally agreed that work can be done to make it more user-friendly - an offline version would be great. This should be done by means of training on the tool and capturing indicators.
The baseline results: these are extracted from very large data sets and reports. IFPRI could invest a little bit more in producing country summary reports and make the reports user-friendly for research teams.
Clarify with research teams on the monitoring role. W<span style="line-height: 1.5;">e always hear with our partners that 'this is IFPRI's role - but it's not very clear, so let's sit together and clarify this with them.</span>

==Ethiopian Highlands group== 
Gaps and additions:
* <span style="line-height: 1.5;">A methodology to find the impact of 'before AR' interventions before the baseline</span>
* <span style="line-height: 1.5;">Need for disaggregated information on gender, based on beneficiaries, non-beneficiaries and control (much generalization)</span>
* <span style="line-height: 1.5;">To what extent the research kebeles/districts represent Ethiopian Highlands scaling?</span>
* Policy constraints on production / productivity as a variable
* Causality on the different factors (we need more correlation analysis?) these may not be independent variables between different farms.
* Comparative analysis among countries/projects.
* So much data has been collected but there was no presentation about <u>nutrition</u>.

==West Africa group== 
Gaps:
* Disaggregate data by district (Mali) or by region (Ghana), both economically and socially.
* Better monitoring of key indicators e.g. dietary diversity (in survey) - regular (annual?) monitoring
* Lack of M&E capacity in the region
* Collaboration with ag economists at IITA + M&E people (coordination challenges)
* Policy + economic aspects connecting with extension - reinforcing extension services
Additional activities:
* Linking households to compounds
* Capacity building on PMMT
* Linking with additional USAID (and other) survey data
* How to move from AR zone to larger FtF priority areas - extending info --> activities
* What relevance does AR info have for other projects / donors / priorities?

===Reactions from Carlo=== 
In the data management session we had last night, we agreed that all data need to be uploaded to the PMMT. The data will be available for sharing as draft for internal and external sharing.
On the feedback from the session:

<u>Re: West Africa</u>: We share the same concerns about the FtF indicators. They are not immediately comprehensible. We'll be able to provide training on the PMMT and on the FtF indicators. We'll be able to link up information about compounds and households. We can disaggregate information by district and region as requested. We are aware of the differences in Ghana. On nutrition, we collected (with the survey) info on food consumption so we can collect scores, nutritional diets, calorie intake etc. but we have to decide if we'll be using these as custom indicators. Expansion?

<u>Re: Ethiopian Highlands</u>: We discussed a lot of this already. In 2012 we abandoned a few things that were discussed. In 2013 activities started in the sites and we weren't there but a lot of the variables matter after 10 years' time so one year of delay won't impact this too much. Hiring someone in situ is in the pipeline. We are trying to evaluate the framework etc., not just a single project or a single scientist.

<u>Re: East & Southern Africa</u>: Yes we want to make the PMMT more user-friendly. It's not in its final stage. Training yes and someone will be based here to provide this. Country results: someone is looking at gender overview, basic tabs etc. but I can't imagine it will be useful for everything so we need to work around the ''specific'' requests as they come through, not on the non-specific ones. M&E: We are entirely responsible for evaluation but monitoring is everyone's collective responsibility.


=Presentation by Moffat Ngugi= 
We work in 18 countries globally, 12 of which are in Africa and a few coincide with those from Africa RISING. We are looking for synergies.
Three points I would like to emphasize as opportunities:
# <span style="line-height: 1.5;">Policy updates in USAID - A new directive will require every new contract and engagement to report back data that have been collected, as of October 1, 2014. That will be a big treasure trove for information. If we define what information is most useful for Africa RISING we need to use that opportunity. This is also a broader commitment from the British government, the World Bank, the BMGF etc. By default we are open. We know that the CG system is having that discussion about incentivizing open data.</span>
# <span style="line-height: 1.5;">An executive order from President Obama as of September 23 about climate resilient development. The portfolio of USAID has been ordered to consider what the key climate change pressures are going to be and what options for adaptation and mitigation are - this is a right opportunity for Africa RISING to look into sustainable intensification's implications for food security etc. This also presents opportunities for climate-proof work / initiatives.</span>
# <span style="line-height: 1.5;">Climate-smart agriculture: there was a global forum launched on African climate-smart agriculture alliance under the auspices of the AU and CAADP. CAADP was launched over 10 years ago and this year was about agriculture. Heads of states of all 54 countries met in Malabo and talked about climate-smart agriculture. They agreed on pursuing this and that's a good opportunity to show the transformative impact that the Africa RISING program could have in informing that. This will be the marching order on which each country on the continent will work related to the agricultural investment plan (2015-2025). We are looking at increasing productivity and income, improving resilience to climate change and adapting to climate change.</span>
I want to encourage everyone to be in touch and share opportunities to engage with each other. We don't really link sufficiently with a lot of FtF efforts. Africa RISING is liaising with Nafaka etc. but in most other places we have opportunities to expand the impact of the research you're carrying out.
Think about the recent FtF agenda. How are we going to think about larger boundaries than those from Africa RISING? I invite you to be aware of that. There are various follow-ups we can think about:
* Linkages between USAID FtF activities. We are frequently in touch with each other and if sthg is going on in Babati we relate feedback to what is happening here. We can clarify ways to strengthen linkages.
* Data management: there's now a requirement to share data. We are looking at which data are scalable etc.
* New investments for the Sustainable Intensification Innovation Lab with CIAT etc. We want to use the whole set of FtF countries in Senegal, Ethiopia, Bangladesh etc. We can influence the design, implementation, monitoring and evaluation...

<span style="display: block; text-align: center;"><u>'''Day 2'''</u></span>
=Ex-ante analysis of crop outcomes= 
===The CSISA experience=== 
[http://www.slideshare.net/africa-rising/ar-me-spielmannov2014 See presentation by David Spielman on CSISA]<br/>
There are a lot of lesser-known, harder to measure and unobserved constraints (informational constraints) and individual preferences. Your learning approach might not be the right one... And we need rigor in our design.

'''Q&A'''<br/>
:-Comment: Africa RISING is at a different stage than CSISA and it seems you are more focused on extension and development. It's great that Africa RISING can learn from CSISA. What skillsets do you need to learn about learning?
A: Our NARS partners are quite weak on this kind of methodologies and that's a struggle for us. We brought in some very good partners from Univ. Davis, Georgia etc. We have 1.45 FTE working on this out of 25 people working. You have to find collaboration outside to work on these things. SIMLESA etc. also have the same partnerships.<br/>
:- Comment: Your research on redundancy of diagnostics - you challenged yourself to look for less tangible aspects of adoption. A lot of projects are measuring things that we measured before and we need to beef up our diagnostics.
''' Yes, we need to work on diagnostics. We sometimes work on our research questions and our diagnostics reveal that our research questions are not correct.
Q: You talked about extension approach work etc. This is RO3.
* A: There are some things that we are not doing e.g. Ex-ante modelling was lost. Evaluating extension approaches is our dream e.g. 'try chalk and talk here, demonstration plots there, mother-baby trials there etc.' and I want to find out what has been helpful. We have hypotheses on some of that work. This is a research project so we need to learn from what we're doing.
Q: On scaling up, CSISA and AR are such a small segment of what needs to happen. How would you compare the 2 projects around policy level engagement?
* A: It's complex. We try to pitch our work at policy level. You should have some real policy domain engagement and extension engagement strategies. CAADP is one of the best channels to do that through.
Q: Governments and government extension... in Mali there's no governmental extension. Apart from telling them 'you should do extension', how do you suggest going to scale?
* A: Turn back to the CAADP process and ask why there's no extension. Ethiopia has one of the most extensive extension systems.
Q: RCT approach to evaluation but RCTs by definition use a baseline survey. How have you managed that?
* A: I use surveys. We evaluate smaller areas of work but we use surveys also. A baseline survey of an entire project is huge and probably not advisable.

==Ex-ante modeling analysis. Maize yield responses to nutrient management options in Malawi== 
[http://www.slideshare.net/africa-rising/ar-me-koonov2014 See presentation by Jawoo Koo]<br/>
Action sites Dedza and Ntcheu.

===Q&A===
:- Comment: What will IFPRI do for us and what can we do with you? You and Bekele should talk together about this work.
:- What we do with action sites is looking at modeling tools etc. There are many things we can do together.
Q: You should use the data we have to validate this. We have sufficient data for treatment site ex-ante analysis. In Malawi, we started a similar approach and we should work together on the formats.
Q: In the maize yield/maize-cowpea rotation, is that the result of every 2 years? (Yes). You get yield of 4 tons/Ha without fertilizer, is that realistic?
* A : We are simulating and something could be wrong there, we'll have to look into this. We can correct these data. If we look at a several years' time period, 4 tons/Ha. The first few years of the research should not be considered.
Q: To what extent can you see any of Eric Crawford's data in Zambia. It would be good to compare these.
* A : I haven't seen the ICRISAT data. Another project is indeed comparing crop models etc. and they are looking at different agro-ecologies and different modeling tools to inform system stability etc. over time. This presentation is also premature compared with the quality of our research publications.

==DAHBSIM Dynamic Agricultural Household Bio-economic Simulation Model== 
''[http:''www.slideshare.net/africa-rising/ar-me-komarekinov2014| See presentation by Adam Komarek]//
Q&A
* '''Q''': What is the relation between production and consumption? How do you deal with that?
* '''A''': The model is capable of looking at production systems and consumption systems.
* '''Q''': This looks like a proper model for sustainable intensification. Where does the livestock component get integrated here?
* '''A''': We have a separate livestock module and 3 different levels of production (extensive, semi-intensive and intensive). We have a series of standard calculations (nutrient demand) which calculate average nutrient supply and the model says how many animals you can have in that system. We choose the production level that we think matches the system but we don't do it at a continuous level of growth.
* '''Q''': What are the cost benefits of using this module? There are sensitivities around rainfall etc. so how do you deal with these. How do you reconcile different scales for models to remain sensitive?
* '''A''': In a month's time we'll have more data and we can talk to the agronomists to build that module.
* '''Q''': What is the decision-making concept. You don't mention the other factors determining farmers/households expectations. Is the decision-making seems to be based around farmers, not households, so is that not biasing the results? If you model doesn't explain that, what is your plan B to include non farmer-based but household-based decision-making? There are lots of decision-makers in a household.
* '''A''': The model is calibrated on the basis of how farms work. I'll have to look into this more and we still need to do more farm analysis. We need to simplify the model.
* '''Comment''': The best models I've seen are when they are integrated with various disciplines and you get opportunities with such projects as this.

=Africa RISING external evaluation results= 
''(This was a verbal presentation by Christine Okali)''
Three weeks, three people, covering Drylands, Humidtropics. Big challenge for us because we had to look at very different projects but we saw some coherence.
At the time of the review there was no logframe and the program framework was not finalized. We learned it the hard way. This type of documents structures our thinking. But as a result we didn't have a clear vision of what was going on (especially in Mali).
Platforms and partners, work packages and management were looked into.
The final draft was submitted last Friday night and that will be available online. It is not yet read by most people.

Platforms: they're central.
Ghana: 2 platforms operating, one-two meetings. In Mali the point was made 12 months ago but still no meeting.
Results are just quick wins really.
Recommendations: concerns about farmer participation. It has to be part of future research.

Partners: very very complex. CGIAR partners are many. 2-3 of them have only one person's time.
ICRISAT has its head office in India, which has implications for contracts, project time etc.
National level partners: they played a major role in Mali's business on the ground. I like their ag extension work and they are also sitting on the platform. There were some problems in Ghana viz some national partners (related to budgets, work plans, deliverables). In Ghana, because the national institutions are on site it works much better whereas in Mali there's just NGOs (and you need to contract, finance them which is labour-intensive, it's not straightforward etc.
In Ghana 1-2 partners are not yet contracted. Some partners from USAID are necessary.
Problems: The farmers complained that CG centres come at different times etc. (in Mali particular). This has to be improved, resolved, to make it easier and to reduce money wasting.

Work packages: Supply- rather than demand-led. The idea of quick wins in year 1 led to problems with villages that saw the project leave.
Impressive on the ground trials, mostly in Ghana but the country is very different to Mali, where people are scattered widely etc. In Mali you get 5 people discussing with you as opposed to having lots of people to interact with you generally in Ghana.
Impressive and palpable interest for technologies.
Farmers wanted and were waiting for seeds (in Babati). All CG partners together in the demonstration sites, technology parks in Ghana etc. were seen and that was a useful place/approach. That would be difficult to implement in Mali. It is difficult to do such trials in Mali and to find land to do that. Land access for trials is an issue.
Livestock integration was difficult.
CG not together in Mali so we would visit the work of ICRISAT one day, ICRAF another day etc. whereas we were trying to present an integrated research approach with multidisciplinarity. This last aspect just <u>has to</u> work. In Mali, there has been much disorganization due to change of government etc. so the situation has been challenging. Very early days in some ways but the roll out has been done in others. Scientists have to follow up.
WP1 (socio-economic work): not much done. The work has been done to establish that they work but there's very little done. In some ways partners need to pre-finance the activities and some refused that. The economic work wasn't done so we couldn't tell farmers 'this is what you might get'...
Not much discussion about what households really look like etc. little done about socio-economic characterization. Mali seems to have much more of that.
Old men etc. all sat there when we arrived in the villages - it's a different scenario to interact with them. Almost no female-headed household in Mali whereas Ghana has one of the biggest concentrations...
Gender: not much there, but women's groups yes, especially in Mali where there's evidence that women/widows are subscribing to 'marrying to a family' not to a husband.
Policy was not addressed.
Wetlands not considered...
Nutrition - there was a problem of baseline. Cross-country program would be good.

There's national and international expertise in both countries and the two should cooperate more. In Mali it was much more about NGOs engaging etc. rather than focusing on national/local government.

Management: Contracting systems etc. we are wasting time. Team leaders mapped it all out to impress that it is hard work when contracts take so long.
How will you scale to national partners in Mali? You can't scale with NGOs. We have various recommendations for all those issues. That will be dealt with in the next 1.5 years... IITA is hosting Humidtropics and we concluded from all of it that if you think that Africa RISING has sthg to contribute, that relationship has to allow for your identity to be kept.

===Project monitoring and mapping tool (PMMT)===
[http://www.slideshare.net/africa-rising/ar-me-bacounov2014 See presentation by Mélanie Bacou]<br/>

=Data management plan= 
''(The data management plan presented by Carlo Azzarri).''

There are two types of data that we need to share: the first data are project metadata and the second the dataset metadata...
There are data access tools. The data management is compliant with CGIAR data access policies on data sharing, open access etc. which require.
The idea is that all researchers in AR adhere to these rules for future data sets:
* Datasets need to be uploaded by April 30 2015 with appropriate metadata.
* We distinguish between internal and external diffusion. Every external data set will be made available after 6 months. For internal data sets there will be no enforcement because we need to be able to share the data. In all cases, the team that collected the data retains the rights over the data for one year.
<span style="line-height: 1.5;">We will make the data sets available to CKAN. You will be sent instructions for how to do it.</span>
<span style="line-height: 1.5;">The data sharing agreement form will be available online and will be uploaded by 30 April 2015. For sharing, each of us (data providers) will receive an email from a requestor to upload data for module xyz etc. and we just have to say 'yes' and 'no'. So we can say 'no' but the requestor can report the fact that we are not sharing the data and that would be discussed. There could be instances when we share only if we collaborate together. </span>
Some of the rules will make the collaboration more prominent.
There is no legal requirement to share the data but it's a cultural issue.
For new contracts, the clause will be included in the contract upon signature, to explicitly endorse this data management...
These rules are clear, accepted and shared by CG Centres but we might have some issues with our national partners. If they're not willing to adhere to these rules their contracts will not be renewed.

===Q&A===
Q: At what point does the data collection period stop and the grace period starts?
* A: We have grey areas and this is one of them... We want to put the meta data and the trial data from the first year also. In any case the data needs to be available and uploaded. You can say no to this because it's part of the trial, or you can say 'yes but please use it this way or that way'.
Q: What type of data are you requesting? Qualitative, quantitative, raw data etc.?
* A: The raw data at micro level (at the lowest level possible). It's also about e.g. interviews... MP4, MP3s etc. The metadata about this should be made available also (location etc.). Data types are observational, experimental, simulated, derived/compiled/analyzed, physical, metadata...
Q: We're not expected to ship soil samples, are we?
* A: No.
Q: For survey tools, will we attach the tools (e.g. questionnaires) etc.?
* A: Yes, manuals, codes etc. so that other researchers can use the data we have. Sampling, evaluation design etc.
Q: We have CG Space also - to what extent do we try and avoid duplication?
* A: We need to talk about this with Peter Ballantyne.
Q: For research that involves wild animals, do we need to upload ethical guidelines?
* A: Yes we need to have a review from our ethical committee. We need the informed consent from the farmers about their participation in the survey. If there is any information that the requestor needs to have we may need to discuss and request additional information. This applies during the grace period. After that grace period everything will be open access but some confidential information will never be made available (e.g. GIS info about plots and households - that cannot be released).
Q: Who decides about this?
* A: It is not controlled - there's no system checking that all information is uploaded. But if I request information you uploaded and if you don't want to share it I can bring up the case for discussion.
Q: After the grace period everything becomes open access - does it mean that anyone can access or request that data.
* A: It's open and available to the world. We can track the users to find out who is downloading what. But we can have it either way. Some researchers are interested in tracking downloads. You can also control whether there will be a sign-up form. You can track the user.
Q: If Trials data doesn't have the geographic information of the plots, does it need to have other data?
* A: In the case of our survey, we can't release GIS coordinates but we can either offset the location by some radius using Demographic Health Services use, or release the information of the grid cell that this particular household belongs to... It's a 10x10 grid cell.


<br/>
<u>Reactions</u><br/>
* There is always a risk but it's going in that direction. Once open access it is left to the discretion of the user to use it correctly or not.
* In terms of anonymizing GPS coordinates for plots it's fairly time consuming. We have to format our data to make them anonymized. It becomes quite a lot of work...
''' Yes. In the case of our baseline surveys the name was anonymous but...

=Final staff evaluations and thank you's= 
I am happy. Nothing's ever perfect but there's been real progress and the IFPRI team should be congratulated on the amazing work they have been doing on this. You have a really nice data set and you can be happy about how to map some of these data fields etc. So all credit to you.

Compared with other projects we are rather successful. CSISA has the advantage of time. We are working on 5 countries with evaluation design. We have put a lot of efforts in the PMMT. I have put a lot of my time in this and in the survey. Still I think they're not going to solve the problems of this project because they are focusing on something else than what the project researchers are aiming at. We need to collect more data that will be useful to you e.g. soil characteristics etc. For monitoring, I don't think that it is in our mandate (IFPRI) to do monitoring. We hope to have a colleague based in Arusha soon to do his research and to help but that is also limited because he won't be judged on the indicators we use but on published research papers... The bottomline is that we need to redefine our roles and expectations. The megasite breakout session showed that we are expected to work on many things and all these many things have a time and cost implication. Our budget is closed. We spent more than anticipated. We are not here to evaluate anybody but to help you with data generation and carrying out your work with information.
Thank you for coming. You will hear more from us about the data side. We are expected to upload the data by April 30, 2015. You will receive an email from us explaining this and you will need to sign when you requested data. Is there any implication about the new rules of the data management plan? We will have to discuss this and we will be able to accept the consequences. We will apply these rules in a multi-institution project.












































[[Moneval3_organizers|  Organizers' page]]