Program strategy workshop Indicators

(This work stream was led by Cheryl Palm, Sieg Snapp and Phil Grabowski)

See the result as a fileFile:SI indicator tables by domain and scale - revised 7 Oct 15.docx

See the notes of the meeting: File:Notes from SI indicator meetings and Visualizing SI indicators.docx

See the SI indicator framework: File:SI indicators framework oct6 Malawi Africa RISING.docx

Introduction

The SI indicator framework is a decision making tool to improve probability of success for farm and farmer level impact. It aims to be realistic for donor investment and practitioner needs. There are two fundamental questions that, by using these indicators, we are trying to answer about sustainability:

1. How sustainable is this plot / household / community in comparison with other plots / households / communities?
2. If we change something, does the sustainability of the plot / community / household change and, if so, in what way does it change? By monitoring all domains relevant to a project we will be better able to detect synergies and tradeoffs and minimize unintended negative consequences.

It is also important to clarify what this framework is not aiming to achieve

• The framework is not intended to define or quantify ‘sustainability’ or pre-determine an ultimate state of sustainability or specific practices that lead to sustainability, but rather to guide decision making, based on evidence-based outcomes, resulting in agricultural systems with improved productivity, environmental, social, and economic outcomes.
• It is not intended to cover all dimensions or scales of sustainability but only those commonly focused on by agricultural R&D projects, and it is intended be amendable the development of standardized methods but flexible enough to be adaptable to different scales of interest.
• It is not intended to replace other frameworks used by individual programs or projects, but rather to provide a simplified, common framework that facilitates cross-program learning and assessment.
• It is not intended to replace adoption studies or identify the best means of up-scaling an intervention. Nevertheless in some contexts it may offer the potential to inform these efforts as well as to inform policy debates related to sustainable intensification.

Explanation for how the framework could be used The following tables show the indicators by domain that are likely to be important when assessing sustainable intensification. The metrics for each indicator are separated by the scale of measurement. Ideally researchers will critically examine the tables and carefully select the indicators and metrics that are most suitable for the specific sustainability assessment considering the unique aspects of the intervention and the bio-physical and socio-economic environment. Below each table we have started making notes about the conditions where each indicators may be applicable. The indicators most likely to be used across interventions and contexts are presented at the top of each table. The next step will be to develop a guide or protocol detailing how to collect the data needed for each metric. In that section we hope to provide the “gold standard” method for collecting the necessary data as well as potential proxy indicators that may be more feasible. The researcher carrying out the sustainability assessment will have to decide which metrics are worth measuring robustly and which metrics can be estimated more coarsely. We will aim to describe the conditions where the proxy may be useful and where it should not be used.

Productivity

Concept	Field/plot (NPP)	Farm	Household	Landscape or Administrative Unit
Yield (partitioned by species and tissue type) and residues (total NPP)	kg (yield, fodder, residue) / ha / season kg (yield, fodder, residue) / ha / year	kg (yield, fodder, residue) / ha / season kg (yield, fodder, residue) / ha / year	N/A	Net Primary Productivity (above ground)
Fodder production considering quality	Crude protein production / ha Metabolisable energy production / ha	Crude protein production / ha Metabolisable energy production / ha	N/A
Animal productivity considering land area	N/A	Animal production (milk yield, weight gain, meat) / ha grazing and fodder land used Animal product/farm/year		Net commercial off-take relative to the total grazing and fodder production area
Variability of yield and animal productivity (over time and space)	Coefficient of variability, distribution, etc. of representative sample	Coefficient of variability, distribution, etc. of representative sample	N/A
Livestock herd composition	N/A	Heifers per cow Calves per cow Oxen per TLU	TLU/ha of farm
Yield gap (use attainable yield or yield target)	Target – Actual (kg / ha / season)	Target – Actual (kg / ha / season)	N/A	Mean yield compared to maximum observed yield in similar location
Cropping intensity (annual count)	Number of crops grown per year (per crop) Plant population density (seeds/ha/season or seeds/ha year)		N/A	# crops grown per year across a landscape Plant population density of crops within a landscape

Notes on productivity indicators:

• o The focus in on yield, partitioned by species and tissue type, in kg/ha/season and year. At a minimum, a better standard for measuring productivity is needed, especially for research projects, and may best be examined by partition among crops to have commodity specific measures of yield.
• o The productivity of the land needs to be assessed in terms of all that is produced (not just grain yield), especially where fodder production is significant or where fruit trees are mixed into the landscape.
• o Comparisons of productivity across crops (e.g. cotton vs. maize) are difficult unless they are put into a common unit that is meaningful for farmer decision-making (such as local currency if both crops are sold). We need to look into the feasibility of using farmers’ subjective valuation of each crop to assign weights if a large portion of the crops are not sold.
• o Farmer recall of harvest is pretty reliable but farmer estimates of area is not very accurate. Measuring is important.
• o Fodder production amounts need to be adjusted based on the quality of the fodder
• o Animal productivity must be broken down into all the parts of the animal that are used (including manure). Estimating the land area used for grazing or fodder production is likely to be difficult or imprecise.
• o Net commercial off take = total sales of region – total purchase into region divided by region herd size
• o Yield variability will be more easily interpreted as a percent of total production. It could also be used to estimate the probability of falling below food self-sufficiency or of achieving a production target (break-even point for a cash crop).
• o Livestock composition is one way of observing if livestock production is intensifying. A herd with more cows is more commercial as the heifers and calves are more retained by those holding the herd for savings; below 4 TLU per ha can feed on farm
• o Yield gap allows for comparison across agro-ecological zones by converting productivity information into the ratio of actual production to the highest observed production in similar conditions. There are limitations to inferences that can be made from this due to strong assumptions about the feasibility of reaching the highest yield.
• o Cropping intensity is likely to be important to monitor where the intervention affects the likelihood of irrigation or planting during short season rains.
• o Seeding density can be over space and time, more crops and more/less of each crop - it is difficult to interpret yield and SI without knowing population density. So this should always be measured, either as seeds planted and/or ideally as plant population density at harvest.

Concept	Field/plot	Farm	Household	Landscape or Administrative Unit
Profitability	Profitability (output sold X price – input costs)	Enterprise budget for livestock Profitability for all crops	Total agricultural profits per household	Contribution to regional or national GDP
Variability of profitability	Yield Risk Livestock mortality Price volatility	Probability that profits < threshold	Probability that profits < threshold	% hh in community with profits < critical level
Income diversification	N/A	Diversification index for all marketable agricultural activities	Diversification index for all income activities
Input Use Efficiency (water, fertilizer, etc.)	kg output / unit input
Limitations of land, labor and capital	Net returns per unit labor input, land input, capital input	Net returns per unit labor input, land input, capital input	% of total labor represented by the peak % of total expenses represented by the peak (Cash flow constraints)
Poverty rates (or estimates)	N/A	N/A	Assets, ability to mitigate losses from any one activity with another	Poverty head count Ag or rural wage/staple food price index Household expenditure
Market participation	N/A	% of production sold (by crop, animal product)	% of total income from agriculture	% households selling an agricultural product
Market orientation	N/A	% of land allocated to cash crops	% of total consumption from own production

Economic Notes on economic indicators

• o The focus is on profitability, variability of profitability and productivity of inputs that may be of critical importance in a particular context.
• o Focus on profitability instead of gross agricultural income (cost/benefit analysis, per enterprise, crop etc.) as the former can be easily linked poverty
• o Market orientation and Market participation:Market orientation measures the degree to which households produce for sale whereas market participation index measures the degree to which households participate in market as suppliers of their own produce. Bothmeasure the degree of commercialization. Commercialization is important to monitor for sustainable intensification because it can have direct links to food security, equity, nutrition and investment in further productivity. Furthermore in some contexts commercialization is a concern as it exposes farmers to price volatility risk. For more details on market orientation see Thom Jayne’s work on a commercialization index
• o Returns to labor is related to income which can be linked to consumption and wellbeing.
• o Input use efficiency: this is especially important to measure where a specific input is limiting (such as water in some contexts). However, in general it will be better not to over emphasize it but rather focus on output per unit of land (as is done in the productivity domain).
• o Poverty rates at various scales have been suggested but may be difficult to measure as poverty is multidimensional. Some of the variables that may be included are: Food consumed at home, food purchases, in-kind food consumption, household demographics (hh size, age, gender), non-food purchases, ownership of durable and productive assets, housing construction, household rentals, asset rentals
• o Aggregate to higher levels
• o Bio-economic household modeling would be a great tool for looking at profitability, labor limitations, etc.

Environment Part 1: Vegetation and water

Concept	Field/plot	Farm	Household	Landscape or Administrative Unit
Vegetative Cover	% bare ground (length of time per year) Months of living cover % cover of noxious (invasive) plants	% vegetative cover and % tree cover (end of wet season, end of dry season, per year)	N/A	% vegetative and tree cover (end of wet season, end of dry season) % of animal feed coming from landscape (not farm) Spatial arrangement of feed sources in landscape
Plant Biodiversity	# species – agro-diversity and other diversity Shannon Diversity	# species (agro-diversity and other diversity) Shannon Diversity	N/A	% cover of natural habitat for rare species (conservation) % cover invasive species
Fuel	% residues used for fuel	% of manure used for fuel	Time spent obtaining fuel % of household fuel coming from farm	Spatial arrangement of fuel availability % of fuel coming from off-farm landscape
Water availability	# days without adequate supply of water (rain, irrigation) for crop growth; # days with too much water (number of days waterlogged during critical crop growth)	Depth to shallow ground water;	# months without adequate supply of clean drinking water within 500m	% hh with year round access to drinking water % of livestock farmers with year round access to water % of irrigable land (given current investment) with sufficient irrigation water % of stream flow not diverted for agriculture or drinking water
Water quality	Water salinity level NO3 level Zoonotics - #/ml	N/A	Number of months without adequate supply of clean drinking water within 500m	% of water sources (wells, streams) with clean water

Concept	Field/plot	Farm	Household	Landscape or Administrative Unit
Erosion	Estimate changes in components affecting soil loss C factor (crop type, tillage) P factor (practice to reduce erosion)		N/A	Erosion (t/ha/yr) (MUSLE or RUSLE)
Soil C	% C at each soil depth Carbon budget - Organic matter quantity and quality of inputs (to calculate C)		N/A
Soil acidity	pH and % Al saturation if pH <5	N/A	N/A
Soil salinity	EC (electrical conductivity)		N/A
Nutrient Partial Balance	kg N,P, and K inputs (fertilizer, manure, etc.) less kg N, P, and K in total biomass removed (harvest, grazing) per hectare per year	kg N,P, and K inputs (fertilizer, manure, etc.) less kg N, P, and K in total biomass removed (harvest, grazing) per hectare per year	N/A	kg N,P, and K inputs (fertilizer, manure, etc.) – kg N, P, and K in total biomass removed (harvest, grazing) per hectare per year
GHG Emissions	CO2 equivalents per hectare (also broken down by CO2, CH4, and N2O).	CO2 equivalents per hectare (also broken down by CO2, CH4, and N2O).	N/A	CO2 equivalents per hectare (broken down into CO2, CH4 and N2O)

Environment part 2: Soil and pollution

Notes on environmental indicators

• o Each indicator is dependent on scale and time, and need to remain simple despite the potential losses in nuance in remaining simple. The indicators can be specified per context.
• o Generally, the percent cover and percent tree cover are inclusive of each level despite needing slightly different metrics between farm and field. They are inclusive of both perennials and non-perennials. As scales increase, species diversity is of greater concern.
• o Nutrient balance can be particularly useful to avoid over application of nutrients and the environmental problems associated with it. It is less useful for diagnosing nutrient deficiencies. Phosphorous availability depends on soil type and organic inputs and its recycling is just as important as input and output balances. Nitrogen requires considering amount of biological nitrogen fixation by legumes. Potassium in most systems is adequately available through mineralization of parent material and its importance depends on the particular crop (such as bananas and tobacco).
• o Erosion is often measured at landscape level but can be adapted for use at the plot level. Potential soil erosion loss is commonly assessed with the Universal Soil Loss Equation (USLE) as follows:

o A = R x K x LS x C x PWith: o A representing the potential long term average annual soil loss in tons per acre per year. o R is the rainfall and runoff factor. The greater the intensity and duration of the rain storm, the higher the erosion potential. o K is the soil erodibility factor. It is the average soil loss in tons/acre per unit area for a particular soil in cultivated, continuous fallow with an arbitrarily selected slope length of 72.6 ft. and slope steepness of 9%. K is a measure of the susceptibility of soil particles to detachment and transport by rainfall and runoff. Texture is the principal factor affecting K, but structure, organic matter and permeability also contribute. o LS is the slope length-gradient factor. The LS factor represents a ratio of soil loss under given conditions to that at a site with the “standard” slope steepness of 9% and slope length of 72.6 feet. The steeper and longer the slope, the higher is the risk for erosion. o C is the crop/vegetation and management factor. It is used to determine the relative effectiveness of soil and crop management systems in terms of preventing soil loss. The C factor is a ratio comparing the soil loss from land under a specific crop and management system to the corresponding loss from continuously fallow and tilled land. The C Factor can be determined by selecting the crop type and tillage method that corresponds to the field and then multiplying these factors. o P is the support practice factor. It reflects the effects of practices that will reduce the amount and rate of the water runoff and thus reduce the amount of erosion. The P factor represents the ratio of soil loss by a support practice to that of straight-row farming up and down the slope. The most commonly used supporting cropland practices are cross slope cultivation, contour farming and strip cropping

• o

Social

Concept	Field/plot	Farm	Household	Landscape or Administrative Unit
Equity (Gender, Marginalized group)	N/A	% labor for power vs. control activities* Rank drudgery from 1 to 10*	Women Empowerment in Agriculture Index Access to production factors (mechanization, land)* Decision-making about production, marketing (by crop)*	Variability and distributions of productivity, income and assets*
Level of social cohesion	N/A	N/A	hh participation in community activities hh receiving help from community, other community, government (bonding, bridging, linking) Level of integration of family activities	Active farmer groups Active innovation platforms % of community participating in some form of social group
Level of collective action	N/A	N/A	hh participation in a collective action group	# of collectively managed resource groups that are functioning beyond just existing to meet regulations # cooperative marketing groups # labor sharing groups # problems addressed by innovation platforms Conflict resolution measures Capacity of these groups to organize (low staff turnover, transition plans)
Conflicts over resources	N/A	N/A	Level of dissatisfaction with household decision-making	# of conflicts over resources

• Disaggregated by gender, age, ethnic background

Notes on Social Indicators:

• o Note from San Jose: The focus depends on the definitions of each indicators despite that the social indicators look at the impact of SI is for gender equality. For example, the goal is to evaluate if SI improves women’s access and use of resources in ways that improve their household and community decision-making power. However, evaluating such a relationship still require definitions of terms such as equity. It may be important to finalize the social theory of change for SI to finalize indicators.
• o The Women Empowerment in Agriculture Index survey process may be too demanding for many programs but we will be looking at how to select components from it that can be selected as needed.
• o The social cohesion, collective action and conflict indicators need to be developed in terms of the feasible metrics

Human Condition

Concept	Field/plot	Farm	Household*	Landscape or Administrative Unit
Nutrition	Diversity of crops grown (disaggregated by consumption versus sale) kg protein / ha Micronutrients g/ha	Availability of diverse food crops from own production kg protein / farm Micronutrients g/farm	Dietary diversity (individual and household) 24 hour and 7 day recall of protein, iron rich foods and Vitamin A rich foods. # of children not meeting minimum acceptable diet and minimum meal frequency Weight for age/height and height for age Uptake of essential nutrients (Zn, Fe, Vitamin A, Calcium)	Rate of stunting and wasting (large scale development project only) Market supply of diverse food Landscape supply of diverse foods (natural areas – not on-farm)
Nutrition Awareness			# of members with adequate knowledge on breastfeeding, complementary feeding, Vitamin A, Iron, Iodine, and other essential nutrients.	% hh with adequate nutrition knowledge
Food Safety	Pesticide application #/ha and toxicity/ha		Food safety and toxicity (resulting from production, processing and storage) Aflatoxin contamination of key crops consumed by household (toxicity units per gram) Pesticide contamination
Food security	Calories/ ha	Calories/farm (potential food self-sufficiency)	Months without food in hh Meals/day across seasons Food security index (various)	Total food production Total food reserves Infrastructure (e.g. warehousing, access to markets/roads, irrigation; dependent on geography) Enabling trade policies
Capacity to experiment		# of new practices being tested		Number of farmers experimenting

• Disaggregated by gender, age, ethnic background

Notes on human condition indicators:

• Note from San Jose: For the human domain, population level data is possible across households and communities, through census style survey. Farm-level data collection is possible if using the same collection tools towards the same measures and indicators. Basic assumptions: Availability of diverse foods à Equitable Access (at markets)à Uptake of nutrients (level of disease and health e.g. suffering from worms or aflatoxins then can’t take in nutrients).
• For micronutrient production use look up tables of nutrient content based on kg crop produced
• Anthropometric measures are only suggested for large scale development projects with sufficient time and sample size to detect this
• We have yet to fully explore all of the relevant food security index tools. For example there is a FANTA tool for FS assessment, FAO also has one
• Farmer capacity is a recent addition to the framework and we have yet to explore the best ways to measure this

Notes from the work stream

Notes from 7 October 2015 AM discussion on SI indicators

Human Condition Diversity of crops – grams protein per ha – also micronutrients important (Zinc content)

• o Vitamin A, Iron,
• o Grain content on plot

No need to measure stunting for research (would be useful for development project) - Minimum 3 years and 500-1000 samples for accurate power - Few agricultural interventions would reduce stunting - Anemia and Vitamin A status – possible to measure but expensive and requires IRB Better to measure capacity (knowledge and practices) – awareness of nutrition (look for standard questions) Quantifying nutrient need for a particular household and match with farming systems. - Kg protein per person per household – hard to get measures, proteins not usually the most important - Micronutrients more important – Daily requirements of hh compared to annual production of vegetables - Use look up tables for quantities in each crop Diet diversity is gold standard – requires continuous monitoring – baseline 24 hour recall, then needs to be done in each season, Fred – did it another project 4 times per year, - Intrahousehold – days in the past week by person – how many times consumed from each group - Changes in diet diversity – will they stem from new production of nutrient rich crops or from increased income coupled with nutrition education; - Monitoring this could also be used to catch negative effects from intensification on diet diversity - Qualitative interviews on how project is affecting nutrition may also be simpler for detection - Consumption diversity NOT just production diversity: Often nutrient rich foods are sold to the market; nutrition education and behavior change needed in this context Landscape level food system analysis - Proximity to forest important for nutrition (fruits) – part of the food system, - Closer to market also has improved nutrition, valley of death in between with only grains - Lots of tradeoffs due to impacts on natural habitat from agriculture - Some projects like agro-forestry specifically try to address this - Add landscape level indicator on landscape supply of diverse foods (non-farm) Food safety and toxicity - Aflatoxin risk for many crops; Grinding tef can lead to iron toxicity - We need to add indicators related to food safety and toxicity that stem from production, processing and storage (metrics include amount of harvest with aflatoxins, pesticide levels, iron toxicity)

Food Security - Farm level self-sufficiency only shows potential food security

• Post-harvest losses – not needed here except in relation to self-sufficiency estimate
• Income not easily translated into food security, though it can be used for consumption

- Revise duration of food supply to months of food insecurity - Measure consumption (meals per day) across multiple seasons (lean and surplus seasons) Social aspects

Equity and distribution of labor - Disaggregation by gender - List activities – who does what by task – power tasks and control tasks for drudgery estimate - Who controls which crops, livestock and land? - Ownership can be difficult but access is more important – decision making about management and control Identifying which groups are marginalized is a key step - Livelihood types (pastoralist, agro-pastoralist, landless) - Gender, age, ethnicity, HIV status Number of collective action (village natural resource regulation, cooperative marketing, labor sharing) Number of conflicts Social capital / social cohesion – number who received support – who do you reach out to for various problems

Visualizing SI indicators October 8th 2015 As a means to determine which technologies perform well under which conditions, and to visualize and assess the impact of these technologies in the five domains of SI, we discussed which tools could be used. Spider diagrams (radar charts) are a useful visual way of comparing technologies across multiple attributes, including representative indicators of all five domains. Bi-plots (bar charts comparing two contexts/technologies across multiple attributes) allow for including even more variables Could threshold values be included? Yes, that may be helpful. It would be easier to see on bi-plots. There is some resistance to thresholds due to the context-specificity and the uncertainty. For whichever visualization method it will be important to easily tell what information pertains to what domain. This could be done through color-coding. For the proposal it would be useful to present a few examples of these visualizations for interventions carried out in Africa RISING, both in terms of teams sharing examples of evaluating technologies to share to partners for scaling; and for showing how we as SI farming systems researchers are evaluating potential impact of SI technologies as they are adopted. Another way to visualize all of this information is through a dashboard with multiple graphs, dials and charts. These can be put on the web and be linked to M&E so that they can be regularly updated Finally, we discussed developing one page infographics for innovations or technologies that can show a map of where it would be suitable, and what would be the impact if the technologies quantitative graphs, and icons with brief qualitative results Does the communication team have capacity to make these infographics? No, they currently outsource this type of work.