Evaluation of the Impact of the Agricultural Support Program

CRRC-Georgia carried out a quasi-experimental, post-hoc, mixed methods impact evaluation of the Agricultural Support Program (ASP) between December 2016 and April 2017 in collaboration with the Independent Office of Evaluation (IOE) of the International Fund for Agricultural Development (IFAD). The Agricultural Support Program took place in Georgia between 2010 and 2015. It consisted of two components: 1) Small Scale Infrastructure Rehabilitation and 2) Support for Rural Leasing. For the infrastructure component, the project aimed “to remove infrastructure bottlenecks which inhibit increasing participation of economically active rural poor in enhanced commercialization of the rural economy” according to project documentation. Within the infrastructure component, three types of infrastructure were rehabilitated or built: 1) Rehabilitation of primary and secondary irrigation canals; 2) Rehabilitation of bridges used to bring cattle to pasture; 3) The construction of drinking water infrastructure.

In line with the Internal Office of Evaluation of IFAD’s methodology, impact was assessed across five specific domains. These include: (i) household income and assets; (ii) human and social capital and empowerment; (iii) food security and agricultural productivity; (iv) natural resources, the environment and climate change; and (v) institutions and policies. While the focus of the evaluation is on the rural poverty impact criterion, the performance of the programme has also been evaluated for impact on gender equality and women’s empowerment.

The evaluation assessed not only “if”, but also “how” and “why” the programme has or has not had an impact on selected households and communities in the programme area. To this end, the evaluation team adopted a mixed methods approach including a household survey, focus group discussions, in depth interviews, and key informant interviews. The survey consisted of 3190 interviews, with 1778 interviews in control households and 1412 in treatment households.

In order to test for impact, the team used a quasi-experimental survey design. The driving idea behind quasi-experimental analysis is to use counterfactuals to understand what would have happened in the communities which received interventions had the intervention not taken place. Given that ASP did not make use of randomization, a two staged matching procedure was used to achieve balance on observable variables. First, treated communities were matched with non-treated communities on a number of variables. Second, after data collection households were matched using multivariate matching with genetic weights. Finally, when feasible, a differences in differences approach was used, with changes measured rather than only the 2016 outcome. Regression analyses were then used to estimate effects.

The key findings of the impact analysis include:

Indirect beneficiaries of the leasing component – individuals who sold grapes to companies that received leases – had substantively large increases in agricultural incomes;

Analyses often suggest little if any impact when it comes to rural poverty. However, context is important. During the project period, ASP was a small part of the very large aid inflows to Georgia, much of which was directed to the area where ASP activities took place. Hence, a lack of significant changes suggests that ASP performed on par with, but not better than other aid projects which took place in control communities;

Project outreach in the small scale infrastructure communities was inadequate, resulting in less effective project design and missing an opportunity for the development of human and social capital;

The project’s main success within the food security and agricultural productivity impact domain is the increase in amount of land irrigated; the project does not appear to have had any detectable impact on food security;