Institutional Context

Agricultural Policy and RDP

After its independence from Yugoslavia, FYR Macedonia experienced turbulent agricultural policies with many reforms and ad hoc policy decisions. In 2005, the country received the status of an EU candidate. This new trend of European integration brought about changes in the national agricultural policy which had to be adjusted to the CAP. Therefore, FYR Macedonia focused on harmonization of the national policy for development of agriculture and rural areas. The rural development policy is to a large extent aligned with (the second pillar of) the CAP. It has four priority areas and instruments to support them: (1) increasing the competitiveness of the agricultural and forest holdings, (2) protecting and improving the environment and rural areas, (3) improving the quality of life and encouraging diversification of economic activities in rural areas, and (4) supporting local development (Dimitrievski et al. Reference Dimitrievski, Kotevska, Janeska Stamenkovska, Tuna, Nacka, Volk, Erjavec and Mortensen2014, 128). In addition, rural development is financed by the EU via the Instrument for Pre-Accession Assistance for Rural Development (IPARD) (Dimitrievski et al. Reference Dimitrievski, Kotevska, Janeska Stamenkovska, Tuna, Nacka, Volk, Erjavec and Mortensen2014), which is not investigated in this study.

After the 2007 introduction of the National Strategy for Agriculture and Rural Development, the Ministry of Agriculture, Forestry and Water Economy has been preparing and announcing annual programs for rural development. The rural development budget is planned on an annual basis and realized through up to eight calls per year. However, because investments require time to be organized and implemented, and due to limited institutional capacity, budget transfers planned for one year are often conducted only in successive years.

In the period from 2008 to 2014, projects of about EUR 31.4 million were funded under the national program for rural development (see Table 1). In the first few years of implementation, the budget was mainly used to increase competitiveness of agricultural holdings, mostly through farm modernization of primary producers. In 2014, a substantial increase in the budget was devoted to the agrifood processing sectors and for improving the quality of life and infrastructural improvement of rural areas. According to information provided by the Agency for Financial Support in Agriculture and Rural Development, in 2014, funds for increasing competitiveness were allocated to 700 applicants (farmers and companies) of relatively small investments averaging EUR 4,460, whereas funds for improving quality of life in rural areas were used by 80 municipalities, averaging EUR 64,470 (APM Database 2015).

Table 1.

Annual Payments for Structural and Rural Development in FYR Macedonia per Priority Area (2008–2014, million EUR)

Source: Own calculation based on data in the Macedonian APM database (APM Database 2015).

Study Design

Our study is based on an information campaign experiment conducted in the Southeast of FYR Macedonia in May–June 2015. A brochure was prepared for this purpose in cooperation with the Agency for Financial Support of Agriculture and Rural Development of the Republic of Macedonia. The assessment of the campaign's effectiveness to promote interest in the RDP is motivated by the relatively low number of applications, despite the government's willingness to support the agricultural sector and the availability of funding.

The causal effect of information provision was intended to be evaluated by means of an experiment. We planned to randomly select 600 farmer households in the largest villages in the chosen region. Every second household on a list of households per village would be treated, while the remaining households would comprise the control group. The treatment probability would thus be asymptotically independent of farmers’ characteristics. The treatment group would receive an information brochure on selected RDP measures delivered in person, whereas the control group would receive no such brochure. A survey would be conducted for the entire sample about two weeks later, collecting information on personal and farm characteristics, previous experiences with the RDP application and participation, awareness about the RDP and its potential benefits for the community and the farm, and, importantly, on the farmers’ intention to apply for RDP measures and to cofinance RDP projects.

The actual implementation of the campaign deviated from the initial plan. Due to an unstable political situation and generally low levels of trust in the country, data collectors did not manage to fully follow the protocol. Reportedly, farmers were reluctant to communicate with strangers and accept brochures when the surveyors tried to approach the farmers at their homes. Therefore, instead of going to every second house when delivering the brochure, and going house to house to conduct the survey in preselected villages, the surveyors distributed them in several villages in public places, such as local shops, markets, pharmacies, fields, gardens, and water supply stations. They distributed the brochures in person and collected farmers’ contact information to survey them 1–2 weeks later. Reportedly, the brochures were more likely to be given to younger farmers, owners of small farms, and those who had not had experience with RDP participation, who were supposedly the types of farmers one predominantly meets in public places in rural areas. The face-to-face survey for the control group took place while the brochures were still being distributed to the treatment group. Once brochure dissemination was completed, the treatment group was surveyed. All treated individuals were interviewed, so there was no unit nonresponse. In the control group, an interviewer would go to the next available household in case of a refusal. The violations of the experimental design required the restriction of the evaluation sample to a specific subset of observations and the application of estimation methods that account for the fact that the intervention was not properly randomized.

The distributed brochure (see Appendix B) contains information about four selected RDP measures. The face-side of the brochure presents the title and the logo of the Agency for Financial Support in Agriculture and Rural Development, the phrase “Every year the Government of the Republic of Macedonia prepares financial support programs for rural development,” and three major goals of the program: modernization and structural adjustment of the agrifood sector, support of economic activities related to nature protection and development of rural areas, and transition of national agricultural policy towards the EU CAP. The rest of the brochure describes selected RDP measures along with eligibility criteria, application processes, required documents, and contact details for the responsible authorities. The selected RDP measures include (1) Support of young farmers (Measure 112), (2) Investments in farm modernization (Measure 121), (3) Investments in increasing the economic value of forestry (Measure 122), and finally (4) Support of economic associations of farms for joint agricultural activity (Measure 131). Three of the four listed measures require cofinancing from the farmers’ side. Measures 121 and 122 require 50 percent cofinancing by the farmer, whereas measure 131 requires up to 20 percent, depending on the submeasure (Zakon za Zemjodelstvo i Ruralen Razvoj 2010: 17–20). Importantly, the farmer must first personally finance the full amount of investment while actual RDP support is received upon the realized costs, if previously approved to be eligible. Measure 112 represents a grant of up to 600,000 Macedonian denars (EUR 9,760)Footnote ¹ paid to a successful application in three instalments over a three-year period (hence, cofinancing is not required). The brochure targets various groups of farmers and provides the most relevant information regarding RDP measures and the application process. If farmers wanted to obtain more details on the program, the contact information of the responsible authorities could be found on the back of the brochure.

Data and Balancing Tests

In our survey, cross-sectional data on 597 farmer households (represented by a household head), including 292 treated and 305 nontreated farmers, were collected. The dataset contains observations from 34 villages of the Southeastern region.

Respondents were asked about their attitudes and opinions about the RDP measured on a Likert scale ranging from 1 “strongly disagree” to 5 “strongly agree.” The variables generated from these questions are used as outcomes in our analysis. One group of questions relates to farmers’ willingness to apply and participate in the program in the near future (3–5 years): “How do you assess the possibility to use RDP support for your household (e.g., for mechanization, equipment purchases) in the next 3–5 years?” and “How do you assess your intention to use RDP support for your household in the next 3–5 years?” Another group of statements covers awareness and opinions about RDP application process and participation: “I have enough information to independently prepare the application (procedure and documents),” “I have enough knowledge and experience to independently prepare the application (procedure and documents),” “The RDP application (procedure and documents) is easy,” and “The RDP increases the administrative work.” Information on farmers’ previous experiences with RDP was collected, including application for the program in the last three years, use of support in the last three years, and received value of support (in denars).

Background characteristics were also gathered, describing household size; household head's age, sex, educational attainment (primary education, high school, or college/university and higher), and experiences with farming activities, including number of years spent working on a farm, and the primary occupation (whether in agriculture or other industries). Information related to farming activities was available from the survey: farm profitability in the last three years (measured on a scale from 1 to 5: “very unprofitable,” “moderately unprofitable,” “break-even,” “moderately profitable,” “very profitable”), ease of getting a loan (1 to 5: “very difficult,” “difficult,” “medium,” “easy,” “very easy”), dependence on subsidies to break even financially (1 to 3: “not dependent,” “slightly dependent,” “very dependent”), frequency of cooperation with other agricultural producers (1 to 5: “never,” “rarely,” “not sure,” “sometimes,” “always”), share of agricultural production sold on a market, share of household income from farming, whether or not there are additional workers besides family members working on the farm, total farmed area (in hectares), and total livestock (in heads). Finally, the data contain binary indicators for receiving the brochure, reading it, and learning new facts about RDP measures.

Balancing t-tests comparing the mean values of the characteristics between the treatment and control groups revealed statistically significant (at the 5 percent level) differences in age, education, years in farming, having additional workers on the farm, the share of agricultural production sold on a market, farm profitability, farm capacity (in hectares), and some missing indicators, which points to a failure of randomization. For this reason, we use a restricted sample for our evaluation based on the information about the brochure assignment process (i.e., brochures were more likely distributed to younger farmers, owners of small farms, and those who had not participated in the RDP previously) provided by the field personnel and reflected in the data. Specifically, we disregard observations from older age groups and only keep prime-age household heads that are up to 55 years old. Furthermore, we only include households that have not previously received RDP support and do not have any employees working on their farm.

As demonstrated in Table 2, which provides descriptive statistics and balancing t-tests for the covariates, the subsample is relatively well balanced in terms of mean values of a range of selected characteristics. Apart from primary education, farm profitability and a missing indicator for the share of agricultural production sold on a market, no mean is statistically significantly different across treatment states at the 5-percent level. We consider this subsample in our analysis of the brochure's effect outlined further below.

Table 2.

Mean Covariate Values by Treatment Status in the Selected Subsample

Notes: Standard deviations are in parentheses. Robust standard errors are in brackets.

^a Profitable farm: 1 = “very unprofitable;” 2 = “moderately unprofitable;” 3 = “break-even;” 4 = “moderately profitable;” 5 = “very profitable.”

^b Subsidy dependent: 1 = “not dependent;” 2 = “slightly dependent;” 3 = “very dependent.”

^c Frequency of cooperation: 1 = “never;” 2 = “rarely;” 3 = “not sure;” 4 = “sometimes;” 5 = “always.”

The evaluation sample includes 257 observations, out of which 156 are treated and 101 comprise the control group. As can be seen from Table 2, farmers are, on average, about 45 years old, predominantly males, with a high school degree, who have spent almost half of their life working in farming. For half of the farmers, agriculture is the main occupation. They sell most of what they produce on the market, and more than half of their income comes from farming. Farms in the sample are, on average, moderately profitable or break-even and somewhat dependent on subsidies. Table 3 provides additional insight into how our evaluation sample compares to the average farm household in the Southeast region and in the entire FYR Macedonia, in terms of characteristics available from the 2013 Farm Structure Survey. Household heads in the selected sample are typically younger, more educated, more likely to be female, and their household size tends to be larger, compared to the respective averages in the region and country. The average farm size in the sample is comparable to the regional and national averages but smaller in terms of total livestock.

Table 3.

Mean Values of Characteristics at Sample, Regional, and National Levels

Source: Own 2015 survey and Farm Structure Survey 2013.

Item nonresponse was moderate. In 21 cases (8.2 percent) the educational level was reported in the selected sample. The number of missing values in other covariates is even smaller. For the purpose of our analysis, we introduce binary indicators for missing values in covariates while replacing actual missing values with zeros.

Estimation Methods

To evaluate the impact of the information brochure on farmers’ willingness to apply and participate in the RDP, as well as on other outcome variables, four econometric methods are used: the simple difference in means, OLS, and two non-/semiparametric estimation techniques, namely, propensity score matching and nonparametric multivariate reweighting (entropy balancing). Formally, we estimate regression specifications of the following kind:

(1)$$Y_i = {\rm \beta} _0 + {\rm \beta} _1T_i + {\rm \beta} _2X_i + {\rm \epsilon} _i\comma \; $$

where the variable Y _i measures various outcomes, e.g., farmers’ intention to apply for the RDP, for individual i. T _i is a binary indicator that takes a value of 1 if individual is “treated,” i.e., received the information brochure, while β₁ is the coefficient of interest, as it captures the treatment effect. X _i is the vector of covariates used in the OLS regression, propensity score estimation, and entropy balancing.

First, we consider the simple differences in mean outcomes between treatment and control groups. If randomization had been successful, both groups would have been comparable in all their background characteristics (both observed and unobserved), and the differences in mean outcomes across treatment groups would have been unbiased estimates of the average casual effects of the intervention. However, the randomization was not successful, and even after restricting the original sample, some characteristics are not fully balanced across treatment states. For this reason, the simple difference is unlikely to provide an unbiased estimate of the average casual treatment effect.

As an alternative strategy, we control for a range of observed characteristics X _i in the estimation. We rely on the conditional independence assumption (CIA), which states that after conditioning on observed characteristics that jointly affect the treatment probability and the outcome, the independence of the treatment and the potential outcomes hold, such that there are no unobservables jointly affecting the treatment and the outcome (Imbens Reference Imbens2004: 7):

(2)$$\lpar Y\lpar 0\rpar \comma \; Y\lpar 1\rpar \rpar \bot T \vert X\comma \; $$

where Y(0) and Y(1) are potential outcomes under, respectively, nontreatment and treatment, T is a binary treatment indicator and X is the covariate set.Footnote ²

The probability of receiving the brochure was reportedly negatively associated with farmers’ age, farm capacity, and previous participation in the RDP. This is why it is important to control for these and related characteristics. Our dataset contains information about farmers’ age. Farm capacity can be controlled by including variables such as farmed area and total livestock. Farmers who previously participated in the program are excluded from the evaluation sample.

However, we believe it is critical to account for additional characteristics that can be simultaneously related to the outcome variables and the treatment probability, because the brochures were more often distributed to relatively poorly informed farmers. Educational level is likely to affect farmers’ awareness about the RDP and, hence, their potential interest in applying for agricultural support. As mentioned in a recent version of The National Programme for Agriculture and Rural Development, Macedonian small-scale farmers appear to have low educational levels (European Commission 2013: 27). Because the brochure was more often distributed to the owners of smaller farms, it is possible that those who received it had lower educational levels. We also suspect that the relative importance of farming and farm profitability might have affected the probability of receiving the brochure and, at the same time, intention to participate. Individuals for whom farming is the main occupation and whose income is mostly generated by farming should be more interested in obtaining information about the RDP. For this reason, household's head occupation, the share of agricultural production sold on the market, and the share of income from farming are included in the regressions. Furthermore, farm profitability and subsidy dependence should be controlled, because some RDP measures require cofinancing. Given that it is easier for profitable and subsidy-independent farmers to cofinance a project, they might be more interested in learning about RDP measures and obtaining the brochure. Table 2 provides supporting evidence for this, because treated farmers are, on average, more likely to have profitable farms and be less subsidy dependent. Finally, we include an indicator for the frequency of cooperation with other farmers as a control variable. More cooperative farmers might be more socially open and active, which increases their chances of receiving the brochure and being interested in the RDP.

Our first approach to control for the observed confounders is a standard OLS regression of the outcome on a constant, the treatment indicator, and the covariates. However, an important drawback of OLS is that it assumes a linear relationship between regressors and the outcome variable, which may be violated in practice. Hence, we also apply more flexible semi- and nonparametric estimators, relying on less rigid functional form assumptions.

One of the most well-known approaches for the evaluation of treatment effects in nonrandomized studies is propensity score matching. The idea is to find for each treated observation one or more nontreated units with a similar conditional treatment probability, i.e., propensity score. In a general form, the treatment effect $ \lpar {\hat \Delta}_{\rm match}\rpar $ is defined as the average difference in the outcomes of the treated and the weighted nontreated matched units (see, for instance, Smith and Todd Reference Smith and Todd2005):

(3)$${\hat \Delta} _{\rm match} = \; \displaystyle{1 \over {N_1}}\mathop \sum \limits_{\lcub i\colon \; T_i = 1\rcub } \lpar{Y_i - \mathop \sum \limits_{\lcub i\colon \; T_j = 0\rcub } W_{i\comma j}Y_j} \rpar\comma \; $$

where W _i,j is the weight given to the outcome of a nontreated observation j, when j is matched to a treated unit i, and N ₁ is the number of treated observations. In this study, we conduct semiparametric kernel matching. First, the propensity score: p(X) = Pr(T = 1|X) is estimated in a probit regression (see Table 1A in Appendix A for the propensity score specification). Then, kernel regression of the outcome on the estimated propensity score among the nontreated is conducted to estimate the conditional mean outcome given the propensity score without treatment, E(Y|T = 0, p(X)) = :m(0, p(X)) (Huber, Lechner and Wunsch Reference Huber, Lechner and Wunsch2013). Formally,

(4)$$\hat m\lpar 0\comma \; \hat p\lpar X_i\rpar \rpar = \displaystyle{{\mathop \sum \nolimits_{\lcub j\colon T_j = 0\rcub } K\lpar \hat p\lpar X_i\rpar - \hat p\lpar X_j\rpar /h\rpar Y_j} \over {\mathop \sum \nolimits_{\lcub j\colon T_j = 0\rcub } K\lpar \hat p\lpar X_i\rpar - \hat p\lpar X_j\rpar /h\rpar }}\comma \; $$

where ${\hat m}\lpar 0\comma \; \hat p\lpar X_i\rpar \rpar$ is an estimate of m(0, p(X)), K is a kernel function and h is a bandwidth operator. In the estimations, the Epanechnikov kernel and a bandwidth of 0.6 is used.Footnote ³ Thereafter, the treatment effect for the treated is estimated by averaging the estimated function by the empirical distribution of p(X) for the treated:

(5)$$\hat \Delta _{{\rm kernel}\_{\rm match}} = \; \displaystyle{1 \over {N_1}}\mathop \sum \limits_{\lcub i\colon \; T_i = 1\rcub } \lpar Y_i - \hat m\lpar 0\comma \; \hat p\lpar X_i\rpar \rpar \rpar .$$

Matching estimators rely on a common support assumption that ensures units with comparable characteristics exist in both treatment states. Figure 1 provides the distribution of the estimated propensity score before and after matching. The upper panel shows some non-overlapping areas in the distribution of the propensity score in the treated and nontreated groups prior to matching. Matching achieves a decent overlap in the propensity score distributions, as illustrated by the lower panel of Figure 1. Only four observations in the treatment group lie outside the common support and therefore need to be excluded from propensity score matching. Additionally, Table 4 presents post-matching mean covariate values by treatment status, standardized differences, and percentage-reduction in standardized differences compared to the original (unmatched) sample, and balancing t-tests on the matched sample. Based on standardized differences and the percentage-reduction in standardized differences, we conclude that matching considerably improved balance in all characteristics (the average reduction in standardized differences was 75 percent), except for farm size.

Figure 1.

Density Estimates of the Estimates of the Propensity Score ${\bf Pr}\lpar {\bi T} = 1\vert {\bi X}\rpar $

Notes: The density estimations are based on pstest command in STATA. The bounds of the support of the propensity score are set to be 0 and 1.

Table 4.

Covariate Balance after Propensity Score Matching

Notes: “Std. diff. %” stands for standardized difference $\displaystyle{{100\; \lpar {\bar x}_T - {\bar x}_C\rpar } \over {\sqrt {\lpar {\rm var}\lpar x_T\rpar + {\rm var}\lpar x_C\rpar \rpar /2}}}$ (Rosenbaum and Rubin Reference Rosenbaum and Rubin1985). “% reduction” is percentage reduction in the absolute value of standardized difference after matching as compared to before matching. “t-value” and “p-value” refer to two-sample t-tests for equality of means.

^a Profitable farm: 1 = “very unprofitable;” 2 = “moderately unprofitable;” 3 = “break-even;” 4 = “moderately profitable;” 5 = “very profitable.”

^b Subsidy dependent: 1 = “not dependent;” 2 = “slightly dependent;” 3 = “very dependent.”

^c Frequency of cooperation: 1 = “never;” 2 = “rarely;” 3 = “not sure;” 4 = “sometimes;” 5 = “always.”

The next estimation technique employed in our analysis is entropy balancing,Footnote ⁴ a fully nonparametric multivariate reweighting method proposed by Hainmueller (Reference Hainmueller2012). It does not rest on any propensity score model, but on user-specified initial base weights for nontreated observations. Reweighting is based on computing new weights in a way that the Kullback-Leibler divergence from the baseline weights is minimized, subject to the balancing constraints. Weights of the nontreated are defined in such a way that exact balance in prespecified covariate moments like the mean is attained for the reweighted nontreated group and the treated. Formally, the weights are chosen by minimizing the following loss function, while balancing the (multidimensional) vector of covariates X _i:

(6)$$\mathop {\min} \limits_{w_i} \mathop \sum \limits_{\{ i:\; T_i = 0\}} h({\rm \omega} _i)$$

subject to the balance constraint

(7)$$\mathop \sum \limits_{\{ i:\; T_i = 0\}} {\rm \omega} _iX_i = \displaystyle{1 \over {N_1}}\mathop \sum \limits_{\{ i:\; T_i = 1\}} X_i$$

and the normalizing constraints

(8)$$\mathop \sum \limits_{\left\{{i\colon \; T_i = 0} \right\}} {\rm \omega} _i = 1$$

(9)$${\rm \omega} _i \ge 0\; \forall \; i\; {\rm with}\; T_i = 0\comma \; $$

where ω_i is a weight estimated for each nontreated observation i, and h( · ) is a distance metric. Hainmueller (Reference Hainmueller2012) uses the directed Kullback (Reference Kullback1959) entropy divergence: h(ω_i) = ω_ilog(ω_i/q _i), where q _i is the initial base weight. The loss function $\mathop \sum \nolimits_{\lcub i\colon \; T_i = 0\rcub } h\lpar {\rm \omega} _i\rpar$ measures the distance between the distribution of the estimated weights ${\rm \omega} _1\comma \; \ldots\comma \; {\rm \omega} _{N_0}$ and the initial base weights $q_1\comma \; \ldots\comma \; q_{N_0}$, where N ₀ is the number of nontreated units. The distribution of the base weights is usually set to be uniform with q _i = 1/N ₀. The constraint (7) balances the distribution of X _i between the treatment and the reweighted nontreated groups, so that the latter resembles the former in its covariate distribution. The normalizing constraints (8) and (9) force the weights to sum up to 1 and be nonnegative. The treatment effect on the treated can be estimated as the difference in mean outcomes between the treatment and the reweighted control groups:

(10)$$\hat \Delta _{{\rm ebalance}} = \displaystyle{1 \over {N_1}}\mathop \sum \limits_{\lcub i\colon \; T_i = 1\rcub } \left({Y_i - \; \displaystyle{{\mathop \sum \nolimits_{\lcub i\colon \; T_i = 0\rcub } Y_i{\rm \omega} _i} \over {\mathop \sum \nolimits_{\lcub i\colon \; T_i = 0\rcub } {\rm \omega} _i}}} \right).$$

Finally, it needs to be pointed out that our analysis relies implicitly on the stable unit treatment value assumption (SUTVA) that precludes any interaction, spill-over, and general equilibrium effects related to individual treatment assignment. However, it is possible that some study participants in the treated group spread information about the brochure in their villages, which would result in the contamination of the control group. In the event this happened, we estimate the lower bound of the absolute value of the treatment effect.