2.1. Institutional setting

In Germany, two types of unemployment benefits are available. Unemployment benefits I (UB I) cover unemployed workers who contributed to the unemployment insurance system for a minimum of one year within the last two years of employment. UB I-beneficiaries receive up to 67% (60%) of their last net wage as parents (childless persons), usually for a maximum duration of one year (Riphahn et al., Reference Riphahn, Sander and Wunder2013).

Unemployment benefits II (UB II) or simply welfare benefit, are tax-financed means-tested flat rate benefits covering persons capable of workingFootnote ¹ for a minimum of three hours per day, and whose household income falls short of the legal social minimum. So even persons who are working or receive UB I can be eligible to UB II if their household income is sufficiently low (Riphahn et al., Reference Riphahn, Sander and Wunder2013; Wapler et al., Reference Wapler, Wolf and Wolff2018). Most refugees who arrived between 2013 and 2016 and who have a legal residence permit in Germany are also eligible for welfare benefits as their household income is likely below the social minimum. Moreover, most of them have not worked long enough to be eligible for UB I benefits.

Welfare beneficiaries are obliged to make efforts to increase their employability to reduce their welfare dependency. Public employment offices, referred to as job centers, support welfare beneficiaries to this purpose through job search assistance and targeted placement in ALMPs (Wapler et al., Reference Wapler, Wolf and Wolff2018). To combat moral hazard, welfare beneficiaries refraining to participate in an ALMP may be subject to sanctions that can result in a reduction of welfare benefits for a period up to three months (van den Berg et al., Reference van den Berg, Uhlendorff and Wolff2014). The same sanctions apply for refugees assigned to participate in an ALMP.

2.2. Which programs do we evaluate?

The first two programs examined are part of the so-called Schemes for Activation and Integration (SAI). In Germany, SAIs are performed either by private training providers (Schemes by Providers or SP) or by employers in the form of In-Firm Training (IFT), also referred to as on-the-job training (Harrer et al., Reference Harrer, Moczall and Wolff2020).

SP are the most frequently used ALMPs for welfare recipients in Germany. They provide training whose goals include guiding into apprenticeships and work, identifying and reducing employment impediments, providing placement services into contributory employment, preparing for self-employment and stabilizing an employment take-up (Wapler et al., Reference Wapler, Wolf and Wolff2018). For example, Guiding into Apprenticeships and into Work is a type of SP that provides guidance on finding suitable job offers and writing resumes and application letters. Another type of SP focuses on detecting individual employment impediments, improving skills and providing information on suitable employment opportunities (Harrer et al., Reference Harrer, Moczall and Wolff2020). SP may also be combined with In-Firm Training for a period of up to 4 weeks. Moreover, SP focusing on teaching a professional occupation may not exceed a period of 8 weeks (Harrer et al., Reference Harrer, Moczall and Wolff2020). The median duration of SP in our sample is about 3 months.

While SP occur mainly in the form of classroom training, IFTs occur in establishments and are shorter in duration. At the median, IFTs last one month in our sample. IFT measures focus on gaining practical skills and applying them directly on the job. In this manner, they are comparable to unpaid internships (Harrer et al., Reference Harrer, Moczall and Wolff2020). A number of refugee-specific programs were additionally developed within the context of a SAI. However, the numbers of treated individuals in these programs are typically too low to be analyzed by themselves or cannot be properly identified in the data due to initial limitations after the introduction of these programs.

The third program considered is Further Vocational Training (FVT), whose objective is to enable employability through human capital accumulation. FVT include short-term programs providing professional and practical skills along with long-term programs (up to two years) providing a certified vocational training degree (Bernhard and Kruppe, Reference Bernhard and Kruppe2012; Wapler et al., Reference Wapler, Wolf and Wolff2018). Refugees can also receive accreditation of their foreign vocational qualifications in the context of FVT (German Federal Employment Agency, 2013).

As FVT programs usually require longer and more intensive investments in human capital, participants have less time and resources to invest in job search compared to non-participants, creating a so-called “lock-in-effect”. The lock-in-effect reflects a period of lower employment rates for participants compared to non-participants due to being “locked-in” in the training. This lock-in-effect is likely to occur only during program participation, which, on average, lasts about five months in our sample. Soon after training completion, participants’ employment should catch up and even exceed those of non-participants due to higher returns from human capital investments (Arendt, Reference Arendt2022; Card et al., Reference Card, Kluve and Weber2017).

Lastly, we examine a public employment program referred to as One-Euro-Jobs (OEJs) targeting unemployed welfare recipients strongly detached from the labor market. OEJs are temporary (part-time) public sector jobs, typically lasting between three and twelve months, in which participants receive one to two Euros per hour that is not deducted from their welfare benefit (Harrer and Stockinger, Reference Harrer and Stockinger2021). The median duration of OEJs in our sample is three months. OEJs provide basic work activities for participants to increase long-term employability. Jobseekers should only be placed in a One-Euro-Job if none of the above-mentioned ALMPs is (yet) suitable (Hohmeyer and Wolff, Reference Hohmeyer and Wolff2018). All the above-mentioned programs, apart from the refugee-specific programs, were therefore pre-existing and extended to refugees. Access considerations according to the Federal Employment Agency are hence identical for refugees as non-refugees and include the necessary language skills.

Comparing treatment probabilities across programs, Tübbicke and Kasrin (Reference Tübbicke and Kasrin2021) show that unemployed refugees have a higher probability of participating in SP and IFT but a lower participation probability in FVT and OEJ compared to non-refugees. These differences may be due to different employment impediments among refugees on the one hand and caseworker preferences or perceptions about program effectiveness on the other. Case workers assign participants to programs after weighing potential program benefits against costs.

Higher participation in SP and IFT relative to FVT could be due to their shorter duration and due to lower human capital required for successful participation. FVT programs are also more expensive than SP and IFT. On the other hand, OEJs are programs of ‘last resort’ aimed at the long term unemployed who are extremely distant to the labor market. Previous results for more general populations also show that participants tend to differ across these programs. For example, Harrer et al. (Reference Harrer, Moczall and Wolff2020) show that IFT participants have more favorable background characteristics, leading to better labor market prospects than participants of SP in the absence of treatment. Hohmeyer and Wolff (Reference Hohmeyer and Wolff2012) show that OEJ participants are expected to perform relatively poorly in the labor market without an intervention. These differences must be addressed in order to obtain unbiased estimates of treatment effects (see Section 3.1).

3.1. Identification and estimation strategy

We aim to estimate the causal effects of a series of active labor market programs on participating refugees in Germany. In the notation of the potential outcomes model by Roy (Reference Roy1951) and Rubin (Reference Rubin1974), we want to estimate the average treatment effect on the treated (ATT), defined as

(1) $$\tau _{ATT}^g = E\left[ {Y_g^1{\rm{|}}{D_g} = 1} \right] - E{\rm{[}}Y_g^0{\rm{|}}{D_g} = 1],$$

where ${D_g}$ is a treatment indicator, taking on the value of one if the person received treatment $g \in $ {SP, IFT, FVT and OEJs}, and zero if not. While $Y_g^1$ refers to the outcome that is realized when an individual receives treatment $g$ , $Y_g^0$ denotes the outcome that occurs when the same individual is not treated with $g$ (but potentially other programs). Hence, the ATT defined by (1) measures the additional effect of program $g$ relative to all other programs available. Due to the fundamental evaluation problem, $Y_g^0$ is unobservable for participants and hence, the second expectation in the equation above has to be imputed from data on non-participants.

To do so, we use inverse probability weighting (IPW) based on a selection-on-observables approach (Heckman and Robb, Reference Heckman and Robb1985). IPW re-weights non-participants based on their propensity score $\;{P_g}\left( X \right) = {\rm{Pr}}({D_g} = 1\;|X)$ , i.e. the conditional treatment probability, in order to undo the selection into treatment. Under the selection-on-observables assumption, i.e. the assumption that the set of observed characteristics $X$ includes all relevant factors that simultaneously govern the selection process and the outcome of interest, and the so-called common support assumption $\;{P_g}\left( X \right) \lt 1\;\forall \;X$ , the missing counterfactual can be recovered from data on non-participants by reweighting their outcomes by the odds of treatment $\;{P_g}\left( X \right)/\left( {1 - {P_g}\left( X \right)} \right)$ .

While the selection-on-observables assumption is strong, we are equipped with administrative data that allows us to control for a large number of observed characteristics measured prior to the start of treatment. To specify a model for ${P_g}\left( X \right)$ , we largely follow other studies evaluating the programs of interest for more general populations and pick covariates that have been shown to be predictive of at least one of the treatments under consideration (Bernhard and Kruppe, Reference Bernhard and Kruppe2012; Harrer et al., Reference Harrer, Moczall and Wolff2020; Harrer and Stockinger, Reference Harrer and Stockinger2021).

We adjust for socio-demographics (age, time in Germany, nationality, schooling and occupational qualificationFootnote ² , presence of children by age category), household characteristics (household type, total income, income without welfare benefits), the type of last job (regular or minor employment) along with minor employment at sampling date, individuals’ labor market historyFootnote ³ (cumulated durations in regular employment, vocational training, ALMP participation, unemployment benefit I and welfare receipt, job search, information on regular employment, and real labor earnings at specific dates one year prior to the sampling date) and regional labor market information (unemployment rate, long-term unemployment rate, vacancy-to-unemployment ratio, inflow rates into sanctions and ALMPs).

Studies by Brücker et al. (Reference Brücker, Kosyakova and Schuß2020) and Kosyakova et al. (Reference Kosyakova, Gundacker, Salikutluk and Trübswetter2021) show that age, time in Germany, schooling, the presence of children in the household and regional labor market conditions are important predictors of refugees’ labor market success. To minimize the impact of any parametric assumptions on our analysis, we discretized all covariates into sensible categories.Footnote ⁴

If the set of covariates X does not include all factors that influence the treatment decision and the outcome, then selection-on-observables assumption will fail and estimates will be biased. For example, this could be the case if physical or mental health – characteristics we do not observe – are decisive for selection into treatment and the outcome-generating process. Similarly, bias may arise if unobserved language skills are important predictors of treatment and outcomes. Fortunately, (Caliendo et al., Reference Caliendo, Mahlstedt and Mitnik2017) have shown that the inclusion of typically unobserved covariates tends not to significantly alter effect estimates of ALMPs upon conditioning on a large set of observed covariates. This is likely because covariates and especially pre-treatment outcomes have already been affected by those unobserved factors before the start of treatment.

Moreover, our list of control variables includes several strong predictors that can proxy language skills, including age, time since entry, education and country of origin (Fennelly and Palasz, Reference Fennelly and Palasz2003; van Tubergen, Reference van Tubergen2010). Nonetheless, we inspect how sensitive our results are to unobserved confounding in Section 4.4. Results shows that our estimates are highly robust. Hence, we believe that the selection-on-observables assumption is a reasonable approximation for our application. Because we have fairly low treatment to control ratios and differences in covariate distribution are modest, overlap between groups is strong and we do not need to trim the sample.Footnote ⁵

In order to estimate the propensity score, we use the so-called covariate balancing propensity scores (CBPS, Imai/Ratkovic, Reference Imai and Ratkovic2014). The CBPS does not maximize the maximum-likelihood, instead it estimates logit coefficients by algorithmically optimizing the resulting covariate balance, leading to (near) perfect finite-sample balance.Footnote ⁶ In addition to obviating the need for iterative balance checking and re-estimation of the propensity score, it was shown to perform well in a recent empirical Monte-Carlo simulation by Frölich et al. (Reference Frölich, Huber and Wiesenfarth2017) based on data similar to ours. Denoting person i’s estimated propensity score by ${{\hat{P}}_{gi}}( X )$ and its observed labor market outcome by ${Y_{i}}$ , estimates of the ATT are obtained using the normalized IPW estimator as

(2) $$\hat \tau _{ATT}^g = {1 \over {\mathop \sum \nolimits_{i = 1}^N {D_{gi}}}}\mathop \sum \limits_{i = 1}^N {D_{gi}}{Y_i} - {1 \over {\mathop \sum \nolimits_{i = 1}^N {{\left( {1 - {D_{gi}}} \right){{\hat P}_{gi}}} \over {1 - {{\hat P}_{gi}}}}}}\mathop \sum \limits_{i = 1}^N {{\left( {1 - {D_{gi}}} \right){{\hat P}_{gi}}{Y_i}} \over {1 - {{\hat P}_{gi}}}}{\rm{\;}}.$$

Intuitively, IPW re-weights non-participants by increasing (decreasing) the weight of individuals with a large (small) propensity score, thereby eliminating bias due to differences in covariate distributions between treated and untreated units. For statistical inference, we rely on the weight-based asymptotic approximation of standard errors by Lechner (Reference Lechner2001) as it was shown to be somewhat conservative and to perform rather well in large samples (see, Bodory et al., Reference Brücker, Kosyakova and Schuß2020, for details).Footnote ⁷

3.2. Data and descriptive statistics

As a data source, we mainly use the Integrated Employment Biographies (IEB), an administrative dataset from the statistics department of the Federal Employment Agency in Germany. The data includes the universe of employees liable to social security, the registered unemployed, registered jobseekers, benefit recipients, and ALMP program participants. Using the IEB has a number of advantages. First, due to its extensive nature, the IEB allow to condition on a large number of observed characteristics. Second, as the IEB includes the universe of person groups, it allows observing the full count of individuals, eliminating the need to rely on a random sample as is the case for most survey data. Lastly, the IEB does not suffer from panel attrition, which is likely to be more problematic among refugees than among the general population. All of these factors should greatly improve inference in our application.

We use the full count of refugees with a residence permit according to §§ 22–26 Residence Law (“Aufenthaltsgesetz”) who were unemployed and on welfare as of September 30, 2016. As the great majority of refugees arriving to Germany between 2013 and 2016 was male and their treatment probabilities are higher than those of female refugees, we restrict our analysis to men.Footnote ⁸ To get a more homogenous sample, we restrict our attention to individuals who first entered Germany in 2013 or later.Footnote ⁹ Moreover, we exclude individuals who were younger than 17 or older than 62 at the sampling date to avoid including people still in high school or who have reached the legal retirement age before the end of our analysis time frame. This is necessary as neither being in school nor in retirement can be observed in the data.

Based on this sample, we classify individuals as treated if they entered SP, IFT, FVT or OEJs during the entry window between October 2016 and March 2017.Footnote ¹⁰ For each treatment group we define a separate control group (or simply non-participants hereafter), i.e. individuals who did not participate in the respective program during the entry window. However, non-participants may participate in other programs during the entry window or participate later in the same program. Using a six months entry window allows sufficient observation numbers among the treated, while avoiding selectivity issues as a longer entry window may lead to a negative selection of non-participants (Sianesi, Reference Sianesi2004). For each treatment, non-participants are randomly assigned a hypothetical entry month from the distribution of entry months among the respective treatment group. Individuals who are either gainfully employed or participating in one of the other programs at the hypothetical entry date are dropped from the respective sample.Footnote ¹¹ This leads to slightly different groups of non-participants across treatments.

Table 1 shows selected descriptive statistics for the four estimation samples. SP is clearly the most common ALMP assigned to refugees with 6,118 participants. Moreover, 747 individuals participated in IFT, followed by 465 OEJ and 424 FVT participants. The groups of non-participants range in size from around 27,000 to 30,000 individuals.

TABLE 1. Selected descriptive statistics

Note: Statistically significant differences at the 10/5/1% level between participants and non-participants are indicated by */**/*** on the respective value among the non-participants.

Panel A shows that the majority of both participants and non-participants are Syrian. Refugees from Iraq, Eritrea or Afghanistan make up between one and six percent, depending on the sample.Footnote ¹² The mean age ranges from just over 30 years among participants of SP to around 33 years among individuals in FVT, while non-participants are about 32 years old. ALMP participants also differ by their time of arrival to Germany: on the one hand, OEJ and SP participants have been in Germany for 1.33 and 1.38 years, respectively. Both are significantly less than the 1.41 years that their non-participants spent in Germany. On the other hand, with an average duration of 1.67 and 1.9 years for IFT and FVT participants, respectively, both groups have spent more time in Germany than their non-participants.

Moreover, FVT and IFT participants have the highest percentage of persons with at least some vocational training or university education, with 33 percent for FVT and 16 percent for IFT participants. SP and OEJ participants have a lower share of persons with some professional qualification, at 10 percent for SP participants and 7 percent for OEJ participants. Among non-participants, 11 percent have at least some vocational training or university education. Similarly, IFT and FVT participants have the highest high school completion rates with 62 and 74 percent across all groups. Among OEJ participants, 53 percent completed high school, which is lower than the completion rates of non-participants with 59 percent. This indicates that IFT and FVT participants might have more favorable labor market prospects than SP participants, while OEJ participants seem to have the lowest employment prospects.

Our outcomes of interest are monthly regular employment, i.e. an indicator for whether an individual is in unsubsidized dependent employment subject to social security contributions, and real monthly labor earnings. For our causal analysis, we make use of outcomes from 12 months prior to the start of (hypothetical) treatment up to 33 months after. After 33 months, IFT and FVT participants perform best in comparison to the non-participants in both dimensions, as shown in panel B of Table 1. However, due to the non-random selection into treatment, such a simple mean comparison is plagued by selection bias.

4.1. Checking balance

Before turning to the causal analysis, we present some evidence on the balancing quality before and after re-weighting the respective group of non-participants using our IPW approach. Table 2 displays several common balancing measures given by the mean absolute standardized bias (MSB, Rosenbaum and Rubin, Reference Rosenbaum and Rubin1983), the pseudo- ${R^2}$ from a probit regression, along with its corresponding p-value of overall significance (Sianesi, Reference Sianesi2004), and Rubin’s B (Rubin, Reference Rubin2001). If the CBPS methodology works, MSB, pseudo- ${R^2}$ and Rubin’s B should be very close to zero after re-weighting. Indeed, all three measures only show negligible deviations from zero after weighting, indicating (near) perfect covariate balance. Similarly, the p-value rises from essentially zero to one in all samples after weighting, thus it is impossible to reject the null hypothesis that covariates are not related to treatment. Hence, our balancing approach delivers excellent balance, successfully purging the outcome gap between participants and non-participants from its association with background characteristics.Footnote ¹³

TABLE 2. Balancing quality before and after weighting

Note: The absolute standardized bias is defined as the difference in means between treatment and control divided by the square root of the mean of variances prior to weighting in both groups. Rubin’s B is the standardized mean difference of the linear index of the propensity score of treatment and control group.

4.2. Main results

Figure 1 presents effects on regular employment, with effects on real monthly labor earnings displayed in Figure 2. Estimated effects range from twelve months prior to the start of treatment until 33 months after. Estimates prior to treatment are close to zero and statistically insignificant, showing that participants and non-participants follow the same path in terms of employment and earnings, providing additional evidence that our weighting strategy was successful.

FIGURE 1. Effects on regular employment in percentage points.

Note: Statistically significant estimates at the 5% level are marked with a cross.

N(Participants): 6,118 for Schemes by Providers; 747 for In-Firm Training; 424 for Further Vocational Training; 465 for One-Euro-Jobs.

N(non-Participants): range from 27,135 for Schemes by Providers to 29,903 for Further Vocational Training.

FIGURE 2. Effects on real monthly labor earnings in Euro.

Note: Statistically significant estimates at the 5% level are marked with a cross.

N(Participants): 6,118 for Schemes by Providers; 747 for In-Firm Training; 424 for Further Vocational Training; 465 for One-Euro-Jobs.

N(non-Participants): range from 27,135 for Schemes by Providers to 29,903 for Further Vocational Training.

For OEJs, there is a rather long-term and statistically significant negative effect on labor earnings and employment. After about 30 months, point estimates become positive but insignificant. Actually, this result is not surprising: OEJs are a last resort measure for individuals who are particularly distant from the labor market and so are likely to only increase employment prospects in the long run, possibly also in combination with other ALMP measures (Kiesel and Wolff, Reference Kiesel and Wolff2018). Indeed, the majority of OEJ participants in our sample received some additional treatment after the OEJ program. This partly explains the long-lasting negative effects of OEJ participation despite its median duration of around three months. Similar results are found by Harrer and Stockinger (Reference Harrer and Stockinger2021) who analyze the effectiveness of OEJs for a more general population of unemployed welfare recipients.

SP show a lock-in effect for the first three months after the start of the program, which coincides with their median duration in our sample. After five months, employment effects become statistically significant and after seven months, earnings are positively affected by the treatment. After 33 months, employment effects are roughly 3.8 percentage points and effects on earnings are about 65 Euro. While in absolute value, these effects may seem rather small, they are considerable when compared to mean outcomes of participants (see Table 1). Relative to the mean, these effects are equivalent to an 11.5 percent increase in employment and a 13.4 percent increase in monthly labor earnings.

For IFT, we find positive effects on participants’ employment probabilities almost immediately. This highlights that participating in an IFT, which has a medium duration of one month, speeds up labor market integration considerably. Over time, employment effects increase, peaking at around 25 percentage points after 23 months. Afterwards, estimates of employment effects subside somewhat, reaching 18 percentage points at the end of the observation period.Footnote ¹⁴ Effects on earnings follow a similar path, with an impact on monthly labor earnings of 360 Euro after 33 months. The effect sizes for IFT and SP participants are comparable to those found for a more general population of unemployed welfare recipients by Harrer et al. (Reference Harrer, Moczall and Wolff2020), who examined treatment effects of program entry into SP and IFT in 2010.

The results for FVT show a significant lock-in effect for three months, which is not surprising given its relatively long median duration of 5 months. After around 10 months, positive impacts on employment rates and average earnings materialize. While employment effects quickly stabilize at around 14 percentage points, effects on earnings rise to around 710 Euro after 33 months. Again, the effect sizes are similar to the ones found by Bernhard and Kruppe (Reference Bernhard and Kruppe2012) for German and non-German welfare participants who entered FVT in 2005.

Thus, our findings indicate that all programs except OEJ are effective in improving refugees’ labor market success. As groups of program participants differ in their composition, it should be noted that effect sizes across programs cannot be directly compared.

4.3. Effect heterogeneity

For our heterogeneity analysis, we split our samples according to median values of certain background characteristics, given by age, time in Germany and regional unemployment. The analysis then repeats the estimation steps for all sub-samples. As before, the CBPS methodology yields excellent balancing quality after weighting (see Table A.1 in the appendix for details).

First, we split the sample according to individuals’ age, i.e. under 30 or at least 30 years old. This distinction is important because younger refugees might learn German and adapt to their new environment quicker than older refugees, which may influence how much they benefit from training programs. Second, we split the sample by duration of stay in Germany since time of arrival; here we analyze effects for individuals with a short and a long duration, i.e. below and above the median of roughly 1.4 years. Along this dimension, those with a longer duration are expected to be better integrated socially and culturally – and thus may benefit more from programs targeting short-term labor market integration such as SP and IFT.

We also analyze effects for individuals in different local labor market environments, splitting the sample by the local unemployment rate, with 9 percent serving as the cut-off. Refugees living in regions with relatively high unemployment may profit the most from programs focusing on human capital acquisition such as FVT, due to higher skill mismatch in regions with higher unemployment (Wapler et al., Reference Wapler, Wolf and Wolff2018). Table 3 shows estimates of employment effects for all programs and sub-groups to keep the presentation concise.Footnote ¹⁵

TABLE 3. Heterogeneous effects regarding regular employment in percentage points

Note: Statistically significant estimates at the 10/5/1% level are marked with */**/***. Italics indicate statistically significant differences at the 5% level between group-specific ATTs. Numbers of observations (treated; comparison) are in brackets below the estimates.

While all subgroup effect estimates are statistically significant except for OEJs, only SP shows statistically significant effect heterogeneity for duration of stay in Germany (indicated by the italics in Table 3). Specifically, the employment probability of persons with a longer duration increases by 5.5 percentage points through participation, more than twice its impact on persons with a shorter duration of stay. Although heterogeneity effects for all other estimates are statistically insignificant, gaps are sizable among IFT, FVT and OEJ participants along several dimensions. As these are relatively small treatment groups, a lack of statistical power makes the detection of meaningful effect heterogeneity difficult. Particularly interesting are the results found for IFT and OEJs. Similar to SP, IFT participants with longer durations of stay appear to benefit more compared to participants with shorter durations. Regarding OEJs, point estimates suggest that this type of program may be effective for individuals with a short duration of stay in Germany and for individuals in regions with low unemployment rates. The latter may be due to an easier transition from a OEJ to a regular job in low unemployment environments.

4.4. Sensitivity analysis

The results presented so far are based on the selection-on-observables assumption. Hence, it is crucial to assess the risk of those estimates being plagued by “hidden bias” through unobserved confounding (Rosenbaum, Reference Rosenbaum2002). As we cannot rule out this possibility, we follow much of the literature on sensitivity analyses and explore how point estimates and possibly inference may change under certain assumptions regarding a relevant but omitted binary confounder.

Vanderweele and Arah (Reference Vanderweele and Arah2011) show that – under some simplifying assumptions – the bias ${B_g}$ due to an unmeasured binary confounder U can be written as:

(3) $${B_g} = {{\rm{\delta }}_{\rm{y}}}\left\{ {\Pr \left( {{\rm{U}} = 1{\rm{|}}{{\rm{D}}_{\rm{g}}} = 1} \right) - \Pr \left( {{\rm{U}} = 1{\rm{|}}{{\rm{D}}_{\rm{g}}} = 0} \right)} \right\},\;$$

where ${{\rm{\delta }}_{\rm{y}}}$ is the population average effect of switching the unobserved confounder ${\rm{U}}$ from zero to one. Similar to Ichino et al. (Reference Ichino, Mealli and Nannicini2008), we use the distribution of observed covariates to inform our analysis about plausible potential unobserved confounder distributions. To provide the most credible results possible, the entire list of available covariates is used. The bias components in Formula (3) are estimated as follows: estimates of ${{\rm{\delta }}_{\rm{y}}}$ are obtained through linear regression of the employment indicator after 33 months on all available covariates among non-participants. The sample shares of covariates among participants and non-participants serve as estimates of their probabilities of possessing the respective characteristic. Based on this information, the true ATT is re-estimated under the assumption that there exists an unmeasured confounder, which follows the same distribution as the observed covariate. Table 4 shows our baseline estimates as presented in Section 4.2 along with information on the distribution of results for the sensitivity analyses.

TABLE 4. Sensitivity analysis

Note: Statistically significant estimates at the 10/5/1% level are marked with */**/***. This table shows estimation results using the sensitivity analysis suggested by Vanderweele and Arah (Reference Vanderweele and Arah2011). The analysis assumes that there is some unobserved covariate that is distributed the same as an observed covariate and re-estimates bias-adjusted causal effects.

As expected, there is some variation in effect estimates, albeit rather limited considering the strong predictiveness of covariates such as age, unemployment and job search history along with regional unemployment on individuals’ employment probabilities.Footnote ¹⁶ Moreover, statistical inference remains unchanged, independent of the type of unobserved confounder assumed. This lack of variation in the bias-adjusted estimates supports our claim that the selection-on-observables assumption seems reasonable. If there is some bias remaining after conditioning on observed characteristics, it appears unlikely to overturn our inference.