Synthetic Control

In September 2023, the Virginia General Assembly amended the commonwealth’s budget to include funding for the upcoming ODU & EVMS merger. This represents significant forward progress integrating EVMS with ODU. Both institutions’ websites list quite a few potential benefits of this merger. Since we’re skeptical economists, we might be interested in evaluating some of the claims made by both institutions. How could we investigate the effects of the merger on some outcome?

At first, you might think, “no sweat, just use DiD,” and you’d be right that we probably could use difference-in-differences. However, we’d run into a few issues with this sort of approach. First, with only one treated unit, we’d have very low statistical power and our standard errors would be very large. Therefore, we’d have a lot of variance in our estimate and it would be hard to differentiate it from zero. Second, it might be difficult to select a control group of universities. For example, is JMU a good control unit? What about VCU? UVA? Norfolk State? Even if we tried to calculate inverse probability weights, or use matching, having a single treated unit would severely limit our statistical power.

So, if we can’t use DiD or PSM/IPW, are we out of luck? In this module, we are going to use a relatively new econometric method called Synthetic Control to estimate causal effects. This method is designed for case studies like the ODU/EVMS merger. Unfortunately, since ODU and EVMS have yet to merge, we cannot investigate the effects of the merger. Rather, to demonstrate this method, we are going to replicate a famous study by Abadie et al. (2015) which examines the effects of reunification in Germany in 1990.

These data include per capita GDP (the outcome variable) in addition “inflation rate, industry share of added, investment rate, schooling, and a measure of trade openness”. Variable definitions can be found on page 15. The authors start with 23 countries plus West Germany, but explain why they drop some of them from the analysis in footnote 14. The exclude Luxembourg and Iceland due to their size, Turkey due to its outlier per capita GDP, and Canada, Finland, Sweden, and Ireland due to “profound structure shocks” during the sample period. The authors are left with 17 total countries including West Germany from 1960 to 2003. While the sample of countries are chosen for particular reasons, the time period less so.

In the following code chunk, produce a time series plot of per capita GDP for West Germany and the average per capita GDP across all other countries in the sample. Hint: use aggregate() to calculate averages by time and group.

The plot you generate should look something like the following:

df <- read.csv("https://alexcardazzi.github.io/econ708/data/repgermany.csv")
aggregate(list(real = df$gdp),
          list(year = df$year,
               wg = df$index == 7), mean) -> agg
plot(agg$year, agg$real, type = "n",
     xlab = "Year", ylab = "Per Capita GDP")
lines(agg$year[agg$wg], agg$real[agg$wg], col = "tomato", lwd = 2)
lines(agg$year[!agg$wg], agg$real[!agg$wg], col = "dodgerblue", lwd = 2)
legend("topleft", legend = c("OECD", "W. Germany"),
       lty = 1, col = c("dodgerblue", "tomato"), lwd = 2,
       bty = "n")
abline(v = 1990, lty = 2)

Hopefully it’s clear that an (equally weighted) average of the other OECD countries may not look so similar to West Germany. For example, you can see the two trends diverging over the course of the sample period. In a difference-in-differences framework, diverging pre-trends often signals potential violations of the parallel trends assumption. Of note, so far we have a likely violation of parallel trends and low statistical power resulting from a single treated unit. Let’s continue with our DiD analysis for instructional purposes and completeness. In the next code block, estimate a DiD model of the following form:

First, let’s dive a bit deeper into the units in the control group. We will call this sample of countries the donor pool, since they are candidate contributors to the synthetic West Germany. Some of these “donors” will have more relative importance to the synthetic than others. For example, New Zealand is probably less similar to West Germany than Austria, Belgium, or Denmark, all of which share a border with West Germany. Therefore, New Zealand will probably get less weight than these other countries when constructing the synthetic. Intuitively, this is fairly easy to think through. Mathematically, though, this can get quite complicated. Hopefully, the DiD and PSM/IPW math was not too difficult to follow. The math behind synthetic control is much more cumbersome, and ultimately outside the scope of this course, so we’ll move ahead straight into coding the method.

To estimate a synthetic control for West Germany, we are going to use the Synth package. We will be able to do everything we want using Synth, but there are other packages that make some of the following steps a bit easier. I am choosing not to demonstrate these other packages for two main pedagogical reasons. First, one of these packages makes use of R syntax that is inconsistent with what we’ve used in this course thus far. I can imagine this being confusing, so I would like to stay away from that. Second, some of the packages require additional installation steps that differ across operating systems. I am happy to point interested students to these alternate packages, but I just wanted to make that disclaimer.

Now that we have our weights, let’s generate the per capita GDP time series for our synthetic West Germany. To do this, we need two pieces of information. First, dataprep_out contains the per capita GDP trend for each unit in the donor pool. We can access it via dataprep_out$Y0plot. These data are organized such that each column represents a control unit and each row represents a year. Next, we need the weights for each unit from the synth_out object. We can access this via synth_out$solution.w. We are going to multiply each column in dataprep_out$Y0plot by its respective unit weight in synth_out$solution.w. Then, we are going to add the values across rows so we are left with a single column and many rows. In math, this is called matrix multiplication. To do this in R, we are going to use %*%. dataprep_out also contains the time units (dataprep_out$tag$time.plot) and real West Germany data (dataprep_out$Y1plot). Let’s create a data.frame with this information.

Given this figure, it should be fairly obvious that the results of the synthetic control suggest a negative effect of German reunification on per capita GDP. However, since we have \(Y(D = 1)\) and \(Y(D = 0)\), we can directly calculate a treatment effect via subtraction. This is usually shown as the “gap” between the real and the synthetic. Let’s plot this gap:

plot(sc_w_germany$time, type = "l",
     sc_w_germany$real - sc_w_germany$synth,
     ylab = "Gap in Per-Captia GDP", xlab = "Year", lwd = 3)
abline(h = 0, v = 1990, lty = c(1, 2))

How should we think about this? Initially, there appears to be a positive bump in per capita GDP followed by a large reduction in per capita GDP. An important thought/question you might have now is the following: using DiD, we found a positive effect, but it was insignificant. How do we know this increase (or decrease) is not also insignificant? How can we create a confidence interval for these effects? If you’ve asked these questions, you’re touching one of two most critical weaknesses of synthetic control (the other weakness being how computationally expensive it is compared to other estimators like OLS). The issue of inference in synthetic control is on the current cutting bleeding edge of econometric research.

Inference is unique in synthetic control environments in that we do not calculate standard errors nor confidence intervals. Rather, we perform placebo tests, sometimes called permutation tests, for each control unit in the donor pool. Essentially, one at a time, we pretend that each other unit in the donor pool was treated instead of West Germany. Of course, none of these units were in fact treated, so there shouldn’t be any large changes between the actual and the synthetic. These placebos should provide some variance for us to use when making inference. If we find that most of the placebo effects are as large, or larger, than the estimated effect of our treated unit, then we should reduce our confidence in our estimated treatment effect. On the other hand, though, if the placebo effects for all other treated units are much smaller in magnitude compared to the treatment effect for the treated unit, then we should have more confidence in our estimate.

Now that we have seen the distribution of gaps, we want to create some measure similar to a p-value for inference. Recall the definition of a p-value: the probability of observing an effect at least as large as this given the null hypothesis is true. Of course, our null hypothesis is that reunification did not cause a change in per capita GDP. So, how do we get from gaps to p-values? First, we want to calculate the average (post-period) gap for each unit. Next, we want to order the effect sizes by magnitude. Then, we can directly calculate the probability of observing an effect at least as large as the effect for the treated unit. To calculate the exact p-value, we would divide the position of the treatment effect (for West Germany) in the ordered list by the number of total number of elements in the list. For example, if West Germany had the 3^rd largest effect size out of 17, then its p-value would be \(\frac{3}{17}\), or \(0.176\). Let’s calculate the p-value for West Germany’s estimate using this method.

Now, let’s return to what was previously mentioned about poor fits in the pre-period. If we cannot generate a good fit in the pre-period, we’re also likely to struggle to estimate a valid counterfactual in the post-period. So, how can we quantify “poor” in terms of pre-treatment fit? We determine pre-treatment fit using Root Mean Squared Prediction Error (RMSPE). Essentially, we take an average of the squared gap across pre-period observations. The, we take the square root of this. This is very similar to converting from variance to standard deviation. Then, if the fit is worse than two or three times that of the treated unit’s RMSPE, we remove it from the p-value calculation. For some code that does this, see below:

# Calculate RMSPE for each unit in the pre-period.
aggregate((sc$real-sc$synth)[sc$time < 1990]^2,
          list(sc$i[sc$time < 1990]),
          function(x) sqrt(mean(x))) -> rmspe
# Using 3x as the cutoff.
keep <- rmspe$Group.1[which(rmspe$x <= 3*rmspe$x[rmspe$Group.1==7])]

numerator <- which(effect_size$Group.1[effect_size$Group.1 %in% keep] == 7)
denominator <- nrow(effect_size[effect_size$Group.1 %in% keep,])
cat("p-Value:", round(numerator / denominator, 3))

p-Value: 0.1

Synthetic Control is a novel, intuitive, and revolutionary econometric method. While some economists are skeptical of the method for various reasons (computational issues, lack of standard errors, reliance on case studies), it’s clearly one of the directions applied research is headed.

Often times, casual inference requires a particular setting that satisfies a slue of assumptions before we’re ready to make causal claims. Synthetic control, for all of its weaknesses, it probably one of the more applicable methods in a business setting. Returning to the ODU & EVMS example, it would be quite easy to gather data on other comparable institutions of higher education to estimate a synthetic ODU.