¹

1 Introduction↩︎

Everyday business inquiries frequently revolve around causal inference, specifically seeking to understand the impact of particular business decisions. To address this, three common approaches are typically employed: A/B testing, quasi-experimentation, and observational causal inference methods. While A/B testing and quasi-experimentation are often preferred due to their ability to provide exogenous variation for identification, their implementation can be prohibitively costly or subject to biases resulting from business or technical constraints. Observational causal inference methods, such as matching methods, synthetic control, and double machine learning are designed to mitigate biases, but are not applicable to some questions we aim to address given the data properties. Furthermore, the aforementioned approaches are more effective in measuring the causal impact of single interventions rather than attributing causal impact holistically across multiple interconnected factors that may contribute to the final outcome. In such scenarios, regression approaches provide a more suitable alternative. Regression methods only necessitate aggregated panel data to concurrently identify the causal impact of multiple factors. However, two challenges undermine our ability to confidently affirm that the estimated parameters from regression methods represent the true causal impact. These challenges are the existence of confounding factors and multicollinearity among covariates. While common solutions such as shrinkage estimators, principal component regressions, and partial linear regression are helpful in prediction problems, a crucial limitation hinders their applicability to causal inference problems — they cannot provide the original causal relationships.

This paper introduces a novel approach that specifically addresses the second challenge, namely multicollinearity. To illustrate the practical application and effectivenss of this approach, we demonstrate its implementation in a marketing measurement context (Marketing Mix Marketing) at Airbnb. We also conclude this paper with a discussion on the broad applicability of this approach.

In marketing, one question of paramount importance is to causally attribute sales to spend across channels - such as Google Search, YouTube, Display, etc. However, advertisers often allocate their expenditures across ad channels in a correlated manner, particularly during peak seasons. When attempting to estimate a regression model, highly correlated variables result in larger estimate variances and imprecise attribution of channel contributions to sales. It is not uncommon to observe regression coefficients switching signs when highly correlated inputs are introduced, consequently undermining the confidence of business stakeholders in the model results.

In this marketing application, we have access to panel data consisting of ad impressions categorized by channel and geographic location (Designated Market Area, or DMA) over a specific time period. When we analyze the data by pooling all geographic locations together, we observe a high level of cross-channel correlation. However, it is worth noting that certain geographic locations exhibit higher cross-channel correlations compared to others. To address the issue of multicollinearity, we propose a novel approach that leverages the variations in correlation patterns across different geographic locations. The objective is to restructure the data in a way that significantly reduces the multicollinearity problem. Our proposed method involves utilizing hierarchical clustering to group geographic locations based on their correlation patterns. The key aspect of this approach lies in defining the distance metric used in the clustering algorithm.

In our methodology, we define the distance between two DMAs as the sum of channel-specific distances. Each channel-specific distance measures the similarity in the cross-channel correlation between the two geographic locations. By incorporating this distance metric into the hierarchical clustering process, we can effectively group the DMAs in a manner that minimizes multicollinearity across channels. This innovative approach allows us to transform the data structure, mitigating the challenges posed by multicollinearity and providing a more robust foundation for further analysis. By adopting this methodology, we can improve our understanding of the causal relationships between channels and accurately attribute their impact on sales or other relevant outcomes. We will demonstrate this improvement with both data descriptive evidence and regression results.

The remainder of this paper is organized as follows. In section 2, we describe the Marketing Mix Modelling (MMM) problem formulation and the related work. In section 3, we present the data properties that motivate our approach to reduce multicollinearity. In section 4, we introduce Hierarchical Clustering and the distance metric designed specifically to address the multicollinearity problem. In section 5, we show how this novel method improves results, and in section 6, we briefly discuss other applications that can utilize this methodology.

2 Bayesian Structural Model Formulation↩︎

This paper focuses on providing an innovative and intuitive solution to the notorious multicollinearity problem. To demonstrate the effectiveness of their approach, we apply it to a specific application called Bayesian Marketing Mix Modeling (MMM), built upon [1]. MMM is a widely applied method in the industry for estimating the performance of various marketing channels in a holistic manner. It takes into account factors such as seasonality, trend (representing organic demand), and mix of different marketing channels when forecasting sales. ²

2.1 Marketing Mix Model Setup↩︎

We model sales as a non-linear function of seasonality, and advertisement impressions of each channel with a Bayesian Model. Let \(g\) denote a DMA and \(t = 1, . . . , T\) denote time (we use weekly data). \[y_{g,t} = \mu t^\lambda + seasonality_{t} + \alpha Z_{g,t} + \sum_{k=1}^K \beta_k AdStock(x_{k, g,t})\;+ \epsilon_{g,t}\]

There are \(K\) media channels, and \(G\) DMAs. \(x_{g,t}\) is the impression of channel \(k\) at week \(t\). Let \(y_{g,t}\) be the response variable at week \(t\), which could be sales or log transformed sales. We include \(\mu t^k\), \(seasonality_{g,t}\), and contemporaneous correlation with covariates \(Z_{g,t}\) to capture the evolution of organic sales over time. \(AdStock(x_{g,t})\) is the transformed impressions that captures: (1) diminishing return; (2) lag of the effect; (3) carryover effect of the impressions. [2] uses a different formulation which only estimates the saturation effect. In this paper, we adapt Google’s proposed shape formulation of marketing effects that is more flexible.[1] The \(AdStock\) function can be defined as:

\[AdStock_{k,g} = \bigg(\frac{\sum_{l=0}^{L}\tau_{k}^{(l-\theta_{k})^2} x_{t-l,m}}{\sum_{l=0}^{L}\tau_{k}^{(l-\theta_{k})^2}}\bigg )^\rho\]

\(\rho \in (0, 1]\) captures (potentially diminishing) returns to scale; \(\tau \in (0, 1)\) governs the carryover rate over time; and \(\theta\) indicates the lagged peak effect.

2.2 Account for Confounding factors↩︎

While it is not the main focus of this paper, we take into account confounding factors when modeling trend and seasonality in order to properly capture organic demand. When modeling trend and seasonality, it is crucial to strike the right balance between flexibility and strictness – excessive flexibility may lead to overfitting, whereas overly rigid parametric formulations can result in a poor fit of the model. In this marketing use case, we can easily overfit a model that performs poorly out of sample because we have high dimensional parameters space - we have to estimate 4 parameters per channel. Keeping this tradeoff in mind, in addition to including exponential trend and sinusoidal seasonality following [1], we also include as an additional covariate an index of Google Search query volume for travel and accommodation brands excluding Airbnb (\(Z_{g,t}\) in the above equation), to capture confounding factors that affect organic demand contemporaneously.

3 Data Properties↩︎

3.1 Pre-Process Data↩︎

We take two important steps to pre-process data in preparation for descriptive analysis and modeling. First, we normalize bookings, channel impressions, and the covariate to establish a common scale across DMAs of different sizes. This will make it easier (A) to interpret the impact of a certain level of marketing activity; and (B) to model the common trend and seasonality later. Second, we decompose channel impressions into the common trend and seasonality and residual variation across DMAs, so we can focus on correlation in the residual variation next.

3.2 Characteristics of DMA-Level Data↩︎

There is a decent level of correlation between marketing channels and DMAs, even after eliminating the common trend and seasonality across DMAs for each channel. As Figure 1 shows, the variation in residual impressions is moderately to strongly correlated across the five channels.³

Such correlation is more pronounced across some DMAs than other DMAs. Figure 2 compares two sets of DMAs – DMAs in the \(X\)th ventile of baseline sales exhibit extremely high correlation for four out of the five marketing channels, while DMAs in the \(Y\)th ventile exhibit relatively little correlation for all channels.⁴

4 Hierarchical Clustering as a Novel Solution to Multicollinearity↩︎

As overviewed in Section 1 and illustrated in Section 3, multicollinearity poses a fundamental challenge in separately measuring the impact of different marketing channels. While common solutions such as shrinkage estimators, principal component regressions, and partial linear regression are helpful in prediction problems, a crucial limitation hinders their applicability to causal inference problems — they cannot provide the original causal relationships for business interpretability.

To overcome this limitation, we propose a novel and intuitive approach that defines distances and hierarchically clusters geographic areas in a way that effectively mitigates cross-channel multicollinearity. We first calculate a pairwise distance or dissimilarity metric to summarize marketing spend correlation between geos and then use that metric to cluster the geos with moderate to strong correlation. For each channel \(k\), we calculate the distance between two DMAs \(i\) and \(j\) as follows, where \(X_{ik}\) denotes the time series of residual impressions, after eliminating the common trend and seasonality, for channel \(k\) in DMA \(i\). \[Distance_{ijk} = 1 - Correlation(X_{ik}, X_{jk})\] We then calculate an overall distance across all channels, which is the square-root of the sum of squared distances across the channels. \[Distance_{ij} = \sqrt{\sum_{k=1}^K Distance_{ijk}^2}\]

This distance measure reflects correlation between DMAs across multiple channels and is used to hierarchically cluster DMAs. We adopt a complete-linkage hierarchical clustering algorithm which works as follows:[[3]][4]

1. Start with assigning each DMA to its own cluster;

2. Then proceed iteratively, joining the two most similar clusters at each step, continuing until there is just a single cluster. Distance or dissimilarity between two clusters is based on the farthest pair.

This algorithm produces a dendrogram in Figure 3, which illustrates how DMAs are clustered at each step. The horizontal axis lays out the DMAs while the vertical axis shows the distance. This algorithm offers a lot of flexibility in how aggressively we want to cluster DMAs or how many clusters we want to have – we can pick any cutoff distance between 0 (i.e., each DMA in a separate cluster) and 4 (i.e., all DMAs in one cluster).

We use a cutoff distance of 1.5, which corresponds to correlation of at least 0.33 on average for each channel and produces 42 clusters, but also consider alternative clustering strategies for sensitivity. Intuitively speaking, we group DMAs that feature moderate to strong correlation into the same cluster.

5 Results↩︎

5.1 Hierarchical Clustering of DMAs↩︎

The hierarchical clustering approach produces intuitive results. We visualize channel impressions over time across DMAs within each cluster, confirming that DMAs within the same cluster tend to have highly correlated impressions over time for at least some channels. Figure 4 exemplifies such patterns using one small cluster (Cluster 1) and one larger cluster (Cluster 4). Each chart illustrates the variation in channel residual impression over time (the horizontal axis) and across DMAs (the vertical axis). Within each cluster, the color patterns over time are quite similar across DMAs for most channels, reflecting moderate to high correlation.

5.2 Descriptive Evidence Shows Clustering Mitigates Multicollinearity↩︎

Descriptive evidence confirms our intuition that hierarchical clustering can effectively mitigate collinearity. By grouping moderately to highly correlated DMAs into the same cluster, we have significantly reduced correlation in residual channel impressions. As Figure 5 visualizes, correlation decreased generally across channels, by \(8\%\) to \(43\%\).

Further, as Figure 6 demonstrates, clustering preserves variation in channel impressions both (A) within clusters over time and (B) across clusters within the same time period. This is promising for separately identifying the impact of different channels using panel data at the cluster-week level. When testing alternative clustering strategies, we consider the reduction in correlation and the preservation of variation as two important criteria.

5.3 Regression Results Confirm Clustering Alleviates Multicollinearity↩︎

Panel linear regression analysis affirms the effectiveness of this hierarchical clustering method in mitigating collinearity and facilitating the separate identification of the impact of different marketing channels. After clustering, channel coefficient estimates are no longer subject to the problem of flipped signs when included together with other channels, and instead produce mostly intuitive results.

Table 1 summarizes panel linear regression results before and after clustering, with geo (DMA or cluster) fixed effects and week fixed effects included throughout the different specifications.⁵ As Column 1 summarizes, most channel coefficients are negative if we use DMA-level data, while they would be positive if included individually. After clustering, in Column 2, the results become mostly intuitive – all four lower-funnel channels have positive estimated impact on sales (three of which are significant at 0.001 level). The remaining channel with a negative coefficient is upper-funnel, where we expect extreme difficulty in detecting a lower funnel impact. Finally, these findings are robust to weighting the cluster based on the natural log of their baseline size.

5.4 Bayesian Model Results Using Cluster-Level Data↩︎

Now that both descriptive evidence and frequentist regression analysis affirm our hierarchical clustering approach effectively mitigates collinearity, we move forward to estimate the Bayesian model described in Section 2 using data at the cluster level. Similar to the frequentist regressions, the Bayesian model also produces intuitive results, even with uninformative priors. Figure 7 visualizes the posterior distribution of channel-specific impact parameters (\(\beta\)) and carryover rate parameters (\(\tau\)). Consistently with frequentist regression results earlier, we estimate a higher impact for channels A and B than the other channels.⁶ Furthermore, the carryover estimates are also intuitive and consistent with our previous learning and knowledge about the different channels. For example, we expect some carryover for Channels C, D, and E, but not for Channels A and B, and it is reassuring that the estimates confirm that understanding even though we are using uninformative priors.⁷

6 Discussion of Broader Applications↩︎

We have focused on an application in marketing mix modeling to demonstrate how and why hierarchical clustering can mitigate multicollinearity. However, this method is not constrained to the setting of marketing and instead is generally applicable to observational causal inference problems featuring multicollinearity. In our example, marketing data properties motivated us to cluster geographic units based on correlation in marketing activities. In other settings, one can decide which dimensions and criteria to use for clustering based on relevant data properties. The dimension to cluster data does not always have to be geographic. Futher more, in some settings, natural clusters might exist for one to consider.

For example, in the context of Customer Service, we would like to understand how customers’ each interaction with our support agents contribute to the long term retention. However, oftentimes, these interaction experience metrics are highly correlated, such as wait time and abandon rate, etc. In this case, we can leverage clustering to segment customer issue types into groups that have different degrees of correlation between wait and abandon rate.

7 Conclusion↩︎

In this paper, we propose to employ hierarchical clustering as an innovative and effective approach to address multicollinearity in regressional causal inference studies. It has several advantages. Firstly, hierarchical clustering provides a systematic and comprehensive method for identifying clusters that exhibit varying levels of multicollinearity, thus reducing the correlation of covariates across clusters. Furthermore, clustering circumvents the need to transform data into non-interpretable entities, as required by techniques such as Principal Component Analysis or Partial Linear Regressions. This ensures that the interpretability and meaningfulness of the variables are preserved throughout the analysis. In addition to its effectiveness, the proposed methodology is characterized by its ease of implementation. It can be readily applied to diverse applications facing similar challenges related to multicollinearity. The key lies in understanding the inherent properties of the data to define an appropriate distance metric for clustering that effectively reduces multicollinearity. This research contributes to enhancing the robustness and reliability of regressional causal inference studies.

8 Acknowledgement↩︎

The authors would like to thank Carolina Barcenas (Airbnb, Inc.), Mike Anderson (Google LLC), Fei Cen (Google LLC) for their help and guidance on this project.

[5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18]

References↩︎