1 Introduction↩︎

Election manifestos play a critical role in structuring political campaigns. Campaign communication can influence a party’s reputation, credibility, and competence, which are primary factors in voter decision making [1]. Among the various campaign-related functions fulfilled by manifestos [2], perhaps the most important is the contract they represent between parties and voters in terms of pledges and prioritisation of political issues [3]. Political scientists have long studied how specific pledges translate into government programs and actual policy [4]–[7]. Other work relates specific pledges to the issue clarity of a political party through selective emphasis, which complements salience theory [8]–[10]. For example:

we commit ... 30 per cent tax rebate or cash benefit on the cost of private health insurance premiums

conveys the party’s support for private health insurance, and is more verifiable than:

we will improve the health system.

Issue clarity has also been shown to be influenced by a party’s ideological position and its role in government [10].

Although pledge specificity prediction is an important task for the analysis of party position, priorities, and post-election policy framing, to date, almost all research has relied on manual analysis. is a recent exception to this, in performing speech act classification over political campaign text, where the class schema includes the distinction between specific and vague pledges (binary specificity class).

In this paper, we perform fine-grained pledge specificity prediction, which is more expressive than binary levels [11], [12]. We use a class schema proposed by [13] as detailed in Table ¿tbl:tab:speceg?, which captures seven levels of specificity, forming a non-linear increasing order of commitment and specificity [13]. Given the non-linear nature of the scale, we use deep ordinal regression models for this task, with distributional loss [14], where we model the output as a uni-modal distribution [15]. Our goal is to capture the intuition that a pledge with specificity level \(k\), has higher commitment than all the levels \(< k\), producing a smoothly varying prediction over the ordinal classes. This can be modeled as a uni-modal distribution which has a probability mass that decreases on both sides of the most probable class. Lastly, as it is expensive to obtain large-scale annotations, in addition to developing a novel annotated dataset, we also experiment with a semi-supervised approach by using unlabeled text.

The contributions of this paper are as follows: (1) we develop and release a dataset¹ for fine-grained pledge specificity prediction based on election manifestos covering eleven Australian federal election cycles (1980–2016), from the two major political parties — Labor and Liberal; (2) we propose to use deep ordinal regression models for the prediction task, and evaluate the model under sparse supervision scenarios using the teacher–student framework; and (3) we evaluate the utility of pledge specificity towards ideology prediction, and provide further qualitative analysis by correlating model predictions with party-specific issue salience across major policy areas.

2 Related Work↩︎

Political manifesto text analysis is a relatively novel application, at the intersection of Political Science and NLP. Research has focused primarily on fine-grained policy topic classification and overall ideology prediction tasks [16]–[19]. Most work dealing with pledge specificity analysis in manifestos has been based on manual analysis, as outlined in Section 1.

Specificity is a pragmatic property of text which has been studied across various fields of research. In cognitive linguistics, [20] showed that specificity of information in text impacts reading comprehension speed. In Political Science, it has been used to analyze salience, party position and post-election policy framing (see Section 1). There has also been research on the association between text specificity and communication style. In terms of automated specificity analysis, [21] found specificity in the context of congressional hearings to vary between speakers belonging to the same vs.different ideologies. Namely, it was shown that specificity increases as the ideological distance between the committee chair and the witness decreases. [22] addressed two levels of pledge specificity, as part of speech act classification task. Specificity has also been studied in news [23] and classroom discussion domains [24], [25].

These studies have dealt with a restrictive coarse-level analysis (2–3 categories), whereas a fine-grained scale better captures and allows for comparison of election manifestos [13]. was the first attempt at fine-grained text specificity prediction, in the context of social media posts. Here, we target the novel task of fine-grained pledge specificity prediction, which can be used in a range of downstream applications, including capturing party priorities (salience) and ideological position across election cycles.

All the text specificity analysis work in NLP has modeled the task as classification or regression. As the 7-step pledge specificity levels used in this research [13] do not form a single real-valued scale, we model it as an ordinal regression task. Some examples of ordinal regression tasks include sentiment rating prediction [26], stages of disease prediction [27], and age prediction [28]. Recent work has shown that adding a distributional (auxiliary) loss alongside a regression loss, and using expectation to obtain the predicted value [14], provides label smoothing and improves regression performance [29]. Approaches based on a uni-modal probability distribution (e.g., Poisson) as output [15], [30] can be seen as related to the former approach [14] where the discrete probability mass function replaces the histogram density. We propose to use a uni-modal distributional loss-based ordinal regression for pledge specificity prediction.

Secondly, as it is difficult to obtain large amounts of labeled data, existing approaches have used semi-supervised learning [22], [31]. Here we use a cross-view training approach [22], [32], where we enforce consensus between the intermediate class distributions or the final real-valued output.

3 Pledge Specificity Dataset↩︎

We annotated 22 election manifestos from the Australian Labor and Liberal parties, covering eleven Australian federal election cycles from 1980–2016. The dataset has 12,185 sentences annotated with seven levels of specificity [13]. See Table ¿tbl:tab:speceg? for class definitions and an example of each class. We obtained annotations using the Figure Eight crowdsourcing platform. For each sentence we provided the previous two sentences from the manifesto (as context), the party which published the manifesto, election year, and incumbent and opposition party details. Each sentence was annotated by at least 3 workers after passing quality control (at least 70% accuracy on test questions). After obtaining annotations, the label which has the highest confidence score is chosen for each sentence. Confidence is the level of agreement between multiple contributors, weighted by the contributors’ trust scores. Overall agreement based on the Krippendorf’s Alpha is \(\alpha=0.58\), indicating moderate agreement [33], [34], on par with related studies [12]. The class distribution in the final dataset is given in Table ¿tbl:tab:specdis?, alongside the average sentence length in tokens. It can be seen that more specific pledge categories have higher average length. Average specificity values of Labor and Liberal party manifestos across elections are given in Figure 1. The length of the manifesto (in terms of number of sentences) influences average specificity values, with exceptions such as the Liberal party’s 2010 election manifesto which is the shortest document but has the highest average pledge specificity value. Further detailed analysis, to decipher the pledge specificity trends in general, is a potential task for future work.

4 Proposed Approach↩︎

4.1 Base Model↩︎

We first obtain representations for each sentence via a sequence of word embeddings, to which we apply a bidirectional GRU (“biGRU”: [35]), and concatenate the final hidden state of both the forward and backward GRUs, \(\mathbf{h}_{i} = \left[\overrightarrow{\mathbf{h}}_{i}, \overleftarrow{\mathbf{h}}_{i}\right]\). Rather than using a linear activation layer for the output, we study the effect of learning a distribution over ordinal classes, and using an expectation layer to get the final prediction, which we now expound upon.

4.2 Distributional Loss↩︎

Let us assume that the continuous target variable \(Y\) is normally distributed, conditioned on inputs \(x \in \mathbb{R}^d\), \(Y\! \sim\! \mathcal{N}(\mu\!=\!f(x), \sigma^2)\) for a fixed variance \(\sigma^2\) > 0. In regression, the maximum likelihood function \(f\) for \(n\) samples {\(x_i\), \(y_i\)} corresponds to minimizing \(l_2\) loss, such that \(f(x) = \mathop{\mathrm{\mathbb{E}}}(Y|x)\). Alternatively, we can learn a categorical distribution (\(q_{x}\)) over the ordinal classes \(\mathcal{Y}\), and use the expected value as the prediction, \(f(x)\) [36]. In this work, we follow the latter method, but parameterise the categorical distribution based on uni-modal probability distribution, a technique which has been shown to perform well for ordinal regression tasks [15]. This modification converts the problem to a more difficult (multi-task) problem, that promotes generalization and reduces over-fitting [14]. The overall objective is to jointly minimize the squared loss for the regression task (\(\mathcal{L}_{S}\)), and cross-entropy for the distributional loss over \(\mathcal{Y}\) (\(\mathcal{L}_D\)), based on the objective \(\mathcal{L}_J = \alpha \mathcal{L}_{S} + \mathcal{L}_D\), where the hyper-parameter \(\alpha\) is tuned using a validation set. We experiment with different distributions in generating the intermediate representations \(q_{x}\), including categorical (as a baseline approach, see Section 5: ), discrete uni-modal (Binomial and Poisson: ), and truncated Gaussian [14]. The final prediction is obtained using expectation, which has been shown to be effective for various regression tasks in the vision domain. Here we study the use of uni-modal distributional loss-based ordinal regression approaches [14], [15] for text specificity analysis (Section 5 has results demonstrating its superiority over the other choices). We detail the different ways to obtain \(q_{x}\), and the corresponding loss functions \(\mathcal{L}_D\) below, and provide an overall summary in Figure 2.

4.2.1 Binomial↩︎

With the biGRU model, we estimate the parameter (\(p\)) of the Binomial distribution (with a sigmoid output), based on which the distribution over classes can be obtained via the probability mass function, \[p(k; K - 1, p) = \binom{K-1}{k} p^k (1 - p)^{K-1-k},\] \(k \in \{0, 1, \ldots K-1\}\), where \(p\) \(\in\) [0, 1]. As the final layers (post sigmoid) are under-parametrized, we have a softmax layer with \(\tau\) after obtaining the probability masses, \[q_k = \frac{\exp(\phi_k/\tau)}{\sum^{K-1}_{j=0} \exp(\phi_j/\tau)},\] where \(\tau \sim \text{SoftPlus}(\tau')\), and \(\tau'\) is learned by the deep net, conditioned on the input (\(x\)). We then have an expectation layer to obtain the final output \(f(x)\). Output of the softmax layer is fit to the one-hot encoded ordinal classes for each input (\(\boldsymbol{y}\)), by minimizing the cross-entropy loss (\(\mathcal{L}_{\mathcal{D}\mathrm{\small Binomial}}\)).

4.2.2 Poisson↩︎

Poisson is similar to the binomial case, in that we obtain the parameter (\(\lambda\)) of the Poisson distribution using the biGRU, with a \(\text{SoftPlus}\) activation. We then use the probability mass function of the Poisson distribution to get the probabilities over different classes, \(k \in \mathcal{Y}\), \[p(k; \lambda) = \frac{\lambda ^ {\kern 0.08 em k} e ^ {-\lambda}}{k!},\] which is again passed through a softmax layer to obtain \(q_{x}\), fit by minimizing cross-entropy loss (\(\mathcal{L}_{\mathcal{D}\mathrm{\small Poisson}}\)), and an expectation layer is used to obtain the final prediction.

4.2.3 Gaussian (Gauss)↩︎

To compute \(\mathop{\mathrm{\mathbb{E}}}(Y|x)\) (\(\mu\) of the Gaussian), here we fit the intermediate distribution \(q_{x}\) directly to histogram density of a truncated Gaussian distribution with support \([1, K]\) (target distribution: \(p_{*}\)). We achieve this by learning a prediction distribution with the biGRU model, \(q_{x}\) : \(\mathcal{Y}\) \(\rightarrow [0,1]\). For this, the ordinal label of training instances is transformed into a truncated Gaussian PDF. The mean \(\mu\) for this Gaussian is the target \(y\) of each data-point, with fixed variance \(\sigma^2\), which we set to the radius of the bins in \(\mathcal{Y}\) (1 in this case). The CDF for the chosen target distribution is computed as \(\frac{1}{2} (1+\text{erf}(\frac{x-\mu}{\sigma\sqrt{2}}))\) and \(p_{*}\) is obtained for each class, \(\text{k} \in \mathcal{Y}\), as, \[\frac{1}{2} \bigg(\text{erf}\big(\frac{k-\mu}{\sigma\sqrt{2}}\big)-\text{erf}\big(\frac{k-1-\mu}{\sigma\sqrt{2}}\big)\bigg).\] This formulation allows efficient computation of divergence between \(p_{*}\) and \(q_{x}\) for optimization, which results in cross-entropy minimization (\(\mathcal{L}_{\mathcal{D}\mathrm{\small Gauss}}\): ). Note that the training target \(p_{*}\) is uni-modal, and no constraints are explicitly enforced on the shape of \(q_{x}\).

4.3 Incorporating Context↩︎

We incorporate context in the form of information from adjacent sentences following the approach of : for each training sentence, we use the predicted (intermediate) probability distribution across ordinal classes of the previous \(L\) sentences as context. A new biGRU model is trained with the sentence and the additional contextual information, concatenated to \(\mathbf{h}_{i}\). We refer to this model as \(\text{biGRU}_{\mathrm{\small ord\! \text{+}\! context}}\). In the test phase, biGRU\(_{\mathrm{\small ord}}\) provides contextual information, and the newly trained model (\(\text{biGRU}_{\mathrm{\small ord\! \text{+}\! context}}\)) is used to predict the test sentence output.

4.4 Semi-supervised Learning↩︎

As it is expensive to get large-scale specificity annotations we employ a cross-view training approach [22], [32] for semi-supervised learning, which can leverage additional unlabeled text. Cross-view training is a kind of teacher–student method, whereby the model “teaches” a “student” model to classify unlabelled data. The student has a restricted view over the data, e.g., through the application of noise [37], [38]. We use biGRU\(_{\mathrm{\small ord\! \text{+}\! context}}\) with word-level dropout and zero vector set to contextual information as the auxiliary model. This procedure regularizes the learning of the teacher to be more robust, as well as increasing its exposure to unlabeled text. We augment our dataset with over 32k sentences from UK and US election manifestos released from the same time period. On these unlabeled examples, the model’s output is used to fit the auxiliary model by enforcing consensus in their predictions. This consensus loss \(\mathcal{L}_{\text{U}}\) is added to the supervised training objective (\(\mathcal{L}_{J}\)). Under the semi-supervised setting, we evaluate the following approaches:

5 Experimental Results↩︎

To evaluate model performance we use macro-averaged mean absolute error (MMAE: ) given the class imbalance, and Spearman’s \(\rho\). MMAE is given as \(\frac{1}{|K|}\! \sum \limits_{j=1}^{|K|} \frac{1}{|S_{j}|}\! \sum \limits_{x_{i} \in S_{j}} |f(x_{i}) - y_{i}|\), where \(S_{j}\) denotes the subset of instances annotated with (true) ordinal class \(j\). We consider the following baselines:

All the baseline and proposed biGRU models use ELMo embeddings [40]. The regression models minimize \(l_2\) loss, unless otherwise specified. We compare the average performance across five runs with an 80:20 train:test split. We randomly choose 10% of instances from the training set as validation data. We compare the baseline approaches with our proposed ordinal approaches, which have an intermediate distributional loss in conjunction with the final prediction loss: biGRU\(_{\mathrm{\small Gauss}}\) (Section 4.2.3), biGRU\(_{\mathrm{\small Binomial}}\) (Section 4.2.1), or biGRU\(_{\mathrm{\small Poisson}}\) (Section 4.2.2). We also evaluate biGRU\(_{\mathrm{\small Categorical}}\), where the softmax layer is fitted to one-hot encoded class labels [29], [36]. Note that this is not uni-modal.

used a combination of bag-of-words representation, surface features, social-media-specific features (eg., Tweet mentions), and emotion-related features, with a support vector regression model which minimizes squared loss. Social media and emotion-related attributes are not relevant to our data, and other surface features did not provide improvements. Hence we show the performance of the bag-of-words representation with squared loss objective (NN\(_{\mathrm{\small reg}}\) in Table ¿tbl:tab:res?). From the results in Table ¿tbl:tab:res?, we can see that sequential models with ELMo embeddings (biGRU) perform better than neural bag-of-words models (NN\(_{\mathrm{\small reg}}\)). The \(l_{2}\) regression model (biGRU\(_{\mathrm{\small reg}}\)) performs better than \(l_{1}\) regression (biGRU\(_{\mathrm{\small reg}_{l_1}}\)) and classification (biGRU\(_{\mathrm{\small class}}\)).

With respect to deep ordinal approaches, biGRU\(_{\mathrm{\small Poisson}}\) performs better than classification (biGRU\(_{\mathrm{\small class}}\)), but does not improve upon regression (biGRU\(_{\mathrm{\small reg}}\)). The Binomial performs better than the Poisson, consistent with previous work [15], [30]. biGRU\(_{\mathrm{\small Categorical}}\) performs better than biGRU\(_{\mathrm{\small reg}}\), but not over unimodal approaches (biGRU\(_{\mathrm{\small Binomial}}\), biGRU\(_{\mathrm{\small Gauss}}\)). Overall, the model which fits intermediate distribution to a truncated Gaussian (histogram density) target distribution provides the best performance. It gives over 6% improvement in terms of MMAE, and over 4% in \(\rho\), compared to biGRU\(_{\mathrm{\small reg}}\). Adding context to biGRU\(_{\mathrm{\small Gauss}}\) (biGRU\(_{\mathrm{\small Gauss\! +\! context}}\)) provides a slight reduction in error.

5.1 Semi-supervised Learning↩︎

We next compare the performance of biGRU\(_{\mathrm{\small Gauss\! +\! context}}\) (Sup) and the semi-supervised extensions of it (Section 4.4) which leverage additional unlabeled data: minimizing \(\mathcal{L}_{\text{UMSE}}\) (Mse), \(\mathcal{L}_{\text{UKLD}}\) (Kld), and \(\mathcal{L}_{\text{UEMD}}\) (Emd). The amount of labeled data in the training split is varied from 10% to 90%. Results are presented in Figure 3 for MMAE and \(\rho\). From the results, semi-supervised approaches provide large gains in terms of both MMAE and \(\rho\), especially when training with fewer instances, ratio \(\leq\) 30%. Overall, the semi-supervised learning approach which minimizes Emd performs best across all training ratios compared to both supervised and other semi-supervised approaches. It provides .10 and .06 absolute improvements in \(\rho\) under sparse supervision scenarios (10% and 30% of training data, resp.). Even under richer supervision settings (\(\geq\) 70 %), it provides higher \(\rho\).

6 Political Analysis Using the Models↩︎

Political scientists utilize pledge specificity for a variety of applications (see Section 1). Here, we extrinsically evaluate our specificity model using two tasks related to campaign strategy: (1) party position or ideology prediction (Section 6.1), and (2) issue salience analysis (Section 6.2). For both tasks, we compare the use of pledge specificity across policy issues vs.a count-based representation of policy mentions.

6.1 Ideology Prediction↩︎

Estimating the manifesto-level ideology score on the left–right spectrum using sentence-level policy topic annotations is a popular task [19], [41]–[43], for which the policy scheme provided by CMP [16] is commonly used. It has 57 political themes, across 7 major categories. Among those approaches, the RILE index is the most widely adopted [44], [45], and has been shown to correlate highly with other popular scores [42]. RILE is defined as the difference between count of (pre-determined) right and left policy theme mentions across sentences in a manifesto [46]. Here we evaluate the effectiveness of using the proposed specificity modeling across those policy issues, compared to using RILE-based party position scores [46].

We compute the specificity weight [13] from the average specificity score across sentences, \(\frac{1}{|I|} \sum \limits_{S_{i} \in I} \text{Spec}(S_{i})\) for each policy issue (\(I\)). With specificity weight as the basic feature, we also model global signals such as party coalition and temporal dependencies across elections, which can enforce smoothness in manifesto positions [19], [47] based on probabilistic soft logic.

6.1.1 Probabilistic Soft Logic↩︎

To address this, we propose an approach using hinge-loss Markov random fields (“HL-MRFs”), a scalable class of continuous, conditional graphical models [48]. These models can be specified using Probabilistic Soft Logic (“PSL”: ), a weighted first order logical template language. An example of a PSL rule is \(\lambda: \mathtt{P(a)} \wedge \mathtt{Q(a, b)} \rightarrow \mathtt{R(b)}\), where \(\mathtt{P}\), \(\mathtt{Q}\), and \(\mathtt{R}\) are predicates, \(\mathtt{a}\) and \(\mathtt{b}\) are variables, and \(\lambda\) is the weight associated with the rule. PSL uses soft truth values for predicates in the interval \(\big[0,1\big]\). The degree of ground rule satisfaction is determined using the Lukasiewicz \(t\)-norm and its corresponding co-norm as the relaxation of the logical AND and OR, respectively. The weight of the rule indicates its importance in the HL-MRF probabilistic model, which defines a probability density function of the form: \[\! P(\mathbf{Y}|\mathbf{X}) \propto \exp \left(-\sum_{r=1}^{M} \lambda_{r} \phi_{r}(\mathbf{Y}, \mathbf{X}) \right),\] where \(\phi_r(\mathbf{Y}, \mathbf{X})\!\) \(=\) \(\!\max{\{l_{r}(\mathbf{Y}, \mathbf{X}), 0 \}}^{\rho_{r}}\) is a hinge-loss potential corresponding to an instantiation of a rule, and is specified by a linear function \(l_{r}\) and optional exponent \(\rho_r\) \(\in \{1, 2\}\).

6.1.2 PSL Model↩︎

Here we elaborate on our PSL model based on manifesto content-based features (specificity weight across 57 policy issues), coalition information, and temporal dependencies. Our target \(\mathtt{pos}\) (left–right position) is a continuous variable \(\big[0,1\big]\), where 1 indicates an extreme right position, 0 denotes an extreme left position, and 0.5 indicates center. We also model the social and economic positions explicitly (\(\mathtt{socpos}\) and \(\mathtt{econpos}\)), which influence the overall \(\mathtt{pos}\). Each instance of a manifesto, its party affiliation and policy issues, are denoted by the predicates \(\mathtt{Manifesto}\), \(\mathtt{Party}\) and \(\mathtt{Policy}\). Other predicates are given as follows:

Representative rules of our PSL model, based on the predicates presented above, are given in Table ¿tbl:tab:pslrule?. They include:

6.1.3 Evaluation↩︎

We use manifestos from Australia and UK for our analysis. We use data from Voter Survey [50] for Australia and CHES Expert Survey [51] for the UK as the gold-standard party position. A primary step (related to the model given in Section 6.1.2) is to obtain policy topic classification for sentences in each manifesto. If annotations are not available from , one out of 57 political themes are predicted using the method of [19]. Specificity scores of sentences are obtained using the proposed ordinal regression approach (biGRU\(_{\mathrm{\small Gauss+context}}\)). Using social, economic and a combined list of left–right policy themes (\(\mathtt{IdeologyMap}\)), and with the RILE formulation, we bootstrap \(\mathtt{socpos}\), \(\mathtt{econpos}\) and \(\mathtt{pos}\). We then use the PSL model (Table ¿tbl:tab:pslrule?) to re-calibrate the scores based on specificity scores and the global signals.

We compare the performance of bootstrapped \(\mathtt{pos}\) (RILE or policy count-based) with the PSL model. Principal component analysis (“PCA”: [52]) on the frequency distribution, and projection on its principal component, is used as an additional baseline. Spearman’s correlation (\(\rho\)) against the gold-standard positions is given in Table ¿tbl:tab:psl?. Overall, pledge specificity, especially on a relative scale (which differentiates emphasis between parties) provides large gains, and global signals give only mild improvements.

6.2 Capturing Issue Salience↩︎

For the Australian manifestos (from the Greens, Labor, Liberal, and National parties) we perform a qualitative study of specificity weight across policy themes, by correlating it against the salience of major policy areas given by the Voter Survey [50]. Again we compare its utility over the use of counts across policy themes in a manifesto. Using sentences classified with policy themes and specificity scores using our proposed approach, we construct the following \(|\)Manifestos\(|\) \(\times\) 57 features — frequency distribution (\(\mathbf{C}\)) and pledge specificity weight (\(\mathbf{S}\)) across policy themes. The features are used as independent variables, and voter survey salience scores across major policy areas — health, education, environment, tax, and economy — are treated as dependent variables. Note that the voter survey scores are available for each party and election cycle across policy areas. We build separate multi-variate linear regression models and compare them based on the goodness of fit (log-likelihood). Log-likelihood values are given in Table ¿tbl:tab:lr?: across all policy areas, pledge specificity better captures salience than a count-based representation.

7 Conclusion and Future Work↩︎

In this work we present a new dataset of election campaign texts, annotated with pledge specificity on a fine-grained scale. We study the use of deep ordinal regression approaches using an auxiliary uni-modal distributional loss for this task. The proposed approaches provide large gains in performance under both supervised and semi-supervised settings. Specificity weight across policy issues benefits ideology prediction and also better captures issue salience, compared to the traditional policy theme count-based representation. This aligns with previous studies done based on manual annotations [10]. In future work, we aim to expand this study to multiple languages.

Acknowledgements↩︎

We thank Robert Thomson for his inputs on the task definition. This work was funded in part by the Australian Government Research Training Program Scholarship, and the Australian Research Council.

References↩︎