1 Introduction↩︎

Abstractive summarization [1]–[3] aims to generate summaries by rephrasing or introducing novel words to capture the most salient information in the source text. Many abstractive summarization models [4]–[7] are based on the Transformers architecture [8] and have consistently produced state-of-the-art summarization quality. However, issues such as exposure bias [9], [10], lack of text generation diversity [11], and insufficient capturing of semantic information [12], [13] remain.

Variational models have gained increasing research interest [14]–[17] as they address these issues by introducing uncertainty in predictions through learning a probability distribution over latent variables. A variational model enables diverse text generation [18], smoother output spaces, and semantically meaningful latent codes [16] that guide the generation of coherent and informative summaries.

Nonetheless, existing variational models have not fully achieved the aforementioned desirable properties due to two main challenges. Firstly, the semantic information in the source text may possess a complex structure. However, since introducing latent variables complicates parameter estimation, many current models [17], [19] represent latent codes using a Gaussian distribution, which is insufficient for capturing the intricacies of the latent space and could potentially reduce model performance. To enrich latent distributions, researchers suggest replacing the highly restricted isotropic Gaussian with normalizing flows [20]. Normalizing flows can generate complex distributions while preserving density in an analytical form, and they have been integrated into variational autoencoder (VAE) [21], [22] and variational encoder-decoder (VED) [23], [24] frameworks to better approximate the latent posterior. This approach has found application in various domains, including text generation [16], neural machine translation [25], and dialogue generation [26]. Despite this progress, the operating characteristics of normalizing flows on summarization tasks have yet to be investigated.

Secondly, as reported by previous studies [27]–[29], variational models tend to experience posterior collapse during training, which occurs when the KL term vanishes to zero, indicating that the model fails to learn meaningful latent codes. This problem becomes more severe when modeling discrete data with a strong auto-regressive decoder [30], which is the case for Transformer-based summarization models. To resolve this issue, several solutions have been proposed, such as employing a less auto-regressive decoder network [31]–[33], modifying the training objective [34]–[36], and proposing new training strategies [30], [37]. However, most existing work focuses on the VAE framework with Gaussian latent distribution, yet limited work considers the VED framework with normalizing flows. In particular, two questions remain unclear: (1) when the latent distribution is modeled by normalizing flows, does the posterior collapse problem still exist? (2) when posterior collapse exists, what are the appropriate strategies to achieve good summarization quality within the VED framework?

This paper introduces FlowSUM¹, a normalizing flows-based VED framework for Transformer-based summarization, along with a controlled alternate aggressive training (CAAT) strategy and a refined gate mechanism to resolve the two challenging issues. Our contributions include:

1. We employ normalizing flows to enrich the latent posterior distribution and integrate the latent code into Transformer-based models in a plug-and-play manner, demonstrating its effectiveness through extensive experiments.

2. We propose a controlled alternate aggressive training strategy and a refined gate mechanism to mitigate the posterior collapse problem and improve training efficacy.

3. Our findings suggest that FlowSUM facilitates knowledge distillation while having a negligible effect on inference time, implying normalizing flows’ potential for transferring knowledge from advanced large language models.

4. We investigate the posterior collapse problem for different normalizing flows and examine how the quality of a summary is impacted by the training strategy, gate initialization, and the type and depth of normalizing flows.

This article consists of five sections. Section 2 provides an overview of normalizing flows, VED, and a summary of related studies. Section 3 describes the proposed model architecture and the training strategies employed. Section 4 presents the experimental setup and results, and Section 5 concludes the paper with some discussions.

2 Backgrounds↩︎

2.1 Normalizing Flows↩︎

Normalizing flows (NF) [20] is a type of generative model that has gained popularity in recent years. The fundamental idea involves mapping a simple probability density (e.g., Gaussian) to a more complex one through a series of invertible transformations. One of the key advantages of NF is that it allows for exact likelihood evaluations, which is crucial for many applications such as density estimation [38], data generation [39], and variational inference [28]. A flow-based model consists of two components: a base distribution \(p_{\mathrm{u}}(\mathbf{u})\) and a transformation \(f(\cdot): \mathbb{R}^D \rightarrow \mathbb{R}^D\), where \(f\) must be invertible and both \(f\) and \(f^{-1}\) must be differentiable. Let \(\mathbf{x} = f(\mathbf{u})\) where \(\mathbf{u} \sim p_{\mathrm{u}}(\mathbf{u})\), then the density of \(\mathbf{x}\) can be obtained via a change of variables [40]: \[\label{eq:32nf-px} \begin{align} p_{\mathrm{x}}(\mathbf{x}) & = p_{\mathrm{u}}(\mathbf{u})\left|\operatorname{det} J_f(\mathbf{u})\right|^{-1} \\ & = p_{\mathrm{u}}(f^{-1}(\mathbf{x}))\left|\operatorname{det} J_{f^{-1}}(\mathbf{x})\right|. \end{align}\tag{1}\] In this paper, we examine several NFs, including planar flows [20], radial flows [20], Sylvester flows [41], real-valued non-volume preserving (RealNVP) transformation [42], inverse autoregressive flow (IAF) [28], rational-quadratic neural spline flows (RQNSF) [43], and rational-linear neural spline flows (RLNSF) [44]. We delegate the detailed discussion of transformation and invertibility to Appendix 15. Throughout the paper, for each type, we compose \(K\) layers of transformation \(f_K \circ \cdots \circ f_1 (\cdot)\), which remains invertible and differentiable.

2.2 Variational Encoder-Decoders↩︎

Variational encoder-decoders (VEDs) [14], [23], [24], [45], which can be seen as an extension of variational autoencoders (VAEs) [21], [22], have been widely used to understand the conditional data generation process. Given an input \(x\), the framework posits the existence of a latent variable \(z \sim p(z \mid x; \phi)\), and the generation of \(y\) relies on \(p(y | x, z; \theta)\). With this premise, the conditional data generation can be formulated as in Eq. 2 . \[\label{eq:32conditional-dg} p(y \mid x; \phi, \theta) = \int p(z \mid x; \phi) p(y \mid x, z; \theta) dz\tag{2}\] Since the marginal \(p(y \mid x; \phi, \theta)\) is intractable, we employ variational inference to estimate the parameters. This involves maximizing the evidence lower bound (ELBO), a surrogate of the log-likelihood, as defined in Eq. 3 . The underlying idea is to propose a parameterized distribution \(q(z \mid x, y; \psi)\), known as the variational posterior, to approximate the true posterior distribution \(p(z \mid x, y; \phi, \theta)\). The greater the flexibility in \(q(z \mid x, y; \psi)\), the better the approximation, and the more effective the surrogate ELBO becomes. See more details in Appendix 7. \[\label{eq:32elbo-ved} \begin{align} & \text{ELBO}_{\text{VED}} \\ = & {!}{\displaystyle\underset{q(z \mid x, y; \psi)}{\mathbb{E}}[\log p(y \mid x, z; \theta)]-\text{KL}(q(z \mid x, y; \psi) \| p(z \mid x; \phi))} \end{align}\tag{3}\] For summarization, we parameterize \(p(y \mid x, z; \theta)\) as an encoder-decoder model that generates summaries conditioned on the input text and latent code.

2.3 Related Work↩︎

2.3.1 Transformer-based Summarization Models↩︎

Transformer-based models equipped with pre-training and fine-tuning techniques have enjoyed significant success in many NLP tasks, including text summarization. [4] proposed BertSUM for extractive and abstractive tasks, utilizing the pre-trained BERT encoder [46]. To better align the pre-trained encoder for document understanding with the decoder trained from scratch for text generation, [6] demonstrated the effectiveness of leveraging pre-trained BERT [46], GPT-2 [47], and RoBERTa [48] checkpoints to build sequence-to-sequence (S2S) models for tasks including summarization. Another approach is to address both document understanding and generation in a unified framework by first pre-training some general-purpose S2S models and then fine-tuning on downstream tasks, for instance, BART [49], MASS [50], UniLM [51], ProphetNet [10], and T5 [7]. In addition, [5] proposed PEGASUS with a pre-training objective tailored for abstractive summarization, achieving significant improvements across multiple datasets.

2.3.2 Variational Summarization↩︎

Variational summarization models come in two different flavors: unsupervised and supervised. In the unsupervised domain, researchers commonly utilize variational autoencoders in conjunction with specific control mechanisms for summary generation, as exemplified by prior work such as [52]–[54]. In the supervised realm, there are generally two primary approaches. The first approach models the conditional probability of the target sentences \(p(y \mid x)\) as in Eq. 2 , whereas the second approach models the joint probability of the source and target sentences \(p(x, y)\) with \(\int p(z) p(x \mid z) p(y \mid z, x) dz\). Our model belongs to the first category, akin to prior studies like [17], [25]. In contrast, other works, including [19], [55], [56], adopt the second type by jointly modeling topics and sequence-to-sequence generation. Most of them assume a simple Gaussian latent prior, except for [55], which employs normalizing flows to model neural topic models and enrich global semantics. However, they did not specify the choice of normalizing flows and how they addressed posterior collapse. To the best of our knowledge, there remains limited research on the application of normalizing flows in variational summarization models and their operating characteristics.

3 Normalizing Flows Enhanced Summarization Model↩︎

3.1 FlowSUM Model Architecture↩︎

As illustrated in Fig. 1, FlowSUM consists of three components: an NF latent module, a Transformer-based encoder-decoder, and a refined gate mechanism. The NF latent module focuses on modeling the variational posterior \(q(z \mid x, y; \psi)\), whereas the encoder-decoder, combined with the refined gate, models the conditional generation \(p(y | x, z; \theta)\) with latent code. As a simplification, we assume the conditional prior \(p(z \mid x; \phi)\) is a standard Gaussian as in [25]. Throughout this section, let \(e\) be the embedding size, \(m, n\) be the length of the input source and target summary respectively, \(\ell\) be the latent dimension of the NF latent module, \(d\) be the dimension of the decoder’s hidden states, \(\{x_i\}_{i=1}^{m}\) be the input source text, \(\{y_j\}_{j=1}^{n}\) be the target summary text, and \(\overline{x} \in \mathbb{R}^e\) be the average embedding of the untruncated input source text².

NF Latent Module. To model the variational posterior \(q(z \mid x, y; \psi)\), we follow [24] and assume all the information in \(y\) is contained in \(x\)³. Therefore, we have \(q(z \mid x, y; \psi) = q(z \mid x; \psi)\), which allows us to parameterize \(q(z \mid x; \psi)\) with neural networks (NNs) and normalizing flows using the amortization and reparameterization tricks [21]. The NF latent module comprises of an inference network \(q_0(\cdot)\) and a normalizing flows model. The inference network takes \(\overline{x}\) as input and produces two output vectors, \(\mu_0 \in \mathbb{R}^{\ell}\) and \(\log (\sigma_0) \in \mathbb{R}^{\ell}\). Using the reparameterization trick, a random sample \(z_0 \in \mathbb{R}^{\ell}\) is drawn from \(N(\mu_0, \text{diag}(\sigma_0^2))\). Afterward, the normalizing flows model applies a sequence of \(K\) invertible transformations to \(z_0\) to obtain the latent code \(z = z_K = f_K \circ \cdots \circ f_1(z_0) \in \mathbb{R}^{\ell}\).⁴ Note that when \(K = 0\), the model reverts to the traditional VED framework, and we refer to this degenerated version as VEDSUM.

Gated Transformer-based Encoder-Decoder. Our model adopts the Transformer-based encoder-decoder. The encoder processes the input text and learns a sequence of hidden representations, and the decoder generates a summary based on the encoder’s hidden states and the previously generated tokens. We incorporate the latent information into the decoder with a gate mechanism, which mixes the latent vector \(z_K\) with the decoder’s last layer of hidden states \(\{h_j\}_{j=1}^n\). As pointed out in [57], the saturation property of traditional gating mechanisms hinders gradient-based optimization. Therefore, following their proposal, we use a refined gate mechanism designed to allow for better gradient flow. Let \(\sigma(\cdot)\) be the sigmoid function. We generate the gated fused hidden states \(\{h_j^\prime\}_{j=1}^n\) as in Eq. 4 . \[\label{eq:32refined-gate} \begin{align} & z_K^{\prime}=W^{z} z_K \in \mathbb{R}^d, \text{ where } W^z \in \mathbb{R}^{d \times \ell} \\ & f_j=\delta\left(W^f\left[h_j ; z_K^{\prime}\right]\right) \in \mathbb{R}^d, \text{ where } W^f \in \mathbb{R}^{d \times 2 d} \\ & r_j=\delta\left(W^r\left[h_j ; z_K^{\prime}\right]\right) \in \mathbb{R}^d, \text{ where } W^r \in \mathbb{R}^{d \times 2 d} \\ & g_j=\left(1-r_j\right) \cdot f_j^2+r_j\left(1-\left(1-f_j\right)^2\right) \in \mathbb{R}^d \\ & h_j^{\prime}=\left(1-g_j\right) \cdot h_j+g_j \cdot z_K^{\prime} \in \mathbb{R}^d \end{align}\tag{4}\] Afterward, the fused hidden states are passed to a language model (LM) Head layer, where they are transformed into vectors modeling the probabilities of each word in the vocabulary.

3.2 Training Objective↩︎

Traditional VEDs usually assume \(q(z \mid x; \psi)\) to be a Gaussian, allowing analytical computation of the KL term in ELBO. However, in our normalizing flows-based VED, the variational posterior \(q(z \mid x) = q_K (z_K \mid x)\) can be complex and hence the KL term in Eq. 3 lacks an analytical form. Therefore, we rewrite the ELBO via a change of variables to enable analytical evaluation⁵: \[\label{eq:32elbo-nf} \begin{align} & \text{ELBO}_{\text{NF-VED}} \\ = & \mathbb{E}_{q_0\left(z_0\right)}\left[\log p\left(y \mid x, z_K\right) + \log p\left(z_K \mid x\right)\right] \\ - & {!}{\mathbb{E}_{q_0\left(z_0\right)}\left[\log q_0\left(z_0\right)-\sum_{k=1}^K \log \left|\operatorname{det} J_{f_k}\left(z_{k-1}\right)\right| \right]}, \end{align}\tag{5}\] where \(q_0\) is \(z_0\)’s probability density function, a Gaussian distribution modeled by NNs, and \(\operatorname{det} J_{f_k}(\cdot)\) is the determinant of \(f_k\)’s Jacobian.

Let \(\mathcal{L}_{\text{CE}}\) denote the cross-entropy loss and \(\mathcal{L}_{\text{VI}}\) denote the loss introduced by the variational latent module. Applying the idea of Monte Carlo to Eq. 5 , we obtain the training objective as below. Note that \(\mathcal{L}_{\text{VI}}\) is a Monte Carlo estimate of the KL divergence between the variational posterior \(q_K\) and the conditional prior distribution \(p(z_K \mid x)\). \[\label{eq:32loss} \begin{align} \mathcal{L} & = \mathcal{L}_{\text{CE}} + \mathcal{L}_{\text{VI}} \\ & = -{\textstyle\sum}_{j=1}^n \log p\left(y_j \mid\left\{x_i\right\}_{i=1}^m, z_K, y_{<j}\right) \\ & + \log q_0\left(z_0\right)-{\textstyle\sum}_{k=1}^K \log \left|\operatorname{det} J_{f_k}\left(z_{k-1}\right)\right| \\ & - \log p\left(z_K \mid x\right) \end{align}\tag{6}\]

3.3 Mitigating Posterior Collapse↩︎

To remedy posterior collapse, we consider two strategies, aiming to preserve the expressiveness of the latent variable and improve the overall summary quality. The first approach, called \(\beta_C\)-VAE [36], replaces the KL term with \(\beta|KL-C|\), where \(\beta\) is a scaling factor, and \(C \geq 0\) is a threshold that regulates the magnitude of the KL term. When \(C > 0\), the KL term is expected to be discouraged from getting close to \(0\).

We propose the second approach, Controlled Alternate Aggressive Training (CAAT), inspired by the lagging inference strategy [30]. This strategy uses the observation that the inference network cannot accurately approximate the true posterior in the initial stages of training. As outlined in Alg. 2 in Appendix 6, CAAT comprises two stages. In the first stage, we alternately update the variational parameters and the entire parameters⁶ for a specified number of steps. In the second stage, we train all parameters jointly, as in basic VAE training, for the remainder of the training.

3.4 NF-enhanced Knowledge Distillation↩︎

Normalizing flows can learn complex and multi-modal distributions [38], which makes them a promising approach for knowledge distillation tasks that involve integrating information from multiple sources [58]. To investigate the impact of normalizing flows on knowledge distillation, we adopt two knowledge distillation methods by [59]: Shrink and Fine-Tune (SFT) and Pseudo-labels (PL). SFT shrinks the teacher model and re-finetunes the shrunk model. In contrast, the PL method initializes the student model with the compressed version produced by SFT and then fine-tunes using the pseudo-labeled data generated by the teacher model. In this study, we fine-tune the model on the augmented data with both original and pseudo-labeled data, enabling it to more effectively switch between generated summaries and ground truth, thereby mitigating exposure bias.

4 Experiments↩︎

4.1 Datasets↩︎

We evaluate the effectiveness of FlowSUM on six public benchmark datasets⁷, including CNN/Daily Mail (CNN/DM) [60], XSum [61], Multi-News [62], arXiv, PubMed [63], and SAMSum [64]. These datasets exhibit various summary styles and lengths, and their corresponding statistics are shown in Table 1. Refer to Appendix 10 for more details.

4.2 Implementation Details↩︎

We configure the inference net \(q_0(z_0|\overline{x})\) to be a feedforward neural network and set the latent dimension \(\ell\) to 300 and the number of NF layers \(K \in \{2, 4, 6, 8\}\). For models that use \(\beta_C\)-VAE, we set \(\beta=1\) and \(C = 0.1\), and for those using CAAT, we conduct one epoch of aggressive training with \(n_{alt} = 15\) and two epochs of non-aggressive training. See more details in Appendix 12.

4.3 Baselines↩︎

We use BART [49] and BERT2BERT [6] as two backbone models. We refer to the PL knowledge distilled FlowSUM as FlowSUM-PLKD. Our comparison involves the following baselines: PG+Cov [1], BERT2BERT [6], BERTSUM [4], BART [49], PEGASUS [5], VHTM [17], TAS [19], and PEGASUS+Flow-NTM [55]. See Appendix 11 for more detailed descriptions.

4.4 Results↩︎

4.4.1 Automatic Evaluation↩︎

We evaluate the generated summary quality using ROUGE scores [65] and BERTScore [66]⁸. Specifically, we utilize the overlap of unigrams and bigrams (ROUGE-1 and ROUGE-2) to evaluate the informativeness, and the longest common subsequence (ROUGE-L) for fluency. Moreover, we report BERTScore, which gauges semantic similarity based on contextual embeddings. Furthermore, we present rep-w [67]⁹ and the average length of summaries to gain a better understanding of the quality.

We compare the proposed model against baseline models in ROUGE scores in Tables 2 and 3. On CNN/DM, FlowSUM (BERT2BERT) greatly outperforms BERT2BERT, whereas VEDSUM adds noise to the model and leads to a decrease in performance. With the BART backbone, FlowSUM achieves an absolute improvement over the BART model with +0.48, +0.08, and +0.75 in R-1, 2, and L scores, respectively. However, on XSum, the variational models do not perform well when the gold summaries involve only one sentence. VEDSUM leads to a significant decrease in performance, whereas with FlowSUM, the decrease in ROUGE scores is less severe, leading to +0.12, -0.15, and -0.25 in R-1, 2, and L scores, respectively.

Table 4 uses BART as the backbone and compares BART, VEDSUM, and FlowSUM across all datasets. Overall, variational models produce summaries of superior quality for datasets with long summaries, such as CNN/DM, Multi-News, arXiv, and PubMed, and FlowSUM further enhances the performance beyond VEDSUM. However, when it comes to datasets featuring short summaries such as XSum and SAMSum, the variational component markedly diminishes the model performance. We hypothesize that brief summaries may be more susceptible to disturbances and are more prone to being affected by noise. Nevertheless, incorporating NF modules alleviates these reductions and accomplishes comparable outcomes. Furthermore, we observe that both variational models tend to generate lengthier summaries, while FlowSUM exhibits fewer issues with repetition compared to VEDSUM.

4.4.2 On NF-enhanced Knowledge Distillation↩︎

We use PEGASUS as the teacher model to generate pseudo-labels on the CNN/DM training set. In this study, we explore the effects of knowledge distillation on BART and DistilBART, a shrunken version of BART. We examine two variations of DistilBART: dBART-6-6, which replicates 6 layers¹⁰ of the BART encoder and decoder, and dBART-12-3, which duplicates all layers of the BART encoder and 3 layers¹¹ of the decoder.

Table ¿tbl:table:32kd95on95bart95cnndm? presents the impact of the PL approach on the original BART model. Training the BART model on augmented data worsens the performance compared to training on the original data. In contrast, VEDSUM-PLKD achieves improvements in all three ROUGE scores, and FlowSUM-PLKD with RQNSF achieves the highest R-2 score, albeit with some sacrifice in R-1 and R-L¹². However, planar flows appear to be unsuitable for knowledge distillation via PL. To better understand FlowSUM-PLKD, we visualize the latent distribution (see Appendix 14) and demonstrate how the NF’s ability to capture multi-modality could account for its impressive performance.

Table 5 investigates the two DistilBART variants with RQNSF. With FlowSUM, both variants achieve improvements, suggesting that NF is beneficial for the SFT approach. Previous experiments from [59] showed that PL performed worse than SFT on CNN/DM. However, our experiments reveal that the NF latent module unleashes the potential of PL. When trained on augmented data, FlowSUM-PLKD (dBART-6-6) achieves R-1/2/L improvements of 0.92/0.47/1.01 over dBART-6-6, and FlowSUM-PLKD (dBART-12-3) achieves improvements of 0.66/0.49/0.63 over dBART-12-3, much more than the SFT approach. Furthermore, FlowSUM does not introduce additional computational burden at inference, and the time cost is primarily related to the length of the generated summaries.

4.4.3 Analysis on NF Types and Depth↩︎

We investigate the effect of NF types and the number of NF layers on the Multi-News dataset¹³. Table 6 explores the effect of NF types. Simple flows like Planar and Radial yield inferior performance compared to the VAE counterpart, whereas more complex flows tend to achieve greater improvements. Overall, IAF and RQNSF emerge as the best-performing NF types.

Table 7 delves further into IAF and RQNSF, investigating the effect of NF depth. The findings indicate that adding more layers does not always lead to improved performance. We hypothesize that when the encoder-decoder model is well-trained, the increased complexity of the NF module may introduce more noise, outweighing the benefits of better latent modeling and subsequently worsening the summary quality.

4.4.4 Analysis on Training Strategies↩︎

We implement standard VAE training, \(\beta_C\)-VAE, and CAAT on VEDSUM and FlowSUM models, and we evaluate their effectiveness with different types of normalizing flows. Table ¿tbl:table:32effect-of-training-strategies? shows that VEDSUM and FlowSUM models with residual flows, including planar, radial, and Sylvester flows, suffer from posterior collapse, whereas those with more complex flows do not. Moreover, applying \(\beta_C\)-VAE to VEDSUM and FlowSUM models with residual flows does not effectively mitigate posterior collapse but even exacerbates the issue. Furthermore, for models with planar, RealNVP, and IAF flows, training with \(\beta_C\)-VAE worsens ROUGE scores, while for radial and Sylvester flows, it improves performance. Notably, the two neural spline flows are not impacted by \(\beta_C\)-VAE training.

Concerning CAAT, we note that applying it to treat severe posterior collapses such as VEDSUM and FlowSUM with residual flows can cause instability in training while producing NaN values. Hence, it is only effective for models with KL divergence that is not close to zero. Nonetheless, when applicable, CAAT enhances the quality of summaries, particularly when utilized with the top-performing NFs, namely IAF and RQNSF.

In addition, we explore the impact of gate score initialization. The standard method initializes gating weights with small deviations from zero, resulting in an initial gate score close to 0.5. In contrast, the near-zero initialization method initializes gating weights such that the resulting gate score is approximately 0.05. Our experiments using FlowSUM (BERT2BERT) with RQNSF as the base model reveal that CAAT + Standard Gate Score Initialization yields the best results and the most stable training process, as illustrated in Table 8 and Figures 3 to 4 in Appendix 13. This suggests that by setting a large initial gate score and forcing the model to learn from the NF latent module, we can better capture latent code information.

5 Conclusions and Discussions↩︎

This paper introduces FlowSUM, a normalizing flows-based Variational Encoder-Decoder (VED) framework for text summarization. It outperforms a leading non-latent model across multiple datasets. This enhanced performance is attributed to the flexible posterior distributions provided by normalizing flows. We also analyze the operating characteristics and the posterior collapse problem of normalizing flows and propose an effective training strategy for complex flows. Moreover, we demonstrate that incorporating normalizing flows is highly effective for knowledge distillation with minimal impact on inference time.

FlowSUM illustrates the advantages of incorporating flexible latent modeling. Considering the remarkable achievements of Latent Diffusion Models (LDMs) in generating images [68], adopting LDMs for capturing latent representation may produce comparable or even superior outcomes in text summarization. In this scenario, the gating mechanism may not be an appropriate choice. A direct correlation between the latent vector and the target text may be more suitable for executing the diffusion process. Enhancing the architecture to leverage diffusion models could be a potential avenue for future research.

Limitations↩︎

FlowSUM has demonstrated excellent results on datasets with long summaries. However, its performance on short-summary datasets like XSum and SAMSum has been unsatisfactory. The underlying cause could be attributed to suboptimal hyperparameter tuning or the incompatibility of FlowSUM with short summaries. Additional investigations are needed to identify the root cause.

Furthermore, we did not fine-tune the hyper-parameters of the normalizing flows model, such as the latent dimension, the number of bins in spline coupling layers, and the neural network in IAF, RealNVP, RLNSF, and RQNSF. Moreover, we opted for a small batch size due to memory limitations. Adjusting these hyperparameters could potentially enhance the model’s performance.

Due to limited computational resources, we utilized BART and BERT2BERT as the backbone models instead of newer architectures. Further research may focus on verifying the effectiveness of FlowSUM on more advanced structures.

Ethics Statement↩︎

Our research entailed developing a new text summarization framework. Although no private data were utilized, we acknowledge the potential societal impacts of our work. Therefore, we adhered to pertinent ethical guidelines and implemented rigorous procedures to guarantee the accuracy of our results.

Acknowledgements↩︎

This work was supported in part by NSF grant DMS-1952539 and NIH grants R01AG069895, R01AG065636, R01AG074858, U01AG073079.

6 Controlled Alternate Aggressive Training (CAAT)↩︎

Another advantage of the controlled alternate aggressive training (CAAT) strategy is that it provides us with more control. It is commonly assumed that allowing the model more freedom to learn, even if the NF latent module is not helpful, will not harm performance. However, our experiments suggest that this assumption does not hold, particularly for short-summary datasets where the model will not learn on its own to avoid hurting the original performance. The CAAT strategy allows us to effectively freeze the encoder-decoder parameters by setting \(n_{agg}\) and \(n_{alt}\) to large values, ensuring that when the nf module is unhelpful, it will not significantly harm performance.

7 Deeper Dive into the Evidence Lower Bound (ELBO)↩︎

Within the VED framework, the conditional data generation process can be expressed as follows: \[p(y \mid x; \phi, \theta) = \int p(z \mid x; \phi) p(y \mid x, z; \theta) dz.\] The subsequent challenge revolves around parameter estimation. Typically, the conditional latent prior is assumed as \(p(z \mid x; \phi) = N(0, I)\) for simplification (hence eliminating the \(\phi\) parameter). Despite this, the likelihood \(p(y \mid x; \theta)\) remains computationally intractable to evaluate. Variational inference tackles this issue by introducing a variational distribution \(q(z \mid x, y; \psi)\) from a specific parametric family, aiming to approximate the actual posterior \(p(z \mid x, y)\). Here, \(\theta\) denotes the model parameters, and \(\psi\) refers to the variational parameters. Instead of attempting to estimate \(\theta\) solely through maximizing the challenging log-likelihood, the approach involves joint estimation of both \(\theta\) and \(\psi\) by optimizing the ELBO.

Examining Eq. 7 and 8 , it’s evident that the ELBO represents a lower bound of the log-likelihood. Moreover, a smaller value of \(\mathrm{KL}(q(z \mid x, y) \| p(z \mid x, y))\) indicates a closer alignment between the variational posterior and the true posterior, thereby bringing the ELBO closer to the log-likelihood. This insight propels the adoption of normalizing flows to model a flexible family of variational posterior.

where \(q_0\) and \(q_K\) are the probability density function for \(z_0\) and \(z_K\) respectively.

8 Discussion on \(q(z \mid x, y) = q(z \mid x)\)↩︎

we choose to assume \(q(z \mid x, y) = q(z \mid x)\) for the following reasons. Firstly, this assumption is grounded in the nature of summarization, where \(y\) can be viewed as a condensed form of \(x\) and hence it is sensible to assume all the information in \(y\) is contained in \(x\). Secondly, as evidenced by [14], it is plausible to condition the posterior on both \(x\) and \(y\). However, their approach suffers from difficulties during prediction. In prediction, the target text \(y\) is not accessible, making it hard to sample from \(q(z \mid x, y)\). [14] suggests taking the prior’s mean as the latent code, but in our paper, the prior is a Gaussian whereas the posterior is a complex distribution modeled by normalizing flows, and taking such a strategy would diminish the benefit of using normalizing flows. Thirdly, it has been shown empirically by [69] that by restricting the conditioning of the posterior to \(x\) alone, their model achieves higher accuracy. Therefore, we consider \(q(z \mid x, y) = q(z \mid x)\) as our modeling strategy.

9 Repetition Measures↩︎

Let \(s\) represent the sentences in a result set \(\mathcal{D}\), \(|s|\) be the number of tokens in \(s\), \(s_t\) be the \(t\)th token, and \(s_{i:j}\) be the sub-sequence of \(s\) from the \(i\)th token to the \(j\)th token. The rep-w [67] is then defined by Equation 10 . \[\label{eq:32repw} \textrm{rep-w} = \frac{1}{|\mathcal{D}|} \sum_{s \in \mathcal{D}} \frac{1}{|s|} \sum_{t=2}^{|s|} \mathbb{1}\left[s_{t} \in s_{\max\left(t-w, 1\right): t-1}\right]\tag{10}\]

10 Datasets↩︎

CNN/Daily Mail [60] consists of 312,085 online news articles, with one article paired with a multi-sentence summary. We use the non-anonymized version as in [1] and follow the text processing¹⁴ in [49].

XSum [61] contains 227k BBC articles, each summarized in a single sentence.

Multi-News [62] is a multi-document dataset comprising 56k pairs of news articles and multi-sentence summaries.

arXiv, PubMed [63] are two scientific paper document datasets from arXiv.org (113k) and PubMed (215k). Each pair consists of a scientific article’s body document and its abstract.

SAMSum [64] includes 16k conversations annotated with summaries by linguists. Unlike structured texts, the information in dialogues is scattered across different speakers’ utterances, increasing the summarization difficulty.

11 Baseline Models↩︎

PG+Cov [1] is a pointer-generator (PG) network supplemented with a coverage mechanism that addresses the Out-Of-Vocabulary problem and minimizes word repetition.

BERT2BERT [6] initializes both the encoder and the decoder with the pre-trained BERT checkpoints and adds cross-attention layers.

BERTSUM [4] builds on top of BERT and applies a fine-tuning scheduler to better align the encoder and the decoder.

BART [49] is a pretrained denoising autoencoder with the standard sequence-to-sequence Transformer architecture. In this paper, we use BART as the encoder-decoder backbone.

PEGASUS [5] is a large Transformer-based S2S model, pre-trained on massive text data using a self-supervised objective called gap sentence generation, designed for abstractive summarization.

VHTM [17] is a variational hierarchical model built on the PG network. It models the topic proportion vector with isotropic Gaussian and fuses in topic information at diverse granularity levels.

TAS [19] is a topic-guided Transformer-based S2S model that injects the topic-word matrix into the LMHead layer and jointly trains the NTM and encoder-decoder model.

PEGASUS+Flow-NTM [55] is a topic-aware model built on PEGASUS. It utilizes a Flow-based NTM and a contextualized gating mechanism to integrate topic information into the encoder and the decoder.

12 Implementation Details↩︎

12.1 NF Latent Module↩︎

We configure the inference net \(q(z_0|\overline{x})\) to be a feedforward neural network with three hidden layers of dimension \(\in \{300, 600\}\), Tanh activations, and a 0.1 dropout rate. We set the latent dimension \(\ell\) to 300 and the number of NF layers \(\in \{2, 4, 6, 8\}\). For spline coupling layers (RLNSF and RQNSF), we set the number of bins to 4, the bound to 3.0, the split dimension to \(\ell/2\), and the neural network to have two hidden layers with the dimension \(\ell\). For RealNVP, the split dimension is \(\ell/2\), and the neural network has one hidden layer with a dimension of \(10\ell\). For IAF, the neural network features one hidden layer of the dimension \(3\ell + 1\). Moreover, we set \(\beta=1\) and \(C = 0.1\) for models that use \(\beta_C\)-VAE, and for models that use CAAT, we conduct one epoch of aggressive training with \(n_{alt} = 15\), followed by two epochs of non-aggressive training.

12.2 Optimization↩︎

We train the models using the Adam optimizer [70] with \(\beta_1 = 0.9, \beta_2 = 0.999\), and \(\epsilon = 10^{-8}\). The initial learning rate is \(5\times 10^{-5}\). We employ a linear learning rate scheduler that increases the learning rate from 0 to the initial learning rate during the warmup stage and decreases it from the initial learning rate to 0 after the warmup stage. We also apply the gradient clipping technique with a maximum gradient norm of 1.0. Furthermore, we terminate the training early when the perplexity fails to improve for eight or sixteen consecutive evaluation calls.

12.3 Model Hyper Parameters↩︎

Table ¿tbl:table:32model95hyper-params? provides the hyper-parameters for the models discussed in Table 4 - 6, for the sake of reproducibility. To ensure fair comparisons, unless otherwise specified, the VEDSUM models typically employ the same set of hyper-parameters as their FlowSUM counterparts, except with standard training and no NF layers applied. Additionally, the models in Table 7 have the same hyper-parameters as those in Table 6, except for the number of NF layers used. Lastly, in Table ¿tbl:table:32effect-of-training-strategies?, all FlowSUM models use 4 NF layers and the same set of hyper-parameters as those in Table 6 but vary in their training strategies.

13 Experiments on Training Strategies and Gate Initialization↩︎

The training curves for the methods in Table 8 are illustrated in Figure 3. The plot demonstrates that the gate score decreases gradually and remains high during aggressive training when CAAT is combined with standard initialization. This combination compels the model to utilize the latent code information effectively. Moreover, as presented in Figure 3 (c), even though CAAT combined with standard initialization starts with a high perplexity, it achieves a lower perplexity level than other approaches by the end. By examining the training procedure in detail, Figure 4 further indicates that CAAT contributes to greater training stability than standard training.

14 Visualization of Latent Distribution↩︎

To gain a better understanding of how normalizing flows contribute to knowledge distillation, we selected several examples from the CNN/Daily Mail and XSum datasets and visualized the resulting latent distribution generated by the FlowSUM-PLKD model, as shown in Figure 5 and 6. For both cases, the transformed latent code \(z_K\) exhibited a highly flexible distribution. Notably, in the CNN/Daily Mail example, the first dimension of the second example demonstrated a clear bi-modal distribution, indicating the model’s ability to capture information from multiple sources. Similarly, in the XSum dataset examples, we observed distinct multi-modal patterns.

15 Normalizing Flows↩︎

Planar flow Proposed by [20], the planar flow can be expressed as in Eq. 11 . It applies contractions or expansions in the direction perpendicular to the hyperplane \(\mathbf{w}^{\top} \mathbf{z}+b=0\). Its Jacobian determinant can be computed in time \(\mathcal{O}(D)\) as in Eq. 12 , using the matrix determinant lemma. In addition, we need to note that this flow is not invertible for all values of \(\mathbf{u}\) and \(\mathbf{w}\). When the derivative of the activation function \(h^{\prime}(\cdot)\) is positive and bounded from above, \(\mathbf{w}^{\top} \mathbf{u}>-\frac{1}{\sup _x h^{\prime}(x)}\) is sufficient to ensure invertibility¹⁵.

\[\label{eq:32planar-appendix} f(\mathbf{z})=\mathbf{z}+\mathbf{u} h\left(\mathbf{w}^{\top} \mathbf{z}+b\right),\tag{11}\] \[\label{eq:32planar-jacobian-det-appendix} \operatorname{det} J = 1 + h^{\prime}\left(\mathbf{w}^{\top} \mathbf{z}+b\right) \mathbf{w}^{\top} \mathbf{u}\tag{12}\] where \(\{\mathbf{u}, \mathbf{w} \in \mathbb{R}^D, b \in \mathbb{R} \}\) are free parameters and \(h(\cdot)\) is a smooth element-wise non-linear activation function with derivative \(h^{\prime}(\cdot)\).

Radial flow The radial flow [20], [71] takes the form of Eq. 13 . It applies radial contractions and expansions around a reference point. Similar to the planar flow, we can apply the matrix determinant lemma to calculate the Jacobian determinant in \(\mathcal{O}(D)\) time, as in Eq. 14 . To guarantee invertibility, we usually require \(\beta>-\alpha\)¹⁶.

\[\label{eq:32radial-appendix} f(\mathbf{z})=\mathbf{z}+\beta h(\alpha, r)\left(\mathbf{z}-\mathbf{z}_0\right),\tag{13}\] \[\label{eq:32radial-jacobian-det-appendix} \operatorname{det} J = \left(1+\frac{\alpha \beta}{h^2}\right)\left(1 + \beta h\right)^{D-1}\tag{14}\] where \(\mathbf{z}_0 \in \mathbb{R}^D\) is the reference point, \(\beta \in \mathbb{R}, \alpha \in \mathbb{R}^{+}\) are free parameters, \(r=\left\|z-z_0\right\|\) is the norm of \(z-z_0\), and \(h(\alpha, r)=\frac{1}{\alpha+r}\).

Sylvester flow The Sylvester flows [41] generalize the planar flows to have \(M\) hidden units, as in Eq. 15 . To achieve better computational efficiency, [41] proposes the parameterization as in Eq. 16 , with which the Jacobian determinnant reduces to Eq. 17 and can be computed in \(\mathcal{O}(M)\). Similar to the planar flows, when \(h^{\prime}(\cdot)\) is positive and bounded from above, \(\tilde{\mathbf{R}}_{ii} \mathbf{R}_{ii} > -\frac{1}{\sup _x h^{\prime}(x)}\) for all \(i \in\{1, \ldots, D\}\) is sufficient to ensure invertibility.

\[\label{eq:32Sylvester-appendix} f(\mathbf{z}) = \mathbf{z}+\mathbf{U} h\left(\mathbf{W}^{\top} \mathbf{z}+\mathbf{b}\right),\tag{15}\] where \(\{\mathbf{U} \in \mathbb{R}^{D \times M}, \mathbf{W} \in \mathbb{R}^{D \times M}, \mathbf{b} \in \mathbb{R}^M \}\) are the free parameters and \(h(\cdot)\) is an element-wise activation function.

\[\label{eq:32Sylvester-2-appendix} f(\mathbf{z}) = \mathbf{z}+\mathbf{Q} \mathbf{R} h\left(\tilde{\mathbf{R}} \mathbf{Q}^T \mathbf{z}+\mathbf{b}\right),\tag{16}\] \[\label{eq:32Sylvester-jacobian-det-appendix} {!}{\operatorname{det} J = \operatorname{det} \left( \mathbf{I}_M + \operatorname{diag} \left( h^{\prime} \left( \tilde{\mathbf{R}} \mathbf{Q}^T \mathbf{z}+\mathbf{b}\right)\right) \tilde{\mathbf{R}} \mathbf{R}\right)}\tag{17}\] where \(\mathbf{R}\) and \(\tilde{\mathbf{R}}\) are upper triangular \(M \times M\) matrices, and \(\mathbf{Q}=\left(\mathrm{q}_1 \ldots \mathrm{q}_M\right)\) consists of an orthonormal set of vectors.

Autoregressive Flows The masked autoregressive flow (MAF) [38] was motivated by MADE [72], which is an autoregressive model for density estimation. MAF generalizes the conditional distribution to be Gaussian and generates data in a recursive way as in Eq. 18 . Given a data point \(\mathbf{x}\), the inverse transformation can be performed in parallel as in Eq. 19 . The Jacobian of the inverse transformation is lower-triangular by design due to the autoregressive structure, hence its absolute determinant can be expressed as in Eq. 20 . The set of functions \(\left\{f_{\mu_i}, f_{\alpha_i}\right\}\) are autoregressive neural networks following the approaches in MADE.

\[\label{eq:32maf-appendix} x_i=u_i \exp \alpha_i+\mu_i,\tag{18}\] where \(\mu_i=f_{\mu_i}\left(\mathbf{x}_{1: i-1}\right), \alpha_i=f_{\alpha_i}\left(\mathbf{x}_{1: i-1}\right)\) and \(u_i \sim \mathcal{N}(0,1)\).

\[\label{eq:32maf-inverse-appendix} u_i=\left(x_i-\mu_i\right) \exp \left(-\alpha_i\right)\tag{19}\]

\[\label{eq:32maf-jacobian-appendix} \left|\operatorname{det}J^{-1}\right|=\exp \left(-\sum_i \alpha_i\right)\tag{20}\]

Likewise, the inverse autoregressive flow (IAF) [28] uses MADE with Gaussian conditionals and generates data as in Eq. 21 . Its Jacobian determinant has a simple form as in Eq. 22 . The main difference between IAF and MAF lies in the history variables. MAF uses previous data variables \(\mathbf{x}_{1: i-1}\) to compute \(\mu_i\) and \(\alpha_i\), whereas IAF uses previous random variables \(\mathbf{u}_{1: i-1}\) for the computation. In terms of sampling and density evaluation, IAF can sample in parallel and need to evaluate sequentially, whereas MAF has to sample sequentially and can evaluate in parallel. Since we care more about the sampling efficiency in variational inference, we choose IAF in the paper.

\[\label{eq:32IAF-appendix} x_i=u_i \exp \alpha_i+\mu_i,\tag{21}\] where \(\mu_i=f_{\mu_i}\left(\mathbf{u}_{1: i-1}\right)\) and \(\alpha_i=f_{\alpha_i}\left(\mathbf{u}_{1: i-1}\right)\).

\[\label{eq:32IAF-jacobian-appendix} \left|\operatorname{det}J\right|=\exp \left(\sum_i \alpha_i\right)\tag{22}\]

Affine Coupling The affine coupling layer, proposed in NICE [73] and later generalized in RealNVP [42] takes the following form.

\[\label{eq:32affine-coupling-appendix} \begin{cases}y_{1: d} & =x_{1: d} \\ y_{d+1: D} & =x_{d+1: D} \odot \exp \left(s\left(x_{1: d}\right)\right)+t\left(x_{1: d}\right)\end{cases}\tag{23}\] where \(s: R^d \mapsto R^{D-d}\) and \(t: R^d \mapsto R^{D-d}\) are scale and translation transformation function respectively, and \(\odot\) is the element-wise product.

Its Jacobian determinant can be efficiently computed as \(\operatorname{det} J = \exp \left[\sum_j s\left(x_{1: d}\right)_j\right]\). Since the computation does not involve the Jacobian of \(s\) or \(t\), we can make these two functions arbitrarily complex and use neural networks to model them. The coupling layers are usually composed of permutation layers to ensure every component gets modified, and since the Jacobian determinant of permutation is 1, the Jacobian determinant remains tractable.

Spline Coupling Neural spline flows [43], [44] use monotonic rational-quadratic splines or monotonic rational-linear splines as the coupling transformation to achieve more flexibility and yet remain differentiable and invertible. The monotonic rational-quadratic spline uses \(K+1\) monotonically increasing knots \(\left\{\left(x^{(k)}, y^{(k)}\right)\right\}_{k=0}^K\) to set up \(K\) bins, each of which is defined as a rational-quadratic function¹⁷ that is monotonically increasing. It maps \([-B, B]\) to \([-B, B]\) and defines the transformation outside the range to be identity transformation. Let \(s_k = \left(y^{k+1}-y^k\right) /\left(x^{k+1}-x^k\right)\) and \(\xi(x)=\left(x-x^k\right) /\left(x^{k+1}-x^k\right)\), the rational-quadratic function in the \(k\)th bin takes the form of Eq. 24 and the Jacobian determinant of the rational-quadratic neural spline flows (RQNSF) can be written as in Eq. 25 . \[\label{eq:32rational-quadratic-appendix} {!}{\frac{\alpha^{(k)}(\xi)}{\beta^{(k)}(\xi)} = y^{(k)}+\frac{\left(y^{(k+1)}-y^{(k)}\right)\left[s^{(k)} \xi^2+\delta^{(k)} \xi(1-\xi)\right]}{s^{(k)}+\left[\delta^{(k+1)}+\delta^{(k)}-2 s^{(k)}\right] \xi(1-\xi)}}\tag{24}\] \[\label{eq:32rqs-jacobian-appendix} \begin{align} & \operatorname{det} J = \prod_{k} \frac{\mathrm{d}}{\mathrm{d} x}\left[\frac{\alpha^{(k)}(\xi)}{\beta^{(k)}(\xi)}\right] \\ = & {!}{\prod_{k} \frac{\left(s^{(k)}\right)^2\left[\delta^{(k+1)} \xi^2+2 s^{(k)} \xi(1-\xi)+\delta^{(k)}(1-\xi)^2\right]}{\left[s^{(k)}+\left[\delta^{(k+1)}+\delta^{(k)}-2 s^{(k)}\right] \xi(1-\xi)\right]^2}} \end{align}\tag{25}\]

The rational-linear neural spline flows (RLNSF) work similarly, except with monotonically increasing linear rational functions in each bin. Neural splines combine the best of autoregressive flows and coupling layers (such as NICE and RealNVP) in that it has both an analytic single-pass inverse and sufficient flexibility, as demonstrated in [43].

16 Example Analysis↩︎

In this section, we analyze several instances from CNN/Daily Mail and XSum, showcasing diverse outcomes generated by different summarization models.¹⁸

References↩︎