1 Introduction↩︎

With the rising demand for visualizing quantitative data, the growing adoption of digital media has played a role in the rapid growth of data visualization, which has led to the task of automatic chart understanding, information extraction, and summarization, critical areas of research [1]–[3]. Recent advancements in Visual Language Models (VLMs) have shown promise in this area [2], [4]–[9]; however, existing methods still struggle with achieving high-quality summaries, especially for L2/L3 content - which is identified as statistics and relations (e.g., min, max) / perceptual and cognitive phenomena (e.g., trends) [10]–[12], as shown in Figure 1. The challenge is around the highly inconsistent matching between the generated summary and the chart’s actual data content, which yields factual inconsistencies and hallucinations. This is either due to failing to parse the text in the chart or to demarcate the numerical value of the visualized data. Additionally, with semantic parsing of the chart elements, VLMs struggle at performing complex reasoning about chart patterns and incorporating statistical reasoning with chart elements [9]. Despite general challenges, although current VLM-based chart understanding methods have shown a certain level of performance, they still face two main challenges: (1) Existing implementations are fine-tuned or pre-trained specifically on chart-related instruction data. While this alignment between the vision encoder and language decoder enhances generalization performance, such training processes introduce significant computational overhead, making them resource-intensive and challenging under computational constraints; (2) These tasks continue to remain a challenge in understanding the structural interplay between the different elements of a chart. Effective visual language understanding in particular requires two key processes: (a) comprehensive semantic layout understanding of the chart; (b) robust statistical reasoning to accurately capture and analyze the underlying data [13].

In light of these challenges, we investigate zero-shot and training-free approaches for VLMs in chart summarization. Program-of-Thoughts (PoT) [14] is a zero-shot prompting method, which was originally proposed to disentangle computation from reasoning to augment a model’s statistical reasoning capability. The success of PoT in chart question answering (QA) [2] with Python programs has motivated our exploration of chart summarization, investigating the effectiveness of the PoT guiding VLMs to perform numerical computations and logical reasoning via Python programs as intermediate steps in the chart summarization process, which focuses on generating more structurally complex and extensive sentences, rather than just concise answers. Instead of relying on the provided real chart data tables for PoT in recent PoT research works, we acknowledge that in real-world scenarios, most charts lack accompanying data tables. Therefore, we investigate a PoT strategy pipeline for chart summarization with simultaneously generated chart data tables. Our key contributions are as follows:

• We propose a PoT-integrated, training-free pipeline, enhancing lightweight VLMs for chart summarization in a zero-shot learning setting.

• We demonstrate the PoT prompting strategy outperforms Direct and MCoT approaches in certain scenarios, particularly across diverse types of VLMs, charts, and supplementary textual data in chart summarization.

• We conduct comprehensive evaluations across lexical, semantic, and factual dimensions to validate the effectiveness of the PoT prompting strategy for chart summarization.

2 Literature Review↩︎

2.1 Chart Understanding↩︎

Template-Based Early approaches to automatic chart understanding, particularly the sub-task of chart summarization, often relied on planning-based architecture and template-based generation methods [15]–[20]. Recent template-based research has focused on utilizing statistics (e.g., min, max, trends) from chart numerical data for presenting the facts [21]–[24], forming the statistics analysis into textual summarization output. Some research utilized the off-the-shelf OCR (Optical Character Recognition) tools or detectors to represent chart data into textual tables and other representations, relying on pipeline methods [11], [25]–[30]. More recently, ResNet [31] encoder and LSTM decoder were used to process the chart and create the caption [32]. However, compared to data-driven models, template-based approaches struggle with complex visual patterns and numerical reasoning, with high costs in producing generics and matching variations in vocabulary choices.

Pretrained With the progression of deep learning techniques, which subsequently improved general computer vision using neural networks and Transformer [33], recent work began to adopt encoder-decoder architectures to improve chart understanding [34], including Transformer [11], [35]–[37], LSTM [38], CNN+LSTM [39], and VLMs [13], which are pre-trained on both visual and text data, often with specialized text and image encoders, and have shown significant promise in tasks requiring joint understanding of multiple modalities. However, challenges remain in grounding the factual and logical coherence in generated summaries, particularly when dealing with complex charts requiring numerical reasoning.

Fine-Tuned Aside from pre-training the model, fine-tuning the pre-training model [12] and instruction fine-tuning [40] have also become widely adopted as an alternative to improve the performance of LLMs and VLMs [2], [4]–[9], [13], [41]–[44]. Instruction tuning is used to generalize the language capability of the model, reducing repetitions and hallucinations generated in summarization compared to pre-training approaches [8]. However, these methods typically rely on the data tables of charts, failing to capture the nuance of the visual artifacts present in charts. Furthermore, their heavy parameter sizes present notable challenges for deployment in computationally constrained environments.

2.2 Chart Representations↩︎

Representing the chart in structured data, the chart-to-table [8] task represents it in the tabular format, but often comes at the cost of losing finer details in the chart. Performing similarly to data tables, scene graphs are easily formatted for web-based charts [12]. Code format is considered, and existing methodologies define two typical chart-to-code approaches: (1) Chart Derendering [13], [37]; and (2) Program of Thoughts [2], [14]. However, codes mainly aim to run for the chart recreation or question answering tasks on narrowly defined questions, rather than representing the whole chart. This paper proposes an auxiliary task of chart-to-table, which is chart-to-dictionary in Python code format, which uses VLM’s chart understanding capability to represent the chart as a Python dictionary.

2.3 Prompting↩︎

Inspired by the success of Chain-of-Thought (CoT) prompting [45] for improving reasoning capabilities, researchers are extending similar mechanisms to VLMs for chart understanding, seeking to mirror the human cognitive process of visual analysis. This is achieved through multimodal-purpose prompting Multimodal Chain of Thought (MCoT) [9], [34] reasoning, which extends the rationale from texts to visual modalities [3]. To contrast with MCoT, PoT [14], [46] intermediate reasoning steps are articulated as executable programs, while executing the program to generate reasoning and statistical computation about the chart data in complex numerical reasoning tasks.

In this work, our pipeline method builds upon these advancements by focusing on PoT prompting in zero-shot chart summarization. By extending the PoT concept to the visual domain of charts, it could decrease hallucinations that language models typically have when outputting calculations, as it provides more explicit and verifiable numeric reasoning processes for VLMs [2], potentially leading to more accurate and factually grounded summaries by delegating complex calculations to a code interpreter. This work differentiates itself from existing works by specifically investigating the benefits and limitations of generating executable code as intermediate reasoning steps for chart summarization with lightweight VLMs.

3 Method↩︎

We propose a pipeline with the PoT integrated to augment a VLM’s capability for statistical reasoning on chart data summarization. An illustration of the proposed PoT-integrated chart summarization pipeline is presented in Figure 2. Our prompts can be found in Appendix 10.

3.1 Chart Representation as a VLM-Generated Python Dictionary↩︎

In order for the chart to interface with the code, the chart needs to be represented in a manner that can interact with the Python interpreter. As shown in Figure 3, Python dictionaries can represent the code in a more free-form structure, allowing for grounding the values compared to the data table, which is more flexible compared to a markdown table and usable by the LLM-generated program. However, lightweight VLMs can struggle to create executable Python code, which consists of wrong syntax, incomplete messages, and even meaningless code-agnostic terminologies when facing the complex code generation request, adding noise. Given that, aside from reflecting understanding from charts, the code needs to be valid and executable. In Appendix 13, we list more details of the failure case analysis. To handle failure cases in dictionary generation, we mainly used InternVL-2.5-4B [47] on dictionary generation in a zero-shot setting, and if the generated Python dictionary is not executable, it is converted with ChatGPT (GPT-4o-mini) [48] instead.

3.2 Statistical Analysis with PoT Prompting↩︎

Since the chart is represented as a Python dictionary, it can be more free-form in containing data and being passed to a Python program. We adopt a similar methodology described in PoT, as the work [14] stated that the program’s line-by-line structure acts as a proxy for the numerical reasoning steps of the model. Code is passed to an LLM to generate a program to do statistical analysis as an intermediate result to provide more context for chart summarization. Compared to QA as a task, statistical analysis with PoT demonstrates numerical reasoning since it demonstrates how the models understand which data points or statistics are necessary to create summary statistics. This paper uses Qwen-2.5-Coder-14B [49] for the complex statistics code generation conversion. The LLM is instructed to generate a Python program using the Python dictionary in the prompt to generate summary statistics relevant to the chart dictionary. This adapts PoT for the chart summarization task as the generated program provides more context to be used for text generation while providing accurate calculations. Code generated by the LLM is constrained to use only the functions from Python’s built-in library. To validate and execute the generated Python program by the PoT strategy, we used the built-in function in Python for automatic code validation.

3.3 Program Execution↩︎

The generated Python program for statistics calculation is executed using a Python interpreter. This step ensures the accuracy of the statistical results, mitigating potential errors that LLMs might make when generating tokens through direct calculations. The program returns a Python statistics dictionary that contains key-value pairs of the summary statistics and the calculated values. At the end, the statistical results in our pipeline are input with the chart into a VLM to assist the chart summarization task.

4 Experiment↩︎

We present our experimental setup in Appendix 8. The overview of our datasets, evaluation metrics, baseline methods, and benchmark and backbone models is provided in the following subsections.

Evaluation. We evaluated PoT prompting for chart summarization on both the test sets of the Pew [11] and VisText [12], following the previous evaluation works [4], [8] for evaluating the PoT on varying degrees of complex charts to show its generalizability. The VisText is built upon Statista [11] with richly labelled L2/L3 captions. Chart type distributions of datasets are summarized in Table 1, which across a variety of simple and complex charts. More details on the dataset statistics and topic distribution information are presented in the Appendix 9.1. To evaluate the effectiveness of the methods, we employ BLEU [50] and CIDEr [51] as the evaluation metric following previous works [4], [8], [11], [13]. Additionally, we use F1 scores of ROUGE [52] and BERTScore [53] for semantic evaluation; UniEval [54] and AlignScore-large [55] for factual evaluation; and human evaluation with 3 human evaluators. We provide details of evaluation metrics in the Appendix 9.3 and Appendix 14.

Baselines. We compared two other types of prompting strategies as baselines: (1) Directly prompting (Direct) the model to summarize the chart, given that this approach is also what is done by fine-tuned end-to-end models [1], [9]; (2) Multimodal CoT (MCoT), which adheres to the framework in [34], prompting to return an outline of all key information and trends derived from the chart.

Backbones. To understand the effects of the PoT, we compared (1) Existing models and methods in the chart-to-text domain: (a) Pretrained Chart-To-Text Models: OCR-Field-Infuse [11], [56], Monkey [57]; (b) Prefix-tuning Chart-To-Text Models: image-scene-graph-PT [12], image-data-table-PT [12]; (c) Commonly used VLMs: Blip2-flant5xl [58], Qwen-VL [59]; and (2) Lightweight VLMs: DeepSeek (DeepSeek-VL2-tiny) [60], InternVL (InternVL-2.5-4B) [47], LLaVA (LLaVA-v1.6-mistral-7B-hf) [61], and Qwen (Qwen2.5-VL-3B-Instruct) [62] on the representative datasets of Pew and VisText. All experiments were done with the zero-shot setting models.

5 Results & Discussion↩︎

5.1 PoT Approach Against Existing Chart-To-Text Models↩︎

Table 2 shows the BLEU and CIDEr scores for each model on the Pew and VisText datasets. We referenced evaluation results from Chart-To-Text [11], VisText [12], and ChartAssistant [8]. As shown in the table, we observed that our PoT prompting approach overperforms baseline Chart-To-Text methods in the BLEU evaluation scores, but underperforms in the CIDEr evaluation scores. This may be due to CIDEr placing more emphasis on important and rare words, as it calculates TF-IDF weighted n-gram similarity. While BLEU also focuses only on surface-level word matching and ignores semantic consistency, we subsequently evaluate our PoT prompting approach using BERTScore and ROUGE to capture semantic relevance, and UniEval and AlignScore to assess factual correctness beyond lexical overlap.

5.2 PoT Approach Against Baselines↩︎

We evaluate baseline prompting strategies and our PoT prompting strategy and report results of our experiment in Table [tab:ablation] and Table [tab:ablation95asueval]. Full experimental results and extended ablation studies are in the Appendix 11.1. Across the evaluated models, the impact of the PoT prompting strategy varied significantly with models and chart types. We observed instances where the PoT led to substantial improvements in performance, while in other cases, its impact was less pronounced or even negative compared to the Direct and MCoT approaches.

PoT Effectiveness Against Chart Types. We notice that the results from different charts are varied, and we suppose this may be due to the uniqueness of each chart structure, texts included in the chart, chart data size, and data complexity. For example, in the case of the Qwen2.5-VL model, the BLEU score increases from 1.94 to 3.34 with PoT, demonstrating the effectiveness of the PoT strategy in enhancing information collection from area charts, which are with limited data information.

PoT Effectiveness Against VLMs. Regarding influences by VLMs, for the DeepSeek-vl2-tiny model, the application of the PoT resulted in considerably lower scores across all reported metrics compared to both the Direct and MCoT methods. This suggests that for this particular model architecture, the PoT strategy in its current implementation might not be beneficial or could even hinder performance on the evaluated tasks. This reveals that the PoT strategy may introduce additional noise or mislead the emphasized information, and may interfere with the model’s original processing and understanding of the chart. In contrast, the InternVL-2.5 model demonstrated a more nuanced response to the PoT prompting strategy. While the Direct method often yielded the highest scores, the PoT strategy achieved comparable or even slightly better results on certain metrics compared to the MCoT strategy in most cases. For example, the PoT strategy achieved a BLEU score of 2.85, which is lower than the Direct method (3.65) but higher than the MCoT strategy (2.38) of the area charts in the Pew dataset. Even on considering all chart types, these trends hold. This indicates that for InternVL-2.5, the PoT strategy can be a viable alternative to the MCoT strategy in certain scenarios, especially from the results of factual evaluation. Similarly, LLaVA-NeXT also had a mixed response given the two datasets, where no conclusive trends can be observed between the different prompting methods. One interesting observation from this comparison is that while the BLEU values of the PoT strategy are lower than the other methods, on average, it outperforms the other prompting techniques on CIDEr, AlignScore or UniEval, indicating some of its effectiveness in these cases. We suggest that this may arise from the inherent design and pertaining data differences in the VLMs with respect to chart understanding. Specifically, DeepSeek-vl2 is equipped with a dedicated vision encoder and a vision-language adapter, originally designed to optimize performance on visual tasks such as chart interpretation. In contrast, InternVL-2.5 is built upon a Vision Transformer architecture integrated with a large language model, while pre-trained with one of the benchmark datasets [47], the VisText dataset, placing more confidence on the fusion of textual information in the chart-to-text task. As a result, when we enlarge the textual information using the PoT strategy, the performance outcomes of DeepSeek-vl2 and InternVL-2.5 can diverge, potentially yielding opposite trends. This observation suggests that the PoT strategy does not universally benefit all VLMs in chart summarization, but is particularly advantageous for those that emphasize textual information.

PoT Compared with MCoT. On the other hand, Qwen-2.5VL-3B showed that the PoT strategy consistently outperformed the MCoT strategy while underperforming relative to the Direct prompting. This suggests that for the Qwen2.5-VL-3B model, the PoT strategy appears to be a more effective CoT prompting strategy compared to the standard MCoT approach across the evaluated tasks. This may be due to the PoT strategy introducing more new statistical content into the chart summarization process during chart data interpretation compared to the MCoT approach. While the PoT generates additional statistical information, MCoT primarily offers a high-level data outline and trends.

5.3 PoT Approach Against VLM Backbones↩︎

While the PoT strategy demonstrated potential for improving performance, particularly for the InternVL-2.5 and Qwen2.5-VL-3B models in certain scenarios, we conducted further investigations to valid the effectiveness of adopting PoT and identify the potential information factors contributing to the varying effectiveness of the PoT strategy and to estimate the extent to which information influences the performance of the PoT strategy pipeline. We compared: (1) Template-based: using the predefined Python program template; and (2) PoT-based: using the PoT for generating the statistics dictionary in chart summarization with human evaluation, as shown in Table 3. In addition, we conducted a series of experiments focusing on the textual components that serve as supplementary inputs to the VLM alongside the input chart. The experimental settings are as follows: (1) Title: Use only the title as input to the VLM, without applying the PoT strategy; (2) Dict+Title: Use the PoT-generated Python dictionary along with the title as input to the VLM; (3) Stats+Title: Use the PoT strategy to generate a statistics dictionary, combined with the title as input to the VLM; (4) Dict+Stats+Title: Use the full set of inputs, including the PoT-generated Python dictionary, the PoT-generated statistics dictionary, and the title as input to the VLM; (5) Dict+StatsT+Title: Replace the LM with a predefined Python program template for generating the statistics dictionary, and use the generated statistics dictionary together with the Python dictionary and title as input to the VLM. The experimental results that were evaluated on ROUGE-L and BERTScores are illustrated in Table [tab:ablation95rouge95bert].

PoT Effectiveness Influenced by Input Textual Data. While the evaluation results remain influenced by the underlying VLM performance, we observed that in over half of the cases, the combination of the title and Python dictionaries outperformed using the title alone. We attribute this to the fact that directly extracted data, despite potential noise, can retain more valuable information than purely generated text, potentially steering the model toward more accurate outputs. However, this also highlights the power of using the PoT strategy, as it guides the model to emphasize more on the enhanced inaccuracies and noise with the poorly extracted data, while weakening the chart analysis, which negatively impacts the overall performance of the model pipeline. In addition, we observed that the PoT strategy can consistently outperform in most cases with the InternVL model. This indicates the effectiveness of the PoT strategy with a pretrained VLM, which is better than directly using the title to enhance the overall pipeline performance in the chart summarization.

6 Future Work↩︎

From experimental results, it is observed that the summarization with the PoT strategy varied by different types of charts that the model was captioning. Most models performed well on relatively simpler bar and pie charts, while struggling with more complex charts, such as multiple line or scatter plots. This indicates that the generalizability requirement of the summarization task may involve some sort of normalization or some way to bridge the gap between the varying levels of complexity presented by the chart. In future work, we would like to explore more sophisticated PoT approaches capable of generating longer and richer statistical information during the pipeline, thereby enhancing the quality of chart summaries. Since the PoT strategy in this work only extends outputs from short, answer-like responses to relatively concise statistical dictionaries. However, for the chart summarization task, we believe the PoT strategy contains untapped potential to capture factual numeric data by its statistical reasoning capability. Moreover, given the significant influence of the PoT-generated information in model inference, we will also further investigate whether the PoT can contribute to mitigating hallucination errors in the chart summarization process, improving the overall factual accuracy of generated chart summaries.

7 Conclusion↩︎

In this work, we conducted a systematic evaluation of the Program-of-Thought (PoT) prompting strategy across currently used lightweight vision-language models under the zero-shot settings on the Pew and VisText benchmarks for the chart summarization task. Our experiments reveal that the efficacy of the PoT varies markedly with model architectures and sizes, pretrained data, corresponding to types of charts, including area, bar, line, pie, and scatter. In conditions of VLMs and chart types, the PoT proved to be a competitive alternative to the Direct and MCoT prompting approaches with pretrained model, such as the InternVL. Beyond prompting strategies, we introduced a novel chart-to-dictionary auxiliary task, demonstrating its promise for capturing robust and semantic nuances in chart understanding, which is also conveniently applicable with the PoT. As charts grow more complex along with the data they represent, there is a need to establish a data structure to evaluate chart-parsing outside the table due to data loss that occurs from the chart to the table.

Limitations↩︎

The diverse performance of the PoT strategy across the evaluated models raises several important considerations. The model architecture and size likely play a significant role in determining the effectiveness of different prompting strategies. The models used in this paper were of lightweight VLMs. While effective in the presented lightweight models, the language decoder may have yielded too low conclusive powers on the efficacy of the PoT and CoT prompting methods relative to direct prompting. However, it is seen that the PoT strategy still can offer comparable results to the other prompting methodologies using lightweight VLMs in some cases or for some chart types, which indicates that on higher parameter models, it can be assumed that, in the worst case, these different prompting techniques may offer similar results. The research design, comparing three zero-shot prompting methods across four distinct vision-language models and a set of tasks, provides a valuable initial exploration of the PoT’s potential on chart summarization with VLMs. Further research can implement few-shot reasoning with examples that can hypothetically increase performance. Additionally, the study focused its experimentation on lightweight VLMs, which might have contributed to the poor results in text generation. Expanding the scope of the study to larger parameter models might lead to more conclusive results.

Ethics Statement↩︎

To the best of the researchers’ knowledge, all datasets used in this study were sourced from publicly available benchmarks. The authors of the benchmark dataset also have obtained the license to distribute the dataset for non-malicious purposes intent which this research has abided by.

8 Experiment Set-up↩︎

The experiments are conducted with loaded pre-trained models from the vLLM API. As much as possible, the default parameters were used, unless suggested otherwise from official documentation. The temperature is set to 0.2, and the repetition penalty is set to 1.2 across all runs. All experiments are carried out on our machine (CPU: Intel(R) Core(TM) i9-9920X CPU @ 3.50GHz, GPU: 1 NVIDIA RTX3090). Python code generation for producing statistics by the Qwen2.5-Coder-14B-Instruct model is the most computationally costly task, which costs 10-12 hours on 1 GPU.

9 Extended Evaluation Details↩︎

9.1 Dataset Analysis↩︎

We chose the Pew [11] (GPL-3.0 license) and VisText [12] (GPL-3.0 license) large-domain English datasets to investigate and evaluate our PoT strategy for generating L2/L3 content in chart summarization, as they provide rich and suitable L2/L3 captions for this task. The VisText is built upon the Statista [11] dataset, but with additionally detailed labelled L2/L3 captions. Since the chart labelled in the VisText may have multiple L2/L3 captions, we automatically selected the longest L2/L3 captions in the test set of the VisText dataset as gold summaries paired to charts for the chart summarization task. The statistics of the Pew and VisText datasets used in this paper are presented in Table 4. In addition, the distribution of topics covered in the Pew and VisText datasets is illustrated in Figure 4.

9.2 Experiment Implementations↩︎

We mainly used DeepSeek-VL2 (deepseek-VL2-tiny) [60] for testing and our experiments. Additionally, we also tested the following models: InternVL (internVL-2.5-4B) [47], LLaVA-NeXT (llava-v1.6-mistral-7b-hf) [61], and Qwen-2.5 (qwen2.5-VL-3B-Instruct) [62] for main and ablation experiments. InternVL was reported to have one of our benchmark datasets, VisText, in its pretraining datasets [47]. This likely contributed to its stronger performance, highlighting the potential benefits of pretraining. For the other models, there is no overlap with our benchmark datasets, nor evidence suggesting that the models were semantically aligned with the test distributions or were familiar with recurring chart patterns from their source papers. All experiments were done in Python 3.12 using the vLLM [63] library, with the models being implemented at the zero-shot setting. Similar to the previous work, the usage of ‘#’ tokens in the generated tokens was restricted to avoid the pitfalls of only generating the reasoning chain as comments instead of executable code.

9.3 Evaluation Metric Descriptions↩︎

To quantitatively measure the performance of our proposed method in chart summarization, we employ two popular automatic evaluation metrics in chart understanding: BLEU (Bilingual Evaluation Understudy) and CIDEr (Consensus-based Image Description Evaluation), in addition to two also well-known automatic evaluation metrics in text summarization: ROUGE [52] and BERTScore [53]. In order to evaluate factual correctness in chart summarization, we additionally adopt UniEval [54] and AlignScore [55].

BLEU [50] This score calculates the n-gram overlap between the ground-truth summary and the generated summary. It indicates lexical similarity between the generated and ground-truth text, assessing how closely the generated text replicates word sequences that occur in the reference.

CIDEr [51] This score measures the TFIDF weighted n-gram overlaps between reference and generated text. By weighting n-grams according to their value in a reference summary corpus, CIDEr seeks to more accurately capture the informativeness and relevance of generated descriptions, especially in image and chart captioning tasks.

BLEU and CIDEr are commonly used metrics throughout natural language generation, image captioning, and chart summarization. Together, they capture a more nuanced quantitative measure of model performance in terms of surface similarity and content alignment with reference summaries. While we note that reference-based measures like BLEU and CIDEr do have some limitations, since they can have loose correlation with human preference for aspects of semantic equivalence and factuality, their popularity and ability to provide an initial quantitative score make them effective measures in chart summarization model evaluation. As a result, we consider additional metrics for evaluating the chart summarization.

ROUGE [52] This score is a prevailing metric in text summarization research based on semantic similarity.

BERTScore [53] This score offers a complementary perspective by quantifying semantic similarity between system outputs and reference texts.

UniEval [54] This score assesses factual correctness by framing the evaluation as a Boolean question-answering task.

AlignScore [55] This score evaluates factual correctness by quantifying the information alignment between two arbitrary text pieces.

10 Prompts↩︎

10.1 LM Chat Templates↩︎

We show an exemplar from our chat templates for internVL here. The full chat templates can be found in our repository.

{{{ bos_token }}
{    {        {{- '<|im_start|>system\n' + message['content'] + '<|im_end|>\n' -}}
    {        {        {            {            {        {        {{- '<|im_start|>user\n' + content + '<|im_end|>\n' -}}
    {        {{- '<|im_start|>assistant\n' + message['content'] + '<|im_end|>\n' -}}
    {{{    {{- '<|im_start|>assistant\n' -}}
{}}}}}}}}}}}}}}

10.2 Chart-to-Dictionary Extraction with Program of Thoughts↩︎

Similar to the chart-to-table task, this is done in a zero-shot setting. We employ the core concept of PoT to guide the VLM in generating a valid and executable Python dictionary from the input chart.

user_prompt = "<img_placeholder>\nConvert the chart into a python dictionary `chart_dict`. Only consider the chart's data when summarizing."
assistant_ = "```python\n chart_dict ="

We discover that with the request message of "check errors" within the prompt, the LM can implicitly check and correct both syntax errors in the output format and the facts in the data.

user_prompt = "<img_placeholder>\nConvert the chart into a python dictionary `chart_dict`. Check json syntax errors. Only consider the chart's data when summarizing, no punctuations. Only return the valid version."

10.3 Dictionary-to-Statistics with Program of Thoughts↩︎

The illustrated prompt content is the same used in VLMs tested in this work, but formatted specifically with each VLM’s template.

system_prompt = "You are a data analyst. You are given a dictionary that represents a chart called `chart_dict`. \;
You need to implement the function `get_summary_statistics(chart_dict)` that takes the dictionary as input and returns a dictionary with the relevant statistics that can be used to summarize the chart. \;
Avoid sorting dictionary objects directly and USE ONLY PYTHON BUILT-IN FUNCTIONS. Name the keys of the dictionary to elaborate how it is a descriptive statistic. When writing Python, follow the PEP style guide. \;
Return ONLY the code of the function that will run without any errors and can work using `eval()`."

user = "Implement the function `get_summary_statistics` that takes a dictionary as input and returns a dictionary with the relevant statistics that can be used to summarize the chart using only built-in Python functions. Make sure to label the keys of the `summary_dict` to be descriptive The input dictionary is defined as {chart_dict}."

assistant_ = "```python\ndef get_summary_statistics(chart_dict):\n    # Define output dictionary `summary_dict` to store the summary statistics\n"

10.4 Chart-to-Summary with Program of Thoughts↩︎

user = "Summarize the insights of the chart with title: '{title}'. The summary use language similar to the chart. Don't explicitly describe chart elements such as chart type. NEVER START A SENTENCE WITH A NUMBER. The chart has the dictionary: {dictionary_str} and the summary_statistics: {summary_dict}."

assistant_ = "Let's think step by step to with as few steps as possible to summarize the chart: "

11 Extended Results↩︎

11.1 Ablation Studies↩︎

Table [tab:apix95text95ablation] and Table [tab:apix95ablation95rouge] present BLEU and CIDEr evaluation results, and ROUGE-1 and ROUGE-L evaluation results, respectively, for various VLMs, tested with different prompting strategies. Table [tab:apix95text95ablation95unieval95cohcons] and Table [tab:apix95text95ablation95unieval95flurel] present coherence, consistency, fluency, and relevance from UniEval evaluation results. Table [tab:apix95ablation95textpye95aliuni], Table [tab:apix95ablation95textpye95aliunicohcon], and Table [tab:apix95ablation95textpye95aliuniflurel] present AlignScore-large score and UniEval score results for various VLMs regarding textual data types with our PoT chart summarization pipeline. We discover that even the evaluation scores of template-based statistics generation mostly outperform other PoT-based statistics generation, but most summaries generated by template-based statistics are low-quality. The factual evaluation results can also indicate that the simple rule-based transformation from a Python dictionary to a structured key-value statistical analysis is rigid and incurs higher computational and development costs.

With empirical results, how a candidate task to represent charts structurally can be an effective auxiliary to the existing chart-to-table task can be sort of answered. While the evaluation of chart-to-table might be more objective in its evaluation, there might be merit to exploring the chart-to-dictionary task for chart understanding. Not only this enable the integration of the chart in a PoT context, but it also facilitates a more robust representation of the chart, given the increasing complexities of charts in the wild. This work acknowledges that there is an overlap between chart redrawing and this task, but the chart redrawing tends to focus more on the reconstruction of the chart with executable matplotlib code rather than capturing the semantic nuances of the chart elements explored in this work.

11.2 Comparison between Manual Template-based and PoT-based Statistics Generation↩︎

We show an exemplar of the predefined extracting data rules in our manual template method. The full rules can be found in our repository.

if isinstance(values, list) and values and all(isinstance(x, (int, float)) for x in values):
    return [{
        "Category": prefix,
        "Total": len(values),
        "Sum": sum(values),
        "Average": statistics.mean(values),
        "Minimum": min(values),
        "Maximum": max(values),
        "Range": max(values) - min(values)
    }]

Figure 5 shows a comparison of the numbers of short failed summaries generated by using the PoT-based and Template-based Deepseek model on the Pew dataset, indicating the effectiveness of using the PoT instead of a simple rule-based template with the VLM in chart summarization.

12 Case Study↩︎

A case study in Figure 7 demonstrates an end-to-end chart-to-text method using the PoT. In this specific instance, the chart-to-dictionary properly captures the appropriate format of how to organize the data, but fundamentally mislabels or misreads the values of which values go to which parties. However, it can be observed that in terms of observing the increasing trend in the time-series data, the dictionary was able to somewhat capture this. The generated PoT is agnostic of the actual values of the functions and is able to correctly identify the relevant keys needed to create summary statistics of total, average, and min and max values. The generated caption captures the general ideas that the chart was able to portray, specifically describing the chart elements of date in the x-axis and anger in the y-axis. While not as verbose as the original text, the generated summary was able to capture the key ideas and trends in the caption.

13 Failure Case Analysis↩︎

13.1 Python Dictionary Generation↩︎

In order to keep the desired quality of the statistics in this work, we decided to use InternVL-2.5-4B [47] with ChatGPT-4o-mini [48] to generate the data dictionary. Figure 6 shows comparisons of failure numbers of the chart data dictionary generation by each VLM, presenting InternVL has the best capability on handling and generating more data dictionaries from the chart data. Since LLaVA is primarily an LLM (LLaMA) with a vision adapter, whereas DeepSeek, InternVL, and Qwen are specialized vision-language models with strong visual encoding, we test DeepSeek, InternVL, and Qwen on generating the dictionary for chart data on Pew and VisText datasets, respectively.

But we are aware that most failure cases are due to (1) limitation on maximum LLM output length, so the output Python code is cut off a part; (2) complex structure or format of the JSON data or Python style, which cannot be generally read, recognized, or pass the execution tests, and are consequently categorized as failure cases, rather than nonsense or empty outputs. Common error message instances are collected and listed in Table 5. In future work, we will implement a module to refine the Python code into their correct format, ensuring the collection of all valuable data.

13.2 Python Code Generation↩︎

Figure 8 presents a comparison between the failure-prone code generated by general-purpose LLMs and the acceptable code produced by code-specialized LLMs, where those models were specifically pre-trained and fine-tuned on programming codes, such as Qwen-Coder. With this observation, we chose to use Qwen-2.5-Coder-14B [49], which is optimized for generating accurate and efficient code outputs, to ensure the quality of the generated code.

14 Human Evaluation Details↩︎

We randomly selected 50 chart samples from both datasets, comprising outputs from both the template-based and PoT-based methods, with 10 charts sampled for each chart type across area, bar, line, pie, and scatter charts. Three graduate students (Master’s and PhD) who research in natural language processing were invited for the human evaluation as volunteers. As participation was voluntary, no payment-related considerations apply. We developed a webpage where human evaluators were requested to select their preferred summary for each pair of summaries of the provided chart. The average total number of selections of each summary category serves as its human evaluation score. An exemplary screenshot of an instance shown on our webpage is shown in Figure 9. The evaluation results indicate the effectiveness of using PoT in generating statistics content for improving the summary quality in chart summarization compared to rule-based statistics extraction.

References↩︎