1 Introduction↩︎

The immense value created by data-driven machine learning models stems from accessible, diverse, and well-structured data [1]. However, publicly available data are running out, as noted in recent discussions of AI data scarcity [2]. Moreover, such data often lack domain specificity and structural consistency. In contrast, private data sources not only retain vast amounts of unused data but also provide curated, task-specific datasets. This gap between the need for high-quality data and the limitations of public sources drives the growth of private data transactions [3], giving rise to dedicated marketplaces such as AWS Marketplace [4], Databricks [5], and Datarade [6]. These marketplaces provide a convenient avenue for showcasing data products and facilitate connections between data sellers and buyers.

A critical function of marketplaces is enabling pricing for products, which determines transaction viability and market liquidity [7]. There are three commonly used pricing methods: (1) the cost approach, (2) the income approach, and (3) the sales comparison approach [8]–[10]. The cost approach sets the price of a product by assessing the cost of replacing or reproducing it. Evaluating the cost of data products is difficult due to the fact that they can be replicated and distributed at almost zero cost. The income approach determines the price of a product based on the income that it is expected to bring. Predicting the income of data products is challenging because they can generate different revenues across use cases and be sold multiple times to distinct customers. Due to inherent issues with these two methods in the context of data products, the sales comparison approach is more applicable and widely adopted in data markets. The sales comparison approach utilizes existing pricing information of similar products to estimate the price of the commodity to be sold. This reduces information asymmetry, fosters mutual acceptance between buyers and sellers, and ultimately facilitates transaction completion in data markets.

With the rapid development of data marketplaces, the sales comparison approach holds significant potential for proper data product pricing by leveraging aggregated pricing signals across platforms. However, its practical implementation faces a critical barrier: comparing and evaluating data products across marketplaces is challenging due to different taxonomies and inconsistent metadata granularity. This complexity is further compounded by the limited capacity of pricing agents to conduct exhaustive cross-marketplace comparisons, as the cognitive burden and time required to manually align disparate product specifications often exceed practical constraints. This structural fragmentation limits the applicability of the sales comparison approach despite its theoretical advantages, thereby constraining market efficiency through suboptimal pricing decisions and reduced transaction volumes. Consequently, enabling automated cross-marketplace price benchmarking represents a pivotal challenge for realizing the full potential of data markets.

To enhance the applicability of the sales comparison approach in data markets, we propose DaDaDa (A Dataset for Data Pricing in Data Marketplaces), containing metadata for 16,147 data products from 9 major data marketplaces and providing price references for pricing decisions. We investigate 60 data marketplaces and carefully select 9 of them based on specific criteria, including the accessibility of their data product catalogs, the diversity of data products they offer, and the comprehensiveness of the metadata they provide. This careful selection process ensures that DaDaDa offers a diverse and representative sample of a broad data marketplace ecosystem. To create DaDaDa, we use advanced web crawling techniques to extract detailed metadata for each data product, then create fields such as title, description, provider, price, volume, size, and category. Following metadata scraping, we conduct thorough data preprocessing to unify various metadata fields into a consistent format, making it easy to analyze and use for various applications. The resulting DaDaDa dataset serves as a unified, analyzable benchmark, making the systematic research and application of the sales comparison approach for data product pricing feasible.

We conduct experiments to demonstrate the effectiveness of using DaDaDa for essential tasks in data markets.

• Data product pricing. This task involves pricing data products based on their metadata. We train several regression models on DaDaDa to predict product prices, providing empirical baselines for future research on automated data pricing.

• Data product classification. This task involves classifying data products into predefined categories. We fine-tune pretrained multilingual models using product titles and descriptions, showing that DaDaDa supports semantic analysis and category prediction in data marketplaces.

• Data product retrieval. This task involves retrieving relevant data products from a large collection of data products in response to user queries. We use Elasticsearch [11] to develop a search engine within DaDaDa, allowing users to quickly find interesting data products along with corresponding pricing information.

The rest of the paper is organized as follows. We first review related work on data product pricing in Section 2. Section 3 provides an overview of the DaDaDa dataset. The dataset construction pipeline is detailed in Section 4. Section 5 explores the various applications of DaDaDa. Finally, we conclude this paper and discuss the limitations and future work in Section 7.

2 Related Work↩︎

2.1 Theoretical Data Pricing Mechanisms↩︎

Existing theoretical studies on data pricing can be broadly organized around three types of mechanisms. The first line studies how to price data access and query answers under formal economic constraints. Query-based pricing work establishes arbitrage-free and discount-free prices for database queries, and further develops scalable query-pricing systems and revenue-maximization models [12]–[14]. Closely related, Li et al. [15] develop a theoretical framework for pricing noisy query responses with compensation mechanisms based on privacy loss.

A second line designs market mechanisms for acquiring, selling, or allocating data when participants have private information. Price-quantity mechanisms account for buyers’ private valuations of data subsets [16], while auction- and Shapley-value-based marketplace designs aim to elicit truthful bids and ensure fair revenue distribution among data owners [17]. Peer-prediction mechanisms further support truthful data acquisition when no test dataset is available [18]. Related procurement work considers scenarios where individuals incur privacy costs, using truthful auctions under differential privacy to purchase private data subject to budget or accuracy constraints [19].

A third line extends pricing theory from raw data to machine-learning artifacts and privacy-aware marketplaces. Model-pricing frameworks prevent arbitrage while balancing seller revenue and buyer affordability [20], and end-to-end model marketplaces incorporate differential privacy into compensation and pricing functions to support fairness, privacy preservation, and revenue maximization [21]. Despite their theoretical rigor, these approaches share a critical limitation: they lack empirical validation using real-world pricing data from operational data marketplaces.

2.2 Empirical and Commercial Studies↩︎

Complementing theoretical work, empirical studies examine commercial data marketplaces from an empirical perspective. Azcoitia et al. [22] conduct a comprehensive survey of commercial data trading entities, analyzing data types, business models, and underlying technologies. In subsequent work, they investigate specific features that influence data product pricing [23]. Hanspach et al. [24] and Wieting and Sapi [25] conduct empirical analyses of pricing on major e-commerce platforms, analyzing pricing data from thousands of products to understand automated pricing behavior in digital marketplaces. Azcoitia et al. [26] analyze price distributions and price-driving features, highlighting both the feasibility and the challenges of cross-marketplace comparisons. However, these studies primarily focus on descriptive analysis rather than providing structured datasets for algorithmic development and benchmarking.

In summary, existing research suffers from a critical disconnect in that theoretical pricing mechanisms remain unvalidated due to the absence of real-world pricing benchmarks, while empirical marketplace studies lack structured datasets needed for algorithmic development.

3 Dataset Overview↩︎

In this section, we present an overview of DaDaDa. A detailed description of the construction pipeline is provided in Section 4.

Overall, DaDaDa distinguishes itself from existing benchmark datasets through three key contributions. (1) Bridging the Gap. It is the first real-world dataset for data product pricing, enabling both empirical validation of pricing theories and the development of practical marketplace algorithms. (2) Extensive Coverage. It provides detailed metadata for 16,147 data products sourced from 9 leading data marketplaces worldwide, facilitating in-depth analysis of the commercial data landscape. (see Section 4.1 for the full list). (3) Standardized Metadata. Each data product is described using a standardized metadata template, curated through web crawling, scraping, and thorough preprocessing to capture a comprehensive set of relevant attributes. The complete set of metadata attributes is detailed below.

•Title. The title of the data product.

• The marketplace name hosting the data product.

• The name of the data provider as listed on the data marketplace. The released table contains 1,991 named data providers. Among them, “Techsalerator” is the leading provider, offering a total of 644 data products, making it the supplier with the highest volume of offerings on the market. For products whose source platform does not disclose the provider identity, we preserve an empty provider value rather than imputing a potentially incorrect name.

• The detailed description of the data product.

• The number of records.

• The data size (in Byte) provided by the data product.

• The number of data features.

• The countries covered by the data product.

• The frequency between data product updates as announced by the seller, such as “monthly”, “daily”, and “real-time”. Most data products adopt “no-update” and “daily”.

• The filename of the data sample if available. We download and store the data sample of data products in an additional folder.

• The original category of data product may vary across different data marketplaces, each with its own way of categorization. We align the data categories from other marketplaces with the AWS Marketplace categories through manual labeling. As shown in Figure 1, Retail (Retail, Location, and Marketing Data) and Financial (Financial Services Data) have the highest proportions.

• The pricing mode of the data product. There are five pricing modes: (1) negotiation mode where data buyers need to negotiate the price with data providers, (2) free mode where the data is provided at no cost, (3) subscription mode where data buyers are charged a recurring fee on a monthly or annual basis, (4) one-off mode where data buyers pay a one-time fee to access the data permanently, and (5) usage-based mode where data buyers are charged based on the amount of data they consume, such as the volume of data downloaded or the number of API calls. As shown in Figure [fig:mode], most data products in the data marketplaces are either free or priced through negotiation. A smaller proportion (subscription, one-off, or usage-based) has clear prices.

• We use USD ($) as the currency unit. If the pricing mode is free, the price is set to 0. Since data products with negotiation mode do not have a clear price, we exclude them from the data product pricing task in Section 5.1. As a result, negotiation-mode data products are assigned a price of 0. If the pricing mode is subscription, the price represents the subscription cost for 12 months. If the pricing mode is usage-based, the price reflects the cost for a single usage. These two standardizations facilitate price comparisons within the same mode. We do not directly compare data products across different modes, as the price mode is the most significant factor influencing the price of a data product when training the pricing model.

• The web address of the detail page of the data product.

Dataset Statistics and Release. The released dataset contains 16,147 data products described by 14 standardized metadata fields. The products come from nine marketplaces with different geographic focuses and platform designs: Datarade (4,396 products), AWS Data Exchange (3,702), Snowflake (2,585), Databricks (2,004), Zhejiang Big Data Exchange (838), Canton Data Exchange (792), Beijing International Data Exchange (714), Guiyang Global Big Data Exchange (669), and Shanghai Data Exchange (447). This coverage is important for pricing research because it includes both international cloud marketplaces and regional data exchanges, whose metadata conventions and pricing disclosure practices differ substantially. In addition to the final table, we release the raw crawled files, intermediate preprocessed files, preprocessing scripts, crawler implementations, downstream experiment code, and 255 data product samples when samples are publicly available from the source platforms. These artifacts allow users to inspect how each product moves from raw marketplace metadata to the unified DaDaDa schema.

Price Coverage. Some products provide explicit nonzero prices under subscription, one-off, or usage-based modes; these products constitute the labeled subset used in the data product pricing task. The remaining products are free or negotiation-based. We retain them in the complete dataset because they are still useful for studying marketplace composition, category classification, retrieval, and pricing-mode prediction. This design follows the practical structure of current data marketplaces: many sellers intentionally hide exact prices and require buyers to negotiate, while buyers still need comparable metadata to identify alternatives and understand market positioning. By separating price_mode from price, DaDaDa supports both direct price regression on products with explicit prices and broader analysis of disclosure patterns across platforms.

Marketplace Coverage and Benchmark Scope. Table 1 summarizes the platform-level coverage of DaDaDa, including product counts, pricing-mode composition, and public sample availability. The purpose of this profiling table is to make the released snapshot auditable: users can identify which marketplaces contribute products to the unified schema and choose task-specific subsets according to their experimental needs.

This platform-level profile documents the scope of the released snapshot rather than defining a single downstream task. Since public data marketplaces organize listings using different pricing and disclosure conventions, the table helps users understand the composition of DaDaDa and select task-specific subsets for pricing, classification, retrieval, or marketplace analysis. The same principle applies to structured metadata. Attributes such as volume, size, and dimension are used when they are reliably disclosed, while textual fields such as title and description provide a consistent semantic representation across platforms. This design preserves broad marketplace coverage and allows downstream models to combine textual and structured signals according to the requirements of each task.

Metadata Sparsity. The dataset also preserves the incompleteness of real marketplace metadata. Fields such as title, description, platform, category, price_mode, and url are complete after preprocessing, while numerical fields such as volume, size, and dimension are disclosed unevenly across platforms. Instead of discarding products from metadata-poor marketplaces, we use explicit default values when a field cannot be reliably inferred from the product page. This choice makes the benchmark more representative of real data marketplace search and pricing conditions, where buyers often rely on textual descriptions and partial structured metadata. It also explains why the textual representation of title and description plays a major role in the pricing experiments in Section 5.1.

Benchmark Implications. Structured numerical attributes in data marketplaces are not disclosed as consistently as textual fields. When fields such as volume, size, and dimension cannot be reliably extracted from the original product pages, we preserve explicit default values rather than inventing product properties that sellers do not publish. The implication is that real data-product metadata is largely text-centric: title and description are the most consistently available high-content fields across marketplaces, while numerical attributes are useful complementary signals only when sellers or platforms expose them reliably.

This observation directly shapes the benchmark design of DaDaDa. First, the pricing task should combine semantic representations with structured features instead of relying on purely tabular metadata, because a structured-only model would either discard many products or depend heavily on artificial imputation. Second, the classification task is naturally formulated as a multilingual text understanding problem, since category evidence is primarily expressed in product titles and descriptions. Third, the retrieval task should combine full-text search with optional structured filters: buyers can search semantically over all products, while using price, coverage, update frequency, size, volume, or dimension filters only when those fields are available. The SHAP analysis in Section 5.1 further supports this design choice: description is the most influential feature group across subscription, one-off, and usage-based pricing modes, while structured attributes such as coverage, update frequency, and dimension provide mode-specific signals when available.

4 Construction Pipeline↩︎

We develop a comprehensive data collection pipeline to gather and organize data from various data marketplaces. As shown in Figure 2, this pipeline consists of three main stages: (1) data marketplace selection, (2) metadata scraping, and (3) data preprocessing. The details of each stage are described as follows.

4.1 Data Marketplace Selection↩︎

Ensuring that raw data sources satisfy rigorous quality and robustness standards is fundamental to developing reliable pricing benchmarks. To this end, we first identify candidate data marketplaces through a selection process guided by three criteria:

1. Data Accessibility. Data marketplaces must provide open access to their data product catalogs and permit data extraction through web crawling techniques.

2. Product Variety. Data marketplaces must encompass a sufficient range of product categories to ensure diversity. In our framework, we adopt a practical threshold of eight categories and manually assess coverage using AWS Marketplace classification criteria as the benchmark.

3. Metadata Comprehensiveness. Data marketplaces must include a substantial portion of the essential metadata fields associated with each data product. Specifically, we require that each marketplace cover at least 70% of the metadata attributes listed in Section 3.

Selection Rationale. The three criteria balance coverage, diversity, and reproducibility. Marketplace size alone is insufficient: a large catalog may be concentrated in a narrow domain or expose minimal metadata, while a smaller marketplace can still contribute structured product pages, multiple categories, and stable public listings. We therefore first require crawlable catalogs and then assess whether each candidate adds diverse and comparable metadata to the unified schema. This also prevents the benchmark from being dominated by a few high-volume but narrow catalogs. This protocol helps DaDaDa represent a broad marketplace ecosystem while remaining reproducible from public sources.

Our selection process starts with an extensive survey of data marketplaces worldwide, identifying 60 initial candidates (see Table 2 for more details). Candidates that do not meet the first criterion are eliminated in the first stage. For example, Azure [27] is removed because it requires payments to access data products. This filtering step yields a shortlist of 20 accessible candidates, as shown in Table 3. Next, we evaluate these candidates against the second and third criteria, gathering information regarding the number of data products and categories available in each marketplace. Our analysis reveals that while some marketplaces offer a large volume of data products, they are often concentrated in specific domains. For example, our manual inspection of representative product samples shows that Advaneo specializes in IoT data and Carto focuses on spatial data. Additionally, data marketplaces like Western China Data Exchange [28] and Google Cloud [29] provide only limited product information, typically restricted to titles and brief descriptions. After removing these cases from consideration, we ultimately identify 9 candidates, which are listed in the upper section of Table 3.

4.2 Metadata Scraping↩︎

After identifying the candidate data marketplaces, we apply web scraping techniques to extract the required metadata. This process is non-trivial due to heterogeneity in platform design. Specifically, some websites provide complete metadata for all products in a single request, whereas others restrict access to product-level detail pages, necessitating item-by-item extraction.

For the former case, we save the returned results. For the latter, we use the Selenium library [45] to scrape metadata for data products that are useful for data pricing. Besides, we save the URL of the data product detail page for potential applications. We create XPath and CSS selector rules to precisely target and extract information by analyzing the HTML structure of the target website. In Figure 3, we illustrate the sources from which each metadata field is scraped. When dealing with textual fields such as title and description, we simply extract the text from the corresponding HTML elements. As for the data sample, we extract the title and rows of the sample, save this data to a file, and then store the path to the saved file in the data_sample field of the corresponding data product. Once metadata is successfully extracted, it is saved in CSV format, allowing easy inspection and simplifying subsequent analysis. To avoid potential service disruptions, requests are scheduled at regular intervals, typically every three minutes. Furthermore, our crawler strictly adheres to each platform’s directives and terms of service to ensure ethical and legally compliant data collection practices.

4.3 Data Preprocessing↩︎

Finally, the scraped metadata is preprocessed through two primary procedures: data cleaning and data integration.

Data Cleaning. We begin by cleaning raw metadata through three primary operations: removing duplicate items, rectifying errors, and imputing missing values. These steps lay the foundation for subsequent preprocessing while preserving dataset integrity.

1. Removing duplicate items. Data crawling may result in the retrieval of duplicate data products. To ensure dataset uniqueness, we detect duplicates by comparing the title and provider fields. If two data products have identical fields, they are considered duplicates, and the redundant entries are removed.

2. Rectifying errors. During the crawling process, non-UTF-8 characters may be retrieved, leading to errors when reading files containing such characters. We use a script to detect and automatically remove non-UTF-8 characters from the data.

3. Imputing missing values. Some critical fields like coverage and size are often not explicitly provided on many data marketplaces and are only mentioned in the description. To address these missing fields, we use manual annotation. Specifically, we read the description field to extract relevant information to fill in the missing fields. If no relevant information is available, we use default values.

Data Integration. The objective of data integration is to merge the cleaned metadata from diverse data marketplaces into a standardized schema, thereby facilitating subsequent analysis and application deployment. Given the varying granularity of metadata, we first define a comprehensive superset that encompasses all potentially relevant attributes across data products. During integration, it is essential to account for each attribute’s characteristics. Attributes such as title and provider are independent and can be merged directly without any issues. However, when it comes to attributes like category, combining them directly is problematic due to the lack of a consistent pattern across different platforms. As shown in Table 3, every data marketplace has its own way of classifying data products. For example, AWS Marketplace [4] has 12 general categories, while Datarade [6] has 25 general categories and over 560 detailed classifications. Different classification standards increase the difficulty of finding similar data products across marketplaces and may result in the omission of some data products. To solve this problem, we manually map the diverse categorizations from various platforms onto the taxonomy of AWS Marketplace [4], ensuring consistency and creating a unified pattern to avoid the confusion that can result from a direct combination. The AWS Marketplace classification system covers a wide range of industries and application areas, ensuring that most data products can be categorized. Additionally, as a well-established data marketplace, its classification system is widely recognized and used by a broad user base. Therefore, we choose it as our benchmark.

Besides, most of the data products in the data marketplaces are denominated in USD ($) and RMB (CN¥), with smaller amounts in EUR (), GBP (£), and JPY (JP¥). To maintain consistency, we convert the other currencies to USD ($) using the Yearly Average Currency Exchange Rates [46]. Such integration eliminates redundancy and resolves conflicts, leading to a more coherent dataset.

Metadata Validation. After integration, we perform consistency checks to reduce avoidable noise before releasing the dataset. First, we verify schema consistency by ensuring that every marketplace-specific file can be projected into the same set of standardized attributes. Second, we normalize categorical values for price_mode and update_frequency; for example, platform-specific phrases such as monthly billing, yearly billing, real-time updates, and on-demand updates are converted into a fixed vocabulary. Third, we validate numerical attributes by converting storage units to bytes, normalizing subscription prices to a 12-month cost, converting usage-based prices to per-use costs when the unit is disclosed, and checking that non-free products with explicit prices have positive values. Finally, for geographic coverage, we convert country abbreviations to country names and keep multi-country coverage as a list, allowing both human inspection and machine-readable filtering.

We deliberately separate automatic normalization from manual annotation. Fields that can be parsed deterministically, such as currency conversion and many update-frequency expressions, are handled by scripts. Fields that require semantic judgment, such as aligning marketplace-specific categories to the AWS taxonomy or extracting missing coverage information from long descriptions, are manually reviewed. This separation makes the construction process auditable: users can reproduce the deterministic steps from the released scripts and inspect the curated fields in the preprocessed files. Such transparency is particularly important for a pricing benchmark, since small changes in price modes, currency conversion, or category alignment can affect both downstream model performance and the interpretation of price benchmarks.

5 Applications↩︎

In this section, we show that DaDaDa can be used for multiple downstream tasks, including data product pricing (Section 5.1), data product classification (Section 5.2), and data product retrieval (Section 5.3). We use a server with an NVIDIA A100 GPU (80GB) and 160 Intel(R) Xeon(R) Platinum 8383C CPU @ 2.70GHz for all empirical experiments in this manuscript. The operating system is Ubuntu 20.04.6.

5.1 Data Product Pricing↩︎

DaDaDa collects important metadata and pricing information for data products, which can greatly assist in making pricing decisions. To validate the effectiveness of DaDaDa, we use it to train models and evaluate pricing performance. Figure 4 shows a histogram and the corresponding CDF of data product prices using different pricing modes. We can observe that usage-based prices are mostly concentrated between $0.01 and $100, one-off prices are primarily between $10 and $10,000, and subscription prices are mainly between $100 and $100,000. The prices for usage-based data are generally associated with API-type data, which is typically priced per use, resulting in lower overall prices. One-off data usually consists of static datasets, while subscription data often includes live data that is periodically updated with the latest information. This dynamic nature of subscription data makes it more valuable compared to static data, which is reflected in its higher overall price.

Benchmark Protocol. We formulate data product pricing as a supervised listed-price regression task over products with explicit nonzero prices. The input consists of standardized product metadata, including semantic fields (title and description), categorical fields (category, coverage, update_frequency, and price_mode), and numerical fields (volume, size, and dimension) when available. The prediction target is the normalized USD price after applying the mode-specific standardization in Section 3. Free and negotiation-based products are excluded from this supervised regression task because their price values are set to zero by construction and do not represent observed listed prices. Including them would mix price regression with pricing-mode prediction and substantially change the task definition.

We predict \(\log(\texttt{price}+1)\) rather than the raw price because listed data prices span several orders of magnitude across subscription, one-off, and usage-based modes. The log transformation reduces the dominance of extremely expensive products and makes errors more comparable across price ranges. To avoid information leakage, fields that directly identify a product page, marketplace, or seller—including url, platform, provider, and data_sample—are removed before model training. Raw title and description strings are also removed after conversion into embeddings. We evaluate all pricing models using shuffled 5-fold cross-validation with a fixed random seed and report the average \(R^2\), MAE, MSE, and standard deviation across folds. This protocol makes the task a metadata-based benchmark for estimating public listed prices rather than a memorization task over marketplaces or providers.

We preprocess the data to ensure it meets the format requirements of machine learning models. The preprocessing procedure includes the following steps.

• Feature extraction. We extract features utilizing the fine-tuned XLM-RoBERTa-Large model [47]. Specifically, the model encodes description and title, obtaining word embeddings from the intermediate layer (12th layer). These high-dimensional word embeddings are then reduced in size using Principal Component Analysis (PCA) [48] to enhance computational efficiency and decrease feature count. We find the minimum number of principal components required to achieve a 95% cumulative explained variance ratio, reducing the original 1024-dimensional word embeddings to 223 dimensions.

• Categorical and numerical feature processing. For the categorical features such as category, update_frequency, and price_mode, we employ One-Hot Encoding to allow the model to interpret these data points accurately. Numerical features, including size, dimension, and volume, are standardized to ensure all data points are on a unified scale, thus preventing discrepancies in feature magnitudes from affecting the performance of the model.

• Price processing. Considering the wide range of the target variable price, we apply a log transformation to stabilize the values and improve the robustness of the training process. This comprehensive preprocessing ensures that the data is well-prepared for the subsequent predictive modeling steps.

We select several regression models to predict the prices of data products, including random forest [49], \(k\)-nearest neighbors [50], gradient boosting [51], and XGBoost [52]. For each of these models, we use grid search combined with cross-validation to find the best parameters and perform 5-fold cross-validation to evaluate the performance. In addition to these traditional regression models, we develop a basic deep neural network (DNN). We use ReLU [53] as the activation function, Adam [54] as the optimizer, and apply Dropout [55] to reduce overfitting.

A variety of metrics are used to evaluate the performance of models, including the coefficient of determination (\(R^2\) score), mean absolute error (MAE), mean squared error (MSE), and standard deviation (SD). Table 4 presents the performance of different models on the data product pricing prediction task. The experimental results show that XGBoost consistently outperforms the other models in most evaluation metrics.

We also extract word embeddings from the last (24th) layer. The remaining experimental procedures are consistent with the 12th layer, where we identify the best parameters through grid search and evaluate model performance using 5-fold cross-validation. Table 5 summarizes the experimental results. XGBoost outperforms other models across all metrics, but it performs worse than the method that uses the intermediate layer as word embeddings in Section 5.1. This is because the last layer of the model is more focused on the classification task, potentially losing some information.

Ablation Study. To quantify the contribution of textual and structured metadata, we conduct an ablation study using Gradient Boosting under the same 5-fold cross-validation protocol. The structured-only setting excludes text embeddings, the text-only setting uses only PCA-reduced XLM-RoBERTa-Large embeddings of title and description, and the full setting combines both feature groups. The “Features” column in Table 6 reports the final input dimensionality after text PCA, categorical one-hot encoding, and numerical standardization, rather than the number of raw metadata fields.

As shown in Table 6, structured metadata alone provides useful pricing signals, achieving an \(R^2\) score of 0.779. Text embeddings alone are less accurate, but they provide complementary semantic information: combining 12th-layer text embeddings with structured metadata improves \(R^2\) to 0.834. The 12th-layer representation also outperforms the 24th-layer representation in both text-only and full settings.

Pricing Results for Two Major Categories. For this experiment, we specifically select the two major data categories in the dataset: “Financial Services Data” and “Retail, Location and Marketing Data”. We train and test the pricing models separately on these two categories to assess their domain-specific performance. As shown in Table 7, the pricing model achieves higher accuracy on “Financial Services Data” than “Retail, Location and Marketing Data”. This discrepancy is likely attributable to more consistent and well-defined pricing patterns in “Financial Services Data”, which enable the model to better capture the underlying pricing logic and generate more accurate estimates.

Pricing Result Explanation. The above experimental results indicate that models trained on DaDaDa can accurately predict data product prices. To identify the key determinants of pricing, we utilize SHAP (SHapley Additive exPlanations) [56], a game theory-based interpretability method that breaks down model predictions into contributions from individual features.

Figure 4 highlights significant differences in price distributions across different pricing modes, so we conduct separate feature importance analyses for each mode using three top-performing models: Random Forest (RF), Gradient Boosting (GB), and XGBoost (XGB). As illustrated in Figure 5, the findings are as follows:

• “Description” is recognized as the most critical feature across all pricing modes, reflecting market realities where buyers mainly evaluate products based on descriptive content.

• “Update frequency” holds significance in both one-off and subscription modes. One-off buyers appreciate frequent updates that enhance the utility of the data, while subscription users are willing to pay a premium for regularly refreshed information.

• “Coverage” is crucial in the subscription mode, as subscribers need comprehensive data for long-term decision-making.

• “Dimension” is particularly important in the usage-based mode, where higher-dimensional data commands premium pricing due to the increased information density offered per call.

• Model comparisons reveal consistent feature attention patterns, with XGBoost demonstrating the most balanced SHAP value distribution, which likely contributes to its superior pricing performance.

Table 8 further summarizes the top feature groups by pricing mode. The values are grouped mean absolute SHAP scores averaged across Random Forest, Gradient Boosting, and XGBoost over the cross-validation folds. This quantitative summary confirms that description dominates all three pricing modes, while structured fields provide complementary mode-specific signals.

Generalization. We experimentally validate the generalization ability of our pricing model trained on the DaDaDa dataset. The DaDaDa dataset contains data products from 9 different data marketplaces. We select the data product information from 8 of these marketplaces as the training set to train our pricing model and then test the pricing prediction capability of the model on the 9th marketplace (Snowflake). This approach allows us to assess the generalization ability of the model across different data marketplaces and evaluate its performance in a new market environment. The experimental results in Table 9 demonstrate that our model achieves strong generalization performance on the Snowflake marketplace after training on data from eight distinct marketplaces. Although data products from different marketplaces may have distinct characteristics, our model is still capable of achieving high predictive accuracy in an unseen marketplace.

5.2 Data Product Classification↩︎

DaDaDa supports training classification models to assist in classifying new data products into predefined categories based on their metadata. To validate its effectiveness, we fine-tune two multilingual pretrained models: mBERT (bert-base-multilingual-cased) [57] and XLM-RoBERTa-Large [47], testing their classification performance.

We divide DaDaDa into three parts: a training set (60%), a validation set (20%), and a test set (20%). The category of a data product can be inferred from its title and description. These two fields are mainly in English and Chinese, with a small portion in Japanese. To work with this multilingual data, we choose mBERT and XLM-RoBERTa-Large, which have been pretrained in multiple languages. We concatenate title and description, and input them into the models for fine-tuning. We assess the performance using the validation set and select the best fine-tuned model after multiple rounds of training. Finally, we evaluate the performance of the best fine-tuned model using the test set.

Table 10 presents the performance of the fine-tuned models on the test set. XLM-RoBERTa-Large shows better classification performance in most categories, outperforming mBERT overall. Both models achieve an average F1-score of over 0.8, demonstrating that our dataset can effectively aid in classifying data products and promote the standardization of classification. Table 11 reports the corresponding confusion matrix.

5.8pt

5.3 Data Product Retrieval↩︎

When searching for a suitable dataset for machine learning or other purposes, it can be frustrating to sift through many data marketplaces and thousands of data products, only to find that very few meet the requirements for training. While general search engines are the first thing that comes to mind for online searches [58], they are not well-equipped to handle the specific task of searching for data products. This is due to the fact that different data marketplaces offer different types of data products with varying levels of detail in the metadata, making it a daunting task to gather comparable results and decide which product to choose. DaDaDa makes it possible to create a vertical search engine for data products collected from major open-access data marketplaces around the world. To demonstrate how DaDaDa can streamline the data search process, we use it to create a search engine based on Elasticsearch.

Figure 6 displays the search engine based on DaDaDa. The left part of the figure shows the metadata filter, where users can adjust settings to obtain filtered results. The available options in DaDaDa can be categorized into two types of filters: range filters and keyword filters. Price, volume, size, and dimension are examples of range filters, allowing users to set minimum and maximum values for each option. The remaining features are keyword filters, offering multiple options for users. The system generates a query based on the filter parameters and the search text in the input box at the top of the page. The most relevant query results are displayed below the input box, along with necessary metadata such as title, price, platform, and part of the description. Users can click the title to navigate to the detail page of the product for more information or to make a purchase.

For example, to find data products that contain B2B data covering the United States, the user can simply input “B2B data US”, and all results matching the search text are displayed below. If the user has more detailed requirements, such as products from Datarade with a one-off pricing mode and a price higher than 2000 USD, the user can set the “Platform”, “Pricing Mode”, “Price Range”, and “Data Coverage” filters accordingly for a refined result.

Retrieval Implementation. The search engine indexes each data product as a document containing textual fields, structured categorical fields, and numerical fields. The title, description, and geographic coverage fields are indexed for full-text matching, while platform, provider, category, price_mode, and update_frequency are represented as keyword filters. Numerical attributes such as price, volume, size, and dimension are represented as range-query fields. This mixed indexing design is useful for data-product search because user intent is usually both semantic and constrained: a buyer may search for “mobility data”, but also require a particular region, a maximum price, or a specific pricing model.

We additionally include a country synonym analyzer for geographic coverage. This helps bridge query variations such as country names, abbreviations, and region-related terms, which frequently appear in marketplace descriptions. Because every retrieved item links back to its original marketplace URL, the search engine acts as a cross-marketplace discovery layer rather than replacing the original transaction platforms. The implementation demonstrates that DaDaDa can support not only offline model training, but also user-facing data discovery systems where pricing information, metadata filters, and original product pages are presented together.

Retrieval Case Study. The Elasticsearch prototype maps each request to three query channels: full-text matching over title, desc, and geo_coverage; keyword filtering over platform, provider, category, pricing_type, and freq; and range filtering over price, volume, size, and dimension. These channels mirror the interface in Figure 6: buyers first express a semantic need, such as “B2B data US” or “mobility data Germany”, and then narrow results by marketplace, geography, billing model, update frequency, or numerical constraints. The mapping also keeps the case reproducible because each interface constraint corresponds to a field in the released index schema.

For example, if a buyer needs B2B data covering the United States from Datarade with one-off pricing above 2,000 USD, the prototype matches the query text, uses country synonyms to align “US”, “U.S.”, and “United States”, and applies platform, pricing_type, and price filters. Returned product cards show comparable metadata, including marketplace, title, provider, category, update frequency, coverage, price or free/negotiation status, a short description, and the original URL. This case study does not claim end-user retrieval effectiveness; rather, it shows that the unified DaDaDa schema is sufficient to build a practical vertical search layer over heterogeneous data marketplaces.

The example also exercises all three field families in the released schema, making it useful for checking whether text analyzers, keyword filters, and numerical range predicates work together. Since each result links back to the original product page, the prototype remains a discovery layer that helps buyers compare candidates before inspecting licensing and purchase details on the source marketplace.

6 Availability, Ethics, and Limitations↩︎

Availability and Reproducibility. We release DaDaDa through the project repository at https://github.com/ZJU-DIVER/DaDaDa, together with a Kaggle mirror for convenient dataset access. The repository contains the final standardized dataset, raw crawled files, intermediate preprocessed files, sample data files, crawling code, preprocessing scripts, experiment scripts, and metadata documentation. The pricing pipeline includes scripts for extracting XLM-RoBERTa-Large embeddings, applying PCA, constructing regression features, training pricing models, and computing SHAP-based feature importance. The classification pipeline includes model fine-tuning and test scripts for multilingual pretrained models. The retrieval pipeline includes the Elasticsearch index schema and a frontend demonstration. Releasing these components together allows researchers to reproduce our task results and to extend the benchmark with new pricing models, classification methods, or retrieval interfaces.

Ethical Considerations. The dataset is constructed from publicly accessible product metadata rather than private transaction records or raw customer data. We collect product-level descriptions, public seller-provided attributes, prices, and URLs that are visible on data marketplace pages. During crawling, requests are scheduled at controlled intervals and platform directives are respected to avoid service disruption. We do not redistribute private buyer information, transaction histories, or any non-public information. When product samples are publicly provided by the source marketplace, we store only the sample files as released by the marketplace; otherwise, the data_sample field records that no sample is available.

Because the dataset describes commercial data products, it should be used as a benchmark for research, marketplace analysis, and decision support rather than as a tool for copying or reselling products. Prices and descriptions remain attributable to their original marketplaces through the public URLs. Users should check the original product page and licensing terms before purchasing, using, or redistributing any referenced data product. This is especially important for sensitive domains such as healthcare, finance, mobility, and personal-data-related products, where marketplace listings may involve additional legal or contractual constraints.

Limitations. DaDaDa is a versioned snapshot of public marketplace listings, so it is intended as a benchmark for listed-price estimation and metadata-driven comparison under transparent and reproducible assumptions rather than a record of realized transactions. Public pages may change over time, and private discounts, enterprise contracts, bundling, or custom licensing terms are usually not visible. Some structured fields are also disclosed unevenly across platforms; we preserve such products to maintain broad marketplace coverage and encode unavailable numeric values consistently instead of inventing unsupported properties. Negotiation-based products remain useful for marketplace and retrieval analysis even when they are not included in listed-price regression. These design choices keep the dataset aligned with how buyers observe data products in practice, while the released URLs, crawlers, preprocessing scripts, and unified schema allow future snapshots to refresh listings and compare new methods under the same public-data protocol.

7 Conclusion↩︎

In this paper, we introduce DaDaDa, the first comprehensive dataset for data product pricing. This dataset fills the gap in data product pricing by including various features that describe and influence the price of a data product. Additionally, we develop a data product search engine to improve the data retrieval process. Experimental results show that both the pricing and classification tasks achieve high accuracy, highlighting the effectiveness of DaDaDa in enhancing data product pricing and classification. Beyond these tasks, DaDaDa provides a reusable benchmark for empirical data-marketplace research. The released metadata, URLs, crawlers, preprocessing scripts, and task code allow future work to refresh public listings and compare pricing, classification, and retrieval methods under a shared protocol. This gives researchers a common reference point for studying data products across marketplaces.

The importance of data markets is increasingly recognized and the field continues to develop rapidly. In contributing to this evolving landscape, we plan to develop an online metadata crawling system that can automatically gather the latest product metadata and process it into a standardized format to ensure regular updates for the dataset. Such temporal extensions will make it possible to study how sellers revise prices and how marketplace-level disclosure practices evolve.

References↩︎