January 01, 1970
Bangladeshi e-commerce platforms host millions of product reviews written in Bengali Unicode, English, and Banglish, where Bengali is phonetically transcribed in Latin script. However, the impact of code-mixed reviews on recommendation performance remains largely unexplored. We present the first such benchmarking on product reviews from Daraz Bangladesh, evaluating six model families under a per-user chronological leave-last-out protocol. To address the severe long-tail sparsity of the dataset, where 59.3% of users have exactly one interaction, we conduct a systematic k-core threshold ablation across five density configurations. The results reveal that Item-based Collaborative Filtering remains stable across settings, Implicit Matrix Factorization degrades sharply with decreasing density, and Explicit Matrix Factorization uniquely improves at higher thresholds. To characterize the impact of code-mixing on recommendation quality, we perform a language-stratified evaluation of content-based filtering using character n-gram TF-IDF profiles. The results provide empirical evidence that fragmentation of the Banglish vocabulary reduces NDCG@10 by 46.8% relative to Bengali-script users, a degradation traceable to transliteration inconsistency across surface forms. This work establishes a reproducible evaluation foundation for recommendation research in code-mixed, low-resource e-commerce settings. The code is publicly available at https://github.com/os-car-war-thy/daraz-recsys.
Recommender systems underpin product discovery in modern e-commerce [1]. Existing canonical testbeds—MovieLens [2] and Amazon Product Reviews [3]—have driven substantial progress in recommendation research. However, these benchmarks are predominantly derived from mature Western digital marketplaces, leaving rapidly growing e-commerce ecosystems in emerging markets comparatively underrepresented.
Bangladesh presents a compelling case. With a population exceeding 170 million people [4] and a rapidly expanding digital commerce sector, the country represents one of the largest emerging consumer markets in South Asia, with its e-commerce market projected to reach approximately USD 10 billion [5]. Yet, little is known about how recommendation systems perform in such environments, where consumer behavior, product ecosystems, and user-generated content differ substantially from those represented in existing benchmarks. In particular, online reviews in Bangladesh frequently exhibit a distinctive code-mixing phenomenon [6], [7], where users seamlessly combine Bengali and English and often write Bengali words using Latin characters (commonly known as Banglish). For example, “valo”, “bhalo”, “balo”, and “vhalo” are common Banglish variants of the Bengali word meaning “good” in English. These variants appear 19,834 times collectively in BanglishRev, a product review dataset from Daraz [6]. Such linguistic variation is not merely incidental noise; it introduces vocabulary fragmentation and representation challenges that are rarely encountered in conventional recommendation benchmarks and can substantially affect the modeling of textual side information [7], [8].
To date, no systematic recommendation benchmark exists for Bengali e-commerce product data . Existing Bangladeshi e-commerce datasets have been developed primarily for sentiment analysis [6], [9]–[11] and lack the user–item interaction structure required for recommendation research. BanglishRev dataset [6] contains user–item interaction signal but the authors focused solely on sentiment analysis and did not explore product recommendation.
Our work makes three contributions:
First systematic benchmark. We evaluate six model families on BanglishRev, establishing the first reproducible Top-N recommendation benchmark for Daraz product recommendation.
Code-mix impact analysis. A language-stratified evaluation of content-based filtering provides the first empirical measurement of how Banglish vocabulary fragmentation degrades recommendation quality, with Banglish-dominant users experiencing a 46.8% reduction in NDCG@10 relative to Bengali-script users.
Sparsity ablation. We systematically vary k-core filtering thresholds across five density configurations, revealing that Implicit Matrix Factorization (ImplicitMF) degrades sharply with decreasing density, whereas Explicit Matrix Factorization (ExplicitMF) uniquely improves. This finding has important implications for model selection in sparse interaction settings.
BanglishRev [6] contains 3.24 million ratings from 934,949 users across 128,494 products collected from Bangladeshi online retail platforms. After deduplication—retaining the highest observed rating for each \((\text{user}, \text{item})\) pair as the strongest preference signal—the interaction table contains 2.67 million unique interactions. The resulting user–item matrix has a density of 0.0022%, highlighting the extreme sparsity of the dataset.
User activity exhibits a pronounced long-tail distribution: 59.3% of users have exactly one interaction and 75.9% have at most two interactions, while only 1.5% have more than 20 interactions. Item popularity is similarly skewed: 35.4% of products receive a single review and 50% receive at most two reviews. The rating distribution is strongly positive-skewed, with 77.93% of ratings assigned the maximum five-star score, consistent with the voluntary review behavior observed on other e-commerce platforms. Furthermore, 45.89% of the 3.24 million reviews contain no review text, creating a substantial challenge for text-based recommendation approaches.
Language and script characteristics. To obtain reliable estimates of user language behavior, script composition is analyzed on the primary \((10,3)\) k-core filtered dataset, where each user has at least 10 interactions and each item has at least 3 interactions after convergent filtering. A user’s dominant script is defined as the script accounting for the largest proportion of their reviews. Among the 39,550 retained users, 61.1% wrote no review text (dominant script: empty), 23.0% are English-dominant, 13.5% Bengali-dominant, and 2.2% Banglish-dominant. Only 27.6% of users are effectively monolingual (more than 90% of reviews in a single script), confirming the prevalence of multilingual and code-mixed behavior within the platform.
K-Core Filtering: We apply convergent iterative k-core filtering, repeatedly removing users with fewer than Minimum User Interactions (MUI) and items with fewer than Min Item Interactions (MII) until no further removals occur. This procedure guarantees that all remaining users and items simultaneously satisfy the specified interaction thresholds. We evaluate five k-core configurations, denoted as \((u,i)\), where \(u\) and \(i\) represent the minimum number of interactions required per user and per item, respectively, as shown in Table 1.
Our primary benchmarking configuration is \((10,3)\), which retains 876,433 interactions. This setting provides a favorable trade-off between dataset size and interaction density while preserving sufficient observations for both user-based and item-based collaborative filtering methods. The resulting dataset contains 39,550 users and 31,713 items, yielding a relatively balanced user–item ratio.
| (MUI, MII) | Users | Items | Interactions | Density | Retention |
|---|---|---|---|---|---|
| (5, 3) | 107,075 | 42,462 | 1,327,720 | 0.029% | 49.7% |
| (10, 3) | 39,550 | 31,713 | 876,433 | 0.069% | 32.8% |
| (15, 3) | 20,957 | 25,074 | 647,838 | 0.12% | 24.3% |
| (15, 5) | 19,032 | 15,022 | 587,828 | 0.19% | 22.0% |
| (20, 5) | 11,454 | 11,499 | 447,773 | 0.32% | 16.8% |
Each successive threshold substantially reduces coverage — the primary (10,3) configuration retains 32.8% of interactions and 4.2% of the original user population, concentrating evaluation on the platform’s most active users and representing a practical limitation on result generalizability.
Temporal Split: For each user, interactions are sorted chronologically by timestamp. The most recent interaction is assigned to the test set, the second-most recent interaction to the validation set, and all preceding interactions to the training set. This leave-last-out protocol simulates next-interaction prediction while preventing future interactions from contaminating model training [12], [13]. Explicit leakage checks verify zero overlap among training, validation, and test partitions. After k-core filtering, users with fewer than three interactions are excluded because at least one interaction is required for each of the training, validation, and test splits.
GlobalPop: A non-personalized baseline that recommends the globally most-interacted items to all users [14].
CatPop: A category-aware popularity baseline that identifies each user’s dominant training category and recommends the most popular items within that category. For users with insufficient interaction history, recommendations default to GlobalPop [15].
UserCF: A neighborhood-based collaborative filtering model that constructs a sparse user–item interaction matrix, computes pairwise cosine similarity between users, and aggregates preferences from the top-50 most similar neighbors to generate recommendations [16].
ItemCF: An item-based collaborative filtering model that computes cosine similarity between items and scores candidate items using similarity-weighted aggregation over a user’s interaction history [16]. ItemCF is expected to be more robust than UserCF under extreme sparsity because item similarities are generally more stable than user similarities.
ExplicitMF (SVD): A matrix factorization model that decomposes the observed rating matrix into latent user and item representations by minimizing squared reconstruction error [17]. We use 100 latent factors and an \(\ell_2\) regularization coefficient of 0.02.
ImplicitMF (ALS): An implicit-feedback matrix factorization model that treats interactions as binary preference signals with confidence weighting and optimizes latent representations using Alternating Least Squares (ALS) [18]. The model uses 64 latent factors.
| Model | Hit@10 | Hit@30 | Hit@50 | MRR@10 | MRR@30 | MRR@50 | NDCG@10 | NDCG@30 | NDCG@50 |
|---|---|---|---|---|---|---|---|---|---|
| GlobalPop | 0.0215 | 0.046 | 0.0613 | 0.006 | 0.0072 | 0.0076 | 0.0096 | 0.0151 | 0.0179 |
| CatPop | 0.0202 | 0.037 | 0.0468 | 0.0076 | 0.0085 | 0.0087 | 0.0105 | 0.0145 | 0.0163 |
| UserCF | 0.0444 | 0.081 | 0.108 | 0.0167 | 0.0187 | 0.0194 | 0.0231 | 0.0317 | 0.0368 |
| ItemCF | 0.0525 | 0.095 | 0.1241 | 0.0194 | 0.0219 | 0.0226 | 0.0270 | 0.0373 | 0.0426 |
| ExplicitMF | 0.0361 | 0.066 | 0.0877 | 0.0148 | 0.0165 | 0.017 | 0.0198 | 0.0269 | 0.0309 |
| ImplicitMF | 0.0399 | 0.082 | 0.1123 | 0.0135 | 0.0159 | 0.0167 | 0.0196 | 0.0296 | 0.0352 |
Content-Based Filtering (CBF): A content-based recommender that represents items using TF–IDF features extracted from aggregated review text. To improve robustness to Banglish transliteration variation, we employ character 1–2 gram features with a vocabulary size of 15,000 [7]. When review text is unavailable, a three-stage fallback strategy is applied: (1) TF–IDF features derived from review text, (2) category-string features, and (3) a zero vector representation. The final fallback applies to approximately 8% of items. User profiles are constructed as the \(\ell_2\)-normalized mean of the vectors corresponding to items in the user’s training history [19]. Due to computational constraints, CBF is evaluated only under the primary (10,3) filtering configuration.
Recommendation quality is evaluated using Hit Rate (HR@K), Mean Reciprocal Rank (MRR@K), and Normalized Discounted Cumulative Gain (NDCG@K) at \(K \in \{10, 30, 50\}\). Among these metrics, NDCG@10 serves as the primary evaluation criterion. To ensure fair evaluation, items observed during training are excluded from each user’s candidate set at inference time. All metrics are computed using a vectorized NumPy-based evaluation pipeline.
For the code-mixing analysis, users are partitioned according to the dominant script used in their review history (e.g., Bangla, English, or Banglish). NDCG@10 is then computed separately for each user segment to assess whether recommendation performance varies across different linguistic usage patterns.
ItemCF achieves the best overall performance across all evaluation metrics and cutoff values, attaining an NDCG@10 of 0.027 and an NDCG@50 of 0.043. UserCF ranks second at every cutoff, establishing neighborhood-based collaborative filtering as the strongest model family on this benchmark. This result is consistent with prior observations that latent-factor methods often struggle in extremely sparse settings, where limited interaction data hinders the estimation of reliable user and item representations [18].
The two matrix factorization approaches exhibit different ranking characteristics. ExplicitMF slightly outperforms ImplicitMF at short recommendation lists (NDCG@10: 0.0198 vs.), whereas ImplicitMF surpasses ExplicitMF at larger cutoffs (NDCG@30: 0.0296 vs.), with the performance gap increasing further at K=50. This pattern suggests that ExplicitMF produces more accurate top-ranked recommendations, while ImplicitMF achieves greater coverage and recall over longer recommendation lists.
Finally, the non-personalized baselines perform substantially worse than all personalized methods across every cutoff. Despite the extreme sparsity of the interaction matrix, user-specific interaction histories still provide sufficient signal to support meaningful personalization.
| Model | (5,3) | (10,3) | (15,3) | (15,5) | (20,5) | \(\Delta\) |
|---|---|---|---|---|---|---|
| GlobalPop | 0.0078 | 0.0096 | 0.0106 | 0.0118 | 0.0123 | +0.0045 |
| CatPop | 0.0126 | 0.0105 | 0.0102 | 0.0114 | 0.0122 | \(-\)0.0004 |
| UserCF | 0.0305 | 0.0231 | 0.0199 | 0.0215 | 0.0191 | \(-\)0.0114 |
| ItemCF | 0.0302 | 0.0270 | 0.0244 | 0.0270 | 0.0255 | \(-\)0.0047 |
| ExplicitMF | 0.0200 | 0.0198 | 0.0201 | 0.0228 | 0.0243 | +0.0043 |
| ImplicitMF | 0.0275 | 0.0196 | 0.0156 | 0.0170 | 0.0136 | \(-\)0.0138 |
The density ablation reveals four distinct sensitivity profiles, indicating that recommendation algorithms respond differently to increasingly sparse interaction data. ItemCF is the most robust model, exhibiting only a modest performance decline (\(-0.0047\)). This stability is consistent with item-based similarity estimation being less sensitive to reductions in user coverage than user-based approaches.
In contrast, ImplicitMF is the most adversely affected (\(-0.0138\)). A plausible explanation is that confidence-weighted implicit-feedback models require sufficient interaction volume to differentiate user preferences; under extreme sparsity, confidence values become increasingly homogeneous, reducing the model’s ability to learn discriminative latent representations.
ExplicitMF exhibits the opposite trend, improving as density increases (\(+0.0043\)). One possible explanation is the highly skewed rating distribution, where 77.93% of ratings are five stars. Under sparse conditions, this skew may encourage near-constant rating predictions. As denser user–item interactions are retained, greater rating diversity becomes available, yielding a more informative supervision signal for factorization.
UserCF experiences a moderate performance degradation (\(-0.0114\)), consistent with the well-documented sensitivity of user-based neighborhood methods to reductions in user overlap. Finally, GlobalPop benefits from increasing density, suggesting that popularity estimates become more reliable as interaction counts accumulate.
| Script Segment | N Users | NDCG@10 | Hit@10 | \(\Delta\) vs Bengali |
|---|---|---|---|---|
| Bengali | 5,354 | 0.0094 | 0.0211 | – |
| Empty (no reviews) | 24,183 | 0.0081 | 0.0166 | \(-\)13.8% |
| English | 9,096 | 0.0062 | 0.0133 | \(-\)34.0% |
| Banglish | 866 | 0.0050 | 0.0115 | \(-\)46.8% |
| Overall CBF | 39,550 | 0.0077 | 0.0160 | – |
At the (10,3) threshold, ItemCF achieves NDCG@10 = 0.027 while CBF achieves 0.0077 — a 3.5× gap demonstrating that collaborative signals substantially outperform text-based profiles in this domain.
Within CBF, Banglish-dominant users suffer the steepest degradation (\(-46.8\)% relative to Bengali). This is directly attributable to transliteration inconsistency: a single concept is split across multiple surface forms (e.g., *valo/bhalo/balo/vhalo* for "good"; *sundor/shundor* for "nice"; *khub/kub/khob* for "very"), fragmenting the TF-IDF vocabulary and reducing inter-item similarity. Character 1-2 gram features partially mitigate this, but cannot fully bridge orthographic distance across arbitrary romanization choices. English-dominant users also degrade meaningfully (\(-34.0\)%), likely due to shorter average review length (mean 40.4 chars vs 73.9 for Bengali) and higher generic vocabulary overlap reducing discrimination.
A counterintuitive finding: users with no dominant review script ("empty") outperform both English and Banglish users (NDCG@10 = 0.0081). Since user profiles are computed as the mean of their interacted item vectors, "empty" users benefit from item profiles constructed from *other users’* text — circumventing the transliteration fragmentation problem by never relying on their own noisy textual history.
| Work | Domain | Language | Interactions | Code-Mix | |
|---|---|---|---|---|---|
| MovieLens [2] | Movies | English | 20M | \(\times\) | |
| Amazon [3] | Fashion | English | \(\sim\)1M | \(\times\) | |
| RokomariBG [20] | Books | Bangla | 210K | \(\times\) | |
| BLaIR [21] | E-comm | English | 570M | \(\times\) | |
| This work | E-comm | BN+EN+Mix | 2.67M | \(✔\) |
Recommendation benchmarking has been anchored to a small set of English-language datasets. MovieLens [2] and Amazon Reviews [3] remain dominant testbeds for collaborative filtering [16], [18] and matrix factorization [17], [22] research. In the Bangla domain, RokomariBG [20] is the first published recommendation benchmark, evaluating models from CF and MF to LightGCN and neural two-tower retrieval on Bangla book data. However, it does not study code-mixed text effects. Prior Bangladeshi e-commerce datasets are predominantly constructed for sentiment analysis [9], [10], [23], lacking the user-item interaction structure required for personalized recommendation. BLaIR [21] benchmarks LLMs as semantic encoders across 570M English reviews, finding poor correlation with general embedding benchmarks — reinforcing that recommendation is a distinct task requiring domain-specific evaluation. For code-mixed NLP, shared tasks on Dravidian languages [24] and Bangla-English mixing [7] establish that transliteration inconsistency severely fragments vocabulary in word-level models, motivating character n-gram approaches [8].
This work presents the first Top-\(N\) recommendation benchmarking study on interaction data collected from Daraz Bangladesh, the country’s largest e-commerce platform. As Bangladesh’s e-commerce sector continues to expand, effective recommender systems are becoming increasingly important for product discovery and customer engagement. Our benchmarking results provide an empirical foundation for understanding recommendation performance in this emerging market setting.
Across five \(k\)-core density configurations and a temporally consistent evaluation protocol, neighborhood-based collaborative filtering methods consistently outperform matrix factorization approaches, highlighting the challenges posed by extreme interaction sparsity. The density ablation further provides practical deployment insights: ItemCF is the most robust under sparse conditions, ExplicitMF benefits from increased interaction density, and ImplicitMF is particularly sensitive to sparsity.
Beyond benchmarking, our language-stratified analysis identifies Banglish vocabulary fragmentation as a challenge for content-based recommendation. The coexistence of Bangla, English, and multiple Banglish transliterations introduces linguistic variability that degrades text-based item representations. This finding motivates future work on transliteration normalization, multilingual representations, and language-aware recommender systems for Bengali and other low-resource multilingual markets.