BamiBERT: A New BERT-based Language Model for Vietnamese

arXiv cs.CL4w4 min read

arXiv:2607.02259v1 Announce Type: new Abstract: In this paper, we introduce BamiBERT, a new BERT-based pre-trained language model for Vietnamese that addresses key limitations of PhoBERT -- the current de facto Vietnamese text encoder. Trained from scratch on a 129GB corpus of general-domain Vietnamese text for 20 epochs, BamiBERT supports an extended context length of up to 2048 tokens and operates directly on raw input, eliminating the need for external word segmentation. Across 8 Vietnamese benchmarks, it achieves the best score on 11 of 15 metrics and the second-best on 3 others, setting a

Read the full story at arXiv cs.CL →

BamiBERT: A New BERT-based Language Model for Vietnamese

More in Machine Learning

Reducing Human Annotation with ML Active Learning

Learning What Matters: Supervising Sparse Attention Routing with Causal Evidence Sets

On the Depth Scalability of Logic Gate Networks

Pretraining EHR Foundation Models with Patient-Aware Sampling

From Perturbation Correction to Geometry-Aware Sampling: Sharpness-Guided Equilibrium Sampling for Balanced Flat Minima in Long-Tailed Learning

Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection