Create new
Search
My space
Scheduled
Library
Blog
SUN API
Affiliate
Recent
Nothing yet
Log in
Give feedback
Was this episode page helpful?
Yes
No
Sign in
to share
Share detailed feedback
Why BERT Encoders Hit a Ceiling and How UltraBERT Breaks It · SUN
Podcast
History & Politics
Why BERT Encoders Hit a Ceiling and How UltraBERT Breaks It
Introduction
0:00
2:23
Episodes
Transcript
0. Introduction
2:23
1. Why Encoders Stall and How UltraBERT Breaks the Barrier
5:17
2. Dissecting the Pre‑Training Recipe: Masking Ratios, Losses, and Geometry
5:34
3. Distillation Dilemmas: When Logit‑Based Distillation Helps and Hurts
3:28
4. UltraBERT Architecture: LM‑Head Decoupling and Long‑Context Design
5:57
5. Benchmark Showdown: UltraBERT vs. ModernBERT, Ettin Encoder, and Others
4:18
6. From Theory to Practice: Applying UltraBERT Insights to Your Projects
6:05
