MehdiIntroducing SubQ
Introducing SubQ - a major breakthrough in LLM intelligence.
It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), and the first frontier model with a 12 million token context window which is:
- 52x faster than FlashAttention at 1MM tokens
- Less than 5% the cost of Opus
Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter.
@subquadratic finds and focuses only on the ones that do.
That’s nearly 1,000x less compute and a new way for LLMs to scale.