Natural Language Processing

Extended Transformer architecture that captures much longer dependencies than RNNs through segment-level recurrence and relative positional encoding.

A company offering open-source large language models, including general-purpose chat models and specialized models for code generation and mathematics.

Robustly optimized BERT algorithm that improves upon BERT with new pretraining objectives, larger mini-batches, and higher learning rates for better NLP performance.

Text-to-Text Transfer Transformer that treats every NLP task as a text-to-text problem, enabling a unified framework for translation, summarization, Q&A, and other language tasks.

Identifier-aware unified encoder-decoder model that leverages code semantics for both code understanding and generation tasks including defect detection and summarization.

A lighter BERT variant that reduces parameter count sharply while retaining most of BERT's performance on NLP benchmarks through factorization and parameter sharing.

Autoregressive pre-trained transformer using permutation language modeling to learn bidirectional context and outperform BERT on multiple language understanding tasks.

Distilled version of BERT with far fewer parameters and faster inference while preserving most of BERT's performance on language understanding tasks.

A bidirectional transformer pre-trained with masked language modeling; strong on many NLP tasks.

A large multimodal language model that understands text and images, more accurate and creative than GPT-3.5.

A family of open foundation models optimized for efficiency and performance across sizes and tasks.