Infrastructure page 157

OpenAI News July 28, 2022 07:00

Efficient training of language models to fill in the middle

We show that autoregressive language models can learn to infill text after we apply a straightforward transformation to the dataset, which simply moves a span of text from the middle of a document to its end. While this data augmentation has garnered much...

Topics: Models Infrastructure

Entities: Models Infrastructure Perplexity

OpenAI News June 28, 2022 07:00

DALL·E 2 pre-training mitigations

In order to share the magic of DALL·E 2 with a broad audience, we needed to reduce the risks associated with powerful image generation models. To this end, we put various guardrails in place to prevent generated images from violating our content policy.

Topics: Models Policy Infrastructure

Entities: Models Policy Infrastructure

OpenAI News June 17, 2022 07:00

Evolution through large models

This paper pursues the insight that large language models (LLMs) trained to generate code can vastly improve the effectiveness of mutation operators applied to programs in genetic programming (GP). Because such LLMs benefit from training data that includes...

Topics: Models Infrastructure

Entities: Models Infrastructure

OpenAI News June 09, 2022 07:00

Techniques for training large neural networks

Large neural networks are at the core of many recent advances in AI, but training them is a difficult engineering and research challenge which requires orchestrating a cluster of GPUs to perform a single synchronized calculation.

Topics: Infrastructure

Entities: Infrastructure

OpenAI News January 24, 2022 08:00

Text and code embeddings by contrastive pre-training

Text embeddings are useful features in many applications such as semantic search and computing text similarity. Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture....

Topics: Models Infrastructure

Entities: Models Infrastructure

OpenAI News September 08, 2021 07:00

TruthfulQA: Measuring how models mimic human falsehoods

We propose a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. We crafted questions that some humans...

Topics: Models Infrastructure

Entities: Models Infrastructure

OpenAI News July 28, 2021 07:00

Introducing Triton: Open-source GPU programming for neural networks

We’re releasing Triton 1.0, an open-source Python-like programming language which enables researchers with no CUDA experience to write highly efficient GPU code—most of the time on par with what an expert would be able to produce.

Topics: Infrastructure

Entities: Infrastructure

OpenAI News June 10, 2021 07:00

Improving language model behavior by training on a curated dataset

Our latest research finds we can improve language model behavior with respect to specific behavioral values by fine-tuning on a small, curated dataset.

Topics: Infrastructure

Entities: Infrastructure

OpenAI News January 25, 2021 08:00

Scaling Kubernetes to 7,500 nodes

We’ve scaled Kubernetes clusters to 7,500 nodes, producing a scalable infrastructure for large models like GPT-3, CLIP, and DALL·E, but also for rapid small-scale iterative research such as Scaling Laws for Neural Language Models.

Topics: Models Infrastructure

Entities: Models Infrastructure

OpenAI News May 28, 2020 07:00

Language models are few-shot learners

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific...

Topics: Models Infrastructure

Entities: Models Infrastructure

OpenAI News January 23, 2020 08:00

Scaling laws for neural language models

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of...

Topics: Models Infrastructure

Entities: Models Infrastructure

OpenAI News December 13, 2019 08:00

Dota 2 with large scale deep reinforcement learning

On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as long time horizons, imperfect information, and complex, continuous state-action...

Topics: Models Infrastructure

Entities: OpenAI Models Infrastructure