The 500 Petabyte Architecture: How OpenAI Scaled Storage for 1 Billion ChatGPT Users

OpenAI has surpassed the milestone of 1 billion users served by ChatGPT and its API infrastructure, requiring an extreme overhaul of how the company manages data. On October 23, 2026, engineers Jon Lee, Hrachya Hakobyan, and Chaomin Yu detailed "Part II" of the technical evolution of Habitat, OpenAI's proprietary storage platform. The system has evolved from a Python client library into a distributed colossus spanning nearly 40 geographic regions, processing more than 70 million requests per second and managing a data volume exceeding 500 petabytes. The core of this technical leap relied on pushing Microsoft's Azure Cosmos DB to its absolute limit, managing partition bottlenecks, and migrating from single-tenancy to managed multi-tenancy environments.

The Context and Timeline of Habitat's Evolution

OpenAI's data storage journey reflects the challenges of unprecedented hyper-scale. Before 2023, the company based much of its infrastructure on traditional relational systems, such as PostgreSQL. However, the rapid adoption of ChatGPT highlighted severe scalability limitations, especially regarding the management of "hot keys." When a single organization or client generated a disproportionate load of queries, the entire database suffered bottlenecks, requiring complex caching and traffic control solutions.

To solve these structural pain points, OpenAI initially introduced Habitat during DevDay 2023. At that time, the platform started relatively modestly, operating as a Python-based client-side library that connected to a single database. The product's success, however, demanded a rapid transition from a local library to an autonomous, globally distributed service, characterizing Phase I of this technical transformation, where OpenAI adopted standard Azure Cosmos DB containers.

In Phase I, the strategy involved directing traffic to a single standard Cosmos DB container and relying on the theoretical limit of 10,000 Request Units per second (RU/s) per logical partition key to generate back pressure. However, initial API decisions exposed the selection of these keys (such as user_id and workspace_id) directly to clients. This created collisions and bottlenecks, as multiple product features shared the same logical partition limits, strangling overall throughput.

Phase II marked the attempt at workload isolation. The team implemented dedicated (single-tenancy) containers for new and critical use cases, mitigating direct partition conflicts. Although this approach solved the isolation problem, individual capacity planning for each new container created significant waste and operational overhead, leading to the current status of refinement of multi-tenancy managed natively through Azure primitives.

The Numbers That Define OpenAI's Scale

The magnitude of the challenge solved by the Habitat platform is quantified by operational metrics that define frontier engineering in the AI sector. The operational data published in Part II of the technical report (October 23, 2026) reveals:

• Active Users: Over 1 billion users worldwide.
• Data Volume: Over 500 petabytes managed by Habitat.
• Request Throughput: Over 70 million requests processed every second.
• Global Presence: Operations distributed across nearly 40 distinct geographic regions.
• Infrastructure Limits: 10,000 RU/s (Request Units per second) stipulated per logical partition key in Azure Cosmos DB.
• Architecture Authorship: The "Part II" technical report is authored by engineers Jon Lee, Hrachya Hakobyan, and Chaomin Yu.

The "Physics" of Cloud Databases and Cosmos DB Challenges

The OpenAI document exposes the inevitable side effects when data volume clashes with the physical architecture of the cloud database. The team dealt with three fundamental obstacles that affect any distributed system of this size:

1. Logical Partition Key Limits: OpenAI exposed logical keys (e.g., user_id) to different competing product use cases. Because these workloads shared Azure's request limit thresholds on the same key, the design throttled end-user read and write speeds, even when the rest of the infrastructure had excess capacity.

2. The "Noisy Neighbors" Phenomenon: Although official limits (like the 10,000 RU/s) apply to logical partitions, actual execution occurs on physical server nodes. Multiple logical partitions inhabit the same physical partition. Thus, niche feature workloads consumed physical capacity, degrading essential operations, such as login and registration flows.

3. Weak Partitions and Passive Growth: Cosmos DB container provisioning is distributed relatively equally among physical partitions. When storage hits its ceiling, a physical partition splits in two, and each receives half of the original throughput capacity. In OpenAI's high-storage use cases, the simple growth of inactive data volume reduced the effective request limit of physical partitions, breaking active workloads even with stable user request rates.

To circumvent these scenarios, the final technical shift was not just to build isolation (Phase II), but to implement an advanced synchronization script between provisioned throughput and the number of partitions, in addition to returning to a multi-tenant model optimized through Azure primitives.

The Impact on the Sector and the Redefinition of Multi-Tenancy

Habitat’s architectural leap directly impacts four groups:

OpenAI itself: The technical gain ensures that the scalability of ChatGPT and Foundation Model APIs does not collapse under its own weight. They now achieve data reads up to 12 times faster, translating directly into lower Time to First Token (TTFT) and drastically reducing costs of over-provisioning on servers.

Infrastructure Engineers worldwide: OpenAI's approach exposes a paradigm in which "off-the-shelf" third-party tools, such as the documented limits of cloud providers, require deep proprietary engineering to function at hyper-scale. The trade-off between the operational cost of single-tenancy (waste due to over-provisioning) versus performance losses in multi-tenancy (noisy neighbors) is a direct case study for teams operating Kubernetes and non-relational databases.

Product Users and the Latin American Market: The Brazilian and Latin American tech market, composed of hundreds of thousands of developers who rely on the GPT API and end-users in the region (a significant portion of the global billion), benefits from resilience. Global latency across "nearly 40 regions" stabilizes critical integrations in Brazilian fintechs and startups, preventing peak oscillations in the United States from taking down regionally hosted instances.

Microsoft and the Azure Ecosystem: OpenAI is literally stressing Azure cloud infrastructure to levels rarely documented commercially. OpenAI's learnings inevitably pressure Microsoft to evolve the standard Cosmos DB architecture and commercialize these enhancements for other Fortune 500 clients in the future.

What Remains Unknown

The transparency of the technical report leaves out vital aspects of the operation that affect finances and corporate migrations. The points that still need clarification are:

• Exact Global Cache Mechanism: Although peripheral publications and third-party materials (as reported by NewXC and HotON.ai) mention that "global caching" was a key factor for the 12x read speed acceleration, the original report focuses mainly on the optimization of logical and physical partitions of Azure Cosmos DB. The expiration algorithms, global coherence, and specific Redis or Memcached technologies (if used) remain a black box.
• Financial Impact of Migration: The documentation points out that the single-tenant model was "expensive to operate," but it does not detail whether operational savings absorbed the costs of complex multi-tenancy routing engineering.
• Legacy Data Migration: The text cites that "many critical legacy cases were trapped" in the Phase I default container, requiring maneuvering for protection. It is unclear how OpenAI will orchestrate the real-time migration of petabytes of legacy data to new containers without downtime.

What to Expect Next

The next concrete step for the OpenAI infrastructure team involves the development of a definitive multi-tenancy ecosystem, which absorbs the raw load left by legacy systems in shared (default) containers.

The company is expected to fully automate intelligent query routing and improve monitoring of physical partition splitting in Cosmos DB. The strategy points toward deepening the use of primitives built into Azure's Cosmos DB to postpone the creation of overlapping proprietary tools as much as possible, keeping Habitat lean, even while serving millisecond-level responses to one-third of the digital population connected to the internet.

Frequently Asked Questions

What is OpenAI's Habitat platform? Habitat is OpenAI's proprietary online data storage platform, responsible for ensuring near-instant access to the data required for the operation of products such as ChatGPT and corporate APIs. It currently manages over 500 petabytes of data across nearly 40 regions.

What was the main bottleneck overcome by using Cosmos DB? The system dealt with limits of 10,000 Request Units per second (RU/s) on logical partitions, generating instability. Because multiple logical keys shared the same physical resource, overloaded clients (noisy neighbors) reduced database performance for other regular services.

What is the difference between single-tenancy and multi-tenancy implemented by the company? In single-tenancy, each product has its own database, which is secure but extremely expensive, resulting in capacity underutilization. In managed multi-tenancy, cloud resources are grouped, and OpenAI used built-in Azure primitives to control load and prevent services from negatively impacting each other's storage and reading, optimizing infrastructure and costs.