From PMEM to XRMEM in Exadata

How Exadata Evolved from Persistent Memory  PMEM to a Modern Memory Architecture – XRMEM

Oracle Exadata has always focused on one goal: moving data faster between storage and the database.

XRMEM Exadata represents a major leap in Oracle Exadata architecture, evolving from traditional PMEM-based designs to RDMA-backed memory access.

The original problem Oracle Exadata wanted to solve:-

Traditional database I/O looks like this:

Storage → OS Kernel → TCP/IP → CPU → Database

This path causes:

• High latency
• CPU overhead
• Multiple memory copies

Even fast flash storage became a bottleneck for:

• High-concurrency OLTP
• RAC workloads
• Commit-heavy systems

Exadata needed data closer to the database and faster than flash.

Step 1: PMEM (Exadata X8M / X9M)

What was PMEM?

PMEM (Persistent Memory) was a special type of memory installed in Exadata storage servers.

PMEM was:

• Non-volatile (data survives power loss)
• Much faster than flash
• Slightly slower than DRAM

How PMEM was used

PMEM acted as a very fast storage layer:

• Hot data blocks were cached in PMEM
• Redo was written to PMEM for faster commits
• RDMA was used to access PMEM quickly

This gave Exadata microsecond-level I/O latency for the first time.

Why PMEM was not the final solution

PMEM worked very well, but it had drawbacks:

• Specialized hardware
• Limited vendor ecosystem
• Memory technology evolves quickly

Oracle realized something important:

Performance should not depend on one specific memory technology.

So instead of building Exadata around PMEM forever, Oracle focused on architecture.

Read more about PMEM on – https://dbatrainer.com/pmem-and-rdma-in-oracle-exadata-x8m-and-x9m/

Step 2: Introduction of XRMEM (the architectural shift)

XRMEM Exadata Architecture Overview

XRMEM (Exadata RDMA Memory) is not hardware.

XRMEM is a software-defined memory architecture that uses RDMA to allow the database server to directly access memory on storage servers with ultra-low latency.

Key idea:

• XRMEM defines how memory is accessed
• It does not care what type of memory is used

How PMEM and XRMEM are related:-

This is the most important clarification.

In Exadata X8M and X9M:

• XRMEM used PMEM as its backend

So people often thought:

PMEM = XRMEM

But that was never architecturally true.

Step 3: Formal naming change (System Software 23.1)

Oracle made the architecture explicit:

Why this mattered:

• PMEM was positioned as one implementation
• XRMEM was positioned as the long-term design
• Backward compatibility was preserved

Step 4: PMEM removed, DRAM adopted (Exadata X10M and later)

What changed in X10M?

• PMEM hardware was removed
• XRMEM remained
• XRMEM now uses high-performance DDR5 DRAM

Is DRAM volatile?

Yes , DRAM is volatile, unlike PMEM.

This leads to the common question:

Does this break data safety?

Why volatile DRAM does NOT cause data loss:-

XRMEM is used as:

• A cache
• A performance acceleration layer

It is not the system of record.

Data durability is still guaranteed by:

• Redo logs
• Flash storage
• Disk storage
• Oracle Database recovery mechanisms

If a storage server restarts:

• XRMEM cache is rebuilt automatically
• No committed data is lost
• No DBA action is required

XRMEM improves speed, not durability. Durability is handled elsewhere

XRMEM today:-

What XRMEM does

• Caches the hottest OLTP data
• Accelerates small block reads
• Reduces latency to ~14–17 microseconds
• Uses RDMA to bypass OS and network stacks

What XRMEM does NOT do

• It does not replace redo logs
• It does not store permanent data
• It does not change Oracle Database behavior

PMEM → XRMEM evolution timeline:-

The architecture stayed.

The memory technology changed.

Why Oracle moved from PMEM to DRAM:-

Oracle chose DRAM because it is:

• Faster
• More scalable
• Widely available
• Cost-effective
• Easier to evolve

By combining:

• DRAM
• RDMA
• XRMEM software intelligence

Exadata achieved better performance than PMEM, without hardware dependency

Final simple explanation:-

PMEM was a special memory used in early Exadata systems.XRMEM is the software architecture that made PMEM useful.

Today, XRMEM uses DRAM instead of PMEM, but delivers the same or better performance while keeping data safe.