PMEM and RDMA in Oracle Exadata X8M and X9M

How Persistent Memory and RDMA Redefined Database I/O Performance

Modern Oracle Exadata systems introduced a major shift in database I/O architecture with the use of Persistent Memory (PMEM) and Remote Direct Memory Access (RDMA).

Together, these technologies significantly reduced I/O latency, CPU overhead, and data movement between storage and database servers.

This blog explains what PMEM and RDMA are, why they were introduced, and how they work together in Exadata.

Why traditional storage was no longer enough..?

Before Exadata X8M, even with NVMe flash, database I/O followed this path:

Storage → OS Kernel → TCP/IP → CPU → Database

This introduced:

• • Multiple memory copies

• • Kernel context switches

• • High CPU usage

• • Network stack latency

As databases became:

• • More latency-sensitive (OLTP)

• • Highly concurrent

• • Mixed with analytics

Flash alone could not deliver consistent microsecond performance.

This led Oracle to introduce PMEM for speed and RDMA for transport.

What is PMEM (Persistent Memory)?

Persistent Memory (PMEM) is a storage-class memory technology that combines the speed of memory with the persistence of storage.

Key characteristics of PMEM

• • Non-volatile (data survives power loss)

• • Much faster than NVMe flash

• • Slower than DRAM, but close

• • Byte-addressable

In Exadata, PMEM is installed inside storage servers and is managed entirely by Exadata software.

How PMEM is used in Exadata.?

PMEM acts as a new tier in the storage hierarchy:

Data Temperature	Storage Tier
Hot	PMEM
Warm	NVMe Flash
Cold	Disk

PMEM use cases in Exadata

• • Caching frequently accessed data blocks

• • Accelerating Smart Scan reads

• • Speeding up redo log commits

PMEM allows Exadata to keep hot data extremely close to the database, without sacrificing durability.

What is RDMA (Remote Direct Memory Access)?

• • Operating system kernel

• • TCP/IP stack

• • Remote CPU involvement

Exadata uses RoCE (RDMA over Converged Ethernet).

Key benefits of RDMA

• • Microsecond-level latency

• • Zero-copy data transfers

• • Very low CPU usage

• • Predictable performance under load

RDMA is not storage or memory—it is the transport mechanism.

Traditional I/O vs RDMA-based I/O

Traditional TCP/IP I/O

Storage → Kernel → CPU → Network → CPU → Kernel → Database

RDMA-based I/O

Storage Memory → Database Memory (Direct)

RDMA removes multiple layers from the data path, making memory access across servers almost as fast as local access.

How PMEM and RDMA work together in Exadata

PMEM and RDMA solve different problems, but complement each other perfectly.

• • PMEM provides a fast, persistent place to store data

• • RDMA provides the fastest possible way to move that data

Simplified architecture

PMEM (Storage Server)

   ↓ RDMA

Database Server Memory

With this design:

• • Data is read directly from PMEM

• • RDMA transfers it without CPU or kernel overhead

• • Latency drops to microseconds

PMEM + RDMA for read operations

For large scans and analytics:

1. 1. Data resides in PMEM

1. 1. Database server requests data

1. 1. RDMA transfers data directly from PMEM

1. 1. Only relevant rows and columns are returned

This dramatically reduces:

• • I/O volume

• • Database CPU usage

• • Query response time

PMEM + RDMA for write operations (commits)

For commit-heavy OLTP workloads:

1. 1. Redo is written to PMEM (non-volatile)

1. 1. Commit is acknowledged immediately

1. 1. Data is later flushed to flash or disk asynchronously

This results in:

• • Faster commits

• • Lower log file sync waits

• • No loss of durability

Why PMEM + RDMA was a breakthrough

Together, PMEM and RDMA delivered:

• • Up to 10x lower latency

• • Higher throughput

• • Better RAC scalability

• • More predictable performance

Most importantly:

Performance gains came without application or schema changes.

PMEM and RDMA scope in Exadata

• • PMEM is available in:
    

     

    • • Exadata X8M
    • • Exadata X9M

• • RDMA remains a core transport technology in Exadata

• • Both are fully integrated and automatically managed

DBAs do not need to:

• • Tune applications

• • Modify SQL

• • Manage memory tiers manually

Simple analogy

• • PMEM → A high-speed, persistent warehouse

• • RDMA → A frictionless express highway

• • Database server → The consumer of data

The faster the warehouse and highway, the faster the business runs.

One-line summary

PMEM provides fast, persistent storage close to the database, and RDMA deliver it with minimal latency and CPU overhead.

Final takeaway

PMEM and RDMA together marked a fundamental redesign of database I/O in Oracle Exadata.

Instead of pushing more data through traditional storage stacks, Exadata brought data:

• • Closer to the database

• • Faster than flash

• • With minimal overhead

This architecture laid the foundation for everything that followed in modern Exadata systems.