Executive Summary
Intel and StreamNative collaborated to develop a new storage solution for mission-critical data. Together, they developed a plug-in that uses high-performance, low-latency Intel® Optane™ persistent memory (PMem) as the storage medium that enables Apache Pulsar’s messaging and event-streaming platform to deliver improved system latency, message throughput, and overall performance while ensuring data persistence.
Apache Pulsar ensures data persistence by writing data entering the message queue to the disk in real time, instead of the memory or cache, preventing data loss even if the system is down. This data persistence and a decoupled storage design enable Pulsar to meet the needs of many mission-critical applications. However, Pulsar is just one piece of the puzzle. For high-scale, business-critical applications such as stock trading, futures trading, financial billing, crypto-currency trading, critical event logging, and state data synchronization, organizations need performance at 1 ms latency. If you are using Pulsar with a typical SSD, you may have issues delivering this latency above a certain scale.
Test results show that using Intel Optane PMem as Pulsar's journal disk for storage reduces system latency to less than half of the SSD, delivering a sub-millisecond latency, and improves the maximum throughput significantly.
Intel Optane PMem is well suited to deal with scenarios that require high performance, such as finance and trading applications.