IBM's New Architecture Boosts Big Data Processing Speeds

At the Supercomputing 2010 conference, IBM announced a new storage architectural design, developed by IBM scientists, which can drastically boost the speed of processing terrabytes of information into actionable insights. According to IBM, they could double the speed of processing without requiring heavy infrastructure investment. One could easily shave off hours from a complex computational task with this new architecture.

The Problem

Today’s organizations create vast amounts of data on a daily basis but they cannot increase the size of storage indefinitely. They soon run into walls unable to keep vast amounts of data. To make sense of these vast amounts of data, organizations have to run advanced analytics on extremely large datasets to get actionable business insights. In order to do this, they need increasing processing power and an ability to access the data without running into performance problems.

IBM Solution

In order to solve this problem, IBM scientists had developed a new way of storage partitioning giving businesses faster time-to-insight without concern for traditional storage limitations. Called General Parallel File System-Shared Nothing Cluster (GPFS-SNC) architecture, it is designed to provide higher availability through advanced clustering technologies, dynamic file system management and advanced data replication techniques. By “sharing nothing,” new levels of availability, performance and scaling are achievable. It is a distributed computing architecture in which each node is self-sufficient; tasks are then divided up between these independent computers and no one waits on the other.

An example quoted by IBM to explain the importance of this new architecture is large financial organizations running complex algorithms to analyze the risk based on petabytes of data. With millions of bytes residing on computing platforms spread out around the world, these mission critical calculations require significant investments on IT. With IBM’s new GPFS-SNC design, running complex workloads became more efficient because the new architecture provides a common filesystem and namespace across multiple disparate platforms.

Conclusion

As big data becomes all pervasive, there is definitely a need for more optimized architectural designs that will not just make storage efficient but also make analytics get higher performance gains. IBMs new architecture is aimed to solve this problem.