LexisNexis Risk Solutions, a leader in offering information that helps customers across different verticals assess and manage risk, today announced the release of their big data platform built on top of High Performance Computing technology under an open core approach where the community edition will be available for free and with source code and an enterprise edition is available with the core open source platform, enterprise class support and advanced modules needed for enterprise use. Called HPCC Systems, full binaries for the community edition will be released in the coming weeks and source code will be made available in the weeks following the binary release. However, LexisNexis Risk Solutions will not release its data sources, data products, the unique data linking technology, or any of the linking applications that are built into its products. These assets will remain proprietary and will not be released as open source.
HPCC Systems can process, analyze, and find links and associations in high volumes of complex data significantly faster and more accurately than current technology systems. The platform scales linearly from tens to thousands of nodes handling petabytes of data and supporting millions of transactions per minute. HPCC Systems is comprised of a single architecture, a consistent data-centric programming language, and two processing platforms: the Thor Data Refinery Cluster and the Roxie Rapid Data Delivery Cluster.
Whether we like it or not, we are living in the big data world and Hadoop is a leading player in this space. LexisNexis has been working with big data for a long time and their technology is battle proven and enterprise grade. There is a great market opportunity for them here but the only way they can get some traction is by reducing the barrier to adoption and by gaining developer interest. The time tested way to achieve these goals in by releasing the platform under open source. By releasing HPCC Systems as open source, they are hoping that the platform will emerge as an alternative to Hadoop. In fact, as highlighted in the interview with Derrick Harris of GigaOm, they are even hoping that their platform will emerge as a Hadoop killer because the Enterprise Control Language driving HPCC abstracts many complexities of MapReduce and runs much faster than Java based Hadoop. The ECL specs are released under creative commons license and this allows anyone to improve the language over time.
Even though HPCC Systems claim this to be much superior to Hadoop, we cannot write off Hadoop so fast. The Hadoop ecosystem is very large and there are many interesting players in the ecosystem helping to improve Hadoop. Having a solid alternative to the market leader is important any day and this will foster competition, ultimately benefitting the users in a big way. It will be interesting to see who captures the market lead as our data production increases significantly in the coming years.
Related articles
- LexisNexis open sources Hadoop challenger (go.theregister.com)
- LexisNexis open sources its Hadoop killer (gigaom.com)
- LexisNexis Will Open-Source Its Hadoop Alternative for Handling Big Data (readwriteweb.com)
- Hadoop, Hadoop, and More Hadoop (fakeiitian.com)
- Product Review: Informatica Addresses The Impending Big Data Challenge With Release 9.1 (enterpriseirregulars.com)
Open Core –> Open Source –> Vendor Lock-in?
Seeking developer interest, in this case, may also be seeking to draw developers into a proprietary trap. As this article describes it, LexisNexis is going to offer the two tier setup of a community edition and a proprietary product in which the proprietary extensions are critical to the full use of the system.
Algot Runeman has this exactly right. Open Core almost always represents a huge danger to real open usability of a working end product unless users pay up for the use of proprietary components and thus it is worthy of great suspicion, at best.
Hadoop is offered under an Apache license and has an active and diverse developer network behind it already; LexisNexis offering, in its current form, does not seem an attractive alternative.
I fully agree with both of you. Open core has the danger of locking in developers. I found it interesting because this move by LexisNexis is an acknowledgement that Hadoop is a leader in the space.
Interesting. I thought (was listening to a webinar) LexisNexis was using MarkLogic software for their Bigdata processing requirements.