The Wayback Machine - https://web.archive.org/web/20160315024506/http://apache.sys-con.com/node/3485180

Click here to close now.



Welcome!

Apache Authors: Talend Inc., Adrian Bridgwater, Pat Romanski, Jim Scott, Jnan Dash

Related Topics: @DevOpsSummit, IBM Cloud, Apache

@DevOpsSummit: Blog Feed Post

IBM’s Big Commitment to Apache Spark | @CloudExpo #DevOps #Microservices

It will offer Apache Spark as a service on Bluemix

Last June IBM made a serious commitment to the future of Apache Spark with a series of initiatives:

  • It will offer Apache Spark as a service on Bluemix (Bluemix is an implementation of IBM's Open Cloud Architecture based on Cloud Foundry, an open source Platform as a Service (PaaS). Bluemix delivers enterprise-level services that can easily integrate with your cloud applications without you needing to know how to install or configure them.
  • It committed to include 3500 researchers to work on Spark-related projects.
  • It will donate IBM SystemML (its machine learning language and libraries) to Apache Spark open source

The question is why this move by IBM?

First let us look at what is Apache Spark? Developed at UC Berkeley's AMPLab, Spark gives us a comprehensive, unified framework to manage big data processing requirements with a variety of data sets that are diverse in nature (text data, graph data etc) as well as the source of data (batch v. real-time streaming data). Spark enables applications in Hadoop clusters to run up to 100 times faster in memory and 10 times faster even when running on disk. In addition to Map and Reduce operations, it supports SQL queries, streaming data, machine learning and graph data processing. Developers can use these capabilities stand-alone or combine them to run in a single data pipeline use case. In other words, Spark is the next-generation of Hadoop (came with its batch pedigree and high latency).

With other solutions for real-time analytics via in-memory processing such as RethinkDB, an ambitious Redis project or commercial in-memory SAP Hana, IBM needed a competitive offering. Other vendors betting on Spark range from Amazon to Zoomdata. IBM will run its own analytics software on top of Spark, including SystemML for machine learning, SPSS, and IBM Streams.

At this week's Strata conference, several companies like Uber described how they have deployed Spark all the way for speedy real-time analytics. 

More Stories By Jnan Dash

Jnan Dash is Senior Advisor at EZShield Inc., Advisor at ScaleDB and Board Member at Compassites Software Solutions. He has lived in Silicon Valley since 1979. Formerly he was the Chief Strategy Officer (Consulting) at Curl Inc., before which he spent ten years at Oracle Corporation and was the Group Vice President, Systems Architecture and Technology till 2002. He was responsible for setting Oracle's core database and application server product directions and interacted with customers worldwide in translating future needs to product plans. Before that he spent 16 years at IBM. He blogs at http://jnandash.ulitzer.com.

@ThingsExpo Stories
transform operational efficiency and safety for businesses and communities, especially during critical situations. During these critical events, man-made incidents or natural disasters, identifying and reaching employees with reliable and automated communications can not only protect business assets, but can be the difference between life and death. In his session at @ThingsExpo, Imad Mouline, chief technology officer for Everbridge, will highlight incident communications best practices and ...
SYS-CON Events announced today Object Management Group® has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
For basic one-to-one voice or video calling solutions, WebRTC has proven to be a very powerful technology. Although WebRTC’s core functionality is to provide secure, real-time p2p media streaming, leveraging native platform features and server-side components brings up new communication capabilities for web and native mobile applications, allowing for advanced multi-user use cases such as video broadcasting, conferencing, and media recording.
*This is part of a series of blogs examining Sensor-2-Server (S2S) communications, development, security and implementation. For the past two weeks, we’ve taken an in-depth look at what Sensor-2-Server communications are, how to implement these systems, and some of the specific aspects of communication that these systems facilitate. This week, for our final installment, we’ll examine some of the benefits, as well as security considerations, for S2S communications.
Big Data, cloud, analytics, contextual information, wearable tech, sensors, mobility, and WebRTC: together, these advances have created a perfect storm of technologies that are disrupting and transforming classic communications models and ecosystems. In his session at @ThingsExpo, Erik Perotti, Senior Manager of New Ventures on Plantronics’ Innovation team, will provide an overview of this technological shift, including associated business and consumer communications impacts, and opportunities...
SYS-CON Events announced today that iDevices®, the preeminent brand in the connected home industry, will exhibit at SYS-CON's 18th International Cloud Expo® | @ThingsExpo, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. iDevices has announced that it has entered into a definitive agreement to sell the world’s #1 app-enabled grilling and cooking thermometer brands, namely iGrill® and Kitchen Thermometer, to Weber-Stephens Products, LLC. Weber®, the world’s lar...
Exosite has announced its Taipei Office has formally been established. Exosite’s Taiwan operations have grown from the Taichung team, primarily focused on research and development, into a full-service IoT enablement organization with development and consulting capabilities. The new Taipei office is located in the Taipei Minsheng Dunhua district, surrounded by all its hardware and channel partners. Exosite will devote its time to the Taiwan market and expand its business into the Asia-Pacific mar...
SYS-CON Events announced today CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. CrowdReviews.com is the first buyer’s guide that ranks products and services based on client reviews.
SYS-CON Events announced today BZ Media LLC has been named “Media Sponsor” of SYS-CON's 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. BZ Media LLC is a high-tech media company that produces technical conferences and expositions, and publishes a magazine, newsletters and websites in the software development, SharePoint, mobile development and Commercial Drone markets.
As devices, sensors, objects and people are given digital identities that connect them to the Internet by the billions, the need for security and privacy becomes a critical factor for both market adoption and safety. The 40-year-old security methods we now use on our PCs and networks cannot address many of these IoT devices.
SYS-CON Events announced today TMCnet has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Technology Marketing Corporation (TMC) is the world's leading business-to-business and integrated marketing media company, servicing niche markets within the com...
SYS-CON Events announced today that Pythian, a global IT services company specializing in helping companies adopt disruptive technologies to optimize revenue-generating systems, has been named "Bronze Sponsor" of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2015 at the Javits Center in New York, New York. Pythian, a 400-person global IT services company that helps companies compete by adopting disruptive technologies, announced today that CEO Paul Vallée has been named “Diversi...
Internet of Things (IoT) platforms have evolved. Advances in technology make it possible for facilities to become more data-driven by connecting to energy assets, thereby improving building resiliency and reducing energy costs. By creating an Energy Network of Things, facilities can manage energy assets in real-time, institute system-wide operational oversight and compliance, execute energy efficiency programs, utilize real-time data, and support budgeting and asset planning across local and geo...
Two-thirds of organizations implementing hybrid cloud report they’re already gaining competitive advantage from their hybrid environments and are nearly three times as likely to use it to assemble data assets or monetize data, according to findings released by IBM (NYSE: IBM). With a hybrid cloud approach, organizations can be selective about when to use cloud and when to use traditional IT infrastructure – delivering the best functionality while meeting speed and flexibility needs, as well a...
The emerging Internet of Everything creates tremendous new opportunities for customer engagement and business model innovation. However, enterprises must overcome a number of critical challenges to bring these new solutions to market. In his session at @ThingsExpo, Michael Martin, CTO/CIO at nfrastructure, will outline these key challenges and recommend approaches for overcoming them to achieve speed and agility in the design, development and implementation of Internet of Everything solutions ...
In his session at @ThingsExpo, Noah Harlan, Founder of Two Bulls and President of AllSeen Alliance, will discuss the coming move from Cloud to Edge and what this means for business. Noah Harlan is President of AllSeen Alliance and a Founder of Two Bulls, a leading mobile software development company with offices in New York, Berlin, and Melbourne. He is also Managing Director of Digital Strategy for Sullivan NYC, a brand engagement firm based in New York. He has served as an advisor for the Whi...
Connected devices and the industrial internet are growing exponentially every year with Cisco expecting 50 billion devices to be in operation by 2020. In this period of growth, location-based insights are becoming invaluable to many businesses as they adopt new connected technologies. Knowing when and where these devices connect from is critical for a number of scenarios in supply chain management, disaster management, emergency response, M2M, location marketing and more. In his session at @Th...
There are over 120 breakout sessions in all, with Keynotes, General Sessions, and Power Panels adding to three days of incredibly rich presentations and content. Join @ThingsExpo conference chair Roger Strukhoff (@IoT2040), June 7-9, 2016 in New York City, for three days of intense 'Internet of Things' discussion and focus, including Big Data's indespensable role in IoT, Smart Grids and Industrial Internet of Things, Wearables and Consumer IoT, as well as (new) IoT's use in Vertical Markets.
Internet of @ThingsExpo, taking place June 7-9, 2016 at Javits Center, New York City and Nov 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with the 18th International @CloudExpo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo New York Call for Papers is now open.
SYS-CON Events announced today that Kintone has been named "Bronze Sponsor" of SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. kintone promotes cloud-based workgroup productivity, transparency and profitability with a seamless collaboration space, build your own business application (BYOA) platform, and workflow automation system.