What’s Next for Data Analysis? Part I

I’m commonly asked this question in a variety of circles and on several continents. I consider it one of the more interesting aspects of my job to have these types of predictive, future-oriented conversations with a wide variety of people: customers, partners, technologists, journalists, analysts, consultants, entrepreneurs, financial analysts, researchers, and so on. So, I sat down recently to simply document my answer to the question “What’s Next for Data Analysis?” Here are my thoughts.

I believe the next few years will be an amplified continuation of what we've seen in the past few years, albeit with a severely web-focused, any-client device emphasis. Many of the patterns I’ll cite have been underway for some time; it’s just that the more simple and affordable ability to implement them has only recently caught up with the ambition.

Here goes. Next-generation data analysis will . . .

. . . embrace the rise and use of “big data” for nearly any size organization. I’ve long said that big, distributed data is not limited to big enterprise and many companies will find valuable insight because they can sift through it all to find new answers.

. . . require a more varied set of "analytics" - from responsive and unstructured to predictive and highly-structured. There are many ways to use data productively. Sometimes understanding the past is the most important point. Sometimes being able to predict the future, based on the patterns of the past, is required. The lines here will blur and more organizations will need both and many types in between.

. . . demand greater use of low latency analysis, which decreases the time between a business event occurring, its analysis, and a decision that is based on it. Sometimes this is referred to as “real-time BI” and other times “streaming analytics”. Let’s just call it useful and necessary in more environments each year.

. . . be untethered from traditional client computing network practices. I’ll repeatedly say “analytics anywhere”, which means purely web-based, open standards-aware, (multi-) thin client architecture. That’s a mouthful, but it needs to be the starting point for business intelligence system design (and this is where the mega-vendors’ mostly aged architectures fall short, btw). The most appropriate analytic experience should be deliverable to any device, because both the thin client or native mobile front-end application (where necessary) and the back-end (web) services understand and adapt. More users will expect this every year.

The combination of these clear patterns and trends is surely driving our product and technology roadmap at Jaspersoft. It’s also positively impacting our work with our far-flung community, where so many of our ideas become grounded. Internally, we talk about speeding the movement of “data to dashboard” so the user has the advantage of working with timely and relevant data, presented elegantly and in the proper context. This is our central thought. If you’re interested in learning more about the analytics landscape, from Jaspersoft’s perspective, we’ve published this white paper.

Next post, I’ll describe the recent BI and data management technologies that stand the most to gain as these shifts occur. In the meantime, let me know what you think.

Brian Gentile
Chief Executive Officer
Jaspersoft

Survey Results Point to the Future of Java and MySQL

I’ve been enthralled with Oracle’s acquisition of Sun since the day it was announced, because of what it represents to the foundation of open source software. My very first blog post on this matter, in April of last year, focused on the hearts and minds of the developer, in this case through two of Oracle’s most important acquired assets: Java and MySQL. My concluding remark was “stay tuned”.

Now we fast-forward more than one year. The deal is (finally) concluded and things are settling, right? Not taking anything for granted, I commissioned a survey of Jaspersoft’s substantial registered community members (numbering more than 130,000 at the time of the survey, April 2010, but now at 150,000). Because nearly 70% of Jaspersoft’s community is comprised of software developers (substantially using Java, MySQL and related tools), we felt our findings could be important. And, indeed, they did not disappoint.

We asked the Community about their planned future use for Java and MySQL versus competing technologies and products. We asked them about the expected pace of innovation for these two products and if they believe the community processes would be maintained. In short, we gained a capsule of information regarding the likelihood that these two foundational technologies would remain relevant, or become more so. Here’s a summary of our findings:

• 43 percent of respondents believe MySQL development and innovation will improve under Oracle;
• 80 percent of respondents believe the Java process will improve or stay the same;
• The most popular alternative to MySQL cited by respondents who said they planned to switch databases away from Oracle, is PostgreSQL. But only 5% indicated they would switch.
• 59 percent of survey respondents were not aware that Oracle reorganized and established a completely separate MySQL business unit from the Oracle’s traditional RDBMS business, which includes separate and dedicated engineering, professional services, sales and marketing.

So, it seems that not only is Oracle’s stewardship and ownership of these critical open source products safe for now, but it could lead to a resurgence for Java and even greater success for MySQL. We considered these findings surprising and hopefully good for open source software overall.

You can find our press release, announcing these findings here. And, you can find a complete copy of the survey results white paper here. You can imagine, given my investment of time on this topic, that I’m interested in your thoughts and comments.

Brian Gentile
Chief Executive Officer
Jaspersoft

Columnar Databases and the Future of Analytics

Delivering deeper data analysis, in formats more consistent with end-user expectations, has been a passion at Jaspersoft for several years. This passion will continue because making analytics, up to and including the sophisticated multi-dimensional type, more accessible and usable must be the goal of the BI provider who claims “pervasiveness” as its mantle. To help our community and customers know and influence where we’re going, we’ve published white papers that describe our view of the broader analytics landscape as well as one that describes the different types of analysis that our tools offer (e.g., in-memory analysis versus traditional, OLAP analysis).

Making data analysis a more common part of every business person’s day is fast becoming the province of both the database and the analytic tool. To advance the analytic agenda, columnar data storage engines, complete with speedy, in-memory techniques, help streamline query processing - delivering results in an instant and at a very low cost. Jaspersoft and its customers have been taking advantage of this trend growingly during the past two years. For example, in addition to analytic databases for enterprise data warehouses like GreenPlum, Jaspersoft has certified its products on columnar analytic databases including Vertica and Infobright, and we have many joint customers together. In many cases, the combination of in-memory-based analytics (within our BI tool) and the strengths and speed of the analytic database, obviate the need for building and maintaining not only star schemas in the data base but traditional OLAP cubes for the BI tool as well. Now that’s the beginning of analytics for everyone.

Interestingly, the pioneer in columnar technology for business intelligence usage, Sybase IQ, is one of the gems to be picked up by SAP. It recently announced its intention to acquire Sybase, and most speculation around the intentions have focused on Sybase’s treasure chest of mobile solutions. While I appreciate the mobile aspects, the combination of SAP’s Business Objects with Sybase IQ could be a formidable, albeit expensive and proprietary, competitor in the changing analytics landscape.

Furthering this analytic future solidly, Ingres last week announced the general availability of its VectorWise commercial analytic database that takes columnar technology even further. And, Ingres plans to provide an open source version before the year is out . Ingres describes VectorWise as next-generation analytic database technology as its announcement summarizes:

“Ingres VectorWise unlocks the power of modern commodity CPUs with a revolutionary database engine that leverages vector-based processing and on-chip memory to provide dramatic 10x - 70x performance gains over other databases.”

The performance claims are being validated by both customers and partners – and it just may be that Ingres has some groundbreaking new technology on its hands. With free downloads, trials, tools and documentation, Ingres is inviting the world to check it out. If it can use low-cost leadership along with super-fast processing to drive faster, interactive analytics and richer data visualization onto the laptops of more knowledge workers, I think Ingres VectorWise will have an important seat at the table of the future of analytics in the enterprise. Imagine that.

Brian Gentile
Chief Executive Officer
Jaspersoft

The Gulf Oil Spill should use an open source response

When BP CEO Tony Hayward first jumped into the spotlight in the days after the tragic explosion at the start of this massive leaking oil well catastrophe, he initially seemed reasoned, convicted and in control. As days passed and technique after technique failed to stem the flow of oil into the Gulf, the executive leadership and companies involved seem inept and even insufficiently skilled. At this point, there is rightly a great deal of blame being shared among the three companies involved and, indeed, within the Obama Administration. The net result of the spill will surely rank among the oil industry’s greatest environmental catastrophes.

So what could have been done differently during those first few days of the tragedy? What leadership methods might have been summoned to yield a different result? No surprise that I suggest an open source response.

What if on the third day after the explosion, Mr. Hayward called a news conference and explained the situation. It might have gone something like this...

“We currently estimate that more than 5,000 barrels of oil per day are leaking into the Gulf. Importantly, the source of the leak is, essentially, a broken pipe within our well, more than one mile below the surface of the ocean. At that depth there is almost no experience, even among the most tenured oil well and rig technicians, in capping and sealing a leak. We simply don’t know what we are up against. What we do have is extensive knowledge of the situation, tremendous and relevant experience capping leaking wells, and an enormous conviction to work quickly and tirelessly to prevent further environmental damage. So, as a team – BP, its affiliated partners and President Obama’s Administration – we’re calling on our global community to help. If you have ideas or expertise that might help us to more quickly implement a solution, we want to hear from you. No thought could be too big or too small. We’ll take ideas from any corner of the planet or any scientific discipline. And, we’ve set up a community web site to help receive and manage input. If you have ideas, I’m asking you to go to http://bp.savethegulf.org and let us know as soon as possible. Together, as a global community united to protect our environment, we can fix this tragedy more quickly and completely. Thank you.”

This leadership response has the benefit of authenticity (admitting vulnerability and limitation) and rallies support for a common cause rather than deflecting or trying to prove everything is all right (when it is clearly not). Most importantly, asking for global community involvement would almost assuredly yield ideas and possibilities to more quickly cap the flow of oil into this precious ecosystem. What could be more important?


Brian Gentile
Chief Executive Office
Jaspersoft

Green Business Intelligence and Overweight Dogs

One of the best parts about being CEO of Jaspersoft is that I get to meet with so many customers and community members. I always learn something interesting. One of my favorite examples of Jaspersoft in action is at Fat Spaniel. You have to love that company name, right? Let me tell you a little more about their products and services.

Founded in 2003, Fat Spaniel Technologies is the leading independent provider of critical software-as-a-service monitoring and reporting tools for the renewable energy industry. Today, the company’s solutions are used by more than 4,000 renewable power plants in 28 countries. Their customers include solar power producers, financiers, systems integrators, OEMs and a growing number of wind power companies. Fat Spaniel solutions help optimize energy output, reduce operations and maintenance costs, contain risk, and increase the return on investment for renewable energy power plant projects. And, they do this using Jaspersoft’s business intelligence suite as their embedded reporting and analysis tool.

Because reporting and analysis are so core to their solution, Fat Spaniel naturally went through a “make versus buy” decision process. It didn’t take them long to realize that partnering with Jaspersoft was the best decision. “We calculated the numbers and realized it didn’t pay off for us to build our own”, said Brett Francis, vice president of engineering and software architecture at Fat Spaniel. After performing what he refers to as “an exhaustive survey of open source BI solutions,” Brett chose Jaspersoft – and we couldn’t be more proud. Choosing Jaspersoft also made sense because Fat Spaniel is a self-proclaimed open source advocate, long committed to both Red Hat Enterprise Linux and Red Hat JBoss middleware.

The company name? The founder literally named the company after glancing at his overweight dog lying on the floor. Good thing he owned a Spaniel and not a Chihuahua.

Here’s a screen shot of the Fat Spaniel Insight Platform:

The Fat Spaniel renewable energy management solution is used by system installers to schedule service and optimize performance, and by owners to track and display renewable energy generation and usage – even performance-based incentives that allow emerging carbon and renewable energy credits. I’m thrilled that, through Fat Spaniel, Jaspersoft is right in the middle of the Green BI movement ☺.

The solution and services running on JBoss Enterprise Application Platform have connected Fat Spaniel’s Insight Platform with JasperServer v3, enabling custom source-data collection, a reporting function for product interaction, and multi-lingual/standard and custom distributed energy reports. Insight and analysis is provided against thousands of plant sites, hundreds of customers, and more than 200 MWs of renewable energy generation and millions of new packets of energy intelligence information daily. Pretty awesome.

If you want to learn more about their solution, check out the press release and the case study, both created jointly with Red Hat.

I’m eager to ensure that every company developing modern software will choose Jaspersoft as its embedded BI platform. Being involved with customers like Fat Spaniel (and partners like Red Hat) reminds me why this job can be so much fun.

Brian Gentile
Chief Executive Officer
Jaspersoft

Visualizing The Data of Our World

Jaspersoft is part of a community of millions of people spread out across many cities and many countries. It occurred to me that looking at our own community and applying our own software to visualize that data could be an interesting way to illustrate just how easy, useful and compelling great BI software can be.

I invite you to send me screenshots of your own community (or other geographically-relevant data) after you try some of the things I describe below.

To “see” our Jaspersoft community, we first began looking at Google Earth to help us visualize the dispersed information we had. Our curiosity and technical development with JasperReports helped spawn the free Google Earth Visualization Project, which in itself is a compelling example of how to leverage the flexibility of an open source tool such as JasperReports to visualize data in new ways. My thanks go out to Lucian Chirita of Jaspersoft Romania who initiated this project. I encourage those who want to explore and contribute to join.

With the Google Earth Visualization package and JasperReports, people can read in address data from Excel spreadsheets or a database and export Google Earth data files in KML or KMZ format, similar to how JasperReports can export PDF, XML, HTML or XLS. JasperReports works with almost any data source, including JDBC-wrapped data for relational databases, JavaBeans (EJB, Hibernate), plain old Java objects (POJO), and XML.

The data can either be processed as a batch data job with a popular and free tool such as batchgeocode or processed as an automated web services job that outputs the KMZ file. Even if you are not a developer, both methods are fun to play around with. However, the web services method is the more compelling and interesting way to go for the more technically savvy.
The Jaspersoft Google Earth Visualization package comes with examples and some sample data. Download it and give it a try! Click here for more information on how to download the source code and example for the package.

Here’s a quick example of the locations of where our JasperServer customer community in Europe are based:
Jaspersoft isn’t alone in offering cool new tools to democratize the ability for all of us to visualize the data in our lives. Google recently announced an interesting lab project called Google Public Data Explorer. This is a quick and simple way to apply some of their SaaS tools on publicly-available data Google stores on its cloud. The resulting output can be line charts, bar charts, maps or bubble charts. The downside is that you can only use the data Google has and you don’t have the flexibility and power of Jaspersoft’s tools. Still, it’s another start in the right direction.

The visualization of Jaspersoft’s global community showcases just one example of the flexibility of open source to input and output data in a wide range of formats. Our goal is to enable people to visualize data in new ways and bring new meaning or insight to everyday information.

Visualizing data also helps us all better understand community. Here, community can be used as a metaphor for any business environment, marketplace or passion. In Jaspersoft’s case, our community is the global population of developers and customers using our software. Visualizing our community data helps us focus resources where developers in a specific geographies might be doing particularly inventive things that we want to sponsor or encourage. Commercially, it helps us focus more sales attention in specific countries or regions showing the most promise.

You get the idea.

I invite you to send in your own examples of data you’ve visualized. I’ll share cool screenshots with everyone in a later posting. I’ll also call for screen shot submissions on our JasperReports forum. Then maybe we can run a contest allowing you, the community, to vote on your choice of the best “Visualizing Your World” screen shot examples.

Brian Gentile
Chief Executive Officer
Jaspersoft

Jaspersoft 3.7: Audit Logs, Search and Help!

Just last week, we made available Jaspersoft 3.7.0.1 – the version that delivers all the functionality of our latest release, now running on an even wider range of the most popular software platforms such as WebSphere 7, WebLogic 11g, SpringSource tc Server 6.0, Firefox 3.6, and Windows 7. So, to conclude my feature series on Jaspersoft 3.7 and our new Enterprise Edition, I will summarize just a few of the remaining core advancements, highlighting how customer and community feedback helped us drive these in the right direction. In previous posts, I’ve covered Jaspersoft 3.7’s more advanced in-memory analysis and its enhanced visualization (via Flash). Now I’ll briefly describe our new Audit Log, Repository Search and on-line, context-sensitive Help System.

The Audit Log was conceived for and with some of our most active customers, specifically to help them monitor, report, and analyze all events occurring in JasperServer such as running reports and analyses, and adding and updating users. Technically, we added an extensible audit-logging framework into the server along with a pre-populated set of audit objects including an audit log data source, Domain, and set of pre-built reports. Using these, any variety of reports and dashboards can be constructed (drag-and-drop) to create a customized view of what is happening within JasperServer, for usage monitoring, performance tuning, and compliance. Here’s a sample screen of that domain and some pre-built reports.

Next, the Repository Search feature adds much-needed functionality especially for our enterprise customers. Greater data volumes and more varied data types usually yield a repository full of folders, reports and myriad data objects. Navigating only a hierarchical folder system becomes unwieldy and a Search function is required. Our new integrated Search capability allows for free-form searching of any metadata maintained in the JasperServer repository. This means you can find a report by date, name, type and so on . . . very useful and time-saving. Here’s a screen shot showing the search box and a sample result.

Lastly, Jaspersoft 3.7 includes an on-line, context-sensitive Help System that takes users directly to the most relevant documentation. We store this Help content in a centralized location, re-purposing in HTML content taken directly from the wildly popular “Ultimate Guides” series of print documentation. This makes for not only a fast but efficient Help solution. Our customers will appreciate the always up-to-date information and examples. Below, I’ve provided an example of the Help provided when viewing a dashboard.

Beyond these three capabilities, we’ve certified Jaspersoft 3.7 against some of the most popular analytic data stores, including Infobright, Vertica and GreenPlum – as well as JBoss Data Services (formerly MetaMatrix). There are so many new features and capabilities in Jaspersoft 3.7 that I could just keep on writing. But, I won’t. Instead, I’ll suggest that you check it out yourself by viewing a brief demo and downloading our latest release highlights. Your comments and questions are encouraged.

Brian Gentile
Chief Executive Officer
Jaspersoft