Hive is not a good store as a backend for a BI product, since it offers no caching at all.  However, a workflow where you crunch data in Hadoop/Hive and then export to a MySQL table (or an Endeca instance) for use in a BI tool works very well.

• An overload of choices make decision-making much more difficult since our pre-frontal cortex can only handle a small number of choices at a time (somewhere between 5 and 12)
• Several experiments suggest that emotion is a critical ingredient to enable fast decision making and prevent analysis paralysis.

Of course, Jonah Lehrer is not the first person to cover this topic. I remember reading Barry Schwartz’s Paradox of Choice a couple of years ago, and it included references to similar ideas.

What intrigues and concerns me is the implication of this research for business intelligence.  It would suggest that:

• Additional data / information can lead people (and hence companies) astray while they assume that more data is always better.  A lot of work has been done in this area.  I vaguely remember reading about such research in a Malcolm Gladwell-genre book.
• If people are making decisions based on data that may not be relevant, then we are, yet again, underestimating the role of randomness in the success or failure of companies (and CEO’s).  Someone else recently wrote about this, but I do not recall who.
• Emotion is a critical ingredient of important decision making; however, emotion can be significantly influenced by many other factors than data.  This suggestion in turn begs the question: just how important / relevant is the data?  Is it more often used for rationalizing emotional decisions than it is used for arriving at rational decisions?

While businesses will continue to move towards being run more and more by data, research like this heightens the importance of the non-technical parts of a good business intelligence implementation – understanding the data that you have available, truly understanding what you’re measuring, how it may impact the business and of course good data  visualization in the final implementation.

PS. The title of this blog post refers to the difficulty the author, Jonah Lehrer, has in picking cereal due to the vast number of choices available.

I find several of his assertions wrong and in fact contradictory to other statements that he makes in the same interview. My takeaway: this reads like a classic case of a company not wanting to cannibalize its current revenue streams.

He makes a fair point that much of SaaS hype has been catalyzed by the success of a single company, Salesforce.com. He also points out that SAP’s SaaS offering, Business ByDesign, was a failure in the market with fewer than a 100 users after a couple of years. However, I see two major problems with his assertions:

• Salesforce.com’s success has little to do with the fact that it’s been licensed under a SaaS model.
• The way to retain customers is via vendor lock-in and retaining customers in SaaS is much harder because switching costs are lower.

First, to his point that he and others would use Salesforce.com because the software is very good, not because it’s SaaS, says nothing about whether or not the SaaS licensing model has helped to increase the market penetration of Salesforce.com. Fundamentally, if I’m a customer inside an enterprise and Salesforce.com gives me a packaged server I can install locally and have my users hit my local server via their browser, the primary benefit I derive is that I have more control over the infrastructure, but correspondingly, the downside is that I have to pay to maintain it. From the customer’s standpoint, the major difference is not the user experience, but the maintenance cost and the license cost structure.

From the vendor’s standpoint, the major difference is not the software, how it’s deployed, or how capital must be deployed in order to build the software (“the more you sell, the more you lose” with SaaS is his assertion). The major difference between traditional enterprise software licenses and SaaS licenses is that with SaaS, the revenues are spread out over a longer time period. Two similar situations, related to the difference between SaaS and traditional enterprise licenses are:

1. Term licensing vs. perpetual licensing – one option when doing an enterprise sale is to give your customer a license to your software for a certain length of time e.g. a 3-year term and then require them to renew their license at the end of that period. Say you do a software deal for $300,000 for a 3-year term. Then, the customer would pay you $25,000 quarterly. According to Mr. Debes, this is a “more you sell, more you lose” kind of model as well. In practice, it works quite well – you have very predictable revenue recognition and cash flow – as long as you are not a start-up. With a start-up, you are investing significantly ahead of revenue and the term license defers revenue, stretching out your breakeven point further.
2. Seat-based licenses vs. all-you-can-eat licenses – When I was selling BI software, I noticed that almost all the other major BI vendors sold seat-based licenses. That is, if you want 2000 people to read a report, you need to buy 2000 “report reading” licenses. If you want 500 people to able to write or modify a report, then you pay 500 “report writing” licenses. We decided that if we were to truly offer our customers “BI for the masses” – that is, large scale adoption of BI – we had to change our license as well. We tried out a model of per-application or per-enterprise. The other vendors could not offer the same easily; their sales incentives and organization were not structured to handle a different license structure.

I would suggest that Lawson may not be able to switch its revenue and cost structure over to a radically different licensing model. Without knowing much more, I would hypothesize that SAP’s experience may have been related to something similar. I also don’t have faith that SAP actually built a compelling SaaS product in the first place.

To Mr. Debes’ second point – that you can only accomplish customer lock-in via enterprise software, I would agree that you can accomplish stronger customer lock-in via a traditional enterprise software deployment. Usually, enterprise software deployments involve IT approval processes, small fiefdoms being built in IT around the software package, and “expertise” in maintaining the software and hardware to keep the application going. SaaS deployments, at least for initial adoption, require none of those things. So yes, it’s far easier for a customer to try out a different SaaS deployment because the barrier to use is lower; however, is the traditional enterprise model truly a better user experience for the customer?

I would argue that Salesforce.com has just as much lock-in as any other “in-the-enterprise” CRM product. There’s no lock-in because of IT approval processes and fiefdoms; the lock-in comes from Salesforce.com being a great product and the fact that companies customize their business processes around their usage of the tool. Once you have companies molding their business process around your tool, you have the ultimate form of lock-in.

On a slightly-related note, check out YC’s list of ideas that they’d like to fund. Having spoken with many enterprise customers that deal with large IT organizations, I particularly like the idea described under the heading “Outsourced IT.”

My most recent discovery is Good Data. They’re Cambridge-based and offer outsourced BI. They have pulled down funding from Esther Dyson, Tim O’Reilly and others and have a small war chest to try out their ideas. Three areas of market reaction that I’m particularly curious about:

1. Are companies willing to let their most valuable data out of their doors? Clearly, they have been open to it in particular areas of their business as they have moved CRM, web analytics for their retail sites, and marketing analytics off-site. However, in this case, they’re potentially moving the entire BI stack out. Would they be willing to move their financial data out?

   Good Data’s success does not depend on whether or not companies choose to ship out their most private data. There’s plenty of pain around BI in organizations that doesn’t involve sensitive data. However, it will be interesting to see corporate attitudes evolve as people get more used to sending data off-site for analysis.

2. Is Good Data able to improve the user experience around business intelligence? When we were bringing the Endeca business intelligence offering to market, the two frustrations that we addressed for our users were:
   • IT often takes a long time to turn reports around
   • The reports that they do provide are static and the tools and UI to manipulate them are really only useful to the analytics “high priests”

   The former is something Good Data can address just by providing infrastructure online; the latter is much harder to do whether you are an outsourced provider or you are in house. It often requires participation of the business users and requires mapping the analysis to the business processes involved. Metrics and analytics do not mean much unless you understand what the numbers are telling you!

3. Can Good Data (and others in this category) truly provide outsourced BI without having a significant services component to their business? This question follows from the previous comment: that analytics becomes much more useful once you have some business context and know how to encode the business context into numbers and interpret the resulting analysis.

   Furthermore, I have always believed that to truly embed BI in the organization requires moving decision-making to being part of each business process as opposed to an after-the-fact activity. Such an approach often requires a deep services component that cannot be standardized easily.

Which brings me to my final thought: Will the very existence of these tools and the pre-defined templates that they come with result in some standardization of business process across organizations? Has anyone modeled their sales process around what Salesforce.com provides out of the box? Will that lead to better efficiency? Why should this happen now when it didn’t happen during the ERP days? Companies spent 10′s of millions to change the software rather than change their business processes. Presumably it’s because a company’s business processes are one of its core sources of value.