Hive is not a good store as a backend for a BI product, since it offers no caching at all.  However, a workflow where you crunch data in Hadoop/Hive and then export to a MySQL table (or an Endeca instance) for use in a BI tool works very well.

My most recent discovery is Good Data. They’re Cambridge-based and offer outsourced BI. They have pulled down funding from Esther Dyson, Tim O’Reilly and others and have a small war chest to try out their ideas. Three areas of market reaction that I’m particularly curious about:

1. Are companies willing to let their most valuable data out of their doors? Clearly, they have been open to it in particular areas of their business as they have moved CRM, web analytics for their retail sites, and marketing analytics off-site. However, in this case, they’re potentially moving the entire BI stack out. Would they be willing to move their financial data out?

   Good Data’s success does not depend on whether or not companies choose to ship out their most private data. There’s plenty of pain around BI in organizations that doesn’t involve sensitive data. However, it will be interesting to see corporate attitudes evolve as people get more used to sending data off-site for analysis.

2. Is Good Data able to improve the user experience around business intelligence? When we were bringing the Endeca business intelligence offering to market, the two frustrations that we addressed for our users were:
   • IT often takes a long time to turn reports around
   • The reports that they do provide are static and the tools and UI to manipulate them are really only useful to the analytics “high priests”

   The former is something Good Data can address just by providing infrastructure online; the latter is much harder to do whether you are an outsourced provider or you are in house. It often requires participation of the business users and requires mapping the analysis to the business processes involved. Metrics and analytics do not mean much unless you understand what the numbers are telling you!

3. Can Good Data (and others in this category) truly provide outsourced BI without having a significant services component to their business? This question follows from the previous comment: that analytics becomes much more useful once you have some business context and know how to encode the business context into numbers and interpret the resulting analysis.

   Furthermore, I have always believed that to truly embed BI in the organization requires moving decision-making to being part of each business process as opposed to an after-the-fact activity. Such an approach often requires a deep services component that cannot be standardized easily.

Which brings me to my final thought: Will the very existence of these tools and the pre-defined templates that they come with result in some standardization of business process across organizations? Has anyone modeled their sales process around what Salesforce.com provides out of the box? Will that lead to better efficiency? Why should this happen now when it didn’t happen during the ERP days? Companies spent 10′s of millions to change the software rather than change their business processes. Presumably it’s because a company’s business processes are one of its core sources of value.

In addition to these tools, one can also use scripting languages like Perl to manipulate and modify text or XSLT to manipulate and modify XML content.

There are some interesting commonalities and differences between these categories of data processing tools. Over the next couple of posts, I will talk about each of these classes of tools in more depth, including sample use cases for each, the data structures that they use (and hence what types of data they can easily handle), and what lessons one might be able to use from each if one were going to build a new data transformation tool. Note that this is by no means an exhaustive set of these kinds of tools, but rather a small collection of those that I have used and know a little bit about.