While evaluating (and now using) SEM platforms, I have been impressed by:

1) how well-suited the tools are for the day-to-day activities of the paid search marketer
2) the dramatic improvements in process and time-savings realized with having reporting and intelligence integrated with the ability to act

We are able to find outliers and immediately cut or increase spend. Tasks that previously used to take several hours can now be completed in minutes.

Demand-side platforms (DSPs) for display advertising are poised to deliver similar operational improvements to display marketing.

Abstracting up one level, I find these tools fascinating in that they’ve been able to productize (and consequently scale) the integration of BI and business processes. What is it about digital marketing that allows this approach to work, as opposed to say a tool built on top of Salesforce? The data sources are fairly homogenous (the three major search engines) and the data itself is even more so (campaign structure, bids, clicks, and CPC’s constitute the majority of the data).

The major difference between users is how they define conversion and revenue; however, even here, every marketer has a revenue number and a fairly standard set of conversion events (registration, revenue event, login, etc). You specify the definition of conversion and then upload one or more related metrics to support analysis and decision-making in the tool. Bidding is driven by revenue and cost and improving CTR and revenue/click allows marketers to focus on improving ad copy and landing pages.

Thus, in this case, limited data and a fairly homogenous cross-vertical definition of success metrics allows a scalable, tight integration of business process and business intelligence.

With regards to the platforms, you do start hitting limits once you start getting very sophisticated e.g. custom bidding algorithms, complex keyword expansion, etc. But even in this area, the vendors are increasingly offering solutions that are more sophisticated than what most in-house digital marketing and analytics operations will develop e.g. attribution models for bidding, automated SQR-driven keyword expansion, etc.

I’m looking forward to watching these products expand over the next year and simultaneously watching how digital marketing processes evolve.

In most cases, the vendors have taken a horizontal platform (whether Quickbooks or Salesforce) and developed a reporting / analytics offering on top of it. The real question for me goes back to some of the posts that I wrote when I started this blog – these plays are interesting, but they don’t get to the heart of the highest value problems:

• understanding a business (i.e. vertical) and offering BI / reporting that factors in specific challenges
• integrating BI as part of the business process of an organization (e.g. within Quickbooks usage or within Salesforce)

Some of the major players (e.g. Siebel Analytics) have offered a “matrix” (i.e. functional area x vertical) suite of applications for several years (e.g. sales for manufacturing or finance for retail). More accurately, at least they offered one several years back; however, it doesn’t seem like these got much traction in their target market as I don’t hear about them any more. I also haven’t seen other enterprise software vendors that sell to large companies universally adopt this packaging for their go-to-market. Presumably, they would have if the market were responding strongly.

- Summing the 5th column of numbers:

cat tmp.txt | awk '{t+=$5}END{print t}'

You may also find the above command useful for processing the output of ls -l

- cut, uniq, sort, grep, and wc are great for filtering and computing aggregates for a column of values e.g. for filtering a list of session ids in the 17th field, looking only for the duplicate ones:

cut -f17 | sort -n | uniq -c | grep -v ' 1 '

- And finally, a random aside that I learned about last week – bash treats the output of the following two commands differently:

echo $output

echo “$output”

If $output has no line breaks, then the output will be identical.  If it does have line breaks, then the former will flatten the multiple lines into a single line, while the latter will print out $output exactly as it was captured.

Added bonus that it easily integrates all the traditional data sources as well e.g. it’s easy to join a MySQL table against a “table” in Hadoop / Hive.  Would be cool if they could automatically discover the schema of your Hive tables and your traditional DB tables. They may already be able to; I didn’t see anything specific on their website indicating that though.

Tools like Datameer are a great addition to any BI practitioner’s toolset.  As datasets get larger, Hadoop allows ready access to large amounts of data at a reasonable price.  Datameer will now allow us to make that data more broadly accessible to a larger group of analysts.

1) a decent quality CLI (command line interface) for Hive.  Editing of a query in Hive is very limited. You can’t use custom keybindings – ideally, you’d want the CLI to get editing mode (emacs or vi) from your inputrc.  History and history search is poor. My workaround to date has been to instead use the shell’s command line and execute each query using the command line

hive -e "... query ..."'

Another benefit of using the Bash shell as your Hive CLI is that you can include variables in your SQL statement e.g. in my script, I can do something like:

TS=$1

hive -e "select * from foo where ts='$TS'"

2) Hive’s inability to map columns correctly.  I am unable to systematically reproduce this behavior, but when you do complex queries with multiple selects, Hive gets confused about the order of columns.  In particular, if the order of the columns in your hive query does not match the order of the columns on disk in the file backing the table, then you get lots of junk back.

3) Lack of documentation and confusing documentation.  The Hive Language Manual looks to be the definitive reference for Hive; however, it is missing quite a bit of documentation and mixes documentation from various versions of Hive, including the most recent, unreleased version, v0.6.0.

Gripes aside, Hadoop / Hive have been a great platform, and we’re continuing to expand our usage of them daily.

Hive is not a good store as a backend for a BI product, since it offers no caching at all.  However, a workflow where you crunch data in Hadoop/Hive and then export to a MySQL table (or an Endeca instance) for use in a BI tool works very well.

In the context of digital marketing, I’ve spent additional time looking into BI and analytics tools, so more on that shortly.

• An overload of choices make decision-making much more difficult since our pre-frontal cortex can only handle a small number of choices at a time (somewhere between 5 and 12)
• Several experiments suggest that emotion is a critical ingredient to enable fast decision making and prevent analysis paralysis.

Of course, Jonah Lehrer is not the first person to cover this topic. I remember reading Barry Schwartz’s Paradox of Choice a couple of years ago, and it included references to similar ideas.

What intrigues and concerns me is the implication of this research for business intelligence.  It would suggest that:

• Additional data / information can lead people (and hence companies) astray while they assume that more data is always better.  A lot of work has been done in this area.  I vaguely remember reading about such research in a Malcolm Gladwell-genre book.
• If people are making decisions based on data that may not be relevant, then we are, yet again, underestimating the role of randomness in the success or failure of companies (and CEO’s).  Someone else recently wrote about this, but I do not recall who.
• Emotion is a critical ingredient of important decision making; however, emotion can be significantly influenced by many other factors than data.  This suggestion in turn begs the question: just how important / relevant is the data?  Is it more often used for rationalizing emotional decisions than it is used for arriving at rational decisions?

While businesses will continue to move towards being run more and more by data, research like this heightens the importance of the non-technical parts of a good business intelligence implementation – understanding the data that you have available, truly understanding what you’re measuring, how it may impact the business and of course good data  visualization in the final implementation.

PS. The title of this blog post refers to the difficulty the author, Jonah Lehrer, has in picking cereal due to the vast number of choices available.

A friend recently showed me a demo of a product idea that he’s been playing with that brings a physical world interface to the iPhone via its touch / gesture capabilities. What truly struck me during the demo was not how useful the feature was in itself, but rather the value of the action going from being an abstract action (i.e. mouse motion / click translated into a visual representation on the monitor) to a physical action again – something that you touch and interact with ( you can’t feel the texture – yet!).

Many have talked about the advantage of a physical book over the many e-versions that we have seen over the years. And of course, simply mimicking the act of turning a page with a gesture on the iPhone screen does not dramatically change the user experience of reading. However, gestures on the iPhone (and other similar touch interfaces) do get one step closer to the physical world that we are used to and for certain applications, that will be good enough for success.

Given that users who are not tech savvy often type search queries into the URL bar, I always wondered what the value was of the URL bar. Clearly, their intention was not to go to the specific URL (unique resource locator) but rather to find out what other information there is on what they typed in. Similarly, as a tech savvy user, I rarely type in URL’s other than for a few often-used sites that I expect my browser to auto-complete. In almost every case, I search instead since I’m never sure what domain a company is using (often, due to domain name squatters). Usually, I find my desired link in the first page of search results.

My follow-up question has been – why not get rid of the URL bar and only provide a search bar? Seems that Google Chrome finally does that.

Example of a word typed into the Google Chrome Address Bar (source: http://www.google.com/support/chrome/bin/answer.py?answer=95440&hl=en)

Search queries and results are listed in line with the potential domain names that could match your query. Hopefully, Firefox 3′s AwesomeBar will provide similar functionality sometime soon (maybe another FF plugin already does?). And if there is broad adoption of Google Chrome, can we hope that the value of typo domain names and squatted domain names will decline?

I acknowledge the obvious value to Google (and Yahoo! and MSN) of additional traffic and searches; I infinitely prefer that option to the current state of affairs.