It’s all about connections

I have not written in a while (not a great thing for a blog) but hopefully that will change beginning with this quick reflection on a post that has already been quoted multiple times. Tim Berners-Lee writes in Giant Global Graph:

People running Internet systems had to let their computer be used for forwarding other people’s packets, and connecting new applications they had no control over. People making web sites sometimes tried to legally prevent others from linking into the site, as they wanted complete control of the user experience, and they would not link out as they did not want people to escape. Until after a few months they realized how the web works. And the re-use kicked in. And the payoff started blowing people’s minds.Letting your data connect to other people’s data is a bit about letting go in that sense.

It is still not about giving to people data which they don’t have a right to. It is about letting it be connected to data from peer sites. It is about letting it be joined to data from other applications.

It is about getting excited about connections, rather than nervous.

Specifically, Tim speaks of the graph above the Net and the Web as providing the level of data connection that will usher in a new level of thinking.  Not about sites and destinations but about things.  Meanwhile, in the Enterprise we still have companies talking about the unified BI stack.  Are you kidding me?  Sure, unification does allow for simplification (or so it may seem) and some BI goals require it (for now) but it is time to get excited about connections and representations above the data noise and that sweet pie chart report.  This will allow for new power as data is easily linked and explored instead of constantly being re-positioned, re-organized and re-published to provide “simpler” access.

Tim goes on to say that being able to connect what a document is about instead of just connect documents themselves as a next step is “obvious, really“.  Well, I am not sure the elaborate trappings of data, systems and IT are allowing the obvious to take place. Where is the open, semantic integration of corporate information that allows for open information exchange? (One might quietly whisper services at this point or indexed feeds but I don’t think that gets it done and refuse to go off on a tangent here).

What I think is obvious, is that interconnected data and how people share and relate to that interconnection will define business intelligence in the years ahead.  The knowledge work that will be done in the context of relationships and visibility across department and organizational boundaries will be as simple as writing a message on a Facebook wall.  In fact, the ability to speak about the graph of an organization and focus on things like views of a transaction, views of a supplier interaction, views of a customer — instead of worrying about the application, the individual process or the report — will create the right framework for increasingly flexible and powerful organizations.

Even at Zua Scene, we see the instant power of what we call a simplified semantic layer around locations and things as granting instant collaboration across user communities by linking spreadsheets by what the rows are about — not just the sheets.  We see the power of creating a simple graph to represent vulnerable populations and disease outbreaks in improving global healthcare.  We are starting to explore how to look at the impact of knowing about a healthcare procedure as a thing with many sides (not bound by a system) with partners and clients. So, yes, we support the Graph and hope that the BI community learns it must support the Graph sooner rather than later to provide user the real visibility they need.

Users, front and center please

In a comment exchange on a Bill Ives post regarding IBMs mashup moves I suggested that many of the current mashup tools still take us to a place where the developer is central in the conversation. In fact, shouldn’t the success of social media and Web 2.0 found in the often referenced of FaceBook, Wikipedia and blogoshpere inspire us to place the enterprise user right in the middle of the technology?

After considering Mr. Ives’ response,

“I agree that mashups do not directly put the user in the center. They are more of a developer short cut. However, by easing the developer’s burden, more time can be spent on user requirements and thus putting the user in the center. They also allow for business units to more easily develop their own applications”

I thought about my own experience with users and I am not sure they want more time spent on requirements (no matter how extreme the techniques). No, in the BI 2.0 world, knowledge management and corporate information worlds we need to push not only for easier to develop applications but continue the hunt for the elusive user developed application.

I feel that mashup technology done correctly will lead to a partial answer to user-developed applications and bring about self-service solutions like never before. Don’t get me wrong — my own 10+ years around start-ups and big shop IT leaves me painfully aware that certain tasks and solutions will always be the domain of experienced developers teaming with quality subject matter experts to arrive at powerful solutions. But it is clear, that much of the energy of crowdsourcing, the READ-WRITE web and web 2.0 should hit the enterprise as user directed, created and deployed application and content experiences.

It is with that thought I applauded Intel, as reported by Slashdot, on its work in the mashup world. Like Zua Scene, it appears the semantic bug has bitten Mash Maker (and then some) to continue to push self-service forward. As the demo shows legroom finding its way into an open flight search, airline enterprise users should just as easily be able to mix delayed flights with important traveler information and baggage handling systems to increase customer service and loyalty (and it should not require the development of a new customer service-baggage-traveler information system to do it).

Need totals, have services

One of the pillars of BI is aggregation. Totals (an aggregate) — every banded report has them. Often metrics are some variation of aggregation or set math. Give me my division numbers please! Services, however, are usually fine grained, transactional. Can they be aggregated? Should they?

How do you take a single service that returns the sales of an invoice line and get those division numbers? Can you? Are you forced to hide a proper aggregating expression behind a division total service? And if you do have to hide or pre-calculate the aggregation (can you say OLAP) is the ability of users to construct visual integrations (mashups) of these services of these special aggregate services on the fly really more useful than traditional approaches? (Good questions and unfortunately the complete answer probably won’t appear in the next paragraph)

I can say this, however. First, let’s take a service that returns a set of data — columns/rows — and assigns an awareness semantic to the specific columns of interest. Second, take a user with access to the service acting in a visualization who’s default actions which are separate from the service are to aggregate based on the semantic. Then you create the desired effect – totals. And all of this without the user necessarily having to make an explicit determination to sum a value and the service not having to resort to pre-aggregation of values. Does that really work? (And before you object I realize that this won’t work at a certain scale but lets give it a chance)

Well Google has a pretty good set of services around the spreadsheet. If you look at each column as an element and each sheet as a service then you end up with a nice set-based service. Throw an interesting element such as number in one of the columns. Define the service so that it tags the numeric column. Then as the user uses the service in a visual designed to return totals it can sum the properly tagged columns. If that spreadsheet is division sales then without the burden of significant definition we use the context and the content in concert to achieve the answer. My division totals!

That said — this thought is still in progress — and it is true that most “mashups” serving data are little more than portal views of discrete service responses and often focused on non-structured information (regardless of what the marketing slick says). And when people want a total – it is off to the meta-data layer and the banded report model or metric calculator.

Regardless, let’s look at the future a bit by considering one of my favorite sites Swivel for a moment. It is true that the last time I built a data set the definitions resembled some of the heft of a traditional data definition in your favorite BI tool. However, I can chart sales of tennis balls from one “data service” against sales of tennis rackets from another with little more than two defined sets and the desire to chart the results. Multiple users, delivering data and collaborating in mixed, complex visuals.

Need totals? (Not all the time and less then we probably have been conditioned to think) Have services? You may still be in business with a little context and a bit of glue driving resulting calculations.

Knowledge sharing meets BI

In a recent blog post by JP Rangaswami (confused of calcutta) entitled Facebook and the Enterprise: Part 5: Knowledge Management
JP writes:

“I believe there are three primary reasons why an enterprise would want to “manage its knowledge”:

One, to share learning, so that the same mistake is not made multiple times.

Two, to share learning, so that activities get sped up.

Three, to share learning, so that people are motivated to learn and to teach.

To share learning.

Knowledge management is not really about the content, it is about creating an environment where learning takes place. Maybe we spend too much time trying to create an environment where teaching takes place, rather than focus on the learning.

Since people want to learn by watching others, what we need to do is to improve the toolsets and the environment that allows people to watch others. It could be as simple as: What does my boss do? Whom does she talk to? What are her surfing habits like? Whom does she treat as high priority in terms of communications received? What applications does she use? Which ones does she not use? When she has a particular Ghost to deal with, which particular Ghostbuster does she call?”

“To share learning” by watching others is indeed an interesting point.  A point possibly lost on today’s BI community.  For business intelligence is “the enterprise place” to find information, right?  It is all about enhancing knowledge, right?  So maybe it is time to ask the BI community what does a user, viewing a report (or even searching for one in isolation) have to truly do with shared learning.

I recently received an e-mail inviting me to a web demo of a data visualization tool I really like (to remain nameless for this post).  It talked in the subject line about collaboration but in the end described a user performing a visual reporting task.  Maybe shipping it to a dashboard or PowerPoint demonstration.  I was a little disappointed.

If we are to truly “look over each other’s informational shoulders” should we not have the ability to publish our BI thoughts, pick freely from services (not always just those prepared in a universe or framework by an integration team) and mix in the thoughts of others?  In seeing other people’s thoughts on what is interesting, seeing them build information combinations and viewing important services through shared visualizations we can start (baby steps of course) “To share learning”.  Learning through others’ behaviors as they interact with information sources.

This is where the tried and true banded report sees its end?  (Or at least re-defines itself for what it is.)  So, yes it is good to have sliders, dimensions and drill-through views for exploration of questions — but frankly I would rather know what my experienced colleague knows and build from that basis rather then try to always discover my own result from a roll-up of my division’s sales history.  Maybe the point here is overstated but if the work on collaboration is going to move out of the Ambient BI labs of BusinessObjects and actually change user behavior maybe a few bold predictions such as the death of the banded report are not as crazy as they seem.

Community and BI

There is a lot to be said for all of the activity around social networks in the Web 2.0 world. In short, these networks allow individuals to publish and find information among friends and contacts. Recent work by Microsoft in SharePoint and BEA through AquaLogic demonstrate the desire to team community with content management for the enterprise. Is it merely the realm of portals where community and content merge?

Business intelligence can also take advantage of community networks. Instead of looking at dashboards by role, BI systems could provide users with the ability to collaborate with data services and each other in very visual ways. In theory, users could take sources of information published through BI and related architectures and combine them into visuals, alter and share them in community. The ranking, comment and combination of these services by individuals would further free information and assure the best content (and related reports/visuals) surfaces to assist business knowledge workers.

Sure, today’s BI can allow users to create reports without IT intervention (maybe) and see specific dashboards related to role and function but they still focus predominately on a single user exploring content. Even embedded BI (or BI 2.0) concentrates metrics in point processes.

Meanwhile, the contribution tools of social networks from blogging to exchanging messages on a friend’s wall (see FaceBook) are creating rich tapestries of information (a tapestry that BI attempts to create for the business user). So why not use these same concepts around data sources and informational services to produce viable business content through user relationships around information. This breaks the single user asking questions of a reporting system paradigm that often exists today and makes business intelligence a self-organized conversation.

SOA and BI 2.0 Come Together

It is interesting to see that Tibco acquired Spotfire recently. This highlights the importance of closely linking business intelligence with services. With BI 2.0 seeking a reduction in “time to intelligence” and BI companies looking closely at guided, embedded and event driven analysis this type of marriage make a lot of sense. In order for BI to continue to advance and move toward right-time information it will need in many cases to divorce itself of the extract, transform, cleanse and load process of duplication and seek to create more clarity in the often “dirty” transactional world.

With BI 1.0 pounding the better reporting, 360 degree view theme the move to put BI closer to the business process (if not within) will continue to pose practical challenges to user experience and analysis engines. I mean look at Spotfire’s home page itself. Lots of charts and graphs which usually result from combinations of data streams and transactional history which have been pulled from systems in bulk or through incremental processing and dimensional analysis.

As services deliver more real-time information into the conversation what natural mechanisms will allow for insightful views and trend analytics? Will a new crop of services emerge in the business engine to server sets or will we still be quietly amassing data marts and warehouses through direct service integration (duplicating those streams)? It will be interesting to watch the developments of this pairing and see those client case studies emerge from the combination of well governed and executed SOA enabled business processing and tightly coupled event based business intelligence.

Open BI

In reading the recent book Wikinomics the power of mass collaboration and the profound impact of the new age of communications reach and capability is explored in depth. New economic and social models are discussed. As you would expect, open source communities are a big topic. In fact, the business intelligence open source community Pentaho is referenced in the book. It will be interesting to see the advancement of open BI and if traditional players will continue to dominate or change to open models themselves.

In the case of Zua Scene, however, the power of mass collaboration has additional meaning. How does the user base of a BI tool assure that the content created (reports, insights and metrics) are of the highest quality? How is collaboration facilitated beyond the “Share My Report” option? How does the crowd (or at least the group of analysts, report writers and business process specialists) shape the content and its value?

One answer lies in adapting social networking tools into the BI realm. Users should be able to explore combinations of resources, rank them and have fellow users be able to extend and alter those sources until the strongest content remains. When you look at collaboration today you find much being said about the power of the Wiki or portal (widget) based collaboration. Unfortunately, depending on structure, search technology and available content these tools can become cluttered and difficult to manage. The value decreases as the content builds in many cases (not the desired effect). Well organized sites like the commercial site Wikipedia tend to have the structure and search characteristics required to make the content more approachable and user interactions that allow for governance.

So as data sources open up and tools allow users to glean more information from web, partner and private sources it is important to make sure we just don’t contribute to the already noisy landscape of constant information. By embedding metrics into business processes (a major goal of BI 2.0) and allowing better scenario-based access to data we see some attempts to strengthen BI and reduce clutter. We feel that current steps are just the beginning.

User contribution to content development and quality is also vital. Can you search, connect, rank and share information freely? Are the conclusions of your peers building blocks for developing insight or silos of understanding? Can you grab semi-structured public information (or unstructured content) and drive it into your business visuals with minimal effort? You should be able to. And hopefully, the same collaborative traits of the open source community driving speed, innovation and quality will be found in your next BI assited, collaborative business decision!

Not quite the same page

I have mentioned BI 2.0 several times and like most references to a big, generational topic it can have several flavors. A good DM Review article by Charles Nicholls provides a definition emphasizing real-time, event driven BI as BI 2.0. I remembered this article as I was writing and went to check it after my last post in which I suggested we might be at BI 1.5.

Well, I thought I would clear up my definition a little. I see a lot of the natural conversation around Web 2.0 being about social networking, rich internet applications and mashups (and perhaps a lot more). So, from my perspective this should also be in the BI 2.0 conversation.

No doubt real-time BI and operational BI intelligence is critical to the growth of BI in support of business processes. But if “the goal of BI 2.0 is to reduce latency – to cut the time between when an event occurs and when an action is taken – in order to improve business performance” then I don’t feel you need to always skip to the injection of BI into the event stream as the complete answer. It’s just an opinion, but users will still need to play an important part in this conversation for quite some time to come and the social impact of picking the right information at the right time from the right services can benefit from a flavor of BI 2.0 that includes user-driven tools as well as event driven and embedded options. So, it’s probably still not clear to you or me what the shades of 1.0, 2.0 and 3.0 are in every context but that is why this blog entry and many like it exist I suppose.

Are we there yet?

In a brief blurb by Jason Bloomberg of Zapthink he writes that the enterprise mashup is the future of service-oriented business applications. Several solid points regarding loose coupling (we do aim to support this in our mashup solutions) and governance were made regarding the required maturity of mashups to play ball inside business. And frankly, I have nothing original to add to those thoughts (as we generally agree) but it did get me thinking about the significance of Web 2.0 (social aspects) and trends in BI.

BI 2.0 is clearly trying to pick up on a theme (hence 2.0) but in my conversations with individuals within IT and creating products around business intelligence they are still blocking and tackling. In many ways, thinking about the impact of social networking or instant service visibility is well outside the daily focus on e-mail report distribution of banded reports and the appropriateness of conformed dimensions. For example, I am certain Gartner’s next BI conference this month will proclaim much success in BI 2.0 and real-time BI but it is not clear in my limited view of the world that the seeming simplicity and power of social networks for content creation and the needs of structured BI performance and big data have really met.

I would say BI 2.0 may be BI 1.5 where at least the user is in control of increasing amounts of information and data exploration but seamless mixing and sharing? I would like to be introduced to those products. Keeping all the cards on the table we at Zua Scene want to offer solutions that help with that mixed and share problem. But really, is having a personalized dashboard or a dynamic dimensional view really BI 2.0? Given the proper environment users will have a lot to say about the quality, quantity and applicability of the powerful views they need and want to drive business but I would say this conversation is barely getting started.

The power of combination

With the 1,000s of map mashups now floating around the web and the apparent success of Yahoo! Pipes (time will tell) there can be little doubt as to the power of combination. See GoogleMaps hybrid view or recently added traffic view as quick examples of combinatorial muscle! We clearly like information best when its in relationship (and usually when it is very specific to the task at hand). And now services provide some powerful options and interesting issues.

One issue is that often web services encourage us to see bits and pieces (hopefully reusable or interesting bits and pieces) of information. And in some ways, the bits and pieces approach of web services reverse all that hard work of data cleansing, ETL and warehouse building that has been trying to give the enterprise user a 360 degree combined view of information.

So now, we are faced with a BI 2.0 challenge. How to take advantage of the new infrastructure? Is it as simple as changing the question and letting the bits and pieces flow into a specific business process in real-time so that consolidated views and reports no longer rule? Maybe. Will it be adding the service invocation to information integration tools or as part of the big meta-data universe that rules the organizational view today? Maybe. And if so, will it be so easy to just lay in all of these services at various levels of granularity into the current models? I am sure many vendors will say yes! But it will not likely be as easy as just saying yes.

In fact, new tools will now emerge (probably daily) to solve various aspects of this service-based power and peril. And we selfishly hope we can contribute in our own focused way. One key is to look hard at service relationships themselves. Regardless of your technology options, services move fast and you need to be able to quickly identify a service and where it can be mixed. Better if that is a simple process and leaves the user some flexibility when it comes to what to mix and when (flexibility that is sometimes missing in today’s reporting environments). So, instead of just building bits and pieces all day long and talking about how systems use them to communicate (isn’t that great) — you can use the power of relationships and combinations of services to deliver the full picture!

And it is that power of combination that turns a plotted address into a well understood destination – a map into a guide – or the bits and pieces into a path instead of just individual stepping stones.