SOA and BI 2.0 Come Together

It is interesting to see that Tibco acquired Spotfire recently. This highlights the importance of closely linking business intelligence with services. With BI 2.0 seeking a reduction in “time to intelligence” and BI companies looking closely at guided, embedded and event driven analysis this type of marriage make a lot of sense. In order for BI to continue to advance and move toward right-time information it will need in many cases to divorce itself of the extract, transform, cleanse and load process of duplication and seek to create more clarity in the often “dirty” transactional world.

With BI 1.0 pounding the better reporting, 360 degree view theme the move to put BI closer to the business process (if not within) will continue to pose practical challenges to user experience and analysis engines. I mean look at Spotfire’s home page itself. Lots of charts and graphs which usually result from combinations of data streams and transactional history which have been pulled from systems in bulk or through incremental processing and dimensional analysis.

As services deliver more real-time information into the conversation what natural mechanisms will allow for insightful views and trend analytics? Will a new crop of services emerge in the business engine to server sets or will we still be quietly amassing data marts and warehouses through direct service integration (duplicating those streams)? It will be interesting to watch the developments of this pairing and see those client case studies emerge from the combination of well governed and executed SOA enabled business processing and tightly coupled event based business intelligence.

Open BI

In reading the recent book Wikinomics the power of mass collaboration and the profound impact of the new age of communications reach and capability is explored in depth. New economic and social models are discussed. As you would expect, open source communities are a big topic. In fact, the business intelligence open source community Pentaho is referenced in the book. It will be interesting to see the advancement of open BI and if traditional players will continue to dominate or change to open models themselves.

In the case of Zua Scene, however, the power of mass collaboration has additional meaning. How does the user base of a BI tool assure that the content created (reports, insights and metrics) are of the highest quality? How is collaboration facilitated beyond the “Share My Report” option? How does the crowd (or at least the group of analysts, report writers and business process specialists) shape the content and its value?

One answer lies in adapting social networking tools into the BI realm. Users should be able to explore combinations of resources, rank them and have fellow users be able to extend and alter those sources until the strongest content remains. When you look at collaboration today you find much being said about the power of the Wiki or portal (widget) based collaboration. Unfortunately, depending on structure, search technology and available content these tools can become cluttered and difficult to manage. The value decreases as the content builds in many cases (not the desired effect). Well organized sites like the commercial site Wikipedia tend to have the structure and search characteristics required to make the content more approachable and user interactions that allow for governance.

So as data sources open up and tools allow users to glean more information from web, partner and private sources it is important to make sure we just don’t contribute to the already noisy landscape of constant information. By embedding metrics into business processes (a major goal of BI 2.0) and allowing better scenario-based access to data we see some attempts to strengthen BI and reduce clutter. We feel that current steps are just the beginning.

User contribution to content development and quality is also vital. Can you search, connect, rank and share information freely? Are the conclusions of your peers building blocks for developing insight or silos of understanding? Can you grab semi-structured public information (or unstructured content) and drive it into your business visuals with minimal effort? You should be able to. And hopefully, the same collaborative traits of the open source community driving speed, innovation and quality will be found in your next BI assited, collaborative business decision!

A disconnect?

In a recent Web 2.0 survey by the McKinsey group about 1,800 company executives responded to questions about investment trends they see related to blogs, wikis, RSS, web service, mashup and other Web 2.0 technologies within the enterprise.

Web service investments were high as you would expect with 80% saying they were working or planning to work with services. Everything else came in around 30-50% in terms of current implementations and plans except for mashups with over 50% not planning to use or consider mashup technology.

Does anyone else think this 0dd? At Zua Scene, we believe it is time for users to reap direct advantage from web service investments. And it is hard to imagine a collective intelligence system (relatively high on the survey respondents list) without services being used to expose information into the shared environment. I would site work with QEDWiki at IBM as an example of this trend. Well, when you take and mix services or even better but the end result of services into the hands of end users to empower decision making, planning and knowledge sharing is this not a type of mashup?

So, massive investment in web services just to get better integration between system components or extend the life of that legacy back-end system? All this investment in ESB’s and service governance tools to route and secure services and events is just for stronger application messaging capabilities?

Surely not. In fact, shouldn’t mashups be a natural conclusion for those looking to derive increasing value from services and a disruptive trend for application composition (if not already)? We certainly think so!

Four paths to business visibility

For the last decade (plus some), business intelligence has focused on assembly of information into structured repositories.  And now, it may be a well structured meta-data repository with query capabilities on the fly – but regardless it requires a complete definition (and perhaps a separate physical storage component outside the system of record).  Call them marts, cubes or warehouses this trend has produced 360 degree views, banded report writers, meta-data management, data appliances and all sorts of beneficial tools for the visualization of information.  Path 1 in this case is the robust world of the data warehouse.  It has also led to data lineage issues, processing windows, inconsistencies from the system of record and in some cases lengthy wait times for end users to get the access they need to data.  Nonetheless, the concepts of shared dimensions, fact tables and the like are at clearly beneficial and will remain at the core of business intelligence for a while.

Path 2 is more along the lines of operational BI (BI 2.0 I suppose).  Collecting information from the service stream and injecting metrics into the business process.  Or at least providing the user with better visibility control on top of the dimensional and in many cases non-dimensional sources of information.  We still have a lot of description going on but things are clearly more fluid as tools advance and far more specific in many cases as BI is pushed closer to or derived from the transaction environments themselves.  For sake of argument we can throw the onDemand (model predefined and hosted) providers in this class as well as the attempts are less about defining everything and look to defining the right thing or analysis based on observation.  Still, often a significant repository sits behind these tools.  (I admit that I likely have not described or know the full essence of this path at this point but look forward to learning more.)

Paths 3 and 4 may be less obvious.  They involve the mashup.  So, on one side (path 3) you have composition after the fact.  The dashboard made of widgets or portal page come to mind.  The collection perhaps sharing a set of injected parameters or integration at the glass to provide a view into the information.  In this case, you may be asking several core systems to collaborate or integrating from a system of record with a back-end mart view by executing tasks through a shared session element such as the user.  These clearly allow more information to become transparent if executed correctly and reduce reliance on back end aggregates.  I don’t know if these are slice and dice worthy implementations but it is an option in the quest for reduced time between data capture and visibility.

So what is path 4?  Path 4 is composition and service-based as well but prior to visualization not after (or at the glass).  This can bring the promise of integrated charts, graphs and visuals typically reserved for more traditional BI architectures forward into the mashup world.  Indeed, some glue needs to wind through the data to empower the combination of disparate elements and services but the need for physical storage and full meta-data descriptions may not be required.  Views that may be embeddable into business process applications much like BI 2.0 without the aggregations of predefined models.

What path to choose?  Well as in most business scenarios it depends (and I will likely comment on what it depends on later).  But it is nice to see the conversation becoming more open and services helping everyone redefine visibility as we speak.

Not quite the same page

I have mentioned BI 2.0 several times and like most references to a big, generational topic it can have several flavors. A good DM Review article by Charles Nicholls provides a definition emphasizing real-time, event driven BI as BI 2.0. I remembered this article as I was writing and went to check it after my last post in which I suggested we might be at BI 1.5.

Well, I thought I would clear up my definition a little. I see a lot of the natural conversation around Web 2.0 being about social networking, rich internet applications and mashups (and perhaps a lot more). So, from my perspective this should also be in the BI 2.0 conversation.

No doubt real-time BI and operational BI intelligence is critical to the growth of BI in support of business processes. But if “the goal of BI 2.0 is to reduce latency – to cut the time between when an event occurs and when an action is taken – in order to improve business performance” then I don’t feel you need to always skip to the injection of BI into the event stream as the complete answer. It’s just an opinion, but users will still need to play an important part in this conversation for quite some time to come and the social impact of picking the right information at the right time from the right services can benefit from a flavor of BI 2.0 that includes user-driven tools as well as event driven and embedded options. So, it’s probably still not clear to you or me what the shades of 1.0, 2.0 and 3.0 are in every context but that is why this blog entry and many like it exist I suppose.

Are we there yet?

In a brief blurb by Jason Bloomberg of Zapthink he writes that the enterprise mashup is the future of service-oriented business applications. Several solid points regarding loose coupling (we do aim to support this in our mashup solutions) and governance were made regarding the required maturity of mashups to play ball inside business. And frankly, I have nothing original to add to those thoughts (as we generally agree) but it did get me thinking about the significance of Web 2.0 (social aspects) and trends in BI.

BI 2.0 is clearly trying to pick up on a theme (hence 2.0) but in my conversations with individuals within IT and creating products around business intelligence they are still blocking and tackling. In many ways, thinking about the impact of social networking or instant service visibility is well outside the daily focus on e-mail report distribution of banded reports and the appropriateness of conformed dimensions. For example, I am certain Gartner’s next BI conference this month will proclaim much success in BI 2.0 and real-time BI but it is not clear in my limited view of the world that the seeming simplicity and power of social networks for content creation and the needs of structured BI performance and big data have really met.

I would say BI 2.0 may be BI 1.5 where at least the user is in control of increasing amounts of information and data exploration but seamless mixing and sharing? I would like to be introduced to those products. Keeping all the cards on the table we at Zua Scene want to offer solutions that help with that mixed and share problem. But really, is having a personalized dashboard or a dynamic dimensional view really BI 2.0? Given the proper environment users will have a lot to say about the quality, quantity and applicability of the powerful views they need and want to drive business but I would say this conversation is barely getting started.

The power of combination

With the 1,000s of map mashups now floating around the web and the apparent success of Yahoo! Pipes (time will tell) there can be little doubt as to the power of combination. See GoogleMaps hybrid view or recently added traffic view as quick examples of combinatorial muscle! We clearly like information best when its in relationship (and usually when it is very specific to the task at hand). And now services provide some powerful options and interesting issues.

One issue is that often web services encourage us to see bits and pieces (hopefully reusable or interesting bits and pieces) of information. And in some ways, the bits and pieces approach of web services reverse all that hard work of data cleansing, ETL and warehouse building that has been trying to give the enterprise user a 360 degree combined view of information.

So now, we are faced with a BI 2.0 challenge. How to take advantage of the new infrastructure? Is it as simple as changing the question and letting the bits and pieces flow into a specific business process in real-time so that consolidated views and reports no longer rule? Maybe. Will it be adding the service invocation to information integration tools or as part of the big meta-data universe that rules the organizational view today? Maybe. And if so, will it be so easy to just lay in all of these services at various levels of granularity into the current models? I am sure many vendors will say yes! But it will not likely be as easy as just saying yes.

In fact, new tools will now emerge (probably daily) to solve various aspects of this service-based power and peril. And we selfishly hope we can contribute in our own focused way. One key is to look hard at service relationships themselves. Regardless of your technology options, services move fast and you need to be able to quickly identify a service and where it can be mixed. Better if that is a simple process and leaves the user some flexibility when it comes to what to mix and when (flexibility that is sometimes missing in today’s reporting environments). So, instead of just building bits and pieces all day long and talking about how systems use them to communicate (isn’t that great) — you can use the power of relationships and combinations of services to deliver the full picture!

And it is that power of combination that turns a plotted address into a well understood destination – a map into a guide – or the bits and pieces into a path instead of just individual stepping stones.

When business intelligence does the mash

There is a lot to be said about the BI 2.0 movement (and I hope to chime in). However, for all the writing about SOA and the integration of business analytics directly into business processes with a user centric focus — all key themes for BI 2.0 I have not seen specific conversations about where mashup and social networking themes fit into this new world of business intelligence. (Note: I have more reading to do on the topic but until then…)

We believe that the ease of social networking in sharing community insight and the power of service mashing will emerge as key themes in the business intelligence discussion. It is true that users need the power to mine data quickly as emplified by organizations like Spotfire or the recent aquisition of Celequest by Cognos. And that the ability to inject analytics directly into event driven business process streams like SeeWhy is also important.

But many organizations don’t have sophisticated event driven application architectures. And while many BI users like what they see in dimensional analysis, they rarely use all the capability of mining systems. So where does that leave the current BI movement?

Well these trends all point in one direction. Users want to be the focus of business intelligence. And we all want the right information at the right time. Unfortunately, this is largely situational. So it would appear that a future key to better business intelligence is reducing the complexity of implemenation and making sure users have the ability to direct the process once the right problem domain has been defined (no searching the sea of the data warehouse).

Let’s make this real.

Suppose several healthcare organizations are working on disease management. They may need to share information collected in the field and have instant access to resources required for response to an outbreak. This is a specific situation indeed. So, with little time to react how do users get the information they need?

Services may provide the answer. We may not know exactly who in the user community needs what information but loosely coupled services (if combined on the fly) could provide field observations and proximity information regarding clinics, hospitals, population statistics and infrastructure issues. As simple in some cases as combining a hospital yellow pages search with observations in the field captured through a mobile phone application. For government entities this may mean putting together mutliple agencies’ service feeds along these lines and letting users mix (and mash) these feeds to create the right picture for response in a local area.

So yes — the power of the mashup has a definate place in the future of BI 2.0. Will combinations of services solve every issue? No (well not yet). For example, latency and volume may prohibit some analysis and mining tasks but watch closely because even public sites like Swivel are showing that the future of information analysis is changing. Regardless of what emerges over time situational implementations can take advantage of the growing simplicity of mashup creation, the increased availability of services and the power of a distinct user focus.

It begins

I started Zua Scene in search of an interesting idea and an even more interesting learning experience. The learning has been no problem. Every day brings something new to the table. Good or bad, hard to tell to sometimes. We’ll see…