Business Transaction Management in the News

As you may or may not know, many of the major Transaction Monitoring players have their R&D centers based in Israel; Correlsense, Optier, Correlix and B-Hive. Considering the important role that "hi-tech" plays in the Israeli economy, it is no surprise that financial publications publish technology related articles on a regular basis; I have taken the liberty to translate parts of a recent article that was published in "Globes Israel".

"After evaluating both Correlsense and Optier; we have decided to invest in Correlsense"

The article talks about how the IT systems management world is currently going through some "tectonic movements" and that the companies that emerge once the dust clears will have a pivotal role in the market. Similarly, there are forces which are acting on the enterprise computing world in general - virtualization, the large growth in the availability of information and applications within the organization and outside of it and of course, the constant need to reduce operational costs – that are causing executives to think differently.

There are a number of Israeli companies who are in this position; Optier is an example of a company that is trying to become a big player in the IT systems management market. This is also the case with Correlsense, says Sam Somech. Somech is an Angel that invested in Correlsense, along with the VC firm eXeed, and is now serving as the company's chairman of the board.

Both Correlsense and Optier have developed technologies for the troubleshooting and optimization of data centers, with the belief that the best way to deal with the complexities of the modern data center is by tracking transactions from the user, through the data center and all the way to the back end data bases.

"The surviving company will have a high value"

According to the evaluation of Correlsense's investors and founders, the leading IT systems management companies – IBM, HP, BMC, CA or Microsoft – do not have this kind of technology yet. "We evaluated the two companies - Optier and Correlsense", Says Somech, "and in our opinion Correlsense's technology is favorable". Optier had no comment.

Despite the uncertainties, "if a start-up can provide real innovation that will give it the edge in a sale, then it has what to offer the market", says Somech. "The technological gap between what IBM has to offer in relation to Correlsense is large. In this world, he who has the faster, more economical offering is at an advantage. It is not likely that the companies in the market today [the big four] are going to be able to invest heavily in R&D, they have also been affected [by the financial situation] and they are not going to invest tens of millions of dollars when they do not know what will come out of them. Therefore the start-up companies that will survive this downturn will have a high value".

With all due respect to Mr. Somech, the IT systems management market will not wait for Correlsense or even Optier. The Gartner Analyst, Will Cappelli for instance told "Globes" at the beginning of the year that Optier will eventually have to find itself as part of a larger company that will acquire it. Even BMC CEO, Bob Beauchamp advises the companies that are active in the field to go for acquisition, as it is a mature field with no room for niche solutions.

"We don't want to cut our valuation"

"The transaction monitoring market is not mature, it's just starting", replies Somech. Oren Elias, CEO and founder of Correlsense, says "transaction management is a big leap forward. Therefore, in times like these when the "big four" are investing less [in R&D], there is an opportunity for a new corporation to rise up and take business even from the "big four"".

There are a number of players in the IT Systems Management field and the Israeli ones among them especially stand out; Optier, Correlsense, Precise, B-Hive and Correlix. Elias and Somech are convinced that their solution is unique. "The only companies that built their transaction monitoring technologies from scratch are Correlsense and Optier", they claim.

In order to turn into one of the hottest names in software in the world, Optier has raised $116M. Correlsense, still at the beginning of their route, has been satisfied by their $1.5M round that was completed half a year ago from eXeed, Vertex and Proseed and is now in the process of raising a few more million.

"We believe in the company, it's not a good idea for us to go for additional external funding at this point, we don't want to cut our valuation. I think that it is going to be a good investment for our firm, because of the advanced technology. We are looking for just enough money to keep the company going for the next two years at which point the market is going to look much different. At this point there isn't a lot of competition".

Business Transaction Management’s Challenges

Business Transaction Management (BTM) is a natural continuation of the past decade and a half's evolution of IT systems management. In the past few years the modern data center has finally started to stabilize; the number of node types has become constant and each node has had tools developed for it.

These silo specific tools are now able to solve 90% of the problems; leaving us with the hardest to solve - last ten percent. This last ten percent is characterized, for example, by those application bottlenecks that occur even though all of the silo specific tools are showing 100% availability.

If the monitoring tools at all tiers are showing 100% availability then how does one know that there is a problem? Well, either the enterprise has put in place an end user measurement tool or the help desk is receiving user complaints.

The IT organization's number one priority is very simple; ensure that all transactions are executing correctly and in a timely manner – it's that simple.

The demand for Business Transaction Management tools comes from the need fulfill that priority; especially when traditional tools are unable to go that last mile. What has enabled the development of these Business Transaction Management tools is the datacenter's recent stabilization; with the number of node types staying constant for a few years, it is finally possible to catch up and enable the full visibility into the entire datacenter that BTM solutions must provide.

BTM Enables True Availability

The only way to resolve the problems that traditional monitoring tools cannot cope with is to take into account the interactions between the nodes and to connect every click of the user to the many events that are triggered by that activation. This enables both being able to see everything in the business context - the user's click of a button - and solving problems that have more than one source.

Business Transaction Management tools are able to monitor the entire system by tracking every single transaction that is activated by the users throughout the entire datacenter, collecting information on all of the interactions along the way. This is the essence of connecting business and IT; understanding the business context by linking every single event in the datacenter to the click of a user.

With Business Transaction Management, a common language is created, where performance can be measured as the time it takes for the transaction to travel between all of the nodes and back. Since all interactions are recorded along the way, if something goes wrong, i.e. a latency that is out of specification, then the event that caused the problem can be immediately singled out since the latencies at each tier are known and all of the necessary data has been collected. 

What Qualifies as a BTM Solution?

Every Business Transaction Management solution should be able to do the following:

Track a transaction that was sent out from a browser, through a load balancer and to a proxy. This proxy is not a known vendor's proxy, just some start-up's which developed the proxy five years ago - the code is still there and still working, but no one has the code - just the binary.

That proxy is sending it to another proxy and its being split to - making it easy - two different applications, each with its own different web server. Each application is running on its own app server - one is java based, the other is not; it's a homegrown C++ program. The first app server is sending out data base requests to four different databases; Oracle, DB2, Sybase and SQL server. The second app server is SOAP based; sending SOAP requests to an external application. There is an MQ at the external application, 'puts' are being sent to the MQ and a Mainframe is receiving the messages from the MQ. 

If a vendor wants to be called a BTM vendor he has to be able to track the transaction across all of these components. He has to be able to account for the time every transaction spends at every point in the datacenter.

Developing a Business Transaction Management Solution

Developing this kind of solution is far from trivial. The task of connecting each event within the datacenter to a specific transaction is a big challenge in today's distributed and heterogeneous systems.

The general concept of how to execute a Business Transaction Management solution is pretty straight forward. Agents must be installed at all tiers, collecting information about everything that flows through that particular tier and all of the collected data is sent back to a central dedicated server that is able to correlate all of the events to the single user's click of a button in the application. The various methods of gluing these events together are the core competency of the solution, along with the ability to develop an agent that complies with all of the different kinds of servers that one finds in the data center.

Some big players have been claiming to provide these end-to-end solutions for a number of years. Only recently have an extremely limited number of vendors been able to prove Business Transaction Management solutions in production – buyers beware.

CMG '08 Exhibition Photos

Here are some photos of the exhibition area right before opening:

G4 IT Systems Management is Here

Information Technology contributes to the efficiency of the human race like nothing else. Keeping up with the management of complex IT systems requires constant innovation; systems management has come a long way since the days of "big iron". The fourth generation of systems management, widely referred to as "end to end transaction management", enables IT to reach levels of maturity and stability like never before.

Mainframe Systems Management – G1

The Birth of application performance management tools came with the development of the Mainframe Omegamon and Tmon; these tools monitor program names, Customer Information Control System (CICS) jobs and Job Control Language (JCL) tickets. As far as application performance management goes, this was really the first transaction performance management solution; where everything was centered on the mainframe.

Client/Server – SQL Management – G2

Next up was the client/server; back then, your server was essentially the database, the database was the single point of failure, it became the place where all of the data was stored and SQL queries were everything. If you had a problematic SQL query then you had a slow transaction.

That is really when the true application performance management market emerged; with companies like Precise and Quest. The first tools for Data Base monitoring utilized the SQL command line to extract system information. These first application performance management vendors built smart agents that would collect all of the information all of the time and sometimes it actually worked in production. These application performance management tools had a nice (relative to the time) dashboard and suddenly you had this sexy, easy to work with tool for optimizing your database, DBAs finally had something to work with that gave them value. This was all around 1997, when Windows 95 was just becoming stable.

J2EE Profilers and the User Experience – G3

G3 is the generation that is currently coming to a close, as will be explained. This generation began with the emergence of Java Application Servers, Java's rise within large organizations was seen in 1999-2002, suddenly a lot of application vendors started working with Java, they built these applications and started selling them to enterprises and suddenly you had Java Application Servers everywhere. This brought on a whole new black box that needed a tool that could manage its performance – that tool, Java profilers, were brought on by Wily, BMC and others.

At the same time, with the dot com bubble, it became important to track user experience; your website was suddenly your business. With the website so critical to doing business, it was important to make sure that things were performing as expected; running synthetic transactions with Gomez, Mercury Topaz, and others in order to ensure availability was the way to do it.

Alongside the JVM profilers and user experience tools, there were always network probes that made sure that the network was working. As TCP/IP became the standard, you had solutions that were parsing and sniffing these protocols, parsing network traffic in order to monitor network performance.

What Marks the End of 3G Systems Management?

In the last couple of years the last two major silos that we have not yet listed; Middleware and Storage, have had tools developed for them. For example; companies like Onaro (acquired by NetApp) developed tools that extended application performance management into the storage tier (Application Insight) and MQSoftware. Today, every silo has its performance management tool, all with the objective of achieving greater application performance.

All of this happened over the past 5-6 years; every single silo was developed on its own and was embraced by the industry. What we see happening in the past two years is some kind of stability, we do see new concepts for application servers, other languages - not just java based, and we see a lot of integration - web services is the new buzz word - really it's just an easy way to do integration (enterprises where doing http based xml ten years ago). All of these changes not withstanding; you do not see new silos. You do see more and more silos, but not different types of silos.

Stability & Maturity => 4G (End to End)

The market has matured to a level where you can deploy end to end solutions, if you would have tried a few years ago you would have run into complications, right now there is enough stability in new technology as far as enterprise applications are concerned in order to enable the proliferation of end to end management. Of course there is always constant change and progress in things like storage and network throughput, these technologies will continue to move forward but the technology for tracking a transaction is here to stay for the foreseeable future. TCP/IP is going to stay with us for the next 10-20 years, HTTP protocols aren't going to change in the next 5-10 years, Java will not go away as an application server, there will be new methods for implementing an application server - it will not necessarily be JVM it could be a new process – but eventually it's going to be some sort of process, multi threaded or not, which will handle the transaction. The overall architecture of the application is going to stay more or less the same; a rich, mixed and complex topology that is hard enough to manage as it is. This maturity is exactly what is enabling transaction management to become the fourth generation of IT systems management.

4G Systems Management – Bringing IT all together

Forget the buzz words for a moment - Business Service Management, Business Transaction Management, Service Level Management and others - what we are talking about here is the evolution of IT systems management itself - which is defined as everything that is needed in order to make the IT data center work. Transaction management is going to compose 70% of the systems management space – it will be critical (and already is) in making sure that your datacenter actually works.

A big thanks to Doug McClure of for creating an informative podcast with Lanir Shacham from which the ideas for this article were taken. Take time to listen to Doug's Podcast which aims to shed light on the next generation of IT systems management.

Linking Business to IT

You may have heard recently about the new paradigm of managing your IT by linking it to your Business. The first question that arises when one first hears of this statement is; what do you mean? My IT infrastructure is part of my business; it is integral to it. What is the meaning of linking my IT to my Business?

Business Transaction Management – The Link

What is your IT essentially? It is an infrastructure of hardware components that interact in order to provide a service or application to your users, those users are either paying customers or employees who in turn provide services or products to paying customers. Those users are the business, they are the source of revenue that enables the business to thrive or dwindle.

Business Transaction Management solutions provide the link between all of those processes and interactions that run throughout your IT and the business' source of income, the individual transaction activations.

How does Business Transaction Management do this? The concept is simple, but the means have been sought after for many years (and have finally been realized). Every single piece of data that enters or leaves every single server, along with the resource consumption of every process in every server is linked back to a transaction activation. In this manner, not only can you connect between the different "dots" within your data center, but you can connect to the most important dot of them all, the User.

Any problem that arises within the system will always be identified by the user (unless it is a false alarm), with the power to "connect between the dots" and see every parameter and statement within its business context (the transaction activation) resolution of problems becomes child's play.

The following video gives a clear view of the importance of Business Transaction Management and its ability to link Business and IT:

The BTM Business Case for Retailers

Convincing retailers to invest in Business Transaction Management is a big challenge.

• The margins in retail are rather slim
  
• The impact of a single transaction is low (compared to the Finance Industry for instance)
  
• Competition is fierce due to today's retail search engine proliferation
  
• Expanding into online sales has been enough of a challenge as it is
  
• ROI has to be clearly shown in order to justify the investment
  

I have taken the liberty to try and come up with a business case for why retailers should consider a BTM solution; I truly believe that those that will take the step will find it hard to understand how they lived without Business Transaction Management. How can you run a business that is so transaction dependant and not be able to make the connection between business and IT?

Figures for this case study have been inspired by Aberdeen Research's well cited APM study from June 2008 (available for free). I had a tough time finding any analysts, or vendors that published concrete numbers on the retail industry. If you have seen anything, please let me know.

ROI case for Business Transaction Management

A large retail company with an annual revenue of $563M in both online sales (30%) and through its chain of nationwide stores was unsatisfied with the performance of the business critical applications they were using. They were unable to identify issues before they impacted end users, their applications were increasing in complexity, and they were unable to test the performance of their applications whose development had recently been moved offshore.

    Growth in their online division had forced them to rapidly increase capacity over the past few years and now they were stuck in a situation where they had very heterogeneous infrastructure, from the old legacy systems that they had implemented in the early nineties, to their newer SOA based applications, things were such a mess, that they were estimating that IT employees were spending 30% of their time just searching for the cause of the problems, let alone resolving them. The Retailer had also estimated a 5-7% revenue loss due not only to their customer facing web applications, but also due to productivity issues with their CRM and ERP applications. They also calculated total downtime cost per hour to be $80,000.

In order to increase productivity, customer satisfaction, brand image, and to reduce the lost revenue opportunities (in online sales) the Retailer had come to the conclusion that the monitoring tools that they had implemented for each one of their components was not enough, and that it was time to invest in a business transaction management tool. They were looking for a tool that would give them an understanding of the business context of their problems. They needed a single solution that would work for both production and pre-production testing in a way that developers could gain visibility into the interactions between their different applications. Their current tools were retrieving large amounts of data at every node, but the information was useless when it came to increasing productivity and sales, they needed something that would provide visibility across their entire infrastructure. Most importantly they didn't have time to waste weeks or months in order to implement one of the solutions provided by the larger vendors, and didn't want to have professional services people modifying their code. They wanted to go with a solution that was not service intensive, and wouldn't cost them an arm and a leg, due to the tight margins that the retail industry was as a result of a tightening economy and rising fuel costs.

For all of the reasons stated above, the Retailer should go with a Business Transaction Management Solution. In order to justify this claim, the following presents a hypothetical continuation to this case.

The organization had requested an initial BTM POC for their online division, in order to better understand the potential ROI that could be yielded from increased sales revenue which is easier to quantify, unlike increased employee productivity and mean time to repair which is tougher to justify.

On the first day of the POC, the BTM agents were already in place and transactions were being monitored from the user end all the way through the webservers, app servers and to the legacy back end. The value became clear immediately, during peak hours, numerous transactions had been taking much too long. The retailer had always wondered why it was experiencing a high number of abandoned shopping carts online. One guess was that users that were shopping the popular search engines ended up opting for competitors that were able to be more responsive to customers. Not only that, various members of the IT department were able to connect the dots on many of the performance problems that they were experiencing and resolve them immediately.

    A full installation was put into place and suddenly issues with the application's performance could be pinpointed immediately without the need to perform long grueling meetings between the IT department's professionals. In the first few months of using their BTM solution, a steady decrease in empty shopping carts could be charted. They were able to achieve a 600% average improvement for response times on business critical transactions. Developers were very satisfied to have a solution that they could directly log on, test the changes and gain immediate insight as to how the overall application performance had been affected. The MTTR was decreased and problems could be spotted proactively before they reached the customer at an 85% improved success rate.

    Since then, the retailer has expanded the scope of their BTM solution to include its internal applications. They were able to raise employee productivity since they saw an average improvement of application availability of 80%, and best of all, IT headcount, did not have to be increased. The Retailer has also been able to increase capacity exactly where it needs it; the justification for increased capacity becomes easy when IT problems are put in the business context.

If you can think of any way to make this study more robust, please write!

Transaction Monitoring – Network Appliances

Yet another way to implement Transaction Monitoring solutions is via a Network Appliance. This approach is defined here as any approach that collects data by non intrusive "Network Sniffing". Two good examples of vendors that provide this type of solution are B-Hive and Correlix.

How it Works

Network appliance solutions usually connect to a port mirror in order to collect the traffic, and then try and re-construct the entire transaction. Information needs to be collected directly from every node that is of interest.

Applications

• Any application where transaction latencies need to be monitored in a production environment
  
• Managing SLAs
  
• Systems which cannot be tempered with at all and need a "plug and play" solution
  

Advantages

• Zero Overhead
  
• Full time monitoring
  
• Immediate installation
  
• Instillation only concerns the network administrator
  
• There is no risk of crashing the system ( some "Deep Dive" solutions will cause system failure if they are used to monitor too many transaction due to high overhead)
  

Drawbacks

• Uses an algorithmic approach to track transactions which limits accuracy of metrics, latencies are not right, you do not know the accurate flow of the transaction through different tiers and so on.
  
• Tracking is not really end-to-end since you cannot see what is actually happening within the servers (cannot achieve full visibility)
  
• Even if you collect data from all nodes, correlating that data into a single transaction path (or topology of the entire transaction) accurately has yet to be done (if you can give a concrete example, then let me know and I will post it)
  
• Receiving data at the network level makes measuring encrypted data close to impossible
  
• Once an event has begun processing, it cannot be controlled (say for resource allocation purposes)
  

Disclaimer

When trying to give an understanding of a general approach to a solution, all potential advantages and drawbacks (which people who develop or promote the specific solution would prefer to ignore) are listed. Comment with any objections (as people have done in the past) and I will at some point post everything.