BTM vs. Traditional Monitoring

Business Transaction Management (BTM) is an IT systems management paradigm that seeks to manage IT from the business perspective. The transactions that Customers and Employees execute represent the Business. The IT department's number one goal is to ensure that those transactions are being executed in a timely and reliable manner. BTM solutions are able to trace the path of these business transactions throughout the Data Center, across all of the tiers, so that if something goes wrong the exact problematic location is identified instantly.

While traditional monitoring tools seek to monitor the health of individual components, BTM solutions monitor the organization's bottom line – the transaction. The measurement granularity of traditional tools is limited, the can only measure total resource consumption for a specific server, BTM's measurement granularity is a single event which could be a web service call, SQL statement or HTTP request.

Traditional tools monitor performance as server up time, the problem with this is that while "all lights are green" for a specific server, the transactions that are flowing through that server are not necessarily doing so in a timely manner. BTM tools monitor the performance of what really counts – the user's quality of experience.

End to End Transaction Monitoring

The term "end to end" - in relation to transaction monitoring is very over used. "End to end" Website monitoring, "end to end" Server performance monitoring, "End to end" Database monitoring, monitoring traffic, "end to end" Network performance monitoring; the list goes on of tools that claim to provide "end to end" monitoring solutions when in fact they only provide monitoring for one small part of the bigger picture - the diagram below explains it all.

Why True End to End Transaction Monitoring is Different than Monitoring Server Performance

Monitoring server performance traditionally means that you are trying to make sure that CPU utilization and memory consumption are not maxed out, ensuring that resources are not maxing out does not promise true availability of applications. Traditional "end to end" monitoring tools provide a dashboard that shows the health of each one of the individual components within the data center. But what if someone forgot to re-connect a network cable after maintenance or what if a load balancer was configured incorrectly and is not executing a proper round robin? An alert will indicate that something is wrong with the servers that are still connected when in fact the problem is that traffic is simply not being equally distributed. You could implement some sort of network performance monitoring solution which would aid by monitoring the usage of the network, but that is not a single end to end solution anymore.

What if a runaway process in one server is "bombing" second server with requests? The server monitoring solution will blame the poor server that is being bombed by the server which has a runaway process running on it as opposed to indicating the source of the problem.

Real User Monitoring – An End to End Solution?

If you only take the user's tier into account, real user monitoring could be considered an end to end solution for that tier alone. You could also say that with real user monitoring you can see both ends of a transaction – hence the claim to provide "end to end". What about everything in the middle? What action is to be taken when a problem is detected by the end user experience tool? What if the root of the problem resides within the database or the mainframe? Shouldn't an end to end solution be able to take care of everything, including triage within the datacenter? Website performance monitoring is important for understanding the quality of service that your customers are receiving and there is a lot of value in that, but an end to end solution should also help you find the cause of the problem along with simply reporting its existence.

End to End Transaction Monitoring Tools Deliver

What do the following components have in common – the User Desktop, a Firewall, a Proxy, a Load Balancer, a Web Server, an Application Server, a message broker, a Database and a Mainframe? The answer is - you guessed it - transactions. The only way to provide a true end to end solution is by monitoring every transaction from the moment any user clicks any button and all the way through all of the different tiers and only business transaction management solutions can promise that.

Performance monitoring tools that are not showing end user performance are not focusing on what is most important to the business. Website monitoring tools that send synthetic transactions to the website and check response times are not showing what users are really experiencing or monitoring the usage of different services. Database monitoring is important but without knowing the context of that problematic SQL server transaction, resolution can be a shot in the dark.

Transaction Monitoring With Business Transaction Management

End to end solutions must be able to monitor all of the infrastructure's components. Transaction monitoring solutions do that automatically since they monitor the object that ties all of those different components together – the transactions. Business transaction management solutions enable a drill down that begins from each transaction type that is running on the system and ends with the smallest event that composes a single transaction instance. In this manner, not only are you ensuring that everything is running smoothly, but when things start to go wrong you can perform immediate triage and resolution.

Application Performance Management Solutions - Quick Tips

Any organization that is running business critical – time sensitive applications is in need of some sort of performance management system. When looking into what type of performance management tool to invest in - things can get confusing – surfing the web throws you into a sea of empty marketing messages. The label “application performance management” is vastly over used; the following article covers a few things that are not to be missed when considering an application performance management solution.

An Application Performance Management Offering Must Include an End User Monitoring Capability

How else can you know how your application is really performing?

End user monitoring tools provide a wonderful presentation of the bottom line of how your applications are performing. When it comes to web application management there is a long list of network appliances that serve the purpose - giving you a plug and play solution – with no need to install anything on the user’s desktop. But what happens when the end user monitoring tool that you are using for your web application management shows that latencies have gone wild?

You need an application performance management solution that knows how to connect between the latencies that your users are seeing and problem that is causing the latency in the data center.

Now of course in these troubled times the performance management system that you may be looking to invest in must not eat up your entire IT budget. Purchasing more than one performance management tool is simply out of the question and network appliances cannot make that full connection, enabling true application performance management.

An Application Performance Management Tool should aid with Data Center Management

Why settle for an application performance management tool that sees only the application server?

True application performance management cannot be done with your run of the mill server management software; true performance management solutions offer the ability to perform data center management. The need for application performance monitoring tools to provide data center management comes from the complex, distributed and interdependent nature of applications these days. Application performance monitoring solutions must take into consideration the entire data center if they are to perform proper triage of a problem. A lot of data center management tools these days utilize information that is collected from server management software that is installed on various tiers, the problem with this is that you end up collecting a whole bunch of resource consumption metrics that do not correlate to what the end user sees which directly reflects your performance. Not monitoring what the end user sees means that your users could be experiencing major problems with the application which you will only hear about if those users take the time to call customer service. Since there are many cases where server management software shows that availability is fine but transactions are taking too long to process.

Application Performance Management – Monitoring Network Performance

What is your application without the network that it is sitting on?

Monitoring network performance also plays a role in application performance management. Imagine that your end user monitoring tool shows that latencies are too high while the latencies that you measure within your application server are only a small percentage of the total latency that your users are seeing. Only by monitoring network performance will your application performance management tool provide the performance management solution that will cover all of your bases.

Only Business Transaction Management Provides the Single Solution

Monitoring every single transaction from the end user - through the network and all the way to the back end of the data center is the only way for an application performance management solution to cover all of the bases that are listed above. The discipline of doing this is called Business Transaction Management or BTM.

Business Transaction Management in the News

As you may or may not know, many of the major Transaction Monitoring players have their R&D centers based in Israel; Correlsense, Optier, Correlix and B-Hive. Considering the important role that "hi-tech" plays in the Israeli economy, it is no surprise that financial publications publish technology related articles on a regular basis; I have taken the liberty to translate parts of a recent article that was published in "Globes Israel".

"After evaluating both Correlsense and Optier; we have decided to invest in Correlsense"

The article talks about how the IT systems management world is currently going through some "tectonic movements" and that the companies that emerge once the dust clears will have a pivotal role in the market. Similarly, there are forces which are acting on the enterprise computing world in general - virtualization, the large growth in the availability of information and applications within the organization and outside of it and of course, the constant need to reduce operational costs – that are causing executives to think differently.

There are a number of Israeli companies who are in this position; Optier is an example of a company that is trying to become a big player in the IT systems management market. This is also the case with Correlsense, says Sam Somech. Somech is an Angel that invested in Correlsense, along with the VC firm eXeed, and is now serving as the company's chairman of the board.

Both Correlsense and Optier have developed technologies for the troubleshooting and optimization of data centers, with the belief that the best way to deal with the complexities of the modern data center is by tracking transactions from the user, through the data center and all the way to the back end data bases.

"The surviving company will have a high value"

According to the evaluation of Correlsense's investors and founders, the leading IT systems management companies – IBM, HP, BMC, CA or Microsoft – do not have this kind of technology yet. "We evaluated the two companies - Optier and Correlsense", Says Somech, "and in our opinion Correlsense's technology is favorable". Optier had no comment.

Despite the uncertainties, "if a start-up can provide real innovation that will give it the edge in a sale, then it has what to offer the market", says Somech. "The technological gap between what IBM has to offer in relation to Correlsense is large. In this world, he who has the faster, more economical offering is at an advantage. It is not likely that the companies in the market today [the big four] are going to be able to invest heavily in R&D, they have also been affected [by the financial situation] and they are not going to invest tens of millions of dollars when they do not know what will come out of them. Therefore the start-up companies that will survive this downturn will have a high value".

With all due respect to Mr. Somech, the IT systems management market will not wait for Correlsense or even Optier. The Gartner Analyst, Will Cappelli for instance told "Globes" at the beginning of the year that Optier will eventually have to find itself as part of a larger company that will acquire it. Even BMC CEO, Bob Beauchamp advises the companies that are active in the field to go for acquisition, as it is a mature field with no room for niche solutions.

"We don't want to cut our valuation"

"The transaction monitoring market is not mature, it's just starting", replies Somech. Oren Elias, CEO and founder of Correlsense, says "transaction management is a big leap forward. Therefore, in times like these when the "big four" are investing less [in R&D], there is an opportunity for a new corporation to rise up and take business even from the "big four"".

There are a number of players in the IT Systems Management field and the Israeli ones among them especially stand out; Optier, Correlsense, Precise, B-Hive and Correlix. Elias and Somech are convinced that their solution is unique. "The only companies that built their transaction monitoring technologies from scratch are Correlsense and Optier", they claim.

In order to turn into one of the hottest names in software in the world, Optier has raised $116M. Correlsense, still at the beginning of their route, has been satisfied by their $1.5M round that was completed half a year ago from eXeed, Vertex and Proseed and is now in the process of raising a few more million.

"We believe in the company, it's not a good idea for us to go for additional external funding at this point, we don't want to cut our valuation. I think that it is going to be a good investment for our firm, because of the advanced technology. We are looking for just enough money to keep the company going for the next two years at which point the market is going to look much different. At this point there isn't a lot of competition".

Business Transaction Management’s Challenges

Business Transaction Management (BTM) is a natural continuation of the past decade and a half's evolution of IT systems management. In the past few years the modern data center has finally started to stabilize; the number of node types has become constant and each node has had tools developed for it.

These silo specific tools are now able to solve 90% of the problems; leaving us with the hardest to solve - last ten percent. This last ten percent is characterized, for example, by those application bottlenecks that occur even though all of the silo specific tools are showing 100% availability.

If the monitoring tools at all tiers are showing 100% availability then how does one know that there is a problem? Well, either the enterprise has put in place an end user measurement tool or the help desk is receiving user complaints.

The IT organization's number one priority is very simple; ensure that all transactions are executing correctly and in a timely manner – it's that simple.

The demand for Business Transaction Management tools comes from the need fulfill that priority; especially when traditional tools are unable to go that last mile. What has enabled the development of these Business Transaction Management tools is the datacenter's recent stabilization; with the number of node types staying constant for a few years, it is finally possible to catch up and enable the full visibility into the entire datacenter that BTM solutions must provide.

BTM Enables True Availability

The only way to resolve the problems that traditional monitoring tools cannot cope with is to take into account the interactions between the nodes and to connect every click of the user to the many events that are triggered by that activation. This enables both being able to see everything in the business context - the user's click of a button - and solving problems that have more than one source.

Business Transaction Management tools are able to monitor the entire system by tracking every single transaction that is activated by the users throughout the entire datacenter, collecting information on all of the interactions along the way. This is the essence of connecting business and IT; understanding the business context by linking every single event in the datacenter to the click of a user.

With Business Transaction Management, a common language is created, where performance can be measured as the time it takes for the transaction to travel between all of the nodes and back. Since all interactions are recorded along the way, if something goes wrong, i.e. a latency that is out of specification, then the event that caused the problem can be immediately singled out since the latencies at each tier are known and all of the necessary data has been collected. 

What Qualifies as a BTM Solution?

Every Business Transaction Management solution should be able to do the following:

Track a transaction that was sent out from a browser, through a load balancer and to a proxy. This proxy is not a known vendor's proxy, just some start-up's which developed the proxy five years ago - the code is still there and still working, but no one has the code - just the binary.

That proxy is sending it to another proxy and its being split to - making it easy - two different applications, each with its own different web server. Each application is running on its own app server - one is java based, the other is not; it's a homegrown C++ program. The first app server is sending out data base requests to four different databases; Oracle, DB2, Sybase and SQL server. The second app server is SOAP based; sending SOAP requests to an external application. There is an MQ at the external application, 'puts' are being sent to the MQ and a Mainframe is receiving the messages from the MQ. 

If a vendor wants to be called a BTM vendor he has to be able to track the transaction across all of these components. He has to be able to account for the time every transaction spends at every point in the datacenter.

Developing a Business Transaction Management Solution

Developing this kind of solution is far from trivial. The task of connecting each event within the datacenter to a specific transaction is a big challenge in today's distributed and heterogeneous systems.

The general concept of how to execute a Business Transaction Management solution is pretty straight forward. Agents must be installed at all tiers, collecting information about everything that flows through that particular tier and all of the collected data is sent back to a central dedicated server that is able to correlate all of the events to the single user's click of a button in the application. The various methods of gluing these events together are the core competency of the solution, along with the ability to develop an agent that complies with all of the different kinds of servers that one finds in the data center.

Some big players have been claiming to provide these end-to-end solutions for a number of years. Only recently have an extremely limited number of vendors been able to prove Business Transaction Management solutions in production – buyers beware.

CMG '08 Exhibition Photos

Here are some photos of the exhibition area right before opening:

G4 IT Systems Management is Here

Information Technology contributes to the efficiency of the human race like nothing else. Keeping up with the management of complex IT systems requires constant innovation; systems management has come a long way since the days of "big iron". The fourth generation of systems management, widely referred to as "end to end transaction management", enables IT to reach levels of maturity and stability like never before.

Mainframe Systems Management – G1

The Birth of application performance management tools came with the development of the Mainframe Omegamon and Tmon; these tools monitor program names, Customer Information Control System (CICS) jobs and Job Control Language (JCL) tickets. As far as application performance management goes, this was really the first transaction performance management solution; where everything was centered on the mainframe.

Client/Server – SQL Management – G2

Next up was the client/server; back then, your server was essentially the database, the database was the single point of failure, it became the place where all of the data was stored and SQL queries were everything. If you had a problematic SQL query then you had a slow transaction.

That is really when the true application performance management market emerged; with companies like Precise and Quest. The first tools for Data Base monitoring utilized the SQL command line to extract system information. These first application performance management vendors built smart agents that would collect all of the information all of the time and sometimes it actually worked in production. These application performance management tools had a nice (relative to the time) dashboard and suddenly you had this sexy, easy to work with tool for optimizing your database, DBAs finally had something to work with that gave them value. This was all around 1997, when Windows 95 was just becoming stable.

J2EE Profilers and the User Experience – G3

G3 is the generation that is currently coming to a close, as will be explained. This generation began with the emergence of Java Application Servers, Java's rise within large organizations was seen in 1999-2002, suddenly a lot of application vendors started working with Java, they built these applications and started selling them to enterprises and suddenly you had Java Application Servers everywhere. This brought on a whole new black box that needed a tool that could manage its performance – that tool, Java profilers, were brought on by Wily, BMC and others.

At the same time, with the dot com bubble, it became important to track user experience; your website was suddenly your business. With the website so critical to doing business, it was important to make sure that things were performing as expected; running synthetic transactions with Gomez, Mercury Topaz, and others in order to ensure availability was the way to do it.

Alongside the JVM profilers and user experience tools, there were always network probes that made sure that the network was working. As TCP/IP became the standard, you had solutions that were parsing and sniffing these protocols, parsing network traffic in order to monitor network performance.

What Marks the End of 3G Systems Management?

In the last couple of years the last two major silos that we have not yet listed; Middleware and Storage, have had tools developed for them. For example; companies like Onaro (acquired by NetApp) developed tools that extended application performance management into the storage tier (Application Insight) and MQSoftware. Today, every silo has its performance management tool, all with the objective of achieving greater application performance.

All of this happened over the past 5-6 years; every single silo was developed on its own and was embraced by the industry. What we see happening in the past two years is some kind of stability, we do see new concepts for application servers, other languages - not just java based, and we see a lot of integration - web services is the new buzz word - really it's just an easy way to do integration (enterprises where doing http based xml ten years ago). All of these changes not withstanding; you do not see new silos. You do see more and more silos, but not different types of silos.

Stability & Maturity => 4G (End to End)

The market has matured to a level where you can deploy end to end solutions, if you would have tried a few years ago you would have run into complications, right now there is enough stability in new technology as far as enterprise applications are concerned in order to enable the proliferation of end to end management. Of course there is always constant change and progress in things like storage and network throughput, these technologies will continue to move forward but the technology for tracking a transaction is here to stay for the foreseeable future. TCP/IP is going to stay with us for the next 10-20 years, HTTP protocols aren't going to change in the next 5-10 years, Java will not go away as an application server, there will be new methods for implementing an application server - it will not necessarily be JVM it could be a new process – but eventually it's going to be some sort of process, multi threaded or not, which will handle the transaction. The overall architecture of the application is going to stay more or less the same; a rich, mixed and complex topology that is hard enough to manage as it is. This maturity is exactly what is enabling transaction management to become the fourth generation of IT systems management.

4G Systems Management – Bringing IT all together

Forget the buzz words for a moment - Business Service Management, Business Transaction Management, Service Level Management and others - what we are talking about here is the evolution of IT systems management itself - which is defined as everything that is needed in order to make the IT data center work. Transaction management is going to compose 70% of the systems management space – it will be critical (and already is) in making sure that your datacenter actually works.

A big thanks to Doug McClure of for creating an informative podcast with Lanir Shacham from which the ideas for this article were taken. Take time to listen to Doug's Podcast which aims to shed light on the next generation of IT systems management.