Healthcare Interests

Monitoring Patient Record Access Performance: How Partners HealthCare Monitors and Streamlines

To allow for greater distribution of clinical information to hospitals, clinics, and individual physician offices, Partners HealthCare moved parts of its Longitudinal Medical Record (LMR) application from a Visual Basic desktop application to the web environment. The conversion significantly increased resource utilization from individual desktops to a web server tier. Partners HealthCare now averages seven to ten million web transactions per day, which are mostly the result of over 6,000 clinicians, physicians, and nurses retrieving information from the LMR and many other clinical applications. The LMR provides all medical, demographic, prescriptions, notes, and physician information for each patient as far back as their original records. Nothing is archived, and all information in patients' medical records is accessible online, in real-time, to the Partners HealthCare medical community.

Efficiency is key when tracking millions of web transactions, and it was essential for Partners to be able to analyze and monitor activities to identify potential problems. Strained systems, brief outages, and delays in accessing clinical information were considered unacceptable.

Five production web servers providing clinical information connect to an application server tier and a database tier - Intersystems' Cache. This complex multi-tiered infrastructure made it difficult to define where problems were originating, and how to create a baseline to show increases in utilization. Partners needed a solution that would show when and why more resources were being used in certain areas by monitoring the clinical applications within the web server environment in both the production and QA platforms.

The Solution

Partners extracts web data elements as objects from Intersystems' Cache post-relational database through ODBC. So, for each of the 7-10 million web transactions per day, the solution collects data such as username, IP address, location, machine name, application name, function (sub-routine within the LMR), various time intervals, network time, database time, render time on the PC, web link time (the name of the DLL that is used to connect to the database), workstation number, the building, the floor - even down to the pod that the PC is in. Once Partners objectified each of these data elements, they were able to instruct the Diver Solution to summarize, index and preprocess the input data stream into a multidimensional data structure that is optimized for query, analysis, and reporting purposes.

Intersystems and Dimensional Insight are united in the view that an object-oriented development environment consistently delivers higher business value than traditional approaches. The Diver Solution's ETL (Extract, Transform and Load) tool represents both data sources and data transformations as objects, enabling the foundation for business intelligence applications to be rapidly created.

Not only can Partners now identify LMR transactions, what machines the requests came from, who the request came from, and metrics on how many reads and writes they all do, (data reads and writes and routine lines executed), they can also analyze individual sub-components of the applications and create individual application metrics. For example, within the LMR application, they can see the page or pages that a request came from, and break out utilization into sub-components
within the application.

"Suddenly we had this reporting capability where we could see average utilization of any metric, whether it be users or routines, or global reads and writes, and we could quickly identify anomalies. We have many "airline screens" and tools to monitor the machines, so when we found, for example, that resource utilization was up, we had a hard time figuring out what specifically was causing the increase. With the DI solution we can now get real-time results and begin to look at individual transactions over a crisis period to see what's changed and what we need to do to correct the problem," says Larry Williams, Senior Corporate Team Lead.

Real Time Results

An automated daily compile is scheduled, and weekly and monthly aggregate compiles run at midnight for the previous day. If necessary, Partners will generate a compile for a part of the day if they notice irregularities.

"We may go through an automatic hourly quick compile of the day's information because it would reduce the time for the day to only the last hour if we needed to deal with a crisis. Within ten to fifteen minutes, depending upon the data elements we're collecting, we'll have the information necessary to correct the issue," explained Williams.

For example, one Monday morning, staff noticed a significant increase in web transactions on the production web servers. By analyzing the data with Diver, they were able to identify that the increase started at 10 am on Saturday morning when web transactions went from 7 million to 25 million. "We had absolutely no idea why, and if we didn't have Diver we wouldn't have even known it was happening," said Williams. By analyzing baseline transaction volumes, various time intervals, and user counts, Partners was able to identify the application that was causing the spike in web transactions.

By using the Diver Solution, members of the support staff were able to identify the application that was causing the increase, and pinpoint exactly nine PCs that were caught in an error loop, generating thousands of transactions per second. Partners was able to control these PCs remotely and eliminate the problem before it caused a real performance issue within Massachusetts General Hospital, the OEM, and the LMR application.

"It was a real eye opener for us. It is a great example of the power of the Diver Solution that we had implemented. Without DI we would have never been able to figure out what the problem was and where it was. Nobody had even complained of slowness yet. We were finally able to identify an issue before it became a real problem," stated Williams.

The Diver Solution's Unique Value

With over 150 developers on staff, Partners does most of their application development in-house. They were not an organization that wanted a team of consultants to set the system up and pump out generic information, because it was not only to be used for infrastructure personnel to monitor web transactions, but also for future clinical applications. "We're a hands-on team, so we wanted the ability to learn the software, figure it out ourselves, and make any changes we needed, and we saw that flexibility with the Diver Solution. We were also looking for a reporting and monitoring tool that would only require part time resources of one person. All other products seemed like they would need three to four people working on it all day, every day," states Williams. Partners perception was correct: only one part time person is necessary to perform all configuration and maintenance of the Diver Solution.

The benefit of representing data sources (Cache) and data transformations (DI's business intelligence solution) as objects has two major benefits: the cost of developing applications reduces in line with the development time required, and shorter development time accelerates the process of getting users the answers they need.

About Partners Healthcare

Partners Healthcare, based in Boston, Massachusetts, is an integrated health care delivery system founded by Massachusetts General Hospital and Brigham and Women's Hospital. The system includes primary care and specialty physicians, community hospitals, the two founding academic medical centers, specialty facilities, community health centers, home-based services, and other health-related entities. In addition, Partners is one of the nation's premier biomedical research organizations and is a major teaching affiliate of Harvard Medical School. Partners HealthCare is a non-profit organization.