Free Wired EMR Practice Newsletter Want to receive the latest updates on EMR from a Doctor's perspective sent straight to your email? Get all the latest EMR and EHR updates from a practicing doctor for FREE!

Lessons Learned from our EMR Upgrade – Part 3

It is after 11 PM and I have just arrived home after a meeting with our practice leadership.   Why so late?  The meeting doesn’t start until 7 PM.  We docs can’t afford to take time out of our practices to meet during the day.  We moonlight as CEOs, CIOs, managers, etc. for our own practices.

This was the first meeting since March that was not dominated by unhappy discussions about the system upgrade.  It wasn’t even mentioned.  Tonight’s EMR discussions were forward looking, including e-prescribing, which just went live for us yesterday, and the pending results of our meaningful use gap analysis that will come out next week.  I think we have reached an appropriate point to take some perspective on our difficult upgrade.

To state the obvious first, we bit off too much at once.  Going 6 years without a software upgrade is bad enough.   But doing a major database conversion at the same time?  And buying all new servers?  And switching to VMware?  What the heck were we thinking?

As I mentioned yesterday we were afraid of using the database merge program (a.k.a. the migration tool) on our precious database until the vendor got more experience with it.  We also thought it was a reasonable strategy to feel all the pain all at once rather than spread it out over several smaller steps.  Regarding our 6 figure server purchase we were trying to cheat the old rule that any computer you buy will be obsolete by the time you get it home and plug it in.

In retrospect those were all good thoughts.  They just weren’t enough.  We failed to realize that while the migration tool was getting better through time, our database and applications were at the same time getting bigger and more complicated.  Every year we added an average of 50,000 new patients to our database.  We also added applications like our web portal and more automated document scanning / indexing.  Time also allows strange things to happen…such as when one office accidentally started scanning clinical documents into the practice management database.  Tens of thousands of documents were in the wrong place.  We picked up on it ahead of time and thought we had fixed it but the migration tool still had a problem with those image files.  Sometimes I wonder if we should have upgraded sooner and taken our chances with a less mature migration tool running on a smaller, less complicated, less entropy-riddled database.

The upgrade was harder and far more stressful than the original implementation in 2005.  I think this was because we no longer had paper charts as a lifeboat when the system wasn’t working well.  The gradual, no-hassle approach to EMR implementation that I wrote about months ago is not an option when you are switching databases.  I have a new found respect for practices that are forced to switch EMR programs.

VMware was a much bigger hassle than I expected.

When one considers that the upgrade occurred at the end of 6 years of relatively hassle-free system performance it really wasn’t that bad.   But it sure felt bad at the time, not knowing when or if we were going to get the bugs fixed.

 

Post a Comment(5)
June 29, 2011 I Written By

Dr. Michael J. Koriwchak received his medical degree from Duke University School of Medicine in 1988. He completed both his Internship in General Surgery and Residency in Otolaryngology-Head and Neck Surgery at Vanderbilt University Medical Center. Dr. Koriwchak continued at Vanderbilt for a fellowship in Laryngology and Care of the Professional Voice. He is board certified by the American Board of Otolaryngology-Head and Neck Surgery. After training Dr. Koriwchak moved to Atlanta in 1995 to become one of the original physicians in Ear, Nose and Throat of Georgia. He has built a thriving practice in Laryngology, Care of the Professional Voice, Thyroid/Parathyroid Surgery, Endoscopic Sinus Surgery and General Otolaryngology. A singer himself, many of his patients are people who depend on their voice for their careers, including some well-known entertainers. Dr. Koriwchak has also performed thousands of thyroid, parathyroid and head and neck cancer operations. Dr. Koriwchak has been working with information technology since 1977. While an undergraduate at Bucknell University he taught a computer-programming course. In medical school he wrote his own software for his laboratory research. In the 1990’s he adapted generic forms software to create one the first electronic prescription applications. Soon afterward he wrote his own chart note templates using visual BASIC script. In 2003 he became the physician champion for ENT of Georgia’s EMR implementation project. This included not only design and implementation strategy but also writing code. In 2008 the EMR implementation earned the e-Technology award from the Medical Association of Georgia. With 7 years EMR experience, 18 years in private medical practice and over 35 years of IT experience, Dr. Koriwchak seeks opportunities to merge the information technology and medical communities, bringing information technology to health care.

Filed Under: EHR  Electronic Health Record  Electronic Medical Record  electronic medical records  EMR  Meaningful use  Upgrades  VMware  workflow
Tags:                  

Lessons Learned from our EMR Upgrade – Part 2

As I discussed in the last blog post we were very busy this past spring with our biggest EMR upgrade to date:

  1. Upgrade from the 2005 software to the 2010 version (2011 upgrade delayed on the advice of our VAR), making a big jump.
  2. Purchase new servers and new memory (SAN)
  3. Switch to virtual servers / VMware
  4. Convert our database from 2-database structure to single database to accommodate the 2010 software.

This was more of a system replacement than an upgrade.  The only parts that weren’t completely replaced were the network components and some peripheral applications (web portal and document scanning).

Despite realistic expectations the upgrade took longer than expected.  Some problems took many weeks to solve.

  1. Despite a successful test run, the dual-to-single database conversion was fraught with problems and took longer than expected.  The computer that was running the conversion software (called a “migration tool”) had a RAM failure during the operation, which slowed the conversion down but didn’t kill it.  When we saw the operation slow down we had a dilemma – do you stop to troubleshoot or let it keep running slowly?  We have over 250,000 patient records in our database so the conversion was expected to take well over 72 hours – longer than a weekend.  That meant we were already looking at EMR down time during office hours.  We stopped the migration to diagnose and replace the RAM.  Then the migration tool itself failed, forcing another interruption and requiring our vendor to troubleshoot and patch the migration tool.  The migration tool is an unusual piece of software.  You only need it once so about the time you have learned to use it you don’t need it anymore.  On the vendor side, every customer’s database / hardware situation is different, so the migration tool is never totally debugged.  That is why we delayed our upgrade so long – we wanted the vendor to gain some experience with the migration tool before we used it.  We were still by far the largest database conversion they had ever done.  In spite of the difficulties the result was an intact single database that gave us no further trouble once the migration was completed.
  2. Another contribution to our delay in upgrading was waiting for our vendor to support VMware and give us hardware specs.  Even with that accomplished VMware was a nightmare to set up.  Performance was very slow initially and took days to correct.  The biggest problem was the printers.  Printer preferences were lost several times a day and it was not unusual for my documents to get printed at a member practice across town despite having reset my printer preferences several times that day.  That wreaked havoc on clinic operations and took over a month to fix.
  3. We were blindsided by a bizarre “failure” of a T1 line to one of our offices.  The line was somehow put in some sort of diagnostic mode, rendering it unable to function but showing it as normal to our monitoring.  For days we assumed that office’s performance problems were related to the upgrade.
  4. Some issues were purely our fault.  We did not adequately staff our upgrade operations.  We had only our chief operating officer and our IT specialist to handle problems and questions; they couldn’t get off the phone long enough to fix anything.  This also impaired communications significantly.  To make things worse each of them had immediate family members become suddenly ill, requiring that they take some time off during the upgrade.

The next post will be my analysis of this great adventure.

 

Post a Comment(2)
June 28, 2011 I Written By

Dr. Michael J. Koriwchak received his medical degree from Duke University School of Medicine in 1988. He completed both his Internship in General Surgery and Residency in Otolaryngology-Head and Neck Surgery at Vanderbilt University Medical Center. Dr. Koriwchak continued at Vanderbilt for a fellowship in Laryngology and Care of the Professional Voice. He is board certified by the American Board of Otolaryngology-Head and Neck Surgery. After training Dr. Koriwchak moved to Atlanta in 1995 to become one of the original physicians in Ear, Nose and Throat of Georgia. He has built a thriving practice in Laryngology, Care of the Professional Voice, Thyroid/Parathyroid Surgery, Endoscopic Sinus Surgery and General Otolaryngology. A singer himself, many of his patients are people who depend on their voice for their careers, including some well-known entertainers. Dr. Koriwchak has also performed thousands of thyroid, parathyroid and head and neck cancer operations. Dr. Koriwchak has been working with information technology since 1977. While an undergraduate at Bucknell University he taught a computer-programming course. In medical school he wrote his own software for his laboratory research. In the 1990’s he adapted generic forms software to create one the first electronic prescription applications. Soon afterward he wrote his own chart note templates using visual BASIC script. In 2003 he became the physician champion for ENT of Georgia’s EMR implementation project. This included not only design and implementation strategy but also writing code. In 2008 the EMR implementation earned the e-Technology award from the Medical Association of Georgia. With 7 years EMR experience, 18 years in private medical practice and over 35 years of IT experience, Dr. Koriwchak seeks opportunities to merge the information technology and medical communities, bringing information technology to health care.

Filed Under: EHR  Electronic Health Record  Electronic Medical Record  electronic medical records  EMR  Upgrades  VMware  workflow
Tags: