After being forced to pen and paper, Powell Valley Healthcare back online

An unknown ‘anomaly’ made data inaccessible

Posted 5/4/21

A computer problem forced Powell Valley Healthcare to unexpectedly go retro last month, requiring the organization to keep records on pen and paper for nearly three weeks.  

On April 1, the …

This item is available in full to subscribers.

Please log in to continue

E-mail
Password
Log in

After being forced to pen and paper, Powell Valley Healthcare back online

An unknown ‘anomaly’ made data inaccessible

Posted

A computer problem forced Powell Valley Healthcare to unexpectedly go retro last month, requiring the organization to keep records on pen and paper for nearly three weeks. 

On April 1, the healthcare organization’s network went down. It was a brief moment, but during that time, PVHC’s virtual servers lost connectivity to their data storage. Afterwards, Powell Valley Healthcare officials were unable to get their financial software to come back up, because a database file had been corrupted by the anomaly and became inaccessible.

No private information was ever at risk or compromised in any way, said CEO Terry Odom, and the issue was not caused by a computer virus or ransomware attack.  

“We’re not sure what caused the network anomaly,” said Josh Baxter, information technology director for PVHC, but “we know it wasn’t a malicious attack or anything nefarious.”

Computer networks sometimes fail, so each department has a “downtime procedure” to follow when such an eventuality occurs. It’s a system PVHC calls “business continuity” and each department regularly submits and updates their plans.  

“It’s business continuity to keep business going, when you have downtime,” Odom explained. 

Jaime Buck, privacy officer for the provider, said typically it’s not every department at once that needs to execute their plan. 

“It took a bit of adjustment for everyone to go to pen and paper,” Buck said, “but we had a plan in place.”

PVHC personnel kept trying to backup the financial data, but every time they tried, the system would collapse. All of their servers work on Microsoft software, so they opened a case with the company’s tech support. Microsoft’s support people said they could try some things to recover the data, but it was possible the tools wouldn’t work and could damage the database beyond repair. 

Since PVHC couldn’t access the data anyway, they didn’t have much to lose. Unfortunately, the tools that Microsoft used couldn’t recover the data, either.

At that point, the hospital began working with its vendor to rebuild the entire system so they could go back to electronic records keeping. In the meantime, they kept using pen and paper. 

“That’s about as old school as you can get,” CFO Joy Coulston told the PVHC board. Normally Coulston gives a financial update to the boards that govern the hospital, but she provided an update on the server situation instead at the April 26 meeting.

By the time of the board meeting, the system was mostly functioning again. The provider started registering patients electronically on April 21. 

“It was 20 days of running on paper,” Baxter said. 

Baxter added that it wasn’t unusual for PVHC to run into computer problems requiring them to execute downtime procedures. It’s just never gone on more than a day or two — and is usually a matter of hours. 

“We’re pretty much getting caught up,” Sandi Lance, patient financial services director, said Friday. “The frustration with the community is they’re wanting to call in and pay bills … we’re slowly getting that back on board.”

People can now make payments through the hospital website, and patients can access their medical records, which were never impacted by the issues.

Meanwhile, PVHC’s vendor is continuing to investigate exactly what caused the problem by going through the entire log files. It’s a tedious process that will take some time. 

“The level of detail they’ll have to go through is pretty extreme,” Baxter said. 

He still believes it was a power issue, possibly related to the construction that’s going on around the hospital. The hospital didn’t lose power, but there may have been a surge of some sort. Until the investigation is complete, they won’t know more. 

“We’re still going to find out; we’re still looking at it.” Odom said.

Comments