Over the past couple of weeks, TherapyNotes has been experiencing technical difficulties that resulted in slow performance, error pages, and a couple of outages. Resolving these issues have been our top priority. Initially it was very difficult to track down the causes of the problems, so we have extended great efforts to leave no stone unturned, reviewing the web servers, the database, the networking equipment, and the TherapyNotes software itself. We have made a lot of improvements that I would like to share with you.
During these investigations, the IT team has put many improvements into production including our super-fast new database server, updated database driver software on the web servers, optimized firewall configurations, and improvements to our server monitoring and load balancing infrastructure. TherapyNotes is now noticeably faster due to the new database server, and we are very pleased with the results of all of these changes.
At the same time, our developers have carefully examined our TherapyNotes code to see why we have been experiencing poor performance. We implemented new debugging and load testing tools to allow us to better monitor the application. Using these extremely helpful tools we have been able to find the bug that caused the 1/16 outage, as well as other issues.
Since TherapyNotes 4.32 was released last week, we have been holding our breath watching TherapyNotes performance, and we are very happy to report that it’s been running extremely well, with lower CPU utilization and no concerning errors being logged. We were very pleased with the results of the Development team’s efforts and the current state of TherapyNotes software. Plus, our new debugging tools, load testing processes, and infrastructure changes made by the IT team means moving forward we will be able to troubleshoot issues and respond to future issues much faster, or avoid them all together.
The Friday outage we experienced came at a bad time for both team morale and heightened customer frustration. It was not related to the issues we’ve been tackling for the past couple weeks. The database server needed to be restarted for no obvious reason. We are carefully investigating why this occurred, but we do not believe it will be a recurring issue.
We haven’t previously announced this, but a project is already underway to switch to a new database platform entirely. This will open up some very exciting ways to improve the performance and reliability of TherapyNotes. We will provide more details about this in the future.
We are working very hard investigating how we can further improve our infrastructure to further reduce the likelihood of outages. Although entirely unrelated, we are pleased to announce that we are transitioning to a new data center on 2/11. Read this article for details about these exciting upgrades. The new data center is more modern, and we are excited about this move. The timing is only coincidental, as we have been planning this migration for several months.
I personally apologize for these difficulties, and we greatly appreciate everyone’s patience and understanding.
