This week, a new data manager went live in the lab. It’s not important which as what I want to discuss would apply to loads of them. This particular one was in a Blood Science lab but the thought process would also apply to certain Microbiology analysers. Not sure about the Cell Path.
Instrument connections would generally carry orders and results. If there is a single connection, only one thing can happen at any one time. So all the time orders are being sent, results are held back and all the time results are being sent, orders are held up. This is generally okay for lower volume analysers as the frequency or orders and messages doesn’t cause a problem.
When you start hitting the volumes seen in a data manager handling full blood counts and general Biochemistry, it then does becomes a problem. To prevent this issue, a dual connection is used. So orders are sent via a dedicated connection and results are sent back on a different connection. Think of it as a narrow country lane versus a road with a central reservation.
Under no circumstances should anything new just go into use without testing. From a LIMS point of view, you might have a dedicated version of the LIMS to perform testing of new things to ensure everything is working okay before it is used in a production environment.
The scale of duplication of testing and production environments might vary between hospitals but it is also good to have a test system to try things out in before real world use. Which brings me back to the go-live.
The data manager had been tested in the test environment and everything was ready to go live. The first samples were loaded and the order channel was monitored but no orders were being seen. The data manager was happily reporting it was connected but wasn’t seeing the orders.
When you have two systems and you switch over, make sure you connect to the right one! As the testing connection was still running, it latched onto the data manager meaning that the production connection couldn’t connect. It should be a quick fix to rectify this kind of issue.
Then when production was indeed connected to the new data manager, things weren’t running smoothly. Orders were being sent on the results channel and the data manger was getting confused as it wasn’t expecting to get orders from there.
This is a downside of having a complete duplication of production and Live, it means that everything has to be configured twice and allows room for mistakes to creep in. Or potentially problems noted during testing and fixed in testing have forgotten to be applied to the production environment.
Even if testing has gone smoothly, never just assume it will just work in production. Never just assume that when you have made a connection, it definitely is connected to what you think it is. So in this case, when the new connection from production was started, the testing connection should have been stopped so there was no chance of it establishing a connection to the data manager.
Also, make sure the settings in testing match what they are in Production. This step needs some thought as there will probably be some differences e.g. testing IP address could be different to production. So note any differences and think are they justified. Also remember to check spellings. An instrument called ABC in one place but called ACB in another wouldn’t be seen as being the same thing and may cause issues.
Sometimes, no amount of testing will ever guarantee a problem free go-live as you are never able to fully replicate the real world in testing. What you can could is eliminate the predictable problems to reduce the scale of problems. Make sure you have tested the different situations you would expect to see in testing and see how they appear in the downstream systems.
Most problems can be easily fixed. The challenge is spotting where the problem is and finding the right person who can fix it!