Trutegra Service Agreement Success

A National coatings company, that manufactures coating products for a variety of industries, was faced with an emergency following a power outage, leading to a product line failing to startup. Engineers from Trutegra used a computer to go online with the PLC to identify the piece of equipment that had failed to start.

A subsequent audit of the existing control system identified numerous critical failures and fault conditions that had been poorly addressed over the years. It was pointed out to the company that this neglect could result in a major shutdown of the plant for an extended period, the possibility of which the site personnel appeared to be unaware. A total upgrade of the entire SCADA system was recommended to rectify the problems and Trutegra provided a proposal to address all the concerns, highlighting the risk to key decision-makers if the upgrade was not performed.

It was emphasized that the failure to carry out preventative maintenance of routine software updates on the SCADA system had led to the existing situation. Furthermore, it was revealed that the core database, which housed all the product recipes together with batch/lot information, resided on a server with failed hard drives and the absence of any backups.

Implementation of the upgrade faced several challenges making it far from straightforward:

1. Forensic data recovery needed to be performed on the failed drives to restore critical required information.
2. The lack of any working backup, code determination, a source for custom compiled functions, or diagram of the existing architecture, represented a major difficulty.  Thus, Trutegra reverse engineered the entire system in order to gain the understanding necessary to outline the required fix.
3. A major difficulty in implementing the solution to the problem was establishing and adhering to a plan that allowed both the old and new system to work in tandem, avoiding any stop in production.
4. No downtime could be allotted for the upgrade making it necessary to perform the upgrade live while running production.
5. An immediate stop-gap solution was required to allow products to be run through the line manually.
6. Some decision-makers could not see an immediate return on the investment and needed to be coached as to the risks of doing nothing.

The completion of the upgrade has led to flawless operation of the new SCADA system.  An off-site backup routine was implemented to ensure the integrity of the database in the event of a failure, as well as more detailed fault/diagnostic information for the operators to aid in troubleshooting. The company now has a deep archive of historical information that can be searched in the event of quality problems in the product. Root cause analysis can now be performed to determine the process parameter that was out of spec, leading to a higher quality product and a higher yield.

To learn more about our Service Agreements, click here!