Mallinckrodt Pharmaceuticals Upgrades Control System

Background

If you've ever been hobbled by a broken bone or endured an MRI, you probably experienced the products made by Mallinckrodt Pharmaceuticals. The $5 billion global company produces specialty pharmaceutical products and diagnostic imaging agents. As the nation's leading supplier of prescription pain medication, Mallinckrodt sells its products to major wholesalers and retail drugstore chains. Its focus on pain control helped transition the company from solely a drug provider to a manufacturer of equipment used to identify the source of the pain.

Located on the company's 40-acre campus in St. Louis is an active pharmaceutical ingredient (API) plant that makes ingredients used for narcotics and other final salable products. The FDA-validated facility is over 145 years old and employs approx. 450 people.

The plant's control equipment was installed in 1985, and costly to maintain. Mallinckrodt needed to upgrade the aging technology and bring the plant to a common control-system platform. Cost and operational efficiencies were more important than ever since the company went public in 2013 and was now accountable to its shareholders. Ultimately, a control system upgrade meant Mallinckrodt could evolve its business and realize improved uptime.

Challenge

For system operators, one of the most challenging aspects of relying on an outdated system was the limited support. Of all engineering employees at the plant, only two understood the programmable logic controller (PLC) technology running the equipment. Service and vendor support was minimized, as was the availability and cost of spare parts. The result? Longer and more costly downtime.

“Cost avoidance was a huge motivator in updating our system,” said Don Geers, senior maintenance engineer, Mallinckrodt Pharmaceuticals. “The downtime associated with hardware failure was a major concern. We wanted to avoid these issues by putting in a newer control system that's easier to maintain with improved parts availability.”

The old technology did not allow maintenance engineers to perform predictive maintenance. The daily production reports would pinpoint problems, but only after they had occurred. With reactive maintenance as their only solution, unplanned and longer downtimes were inevitable.

Realizing the cost of downtime is greater than the cost of change, Mallinckrodt decided to upgrade its control system. Replacing the brains of a manufacturing process in a 24/7/365 facility requires careful planning. The team wanted to limit transitional downtime and make sure the new system would run efficiently – and at full capacity – as soon as it was brought back online.

Solution

The team opted for a virtualized architecture. Mallinckrodt decided to complete the upgrade by replacing the legacy PLCs with a PlantPAx^® distributed control system from Rockwell Automation built on an existing VMware virtualization infrastructure. The virtualized PlantPAx platform enables plant wide multi-discipline control, and eliminated the number of physical servers needed to run the batch and HMI software. This greatly reduced the maintenance required for servers and workstations.

The system design was completed in two phases to minimize downtime. The first phase – handled by the Mallinckrodt maintenance team, with engineering from Rockwell Automation and local systems integrators Stone Technologies and Syntronix who were involved in developing the programming standards. Once finished, Rockwell Automation completed the documentation and the functional and design specification. Then the team built, tested and validated the programming code. This standard code will be used on the next phase, as well as any new projects being implemented in the facility.

During the development phase, Rockwell Automation had a duplicate system in its Cleveland facility to help troubleshoot and finalize the programming before start-up. Mallinckrodt retained that service in order to call on technical support engineers for future assistance should issues or system changes arise.

In phase two, updates were performed on the functional specifications utilizing the new programming standards. Leveraging the existing production line documentation helped to reduce project risk. It also helped ease the transition to the new system since the plant engineering, quality and validation team members were already familiar with the documentation. This reduced time and cost in system testing and validation.

Factory acceptance testing (FAT) was completed at the Mallinckrodt facility with the help of local system integrators and Rockwell Automation. It took three months to complete the FAT. Then the system was installed in the operating unit where multiple water batches were run to assure all system components operated properly before adding the expensive chemicals used to produce the pharmaceutical and imaging ingredients. Once testing proved successful, Rockwell Automation and Syntronix engineers completed the start-up on site.

As part of the PlantPAx system, reporting and historian software were implemented for improved production intelligence. The software collects, tracks and records key process data to pinpoint production trends, allowing for proactive maintenance activities. Now equipped with validated, electronic reporting capabilities, Mallinckrodt is better equipped to meet regulatory reporting requirements and streamline overall operations through daily electronic production reports.

In addition, Mallinckrodt upgraded its HMI operator terminals. New HMI graphics were re-drawn from the legacy HMI at the facility, and consisted of 140 screens of process graphics for the new API line. The goal was to keep the operations interface similar to the existing equipment in the plant so that operators could easily transition to the new system when it came online.

Results

In the end, commissioning only required five weeks, minimizing system downtime for the install, thanks in part to smart engineering, migration tools and comprehensive functional testing.

“The new PlantPAx system has provided greater diagnostic/troubleshooting information into the control system hardware.” said Geers. “Having a common platform for control and information has also allowed our maintenance personnel to transition from expensive, reactive maintenance to proactive maintenance that was not possible with the old technology. This will reduce our downtime further as we move forward.”

The redundant virtual infrastructure, located in separate buildings, provide an added layer of protection, preventing a catastrophic failure in one location from shutting down the entire system. In addition, a virtualized platform consolidates islands of information. In the event of a system failure on the data center, the Mallinckrodt team will use the backup. They can also run the backup during scheduled maintenance outages, enabling the production line to keep running.

If a failure occurs, Mallinckrodt estimates that virtualization will save at least eight hours – and as many as 24 hours – to re-start the system and get the line back into service. With three separate facilities involved--one acting as the main power supplier for all other buildings--virtualization provides a quick disaster-recovery option.

From a maintenance perspective, the control-system upgrade was deemed a success, but the benefits extend far beyond the maintenance aspect. Mallinckrodt's environmental health and safety (EH&S), plant-floor, production, and leadership teams now have access to continual process data via the PlantPAx process historian. This allows them to more easily meet regulatory reporting requirements, access production information in real-time and initiate action if required.

In fact, operators and managers across the campus are benefiting from new levels of access to system data. Data from the historian software is accessible to anyone in the facility with Excel^® spreadsheet software, and can be analyzed from any building on campus. EH&S professionals can now analyze real-time data to help ensure hazardous chemicals are not released into the environment. Operators entering production data on the HMI can now generate an electronic batch record, reducing the risk of error in compliance reports.

Daily production reports provide engineers and managers with correlated process data, collected from batches completed over several days, to quickly identify trends and institute changes to streamline production or to pinpoint potential bottlenecks. This enables Mallinckrodt to predict potential problems, respond faster, and keep consistency and quality high.

“When Rockwell Automation originally approached us about the upgrade, their team consisted of engineers who work with life sciences customers,” said Geers. “Having a dedicated team that understood our pharmaceutical syntax and vocabulary reassured our decision-making team and allowed us to meet an aggressive timeline with minimal disruption and problems.”

While it may have been a maintenance issue that prompted the control system and reporting upgrade, the entire operation has reaped the benefits associated with a better operating system and improved access to production data. Mallinckrodt now has the ability to build tools for all engineering departments and to benefit from a control system infrastructure that empowers the entire team.