High availability is important for all organizations in the process industries, but for a company tasked with supplying an essential vaccine to the U.S. government, high availability simply is not optional.
For Emergent BioDefense Operations Lansing LLC, a subsidiary of Emergent BioSolutions Inc., the ability to produce and deliver high-quality product on time is a matter of national security. Emergent produces and supplies the anthrax vaccine, BioThrax (Anthrax Vaccine Adsorbed), in the United States. Because BioThrax is the only U.S. Food and Drug Administration-licensed vaccine for the prevention of anthrax disease in both a pre- and post-exposure setting, it is essential that Emergent maintains the ability to deliver a reliable supply to individuals whose occupation or other activities place them at high risk of exposure to the disease.
To maintain its essential commercial license, Emergent facilities, employees, and equipment face strict compliance regulations. To facilitate compliance, availability and reliability of its control system is essential. The company cannot afford unexpected problems that cause data collection or production outages. To maintain uptime, automation engineers keep a robust development server environment running so that they can test any changes to the system before implementation, to ensure that software upgrades and changes won’t cause unanticipated problems.
The case for change
For years, Emergent had been relying on a DeltaV distributed control system (DCS) from Emerson Automation Solutions. The engineers’ familiarity with the system made it a critical part of their everyday functionality. When version 12 was released, it contained several features that the company found advantageous. Normally, they would download the software, perform the upgrades and validate the changes; however, their server racks were filled with legacy equipment that was unlikely to be able to support the newest version of their control system. Moreover, the Windows operating systems used were no longer supported. With system requirements chaining hardware and software together, they found themselves running servers (some operating as workstations) with operating systems that were obsolete.
In a sense, they felt trapped: The old hardware could not be replaced because new equipment wouldn’t run the required operating system for their control system. On the other hand, they could not update the software, because it wouldn’t run on the old operating system. As a result, they found themselves making costly repairs to old, outdated equipment, spending far more effort than necessary to keep the servers and workstations running.
The automation engineers concluded the only solution was to change the hardware and the software at the same time, but this came at a cost. Making such a sweeping change meant taking the control system offline to make the switch. This would result in costly downtime for the organization. With proper planning, they knew the outage was something they could manage, but a problem remained: Were they going to have to face such an outage every time they wanted to upgrade hardware or software? They knew they had to find a better way.
Finding a solution in virtualization
Initially, they planned to do a like-for-like swap on all of the hardware and then upgrade the software at the same time. It seemed that swapping the hardware and software in this manner would be the fastest solution, and would, therefore, save the most money as a result of reduced downtime during the upgrade.
When they priced out their options and discussed their plan with Emerson, a virtual environment was recommended. The control system network had reached the size and scope where a virtual solution was effectively the same cost as a standard hardware solution. A virtualization solution offered many features needed, but that weren’t available in a traditional server environment. By choosing a virtual solution, they could meet all of their needs effectively at the same price point.
The benefits of virtualization
Automation engineers said they immediately saw the benefits of a virtualized control system network. The most important advantage offered by virtualization was the addition of high availability to the control system and historians. When a hardware-based server fails, redundancy is limited or nonexistent. Everything shuts down while a new hardware server is rebuilt. In addition to the time that it takes to get new hardware up and running, simply waiting for new parts or machines to arrive can lead to lengthy outages, resulting in failure to meet customer commitments and lost revenue.
With the new virtual solution, workstation and server failures are a lesser concern, as virtual machines can be easily recreated in minutes on another host. Because machines can be moved automatically by the high availability management software with just a reboot of the server or computer, users should see little or no downtime when a server is moved between hosts, as opposed to the hours-long outages that could occur with traditional hardware servers.
Virtualization also provides access to a snapshotting feature that allows end users to take a picture in time of any server state. The snapshot can be used to several different outcomes. If users suspect that an impending change might cause a problem, having a snapshot allows them to instantly roll back any changes, returning everything to a working state. The snapshot feature can also be used for development environments, for example, moving an image of the current system to new hardware, where users can make changes or implement patches or upgrades, and see what impact the change has on the environment in a risk-free development state. Now, nearly every change Emergent engineers make is implemented only after it has been tested on a snapshot running on their development server.
In addition, hardware and software are decoupled going forward, therefore, eliminating compatibility problems in future upgrades. Hardware and software upgrades will be cheaper and easier in the future as well. Software can be upgraded without changing the hardware, and vice versa. Moreover, all of these changes can be made without taking the servers offline, avoiding production outages during upgrades.
Another benefit to virtualization is that it has dramatically reduced the organization’s hardware footprint. The original configuration had 10 operator workstations that ran on physical server hosts, accessed from KVM switches in the field. In addition, there were seven other servers. These 17 total servers took up four full-size racks, while consuming a great deal of power and producing significant heat (resulting in increased costs to cool the server room). With a virtual solution, more machines were able to be run on only five physical server hosts and only two full-size racks, with field operators accessing their workstations on virtual machines using thin clients in the field.
Everyday successes
The new virtualized system has proven successful in day-to-day operation of the plant. The redundancy of the built-in high availability has been a key benefit on the plant floor, as operators know they can now rely on the applications they need to be available at all times. As a result of comparability and trial runs using the new system, Emergent estimates that the effective elimination of unplanned outages will save tens of thousands, if not millions, of dollars in lost production. In addition, the time saved from having employees troubleshooting and documenting hardware failures will save thousands of dollars, as will removing the need to source equipment to repair aging hardware.
Not all the benefits of the virtualized system are quantifiable. In one instance, a workstation was experiencing significant technical issues. The problem could not be replicated on a test system, and configuration changes on the live system had no effect. Fortunately, operators had access to the snapshot feature of the virtualized environment, which allowed them to quickly and easily revert to an earlier snapshot that was validated and documented, allowing them to move forward without additional testing and documentation.
In addition, whenever Emergent or a vendor makes changes to the production system, they always do a dry run of the changes on a snapshot on their development system, and run all validation protocols before anything goes live. By keeping snapshots on the development system that are as close as possible to the production environment, operators can be confident that any implemented changes or updates will work.
Why virtualize?
Emergent’s move to virtualization has delivered a number of significant benefits for day-to-day operations. On the production floor, they can see the results, as uptime has improved significantly, and no longer worry about technology failures impacting their ability to deliver the critical anthrax vaccine. Just as important, however, is that the virtualized solution gives them flexibility for the future, allowing them to grow the control system on-demand in order to meet needs. Because hardware and software no longer will be chained together, they won’t have to worry about being stuck without an upgrade path when new applications deliver features that are critical to production or compliance. “The flexibility of virtualization has been liberating, and it has delivered us the flexibility that we need to meet regulatory requirements and our customer’s targets for production,” Emergent automation engineers report.
Author note: BioThrax® and any and all Emergent BioSolutions Inc. brand, product, service and feature names, logos and slogans are trademarks or registered trademarks of Emergent BioSolutions Inc. or its subsidiaries in the United States or other countries. All rights reserved.
DeltaV™ is a trademark of Emerson. All other brand, product, service and feature names of trademarks are the property of their respective owners.
Brian Goschka is a senior automation engineer at Emergent BioSolutions with more than 20 ears of experience in the automation field. He has been with Emergent for four years.
Emerson Automation Solutions facilitated this article.
