From Empty Warehouse to Sanitizer Bottling in Three Months
This new hand sanitizer packaging plant uses a unique automation architecture featuring edge computing to run at the pace of modern business.
With demand for hand sanitizer more than doubling across the U.S. and annual growth of more than 20% expected in coming years, 2020 presented the cleaning industry with a big financial opportunity. Emerald 66 Enterprises (E66) mobilized resources to meet this need, setting up shop in an empty denim processing plant in Seminole, Okla.Shown here is the rotary filler/capper running on one of the five packaging lines that E66 now has running smoothly in its Oklahoma facility.
In just three months, E66 had three automated packaging lines producing as many as 1 million HDPE bottles of hand sanitizer a week in a cGMP-compliant facility, and it continues to expand its core capabilities at a rapid clip. “When we say we do stuff quick,” says E66 Chief of Operations Robert Bodnar, “we’re talking days, not weeks or months.”
Let’s step back to the summer of 2020 and examine the technologies and techniques E66 and system integrator Northeast Automation Company, Inc. (NACI) used to achieve competitive advantage in a challenging market environment. When NACI was hired to come up with a bottling and packaging operation, E66 management made it clear that they needed to move fast. They were competing against low-paid, high-volume workforces operating manually and believed they could use technology to do more with a smaller, better-paid workforce.
“Each piece of equipment had to be intelligent because E66 management is so keen on information,” says Thomas Coombs, Principal Engineer at NACI. “We had to make every conveyor and every device smart.”
Coombs planned to use edge computing—a design technique that adds general-purpose data processing and connectivity capabilities to traditional real-time control and sensing applications—to build an information management system at the same time that he scaled up production capabilities. He did this using the Opto 22 groov family of industrial edge controllers and I/O.
Also important to point out is that E66 had also determined that the quickest way to build a new packaging process was by acquiring a variety of equipment at auction. The state of equipment on arrival varied widely, so NACI had to get creative in order to design a cohesive system at the speed that E66 needed.
To address the circumstances the team faced, NACI employed a unique architecture that enabled separate control systems to function together and also laid a foundation for E66’s data acquisition goals. At the top level, NACI used a groov EPIC edge programmable industrial controller, which combines PLC control with embedded HMI, OPC UA, and secure gateway functions in a single backplane. The EPIC controller supervised the packaging lines and connected disparate devices so they could freely communicate over a common communications protocol. Any equipment that arrived with a defunct control system was integrated directly into this network as simple remote I/O.NACI manufactured several of these VFD control panels for local conveyor control using Opto 22’s groov RIO.
Where packaging machines had functional controllers, on the other hand, these were left in place and loosely coupled to the main process using Opto 22’s groov RIO edge I/O modules. These modules provide software-configurable, multi-signal I/O channels and are powered over Ethernet (PoE), making them quick to deploy. NACI placed a module in each piece of packaging equipment upon arrival, connected any I/O wires, and identified the types of signals the equipment provided. These I/O signals were then integrated into the groov EPIC network in parallel with the existing PLC I/O connections, which continued to function independently.
“The ease with which you can do this,” says Coombs, “means you’re talking about a half-hour of wiring. Your biggest problem is finding the documentation from the original packaging machinery manufacturer.”
NACI also engineered an additional layer of control independent of the groov EPIC by building limited local control into each groov RIO module through Node-RED, an embedded, open-source IoT platform from IBM. Living up to the ambition to make every device smart, NACI added motors, photo eyes, load cells, and other instrumentation to many pieces of semi-automated and dumb equipment, connected these to local groov RIO modules, and added Node-RED logic to make them work together and report process data up to the supervisory level.
A former Oracle engineer himself, E66 COO Bodnar has been involved in much of this design. “The top-level process in the EPIC is turning on two or three packaging lines. If you have a conveyor coming in and you have another one going out, they may not be running at the same speed. You’ve got cappers that are running at different speeds. You’ve got label printers and all these different components all kind of running at different speeds, so it’s kinda neat to be able to say, okay, what if we use the groov RIOs to control, for example, the inbound and outbound conveyors and case packers and things like that? It’s all simple stuff, like turning on and off motors to run a conveyor and trying to match up to how fast a filling machine is spinning. They’re kind of like little islands of automation.”
The right architecture
This loosely coupled, distributed architecture allowed NACI to assemble their packaging lines without modifying any of the existing control systems that came as part of their purchased equipment. This strategy ultimately saved them development time, and in three months, Coombs and his team had 15 pieces of equipment up and running spread across three packaging lines.
Go here to see how even Louis Vuitton is making hand sanitizer.
At this point in their rapid development, E66 now has five packaging lines, each of which represents an end-to-end process including conveyors, fillers, cappers, print-and-apply labelers, case packers, and palletizers. Equally impressive is how management was able to roll with some unexpected punches. For example, they originally built their contract packaging business around packaging and distribution for a single large customer. But when that customer suffered a financial setback and had to shut down, E66 had to pivot and become a contract packager for multiple customers. The firm now works with both alcohol- and water-based sanitizers as well as a variety of packaging types, including injection stretch blow molded PET and extrusion blow molded HDPE. They’re even moving into self-manufacturing some of the containers they fill.Opto 22’s groov EPIC edge programmable industrial controller and groov RIO edge I/O module.
Automation grew from processing a high volume of single-formulation, one-gallon containers to working with a variety of sanitizer chemistries in different batch sizes and packaging form factors: from small two-, four-, six-, and eight-oz containers, hand pumps, and spray bottles, to large jugs in excess of one gallon.
Fortunately, NACI’s decision to use loosely coupled packaging lines made it easy to modify individual segments without interrupting their data collection and process integration. In combination with on-site building of electrical panels and the use of 3D printing for rapid prototyping of new mechanical components for the packaging lines, E66’S investment in edge-oriented automation allowed them to retool very quickly and break even on their initial investment within six months.
“There are a lot of good things you can deliver in manufacturing that people might admire, things like quality or throughput,” says Bodnar. ”Being nimble isn’t often on that list. But it’s fun to be nimble in your approach to manufacturing, and I think that’s one of the things Opto 22 has really helped us with.”
As E66’s process expands, so do its ambitions for building out a data infrastructure. In addition to using Node-RED to augment the production process, E66 has begun tapping into the core connectivity functions of the platform to move data from each groov device into relational and time-series databases, financial software, and other connected systems. And since Node-RED is a free, open-source application, E66 is adding it to their back-end systems as well so they can push data down into the control system.
“You might want to trip a lot number forward on a device or increment something based on a date that’s somewhat arbitrary, based on a business event, not necessarily on a machine event,” Bodnar explains. “You can send that over to the control system fairly easily, or just get summarized data from the control system and publish that back to your business systems. It gives you a very lightweight way to run an operational bus or hub.”
For example, E66 purchased a stand-alone pallet wrapping machine that used a proprietary circuit board design and offered only a limited operator interface. However, by adding load cells, connecting them to a groov RIO module, and feeding that data to Node-RED, E66 could readily verify the weight based on the known pallet contents and communicate the pallet number, lot number, shipping weight, and date to its central database to create bills of lading automatically.E66 now is able to process a variety of sanitizer chemistries in different batch sizes and packaging form factors.
Bodnar, who describes his career as back and forth between giant enterprise application integration (EAI) companies, like Oracle, and manufacturing and consumer products companies, was impressed by the ease with which E66 could integrate business and operations data.
“I’ve done smart grid stuff for a bunch of cities and things like that on the big integration systems,” says Bodnar. “We would have multiple enterprise data buses for operational data, reporting data, back-office data, or for communicating with different kinds of devices. I was thinking about how insanely hard that was. It would have been so much easier with Node-RED in an Opto device.”
As demand grows, E66 continues to expand its automation and diversify its business. It is adapting to accommodate new functions like on-site container molding, bulk product blending, and additional quality control procedures.
Bodnar says he is defying the industry norm of investing in a multi-million dollar build for an automated facility and instead is “putting in just enough automation to double my business, literally, which is going to make the company’s year. And then I can grow into that. And I’m not going to have to go back and rip and replace everything … We’ll tweak things, but I’m not going to have to throw away anything that I do because I can scale it.”
Recently, E66 has also spun up a lucrative technology division that is using groov devices to develop filling equipment for overseas export. In one example, using groov EPIC and RIO, NACI took an inline mixer design from concept to implementation in about five days, including a mobile operator interface built using the EPIC’s embedded HMI server, called groov View. By Thomas Coombs’ estimate, NACI implemented the controls in about four hours for a system E66 will sell for $50,000.
“The fluidity and dynamics of modern manufacturing requires extremely fast response to changing market demands,” adds Coombs. “With groov EPIC and groov RIO, Opto 22 puts dynamic manufacturing data at the edge of the production line and into enterprise systems simultaneously in real-time.”