A factory introduces a 10-part product line but only two parts are selling. The other eight parts pile up in a storage room alongside dusty inventory so old that it carries an abandoned brand name. With combined operations technology (OT) and information technology (IT) analytics, the company regroups to focus only on products people buy. Savings: millions of dollars.
A beer company leverages predictive maintenance and prevents a key machine from failing during the week beer is brewed to sell for the 4th of July. Savings: $1666 a minute.
A car company leverages predictive maintenance in a similar fashion and avoids temporarily losing a production line. Savings: $25,000 a minute.
Consumers reject a yogurt company’s new flavor and unsold yogurt sours in grocery store dumpsters. In the past, it would have taken at least a week to find out about the flavor flop. But with OT/IT integration, the yogurt company learns as quickly as possible and pivots to eliminate the failed flavor. Savings: millions of dollars.
These true stories, collected in Smart Manufacturing magazine interviews with more than a dozen people who understand first-hand the importance of OT and IT working together, illustrate how OT and IT can work together and eventually save billions of dollars around the world.
And yet in manufacturing it can still be said that IT and OT professionals use different terminology. They still rank priorities differently. They still focus on different goals. There are still two cultures and mindsets — that rarely overlap.
“We have had projects abandoned before they start, simply because the OT team was not going to listen to the IT team’s recommendations or edict,” Memex CEO David McPhail said. “Or, worse, the IT team would pull support for any OT project that was not perceived to be fully accountable to IT policy and procedure. I believe the majority of IT’s concern stems from the fear of cyber attack, and the fact that they are charged with the responsibility of this risk abatement by the upper management of the company.”
When the two sides don’t work together, consequences include duplicated costs, bloated inventory, compromised plant security, injuries and even deaths.
When the two sides finally come together, companies see better safety and security, tighter inventory control, improved quality, better maintenance, reduction in costly incidents and downtime, reduced cycle time to market and increased ability to customize production.
It’s enough to make you want to pick up a guitar:
Here come OT man
He come groovin’ up slowly
He got data on light boards
He got greasy fingers
He got an old computer
Runs Windows XP
Ain’t worried ’bout them hackers
He just do what he please
Here come IT man
Flashin’ right past the factory
Talkin’ ’bout sensors on switches
’N’ security patches
He got the latest computer
Big IT degree
Ain’t worried ’bout production
Come together right now, OT/IT
They say. “I don’t know you”
And. “You don’t know me”
But they need to work together
To run this factory
Come Together
In part because the business case for OT/IT integration is becoming crystal clear, companies are beginning to bridge the differences.
Technology is also a driving force for unity.
“Five years ago, it was pretty common for operations people to say of IT: ‘They stay away from me, and I stay away from them’,” Scot Wlodarczak, market manager for industry marketing at Cisco, said on behalf of Industrial IP Advantage, a coalition of Cisco, Panduit and Rockwell Automation.
But in the last few years, “the emergence of the Industrial Internet of Things, also called Industry 4.0, and the digital thread have required the new rationale,” Doug Macdonald, Aras Corp.’s marketing director, said.
“Generally, IT and OT have functioned in different spaces, essentially separated,” Bart Weihl, executive, global operations, advanced manufacturing initiatives group at General Electric, said. “The emergence of the Internet of Things, Industrial Internet, Brilliant Factory, etc., have blurred the lines between the two groups.”
And that’s uncomfortable, to be sure. OT and IT professionals “come from different cultures,” Autoware CEO Luigi De Bernardini said. “They are used to managing a different kind of project. It is difficult to put them at the same table. Even creating a team is not so easy.”
The two camps have trouble relating when they do try: The professional data scientists know the latest ITE tools, but may have never set foot in a plant, said Robert Borchelt, VP for enterprise architecture at GE. “Until they get that exposure, they may not be able to communicate. It’s like talking in a foreign language.”
Traditionally, OT has only been concerned with local premises below the switch, Borchelt said. IT meanwhile has been concerned with above the switch. “Now above and below the switch has to blend.”
Christine Frank, global industrial partner manager, IIoT, at Dell, puts it this way: “OT is looking from the bottom up at equipment and processes. IT is looking from the top down at business intelligence. They’re not realizing they have advanced business intelligence until OT and IT come together.”
And even when it is clear that the two camps need to embrace peace and harmony, that cultural change tends to be slow.
“The entrenched positions of OT and IT go back 50 years, generations even,” Kristian Steenstrup, distinguished analyst and Gartner Fellow at Gartner Inc., said. “You can’t overcome it with a Gartner PowerPoint slide.”
Gartner—which takes credit for coining the term OT— in the last decade contacted 4000 companies through surveys or individual dialog to listen to and help those companies better align and integrate OT and IT, he said.
The utility and energy sectors are further down the road than others because of the emergence of the smart grid, which required IT and OT to work together, Steenstrup said.
“Utilities started 15 years ago predicated on the hybrid tech rollout of smart meters, which are part IT and part OT,” he said.
Since then, manufacturing, along with mining, oil, gas and transportation, has made progress.
“There is some natural alignment in the utility sector simply because of the nature of what they do,” Borchelt said. “If they collect any data, they have to blend the two.”
Some companies in the manufacturing sector, such as GE and Cummins, have identified people who can successfully bridge the OT/IT gap and tasked them with identifying areas where change is needed. These firms believe that companies that successfully bridge the gap and integrate the two will thrive, leaving others behind.
“We see the savings in OT,” Eric Green, Dassault Systèmes VP for DELMIA user experience and advocacy, said. “But the linkage to IT is key. The savings might not be realized if the two are not converged.”
At GE, OT/IT integration is all about survival, Weihl said. Manufacturers that can launch a new product or upgrade an existing product quickly stay ahead of the competition, he said. “It’s a requirement to be competitive. With the Brilliant Factory, our ability to implement a product change quickly is a game changer.”
Now, said Frank, companies can finally see that profitability is on the line if they refuse to bring OT and IT together.
“So many times in the past we’ve seen examples of how operating in silos can be catastrophic to an organization,” Pete Tecos, executive VP at 5ME, said. “The OT/IT discussion is no exception. By connecting cross-functional teams through data, and creating an environment of transparency, you enable transformation. For the manufacturer, it becomes the difference between just making parts, or crafting a highly efficient, profitable, sustainable business that can compete on a global scale.”
Market dynamics are driving the integration of OT and IT, Paul Taylor, strategic alliance manager at Cisco, said on behalf of Industrial IP Advantage. “Companies are realizing that to be successful, they have to start pulling their teams together.”
IT must understand OT needs
As they come together, IT must understand OT needs and workflow.
IT might run analytics based on a workday ending at midnight. Meantime, OT counts the entire 7 pm to 3 am third shift as part of the day workers clocked in. Or, IT might be unaware that OT works all weekend.
“IT would decide they were going to do an upgrade of the network,” Taylor said. “They’d schedule it for 6 on a Friday night to do over the weekend. They were off, but they didn’t realize the factory worked 24–7. They didn’t realize the upgrade would have an impact on production. People would come in on Monday and find out production had been down all weekend and it cost the company a million dollars.”
IT leaders sometimes develop a solution without keeping in mind the needs and priorities of OT, Weihl said. That one-size-fits-all solution may be hard to use because it provides data that isn’t actionable or doesn’t tie into OT metrics.
“In a machining environment, I’m looking at spindle run time,” Weihl said. “That isn’t necessarily applicable to an assembly line. Don’t provide me with an assembly-line solution for discrete manufacturing.”
When both sides don’t buy into a proposed change, the process stalls. IT can’t shove changes down the throats of people in OT.
“It’s important that manufacturing operations understands what we’re bringing to them, that they are pulling it instead of IT pushing it,” Rachel Lecrone, director, manufacturing IT systems and industrial controls at Cummins Inc., said. “When we go into a plant, the first thing we do is sit down with the operations folks. We ask: What are they building? What does the project look like? What kind of problems do they have? What does their line look like? If they have a legacy system they’re trying to replace, how that works.”
“We have seen projects where OT people weren’t involved or leading and the projects are ignored or stagnate. They won’t use the system. A lot of it is, ‘It wasn’t invented here.’ They didn’t come up with the idea,” she said. “Or, it’s resistance to change—they haven’t dealt with this equipment. Now IT is putting a computer in front of them where before they just had a machine with a control panel.”
Lecrone can easily see the benefits of better OT/IT integration. But she must work through the human factors first.
“It frustrates me because I see us putting this system in that can help,” she said. “But until you have one up and running and people can see the value they’ll get out of it, OT is reluctant to take it on.”
At one plant, the Cummins team was halfway finished with a project when Lecrone realized the relevant OT people weren’t engaged. “We stopped, rolled it back to the beginning and said we weren’t going to move forward until we got the right people from operations involved,” she said.
“IT may say to OT, ‘I have this great solution,’ but if you can’t address and bring value to one of OT’s key areas, they don’t want to listen,” Weihl said.
Sometimes OT is ready to embrace change and meets resistance from IT, Steve Hanna, now US industry manufacturing leader, Microsoft Corp., said.
“At one big equipment manufacturer in the Midwest, we were trying to install solutions and OT was very receptive,” Hanna said, describing himself as an old manufacturing guy with cutting oil still under his fingernails. “The manufacturing team was chomping at the bit to understand machine utilization, tool utilization and the productivity of their investment. IT didn’t want to have anything to do with it. From IT’s point of view, there were significant security issues and they weren’t going to let anyone touch their controls without vetting it 100%. Even though a senior executive was sponsoring the program, IT was holding it hostage.”
Coming together over data
OT and IT can come together over the tremendous amount of data that now is available via the Internet of Things. OT teams have access to data that, properly analyzed by IT programs and people, can save companies millions of dollars via better product life cycle management.
But often that data, when it is collected at all, has been siloed. Potentially useful information is stuck on devices, machines, cabinets and light boards on the factory floor and never makes it up to the C suite where key decisions are made.
In the past, “we had readings from sensors and other outputs,” Frank said. “People didn’t know what to do with it. We didn’t have the computer power to store it. We dumped the data and no one ever saw it again.”
“One of the realities of dealing with OT data is there’s so much of it,” Borchelt said. “Sometimes, we get large amounts of data at very fast rates. Human beings are not always good at coping with that amount of data.”
The OT side of the house, which may be accustomed to collecting information, may not have an understanding of how to deal with the massive amount of data now generated, Borchelt said. “OT collects it all,” Borchelt said. “But how to mine it?”
That’s where IT comes in. “People are combing through terabytes of sensor data to discern patterns and come up with a better way,” said Stan Przybylinski, vice president of research at CIMdata.
“There are tools on the IT side that can help process that data into usable information,” Borchelt said. “But if you’re not familiar with both sides of the house, getting the right blend of data sources and ways to process that data and turn it into insights can be difficult to achieve.”
“Historically IT stayed away from the plant floor,” Wlodarczak said. “We’re overcoming that. Now Ethernet/IP has spread fingers throughout the plant floor. IT now has a role to play all the way down to a sensor.”
That data needs to be on hand in real time or nearly so. “You need to have the data—not every day, not every shift, but every hour or faster to make the right decisions,” De Bernardini said.
By collecting and analyzing that data in real time, manufacturers can improve production. “Connecting all the devices on a shop floor to other devices in the enterprise allows you to move data to places where action can be taken on that data,” Taylor said. “You turn that data into meaningful information that someone can make a decision on.”
How it worked with the new yogurt flavor: In the old days, stores ran a point-of-sale report once a week on an Excel spread sheet, Frank said. That means at least a week’s worth of unwanted product could pile up in stores and continue to be produced at the company before data was available. Now, that data can be sent instantly via WiFi or cellular transmission.
“In food processing, we can have complete visibility from the supply chain to the customer,” she said. “Now the data from the stores can be sent in near real-time and can be validated and scrubbed by analytics software with data modeling. They can look at the data in so many ways now—by region, most and least popular flavor, size container sold, sales prices, promotional data such as manufacturer coupons used, the temperature, environmental levels, etc. Now if a company puts out a new yogurt flavor and finds out from their information gathering in the grocery stores that it’s not selling very well, they’ll get rid of that manufacturing line.”
This data can also be saved to the cloud for further analysis later, she said. “The data has always been there but companies didn’t have the compute power to store it and the understanding of how to see the data in a meaningful way. Data modeling and data storage is much greater now and the analytics behind it has never been so powerful.”
As for inventory—before OT/IT integration, “people were working on building products that weren’t necessarily aligned to what the demand was,” Green said. “In a machine-intensive industry they were focused on producing product and asset utilization based on what they thought was the real demand. For example, in a machine-intensive industry some were focused on producing product and asset utilization based on forecasted demand. Because of the disconnect with the IT systems, they didn’t have an idea of the real demand or changes in demand and they were producing a volume of product based on demand that didn’t exist.”
Or product being built may not be in line with the latest customer needs. The product may not technically be defective or problem, but it’s out of specification,” Green said. “Orders have to be reproduced or reworked, driving up operational costs.”
That nonexistent demand and other issues become apparent with OT/IT integration. “We had made a huge investment in SAP analytics,” Hanna said about a previous employer. “Leaders wanted to know how we would get a return on the investment. We convinced the team that analytics were the answer. Over a three-month period, we found significant inventory issues inside the manufacturing plant. Before we put the SAP platform in, it was too hard to get the information out. We closed the loop on that, tied in sales and saved $150 million in inventory.”
Savings from improved, automated, connected production processes add up and create demand for more.
“When that happened, the manufacturing leader came to me and said ‘We need to have more analytics in all my plants. How do we make that happen?’” Hanna said. “We took the manufacturing engineering team and started monitoring equipment at a deeper level to better understand the efficiencies of all the operations. We found tons and tons of savings.”
Losses can be stopped, averted
Without such data, companies lose money.
When Hanna worked at a major car manufacturer, the company made a crossover vehicle and heavily marketed the six-foot sunroof that extended over all three rows of seats. The problem was, the carmaker could produce only 10% of the vehicles with the much-advertised sunroof.
“We ended up with 90% of our inventory that nobody wanted,” he said. “Those cars ended up in the rental car fleet. It killed the value. It killed the vehicle.”
Integrating IT and OT also can identify under-utilized assets and misaligned objectives, Hanna said. Manufacturers often are able to close factories and either reduce personnel or redeploy them to other parts of the business.
“A CEO might bet his career on making these huge capital investments in manufacturing facilities and come to find out that, in a lot of cases, these are wasted investments because you’re only getting 40% utilization out of existing equipment,” he said. “They didn’t need to invest capital in new equipment—only needed to run what they had more efficiently.”
Weihl spends time visiting factories within GE and looking for ways to improve production via Brilliant Factory changes.
“I may go into a site and see them using manual calipers and writing down measurements,” he said. “I tell them, ‘We can do this with WiFi or Blue Tooth solutions. There’s a digital insertion I can put in here.’ Then IT will come in and insert the data.”
In one case, GE collected data on oil fill, temperature and humidity at the edge from a process used to make high-voltage capacitors, Borchelt said. “We did data analytics on the collected data and were able to find key input parameters that affected the process of making quality capacitors. The data was always there, but without analytics on it, it had not become apparent that it was relevant to the outcome.”
One of GE’s Brilliant Factory OT/IT integration pilot programs reduced by 70% the cycle time for engineering design changes to hit the plant floor, Weihl said. “Historically, it was a 10-week cycle time at this plant. Our improvements knocked seven weeks off in the first pass.”
Getting ahead of the game
Data helps manufacturers transition to predictive maintenance.
That data can help as manufacturers, who have already moved from reactive maintenance to preventive maintenance, take the next step to predictive maintenance.
For example whether it’s your car or a machine on a factory floor, analyzing viscosity and contaminants in oil could reliably let someone know whether the oil needs to be changed at the standard 7000-mile benchmark or whether it could go to 10,000 miles, Przybylinski said.
“Some people have a strategy of ‘Fix it when it breaks,’” he said. “Some have scheduled maintenance. Now we’re able to monitor machines and fix them before they break. People are able to save a lot of money in maintenance. Given a set of parameters, you can look at what broke and what’s likely to break now.”
Added Frank: “Now we can go in and monitor all these things based on vibration and temperature. They can predict based on the modeling of data, bringing it out of the control system, running analytics and pushing it up to the cloud.”
When something does break or go wrong, fixes are quicker and easier.
Instead of hoping the service guy has the right part for a malfunctioning machine, that machine can be plugged into a diagnostic computer that will identify the broken belt or motor, Przybylinski said. Or the machine can even provide instructions, such as “Take these three bolts off and remove this panel,” he said.
“If there was a color mismatch causing us to have to throw away a particular batch: was it related to temperature or weather?” Cisco’s Taylor said. “It gives you the ability to look at production processes remotely.”
“That level of data collection means that as soon as there’s a failure, the control system allows us to identify what product was affected at the time of the issue,” Lecrone said. “Before, we’d have to quarantine a large batch of product,” she said. “The tracking system allows us to narrow this down to just a few units, 10% of what would formerly be affected. Now we can check out that this incident happened at this time and where and keep issues contained.”
“You’ve gone from having a product to having a smart product,” Przybylinski said.
GE’s Predicts Platform collects data all the way down to the device level and then provides analysis, sometimes in real time, Borchelt said.
The data available now also can help management monitor and improve production at the individual machine level, Lecrone said. In one Cummins plant building, production had been tracked only by the entire factory, she said. “Now we’re seeing machine by machine how many parts per day are being produced. We’re using that information to increase productivity of that line.”
A recent addition to the networking solution stack is a dual-purpose device that appeases both camps, Memex’s McPhail said. The device uses a cost-effective switch at the edge with dual cores that handle both compute (data collection and aggregation) and IT networking policy and procedure.
The OT camp gets its data that it can then use to solve problems and save money. The IT camp can support the network and enforce whatever policy and procedure it uses in the rest of its corporate networks, without compromise, he said. Memex is working with Mazak and Cisco, and results of the IT/OT collaboration are encouraging, he added.
Design changes deployed faster
Another area where OT/IT integration can help is deploying design changes more quickly, GE’s Weihl said.
At one GE plant, the challenge was inspecting holes to make sure they were the right diameter and were in fact true through-holes. “The only way to test it was to manually place a gage through the hole,” he said. “Now automation can be leveraged to confirm this.”
Managers and VPs miles from the factories can monitor production, analyze data and build strategies to boost output.
“You’re able to very easily compare data across various factories to see which are the most productive,” Taylor said. “You can look at production processes remotely. There’s more data coming out of connected devices. You can see improvement in productivity through better data analysis. Which factories are doing well and how can we bring the others up to the same performance?”
Savings on energy possible
With access to energy use data, companies can now factor energy spending into the cost of production as opposed to seeing an energy bill as a fixed cost, Taylor said.
Based on energy costs, company executives can evaluate energy savings from shifting production to a different time of day or even a different factory, he said. Factories can evaluate the pros and cons of staggering machine start-ups to avoid one-time peak demand charges, he said.
“If the factory can be more flexible, it can take advantage of lower energy costs at non-peak times,” he said.
Local results highly valued
But before moving forward, people want to see solid results in their own workspaces. Even results in another division of the company often aren’t enough to convert skeptics into believers.
“Without local, tangible results to demonstrate the value, it’s difficult to get organizational buy-in and acceptance,” Borchelt said. “Even though people see results in other businesses, until they see results in their own place, they have difficulty embracing it. Even within one business, someone might say ‘Yeah that might work in another division, but our division is different.’ Everybody believes their combination of customer demands is unique. The value of a local proof point is hard to overstate.”
With this in mind, GE has focused on what it calls FastWorks, applying startup philosophies in the 125-year-old company.
Instead of wide-scale programs with all the bells and whistles, FastWorks focuses first on small, local changes with quick results, Borchelt said.
“The traditional story is you have to start with a huge, upfront investment,” he added. “We start with some leap-of-faith assumptions, design the minimally viable product or solution and use it to see if we get the expected results. We want to fail fast, learn from it, pivot and end up with a better outcome. The goal is to get something working and then we’ll improve it. The idea is don’t let the perfect get in the way of the good.”
Connect With Us
