One of the big differences I have seen over the years in the OT vs IT realm is the base knowledge of what you have. An active inventory that is kept so you know when it is time to do an equipment refresh. So many facilities grow homogeneously over time that pieces and parts are added randomly as the need arises. What’s even better is that no one documents what they add or put in. Over time you end up with a great big mess of a network that the local team isn’t sure how to support.

This mess is further complicated by vendors and other “helpful” employees. When a vendor comes in and installs new equipment, generally there are some network components involved. From what I have seen, they don’t always tell their clients what they put in. Additionally, employees sometimes think they are fixing a problem by adding their own equipment from home. The resulting rats’ nests become nice little surprises for guys like me to find when trying to assess systems or troubleshoot issues.

At this point, it should come a no surprise that most industrial and manufacturing sites usually have no idea of what’s going on in and around their own airspace. As with network switches, it is becoming common place for vendors to add their own wireless access to the systems that they install. Generally, there is no consultation with the site on any WiFi specifics, they just install and configure whatever they see fit.

To me, this triggers all kinds of alarms. What is being installed is essentially a rogue access point. Does anyone other than the vendor know what the WPA passwords are for access? Not usually. Just who is connecting and what are they doing? No one seems to know. Is there any sort of monitoring in place? Of course not. Does management know about these? Sometimes they do and sometimes they don’t. Sadly, it is usually just shrugged off as something vendors do.

These vendors, they are professionals right? They are at least using the appropriate access points. I am sure that installation is neat, clean and mountings are in appropriate places. I mean, come on man, they get paid a shit ton of money for what they do!

Nope.

Most vendor installed wireless devices are not actually mounted. Usually they are shoved into a cabinet under a panel display or jammed behind a piece of conduit to hold it in place. On the rare occasion when something is mounted, the devices are easily accessible, close to power outlets, and held together with zip ties.

Moreover, there is a laundry list of issues and vulnerabilities of the devices. Physically these little appliances are not designed for the fairly harsh environment that they are placed in. They are made of consumer grade materials that, long term, cannot take the higher temperatures. Many times, the sheer volume of dust generated by the facility gets into the devices because they are not sealed for protection. If dust is getting into them, then you know any moisture is too.

Physical issues aside, there is the matter that these “off the shelf” gadgets are routers as well as wireless access points. If services like DHCP are not disabled and they are blindly plugged into a production network, imagine the chaos it could produce. Also, even if they are configured to get along with everything else network and wireless wise, there is still the problem of the constant updates needed to address the stream of security flaws.

It is kind of scary when you think about it. Especially when you consider that this is not an isolated case. It is a common occurrence across different facilities of completely different companies.

Not long ago, I was at a site doing some maintenance on an outdoor mesh network. I hadn’t been there in quite some time, so I decided to take a look and see how crowded the air space was getting. Keep in mind that the plant was about half a mile off of a main road and surrounded by woods and marshlands. They owned their airspace.

I can’t say that I was surprised by what I found.

• 6 printers (none of which were secured)
• 1 multi-function copier
• 3 MiFi devices
• 5 SOHO wireless routers (Netgear, Linksys and D-Link)
• 6 Cisco wireless devices that were not owned by the client site
• 1 lone Ubiquiti access point
• Three Routerboard devices (I am assuming they are MikroTik APs, but could be home grown custom software on Routerboard hardware)

So, in the one spot that I stopped and looked, I found 27 WiFi sources that were not part of the site’s wireless infrastructure. Some of the items on this list, like the printers and copier, just need to have their wireless access disabled. They should all be hard lined anyway.

There is not much you can do about the MiFi hotspots unless you have policies specifically written that do not allow them. According to the FCC, they cannot be “tar-pitted” or “de-authed” by any WIPs security measures. They are privately owned devices that are freely allowed to use public access space. You can thank Marriott for that little gem (https://www.cnn.com/2014/10/03/travel/marriott-fcc-wi-fi-fine/index.html).

A byproduct of the lack of communication and/or coordination between vendors and site management, is the almost random assortment of WiFi protocols in use. It is a mix of 802.11 A, G, N and AC. There also appeared to be a fair amount of channel bonding going on in 2.4 ad 5GHz. One 802.11AC radio was even blasting out an 80MZ wide broadcast.

It was the Wild West of Wireless!

Is there a moral to this tale of WiFi woes? Well, no… not really. It’s more a word of warning when venturing into the realm of industrial WiFi. Learn what’s in the local airspace, document it and be prepared with remediation recommendations. Odds are, no one else has any idea of what’s going on.

If you have enjoyed what you have read, follow this blog as I share my experiences, blunders, how-to’s, tips and opinions in all things OT Wireless from the wonderful world of industrial WiFi!