Many years ago I went through the Cisco Academy at our local community college while working on my first degree, and I had a great instructor. He was an Air Force veteran, a fighter pilot to be exact, who then moved on to data communications. He had a great sense of humor and his teaching style fit me well. He was also a “no bullshit” kind of guy. I still have the occasional nightmare about his conversion drills though, decimal to hex, hex to binary, binary to decimal… I shiver just thinking about it.

Anyway, when there was a particular concept that he felt was of the utmost importance, he had a way of really drilling it home. A way of burning it into your brain so that you would never forget. One of those concepts was all about layer 1 of the OSI model. That’s right gang, the infamous physical layer (cue dramatic music). His point was that 8 times out of 10, a problem was in some way physical. So he stressed the important message to always check that first. Was the device powered on? Was the Ethernet cable plugged in to the client device? Was the cable plugged into the patch panel? How about the switch- were the connections there solid? Check your port status, any errors? You get the idea.

Over the years, this concept stayed with me. Always check the physical when ever there is problem. If it is not the issue, you can at least rule it out.

This has carried over into my wireless career as well, but a bit differently. Let’s face it, cable issues are fairly uncommon after an initial installation, as are switchport misconfigurations. Once wireless is up and running, it’s pretty static on the Ethernet’s physical side of things. The physical aspects of the RF environment are a different story. Now, when I am asked to check out a client’s Wi-Fi and figure out what’s wrong, the first thing I do is check out the airspace to see what’s going on.

Before I rip open my bag and start pulling out all the fun stuff (like my Surface PC and various adapters) I fire up an app or two on my phone, grab my notebook, and go walkabout. What I am looking for are SSIDs. How many am I seeing? What spectrum are they using, 2.4 or 5GHz? What channels in those problematic spaces are they using? Are they using 802.11n or .11ac channel bonding? If so, what channels are they eating up? I take notes accordingly so that I can refer back to them while using other tools. If there are more than say, seven or eight SSIDs, I am inclined to go back and grab my Surface and open up inSSIDer Office (I’m a big MetaGeek fan by the way) so that I can export the entire list and their corresponding stats. This list can be pretty handy, especially if the client has no idea that there were all of these additional wireless broadcasting devices in their facility.

 While on walkabout, I try to study my surroundings. I try to take note of areas of mass piping and conduit, large metal casings around equipment, equipment that has no shielding, metal grid walkways and any other thing that just sticks out to me. These fixtures can be important when you start to collate reported trouble areas with what is in them.

Now that I have a good idea of my surroundings and a base understanding of what’s happening in the airspace TX wise, its time to break out Chanalyzer (did I mention that I was a big MetaGeek fan?) and take a look at the RF spectrum. I start by choosing specific areas I spotted during my walkabout that stood out to me as possible trouble areas. My goal with spectrum analysis is twofold. I want to either validate clean airspace, or to identify and locate sources of NON-WiFi based interference.

I must say that I am continually surprised at how many people do not make spectrum analysis part of their routine when doing initial site surveys for WiFi installation, ESPECIALLY in industrial, manufacturing and warehouse applications. If you consider yourself a WiFi guy (or girl) and you don’t do this, I highly recommend that you start doing so. (sorry, end of lecture…)

Often, I find that NON-WiFi based interference is the culprit. The nice thing about Chanalyzer (and most other spectrum analysis tools these days), is that you do not have to be a rocket scientist in order to use it. To illustrate what I am talking about, please see exhibit A. This is a snapshot of your typical manufacturing 2.4GHz airspace with no major sources of interference. Notice all the blues and some yellows and maybe a little green representing the ambient energy, so pretty and calm based on the tool’s color scheme. It appears that what WiFi is present is operating on channel 1.

Now let’s take a look at exhibit B. I think that it is fairly obvious that there is a serious problem going on. Look at all of the angry red sitting on top of the three non-overlapping channels. For those not familiar with Chanalyzer, the color red represents well over 50% airtime utilization.

Then there are exhibits C & D. Total destruction of WiFi space as we know it.  But what’s actually going on? What is causing the trouble on the RF side? Let’s talk about that.

Exhibit C is interference caused by an industrial microwave dryer that was imported from Japan many years ago. It was old enough that it was grandfathered in under current FCC regulations requiring shielding to avoid this kind of interference. Exhibit D is interference from infrared adhesive curing equipment on a production line. Once the device was determined to be the source, the user manual was found, and it actually stated that this device caused interference in the 2.4 GHz spectrum.

My point to all of this is, when troubleshooting WiFi in the OT space, remember the golden rule from your early days in network education. Always check Layer 1.

Side Note: all of these spectrum analysis images are from actual site visits that I have done. Names have been sanitized to protect the innocent.

If you have enjoyed what you have read, follow this blog as I share my experiences, blunders, how-to’s, tips and opinions in all things OT Wireless from the wonderful world of industrial WiFi!