Chapter 10: Mission Critical Municipal Network Designs Where Hope is not an Engineering Strategy

Municipal wireless designs…

were supposed to be an all-encompassing solution for residential Internet access and various municipal government needs. Let’s take this up a notch though. Data and connectivity are becoming critical for police officers who are handling traffic stops in dangerous situations without partners. With the technology we currently have, I believe that when a lone police officer leaves a vehicle to approach a suspect or even during routine traffic stop, that officer should, at the very least, have visual backup. We have the technology to deliver a live camera feed from the vehicle back to a central dispatch location. If the situation becomes critical, the officer does not have to reach for a radio to get an immediate response. There have been numerous instances of officers being attacked on routine traffic stops and then becoming too incapacitated to request backup. This is a perfect application for municipal wireless or alternative data technologies.

Realistically, TriadLand is not designed for a mission critical environment with mobile connectivity. Criminal activity or life-threatening situations can occur in a matter of seconds which may involve moving vehicles. The TriadLand system is designed only for stationary users. Mobility requires fast handoffs between access points (APs) and optimized mobile APs. If you do not expect 2.4GHz mobility, then the TriadLand system is more realistic for budgetary and performance reasons. Without fast handoff, it could take 5-20 seconds to reacquire the AP once the police car stops. However, sometimes it’s better to have a basic system with some limitations that you can afford rather than no system at all. Know the limitations and plan around them.

Most of us have seen the results of municipal systems that were grossly under-engineered in terms of number of APs. The news is littered with implementations that needed 2-3 times more APs per square mile than originally estimated due to trees and other obstructions. In most cases, those systems were designed for Internet access with public safety a secondary benefit of the system. However, unless you are planning on deploying 30-60 APs per square mile and site-surveying every inch that a car can go within the city, you simply cannot guarantee that system will deliver the level of mission critical performance that public safety requires. 2.4GHz is easily blocked by obstructions and it is extremely difficult to find every nook and cranny. No amount of hope solves that problem.

I am still seeing companies make promises to police departments like 95% to 98% coverage. What the heck does that mean and how do you actually guarantee that without a full deployment and detailed site survey? Is there a guarantee of a minimum modulation rate within that 95-98%? Without a foot-by-foot, on-site visual analysis of the city, how do you know that you are going to have coverage everywhere? What about the loading dock behind the metal doors in the warehouse district (where the semi trailers park randomly), behind brick buildings, deep in the trees in the city park, alleys, and across fields and forest pockets? 2.4GHz will not get into those areas unless there are boatloads of APs per square mile and LOS. In many cases, the backhaul to the AP is more difficult than the coverage area.

What is really very, very sad is that I just sat in a meeting where 2 companies, one a $20 billion dollar company and the other a local video surveillance company, told a city that 30-45 APs will cover a 2.5 square mile city with a tremendous amount of 50 foot to 70 foot trees in South Florida. Both companies paraded coverage maps where they simply drew circles showing how far the APs would cover. However, these are not wireless companies; they are video surveillance companies who use wireless for backhaul.

These companies did not show ranges based on any RF spectrum surveys to determine what signal level they needed to maintain a useable signal to noise ratio. They did not show any real RF analysis of path coverage using even Radio Mobile, let alone a commercial package that might take into account the buildings or the deep forest coverage that surrounds every park and lines every street. They simply threw big circles on a map with as many APs as they could fit in the budget. They then expanded the circles to show them overlapping. This is typical of many companies that believe 2.4GHz mesh APs can be deployed by anyone without any wireless engineering experience.

Who needs an RF consultant when you can simply draw big circles on a map and proclaim yourself an expert? Apparently the surveillance companies have not researched municipal wireless deployments over the past several years. “Those who fail to learn from the mistakes of their predecessors are destined to repeat them” (my apologies to George Santayana for that variation).

I’m sure they hoped nobody in the room was astute enough to see that these circles were ridiculous in the expected coverage area. These ranges would be unrealistic in Tucson, Arizona (middle of the desert with very little tree foliage), let alone south Florida. It took all my self-control to not jump out of the chair and beat these presenters with their rolled up coverage maps for total incompetence and outright lying. For the record, this system would have taken about 160 APs.

Since the city has little vertical assets, 20 foot to 60 foot poles would have to be placed around the city with electrical infrastructure and trenching. The total cost of the system would realistically be around $2.9 million (give or take a couple hundred thousand dollars depending on trenching and pole installation costs which wasn’t exactly detailed) for 100% coverage.

Some of you are saying that MIMO significantly reduces the number of APs. I do not disagree except in the case of a high-density vegetation environment. A recent test by one of my Texas associates, Justin, has demonstrated that penetration through trees is greatly enhanced with 2×2 MIMO dual-polarity, almost to the point of matching 900MHz radio performance with lower power output. However, move those APs 20 feet either direction and there is absolutely no way to determine if you will get the same result. Move those trees to Florida where they will not exactly be parched for water and it would be much more difficult to penetrate the trees. Keep in mind you are also penetrating wood, which will have significantly higher moisture content in Florida. The test in Texas was done with dual polarity 2×2 MIMO fixed APs set to 24dBm output. Unfortunately, mobility demands that the 2×2 MIMO mobile systems will use dual vertical antennas. Tree obstruction is a huge variable and dual vertical antennas with 2×2 MIMO helps fading, but it’s not as effective as dual polarity in tree penetration. The 3-6dB average improvement that can be calculated in over an 802.11g system is still not going to get that signal through hundreds of feet of thick, overlapping trees, especially in South Florida along the coast.

We also have the problem of antenna gain in a mobile vehicle. If the APs are at lower points such as 20 feet to 30 feet off the ground, then we can run 12-15dBi omni-directional antennas on the APs and 5-7dBi or better antennas on the car. However, if the APs are 50 feet to 80 feet in the air to increase coverage area, then the APs either have to run lower gain antennas or have some type of electronic downtilt which reduces the coverage range. At the same time, the car may have to run 3dBi antennas to keep the towers in the vertical beam pattern. Keeping in mind that the coverage zone doubles with a 6dBi improvement in antenna gain, we may be trading altitude for significantly less coverage area. The idea is that having LOS provides better area coverage than trying to go through obstructions with higher gain as we talked about in earlier articles. Simultaneously, if the area does not have a lot of obstructions, then a 3dBi car antenna talking to a 5-7dBi omnidirectional antenna will surely go half as far as a 9dBi antenna talking to a 7dBi car antenna. You are giving up a lot of range meaning more APs, more hops, and more expense.

Mesh manufacturers are still in the process of converting their product lines to 802.11N 2×2 MIMO, which also caused me a few issues. If you are considering a system today, it makes no sense to put in an 802.11b/g 2.4GHz AP with the 2×2 MIMO APs already released. However, not all manufacturers have transitioned their product lines over to 802.11N. I am finding that without complete product lines, I am modifying typical manufacturer deployment strategies to get the performance I want. The all-in-one solution from a single manufacturer still is not there.

A modeling program is only step one in designating AP locations. It’s very important for engineers to visually see placement of every AP to understand the environment. Until that is done, no cost or performance estimate is ever 100% accurate. Heck, even when it is completed, the wild nature of 2.4GHz and the dynamic noise environment means no guarantees. Some of the early deployments were not even 30% accurate for their designated target clients and expectations. That’s why it is also important to document noise values at every AP during the initial installation and then audit them every 3-6 months to ensure that things have not changed.

Let’s get back to the real world with mission critical mobile designs. This is where I start talking to BelAir, FireTide, Tropos, Mesh Dynamics, Fluid Mesh, and Motorola. They all have systems designed for fast-handoff which is the key feature of a mobile system. They also have years of experience in deploying stable, mature firmware. Unfortunately, they are battling municipal budgets that may not be capable of deploying a system that provides 100% coverage. However, telling a police captain that he gets 95% coverage when lives may be on the line is like telling your spouse you have been 95% faithful. Let me know how that goes.

There should be a penalty for companies and consultants when they tell clients they are getting 98% coverage if clearly they will be lucky to get a useable 50% coverage. If these people were doctors, this would be malpractice. This much differential is simply not a matter of my opinion or a difference in design between engineers; it is gross negligence and complete incompetence. If you are municipal agency or a police department looking at this type of system, make the company put into writing the coverage zone based on signal, signal/noise ratio, and demonstrated performance parameters. Ask for some level of actual engineering specifications similar to what your utility department would ask for. If the system does not perform to those designed specifications, make sure the contract states that they have to add APs at their cost to meet these specs at no loss of bandwidth expectations along with the accompanying engineering documentation. Also make sure they guarantee a minimum bandwidth while a vehicle is moving and is stationary. If they won’t do it, walk away and find another company. At the very least, hire a consultant who has the experience to evaluate designs. If all they hand you is a map with pretty blue circles, show them the door.

On the other hand, the evaluation committee really needs someone on staff also with this experience. Nothing personal, but if your IT person cannot run Radio Mobile or something similar and calculate path loss in his head for simple Point-to-Point links, he probably does not have that experience. If you are spending a million dollars or more on a project, spend a few dollars to have a RF consultant or engineer (I mean a bachelor’s degree, not a 1 week manufacturer’s training class and a engineering certification) with some real-world experience, evaluate the designs to make sure they are actually feasible and not engineered with a Spirograph (children’s drawing toy). In fact, I would go so far at this level to suggest hiring 2 consultants for a design review to make sure you get accurate information that is not subject to personal bias. Personally, I think that the best consultants should have an engineering degree in a technical field since they are used to working with empirical evidence (this should drive some serious discussion by itself).

There are other options out there than can provide a significant advantage in coverage area with a simultaneous reduction in bandwidth. This alternative redundant hybrid design, which I proposed, makes it possible to deploy in areas with lots of vegetation that would be cost prohibitive for standard 2.4GHz deployments along with reducing the cost by at least 65%.

For example, instead of 160 APs at $2.9 million, I reduced the number of APs by a factor of 4, and then added an optional mobile technology which I will cover in the future, capable of penetrating trees and buildings (and no, it’s not 3G or 4G which is another option). The end result of this design was 100% coverage of the area with mobility at rates of 1-2Mbps while moving in an area not covered by 2.4GHz. The 2.4GHz system only covers 30%-40% of the city (I’m being relatively conservative here because I thought of a few new ideas I have to test in Phoenix first before I include them in the design) but this includes all the major city assets and the main streets. If we are near a 2.4GHz AP, then we get full 802.11N 2×2 MIMO speeds and we still have fast handoff on the major streets. A side benefit is that the vehicle now has a GPS for centralized tracking and is its own hot-spot for 2.4GHz. This design change saved the city about $2 million dollars over the cost of a complete coverage 2.4GHz system. It also helps me sleep better at night knowing that no officer has to worry about being in a dead zone.

One other feature of the design, for those of you who are more intimately familiar with hurricanes than I am, is the emergency feature. If a hurricane comes through and completely takes power out in the city for several hours or days, only 5 towers in the city have to get emergency power to keep all the police cars connected. The 2.4GHz mesh system will be completely down as the battery backup on those units will only run 40 minutes. If only 3 towers out of the 5 are kept functioning with long battery backups or generators, depending on which 3, the system should still have very close to 100% coverage. The tower systems also have 2 APs with automatic failover for 100% uptime. For the record, this is why I also didn’t use cellular since in an emergency, there is no guarantee of response time and coverage. However, cellular can also be added to any car in a minute simply by plugging a 3G or 4G card into the car router, thus making 3 connection options instead of 2. This also works if the vehicles have to leave the city limits.

This design works for this city because the city has very few vertical assets, heavy vegetation, limited budget, and focuses on the mobility component. It was also designed to support up to 108 cameras with the ability to expand to 4000 cameras at very little cost. The mesh system is also expandable as vertical assets become available. I’m also comfortable with stating unequivocally that it provides 100% coverage. You cannot ask for anything better than that. And as for you surveillance companies that are designing city-wide municipal or mesh networks for cameras that include 2.4GHz mobility systems, please hire a consultant who is qualified in RF engineering and who has this experience before you propose a system where lives may depend on the performance. Hope is not an engineering strategy and pretty blue circles don’t an engineering design make.