We had the opportunity while visiting with NETSCOUT at Mobility Field Day 2 to get a sneak peek of the v2.0 firmware for the G2 (yes, firmware; no need to purchase new hardware). Many of us were very excited to see some of the newly added features and it seems as though there is even more room to grow in the future. What we have come to trust as one of our favorite go-to tools in the wifi toolbag has only gotten more powerful and useful. I am not going to go into a deep dive of each of the features that Chris Hinsz showed us but I will share the links because he does a way better job explaining it (because he is the Product Manager of course).
Notable features in the v2.0 release are iPerf, captive portal support/testing, interferer identification, and packet captures. Also introduced was the NETSCOUT “Test Accessory” which will function as an iPerf server and can be placed anywhere there is a network connection with POE.
The additional features that are being introduced show that this product is as versatile as ever and should increase the confidence of its user while deployed in the field.
Here is the link to download the new v2.0 firmware.
Video from MFD2 featuring Chris Hinsz, Product Manager at NETSCOUT
Deep dive of AirCheck G2 v2.0 with Chris Hinsz
Video/webinar featuring Chris Hinsz of NETSCOUT and George Stefanick of Houston Methodist Hospital
A few weeks ago at Mobility Field Day 2 I was afforded the opportunity to listen to a presentation from the fine people of NETSCOUT (all capitals because that is how Kendall Hershey told me it should be). It was somewhat of a reunion for Brennan, Rob, and myself as we had established a relationship with NETSCOUT and their Social Media Manager, Kendall Hershey, at Cisco Live 2016 (shameless blog plug) . It was great to see her again as well as take a deeper dive into the NETSCOUT product line, especially the AirCheck G2.
Chris Hinsz, NETSCOUT Product Manager, was the main presenter and he gave a nice overview of the NETSCOUT product line, more specifically the beloved AirCheck G2. Many of us WLAN professionals use the AirCheck G2 at our day jobs and have come to love the simple yet powerful tool. Chris went on to offer a closer look into what makes the AirCheck G2 so great as well as some lesser known tips and tricks. Check out the presentation here.
Chris also gave us a quick demo of the Link-Live dashboard. The Link-Live dashboard is a clean, easy to read area where you can collect the information that your various NETSCOUT tools gather. I have had the opportunity to use the Link-Live dashboard with my LinkSprinters for some time and I appreciate the simple layout. I can quickly get the information I need with a quick glance. The dashboard organizes your tests into a table and allows you to make annotations, as well as create reports from the collected data. NETSCOUT has definitely spent some time refining the layout making it easy to get the information you need from your test equipment.
I have been drooling over the AirCheck G2 since it was released. We recently purchased one for work so I have only recently had the opportunity to use it on a regular basis. I love the features of the G2 especially the touchscreen and well laid out menus. The testing features are straight forward. The G2 is hands down the first device we go to when we get called out for wireless issues. The G2 saves us a significant amount of time troubleshooting. Before the G2, troubleshooting wireless issues took much longer. These handheld testers bring you to a solution much faster with the information that they supply. I personally own the G2’s predecessor, the **Fluke AirCheck. The original AirCheck is still very useful to me because it is able to extract a Meru/Fortinet/FortiRu AP ID out of a beacon’s information elements. This makes it very easy for me to identify an AP that is a member of a Virtual Cell (virtual BSSID). I still love my original AirCheck for this reason, but mostly because Kendall Hershey gave it to me. I am told that they will look into adding the FortiRu information element to the G2 in the near future.
Gratuitous NETSCOUT AirCheck G2 product placement by Brett (@crabby_fi)
The AirCheck G2 has been, and continues to be, a staple in the WLAN professional’s bag of tools. The NETSCOUT product line is built on a solid platform and is sure to deliver for the foreseeable future. Anyone who uses a G2 will let you know how much it enhances their troubleshooting experience. Issues that may have previously taken a few hours to figure out are easily diagnosed in a much shorter time period. NETSCOUT is another one of those companies who I hold near and dear to my heart because they listen to their customers. I am looking forward to what NETSCOUT has coming in the future.
**On July 14, 2015, the following products from Fluke Networks were merged with NETSCOUT Systems; Visual TruView™, OptiView® XG, OneTouch™, LinkSprinter™, LinkRunner™, AirMagnet™ and AirCheck™. Fluke Networks continue to retain the DTX CableAnalyzer™, Versiv Cabling Certification System, LinkWare™ Live and Telecom products.
I recently had the opportunity to listen to a presentation from Mist Systems at Mobility Field Day 2 in San Jose. At first glance/listen Mist might sound like just another controller-less “cloud” wifi solution that touts a shiny dashboard and user analytics. They even toss around trendy new buzzwords such as “AI” and “machine learning.” I was pleasantly surprised to find out that they are much more than that!
Mist is relatively new to the wifi game but their short lived heritage is backed by staff who have worked for the likes of Cisco, Motorola, Symbol, Airespace, Zebra, and Ubiquiti to name a few. As you can tell their staff are seasoned professionals that have worked with companies with a very solid track record. With a list of leaders like this I was confident they would be able to do wireless well, but I was still curious what would differentiate them from other offerings. It didn’t take long to realize their pedigree and vision, when put together, sound as if they can break through the often “cloudy” (pun intended) traditional stalwarts. Their current customer base is impressive, including several very large organizations that span several different verticals.
After a few moments into the introduction given by Jeff Aaron, you realize that Mist isn’t just another controller-less wifi company. Jeff pointed out that Mist has two main objectives:
Bring wifi into the modern smart device era by:
giving data that allows an engineer to be proactive
providing predictive recommendations
allowing better visibility into the user experience
not traditional network visibility, but focus on user experience
making wifi more predictable, reliable, and measurable
Expand wireless. Make wifi work for you. Wireless as a Service
in addition to wifi, providing an indoor personalized location based experience
Next to speak was Bob Friday. Bob has a rich history of providing location based services in the wireless environment. Bob was co-founder of Airespace, a wireless networking company who was the first to offer integrated location based services in their products. Bob went on to explain that wireless has transitioned from just providing wifi into an “end to end” service. Bob made it very clear that the Mist location services were built from the ground up and that it wasn’t just a “BLE laid on wifi” afterthought.
One of the coolest things I got from Bob is that Mist actually employs data scientists to analyze the data that is collected. He said that Mist has Particle Physicists and professors (you know, people who are actually good at sifting through data) going through and helping with design of the Mist product. We have all seen a nice dashboard that is spewing data all over the place. The data may have a nice presentation but is often to much or to little, and then all wrapped up in a pretty box. It is clear that Mist wants the data to be clear and easy to read, as well as useful.
Mist really seems to be ahead of the game with their vBLE location services. It didn’t take me long to realize that there are many use cases spanning several verticals for virtual beacons. App developers can use these virtual beacons to interact with applications designed for all sorts of devices. Applications can use location data to relay relevant information to the device owner based on the location of the device. Some examples include way-finding at your favorite ball park or sending offers or discounts to a customer when they are near a particular product in a retail store.
Mist’s dynamic packet capture is something I think is an absolute game changer. Mist is able to leverage it’s services to detect problems on the wireless network and then collect a packet capture in real-time. We have all been called out to troubleshoot problems at one time or another and by the time we get to the site, the problem has disappeared or changed. When Mist detects an issue, it automatically starts a packet capture. Being able to gather packets immediately when a problem is detected can potentially solve a problem in a matter of minutes where in the past it may have taken several hours.
One other notable feature is Mist’s ability to give meaningful and measurable data that makes it easy to show whether or not Service Level Expectations (SLEs) are being met. Based on the information given in the dashboard it is easy to tell which clients are having trouble connecting or are failing to connect. Being able to evaluate access point health, coverage, capacity, and throughput are also a major advantage for any wifi engineer and are all easily found in the dashboard.
Overall, I thought Mist was very impressive. They are doing things with their product that can make life as a wifi engineer easier. There is no better judge of a wireless network than the actual user experience and it is obvious that is where Mist is focusing their efforts. I am familiar with BLE but overall I am generally “green” and have lots to learn. What I do understand about Mist’s vBLE is that it has potential to be very powerful. I plan on digging a little deeper into vBLE when I get an opportunity. Having listened to Jeff Aaron, Bob Friday, and Sudheer Matta, I can tell they are confident and passionate about Mist. I am hoping to hear more from them in the near future. For now, please watch their presentation from Mobility Field Day 2. There is a lot more impressive information in their presentation. I hope you find Mist as intriguing as I did.
Check out what other #MFD2 delegates are saying about Mist:
The excitement for MFD2 has been building for awhile now and I can tell you we are in store for a great couple of days. I have been looking forward to this event since I accepted the invitation to be a delegate a couple of months ago. After coming together for dinner tonight as our first time as a group, I am even more excited for all that is in store.
I have had a chance to play around with one of Mist’s APs over the past couple weeks and I found it to be very intuitive and easy to set up. It took no time at all to get an SSID on the air and clients connected. The available data on the dashboard is very helpful/useful. I am looking forward to getting a deeper dive into the BLE features of the device.
Mojo Networks, formerly AirTight, has gone through several changes over the last couple years. They have changed their direction and I am looking forward to hearing more about their plans going forward. They had an excellent presentation at WirelessLAN Professionals conference this past February.
Cape is a company that I am not real familiar with yet and I am very interested to see what they have lined up. I did see their presentation at WLPC but haven’t had a chance to work with their gear yet.
Another company that I am really looking forward to hearing more from is Nyansa. From what I have seen, they are able to take wireless client data and present it to an engineer in an easy to read format to get a better understanding of what is actually going on in the wireless network. You can find their presentation from WLPC here.
The AirCheck G2 is our go to tool in the field when a wireless problem is reported at work. I am excited to hear what they have in the pipeline for the future. I am also pumped that we get to see our good friend Kendall Hershey. A couple of us had the pleasure of meeting and hanging out with her at Cisco Live 2016. Her and I both missed this year’s Cisco Live so it will be cool to see her again.
Being able to watch and listen to presenters at previous Wireless/Mobility Fields days and interact with the delegates was something I looking forward to. With this being my first MFD, I would be lying if I said I wasn’t a bit nervous but I am with a great group of professionals so I am sure it will go great! I am very excited for the next two days as they will be packed from the beginning of the day until the end with all kinds of great information and maybe some surprises.
Please feel free to follow along on the Mobility Field Day page here.
Please also feel free to interact with me or any of the other delegates listed in the above link. Interacting with the delegates was almost as fun as listening to the presenters in my past experiences. If you have any questions you would like to see asked of the presenters, feel free to reach out to any of us delegates via Twitter. Make sure to also use the hashtag #MFD2.
In the weeks to come I will be releasing blog posts about the presentations so make sure to check those out when they are released.
It has been a couple weeks since I returned from WLPC in Phoenix. It was a great trip down to the southwest which included catching up with some good friends, listening to great presentations, learning a lot, and also presenting on a topic near and dear to my heart. I was able to put a lot more names to faces and have some great conversation.
As many of you know I presented on how well single channel architecture works for us. You can find the video below:
I have received mostly all positive feedback on the presentation. It was a great opportunity for me to speak in front of a large crowd and give examples of how we have Meru deployed across our school district. As well as it went, I feel like I need to explain a few things. I have come to the realization that a Ten Talk may have been the wrong platform to discuss the topic at hand. I understand that most people wanted more information about SCA rather than “it just works for us.” Ten minutes was simply not enough time to discuss all of this.
First, I understand that being a large network doesn’t equal wired\wireless networking done right. Even though this wasn’t conveyed in the presentation, I have received feedback indicating as such. Although we are very proud of how large and successful our network is, it doesn’t mean that large = successful. A lot of time and effort goes into making a wireless network work well for 60,000 users on average every day. We all know that architecture doesn’t matter if you don’t define, design, implement, and validate.
Second, I gave some confusing information. I realized shortly after the presentation that the stat showing that we “average ~12 connections per AP county wide” was very vague. Some of our access points have upwards of 100 clients on them at once for an extended period of time. Some of our access points have 10 clients total (maybe less) in an entire day. It doesn’t matter whether there is 30+ clients on an AP during every class or another AP that sits unused for the majority of the day. If an AP was placed in a location, it was done there with the intent that wifi could be needed at any point of an instructional day; planned or un-planned. The opinion that there are too many APs or not is irrelevant. Unless you have actually visited our schools and used the network, you won’t know. The proof is in the pudding so to say.
I know that most of us consider our wireless networks as mission critical. The vast majority of our schools don’t have desktop computers other than in administrative areas. Many of our schools don’t have computer labs, but employ several carts of mobile devices. I know that most of us want NUMBERS to back up the user experience, but most of the time I don’t have time to get this type of information. If reports of “wireless problems” are low or non-existent and teachers are able to complete their instruction using mobile devices and be successful, our mission is accomplished. I know a lot of you won’t be happy until you see NUMBERS and that is fine. I am going to try real hard to get some of those and put them out there. To be perfectly honest, I don’t know if some of you will believe it then, and that is fine too.
I understand many of the things that have been said as far as physics, airtime consumption, high density, etc. I don’t necessarily disagree with some of the opinions. You may be absolutely right! The thing that is somewhat discouraging is that there are a lot of opinions based on no experience or an experience that was several years ago. Don’t get me wrong, the first time I saw virtual port, even with my limited wireless knowledge at the time, I couldn’t believe it would work. A lot has been improved upon since then with virtual cell and especially the equipment. I’m not saying Meru will beat another vendor head to head every time, but I bet it will sometimes! Does it really matter how many jigabits we can ram through the air or is it more important for a user to have an experience where they don’t even notice the wifi? A user sits down, opens their laptop, completes a task, closes the laptop and moves on without even acknowledging the presence of wifi. The wifi just works. Please understand, I am not trying to discount numbers such as channel utilization, retries, available bandwidth, etc. Those are all very important things to consider in a wireless environment. We all look to those numbers first when trouble is initially reported. Those are the numbers that give us a baseline to begin troubleshooting. They are absolutely critical to a successful wireless deployment.
Obviously interest has been piqued since the presentation. It has been fun, most of the time, discussing various aspects concerning single channel vs multi channel environments. I have heard a handful of different people give a handful of different explanations on what Meru’s special sauce is, and they are all different. There isn’t a ton of information out there concerning the “magic” of Meru but if you are truly interested please watch any video by Dr. Bharghavan, founder of Meru networks. Many of these videos are a few years old, but still offer great information. To be honest, I need to re-watch most of them as I get tangled in the “it just works,” sometimes. Below are a handful of videos.
If you want to catch the videos later but still want to read the conclusion, please scroll down.
Meru Networks Wireless Virtualization Architecture – Part 1
Meru Networks Wireless Virtualization Architecture – Part 2
Meru Networks Wireless Virtualization Architecture – Part 3
Contention Management Schemes: Part 1 – Single / Multiple AP
Contention Management Schemes: Part 2 – Multiple APs
Maximizing Air Traffic – Part 1: Maximize Channel Reuse
Maximizing Air Traffic – Part 2: Simultaneous Transmissions
Leveraging Single Channel Architecture for Multiple Channels
A Little Dated But Still Good
Very High Density Wireless LAN Demonstration for BYOD: #1
Very High Density Wireless LAN Demonstration for BYOD: #2
Conclusion
For those of us who went to WLPC 2017, we heard more than one person mention that wireless networking can be done in more than one way. We also heard that less than ideal practices may be employed against our better wireless judgement due to other factors such as politics, aesthetics, etc. Sometimes I think we need to remember that just because someone does something different doesn’t mean that it is wrong. We also need to remember that just because we don’t like a technology it doesn’t mean it doesn’t fit someone’s need. I need to remind myself of this from time to time. We deploy wireless networks, in schools, mines, warehouses, large refrigerators, outdoors; you name it, we put wifi in all kinds of places. Our ultimate goal should be to use the knowledge we have to deploy a wireless network that gives a reliable experience to the greatest number of users.
The countdown is on. There is less than a week until the Wireless LAN Professionals conference in Phoenix, Arizona. This will be the first WLPC that I attend and I am really looking forward to it. I am in the process right now of completing a presentation for a TEN Talk that I will be doing during my stay in the desert. I am looking forward to seeing a bunch of guys that I met at Cisco Live, especially a group that I have grown especially close to; Brennan Martin, Rob Boardman, and Stewart Goumans #hosers. I am also looking forward to putting more faces to names with people I have talked to via social media and haven’t met in person.
A lot of things have happened since the last blog post:
Nexus 9k ToR project in our data center
Replace all Avaya equipment with Cisco 3850s one school at a time, typically 2-3 schools a week
Applied for and selected into Cisco Champions
Preparing to open two new school buildings
Presentation to staff members helping with basic wifi troubleshooting
Surpassed 5700 access points
New high score of 68k concurrent clients on the wireless network
Preparing for Nexus 7k core install
Studying and preparing for CWDP
Became a Volunteer Examiner for the ARRL
on and on and on…
As you can see I’ve been pretty busy so WLPC should prove to be a relaxing time where I can refocus on the depths of wifi. I am also looking forward to giving everyone a little taste of how well SCA works for us.
Safe travels to all you wifi peeps that I will see in a few days!
Shout out to Michael Martin at Art Expressions in Saskatoon, Canada. He has brought at least two amazing things to this world that I know of. First, Brennan Martin, aka CdnBeacon and the badger SCA logo at the top of this post. He was awesome to work with, even after bugging him with changes to logo as he was designing it. Please check out his website. I can highly recommend his professionalism and product!
We all find bugs in software from time to time and most of the time they are fixed in relatively short order…..maybe….
It is always an unsettling feeling when those bugs resurface in later versions of code after being “fixed.”
We recently received reports from a school that the wifi was “not working.” Early on in troubleshooting I could see that stations were associated to APs in the reported troublesome area. All the normal quick checks looked ok. There wasn’t high channel utilization, retries were low, the APs were not overwhelmed with clients, etc, etc. I touched base with the person who reported the trouble and they informed me that the issue affected all devices and that it seemed to be in one specific area; the library. I again checked the APs via the web ui and upon quick glance everything still looked ok. I logged into the controller CLI and issued the station database command to see what the clients were doing specifically on the library access points. And then it got cold……..
Station database showing the dreaded TX freeze
As you can see in the graphic above the TX packets are few or none at all. This problem occurred a couple years ago in later versions of System Director 6 code and was fixed with a newer release so I had seen the problem before. I took down all of the necessary information so that I could put a ticket in with TAC then rebooted the two APs in the library. Both of the APs affected were AP 832s and both were experiencing the same problem. All other APs (AP1020s) were working fine. After the reboot the APs returned back to working order.
I began putting my documentation together to submit a ticket when we received word that another school was having wireless issues. This school was having trouble in it’s library as well. Like most of our schools, all high density areas (for the most part) are serviced by AP832s. I logged into the CLI of that school’s controller and found the same TX freeze occurring. It became obvious to me at this point that the problem was likely affecting more than these two schools. I checked a few other controllers and found most AP832s were in a frozen TX state. I started to think what the common denominators could be. All the APs affected were AP832s. All the controllers were running the same code, System Director 8.1-2-0.
And there it was….
I glanced at the uptime of the AP832s on the controllers that I had up. All of the AP832s had been up for 99 days. I started going through other controllers and checking AP832s that had been up for 99 days and found them all to be in a TX frozen state as well. A few I found that hadn’t reached 99 days were still operating normally. Clearly there was a bug that reared it’s head on the 99th day of uptime.
So you are probably asking yourself why so many of the APs had reached the 99 day uptime mark at the same time. I was wondering the same thing at first and then it hit me. I had done a mass upgrade of controllers over the summer to System Director 8.1-2-0 which mostly put all of the controller/AP uptimes in sync.
We launched a preemptive strike and rebooted all AP 832s less a group of three APs at one middle school which hadn’t reached 99 days yet. We knew that if we rebooted all of the APs the issue would be resolved for another 98 days or until FortiRu provided us a fix. All of the Ap832s returned to working order.
At that point I assembled all of my documentation and submitted a ticket to TAC. We did some initial troubleshooting but they needed to have some APs in the frozen state to get the information they needed. We put the ticket on hold until our test group of three APs reached 99 days uptime. That occurred over this past weekend, November 20. When I returned to the office I checked the APs which were at an uptime of 102 days. All three were in a TX frozen state.
AP832s with uptime of 99 plus in a frozen TX stateAs of right now, the issue is with TAC. It looks like we are the first to report the issue. They have all the information that they requested; logs, diags, etc so I should be hearing back from them soon. In the meantime, a quick reboot is just the ticket to thaw out the TX.
Here are a few commands I used to gather AP/radio data from within the AP CLI:
stadb display assigned
stadb display assigned -v <mac-addr>
stadb display rxq_info <client mac-address>
stadb display txq_info <client mac-address>
sys exec /wl -i radio0 msglevel err
sys exec /wl -i radio1 msglevel err
radio txqinfo radio0 (radio zero)
radio txqinfo radio1
radio txq radio0
radio txq radio1
radio display
radio show radio0 (radio zero) – radio specific parameters
radio show radio1 – radio specific parameters
radio stats radio0 (radio zero) – radio specific stats
radio stats radio1 – radio specific stats
dev cmd radio0 reset – resets radio without rebooting the AP
dev cmd radio1 reset – resets radio without rebooting the AP
good friends, listening to great presentations, learning a lot, and also presenting on a topic near and dear to my heart. I was able to put a lot more names to faces and have some great conversation.
