TL;DR - My Sonos devices were causing a spanning tree storm and knocking my network offline for two to thiry seconds a few times an hour. Removing the Sonos devices from the network fixed everything. I still need to update when I get things working with the Sonos again which I'll do today or tomorrow.
I have been experiencing sustained periods of packet loss from all the devices on my network for some time now. It got worse over the summer and I finally had some time to debug over the past few weeks. I got so much help from the prior posts here and I thought I would share my voyage and the successful conclusion.
My network setup is relatively simple:
Modem in bridge mode -> primary google puck -> unmanaged switch 1 -> unmanaged switch 2 -> 4 to 5 child wired google pucks
As you can see above all of my pucks are wired into the primary via two unmanaged switches. I have the first switch there so I can simplify my network for testin purposes and connect only my laptop into the first switch and disconnect everything downstream (including the child pucks).
The symptom was that my 2019 Macbook Pro would show as connected to the wifi, have a very strong signal (-35 to -40dBM) and have no internet connectivity. I immediately assumed that there was some interference so I added more pucks and move the primary into my office (about 6 feet from my computer). I did a variety of other crazy, invasive, annoying things like disconnecting our DECT phone system, making a tinfoil hat for a nearby puck, etc. That all improved my signal to near perfect but did not have any impact on the issue.
I wrote two scripts to diagnose the problem:
a script that would constantly ping the Google Wifi gateway (192.168.86.1 in my case) and timestamp the results
a script that would use the aiport utility I found in other posts here and log and timestamp those results. That would give me the signal and noise for my wifi and help rule out whether or not the issue was interference
// TODO #1 open source these scripts at some point
I started running both scripts on my wife's 2012 Macbook Air several rooms away as well as my computer. After less than an hour I was able to observe the issue - both computers dropped 100% of the packets to the gateway. Remember, this isn't even leaving my network so, barring some terrible config issue, this isn't related to the ISP.
While the issue occurred on both computers at the same time the signal stayed strong and the noise didn't jump. I became pretty sure that, surprisingly, this wasn't a wifi issue at all.
Next I installed a USB-C to ethernet dongle on my macbook pro and disabled the wifi. Wait an hour and same issue - 100% packet loss for 35 seconds. Ok, now things are interesting. I am connected to the puck via only an unmanaged switch in between and I can't ping the gateway!!!
I have been speaking to Google Wifi support this whole time on and off via chat and email. They are completely useless and are clearly reading a script. I tell them I have packet loss to the primary puck on my network and they tell me to change my DNS. The suggestions they gave were offensive to anyone who has the slightest clue what is going on. Nonetheless I restored my whole network to factory settings and the issue persisted. I have a bunch of servers, cameras, smarthome things, etc. running on my network with static IPs, port forwarding, etc. so the factory restore is pretty troublesome for me.
After demonstrating the simple test where I couldn't ping the gateway via ethernet I ditched the useless support and turned to this community on Reddit. I saw so many helpful posts.
This great post explained an issue with wired pucks that have poor mesh connectivity which pointed me to this google wifi support issue. I learned a lot but I couldn't tell if it was still happening in 2020 or not. The Google support people didn't know either.
This great post just did a fantastic job explaining how the wired ethernet backhaul works IMO.
More on this one later about ethernet backhaul showing as mesh.
I used Wireshark on my mac to capture anything that wasn't TCP to see if I could line up any issues when I couldn't ping the gateway. One interesting tidbit is that while I was unable to ping there were still packets flowing - ARP, etc. so the network was functioning. I saw some spanning tree comms going on leading up to the issue but I didn't feel like it was every time. Again, I wish I kept better notes and I could tell you for a fact and not a feeling.
On Friday I came up with a test and a phased approach to setting up a new network.
Modem -> Google Puck for new Network (192.168.87.1) -> unmanaged switch #1 -> Google Puck for old Network (192.168.86.1) -> unmanaged switch -> rest of child pucks and devices.
This creates the dreaded "Double NAT" issue that Google support hates so now I was really on my own beacuse if you ask them any questions they basically just respond "Double NAT" and send you a useless webpage about how to put your modem in bridge mode.
I plugged my laptop into switch #1 (the new Google network I just created). This network was now serving only 2 devices - the puck for the second network and my laptop. I ran the same script as before (pinging every second) and now things behaved like a regular network! Great news. This lasted all day. For the first time in a long time I had
After everyone was done with work and remote school on Friday I made myself a list for rebuilding my network on Saturday morning - what devices I need to check, etc. On my previous restore I gave the same SSID and wanted to eliminate that issue that there was some routing problem with a camera with bad DHCP implementation or something.
Saturday morning I factory restored all the old Google pucks so I could add them to the new network. One by one I added the switches and then the pucks to the mesh in wifi-only first and then, eventually, wired. I kept running my ping logger script throughout and it looked relatively stable. One side benefit of doing this is that, as I added each device, I could name them properly. Now I have well named devices for the thermostats, amazon echos, etc. rather than "android 1," etc.
Unfortunately, this new network started to exhibit the same issues. I'm not sure if it happened right away or not. I wish I did a better job with my experiment logging. Regardless, last night after rebuilding everything I was left with the same problems as before.
I tried some more stabbing in the dark things -> Turn off certain access points because maybe I had the weak-wifi-ethernet-backhaul firmware issue from 2018? Remove the basement point because that one was added most recently and maybe caused the issue? Turn all the wired points back into mesh points? IDK - I was desperate and nothing worked.
I was left with a setup where I had modem -> primary puck -> switch #1 -> switch #2 -> two well placed, wired child pucks + wired devices + wifi devices that could reach those pucks. Somehow the issue was worse and happening more often which I was excited about because it would be easier to test.
Since I was still having the issue in this simple setup I fired up Wireshark again. I kept adding filters for traffic I understood until I was left with basically just ARP, STP and some UPnP stuff. I was "lucky" enough to be having the issue every 15 minutes or so now and I finally saw a pattern! Sonos. I knew that most of the issues that people had with Google Wifi in an ethernet backhaul situation came down to Spanning Tree Protocol (STP) but I "knew" that there were no STP roots on my network except the primary Google puck. Well, I was wrong. Sonos was fighting with Google with some serious STP storms. It happened every time the pings stopped. Every time.
I came back to reddit and there are many posts related to Sonos+Google Wifi and some posts on the Sonos forums that point to this STP issue.
I unplugged the power on my 7 wired Sonos Connects. Got better but still had the issue. I unplugged the Sonos Player 1 in the garage. Got better but still not perfect. I removed the wifi settings in my Sonos app and turned off the Sonos Move. Perfection! The STP topology stopped changing. The packet loss stopped.
I think what was happening is that there was a loop introduced. My hunch is that happened with the addition of the Sonos Move device over the summer but I'm not sure. I have an outood AP (not google with separate SSID) that is wired into the Google network via switch #2 and the Sonos Move was connected to that. That showed up as a wired connection in Google so I think, somehow, the Sonos STP thing was getting involved.
TBH I haven't researched much how the Sonos Connects are connected - my AV installer put them in originally. Now that I understand the issue is Google Wifi + Sonos I have what I need to put the network back together and properly add Sonos back to it. For now, I'm much happier having reliable internet and no Sonos than having unreliable internet and working Sonos.
I'll update when I get the Sonos issue resolve so people know the result. A couple other things I found during my voyage that may be helpful for people:
When you do a mesh test there is more data than weak, ok, good. If you click the "..." in the upper right and click details down below you can see the throughput for each puck. You have to do it form the mesh test results screen and it doesn't persist when you are just looking at the wifi point screen. Why? Who knows. What do these rates mean? There is a lot of discussion of that around but think of them in comparative terms - low is bad and high is good - and not necessarily your actual wifi throughput you should expect.
The post above that talks about wired points showing up as mesh is important. That was an issue that I had as well. I don't know if it is just a mislabeling OR if it indicates that the puck is actually functioning as a mesh puck and not a wired puck. Regardless, the way that I was able to get it to register correctly is to go Wifi Points -> Gear -> pick any non-primary point -> Move Location. I selected the same location and just went through the steps. After doing that all of the wired points would register correctly for me.
Like I said, I'll be back when I get the whole thing working. I have to imagine that my setup of Google Wifi + wired pucks + sonos is very common in new construction homes that are pre-wired and hopefully this helps some of you.