Packet loss at our East NJ PoP
Incident Report for Flowroute
Postmortem

Investigation results: On April 14, 2022 we were provided with the official investigation results from a Senior Interop Engineer at 3CX:

After looking into this we found that the 147.75.65.0/24 range had been added in our global blacklist. We have now removed that value but it will take approximately 6 hours for all systems to receive the change. Normally by now all affected systems must have switched to one of the other IP addresses so it should be OK. If any more customers have issues please let us know.
---Your 3CX Interop Team

Scope of impact: This incident affected some 3CX PBX systems that were directly connected to the New Jersey Edge proxy (us-east-nj.sip.flowroute.com, 147.75.65.192/28) by IP address or had cached DNS records.

Mitigation plan:

  1. Immediately escalate incidents to all involved parties (3CX Support, Flowroute Support)
  2. Whitelist all IP ranges listed in Firewall Requirements
  3. Avoid using direct IP connections and/or cashed DNS records when it’s possible
Posted Apr 15, 2022 - 11:01 PDT

Resolved
Our engineering team has done extensive testing and has not been able to reproduce the reported packet loss at the East NJ PoP. The customers that had issues with this PoP had their issues resolved when switching to the East VA PoP.
Posted Apr 11, 2022 - 12:32 PDT
Update
We are continuing to investigate this issue.
Posted Apr 11, 2022 - 09:03 PDT
Investigating
We are currently investigating reports of outbound packet loss at our East NJ PoP.
Posted Apr 11, 2022 - 08:50 PDT
This incident affected: Voice (Outbound Voice).