Customer cases have been closed. Resolving this major Incident.
Posted Aug 09, 2024 - 22:54 PDT
Update
We continue to see no latency or packet loss for congestion across our provider's network to their AT&T peering point. Our TPx Support team will contact customers to confirm the issue is no longer present. The TPX NOC will continue following up with our provider for a cause and resolution of the next steps to prevent this issue from reoccurring.
If you feel your service is being impacted, please open a case with our technical support teams at www.tpx.com/support/.
Posted Aug 09, 2024 - 16:03 PDT
Monitoring
We have been monitoring our internet drain connection with our provider and have seen no further latency or packet loss. At approximately 9:00 pm Pacific Time, the congestion appears to have cleared. We have continued testing throughout the night and as of 10:00 am Pacific Time, we are still not seeing the latency and packet loss that occurred yesterday.
TPx will continue to monitor the connection throughout the day.
Our Provider has not received a root cause for the congestion on their peering point with AT&T. We continue to escalate for a root cause and corrective action taken.
Posted Aug 09, 2024 - 10:23 PDT
Update
We are seeing 3 to 5 percent packet loss, down from 9 percent, for AT&T traffic going over our providers connections out of Palo Alto and Los Angeles. TPx Engineering is reviewing a plan to adjust preferences for AT&T traffic over the two internet drains and will be able to apply them if necessary tomorrow. In the meantime, the NOC will continue to escalate with our provider for their Peering Management and traffic engineers to determine what can be done on their connection to AT&T.
Posted Aug 08, 2024 - 16:42 PDT
Identified
Our provider has confirmed they are seeing congestion at their 200 GB peering connection to AT&T. They are escalating the issue to their Peering Manager for direction and to see if any alternate routing or traffic shifting can be done. We have not ETTR for this congestion to be cleared.
Posted Aug 08, 2024 - 14:38 PDT
Investigating
We are experiencing a Major service disruption impacting internet speeds in multiple markets.
Users may be experiencing slow internet speeds. We have narrowed the problem to one of our Internet Drain providers. Our connection to them appears to not be the issue as we are seeing high latency further down the path.
Our Engineering and Operations teams are engaged with our Provider technical support team to investigate this issue.
If you feel your service is being impacted, please open a case with our technical support teams at www.tpx.com/support/.
Posted Aug 08, 2024 - 13:20 PDT
This incident affected: Connectivity (3rd Party Internet Access).