Our Vendor has completed all non-customer-facing fixes to their software. Our monitoring systems are functioning as expected and have been stable for a week. This event is now being closed.
If you feel your service is being impacted, please open a case with our technical support teams at www.tpx.com/support/.
Posted Feb 13, 2025 - 07:34 PST
Update
We continue to work with Broadcom on a few other non-service-impacting issues caused by the Orchestrator upgrade. We are communicating daily with their engineers until all issues have been corrected.
If you feel your service is being impacted, please open a case with our technical support teams at www.tpx.com/support/.
Posted Feb 03, 2025 - 14:10 PST
Monitoring
Our Vendor has resolved the issues VCO-176 was experiencing. We are monitoring stability and verifying functions are performing as expected. Customers with access to this VCO should have access and functions as before.
If you feel your service is being impacted, please open a case with our technical support teams at www.tpx.com/support/.
Posted Jan 31, 2025 - 05:23 PST
Update
Our vendor has engaged additional engineering resources to continue their troubleshooting. The NOC coordinated a meeting with Broadcom Engineering and TPx Engineering to discuss a possible rollback to the previous code. Although still on the table as an option, both Broadcom and TPx have decided the risk of possible loss of services to customers was to high at this time. TPx has asked our vendor to apply as many resources as they have to continue troubleshooting efforts.
If you feel your service is being impacted, please open a case with our technical support teams at www.tpx.com/support/.
Posted Jan 30, 2025 - 20:41 PST
Update
Our vendor has escalated this event to their Orchestrator Engineering team for investigation. They are in the process of restarting some services in the Orchestrator that appear to be in a "stuck" condition. They are continuing the debugging of the system to restore it to normal functionality.
Posted Jan 30, 2025 - 12:12 PST
Identified
We have identified a degradation of status communications between our SD-Wan devices and one of the Orchestrators that monitors the devices. This only affects customers on Orchestrators VCO-176. Customer traffic is not affected only monitoring and statistics gathering between the edge device and the Orchestrator. Real-Time statistics are not available for customers on this VCO at this time.
Our engineering team and vendor are actively troubleshooting this event.
If you feel your service is being impacted, please open a case with our technical support teams at www.tpx.com/support/.
Posted Jan 30, 2025 - 08:14 PST
This incident affected: Security & Managed IT Services (MSx WAN (SD-WAN)).