After deploying the fix, we are still seeing a number of issues. Earlier in the day an update to Open Source Intrusion Prevention System (SNORT) rules impacted Windows based machines and servers from making outbound internet connections and, in this case, also impacted machines running the Okta AD agents from connecting to the Okta service.\r\n\r\n \r\n\r\nRemediation Steps:\r\n\r\nOnce customers were able to address the problematic SNORT rule update, the machines running Okta AD agents were able to connect to the Okta service again and resume normal operations. Therefore, the message might reference the wrong device management solution and include a link that points to the wrong enrollment website. Okta also meets FedRAMP FICAM requirements by relying on FIPS-validated vendors. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P1Y000007zHkSUAU"},"Id":"a1P1Y000007zHkSUAU","CreatedDate":"2019-10-01T03:49:19.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Workaround has been deployed to Cell EU1. Here is my code: Okta is closely monitoring as emails are continuing to be processed. If the problem continues, report the issue to Okta (right-click the app icon, and then select Report Issue). This maintenance was not applied to Custom Domain, and traffic to Custom Domains was not impacted during this incident in US Cell 10. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z00000C1dxoEAB"},"Id":"a1P4z00000C1dxoEAB","CreatedDate":"2023-04-11T15:43:17.000+0000","IncidentId__c":"a9C4z000000TXBDEA4","UpdateLog__c":"The Workflows team continues to monitor the current situation with the Workflows service. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z00000BAKwxEAH"},"Id":"a1P4z00000BAKwxEAH","CreatedDate":"2022-04-05T16:47:26.000+0000","IncidentId__c":"a9C4z000000wkCJEAY","UpdateLog__c":"Resolved: Investigation of the the LDAP connection service has been completed and service has been restored on all cells. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P2A00000Fjr2ZUAR"},"Id":"a1P2A00000Fjr2ZUAR","CreatedDate":"2018-03-30T15:49:50.000+0000","IncidentId__c":"a9C2A000000bmK1UAI","UpdateLog__c":"Okta is monitoring progress with Microsoft on this issue. Shortly thereafter, the script was disabled and removed from code. \r\n\r\n \r\n\r\nDuring the duration of the incident, customers within APAC Cell 1 and US Cell 5 would have been unable to access the workflow designer tool and background workflow processes were held in queue for delayed processing. The code change caused an unintended miscalculation of available job processing capacity for specific job types under specific conditions. \r\n\r\n \r\n\r\nRemediation Steps:\r\n\r\nAt 12:29pm PDT the impacted services were identified and action was taken to manually scale workflow services back up to expected levels within the impacted cells. Is there liablility if Alice startles Bob and Bob damages something? ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z000009mRfIEAU"},"Id":"a1P4z000009mRfIEAU","CreatedDate":"2022-11-09T20:38:06.000+0000","IncidentId__c":"a9C4z000000YzhvEAC","UpdateLog__c":"Okta continues to monitor the current issue with our partners regarding the SMS delivery delays using US-based phone numbers. The LDAP Interface service was fully restored to all customers on US Cell 5 by 4:50pm (PST).\r\n\r\n\r\nImmediately after resolution, Okta implemented updates to internal monitoring and alerting in order to close the identified monitoring gap. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P1Y000009QR31UAG"},"Id":"a1P1Y000009QR31UAG","CreatedDate":"2020-08-28T21:43:58.000+0000","IncidentId__c":"a9C1Y0000008PZGUA2","UpdateLog__c":"Root Cause Analysis: On Wednesday August 26th at approximately 7:40 AM PDT, Okta began receiving customer reports that Help desk Admins were unable to trigger a user password reset through the Okta Admin UI.\r\n\r\nUpon investigation, it was determined that Weekly Release 2020.08.2, which included a fix to remove permissions allowing Help desk Admins to expire passwords for all Okta-sourced users in the org, introduced a regression. SMS sent to other networks are unaffected. Engineering is working to identify the list of customers still impacted and is investing a way to proactively address the issue for these customers. We will continue to monitor residual effects for a limited number of customers. The mitigation procedures completed at 7:22AM.\r\n\r\nDetection and Impact:\r\nIn all Cells, some customers who leverage SMS as an authentication factor experienced intermittent delivery delays on their first request to US based phone numbers. Finally, Okta restored job processing capabilities to the affected environments. At 9:09PM service was restored in US Cell 5.\r\n\r\n\r\n\r\nRoot Cause Summary:\r\n\r\nThe root cause of the incident is the instance failures in US Cell 5. Solution: Click the Sign in with your browser instead link to access the app. At Okta trust and transparency are our top priorities. Between 4:28 PM PDT and 4:33 PM PDT, approximately 9% of end-user requests to US Cell 6 experienced latency or errors. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z000009mRhnEAE"},"Id":"a1P4z000009mRhnEAE","CreatedDate":"2022-11-16T18:28:45.000+0000","IncidentId__c":"a9C4z000000Yzi5EAC","UpdateLog__c":"Okta has provided a self-service guide to help resolve the authentication issues for Office 365 applications. SP-Initiated Single-Sign-On is not affected. Outlined below are the facts regarding this incident. This new app version was submitted to the Play Store for approval, and once approved by Google, deployed to the store as Okta Verify 7.9.2. Various trademarks held by their respective owners. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z00000BAKt0EAH"},"Id":"a1P4z00000BAKt0EAH","CreatedDate":"2022-02-05T02:39:26.000+0000","IncidentId__c":"a9C4z000000wkAXEAY","UpdateLog__c":"Okta continues to monitor the current issue with third-party SMS providers to US-based phone numbers. A subset of network traffic received 403 errors. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z000009mRV3EAM"},"Id":"a1P4z000009mRV3EAM","CreatedDate":"2022-10-17T15:07:35.000+0000","IncidentId__c":"a9C4z000000Yzh7EAC","UpdateLog__c":"An issue impacting Okta Verify Application version 7.9.1 functionality for Android devices in all cells is being investigated. You can find the knowledge article at support.okta.com. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z00000BAKkhEAH"},"Id":"a1P4z00000BAKkhEAH","CreatedDate":"2021-10-15T03:00:30.000+0000","IncidentId__c":"a9C4z000000wk4eEAA","UpdateLog__c":"Mitigation steps have been applied. \r\n\r\nThis caused SMS delivery delays and failures for the subscribers of their service. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z000009mRarEAE"},"Id":"a1P4z000009mRarEAE","CreatedDate":"2022-11-01T18:00:18.000+0000","IncidentId__c":"a9C4z000000YzhREAS","UpdateLog__c":"An issue impacting Multifactor Authentication for end users in all cells has been identified and is being investigated. This led to interruptions in flow execution due to a system fault in the affected infrastructure.\r\n\r\nRemediation Steps\r\n\r\nUpon receiving alerts, Okta immediately began diagnosing the issue. If you are using Okta SDK, re-check the config for Okta issuer and org URL. { errorCode: 'E0000006', Additionally, new tooling is being investigated to allow us to flush Redis cache should such a mitigation step be needed in the future.\r\n
\r\nReview the root cause analysis [here](https://support.okta.com/help/Documentation/Knowledge_Article/Root-Cause-Analysis-Service-Disruption-03-07-2017). The design and implementation of such improvements will be added to Okta's infrastructure roadmap. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z000009mSBxEAM"},"Id":"a1P4z000009mSBxEAM","CreatedDate":"2023-02-08T20:47:57.000+0000","IncidentId__c":"a9C4z000000YzlTEAS","UpdateLog__c":"We sincerely apologize for any impact this incident has caused on you and your business, and your customers. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z000009mRhTEAU"},"Id":"a1P4z000009mRhTEAU","CreatedDate":"2022-11-15T23:17:35.000+0000","IncidentId__c":"a9C4z000000Yzi5EAC","UpdateLog__c":"An issue impacting Microsoft O365 Federated Single Sign-On for all end users in all cells has been identified. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z000009mRX4EAM"},"Id":"a1P4z000009mRX4EAM","CreatedDate":"2022-10-20T03:22:23.000+0000","IncidentId__c":"a9C4z000000Yzh7EAC","UpdateLog__c":"We sincerely apologize for any impact this incident has caused on you and your business, and your customers. \r\n\r\nWhile the cluster reported healthy, in post incident analysis, the healthiness was not consistent within the cluster which masked visibility into the problem. Okta continued monitoring and confirmed at 1:18PM, that the service was restored and operating as expected. The deployment completed to all cells at approximately 5:30PM PDT.\r\n
\r\nOkta is implementing additional testing and monitoring to detect this condition in the future. You can also enable Okta FastPass. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z00000BAKm4EAH"},"Id":"a1P4z00000BAKm4EAH","CreatedDate":"2021-10-27T20:40:13.000+0000","IncidentId__c":"a9C4z000000wk5IEAQ","UpdateLog__c":"Our monitoring shows a return to normal conditions with MFA for customers that utilize SMS with AT&T. The guide can be found in the knowledge article at support.okta.com. Engineering is currently investigating the issue. Were reverting the update to remediate the problem","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P0Z000005XZDOUA4"},"Id":"a1P0Z000005XZDOUA4","CreatedDate":"2018-09-05T19:16:27.000+0000","IncidentId__c":"a9C0Z0000008PU1UAM","UpdateLog__c":"We are actively working with Microsoft who has reported this is related to mitigation activities from yesterday's service disruption (MO147606):\r\n\r\n\r\n\"We identified and reverted an update that was causing multiple services to be throttled. The performance of your phone is not affected. The majority of requests within Australia were successful, while connection attempts outside of Australia had higher failure rates. However, retry requests were successful. We are committed to implementing service improvements to prevent similar events in the future.\r\n\r\nImpact and Detection: \r\n\r\nThe following information applies to any customer using Microsoft Office 365 who were configured with domain federation and not using provisioning.\r\n\r\nOn November 15, 2022 at 10:37AM (PT), Okta started an application update for the Microsoft Office 365 integration to remove deprecated Microsoft Dynamics application links. \r\n\r\nPreventative Actions\r\n\r\nTo ensure this does not happen again, Okta is including additional runbook improvements and adding additional capacity to the affected infrastructure component. email developers@okta.com with your org. During this time Workflows may run with unsynchronized data and admin may see errors upon running a subset of Workflows. See below. In addition, some customers were unable to complete the Advanced API Access setup for newly created Office 365 application instances. During this time, customers may have received Http 500 errors, and Directory and IWA agents may have experienced intermittent connections. We are working with the service provider to investigate the root cause of this service disruption and will post the RCA to https://status.okta.com within 48 hours. Okta Verify Does Not Work on My New Device Okta confirmed with our downstream telecommunications providers that this was an issue with Google Voice.\r\n\r\nRoot Cause:\r\n\r\nGoogle Voice experienced issues receiving MFA one-time passcodes across the US. At 9:55 PM, the solution was deployed to all cells in the Okta service.\r\n\r\n \r\n\r\nPreventative Actions:\r\n\r\nOkta is updating our test and monitoring procedures for token replacement changes and actions related to multi-factor authentication, to prevent similar incidents. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z000007sukNEAQ"},"Id":"a1P4z000007sukNEAQ","CreatedDate":"2022-08-03T20:15:20.000+0000","IncidentId__c":"a9C4z0000012OTjEAM","UpdateLog__c":"An issue impacting Okta Workflows in US Cell-5, APAC Cell-8, and US Cell-10 was addressed. We will provide more information once available. Other potential causes can be found here: Other Okta products and features that leverage the impacted streaming service, such as User Search for entity updates and ThreatInsights for event analysis, remained available throughout the impact window and operated with partially stale data until processing delays were resolved. We are exploring a patch with a workaround while we work on a permanent fix. Okta Verify authentication doesn't function properly if HTTP Strict Transport Security (HSTS) is enabled for loopback. On February 18th at 2:12 am (PST) the fix was applied to all production cells and service was fully restored.\r\n\r\nOkta is implementing improvements in alerting to further reduce the response time for these types of integration errors as well as exploring the possibility with Microsoft to improve compatibility testing with pre-release versions of Office 365. The success rate began to gradually improve from 8:07am, and by 8:25am approximately 80% of requests were successfully returned. Users verify their identity by tapping a notification pushed to their mobile device when they're attempting to authenticate. The date and time on your device are set to automatic. Upon further investigation, it was determined that the issue was caused by an on-going AT&T cellular network incident. Further updates can be found in the admin portal under EX147785 and MO147789.\"","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P0Z000005XZU5UAO"},"Id":"a1P0Z000005XZU5UAO","CreatedDate":"2018-09-18T01:23:42.000+0000","IncidentId__c":"a9C0Z0000008PUVUA2","UpdateLog__c":"End users are currently experiencing errors when authenticating to O365. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P0Z000005XZCGUA4"},"Id":"a1P0Z000005XZCGUA4","CreatedDate":"2018-09-05T03:16:40.000+0000","IncidentId__c":"a9C0Z0000008PTwUAM","UpdateLog__c":"Resolved: An issue impacting access to *.oktapreview.com due to a \"Deceptive site\" flagging from the Google Safe Browsing list on Google Chrome and Firefox has been resolved. Remaining impacted customers have been notified via in product notification on additional actions to resolve the situation. Investigation revealed that this issue impacted custom URL domain users on US Cell 12. The team continues to monitor the service.\r\n\r\nAdditional root cause information will be available within 2 Business days. Android Okta Verify Cause When a user has enrolled in Okta Verify and the push notifications are not received and the codes fail, the device time may be out of sync with server time. Okta recommends affected AT&T mobile customers use an alternate second factor as a work around to the issue. On November 16 at 12:51AM (PT), Okta published a knowledge article that customers could use to resolve the Microsoft O365 Federation SSO issue.\r\n\r\nOkta continued to pursue alternative ways to restore authentication without requiring customer action, but eventually concluded that reconfiguring customer domains via the knowledge base article was the best path forward.\r\n\r\nImprovements and Preventative Actions:\r\n\r\nWe understand that this is disruptive to your business and are fully committed to taking the necessary steps to prevent recurrence and improve. Okta subsequently corrected agent status for all customers by approximately 5:00pm PST.\r\n\r\nUpon investigation, Okta determined that a mistake was made when updating references to the database system during the execution of a planned routine database maintenance, This misconfiguration caused a very small number of requests to return an HTTP 500 error code and also resulted in the outdated agent status information.\r\n\r\nOkta has updated run-books and is improving the automation of maintenance verification procedures to prevent future occurrences. At 10:50AM Oktas infrastructure provider began to automatically limit the volume of traffic to US Cell 6 to mitigate impact to downstream services. Additional root cause information will be available within 2 Business days. To mitigate the Multifactor redirect loop issue, please refer to the article: https://support.okta.com/help/s/article/How-to-Clear-Cookies-for-a-Specific-Site?language=en_US. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z00000BAKiHEAX"},"Id":"a1P4z00000BAKiHEAX","CreatedDate":"2021-08-31T19:21:19.000+0000","IncidentId__c":"a9C4z000000wk3CEAQ","UpdateLog__c":"We continue to investigate the increased error rates in US Cells 5, 7, 12, and 14, and Preview 3. Root Cause Analysis is underway and we will post full RCA details here within 48 hours. This prevented external services (ASA, Okta) from authenticating user sessions against Google as an IdP, and no workarounds were provided. A delay in event hook delivery has been identified and is being addressed. At the same time, Okta began receiving reports from customers who were impacted in the following scenarios:\r\nWorkflows Console - Unavailable from 12:45 pm - 3:50 pm\r\nWorkflow API calls experienced timeouts from 12:55 pm - 2:25 pm. Your Okta Verify account might still be tied to your original app installation. Credentials are earned by passing an Okta certification exam, series of exams, or by fulfilling other performance-based activities. Customers experiencing any issues related to API Access Management should contact Support. In addition, please continue to contact support if you are having difficulty setting up O365 provisioning and we will assist with implementation of an applicable workaround for that problem. Additional root cause information will provided within 48 hours. We are committed to implementing improvements to the service to prevent future occurrences of this incident.\r\n\r\nDetection and Impact:\r\n \r\nOn December 15th at 7:14AM (PT) Okta detected errors occurring across US Cells 5, 7, 12 and 14.\r\n \r\nDuring the incident, some customers experienced intermittent network connectivity errors, timeouts and increased request latencies. \r\n\r\nPreventive Actions:\r\nOkta is taking steps to improve the speed of detection and implementation of corrective action. We will provide an update in 30 minutes or sooner if additional information becomes available. \r\n \r\nRoot Cause Summary:\r\nOkta determined that the root cause was a maintenance operation to add router capacity in US Cell 10. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P1Y000007zHnbUAE"},"Id":"a1P1Y000007zHnbUAE","CreatedDate":"2019-10-01T21:26:19.000+0000","IncidentId__c":"a9C1Y0000004DyLUAU","UpdateLog__c":"Workaround has been deployed to Cell OK5. Users aren't able to remove their account from Okta Verify if they're deleted from Active Directory. Okta is closely monitoring as emails are continuing to be processed. Additional root cause information will be within 2 Business days. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z00000C1dxjEAB"},"Id":"a1P4z00000C1dxjEAB","CreatedDate":"2023-04-11T14:38:57.000+0000","IncidentId__c":"a9C4z000000TXBDEA4","UpdateLog__c":"At 7:20 AM PDT, the Workflows team reconfigured the Workflows service to remediate the issue, and we are currently monitoring to validate the improvement of the service.\r\n\r\nWell provide an update in 30 minutes or sooner if additional information becomes available. In addition Okta expanded alerting for issues related to the network rule changes for more rapid detection and response, and updating internal tooling bug fixes. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z000009mRe0EAE"},"Id":"a1P4z000009mRe0EAE","CreatedDate":"2022-11-06T20:06:18.000+0000","IncidentId__c":"a9C4z000000YzhlEAC","UpdateLog__c":"Okta is continuing to work on resolving the issue with email delivery to Yahoo end-users.\r\nImpacted domains include: yahoo.com (yahoo.co.uk, etc. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P1Y000007zKapUAE"},"Id":"a1P1Y000007zKapUAE","CreatedDate":"2020-02-25T23:11:00.000+0000","IncidentId__c":"a9C1Y0000004EBeUAM","UpdateLog__c":"Resolved: Okta has addressed the delayed user import jobs issue on Okta Preview 1, US Cell 6, and US Cell 7. Customers experienced very long delivery delays, or in some cases did not receive their one time passcode requests through SMS.\r\n\r\nOkta confirmed with our downstream telecommunications providers that this was a network issue with multiple US carriers.\r\n\r\nRoot Cause Summary:\r\nUS carriers experienced network issues across the US for messages sent through a short code, which caused SMS delivery delays and failures for the subscribers of their services.\r\n\r\nRemediation Steps:\r\nCustomer Support advised customers with a secondary MFA factor enrolled to utilize that method. Additional root cause information will be available within 2 Business days. On March 29th at 10:39AM(PT) the certificates expired, impacting the service. ","Status__c":"Resolved","Start_Date__c":"2023-02-02","End_Date__c":"2023-02-02","Last_Updated__c":"2023-02-07T08:48:45.000+0000","LastModifiedDate":"2023-02-07T08:48:45.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Small","Incident_Title__c":"US Cell-7 network connectivity issues","Impacted_Cells__c":"okta.com:7","Impacted_Audience__c":"Admin;API Products;End user","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2023-02-02T14:42:00.000+0000","CreatedDate":"2023-02-02T14:43:53.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v57.0/sobjects/Incident__c/a9C4z000000YzkkEAC"},"Id":"a9C4z000000YzkkEAC","Duration__c":8204,"Log__c":"Okta became aware of IdP-initiated login (login via Okta Dashboard) in Microsoft O365 app issue affecting customers in all cells.
Graco Totbloc Playard,
Acqua Di Parma Colonia Splash,
Pediatric Gastroenterologist Cornell,
Articles O