tag:status.leveloneservers.com,2005:/historyLevelOneServers Status - Incident History2024-03-29T05:23:55ZLevelOneServerstag:status.leveloneservers.com,2005:Incident/203451502024-03-24T20:50:40Z2024-03-24T20:51:06ZDallas R04 - Partial Power Outage<p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>20:50</var> UTC</small><br><strong>Resolved</strong> - One of the PDUs in the rack was knocked offline effecting all servers connected to it.<br />Power was restored 18 minutes later.</p>tag:status.leveloneservers.com,2005:Incident/203032802024-03-23T11:00:56Z2024-03-23T11:00:56ZDallas - Subnet Replacement<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>10:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>13:56</var> UTC</small><br><strong>Update</strong> - All affected clients were emailed with new IP information. Please make sure to update to the new IPs.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>17:52</var> UTC</small><br><strong>Scheduled</strong> - During the maintenance we will be replacing the subnets that were involved in the outage (https://status.leveloneservers.com/incidents/dztqmbrlwlnm).<br /><br />If you were affected by the issue, you will receive a ticket with new IP information for your server.</p>tag:status.leveloneservers.com,2005:Incident/203019522024-03-19T17:53:28Z2024-03-22T13:55:21ZDallas - Partial Network Outage<p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>17:53</var> UTC</small><br><strong>Resolved</strong> - We are working on adding new subnets from a different IP broker to the network to replace the ones involved.<br />Maintenance schedule: https://status.leveloneservers.com/incidents/677frm3zx5n0</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>15:35</var> UTC</small><br><strong>Monitoring</strong> - Path (the IP broker) recreated the ROA records. Connectivity to the affected subnets was restored.<br /><br />We will be scheduling a date to move off these subnets.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>14:55</var> UTC</small><br><strong>Identified</strong> - We are aware there are 2 subnets not working in Dallas. It appears the IP broker revoked the ROA records, we are awaiting a response from them and looking on alternative solutions.</p>tag:status.leveloneservers.com,2005:Incident/198948252024-02-02T17:01:15Z2024-02-02T17:06:03ZDallas - Network Outage<p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>17:01</var> UTC</small><br><strong>Resolved</strong> - Due to an change with upstream filtering a 5 minute outage occurred causing our prefixes to not be advertised to certain upstreams.<br />This has been resolved.</p>tag:status.leveloneservers.com,2005:Incident/197646142024-01-20T00:13:52Z2024-01-21T07:27:33ZDallas - Packet Loss<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>00:13</var> UTC</small><br><strong>Resolved</strong> - This has been resolved and we have enabled Path's protection again.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>11:54</var> UTC</small><br><strong>Identified</strong> - We are aware of packet loss issues in Dallas.<br />We have identified this is an issue with Path (our ddos mitigation provider).<br />We are investigating this with them and will update once they follow up.</p>tag:status.leveloneservers.com,2005:Incident/195347602023-12-26T16:35:23Z2023-12-26T16:35:23ZAMS01 - Connectivity Issues<p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>16:35</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>13:45</var> UTC</small><br><strong>Monitoring</strong> - Connectivity is fully restored, we are currently monitoring the status.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>13:27</var> UTC</small><br><strong>Update</strong> - It appears a DDoS attack started a few minutes after disabling our mitigation. We are working with our upstream to resolve it and return full connectivity.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>13:08</var> UTC</small><br><strong>Identified</strong> - We've identified there is an issue with our 3rd party DDoS mitigation provider. We have temporarily disabled it and are reaching out to them.<br /><br />Connectivity is restored at this time.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>13:05</var> UTC</small><br><strong>Investigating</strong> - We are investigating connectivity issues while reaching AMS01.</p>tag:status.leveloneservers.com,2005:Incident/184577922023-09-10T13:07:46Z2024-01-20T00:14:43ZAMS01 - Power Outage<p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>13:07</var> UTC</small><br><strong>Resolved</strong> - One of the ATS devices failed causing a power outage while it was being repaired.</p>tag:status.leveloneservers.com,2005:Incident/182994232023-08-31T23:25:51Z2023-08-31T23:25:51ZAMS01 - Network Outage<p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>23:25</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>23:27</var> UTC</small><br><strong>Update</strong> - Path updated on their website https://status.path.net/incidents/h5qhs6xpqq08.<br />We are awaiting additional updates.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>22:45</var> UTC</small><br><strong>Identified</strong> - We have identified the issue is related to Path. The subnets were moved to upstreams while we await a reply from Path.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>22:42</var> UTC</small><br><strong>Investigating</strong> - We are looking into the network issue in AMS01.</p>tag:status.leveloneservers.com,2005:Incident/182973142023-08-30T17:55:33Z2023-08-30T17:55:33ZTampa Datacenter | Hurricane Idalia<p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>17:55</var> UTC</small><br><strong>Resolved</strong> - The storm has passed without any disruption to services.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>19:13</var> UTC</small><br><strong>Monitoring</strong> - Hello Everyone<br /><br />As Hurricane Idalia approaches, we want to reassure you that our Tampa servers are well-prepared to weather the storm. While the hurricane is expected to impact areas north of Tampa, our data center taking all necessary measures to ensure services remain online and unaffected.<br /><br />Our datacenter provider, Hivelocity, has activated their emergency action plan, which means:<br /><br />Fuel Supply: In case of unstable or lost grid power, diesel fuel is secured to sustain their data center throughout the storm.<br /><br />On-site Staff: Hivelocity will have a dedicated team present throughout the storm to monitor and manage datacenter facilities and client needs.<br /><br />While steps to ensure 100% service availability are taken, we also recommend having your backup plans ready, just to be extra cautious.<br /><br />Your trust is essential to us, and we're here to support you. If you have any concerns or questions, please contact our support team.<br /><br />For those clients in Florida, stay safe and stay connected.</p>tag:status.leveloneservers.com,2005:Incident/181950992023-08-19T17:37:55Z2023-08-19T17:37:55ZDallas, TX - Network Outage<p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>17:37</var> UTC</small><br><strong>Resolved</strong> - Path has corrected the issue and it is back online.</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>15:55</var> UTC</small><br><strong>Update</strong> - It appears there is an issue with Path in AMS as well, we've disabled Path on our network.</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>15:35</var> UTC</small><br><strong>Update</strong> - Path is investigating the issue: https://status.path.net/incidents/50t7b6ts412f</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>15:21</var> UTC</small><br><strong>Update</strong> - We have moved traffic to go through upstream instead of Path while we are awaiting Path's response.</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>15:15</var> UTC</small><br><strong>Identified</strong> - It appears there is an issue with Path, we are in contact with them.</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>15:10</var> UTC</small><br><strong>Investigating</strong> - We are investigating the network outage in Dallas.</p>tag:status.leveloneservers.com,2005:Incident/180602652023-08-05T13:37:57Z2023-08-05T13:37:57ZDallas R02 Outage<p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>13:37</var> UTC</small><br><strong>Resolved</strong> - The PDU breaker tripped, it has been reset and power has been restored.</p><p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>13:18</var> UTC</small><br><strong>Investigating</strong> - It appears there is a partial power outage in R02, we are investigating.</p>tag:status.leveloneservers.com,2005:Incident/180524222023-08-04T18:40:24Z2023-08-04T18:40:33ZAMS01 - Power<p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>18:40</var> UTC</small><br><strong>Resolved</strong> - The ATS has been repaired by the engineer. No downtime was recorded.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>18:17</var> UTC</small><br><strong>Update</strong> - An engineer from the ATS company arrived on site and is investigating.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>16:22</var> UTC</small><br><strong>Update</strong> - The mains are back up but the ATS is failing to switch to it, engineers were dispatched to the site.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>15:34</var> UTC</small><br><strong>Monitoring</strong> - The datacenter has notified us they have an issue with their mains and we are currently running on backup power.</p>tag:status.leveloneservers.com,2005:Incident/176341112023-06-21T02:21:33Z2023-06-21T02:21:33ZDallas Heatwave | ERCOT Curtailment Request<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>02:21</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>19:39</var> UTC</small><br><strong>Monitoring</strong> - Our Dallas, TX datacenter has sent us the following:<br />"Dear Clients,<br />ERCOT has issued a curtailment request for today, 6/20/2023, due to the extreme heatwave that we are experiencing. From 4 to 8PM CST today we will be transferring our load to the generators that feed both DC1 and DC2. We will have staff onsite to ensure that all systems operate as designed and we expect no impact to any client services."</p>tag:status.leveloneservers.com,2005:Incident/174398952023-06-01T09:54:45Z2023-06-01T09:54:45ZDallas Rack 1 switch offline<p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>09:54</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>07:31</var> UTC</small><br><strong>Identified</strong> - We are working to resolve an issue with Dallas, rack 1 switch.</p>tag:status.leveloneservers.com,2005:Incident/169338052023-04-21T23:16:45Z2023-04-21T23:16:45ZDallas - Packet Loss<p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>23:16</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>21:44</var> UTC</small><br><strong>Monitoring</strong> - The attack is being mitigated and things are stable now.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>21:39</var> UTC</small><br><strong>Identified</strong> - Dallas is currently under a large DDoS attack and is suffering from packet loss. We are working with our upstream to mitigate the attack.</p>tag:status.leveloneservers.com,2005:Incident/168589652023-04-14T02:25:56Z2023-04-14T08:00:06ZDallas - Partial Network Outage<p><small>Apr <var data-var='date'>14</var>, <var data-var='time'>02:25</var> UTC</small><br><strong>Resolved</strong> - Our team is awaiting reply from our upstream regarding the issue however it appears to be resolved.</p><p><small>Apr <var data-var='date'>14</var>, <var data-var='time'>02:16</var> UTC</small><br><strong>Investigating</strong> - We are aware of a partial network outage in Dallas. It appears there is an issue with our upstream provider Hivelocity, we are investigating this with them.</p>tag:status.leveloneservers.com,2005:Incident/167833892023-04-07T22:16:05Z2023-04-07T22:16:05ZTampa Connectivity Issues<p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>22:16</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>19:41</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>18:32</var> UTC</small><br><strong>Investigating</strong> - We are aware of connectivity issues in Tampa due to our 3rd party mitigation provider and we're working on fixing it.</p>tag:status.leveloneservers.com,2005:Incident/164273312023-03-10T22:19:27Z2023-03-10T22:19:42ZDallas R08 Network Maintenance<p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>22:19</var> UTC</small><br><strong>Completed</strong> - Unfortunately, this did not go as expected.<br />We've ran a fiber cable between new network equipment and when connecting the QSFP our aggregation switch kernel panicked which resulted in an outage for r08.<br />We'll be looking into alternatives to fix the issue.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>22:09</var> UTC</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>22:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>21:40</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time, no downtime is expected.</p>tag:status.leveloneservers.com,2005:Incident/163802882023-03-07T21:40:52Z2023-03-07T21:40:52ZOutage in Dallas<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>21:40</var> UTC</small><br><strong>Resolved</strong> - While we were performing maintenance on our core network equipment, a human error caused the primary upstream link to get blocked.<br />DC staff were deployed to regain access via the serial console, after which, full network access was regained.<br />We extremely apologize for the issue and we'll look into the cause and setup additional filters in place to prevent this from happening in the future.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>20:23</var> UTC</small><br><strong>Update</strong> - Everything has been online and stable for the last 30 minutes.<br />Our team continues to investigate what caused the outage.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>19:52</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>19:29</var> UTC</small><br><strong>Investigating</strong> - Our team is currently investigating a known outage in Dallas.<br />At this time we are still investigating.<br />What we do know is this was not a DDoS attack, or a power outage.<br />Our team will continue to update this as we know more.</p>tag:status.leveloneservers.com,2005:Incident/162712212023-02-24T22:03:09Z2023-02-24T22:03:09ZDallas - R08 Packet Loss<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>22:03</var> UTC</small><br><strong>Resolved</strong> - We are unable to declare what caused the initial issue, we'll continue monitoring the switch.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>21:20</var> UTC</small><br><strong>Monitoring</strong> - We've power cycled the switch, and the networking on the rack is now online again.<br />We'll be investigating the initial cause of the issue.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>21:06</var> UTC</small><br><strong>Identified</strong> - It appears a switch in the rack failed, we're investigating the cause.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>20:50</var> UTC</small><br><strong>Investigating</strong> - We're looking into the packet loss in R08</p>tag:status.leveloneservers.com,2005:Incident/162708042023-02-24T17:00:00Z2023-02-24T20:00:01ZTampa - High Latency<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>Resolved</strong> - A congested route between Tampa and Miami caused high latency. We've routed traffic around and the issue is now solved.</p>tag:status.leveloneservers.com,2005:Incident/161895612023-02-18T20:45:39Z2023-02-18T20:45:39ZDallas R08 Outage<p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>20:45</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>19:38</var> UTC</small><br><strong>Monitoring</strong> - The PDU was replaced and all services are up, we're monitoring the rack.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>19:35</var> UTC</small><br><strong>Update</strong> - The team has narrowed down the issue to be one of the rack's PDUs.<br />The PDU is currently being replaced.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>19:05</var> UTC</small><br><strong>Identified</strong> - It appears there is a power issue in the rack. On-site staff team is looking into it.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>18:59</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>18:54</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.leveloneservers.com,2005:Incident/160271892023-02-05T18:00:52Z2023-02-05T18:53:40ZDallas R07 - Emergency PDU Maintenance<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>17:27</var> UTC</small><br><strong>Scheduled</strong> - We'll be replacing a faulty PDU in r07. The rack will be partially offline during a small window in the replacement procedure.</p>tag:status.leveloneservers.com,2005:Incident/160047942023-02-03T09:26:36Z2023-02-03T09:26:36ZDallas r04 Outage<p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>09:26</var> UTC</small><br><strong>Resolved</strong> - A flash fault in the primary switch of R04 caused downtime. The issue is now marked as resolved.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>09:59</var> UTC</small><br><strong>Monitoring</strong> - Our switch in Dallas R04 had an issue early AM EST. Rack 04 is now fully back online. Our networking team is investigating the cause of the outage and will update once more information is found.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>09:28</var> UTC</small><br><strong>Update</strong> - We are investigating an isolated issue with Dallas rack R04. Our datacenter techs have been dispatched to the rack to investigate the outage.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>09:23</var> UTC</small><br><strong>Investigating</strong> - We are investigating the outage in r04.</p>tag:status.leveloneservers.com,2005:Incident/159891342023-02-01T22:54:17Z2023-02-01T22:54:17ZMaintenance Notification - TPA2 UPS battery replacement<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>22:54</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>22:53</var> UTC</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>14:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>21:45</var> UTC</small><br><strong>Scheduled</strong> - NO DOWNTIME OR OUTAGES IS EXPECTED<br /><br />Our datacenter has sent us the following message:<br /><br />"<br />Attention:<br />———————————————<br />We will be replacing the batteries in each of the UPS systems at our TPA2 data center throughout tomorrow. Our TPA2 data center is outfitted with N+2 UPS redundancy so no impact to your service is expected.<br /> <br /><br />Maintenance Window:<br />---------------------------------------------<br />2/01/23, 9:00AM - 5:00PM EST<br /><br /><br />Summary:<br />———————————————<br />In order to provide you with the best service and performance we are routinely required to perform maintenance to our facility critical power assets. Tomorrow we will be replacing older batteries with new batteries in each of our UPS systems inside TPA2. This update will ensure our ability to continue providing our customers with the utmost reliability."</p>