Category: Issues

We are aware of issues with the secondary host!

We are aware of some ongoing issues with the network that services our secondary host which runs our status page and slave name server. The data center has been contacted and has let us know that they are having internal issues and are working to resolve it. During this time secondary services might become unusable off and on.


Additionally, our volunteers are working off and on on some updates and changes to the hypervisor on the secondary. Expect some minor outages today caused by our team.

Name Server Attack

UPDATE OCT 24TH 2024@1110EST:
It appears that the attack has stopped and we have removed the drop rules for the affected domains. We have also restored our usual name query limits.

UPDATE OCT 22ND 2024@0950EST:
The attack is still ongoing and our team will continue to drop queries for ‘cloudflare.com’ for the time being. We will perform another check around 1500EST.

UPDATE OCT 21ST 2024@2250EST:
It appears that the attack is largely focused on various regions in Brazil. While our team cannot ID the source of the attack, the destination for the requested data is pretty localized to that region. As of this time, the attackers seem to be settling for bogus TXT lookups against cloudflare.com and our team will begin dropping traffic with the hex for that domain until sometime tomorrow morning. This may break things on a temp basis as DNS queries for their domain will not resolve for users hitting our servers.


We are seeing what looks like a DNS attack at the moment, appears to have started around 1345EST until it died off near 1420EST, and then picked up again around 1700EST and it still ongoing at the time of posting. The majority of the request are TXT records for only a handful of domains. Our team will be keeping an eye on this.

We have already stepped in to significantly lower the responses per second allowed and widen the tracking bitmask to /24 for v4 and /58 for v6. These lower limits on responses will remain in effect until six hours after the attack ends in order to limit our impact without dropping our service entirely.

While the RPS is not super high, it is out of the norm for our name servers and we are reacting to it in order to limit the outgoing traffic amount and keep our name servers accessible.

Working to restore connectivity!

There were some upstream issues that resulted in us loosing power a few times. Our team will be checking disk images and slowly getting connectivity back online, please have a bit of patience with us while we work on this.

–updates–

Core and Catos routers were restored from local images and should be alive again, we are still working on Ikus to see if it was damaged or not.

Ikus and CX were also damaged, Ikus has been restored from local. CX might need to be rolled back.

IPv6 Edge router is fine and is working as should. Web, email, NMS and Minecraft all seem to be okay as well.