Month: March 2024

Hey guys,
We had some unexpected downtime tonight that started around 1700EST
and lasted for about two hours. We started seeing disk errors in
the system log of the hypervisor followed by VMs crashing very soon
after. We shutdown the physical server to prevent further damage to
our running VMs, but the damage to our CFX and Minecraft servers was
already done.

We do have backups made nightly during the weekdays and had to roll
these two machines back to get them playable again. The root of the
problem was a bad SATA connector that was causing the disk those
machines live on to drop to 1.5Gbps and then fall off the bus.

The issue appears to have been resolved with the help of the data
center and having our backups in place. We apologize for the interruption
in service tonight.

Hey guys,
The volunteer staff here at Marbled Fennec Networks have been busy over
the past few days processing various updates and changes to our network
and policies.

First thing that you will notice is that as of March 25th, we are no longer
allowing project members to use their own DNS servers when on network. This
change has come about due to someone who couldn’t behave and decided to make
an attempt at going places they shouldn’t go. As a result, everyone using the
project’s network will now have to rely on our internal DNS service which has
filtering deployed against unwanted domains and as a step further, we are
working towards deploying IPv4/v6 blocklist against known problematic subnets
and content servers. While we really do try to stay positioned as a neutral
carrier, we cannot allow the actions of our members to possibly put the project
in a bad light with our upstream providers. Sorry.

The second thing that is going on in the background is that we are looking into
setting up QoS based on data moved. This has been a manual process for a while,
but we are toying with the thought of automating it since we now have an NMS setup
that provides decent accounting and has an API that we can use.

The third thing that is being discussed is that we might, in the future, make it a
requirement for project members that the first address in their routed IPv6 subnet
is used as a PTR record to identify their subnet and traffic. For most subnets, we
do not use the ::0 address and instead place our router on ::1 and the member’s
first client on ::2 and so on. If the volunteer staff do vote on this policy, the
::0 address will always get a PTR record created to the effect of something like
“device_name.member_handle.marbledfennec.net”. This will help identify who the
traffic belongs to as well as which network provider it came from for the outside
world, should anything unwanted occur. With us picking up more members and seeing
more traffic routed through us, it might be time to consider accountability on
both ends of the tunnel.

Anywho, yea, that is what we have been up to.

You would think that I would be aware of the order of parsing for
firewall rules when placing them into effect at four in the morning…
Apparently not. I apologize for the issues this caused in routing and
I have swapped the rules into the correct order for processing DNS
queries on our network.

If you guys notice that things break, please email me when you first
notice that it is broken! I want to keep the ship afloat and as you
guys can see, without extra eyes on the project, I sometimes make mistakes.

—
Something else we are trying out and seeing how it works out with the DNS
resolver. Under the old method, domains that were blocked would simply be
responded to as 0.0.0.0 which makes the client have to timeout after a
while when loading webpages. On Catos, we are piloting the use of the
NXDOMAIN flag to see if it results in a better user experience when browsing
webpages that may have some of their content blocked by the DNS filtering.