What an exciting, eye catching headline!
We took a call from a potentially new customer who were unhappy with their current support provider.
I went to take a look at the system and the main technical issues they were describing all seemed to be DNS related.
There were immediately several things that jumped out to me that would cause problems with DNS resolution so I was understandably confident of getting to the bottom of the issues.
So prospect becomes customer. We do some work to fix various bits and pieces and all appears well.
Couple of nights ago our monitoring agent reported the Domain Controller was offline.
So I logged onto the server and immediately spotted that the monitoring agent was having trouble resolving DNS.
I couldn’t find anything in the event log that pointed to a problem, other computers and servers were able to do DNS fine but the domain controller itself was unable to do any name resolution against itself.
After doing various troubleshooting steps I eventually rebooted the server, everything came up and was working fine. So I left it at that though I was fully expecting it to happen again because I hadn’t actually changed anything.
What I’d initially thought were the cause of the DNS issues were clearly only part of the problem.
Fast forward a couple of days and the same thing happened again.
So I duly logged on and confirmed exactly the same thing
Since the DNS server was operating fine for other devices I connected to an external DNS server using nslookup just to check the DNS client itself was ok. That worked without issue.
I then used nslookup against the loopback address (127.0.0.1)
DNS resolved fine that way too.
Feeling a little confused I double checked all the things I’d previously identified could be a DNS issue. Everything seemed in order on that front.
I was just about to restart the server again when I remembered there was something on the server that was a bit of an unknown to me – the anti-virus.
Specifically what was unknown to me was the product
They were using MalwareBytes Premium. (version 3.0.5)
Now I’m fully aware of what MalwareBytes is but I’ve never seen it installed on a server. I am aware they have paid for home and business products but my experience is limited to the free home product.
So I disabled the software.
DNS returned to normal!
I enabled the software. DNS fell over again.
So after a bit of trial an error it turned out to be this setting
Kind of makes sense. Like a lot of anti-virus products it intercepts your DNS request, passes it to their own DNS servers and checks to see if the website you are trying to visit is on their block lists.
Why this issues only surfaced every couple of days I’m not sure (and I only saw it twice..so I don’t know if there is any pattern to it)
But the secondary issue is that MalwareBytes Premium isn’t even supported on server operating systems (2012 R2 in this case)
So the fact it’s causing a problem isn’t a surprise
I also found it specifically mentioned on their support forum
According to that they shouldn’t have even been able to install it on the server so that’s kind of interesting all by itself.
So as it’s not supported I’ve removed it from the server.
Server has been fine since!