IT Issue, Oddest I Have ever seen.
NWCCTV
Posts: 3,629
Having been in IT for 15 years I thought I had seen it all until now:
Here's the scenerio. I have a client that is using MS Small Business Server 2003. No changes have been made. Using dual NIC's, one for Internal Network, one for external. This has been working for years. Last Friday I received a call that they were not receiving email. I remoted in to the Server, which works just fine. Here's the kicker. Suddenly they could not reach their domain name that is hosted by a web hosting company and resold through me. My own web site is on the exact same server. They were unable to bring my site up either. From my office I am able to bring up both sites and email works fine. After some calls to Comcast tech support we were told a hundred different solutions, none of which worked. I went on site and connected my laptop directly to the modem. Still no luck but all other sites worked fine. After 2 days of waiting for a call back from Comcast, I once again connected my laptop directly to the modem and suddenly their web site is now working but mine, which is hosted on the same server and uses the same DNS still will not come up. When performing an NSLookup for my site from inside their network all shows OK. So, any of the other IT people here ever seen anything like this? Like I said, NO changes had been made at all.
Here's the scenerio. I have a client that is using MS Small Business Server 2003. No changes have been made. Using dual NIC's, one for Internal Network, one for external. This has been working for years. Last Friday I received a call that they were not receiving email. I remoted in to the Server, which works just fine. Here's the kicker. Suddenly they could not reach their domain name that is hosted by a web hosting company and resold through me. My own web site is on the exact same server. They were unable to bring my site up either. From my office I am able to bring up both sites and email works fine. After some calls to Comcast tech support we were told a hundred different solutions, none of which worked. I went on site and connected my laptop directly to the modem. Still no luck but all other sites worked fine. After 2 days of waiting for a call back from Comcast, I once again connected my laptop directly to the modem and suddenly their web site is now working but mine, which is hosted on the same server and uses the same DNS still will not come up. When performing an NSLookup for my site from inside their network all shows OK. So, any of the other IT people here ever seen anything like this? Like I said, NO changes had been made at all.
Comments
1. Are you using a certain freebsd based firewall and using that as dns server (NS)... try restarting the dns server (tinydns?) service and also if you have 1:1 nat and override firewall rule to dns server sharing an IP, then re-apply the firewall rule. If that does not make sense, probably not the problem.
2. Did a recent windows automatic update change something (especially a problem on older MS versions)... try to roll back before the last update, or better still just ensure the links in the chain from internet to the webserver are all good. Try to check every link. could be a firewall, could be a dns server, could be permissions on the firewall only allowing connection from certain ip ranges (something silly like that). A recent MS update might have started the basic firewall for you (!), which might not be great in your situ.. try switching it off for test.
3. Any ip changes? Is iis or apache bound to the old public ip or wrong interface... check the web server conf. files.
Gosh, could be lots of things... but as mentioned just work methodically testing each link in the route which ever way you can to find the point where things go wrong. It's a simple thing really when you step back and deal with it all in little blocks!
couple tools I often use to help through up dns / routing issues:
http://dnscheck.iis.se/
http://www.webdnstools.com/dnstools/dns-lookup-results
Good luck!
Edit: I Figured it out!! The thing that threw me was the fact that when connecting my laptop directly to the modem all web sites worked with the exceptoion of mine. The Companies web site worked and is hosted on the same server and DNS as my own. Turns out it was the cable that was connected from the modem to the router/my laptop. What a relief!!!!!
Still the oddest issue I have come across in my 15 years of doing IT work.
They FINALLY agreed to send a tech with a new modem. After he was on the phone with his direct tech support line for over an hour doing everything that I had already done, they finally decided to change the modem. The modem was set to factory defaults with no static IP address. As soon as it was up and running everything worked just fine. However, once they entered the static IP info again it all went back to the same issue.
So, They set the modem up with a new static IP address and everything works now. Apparently, for some unknown reason the static IP that was being used was not forwarding DNS correctly. Very odd indeed but it is now working. I said from the begining it was an ISP issue but they (as always) pointed the finger elsewhere. I made sure I let them know that I was not real happy with the support they gave. I have been on and off the phone with them for a week and every time I was told something different. I told my client if I were them I would forward my bill to Comcast for payment!!!! However, I am just relieved that it is finally solved.
A friend of mine here in Taiwan claims the US Department of Homeland Security is now playing around with blocking some IP#s to DNS servers. The IP#s are still operational, but the DNS servers apparently divert to a government site. That may not be the case here, but there is one more finger in the DNS servers ot confound the issues.
Whatever is going on, the WWW is being fragmented by regionalization. There are a growing number of sites in the US I can't reach from Taiwan. I just wonder who is doing what. I never wanted a search engine to just provide me with advertisers and local news.
For instance, if you are traveling abroad, you might find that you cannot reach your Senator or Congressman via email. And there are very real situations where that might be the only person that will help an American citizen abroad with immediate results.
Seems to me there is some goofy DNS issue in play here.
Edit: I have also tried a few of the free DNS services with the same results.
But if it is an unstable router issue, you might try using a router with Linux, such as OPENWRT firmware installed to monitor the problem.
I do realize that Cisco may be considered the 'gold-standard' of secure routers, but if that has been buggy or hacked or defective ... you need a router with an OS that is more visible.
Using a customized LInux router might even become a useful service tool in the future. As you are free to load up diagnostic software and use it to monitor and debug all sorts of behavior. As it is, you seem to have a black box solution of just swapping routers and getting lucky.
Of course, there is yet another issue, the ADSL modem may be rebooting due to terminated leases or power down by someone or something on site, and the ISPN loosing track of the IP# associated with the DNS server. Or browsers could be looking to old IP#. Have you lost your static IP# with your service provider?
I truly do hope this is just a case of an ADSL modem that wore out and needed replacement. After all, they do sit around in an on state for years without anyone thinking about them and they do have wall warts that are capable of failure as well.
Isn't that the company that advices against using any form of router between the modem and your PC(s)?
Cable modems...
DOCSIS 1 & 2 is inherently flawed when it comes to security. It's also way too easy for anyone on the same network to spoof originator and to 'listen in' on anything that goes on.
Here's the document from the Cable-modem hack demonstrated(Live) at HackCon #4 in 2009
www.packet-o-matic.org/.../sniffing-cable-modems-hackcon4.odp
The 'Cisco E1200' is now Linksys E1200 as Cisco is separating the 'home division' stuff from the corporate goods.
I'd study the logs in it very carefully.
When you replaced the router, did you also replace the wall-wart?
Can you see any difference in your PCs DNS/domain settings when connected to the modem directly compared to when connected through the Router?
Linksys E1200 can be loaded with Linux router firmware and used as a custom tool.
Always give the webserver a static internal IP outside the routers dhcp window.
Can be DNS problems, try actual IP address.
If port 80 and 8080 does not work, try another as your internet provider may be blocking them.
Nope. Works just fine from my office.
Nope. This is all happening when I am on site and no DHCP is used in either modem or router.
Yes
No, I am able to ping most all DNS servers on outside when laptop is connected. It is mainly the companies and a couple other sites that are not pingable.
Internal Server not even connected. Direct Laptop connection to modem.
IP Address does not work and is not pingable when router connected. Just fine when using just the laptop. When router connected a Trace Route fails at 14 Hops but completes with direct laptop connection.
.This works fine. I am able to remote in from the outside with no issues. ALL web sites work when I only have Laptop connected. As soon as router is connected is when it fails.
This is the most INSANE issue I have had in 15 years. I am going to try a different router later today. For now I have them directly connected from Server to Modem. I do not like doing this but client needs to be able to function. I read an article last night that said to change the routers MTU setting. What I do not get is why it works fine in my office but fails when connected to the Comcast modem.
It's probably because the LAN IP of the modem sees your laptop on a different subnet and treats it as a remote connection in which case ICMP echo would be blocked for security. To ping the modem from the LAN make sure you're pinging your WAN gateway, if you weren't.
Sounds like you got your hands full... Tlime to make robots!
.
Oh and BTW, the original router that suddenly stopped working was a Cisco Pix. Starnge situation all around.
From what I understand, your ISP is supposed to be providing 24/7 service to a static IP#, but what is supposed to be and how the service provider goes about fulfilling their contract can be two different things.
The truth is that I have no idea what lurks inside an ADSL modem, but I have been regularly forced to reset mine to reach the internet. It is rather annoying and seems that the cycle is that once shut down it requires the router to reboot in order to log in for a new lease. It wasn't always that way, but the iPads and other touchscreens are demanding more bandwidth and it has to come from somewhere.
You ISP may be tweaking things to avoid having to purchase more bandwidth.
The ISP may not want routers included as they login and never logout.
Is there any chance that an individual internal node is doing something funky? For example, a test station PC that needs email, or anything else that might seem to warrant a "custom" configuration? When I started messing with internet servers, I was very careful about the "Don't do this, you'll screw up the whole network" type of issues, and yet I still managed to mess things up royally. For DAYS. Could there be somebody that knows just enough to be dangerous? I'd look for an intern that doesn't know anything, or a senior engineer that knows "everything", except DNS (as that was me, both times).
You issues reminds me of how my misconfigured workstation would appear to work at first, but after all the updates propagated, it would send the DNS requests to neverland.
http://www.techinstyle.tv/20112418220/post-types/guides/10-causes-of-adsl-connectivity-dropping-out/
Hello!
George, I believe you are right. Very right.
Outside of very new sites such as the one our correspondent has setup himself (And I am not being critical) most sites are approaching ten years of age. And sadly most routers only last about 3 to about 5 years before things internal and external (power thing on wall) for those of us before they crash and burn.
For the bigger Cisco guys they have other issues as they age.
Incidentally the Cable delivered Internet service uses a form of DHCP delivered IP addresses, they are not static. But neither are they being delivered via PPPoE which is what ADSL delivers. And here's where it gets peculiar, the service Comcast delivers for tech support is atrocious, that's the clean way of stating it.
As for what clobbered Cox I don't know, but I do know from discussing issues with a friend once who lives in an area served by them, it's, ah, worse then that.
Last night I wanted to get on to the internet and the ADSL was once again down. I picked up the telephone and there was no dial tone. I rebooted the ADSL modem about six times and fooled with telephone until I got a dial tone.
I guess I need to complain to my ISP, but I do see a service truck arond the corner has been pulling fiber optic cable ( the unfinished spice was poking out of a manhole).
When the economy gets bad, a lot of things just get funky as cost savings become the only task that managers can do to justify their paychecks.
I am having to reboot my ADSL at least once a week. I don't have a 24/7 web server, so it isn't a big issue for me. I seem to reach the outside when I want to. But service is not what it used to be and this is with Taiwan's national telephone company.
But after I arrived, it became very obvious as the rapid expansion in electronics had telephone, cable TV, and electrical power wires strung everywhere in chaos. Cable TV companies were tying wire to light poles rather than provide their own poles. And new telephone lines for internet services have wires pulled under doors rather than drill through concrete walls.
Much of the this mess still exists and my telephone line was installed at the height of this rush for voice and a 56K modem service. I have no idea how much they have cleaned up since then. I do know that much of the service has gone over to fiber optic. But typhoon add to the infrastructure problems every year.
If I need a repair it is a rather involved process as I have to visit the telephone company in person or have a friend call in Chinese to get them to address the problem.
I suppose you would be appalled with all this, but for an ex-pat living in Taiwan, everything gets done at a slower pace.
I did have a web page and I used Yahoo as nothing was in Chinese or in Taiwan.