I have been working on a migration project and one thing i have seen a couple of times is people love to over engineer solutions when basic troubleshooting and debugging would have found the problem.
One example of this was an engineer was migrating a java based application shared with the testers the URL saying ready for smoke testing. The URL was slightly different to what the tester was expecting, it ended in 8080 e.g. http://migration.example.com:8080/users.do which the user wasnt able to access due to security rules. Once we had a chat with the tester to find out what they where expecting to see which was http://migration.example.com/users.do the engineer in question first assumed it was a NAT rule the network team have implemented, when he got told this was not the case automatically diving into his toolbox for tricks and started to setup iptables e.g.
iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 8080 -j ACCEPT
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
This will fix the port redirects but would have caused static assets within the site to fail. The answer here was to simply stop take a breath and look at the machine in more details.
- Check the command line history
- What applications are installed on the system
- Check what logs are present on the machine
- Is this system clustered or do you have another machine which you can compare application configs or process table
From just performing these basic checks we found that the issue was someone had forgot to setup Apache httpd to start up on boot.
It is always to easy for technical minded people to automatically assume that there is actually a bigger issue or want to find away to use a new technology trick someone has showed you. We are all guilty of this so its always good for us to go back to basics and not assume that we have to solutionise something.