Moving my Dell R630 from my previous rack to a new rack in a new physical location was supposed to be a quick affair but it wouldn’t power on after the relocation. While the PSU and iDRAC port showed the power LED and iDRAC port lighted up after plugging the wires in, it wouldn’t boot despite repeatedly pressing the power button. Using network capture, this is how I managed to regain access to iDRAC to power it back on.
Everything I’d seen suggested that the server should have been able to power on. The PSU lights are on, network activity appears to be active at the iDRAC port but yet despite attempting to connect to it via its previously known subnets and IP addresses, I had no success.
With the other end of the ethernet connected to my local Linux machine, I decided to stop guessing and start listening to what the iDRAC is doing.
I first started by attempting to see what kind of traffic is going on the iDRAC network interface. After all, the moment I connected the server iDRAC port to my Linux machine, the interface started to blink furiously indicating some kind of activity is taking place.
Using tcpdump, I started listening on the network interface to see what’s happening there.
With sudo tcpdump -i <interface> -n -e, I mostly saw ARP packets my Linux machine is sending out. It wasn’t only until I saw a connection from port 68 trying to send out a broadcast packet that it gave me an idea.
Port 68 is a port used mostly by DHCP clients to send DHCP messages to DHCP servers, mostly for the purposes of requesting or releasing DHCP IP addresses.
12:46:26.568351 10:7d:1a:fd:31:90 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 331: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 10:7d:1a:fd:31:90, length 289
Seeing a new device (that is not my own Linux machine) send out a broadcast packet via 68 suggested to me that a DHCP request is being attempted.
My iDRAC is actually trying to get an IP.
I quickly configured dnsmasq to give out IP addresses via DHCP in the subnet that my interface is in with the hope that the iDRAC would accept the DHCP response and assign itself to an IP address in a subnet that I can access.
It worked.
Soon, I saw the confirmation I was looking for - an ARP request showing a new device had received an IP address:
12:51:43.024713 7c:10:c9:1e:5d:7e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.100 tell 192.168.1.41, length 28
ARP requests are messages sent via the ARP protocol (Address Resolution Protocol) that is used to map an IPv4 address to a MAC address. This message is usually sent when an IP address was assigned to a device and the device wants to let everybody know in the network how to reach it.
I quickly reached out to https://192.168.1.100 and luckily with that I was able to regain access to the Dell server and power it on remotely.
While I was able to finally power the server on, the problem of why it is not able to power on using the physical power button still remains. But till I find out how, at least I now am able to power cycle the machine when needed.
This technique works for any network device that falls back to DHCP when its static configuration fails - useful whenever you need to rediscover ’lost’ equipment on your network.