So the new HBA card arrived last week, but I was way too busy to deal with it then. Plus, I like to plan things out when doing installations like this.
Anyways, I woke up earlier than expected this morning, and since my wife was still asleep, I decided that it would be a great time to get the new HBA installed and configured. I figured that should only take about 30 minutes. Little did I know, the universe had other plans.
To take the Server down to swap out the RAID cards would also mean killing the network since pfSense was one of the Virtual Machines under Proxmox, so I decided that I first needed to reconfigure the ASUS RT-AC5300 and elevate it back to the role of being the primary DHCP and DNS Server for the network. But before I could do that, I would need to go into the AT&T Router and reconfigure it to no longer expect to see pfSense on the other side of the WAN connection and instead set it to expect to see the ASUS once again. Done.
Next, I went into the ASUS and re-enabled it’s DHCP Server and added the needed settings for Primary and Secondary DNS. Check!
Then I went to the Server and unplugged both the LAN and WAN feeds from the pfSense NIC, and swung the WAN over to the ASUS. Now the ASUS was getting a direct feed from the AT&T Router, and importantly, the pfSense router was no longer attached to the network. Equally important to me, this also meant that I didn’t need to reconfigure pfSense, which was a huge motivation for the whole process.
It should have worked this way, but did not. I spent the next hour or two troubleshooting that issue. Then it hit me. I was only changing the network because it would make it easier to swap out the RAID Controllers, but if I had just done the HBA card swap, I’d be done by now.
So I stopped troubleshooting the network issues and swapped out the RAID controller cards. But Murphy being Murphy, I was literally almost done with the card swap and Jana poked her head in and asked if there was a Network issue. “Yeah”, I said, “I took the server down to work on that card swap thing.”
Even after swapping out the Dell PERC (PowerEdge Expansion RAID Controller) 6/I card for the HBA (Host Bus Adapter) card, it still didn’t work correctly.
I had to go into the Proxmox config for the TrueNAS VM (Virtual Machine) and add the HBA Card via the “Add PCI Interface” in order to “pass it thru” from Proxmox (the “metal install”) to the VM.
It still didn’t work, but it looked like I had done everything correctly. I was seeing the “heartbeat” LED beating once every second on the HBA. This proved that the motherboard, the HBA, the power supplies, the memory, and the processor were good and working. This also confirms in a round-about fashion that the “IT Mode” Firmware itself was good and working.
Because of the cramped nature of my work area, the server was facing away from me, so I had failed to notice that the drives were not powering-up at POST, but I did notice that they were not be enumerated during the boot sequence when the LSI card was initializing. So that meant it had to be a problem with the cabling. I was 100% certain about the cabling where it connected to the backplane. There was only a single way to connect at the backplane (a 90 elbow, which I would describe like a man with hands straight-up then bent to the face his right-side), and it matched the prior cable set that I replaced.
But the connections at the card could be the problem. Sadly, I was unable to source “straight” connectors on the card side and had to source cables that had a 90 bend (more resembling a man bending at the waist straight forward), and those fit perfectly. Most connectors nowadays only allow a single way to connect unless it was designed to be agnostic (like USB-C connectors).
But it turns out that you can in point of fact plug the 8087 connectors in backward. Which is what I did. So when I turned the cables around, suddenly I could hear the drives initializing. That is when I knew that I had found and fixed that problem. Presto Change-O, TrueNas could now see the drives!
30 Minutes had turned into a four-hour struggle. Well, hopefully, it at least made for an entertaining read…