I found it necessary to rewire the SAP server rack, as we hastily pasted it all together prior to the migration from our HP NetServer LC series to our HP DL380 and MSA500 G2 cluster. During this time, I decided it was necessary to install a gigabit switch. I just wanted anything that was rackmountable and decided to go with a D-Link DGS-1016D, as this was available right down the street at Office Depot, has 16 ports, and I only have 7 nodes in that rack, and it was relatively cheap. Just using normal cat5e cables, and not enabling jumbo frames or anything like that made quite a significant difference in the backup window using BackupExec. I have BackupExec running on the development/test system, which uses the ethernet connection to backup the cluster nodes. check it out:
Backup over Fast-Ethernet (100Mbps):
Backup over Gigabit-Ethernet (1000Mbps):
The speed went from 481 MB/min to 3480 MB/min, cutting the backup window down by 85%.
Overall, I am pretty effin pleased.