top-of-rack switching
What is top-of-rack switching?
Top-of-rack (ToR) switching is a data center architecture design in which computing equipment like servers, appliances and other switches located within the same or adjacent rack connect to an in-rack network switch.
ToR architecture generally places a network fiber switch in every rack to connect easily with every other device in the rack. The in-rack network switch, in turn, connects to aggregation switches, most often via fiber optic cable.
Network architects typically place ToR switches at the top of the rack, but they can be anywhere in the rack. However, to better enable access to the horizontal fiber optic cables that connect in-rack switches to aggregation switches, network architects usually position ToR switches near the top of the rack.
ToR deployment options
In high-density data center deployments, switches are in every rack. In most cases, switches also connect to a computing device such as a bare-metal server used by only one tenant or blade server chassis. Each in-rack switch, in turn, connects to the aggregation switch block with fiber optic cables. Connections within the rack can be any combination of copper, fiber or direct access cabling.
Among other capabilities, ToR switches handle operations, including Layer 2 and Layer 3 frame and packet forwarding, data center bridging and the transport of Fibre Channel over Ethernet for the racks of servers connected to them.
Benefits and challenges of ToR switching
ToR offers many benefits, along with some drawbacks. With ToR, cabling complexity reduces because all the servers connect to the switch on the same rack and fewer connections need to run outside the rack to reach the aggregation switch.
ToR also reduces cable length and the amount of cabling needed. Network architects can easily upgrade a ToR design to 10 Gigabit Ethernet (GbE), 40 GbE or 100 GbE without incurring many costs or necessitating a change in cabling.
Other benefits of ToR include the ability to combine one network switch for two or three racks, if they are low-density deployed racks. In other instances, ToR architecture boosts modular deployment. A preassembled rack with all the necessary cabling and switches can be quickly connected and deployed on site.
In terms of challenges, ToR could have higher capital and maintenance costs. The distributed architecture of a ToR design requires the need for more physical switches. ToR switches also might end up underused, which can result in unnecessary power usage and increased cooling with no direct benefit to performance. Finally, if the ToR architecture requires the deployment of a single in-rack switch per rack, an entire rack will go offline if that switch fails.
ToR switching vs. end of row switching
ToR and end-of-row (EoR) switching are both popular data center architecture options that require a large number of servers to connect. EoR switching differs from ToR switching because EoR design calls for each server in a rack to directly connect to a common aggregation switch without connecting to individual switches within the rack.
EoR designs almost always require a much larger horizontal cable plant. A data center often has multiple EoR switches, one per row or sometimes one to connect a certain number of racks. If an EoR cable plant is already in use in a data center, it's often easier and more cost-effective for network architects to reuse it when they upgrade switch hardware, as opposed to ripping and replacing the horizontal cable plant to convert it to a ToR design.
The recabling cost, in addition to increased capital expense due to the additional hardware requirements of a ToR architecture, can sway data center operators to stick with an EoR architecture. In a greenfield data center deployment where an existing EoR cable plant doesn't exist, the benefits of a ToR architecture supersede the nominal increase in hardware capital expenditures.
Editor's note: This article was reformatted to improve the reader experience.
