Fabric Emulator

Problem: Complexity of Reproducing Realistic Behavior of System Test Network

Performance of modern distributed compute-intensive systems is often limited by throughput of network interconnections between cluster nodes. Innovators in this field disaggregate infrastructure components into specialized modules with hardware offload capabilities, coupled with development of new transport protocols.

Such designs must account for the data center topologies that their target customers employ.

Complexity and friction in reproducing realistic datacenter network behavior in a lab—how much equipment do you have to set up and maintain?—is a major factor contributing to excessive cost of innovation in this area.

Solution: System Network Emulation with  Fabric Emulator

The Fabric Emulator product brings the realism of customer networks into experimentation and test workflows, for developers of the next generation of system components ranging from Data Processing Units, SmartNICs, and storage to communication libraries and transport protocols.

The Fabric Emulator, is a specialized network operating system (NOS) capable of:

• Emulating complex data center topologies on a single open switch hardware unit.

• Fairly reproducing real data plane behavior.

• Generating background traffic in the fabric to introduce buffer pressure.

• Impairing workload communications with crafted packet loss scenarios.

To reproduce topologies and conditions found in real data center environments, Fabric Emulator leverages the Open Network Experiments model and API (ONEx). With ONEx, you can share topology data and fabric utilization patterns, and quickly and programmatically iterate through various scenarios as part of an experimentation batch run or continuous-integration pipeline job.

Highlights

• Logically create a multi-stage ECMP network to mimic data center fabric Clos topologies on a single white box switch

• Fairly reproduce traffic forwarding behavior of a data center fabric by leveraging the same switching ASIC as those deployed in production network

• API first, declarative models along with a developer centric UI simplifies configuration of all elements, including oversubscription between stages, QoS knobs, shared/reserved buffer allocations

• Chaos experiments with fabric load enabled by support for injecting up to 1 Tb background traffic into various points of the fabric, or controlled faults like configuring the emulated topology to periodically drop frames on specific links

• 25GE, 50GE, 100GE, and 200GE front panel and fabric ports

• Insights and observability with the ability to mirror ports and visualize dimensional metrics through Prometheus/Grafana integration

• Wire external servers and appliances once, and logically re-attach to different points in the topology during experiments

Product Capabilities

Fabric Emulator provides the following capabilities by leveraging the functionality of a switching ASIC it is running on:

• Emulate two or three tier Clos fabrics, with programmable oversubscription.

• Single or multiple disjoint L2 or L3 switches.

• Multi-stage ECMP with traffic load balancing by using ECMP random spray mode, 5-tuple hashing, and 3-tuple hashing, configurable per tier and direction.

• Layer 2 flooding for L2 discovery protocols.

• QoS at tier or switch level: packet classification and scheduling, ECN marking, control of ingressadmission shared, and reserved buffer sizes.

• Congestion in the fabric by injecting background traffic in different points at the topology.

• Internal traffic sinks, to send packets to a certain IP address without having a destination server with the IP physically connected to the front panel.

• Impairments: drop frames on up to six links during experiments.

• Traffic insights by mirroring any port in the emulated topology to a front panel port.

• Configurable through a gRPC API.

• Hop by hop statistics provided by a Prometheus monitoring agent.

• 25GE, 50GE, 100GE, and 200GE port speeds, uniform throughout the fabric.

• Support for FEC and for auto-negotiation with the option to advertise FEC.