$Amazon(AMZN.US) $Alphabet(GOOGL.US) Google can reach nearly 10k accelerators with direct connect. Why didn’t AWS stay on that path?
The short answer: AWS’s 3D torus is a stitched-together design and lacks Google’s prerequisites for scaling. A 3D torus—chips arranged as a cube with only nearest-neighbor links—does not require switches.Trn2-Ultra daisy-chains four 16-accelerator servers into a 4×4×4 topology, with all links on copper. Google’s approach looks similar, but the wrap-around links use a different medium: Google uses fiber into OCS, whereas AWS stays on copper.
The immediate consequence is a hard cap at 64 accelerators. With OCS plus its software stack, Google can tile cubes flexibly to approach nearly 10k cards. Trn2 has no switching elements, the backplane copper is fixed, and AECs (with retimers for longer reach) have length limits, so you can’t span more racks.
Overall, Trn2’s all-copper, switchless design limits scale. With too many gaps to fill and pressure to deploy, AWS pivoted to a switched fabric.Trn3 adds switch silicon, taking a domain to 144 accelerators and preventing hop count from growing with scale.