Transcription
EW-ServerGen-Z Technology: EnablingMemory Centric ArchitectureGreg Casey, Server Strategist, Gen-Z & Dell/EMC
SERVER Launched in October of 2016 to create an open, industry standard for a high speed, lowlatency, scalable, memory centric fabric Demos of memory pooling with multiple servers shown over the 2 years Members have released design IP and silicon vendors have started detailed designs forGen-Z devices Released and draft GenZ Documentation is available for public review and comment:www.GenZConsortium.org
4Consortium Members AcesAllion LabsAMDAmphenolArmAvery Design SystemsBroadcomCadenceCiscoCrayDell EMCEverspinETRIFITGenesis rop ITT MadrasJess tAppNode HavenNokiaOak Ridge Natl LabsPLDA GroupQualcommRed HatSamsungSamtec SeagateSenko Advanced CompSimula Research LabSK hynixSmart ModularSony SemiSpin Transfer TechTeledyne LeCroyTEToshiba Memory CorpUniv. New HampshireVMwareWestern DigitalXilinxYadroYonsei University3M
Computer-Memory Balance is Degrading87Processor memory and I/O technologies Normalized Properties ofTypical Server Processors7 are being stretched to their limitsMemory & I/O BandwidthCapacity per Core (GB/s)64 Cores66554433 4000 pins8 DDR Channels264 PCIe LanesAdded DDR Channels gave a bump in 2017,but core count growth offsets 8 DDR channels2110Cores20122013Pins2014DDR Channels201520162017PCIe Lanes201820190201220132014DRAM Bandwidth/Core201520162017PCIe Bandwidth/Core20182019
Layered GADSPI/OCustomPacketCoreArchitecturePhysical Layer AbstractionPhysical XPhysical YPhysical Z Core architecture defines operations, protocol, and physical layer abstraction10s-100s GB/s to TB/s per link bandwidthMultiple physical layers and signaling rates specified per marketLeverage existing IEEE 802.3 electrical standards with Gen-Z-specific optimizationsSupports PCIe electrical, logical, and LTSSM at all signaling rates
Gen-Z Architecture Attributes Feature-scalable packetized transport Scalable and power-proportional link, physical layers, and underlying memory media access. Split memory controller and media controller paradigmBreaks processor-memory interlock—numerous benefits, e.g., Abstracts media to enable memory controller to transparently support multiple media types andmedia generations Accelerate solution innovation and industry agility (eliminates “big bang” events) Transparently integrate performance acceleration techniques to reduce load-to-use latency andincrease aggregate bandwidth, mitigate NVM latencies, etc. Supports processor-centric and memory-centric architectures Processor-centric provides solution evolution path Memory-centric provides enables new solution architectures not possible / practical with processorcentric
Gen-Z Architecture Attributes (continued) Supports unmodified OS and unmodified applications MMU memory mapping to directly access Gen-Z-attached memory Supports logical PCI / PCIe devicesAbstract physical layer interface supporting multiple physical layers and media Easily tailored to market-specific needs. Rapid evolution or replacement without waiting for entire ecosystem to move in lock-stepMarket-driven packaging and fabric topologies Co-packaged and discrete components Single or multi-link point-to-point topologies Switched fabric topologies—component-integrated switch logic or discrete switchcomponents Single enclosure (client, server, storage, network, etc.) to multi-enclosure / rack scaleSupports legacy connectors and mechanical form factorsSupports a new, scalable connector and new modular mechanical form factorsCommon protocol enables democratized communications among all component types
Datagram Packets Datagram packet modelRequesters ensure reliability (if required) Responders simply execute requests and generate responses (if required) Datagrams operate over: Point-to-point and switch topologies Multipath options to provide aggregate bandwidth and resiliency Optional encapsulation and strong-ordering domain for specialized communications For example, transparently augment communications without changing primary / third-partyprotocols For example, transparently tunnel third-party protocols without end-component modifications
Gen-Z Allows Memory InnovationProcessorMedia ModuleMedia ModuleProcessorDRAMMemory BusMediaControllerDRAMGen-Z LogicMemory BusDRAMGen-Z LogicMediaControllerDRAMDRAM288pins / DIMMSynchronous InterfaceDRAMDRAMDRAMSplit Memory ControllerAsynchronous InterfaceProcessor is media agnostic
Gen-Z Connects Disaggregated Components High Performance–––High Bandwidth, Low Latency, ScalableEliminates protocol translation cost / complexity / latencyEliminates software complexity / overhead / latency Reliable–––No stranded resources or single-point-of-failuresTransparently bypass path and component failureEnables highly-resilient data (e.g., RAID / erasure codes) Secure–Provides strong hardware-enforced isolation and security Flexible–––Multiple topologies, component types, etc.Supports multiple use cases using simple to robust designsThorough yet easily extensible architecture Compatible–Use existing physical layers, no OS modifications required Economic Lowers CAPEX / OPEX, unlocks / accelerates innovationGen-Z speaks the language of compute
32GB/s (PCIe 3.0) – 64GB/s (PCIe 4.0)25GB/s (DDR4) – 50GB/s (DDR5)BridgeOption 1 Gen-Z port 4 – 8 DDR5 memory channels1 Gen-Z port 3 – 7 PCIe Gen4 ports200GB/s – 448GB/sLet’s Help the Cores12
All resources are collected into shared pools High-speed, low-latency fabric connects pools Management software: Configures network to connect components Assigns resources Result: Disaggregated serverServer Builder (SW)Server DisaggregationPooled ResourcesSCMCPUDRAMNetworkStorageGPU /FPGARack Scale FabricsDisaggregatedServer True bare-metal bootable serverDisaggregatedServer Ready for installation of any OS and applicationApplication 1Application 2
Flexible: Universal Connector SystemVertical, horizontal, right angle, straddle mountSame connectors for memory, I/O, storage, etc.Cabled solutions: for copper & opticalEliminates “hard choices” Universal connector eliminates industry fragmentation Simplifies supply chain—drives volume and lowers cost Any component, any slot, any time Any mix of static and hot-plug form factors Multi-connector option to provide added scalability 80W incremental power Incremental bandwidth Supports internal and external cable applications Enables modular system design Enables system disaggregation Eliminates expensive board materials Multiple technologies—Gen-Z, PCIe, etc. OCP NIC 3.0 Spec uses the 4C ConnectorGen-Z members contributed mechanical & electrical specification to SNIA—see SFF-TA-1002Gen-Z Scalable Connector specification (final version is publicly available) covers remainingfunctionality.
Scalable Form Factor1105 mm Supports any component type Flash, SCM, DRAM, NIC, GPU, FPGA, DSP, ASIC, etc. Supports multiple interconnect technologies—Gen-Z, PCIe, etc. Single and double-wide—scale in x-y-z directions Increased media, power, performance, and thermal capacity Double-wide can be inserted into pairwise single slots Supports 1C, 2C, and 4C scalable connectors76mm Larger modules can support multiple connectors—scale power & performance Scalable Form Factor Benefits: Simplifies supply chain Lower customer CAPEX / OPEX Consistent customer experience Increases solution and business agility @ lower dev cost Eliminates Potential ESD Damage105 mm154 mm Can safely move modules from failed / old to new enclosure Eliminates SPOF or stranded resources Multiple links per connector, multiple connectors per module Scalable thermal plus improved airflow across components35.4mmDraft specification publicly available—see www.genzconsortium.org* Bandwidth calculated using 32 GT/s Signaling** DRAM module provides 3.5x the highest-capacity DDR5 DIMM1
2018 Deliverables Gen-Z Core Spec 1.0 was released in February 2018 The Gen-Z Phy 1.0 spec release and includes 25G NRZ fabric & local requirements PCIe G4/G5 Gen-Z Scalable Connector 1.1 spec release and adds Internal cables, 48V power delivery, new 4C-HP version SFF 8639, SFF 8639 Compact, and SFF8201 1.0 contain form factor requirements based onindustry specs with exceptions for Gen-Z ZSFF 1.0 contains requirements for form factors unique to Gen-Z PECFF 1.0 contains a Gen-Z form factor that has a CEM compatible outline Working on deliverables for Management/SW, Test/Compatibility & Design Guides
Call to Action Now is the time to engage with Gen-Z. Do it now while there is the opportunity to influence first products. Gen-Z embraces OPEN. We shared the Gen-Z connector with the industry SFF-TA1002 We embraced EDSFF recommended form factors All of our released specs are publicly available Gen-Z allows companies to focus on innovation in their area(s) of expertise and provides the interconnectfor these innovative products Gen-Z is pleased to welcome CXL (Compute Express Link) to the industry and sees opportunity forsolutions that bridge between these two interconnects.See computeexpresslink.org for more information How can OCP and Gen-Z collaborate ? The Gen-Z and OCP organizations must work together to continuedriving common elements into designs that benefit our members and the y/gen-zconsortium/Twitter.com/GenZConsortiumGen-Z Consortium Channel
Gen-Z Architecture Attributes Feature-scalable packetized transport Scalable and power -proportional link, physical layers, and underlying memory media access. Split memory controller and media controller paradigm Breaks processor-memory interlock—numerous benefits, e.g., Abstracts media to enable memory con
