WO2015164049A1

WO2015164049A1 - Memory mirroring

Info

Publication number: WO2015164049A1
Application number: PCT/US2015/024230
Authority: WO
Inventors: Steven Woo; David Secker; Ravindranath Kollipara
Original assignee: Rambus, Inc.
Priority date: 2014-04-25
Filing date: 2015-04-03
Publication date: 2015-10-29

Abstract

Memory system enabling memory mirroring in single write operations for primary and backup data storage. In one aspect, the system utilizes a memory channel including one or more latency groups, each encompassing memory modules that have the same signal timing to the controller. A primary and a backup copy of a data element can be written to two memory modules in the same latency group of the channel in a single write operation. In another aspect, a memory channel can store duplicate copies of data into multiple locations disposed in different memory modules and have different data propagation times. The relative timings of the chip select, command and address signals among the multiple locations are adjusted according to the data propagation delay, such that a data element can be written into the multiple locations responsive to a data signal transmitted from the memory controller in a single transmission event.

Description

MEMORY MIRRORING

BACKGROUND

[001] Server memory systems play vital roles in enabling the information technology and business goals of a data center. To maximize system uptime for mission-critical applications and to help safeguard servers against certain types of memory errors, various technologies of utilizing Reliability, Availability, and Serviceability (RAS) features have been developed.

[002] Memory mirroring generally refers to a RAS feature involving writing memory content to different places in physical system memory. In general, one set of memory locations is designated as the "primary" memory and the other set is designated as the "backup" memory. This second set of memory locations is not available to the operating system or applications for general access, but is rather reserved to store the backup copy of data in case the primary memory fails.

BRIEF DESCRIPTION OF THE DRAWINGS

[003] Fig. 1 A illustrates a memory system operable to perform intra-socket memory mirroring on different channels coupled to a single memory controller.

[004] Fig. IB illustrates a memory system operable to perform inter-socket memory mirroring on channels coupled to different controllers.

[005] Fig. 1C illustrates a memory system operable to perform intra-socket memory mirroring on a single channel.

[006] Fig. 2A illustrates an exemplary channel topology in a memory system capable of performing a single write operation to store data to two modules simultaneously within the same channel in accordance with an embodiment of the present disclosure.

[007] Fig. 2B is a flow chart depicting an exemplary process of storing data by memory mirroring in accordance with an embodiment of the present disclosure. [008] Fig. 3 illustrates another exemplary channel topology in a memory system capable of performing a single write operation to broadcast data to all modules simultaneously within the channel in accordance with an embodiment of the present disclosure.

[009] Fig. 4 illustrates an exemplary memory system having a memory controller coupled to four memory channels of different topologies in accordance with an embodiment of the present disclosure.

[010] Fig. 5 A illustrates an exemplary memory channel configured to store a data element to two memory modules in a single write operation where the data element arrives at the two modules at different times, e.g., with different signal delay, in accordance with an embodiment of the present disclosure.

[011] Fig. 5B illustrates another exemplary memory channel configured to store a data element to two memory modules in a single write operation where the data element arrives at the two modules at different times in accordance with an embodiment of the present disclosure.

[012] Fig. 6 illustrates sample timing diagrams of various signals at two memory modules used to store a data element in an exemplary single write operation according to an embodiment of the present disclosure.

[013] Fig. 7 illustrates exemplary memory systems capable of storing data into two memory locations in a single write operation in accordance with an embodiment of the present disclosure.

[014] Fig. 8 is a flow chart depicting an exemplary memory mirroring process in which a data element is written to two memory locations in a single write operation in accordance with an embodiment of the present disclosure.

DETAILED DESCRIPTION

[015] Embodiments of the present disclosure provide memory systems and processes enabling duplicate copies of a data element to be stored in different memory locations using a single write operation, thereby increasing memory throughput. In one embodiment, duplicate copies of a data element are written to different memory locations having the same signal latency (or propagation time) regarding command and data signals. As used herein and throughout this disclosure, "the same" refers to an intent for a value to be as close to the same as necessary such that any differences between the two items are immaterial in terms of function. For example, with respect to signal latency, the same signal latency for two DIMMs as discussed herein refers to a signal latency as close to the same as necessary such that data, clock and command signals can be issued by the memory controller without adjusting or accounting for any difference in signal latency between the two DIMMs. An exemplary memory system includes multiple groups of memory modules coupled to a memory channel. The modules in each group are configured to have the same latency for signals sent from the associated memory controller to the memory units. In another embodiment, the multiple memory locations assigned to store duplicate copies of a data element have different signal latency. In a write operation for this embodiment, the relative timings of the chip select, command and address signals among the multiple locations are adjusted according to the latency difference thereof such that the data element is written to the multiple locations using a single write data operation from the memory controller.

[016] Accordingly, a data element can be concurrently written to multiple modules within a memory group in a single write operation, which may advantageously allow memory mirroring without consuming additional channel bandwidth and power as would be needed by conventional memory systems.

[017] In a typical memory system, a memory controller is associated with a few memory channels. In one system type, each channel has a bus coupled to a number of dual in-line memory modules (DIMM) through respective DIMM sockets. In typical commercial CPU-based server memory system designs, each channel is configured to connect with three DIMMs. Each DIMM module further bears several memory chips, see Fig. 1 A.

[018] Based on the relative locations of the memory designated for the primary and backup copy, memory mirroring can be implemented as (1) intra-socket mirroring on the same channel, (2) intra-socket mirroring on different channels, or (3) inter-socket mirroring.

[019] In intra-socket mirroring on the same channel, primary and backup copies are stored in the same memory channel, e.g., in different DIMM modules. In intra-socket mirroring on different channels, the primary and backup copies are stored in different channels connected to the same memory controller. In inter-socket mirroring, the primary and backup copies are stored in memory channels that are coupled to different memory controllers.

[020] Fig. 1 A illustrates a memory system operable to perform intra-socket memory mirroring on different channels coupled to a single memory controller. The memory system includes a memory controller (not explicitly shown) integrated in the CPU 110. The CPU 1 10 is coupled to four exemplary memory channels 121-124, with each channel coupled to three DIMMs. As shown, the three DIMMs on channel 121 are used to store primary data, while the other three DIMMs on a different channel 122 are used to store the backup copy of the data. A tradeoff of such a design is that half of the channel bandwidth and capacity cannot be used for memory operations because two of the channels are dedicated to store backup data. That is, the CPU can access only half of the channel bandwidth during normal operation.

[021] Fig. IB illustrates a memory system operable to perform inter-socket memory mirroring on channels coupled to different controllers. All the memory modules (associated with channels 151-154) that are coupled to a first memory controller 130 (e.g., integrated in a CPU) are used to store a primary copy of data, while all the memory modules (associated with channels 161- 164) that are coupled to a second memory controller 140 are used to store a backup copy of the data. In such a design, an associated CPU 130 or 140 can access full bandwidth as all four channels can be used at one time. However, half of the memory capacity and CPUs are not regularly used in this configuration, because they are dedicated to the backup copy.

[022] Fig. 1C illustrates a memory system operable to perform intra-socket memory mirroring on a single channel. The memory system includes a memory controller (not explicitly shown) integrated in the CPU 170. The CPU is coupled to four memory channels 181-184, with each channel coupled to three DIMMs. Within each channel, the first DIMM is used to store a primary copy of data, and the second DIMM is used to store a backup copy of the data. The drawback of this configuration is that the third module slot ("Empty slot") is usually unused due to lack of a pairing partner for mirroring within the same channel because it is an odd number. Thus, only 2/3 of the memory capacity is used, resulting in wasted memory capacity. [023] Fig. 2A illustrates an exemplary channel topology in a memory system 200 capable of performing a single write operation to store data to two modules within the channel in accordance with an embodiment of the present disclosure. In this example, the memory controller 210 is coupled to one memory channel including the communication channel (or buses) 221 and to 6 DIMMs 201-206. However, it will be appreciated that the present disclosure is not limited by the number of channels coupled to a memory controller or by the number of modules encompassed in a channel.

[024] It will be appreciated that each memory module has an array of memory chips. The memory modules within a channel may have the same type or different types of memory chips selected from Dynamic Random Access Memory (DRAM), Non-volatile memory (NVM) e.g., Flash memory, erasable programmable read-only memory (EPROM), programmable read-only memory (EPROM), and etc. Further, the DIMM modules within a channel can be single data rate (SDR), double data rate (DDR), load-reduction (LR), or registered (R)DIMMs, or a combination thereof, etc.

[025] According to the illustrated channel topology, the 6 DIMMs 201-206 are arranged symmetrically with respect to the memory controller 210. It will be appreciated that the buses 221 have wires directed to each DIMM and include data buses, command buses, and address buses. The address buses include chip select buses directed to individual memory chips on each DIMM. Further the buses 221 include a single primary transmission line 222 (e.g., 8.0" long maximum) routed from the memory controller 210 to a T-split 223 disposed proximate to the 6 DIMMs 201-206. At the T- split 223, the primary transmission line is split into secondary transmission lines (e.g., 224) directed to individual DIMMs, e.g., through DIMM sockets. For instance, two adjacent parallel sockets are spaced apart by 0.4".

[026] As shown, the channel topology 200 has two symmetric halves, the upper half including DIMM 1-3 201-203 and the lower half including DIMM 4-6 204-206. The buses directed to DIMM2 and DIMM5 have the same trace length and thus have the same signal latency. These are a matching pair of memory DIMMs. The buses directed to DIMM 1 , 3, 4, and 6 have the same trace length and thus have the same signal latency. These are a matching group of DIMMs. Effectively, the DIMMs 201-206 in the memory channel are divided into two latency groups, one group including DIMMs 2 and 5 (202 and 205) and the other group including 1 , 3, 4, and 6 (201, 203, 204 and 206). It is appreciated that the DIMMs within the same latency group share the same signal timing, e.g., flight time for data signals, command signals, or address signals, etc.

[027] More specifically, each memory chip on a DIMM has a counterpart chip in every other DIMM in the same latency group. The signal latency to the memory chip is the same as to its counterpart chips.

[028] Because all the DIMMs within each latency group have the same signal latency, two DIMMs within a latency group can be paired-up to store primary data and backup data respectively and can effectively receive the data in a single write operation. For example, DIMM 2 202 can be paired with DIMM 5 205, DIMM 3 203 can be paired with DIMM 4 204, and DIMM 1 201 can be paired with DIMM 6 206 for memory mirroring purposes. DIMMs 1, 2, 3 can be used to store primary copies while DIMMs 4, 5, 6 can be used to store corresponding backup copies.

[029] In some embodiments, all the mirrored DIMMs 201-206 are included in a single packaged device which also includes the T-splits 222 and 223.Because data as well as the corresponding write command are sent from the memory controller and can respectively arrive at a pair of DIMMs (e.g., DIMM 2 and 5) at the same time, the data can be written on to the pair of DIMMs simultaneously, and more particularly, in a single write operation. Fig. 2B is a flow chart depicting an exemplary process 250 of storing data by memory mirroring in accordance with an embodiment of the present disclosure. For example, process 250 can be performed by a memory system as shown in Fig. 2A, Fig.3, Fig. 4, Fig. 5A, Fig. 5B or Fig. 7.

[030] Referring to the system 200 in Fig. 2A by way of example. At 251 , the system 200 reads registers and, at 252, determines if memory mirroring is enabled and which module will be used. For instance, the step 251 may be performed at boot time. If memory mirroring in enabled, at 253, the system 200 automatically configures the address space mappings to place primary and backup copies of data where they need to be with respect to the mirror pairs. At 254, the

configuration is stored for use during subsequent write operations. At 255, write operations are performed responsive to write requests.

[031] As will be described in greater detail with reference to Fig. 8, during a write operation, the memory controller 210 transmits each of the command, address and data signals to the pair of DIMMs once, e.g., pair of DIMM 5 and DIMM 2, for instance. Chip selects are enabled for the pair of DIMMs at the same time, and more specifically for the selected chips on the two DIMMs. Thus, responsive to a single write command, a primary copy and a backup copy of the data are saved respectively in different memory locations simultaneously and with a single write operation.

[032] Advantageously, there is no additional write latency introduced for purposes of generating the backup copy. Compared with the conventional approach wherein two separate write operations are needed to store the primary and backup copies of data, the memory topology 200 according to the present disclosure advantageously enables memory mirroring with reduced power consumption and improved write performance of the memory system.

[033] Moreover, because the memory channel includes an even number of DIMMs, each DIMM can be paired up with another DIMM within the same channel. A memory channel can be efficiently utilized without causing a DIMM slot to be unused.

[034] Since the same memory write operation feeds both DIMM units of a matching latency group, in one embodiment, it may be beneficial to increase the nominal drive strength of the signals over the shared bus lines to accommodate both DIMMs.

[035] It will be appreciated that the present disclosure is not limited by techniques and configurations used to achieve matching latency for the modules within a latency group. In some other embodiments, suitable delay elements well known in the art can be inserted in a

communication channel and configured to obtain the same latency among the multiple modules within a group.

[036] Fig. 3 illustrates another exemplary channel topology in a memory system 300 capable of performing a single write operation to broadcast data to all modules within the channel in accordance with an embodiment of the present disclosure. The memory topology in Fig. 3 is similar with the memory topology 200 illustrated in Fig. 2. However, the buses are configured such that all the DIMMs (1-6) in the memory channel are in the same latency group, or share the same timing. As shown, the secondary transmission lines 332 and 335 directed to DIMMs 2 and 5 from the T-split 323 are routed with extra lengths to match the trace length of the buses directed to DIMMs 1, 3, 4, and 6. In this configuration, a data can be stored at six locations in a single write operation responsive to one write command as all bus lengths and times are matched. That is, the data can be concurrently broadcast to all the DIMMs within the channel. For instance, such an operation can be used to broadcast code into buffers on all the modules and the code can be executed in the buffers.

[037] As discussed above, in a write operation in system 300 according to the present disclosure, as a single signal (e.g., a data signal) needs to be transmitted to and detected by multiple memory locations, the drive strength (e.g., voltage) for the signal may be adjusted higher at the memory controller accordingly.

[038] It will be appreciated the memory controller can be implemented as a separate chip or integrated in a host device, such as a CPU, a coprocessor, a GPU, a Field-Programmable Gate Array (FPGA), Application-Specific Integrated Circuit (ASIC), etc.

[039] Fig. 4 illustrates an exemplary memory system 400 having a memory controller 410 coupled to four memory channels 420-450 of different topologies in accordance with an embodiment of the present disclosure. The memory controller is a FPGA device in this example. Each memory channel has an even number of DIMMs that are grouped into one or more latency groups, as described in greater detail above. Within each latency group, the DIMMs can be paired up to store primary and backup copies of data. In some other embodiments, the FPGA could be replaced with an ASIC.

[040] The memory system combines two types of channel topologies. The channels 440 and 450 are each configured to have a single latency group as described with reference to Fig. 3. The other two channels 420 and 430 are each configured to have two latency groups as described with reference to Fig. 2. In some other embodiments, all the channels of a memory system have the same channel topology.

[041] Embodiments of the present disclosure also provide an approach of performing a single write operation to store duplicate copies of a data element in multiple memory locations that have different propagation times for data signals transmitted from the memory controller. The approach presented herein involves controlling the timing offsets of respective clock and control signals between the two memory locations based on corresponding propagation delay. Thus, the data element can be written to these locations substantially in parallel in accordance with a single write operation rather than with, for example, two sequential write operations to store a primary and a backup copy of a data element with each write operation using a different propagation delay between the memory locations.

[042] Fig. 5 A illustrates an exemplary memory channel design 510 configured to store a data element to two memory modules in a single write operation where the data element arrives at the two modules at different times (due to different propagation delay) in accordance with an embodiment of the present disclosure. The two memory locations are located on module A 511 and module B 512 respectively, which may be DIMM modules. The modules 511 and 512 are assigned to store primary copies and backup copies respectively. A data signal sent from the memory controller (not explicitly shown) arrives at the modules 511 and 512 in different times due to the propagation delay, on the channel 513.

[043] As shown, each module 511 or 512 is coupled to separate chip select (CS), clock (CK), on-die termination (ODT), and clock enable (CKE) signal lines. The command and address (C/A) lines are shared between the modules A and B, which allows the same data to be written to the same addresses in the two modules 51 1 and 512. This configuration can reduce the number of pins on the host device (e.g., a CPU) controlling the memory channel 510 and the number of wires or lines on the board.

[044] During initialization, the memory controller (not shown) determines the best timing relationship between CS, CK and the C/A bits so that data can be reliably stored to each module. When memory mirroring is not being enabled, the memory controller determines which modules the data is being written to, and drives the C/A bits, as well as the CS, CK signals at appropriate timing for the proper memory module.

[045] When memory mirroring is enabled, the memory controller activates the CS and CK signals for both modules at the appropriate times (e.g., with a predetermined and slight time offset) in a single write operation, such that the data can be written to the two modules at substantially the same time. More specifically, during a single write operation, a write command is issued from the memory controller once, and the CS, CK, CKE and ODT signals are driven at appropriate times relative to the command and address (C/A) signals to each module 511 and 512 to capture and store data as the data passes along the channel 513 from the memory controller. [046] Fig. 5B illustrates another exemplary memory channel design 520 configured to store a data element to two memory modules in a single write operation where the data element arrives at the two modules at different times in accordance with an embodiment of the present disclosure. The channel 520 has a similar configuration with the channel 510 in Fig. 5 A except that the two modules 521 and 522 are coupled to separate command and address lines 524 and 525 (C/A_A and C/A_B). This allows the same data to be written to different or the same addresses in the two modules 521 and 522 using the same write operation. Although using separate command and address lines for the two modules requires additional pins on the associated host device and additional wires on the circuit board, it can provide better timing characteristics and in some cases higher operating speed than the configuration shown in Fig. 5A, and generally provides increased memory addressing flexibility.

[047] When memory mirroring is enabled, the memory controller activates the C/A, CS and CK signals for both modules at the appropriates times to make sure that both can store the write data as it passes by each module on the way down the channel within the single write operation.

[048] Fig. 6 illustrates sample timing diagrams of various signals at two memory modules used to store primary and backup data in an exemplary single write operation according to an embodiment of the present disclosure. The write operation in this example is performed at an exemplary memory channel with two modules disposed in series along the buses, with each module coupled to separate chip select, clock, command and address lines, as shown in Fig. 5B.

[049] The clock diagrams 611 and 621 (CK A and CK B) represent the clock signals at module A and module B respectively. The command diagrams 612 and 622 (CMD A and CMD B) represent write command signals at the two modules respectively. The address diagrams 613 and 623 (ADDRESS A and ADDRESS B) represent address signals at the two modules respectively. The DQS diagrams 614 and 624 represent the data strobe signals at the two modules respectively. The DQ diagrams 615 and 625 represent the data signals at the two modules respectively.

[050] This write operation is configured to have a burst length of 8 (BL=8), write latency of 5 (WL=5), additive latency of 0 (AL=0), column address strobe write latency of 5 (CWL=5). During the write operation, the memory controller transmits write data signals (DQ) and a strobe (DQS) asserting data validity on the data bus to both modules in the memory channel. Due to propagation delay (Δί), the strobe (DQS) first appears at module A at time 601 and then at module B at time 602. Also, the data signal (DQ) appears at module A at 603 and module B at 604 with the same delay (Δί). It will be appreciated that the memory controller is configured to provide a sufficient drive strength for the data signal to be suitably detected by both memory locations.

[051] To compensate the propagation delay (Δί) of the data signals along the data buses, the memory controller activates the clock and control signals for module B (CK B 623, CMD B 624, ADDRESS B 625, and chip select signal (not shown)) relative to when the data and strobe signals reach module B. More specifically, as shown, a time offset Δί is added between CMD_A and CMD B (see 605 and 606), between CK A and CK B (see 607 and 608), and between

ADDRESS A and ADDPvESS B (see 609 and 610).

[052] In this write operation, the memory controller issues a write command (through separate command lines) and transmits the write data and strobe to the two modules only once for the data element. The eight data elements (Dj_n n to Dj_n n+7 in diagram 615 and 625) are written to the two modules in parallel with a slight time offset (Δί).

[053] Fig. 7 illustrates exemplary memory systems 710 and 720 capable of storing a data element into two memory locations in a single write operation in accordance with an embodiment of the present disclosure. The exemplary memory system 710 has a memory controller (not explicitly shown) integrated in the CPU 711 and four channels, each channel including two modules (e.g., 713 and 714) of the same memory type disposed in series along the buses (e.g., 712). Each module in system 710 is a DDR3 LR-DIMM. The memory system 720 has a memory controller (not explicitly shown) integrated in the CPU 711 and four channels, each channel including two modules of different memory types disposed in series along the buses (e.g., 722). In this example, the module 723 is a DDR3 LR-DIMM while the module 724 is a DIMM with NVM (e.g., Flash memory).

[054] Both systems 710 and 720 are configured to write a data element to different modules within a channel in a single write operation, whether the modules are of the same type or of different types. In such a single write operation for memory mirroring, the memory controller sends only one write command to both modules (through the same or separate command lines) at substantially the same time. The memory controllers sends data signals only once to both modules through the data buses, which advantageously eliminates the write latency and additional power consumption caused by performing two separate write operations for storing a data element, one for each module, as required in conventional memory mirroring techniques.

[055] Fig. 8 is a flow chart depicting an exemplary memory mirroring process 800 in which a data element is written to two memory locations in a single write operation in accordance with an embodiment of the present disclosure. The process can be performed on a memory system including a memory controller and a memory channel having at least two memory modules (e.g., DIMMs), for example as shown in Figs 2-4, Fig. 5A, Fig. 5B or Fig. 7. Referring to the Memory System 200 by way of example, module "A" or DIMM1 201 is used to store primary data and module "B" 202 is used to store backup data. At 801 , the memory controller receives a write request to store a data element in the memory. If it is determined that memory mirroring is disabled at 802 (e.g., by reading the configuration stored at step 254 in Fig. 2B), the memory controller 210, at 803, sends a write command and the data element to the designed address in module A 201. As such, only one copy of the data is stored in the memory.

[056] If memory mirroring is enabled, the memory controller sends a write command only once to both memory locations on the two modules 201 and 202 at 804. At 805, the

corresponding chip selects are activated for both modules 201 and 202. At 806, the memory controller 210 sends the data only once to both memory locations. In response to this single data transmission event from the controller, the data element is captured and written to both locations in the two modules 201 and 202, thereby creating a primary copy and a backup copy of the data using a single write operation.

[057] Reference has been made in detail to the preferred embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. While the disclosure is described in conjunction with the preferred embodiments, it will be understood that they are not intended to limit the disclosure to these embodiments. On the contrary, the disclosure is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the disclosure as defined by the appended claims. Furthermore, in the detailed description of embodiments of the present disclosure, numerous specific details have been set forth in order to provide a thorough understanding of the present disclosure. However, it will be recognized by one of ordinary skill in the art that the present disclosure may be practiced without these specific details.

In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the present disclosure. The drawings showing embodiments of the disclosure are semi-diagrammatic and not to scale and, particularly, some of the dimensions are for the clarity of presentation and are shown exaggerated in the drawing Figures. Similarly, although the views in the drawings for the ease of description generally show similar orientations, this depiction in the Figures is arbitrary for the most part. Generally, the disclosure can be operated in any orientation.

[058] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present disclosure, discussions utilizing terms such as "processing" or "accessing" or "executing" or "storing" or "rendering" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories and other computer readable media into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or client devices. When a component appears in several embodiments, the use of the same reference numeral signifies that the component is the same component as illustrated in the original embodiment.

[059] Although certain preferred embodiments and methods have been disclosed herein, it will be apparent from the foregoing disclosure to those skilled in the art that variations and modifications of such embodiments and methods may be made without departing from the spirit and scope of the disclosure. It is intended that the disclosure shall be limited only to the extent required by the appended claims and the rules and principles of applicable law.

Claims

WHAT IS CLAIMED IS:

1. A system comprising:

a memory controller configured to transmit memory data to dual-inline memory modules (DIMMs) using two or more communication channels;

a first communication channel coupled to said memory controller comprising a first bus; a first set of DIMMs coupled to said first bus;

a second communication channel coupled to said memory controller, wherein the second communication channel comprises second and third buses;

a second set of DIMMs coupled to said second bus; and

a third set of DIMMs coupled to said third bus,

wherein said memory controller is configured to enable storage of a primary copy of data within said second set of DIMMs coupled to said second bus and a secondary copy of data within said third set of DIMMs coupled to said third bus by transmitting a single copy of the data over the second communication channel.

2. The system of Claim 1, wherein the first communication channel coupled to said memory controller comprises the first bus and a fourth bus, and further comprising a fourth set of DIMMs coupled to said fourth bus.

3. The system of Claim 1 , wherein the second communication channel comprises a transmission split comprising the second and third buses, wherein the transmission split comprises splitting a trace line coupled to the memory controller.

4. The system of Claim 3, wherein the transmission split is disposed within a device package housing the second and third buses and the second and third sets of DIMMs.

5. The system of Claim 1, wherein said primary copy of data is stored in a first DIMM encompassed in said second set of DIMMs, wherein said secondary copy of data is stored in a second DIMM encompassed in said third set of DIMMs, wherein said first DIMM and said second DIMM are in a first latency group, wherein said second communication channel is configured to deliver a signal representing said single copy of data sent by said memory controller to DIMMs in said first latency group with substantially a same propagation time.

6. The system of Claim 4, wherein said first latency group comprises an even number of DIMMs.

7. The system of Claim 5, wherein said second bus and said third bus have an approximately same trace length to every DIMM in said first latency group.

8. A memory system comprising:

a memory controller;

a communication channel coupled to said memory controller and comprising buses; and a plurality of memory devices coupled to said communication channel and divided into one or more latency groups, wherein each latency group comprises a respective number of memory devices, wherein said communication channel is configured to deliver a data signal sent from said memory controller to said respective number of memory devices with substantially a same propagation time.

9. The memory system of Claim 8, wherein each of said plurality of memory devices comprises a dual in-line memory module (DIMM).

10. The memory system of Claim 8, wherein each latency group comprises a respective even number of memory devices.

11. The memory system of Claim 8, wherein said one or more latency groups comprise a first latency group, wherein said first latency group comprises a first memory device coupled to a first bus and a second memory device coupled to a second bus, wherein said data signal represents a data element, and wherein further said first memory device is configured to store a primary copy of said data element, and said second memory device is configured to store a backup copy of said data element.

12. The memory system of Claim 1 1, wherein said data element is written into both said first memory device and said second memory device in a single write operation performed by said memory controller.

13. The memory system of Claim 8, wherein said memory controller comprises a field programmable gate array (FPGA) device.

14. The memory system of Claim 1 1, wherein said first bus and said second bus have a substantially same trace length.

15. The memory system of Claim 1 1 , wherein said one or more latency groups further comprise a second latency group, and wherein further each memory device in said second latency group is coupled to a bus that has a different trace length than said first bus.

16. A method of storing data to memory devices, said method comprising:

sending, with a memory controller, a single write operation to a first memory device and a second memory device via a communication channel, wherein said communication channel comprises respective chip selects for said first memory device and said second memory device; concurrently activating chip selects for said first memory device and said second memory device responsive to said single write operation; and

concurrently sending, with said memory controller, a data element to said first memory device and said second memory device via said communication channel during said single write operation such that a primary copy of said data element is stored in said first memory device and a backup copy of said data element is stored in said second memory device.

17. The method of Claim 16, wherein said communication channel has substantially a same trace length to said first memory device and said second memory device.

18. The method of Claim 16, wherein said communication channel comprises shared command/address buses, and wherein said write operation is sent to said first memory device and said second memory device via shared command/address buses of said communication channel.

19. The method of Claim 16 further comprising storing said data element to other memory 95 devices coupled to said communication channel responsive to said single write operation, wherein said communication channel has a same trace length to said first memory device and each of said other memory device.

20. The method of Claim 16, wherein said first memory device and said second memory device 100 comprise dual in-line memory modules (DIMM) disposed symmetrically with reference to said memory controller.

21. A system comprising:

a memory controller; and

105 a memory channel coupled to said memory controller, said memory channel comprising:

a communication channel coupled to said memory controller;

a first dual-inline memory module (DIMM) coupled to said communication channel and having a first signal latency for signals sent from the memory controller to the first DIMM; and

1 10 a second DIMM coupled to said communication channel and having a second signal latency for signals sent from the memory controller to the second DIMM,

wherein said memory controller is configured to enable storage of a primary copy of data within said first DIMM and a secondary copy of the data within said second DIMM in a single write operation by transmitting:

1 15 a copy of the data over the memory channel to the first DIMM and the second DIMM;

a first clock signal to the first DIMM; and

a second clock signal to the second DIMM, wherein the second clock signal is delayed with respect to the first clock signal based on the first and second signal latencies.

120 22. The system of Claim 21 , wherein, in said single write operation, said memory controller is further configured to: transmit first write command and address signals to said first DIMM at a first time; and transmit second write command and address signals to said second DIMM at a second time, wherein said second time is delayed with respect to said first time based on the first and second signal latencies.

125

23. The system of Claim 21 , wherein, in said single write operation, said memory controller is further configured to transmit same command and address signals concurrently to said first DIMM and said second DIMM.

130 24. The system of Claim 21 , wherein said first DIMM and said second DIMM are coupled to different command and address buses.

25. The system of Claim 21 , wherein, in said single write operation, said first DIMM is configured to start storing said data at a first time, and said second DIMM is configured to start

135 storing said data at a second time, wherein said second time is delayed with respect to said first time based on the first and second signal latencies.

26. The system of Claim 21 , wherein, in said single write operation, said memory controller is further configured to transmit a strobe signal concurrently to said first DIMM and said second

140 DIMM .

27. The system of Claim 23, wherein, in said single write operation, said memory controller is further configured to activate chip selects of said first DIMM and said second DIMM respectively and sequentially based on a difference between the first and second signal latencies.

145

28. A system comprising:

a memory controller; and

a memory channel comprising:

a communication channel coupled to said memory controller; and

150 a plurality of memory modules comprising a first module and a second module, wherein said first module and said second module are coupled to said communication channel,

wherein said memory controller is configured to: determine a propagation delay between said first module and said second module 155 with respect to signals transmitted from said memory controller;

send a data signal representing a data element in a single transmission to said first module and said second module via data buses of said communication channel; and

send clock signals and control signals respectively to said first module and said second module, wherein said clock signals and said control signals are respectively timed 160 based on said propagation delay, and

wherein said first module and said second module each are configured to store said data element responsive to said data signal and said control signals.

29. The memory system of Claim 28, wherein said first module and said second module are

165 coupled to same command and address buses of said communication channel, and wherein further said memory controller is further configured to send a write command signal in a single

transmission to said first module and said second module.

30. The memory system of Claim 28, wherein said control signals comprise chip select signals 170 and on-die termination signals.

31. The memory system of Claim 28, wherein said first module is configured to start storing said data element at a first time and said second module is configured to start storing said data element at a second time respectively, and wherein said second time differs from said first time by said

175 propagation delay.

32. The memory system of Claim 28, wherein said memory controller is integrated in a central processing unit (CPU).

180 33. The system of Claim 28, wherein said data buses have different trace lengths between said memory controller and said first module and between said memory controller and said second module.

34. The system of Claim 28, wherein said first module comprises a dual in-line memory module 185 (DIMM) and is assigned to store a primary copy of said data element, and wherein further said second module comprises a non-volatile memory module and is assigned to store a backup copy of said data element.

35. The system of Claim 28, wherein said first module and said second module are coupled to 190 separate command and address buses of said communication channel, and wherein further said memory controller is further configured to send write command signals in respective transmissions to said first module and said second module, wherein said respective transmissions are timed based on said propagation delay.

195 36. A method of storing data by memory mirroring, said method comprising:

receiving at said memory controller a write request for storing a data element; transmitting by said memory controller a data signal representing said data element in a single transmission to a primary memory device and a backup memory device; and

transmitting by said memory controller a first control signal to said primary memory device 200 at a first time, and transmitting a second control signal to said backup memory device at a second time that differs from said first time by a propagation delay between said prime and said backup memory devices, wherein said first and said second control signals are operable to enable said primary memory device and said backup memory device to store said data element respectively, wherein said data signal and said first and said second control signals enable storage of a 205 copy of said data element to said primary memory device and a copy of said data element to said backup memory device.

37. The method of Claim 36, wherein said first and said second control signals are chip select signals, and further comprising transmitting a write command signal by said memory controller to

210 said primary memory device and said backup memory device in separate transmissions that are timed based on said propagation delay.

38. The method of Claim 36 further comprising transmitting by said memory controller a first clock signal to said primary memory device and a second clock signal to said backup memory 215 device, wherein said first clock signal and said second clock signal are timed based on said propagation delay.

39. The method of Claim 36 further comprising transmitting by said memory controller an address signal to said primary memory device and said backup memory in a single transmission via

220 an address bus, wherein said address bus is shared by said primary memory device and said backup memory device.

40. The method of Claim 36, wherein said data element is sent to different addresses in said primary memory device and said backup memory device for said storage.