DDR3 - The quest to 128 MBytes
Ale
Posts: 2,363
Hei all,
I want DDR3, want it want it want it, did I say that I want it ? , no ?... well... i do.
Here I'll post my progress.
Testbench:
BeMicroCV. Cyclone V, DDR3 chip: Micron MT41J64M16JT-15E here: http://www.micron.com/parts/dram/ddr3-sdram/mt41j64m16jt-15e?source=ps
Code: my own Cog, Ale's Cog
, just because I know it and can modify it to suit this test, it runs normal cog's assembler, it has one difference, a hw-uart .
I'll post the code to my github account as soon as I have found my github password ;-) again....
The idea is to get it to work, I have compiled the core before... but didn't have a suitable environment to test it, now that the environment works well, I can go on with the DDR3 config itself.
Note on speed and bandwidth:
The board's docs don't really say much more than it "works", let's take they word for it at 300 MHz. The chip is a CL 9 chip, meaning from command to answer we have 9 clocks, plus all the overhead and so on, like 20 clocks per transfer... I'm sure it can be integrated as a HUB peripherial, the hub waits anyways up to 16 80 MHz clocks, maybe it fits somehow.
Controller setup:
- Hard external memory controller, select the chip from the list on the right and 50 MHz as base clock, and 300 MHz Memory clock, avalon at full speed.
- I'll use a 32 bit port to read and write.
- Compiling the thing takes like forever, welcome to the slow lane
Edit:
I got it to compile, it costs like 800 extra ALMs... but it did something strange for the dqs pins during assignment...
I want DDR3, want it want it want it, did I say that I want it ? , no ?... well... i do.
Here I'll post my progress.
Testbench:
BeMicroCV. Cyclone V, DDR3 chip: Micron MT41J64M16JT-15E here: http://www.micron.com/parts/dram/ddr3-sdram/mt41j64m16jt-15e?source=ps
Code: my own Cog, Ale's Cog
, just because I know it and can modify it to suit this test, it runs normal cog's assembler, it has one difference, a hw-uart .I'll post the code to my github account as soon as I have found my github password ;-) again....
The idea is to get it to work, I have compiled the core before... but didn't have a suitable environment to test it, now that the environment works well, I can go on with the DDR3 config itself.
Note on speed and bandwidth:
The board's docs don't really say much more than it "works", let's take they word for it at 300 MHz. The chip is a CL 9 chip, meaning from command to answer we have 9 clocks, plus all the overhead and so on, like 20 clocks per transfer... I'm sure it can be integrated as a HUB peripherial, the hub waits anyways up to 16 80 MHz clocks, maybe it fits somehow.
Controller setup:
- Hard external memory controller, select the chip from the list on the right and 50 MHz as base clock, and 300 MHz Memory clock, avalon at full speed.
- I'll use a 32 bit port to read and write.
- Compiling the thing takes like forever, welcome to the slow lane
Edit:
I got it to compile, it costs like 800 extra ALMs... but it did something strange for the dqs pins during assignment...
Comments
/* * DDR3 Test top file * * * */ `include "../../01_Verilog/acog_defs.v" module ddr3_cog( input wire clk_50, output wire [7:0] leds, // DDR3 inout wire [15:0] mem_dq, output wire [12:0] mem_a, output wire [2:0] mem_ba, output wire mem_cas_n, output wire mem_ck, output wire mem_ck_n, output wire mem_cke, output wire mem_cs_n, output wire [1:0] mem_dm, inout wire [1:0] mem_dqs, inout wire [1:0] mem_dqs_n, inout wire mem_odt, output wire mem_ras_n, output wire mem_reset_n, output wire mem_we_n, input wire oct_rzqin, inout wire [31:0] port_a, output wire tx_o ); //`define TWO_COGS 1 //`define HW_DEBUG 1 wire [31:0] CNT; wire [31:0] port_a_in, mux_port_a_out; wire [31:0] port_a_dir; wire [31:0] port_a_o_0, port_a_dir_o_0; wire [31:0] port_a_o_1, port_a_dir_o_1; wire [31:0] hub_data_to_cog_0, hub_data_to_hub_0; wire [`HUB_MEM_WIDTH-1:0] hub_mem_addr_0; wire [1:0] hub_mem_size_0; wire hub_data_rdy_0, hub_mem_ack_0; wire [4:0] hub_op_0; wire cog_reset_0; wire internal_clk, main_clk; wire [95:0] debug_data_cog_0; wire debug_data_valid_0; wire debug_data_o; wire uart_tx; wire pll_locked; reg [3:0] reset_cnt; `ifdef HW_DEBUG assign tx_o = debug_data_o; `else assign tx_o = uart_tx; `endif assign port_a_in = port_a; assign mux_port_a_out = port_a_o_0;// | port_a_o_1; assign port_a_dir = port_a_dir_o_0;// | port_a_dir_o_1; assign port_a[ 0] = port_a_dir[ 0] ? mux_port_a_out[ 0]:1'bz; assign port_a[ 1] = port_a_dir[ 1] ? mux_port_a_out[ 1]:1'bz; assign port_a[ 2] = port_a_dir[ 2] ? mux_port_a_out[ 2]:1'bz; assign port_a[ 3] = port_a_dir[ 3] ? mux_port_a_out[ 3]:1'bz; assign port_a[ 4] = port_a_dir[ 4] ? mux_port_a_out[ 4]:1'bz; assign port_a[ 5] = port_a_dir[ 5] ? mux_port_a_out[ 5]:1'bz; assign port_a[ 6] = port_a_dir[ 6] ? mux_port_a_out[ 6]:1'bz; assign port_a[ 7] = port_a_dir[ 7] ? mux_port_a_out[ 7]:1'bz; assign port_a[ 8] = port_a_dir[ 8] ? mux_port_a_out[ 8]:1'bz; assign port_a[ 9] = port_a_dir[ 9] ? mux_port_a_out[ 9]:1'bz; assign port_a[10] = port_a_dir[10] ? mux_port_a_out[10]:1'bz; assign port_a[11] = port_a_dir[11] ? mux_port_a_out[11]:1'bz; assign port_a[12] = port_a_dir[12] ? mux_port_a_out[12]:1'bz; assign port_a[13] = port_a_dir[13] ? mux_port_a_out[13]:1'bz; assign port_a[14] = port_a_dir[14] ? mux_port_a_out[14]:1'bz; assign port_a[15] = port_a_dir[15] ? mux_port_a_out[15]:1'bz; assign port_a[16] = port_a_dir[16] ? mux_port_a_out[16]:1'bz; assign port_a[17] = port_a_dir[17] ? mux_port_a_out[17]:1'bz; assign port_a[18] = port_a_dir[18] ? mux_port_a_out[18]:1'bz; assign port_a[19] = port_a_dir[19] ? mux_port_a_out[19]:1'bz; assign port_a[20] = port_a_dir[20] ? mux_port_a_out[20]:1'bz; assign port_a[21] = port_a_dir[21] ? mux_port_a_out[21]:1'bz; assign port_a[22] = port_a_dir[22] ? mux_port_a_out[22]:1'bz; assign port_a[23] = port_a_dir[23] ? mux_port_a_out[23]:1'bz; assign port_a[24] = port_a_dir[24] ? mux_port_a_out[24]:1'bz; assign port_a[25] = port_a_dir[25] ? mux_port_a_out[25]:1'bz; assign port_a[26] = port_a_dir[26] ? mux_port_a_out[26]:1'bz; assign port_a[27] = port_a_dir[27] ? mux_port_a_out[27]:1'bz; assign port_a[28] = port_a_dir[28] ? mux_port_a_out[28]:1'bz; assign port_a[29] = port_a_dir[29] ? mux_port_a_out[29]:1'bz; assign port_a[30] = port_a_dir[30] ? mux_port_a_out[30]:1'bz; assign port_a[31] = port_a_dir[31] ? mux_port_a_out[31]:1'bz; `ifdef HW_DEBUG reg [15:0] baud_acc; reg [7:0] second_div; reg [95:0] debug_sr; reg [9:0] debug_cnt; assign internal_clk = debug_cnt[8]; // one tick every 512 clocks assign main_clk = baud_acc[15]; // 50 MHz clock and 115200 Baud debug debuguint( .reset_in(cog_reset_0), // main tick .clk_tx_in(main_clk), // uart transmitter tick, 1 MHz .new_cycle_in(), // assrted when we have to send the info .debug_data_in(debug_data_cog_0), .uart_tx_o(debug_data_o) ); always @(posedge clk_50) begin baud_acc <= { 1'b0, baud_acc[14:0] } + 16'd75; // 50 MHz clock and 115200 Baud end always @(posedge main_clk) begin debug_cnt <= debug_cnt + 10'd1; end always @(posedge internal_clk) begin second_div <= second_div + 8'd1; end `else assign internal_clk = clk_50; // clock for clock `endif assign cog_reset_0 = reset_cnt != 4'd3; assign leds = { pll_locked, cog_reset_0, CNT[7:1] }; always @(posedge internal_clk) if (reset_cnt != 4'd3) reset_cnt <= reset_cnt + 4'd1; wire avl_rdata_valid, avl_ready; assign hub_mem_ack_0 = (avl_rdata_valid & (hub_op_0[4:3] == 2'b11)) | ((hub_op_0[4:3] == 2'b11) & avl_ready); ACog cog0( .clk_in(internal_clk), .reset_in(cog_reset_0), .CNT_in(CNT), // debug .debug_data_o(debug_data_cog_0), .debug_data_valid_o(debug_data_valid_0), // Ports access .port_a_in(port_a_in), .port_a_o(port_a_o_0), .port_a_dir_o(port_a_dir_o_0) `ifdef WITH_PORTB ,.port_b_in(port_b_in), .port_b_o(port_b_o_0), .port_b_dir_o(port_b_dir_o_0) `endif // HUB access , .hub_op_o(hub_op_0), .hub_addr_o(hub_mem_addr_0), .hub_data_in(hub_data_to_cog_0), .hub_data_rdy_o(hub_data_rdy_0), .hub_sz_o(hub_mem_size_0), .hub_data_o(hub_data_to_hub_0), .hub_ack_in(hub_mem_ack_0), .uart_tx_o(uart_tx) ); GblCounter gblcnt( .clk_in(internal_clk), .reset_in(cog_reset_0), .CNT_o(CNT) ); `ifndef SIMULATOR ddr3_ctrl ( .pll_ref_clk(clk_50), // pll_ref_clk.clk .global_reset_n(!cog_reset_0), // global_reset.reset_n .soft_reset_n(!cog_reset_0), // soft_reset.reset_n .afi_clk(), // afi_clk.clk .afi_half_clk(), // afi_half_clk.clk .afi_reset_n(), // afi_reset.reset_n .afi_reset_export_n(), // afi_reset_export.reset_n .mem_a(mem_a), // memory.mem_a .mem_ba(mem_ba), // .mem_ba .mem_ck(mem_ck), // .mem_ck .mem_ck_n(mem_ck_n), // .mem_ck_n .mem_cke(mem_cke), // .mem_cke .mem_cs_n(mem_cs_n), // .mem_cs_n .mem_dm(mem_dm), // .mem_dm .mem_ras_n(mem_ras_n), // .mem_ras_n .mem_cas_n(mem_cas_n), // .mem_cas_n .mem_we_n(mem_we_n), // .mem_we_n .mem_reset_n(mem_reset_n), // .mem_reset_n .mem_dq(mem_dq), // .mem_dq .mem_dqs(mem_dqs), // .mem_dqs .mem_dqs_n(mem_dqs_n), // .mem_dqs_n .mem_odt(mem_odt), // .mem_odt .avl_ready_0(avl_ready), // avl_0.waitrequest_n .avl_burstbegin_0(1'b0), // .beginbursttransfer .avl_addr_0(hub_mem_addr_0), // .address .avl_rdata_valid_0(avl_rdata_valid), // .readdatavalid .avl_rdata_0(hub_data_to_cog_0), // .readdata .avl_wdata_0(hub_data_to_hub_0), // .writedata .avl_be_0(4'hf), // .byteenable .avl_read_req_0(hub_op_0[4:3] == 2'b11), // .read .avl_write_req_0(hub_op_0[4:3] == 2'b10), // .write .avl_size_0(3'h1), // .burstcount .mp_cmd_clk_0_clk(clk_50), // mp_cmd_clk_0.clk .mp_cmd_reset_n_0_reset_n(!cog_reset_0), // mp_cmd_reset_n_0.reset_n .mp_rfifo_clk_0_clk(clk_50), // mp_rfifo_clk_0.clk .mp_rfifo_reset_n_0_reset_n(!cog_reset_0), // mp_rfifo_reset_n_0.reset_n .mp_wfifo_clk_0_clk(clk_50), // mp_wfifo_clk_0.clk .mp_wfifo_reset_n_0_reset_n(!cog_reset_0), // mp_wfifo_reset_n_0.reset_n .local_init_done(), // status.local_init_done .local_cal_success(), // .local_cal_success .local_cal_fail(), // .local_cal_fail .oct_rzqin(oct_rzqin), // oct.rzqin .pll_mem_clk(), // pll_sharing.pll_mem_clk .pll_write_clk(), // .pll_write_clk .pll_locked(pll_locked), // .pll_locked .pll_write_clk_pre_phy_clk(), // .pll_write_clk_pre_phy_clk .pll_addr_cmd_clk(), // .pll_addr_cmd_clk .pll_avl_clk(), // .pll_avl_clk .pll_config_clk(), // .pll_config_clk .pll_dr_clk(), // .pll_dr_clk .pll_dr_clk_pre_phy_clk(), // .pll_dr_clk_pre_phy_clk .pll_mem_phy_clk(), // .pll_mem_phy_clk .afi_phy_clk(), // .afi_phy_clk .pll_avl_phy_clk() // .pll_avl_phy_clk ); `else assign avl_rdata_valid = hub_op_0[4:3] == 2'b11; `endif initial begin reset_cnt = 0; `ifdef HW_DEBUG baud_acc = 0; second_div = 0; debug_sr = 0; debug_cnt = 0; `endif end endmodulehier the cog's code. VSCL is used as UART interface
equ1 = 500 equ2 = 700 mov DIRA, #$3 mov $1ef, #0 ; end boot mode call outcrlf testloop mov waddr, addr0 mov wvalue, startvalue0 mov scnt, CNT ; start time rdlong wvalue, waddr mov ecnt, CNT mov VSCL, #87 ; W call outuart call outaddr mov waddr, addr1 mov wvalue, startvalue1 mov scnt, CNT ; start time rdlong wvalue, waddr mov ecnt, CNT mov VSCL, #87 ; W call outuart call outaddr mov waddr, addr2 mov wvalue, startvalue2 mov scnt, CNT ; start time rdlong wvalue, waddr mov ecnt, CNT mov VSCL, #87 ; W call outuart call outaddr mov waddr, addr0 mov scnt, CNT ; start time rdlong wvalue, waddr mov ecnt, CNT mov VSCL, #87 ; W call outuart call outaddr mov waddr, addr1 mov scnt, CNT ; start time rdlong wvalue, waddr mov ecnt, CNT mov VSCL, #82 ; R call outuart call outaddr mov waddr, addr2 mov scnt, CNT ; start time rdlong wvalue, waddr mov ecnt, CNT mov VSCL, #82 ; W call outuart call outaddr lbl1 mov OUTA, #$2 mov OUTA #$1 jmp #lbl1 ; outs a string of 8 hex chars from hexvalue hex8 mov temp, hexvalue mov digitcnt, #8 hexd mov dtemp, temp shl temp, #4 shr dtemp, #28 cmp dtemp, #10 wc add dtemp, #48 if_nc add dtemp, #7 ; for values > 9 mov VSCL, dtemp hexdl mov VSCL,VSCL nr wz if_nz jmp #hexdl djnz digitcnt, #hexd hex8_ret ret ; outputs crlf outcrlf mov VSCL, #13 outcrlf1 mov VSCL, VSCL nr wz if_nz jmp #outcrlf1 mov VSCL, #10 outcrlf2 mov VSCL, VSCL nr wz if_nz jmp #outcrlf2 outcrlf_ret ret ; outputs a space outsp mov VSCL, #32 outsp1 mov VSCL, VSCL nr wz if_nz jmp #outsp1 outsp_ret ret ; ouputs a formatted address/value : [01234567]=BABEFACE 01234567 89ABCDEF outaddr call outsp mov VSCL, #91 ; [ call outuart mov hexvalue, waddr call hex8 mov VSCL, #93 ; ] call outuart mov VSCL, #61 ' = mov hexvalue, wvalue call hex8 call outsp mov hexvalue, scnt call hex8 call outsp mov hexvalue, ecnt call outcrlf outaddr_ret ret outuart mov VSCL, VSCL nr wz if_nz jmp #outuart outuart_ret ret waddr long 0 wvalue long 0 raddr long 0 rvalue long 0 startvalue0 long $BABEFACE startvalue1 long $DEADBEEF startvalue2 long $01234567 scnt long 0 ecnt long 0 addr0 long $40 addr1 long $1120 addr2 long $2360 digitcnt long 0 dtemp long 0 temp long 0 hexvalue long 0 long 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 long 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 long 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 long 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 long 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 long 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 long 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 long 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 long 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 long 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 long 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 long 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 long 0,0,0,0,0 jj jmp #0So instead of DDR3 I am starting with the DE-0 nano with regular SDRAM. Was looking at it last night and even there the memory timing is tight. To fit it in with standard hub read timing once locked to the hub, you need to be able to latch the result back into a COG register within 5 COG clocks of latching the first read S parameter of say RDLONG D,S which is holding the address. In the case of SDRAM that becomes 10 clocks as I plan to run the SDRAM at 2x the Prop clock (eg. 144MHz and 72Mhz). In theory it should just fit because the SDRAM on that board will return the last of the 32 bits on the 7th clock after the ACTIVE command is issued. There might be one or two registers in the pipeline that delay it further but I have 3 memory clocks left.
In your case with DDR3 if you run your memory at say 3x the Prop clock (100MHz COGs, 300MHz memory) you'll have 15 DDR3 clocks to get the results back in time. I think that is beyond what the DDR3 on the BeMicroCV can do as according to the Micron device datasheet its CAS latency is 9 and its tRCD is 9 so your first 16 bit result won't be back until 18 clocks after you issue the read. I think you might want to look at running the device at 4x the Prop (75MHz COGs, 300MHz memory) to give yourself 20 memory clocks for returning the result in time. It's either that or add more M4 wait states to the external memory reads, which is possible but not as nice for performance/timing compatibility with existing code.
Looking forward to hearing how it goes.
Success ! I was able to re-enter the pins in the pin planner and it compiled !
Now, I have to write a simple mem test and see what happens
I think it is time I read the hard memory controller manual
Yes, and that is what I am using... or trying anyways... I think that the problem lays in the pll/reset part... I am reading some docs but haven't found what I need yet, this is not for beginners like me... there is only experiment..
As I said above, it uses my version of a cog, not chip's one
log file
Is this one working?
It is a NIOS-II with DDR3 example.
This one, the one I posted before is quite nice because it is not only a working DDR3 example, it also has SignalTap, here : http://www.alterawiki.com/uploads/1/1c/QuartusII_projects.zip
I hope I can make some progress...
in c:\altera\15.0\nios2eds\nios2_command_shell
and from there the nios2-terminal.
- Compile the example
- Load it to the BeMicroCV A2
- run nios2-terminal
- Enjoy blinking LEDs
This is a QSys example. It would be great if
- we could build our own QSys components, a P1V would be great, the HUB would be DDR3 DRAM, no idea how much time from request to data you get, but if we have 8 80 MHz cycles (at least), that means 100 ns, we should be getting data in this time, I think.
It would mean an avalon-enabled P1V, whatever that exactly means (I have a only rough idea that is improving while reading the document posted below).
Happy coding !
Link to a couple of useful pdfs: JTAG-UART
Using Terminals
I found it ! Making QSys components it wasn't that difficult to find...
Your efforts will pay off nicely if you can get it going.
I've been working on similar concept but with regular SDRAM, not DDR3 in order to keep things simpler. On paper at least I found you can just about manage when running memory at 144MHz (eg. for SDRAM on DE0-Nano or BeMicro MAX10) and the P1V at 72MHz to get the data back in a single hub cycle for one or two COGs. The problem is you basically don't know the memory address to read/write until near the end of the second P1V CPU cycle, really just leaving you with 4 more CPU clocks before your data needs to be ready to latch if use the regular ALU path (or possibly 5 if you bypass it). So latency will likely become the major issue depending on propagation delay and DRAM/FPGA setup time etc. I also think there is a possibility of inserting an extra M4 cycle to give you one extra clock (ie. 8 clocks instead of 7 in the hub) and still fit in the existing overall hub cycle timing window which would be ideal for LMM and other deterministic applications.
So far I've started planning some Verilog state machine stuff to try to do this but still have to figure out a few more things like SDRAM initialization and FPGA clock generation etc which will no doubt take me a while to nail it all.
I'll have to keep an eye out if you get your Nios approach working with the P1V. It's likely there's a Nios driver for regular SDRAM lying about somewhere so it might still be an option for me on the DE0-Nano if it can access external memory in a single hub cycle otherwise I'm still gonna have to do my own thing. I'm reasonably confident what we want is doable in principle, it's just a matter of time and effort and also whether the board layout and FPGA timing allows the maximum memory speed to be attained.
Cheers,
rogloh
I am trying to understand the trace given by signal tap... and I really seem to be missing something: The problem is I don't really know how this whole "bus traffic generator" works.
I see that writes are issued and reads... but I do not see that the reads get the written data...
It would be great if someone would also help a bit here
Here the avalon handbook
I wish I could offer help, but I just came to the FPGA party. (Very recently, I bought a BeMicroCV, a BeMicroCVA9, and a LatticeXP Bravia2). I have loaded the Propeller 1V into the BeMicroCV and am interested in using it with the DDR3 Ram. I see that you have been working on this for quite some time.
I do suspect that the DDR3 ram will demand quite a bit more power. Has that been taken care of?
For now, I should just follow along and learn. At least I do understand most of what you are trying to do. I am not very clear on Ale's Cog image which included a hardware UART. So I will poke around and try to catch up.
It does seem that your approach of clocking the DDR3 at 300mHz is conservative. I have read that some of the Cyclone V E devices might clock 400mHz and all do at least 333mHz. Maybe I missed something.
I'll be quiet and just lurk.
The 2 examples that I found also use 300 MHz, the BeMicro handbook says somewhere so. Maybe it can be pushed to 400 no idea, but for testing I'd go even lower if necessary. My first step is to replace the traffic generator with somethig i made myself my cog or p1v or something, and to measure turnaroud times, those times will tell what can we expect from this memory pool, I think.
If it is too slow, what I expect, it means that we may have to make a dummy read, and then we get say 2 longs of data or four, and lock it to a specific cog. One can configure up to 6 bus masters so the arbitration is not in your hands but that will need a bit of work.