Does the company that did the synthesis offer any support as to the blob they created?
This is like sending very detailed specs to someone for a black box that has an input and output. You hook it up and it doesn't work and then you get to guess what they did in the black box and how they implemented your specifications. I'm sure there's more concrete engineering behind it but that's my management view
The synthesis guy has been helping Chip. The problem is that Chip built the silicon on a process that is different than originally specified. The transistors have a lower turn on potential, so they saturate quicker and leak more. It's analogous to designing a part with tolerances for 4140 steel, then making it from 6061 aluminum. The thermal growth properties are considered in the original design and thus aluminum wouldn't be suitable.
Another concrete example in the vein of the above, car pistons. They used to make pistons from cast iron (ford flathead) and now they make them from aluminum typically. If you tried to put an aluminum piston in an application where cast iron was used, without compensating for the material differences, you'd end up with a seized piston.
The chip was designed for one process and tolerances, but manufactured in a different process that has different tolerances, now they are trying to cheat it to work.
Testing this morning at lower voltages had mixed results, none very promising. Dave's heading up to my place with the test chips now. Maybe this evening we'll have a better idea about what the problems are.
Can you run some numbers on the process you used (see KeithE's comments above) to see if you do have any predicted windows of operation, with Vcc and Temperature ?
Aggressive (aka paranoid) decoupling, and a power plane Board are probably dictated, as any Vcc noise is more variables.
Aggressive (aka paranoid) decoupling, and a power plane Board are probably dictated, as any Vcc noise is more variables.
Ditto that. I know Beau designed in as much capacitance as practical in the chip itself, but I'm still a big fan of paranoid decoupling. (An excellent way to say it, btw.)
Can you run some numbers on the process you used (see KeithE's comments above) to see if you do have any predicted windows of operation, with Vcc and Temperature ?
Aggressive (aka paranoid) decoupling, and a power plane Board are probably dictated, as any Vcc noise is more variables.
I will be running some simulations of our level shifters using OnSemi's SPICE models. They may show something. It seems that the level shifters may not be working right, as some pin states don't clear during reset.
Have you double checked that OnSemi can't dial in their process to match TSMC? I thought that this was common, but I don't deal at this level. I just know that we tapeout to multiple foundries in parallel since our customers require it.
For our volumes, they will only do their standard processes.
I really feel sorry for you guys. It is bad enough for us anxiously waiting.
So what else can be tested with this chip...
Can you probe and test the i/o - analog, 75 ohm impedance, etc
Can you test the fuses - last test did not work properly IIRC
Can you test probe m/c get to cog ram to inject an instruction or some other small tests
Any other block test;s possible
Can you mod the RC osc freq or inject
Back to the hold problem...
Seems to me that a faster clock would reduce the hold requirement, rather thanlower freq. So can your test m/c inject the clock?
It is done when it is done. Parallax is sharing what happens with us. The goal is to get it produced right as soon as possible and with the least cost possible.
They will explore this chip and learn what they can, then a decision will be made on how best to continue.
I think the goal is to get it done as reasonably as possible. There isn't a commit date or anything like that out there, however you can follow the journey and run the FPGA emulation along the way.
There is no goal date if that's what you mean. The goal of this effort is to produce a working chip that behaves the way it was designed and behaves reliably. The design is in the process of being documented and these documents are mostly scattered throughout this thread. Obviously, it's to Parallax's benefit if the chip is available soon and there were hopes of this round being successful, but that's not going to happen. Things get learned each step along the way. Think of this as being a process oriented effort rather than a goal oriented one.
I will be running some simulations of our level shifters using OnSemi's SPICE models. They may show something. It seems that the level shifters may not be working right, as some pin states don't clear during reset.
The level shifter circuit you posted before should be quite supply skew tolerant, up to a point.
At some level, the S-R steering current shunts, will not be able to overcome the buffer-drive, and snap-action will stop.
So you may need to lower VccIO (3v3), as Core Vcc falls ?
I'm really sorry to hear that this time it didn't quite work out as you expected Chip.
Things go wrong all the time and the true measure of a person/company is how they resolve them.
I for one have the utmost faith that you guys will succeed.
The forum members that have access to the FPGA synthesis should get together now and try to make a more concerted effort to fully test what we have already.
I know I drifted away from the P2 FPGA as I had lots of P1 jobs to get out the door, it would be interesting to know what other forum members are doing development wise....
So if you put the electron probe on where the IOs come out of the core, they would show the same exact problem? Why does the CS pin blink with the right timing but not anything else? Do only certain pins not work? If all of the pins were toggled at the same time by a reps instruction, would they all toggle or only some of them? Or would the xor instruction only work sometimes or the reps fail and only toggle the pins once? Do the hold time problems only happen in a few places but enough to make the chip not work, or are they everywhere?
Also, when it works, we don't want a video of Chip playing Space Invaders on it while wearing a propeller hat. We want a video of 8 people, including Chip, playing 8 instances of space invaders, 1 on each cog, all wearing propeller hats.
The chaos of summertime and the imminence of the actual chip dissuaded me from emulating the P2 on an FPGA. Given the developments of the past two days and considering all that others have done (including Ozpropdev and Coley) I'm going to break out the Terasic board and get busy.
The forum members that have access to the FPGA synthesis should get together now and try to make a more concerted effort to fully test what we have already.
I know I drifted away from the P2 FPGA as I had lots of P1 jobs to get out the door, it would be interesting to know what other forum members are doing development wise....
I've been working on P1 projects recently but will likely get back to using my FPGA board sometime soon. I'd love to try running the Space Invaders program on my DE0-Nano. I haven't ever tried that since I got the add-on board from Parallax. All of my recent work has been with the DE2-115 board.
I got my DE2 fired up this week. Back to work troubleshooting the TV driver tearing nobody has sorted out yet. I did run and tinker with the space invaders game. Great piece of code to work with.
During the summer, I helped a friend do some P1 IR related things and learned a lot. Had some career / life issues to get sorted along with a contract project. Just got busy.
Oh, and a cat knocked coffee onto the Mac. I'm thinking this is the weekend to take it all apart and either fix the keyboard, or get a new one. Gotta hate that. So I've had no Mac for a while. That was a bummer too! I had Prop-GCC building and was using it a little with the DE2...
Just want to express my appreciation for the open-ness and this look into the process. From my first experience with the Basic Stamp many years ago, I've noticed a unique pattern: The actual performance of a Parallax product will not just meet, but exceed the claims. This thread is no exception.
BTW, I remember reading about the proposed package for the P2, but I can't seem to find it. I want to have my basic layout library ready when P2 arrives!
My proposal for Propeller II test is to start with controllable environment. Liquid Nitrogen seems like a good idea but in reality it's a brute force method (best left for the last resort) - I would start with using the fan-cooled peltier (Thermoelectric Cooler) so the Propeller II die temperature can be directly controlled, to the point the entire processor starts to operate correctly. Two more thing I would toss in: obtain a digital power supply for CPU (COG) voltage rail (1.8 V) and pin IO voltage rail (3.3 V), and frequency generator so we can start at 1.8 Volts then turn it down until it quit working (with brownout reset disabled, of course), with a way to change CPU core frequency and monitor the current consumption at given core voltage.
Hot chip that sucks lot more power? Sure sounds like it could be able to operate at over 200 MHz. Who knows? First, getting the Propeller II to boot up is the priority.
Dave and I spent time today going over the Prop2 chips some more. We tried running at lower voltages and with high and low temperatures, to no avail.
There are at least three certain problems, it seems:
1) The chip takes way to much quiescent current: 44mA at 1.8V with no clocking. This indicates a likely short somewhere. Where the boundary of the synthesized block meets our frame, things were a little hairy, and I suspect something got shorted. We could only perform LVS (layout-versus-schematic) tests on our frame, and not the synthesized block, so we couldn't do a whole-chip LVS, which would have caught something like that.
2) In the Verilog code, I handled the reset signal as:
always @(posedge clk or negedge res_n)
if (!res_n)
q <= 1'b0;
else
q <= d;
I've read from a few sources that this is how to generate a D flip-flop with an asynchronous 'clear' input. It turns out that this is behaving from power-up such that there is no clear, since res_n is low, as power rises, causing no negative edge event, and clk is still the whole time. This problem is apparent by the I/O pins not going high-z until clk begins cycling while internal res_n is still low, but after the RESn pin is raised. This is a headache, but not a showstopper. I need to figure out how to recode these flip-flops so that we wind up with a truly asynchronous reset, not one that must be practically correlated with clk.
3) Cog0 seems to launch, and executes part of the booter program from hub ROM, but it acts erratically. I don't know if this is due to hold-time issues resulting from a process change, or perhaps our memories are not working as designed, due to the process change.
I think the way forward is to resynthesize (we really need asynchronous reset behavior) using OnSemi's cells, since OnSemi is going to be a lot less expensive than TSMC, going forward, and their turn-around is quicker. This is probably going to take $100k and 4 months to get straightened out. What a bummer!
This is probably going to take $100k and 4 months to get straightened out. What a bummer!
Sorry to hear that Chip.
Time to buy up big on Prop1 guys to help fund this. (Echoing Heater's comment)
I'm super excited about Prop2 and upgrading to a DE2 board to get going on some new (crazy?) ideas.
Keep up the good work!
I'm not sure. The synthesis guys didn't know what to do about it, either. The problem is that we have two different design methods in use and they don't play together well. We did all the checking we could do through automation, and then Beau did a visual check around the frame/cell interface. Whatever short(s) exist(s), they are probably subtle to the eye.
I did some checking on the internet and the way I did the async flop clearing is supposed to be proper, but it doesn't seem to work in practice. It feels like there are too many things out of our control. What circuitry we could test in the frame (BOEn and RESn) worked perfectly, but things relating to the synthesized block were a mess.
The short check was not visual. I was able to flatten the core and remove all active circuitry within the core, which means only the transistors were removed leaving ONLY metal. From there I was able to do a complete LVS with all of our stuff included. If there was a short of the nature suggested, it would have been caught.
Edit: There were two kinds of short checks that were done with regard to the core.
1) Yes, a visual check was done against the entire chip to make sure that Power and Ground were not shorting. This involves selecting one node or the other (Power or Ground) and making sure that the selected opposite remains un-highlighted. Initially this test was missed in the May tapeout because the Core block was not included. (See Note1 below)
2) The second check involved flattening all of the core cells and removing only Poly and Diffusion (Gate, Source, and Drain) within the Core that makeup a transistor. Assuming that the Core logic passed LVS with the synthesis team, this procedure on our part allowed us to LVS the entire block with reasonable certainty with the Core metal in place that we were not creating a short or other signal problem.
Note1: In Check #1 mentioned above, it was originally done without the Core in place, this is why it was missed during the May tapeout. Poly (Gate material) was causing a short between power and ground within the Core which was removed in step #2 so therefore did not show itself. The over site of not including the Core in Check #1 was my fault and was a poor decision. The difference at the time was the amount of time required for the test.... 7 Hours with the Core present, versus 2 hours with the Core removed.
Note2: The hand-off between the Synthesized block and our circuitry was actually par with industry standard for the most part. We provided all of the signal IO's including Power and Ground which abutted to a Core perimeter that we specified with proper DRC padding. The synthesis team connected all of the signal lines, but failed to connect any of the Power or Ground lines. This procedure was odd, and the Power and Ground should have been part of their script. Since it was not, or since it was missed, I manually traversed the perimeter and connected all of the Power and Ground into the Core. If there were any Problems in connection, then Step #1 and Step #2 would have caught the discrepancy.
Comments
This is like sending very detailed specs to someone for a black box that has an input and output. You hook it up and it doesn't work and then you get to guess what they did in the black box and how they implemented your specifications. I'm sure there's more concrete engineering behind it but that's my management view
Thanks for sharing the process. It's fascinating!
Another concrete example in the vein of the above, car pistons. They used to make pistons from cast iron (ford flathead) and now they make them from aluminum typically. If you tried to put an aluminum piston in an application where cast iron was used, without compensating for the material differences, you'd end up with a seized piston.
The chip was designed for one process and tolerances, but manufactured in a different process that has different tolerances, now they are trying to cheat it to work.
Can you run some numbers on the process you used (see KeithE's comments above) to see if you do have any predicted windows of operation, with Vcc and Temperature ?
Aggressive (aka paranoid) decoupling, and a power plane Board are probably dictated, as any Vcc noise is more variables.
Ditto that. I know Beau designed in as much capacitance as practical in the chip itself, but I'm still a big fan of paranoid decoupling. (An excellent way to say it, btw.)
I will be running some simulations of our level shifters using OnSemi's SPICE models. They may show something. It seems that the level shifters may not be working right, as some pin states don't clear during reset.
For our volumes, they will only do their standard processes.
Dang. Good luck with the spice sims. I've got to deal with some of our own PVT problems...
So what else can be tested with this chip...
- Can you probe and test the i/o - analog, 75 ohm impedance, etc
- Can you test the fuses - last test did not work properly IIRC
- Can you test probe m/c get to cog ram to inject an instruction or some other small tests
- Any other block test;s possible
- Can you mod the RC osc freq or inject
Back to the hold problem...Seems to me that a faster clock would reduce the hold requirement, rather thanlower freq. So can your test m/c inject the clock?
And of course, when is the next TMSC shuttle run?
They will explore this chip and learn what they can, then a decision will be made on how best to continue.
I think the goal is to get it done as reasonably as possible. There isn't a commit date or anything like that out there, however you can follow the journey and run the FPGA emulation along the way.
The level shifter circuit you posted before should be quite supply skew tolerant, up to a point.
At some level, the S-R steering current shunts, will not be able to overcome the buffer-drive, and snap-action will stop.
So you may need to lower VccIO (3v3), as Core Vcc falls ?
Things go wrong all the time and the true measure of a person/company is how they resolve them.
I for one have the utmost faith that you guys will succeed.
The forum members that have access to the FPGA synthesis should get together now and try to make a more concerted effort to fully test what we have already.
I know I drifted away from the P2 FPGA as I had lots of P1 jobs to get out the door, it would be interesting to know what other forum members are doing development wise....
Also, when it works, we don't want a video of Chip playing Space Invaders on it while wearing a propeller hat. We want a video of 8 people, including Chip, playing 8 instances of space invaders, 1 on each cog, all wearing propeller hats.
Also finish up all those Prop 1 project ideas. You will need to buy some Prop 1's of course to help finance the next shuttle run.
I've been working on P1 projects recently but will likely get back to using my FPGA board sometime soon. I'd love to try running the Space Invaders program on my DE0-Nano. I haven't ever tried that since I got the add-on board from Parallax. All of my recent work has been with the DE2-115 board.
During the summer, I helped a friend do some P1 IR related things and learned a lot. Had some career / life issues to get sorted along with a contract project. Just got busy.
Oh, and a cat knocked coffee onto the Mac. I'm thinking this is the weekend to take it all apart and either fix the keyboard, or get a new one. Gotta hate that. So I've had no Mac for a while. That was a bummer too! I had Prop-GCC building and was using it a little with the DE2...
Hell, even if it WORKED it would be worth dipping it in liquid nitrogen. Because LIQUID NITROGEN.
BTW, I remember reading about the proposed package for the P2, but I can't seem to find it. I want to have my basic layout library ready when P2 arrives!
Hot chip that sucks lot more power? Sure sounds like it could be able to operate at over 200 MHz. Who knows? First, getting the Propeller II to boot up is the priority.
There are at least three certain problems, it seems:
1) The chip takes way to much quiescent current: 44mA at 1.8V with no clocking. This indicates a likely short somewhere. Where the boundary of the synthesized block meets our frame, things were a little hairy, and I suspect something got shorted. We could only perform LVS (layout-versus-schematic) tests on our frame, and not the synthesized block, so we couldn't do a whole-chip LVS, which would have caught something like that.
2) In the Verilog code, I handled the reset signal as:
I've read from a few sources that this is how to generate a D flip-flop with an asynchronous 'clear' input. It turns out that this is behaving from power-up such that there is no clear, since res_n is low, as power rises, causing no negative edge event, and clk is still the whole time. This problem is apparent by the I/O pins not going high-z until clk begins cycling while internal res_n is still low, but after the RESn pin is raised. This is a headache, but not a showstopper. I need to figure out how to recode these flip-flops so that we wind up with a truly asynchronous reset, not one that must be practically correlated with clk.
3) Cog0 seems to launch, and executes part of the booter program from hub ROM, but it acts erratically. I don't know if this is due to hold-time issues resulting from a process change, or perhaps our memories are not working as designed, due to the process change.
I think the way forward is to resynthesize (we really need asynchronous reset behavior) using OnSemi's cells, since OnSemi is going to be a lot less expensive than TSMC, going forward, and their turn-around is quicker. This is probably going to take $100k and 4 months to get straightened out. What a bummer!
I know you guys will get it!
I do like the idea of synthesizing for the other companies process, if it means lower costs and greater speed! : ]
Keep up the good work Chip, Dave and the rest of the team over at Parallax!
Sorry to hear that Chip.
Time to buy up big on Prop1 guys to help fund this. (Echoing Heater's comment)
I'm super excited about Prop2 and upgrading to a DE2 board to get going on some new (crazy?) ideas.
Keep up the good work!
Cheers
Brian
I'm not sure. The synthesis guys didn't know what to do about it, either. The problem is that we have two different design methods in use and they don't play together well. We did all the checking we could do through automation, and then Beau did a visual check around the frame/cell interface. Whatever short(s) exist(s), they are probably subtle to the eye.
I did some checking on the internet and the way I did the async flop clearing is supposed to be proper, but it doesn't seem to work in practice. It feels like there are too many things out of our control. What circuitry we could test in the frame (BOEn and RESn) worked perfectly, but things relating to the synthesized block were a mess.
The short check was not visual. I was able to flatten the core and remove all active circuitry within the core, which means only the transistors were removed leaving ONLY metal. From there I was able to do a complete LVS with all of our stuff included. If there was a short of the nature suggested, it would have been caught.
Edit: There were two kinds of short checks that were done with regard to the core.
1) Yes, a visual check was done against the entire chip to make sure that Power and Ground were not shorting. This involves selecting one node or the other (Power or Ground) and making sure that the selected opposite remains un-highlighted. Initially this test was missed in the May tapeout because the Core block was not included. (See Note1 below)
2) The second check involved flattening all of the core cells and removing only Poly and Diffusion (Gate, Source, and Drain) within the Core that makeup a transistor. Assuming that the Core logic passed LVS with the synthesis team, this procedure on our part allowed us to LVS the entire block with reasonable certainty with the Core metal in place that we were not creating a short or other signal problem.
Note1: In Check #1 mentioned above, it was originally done without the Core in place, this is why it was missed during the May tapeout. Poly (Gate material) was causing a short between power and ground within the Core which was removed in step #2 so therefore did not show itself. The over site of not including the Core in Check #1 was my fault and was a poor decision. The difference at the time was the amount of time required for the test.... 7 Hours with the Core present, versus 2 hours with the Core removed.
Note2: The hand-off between the Synthesized block and our circuitry was actually par with industry standard for the most part. We provided all of the signal IO's including Power and Ground which abutted to a Core perimeter that we specified with proper DRC padding. The synthesis team connected all of the signal lines, but failed to connect any of the Power or Ground lines. This procedure was odd, and the Power and Ground should have been part of their script. Since it was not, or since it was missed, I manually traversed the perimeter and connected all of the Power and Ground into the Core. If there were any Problems in connection, then Step #1 and Step #2 would have caught the discrepancy.